NeMo-Speech.cpp

NeMo-Speech.cpp is a lightweight C++ inference runtime for NVIDIA speech models. This multi-architecture container provides Riva-compatible gRPC services for automatic speech recognition (ASR) and text-to-speech (TTS) without requiring Triton or a Python runtime.

The container supports linux/amd64 and linux/arm64 and includes only riva_server and its required runtime libraries. Models and client applications are not included.

Features

• Riva-compatible ASR and TTS gRPC APIs
• NVIDIA CUDA GPU inference
• Streaming and offline ASR
• CTC and cache-aware RNNT ASR models
• Greedy and Flashlight/KenLM CTC decoding
• MagpieTTS with NanoCodec speech synthesis

Requirements

• NVIDIA GPU supported by the model
• NVIDIA driver
• Docker with NVIDIA Container Toolkit
• Ready-to-run NeMo-Speech.cpp GGUF model files from Hugging Face

Download models

The container does not include models. Ready-to-run GGUFs are published alongside the original NVIDIA checkpoints on Hugging Face. Install the Hugging Face CLI if needed:

pip install -U huggingface_hub
mkdir -p models

For the ASR quickstart, download Nemotron-Speech Streaming English 0.6B:

hf download nvidia/nemotron-speech-streaming-en-0.6b \
    nemotron-speech-streaming-en-0.6b.q8_0.gguf --local-dir models

# The container runs as UID/GID 1000 and must be able to read bind-mounted files.
chmod -R a+rX models

For TTS, download the prebuilt MagpieTTS and NanoCodec GGUFs. MagpieTTS also requires tokenizer assets from its original .nemo archive; the archive is extracted but is not converted:

mkdir -p models/magpie-tts models/nano-codec

hf download nvidia/magpie_tts_multilingual_357m \
    --include magpie_tts_multilingual_357m.v2602.f16.gguf \
    --include magpie_tts_multilingual_357m.nemo \
    --local-dir models/magpie-tts

mkdir -p models/magpie-tts/extracted
tar -xf models/magpie-tts/magpie_tts_multilingual_357m.nemo \
    -C models/magpie-tts/extracted

hf download nvidia/nemo-nano-codec-22khz-1.89kbps-21.5fps \
    nemo_nano_codec_22khz_1.89kbps_21.5fps.decoder.f16.gguf \
    --local-dir models/nano-codec

# The container runs as UID/GID 1000 and must be able to read bind-mounted files.
chmod -R a+rX models

Other ready-to-run GGUFs and model details are available on the corresponding Hugging Face pages:

• Parakeet CTC 1.1B
• Nemotron-Speech Streaming English 0.6B
• Nemotron 3.5 ASR Streaming 0.6B
• MagpieTTS Multilingual 357M
• NeMo NanoCodec

Mount prepared models into /models when starting the server. The commands below use the exact 0.1.0 container CLI. See the public source documentation for broader server and per-service configuration:

• ASR models
• TTS models
• Server configuration
• ASR configuration
• TTS configuration

Start an ASR server

After downloading an ASR GGUF into ./models, run the server. This example uses Nemotron-Speech Streaming English 0.6B:

export IMAGE=nvcr.io/nvidia/nemo-speech.cpp:0.1.0

docker run --rm \
    --runtime=nvidia \
    --gpus all \
    -p 50051:50051 \
    -v "$PWD/models:/models:ro" \
    "$IMAGE" \
    --asr.model.path /models/nemotron-speech-streaming-en-0.6b.q8_0.gguf \
    --asr.backend.gpu 0 \
    --bind 0.0.0.0:50051

The server listens for Riva gRPC requests on port 50051.

Input audio should be mono, 16-bit linear PCM WAV at 16 kHz.

Run Riva C++ clients

The standard NVIDIA Riva C++ clients work with riva_server. Follow that repository's instructions to build or install the clients.

Run streaming ASR:

riva_streaming_asr_client \
    --riva_uri=localhost:50051 \
    --audio_file=/absolute/path/to/sample.wav \
    --language_code=en-US \
    --simulate_realtime=true \
    --chunk_duration_ms=160

Run offline ASR:

riva_asr_client \
    --riva_uri=localhost:50051 \
    --audio_file=/absolute/path/to/sample.wav \
    --language_code=en-US

Start a TTS server

After downloading the MagpieTTS GGUF, NanoCodec decoder GGUF, and MagpieTTS tokenizer archive as shown above, run:

export IMAGE=nvcr.io/nvidia/nemo-speech.cpp:0.1.0

docker run --rm \
    --runtime=nvidia \
    --gpus all \
    -p 50051:50051 \
    -v "$PWD/models:/models:ro" \
    "$IMAGE" \
    --tts.magpie-model /models/magpie-tts/magpie_tts_multilingual_357m.v2602.f16.gguf \
    --tts.codec-model /models/nano-codec/nemo_nano_codec_22khz_1.89kbps_21.5fps.decoder.f16.gguf \
    --tts.tokenizer-model-dir /models/magpie-tts/extracted \
    --tts.language-code en-US \
    --tts.threads 8 \
    --bind 0.0.0.0:50051

Send a TTS request with the C++ client:

riva_tts_client \
    --riva_uri=localhost:50051 \
    --language=en-US \
    --text="Hello World"

License and third-party software

NeMo-Speech.cpp is licensed under the Apache License 2.0. License information included in the container is installed under:

/opt/nemo-speech/share/licenses/nemo-speech/

Models are distributed separately and remain subject to their respective licenses and terms.