NeMo-Speech.cpp

NeMo-Speech.cpp is a lightweight C++ inference runtime for NVIDIA speech models. This multi-architecture container serves automatic speech recognition (ASR), speaker-attributed ASR, and text-to-speech (TTS) through HTTP and Riva-compatible gRPC APIs, without requiring Triton or a Python runtime.

The container supports linux/amd64 and linux/arm64. Its default entry point is nemo-speech, which runs the HTTP server with the serve command; the separate riva_server executable provides gRPC services. Models and client applications are not included.

Features

• Riva-compatible ASR and TTS gRPC APIs
• HTTP API for speech inference
• NVIDIA CUDA GPU inference
• Streaming and offline ASR
• Speaker-attributed transcription (up to eight speakers)
• CTC and cache-aware RNNT ASR models
• Greedy and Flashlight/KenLM CTC decoding
• MagpieTTS with NanoCodec speech synthesis

Requirements

• NVIDIA GPU supported by the model
• NVIDIA driver
• Docker with NVIDIA Container Toolkit
• Ready-to-run NeMo-Speech.cpp GGUF model files from Hugging Face

Download models

The container does not include models or a downloader. Ready-to-run GGUFs are published alongside the original NVIDIA checkpoints on Hugging Face. Download them on the host and mount them into the container. Install the Hugging Face CLI if needed:

pip install -U huggingface_hub
mkdir -p models

For the ASR quickstart, download Nemotron-Speech Streaming English 0.6B:

hf download nvidia/nemotron-speech-streaming-en-0.6b \
    nemotron-speech-streaming-en-0.6b.q8_0.gguf --local-dir models

# The container runs as UID/GID 1000 and must be able to read bind-mounted files.
chmod -R a+rX models

To add speaker labels to ASR, download the Nemotron 3 Diarization GGUF:

hf download nvidia/Nemotron-3-Diarization \
    Nemotron-3-Diarization.q8_0.gguf --local-dir models

# The container runs as UID/GID 1000 and must be able to read bind-mounted files.
chmod -R a+rX models

For TTS, download the prebuilt MagpieTTS v2607 and NanoCodec GGUFs. MagpieTTS also needs the matching tokenizer assets from its .nemo archive; only the archive is extracted, not converted:

mkdir -p models/magpie-tts models/nano-codec

hf download nvidia/magpie_tts_multilingual_357m \
    magpie_tts_multilingual_357m.v2607.f16.gguf \
    magpie_tts_multilingual_357m.nemo \
    --local-dir models/magpie-tts

mkdir -p models/magpie-tts/extracted
tar -xf models/magpie-tts/magpie_tts_multilingual_357m.nemo \
    -C models/magpie-tts/extracted

hf download nvidia/nemo-nano-codec-22khz-1.89kbps-21.5fps \
    nemo_nano_codec_22khz_1.89kbps_21.5fps.decoder.f16.gguf \
    --local-dir models/nano-codec

# The container runs as UID/GID 1000 and must be able to read bind-mounted files.
chmod -R a+rX models

Other ready-to-run GGUFs and model details are available on the corresponding Hugging Face pages:

• Parakeet CTC 1.1B
• Nemotron-Speech Streaming English 0.6B
• Nemotron 3.5 ASR Streaming 0.6B
• Nemotron 3 Diarization
• MagpieTTS Multilingual 357M
• NeMo NanoCodec

Mount prepared models into /models when running the container. The commands below use the 0.2.0 container tag. See the public source documentation for broader server and per-service configuration:

• ASR models
• TTS models
• Server configuration
• ASR configuration
• TTS configuration

Start an ASR server

After downloading the ASR and Nemotron 3 Diarization GGUFs into ./models, run the server. This example uses Nemotron-Speech Streaming English 0.6B for ASR:

export IMAGE=nvcr.io/nvidia/nemo-speech.cpp:0.2.0

docker run --rm \
    --runtime=nvidia \
    --gpus all \
    -p 50051:50051 \
    -v "$PWD/models:/models:ro" \
    --entrypoint /opt/nemo-speech/bin/riva_server \
    "$IMAGE" \
    --asr.model.path /models/nemotron-speech-streaming-en-0.6b.q8_0.gguf \
    --asr.diar.model_path /models/Nemotron-3-Diarization.q8_0.gguf \
    --asr.backend.gpu 0 \
    --bind 0.0.0.0:50051

The server listens for Riva gRPC requests on port 50051. When an ASR client requests diarization, results include word-level speaker labels; Nemotron 3 Diarization supports up to eight speakers. Omit --asr.diar.model_path to run ASR without diarization.

Input audio should be mono, 16-bit linear PCM WAV at 16 kHz.

Run Riva C++ clients

The standard NVIDIA Riva C++ clients work with riva_server. Follow that repository's instructions to build or install the clients on your host.

Run streaming ASR:

riva_streaming_asr_client \
    --riva_uri=localhost:50051 \
    --audio_file=/absolute/path/to/sample.wav \
    --language_code=en-US \
    --simulate_realtime=true \
    --chunk_duration_ms=160

Run offline ASR:

riva_asr_client \
    --riva_uri=localhost:50051 \
    --audio_file=/absolute/path/to/sample.wav \
    --language_code=en-US

Start a TTS server

After downloading the models and extracting the MagpieTTS tokenizer as shown above, start the server:

export IMAGE=nvcr.io/nvidia/nemo-speech.cpp:0.2.0

docker run --rm \
    --runtime=nvidia \
    --gpus all \
    -p 50051:50051 \
    -v "$PWD/models:/models:ro" \
    --entrypoint /opt/nemo-speech/bin/riva_server \
    "$IMAGE" \
    --tts.magpie-model /models/magpie-tts/magpie_tts_multilingual_357m.v2607.f16.gguf \
    --tts.codec-model /models/nano-codec/nemo_nano_codec_22khz_1.89kbps_21.5fps.decoder.f16.gguf \
    --tts.tokenizer-model-dir /models/magpie-tts/extracted

Send a TTS request with the C++ client:

riva_tts_client \
    --riva_uri=localhost:50051 \
    --language=en-US \
    --text="Hello World"

License and third-party software

NeMo-Speech.cpp is licensed under the Apache License 2.0. License information included in the container is installed under:

/opt/nemo-speech/share/licenses/nemo-speech/

Models are distributed separately and remain subject to their respective licenses and terms.