Parakeet RNNT 1.1B Multilingual - Speech-to-Text Collection
Parakeet RNNT 1.1B - Speech-to-Text Collection
Overview
Automatic Speech Recognition (ASR) systems are widely used in applications such as voice command interfaces, automated subtitle generation for online media, and transcription of customer interactions for archival. Recent advances in deep learning have enabled speech-to-text models to transcribe audio to text in real time, achieving high accuracy even in adverse acoustic conditions, with robustness to noise and accents, and low word error rates (WER).
This collection provides three Parakeet RNNT 1.1B model variants: a Multilingual model (25 languages), a Prompt / Language ID model (13 languages with language identification), and an Indic model (4 Indic languages). All variants use the same 1.1 billion parameter FastConformer-RNNT architecture and are integrated within the NeMo framework, giving you one collection to choose the right variant for your use case.
In this collection
This collection provides finetunable artifacts and ready-to-use Parakeet NIM containers for all three model variants, enabling flexible deployment and customization as NVIDIA Inference Microservices (NIM). Each variant includes:
- Acoustic Model - Parakeet-RNNT-XXL (FastConformer-RNNT) base model with 1.1B parameters for audio-to-text transcription with punctuation and capitalization capability.
- Speaker Diarization Model (Sortformer) - Identifies and labels different speakers in audio (who spoke when).
- Voice Activity Detection (VAD) - Detects speech segments in audio streams.
Components can be used as provided or finetuned for domain-specific customization.
Model variants and supported languages
1. Multilingual (25 languages)
Wide language coverage for global applications. Supported locales:
- English: en-US (United States), en-GB (United Kingdom)
- Spanish: es-US (United States), es-ES (Spain)
- German: de-DE (Germany)
- French: fr-FR (France), fr-CA (Canada)
- Italian: it-IT (Italy)
- Arabic: ar-AR (Arabic)
- Japanese: ja-JP (Japan)
- Korean: ko-KR (Korea)
- Portuguese: pt-BR (Brazil), pt-PT (Portugal)
- Russian: ru-RU (Russia)
- Hindi: hi-IN (India)
- Dutch: nl-NL (Netherlands)
- Danish: da-DK (Denmark)
- Norwegian: nn-NO (Nynorsk), nb-NO (Bokmål)
- Czech: cs-CZ (Czech Republic)
- Polish: pl-PL (Poland)
- Swedish: sv-SE (Sweden)
- Thai: th-TH (Thailand)
- Turkish: tr-TR (Turkey)
- Hebrew: he-IL (Israel)
2. Prompt / Language ID (13 languages)
Optimized for scenarios where language is specified or identified via prompt. Supported locales:
- English: en-US (United States), en-GB (United Kingdom)
- Spanish: es-ES (Spain), es-US (United States)
- French: fr-FR (France)
- German: de-DE (Germany)
- Arabic: ar-AR (Arabic)
- Portuguese: pt-BR (Brazil)
- Italian: it-IT (Italy)
- Japanese: ja-JP (Japan)
- Korean: ko-KR (Korea)
- Russian: ru-RU (Russia)
- Hindi: hi-IN (India)
3. Indic (4 languages)
Tuned for Indic languages. Supported locales:
- Bengali: bn-IN (India)
- English: en-US (United States)
- Hindi: hi-IN (India)
- Tamil: ta-IN (India)
All variants output text with upper and lower case, punctuation, spaces, and apostrophes.
How does Parakeet RNNT work?
The Parakeet RNNT pipeline processes audio through multiple specialized components that work together to produce accurate, properly formatted transcriptions with speaker labels:
1. Voice Activity Detection (VAD)
Before processing begins, the Silero VAD model identifies segments of the audio that contain speech versus silence or background noise. This preprocessing step:
- Reduces computational overhead by filtering non-speech segments
- Improves noise robustness by focusing processing on relevant audio
- Enhances the ability to detect the beginning and end of speech segments for accurate endpointing.
2. Feature Extraction
The audio preprocessing stage converts raw analog signals into a machine-understandable format:
- Resampling: Converts audio to the appropriate sample rate
- Signal Preprocessing: Applies standardization and windowing techniques
- Spectrogram Transformation: Converts time-domain audio signals into frequency-domain representations that capture acoustic features
3. Acoustic Modeling with Parakeet-RNNT
The core of the system is the Parakeet-RNNT-XXL (FastConformer-RNNT) acoustic model, which:
- Architecture: Uses an optimized Conformer architecture with 8x depthwise-separable convolutional downsampling
- Scale: 1.1 billion parameters for superior accuracy
- RNNT Decoder: Employs Recurrent Neural Network Transducer with Hybrid loss for improved performance
- Tokenizer: Uses a merged tokenizer supporting the languages of each variant
- Input Processing: Accepts spectrogram features and processes them through self-attention and convolution layers
- Output: Produces text with built-in punctuation and capitalization
The FastConformer architecture combines the strengths of Transformers (global context via self-attention) with CNNs (local feature extraction), achieving superior accuracy with improved efficiency through linearly scalable attention mechanisms.
Key Differences from CTC:
- RNNT models use a prediction network that conditions outputs on previously emitted tokens
- Better handling of long-form dependencies and contextual information
- Native support for punctuation and capitalization in the acoustic model output
- More accurate transcription with fewer post-processing steps
4. Speaker Diarization with Sortformer
The Sortformer diarization model identifies and labels different speakers in multi-speaker audio:
- Speaker Segmentation: Determines when different speakers are talking
- Speaker Labeling: Assigns consistent labels to each speaker throughout the audio
- Streaming Support: Provides real-time speaker diarization with low latency
- Multi-speaker Support: Handles up to 4 speakers simultaneously
- Multilingual Support: Works with the languages supported by each model variant
Sortformer uses an innovative arrival-time ordering approach to resolve speaker permutation problems, making it particularly effective for conversational audio, meetings, and interviews. The model employs a FastConformer encoder with Transformer layers trained with Sort-Loss and Permutation Invariant Loss objectives.
Where to Get Started
Prerequisites
Runtime Environment:
- Riva 2.19.0 or higher
- Linux operating system
- NVIDIA GPU (H100, A100, or L40 recommended)
Development Tools:
- NeMo Framework for model finetuning
- NGC CLI for downloading artifacts
- Docker for containerized deployment
Deployment Workflow
-
Download the artifacts: Use NGC CLI to download the artifacts for your chosen variant (Multilingual, Prompt/Language ID, or Indic) from this collection
ngc registry model download-version <artifact> -
Finetune for Your Domain (Optional): Customize models for your specific use case
-
Optimize and Deploy: Package customized models into a NIM
- Use Riva build and deploy flow to convert and optimize models
- Deploy as a microservice with Riva NIMs
- Scale horizontally for production workloads
References
Academic Papers
- Fast Conformer with Linearly Scalable Attention for Efficient Speech Recognition
- Conformer: Convolution-augmented Transformer for Speech Recognition
Software & Tools
- NeMo Framework - Training and finetuning toolkit
- Riva - Deployment and inference platform
Ethical Considerations
NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.
Terms of Use
By using this software or microservice, you are agreeing to the terms and conditions of the license and acceptable use policy.
Governing Terms
This trial is governed by the NVIDIA API Trial Terms of Service (found at https://assets.ngc.nvidia.com/products/api-catalog/legal/NVIDIA%20API%20Trial%20Terms%20of%20Service.pdf). The use of this model is governed by the AI Foundation Models Community License Agreement.
You are responsible for ensuring that your use of NVIDIA AI Foundation Models complies with all applicable laws.
Disclaimer
AI models generate responses and outputs based on complex algorithms and machine learning techniques, and those responses or outputs may be inaccurate or indecent. By testing this model, you assume the risk of any harm caused by any response or output of the model. Please do not upload any confidential information or personal data. Your use is logged for security.
