Parakeet TDT 0.6B - Speech-to-Text Collection

Overview

Automatic Speech Recognition (ASR) systems are widely used in applications such as voice command interfaces, automated subtitle generation for online media, and transcription of customer interactions for archival. Recent advances in deep learning have enabled speech-to-text models to transcribe audio to text in real time, achieving high accuracy even in adverse acoustic conditions, with robustness to noise and accents, and low word error rates (WER).

This collection offers two Parakeet TDT 0.6B variants: v2 (English only) and v3 (25 European languages). Both are 600-million-parameter models with native punctuation, capitalization, and accurate word-level timestamp prediction. Integrated within the NeMo framework, they provide a highly accurate solution that can efficiently transcribe audio segments up to 24 minutes in a single pass.

In this collection

This collection provides finetunable artifacts and ready-to-use Parakeet NIM containers for both v2 and v3, enabling flexible deployment and customization as NVIDIA Inference Microservices (NIM). Each variant includes:

1. Acoustic Model - Parakeet-TDT (FastConformer-TDT) base model with 600M parameters for audio-to-text transcription with timestamps
2. Voice Activity Detection (VAD) - Detects speech segments in audio streams
3. Speaker Diarization Model (Sortformer) - Identifies and labels different speakers in audio (who spoke when)

Each component can be used as provided or further finetuned to suit custom deployment needs, enabling domain-specific customization for various industries and use cases.

Note: These models do not require separate language models, punctuation models, or ITN; they provide native punctuation, capitalization, and proper formatting directly from the acoustic model.

Model variants and supported languages

Parakeet TDT 0.6B v2 — English only

Optimized for English transcription. Use this variant when your content is English-only.

Parakeet TDT 0.6B v3 — 25 European languages

Supports: Bulgarian (bg), Croatian (hr), Czech (cs), Danish (da), Dutch (nl), English (en), Estonian (et), Finnish (fi), French (fr), German (de), Greek (el), Hungarian (hu), Italian (it), Latvian (lv), Lithuanian (lt), Maltese (mt), Polish (pl), Portuguese (pt), Romanian (ro), Slovak (sk), Slovenian (sl), Spanish (es), Swedish (sv), Russian (ru), Ukrainian (uk).

Key Features

• Word-level Timestamps: Accurate timestamp predictions at word, character, and segment levels
• Native Punctuation and Capitalization: Built-in formatting without post-processing
• Long-form Audio Support: Efficiently handles audio up to 24 minutes in a single pass
• Robust Performance: Excellent accuracy on spoken numbers and song lyrics transcription
• High Throughput: Achieves RTFx of 2940 on HF-Open-ASR leaderboard (batch size 128)

How does Parakeet TDT work?

The Parakeet TDT pipeline processes audio through specialized components that work together to produce accurate, properly formatted transcriptions with timestamps and speaker labels:

1. Voice Activity Detection (VAD)

Before processing begins, the Silero VAD model identifies segments of the audio that contain speech versus silence or background noise. This preprocessing step:

• Reduces computational overhead by filtering non-speech segments
• Improves noise robustness by focusing processing on relevant audio
• Enhances the ability to detect the beginning and end of speech segments for accurate endpointing.

2. Feature Extraction

The audio preprocessing stage converts raw analog signals into a machine-understandable format:

• Resampling: Converts audio to 16 kHz mono channel
• Signal Preprocessing: Applies standardization and windowing techniques
• Spectrogram Transformation: Converts time-domain audio signals into frequency-domain representations that capture acoustic features

3. Acoustic Modeling with Parakeet-TDT

The core of the system is the Parakeet-TDT (FastConformer-TDT) acoustic model, which:

• Architecture: FastConformer encoder with TDT (Token and Duration Transducer) decoder
• Scale: 600 million parameters; v2 optimized for English, v3 for 25 European languages
• Full Attention: Trained with full attention mechanism for superior context understanding
• Token Prediction: Jointly predicts tokens and their durations for efficient sequence transduction
• Native Formatting: Produces text with punctuation and capitalization directly
• Timestamp Generation: Outputs accurate word-level, character-level, and segment-level timestamps
• Long Context: Handles audio segments up to 24 minutes efficiently

The FastConformer architecture combines the strengths of Transformers (global context via self-attention) with CNNs (local feature extraction), achieving superior accuracy with improved efficiency through linearly scalable attention mechanisms. The TDT decoder jointly predicts tokens and durations, enabling efficient and accurate transcription.

Key Advantages of TDT:

• More efficient than traditional transducers by jointly modeling tokens and durations
• Better handling of long-form audio with full attention
• Native timestamp prediction aligned with acoustic features
• Built-in punctuation and capitalization without separate models
• Optimal balance between accuracy and computational efficiency

4. Speaker Diarization with Sortformer

The Sortformer diarization model identifies and labels different speakers in multi-speaker audio:

• Speaker Segmentation: Determines when different speakers are talking
• Speaker Labeling: Assigns consistent labels to each speaker throughout the audio
• Streaming Support: Provides real-time speaker diarization with low latency
• Multi-speaker Support: Handles up to 4 speakers simultaneously
• Timestamp Integration: Aligns speaker labels with word-level timestamps from TDT model

Sortformer uses an innovative arrival-time ordering approach to resolve speaker permutation problems, making it particularly effective for conversational audio, meetings, and interviews. The model employs a FastConformer encoder with Transformer layers trained with Sort-Loss and Permutation Invariant Loss objectives.

Where to Get Started

Prerequisites

Runtime Environment:

• NeMo 2.2 or higher (for training and finetuning)
• Riva ASR 1.5.0 or higher (for deployment)
• Linux operating system
• NVIDIA GPU (H100, A100, or L40 recommended)
• Minimum 2GB RAM for model loading (larger RAM supports longer audio)

Development Tools:

• NeMo Framework for model finetuning
• NGC CLI for downloading artifacts
• Docker for containerized deployment

Deployment Workflow

1. Download the artifacts: Use NGC CLI to download the artifacts for your chosen variant (v2 or v3) from this collection

   ngc registry model download-version <artifact>
   

2. Finetune for Your Domain (Optional): Customize models for your specific use case

   • Acoustic Model: Finetune on domain-specific audio (accents, terminology, acoustics) Tutorial
   • Timestamp Tuning: Optimize for specific timing requirements
   • Domain Terminology: Adapt to specialized vocabulary

3. Optimize and Deploy: Package customized models into a NIM

   • Use Riva build and deploy flow to convert and optimize models
   • Deploy as a microservice with Riva NIMs
   • Scale horizontally for production workloads

References

Academic Papers

1. Fast Conformer with Linearly Scalable Attention for Efficient Speech Recognition
2. Efficient Sequence Transduction by Jointly Predicting Tokens and Durations
3. Conformer: Convolution-augmented Transformer for Speech Recognition

Software & Tools

1. NeMo Framework - Training and finetuning toolkit
2. Riva - Deployment and inference platform

Ethical Considerations

NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their supporting model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.

Terms of Use

License

GOVERNING TERMS: Use of this model is governed by the CC-BY-4.0 license.

Deployment Geography

Global

Disclaimer

AI models generate responses and outputs based on complex algorithms and machine learning techniques, and those responses or outputs may be inaccurate or indecent. By testing this model, you assume the risk of any harm caused by any response or output of the model. Please do not upload any confidential information or personal data. Your use is logged for security.