Speech Synthesis: RAD-TTS Model Card

Model Overview

RAD-TTS is a mel-spectrogram generator, designed to be used as the first part of a neural text-to-speech system in conjunction with a neural vocoder. This model uses the International Phonetic Alphabet (IPA) for inference and training, and it can output a female or a male voice for US English. It is also trained for six (6) emotions for the female voice, and four (4) emotions for the male voice, which can be mixed and regulated through SSML tags.

Model Architecture

RAD-TTS is a parallel flow-based generative network for text-to-speech synthesis. It extends prior parallel approaches by additionally modeling speech rhythm as a separate generative distribution to facilitate variable token duration during inference.

Input:

Text Strings in English

Other Properties Related to Input: 400 Character Text String Limit

Output:

Mel spectrogram of shape (batch x mel_channels x time)

Software Integration:

Runtime Engine(s): Riva 2.13.0 or greater

Supported Hardware Platform(s):

• NVIDIA Volta V100
  
• NVIDIA Turing T4
  
• NVIDIA A100 GPU
  
• NVIDIA A30 GPU
  
• NVIDIA A10 GPU
  
• NVIDIA H100 GPU
  
• NVIDIA L4 GPU
  
• NVIDIA L40 GPU
  
• NVIDIA Jetson Orin
  
• NVIDIA Jetson AGX Xavier
  
• NVIDIA Jetson NX Xavier
  

Supported Operating System(s):

• Linux
  
• Linux 4 Tegra
  

Model Version(s):

RadTTSpp_Emo_mix_44k_EnglishUS_Emotion_IPA v2.16.0

Training & Evaluation:

Training Dataset:

** Data Collection Method by dataset

• Human
  Properties (Quantity, Dataset Descriptions, Sensor(s)): This model is trained on a proprietary dataset sampled at 44100Hz, and can be used to generate English voices with an American accent. This model supports one (1) male voice and one (1) female voice. The female voice comes with neutral, calm, happy, angry, sad, and fearful emotions. The male voice comes with neutral, calm, happy, and angry emotions. While both genders are trained for all emotions, this dataset only releases those that passed the evaluation standard for expressiveness and quality. Each emotion is accessed as a speaker. For example Female-Sad, Male-Happy, and so on. The dataset also contains a subset of sentences with different words emphasized.

Evaluation Dataset:

** Data Collection Method by dataset

• Human
  Properties (Quantity, Dataset Descriptions, Sensor(s)): This model is trained on a proprietary dataset sampled at 44100Hz, and can be used to generate English voices with an American accent. This model supports one (1) male voice and one (1) female voice. The female voice comes with neutral, calm, happy, angry, sad, and fearful emotions. The male voice comes with neutral, calm, happy, and angry emotions. While both genders are trained for all emotions, this dataset only releases those that passed the evaluation standard for expressiveness and quality. Each emotion is accessed as a speaker. For example Female-Sad, Male-Happy, and so on. The dataset also contains a subset of sentences with different words emphasized.

How to Use this Model

RAD-TTS is intended to be used as the first part of a two stage speech synthesis pipeline. RAD-TTS takes text and produces a mel-spectrogram. The second stage takes the generated mel-spectrogram and returns audio.

Inference:

Engine: Triton
Test Hardware:

• NVIDIA Volta V100
• NVIDIA Turing T4
• NVIDIA A100 GPU
• NVIDIA A30 GPU
• NVIDIA A10 GPU
• NVIDIA H100 GPU
• NVIDIA L4 GPU
• NVIDIA L40 GPU
• NVIDIA Jetson Orin
• NVIDIA Jetson AGX Xavier
• NVIDIA Jetson NX Xavier

References

[1] RAD-TTS: Parallel Flow-Based TTS with Robust Alignment Learning and Diverse Synthesis

Suggested Reading

Refer to the Riva documentation for more information.

Ethical AI

NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their supporting model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse. For more detailed information on ethical considerations for this model, please see the Model Card++ Explainability, Bias, Safety & Security, and Privacy Subcards. Please report security vulnerabilities or NVIDIA AI Concerns here.

License

By downloading and using the models and resources packaged with Riva Conversational AI, you accept the terms of the Riva license.