Speech Synthesis: German (de-DE) Male - FastPitch Model Overview

Description:

The German Male FastPitch-HifiGAN model transcribes text into audio representations using two model components: FastPitch and HifiGAN. This model is ready for commercial use.

FastPitch is a mel-spectrogram generator used as the first part of a neural text-to-speech system with a neural vocoder. This model uses the International Phonetic Alphabet (IPA) for inference and training, and it can output a male voice for German.

This FastPitch model was trained to be used with a corresponding HifiGAN model available here. HifiGAN is a neural vocoder model for text-to-speech applications. It is the second part of a two-stage speech synthesis pipeline.

References:

FastPitch paper: https://arxiv.org/abs/2006.06873

HifiGAN paper: https://arxiv.org/abs/2010.05646

Model Architecture:

Architecture Type: Transformer + Generative Adversarial Network (GAN)

Network Architecture: FastPitch + HifiGAN

FastPitch is a fully-parallel text-to-speech transformer-based model, conditioned on fundamental frequency contours. The model predicts pitch contours during inference. By altering these predictions, the generated speech can be more expressive, better match the semantic of the utterance, and be engaging to the listener. FastPitch is based on a fully-parallel Transformer architecture, with much higher real-time factor than Tacotron2 for mel-spectrogram synthesis of a typical utterance.

HifiGAN is a neural vocoder based on a generative adversarial network framework. During training, the model uses a powerful discriminator consisting of small sub-discriminators, each one focusing on specific periodic parts of a raw waveform.

Input:

For FastPitch (1st Stage): Text strings in German

Other Properties Related to Input: 400 Character Text String Limit

Output:

For HifiGAN (2nd Stage): Audio of shape (batch x time) in wav format

Other Parameters Related to Output: Mono, Encoded 16 bit audio; 20 Second Maximum Length; this model can output a male voice for German.

Software Integration:

Runtime Engine(s): Riva 2.13.0 or greater

Supported Hardware Platform(s):

• NVIDIA Volta V100
  
• NVIDIA Turing T4
  
• NVIDIA A100 GPU
  
• NVIDIA A30 GPU
  
• NVIDIA A10 GPU
  
• NVIDIA H100 GPU
  
• NVIDIA L4 GPU
  
• NVIDIA L40 GPU
  
• NVIDIA Jetson Orin
  
• NVIDIA Jetson AGX Xavier
  
• NVIDIA Jetson NX Xavier
  

Supported Operating System(s):

• Linux
  
• Linux 4 Tegra
  

Model Version(s):

FastPitch_De-DE-Male-1.0

Training & Evaluation:

Training Dataset:

** Data Collection Method by dataset

• Human
  Properties (Quantity, Dataset Descriptions, Sensor(s)): Proprietary audio-text pair dataset sampled at 22050 Hz, which contains one Male voice speaking German.
  

Evaluation Dataset:

** Data Collection Method by dataset

• Human
  Properties (Quantity, Dataset Descriptions, Sensor(s)): Private audio-text pair dataset used for training, a proprietary dataset sampled at 22050 Hz, which contains one Male voice speaking German.
  

Inference:

Engine: Triton
Test Hardware:

• NVIDIA Volta V100
• NVIDIA Turing T4
• NVIDIA A100 GPU
• NVIDIA A30 GPU
• NVIDIA A10 GPU
• NVIDIA H100 GPU
• NVIDIA L4 GPU
• NVIDIA L40 GPU
• NVIDIA Jetson Orin
• NVIDIA Jetson AGX Xavier
• NVIDIA Jetson NX Xavier

Ethical Considerations:

NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their supporting model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse. For more detailed information on ethical considerations for this model, please see the Model Card++ Explainability, Bias, Safety & Security, and Privacy Subcards. Please report security vulnerabilities or NVIDIA AI Concerns here.

License

By downloading and using the models and resources packaged with Riva Conversational AI, you accept the terms of the Riva license.