NGC | Catalog
Welcome Guest
CatalogModelsTTS En E2E Fastspeech2 Hifigan

TTS En E2E Fastspeech2 Hifigan

For downloads and more information, please view on a desktop device.
Logo for TTS En E2E Fastspeech2 Hifigan

Description

FastSpeech2+HiFiGAN End-to-End Speech Synthesis model trained on female English speech.

Publisher

NVIDIA

Use Case

Other

Framework

PyTorch with NeMo

Latest Version

1.0.0

Modified

June 30, 2021

Size

352.87 MB

Model Overview

FastSpeech2HifiGanE2E is an end-to-end, non-autoregressive model that generates audio from text. It combines FastSpeech2 and HiFiGan into one model and is traned jointly in an end-to-end manner.

Model Architecture

The FastSpeech2 portion consists of the same transformer-based encoder, and a 1D-convolution-based variance adaptor as the original FastSpeech2 model. The HiFiGan portion takes the discriminator from HiFiGan and uses it to generate audio from the output of the fastspeech2 portion. No spectrograms are used in the training of the model. All losses are taken from HiFiGan plus additional losses for the variance adaptor.

Training

This model is trained on LJSpeech sampled at 22050Hz, and has been tested on generating female English voices with an American accent. Supplementary data (durations, pitches, energies) were calculated using dataset preprocessing scripts that can be found in the NeMo library.

Performance

No performance information available at this time.

How to Use this Model

This model can be automatically loaded from NGC.

import soundfile as sf
from nemo.collections.tts.models import FastSpeech2HifiGanE2EModel

# Load the model from NGC
model = FastSpeech2HifiGanE2EModel.from_pretrained(model_name="tts_en_e2e_fastspeech2hifigan")

# Run inference
tokens = model.parse("Hey, I can speak!")
audio = model.convert_text_to_waveform(tokens=tokens)

# Save the audio to disk in a file called speech.wav
sf.write("speech.wav", audio.to('cpu').numpy(), 22050)

Input

This model accepts batches of text.

Output

This model generates audio.

Limitations

This model outputs audio at 22050Hz.