Model Overview

Trained or fine-tuned NeMo models (with the file extenstion .nemo) can be converted to Riva models (with the file extension .riva) and then deployed. Here is a pre-trained HiFiGAN text-to-speech (TTS) Riva model.

Model Architecture

HiFi-GAN is a generative adversarial network (GAN) model that generates audio from mel spectrograms. The generator uses transposed convolutions to upsample mel-spectrograms to audio.

Training

Dataset

This model is trained on LJSpeech sampled at 22050Hz, and has been tested on generating female English voices with an American accent.

Performance

No performance information available at this time.

How to Use this Model

This model can be automatically loaded from NGC.

NOTE: In order to generate audio, you also need a spectrogram generator from NeMo. This example uses the FastPitch model.

# Load PastPitch
from nemo.collections.tts.models import FastPitchModel
spec_generator = FastPitchModel.from_pretrained("tts_en_fastpitch")

# Load HiFi-GAN
from nemo.collections.tts.models import HifiGanModel
model = HifiGanModel.from_pretrained(model_name="tts_en_lj_hifigan_ft_tacotron2_talknet_fastpitch")

# Generate audio
import soundfile as sf
parsed = spec_generator.parse("You can type your sentence here to get nemo to produce speech.")
spectrogram = spec_generator.generate_spectrogram(tokens=parsed)
audio = model.convert_spectrogram_to_audio(spec=spectrogram)

### Save the audio to disk in a file called speech.wav
sf.write("speech.wav", audio.to('cpu').numpy(), 22050)

Input

This model accepts batches of mel spectrograms.

Output

This model outputs audio at 22050Hz.

Limitations

There are no known limitations at this time.

Versions

1.0.0: Add model (tts_en_lj_hifigan_ft_tacotron2_talknet_fastpitch.nemo) which was released with NeMo 1.0.0. This model was trained on ground-truth mel-spectrograms and additionally fine-tuned on generated spectrograms from Tacotron 2, TalkNet 2 and FastPitch.

1.6.0: Add models (tts_en_lj_hifigan_ft_mixertts.nemo and tts_en_lj_hifigan_ft_mixerttsx.nemo) which were released with NeMo 1.6.0 and model (tts_en_lj_hifigan_ft_tacotron2_talknet_fastpitch.nemo) from 1.0.0 without changes. The tts_en_lj_hifigan_ft_mixertts.nemo and tts_en_lj_hifigan_ft_mixerttsx.nemo models were fine-tuned on generated spectrograms from Mixer-TTS and Mixer-TTS-X respectively. The tts_en_lj_hifigan_ft_tacotron2_talknet_fastpitch.nemo model were used as initial checkpoint for fine-tuning.

References

HiFi-GAN paper: https://arxiv.org/abs/2010.05646

Licence

License to use this model is covered by the NGC TERMS OF USE unless another License/Terms Of Use/EULA is clearly specified. By downloading the public and release version of the model, you accept the terms and conditions of the NGC TERMS OF USE.