NGC | Catalog
Welcome Guest
CatalogModelsTTS En LJ Mixer-TTS

TTS En LJ Mixer-TTS

For downloads and more information, please view on a desktop device.
Logo for TTS En LJ Mixer-TTS

Description

Mixer-TTS Speech Synthesis model trained on female English speech (LJSpeech dataset)

Publisher

NVIDIA

Use Case

Speech Synthesis

Framework

PyTorch

Latest Version

1.6.0

Modified

December 29, 2021

Size

71.91 MB

Model Overview

Mixer-TTS is non-autoregressive model for mel-spectrogram generation. The model is based on the MLP-Mixer architecture adapted for speech synthesis. It contains pitch and duration predictors, with the latter being trained with an unsupervised TTS alignment framework.

Model Architecture

For more information about the model architecture, see the Mixer-TTS paper.

Training

Dataset

This model is trained on LJSpeech sampled at 22050Hz, and has been tested on generating female English voices with an American accent.

Performance

No performance information available at this time.

How to Use this Model

This model can be automatically loaded from NGC.

NOTE: In order to generate audio, you also need a 22050Hz vocoder from NeMo. This example uses the HiFi-GAN model which was additionally fine-tuned on Mixer-TTS outputs.

# Load Mixer-TTS
from nemo.collections.tts.models import MixerTTSModel
spec_generator = MixerTTSModel.from_pretrained("tts_en_lj_mixertts")

# Load vocoder
from nemo.collections.tts.models import Vocoder
model = Vocoder.from_pretrained(model_name="tts_en_lj_hifigan_ft_mixertts")

# Generate audio
import soundfile as sf
parsed = spec_generator.parse("You can type your sentence here to get nemo to produce speech.")
spectrogram = spec_generator.generate_spectrogram(tokens=parsed)
audio = model.convert_spectrogram_to_audio(spec=spectrogram)

# Save the audio to disk in a file called speech.wav
sf.write("speech.wav", audio.to('cpu').numpy(), 22050)

Input

This model accepts batches of text.

Output

This model generates mel spectrograms.

Limitations

This checkpoint only works well with vocoders that were trained on 22050Hz data. Otherwise, the generated audio may be scratchy or choppy-sounding.

Versions

1.6.0: The original version of Mixer-TTS model which was released with NeMo 1.6.0.

References

Mixer-TTS paper: https://arxiv.org/pdf/2110.03584.pdf

Licence

License to use this model is covered by the NGC TERMS OF USE unless another License/Terms Of Use/EULA is clearly specified. By downloading the public and release version of the model, you accept the terms and conditions of the NGC TERMS OF USE.