Mixer-TTS-X is non-autoregressive model for mel-spectrogram generation. The model is based on the MLP-Mixer architecture adapted for speech synthesis. It contains pitch and duration predictors, with the latter being trained with an unsupervised TTS alignment framework, and additionally uses token embeddings from a pre-trained language model ALBERT.
For more information about the model architecture, see the Mixer-TTS paper which describes Mixer-TTS-X.
This model is trained on LJSpeech sampled at 22050Hz, and has been tested on generating female English voices with an American accent.
No performance information available at this time.
This model can be automatically loaded from NGC.
NOTE: In order to generate audio, you also need a 22050Hz vocoder from NeMo. This example uses the HiFi-GAN model which was additionally fine-tuned on Mixer-TTS-X outputs.
# Load Mixer-TTS-X from nemo.collections.tts.models import MixerTTSModel spec_generator = MixerTTSModel.from_pretrained("tts_en_lj_mixerttsx") # Load vocoder from nemo.collections.tts.models import Vocoder model = Vocoder.from_pretrained(model_name="tts_en_lj_hifigan_ft_mixerttsx") # Generate audio import soundfile as sf raw_text = "You can type your sentence here to get nemo to produce speech." parsed = spec_generator.parse(raw_text) spectrogram = spec_generator.generate_spectrogram(tokens=parsed, raw_texts=[raw_text]) audio = model.convert_spectrogram_to_audio(spec=spectrogram) # Save the audio to disk in a file called speech.wav sf.write("speech.wav", audio.to('cpu').numpy(), 22050)
This model accepts batches of text.
This model generates mel spectrograms.
This checkpoint only works well with vocoders that were trained on 22050Hz data. Otherwise, the generated audio may be scratchy or choppy-sounding.
1.6.0: The original version of Mixer-TTS-X model which was released with NeMo 1.6.0.
Mixer-TTS paper which describes Mixer-TTS-X: https://arxiv.org/pdf/2110.03584.pdf