Tacotron2 and Waveglow 2.0 for PyTorch

NVIDIA Deep Learning Examples

Resource

NVIDIA Deep Learning Examples

Tacotron2 and Waveglow 2.0 for PyTorch

The Tacotron 2 and WaveGlow model form a text-to-speech system that enables user to synthesise a natural sounding speech from raw transcripts.

To train your model using mixed precision with Tensor Cores or using FP32, perform the following steps using the default parameters of the Tacrotron 2 and WaveGlow model on the LJ Speech dataset.

Clone the repository.

git clone https://github.com/NVIDIA/DeepLearningExamples.git
cd DeepLearningExamples/PyTorch/SpeechSynthesis/Tacotron2

Download and preprocess the dataset. Use the ./scripts/prepare_dataset.sh download script to automatically download and preprocess the training, validation and test datasets. To run this script, issue:
```
bash scripts/prepare_dataset.sh
```
Data is downloaded to the ./LJSpeech-1.1 directory (on the host). The ./LJSpeech-1.1 directory is mounted to the /workspace/tacotron2/LJSpeech-1.1 location in the NGC container.
Build the Tacotron 2 and WaveGlow PyTorch NGC container.
```
bash scripts/docker/build.sh
```
Start an interactive session in the NGC container to run training/inference. After you build the container image, you can start an interactive CLI session with:
```
bash scripts/docker/interactive.sh
```
The interactive.sh script requires that the location on the dataset is specified. For example, LJSpeech-1.1. To preprocess the datasets for Tacotron 2 training, use the ./scripts/prepare_mels.sh script:
```
bash scripts/prepare_mels.sh
```
The preprocessed mel-spectrograms are stored in the ./LJSpeech-1.1/mels directory.
Start training. To start Tacotron 2 training, run:
```
bash scripts/train_tacotron2.sh
```
To start WaveGlow training, run:
```
bash scripts/train_waveglow.sh
```
Start validation/evaluation. Ensure your loss values are comparable to those listed in the table in the Results section. For both models, the loss values are stored in the ./output/nvlog.json log file.

After you have trained the Tacotron 2 and WaveGlow models, you should get audio results similar to the samples in the ./audio folder. For details about generating audio, see the Inference process section below.

The training scripts automatically run the validation after each training epoch. The results from the validation are printed to the standard output (stdout) and saved to the log files.
Start inference. After you have trained the Tacotron 2 and WaveGlow models, you can perform inference using the respective checkpoints that are passed as --tacotron2 and --waveglow arguments. Tacotron2 and WaveGlow checkpoints can also be downloaded from NGC:

https://ngc.nvidia.com/catalog/models/nvidia:tacotron2pyt_fp16/files?version=3

https://ngc.nvidia.com/catalog/models/nvidia:waveglow256pyt_fp16/files?version=2

To run inference issue:
```
python inference.py --tacotron2 <Tacotron2_checkpoint> --waveglow <WaveGlow_checkpoint> --wn-channels 256 -o output/ -i phrases/phrase.txt --fp16
```
The speech is generated from lines of text in the file that is passed with -i argument. The number of lines determines inference batch size. To run inference in mixed precision, use the --fp16 flag. The output audio will be stored in the path specified by the -o argument.

You can also run inference on CPU with TorchScript by adding flag --cpu:
```
export CUDA_VISIBLE_DEVICES=
```
```
python inference.py --tacotron2 <Tacotron2_checkpoint> --waveglow <WaveGlow_checkpoint> --wn-channels 256 --cpu -o output/ -i phrases/phrase.txt
```