To train your model using mixed or TF32 precision with Tensor Cores or using FP32, perform the following steps using the default parameters of the FastPitch model on the LJSpeech 1.1 dataset. For the specifics concerning training and inference, see the Advanced section. Pre-trained FastPitch models are available for download on NGC.

1. Clone the repository.

   git clone https://github.com/NVIDIA/DeepLearningExamples.git
   cd DeepLearningExamples/PyTorch/SpeechSynthesis/FastPitch
   

2. Build and run the FastPitch PyTorch NGC container.

   By default the container will use all available GPUs.

   bash scripts/docker/build.sh
   bash scripts/docker/interactive.sh
   

3. Download and preprocess the dataset.

   Use the scripts to automatically download and preprocess the training, validation and test datasets:

   bash scripts/download_dataset.sh
   bash scripts/prepare_dataset.sh
   

   The data is downloaded to the ./LJSpeech-1.1 directory (on the host). The ./LJSpeech-1.1 directory is mounted under the /workspace/fastpitch/LJSpeech-1.1 location in the NGC container. The complete dataset has the following structure:

   ./LJSpeech-1.1
   |-- mels             # (optional) Pre-calculated target mel-spectrograms; may be calculated on-line
   |-- metadata.csv     # Mapping of waveforms to utterances
   |-- pitch            # Fundamental frequency countours for input utterances; may be calculated on-line
   |-- README
   |-- wavs             # Raw waveforms
   

4. Start training.

   bash scripts/train.sh
   

   The training will produce a FastPitch model capable of generating mel-spectrograms from raw text. It will be serialized as a single .pt checkpoint file, along with a series of intermediate checkpoints. The script is configured for 8x GPU with at least 16GB of memory. Consult Training process and example configs to adjust to a different configuration or enable Automatic Mixed Precision.

5. Start validation/evaluation.

   Ensure your training loss values are comparable to those listed in the table in the Results section. Note that the validation loss is evaluated with ground truth durations for letters (not the predicted ones). The loss values are stored in the ./output/nvlog.json log file, ./output/{train,val,test} as TensorBoard logs, and printed to the standard output (stdout) during training. The main reported loss is a weighted sum of losses for mel-, pitch-, and duration- predicting modules.

   The audio can be generated by following the Inference process section below. The synthesized audio should be similar to the samples in the ./audio directory.

6. Start inference/predictions.

   To synthesize audio, you will need a WaveGlow model, which generates waveforms based on mel-spectrograms generated with FastPitch. By now, a pre-trained model should have been downloaded by the scripts/download_dataset.sh script. Alternatively, to train WaveGlow from scratch, follow the instructions in NVIDIA/DeepLearningExamples/Tacotron2 and replace the checkpoint in the ./pretrained_models/waveglow directory.

   You can perform inference using the respective .pt checkpoints that are passed as --fastpitch and --waveglow arguments:

   python inference.py \
       --cuda \
       --fastpitch output/<FastPitch checkpoint> \
       --energy-conditioning \
       --waveglow pretrained_models/waveglow/<WaveGlow checkpoint> \
       --wn-channels 256 \
       -i phrases/devset10.tsv \
       -o output/wavs_devset10
   

   The speech is generated from a file passed with the -i argument, with one utterance per line:

   `<output wav file name>|<utterance>`
   

To run inference in mixed precision, use the --amp flag. The output audio will be stored in the path specified by the -o argument. Consult the inference.py to learn more options, such as setting the batch size.