The following sections provide greater details of the dataset, running training and inference, and the training results.

Scripts and sample code

The repository holds code for FastPitch (training and inference) and WaveGlow (inference only). The code specific to a particular model is located in that model's directory - ./fastpitch and ./waveglow - and common functions live in the ./common directory. The model-specific scripts are as follows:

• <model_name>/model.py - the model architecture, definition of forward and inference functions
• <model_name>/arg_parser.py - argument parser for parameters specific to a given model
• <model_name>/data_function.py - data loading functions
• <model_name>/loss_function.py - loss function for the model

In the root directory ./ of this repository, the ./train.py script is used for training while inference can be executed with the ./inference.py script. The script ./models.py is used to construct a model of requested type and properties.

The repository is structured similarly to the NVIDIA Tacotron2 Deep Learning example, so that they could be combined in more advanced use cases.

Parameters

In this section, we list the most important hyperparameters and command-line arguments, together with their default values that are used to train FastPitch.

• --epochs - number of epochs (default: 1500)
• --learning-rate - learning rate (default: 0.1)
• --batch-size - batch size for a single forward-backward step (default: 16)
• --grad-accumulation - number of steps over which gradients are accumulated (default: 2)
• --amp - use mixed precision training (default: disabled)
• --load-pitch-from-disk - pre-calculated fundamental frequency values, estimated before training, are loaded from the disk during training (default: enabled)
• --energy-conditioning - enables additional conditioning on energy (default: enabled)
• --p-arpabet - probability of choosing phonemic over graphemic representation for every word, if available (default: 1.0)

Command-line options

To see the full list of available options and their descriptions, use the -h or --help command line option, for example:

python train.py --help

The following example output is printed when running the model:

DLL 2021-06-14 23:08:53.659718 - epoch    1 | iter   1/48 | loss 40.97 | mel loss 35.04 | kl loss 0.02240 | kl weight 0.01000 |    5730.98 frames/s | took 24.54 s | lrate 3.16e-06
DLL 2021-06-14 23:09:28.449961 - epoch    1 | iter   2/48 | loss 41.07 | mel loss 35.12 | kl loss 0.02258 | kl weight 0.01000 |    4154.18 frames/s | took 34.79 s | lrate 6.32e-06
DLL 2021-06-14 23:09:59.365398 - epoch    1 | iter   3/48 | loss 40.86 | mel loss 34.93 | kl loss 0.02252 | kl weight 0.01000 |    4589.15 frames/s | took 30.91 s | lrate 9.49e-06

Getting the data

The FastPitch and WaveGlow models were trained on the LJSpeech-1.1 dataset. The ./scripts/download_dataset.sh script will automatically download and extract the dataset to the ./LJSpeech-1.1 directory.

Dataset guidelines

The LJSpeech dataset has 13,100 clips that amount to about 24 hours of speech of a single, female speaker. Since the original dataset does not define a train/dev/test split of the data, we provide a split in the form of three file lists:

./filelists
|-- ljs_audio_pitch_text_train_v3.txt
|-- ljs_audio_pitch_text_test.txt
|-- ljs_audio_pitch_text_val.txt

FastPitch predicts character durations just like FastSpeech does. FastPitch 1.1 aligns input symbols to output mel-spectrogram frames automatically and does not rely on any external aligning model. FastPitch training can now be started on raw waveforms without any pre-processing: pitch values and mel-spectrograms will be calculated on-line.

For every mel-spectrogram frame, its fundamental frequency in Hz is estimated with the Probabilistic YIN algorithm.

Figure 2. Pitch estimates for mel-spectrogram frames of phrase "in being comparatively" (in blue) averaged over characters (in red). Silent letters have duration 0 and are omitted.

Multi-dataset

Follow these steps to use datasets different from the default LJSpeech dataset.

1. Prepare a directory with .wav files.

   ./my_dataset
   |-- wavs
   

2. Prepare filelists with transcripts and paths to .wav files. They define training/validation split of the data (test is currently unused):

   ./filelists
   |-- my-dataset_audio_text_train.txt
   |-- my-dataset_audio_text_val.txt
   

   Those filelists should list a single utterance per line as:

   `<audio file path>|<transcript>`
   

   The <audio file path> is the relative path to the path provided by the --dataset-path option of train.py.

3. Run the pre-processing script to calculate pitch:

    python prepare_dataset.py \
        --wav-text-filelists filelists/my-dataset_audio_text_train.txt \
                             filelists/my-dataset_audio_text_val.txt \
        --n-workers 16 \
        --batch-size 1 \
        --dataset-path $DATA_DIR \
        --extract-pitch \
        --f0-method pyin
   

4. Prepare file lists with paths to pre-calculated pitch:

   ./filelists
   |-- my-dataset_audio_pitch_text_train.txt
   |-- my-dataset_audio_pitch_text_val.txt
   

In order to use the prepared dataset, pass the following to the train.py script:

--dataset-path ./my_dataset` \
--training-files ./filelists/my-dataset_audio_pitch_text_train.txt \
--validation files ./filelists/my-dataset_audio_pitch_text_val.txt

Training process

FastPitch is trained to generate mel-spectrograms from raw text input. It uses short time Fourier transform (STFT) to generate target mel-spectrograms from audio waveforms to be the training targets.

The training loss is averaged over an entire training epoch, whereas the validation loss is averaged over the validation dataset. Performance is reported in total output mel-spectrogram frames per second and recorded as train_frames/s (after each iteration) and avg_train_frames/s (averaged over epoch) in the output log file ./output/nvlog.json. The result is averaged over an entire training epoch and summed over all GPUs that were included in the training.

The scripts/train.sh script is configured for 8x GPU with at least 16GB of memory:

--batch-size 16
--grad-accumulation 2

In a single accumulated step, there are batch_size x grad_accumulation x GPUs = 256 examples being processed in parallel. With a smaller number of GPUs, increase --grad_accumulation to keep this relation satisfied, e.g., through env variables

NUM_GPUS=1 GRAD_ACCUMULATION=16 bash scripts/train.sh

Inference process

You can run inference using the ./inference.py script. This script takes text as input and runs FastPitch and then WaveGlow inference to produce an audio file. It requires pre-trained checkpoints of both models and input text as a text file, with one phrase per line.

Pre-trained FastPitch models are available for download on NGC.

Having pre-trained models in place, run the sample inference on LJSpeech-1.1 test-set with:

bash scripts/inference_example.sh

Examine the inference_example.sh script to adjust paths to pre-trained models, and call python inference.py --help to learn all available options. By default, synthesized audio samples are saved in ./output/audio_* folders.

FastPitch allows us to linearly adjust the rate of synthesized speech like FastSpeech. For instance, pass --pace 0.5 for a twofold decrease in speed.

For every input character, the model predicts a pitch cue - an average pitch over a character in Hz. Pitch can be adjusted by transforming those pitch cues. A few simple examples are provided below.

The flags can be combined. Modify these functions directly in the inference.py script to gain more control over the final result.

You can find all the available options by calling python inference.py --help. More examples are presented on the website with samples.