HiFi-GAN for PyTorch

NVIDIA Deep Learning Examples

Resource

NVIDIA Deep Learning Examples

HiFi-GAN for PyTorch

HiFi-GAN model implements a spectrogram inversion model that allows to synthesize speech waveforms from mel-spectrograms.

To train your model using mixed or TF32 precision with Tensor Cores or using FP32, perform the following steps using the default parameters of the HiFi-GAN model on the LJSpeech 1.1 dataset. For the specifics concerning training and inference, refer to the Advanced section. Pre-trained HiFi-GAN models are available for download on NGC.

Clone the repository.

git clone https://github.com/NVIDIA/DeepLearningExamples.git
cd DeepLearningExamples/PyTorch/SpeechSynthesis/HiFiGAN

Build and run the HiFi-GAN PyTorch NGC container.

By default the container uses all available GPUs.
```
bash scripts/docker/build.sh
bash scripts/docker/interactive.sh
```
Download and preprocess the dataset.
```
bash scripts/download_dataset.sh
bash scripts/prepare_dataset.sh
```
The data is downloaded to the ./data/LJSpeech-1.1 directory (on the host). The ./data/LJSpeech-1.1 directory is mounted under the /workspace/hifigan/data/LJSpeech-1.1 location in the NGC container. The complete dataset has the following structure:
```
./data/LJSpeech-1.1
|-- mels             # Mel-spectrograms generated with the `prepare_dataset.sh` script
|-- metadata.csv     # Mapping of waveforms to utterances
|-- README
|-- wavs             # Raw waveforms
```
Apart from generating mel-spectrograms in data/LJSpeech-1.1/mels directory, the scripts/prepare_dataset.sh script additionally generates LJSpeech-1.1 relevant split metadata files in data/filelists directory.
Start training.
```
NUM_GPUS=8 BATCH_SIZE=16 GRAD_ACCUMULATION=1 bash scripts/train_lj22khz.sh
```
The training produces a HiFi-GAN model capable of generating waveforms from mel-spectrograms. It is serialized as a single .pt checkpoint file, along with a series of intermediate checkpoints. The script is configured for 8x GPU with at least 16GB of memory.

To train with 1x GPU, run:
```
NUM_GPUS=1 BATCH_SIZE=16 GRAD_ACCUMULATION=8 bash scripts/train_lj22khz.sh
```
When training with AMP on Ampere GPU architectures, use an additional --no_amp_grouped_conv flag to speed up the training:
```
AMP=true NUM_GPUS=8 BATCH_SIZE=16 GRAD_ACCUMULATION=1 bash scripts/train_lj22khz.sh  --no_amp_grouped_conv
```
The flag will disable mixed-precision training on selected layers. For more details refer to Known issues.

Consult Training process and example configs to adjust to a different configuration or enable Automatic Mixed Precision.
(optionally) Fine-tune the model.

Some mel-spectrogram generators are prone to model bias. As the spectrograms differ from the true data on which HiFi-GAN was trained, the quality of the generated audio might suffer. In order to overcome this problem, a HiFi-GAN model can be fine-tuned on the outputs of a particular mel-spectrogram generator in order to adapt to this bias. In this section we discuss fine-tuning to FastPitch outputs.

Obtain a model for which HiFi-GAN will be fine-tuned. If the FastPitch model was trained using phonemes, additionally download the CMU Dictionary.
```
bash scripts/download_models.sh fastpitch
bash scripts/download_cmudict.sh
```
Generate mel-spectrograms for all utterances in the dataset with the FastPitch model:
```
bash scripts/extract_fine_tune_mels.sh
```
Mel-spectrograms should now be prepared in the data/mels-fastpitch-ljs22khz directory. The fine-tuning script will load an existing HiFi-GAN model and run several epochs of training using spectrograms generated in the last step.
```
bash scripts/fine_tune.sh
```
This step will produce another .pt HiFi-GAN model checkpoint file fine-tuned to the particular FastPitch model.
Start validation/evaluation.

Ensure your training loss values are comparable to those listed in the table in the Results section. Note the validation loss is evaluated with ground truth durations for letters (not the predicted ones). The loss values are stored in the ./output/nvlog.json log file, ./output/{train,val,test} as TensorBoard logs, and printed to the standard output (stdout) during training. The main reported loss is a weighted sum of losses for mel-, pitch-, and duration- predicting modules.
Start inference.

The audio can be synthesized either:
- from ground truth mel-spectrograms, as a means of assessing the quality of HiFi-GAN, or
- from an output of a mel-spectrogram generator model like FastPitch or Tacotron 2, as a full text-to-speech pipeline. We descibe both scenarios.
Sythesizing audio from ground truth mel-spectrograms In order to perform inference, pass a .pt checkpoint with the --hifigan argument to the inference script:
```
python inference.py --cuda \
                    --hifigan pretrained_models/hifigan/<HiFi-GAN checkpoint> \
                    -i phrases/devset10.tsv \
                    -o output/wavs_devset10
```
The speech is generated from a file passed with the -i argument, with one utterance per line:
```
`<output wav file name>|<utterance>`
```
Synthesizing audio from raw text with mel-spectrogram generator The current implementation allows for convenient inference with the FastPitch model. A pre-trained FastPitch model can be downloaded with the scripts/download_model.sh fastpitch script. Alternatively, to train FastPitch or Tacotron 2 from scratch, follow the instructions in NVIDIA/DeepLearningExamples/FastPitch or NVIDIA/DeepLearningExamples/Tacotron2.

Begin with downloading the model and the pronunciation dictionary:
```
bash scripts/download_models.sh fastpitch
bash scripts/download_cmudict.sh
```
To run inference on FastPitch outputs, pass an additional .pt checkpoint with the --fastpitch flag:
```
python inference.py --cuda \
                    --hifigan pretrained_models/hifigan/<HiFi-GAN checkpoint> \
                    --fastpitch pretrained_models/fastpitch/<FastPitch checkpoint> \
                    -i phrases/devset10.tsv \
                    -o output/wavs_devset10
```

To run inference in mixed precision, use the --amp flag. The output audio will be stored in the path specified by the -o argument. Consult the inference.py to learn more options, such as setting the batch size.

Now that you have your model trained and evaluated, you can choose to compare your training results with our Training accuracy results. You can also choose to benchmark your performance to the Training performance benchmark, or the Inference performance benchmark. Following the steps in these sections ensures you achieve the same accuracy and performance results as stated in the Results section.

The audio can be generated by following the Inference process section below. The synthesized audio should be similar to the samples in the ./audio directory.