This resource is using open-source code maintained in github (see the quick-start-guide section) and available for download from NGC

This text-to-speech (TTS) system is a combination of two neural network\nmodels:

a modified Tacotron 2 model from the Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions\npaper
a flow-based neural network model from the WaveGlow: A Flow-based Generative Network for Speech Synthesis paper

The Tacotron 2 and WaveGlow models form a text-to-speech system that enables\nusers to synthesize natural sounding speech from raw transcripts without\nany additional information such as patterns and/or rhythms of speech.

Our implementation of Tacotron 2 models differs from the model described in the\npaper. Our implementation uses Dropout instead of Zoneout to regularize the\nLSTM layers. Also, the original text-to-speech system proposed in the paper\nuses the WaveNet model to synthesize\nwaveforms. In our implementation, we use the WaveGlow model for this purpose.

Both models are based on implementations of NVIDIA GitHub repositories\nTacotron 2 and\nWaveGlow, and are trained on a publicly\navailable LJ Speech dataset.

The Tacotron 2 and WaveGlow model enables you to efficiently synthesize high\nquality speech from text.

Both models are trained with mixed precision using Tensor Cores on Volta,\nTuring, and the NVIDIA Ampere GPU architectures. Therefore, researchers can\nget results 2.0x faster for Tacotron 2 and 3.1x faster for WaveGlow than\ntraining without Tensor Cores, while experiencing the benefits of mixed\nprecision training. The models are tested against each NGC monthly\ncontainer release to ensure consistent accuracy and performance over time.

Model architecture

The Tacotron 2 model is a recurrent sequence-to-sequence model with attention that\npredicts mel-spectrograms from text. The encoder (blue blocks in the figure\nbelow) transforms the whole text into a fixed-size hidden feature\nrepresentation. This feature representation is then consumed by the\nautoregressive decoder (orange blocks) that produces one spectrogram frame at\na time. In our implementation, the autoregressive WaveNet (green block) is\nreplaced by the flow-based generative WaveGlow.

$\"\"$

Figure 1. Architecture of the Tacotron 2 model. Taken from the\nTacotron 2 paper.

The WaveGlow model is a flow-based generative model that generates audio\nsamples from Gaussian distribution using mel-spectrogram conditioning (Figure\n2). During training, the model learns to transform the dataset distribution\ninto spherical Gaussian distribution through a series of flows. One step of a\nflow consists of an invertible convolution, followed by a modified WaveNet\narchitecture that serves as an affine coupling layer. During inference, the\nnetwork is inverted and audio samples are generated from the Gaussian\ndistribution. Our implementation uses 512 residual channels in the coupling layer.

$\"\"$

Figure 2. Architecture of the WaveGlow model. Taken from the\nWaveGlow paper.

Default configuration

Both models support multi-GPU and mixed precision training with dynamic loss\nscaling (see Apex code\nhere),\nas well as mixed precision inference. To speed up Tacotron 2 training,\nreference mel-spectrograms are generated during a preprocessing step and read\ndirectly from disk during training, instead of being generated during training.

The following features were implemented in this model:

data-parallel multi-GPU training
dynamic loss scaling with backoff for Tensor Cores (mixed precision)\ntraining.

Feature support matrix

The following features are supported by this model.

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n

tr:nth-child(odd)]:bg-surface-sunken\">

Feature	Tacotron 2	WaveGlow
AMP	Yes	Yes
Apex DistributedDataParallel	Yes	Yes

Features

AMP - a tool that enables Tensor Core-accelerated training. For more information,\nrefer to Enabling mixed precision.

Apex DistributedDataParallel - a module wrapper that enables easy multiprocess\ndistributed data parallel training, similar to torch.nn.parallel.DistributedDataParallel.\nDistributedDataParallel is optimized for use with NCCL. It achieves high\nperformance by overlapping communication with computation during backward()\nand bucketing smaller gradient transfers to reduce the total number of transfers\nrequired.

Mixed precision training

Mixed precision is the combined use of different numerical precisions in a\ncomputational method. Mixed precision\ntraining offers significant computational speedup by performing operations in\nhalf-precision format, while storing minimal information in single-precision\nto retain as much information as possible in critical parts of the network.\nSince the introduction of Tensor Cores\nin Volta, and following with both the Turing and Ampere architectures,\nsignificant training speedups are\nexperienced by switching to mixed precision -- up to 3x overall speedup on\nthe most arithmetically intense model architectures. Using mixed precision\ntraining requires two steps:

Porting the model to use the FP16 data type where appropriate.
Adding loss scaling to preserve small gradient values.

The ability to train deep learning networks with lower precision was\nintroduced in the Pascal architecture and first supported in CUDA 8 in the NVIDIA Deep Learning SDK.

For information about:

How to train using mixed precision, see the Mixed Precision Training\npaper and Training With Mixed Precision\ndocumentation.
Techniques used for mixed precision training, see the Mixed-Precision Training of Deep Neural Networks\nblog.
APEX tools for mixed precision training, see the NVIDIA Apex: Tools for Easy Mixed-Precision Training in PyTorch.

Enabling mixed precision

Mixed precision is enabled in PyTorch by using the Automatic Mixed Precision\n(AMP) library from APEX that casts variables\nto half-precision upon retrieval, while storing variables in single-precision\nformat. Furthermore, to preserve small gradient magnitudes in backpropagation,\na loss scaling\nstep must be included when applying gradients. In PyTorch, loss scaling can be\neasily applied by using the scale_loss() method provided by AMP. The scaling value\nto be used can be dynamic or fixed.

By default, the train_tacotron2.sh and train_waveglow.sh scripts will\nlaunch mixed precision training with Tensor Cores. You can change this\nbehaviour by removing the --amp flag from the train.py script.

To enable mixed precision, the following steps were performed in the Tacotron 2 and\nWaveGlow models:

Import AMP from APEX:

from apex import amp\namp.lists.functional_overrides.FP32_FUNCS.remove('softmax')\namp.lists.functional_overrides.FP16_FUNCS.append('softmax')\n

Initialize AMP:

model, optimizer = amp.initialize(model, optimizer, opt_level=\"O1\")\n

If running on multi-GPU, wrap the model with DistributedDataParallel:

from apex.parallel import DistributedDataParallel as DDP\nmodel = DDP(model)\n

\n
Scale loss before backpropagation (assuming loss is stored in a variable\ncalled losses):
\n
- \n
  Default backpropagate for FP32:
  \n
```
losses.backward()\n
```
  \n
- \n
  Scale loss and backpropagate with AMP:
  \n
```
with optimizer.scale_loss(losses) as scaled_losses:\n    scaled_losses.backward()\n
```
  \n
\n

Enabling TF32

TensorFloat-32 (TF32) is the new math mode in NVIDIA A100 GPUs for handling the matrix math also called tensor operations. TF32 running on Tensor Cores in A100 GPUs can provide up to 10x speedups compared to single-precision floating-point math (FP32) on Volta GPUs.

TF32 Tensor Cores can speed up networks using FP32, typically with no loss of accuracy. It is more robust than FP16 for models which require high dynamic range for weights or activations.

For more information, refer to the TensorFloat-32 in the A100 GPU Accelerates AI Training, HPC up to 20x blog post.

TF32 is supported in the NVIDIA Ampere GPU architecture and is enabled by default.

","resourceVersions",[230,231],"resourceVersionsTotal","hasMoreVersions","versionMetadata",{"_209":28,"_210":-5,"_52":115,"_112":-5,"_116":211,"_118":-5,"_212":114,"_123":16,"_120":34,"_213":-5,"_119":34,"_214":215,"_216":89,"_217":218,"_219":89,"_220":221,"_222":89,"_223":224,"_225":89,"_226":227,"_228":89,"_121":229},"versionNotFound","isLoggedIn","latestVersionPath","/orgs/nvidia/dle/resources/tts_taco2-wg_pyt/-","-","imageReference","cliDownloadCommand","ngc registry resource download-version nvidia/dle/tts_taco2-wg_pyt:20.06.0","helmFetchCommand","viewAllVersionsPath","/orgs/nvidia/dle/resources/tts_taco2-wg_pyt/-/version-history","accessCheckResult",{"_208":89},"hasRelatedCollections","context",{"_204":205,"_206":207},1784419599802,[],[],[197,198,199,200,201,202,203],{"_48":62,"_50":72,"_52":64,"_54":65,"_56":73,"_58":59},{"_48":62,"_50":71,"_52":145,"_54":-5,"_56":-5,"_58":59},{"_48":62,"_50":69,"_52":64,"_54":65,"_56":70,"_58":59},{"_48":62,"_50":67,"_52":64,"_54":65,"_56":68,"_58":59},{"_48":62,"_50":63,"_52":64,"_54":65,"_56":66,"_58":59},{"_48":60,"_50":61,"_52":53,"_54":55,"_56":57,"_58":59},{"_48":49,"_50":51,"_52":53,"_54":55,"_56":57,"_58":59},"orgName","nvidia","teamName","dle","allowed","versionId","description",["D",1663857867598],"totalSizeInBytes","malwareScanStatus","releaseNotes","### Changelog\n\nJune 2020\n* Updated performance tables to include A100 results\n\nMarch 2020\n* Added Tacotron 2 and WaveGlow inference using TensorRT Inference Server with custom TensorRT backend in `trtis_cpp`\n* Added Conversational AI demo script in `notebooks/conversationalai`\n* Fixed loading CUDA RNG state in `load_checkpoint()` function in `train.py`\n* Fixed FP16 export to TensorRT in `trt/README.md`\n\nJanuary 2020\n* Updated batch sizes and performance results for Tacotron 2.\n\nDecember 2019\n* Added export and inference scripts for TensorRT. See [Tacotron2 TensorRT README](https://raw.githubusercontent.com/NVIDIA/DeepLearningExamples/master/PyTorch/SpeechSynthesis/Tacotron2//trt/README.md).\n\nNovember 2019\n* Implemented training resume from checkpoint\n* Added notebook for running Tacotron 2 and WaveGlow in TRTIS.\n\nOctober 2019\n* Tacotron 2 inference with torch.jit.script\n\nSeptember 2019\n* Introduced inference statistics\n\nAugust 2019\n* Fixed inference results\n* Fixed initialization of Batch Normalization\n\nJuly 2019\n* Changed measurement units for Tacotron 2 training and inference performance\nbenchmarks from input tokes per second to output mel-spectrograms per second\n* Introduced batched inference\n* Included warmup in the inference script\n\nJune 2019\n* AMP support\n* Data preprocessing for Tacotron 2 training\n* Fixed dropouts on LSTMCells\n\nMarch 2019\n* Initial release\n\n### Known issues\n\nThere are no known issues in this release.","hasReleaseNotes","setup","The following section lists the requirements in order to start training the\nTacotron 2 and WaveGlow models.\n\n### Requirements\n\nThis repository contains Dockerfile which extends the PyTorch NGC container\nand encapsulates some dependencies. Aside from these dependencies, ensure you\nhave the following components:\n\n- [NVIDIA Docker](https://github.com/NVIDIA/nvidia-docker)\n- [PyTorch 20.06-py3 NGC container](https://ngc.nvidia.com/registry/nvidia-pytorch)\nor newer\n- Supported GPUs:\n - [NVIDIA Volta](https://www.nvidia.com/en-us/data-center/volta-gpu-architecture/)\n - [NVIDIA Turing](https://www.nvidia.com/en-us/geforce/turing/)\n - [NVIDIA Ampere architecture](https://www.nvidia.com/en-us/data-center/nvidia-ampere-gpu-architecture/)\n\nFor more information about how to get started with NGC containers, see the\nfollowing sections from the NVIDIA GPU Cloud Documentation and the Deep Learning\nDocumentation:\n\n- [Getting Started Using NVIDIA GPU Cloud](https://docs.nvidia.com/ngc/ngc-getting-started-guide/index.html)\n- [Accessing And Pulling From The NGC Container Registry](https://docs.nvidia.com/deeplearning/frameworks/user-guide/index.html#accessing_registry)\n- [Running PyTorch](https://docs.nvidia.com/deeplearning/frameworks/pytorch-release-notes/running.html#running)\n\nFor those unable to use the PyTorch NGC container, to set up the required\nenvironment or create your own container, see the versioned\n[NVIDIA Container Support Matrix](https://docs.nvidia.com/deeplearning/frameworks/support-matrix/index.html).","hasSetup","quickStartGuide","To train your model using mixed precision with Tensor Cores or using FP32,\nperform the following steps using the default parameters of the Tacrotron 2\nand WaveGlow model on the [LJ Speech](https://keithito.com/LJ-Speech-Dataset/)\ndataset.\n\n1. Clone the repository.\n ```bash\n git clone https://github.com/NVIDIA/DeepLearningExamples.git\n cd DeepLearningExamples/PyTorch/SpeechSynthesis/Tacotron2\n ```\n\n2. Download and preprocess the dataset.\nUse the `./scripts/prepare_dataset.sh` download script to automatically\ndownload and preprocess the training, validation and test datasets. To run\nthis script, issue:\n ```bash\n bash scripts/prepare_dataset.sh\n ```\n\n Data is downloaded to the `./LJSpeech-1.1` directory (on the host). The\n `./LJSpeech-1.1` directory is mounted to the `/workspace/tacotron2/LJSpeech-1.1`\n location in the NGC container.\n\n3. Build the Tacotron 2 and WaveGlow PyTorch NGC container.\n ```bash\n bash scripts/docker/build.sh\n ```\n\n4. Start an interactive session in the NGC container to run training/inference.\nAfter you build the container image, you can start an interactive CLI session with:\n\n ```bash\n bash scripts/docker/interactive.sh\n ```\n\n The `interactive.sh` script requires that the location on the dataset is specified.\n For example, `LJSpeech-1.1`. To preprocess the datasets for Tacotron 2 training, use\n the `./scripts/prepare_mels.sh` script:\n ```bash\n bash scripts/prepare_mels.sh\n ```\n\n The preprocessed mel-spectrograms are stored in the `./LJSpeech-1.1/mels` directory.\n\n5. Start training.\nTo start Tacotron 2 training, run:\n ```bash\n bash scripts/train_tacotron2.sh\n ```\n\n To start WaveGlow training, run:\n ```bash\n bash scripts/train_waveglow.sh\n ```\n\n6. Start validation/evaluation.\nEnsure your loss values are comparable to those listed in the table in the\n[Results](#results) section. For both models, the loss values are stored in the `./output/nvlog.json` log file.\n\n After you have trained the Tacotron 2 and WaveGlow models, you should get\n audio results similar to the\n samples in the `./audio` folder. For details about generating audio, see the\n [Inference process](#inference-process) section below.\n\n The training scripts automatically run the validation after each training\n epoch. The results from the validation are printed to the standard output\n (`stdout`) and saved to the log files.\n\n7. Start inference.\nAfter you have trained the Tacotron 2 and WaveGlow models, you can perform\ninference using the respective checkpoints that are passed as `--tacotron2`\nand `--waveglow` arguments. Tacotron2 and WaveGlow checkpoints can also be downloaded from NGC:\n\n https://ngc.nvidia.com/catalog/models/nvidia:tacotron2pyt_fp16/files?version=3\n\n https://ngc.nvidia.com/catalog/models/nvidia:waveglow256pyt_fp16/files?version=2\n\n To run inference issue:\n\n ```bash\n python inference.py --tacotron2 --waveglow --wn-channels 256 -o output/ -i phrases/phrase.txt --fp16\n ```\n\n The speech is generated from lines of text in the file that is passed with\n `-i` argument. The number of lines determines inference batch size. To run\n inference in mixed precision, use the `--fp16` flag. The output audio will\n be stored in the path specified by the `-o` argument.\n\n You can also run inference on CPU with TorchScript by adding flag --cpu:\n ```bash\n export CUDA_VISIBLE_DEVICES=\n ```\n ```bash\n python inference.py --tacotron2 --waveglow --wn-channels 256 --cpu -o output/ -i phrases/phrase.txt\n ```","hasQuickStartGuide","advanced","The following sections provide greater details of the dataset, running\ntraining and inference, and the training results.\n\n### Scripts and sample code\n\nThe sample code for Tacotron 2 and WaveGlow has scripts specific to a\nparticular model, located in directories `./tacotron2` and `./waveglow`, as well as scripts common to both\nmodels, located in the `./common` directory. The model-specific scripts are as follows:\n\n* `/model.py` - the model architecture, definition of forward and\ninference functions\n* `/arg_parser.py` - argument parser for parameters specific to a\ngiven model\n* `/data_function.py` - data loading functions\n* `/loss_function.py` - loss function for the model\n\nThe common scripts contain layer definitions common to both models\n(`common/layers.py`), some utility scripts (`common/utils.py`) and scripts\nfor audio processing (`common/audio_processing.py` and `common/stft.py`). In\nthe root directory `./` of this repository, the `./run.py` script is used for\ntraining while inference can be executed with the `./inference.py` script. The\nscripts `./models.py`, `./data_functions.py` and `./loss_functions.py` call\nthe respective scripts in the `` directory, depending on what\nmodel is trained using the `run.py` script.\n\n### Parameters\n\nIn this section, we list the most important hyperparameters and command-line arguments,\ntogether with their default values that are used to train Tacotron 2 and\nWaveGlow models.\n\n#### Shared parameters\n\n* `--epochs` - number of epochs (Tacotron 2: 1501, WaveGlow: 1001)\n* `--learning-rate` - learning rate (Tacotron 2: 1e-3, WaveGlow: 1e-4)\n* `--batch-size` - batch size (Tacotron 2 FP16/FP32: 104/48, WaveGlow FP16/FP32: 10/4)\n* `--amp` - use mixed precision training\n* `--cpu` - use CPU with TorchScript for inference\n\n#### Shared audio/STFT parameters\n\n* `--sampling-rate` - sampling rate in Hz of input and output audio (22050)\n* `--filter-length` - (1024)\n* `--hop-length` - hop length for FFT, i.e., sample stride between consecutive FFTs (256)\n* `--win-length` - window size for FFT (1024)\n* `--mel-fmin` - lowest frequency in Hz (0.0)\n* `--mel-fmax` - highest frequency in Hz (8.000)\n\n#### Tacotron 2 parameters\n\n* `--anneal-steps` - epochs at which to anneal the learning rate (500 1000 1500)\n* `--anneal-factor` - factor by which to anneal the learning rate (FP16/FP32: 0.3/0.1)\n\n#### WaveGlow parameters\n\n* `--segment-length` - segment length of input audio processed by the neural network (8000)\n* `--wn-channels` - number of residual channels in the coupling layer networks (512)\n\n### Command-line options\n\nTo see the full list of available options and their descriptions, use the `-h`\nor `--help` command line option, for example:\n```bash\npython train.py --help\n```\n\nThe following example output is printed when running the sample:\n\n```bash\nBatch: 7/260 epoch 0\n:::NVLOGv0.2.2 Tacotron2_PyT 1560936205.667271376 (/workspace/tacotron2/dllogger/logger.py:251) train_iter_start: 7\n:::NVLOGv0.2.2 Tacotron2_PyT 1560936207.209611416 (/workspace/tacotron2/dllogger/logger.py:251) train_iteration_loss: 5.415428161621094\n:::NVLOGv0.2.2 Tacotron2_PyT 1560936208.705905914 (/workspace/tacotron2/dllogger/logger.py:251) train_iter_stop: 7\n:::NVLOGv0.2.2 Tacotron2_PyT 1560936208.706479311 (/workspace/tacotron2/dllogger/logger.py:251) train_iter_items/sec: 8924.00136085362\n:::NVLOGv0.2.2 Tacotron2_PyT 1560936208.706998110 (/workspace/tacotron2/dllogger/logger.py:251) iter_time: 3.0393316745758057\nBatch: 8/260 epoch 0\n:::NVLOGv0.2.2 Tacotron2_PyT 1560936208.711485624 (/workspace/tacotron2/dllogger/logger.py:251) train_iter_start: 8\n:::NVLOGv0.2.2 Tacotron2_PyT 1560936210.236668825 (/workspace/tacotron2/dllogger/logger.py:251) train_iteration_loss: 5.516331672668457\n```\n\n### Getting the data\n\nThe Tacotron 2 and WaveGlow models were trained on the LJSpeech-1.1 dataset.\nThis repository contains the `./scripts/prepare_dataset.sh` script which will automatically download and extract the whole dataset. By default, data will be extracted to the `./LJSpeech-1.1` directory. The dataset directory contains a `README` file, a `wavs` directory with all audio samples, and a file `metadata.csv` that contains audio file names and the corresponding transcripts.\n\n#### Dataset guidelines\n\nThe LJSpeech dataset has 13,100 clips that amount to about 24 hours of speech. Since the original dataset has all transcripts in the `metadata.csv` file, in this repository we provide file lists in the `./filelists` directory that determine training and validation subsets; `ljs_audio_text_train_filelist.txt` is a test set used as a training dataset and `ljs_audio_text_val_filelist.txt` is a test set used as a validation dataset.\n\n#### Multi-dataset\n\nTo use datasets different than the default LJSpeech dataset:\n\n1. Prepare a directory with all audio files and pass it to the `--dataset-path` command-line option.\n\n2. Add two text files containing file lists: one for the training subset (`--training-files`) and one for the validation subset (`--validation files`).\nThe structure of the filelists should be as follows:\n ```bash\n `