[{"_1":2,"_3":-5,"_4":-5},"loaderData",{"_5":6,"_7":8,"_9":10,"_11":12},"actionData","errors","root",{"_68":295,"_296":297,"_298":299,"_300":301,"_302":303,"_304":26,"_305":306,"_307":308,"_309":26,"_310":26,"_311":26,"_312":26,"_313":314,"_315":-5,"_316":317,"_318":319,"_320":321,"_322":95,"_323":97,"_324":325,"_89":-5,"_326":26,"_327":-5,"_328":329,"_118":330,"_331":332,"_333":334,"_335":336,"_337":338,"_339":26,"_21":22,"_23":340,"_25":26,"_27":341,"_29":342,"_31":343},"routes/orgs/$orgName/$teamName/$resourceType/$resourceId/$version/route",{"_104":105,"_17":106,"_107":108,"_109":110,"_111":26,"_112":113,"_114":26,"_115":26,"_116":117,"_118":14,"_119":-5,"_120":121,"_122":-5,"_123":124,"_125":126,"_127":26,"_128":129,"_21":22,"_23":72,"_25":26,"_27":130,"_29":131,"_31":132},"routes/orgs/$orgName/$teamName/$resourceType/$resourceId/$version/_access/route",{"_65":66,"_67":-5,"_68":69,"_70":71,"_21":22,"_23":72,"_25":26,"_27":73,"_29":74,"_31":75},"routes/orgs/$orgName/$teamName/$resourceType/$resourceId/$version/release-notes/route",{"_13":14,"_15":16,"_17":18,"_19":20,"_21":22,"_23":24,"_25":26,"_27":28,"_29":30,"_31":32},"versionId","1","releaseNotes","### Changelog\n\nMay 2020\n- Initial release\n\n### Known issues\n\nThere are no known issues in this release.","html","

Changelog

May 2020

Initial release

Known issues

There are no known issues in this release.

","resourceType","resources","traceId","41d1ae68eb5bbf5ff76d3cbbca7877ac","traceStartTime",1783860652201,"isStaleResponse",false,"notifications",[],"toasts",[],"apiCalls",[33,34,35,36,37,38],{"_39":54,"_41":63,"_43":56,"_45":57,"_47":64,"_49":50},{"_39":54,"_41":61,"_43":56,"_45":57,"_47":62,"_49":50},{"_39":54,"_41":59,"_43":56,"_45":57,"_47":60,"_49":50},{"_39":54,"_41":55,"_43":56,"_45":57,"_47":58,"_49":50},{"_39":51,"_41":52,"_43":44,"_45":46,"_47":53,"_49":50},{"_39":40,"_41":42,"_43":44,"_45":46,"_47":48,"_49":50},"apiTag","subscription-api","path","/v2/users/me/subscriptions","status","error","httpStatus",401,"durationMs",3,"retryCount",0,"oms-api","/v2/users/me",4,"models-api","/v2/resources/org/nvidia/fastpitch_for_pytorch/collections?page-number=0&page-size=1","success",200,62,"/v2/resources/nvidia/fastpitch_for_pytorch/versions/1",56,"/v2/resources/nvidia/fastpitch_for_pytorch/versions?page-size=100&page-number=0&sort-order=CREATED_DATE_DESC",64,"/v2/resources/nvidia/fastpitch_for_pytorch?resolve-labels=true&remove-unresolved-labels=true",130,"canDownload",true,"accessDeniedReason","appConfig",{"_87":88,"_89":90},"debugAccessInfo",{"_83":66,"_84":85,"_86":26},1783860652194,[],[],[76,77,78,79,80,81],{"_39":54,"_41":63,"_43":56,"_45":57,"_47":64,"_49":50},{"_39":54,"_41":61,"_43":56,"_45":57,"_47":62,"_49":50},{"_39":54,"_41":59,"_43":56,"_45":57,"_47":60,"_49":50},{"_39":54,"_41":55,"_43":56,"_45":57,"_47":58,"_49":50},{"_39":51,"_41":52,"_43":82,"_45":-5,"_47":-5,"_49":50},{"_39":40,"_41":42,"_43":44,"_45":46,"_47":48,"_49":50},"pending","canGuestAccess","requiredProducts",[],"hasLicenseRequirement","urls",{"_92":93,"_94":95,"_96":97,"_98":99,"_100":101,"_102":103},"user",{"_91":26},"loggedIn","build","https://build.nvidia.com","ngcApi","https://api.ngc.nvidia.com","ngcHelm","https://helm.ngc.nvidia.com","ngcRegistry","https://registry.ngc.nvidia.com","orgUi","https://org.ngc.nvidia.com","ngcUi","https://ngc.nvidia.com","resource",{"_189":226,"_227":228,"_19":20,"_229":230,"_139":140,"_141":-7,"_231":232,"_143":233,"_234":235,"_148":236,"_237":238,"_239":222,"_240":-7,"_241":-7,"_242":243,"_244":245,"_246":66,"_247":248,"_249":-5,"_250":-7,"_251":-7,"_252":-7,"_253":-7,"_254":-7,"_255":-7,"_256":257,"_258":259,"_260":261,"_262":263,"_264":265,"_266":26},"

FastPitch is one of two major components in a neural, text-to-speech (TTS) system:

a mel-spectrogram generator such as FastPitch or Tacotron 2, and
a waveform synthesizer such as WaveGlow (see NVIDIA example code).

Such two-component TTS system is able to synthesize natural sounding speech from raw transcripts.

The FastPitch model generates mel-spectrograms and predicts a pitch contour from raw input text.\nIn version 1.1, it does not need any pre-trained aligning model to bootstrap from.\nIt allows to exert additional control over the synthesized utterances, such as:

modify the pitch contour to control the prosody,
increase or decrease the fundamental frequency in a naturally sounding way, that preserves the perceived identity of the speaker,
alter the rate of speech,
adjust the energy,
specify input as graphemes or phonemes,
switch speakers when the model has been trained with data from multiple speakers.\nSome of the capabilities of FastPitch are presented on the website with samples.

Speech synthesized with FastPitch has state-of-the-art quality, and does not suffer from missing/repeating phrases like Tacotron 2 does.\nThis is reflected in Mean Opinion Scores (details).

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n

tr:nth-child(odd)]:bg-surface-sunken\">

Model	Mean Opinion Score (MOS)
Tacotron 2	3.946 ± 0.134
FastPitch 1.0	4.080 ± 0.133

The current version of the model offers even higher quality, as reflected\nin the pairwise preference scores (details).

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n

tr:nth-child(odd)]:bg-surface-sunken\">

Model	Average preference
FastPitch 1.0	0.435 ± 0.068
FastPitch 1.1	0.565 ± 0.068

The FastPitch model is based on the FastSpeech model. The main differences between FastPitch and FastSpeech are that FastPitch:

no dependence on external aligner (Transformer TTS, Tacotron 2); in version 1.1, FastPitch aligns audio to transcriptions by itself as in One TTS Alignment To Rule Them All,
explicitly learns to predict the pitch contour,
pitch conditioning removes harsh sounding artifacts and provides faster convergence,
no need for distilling mel-spectrograms with a teacher model,
capabilities to train a multi-speaker model.

The FastPitch model is similar to FastSpeech2, which has been developed concurrently. FastPitch averages pitch/energy values over input tokens, and treats energy as optional.

FastPitch is trained on a publicly\navailable LJ Speech dataset.

This model is trained with mixed precision using Tensor Cores on Volta, Turing, and the NVIDIA Ampere GPU architectures. Therefore, researchers can get results from 2.0x to 2.7x faster than training without Tensor Cores, while experiencing the benefits of mixed precision training. This model is tested against each NGC monthly container release to ensure consistent accuracy and performance over time.

Model architecture

FastPitch is a fully feedforward Transformer model that predicts mel-spectrograms\nfrom raw text (Figure 1). The entire process is parallel, which means that all input letters are processed simultaneously to produce a full mel-spectrogram in a single forward pass.

\n $\"FastPitch$ \n

\n Figure 1. Architecture of FastPitch (source). The model is composed of a bidirectional Transformer backbone (also known as a Transformer encoder), a pitch predictor, and a duration predictor. After passing through the first *N* Transformer blocks, encoding, the signal is augmented with pitch information and discretely upsampled. Then it goes through another set of *N* Transformer blocks, with the goal of\nsmoothing out the upsampled signal, and constructing a mel-spectrogram.\n \n

Default configuration

The FastPitch model supports multi-GPU and mixed precision training with dynamic loss\nscaling (see Apex code\nhere),\nas well as mixed precision inference.

The following features were implemented in this model:

data-parallel multi-GPU training,
dynamic loss scaling with backoff for Tensor Cores (mixed precision)\ntraining,
gradient accumulation for reproducible results regardless of the number of GPUs.

Pitch contours and mel-spectrograms can be generated on-line during training.\nTo speed-up training, those could be generated during the pre-processing step and read\ndirectly from the disk during training. For more information on data pre-processing refer to Dataset guidelines\n and the paper.

Feature support matrix

The following features are supported by this model.

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n

tr:nth-child(odd)]:bg-surface-sunken\">

Feature	FastPitch
Automatic mixed precision (AMP)	Yes
Distributed data parallel (DDP)	Yes

Features

Automatic Mixed Precision (AMP) - This implementation uses native PyTorch AMP\nimplementation of mixed precision training. It allows us to use FP16 training\nwith FP32 master weights by modifying just a few lines of code.

DistributedDataParallel (DDP) - The model uses PyTorch Lightning implementation\nof distributed data parallelism at the module level which can run across\nmultiple machines.

Mixed precision training

Mixed precision is the combined use of different numerical precisions in a computational method. Mixed precision training offers significant computational speedup by performing operations in half-precision format while storing minimal information in single-precision to retain as much information as possible in critical parts of the network. Since the introduction of Tensor Cores in Volta, and following with both the Turing and Ampere architectures, significant training speedups are experienced by switching to mixed precision -- up to 3x overall speedup on the most arithmetically intense model architectures. Using mixed precision training requires two steps:

Porting the model to use the FP16 data type where appropriate.
Adding loss scaling to preserve small gradient values.

The ability to train deep learning networks with lower precision was introduced in the Pascal architecture and first supported in CUDA 8 in the NVIDIA Deep Learning SDK.

For information about:

How to train using mixed precision, see the Mixed Precision Training paper and Training With Mixed Precision documentation.
Techniques used for mixed precision training, see the Mixed-Precision Training of Deep Neural Networks blog.
APEX tools for mixed precision training, see the NVIDIA Apex: Tools for Easy Mixed-Precision Training in PyTorch.

Enabling mixed precision

For training and inference, mixed precision can be enabled by adding the --amp flag.\nMixed precision is using native PyTorch implementation.

Enabling TF32

TensorFloat-32 (TF32) is the new math mode in NVIDIA A100 GPUs for handling the matrix math also called tensor operations. TF32 running on Tensor Cores in A100 GPUs can provide up to 10x speedups compared to single-precision floating-point math (FP32) on Volta GPUs.

TF32 Tensor Cores can speed up networks using FP32, typically with no loss of accuracy. It is more robust than FP16 for models which require high dynamic range for weights or activations.

For more information, refer to the TensorFloat-32 in the A100 GPU Accelerates AI Training, HPC up to 20x blog post.

TF32 is supported in the NVIDIA Ampere GPU architecture and is enabled by default.

Glossary

Character duration\nThe time during which a character is being articulated. Could be measured in milliseconds, mel-spectrogram frames, etc. Some characters are not pronounced, and thus have 0 duration.

Fundamental frequency\nThe lowest vibration frequency of a periodic soundwave, for example, produced by a vibrating instrument. It is perceived as the loudest. In the context of speech, it refers to the frequency of vibration of vocal chords. Abbreviated as f0.

Pitch\nA perceived frequency of vibration of music or sound.

Transformer\nThe paper Attention Is All You Need introduces a novel architecture called Transformer, which repeatedly applies the attention mechanism. It transforms one sequence into another.

","resourceVersions",[171,172,173,174,175,176,177,178,179,180,181,182,183,184,185,186,187,188],"resourceVersionsTotal",17,"hasMoreVersions","versionMetadata",{"_13":14,"_143":-5,"_43":144,"_145":-5,"_146":147,"_148":-5,"_149":150,"_151":152,"_153":26,"_154":-5,"_155":26,"_15":16,"_156":66,"_157":158,"_159":66,"_160":161,"_162":66,"_163":164,"_165":66,"_166":167,"_168":66,"_169":170},"versionNotFound","isLoggedIn","latestVersionPath","/orgs/nvidia/-/resources/fastpitch_for_pytorch/-","version","imageReference","cliDownloadCommand","ngc registry resource download-version nvidia/fastpitch_for_pytorch:1","helmFetchCommand","viewAllVersionsPath","/orgs/nvidia/-/resources/fastpitch_for_pytorch/1/version-history","accessCheckResult",{"_142":66},"hasRelatedCollections","context",{"_139":140,"_141":-7},[],[],[133,134,135,136,137,138],{"_39":54,"_41":63,"_43":56,"_45":57,"_47":64,"_49":50},{"_39":54,"_41":61,"_43":56,"_45":57,"_47":62,"_49":50},{"_39":54,"_41":59,"_43":56,"_45":57,"_47":60,"_49":50},{"_39":54,"_41":55,"_43":56,"_45":57,"_47":58,"_49":50},{"_39":51,"_41":52,"_43":82,"_45":-5,"_47":-5,"_49":50},{"_39":40,"_41":42,"_43":44,"_45":46,"_47":48,"_49":50},"orgName","nvidia","teamName","allowed","description","UPLOAD_COMPLETE","releaseType","createdDate",["D",1592342572576],"updatedDate","totalSizeInBytes",7493489,"totalFileCount",80,"isSigned","malwareScanStatus","isScanned","hasReleaseNotes","setup","The following section lists the requirements that you need to meet in order to start training the FastPitch model.\n\n### Requirements\n\nThis repository contains Dockerfile which extends the PyTorch NGC container and encapsulates some dependencies. Aside from these dependencies, ensure you have the following components:\n- [NVIDIA Docker](https://github.com/NVIDIA/nvidia-docker)\n- [PyTorch 21.05-py3 NGC container](https://ngc.nvidia.com/registry/nvidia-pytorch)\nor newer\n- supported GPUs:\n - [NVIDIA Volta architecture](https://www.nvidia.com/en-us/data-center/volta-gpu-architecture/)\n - [NVIDIA Turing architecture](https://www.nvidia.com/en-us/geforce/turing/)\n - [NVIDIA Ampere architecture](https://www.nvidia.com/en-us/data-center/nvidia-ampere-gpu-architecture/)\n\nFor more information about how to get started with NGC containers, see the following sections from the NVIDIA GPU Cloud Documentation and the Deep Learning Documentation:\n- [Getting Started Using NVIDIA GPU Cloud](https://docs.nvidia.com/ngc/ngc-getting-started-guide/index.html)\n- [Accessing And Pulling From The NGC Container Registry](https://docs.nvidia.com/deeplearning/frameworks/user-guide/index.html#accessing_registry)\n- [Running PyTorch](https://docs.nvidia.com/deeplearning/frameworks/pytorch-release-notes/running.html#running)\n\nFor those unable to use the PyTorch NGC container, to set up the required environment or create your own container, see the versioned [NVIDIA Container Support Matrix](https://docs.nvidia.com/deeplearning/frameworks/support-matrix/index.html).","hasSetup","quickStartGuide","To train your model using mixed or TF32 precision with Tensor Cores or using FP32, perform the following steps using the default parameters of the FastPitch model on the LJSpeech 1.1 dataset. For the specifics concerning training and inference, see the [Advanced](#advanced) section. Pre-trained FastPitch models are available for download on [NGC](https://ngc.nvidia.com/catalog/models?query=FastPitch&quickFilter=models).\n\n1. Clone the repository.\n ```bash\n git clone https://github.com/NVIDIA/DeepLearningExamples.git\n cd DeepLearningExamples/PyTorch/SpeechSynthesis/FastPitch\n ```\n\n2. Build and run the FastPitch PyTorch NGC container.\n\n By default the container will use all available GPUs.\n ```bash\n bash scripts/docker/build.sh\n bash scripts/docker/interactive.sh\n ```\n\n3. Download and preprocess the dataset.\n\n Use the scripts to automatically download and preprocess the training, validation and test datasets:\n ```bash\n bash scripts/download_dataset.sh\n bash scripts/prepare_dataset.sh\n ```\n\n The data is downloaded to the `./LJSpeech-1.1` directory (on the host). The\n `./LJSpeech-1.1` directory is mounted under the `/workspace/fastpitch/LJSpeech-1.1`\n location in the NGC container. The complete dataset has the following structure:\n ```bash\n ./LJSpeech-1.1\n |-- mels # (optional) Pre-calculated target mel-spectrograms; may be calculated on-line\n |-- metadata.csv # Mapping of waveforms to utterances\n |-- pitch # Fundamental frequency countours for input utterances; may be calculated on-line\n |-- README\n |-- wavs # Raw waveforms\n ```\n\n4. Start training.\n ```bash\n bash scripts/train.sh\n ```\n The training will produce a FastPitch model capable of generating mel-spectrograms from raw text.\n It will be serialized as a single `.pt` checkpoint file, along with a series of intermediate checkpoints.\n The script is configured for 8x GPU with at least 16GB of memory. Consult [Training process](#training-process) and [example configs](#training-performance-benchmark) to adjust to a different configuration or enable Automatic Mixed Precision.\n\n5. Start validation/evaluation.\n\n Ensure your training loss values are comparable to those listed in the table in the\n [Results](#results) section. Note that the validation loss is evaluated with ground truth durations for letters (not the predicted ones). The loss values are stored in the `./output/nvlog.json` log file, `./output/{train,val,test}` as TensorBoard logs, and printed to the standard output (`stdout`) during training.\n The main reported loss is a weighted sum of losses for mel-, pitch-, and duration- predicting modules.\n\n The audio can be generated by following the [Inference process](#inference-process) section below.\n The synthesized audio should be similar to the samples in the `./audio` directory.\n\n6. Start inference/predictions.\n\n To synthesize audio, you will need a WaveGlow model, which generates waveforms based on mel-spectrograms generated with FastPitch. By now, a pre-trained model should have been downloaded by the `scripts/download_dataset.sh` script. Alternatively, to train WaveGlow from scratch, follow the instructions in [NVIDIA/DeepLearningExamples/Tacotron2](https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/SpeechSynthesis/Tacotron2) and replace the checkpoint in the `./pretrained_models/waveglow` directory.\n\n You can perform inference using the respective `.pt` checkpoints that are passed as `--fastpitch`\n and `--waveglow` arguments:\n ```bash\n python inference.py \\\n --cuda \\\n --fastpitch output/ \\\n --energy-conditioning \\\n --waveglow pretrained_models/waveglow/ \\\n --wn-channels 256 \\\n -i phrases/devset10.tsv \\\n -o output/wavs_devset10\n ```\n\n The speech is generated from a file passed with the `-i` argument, with one utterance per line:\n ```bash\n `|`\n ```\nTo run\n inference in mixed precision, use the `--amp` flag. The output audio will\n be stored in the path specified by the `-o` argument. Consult the `inference.py` to learn more options, such as setting the batch size.","hasQuickStartGuide","advanced","The following sections provide greater details of the dataset, running training and inference, and the training results.\n\n### Scripts and sample code\n\nThe repository holds code for FastPitch (training and inference) and WaveGlow (inference only).\nThe code specific to a particular model is located in that model's directory - `./fastpitch` and `./waveglow` - and common functions live in the `./common` directory. The model-specific scripts are as follows:\n\n* `/model.py` - the model architecture, definition of forward and\ninference functions\n* `/arg_parser.py` - argument parser for parameters specific to a\ngiven model\n* `/data_function.py` - data loading functions\n* `/loss_function.py` - loss function for the model\n\nIn the root directory `./` of this repository, the `./train.py` script is used for\ntraining while inference can be executed with the `./inference.py` script. The\nscript `./models.py` is used to construct a model of requested type and properties.\n\nThe repository is structured similarly to the [NVIDIA Tacotron2 Deep Learning example](https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/SpeechSynthesis/Tacotron2), so that they could be combined in more advanced use cases.\n\n### Parameters\n\nIn this section, we list the most important hyperparameters and command-line arguments,\ntogether with their default values that are used to train FastPitch.\n\n* `--epochs` - number of epochs (default: 1500)\n* `--learning-rate` - learning rate (default: 0.1)\n* `--batch-size` - batch size for a single forward-backward step (default: 16)\n* `--grad-accumulation` - number of steps over which gradients are accumulated (default: 2)\n* `--amp` - use mixed precision training (default: disabled)\n* `--load-pitch-from-disk` - pre-calculated fundamental frequency values, estimated before training, are loaded from the disk during training (default: enabled)\n* `--energy-conditioning` - enables additional conditioning on energy (default: enabled)\n* `--p-arpabet` - probability of choosing phonemic over graphemic representation for every word, if available (default: 1.0)\n\n### Command-line options\n\nTo see the full list of available options and their descriptions, use the `-h`\nor `--help` command line option, for example:\n```bash\npython train.py --help\n```\n\nThe following example output is printed when running the model:\n\n```bash\nDLL 2021-06-14 23:08:53.659718 - epoch 1 | iter 1/48 | loss 40.97 | mel loss 35.04 | kl loss 0.02240 | kl weight 0.01000 | 5730.98 frames/s | took 24.54 s | lrate 3.16e-06\nDLL 2021-06-14 23:09:28.449961 - epoch 1 | iter 2/48 | loss 41.07 | mel loss 35.12 | kl loss 0.02258 | kl weight 0.01000 | 4154.18 frames/s | took 34.79 s | lrate 6.32e-06\nDLL 2021-06-14 23:09:59.365398 - epoch 1 | iter 3/48 | loss 40.86 | mel loss 34.93 | kl loss 0.02252 | kl weight 0.01000 | 4589.15 frames/s | took 30.91 s | lrate 9.49e-06\n```\n\n### Getting the data\n\nThe FastPitch and WaveGlow models were trained on the LJSpeech-1.1 dataset.\nThe `./scripts/download_dataset.sh` script will automatically download and extract the dataset to the `./LJSpeech-1.1` directory.\n\n#### Dataset guidelines\n\nThe LJSpeech dataset has 13,100 clips that amount to about 24 hours of speech of a single, female speaker. Since the original dataset does not define a train/dev/test split of the data, we provide a split in the form of three file lists:\n```bash\n./filelists\n|-- ljs_audio_pitch_text_train_v3.txt\n|-- ljs_audio_pitch_text_test.txt\n|-- ljs_audio_pitch_text_val.txt\n```\n\nFastPitch predicts character durations just like [FastSpeech](https://arxiv.org/abs/1905.09263) does.\nFastPitch 1.1 aligns input symbols to output mel-spectrogram frames automatically and does not rely\non any external aligning model. FastPitch training can now be started on raw waveforms\nwithout any pre-processing: pitch values and mel-spectrograms will be calculated on-line.\n\nFor every mel-spectrogram frame, its fundamental frequency in Hz is estimated with\nthe Probabilistic YIN algorithm.\n\n

\n $\"Pitch$ \n

\n Figure 2. Pitch estimates for mel-spectrogram frames of phrase \"in being comparatively\"\n(in blue) averaged over characters (in red). Silent letters have duration 0 and are omitted.\n

\n\n#### Multi-dataset\n\nFollow these steps to use datasets different from the default LJSpeech dataset.\n\n1. Prepare a directory with .wav files.\n ```bash\n ./my_dataset\n |-- wavs\n ```\n\n2. Prepare filelists with transcripts and paths to .wav files. They define training/validation split of the data (test is currently unused):\n ```bash\n ./filelists\n |-- my-dataset_audio_text_train.txt\n |-- my-dataset_audio_text_val.txt\n ```\n Those filelists should list a single utterance per line as:\n ```bash\n `