The performance measurements in this document were conducted at the time of publication and may not reflect the performance achieved from NVIDIA's latest software release. For the most up-to-date performance measurements, go to NVIDIA Data Center Deep Learning Product Performance.

Benchmarking

The following section shows how to run benchmarks measuring the model performance in training and inference modes.

Training performance benchmark

To benchmark the training performance for a specific local (per-gpu) batch size <LBS>, with a specific number of GPUs <#GPUs> for a specific number of training iterations <ITER>, run:

bash run_wt103_{base,large}.sh train <#GPUs> --config trainbench --local_batch_size <LBS> --max_step <ITER> [--fp16]

• use the run_wt103_base.sh script to run the benchmark for the base model, and use the run_wt103_large.sh script to run the benchmark for the large model
• it's recommended to launch at least 500 training steps to get a reliable estimate of training performace.
• the --fp16 flag is optional, however, if it's specified, then the script launches mixed precision training with Tensor Cores. If the flag is not present, then the script launches FP32 training on NVIDIA Volta GPUs and TF32 training on NVIDIA Ampere GPUs.

For more information about the available options, refer to the Training process section.

The training script prints information in the following format:

(...)

| epoch   1 step      499 | batches    499 / 16802 | lr 4.990e-03 | ms/batch 219.9 | tok/s   27947 | loss  6.43 | ppl    620.80
| epoch   1 step      500 | batches    500 / 16802 | lr 5.000e-03 | ms/batch 221.4 | tok/s   27747 | loss  6.42 | ppl    611.70
-------------------------------------------------------------------------------
(...)
Training time: 1.81 minutes
Training throughput: 28508.91 tok/s

The last two lines contain information on the total training time and on the average training throughput measured in tokens per second.

Training performance benchmark for multi-node

To benchmark the multi-node training performance of the large model on a specific number of DGX-2H nodes <NODES> and a specific local batch size <LBS>, run:

For mixed precision:

FP16=1 LOCAL_BATCH_SIZE=<LBS> CONFIG=trainbench_multinode sbatch -N <NODES> run.sub train

For FP32:

LOCAL_BATCH_SIZE=<LBS> CONFIG=trainbench_multinode sbatch -N <NODES> run.sub train

Inference performance benchmark

The inference performance and accuracy benchmarks require a checkpoint from a trained model.

To benchmark the inference performance on a specific global batch size <BS> with a specific number of GPUs <#GPUs>, run:

For the base model:

bash run_wt103_base.sh eval <#GPUs> --model <CHECKPOINT> --batch_size <BS> --save_data [--fp16] [--type {pytorch, torchscript}]

For the large model:

bash run_wt103_large.sh eval <#GPUs> --model <CHECKPOINT> --batch_size <BS> --save_data [--fp16] [--type {pytorch, torchscript}]

The inference script prints information in the following format:

Evaluating with math fp16 type torchscript bsz 16 tgt_len 64 ext_len 0 mem_len 640 clamp_len 400
Time : 5.25s, 21.88ms/segment
====================================================================================================

| test loss  3.15 | test ppl    23.304
====================================================================================================
Throughput Avg: 46316.64 tok/s
Latency Avg: 22.09 ms
Latency 90%: 22.22 ms
Latency 95%: 22.25 ms
Latency 99%: 22.37 ms
====================================================================================================

The output contains information on the achieved test loss and test perplexity, average inference throughput (measured in tokens per second), average inference latency and latency at 90%, 95% and 99% confidence intervals (measured in milliseconds).

The scripts/inference_benchmark.sh benchmarking script is provided for convenience, it automatically launches TF32/FP32 and FP16 inference for various batch sizes.

Results

The following sections provide details on how we achieved our performance and accuracy in training and inference.

Training accuracy results

Training accuracy: NVIDIA DGX A100 (8x A100 40GB)

Base model

Our results were obtained by running the pytorch/run_wt103_base.sh training script in the pytorch-20.06-py3 NGC container on NVIDIA DGX A100 with 8x A100 40GB GPUs.

Large model

Our results were obtained by running the pytorch/run_wt103_large.sh training script in the pytorch-20.06-py3 NGC container on NVIDIA DGX A100 with 8x A100 40GB GPUs.

Training accuracy: NVIDIA DGX-1 (8x V100 16GB)

Base model

Our results were obtained by running the pytorch/run_wt103_base.sh training script in the pytorch-20.06-py3 NGC container on NVIDIA DGX-1 with 8x V100 16GB GPUs.

Large model

Our results were obtained by running the pytorch/run_wt103_large.sh training script in the pytorch-20.06-py3 NGC container on NVIDIA DGX-1 with 8x V100 16GB GPUs.

Training accuracy: NVIDIA DGX-2H (16x V100 32GB)

Base model

Our results were obtained by running the pytorch/run_wt103_base.sh training script in the pytorch-20.06-py3 NGC container on NVIDIA DGX-2H with 16x V100 32GB GPUs.

Large model

Our results were obtained by running the pytorch/run_wt103_large.sh training script in the pytorch-20.06-py3 NGC container on NVIDIA DGX-2H with 16x V100 32GB GPUs.

Training accuracy: 8x NVIDIA DGX-2H (16x V100 32GB)

Large model

Our results were obtained by running the pytorch/run.sub training script in the pytorch-20.06-py3 NGC container on 8x NVIDIA DGX-2H with 16x V100 32GB GPUs.

Training accuracy plots

Base model

Large model (single-node)

Large model (multi-node)

Training stability test

Base model

The Transformer-XL base model was trained for 40,000 training steps, starting from 16 different initial random seeds. After every 5,000 training steps, the model was evaluated on the validation dataset and validation perplexity was recorded. The training was performed in the pytorch-20.06-py3 NGC container on NVIDIA DGX A100 with 8x A100 40GB GPUs. The following table summarizes the perplexity of our validation dataset.

After training, the models were evaluated on the test dataset. The following table summarizes the final perplexity on the test set.

Large model (single-node)

The Transformer-XL large model was trained for 100,000 training steps, starting from 16 different initial random seeds. After every 10,000 training steps, the model was evaluated on the validation dataset and validation perplexity was recorded. The training was performed in the pytorch-20.06-py3 NGC container on NVIDIA DGX A100 with 8x A100 40GB GPUs. The following table summarizes the perplexity of our validation dataset.

After training, the models were evaluated on the test dataset. The following table summarizes the final perplexity on the test set.

Large model (multi-node)

The Transformer-XL large model was trained for 25,000 training steps, starting from 10 different initial random seeds. After every 1,000 training steps, the model was evaluated on the validation dataset and validation perplexity was recorded. The training was performed in the pytorch-20.06-py3 NGC container on 8x NVIDIA DGX-2H with 16x V100 32GB GPUs. The following table summarizes the perplexity of our validation dataset.

After training, the models were evaluated on the test dataset. The following table summarizes the final perplexity on the test set.

Training performance results

Training performance: NVIDIA DGX A100 (8x A100 40GB)

Base model

Our results were obtained by running the pytorch/run_wt103_base.sh training script in the pytorch-20.06-py3 NGC container on NVIDIA DGX A100 with 8x A100 40GB GPUs. Performance numbers (in tokens per second) were averaged over 500 training iterations.

To achieve these same results, follow the steps in the Quick Start Guide to download the dataset and setup the container, and then proceed to the Training performance benchmark section for instruction on how to launch the benchmark.

Large model

Our results were obtained by running the pytorch/run_wt103_large.sh training script in the pytorch-20.06-py3 NGC container on NVIDIA DGX A100 with 8x A100 40GB GPUs. Performance numbers (in tokens per second) were averaged over 500 training iterations.

To achieve these same results, follow the steps in the Quick Start Guide to download the dataset and setup the container, and then proceed to the Training performance benchmark section for instruction on how to launch the benchmark.

Training performance: NVIDIA DGX-1 (8x V100 16GB)

Base model

Our results were obtained by running the pytorch/run_wt103_base.sh training script in the pytorch-20.06-py3 NGC container on NVIDIA DGX-1 with 8x V100 16GB GPUs. Performance numbers (in tokens per second) were averaged over 500 training iterations.

To achieve these same results, follow the steps in the Quick Start Guide to download the dataset and setup the container, and then proceed to the Training performance benchmark section for instruction on how to launch the benchmark.

Large model

Our results were obtained by running the pytorch/run_wt103_large.sh training script in the pytorch-20.06-py3 NGC container on NVIDIA DGX-1 with 8x V100 16GB GPUs. Performance numbers (in tokens per second) were averaged over 500 training iterations.

To achieve these same results, follow the steps in the Quick Start Guide to download the dataset and setup the container, and then proceed to the Training performance benchmark section for instruction on how to launch the benchmark.

Training performance: NVIDIA DGX-2H (16x V100 32GB)

Base model

Our results were obtained by running the pytorch/run_wt103_base.sh training script in the pytorch-20.06-py3 NGC container on NVIDIA DGX-2H with 16x V100 32GB GPUs. Performance numbers (in tokens per second) were averaged over 500 training iterations.

To achieve these same results, follow the steps in the Quick Start Guide to download the dataset and setup the container, and then proceed to the Training performance benchmark section for instruction on how to launch the benchmark.

Large model

Our results were obtained by running the pytorch/run_wt103_large.sh training script in the pytorch-20.06-py3 NGC container on NVIDIA DGX-2H with 16x V100 32GB GPUs. Performance numbers (in tokens per second) were averaged over 500 training iterations.

To achieve these same results, follow the steps in the Quick Start Guide to download the dataset and setup the container, and then proceed to the Training performance benchmark section for instruction on how to launch the benchmark.

Training performance: 8x NVIDIA DGX-2H (16x V100 32GB)

Our results were obtained by running the pytorch/run.sub training script in the pytorch-20.06-py3 NGC container. Performance numbers (in tokens per second) were averaged over 500 training iterations.

Large model

To achieve these same results, follow the steps in the Quick Start Guide to download the dataset and then proceed to the Training performance benchmark for multi-node section for instruction on how to launch the multi-node performance benchmark. The numbers presented above were obtained with LOCAL_BATCH_SIZE=4.

Inference performance results

Inference performance: NVIDIA DGX A100 (1x A100 40GB)

Base model

Our results were obtained by running the pytorch/scripts/inference_benchmark.sh inferencing benchmarking script in the pytorch-20.06-py3 NGC container on NVIDIA DGX A100 with 1x A100 40GB GPU.

The command to launch the inference performance benchmark is provided in the Inference performance benchmark section.

FP16, pure Python

FP16, TorchScript

TF32, pure Python

TF32, TorchScript

To achieve these same results, follow the steps in the Quick Start Guide to download the dataset and setup the container, and then proceed to the Inference performance benchmark section for instruction on how to launch the benchmark.

Large model

Our results were obtained by running the pytorch/scripts/inference_benchmark.sh inferencing benchmarking script in the pytorch-20.06-py3 NGC container on NVIDIA DGX A100 with 1x A100 40GB GPU.

The command to launch the inference performance benchmark is provided in the Inference performance benchmark section.

FP16, pure Python

FP16, TorchScript

TF32, pure Python

TF32, TorchScript

To achieve these same results, follow the steps in the Quick Start Guide to download the dataset and setup the container, and then proceed to the Inference performance benchmark section for instruction on how to launch the benchmark.

Inference performance: NVIDIA DGX-1 (1x V100 16GB)

Base model

Our results were obtained by running the pytorch/scripts/inference_benchmark.sh inferencing benchmarking script in the pytorch-20.06-py3 NGC container on NVIDIA DGX-1 with 1x V100 16GB GPU.

The command to launch the inference performance benchmark is provided in the Inference performance benchmark section.

FP16, pure Python

FP16, TorchScript

FP32, pure Python

FP32, TorchScript

To achieve these same results, follow the steps in the Quick Start Guide to download the dataset and setup the container, and then proceed to the Inference performance benchmark section for instruction on how to launch the benchmark.

Large model

Our results were obtained by running the pytorch/scripts/inference_benchmark.sh inferencing benchmarking script in the pytorch-20.06-py3 NGC container on NVIDIA DGX-1 with 1x V100 16GB GPU.

The command to launch the inference performance benchmark is provided in the Inference performance benchmark section.

FP16, pure Python

FP16, TorchScript

FP32, pure Python

FP32, TorchScript

To achieve these same results, follow the steps in the Quick Start Guide to download the dataset and setup the container, and then proceed to the Inference performance benchmark section for instruction on how to launch the benchmark.

Inference performance: NVIDIA T4

Base model

Our results were obtained by running the pytorch/scripts/inference_benchmark.sh inferencing benchmarking script in the pytorch-20.06-py3 NGC container on NVIDIA T4.

The command to launch the inference performance benchmark is provided in the Inference performance benchmark section.

FP16, pure Python

FP16, TorchScript

FP32, pure Python

FP32, TorchScript

To achieve these same results, follow the steps in the Quick Start Guide to download the dataset and setup the container, and then proceed to the Inference performance benchmark section for instruction on how to launch the benchmark.

Large model

Our results were obtained by running the pytorch/scripts/inference_benchmark.sh inferencing benchmarking script in the pytorch-20.06-py3 NGC container on NVIDIA T4.

The command to launch the inference performance benchmark is provided in the Inference performance benchmark section.

FP16, pure Python

FP16, TorchScript

FP32, pure Python

FP32, TorchScript

To achieve these same results, follow the steps in the Quick Start Guide to download the dataset and setup the container, and then proceed to the Inference performance benchmark section for instruction on how to launch the benchmark.