The performance measurements in this document were conducted at the time of publication and may not reflect the performance achieved from NVIDIA's latest software release. For the most up-to-date performance measurements, go to NVIDIA Data Center Deep Learning Product Performance.

Benchmarking

The following section shows how to run benchmarks measuring the model performance in training and inference modes.

Training performance benchmark

Training is launched on batches of text data, different batches have different sequence lengths (number of tokens in the longest sequence). Sequence length and batch efficiency (ratio of non-pad tokens to total number of tokens) affect performance of the training, therefore it's recommended to run the training on a large chunk of training dataset to get a stable and reliable average training performance. Ideally at least one full epoch of training should be launched to get a good estimate of training performance.

The following commands will launch one epoch of training:

To launch mixed precision training on 1, 4 or 8 GPUs, run:

python3 -m torch.distributed.launch --nproc_per_node=<#GPUs> train.py --seed 2 --train-global-batch-size 1024 --epochs 1 --math fp16

To launch mixed precision training on 16 GPUs, run:

python3 -m torch.distributed.launch --nproc_per_node=16 train.py --seed 2 --train-global-batch-size 2048 --epochs 1 --math fp16

Change --math fp16 to --math fp32 to launch single precision training (for NVIDIA Volta and NVIDIA Turing architectures) or to --math tf32 to launch TF32 training with Tensor Cores (for NVIDIA Ampere architecture).

After the training is completed, the train.py script prints a summary to standard output. Performance results are printed in the following format:

(...)
0: Performance: Epoch: 0	Training: 418926 Tok/s	Validation: 1430828 Tok/s
(...)

Training: 418926 Tok/s represents training throughput averaged over an entire training epoch and summed over all GPUs participating in the training.

Inference performance benchmark

The inference performance and accuracy benchmarks require a checkpoint from a fully trained model.

Command to launch the inference accuracy benchmark on NVIDIA Volta or on NVIDIA Turing architectures:

python3 translate.py \
  --model gnmt/model_best.pth \
  --input data/wmt16_de_en/newstest2014.en \
  --reference data/wmt16_de_en/newstest2014.de \
  --output /tmp/output \
  --math fp16 fp32 \
  --batch-size 128 \
  --beam-size 1 2 5 \
  --tables

Command to launch the inference accuracy benchmark on NVIDIA Ampere architecture:

python3 translate.py \
  --model gnmt/model_best.pth \
  --input data/wmt16_de_en/newstest2014.en \
  --reference data/wmt16_de_en/newstest2014.de \
  --output /tmp/output \
  --math fp16 tf32 \
  --batch-size 128 \
  --beam-size 1 2 5 \
  --tables

Command to launch the inference throughput and latency benchmarks on NVIDIA Volta or NVIDIA Turing architectures:

python3 translate.py \
  --model gnmt/model_best.pth \
  --input data/wmt16_de_en/newstest2014.en \
  --reference data/wmt16_de_en/newstest2014.de \
  --output /tmp/output \
  --math fp16 fp32 \
  --batch-size 1 2 4 8 32 128 512 \
  --repeat 1 1 1 1 2 8 16 \
  --beam-size 1 2 5 \
  --warmup 5 \
  --tables

Command to launch the inference throughput and latency benchmarks on NVIDIA Ampere architecture:

python3 translate.py \
  --model gnmt/model_best.pth \
  --input data/wmt16_de_en/newstest2014.en \
  --reference data/wmt16_de_en/newstest2014.de \
  --output /tmp/output \
  --math fp16 tf32 \
  --batch-size 1 2 4 8 32 128 512 \
  --repeat 1 1 1 1 2 8 16 \
  --beam-size 1 2 5 \
  --warmup 5 \
  --tables

Results

The following sections provide details on how we achieved our performance and accuracy in training and inference.

Training accuracy results

Training accuracy: NVIDIA DGX A100 (8x A100 40GB)

Our results were obtained by running the train.py script with the default batch size = 128 per GPU in the pytorch-20.06-py3 NGC container on NVIDIA DGX A100 with 8x A100 40GB GPUs.

Command to launch the training:

python3 -m torch.distributed.launch --nproc_per_node=<#GPUs> train.py --seed 2 --train-global-batch-size 1024 --math fp16

Change --math fp16 to --math tf32 to launch TF32 training with Tensor Cores.

To achieve these same results, follow the Quick Start Guide outlined above.

Training accuracy: NVIDIA DGX-1 (8x V100 16GB)

Our results were obtained by running the train.py script with the default batch size = 128 per GPU in the pytorch-20.06-py3 NGC container on NVIDIA DGX-1 with 8x V100 16GB GPUs.

Command to launch the training:

python3 -m torch.distributed.launch --nproc_per_node=<#GPUs> train.py --seed 2 --train-global-batch-size 1024 --math fp16

Change --math fp16 to --math fp32 to launch single precision training.

To achieve these same results, follow the Quick Start Guide outlined above.

Training accuracy: NVIDIA DGX-2H (16x V100 32GB)

Our results were obtained by running the train.py script with the default batch size = 128 per GPU in the pytorch-20.06-py3 NGC container on NVIDIA DGX-2H with 16x V100 32GB GPUs.

To launch mixed precision training on 16 GPUs, run:

python3 -m torch.distributed.launch --nproc_per_node=16 train.py --seed 2 --train-global-batch-size 2048 --math fp16

Change --math fp16 to --math fp32 to launch single precision training.

To achieve these same results, follow the Quick Start Guide outlined above.

Training stability test

The GNMT v2 model was trained for 6 epochs, starting from 32 different initial random seeds. After each training epoch, the model was evaluated on the test dataset and the BLEU score was recorded. The training was performed in the pytorch-20.06-py3 Docker container on NVIDIA DGX A100 with 8x A100 40GB GPUs. The following table summarizes the results of the stability test.

In the following table, the BLEU scores after each training epoch for different initial random seeds are displayed.

Training throughput results

Training throughput: NVIDIA DGX A100 (8x A100 40GB)

Our results were obtained by running the train.py training script in the pytorch-20.06-py3 NGC container on NVIDIA DGX A100 with 8x A100 40GB GPUs. Throughput performance numbers (in tokens per second) were averaged over an entire training epoch.

To achieve these same results, follow the Quick Start Guide outlined above.

Training throughput: NVIDIA DGX-1 (8x V100 16GB)

Our results were obtained by running the train.py training script in the pytorch-20.06-py3 NGC container on NVIDIA DGX-1 with 8x V100 16GB GPUs. Throughput performance numbers (in tokens per second) were averaged over an entire training epoch.

To achieve these same results, follow the Quick Start Guide outlined above.

Training throughput: NVIDIA DGX-2H (16x V100 32GB)

Our results were obtained by running the train.py training script in the pytorch-20.06-py3 NGC container on NVIDIA DGX-2H with 16x V100 32GB GPUs. Throughput performance numbers (in tokens per second) were averaged over an entire training epoch.

To achieve these same results, follow the Quick Start Guide outlined above.

Inference accuracy results

Inference accuracy: NVIDIA A100 40GB

Our results were obtained by running the translate.py script in the pytorch-20.06-py3 NGC Docker container with NVIDIA A100 40GB GPU. Full command to launch the inference accuracy benchmark was provided in the Inference performance benchmark section.

Inference accuracy: NVIDIA Tesla V100 16GB

Our results were obtained by running the translate.py script in the pytorch-20.06-py3 NGC Docker container with NVIDIA Tesla V100 16GB GPU. Full command to launch the inference accuracy benchmark was provided in the Inference performance benchmark section.

Inference accuracy: NVIDIA T4

Our results were obtained by running the translate.py script in the pytorch-20.06-py3 NGC Docker container with NVIDIA Tesla T4. Full command to launch the inference accuracy benchmark was provided in the Inference performance benchmark section.

To achieve these same results, follow the Quick Start Guide outlined above.

Inference throughput results

Tables presented in this section show the average inference throughput (columns Avg (tok/s)) and inference throughput for various confidence intervals (columns N% (ms), where N denotes the confidence interval). Inference throughput is measured in tokens per second. Speedups reported in FP16 subsections are relative to FP32 (for NVIDIA Volta and NVIDIA Turing) and relative to TF32 (for NVIDIA Ampere) numbers for corresponding configuration.

Inference throughput: NVIDIA A100 40GB

Our results were obtained by running the translate.py script in the pytorch-20.06-py3 NGC Docker container with NVIDIA A100 40GB. Full command to launch the inference throughput benchmark was provided in the Inference performance benchmark section.

FP16

Inference throughput: NVIDIA T4

Our results were obtained by running the translate.py script in the pytorch-20.06-py3 NGC Docker container with NVIDIA T4. Full command to launch the inference throughput benchmark was provided in the Inference performance benchmark section.

FP16

To achieve these same results, follow the Quick Start Guide outlined above.

Inference latency results

Tables presented in this section show the average inference latency (columns Avg (ms)) and inference latency for various confidence intervals (columns N% (ms), where N denotes the confidence interval). Inference latency is measured in milliseconds. Speedups reported in FP16 subsections are relative to FP32 (for NVIDIA Volta and NVIDIA Turing) and relative to TF32 (for NVIDIA Ampere) numbers for corresponding configuration.

Inference latency: NVIDIA A100 40GB

Our results were obtained by running the translate.py script in the pytorch-20.06-py3 NGC Docker container with NVIDIA A100 40GB. Full command to launch the inference latency benchmark was provided in the Inference performance benchmark section.

FP16

Inference latency: NVIDIA T4

Our results were obtained by running the translate.py script in the pytorch-20.06-py3 NGC Docker container with NVIDIA T4. Full command to launch the inference latency benchmark was provided in the Inference performance benchmark section.

FP16

To achieve these same results, follow the Quick Start Guide outlined above.