The performance measurements in this document were conducted at the time of publication and may not reflect the performance achieved from NVIDIA's latest software release. For the most up-to-date performance measurements, go to NVIDIA Data Center Deep Learning Product Performance.

Benchmarking

The following section shows how to run benchmarks measuring the model performance in training and inference modes.

Training performance benchmark

To benchmark training performance, run:

• python nmt.py --output_dir=results --max_train_epochs=1 --num_gpus <num GPUs> --batch_size <total batch size> --amp for mixed precision
• python nmt.py --output_dir=results --max_train_epochs=1 --num_gpus <num GPUs> --batch_size <total batch size> for FP32/TF32

The log file will contain training performance in the following format:

training time for epoch 1: 25.75 mins (3625.19 sent/sec, 173461.27 tokens/sec)

Inference performance benchmark

To benchmark inference performance, run the scripts/translate.py script:

• For FP32/TF32: python scripts/translate.py --output_dir=/path/to/trained/model --beam_width <comma separated beam widths> --infer_batch_size <comma separated batch sizes>

• For mixed precision python scripts/translate.py --output_dir=/path/to/trained/model --amp --beam_width <comma separated beam widths> --infer_batch_size <comma separated batch sizes>

The benchmark requires a checkpoint from a fully trained model.

Results

The following sections provide details on how we achieved our performance and accuracy in training and inference.

Training accuracy results

Training accuracy: NVIDIA DGX A100 (8x A100 40GB)

Our results were obtained by running the examples/DGXA100_{TF32,AMP}_8GPU.sh training script in the tensorflow-20.06-tf1-py3 NGC container on NVIDIA DGX A100 (8x A100 40GB) GPUs.

Training accuracy: NVIDIA DGX-1 (8x V100 16GB)

Our results were obtained by running the nmt.py script in the tensorflow-19.07-py3 NGC container on NVIDIA DGX-1 with (8x V100 16GB) GPUs.

In the following plot, the BLEU scores after each training epoch for different configurations are displayed.

Training stability test

The GNMT v2 model was trained for 6 epochs, starting from 6 different initial random seeds. After each training epoch, the model was evaluated on the test dataset and the BLEU score was recorded. The training was performed in the tensorflow-20.06-tf1-py3 NGC container.

In the following tables, the BLEU scores after each training epoch for different initial random seeds are displayed.

NVIDIA DGX A100 with 8 Ampere A100 40GB GPUs with TF32.

NVIDIA DGX-1 with 8 Tesla V100 16GB GPUs with FP32.

Inference accuracy results

Inference accuracy: NVIDIA DGX-1 (8x V100 16GB)

Our results were obtained by running the scripts/translate.py script in the tensorflow-19.07-py3 NGC container on NVIDIA DGX-1 8x V100 16GB GPUs.

• For mixed precision: python scripts/translate.py --output_dir=/path/to/trained/model --beam_width 1,2,5 --infer_batch_size 128 --amp

• For FP32: python scripts/translate.py --output_dir=/path/to/trained/model --beam_width 1,2,5 --infer_batch_size 128

Training performance results

Training performance: NVIDIA DGX A100 (8x A100 40GB)

Our results were obtained by running the examples/DGXA100_{TF32,AMP}_{1,8}GPU.sh training script in the tensorflow-20.06-tf1-py3 NGC container on NVIDIA DGX A100 (8x A100 40GB) GPUs. Performance numbers (in items/images per second) were averaged over an entire training epoch.

To achieve these same results, follow the steps in the Quick Start Guide.

Training performance: NVIDIA DGX-1 (8x V100 16GB)

Our results were obtained by running the nmt.py script in the tensorflow-19.07-py3 NGC container on NVIDIA DGX-1 with 8x V100 16G GPUs. Performance numbers (in tokens per second) were averaged over an entire training epoch.

To achieve these same results, follow the Quick Start Guide outlined above.

Inference performance results

The benchmark requires a checkpoint from a fully trained model.

To launch the inference benchmark in mixed precision on 1 GPU, run:

python scripts/translate.py --output_dir=/path/to/trained/model --beam_width 1,2,5 --infer_batch_size 1,2,4,8,32,128,512 --amp

To launch the inference benchmark in FP32/TF32 on 1 GPU, run:

python scripts/translate.py --output_dir=/path/to/trained/model --beam_width 1,2,5 --infer_batch_size 1,2,4,8,32,128,512

To achieve these same results, follow the Quick Start Guide outlined above.

Inference performance: NVIDIA DGX A100 (1x A100 40GB)

Our results were obtained by running the python scripts/translate.py --infer_batch_size 1,2,4,8,32,128,512 --beam_width 1,2,5 {--amp} inferencing benchmarking script in the tensorflow-20.06-tf1-py3 NGC container on NVIDIA DGX A100 (1x A100 40GB) GPU.

FP16

TF32

To achieve these same results, follow the steps in the Quick Start Guide.

Inference performance: NVIDIA DGX-1 (1x V100 16GB)

Our results were obtained by running the python scripts/translate.py --infer_batch_size 1,2,4,8,32,128,512 --beam_width 1,2,5 {--amp} inferencing benchmarking script in the tensorflow-20.06-tf1-py3 NGC container on NVIDIA DGX-1 with (1x V100 16GB) GPU.

FP16

FP32

To achieve these same results, follow the steps in the Quick Start Guide.

Inference performance: NVIDIA T4

Our results were obtained by running the scripts/translate.py script in the tensorflow-19.07-py3 NGC container on NVIDIA T4.

Reported mixed precision speedups are relative to FP32 numbers for corresponding configuration.