The performance measurements in this document were conducted at the time of publication and may not reflect the performance achieved from NVIDIA's latest software release. For the most up-to-date performance measurements, go to NVIDIA Data Center Deep Learning Product Performance.

Benchmarking

The following section shows how to run benchmarks measuring the model performance in training and inference modes.

Training performance benchmark

To benchmark the training performance on a specific batch size, run train.py training script. Performance in tokens/s will be printed to standard output every N iterations, specified by the --log-interval option. Additionally performance and loss values will be logged by dllogger to the file specified in --stat-file option. Every line in the output file will be a valid JSON file prepended with DLLL prefix.

Inference performance benchmark

To benchmark the inference performance on a specific batch size, run following command to start the benchmark

for i in {1..10}; do sacrebleu -t wmt14/full -l en-de --echo src; done | python inference.py --buffer-size 5000 --path /path/to/your/checkpoint.pt --max-tokens 10240 --fuse-dropout-add --remove-bpe --bpe-codes /data/code --fp16 > /dev/null

Results will be printed to stderr.

Results

The following sections provide details on how we achieved our performance and accuracy in training and inference.

Training accuracy results

Following the spirit of the paper A Call for Clarity in Reporting BLEU Scores we decided to change evaluation metric implemented in fairseq to SacreBleu score. We have calculated that the new metric has almost linear relationship with the old one. We run linear regression on nearly 2000 checkpoints to discover that the SacreBleu score almost perfectly follows the formula: newScore = 0.978 * oldScore - 0.05.

Figure 2. Linear relationship between old and new BLEU metric.

Figure 3. Validation loss vs BLEU score. Plots are trimmed to certain validation loss threshold.

Training accuracy: NVIDIA DGX A100 (8x A100 40GB)

Our results were obtained by running the run_DGXA100_AMP_8GPU.sh and run_DGXA100_TF32_8GPU.sh training scripts in the pytorch-20.06-py3 NGC container on NVIDIA DGX A100 (8x A100 40GB) GPUs. We report average accuracy over 6 runs. We consider a model trained when it reaches minimal validation loss. Time to train contains only training time without validation. Depending on a configuration and frequency of validation it can take up to additional minute per epoch.

Training accuracy: NVIDIA DGX-1 (8x V100 16GB)

Our results were obtained by running the run_DGX1_AMP_8GPU.sh and run_DGX1_FP32_8GPU.sh training scripts in the pytorch-20.06-py3 NGC container on NVIDIA DGX-1 (8x V100 16GB) GPUs. We report average accuracy over 6 runs. We consider a model trained when it reaches minimal validation loss. Time to train contains only training time without validation. Depending on a configuration and frequency of validation it can take up to additional minute per epoch. Using mixed precision we could fit a larger batch size in the memory, further speeding up the training.

Training performance results

Training performance: NVIDIA DGX A100 (8x A100 40GB)

Our results were obtained by running the run_DGXA100_AMP_8GPU.sh and run_DGXA100_TF32_8GPU.sh training scripts in the pytorch-20.06-py3 NGC container on NVIDIA DGX A100 (8x A100 40GB) GPUs. Performance numbers (in tokens per second) were averaged over an entire training epoch.

To achieve these same results, follow the steps in the Quick Start Guide.

Training stability test

The following plot shows average validation loss curves for different configs. We can see that training with AMP O2 converges slightly slower that FP32 and TF32 training. In order to mitigate this, you can use option --amp-level O1 at the cost of 20% performance drop compared to the default AMP setting.

Figure 4. Validation loss curves

Training performance: NVIDIA DGX-1 (8x V100 16GB)

Our results were obtained by running the run_DGX1_AMP_8GPU.sh and run_DGX1_FP32_8GPU.sh training scripts in the pytorch-20.06-py3 NGC container on NVIDIA DGX-1 with (8x V100 16GB) GPUs. Performance numbers (in tokens per second) were averaged over an entire training epoch. Using mixed precision we could fit a larger batch size in the memory, further speeding up the training.

To achieve these same results, follow the steps in the Quick Start Guide.

Training performance: NVIDIA DGX-2 (16x V100 32GB)

Our results were obtained by running the run_DGX1_AMP_8GPU.sh and run_DGX1_FP32_8GPU.sh training scripts setting number of GPUs to 16 in the pytorch-20.06-py3 NGC container on NVIDIA DGX-2 with (16x V100 32GB) GPUs. Performance numbers (in tokens per second) were averaged over an entire training epoch. Using mixed precision we could fit a larger batch size in the memory, further speeding up the training.

To achieve these same results, follow the steps in the Quick Start Guide.

Inference performance results

Our implementation of the Transformer has dynamic batching algorithm, which batches sentences together in such a way that there are no more than N tokens in each batch or no more than M sentences in each batch. In this benchmark we use the first option in order to get the most stable results.

Inference performance: NVIDIA DGX A100 (1x A100 40GB)

Our results were obtained by running the inference.py inferencing benchmarking script in the pytorch-20.06-py3 NGC container on NVIDIA DGX A100 (1x A100 40GB) GPU.

FP16

TF32

To achieve these same results, follow the steps in the Quick Start Guide.

Inference performance: NVIDIA DGX-1 (1x V100 16GB)

Our results were obtained by running the inference.py inferencing benchmarking script in the pytorch-20.06-py3 NGC container on NVIDIA DGX-1 with (1x V100 16GB) GPU.

FP16

FP32

To achieve these same results, follow the steps in the Quick Start Guide.