The performance measurements in this document were conducted at the time of publication and may not reflect the performance achieved from NVIDIA's latest software release. For the most up-to-date performance measurements, go to NVIDIA Data Center Deep Learning Product Performance.

Benchmarking

The following section shows how to run benchmarks measuring the model performance in training and inference modes.

Training performance benchmark

To benchmark the training performance on a specific batch size, run:

• For 1 GPU

  • FP32 / TF32

    python ./main.py --arch=resnext101-32x4d --mode=training_benchmark --warmup_steps 200 --batch_size <batch size> --data_dir=<path to imagenet> --results_dir=<path to results directory>

  • AMP

    python ./main.py --arch=resnext101-32x4d --mode=training_benchmark --amp --warmup_steps 200 --batch_size <batch size> --data_dir=<path to imagenet> --results_dir=<path to results directory>

• For multiple GPUs

  • FP32 / TF32

    mpiexec --allow-run-as-root --bind-to socket -np <num_gpus> python ./main.py --arch=resnext101-32x4d --mode=training_benchmark --batch_size <batch size> --data_dir=<path to imagenet> --results_dir=<path to results directory>

  • AMP

    mpiexec --allow-run-as-root --bind-to socket -np <num_gpus> python ./main.py --arch=resnext101-32x4d --mode=training_benchmark --amp --batch_size <batch size> --data_dir=<path to imagenet> --results_dir=<path to results directory>

Each of these scripts runs 200 warm-up iterations and measures the first epoch.

To control warmup and benchmark length, use the --warmup_steps, --num_iter and --iter_unit flags. Features like XLA or DALI can be controlled with --xla and --dali flags. For proper throughput reporting the value of --num_iter must be greater than --warmup_steps value. Suggested batch sizes for training are 128 for mixed precision training and 64 for single precision training per single V100 16 GB.

If no --data_dir=<path to imagenet> flag is specified then the benchmarks will use a synthetic dataset. The resolution of synthetic images used can be controlled with --synthetic_data_size flag.

Inference performance benchmark

To benchmark the inference performance on a specific batch size, run:

• FP32 / TF32

python ./main.py --arch=resnext101-32x4d --mode=inference_benchmark --warmup_steps 20 --num_iter 100 --iter_unit batch --batch_size <batch size> --data_dir=<path to imagenet> --results_dir=<path to results directory>

• AMP

python ./main.py --arch=resnext101-32x4d --mode=inference_benchmark --amp --warmup_steps 20 --num_iter 100 --iter_unit batch --batch_size <batch size> --data_dir=<path to imagenet> --results_dir=<path to results directory>

By default, each of these scripts runs 20 warm-up iterations and measures the next 80 iterations. To control warm-up and benchmark length, use the --warmup_steps, --num_iter and --iter_unit flags. If no --data_dir=<path to imagenet> flag is specified then the benchmarks will use a synthetic dataset.

The benchmark can be automated with the inference_benchmark.sh script provided in resnext101-32x4d, by simply running: bash ./resnext101-32x4d/inference_benchmark.sh <data dir> <data idx dir>

The <data dir> parameter refers to the input data directory (by default /data/tfrecords inside the container). By default, the benchmark tests the following configurations: FP32, AMP, AMP + XLA with different batch sizes. When the optional directory with the DALI index files <data idx dir> is specified, the benchmark executes an additional DALI + AMP + XLA configuration. For proper throughput reporting the value of --num_iter must be greater than --warmup_steps value.

For performance benchamrk of raw model, synthetic dataset can be used. To use synthetic dataset, use --synthetic_data_size flag instead of --data_dir to specify input image size.

Results

The following sections provide details on how we achieved our performance and accuracy in training and inference.

Training accuracy results

Training accuracy: NVIDIA DGX A100 (8x A100 40GB)

Our results were obtained by running the /resnet50v1.5/training/DGXA100_RN50_{PRECISION}_90E.sh training script in the TensorFlow 20.06-tf1-py3 NGC container NGC container on NVIDIA DGX A100 (8x A100 40GB) GPUs.

Training accuracy: NVIDIA DGX-1 (8x V100 16G)

Our results were obtained by running the /resnext101-32x4d/training/DGX1_RNxt101-32x4d_{PRECISION}_{EPOCHS}E.sh training script in the TensorFlow 20.06-tf1-py3 NGC container NGC container on NVIDIA DGX-1 with (8x V100 16G) GPUs.

Example training loss plot

Training performance results

Training performance: NVIDIA DGX A100 (8x A100 40GB)

Our results were obtained by running the resnext101-32x4d/training/training_perf.sh benchmark script in the TensorFlow 20.06-tf1-py3 NGC container NGC container on NVIDIA DGX A100 (8x A100 40GB) GPUs. Performance numbers (in images per second) were averaged over an entire training epoch.

Training performance: NVIDIA DGX-1 (8x V100 16G)

Our results were obtained by running the resnext101-32x4d/training/training_perf.sh benchmark script in the TensorFlow 20.06-tf1-py3 NGC container NGC container on NVIDIA DGX-1 with (8x V100 16G) GPUs. Performance numbers (in images per second) were averaged over an entire training epoch.

Training performance: NVIDIA DGX-2 (16x V100 32G)

Our results were obtained by running the resnext101-32x4d/training/training_perf.sh benchmark script in the TensorFlow 20.06-tf1-py3 NGC container NGC container on NVIDIA DGX-2 with (16x V100 32G) GPUs. Performance numbers (in images per second) were averaged over an entire training epoch.

Training Time for 90 Epochs

Training time: NVIDIA DGX A100 (8x A100 40GB)

Our results were estimated based on the training performance results on NVIDIA DGX A100 with (8x A100 40G) GPUs.

Training time: NVIDIA DGX-1 (8x V100 16G)

Our results were estimated based on the training performance results on NVIDIA DGX-1 with (8x V100 16G) GPUs.

Training time: NVIDIA DGX-2 (16x V100 32G)

Our results were estimated based on the training performance results on NVIDIA DGX-2 with (16x V100 32G) GPUs.

Inference performance results

Inference performance: NVIDIA DGX A100 (1x A100 40GB)

Our results were obtained by running the inference_benchmark.sh inferencing benchmarking script in the TensorFlow 20.06-tf1-py3 NGC container NGC container on NVIDIA DGX A100 with (1x A100 40G) GPU.

TF32 Inference Latency

TF32 Inference Latency + XLA

Mixed Precision Inference Latency

Mixed Precision Inference Latency + XLA

Inference performance: NVIDIA DGX-1 (1x V100 16G)

Our results were obtained by running the inference_benchmark.sh inferencing benchmarking script in the TensorFlow 20.06-tf1-py3 NGC container NGC container on NVIDIA DGX-1 with (1x V100 16G) GPU.

FP32 Inference Latency

Mixed Precision Inference Latency

Mixed Precision Inference Latency + XLA

Inference performance: NVIDIA DGX-2 (1x V100 32G)

Our results were obtained by running the inference_benchmark.sh inferencing benchmarking script in the TensorFlow 20.06-tf1-py3 NGC container NGC container on NVIDIA DGX-2 with (1x V100 32G) GPU.

FP32 Inference Latency

Mixed Precision Inference Latency

Mixed Precision Inference Latency + XLA

Inference performance: NVIDIA T4 (1x T4 16G)

Our results were obtained by running the inference_benchmark.sh inferencing benchmarking script in the TensorFlow 20.06-tf1-py3 NGC container NGC container on NVIDIA T4 with (1x T4 16G) GPU.

FP32 Inference Latency

Mixed Precision Inference Latency

Mixed Precision Inference Latency + XLA