The performance measurements in this document were conducted at the time of publication and may not reflect the performance achieved from NVIDIA's latest software release. For the most up-to-date performance measurements, go to NVIDIA Data Center Deep Learning Product Performance.

Benchmarking

The following section shows how to run benchmarks measuring the model performance in training and inference modes.

Training performance benchmark

The training benchmark was run in various scenarios on A100 80GB and V100 16G GPUs. The benchmark does not require a checkpoint from a fully trained model.

To benchmark training, run:

torchrun --nproc_per_node={NGPU} \
       main.py --batch-size {bs} \
               --mode benchmark-training \
               --benchmark-warmup 100 \
               --benchmark-iterations 200 \
               {AMP} \
               --data {data}

Where the {NGPU} selects number of GPUs used in benchmark, the {bs} is the desired batch size, the {AMP} is set to --amp if you want to benchmark training with Tensor Cores, and the {data} is the location of the COCO 2017 dataset.

--benchmark-warmup is specified to omit the first iteration of the first epoch. --benchmark-iterations is a number of iterations used to measure performance.

Inference performance benchmark

Inference benchmark was run on 1x A100 80GB GPU and 1x V100 16G GPU. To benchmark inference, run:

python main.py --eval-batch-size {bs} \
               --mode benchmark-inference \
               --benchmark-warmup 100 \
               --benchmark-iterations 200 \
               {AMP} \
               --data {data}

Where the {bs} is the desired batch size, the {AMP} is set to --amp if you want to benchmark inference with Tensor Cores, and the {data} is the location of the COCO 2017 dataset.

--benchmark-warmup is specified to omit the first iterations of the first epoch. --benchmark-iterations is a number of iterations used to measure performance.

Results

The following sections provide details on how we achieved our performance and accuracy in training and inference.

Training accuracy results

Training accuracy: NVIDIA DGX A100 (8x A100 80GB)

Our results were obtained by running the ./examples/SSD300_A100_{FP16,TF32}_{1,4,8}GPU.sh script in the pytorch-22.10-py3 NGC container on NVIDIA DGX A100 (8x A100 80GB) GPUs.

Training accuracy: NVIDIA DGX-1 (8x V100 16GB)

Our results were obtained by running the ./examples/SSD300_FP{16,32}_{1,4,8}GPU.sh script in the pytorch-22.10-py3 NGC container on NVIDIA DGX-1 with 8x V100 16GB GPUs.

Due to smaller size, mixed precision models can be trained with bigger batches. In such cases mixed precision speedup is calculated versus FP32 training with maximum batch size for that precision

Training loss plot

Here are example graphs of FP32, TF32 and AMP training on 8 GPU configuration:

Training stability test

The SSD300 v1.1 model was trained for 65 epochs, starting from 15 different initial random seeds. The training was performed in the pytorch-22.10-py3 NGC container on NVIDIA DGX A100 8x A100 80GB GPUs with batch size per GPU = 128. After training, the models were evaluated on the test dataset. The following table summarizes the final mAP on the test set.

Training performance results

Training performance: NVIDIA DGX A100 (8x A100 80GB)

Our results were obtained by running the main.py script with the --mode benchmark-training flag in the pytorch-22.10-py3 NGC container on NVIDIA DGX A100 (8x A100 80GB) GPUs. Performance numbers (in items/images per second) were averaged over an entire training epoch.

To achieve these same results, follow the Quick Start Guide outlined above.

Training performance: NVIDIA DGX-1 (8x V100 16GB)

Our results were obtained by running the main.py script with the --mode benchmark-training flag in the pytorch-22.10-py3 NGC container on NVIDIA DGX-1 with 8x V100 16GB GPUs. Performance numbers (in items/images per second) were averaged over an entire training epoch.

Due to smaller size, mixed precision models can be trained with bigger batches. In such cases mixed precision speedup is calculated versus FP32 training with maximum batch size for that precision

To achieve these same results, follow the Quick Start Guide outlined above.

Inference performance results

Inference performance: NVIDIA DGX A100 (1x A100 80GB)

Our results were obtained by running the main.py script with --mode benchmark-inference flag in the pytorch-22.10-py3 NGC container on NVIDIA DGX A100 (1x A100 80GB) GPU.

To achieve these same results, follow the Quick Start Guide outlined above.

Inference performance: NVIDIA DGX-1 (1x V100 16GB)

Our results were obtained by running the main.py script with --mode benchmark-inference flag in the pytorch-22.10-py3 NGC container on NVIDIA DGX-1 with (1x V100 16GB) GPU.

To achieve these same results, follow the Quick Start Guide outlined above.