Benchmarking

The following section shows how to run benchmarks measuring the model performance in training and inference modes. Note that the first 3 steps of each epoch are not used in the throughput or latency calculation. This is due to the fact that the nvFuser performs the optimizations on the 3rd step of the first epoch causing a multi-second pause.

Training performance benchmark

In order to run training benchmarks, use the scripts/benchmark.sh script.

Inference performance benchmark

To benchmark the inference performance on a specific batch size and dataset, run the inference.py script.

Results

The following sections provide details on how we achieved our performance and accuracy in training and inference.

Training accuracy results

We conducted an extensive hyperparameter search along with stability tests. The presented results are the averages from the hundreds of runs.

Training accuracy: NVIDIA DGX A100 (8x A100 80GB)

Our results were obtained by running the train.sh training script in the PyTorch 22.11 NGC container on NVIDIA A100 (8x A100 80GB) GPUs.

Training accuracy: NVIDIA DGX-1 (8x V100 16GB)

Our results were obtained by running the train.sh training script in the PyTorch 22.11 NGC container on NVIDIA DGX-1 with (8x V100 16GB) GPUs.

Training stability test

In order to get a greater picture of the model's accuracy, we performed a hyperparameter search along with stability tests on 100 random seeds for each configuration. Then, for each benchmark dataset, we have chosen the architecture with the least mean test q-risk. The table below summarizes the best configurations.

Training performance results

Training performance: NVIDIA DGX A100 (8x A100 80GB)

Our results were obtained by running the train.sh training script in the PyTorch 22.11 NGC container on NVIDIA A100 (8x A100 80GB) GPUs. Performance numbers (in items/images per second) were averaged over an entire training epoch.

To achieve these same results, follow the steps in the Quick Start Guide.

The performance metrics used were items per second.

Training performance: NVIDIA DGX-1 (8x V100 16GB)

Our results were obtained by running the train.sh training script in the PyTorch 22.11 NGC container on NVIDIA DGX-1 with (8x V100 16GB) GPUs. Performance numbers (in items/images per second) were averaged over an entire training epoch.

To achieve these same results, follow the steps in the Quick Start Guide.

The performance metrics used were items per second.

Inference Performance Results

Inference Performance: NVIDIA DGX A100

Our results were obtained by running the inference.py script in the PyTorch 22.11 NGC container on NVIDIA DGX A100. Throughput is measured in items per second and latency is measured in milliseconds. To benchmark the inference performance on a specific batch size and dataset, run the inference.py script.

Inference Performance: NVIDIA DGX-1 V100

Our results were obtained by running the inference.py script in the PyTorch 22.11 NGC container on NVIDIA DGX-1 V100. Throughput is measured in items per second and latency is measured in milliseconds. To benchmark the inference performance on a specific batch size and dataset, run the inference.py script.