The performance measurements in this document were conducted at the time of publication and may not reflect the performance achieved from NVIDIA's latest software release. For the most up-to-date performance measurements, go to NVIDIA Data Center Deep Learning Product Performance.

Benchmarking

The following section shows how to run benchmarks measuring the model performance in training and inference modes.

Training performance benchmark

To benchmark training, run one of the TRAIN_BENCHMARK scripts in ./examples/:

bash examples/unet_TRAIN_BENCHMARK{_TF-AMP}.sh <number/of/gpus> <path/to/dataset> <path/to/checkpoints> <batch/size>

For example, to benchmark training using mixed-precision on 8 GPUs use:

bash examples/unet_TRAIN_BENCHMARK_TF-AMP.sh 8 <path/to/dataset> <path/to/checkpoint> <batch/size>

Each of these scripts will by default run 200 warm-up iterations and benchmark the performance during training in the next 800 iterations.

To have more control, you can run the script by directly providing all relevant run parameters. For example:

horovodrun -np <num of gpus> python main.py --exec_mode train --benchmark --augment --data_dir <path/to/dataset> --model_dir <optional, path/to/checkpoint> --batch_size <batch/size> --warmup_steps <warm-up/steps> --max_steps <max/steps>

At the end of the script, a line reporting the best train throughput will be printed.

Inference performance benchmark

To benchmark inference, run one of the scripts in ./examples/:

bash examples/unet_INFER_BENCHMARK{_TF-AMP}.sh <path/to/dataset> <path/to/checkpoint> <batch/size>

For example, to benchmark inference using mixed-precision:

bash examples/unet_INFER_BENCHMARK_TF-AMP.sh <path/to/dataset> <path/to/checkpoint> <batch/size>

Each of these scripts will by default run 200 warm-up iterations and benchmark the performance during inference in the next 400 iterations.

To have more control, you can run the script by directly providing all relevant run parameters. For example:

python main.py --exec_mode predict --benchmark --data_dir <path/to/dataset> --model_dir <optional, path/to/checkpoint> --batch_size <batch/size> --warmup_steps <warm-up/steps> --max_steps <max/steps>

At the end of the script, a line reporting the best inference throughput will be printed.

Results

The following sections provide details on how we achieved our performance and accuracy in training and inference.

Training accuracy results

Training accuracy: NVIDIA DGX A100 (8x A100 80G)

The following table lists the average DICE score across 5-fold cross-validation. Our results were obtained by running the examples/unet_TRAIN_FULL{_TF-AMP}.sh training script in the tensorflow:21.02-tf2-py3 NGC container on NVIDIA DGX A100 (8x A100 80G) GPUs.

Training accuracy: NVIDIA DGX-1 (8x V100 16G)

The following table lists the average DICE score across 5-fold cross-validation. Our results were obtained by running the examples/unet_TRAIN_FULL{_TF-AMP}.sh training script in the tensorflow:21.02-tf2-py3 NGC container on NVIDIA DGX-1 with (8x V100 16G) GPUs.

To reproduce this result, start the Docker container interactively and run one of the TRAIN scripts:

bash examples/unet_TRAIN_FULL{_TF-AMP}.sh <number/of/gpus> <path/to/dataset> <path/to/checkpoint> <batch/size>

for example

bash examples/unet_TRAIN_FULL_TF-AMP.sh 8 /data /results 8

This command will launch a script which will run training on 8 GPUs for 6400 iterations five times for 5-fold cross-validation. At the end, it will collect the scores and print the average validation DICE score and cross-entropy loss. The time reported is for one fold, which means that the training for 5 folds will take 5 times longer. The default batch size is 8, however if you have less than 16 Gb memory card and you encounter GPU memory issue you should decrease the batch size. The logs of the runs can be found in /results directory once the script is finished.

Learning curves

The following image show the training loss as a function of iteration for training using DGX A100 (TF32 and TF-AMP) and DGX-1 V100 (FP32 and TF-AMP).

Training performance results

Training performance: NVIDIA DGX A100 (8x A100 80G)

Our results were obtained by running the examples/unet_TRAIN_BENCHMARK{_TF-AMP}.sh training script in the NGC container on NVIDIA DGX A100 (8x A100 80G) GPUs. Performance numbers (in images per second) were averaged over 1000 iterations, excluding the first 200 warm-up steps.

Training performance: NVIDIA DGX-1 (8x V100 16G)

Our results were obtained by running the examples/unet_TRAIN_BENCHMARK{_TF-AMP}.sh training script in the tensorflow:21.02-tf2-py3 NGC container on NVIDIA DGX-1 with (8x V100 16G) GPUs. Performance numbers (in images per second) were averaged over 1000 iterations, excluding the first 200 warm-up steps.

To achieve these same results, follow the steps in the Training performance benchmark section.

Throughput is reported in images per second. Latency is reported in milliseconds per image.

TensorFlow 2 runs by default using the eager mode, which makes tensor evaluation trivial at the cost of lower performance. To mitigate this issue multiple layers of performance optimization were implemented. Two of them, AMP and XLA, were already described. There is an additional one called Autograph, which allows to construct a graph from a subset of Python syntax improving the performance simply by adding a @tf.function decorator to the train function. To read more about Autograph see Better performance with tf.function and AutoGraph.

Inference performance results

Inference performance: NVIDIA DGX A100 (1x A100 80G)

Our results were obtained by running the examples/unet_INFER_BENCHMARK{_TF-AMP}.sh inference benchmarking script in the tensorflow:21.02-tf2-py3 NGC container on NVIDIA DGX A100 (1x A100 80G) GPU.

FP16

TF32

To achieve these same results, follow the steps in the Quick Start Guide.

Inference performance: NVIDIA DGX-1 (1x V100 16G)

Our results were obtained by running the examples/unet_INFER_BENCHMARK{_TF-AMP}.sh inference benchmarking script in the tensorflow:21.02-tf2-py3 NGC container on NVIDIA DGX-1 with (1x V100 16G) GPU.

FP16

FP32

To achieve these same results, follow the steps in the Inference performance benchmark section.

Throughput is reported in images per second. Latency is reported in milliseconds per batch.