The performance measurements in this document were conducted at the time of publication and may not reflect the performance achieved from NVIDIA's latest software release. For the most up-to-date performance measurements, go to NVIDIA Data Center Deep Learning Product Performance.

Benchmarking

The following section shows how to run benchmarks measuring the model performance in training and inference modes.

Training performance benchmark

To benchmark training, run one of the train_benchmark scripts in ./scripts/:

bash scripts/unet3d_train_benchmark{_TF-AMP}.sh <num/of/gpus> <path/to/dataset> <path/to/checkpoints> <batch/size>

For example, to benchmark training using mixed-precision on 4 GPUs with batch size of 2 use:

bash scripts/unet3d_train_benchmark_TF-AMP.sh 4 <path/to/dataset> <path/to/checkpoints> 2

Each of these scripts will by default run 40 warm-up iterations and benchmark the performance during training in the next 40 iterations.

To have more control, you can run the script by directly providing all relevant run parameters. For example:

horovodrun -np <num/of/gpus> python main.py --exec_mode train --benchmark --augment --data_dir <path/to/dataset> --model_dir <path/to/checkpoints> --batch_size <batch/size> --warmup_steps <warm-up/steps> --max_steps <max/steps>

At the end of the script, a line reporting the best train throughput will be printed.

Inference performance benchmark

To benchmark inference, run one of the scripts in ./scripts/:

bash scripts/unet3d_infer_benchmark{_TF-AMP}.sh <path/to/dataset> <path/to/checkpoints> <batch/size>

For example, to benchmark inference using mixed-precision with batch size 4:

bash scripts/unet3d_infer_benchmark_TF-AMP.sh <path/to/dataset> <path/to/checkpoints> 4

Each of these scripts will by default run 20 warm-up iterations and benchmark the performance during inference in the next 20 iterations.

To have more control, you can run the script by directly providing all relevant run parameters. For example:

python main.py --exec_mode predict --benchmark --data_dir <path/to/dataset> --model_dir <optional, path/to/checkpoint> --batch_size <batch/size> --warmup_steps <warm-up/steps> --max_steps <max/steps>

At the end of the script, a line reporting the best inference throughput will be printed.

Results

The following sections provide details on how we achieved our performance and accuracy of training and inference.

Training accuracy results

To reproduce this result, start the Docker container interactively and run one of the train scripts:

bash scripts/unet3d_train_full{_TF-AMP}.sh <num/of/gpus> <path/to/dataset> <path/to/checkpoint> <batch/size>

for example to train using 8 GPUs and batch size of 2:

bash scripts/unet3d_train_full_TF-AMP.sh 8 /data/preprocessed /results 2

This command will launch a script which will run 5-fold cross-validation training for 16,000 iterations on each fold and print:

• the validation DICE scores for each class: Tumor Core (TC), Peritumoral Edema (ED), Enhancing Tumor (ET),
• the mean DICE score,
• the whole tumor (WT) which represents a binary classification case (tumor vs background).

The time reported is for one fold, which means that the training of 5 folds will take 5 times longer. The default batch size is 2, however if you have less than 16 GB memory card and you encounter GPU memory issues you should decrease the batch size. The logs of the runs can be found in the /results directory once the script is finished.

Training accuracy: NVIDIA DGX A100 (8x A100 80G)

The following table lists the average DICE score across 5-fold cross-validation. Our results were obtained by running the scripts/unet3d_train_full{_TF-AMP}.sh training script in the tensorflow:21.10-tf1-py3 NGC container on NVIDIA DGX A100 (8x A100 80G) GPUs.

Training accuracy: NVIDIA DGX-1 (8x V100 16G)

The following table lists the average DICE score across 5-fold cross-validation. Our results were obtained by running the scripts/unet3d_train_full{_TF-AMP}.sh training script in the tensorflow:21.10-tf1-py3 NGC container on NVIDIA DGX-1 (8x V100 16G) GPUs.

Training performance results

Training performance: NVIDIA DGX A100 (8x A100 80G)

Our results were obtained by running the scripts/unet3d_train_benchmark{_TF-AMP}.sh training script in the tensorflow:21.10-tf1-py3 NGC container on NVIDIA DGX A100 with (8x A100 80G) GPUs. Performance numbers (in volumes per second) were averaged over 80 iterations, excluding the first 40 warm-up steps.

Training performance: NVIDIA DGX-1 (8x V100 16G)

Our results were obtained by running the scripts/unet3d_train_benchmark{_TF-AMP}.sh training script in the tensorflow:21.10-tf1-py3 NGC container on NVIDIA DGX-1 with (8x V100 16G) GPUs. Performance numbers (in volumes per second) were averaged over 80 iterations, excluding the first 40 warm-up steps.

To achieve these same results, follow the steps in the Training performance benchmark section.

Inference performance results

Inference performance: NVIDIA DGX A100 (1x A100 80G)

Our results were obtained by running the scripts/unet3d_infer_benchmark{_TF-AMP}.sh inference benchmarking script in the tensorflow:21.10-tf1-py3 NGC container on NVIDIA DGX A100 with (1x A100 80G) GPU. Performance numbers (in volumes per second) were averaged over 40 iterations, excluding the first 20 warm-up steps.

FP16

TF32

To achieve these same results, follow the steps in the Inference performance benchmark section.

Inference performance: NVIDIA DGX-1 (1x V100 16G)

Our results were obtained by running the scripts/unet3d_infer_benchmark{_TF-AMP}.sh inference benchmarking script in the tensorflow:21.10-tf1-py3 NGC container on NVIDIA DGX-1 with (1x V100 16G) GPU. Performance numbers (in volumes per second) were averaged over 40 iterations, excluding the first 20 warm-up steps.

FP16

FP32

To achieve these same results, follow the steps in the Inference performance benchmark section.