The performance measurements in this document were conducted at the time of publication and may not reflect the performance achieved from NVIDIA's latest software release. For the most up-to-date performance measurements, go to NVIDIA Data Center Deep Learning Product Performance.

Benchmarking

The following section shows how to run benchmarks measuring the model performance in training and inference modes.

Training performance benchmark

To benchmark training (A100 GPUs only for now), set --benchmark, --benchmark-steps and --benchmark-warmup-steps, then run training with --run-scope train_only. Refer to Command-line options.

Example:

# For 8 GPUs benchmark for AMP
python -m paddle.distributed.launch --gpus=0,1,2,3,4,5,6,7 train.py \
  --run-scope train_only \
  --amp \
  --scale-loss 128.0 \
  --use-dynamic-loss-scaling \
  --data-layout NHWC \
  --benchmark \
  --benchmark-steps 100 \
  --benchmark-warmup-steps 300

Benchmark will run 300 iterations for warmup and 100 iterations for benchmark, then save benchmark results in the --report-file file.

Inference performance benchmark

Benchmark

To benchmark evaluation (A100 GPUs only for now), set --benchmark, --benchmark-steps and --benchmark-warmup-steps, then run training with --run-scope eval_only. Refer to Command-line options.

Example:

# For 8 GPUs benchmark for AMP
python -m paddle.distributed.launch --gpus=0,1,2,3,4,5,6,7 train.py \
  --run-scope eval_only \
  --amp \
  --data-layout NHWC \
  --benchmark \
  --benchmark-steps 100 \
  --benchmark-warmup-steps 300

Benchmark will run 300 iterations for warmup and 100 iterations for benchmark, then save benchmark results in the --report-file file.

It is also allowed to set batch size for benchmark by adding --batch-size <batch_size> in launching commands.

# For 8 GPUs benchmark for AMP
python -m paddle.distributed.launch --gpus=0,1,2,3,4,5,6,7 train.py \
  --run-scope eval_only \
  --batch-size 32 \
  --amp \
  --data-layout NHWC \
  --benchmark \
  --benchmark-steps 100 \
  --benchmark-warmup-steps 300

Benchmark with TensorRT

To benchmark the inference performance with TensorRT on a specific batch size, run:

• FP32 / TF32

python inference.py \
    --trt-inference-dir <path_to_exported_model> \
    --trt-precision FP32 \
    --batch-size <batch_size> \
    --benchmark-steps 1024 \
    --benchmark-warmup-steps 16

• FP16

python inference.py \
    --trt-inference-dir <path_to_exported_model> \
    --trt-precision FP16 \
    --batch-size <batch_size>
    --benchmark-steps 1024 \
    --benchmark-warmup-steps 16

Note that arguments passed to inference.py should be the same with arguments used in training.

The benchmark uses the validation dataset by default, which should be put in --image-root/val. For the performance benchmark of the raw model, a synthetic dataset can be used. To use synthetic dataset, add --trt-use-synthat True as a command line option.

Results

Training accuracy results

Our results were obtained by running the applicable training script in the PaddlePaddle NGC container.

To achieve these same results, follow the steps in the Quick Start Guide.

Training accuracy: NVIDIA DGX A100 (8x A100 80GB)

Example plots

The following images show the 90 epochs configuration on a DGX-A100.

Accuracy recovering of Automatic SParsity: NVIDIA DGX A100 (8x A100 80GB)

Training performance results

Our results were obtained by running the applicable training script in the PaddlePaddle NGC container.

To achieve these same results, follow the steps in the Quick Start Guide.

Training performance: NVIDIA DGX A100 (8x A100 80GB)

Training performance of Automatic SParsity: NVIDIA DGX A100 (8x A100 80GB)

Note that the train.py would enable CPU affinity binding to GPUs by default, that is designed and guaranteed being optimal for NVIDIA DGX-series. You could disable binding via launch train.py with --enable-cpu-affinity false.

Inference performance results

Inference performance: NVIDIA DGX A100 (1x A100 80GB)

Our results were obtained by running the applicable training script with --run-scope eval_only argument in the PaddlePaddle NGC container.

TF32 Inference Latency

Mixed Precision Inference Latency

Paddle-TRT performance results

Paddle-TRT performance: NVIDIA DGX A100 (1x A100 80GB)

Our results for Paddle-TRT were obtained by running the inference.py script on NVIDIA DGX A100 with (1x A100 80G) GPU.

TF32 Inference Latency

FP16 Inference Latency

Paddle-TRT performance: NVIDIA A30 (1x A30 24GB)

Our results for Paddle-TRT were obtained by running the inference.py script on NVIDIA A30 with (1x A30 24G) GPU.

TF32 Inference Latency

FP16 Inference Latency

Paddle-TRT performance: NVIDIA A10 (1x A10 24GB)

Our results for Paddle-TRT were obtained by running the inference.py script on NVIDIA A10 with (1x A10 24G) GPU.

TF32 Inference Latency

FP16 Inference Latency