Skip to main content
NVIDIA
BART for PyTorch
Resource
NVIDIA
BART for PyTorch

BART is a denoising autoencoder for pretraining sequence-to-sequence models.

Benchmarking

The following section shows how to run benchmarks measuring the model performance in training and inference modes.

Training performance benchmark

To benchmark the training performance on a specific batch size, source length, target length and dataset for one epoch, run:

bash scripts/run_training_benchmark.sh <batch size> <max source length> <max target length> <data dir>

The resulting NUM_GPU and PRECISION vs Throughput is stored in results/bart_pyt_training_benchmark_${DATESTAMP}/inference_benchmark.log

Inference performance benchmark

To benchmark the inference performance on a specific batch size, source length, target length and dataset, run:

bash scripts/run_inference_benchmark.sh <predict batch size> <eval beams> <max source length> <max target length> <model name or path> <data dir> <config path>

The resulting NUM_GPU and PRECISION vs Throughput is stored in results/bart_pyt_inference_benchmark_${DATESTAMP}/inference_benchmark.log

Results

The following sections provide details on how we achieved our performance and accuracy in training and inference.

Training accuracy results

Training accuracy: NVIDIA DGX A100 (8x A100 80GB)

Our results for XSUM dataset were obtained by running the run_summarization.sh training script in the PyTorch 21.02-py3 NGC container on NVIDIA DGX A100 (8x A100 80GB) GPUs. Accuracy column lists rogue1, rogue2 and rogueLSum scores.

GPUsBatch size (TF32, mixed precision)Accuracy - TF32Accuracy - mixed precisionTime to train (hrs) - TF32Time to train (hrs) - mixed precisionTime to train (hrs) speedup (TF32 to mixed precision)
124, 4044.41, 21.02, 35.6644.87, 21.49, 36.173.102.431.27
8192, 32045.34, 21.93, 36.6145.31, 21.83, 36.600.580.451.27

In addition,results for CNN-DM dataset are:

GPUsBatch size (TF32, mixed precision)Accuracy - TF32Accuracy - mixed precisionTime to train (hrs) - TF32Time to train (hrs) - mixed precisionTime to train (hrs) speedup (TF32 to mixed precision)
124, 4044.37, 21.36, 41.1744.43, 21.43, 41.224.883.611.35
8192, 32044.49, 21.48, 41.2844.19, 21.26, 40.970.730.561.30
Training accuracy: NVIDIA DGX-1 V100 (8x V100 32GB)

Our results were obtained by running the run_summarization.sh training script in the PyTorch 21.02-py3 NGC container on NVIDIA DGX-2 with (16x V100 32GB) GPUs. Accuracy column lists rogue1, rogue2 and rogueLSum scores.

GPUsBatch size (FP32, mixed precision)Accuracy - FP32Accuracy - mixed precisionTime to train (hrs) - FP32Time to train (hrs) - mixed precisionTime to train (hrs) speedup (FP32 to mixed precision)
18, 1444.16, 20.66, 35.2444.86, 21.41, 36.0217.236.122.82
864, 11245.42, 21.91, 36.6245.58, 22.01, 36.792.561.092.36

In addition,results for CNN-DM dataset are:

GPUsBatch size (FP32, mixed precision)Accuracy - FP32Accuracy - mixed precisionTime to train (hrs) - FP32Time to train (hrs) - mixed precisionTime to train (hrs) speedup (FP32 to mixed precision)
18, 1444.49, 21.48, 41.2644.55, 21.47, 41.3226.179.742.69
864, 11244.34, 21.42, 41.1244.27, 21.30, 41.063.581.452.46
Training stability test

Our results for XSUM dataset were obtained by running the run_summarization.sh training script in the PyTorch 21.02-py3 NGC container on NVIDIA DGX A100 (8x A100 80GB) GPUs. Accuracy column lists rogue1 scores across 5 different training runs with different seeds on DGX A100.

FP16, 8x GPUsseed 1seed 2seed 3seed 4seed 5meanstd
rogue145.3445.3445.2145.3345.3445.310.055

Training performance results

Training performance: NVIDIA DGX A100 (8x A100 80GB)

Our results were obtained by running the run_summarization.sh training script in the PyTorch 21.02-py3 NGC container on NVIDIA DGX A100 (8x A100 80GB) GPUs. Performance numbers (in items/images per second) were averaged over an entire training epoch.

GPUsBatch size / GPU (TF32, mixed precision)Throughput - TF32Throughput - mixed precisionThroughput speedup (TF32 - mixed precision)Weak scaling - TF32Weak scaling - mixed precision
124, 4031607420761.331.001.00
824, 401630542175141.335.165.17

To achieve these same results, follow the steps in the Quick Start Guide.

The performance metrics used are tokens per second computed from iterating through an entire epoch of XSum dataset with source length = 1024 and target length = 60.

Training performance: NVIDIA DGX-1 V100 (8x V100 32GB)

Our results were obtained by running the run_summarization.sh training script in the PyTorch 21.02-py3 NGC container on NVIDIA DGX-2 with (16x V100 32GB) GPUs. Performance numbers (in items/images per second) were averaged over an entire training epoch.

GPUsBatch size / GPU (FP32, mixed precision)Throughput - FP32Throughput - mixed precisionThroughput speedup (FP32 - mixed precision)Weak scaling - FP32Weak scaling - mixed precision
18, 147527193562.571.001.00
88, 14420241117202.655.585.77

To achieve these same results, follow the steps in the Quick Start Guide.

The performance metrics used are tokens per second computed from iterating through an entire epoch of XSum dataset with source length = 1024 and target length = 60.

Inference performance results

Inference performance: NVIDIA DGX A100 (1x A100 80GB)

Our results were obtained by running the run_eval_summarization.sh inferencing benchmarking script in the PyTorch 20.11-py3 NGC container on NVIDIA DGX A100 (1x A100 80GB) GPU.

FP16

Batch sizeLatency AvgLatency 90%Latency 95%Latency 99%Throughput
10.430.530.570.672.34
40.640.750.810.956.28
80.861.011.091.209.35
161.291.561.651.7612.44
322.383.063.233.3313.42
644.706.066.256.3513.55
12810.1012.2212.3212.9612.61

To achieve these same results, follow the steps in the Quick Start Guide.

The inference performance metrics used are milliseconds per iteration. They are computed by iterating through the XSum test data with source length = 1024, target length = 60 and beam search = 6.

Inference performance: NVIDIA DGX-1 V100 (1x V100 32GB)

Our results were obtained by running the run_eval_summarization.sh inferencing benchmarking script in the PyTorch 20.11-py3 NGC container on NVIDIA DGX-2 with (1x V100 32GB) GPU.

FP16

Batch sizeLatency AvgLatency 90%Latency 95%Latency 99%Throughput
10.670.840.891.041.49
40.961.141.241.434.16
81.331.591.721.906.01
161.992.392.572.698.04
323.414.314.534.639.36
646.668.618.758.929.55

To achieve these same results, follow the steps in the Quick Start Guide.

The inference performance metrics used are milliseconds per iteration. They are computed by iterating through the XSum test data with source length = 1024, target length = 60 and beam search = 6.

Inference performance: NVIDIA T4

Our results were obtained by running the run_eval_summarization.sh inferencing benchmarking script in the PyTorch 21.02-py3 NGC container on NVIDIA T4 with GPU.

FP16

Batch sizeLatency AvgLatency 90%Latency 95%Latency 99%Throughput
10.420.520.560.662.40
40.720.890.961.095.58
81.131.601.731.967.08
162.253.193.383.587.11
324.446.536.967.217.19

To achieve these same results, follow the steps in the Quick Start Guide.

The inference performance metrics used are milliseconds per iteration. They are computed by iterating through the XSum test data with source length = 1024, target length = 60 and beam search = 6.

Modal Content

NVIDIA uses cookies to improve your experience on our web site. We and our third-party partners also use cookies and other tools to collect and record information you provide as well as information about your interactions with our websites for performance improvement, analytics, and to assist in marketing efforts. By clicking "Accept All", you consent to our use of cookies and other tools as described in our Cookie Policy. You can manage your cookie settings by clicking on "Manage Settings." By continuing to use this site or by clicking one of the buttons below, you agree to our Terms of Service (which contains important waivers). Please see our Privacy Policy for more information on our privacy practices.