BART is a denoising autoencoder for pretraining sequence-to-sequence models.
Benchmarking
The following section shows how to run benchmarks measuring the model performance in training and inference modes.
Training performance benchmark
To benchmark the training performance on a specific batch size, source length, target length and dataset for one epoch, run:
bash scripts/run_training_benchmark.sh <batch size> <max source length> <max target length> <data dir>
The resulting NUM_GPU and PRECISION vs Throughput is stored in results/bart_pyt_training_benchmark_${DATESTAMP}/inference_benchmark.log
Inference performance benchmark
To benchmark the inference performance on a specific batch size, source length, target length and dataset, run:
bash scripts/run_inference_benchmark.sh <predict batch size> <eval beams> <max source length> <max target length> <model name or path> <data dir> <config path>
The resulting NUM_GPU and PRECISION vs Throughput is stored in results/bart_pyt_inference_benchmark_${DATESTAMP}/inference_benchmark.log
Results
The following sections provide details on how we achieved our performance and accuracy in training and inference.
Training accuracy results
Pre-training accuracy: NVIDIA DGX A100 (320x A100 80GB)
Our results were obtained by running the run_pretraining.sh training script in the PyTorch 22.08-py3 NGC container on 40 nodes NVIDIA DGX A100 (320x A100 80GB) GPUs.
| Nodes | Sequence Length | Batch size/GPU (BF16) | Accumulation Steps | Final loss - BF16 | Time to train (hrs) - BF16 |
|---|---|---|---|---|---|
| 40 | 128 | 200 | 1 | 0.5095 | 17.38 |
| 40 | 512 | 32 | 3 | 0.6085 | 3.28 |
Fine-tuning accuracy: NVIDIA DGX A100 (8x A100 80GB)
Our results for XSUM dataset were obtained by running the run_summarization.sh training script in the PyTorch 22.08-py3 NGC container on NVIDIA DGX A100 (8x A100 80GB) GPUs. Rogue1, rogue2 and rogueLSum scores list as accuracy.
| GPUs | Batch size (TF32, BF16) | R1 - TF32 | R2 - TF32 | RL - TF32 | R1 - BF16 | R2 - BF16 | RL - BF16 | Time to train (hrs) - TF32 | Time to train (hrs) - BF16 | Time to train (hrs) speedup (TF32 to BF16) |
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | 24, 40 | 45.22 | 22.03 | 36.95 | 44.91 | 21.85 | 36.78 | 2.41 | 1.69 | 1.43 |
| 8 | 192, 320 | 45.04 | 21.92 | 36.82 | 45.01 | 21.86 | 36.81 | 0.64 | 0.39 | 1.64 |
In addition,results for CNN-DM dataset are:
| GPUs | Batch size (TF32, BF16) | R1 - TF32 | R2 - TF32 | RL - TF32 | R1 - BF16 | R2 - BF16 | RL - BF16 | Time to train (hrs) - TF32 | Time to train (hrs) - BF16 | Time to train (hrs) speedup (TF32 to BF16) |
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | 24, 40 | 43.76 | 20.79 | 40.51 | 43.58 | 20.63 | 40.32 | 3.87 | 2.42 | 1.60 |
| 8 | 192, 320 | 43.77 | 20.77 | 40.53 | 43.76 | 20.73 | 40.50 | 0.73 | 0.45 | 1.62 |
Fine-tuning stability test
Our results for XSUM dataset were obtained by running the run_summarization.sh training script in the PyTorch 22.08-py3 NGC container on NVIDIA DGX A100 (8x A100 80GB) GPUs. Accuracy column lists rogue1 scores across 5 different training runs with different seeds on DGX A100.
| FP16, 8x GPUs | seed 1 | seed 2 | seed 3 | seed 4 | seed 5 | mean | std |
|---|---|---|---|---|---|---|---|
| rogue1 | 45.08 | 44.98 | 45.10 | 44.91 | 44.95 | 45.00 |
Training performance results
Pre-training performance: Single-node on NVIDIA DGX A100 (8x A100 80GB)
Our results were obtained by running the run_pretraining.sh training script in the PyTorch 22.08-py3 NGC container on single node NVIDIA DGX A100 (8x A100 80GB) GPUs.
| GPUs | Sequence Length | Batch size / GPU (TF32, BF16) | Throughput - TF32 | Throughput - BF16 | Throughput speedup (TF32 - BF16) | Weak scaling - TF32 | Weak scaling - BF16 |
|---|---|---|---|---|---|---|---|
| 1 | 128 | 100, 200 | 202.53 | 326.53 | 1.61 | 1 | 1 |
| 8 | 128 | 100, 200 | 1556.23 | 2572.86 | 1.65 | 7.68 | 7.88 |
| 1 | 512 | 16, 32 | 41.35 | 69.31 | 1.68 | 1 | 1 |
| 8 | 512 | 16, 32 | 317.85 | 549.67 | 1.73 | 7.69 | 7.93 |
Pre-training performance: Multi-node on NVIDIA DGX A100 (8x A100 80GB)
Our results were obtained by running the run_pretraining.sh training script in the PyTorch 22.08-py3 NGC container on multi node NVIDIA DGX A100 (8x A100 80GB) GPUs.
| Nodes | Sequence Length | Batch size / GPU (TF32, BF16) | Throughput - TF32 | Throughput - BF16 | Throughput speedup (TF32 - BF16) | Weak scaling - TF32 | Weak scaling - BF16 |
|---|---|---|---|---|---|---|---|
| 1 | 128 | 100, 200 | 1556.23 | 2572.86 | 1.65 | 1 | 1 |
| 20 | 128 | 100, 200 | 31067.96 | 52,459.02 | 1.69 | 19.96 | 20.39 |
| 40 | 128 | 100, 200 | 61,538.46 | 97028.51 | 1.58 | 39.54 | 37.71 |
| 1 | 512 | 16, 32 | 317.85 | 549.67 | 1.73 | 1 | 1 |
| 20 | 512 | 16, 32 | 5953.49 | 10520.54 | 1.77 | 18.73 | 19.14 |
| 40 | 512 | 16, 32 | 11,636.36 | 19948.05 | 1.71 | 36.61 | 36.29 |
Fine-tuning performance: NVIDIA DGX A100 (8x A100 80GB)
Our results were obtained by running the run_summarization.sh training script in the PyTorch 22.08-py3 NGC container on NVIDIA DGX A100 (8x A100 80GB) GPUs. Performance numbers (in items per second) were averaged over an entire training epoch.
| GPUs | Batch size / GPU (TF32, BF16) | Throughput - TF32 | Throughput - BF16 | Throughput speedup (TF32 - BF16) | Weak scaling - TF32 | Weak scaling - BF16 |
|---|---|---|---|---|---|---|
| 1 | 24, 40 | 48.61 | 74.59 | 1.53 | 1.00 | 1.00 |
| 8 | 24, 40 | 243.03 | 390.24 | 1.61 | 3.39 | 4.08 |
To achieve these same results, follow the steps in the Quick Start Guide.
The performance metrics used are tokens per second computed from iterating through an entire epoch of XSum dataset with source length = 1024 and target length = 60.
Inference performance results
Inference performance: NVIDIA DGX A100 (1x A100 80GB)
Our results were obtained by running the run_eval_summarization.sh inferencing benchmarking script in the PyTorch 22.08-py3 NGC container on NVIDIA DGX A100 (1x A100 80GB) GPU.
BF16
| Batch size | Latency Avg | Latency 90% | Latency 95% | Latency 99% | Throughput |
|---|---|---|---|---|---|
| 1 | 0.28 | 0.35 | 0.38 | 0.46 | 3.54 |
| 4 | 0.44 | 0.52 | 0.56 | 0.71 | 9.16 |
| 8 | 0.63 | 0.75 | 0.83 | 0.98 | 12.79 |
| 16 | 0.98 | 1.2 | 1.29 | 1.47 | 16.3 |
| 32 | 1.8 | 2.27 | 2.47 | 2.63 | 17.73 |
| 64 | 3.78 | 4.85 | 5.21 | 5.4 | 16.83 |
| 128 | 8.29 | 10.53 | 10.69 | 10.93 | 15.36 |
To achieve these same results, follow the steps in the Quick Start Guide.
The inference performance metrics used are milliseconds per iteration. They are computed by iterating through the XSum test data with source length = 1024, target length = 60 and beam search = 6.