BERT is a method of pre-training language representations which obtains state-of-the-art results on a wide array of NLP tasks.
Benchmarking
The following section shows the steps to run benchmarks measuring the model performance in training and inference modes.
Training performance benchmark
Training performance benchmarks for pre-training can be obtained by running scripts/run_pretraining.sh, and fine-tuning can be obtained by running scripts/run_squad.sh for SQuAD, respectively. The required parameters can be passed through the command-line as described in Training process.
To benchmark the training performance on a specific batch size for pre-training, refer to Pre-training and turn on the <benchmark> flags. An example call to run pretraining for 20 steps (10 steps for warmup and 10 steps to measure, both in phase1 and phase2) and generate throughput numbers:
bash scripts/run_pretraining.sh \
256 6e-3 amp 8 0.2843 7038 200 false \
32 0 bert_pretraining 32 4e-3 0.128 1563 128 \
/path/to/dataset/phase1 \
/path/to/dataset/phase2 \
/workspace/bert \
None \
/path/to/wikipedia/source \
32 128 4 0.9 64 static \
None true 10 10
To benchmark the training performance on a specific batch size for SQuAD, refer to Fine-tuning and turn on the <benchmark> flags. An example call to run training for 200 steps (100 steps for warmup and 100 steps to measure), and generate throughput numbers:
bash scripts/run_squad.sh \
/path/to/pretrained/model \
2 32 4.6e-5 0.2 amp 8 42 \
/path/to/squad/v1.1 \
vocab/bert-large-uncased-vocab.txt \
results/checkpoints \
train \
bert_configs/bert-large-uncased.json \
-1 true 100 100
Inference performance benchmark
Inference performance benchmark for fine-tuning can be obtained by running scripts/run_squad.sh. The required parameters can be passed through the command-line as described in Inference process.
To benchmark the inference performance on a specific batch size for SQuAD, run:
bash scripts/run_squad.sh \
<pre-trained model path> \
<epochs> <batch size> <learning rate> <warmup_proportion> <amp|fp32> <num_gpus> <seed> \
<path to SQuAD dataset> \
<path to vocab set> \
<results directory> \
eval \
<BERT config path> \
<max steps> <benchmark> <benchmark_steps> <benchmark_warmup_steps>
An example call to run inference and generate throughput numbers:
bash scripts/run_squad.sh \
/path/to/pretrained/model \
2 32 4.6e-5 0.2 amp 8 42 \
/path/to/squad/v1.1 \
vocab/bert-large-uncased-vocab.txt \
results/checkpoints \
eval \
bert_configs/bert-large-uncased.json \
-1 true 100 100
Results
The following sections provide details on how we achieved our performance and accuracy in training and inference.
Training accuracy results
Our results were obtained by running the scripts/run_squad.sh and scripts/run_pretraining.sh training scripts in the paddle NGC container unless otherwise specified.
Pre-training loss results: NVIDIA DGX A100 (8x A100 80GB)
| DGX System | GPUs / Node | Precision | Accumulated Batch size / GPU (Phase 1 and Phase 2) | Accumulation steps (Phase 1 and Phase 2) | Final Loss | Time to train(hours) | Time to train speedup (TF32 to mixed precision) |
|---|---|---|---|---|---|---|---|
| 32 x DGX A100 80GB | 8 | AMP | 256 and 128 | 1 and 4 | 1.409 | ~ 1.2 hours | 1.72 |
| 32 x DGX A100 80GB | 8 | TF32 | 128 and 16b | 2 and 8 | 1.421 | ~ 2.5 hours | 1 |
Pre-training loss curves
Fine-tuning accuracy results: NVIDIA DGX A100 (8x A100 80GB)
- SQuAD
| GPUs | Batch size / GPU (TF32 and FP16) | Accuracy - TF32(% F1) | Accuracy - mixed precision(% F1) | Time to train(hours) - TF32 | Time to train(hours) - mixed precision | Time to train speedup (TF32 to mixed precision) |
|---|---|---|---|---|---|---|
| 8 | 32 | 91.13 | 91.11 | 0.078 | 0.056 | 1.39 |
Training stability test
Pre-training stability test
| Accuracy Metric | Seed 0 | Seed 1 | Seed 2 | Seed 3 | Seed 4 | Mean | Standard Deviation |
|---|---|---|---|---|---|---|---|
| Final Loss | 1.409 | 1.367 | 1.528 | 1.434 | 1.470 | 1.442 | 0.049 |
Fine-tuning stability test
- SQuAD
Training stability with 8 GPUs, FP16 computations, batch size of 32:
| Accuracy Metric | Seed 0 | Seed 1 | Seed 2 | Seed 3 | Seed 4 | Seed 5 | Seed 6 | Seed 7 | Seed 8 | Seed 9 | Mean | Standard Deviation |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Exact Match % | 84.07 | 84.39 | 83.94 | 83.78 | 83.85 | 84.47 | 84.13 | 84.20 | 84.03 | 83.80 | 84.07 | 0.225 |
| f1 % | 90.86 | 91.00 | 90.82 | 90.56 | 90.76 | 91.11 | 90.77 | 90.90 | 90.65 | 90.54 | 90.80 | 0.173 |
Training performance results
Training performance: NVIDIA DGX A100 (8x A100 80GB)
Our results were obtained by running the script run_pretraining.sh in the PaddlePaddle:22.12-py3 NGC container on NVIDIA DGX A100 (8x A100 80GB) GPUs. Performance numbers (in sequences per second) were averaged over a few training iterations.
Pre-training NVIDIA DGX A100 (8x A100 80GB)
| GPUs | Batch size / GPU (TF32 and FP16) | Accumulation steps (TF32 and FP16) | Sequence length | Throughput - TF32(sequences/sec) | Throughput - mixed precision(sequences/sec) | Throughput speedup (TF32 - mixed precision) | Weak scaling - TF32 | Weak scaling - mixed precision |
|---|---|---|---|---|---|---|---|---|
| 1 | 8192 and 8192 | 64 and 32 | 128 | 307 | 633 | 2.06 | 1.00 | 1.00 |
| 8 | 8192 and 8192 | 64 and 32 | 128 | 2428 | 4990 | 2.06 | 7.91 | 7.88 |
| 1 | 4096 and 4096 | 256 and 128 | 512 | 107 | 219 | 2.05 | 1.00 | 1.00 |
| 8 | 4096 and 4096 | 256 and 128 | 512 | 851 | 1724 | 2.26 | 7.95 | 7.87 |
Pre-training NVIDIA DGX A100 (8x A100 80GB) Multi-node Scaling
| Nodes | GPUs / node | Batch size / GPU (TF32 and FP16) | Accumulated Batch size / GPU (TF32 and FP16) | Accumulation steps (TF32 and FP16) | Sequence length | Mixed Precision Throughput | Mixed Precision Strong Scaling | TF32 Throughput | TF32 Strong Scaling | Speedup (Mixed Precision to TF32) |
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | 8 | 126 and 256 | 8192 and 8192 | 64 and 32 | 128 | 4990 | 1 | 2428 | 1 | 2.06 |
| 2 | 8 | 126 and 256 | 4096 and 4096 | 32 and 16 | 128 | 9581 | 1.92 | 4638 | 1.91 | 2.07 |
| 4 | 8 | 126 and 256 | 2048 and 2048 | 16 and 8 | 128 | 19262 | 3.86 | 9445 | 3.89 | 2.04 |
| 8 | 8 | 126 and 256 | 1024 and 1024 | 8 and 4 | 128 | 37526 | 7.52 | 18335 | 7.55 | 2.05 |
| 16 | 8 | 126 and 256 | 512 and 512 | 4 and 2 | 128 | 71156 | 14.26 | 35526 | 14.63 | 2.00 |
| 32 | 8 | 126 and 256 | 256 and 256 | 2 and 1 | 128 | 142087 | 28.47 | 69701 | 28.71 | 2.04 |
| 1 | 8 | 16 and 32 | 4096 and 4096 | 256 and 128 | 512 | 1724 | 1 | 851 | 1 | 2.03 |
| 2 | 8 | 16 and 32 | 2048 and 2048 | 128 and 64 | 512 | 3305 | 1.92 | 1601 | 1.88 | 2.06 |
| 4 | 8 | 16 and 32 | 1024 and 1024 | 64 and 32 | 512 | 6492 | 3.77 | 3240 | 3.81 | 2.00 |
| 8 | 8 | 16 and 32 | 512 and 512 | 32 and 16 | 512 | 12884 | 7.47 | 6329 | 7.44 | 2.04 |
| 16 | 8 | 16 and 32 | 256 and 256 | 16 and 8 | 512 | 25493 | 14.79 | 12273 | 14.42 | 2.08 |
| 32 | 8 | 16 and 32 | 128 and 128 | 8 and 4 | 512 | 49307 | 28.60 | 24047 | 28.26 | 2.05 |
Fine-tuning NVIDIA DGX A100 (8x A100 80GB)
- SQuAD
| GPUs | Batch size / GPU (TF32 and FP16) | Throughput - TF32(sequences/sec) | Throughput - mixed precision(sequences/sec) | Throughput speedup (TF32 - mixed precision) | Weak scaling - TF32 | Weak scaling - mixed precision |
|---|---|---|---|---|---|---|
| 1 | 32 and 32 | 83 | 120 | 1.45 | 1.00 | 1.00 |
| 8 | 32 and 32 | 629 | 876 | 1.39 | 7.59 | 7.30 |
Inference performance results
Inference performance: NVIDIA DGX A100 (1x A100 80GB)
Our results were obtained by running scripts/run_squad.sh in the PaddlePaddle:22.08-py3 NGC container on NVIDIA DGX A100 with (1x A100 80G) GPUs.
Fine-tuning inference on NVIDIA DGX A100 (1x A100 80GB)
- SQuAD
| GPUs | Batch Size (TF32/FP16) | Sequence Length | Throughput - TF32(sequences/sec) | Throughput - Mixed Precision(sequences/sec) |
|---|---|---|---|---|
| 1 | 32/32 | 384 | 131 | 158 |
To achieve these same results, follow the steps in the Quick Start Guide.
The inference performance metrics used were items/second.