BERT for TensorFlow | NVIDIA NGC

NVIDIA

BERT for TensorFlow

Resource

NVIDIA

BERT for TensorFlow

BERT is a method of pre-training language representations which obtains state-of-the-art results on a wide array of NLP tasks.

Performance

Benchmarking

The following section shows how to run benchmarks measuring the model performance in training and inference modes.

Benchmarking can be performed for both training and inference. Both scripts run the BERT model for fine tuning. You can specify whether benchmarking is performed in FP16 or FP32 by specifying it as an argument to the benchmarking scripts.

Both of these benchmarking scripts enable you to run a number of epochs and extract performance numbers.

Training performance benchmark

Training benchmarking can be performed by running the script:

scripts/finetune_train_benchmark.sh squad <fp16/fp32> <use_xla> <num_gpu> <batch_size/gpu> <lr>

Inference performance benchmark

Inference benchmarking can be performed by running the script:

scripts/finetune_inference_benchmark.sh squad <fp16/fp32> <use_xla> <batch_size> <path-to-checkpoint>

Results

The following sections provide details on how we achieved our performance and accuracy in training and inference for Question Answering fine tuning.

Training accuracy results

Our results were obtained by running the run_squad.py training script in the TensorFlow 19.03-py3 NGC container on NVIDIA DGX-1 with 8x V100 32G GPUs.

Number of GPUs	Batch size per GPU	Training time with FP16 (Hrs)	Training time with FP32 (Hrs)
8	4	0.51	0.77

Training stability test

The following tables compare F1 scores across 5 different training runs with different seeds, for both FP16 and FP32 respectively. The runs showcase consistent convergence on all 5 seeds with very little deviation.

FP16, 8x GPUs	seed #1	seed #2	seed #3	seed #4	seed #5	mean	std
F1	91.16	90.69	90.99	90.94	91.17	90.99	0.196
Exact match	84.2	83.68	84.14	83.95	84.34	84.06	0.255

FP32, 8x GPUs	seed #1	seed #2	seed #3	seed #4	seed #5	mean	std
F1	90.67	90.8	90.94	90.83	90.93	90.83	0.11
Exact match	83.56	83.96	83.99	83.95	84.12	83.92	0.21

Training performance results

Our results were obtained by running batch sizes up to 3x GPUs on a 16GB V100 and up to 10x GPUs on a 32G V100 with mixed precision.

NVIDIA DGX-1 (8x V100 16G)

Our results were obtained by running the scripts/run_squad.sh training script in the TensorFlow 19.03-py3 NGC container on NVIDIA DGX-1 with 8x V100 16G GPUs. Performance numbers (in tokens per second) were averaged over an entire training epoch.

Number of GPUs	Batch size per GPU	FP32 sentences/sec	FP16 sentences/sec	Speed-up with mixed precision	Multi-gpu weak scaling with FP32	Multi-gpu weak scaling with FP16
1	2	7.41	11.86	1.6	1.0	1.0
4	2	23.699	35.34	1.49	3.2	2.98
8	2	44.29	64.96	1.47	5.98	5.48

Number of GPUs	Batch size per GPU	FP32 sentences/sec	FP16 sentences/sec	Speed-up with mixed precision	Multi-gpu weak scaling with FP32	Multi-gpu weak scaling with FP16
1	3	-	14.86	-	-	1.0
4	3	-	44.17	-	-	2.97
8	3	-	78.89	-	-	5.3

Note: The respective values for FP32 runs that use a batch size of 3 are not available due to out of memory errors that arise. Batch size of 3 is only available on using FP16.