GNMT v2 for TensorFlow

NVIDIA

Resource

NVIDIA

GNMT v2 for TensorFlow

The GNMT v2 model is an improved version of the first Google's Neural Machine Translation System with a modified attention mechanism.

Performance

Results

The following sections provide details on how we achieved our performance and accuracy in training and inference.

Training accuracy results

Our results were obtained by running the nmt.py script in the tensorflow-19.03-py3 NGC container on NVIDIA DGX-1 with 8x V100 16G GPUs.

Commands to launch the training:

for 1 GPUs in mixed precision:
python nmt.py --output_dir=results --batch_size=192 --learning_rate=8e-4

for 8 GPUs in mixed precision:
python nmt.py --output_dir=results --batch_size=1536 --num_gpus=8 --learning_rate=2e-3

for 1 GPUs in FP32:
python nmt.py --output_dir=results --batch_size=128 --learning_rate=5e-4 --use_amp=false

for 8 GPUs in FP32:
python nmt.py --output_dir=results --batch_size=1024 --num_gpus=8 --learning_rate=2e-3 --use_amp=false

Number of GPUs	Mixed precision batch size/GPU	FP32 batch size/GPU	Mixed precision BLEU	FP32 BLEU	Mixed precision training time	FP32 training time
1	192	128	24.58	24.64	789 min	1375 min
8	192	128	24.46	24.51	184 min	262 min

In the following plot, the BLEU scores after each training epoch for different configurations are displayed.

BLEUScore

Training stability test

The GNMT v2 model was trained for 6 epochs, starting from 6 different initial random seeds. After each training epoch, the model was evaluated on the test dataset and the BLEU score was recorded. The training was performed in the tensorflow-19.03-py3 NGC container on NVIDIA DGX-1 with 8 Tesla V100 16G GPUs.

In the following table, the BLEU scores after each training epoch for different initial random seeds are displayed.

Epoch	Average	Standard deviation	Minimum	Maximum	Median
1	19.943	0.240	19.670	20.290	19.855
2	21.750	0.197	21.550	22.110	21.690
3	22.408	0.150	22.160	22.630	22.430
4	23.057	0.219	22.770	23.440	22.985
5	23.897	0.142	23.700	24.080	23.915
6	24.243	0.174	24.030	24.460	24.235

Training performance results

Our results were obtained by running the nmt.py script in the tensorflow-19.03-py3 NGC container on NVIDIA DGX-1 with 8x V100 16G GPUs. Performance numbers (in tokens per second) were averaged over an entire training epoch.

Number of GPUs	Mixed precision batch size/GPU	FP32 batch size/GPU	Mixed precision tokens/s	FP32 tokens/s	Mixed precision speedup	Mixed precision multi-gpu weak scaling	FP32 multi-gpu weak scaling
1	192	128	22 297	12 767	1.74	1.00	1.00
8	192	128	133 992	83 337	1.61	6.01	6.53

To achieve these same results, follow the Quick Start Guide outlined above.

Inference performance results

Our results were obtained by running the scripts/translate.py script in the tensorflow-19.03-py3 NGC container on NVIDIA DGX-1 with a single V100 16G GPUs. The benchmark requires a checkpoint from a fully trained model.

To launch the inference benchmark in mixed precision on 1 GPU, run:

python scripts/inference_benchmark.py --output_dir=/path/to/trained/model --beam_width 1,2,5,10 --infer_batch_size 32,128,512

To launch the inference benchmark in FP32 on 1 GPU, run:

python scripts/inference_benchmark.py --output_dir=/path/to/trained/model --beam_width 1,2,5,10 --infer_batch_size 32,128,512 --use_amp=false

Batch size	Beam size	Mixed precision BLEU	FP32 BLEU	Mixed precision tokens/s	FP32 tokens/s
32	1	23.43	23.47	8180	7555
32	2	24.14	24.13	6870	6359
32	5	24.43	24.47	4729	3991
32	10	24.29	24.29	2344	2495
128	1	23.43	23.47	19343	14810
128	2	24.15	24.13	10267	9956
128	5	24.45	24.47	7009	5224
128	10	24.30	24.29	3873	2855
512	1	23.45	23.47	32325	20782
512	2	24.17	24.13	18917	13702
512	5	24.46	24.47	8789	6036
512	10	24.29	24.29	4684	3149

To achieve these same results, follow the Quick Start Guide outlined above.