This implementation of Transformer model architecture is based on the optimized implementation in Fairseq NLP toolkit.
Performance
Results
Training Accuracy Results
In order to test accuracy of our implementation we have run experiments with different seeds for 100 epochs with batch size 5120 per GPU and learining rate 6e-4 in the pytorch-18.12-py3 Docker container. Plot below shows BLEU score changes.
Training Performance Results
Running this code with the provided hyperparameters will allow you to achieve the following results. Our setup is a DGX-1 with 8x Tesla V100 16GB. We've verified our results after training 32 epochs to obtain multi-GPU and mixed precision scaling results.
| GPU count | Mixed precision BLEU | fp32 BLEU | Mixed precision training time | fp32 training time |
|---|---|---|---|---|
| 8 | 28.69 | 28.43 | 446 min | 1896 min |
| 4 | 28.35 | 28.31 | 834 min | 3733 min |
In some cases we can train further with the same setup to achieve slightly better results.
| GPU count | Precision | BLEU score | Epochs to train | Training time |
|---|---|---|---|---|
| 4 | fp16 | 28.67 | 74 | 1925 min |
| 4 | fp32 | 28.40 | 47 | 5478 min |
Results here are the best we achieved. We've observed a large variance in BLEU, while using random seed. Nearly all setups reach 28.4 BLEU, although the time it takes also varies between setups. We also observed a good rate of week scaling. We measured performance in tokens (words) per second.
| GPU count | Mixed precision | FP32 | FP32/Mixed speedup | Mixed precision week scaling | FP32 week scaling |
|---|---|---|---|---|---|
| 1 | 37650 | 8630 | 4.36 | 1.0 | 1.0 |
| 4 | 132700 | 30500 | 4.35 | 3.52 | 3.53 |
| 8 | 260000 | 61000 | 4.26 | 6.91 | 7.07 |
Inference performance results
All results were obtained by generate.py inference script in the pytorch-19.01-py3 Docker container. Inference was run on a single GPU.
| GPU | Mixed precision | FP32 | FP16/Mixed speedup |
|---|---|---|---|
| Tesla V100 | 5129.34 | 3396.09 | 1.51 |