To train your model using mixed or TF32 precision with Tensor Cores or using FP32, perform the following steps using the default parameters of the Transformer-XL base model on the WikiText-103 dataset.

For the specifics concerning training and inference, see the Advanced section.

1. Clone the repository.

git clone https://github.com/NVIDIA/DeepLearningExamples
cd DeepLearningExamples/PyTorch/LanguageModeling/Transformer-XL

2. Download and preprocess the dataset.

bash getdata.sh

3. Build the Transformer-XL PyTorch NGC container.

bash pytorch/scripts/docker/build.sh

4. Start an interactive session in the NGC container to run training/inference.

bash pytorch/scripts/docker/interactive.sh

5. Start training.

This repository contains a number of predefined configurations to run the training on NVIDIA DGX-1, NVIDIA DGX-2H or NVIDIA DGX A100 nodes.

To start the training on NVIDIA DGX-1 or NVIDIA DGX-2H, run:

bash run_wt103_{base,large}.sh train <#GPUs> --config {dgx1,dgx2}_<#GPUs>gpu_{fp16,fp32}

To start the training on NVIDIA DGX A100, run:

bash run_wt103_{base,large}.sh train <#GPUs> --config dgxa100_<#GPUs>gpu_{fp16,tf32}

• use the run_wt103_base.sh script to train the base model, and use the run_wt103_large.sh script to train the large model
• the training is executed on <#GPUs> GPUs, supported values for <#GPUs> for NVIDIA DGX-1 and NVIDIA DGX A100 are: 1, 2, 4, 8 and for NVIDIA DGX-2H: 1, 2, 4, 8, 16
• use configs with the dgx1 prefix to run on a NVIDIA DGX-1, configs with the dgx2 prefix to run on a NVIDIA DGX-2H and configs with the dgxa100 prefix to run on a NVIDIA DGX A100
• configs with the fp16 suffix are launching mixed precision training, configs with the fp32 suffix are launching FP32 training, configs with the tf32 suffix are launching TF32 training

Examples:

To launch TF32 training of the base Transformer-XL model on a NVIDIA DGX A100 using 8 GPUs, run:

bash run_wt103_base.sh train 8 --config dgxa100_8gpu_tf32

To launch FP32 training of the base Transformer-XL model on a NVIDIA DGX-1 using 8 GPUs, run:

bash run_wt103_base.sh train 8 --config dgx1_8gpu_fp32

To launch mixed precision training of the large Transformer-XL model on a NVIDIA DGX-2H using 16 GPUs, run:

bash run_wt103_large.sh train 16 --config dgx2_16gpu_fp16

To launch mixed precision training of the large Transformer-XL model on a NVIDIA DGX A100 using 8 GPUs, run:

bash run_wt103_large.sh train 8 --config dgxa100_8gpu_fp16

To run on multiple nodes, see the Multi-node section.

For more information on the available options, and for an explanation of what happens at the end of training, refer to the Training process section.

6. Start evaluation.

To start inference on the test set using <#GPUs> GPUs, run:

bash run_wt103_{base,large}.sh eval <#GPUs> [--fp16] [--type {pytorch, torchscript}]

Select run_wt103_base.sh for the base Transformer-XL model and run_wt103_large.sh for the large Transformer-XL model. The --fp16 flag is optional, however, if it's specified, then the script launches mixed precision inference with Tensor Cores. If the flag is not present, then the script launches FP32 inference on NVIDIA Volta and NVIDIA Turing GPUs and TF32 inference on NVIDIA Ampere GPUs.

By default, the script is loading the checkpoint from LM-TFM/checkpoint_best.pt, which contains the model corresponding to the lowest value of the validation loss from the previous training run. Path to the checkpoint can be customized by setting the --model flag.

Inference can use pure Python execution or TorchScript from using the --type flag.

Supported values for <#GPUs> are: 1, 2, 4, 8 for NVIDIA DGX-1 and NVIDIA DGX A100 and 1, 2, 4, 8, 16 for NVIDIA DGX-2H.

Additionally, one can pass the input text directly from the command-line using the --manual flag. This mode of operation supports only 1 GPU and batch size of 1. The script outputs average loss and perplexity for the provided input text.

Examples:

bash run_wt103_base.sh eval 1 \
  --model LM-TFM/checkpoint_best.pt \
  --fp16 \
  --manual "recognize speech"

===============================================================================

| test loss  6.20 | test ppl   494.291
===============================================================================

bash run_wt103_base.sh eval 1 \
  --model LM-TFM/checkpoint_best.pt \
  --fp16 \
  --manual "wreck a nice beach"

===============================================================================

| test loss  8.04 | test ppl  3099.706
===============================================================================

For more information on the available options, refer to the Inference process section.