BERT for PaddlePaddle

NVIDIA Deep Learning Examples

Resource

NVIDIA Deep Learning Examples

BERT for PaddlePaddle

BERT is a method of pre-training language representations which obtains state-of-the-art results on a wide array of NLP tasks.

To train your model using mixed or TF32 precision with Tensor Cores, perform the following steps using the default parameters of the BERT model. Training configurations to run on 8 x A100 cards and examples of usage are provided at the end of this section. For the specifics concerning training and inference, refer to the Advanced section.

Clone the repository.

git clone https://github.com/NVIDIA/DeepLearningExamples.git
cd DeepLearningExamples/PaddlePaddle/LanguageModeling/BERT

Download the NVIDIA pre-trained checkpoint. If you want to use a pre-trained checkpoint, visit NGC. This pre-trained checkpoint is used to fine-tune on SQuAD. Ensure you unzip the downloaded file and place the checkpoint in the checkpoints/ folder. For a checkpoint already fine-tuned for QA on SQuAD v1.1 visit NGC.
Build BERT on top of the NGC container.

bash scripts/docker/build.sh

Start an interactive session in the NGC container to run training/inference.

bash scripts/docker/launch.sh

By default:

Checkpoints of pretraining and fine-tuning routines are stored in the results/ folder.
Paddle native logs are stored in the log/ folder.
DLLogger's outputs are stored in the results/ folder.

Download the dataset.

This repository provides scripts to download, verify, and extract the following datasets:

SQuAD (fine-tuning for question answering)
Wikipedia (pre-training)

To download, verify, extract the datasets, run:

bash data/create_datasets_from_start.sh

Note: For fine-tuning only, downloading the Wikipedia dataset can be skipped by commenting it out.

Note: Ensure a complete Wikipedia download. But if the download failed in LDDL, remove the output directory data/wikipedia/ and start over again.

Start pre-training.

To run on a single node 8 x A100 80G cards from within the container, you can use the following script to run pre-training.

bash scripts/run_pretraining.sh

The default hyperparameters are set to run on 8x A100 80G cards.

To run on multiple nodes, refer to the Multi-node section.

Start fine-tuning with the SQuAD dataset.

The above pre-trained BERT representations can be fine-tuned with just one additional output layer for a state-of-the-art question answering system. Running the following script launches fine-tuning for question answering with the SQuAD dataset.

bash scripts/run_squad.sh /workspace/bert/checkpoints/<pre-trained_checkpoint>

Start validation/evaluation.

For SQuAD, validation can be performed with the bash scripts/run_squad.sh /workspace/bert/checkpoints/<pre-trained_checkpoint>, setting mode to eval in scripts/run_squad.sh as follows:

mode=${12:-"eval"}

Start inference/predictions. Inference can be performed with the bash scripts/run_squad.sh /workspace/bert/checkpoints/<pre-trained_checkpoint>, setting mode to prediction in scripts/run_squad.sh as follows:

mode=${12:-"prediction"}

Note:

Both in prediction and eval mode, the inference process will be performed and the prediction results will be saved into <OUT_DIR>/bert-large-uncased/squad/predictions.json, set in scripts/run_squad.sh as follows:

OUT_DIR=${11:-"/results"} # For SQuAD.

In eval mode, after the inference process is completed, the script will further evalute the predictions.json on the test dateset and output two metrics: the average exact match score and the average F1 score.
predictions.json only contains a dict of mapping the questions' id to their predicted answers. For example:

{
    "56be4db0acb8001400a502ec": "Denver Broncos",
    "56be4db0acb8001400a502ed": "Carolina Panthers",
}

All the reference (such as contexts, questions, answers) can be located in test dataset (dev-v1.1.json) according to the unique id. For example:

{
    "answers": [{"answer_start": 177, "text": "Denver Broncos"}, {"answer_start": 177, "text": "Denver Broncos"}, {"answer_start": 177, "text": "Denver Broncos"}], 
    "question": "Which NFL team represented the AFC at Super Bowl 50?", 
    "id": "56be4db0acb8001400a502ec"
}

This repository contains some predefined configurations to run the pre-training and SQuAD on NVIDIA DGX A100 nodes in scripts/configs/pretrain_config.sh and scripts/configs/squad_config.sh. For example, to use the default DGX A100 8 gpu config, run:

bash scripts/run_pretraining.sh $(source scripts/configs/pretrain_config.sh && dgxa100-80g_8gpu_amp) # For pre-training with mixed precision.
bash scripts/run_pretraining.sh $(source scripts/configs/pretrain_config.sh && dgxa100-80g_8gpu_tf32) # For pre-training with TF32 precision.
bash scripts/run_squad.sh $(source scripts/configs/squad_config.sh && dgxa100-80g_8gpu_amp)  # For the SQuAD v1.1 dataset with mixed precision.
bash scripts/run_squad.sh $(source scripts/configs/squad_config.sh && dgxa100-80g_8gpu_tf32)  # For the SQuAD v1.1 dataset with TF32 precision.