NGC | Catalog
Welcome Guest


For downloads and more information, please view on a desktop device.
Logo for BertBaseUncasedSQuADv1


BERT Base Uncased model for Question Answering finetuned with NeMo on SQuAD v1.1 dataset.



Use Case

Natural Language Processing


PyTorch with NeMo

Latest Version



September 24, 2020


417.7 MB


This is a checkpoint for the BERT Base Uncased for Question Answering finetuned in NeMo on the uncased question answering dataset SQuADv1.1 The model is trained for 2 epochs on a DGX1 with 8 V100 GPUs using Apex/Amp optimization level O2. On the development data this model achieves an exact match (EM) score of 82.74 and F1 score of 89.79. The pretrained BERT model checkpoint was also trained with NeMo, on uncased English Wikipedia and BookCorpus.

Please be sure to download the latest version in order to ensure compatibility with the latest NeMo release.

  • - finetuned BERT encoder weights
  • - finetuned BERT question answering head weights
  • bert-config.json - the config file used to initialize BERT network architecture in NeMo

More Details

BERT, or Bidirectional Encoder Representations from Transformers, is a neural approach to pre-train language representations which obtains near state-of-the-art results on a wide array of Natural Language Processing (NLP) tasks, including SQuAD Question Answering dataset. Stanford Question Answering Dataset (SQuAD) is a reading comprehension dataset, consisting of questions posed by crowdworkers on a set of Wikipedia articles, where the answer to every question is a segment of text, or span, from the corresponding reading passage. Unlike SQuADv1.1, SQuADv2.0 can contain questions that are unanswerable.

Apart from the BERT architecture this model also includes a question answering model head, which is stacked on top of BERT. This question answering head is a token classifier and is, more specifically, a single fully connected layer.

Training works as follows: The user provides training and evaluation data in text form in JSON format. This data is parsed by scripts in NeMo and converted into model input. The input sequence is a concatenatenation of a tokenized query and its according reading passage. The question answering head predicts for each token in the reading passage or context if it is the start or end of the answer span. The model is trained using cross entropy loss.


Source code and developer guide is available at Refer to documentation at Code to pretrain and reproduce this model checkpoint are available at

This model checkpoint can be used for either inference or finetuning on custom question answering datasets, as long as they are in the required format. More details at

In the following we show examples for how to finetune BERT on SQuAD v1.1, how to do evaluation and inference.

Before you start, please download SQuADv1.1 dataset either directly from or using

Usage example 1: Training

  1. Download the pretrained BERT encoder checkpoint and bert-config.json from

  2. Run BERT Base Uncased on e.g. DGX1 with 8 V100 GPUs

     cd examples/nlp/question_answering;
     python -m torch.distributed.launch --nproc_per_node=8 --mode train_eval --amp_opt_level O2 --num_gpus 8 --train_file=/path_to/squad/v1.1/train-v1.1.json --eval_file /path_to/squad/v1.1/dev-v1.1.json --bert_checkpoint /path_to/ --bert_config /path_to/bert-config.json --pretrained_model_name bert-base-uncased --batch_size 3 --num_epochs 2 --lr_policy SquareRootAnnealing --optimizer fused_adam --lr 3e-5 --do_lower_case --no_data_cache

    Checkpoints will be store at --work_dir folder.

Usage example 2: Evaluation

  1. Download finetuned checkpoints,, put them into /path_to/checkpoints_dir. Download bert-config.json.

  2. Run evaluation:

     cd examples/nlp/question_answering;
     python --mode eval --amp_opt_level O2 --eval_file /path_to/squad/v1.1/dev-v1.1.json --checkpoint_dir /path_to/checkpoints_dir --bert_config /path_to/bert-config.json --pretrained_model_name bert-base-uncased --do_lower_case --no_data_cache

Usage example 3: Inference

  1. Download finetuned checkpoints,, put them into /path_to/checkpoints_dir. Download bert-config.json.

  2. Run inference on custom test_file.json:

     cd examples/nlp/question_answering;
     python --mode test --amp_opt_level O2 --test_file test_file.json --checkpoint_dir /path_to/checkpoints_dir --bert_config /path_to/bert-config.json --pretrained_model_name bert-base-uncased --do_lower_case --no_data_cache

Metrics will be logged, model predictions are stored at --output_prediction_file