## Model Overview

BERT is a method of pre-training language representations which obtains state-of-the-art results on a wide array of NLP tasks.

## Model Architecture


BERT's model architecture is a multi-layer bidirectional Transformer encoder. Based on the model size, we have the following two default configurations of BERT:

| **Model** | **Hidden layers** | **Hidden unit size** | **Attention heads** | **Feedforward filter size** | **Max sequence length** | **Parameters** |
|:---------:|:----------:|:----:|:---:|:--------:|:---:|:----:|
|BERTBASE |12 encoder| 768| 12|4 x  768|512|110M|
|BERTLARGE|24 encoder|1024| 16|4 x 1024|512|330M|

BERT training consists of two steps, pre-training the language model in an unsupervised fashion on vast amounts of unannotated datasets, and then using this pre-trained model for fine-tuning for various NLP tasks, such as question and answer, sentence classification, or sentiment analysis. Fine-tuning typically adds an extra layer or two for the specific task and further trains the model using a task-specific annotated dataset, starting from the pre-trained backbone weights. The end-to-end process in depicted in the following image:

![](https://github.com/NVIDIA/DeepLearningExamples/raw/master/TensorFlow/LanguageModeling/BERT/data/images/bert_pipeline.png?raw=true)


Figure 1: BERT Pipeline


    
## Training

This model was trained using script available on [NGC](https://ngc.nvidia.com/catalog/resources/nvidia:bert_for_tensorflow) and in [GitHub repo](https://github.com/NVIDIA/DeepLearningExamples/tree/master/TensorFlow/LanguageModeling/BERT)

## Dataset

The following datasets were used to train this model:
- [SQuAD 1.1 + 2.0](https://rajpurkar.github.io/SQuAD-explorer/explore/1.1/dev/) - Reading comprehension dataset, consisting of questions posed by crowdworkers on a set of Wikipedia articles, where the answer to every question is a segment of text, or span, from the corresponding reading passage, or the question might be unanswerable.


## Performance

Performance numbers for this model are available in [NGC](https://ngc.nvidia.com/catalog/resources/nvidia:bert_for_tensorflow/performance)

## References
- [original paper](https://arxiv.org/abs/1810.04805)
- [NVIDIA model implementation in NGC](https://ngc.nvidia.com/catalog/resources/nvidia:bert_for_tensorflow)
- [NVIDIA model implementation on GitHub](https://github.com/NVIDIA/DeepLearningExamples/tree/master/TensorFlow/LanguageModeling/BERT)

## License

This model was trained using open-source software available in [Deep Learning Examples](https://github.com/NVIDIA/DeepLearningExamples/tree/master/TensorFlow/LanguageModeling/BERT) repository.
For terms of use, please refer to the license of the script and the datasets the model was derived from.

bert_tf_ckpt_large_qa_squad11_amp_384

BERT Large TensorFlow checkpoint trained for QA on SQuAD1.1 using AMP

BERT TF checkpoint (Large, QA, SQuAD 1.1, AMP, SeqLen=384)

Description

Publisher

Latest Version

Modified

Size