BERT is a method of pre-training language representations which obtains state-of-the-art results on a wide array of NLP tasks.
The following section lists the requirements you need to meet to start training the BERT model.
Requirements
This repository contains a Dockerfile that extends the CUDA NGC container and encapsulates some dependencies. Aside from these dependencies, ensure you have the following components:
- NVIDIA Docker
- PaddlePaddle 22.12-py3 NGC container or newer
- Supported GPUs:
For more information about how to get started with NGC containers, refer to the following sections from the NVIDIA GPU Cloud Documentation and the Deep Learning DGX Documentation:
For those unable to use the PaddlePaddle NGC container, to set up the required environment or create your own container, refer to the versioned NVIDIA Container Support Matrix.
For multi-node, the sample provided in this repository requires Enroot and Pyxis set up on a SLURM cluster.
More information on how to set up and launch can be found in the Multi-node Documentation.