Transformer-XL is a transformer-based language model with a segment-level recurrence and a novel relative positional encoding.
The following section lists the requirements that you need to meet in order to start training the Transformer-XL model.
Requirements
This repository contains Dockerfile which extends the TensorFlow NGC container
and encapsulates some dependencies. Aside from these dependencies, ensure you
have the following components:
- NVIDIA Docker
- TensorFlow 20.06-tf1-py3 NGC container
- GPU-based architecture:
- NVIDIA Volta
- NVIDIA Turing
- NVIDIA Ampere architecture
For more information about how to get started with NGC containers, see the following sections from the NVIDIA GPU Cloud Documentation and the Deep Learning DGX Documentation:
- Getting Started Using NVIDIA GPU Cloud,
- Accessing And Pulling From The NGC Container Registry,
- Running TensorFlow
For those unable to use the TensorFlow NGC container, to set up the required environment or create your own container, see the versioned NVIDIA Container Support Matrix.