BART is a denoising autoencoder for pretraining sequence-to-sequence models.
The following sections provide greater details of the dataset, running training and inference, and the training results.
Scripts and sample code
In the root directory, the most important files are:
finetune.py- Serves as entry point for fine-tuningrun_eval.py- Serves as entry point for inferenceDockerfile- Container with the basic set of dependencies to run BART
The scripts/ folder encapsulates all the one-click scripts required for running various functionalities supported such as:
run_summarization.sh- Runs summarization finetuning followed by inference using thefinetune.pyandrun_eval.pyfiles.run_summarization_eval.sh- Runs inference on fine tuned checkpoint using therun_eval.pyfile.get_data.sh- Preprocesses CNN-DM dataset as well as downloads and preprocesses XSum dataset.
Other folders included in the root directory are:
data/- Necessary folder to download datasets required for fine tuning of BART.src/- Modeling, tokenization and configuration functionality files for implementing the BART model.utils/- Necessary utility files for BART model.
Parameters
Aside from the options to set hyperparameters, the relevant options to control the behaviour of the run_pretraining.py script are:
--config_path: The configuration file corresponding to BART Model
--warmup_steps: Number of WARMUP_STEPS
--max_steps: Number of MAX_STEPS
--data_dir: Location to DATA_DIR
--gpus: Number of GPUs
--learning_rate: Learning Rate
--n_val: Number of validation examples to test for early stopping
--train_batch_size: Train batch size
--gradient_accumulation_steps: Number of accumulation steps
--val_check_interval: Periodicity of checking validation score
--max_source_length: Maximum source length
--max_target_length: Maximum target length
--val_max_target_length: Maximum length of validation tokens
--eval_max_gen_length: Maximum length while generating validation tokens
--weight_decay: weight decay
--dropout: drop out
--early_stopping_patience: number of validation trials of no improvement before which to trigger early stopping
--amp_level: amp mode of optimization level to use if training with mixed precision
Command-line options
To see the full list of available options and their descriptions, use the -h or --help command-line option with the Python file, for example:
python finetune.py --help
python run_eval.py --help
Getting the data
We have tested fine tuning the BART model on summarization benchmarks such as CNN-DM and XSum.
CNN-DM is a concatenation of CNN Stories as well as Daily Mail Stories. CNN consists of approximately 90k documents whereas Daily Mail consists of 197k documents.
These documents are preprocessed to have two features:
- Article: text of news article, used as the document to be summarized
- Highlights: joined text of highlights with and around each highlight, which is the target summary
XSum, on the other hand, is also a single-document summarization task dataset but one that favors abstractive modeling. It consists of BBC articles and single sentence summaries. It consists of approximately 230k articles.
Dataset guidelines
The repository contains a script to preprocess and download data. It can be run as:
bash scripts/get_data.sh <path to output data folder>
The script downloads CNN and DM raw data from here. The raw data is preprocessed using scripts from repository. The stories are first tokenized, written to serialized binary files and split into train, test and validation sets.
The script also downloads the XSum dataset from the HuggingFace storage.
The script by default stores the data into the /workspace/bart/data folder.
Training process
Training BART for summarization is done using scripts/run_summarization.sh script that, in turn, uses the finetune.py file to perform training.
For example, it can be invoked by calling:
Bash scripts/run_summarization.sh <DATA_DIR> <CONFIG_PATH> <num_gpu> <LR> <BS> <ACCUM> <PREC> <TRAIN_STEPS> <WARMUP_STEPS> <MAX_SOURCE_LEN> <MAX_TARGET_LEN> <EVAL_BEAMS> <EVAL_BS> <PRED_BS> <VAL_CHECK_INTERVAL> <PATIENCE>
Where:
- DATA_DIR - path to data directory with train/test/val files.
- CONFIG_PATH - path to configuration file of BART Model
- NUM_GPU - number of GPUs to run training with
- LR - Learning rate for training process
- BS - Training batch size
- ACCUM - Number of accumulation steps for an effective larger training batch size
- PREC - fp16/fp32/tf32 precision for training and inference
- TRAIN_STEPS - Maximum number of training steps
- WARMUP_STEPS - Number of warmup steps for learning rate scheduler
- MAX_SOURCE_LEN - Maximum source length of articles
- MAX_TARGET_LEN - Maximum target length of summaries
- EVAL_BEAMS - Number of beams to run during inference
- EVAL_BS - Batch size for inference during validation
- PRED_BS - Batch size for inference on test data
- VAL_CHECK_INTERVAL - Fraction of an epoch after which runs validation if <1. Or number of training samples after which to run validation if >1.
- PATIENCE - Number of validation checks of no improvement after which to stop training
By default, the training script stores results to results/bart_pyt_${DATESTAMP} and runs with:
bash scripts/run_summarization.sh data/cnn_dm/ configs/config.json 8 1e-4 24 1 fp16 20000 500 1024 142 4 128 64 0.3
These parameters train CNN-DM with reasonable rouge scores on a LUNA with 80GB A100 cards. Other tested configurations are available under scripts/params/cnn_dm_params.sh for CNN-DM and scripts/params/xsum_params.sh for XSum datasets.
Inference process
Evaluating BART for summarization is done using scripts/run_eval_summarization.sh script that, in turn, uses the run_eval.py file to perform inference.
For example, it can be invoked by calling:
bash scripts/run_eval_summarization.sh <INIT_CKPT> <PRED_BS> <NUM_GPU> <PRECISION> <EVAL_BEAMS> <MAX_SOURCE_LEN> <MAX_TARGET_LEN> <DATA_DIR> <CONFIG_PATH>
Where:
INIT_CKPT- Model name or path to initialize BART Model weights with.PRED_BS- Batch size for inferenceNUM_GPU- number of GPUs to run training withPRECISION- FP16/FP32/TF32 precision for training and inferenceEVAL_BEAMS- Number of beams to run during inferenceMAX_SOURCE_LEN- Maximum source length of articlesMAX_TARGET_LEN- Maximum target length of summariesDATA_DIR- path to data directory with train/test/val files.CONFIG_PATH- path to configuration file of BART Model
By default, the training script stores results to results/bart_pyt_inference_${DATESTAMP} and runs with:
bash scripts/run_eval_summarization.sh data/nvidia-pretrained/model.ckpt128 8 fp16 4 1024 142 data/cnn_dm/ configs/config.json
These parameters run inference on CNN-DM on a DGX A100 with 80GB A100 cards. For XSum, try EVAL_BEAMS=6, MAX_SOURCE_LEN=1024 and MAX_TARGET_LEN=60. For other GPUS/precisions, reduce PRED_BS as indicated in scripts/params.