Wide & Deep Recommender model.
The following sections provide greater details of the dataset, running training, and the training results.
Scripts and sample code
These are the important scripts in this repository:
main.py- Python script for training the Wide & Deep recommender model.scripts/preproc.sh- Bash script for Outbrain dataset preparation for training, preprocessing and saving into NVTabular format.data/outbrain/dataloader.py- Python file containing NVTabular data loaders for train and evaluation set.data/outbrain/features.py- Python file describing the request and item level features as well as embedding dimensions and hash buckets' sizes.trainer/model/widedeep.py- Python file with model definition.trainer/run.py- Python file with training and evaluation setup.
Parameters
These are model parameters in the main.py script:
| Scope | parameter | Comment | Default Value |
|---|---|---|---|
| location of datasets | --train_data_pattern TRAIN_DATA_PATTERN | Pattern of training file names | /outbrain/data/train/*.parquet |
| location of datasets | --eval_data_pattern EVAL_DATA_PATTERN | Pattern of eval file names | /outbrain/data/valid/*.parquet |
| location of datasets | --use_checkpoint | Use checkpoint stored in model_dir path | False |
| location of datasets | --model_dir MODEL_DIR | Destination where model checkpoint will be saved | /outbrain/checkpoints |
| location of datasets | --results_dir RESULTS_DIR | Directory to store training results | /results |
| location of datasets | --log_filename LOG_FILENAME | Name of the file to store dlloger output | log.json |
| training parameters | --global_batch_size GLOBAL_BATCH_SIZE | Total size of training batch | 131072 |
| training parameters | --eval_batch_size EVAL_BATCH_SIZE | Total size of evaluation batch | 131072 |
| training parameters | --num_epochs NUM_EPOCHS | Number of training epochs | 20 |
| training parameters | --cpu | Run computations on the CPU | Currently not supported |
| training parameters | --amp | Enable automatic mixed precision conversion | False |
| training parameters | --xla | Enable XLA conversion | False |
| training parameters | --linear_learning_rate LINEAR_LEARNING_RATE | Learning rate for linear model | 0.02 |
| training parameters | --deep_learning_rate DEEP_LEARNING_RATE | Learning rate for deep model | 0.00012 |
| training parameters | --deep_warmup_epochs DEEP_WARMUP_EPOCHS | Number of learning rate warmup epochs for deep model | 6 |
| model construction | --deep_hidden_units DEEP_HIDDEN_UNITS [DEEP_HIDDEN_UNITS ...] | Hidden units per layer for deep model, separated by spaces | [1024, 1024, 1024, 1024, 1024] |
| model construction | --deep_dropout DEEP_DROPOUT | Dropout regularization for deep model | 0.1 |
| model construction | --combiner {mean,sum} | Type of aggregation used for multi hot categorical features | sum |
| run mode parameters | --evaluate | Only perform an evaluation on the validation dataset, don't train | False |
| run mode parameters | --benchmark | Run training or evaluation benchmark to collect performance metrics | False |
| run mode parameters | --benchmark_warmup_steps BENCHMARK_WARMUP_STEPS | Number of warmup steps before start of the benchmark | 500 |
| run mode parameters | --benchmark_steps BENCHMARK_STEPS | Number of steps for performance benchmark | 1000 |
| run mode parameters | --affinity {all,single,single_unique, unique_interleaved,unique_contiguous,disabled} | Type of CPU affinity | unique_interleaved |
Command-line options
To see the full list of available options and their descriptions, use the -h or --help command-line option:
python main.py -h
Getting the data
The Outbrain dataset can be downloaded from Kaggle (requires Kaggle account).
Dataset guidelines
The dataset contains a sample of users' page views and clicks, as observed on multiple publisher sites. Viewed pages and clicked recommendations have additional semantic attributes of the documents. The dataset contains sets of content recommendations served to a specific user in a specific context. Each context (i.e. a set of recommended ads) is given a display_id. In each such recommendation set, the user has clicked on exactly one of the ads.
The original data is stored in several separate files:
page_views.csv- log of users visiting documents (2B rows, ~100GB uncompressed)clicks_train.csv- data showing which ad was clicked in each recommendation set (87M rows)clicks_test.csv- used only for the submission in the original Kaggle contestevents.csv- metadata about the context of each recommendation set (23M rows)promoted_content.csv- metadata about the adsdocument_meta.csv,document_topics.csv,document_entities.csv,document_categories.csv- metadata about the documents
During the preprocessing stage, the data is transformed into 87M rows tabular data of 29 features. The dataset is split into training and evaluation parts that have approx 60M and approx 27M rows, respectively. Splitting into train and eval is done in this way so that random 80% of daily events for the first 10 days of the dataset form a training set and remaining part (20% of events daily for the first 10 days and all events in the last two days) form an evaluation set. Eventually the dataset is saved in NVTabular parquet format.
Dataset preprocessing
Dataset preprocessing aims in creating in total 29 features: 16 categorical and 13 numerical. These features are obtained from the original Outbrain dataset in NVTabular preprocessing.
NVTabular GPU preprocessing
The NVTabular dataset is preprocessed using the script provided in data/outbrain/nvtabular. The workflow consists of:
- separating out the validation set for cross-validation
- filling missing data with the mode, median, or imputed values most frequent value
- joining click data, ad metadata, and document category, topic and entity tables to create an enriched table.joining the tables for the ad clicks data
- computing 7 click-through rates (CTR) for ads grouped by 7 features different contexts
- computing attribute cosine similarity between the landing page and ad to be featured on the page features of the clicked ads and the viewed ads
- extracting multi-hot categorical values
- math transformations of the numeric features (logarithmic, normalization)
- categorifying data using hash-bucketing
- storing the result in a Parquet format
Most of the code describing operations in this workflow are in data/outbrain/nvtabular/utils/workflow.py and leverage NVTabular v0.7.1. As stated in its repository, NVTabular, a component of NVIDIA Merlin Open Beta, is a feature engineering and preprocessing library for tabular data that is designed to quickly and easily manipulate terabyte scale datasets and train deep learning based recommender systems. It provides a high-level abstraction to simplify code and accelerates computation on the GPU using the RAPIDS Dask-cuDF library.
The NVTabular Outbrain workflow has been successfully tested on DGX-1 V100 and DGX A100 for single and multigpu preprocessing.
For more information about NVTabular, refer to the NVTabular documentation.
Training process
The training can be started by running the main.py script. By default, the script is in train mode. Other training related configs are also present in the trainer/utils/arguments.py and can be seen using the command python main.py -h. Training happens with NVTabular data loader on a NVTabular training dataset files that match --train_data_pattern. Training is run for --num_epochs epochs with a global batch size of --global_batch_size in strong scaling mode (i.e. the effective batch size per GPU equals global_batch_size/gpu_count).
The model:
tf.keras.experimental.WideDeepModel consists of a wide part and deep part with a sigmoid activation in the output layer (see Figure 1) for reference and trainer/model/widedeep.py for model definition).
During training (default configuration): Two separate optimizers are used to optimize the wide and the deep part of the network:
- FTLR (Follow the Regularized Leader) optimizer is used to optimize the wide part of the network.
- RMSProp optimizer is used to optimize the deep part of the network.
Checkpoint of the model:
- Can be loaded at the beginning of training when
--use_checkpointis set. - is saved into
--model_dirafter each training epoch. Only the last checkpoint is kept. - Contains information about number of training epochs.
The model is evaluated on an evaluation dataset after every training epoch training log is displayed in the console and stored in --log_filename.
Every 100 batches with training metrics: bce loss
Every epoch after training, evaluation metrics are logged: bce loss and MAP@12 value
Evaluation process
The evaluation can be started by running the main.py --evaluation script. Evaluation is done on NVTabular parquet dataset stored in --eval_data_pattern. Other evaluation related configs are also present in the trainer/utils/arguments.py and can be seen using the command python main.py -h.
During evaluation (--evaluation flag):
- Model is restored from checkpoint in
--model_dirif--use_checkpointis set. - Evaluation log is displayed in console and stored in
--log_filename. - Every 100 batches evaluation metrics are logged: bce loss.
After the whole evaluation, the total evaluation metrics are logged: bce loss and MAP@12 value.