The following sections provide greater details of the dataset, running training, and the training results.

Scripts and sample code

These are the important scripts in this repository:

• main.py - Python script for training the Wide & Deep recommender model.
• scripts/preproc.sh - Bash script for Outbrain dataset preparation for training, preprocessing, and saving into NVTabular format.
• data/outbrain/dataloader.py - Python file containing NVTabular data loaders for train and evaluation set.
• data/outbrain/features.py - Python file describing the request and item level features as well as embedding dimensions and hash buckets' sizes.
• trainer/model/widedeep.py - Python file with model definition.
• trainer/run.py - Python file with training and evaluation setup.

Parameters

These are model parameters in the main.py script:

Command-line options

To display the full list of available options and their descriptions, use the -h or --help command-line option:

python main.py -h

Getting the data

The Outbrain dataset can be downloaded from Kaggle (requires a Kaggle account).

Dataset guidelines

The dataset contains a sample of users' page views and clicks, as observed on multiple publisher sites. Viewed pages and clicked recommendations have additional semantic attributes of the documents. The dataset contains sets of content recommendations served to a specific user in a specific context. Each context (i.e., a set of recommended ads) is given a display_id. In each such recommendation set, the user has clicked on exactly one of the ads.

The original data is stored in several separate files:

• page_views.csv - log of users visiting documents (2B rows, ~100GB uncompressed)
• clicks_train.csv - data showing which ad was clicked in each recommendation set (87M rows)
• clicks_test.csv - used only for the submission in the original Kaggle contest
• events.csv - metadata about the context of each recommendation set (23M rows)
• promoted_content.csv - metadata about the ads
• document_meta.csv, document_topics.csv, document_entities.csv, document_categories.csv - metadata about the documents

During the preprocessing stage, the data is transformed into 87M rows of tabular data of 29 features. The dataset is split into training and evaluation parts that have approx 60M and approx 27M rows, respectively. Splitting into train and eval is done in this way so that a random 80% of daily events for the first 10 days of the dataset form a training set and the remaining part (20% of events daily for the first 10 days and all events in the last two days) form an evaluation set. Eventually, the dataset is saved in NVTabular parquet format.

Dataset preprocessing

Dataset preprocessing aims to create a total of 29 features: 16 categorical and 13 numerical. These features are obtained from the original Outbrain dataset in NVTabular preprocessing.

NVTabular GPU preprocessing

The NVTabular dataset is preprocessed using the script provided in data/outbrain/nvtabular. The workflow consists of:

• separating out the validation set for cross-validation
• filling missing data with the mode, median, or imputed values most frequent value
• joining click data, ad metadata, and document category, topic, and entity tables to create an enriched table.joining the tables for the ad clicks data
• computing seven click-through rates (CTR) for ads grouped by seven features in different contexts
• computing attribute cosine similarity between the landing page and ad to be featured on the page features of the clicked ads and the viewed ads
• extracting multi-hot categorical values
• math transformations of the numeric features (logarithmic, normalization)
• categorifying data using hash-bucketing
• storing the result in a Parquet format

Most of the code describing operations in this workflow is in data/outbrain/nvtabular/utils/workflow.py and leverage NVTabular v0.7.1. As stated in its repository, NVTabular, a component of NVIDIA Merlin Open Beta, is a feature engineering and preprocessing library for tabular data that is designed to quickly and easily manipulate terabyte-scale datasets and train deep learning based recommender systems. It provides a high-level abstraction to simplify code and accelerates computation on the GPU using the RAPIDS Dask-cuDF library. The NVTabular Outbrain workflow has been successfully tested on DGX-1 V100 and DGX A100 for single and multi GPU preprocessing.

For more information about NVTabular, refer to the NVTabular documentation.

BYO dataset

This implementation supports using other datasets thanks to BYO dataset functionality. The BYO dataset functionality allows users to plug in their dataset in a common fashion for all Recommender models that support this functionality. Using BYO dataset functionality, the user does not have to modify the source code of the model thanks to the Feature Specification file. For general information on how the BYO dataset works, refer to the BYO dataset overview section.

There are three ways to plug in the user's dataset:

DATASET_PARENT_DIRECTORY=/raid/wd2

docker build . -t wd2

docker run --runtime=nvidia --gpus=all --rm -it --ipc=host -v ${DATASET_PARENT_DIRECTORY}:/outbrain wd2 bash

cp tests/feature_specs/fspec_csv.yaml /outbrain/feature_spec.yaml

python gen_csv.py --feature_spec_in /outbrain/feature_spec.yaml --output /outbrain --size 393216

mkdir /outbrain/data
python transcode.py --input /outbrain --output /outbrain/data --chunk_size 16384 # to get 8 partitions out of 131072 rows

export EMBEDDING_SIZES_FILE="data/outbrain/embedding_sizes.json"

horovodrun -np 1 sh hvd_wrapper.sh python main.py --dataset_path /outbrain/data --disable_map_calculation --embedding_sizes_file ${EMBEDDING_SIZES_FILE}

horovodrun -np 8 sh hvd_wrapper.sh python main.py --dataset_path /outbrain/data --disable_map_calculation --embedding_sizes_file ${EMBEDDING_SIZES_FILE}

Channel definitions and requirements

This model defines three channels:

• multihot_categorical, accepting an arbitrary number of features
• onehot_categorical, accepting an arbitrary number of features
• numerical, accepting an arbitrary number of features
• label, accepting a single feature
• map, accepting zero or one feature

The training script expects two mappings:

• train
• test

For performance reasons:

• The only supported dataset type is parquet. CSV is supported through transcoding.
• Each chunk must contain at least as many parts as there are workers.
• Only integer types are supported for categorical features
• Only floating point types are supported for numerical features

The MAP channel is optional. If present, it is expected to be provided only for the test mapping. Rows with common values are aggregated into a list, and MAP@12 is calculated during evaluation. Please note that the MAP implementation used in this repository may not be correct for datasets other than Outbrain preprocessed with the included script.

BYO dataset constraints for the model

There are the following constraints of BYO dataset functionality for this model:

1. The performance of the model depends on the dataset size. Generally, the model should scale better for datasets containing more data points. For a smaller dataset, you might experience smaller performance improvements than those reported for Outbrain
2. Using other datasets might require tuning some hyperparameters (for example, learning rate, beta1, and beta2) to reach the desired accuracy.

Training process

The training can be started by running the main.py script. By default, the script is in train mode. Other training-related configs are also present in the trainer/utils/arguments.py and can be seen using the command python main.py -h. Training happens with a NVTabular data loader on a NVTabular training dataset files specified by feature spec. Training is run for --num_epochs epochs with a global batch size of --global_batch_size in strong scaling mode (i.e., the effective batch size per GPU equals global_batch_size/gpu_count).

The model: tf.keras.experimental.WideDeepModel consists of a wide part and deep part with a sigmoid activation in the output layer (refer to Figure 1) for reference and trainer/model/widedeep.py for model definition).

During training (default configuration): Two separate optimizers are used to optimize the wide and the deep part of the network:

• FTLR (Follow the Regularized Leader) optimizer is used to optimize the wide part of the network.
• RMSProp optimizer is used to optimize the deep part of the network.

Checkpoint of the model:

• Can be loaded at the beginning of training when --use_checkpoint is set.
• is saved into --model_dir after each training epoch. Only the last checkpoint is kept.
• Contains information about the number of training epochs.

The model is evaluated on an evaluation dataset after every training epoch training log is displayed in the console and stored in --log_filename.

Every 100 batches with training metrics: bce loss

Every epoch after training, evaluation metrics are logged: bce loss and MAP@12 value.

Evaluation process

The evaluation can be started by running the main.py --evaluation script. Evaluation is done on the NVTabular parquet dataset specified by feature spec. Other evaluation-related configs are also present in the trainer/utils/arguments.py and can be seen using the command python main.py -h.

During evaluation (--evaluation flag):

• Model is restored from a checkpoint in --model_dir if --use_checkpoint is set.
• Evaluation log is displayed in the console and stored in --log_filename.
• Every 100 batches, evaluation metrics are logged: bce loss.

After the whole evaluation, the total evaluation metrics are logged: bce loss and MAP@12 value.