The following sections provide greater details of the dataset, running training and inference, and the training results.

Scripts and sample code

The main.py script provides an entry point to all the provided functionalities. This includes running training and inference modes. The behavior of the script is controlled by command-line arguments listed below in the Parameters section. The preprocessing folder contains scripts to prepare data. In particular, the preprocessing/sim_preprocessing.py script can be used to preprocess the Amazon Reviews dataset while preprocessing/parquet_to_tfrecord.py transforms parquet files to TFRecords for loading data efficiently.

Models are implemented in corresponding modules in the sim/models subdirectory, for example, sim/models/sim_model.py for the SIM model. The sim/layers module contains definitions of different building blocks for the models. Finally, the sim/data subdirectory provides modules for the dataloader. Other useful utilities are contained in the sim/utils module.

Parameters

The main.py script parameters are detailed in the following table.

Command-line options

To view the full list of available options and their descriptions, use the --help command-line option, for example:

python main.py --help

Getting the data

The SIM model was trained on the Books department subset of Amazon Reviews dataset. The dataset is split into two parts: training and test data. The test set for evaluation was constructed using the last user interaction from user behavior sequences. All the preceding interactions are used for training.

This repository contains the scripts/download_amazon_books_2014.sh, which can be used to download the dataset.

Dataset guidelines

The preprocessing steps applied to the raw data include:

• Sampling negatives randomly (out of all possible items)
• Choosing the last category as the item category (in case more than one is available)
• Determining embedding table sizes for categorical features needed to construct a model
• Filter users for training split based on their number of interactions (discard users with less than 20 interactions)

Prebatching

Preprocessing scripts allow to apply batching prior to the models dataloader. This reduces the size of produced TFrecord files and speeds up dataloading. To do so, specify --prebatch_train_sizeand--prebatch_test_sizewhile converting data usingscripts/parquet_to_tfrecord.py. Later, while using the main.py` script, pass the information about applied prebatch size via the same parameters.

Example

Start preprocessing from step 5. from Quick Start Guide:

python preprocessing/sim_preprocessing.py \
--amazon_dataset_path ${RAW_DATASET_PATH} \
--output_path ${PARQUET_PATH}

python preprocessing/parquet_to_tfrecord.py \
--amazon_dataset_path ${PARQUET_PATH} \
--tfrecord_output_dir ${TF_RECORD_PATH} \
--prebatch_train_size ${PREBATCH_TRAIN_SIZE} \
--prebatch_train_size ${PREBATCH_TEST_SIZE}

And then train the model (step 6.):

mpiexec --allow-run-as-root --bind-to socket -np ${GPU} python main.py \
--dataset_dir ${TF_RECORD_PATH} \
--mode train \
--model_type sim \
--embedding_dim 16 \
--drop_remainder \
--optimizer adam \
--lr 0.01 \
--epochs 3 \
--global_batch_size 131072 \
--amp \
--prebatch_train_size ${PREBATCH_TRAIN_SIZE} \
--prebatch_train_size ${PREBATCH_TEST_SIZE}

BYO dataset

This implementation supports using other datasets thanks to BYO dataset functionality. BYO dataset functionality allows users to plug in their dataset in a common fashion for all Recommender models that support this functionality. Using BYO dataset functionality, the user does not have to modify the source code of the model thanks to the Feature Specification file. For general information on how the BYO dataset works, refer to the BYO dataset overview section.

For usage of preprocessing scripts refer to Quick Start Guide

There are currently two ways to plug in the user's dataset:

DATASET:
  metadata.json
  test:
    part.0.parquet
    part.1.parquet
    .
    .
    .
  train:
    part.0.parquet
    part.1.parquet
    .
    .
    .

{
  "cardinalities": [
    {"name": "uid", "value": 105925}, 
    {"name": "item", "value": 1209081}, 
    {"name": "cat", "value": 2330}
  ]
}

TF_RECORD_PATH:
    feature_spec.yaml
    test.tfrecord
    train.tfrecord

channel_spec:
  label:
  - label
  negative_history:
  - item_id_neg
  - cat_id_neg
  positive_history:
  - item_id_pos
  - cat_id_pos
  target_item_features:
  - item_id_trgt
  - cat_id_trgt
  user_features:
  - user_id
feature_spec:
  item_id_neg:
    cardinality: 1209081
    dimensions:
    - 100
    dtype: int64
  item_id_pos:
    cardinality: 1209081
    dimensions:
    - 100
    dtype: int64
  item_id_trgt:
    cardinality: 1209081
    dtype: int64
  cat_id_neg:
    cardinality: 2330
    dimensions:
    - 100
    dtype: int64
  cat_id_pos:
    cardinality: 2330
    dimensions:
    - 100
    dtype: int64
  cat_id_trgt:
    cardinality: 2330
    dtype: int64
  label:
    dtype: bool
  user_id:
    cardinality: 105925
    dtype: int64
metadata: {}
source_spec:
  test:
  - features: &id001
    - label
    - user_id
    - item_id_trgt
    - cat_id_trgt
    - item_id_pos
    - cat_id_pos
    - item_id_neg
    - cat_id_neg
    files:
    - test.tfrecord
    type: tfrecord
  train:
  - features: *id001
    files:
    - train.tfrecord
    type: tfrecord

Channel definitions and requirements

This model defines five channels:

• label, accepting a single feature
• negative_history, accepting a categorical ragged tensor for an arbitrary number of features
• positive_history, accepting a categorical ragged tensor for an arbitrary number of features
• target_item_features, accepting an arbitrary number of categorical features
• user_features, accepting an arbitrary number of categorical features

Features in negative_history, positive_history and target_item_features channels must be equal in number and must be defined in the same order in channel spec.

The training script expects two mappings:

• train
• test

For performance reasons, the only supported dataset type is tfrecord.

Training process

Training can be run using main.py script by specifying the --mode train parameter. The speed of training is measured by throughput, that is, the number of samples processed per second. Evaluation is based on the Area under ROC Curve (ROC AUC) metric. Model checkpoints may be stored using Checkpoint manager via the --save_checkpoint_path and --load_checkpoint_path parameters. Training and inference logs are saved to a directory specified via the --results_dir parameter. Mixed precision training is supported via the --amp flag. Multi-GPU training is performed using mpiexec and Horovod libraries.

Inference process

Inference can be run using main.py script by specifying the --mode inference parameter. It is performed using a dummy model initialized randomly, and it is intended to measure inference throughput. The most important parameter for inference is the batch size.

Example usage of training and inference are demonstrated in Quick Start Guide.

Log format

There are three type of log lines during model execution. Each of them have step value, however it is formatted differently based on the type of log:

• step log - step value is in format [epoch, step]:

DLLL {"timestamp": ..., "datetime": ..., "elapsedtime": ..., "type": ..., "step": [2, 79], "data": ...}

• end of epoch log - step value is in format [epoch]:

DLLL {"timestamp": ..., "datetime": ..., "elapsedtime": ..., "type": ..., "step": [2], "data": ...}

• summary log - logged once at the end of script execution. Step value is in fomat []:

DLLL {"timestamp": ..., "datetime": ..., "elapsedtime": ..., "type": ..., "step": [], "data": ...}

In those logs, data field contains dictonary in form {metric: value}. Metrics logged differ based on log type (step, end of epoch, summary) and model mode (training, inference).

Training log data

• step log
  • classification_loss - loss at the final output of the model.
  • dien_aux_loss - loss at the output of auxiliary model.
  • total_loss - sum of the above.
  • samples/s - estimated throughput in samples per second.
• end of epoch log
  • throughput - average throughput during epoch in samples/s.
  • time - epoch time in seconds.
  • train_auc - AUC during evaluation on train set.
  • test_auc - AUC during evaluation on test set.
  • train_loss - loss during evaluation on train set.
  • test_loss - loss during evaluation on test set.
  • latency_[mean, p90, p95, p99] - latencies in miliseconds.
• summary log
  • time_to_train - total training time in seconds.
  • train_auc, test_auc, train_loss, test_loss - results from the last epoch (see above).

Inference log data

• step log
  • samples/s - estimated throughput in samples per second.
• end of epoch log is not present
• summary log
  • throughput - average throughput during epoch in samples/s.
  • time - total execution time in seconds.
  • latency_[mean, p90, p95, p99] - latencies in miliseconds.