The following sections provide greater details of the dataset, running training and inference, and the training results.

Scripts and sample code

The ncf.py script contains most of the training and validation logic. Data loading and preprocessing code is located in dataloading.py. The model architecture is defined in neumf.py. Some initial data preprocessing is located in convert.py. The logger directory contains simple bookkeeping utilities for storing training results. The transcode.py script enables transcoding data from a CSV containing preprocessed data to a format accessible by the model.

Command-line options

To view the full list of available options and their descriptions, use the -h or --help command-line option, for example: python ncf.py --help

The following example output is printed when running the sample:

usage: ncf.py [-h] [--data DATA] [--feature_spec_file FEATURE_SPEC_FILE] [-e EPOCHS] [-b BATCH_SIZE] [--valid_batch_size VALID_BATCH_SIZE] [-f FACTORS]
              [--layers LAYERS [LAYERS ...]] [-n NEGATIVE_SAMPLES] [-l LEARNING_RATE] [-k TOPK] [--seed SEED] [--threshold THRESHOLD] [--beta1 BETA1] [--beta2 BETA2]
              [--eps EPS] [--dropout DROPOUT] [--checkpoint_dir CHECKPOINT_DIR] [--load_checkpoint_path LOAD_CHECKPOINT_PATH] [--mode {train,test}]
              [--grads_accumulated GRADS_ACCUMULATED] [--amp] [--log_path LOG_PATH]

Train a Neural Collaborative Filtering model

optional arguments:
  -h, --help            show this help message and exit
  --data DATA           Path to the directory containing the feature specification yaml
  --feature_spec_file FEATURE_SPEC_FILE
                        Name of the feature specification file or path relative to the data directory.
  -e EPOCHS, --epochs EPOCHS
                        Number of epochs for training
  -b BATCH_SIZE, --batch_size BATCH_SIZE
                        Number of examples for each iteration. This will be divided by the number of devices
  --valid_batch_size VALID_BATCH_SIZE
                        Number of examples in each validation chunk. This will be the maximum size of a batch on each device.
  -f FACTORS, --factors FACTORS
                        Number of predictive factors
  --layers LAYERS [LAYERS ...]
                        Sizes of hidden layers for MLP
  -n NEGATIVE_SAMPLES, --negative_samples NEGATIVE_SAMPLES
                        Number of negative examples per interaction
  -l LEARNING_RATE, --learning_rate LEARNING_RATE
                        Learning rate for optimizer
  -k TOPK, --topk TOPK  Rank for test examples to be considered a hit
  --seed SEED, -s SEED  Manually set random seed for torch
  --threshold THRESHOLD, -t THRESHOLD
                        Stop training early at threshold
  --beta1 BETA1, -b1 BETA1
                        Beta1 for Adam
  --beta2 BETA2, -b2 BETA2
                        Beta1 for Adam
  --eps EPS             Epsilon for Adam
  --dropout DROPOUT     Dropout probability, if equal to 0 will not use dropout at all
  --checkpoint_dir CHECKPOINT_DIR
                        Path to the directory storing the checkpoint file, passing an empty path disables checkpoint saving
  --load_checkpoint_path LOAD_CHECKPOINT_PATH
                        Path to the checkpoint file to be loaded before training/evaluation
  --mode {train,test}   Passing "test" will only run a single evaluation; otherwise, full training will be performed
  --grads_accumulated GRADS_ACCUMULATED
                        Number of gradients to accumulate before performing an optimization step
  --amp                 Enable mixed precision training
  --log_path LOG_PATH   Path for the JSON training log

Getting the data

The NCF model was trained on the ML-20m dataset. For each user, the interaction with the latest timestamp was included in the test set, and the rest of the examples are used as the training data.

This repository contains the ./prepare_dataset.sh script that automatically preprocess the training and validation datasets. By default, the preprocessed data will be placed in /data/cache.

Dataset guidelines

NCF supports all datasets that include a Feature Specification file and are properly formatted. For details, refer to the BYO dataset section.

ML-1m

To preprocess and train on the ML-1m dataset run:

./prepare_dataset.sh ml-1m
python -m torch.distributed.launch --nproc_per_node=8 --use_env ncf.py --data /data/cache/ml-1m

BYO dataset

This implementation supports using other datasets thanks to BYO dataset functionality. The BYO dataset functionality allows users to plug in their dataset in a common fashion for all Recommender models that support this functionality. Using BYO dataset functionality, the user does not have to modify the source code of the model thanks to the Feature Specification file. For general information on how BYO dataset works, refer to the BYO dataset overview section.

There are three ways to plug in user's dataset:

docker build . -t nvidia_ncf

docker run --runtime=nvidia -it --rm --ipc=host  -v ${PWD}/data:/data nvidia_ncf bash

./prepare_dataset.sh like_movielens

python -m torch.distributed.launch --nproc_per_node=1 --use_env ncf.py --data /data/cache/like_movielens

python -m torch.distributed.launch --nproc_per_node=8 --use_env ncf.py --data /data/cache/like_movielens

docker build . -t nvidia_ncf

docker run --runtime=nvidia -it --rm --ipc=host  -v ${PWD}/data:/data nvidia_ncf bash

mkdir /data/conversion_output
python transcode.py --path /data/csv_conversion --output /data/conversion_output

python -m torch.distributed.launch --nproc_per_node=1 --use_env ncf.py --data /data/conversion_output -k 3

python -m torch.distributed.launch --nproc_per_node=8 --use_env ncf.py --data /data/conversion_output -k 3

Channel definitions and requirements for NCF-PyT feature specifications.

This model defines three channels, each accepting a single feature:

• user_ch
• item_ch
• label_ch

The training script expects two mappings:

• train
• test

As this NeuMF implementation computes list ranking metrics, the testing set actually consists of lists of candidates. Usually, all entries in a list share the same user id, although this is not mandatory. All entries from a given list must appear consecutively in the testing set. List boundaries are not marked in the testing set. All lists must have the same length. This length must be set by the metadata:test_samples_per_series parameter in the Feature Specification yaml file.

BYO dataset constraints for the model

There are the following constraints of BYO dataset functionality for this model:

1. The performance of the model depends on the dataset size. Generally, the model should scale better for datasets containing more data points. For a smaller dataset, you might experience slower performance than the one reported for ml-20m
2. As this implementation keeps the training and testing data in GPU VRAM, supported dataset size is limited by the GPU VRAM size.
3. Using other datasets might require tuning some hyperparameters (for example, learning rate, beta1 and beta2) to reach desired accuracy.
4. The transcoding script uses pandas, and the user's dataset needs to fit into the system memory

Training process

The name of the training script is ncf.py. Because of the multi-GPU support, it should always be run with the torch distributed launcher like this:

python -m torch.distributed.launch --nproc_per_node=<number_of_gpus> --use_env ncf.py --data <path_to_dataset> [other_parameters]

The main result of the training are checkpoints stored by default in /data/checkpoints/. This location can be controlled by the --checkpoint_dir command-line argument.

The validation metric is Hit Rate at 10 (HR@10) with 100 test negative samples. This means that for each positive sample in the test set, 100 negatives are sampled. All resulting 101 samples are then scored by the model. If the true positive sample is among the 10 samples with the highest scores we have a "hit," and the metric is equal to 1; otherwise, it's equal to 0. The HR@10 metric is the number of hits in the entire test set divided by the number of samples in the test set.

Inference process

Inference can be launched with the same script used for training by passing the --mode test flag:

python -m torch.distributed.launch --nproc_per_node=<number_of_gpus> --use_env ncf.py  --data <path_to_dataset> --mode test [other_parameters]

The script will then:

• Load the checkpoint from the directory specified by the --checkpoint_dir directory
• Run inference on the test dataset
• Compute and print the validation metric