The following sections provide greater details of the dataset, running training and inference, and the training results.

Command Line Arguments

To see the full list of available options and their descriptions, use the -h or --help command line option, for example:

python ncf.py --help

Aside from options to set hyperparameters, the relevant options to control the behavior of the script are:

--data DATA           path to test and training data files
-e EPOCHS, --epochs EPOCHS
                      number of epochs to train for
-b BATCH_SIZE, --batch-size BATCH_SIZE
                      number of examples for each iteration
--valid-users-per-batch VALID_USERS_PER_BATCH
                      Number of users tested in each evaluation batch
-n NEGATIVE_SAMPLES, --negative-samples NEGATIVE_SAMPLES
                      number of negative examples per interaction
-k TOPK, --topk TOPK  rank for test examples to be considered a hit
--amp                 enable half-precision computations using automatic
                      mixed precision (only available in supported
                      containers)
--xla                 enable TensorFlow XLA (Accelerated Linear Algebra)
--valid-negative VALID_NEGATIVE
                      Number of negative samples for each positive test
                      example
--loss-scale LOSS_SCALE
                      Loss scale value to use when manually enabling mixed precision training
--checkpoint-dir CHECKPOINT_DIR
                      Path to store the result checkpoint file for training, or to read from for evaluation
--mode {train,test}   Passing "test" will only run a single evaluation,
                      otherwise full training will be performed
--no-neg-trick        do not use negative sample generation shortcut to
                      speed up preprocessing (will increase GPU memory
                      consumption)
--eval-after EVAL_AFTER
                      Perform evaluations only after this many epochs
--verbose             Log the performance and accuracy after every epoch

Getting the Data

For each user, the test dataset is generated by removing one movie the user has interacted with. For each removed movie, the data is augmented with a large number of movies (corresponding to the --valid-negative option) that the user has not interacted with.

The repository contains the prepare_dataset.sh that will preprocess the training and test datasets. By default, the data will be downloaded to the /data directory.

Multi-dataset

This implementation is tuned for the ml-20m and ml-1m datasets. Using other datasets might require tuning some hyperparameters (for example, learning rate, beta1, beta2).

If you'd like to use your custom dataset, you can do so by adding support for it in the prepare_dataset.sh and download_dataset.sh scripts. The required format of the data is a CSV file which should follow the pattern outlined below:

userId, movieId
1,2
1,10
...

The CSV file may contain additional columns with extra features such as ratings and timestamps, but only the userId and movieId columns are required.

The performance of the model depends on the dataset size. Generally, the model should scale better for datasets containing more data points. For a smaller dataset, you might experience slower performance as fixed cost operations that do not scale with input size will have a larger impact. Furthermore, it will be difficult for the model to converge.

Training Process

The training can be launched with the ncf.py script. This script will train the NCF model for a number of epochs specified by the --epochs argument, which has a default value of 30.

During training, the script will begin logging after the number of epochs specified by the --eval-after option. After that the script will output a line like the one below:

DLL 2020-07-03 10:58:43.371321 - (26,) train_time : 9.889576196670532  eval_time : 0.674187183380127  hr@10 : 0.9526329850606168  ndcg : 0.7448715819572108

The evaluation metrics are: HR (hit rate), and NDCG (normalized discounted cumulative gain). In the evaluation set, each user will be assigned one item that they have actually interacted with, and a number (by default 99) of items that they have not interacted with. For each user, the evaluation process will rank each of the items assigned to that user based on the user's likeliness to interact with the items. The hit rate measures the percentage of users for which the item that they have interacted with is ranked within the top k items, where k is a number (by default 10) specified by the -k option. NDCG has a similar meaning, except the rank of the positive item is taken into account. Typically, HR is used as the primary evaluation metric.

Additionally, the model parameters that give the best accuracy in validation will be stored at the directory pointed to by the --checkpoint-dir argument.

Multiple GPUs can be used for training through Horovod. The number of GPUs can be controlled by the -np parameter passed to mpirun.

Evaluation Process

The evaluation process can be run by the ncf.py script as well. By passing the --mode=test argument, the script will run evaluation once using the TensorFlow checkpoint specified by the --checkpoint-dir file.