Changelog

June 2020

• Updated performance tables to include A100 results

March 2019

• Initial Release

Known Issues

AMP speedup for Ampere

In this model the TF32 precision can in some cases be as fast as the FP16 precision on Ampere GPUs. This is because TF32 also uses Tensor Cores and doesn't need any additional logic such as maintaining FP32 master weights and casts. However, please note that NCF is, by modern recommender standards, a very small model. Larger models should still see significant benefits of using FP16 math.

Multi-GPU Scaling Efficiency

Currently, this model does not exhibit good scaling efficiency when scaling to 4 and 8 GPUs. Since we could not find hyper-parameters that could hit the target accuracy for batch size of over 1 million samples, we elected to use a strong scaling strategy which generally has worse scaling efficiency compared to a more common weak scaling strategy. Additionally, we believe that the small dataset size does not facilitate great scaling. However, the training scripts allow the use of custom datasets provided they are in the correct format.

Scaling beyond 8 GPUs

Neural Collaborative Filtering (NCF) is a relatively lightweight model that trains quickly with this relatively smaller dataset, ml-20m. Because of the smaller dataset, the high ratio of communication to computation makes it difficult to efficiently use more than 8 GPUs. Typically, this is not an issue because when using 8 GPUs with FP16 precision the training is sufficiently fast. However, if you would like to scale the training to 16 GPUs and beyond, you might try modifying the model so that the communication to computation ratio facilitates better scaling. This could be done, for example, by finding hyper-parameters that enable using a larger global batch size.

Preprocessing Out-of-Memory with 16GB GPUs

When running on GPUs with 16GB of memory, ensure the --no-neg-trick flag is not set. Otherwise, the data augmentation stage of training will consume too much GPU memory, causing TensorFlow to raise an out-of-memory error.

This flag, when it is not set, reduces memory consumption in the negative samples generation phase of training by telling the script not to verify that the randomly generated samples are actually negative samples (verification still occurs for negative samples generated for the test set). Therefore, there is no need to keep the data structures used to verify negative samples in memory during training.