Changelog

1. January 22, 2018
   • Initial release
2. May, 2019
   • Lower memory consumption (down from about 18GB to 10GB for batch size 1M on a single NVIDIA Tesla V100). Achieved by using an approximate method for generating negatives for training.
   • Automatic Mixed Precision (AMP) with dynamic loss scaling instead of a custom mixed-precision optimizer.
   • Performance numbers for NVIDIA DGX-2.
   • Data loading code cleanup.
   • Default container updated to PyTorch 19.05-py3.
   • Updated README.md.
3. June, 2019
   • Updated performance tables.
   • Default container changed to PyTorch 19.06-py3.
   • Caching validation negatives between runs
4. September, 2019
   • Adjusting for API changes in PyTorch and APEX
   • Checkpoints loading fix
5. January, 2020
   • DLLogger support added

Known issues

Scaling beyond 8 GPUs

Neural Collaborative Filtering is a relatively lightweight model that trains quickly with this relatively smaller dataset, ML-20m. Because of that, the high ratio of communication to computation makes it difficult to efficiently use more than 8 GPUs. Typically, this is not an issue because when using 8 GPUs with FP16 precision, the training is sufficiently fast. However, if you’d like to scale the training to 16 GPUs and beyond, you might try modifying the model so that the communication-computation ratio facilitates better scaling. This could be done, for example, by finding hyperparameters that enable using a larger batch size or by reducing the number of trainable parameters.

Memory usage

In the default settings, the additional memory beyond 16G may not be fully utilized. This is because we set the default batch size for ML-20m dataset to 1M, which is too small to completely fill-up multiple 32G GPUs. 1M is the batch size for which we experienced the best convergence on the ML-20m dataset. However, on other datasets, even faster performance can be possible by finding hyperparameters that work well for larger batches and leverage additional GPU memory.