The performance measurements in this document were conducted at the time of publication and may not reflect the performance achieved from NVIDIA's latest software release. For the most up-to-date performance measurements, go to NVIDIA Data Center Deep Learning Product Performance.

Benchmarking

Training performance benchmark

NCF training on NVIDIA DGX systems is very fast; therefore, in order to measure train and validation throughput, you can simply run the full training job with:

./prepare_dataset.sh
python -m torch.distributed.launch --nproc_per_node=8 --use_env ncf.py --data /data/cache/ml-20m --epochs 5

At the end of the script, a line reporting the best train throughput is printed.

Inference performance benchmark

Validation throughput can be measured by running the full training job with:

./prepare_dataset.sh
python -m torch.distributed.launch --nproc_per_node=8 --use_env ncf.py --data /data/cache/ml-20m --epochs 5

The best validation throughput is reported to the standard output.

Results

The following sections provide details on how we achieved our performance and accuracy in training and inference.

Training accuracy results

Training accuracy: NVIDIA DGX A100(8x A100 40GB)

Our results were obtained by following the steps in the Quick Start Guide in the PyTorch 21.04-py3 NGC container on NVIDIA DGX A100 (8x A100 40GB) GPUs.

The following table lists the best hit rate at 10 for DGX A100 with 8 A100 40GB GPUs. It also shows the time to reach this HR@10. Results are averages across 20 random seeds.

Training accuracy: NVIDIA DGX-1 (8x V100 16GB)

Our results were obtained by following the steps in the Quick Start Guide in the PyTorch 21.04-py3 NGC container on NVIDIA DGX-1 with 8x V100 16GB GPUs.

The following table lists the best hit rate at 10 for DGX-1 with 8 V100 16GB GPUs. It also shows the time to reach this HR@10. Results are averages across 20 random seeds. The training time was measured excluding data downloading, preprocessing, validation data generation and library initialization times.

To reproduce this result, start the NCF Docker container interactively and run:

./prepare_dataset.sh
python -m torch.distributed.launch --nproc_per_node=8 --use_env ncf.py --data /data/cache/ml-20m

Training accuracy: NVIDIA DGX-1 (8x V100 32GB)

Our results were obtained by following the steps in the Quick Start Guide in the PyTorch 21.04-py3 NGC container on NVIDIA DGX-1 with 8x V100 32GB GPUs.

The following table lists the best hit rate at 10 for DGX-1 with 8 V100 32GB GPUs. It also shows the time to reach this HR@10. Results are averages across 20 random seeds. The training time was measured excluding data downloading, preprocessing, validation data generation and library initialization times.

To reproduce this result, start the NCF Docker container interactively and run:

./prepare_dataset.sh
python -m torch.distributed.launch --nproc_per_node=8 --use_env ncf.py --data /data/cache/ml-20m

Training accuracy: NVIDIA DGX-2 (16x V100 32GB)

Our results were obtained by following the steps in the Quick Start Guide in the PyTorch 21.04-py3 NGC container on NVIDIA DGX-2 with 16x V100 32GB GPUs.

The following table lists the best hit rate at 10 for DGX-2 with 16 V100 32GB GPUs. It also shows the time to reach this HR@10. Results are averages across 20 random seeds. The training time was measured excluding data downloading, preprocessing, validation data generation and library initialization times.

To reproduce this result, start the NCF Docker container interactively and run:

./prepare_dataset.sh
python -m torch.distributed.launch --nproc_per_node=16 --use_env ncf.py --data /data/cache/ml-20m

Influence of AMP on accuracy

The box plots below show the best accuracy achieved in each run. Twenty experiments were performed for each configuration.

Training validation curves

The plots below show the validation accuracy over the course of training. One sample curve is shown for each configuration.

Training performance results

Results are averages over 20 runs for each configuration.

Training performance: NVIDIA DGX A100 (8x A100 40GB)

Our results were obtained by following the steps in the Quick Start Guide in the PyTorch 21.04-py3 NGC container on NVIDIA DGX A100 (8x A100 40GB) GPUs. Performance numbers (in items per second) were averaged over an entire training epoch.

Training performance: NVIDIA DGX-1 (8x V100 16GB)

Our results were obtained by following the steps in the Quick Start Guide in the PyTorch 21.04-py3 NGC container on NVIDIA DGX-1 with 8x V100 16GB GPUs.

The following table shows the best training throughput:

Training performance: NVIDIA DGX-1 (8x V100 32GB)

Our results were obtained by following the steps in the Quick Start Guide in the PyTorch 21.04-py3 NGC container on NVIDIA DGX-1 with 8x V100 32GB GPUs.

The following table shows the best training throughput:

Training performance: NVIDIA DGX-2 (16x V100 32GB)

Our results were obtained by following the steps in the Quick Start Guide in the PyTorch 21.04-py3 NGC container on NVIDIA DGX-2 with 16x V100 32GB GPUs.

The following table shows the best training throughput:

Inference performance results

Inference performance: NVIDIA DGX A100 (1x A100 40GB)

Our results were obtained by running the inference.py script in the PyTorch 21.04 NGC container on NVIDIA DGX A100 with 1x A100 GPU.

TF32

FP16

Inference performance: NVIDIA DGX-1 (1x V100 16GB)

Our results were obtained by running the inference.py script in the PyTorch 21.04 NGC container on NVIDIA DGX-1 with 1x V100 16GB GPU.

FP32

FP16