NVIDIA HPC-Benchmarks 26.02

The NVIDIA HPC-Benchmarks collection provides four benchmarks (HPL, HPL-MxP, HPCG, and STREAM) widely used in the HPC community optimized for performance on NVIDIA accelerated HPC systems.

NVIDIA's HPL and HPL-MxP benchmarks provide software packages to solve a (random) dense linear system in double precision (64-bit) arithmetic and in mixed precision arithmetic using Tensor Cores, respectively, on distributed-memory computers equipped with NVIDIA GPUs, based on the Netlib HPL benchmark and HPL-MxP benchmark.

NVIDIA's HPCG benchmark accelerates the High Performance Conjugate Gradients (HPCG) Benchmark. HPCG is a software package that performs a fixed number of multigrid preconditioned (using a symmetric Gauss-Seidel smoother) conjugate gradient (PCG) iterations using double precision (64-bit) floating point values.

NVIDIA's STREAM benchmark is a simple synthetic benchmark program that measures sustainable memory bandwidth. NVIDIA HPC-Benchmarks container includes STREAM benchmarks optimized for NVIDIA Ampere GPU architecture (sm80), NVIDIA Hopper GPU architecture (sm90), NVIDIA Blackwell GPU architecture (sm100/sm103), and NVIDIA Grace CPU.

The NVIDIA HPC Benchmark package includes microbenchmarks designed to assess system readiness before running large-scale benchmarks.

• NCCL tests
• NVSHMEM performance tests
• OSU MPI benchmark
• GEMM (matrix-matrix multiplication) benchmark

Container packages

The NVIDIA HPC-Benchmarks collection provides a multiplatform (x86 and aarch64) container image hpc-benchmarks:26.02 based on NVIDIA Optimized Frameworks 26.01 container image.

The HPC benchmarks package stores open-source code inside the container at /workspace/open_source/, in addition to the DLFW open source code.

In addition to NVIDIA Optimized Frameworks 26.02 container images, the hpc-benchmarks:26.02 container image is provided with the following packages embedded:

• NVIDIA HPL 26.02
• NVIDIA HPL-MxP 26.02
• NVIDIA HPCG 26.02
• NVIDIA STREAM 26.02
• NVIDIA NVSHMEM 3.5.19
• NVIDIA NVPL 25.1

NVIDIA HPC-Benchmarks for MPI libraries that are ABI-compatible with MPICH (e.g., MPICH, Cray MPICH, MVAPICH, etc.) and OpenMPI available on NVIDIA.DEVELOPER.

Prerequisites

Using the NVIDIA HPC-Benchmarks Container requires the host system to have the following installed:

• Docker Engine
• NVIDIA GPU Drivers
• NVIDIA Container Toolkit or NVIDIA Pyxis/Enroot, or Singularity version 3.4.1 or later

For supported versions, see the Framework Containers Support Matrix and the NVIDIA Container Toolkit Documentation

The NVIDIA HPC-Benchmarks Container supports the NVIDIA Ampere GPU architecture (sm80), the NVIDIA Hopper GPU architecture (sm90), the NVIDIA Blackwell GPU architecture (sm100/sm103). This version of the container supports clusters featuring DGX A100, DGX H100, DGX B200/B300, NVIDIA Grace Hopper, NVIDIA Grace Blackwell NVL, and NVIDIA Grace CPU nodes. Previous GPU generations are not expected to be compatible.

Notes:

• (G)B300 (sm103) has low native FP64 throughput, therefore, the performance of the NVIDIA HPL Benchmark on this hardware is expected to be low
• The NVIDIA HPL benchmark does not support FP64 emulation on (G)B300

NVIDIA HPC-Benchmarks documentation

The NVIDIA HPC-Benchmarks documentation is available on NVIDIA Documentation Hub.

Running with Pyxis/Enroot

The examples below use Pyxis/enroot from NVIDIA to facilitate running HPC-Benchmarks Containers. Note that an enroot .credentials file is necessary to use these NGC containers.

To copy and customize the sample Slurm scripts and/or sample HPL.dat/hpcg.dat files from the containers, run the container in interactive mode, while mounting a folder outside the container, and copy the needed files, as follows:

CONT='nvcr.io#nvidia/hpc-benchmarks:26.02'
MOUNT="$PWD:/home_pwd"

srun -N 1 --cpu-bind=none --mpi=pmix \
     --container-image="${CONT}" \
     --container-mounts="${MOUNT}" \
     --pty bash

Once inside the container, copy the needed files to /home_pwd.

NVIDIA HPL, NVIDIA HPL-MxP, NVIDIA HPCG and NVIDIA STREAM Benchmarks with support of GPU

Examples of NVIDIA HPL run

Several sample input files are available in the container at /workspace/hpl-linux-x86_64 or /workspace/hpl-linux-aarch64-gpu.

To run NVIDIA HPL on nodes 8 with 8 GPUs using provided sample HPL-64GPUs.dat files:

CONT='nvcr.io#nvidia/hpc-benchmarks:26.02'

srun -N 8 --ntasks-per-node=8 --cpu-bind=none --mpi=pmix \
     --container-image="${CONT}" \
     ./hpl.sh --dat /workspace/hpl-linux-x86_64/sample-dat/HPL-64GPUs.dat

Examples of NVIDIA HPL-MxP run

To run NVIDIA HPL-MxP on 4 nodes, each node with 4 GPUs:

CONT='nvcr.io/nvidia/hpc-benchmarks:26.02'

srun -N 4 --ntasks-per-node=4 \
     --container-image="${CONT}" \
     ./hpl-mxp.sh --n 280000 --nb 2048 --nprow 4 --npcol 4 --nporder row --gpu-affinity 0:1:2:3 --cpu-affinity 0-55:56-111:112-167:168-223

Pay special attention to CPU cores affinity/binding, as it greatly affects the performance of the HPL benchmarks.

Examples of NVIDIA HPCG run

Several sample input file are available in the container at /workspace/hpcg-linux-x86_64 or /workspace/hpcg-linux-aarch64

To run NVIDIA HPCG on a single node with 8 GPUs using your custom hpcg.dat file on x86:

CONT='nvcr.io/nvidia/hpc-benchmarks:26.02'
MOUNT="/path/to/your/custom/dat-files:/my-dat-files"

srun -N 1 --ntasks-per-node=8 --cpu-bind=none --mpi=pmix \
     --container-image="${CONT}" \
     --container-mounts="${MOUNT}" \
     ./hpcg.sh --dat /my-dat-files/hpcg.dat

To run NVIDIA HPCG on nodes 16 with 4 GPUs using script parameters on x86:

CONT='nvcr.io/nvidia/hpc-benchmarks:26.02'
MOUNT="/path/to/your/custom/dat-files:/my-dat-files"

srun -N 16 --ntasks-per-node=4 --cpu-bind=none --mpi=pmix \
     --container-image="${CONT}" \
     --container-mounts="${MOUNT}" \
     ./hpcg.sh --nx 512 --ny 512 --nz 256 --rt 2 --cpu-affinity 0-55:56-111:112-167:168-223 --mem-affinity 0:0:1:1

Examples of NVIDIA STREAM run

To run NVIDIA STREAM for GPU on the device 1 and the number of elements in arrays 10000000. Note: it's recommended to use default value of the number of elements in arrays. The value above is for demonstration only.

CONT='nvcr.io/nvidia/hpc-benchmarks:26.02'

srun -N 1 --ntasks-per-node=4 --cpu-bind=none --mpi=pmix \
     ./stream-gpu-test.sh --d 1 --n 10000000

NVIDIA HPL, NVIDIA HPL-MxP, and NVIDIA HPCG for NVIDIA Grace CPU

Examples of NVIDIA HPL run

Several sample input files are available in the container at /workspace/hpl-linux-aarch64.

To run NVIDIA HPL on two nodes of NVIDIA Grace CPU using your custom HPL.dat file:

CONT='nvcr.io#nvidia/hpc-benchmarks:26.02'
MOUNT="/path/to/your/custom/dat-files:/my-dat-files"

srun -N 2 --ntasks-per-node=2 --cpu-bind=none --mpi=pmix \
     --container-image="${CONT}" \
     --container-mounts="${MOUNT}" \
     ./hpl-aarch64.sh --dat /my-dat-files/HPL.dat --cpu-affinity 0-71:72-143 --mem-affinity 0:1

Examples of NVIDIA HPL-MxP run

To run NVIDIA HPL-MxP on a 4 node of NVIDIA Grace Hopper x4:

CONT='nvcr.io/nvidia/hpc-benchmarks:26.02'

srun -N 4 --ntasks-per-node=4 \
     --container-image="${CONT}" \
     ./hpl-mxp-aarch64.sh --n 380000 --nb 2048 --nprow 4 --npcol 4 --nporder row \
     --cpu-affinity 0-71:72-143:144-215:216-287 \
     --mem-affinity 0:1:2:3

Examples of NVIDIA HPCG run

Sample input files are available in the container at /workspace/hpcg-linux-aarch64

To run NVIDIA HPCG on two nodes of NVIDIA Grace CPU using your custom parameters:

CONT='nvcr.io/nvidia/hpc-benchmarks:26.02'
MOUNT="/path/to/your/custom/dat-files:/my-dat-files"

srun -N 2 --ntasks-per-node=4 --cpu-bind=none --mpi=pmix \
     --container-image="${CONT}" \
     --container-mounts="${MOUNT}" \
     ./hpcg-aarch64.sh --exm 1 --nx 512 --ny 512 --nz 288 --rt 30 --cpu-affinity 0-35:36-71:72-107:108-143 --mem-affinity 0:0:1:1

To run NVIDIA HPCG on NVIDIA Grace Hopper x4 using script parameters on aarch64:

CONT='nvcr.io/nvidia/hpc-benchmarks:26.02'
MOUNT="/path/to/your/custom/dat-files:/my-dat-files"

# GPU + Grace (Heterogeneous execution)
# GPU rank has 8 OpenMP threads and Grace rank has 64 OpenMP threads
srun -N 2 --ntasks-per-node=8 --cpu-bind=none --mpi=pmix \
     --container-image="${CONT}" \
     --container-mounts="${MOUNT}" \
     ./hpcg-arch64.sh --nx 256 --ny 1024 --nz 288 --rt 2 \
     --exm 2 --ddm 2 --lpm 1 --g2c 64 \
     --npx 4 --npy 4 --npz 1 \
     --cpu-affinity 0-7:8-71:72-79:80-143:144-151:152-215:216-223:224-287 \
     --mem-affinity 0:0:1:1:2:2:3:3

Examples of NVIDIA STREAM run

To run NVIDIA STREAM for CPU on 72 threads and the number of elements in arrays 10000000. Note: it's recommended to use default value of the number of elements in arrays. The value above is for demonstration only.

CONT='nvcr.io/nvidia/hpc-benchmarks:26.02'

srun -N 1 --ntasks-per-node=4 --cpu-bind=none --mpi=pmix \
     ./stream-cpu-test.sh --t 144 --n 10000000

Known issues

• Known performance issue with HPC-X 2.25 affects the MPI_Alltoall, MPI_Bcast, and MPI_Allgather operations, which may impact HPL and HPL-MxP performance. Refer to the HPC-X 2.25 release notes for workarounds to resolve this issue.

Resources

• World's Fastest Supercomputer Triples Performance Record | NVIDIA Blog
• Netlib HPL benchmark
• HPL Mixed-Precision Benchmark
• HPCG Benchmark
• Developer Forums
• NGC Catalog
• NVIDA HPCG
• Accelerating the HPCG Benchmark with NVIDIA Math Sparse Libraries

Support

• For questions or to provide feedback, please contact HPCBenchmarks@nvidia.com

License

By pulling and using the container, you accept the terms and conditions of this End User License Agreement and Product-Specific Terms.