DCGM Exporter converts selected DCGM telemetry fields to Prometheus text format. Run one exporter on each GPU node that Prometheus monitors.
Overview
DCGM Exporter converts selected DCGM telemetry fields to Prometheus text format. Install one exporter on each GPU node that Prometheus should monitor.
DCGM Exporter can run as a package-managed systemd service, an OCI container, or a Kubernetes DaemonSet managed by Helm or the NVIDIA GPU Operator.
By default the exporter initializes an embedded DCGM host engine inside its own process through libdcgm. It can instead connect to a separately managed DCGM host engine over TCP, a Unix socket, or VSOCK.
Requirements and version compatibility
Before installation, verify that the target system has:
- An NVIDIA GPU, driver, and DCGM combination covered by the DCGM product support matrix: https://docs.nvidia.com/datacenter/dcgm/latest/user-guide/getting-started.html#supported-platforms
- Access to the NVIDIA GPUs from the selected runtime. Container deployments require the NVIDIA Container Toolkit.
- Network access from Prometheus to the exporter HTTP endpoint when Prometheus runs on another host or cluster.
DCGM determines which GPUs, Linux distributions, and architectures this image supports, and the minimum NVIDIA driver version, CUDA version, host memory, and CPU core count. The support matrix above lists them.
DCGM Exporter releases pair a DCGM version with an exporter version.
Container image tags encode the tested pair as
<dcgm-version>-<dcgm-exporter-version>-<image-variant>.
Running dcgm-exporter --version reports the paired DCGM and exporter versions
embedded in the executable.
Run DCGM Exporter with the DCGM version paired with that exporter release. Mismatched combinations might function, but they are not tested or supported.
Run the container
Use latest to follow the current exporter release, or choose a
semantic-version tag to control upgrades. This command exposes every GPU
and allows DCGM profiling fields such as DCGM_FI_PROF_*:
export DCGM_EXPORTER_TAG=4.6.0-4.8.4-distroless # or latest, 4.8.4
docker run -d --rm --name dcgm-exporter --gpus all --cap-add SYS_ADMIN -p 9400:9400 nvcr.io/nvidia/k8s/dcgm-exporter:${DCGM_EXPORTER_TAG}
Review Docker's runtime privilege and Linux capabilities documentation
before adapting the command to a production host.
The exporter serves metrics on port 9400. DCGM_FI_PROF_* fields require
GPU support and the SYS_ADMIN capability.
Related software
- The NVIDIA driver and NVML provide GPU access and the management library that DCGM uses. Every deployment needs them.
- nvidia-smi verifies that the driver can discover the GPUs, and is the first check when troubleshooting.
- DCGM discovers entities, owns field watches, and supplies cached telemetry.
- The NVIDIA Container Toolkit exposes NVIDIA GPUs to containers, so this image requires it.
- The NVIDIA GPU Operator manages the Kubernetes GPU software stack and the exporter lifecycle.
- Helm installs and upgrades the standalone DCGM Exporter chart, which requires Kubernetes 1.19 or later.
- Prometheus scrapes, stores, and queries the exported time series. The Prometheus Operator supplies the ServiceMonitor custom resource that the chart's default scrape configuration uses.
Documentation
- About DCGM: https://docs.nvidia.com/datacenter/dcgm/latest/about.html
- Install DCGM Exporter: https://docs.nvidia.com/datacenter/dcgm/latest/installation/install-dcgm-exporter.html
- Source and Helm chart: https://github.com/NVIDIA/dcgm-exporter
License
DCGM Exporter source is licensed under the Apache License, Version 2.0:
- https://github.com/NVIDIA/dcgm-exporter/blob/main/LICENSE
- https://www.apache.org/licenses/LICENSE-2.0
Use of this NGC container is also subject to the NVIDIA Software License Agreement:
NVIDIA Corporation publishes this image.
Get Help
Community
Open an issue at https://github.com/NVIDIA/dcgm-exporter/issues
Do not use public issues for security reports. Use https://www.nvidia.com/en-us/security/
Enterprise
Get access to knowledge base articles and support cases or submit a ticket.