Overview

Monitoring stacks usually consist of a collector, a time-series database to store metrics and a visualization layer. A popular open-source stack is Prometheus used along with Grafana as the visualization tool to create rich dashboards. Prometheus is deployed along with kube-state-metrics and node_exporter to expose cluster-level metrics for Kubernetes API objects and node-level metrics such as CPU utilization.

NVIDIA DCGM

NVIDIA DCGM is a set of tools for managing and monitoring NVIDIA GPUs in large scale linux based cluster environments. It's a low overhead tool that can perform a variety of functions including active health monitoring, diagnostics, system validation, policies, power and clock management, group configuration and accounting.

DCGM Exporter

DCGM-Exporter is an exporter for Prometheus to monitor the health and get metrics from GPUs. It leverages DCGM using Go bindings to collect GPU telemetry and exposes GPU metrics to Prometheus using an http endpoint (/metrics). DCGM-Exporter can be used either standalone or deployed as part of the NVIDIA GPU Operator.

Usage

For using the DCGM-Exporter, visit the user guide

License Agreements

• By pulling and using the container, you accept the terms and conditions of this End User License Agreement.

Suggested Reading

• Getting Started with DCGM-Exporter
• Code Repository for DCGM-Exporter