GPUd is designed to ensure GPU efficiency and reliability by actively monitoring GPUs and effectively managing AI/ML workloads. License: Apache License 2.0
Overview
GPUd is an open-source daemon for monitoring NVIDIA GPU health and the surrounding host and container runtime. It collects telemetry, detects hardware and software errors, and provides diagnostic data for individual nodes and large GPU fleets.
GPUd can run directly on a host, as a systemd service, or as a Kubernetes DaemonSet through the official Helm chart.
Core features
| Area | Capabilities |
|---|---|
| GPU telemetry | Monitors utilization, memory, power, temperature, clock speed, processes, and remapped rows. |
| Error detection | Detects Xid, SXid, ECC, hardware slowdown, and other errors through kernel logs, NVML, nvidia-smi, and optional DCGM checks. |
| GPU fabric | Monitors NVLink, NVSwitch, InfiniBand, Fabric Manager, NCCL, and peer-memory status where available. |
| Host monitoring | Reports CPU, memory, disk, operating system, PCI, network, and file-descriptor health. |
| Runtime monitoring | Tracks Docker, containerd, and kubelet status alongside GPU workloads. |
| Observability | Exposes health information and metrics for local diagnostics and integration with centralized monitoring systems. |
See the component reference for the full list of checks.
Typical use cases
- Monitor GPU health across bare-metal, virtual-machine, and Kubernetes fleets.
- Detect Xid, ECC, thermal, power, and fabric problems before assigning new workloads to a node.
- Distinguish software faults that may be resolved by a restart from hardware faults that may require repair.
- Validate node health after driver changes, reboots, or hardware maintenance.
- Correlate GPU failures with host, container-runtime, and Kubernetes state.
- Export GPU and host telemetry to an existing observability platform.
GPUd complements tools such as node_exporter and dcgm-exporter by combining GPU health checks, host monitoring, and operational diagnostics in one agent. See the project's design rationale and comparisons.
Deployment options
- Linux host: Install the standalone binary and run GPUd with
systemd. - Kubernetes: Deploy GPUd as a DaemonSet using the official Helm chart.
- Container environments: Monitor Docker, containerd, and kubelet from the same agent.
- DGX Cloud Lepton: Connect GPUd to the platform with a token for centralized health reporting. GPUd is used in DGX Cloud Lepton production infrastructure.
System requirements
| Area | Minimum requirement or limitation |
|---|---|
| Operating system | The documented installation path targets Linux on amd64. The installer also supports Linux arm64 and macOS, although some Linux-specific checks are unavailable on macOS. |
| GPU hardware | An NVIDIA GPU is required for GPU-specific telemetry and health checks. GPUd can still collect supported host metrics when no GPU is present. |
| NVIDIA software | GPU checks require a working NVIDIA driver with nvidia-smi and NVML. Optional checks may also use DCGM, NCCL, Fabric Manager, InfiniBand, or nvidia-peermem. |
| Driver version | GPUd does not publish one universal minimum NVIDIA driver version. Use a driver supported by the installed GPU and any optional NVIDIA components enabled on the host. |
| Kubernetes | Kubernetes 1.19 or later and Helm 3.16.1 or later. |
| Lepton-connected node | When joining DGX Cloud Lepton with --token: at least 3 CPU cores and 4 GiB of memory. The recommended allocation is 4 or more CPU cores and 8 GiB or more of memory. Standalone GPUd has lower requirements. |
See the installation documentation for current platform details.
Project and support
GPUd is maintained by Lepton AI with contributions from the open-source community. Copyright is held by Lepton AI Inc., as recorded in the project's NOTICE.
- Source code and releases: github.com/leptonai/gpud
- Bug reports and feature requests: GitHub Issues
- Contribution guidelines: CONTRIBUTING.md
- Security reports: security@lepton.ai
License
GPUd is released under the Apache License 2.0.