NVIDIA System Management (NVSM) Prometheus is a part of multinode software framework for monitoring NVIDIA DGX nodes cluster

Introduction

The NVSM Prometheus Container is a Docker container that serves as the metrics storage and querying system for the NVSM multi-node architecture. It collects, stores, and provides access to time-series data from the aggregator and compute nodes, enabling comprehensive monitoring and analysis capabilities.

Key Features

• Time-series metrics storage
• Real-time data collection
• Multi-node metrics aggregation

Architecture

Core Components

1. Prometheus Server

   • Time-series database
   • Data collection and storage
   • Query engine

2. Data Collection

   • Pull-based metrics collection
   • Data retention management

3. Query Interface

   • PromQL query language
   • HTTP API
   • Data visualization support
   • Export capabilities

Container Specifications

System Requirements

• Part of the NVSM multi-node stack
• Requires NVSM Aggregator container

Network Configuration

• Web Interface: 9090/tcp
• Metrics Collection: 9123/tcp (from NVSM Exporter)

Accessing Prometheus

• Web Interface: http://[aggregator-host]:9090
• API Endpoint: http://[aggregator-host]:9090/api/v1
• Query Interface: http://[aggregator-host]:9090/graph

Important Notes

• Requires NVSM Aggregator container for metrics
• Supports custom metric collection
• Configurable data retention policies
• Part of the NVSM multi-node stack

References

• NVSM Documentation
• License
• Sources