NVIDIA System Management (NVSM) Provision is a part of multinode software framework for provisioning the aggregator and NVIDIA DGX nodes cluster.

Overview

The NVSM provision container is a key component in the NVSM multi-node architecture. It contains Ansible playbooks that handle the provisioning of both the aggregator node and compute nodes in the cluster.

Key Functions

• Provisions the aggregator node and compute nodes in the cluster
• Copies certificates from the aggregator node to compute nodes
• Restarts compute nodes' NVSM services to establish connection with the aggregator node
• Manages the multinode inventory file

Prerequisites

Before running the provision container, ensure:

1. Docker and docker-compose are installed on the aggregator node:
   apt-get install docker
   apt-get install docker-compose
   

2. All required containers are loaded:
   docker load -i nvsm-prometheus_25.03.05.tar.gz
   docker load -i nvsm-grafana_25.03.05.tar.gz
   docker load -i nvsm-provision_25.03.05.tar.gz
   docker load -i nvsm-aggregator_25.03.05.tar.gz
   

Inventory File Management

The provision container uses an inventory file (/etc/nvsm/aggregator/inventory.yaml) that contains:

• BMC IP addresses
• Host IP addresses
• Passwords for compute nodes
• Node-specific configurations

The inventory file can be encrypted using admin defined password.

Running the Provision Container

docker run -it --rm -v /etc/nvsm/aggregator:/etc/nvsm/mn nvcr.io/nvstaging/nvsm/nvsm-provision:25.03.05

Post-Provisioning

After provisioning:

1. Check that all containers are running:

   sudo docker ps

2. Verify NVSM services are active

   sudo docker exec nvsm-aggregator nvsm status

   sudo docker exec nvsm-aggregator nvsm_lifecycle status

Important Notes

• The provisioning process restarts NVSM services on compute nodes
• NVSM commands may error out during the initial NVSM INIT phase (3-5 minutes)
• Ensure time synchronization between aggregator and compute nodes

References

• NVSM Documentation
• License
• Sources