SR-IOV Network Operator Config Daemon

Overview

The SR-IOV Network Operator Config Daemon is a Kubernetes component that automates the discovery and configuration of Single Root I/O Virtualization (SR-IOV) capable network interface cards (NICs) on cluster nodes. It enables high-performance, low-latency network connectivity for containerized workloads by managing hardware-level network virtualization.

What It Does

The Config Daemon runs as a privileged DaemonSet on each node and performs the following tasks:

• Hardware Discovery: Automatically detects SR-IOV capable NICs and their virtual functions (VFs) on each node
• Configuration Management: Applies network configurations specified in SriovNetworkNodeState custom resources, including:
  • Virtual function creation and allocation
  • Driver binding and unbinding
  • Hardware offload features (RDMA, hardware checksumming, etc.)
  • NIC firmware management for supported vendors (e.g., Mellanox devices via mstflint)
• Status Synchronization: Continuously monitors NIC hardware state and reports it back to Kubernetes via custom resource status updates
• Safe Configuration: Uses a two-phase configuration process (pre-network and post-network) to safely apply changes without disrupting node connectivity

Key Benefits

• Improved Network Performance: Direct hardware access reduces CPU overhead and latency compared to traditional virtual networking
• Resource Isolation: Dedicated VFs provide network-level isolation between workloads
• Hardware Acceleration: Enables offloading of network processing to hardware (RDMA, SR-IOV, hardware checksumming)
• Automated Management: Eliminates manual NIC configuration, reducing operational complexity and human error
• Multi-Platform Support: Works on bare metal servers and virtualized environments (KubeVirt, OpenStack)

Use Cases

• High-Performance Computing (HPC): Workloads requiring low-latency, high-bandwidth inter-node communication
• Telecommunications (Telco/NFV): Network functions requiring line-rate packet processing
• Data-Intensive Applications: Database clusters, distributed storage systems, and analytics workloads
• GPU-Accelerated Workloads: ML/AI applications needing high-speed data ingestion
• 5G Core Network Functions: User Plane Functions (UPF) and other latency-sensitive network services

Prerequisites and Dependencies

• Kubernetes: Version 1.16 or later
• Node Requirements:
  • SR-IOV capable NICs with supported chipsets (Intel, Mellanox, Broadcom, etc.)
  • IOMMU enabled in BIOS and kernel (intel_iommu=on or amd_iommu=on)
  • Linux kernel with SR-IOV support (3.10+)
• Required Packages: pciutils, hwdata, and optionally mstflint (for Mellanox devices)
• Operator Components: SR-IOV Network Operator must be installed and running

Configuration

The daemon is configured through SriovNetworkNodePolicy and SriovNetworkNodeState custom resources. Configuration includes:

• NIC Selection: Target specific NICs by vendor, device ID, PCI address, or interface name
• VF Parameters: Number of VFs, VF driver, MTU, link type (ETH/IB)
• Advanced Settings: Device plugins, RDMA mode, external device plugin management
• Platform-Specific Options: Switchdev mode for hardware offloading, externally managed devices

Operational Features

• Comprehensive Logging: Detailed logs for troubleshooting configuration issues
• Error Handling: Graceful failure modes with status reporting to prevent node disruption
• Parallel Configuration: Optional parallel NIC configuration for faster deployment
• Idempotent Operations: Safe to run repeatedly without causing unintended changes
• Node Reboots: Automatically triggers node draining and reboots when required for hardware changes

Documentation

For information on platform support and getting started, visit the official documentation

Learn More

The Network Operator is an open-source project. For more information on contributions and release artifacts, see the GitHub repo

Get Help

Enterprise Support Get access to knowledge base articles and support cases or submit a ticket.

NVIDIA AI Enterprise Documentation

Visit the NVIDIA AI Enterprise Documentation Hub for release documentation, deployment guides and more.

License Agreement

The NVIDIA Network Operator is licensed under Apache 2.0 and contributions are accepted with a DCO. See the contributing document for more information on how to contribute and the release artifacts.