feat(vgpu-manager): ship and start Fabric Manager on NVSwitch hosts

[lexfrei]

Description

RFC / Draft — seeking maintainer design direction before completing. Please do not merge.

On NVSwitch (HGX) systems running SR-IOV vGPU, NVIDIA Fabric Manager must run in vGPU multitenancy mode (FABRIC_MODE=2) instead of the default bare-metal / passthrough mode (FABRIC_MODE=0). In mode 0 a whole-card vGPU guest cannot initialize CUDA — cuInit fails with error 802 — because the guest's NVLink fabric never becomes usable.

Today the vGPU Manager container image ships no Fabric Manager and its entrypoint never starts it (Fabric Manager is installed only in the non-vGPU driver image, gated on DRIVER_TYPE != vgpu), so SR-IOV vGPU on NVSwitch cannot be brought up through this image.

This change makes the vGPU Manager image own Fabric Manager on vGPU nodes:

1. Ship Fabric Manager. The Ubuntu 22.04 / 24.04 vGPU Manager images install nvidia-fabricmanager + libnvidia-nscq (and, on driver branch ≥ 570, libnvsdm + nvlsm + infiniband-diags for the NVLink5 path) from the public CUDA repository, mirroring how the non-vGPU driver image installs them.
2. Apply FABRIC_MODE and start Fabric Manager on NVSwitch hosts. Before starting the daemon the entrypoint writes the FABRIC_MODE environment value into fabricmanager.cfg when set (validated as a non-negative integer); when unset the packaged default is left unchanged. Fabric Manager is started only when NVSwitch devices are present — using the NVLink5 start path (nvidia-fabricmanager-start.sh + NVLSM) on NVLink5+ systems, otherwise nv-fabricmanager — and is stopped before the driver is unloaded.

This is the image half of a two-repo change. The operator half — injecting FABRIC_MODE=2 into the vGPU Manager container env on the vm-vgpu sandbox-workload path — is NVIDIA/gpu-operator#2602; that env is inert without this change.

Open design questions (why this is an RFC)

• Which component should own Fabric Manager in a vGPU-on-NVSwitch deployment? On NVSwitch systems Fabric Manager is started today by the driver container, but in a vGPU deployment the vGPU Manager container is what runs on the node (and shipped no Fabric Manager). This PR implements the "vGPU-Manager-owns-Fabric-Manager" option to match #2602; having the driver container own it instead is also viable. I would like direction before finalizing.
• Where should the vGPU Manager image get Fabric Manager, and at which version? (central technical risk of this RFC) The Fabric Manager daemon is tightly version-coupled to the loaded driver — it enforces a driver-interface version check and can refuse to start against a mismatch. The non-vGPU image handles this by installing Fabric Manager from the public CUDA repository pinned to the exact driver version (=${DRIVER_VERSION}*), which resolves because the datacenter driver and its Fabric Manager share a version there. The vGPU Manager image is different: it must carry a Fabric Manager that matches the vgpu-kvm host driver baked into the image, and the public CUDA repository carries datacenter Fabric Manager whose patch versions do not track vgpu-kvm patch versions — so an exact public pin generally would not resolve. As a buildable placeholder this change installs from the public repository bounded to the driver's major branch (=<branch>.*), and Fabric Manager start is fatal on failure so a mismatch surfaces loudly (container restart) rather than silently. Direct question for maintainers: how does NVIDIA intend Fabric Manager to be packaged / sourced for the vGPU Manager image — should it come from the same source as the vgpu-kvm host driver (so the versions always match), rather than the public datacenter repository?
• Should this image default FABRIC_MODE to 2 when the variable is unset? No, by design here: the image is the mechanism (it reads FABRIC_MODE and applies it), and setting the mode is policy that belongs to the operator — #2602 injects FABRIC_MODE=2 on the vm-vgpu sandbox-workload path. So an unset FABRIC_MODE leaves the packaged fabricmanager.cfg default unchanged, which keeps the image a faithful consumer, avoids double-defaulting with the operator, and leaves existing non-multitenancy vGPU deployments unaffected on upgrade. Hardcoding mode 2 in the image would break that backward compatibility for standalone use and duplicate the operator's policy.
• Distro coverage. Implemented for the Ubuntu images. The RHEL vGPU Manager images can follow the same pattern (dnf) once the approach is agreed.

Out of scope: the host-installed-driver path (driver.enabled=false), and per-VM fabric-partition activation (a device-plugin concern).

Related: NVIDIA/gpu-operator#2602 (operator half), NVIDIA/gpu-operator#2594 (adjacent SR-IOV / vendor-specific VFIO vGPU device-creation gap on the same class of hardware).

Testing

• shellcheck on the changed entrypoints and hadolint on the changed Dockerfiles introduce no new findings versus the base branch.
• The entrypoint logic was exercised in isolation with stubs: non-NVSwitch host is a no-op; NVSwitch host without Fabric Manager installed skips with a message; FABRIC_MODE=2 rewrites an existing FABRIC_MODE=0; unset FABRIC_MODE leaves the config untouched; a non-numeric FABRIC_MODE is rejected with a warning; and a stale or empty Fabric Manager pid file no longer aborts driver teardown on shutdown.
• A full docker build of the vGPU Manager image was not run: it requires the proprietary NVIDIA-Linux-<arch>-<version>-vgpu-kvm.run host-driver installer in the build context, which is not publicly downloadable. End-to-end behavior on NVSwitch hardware depends on the companion operator change and cannot run in CI here.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.