feat(validator): commit pending MIG-mode change via targeted GPU reset

[lexfrei]

Description

RFC / DRAFT — opening for design feedback before the runtime path is exercised on GPU hardware. See the "Destructive GPU reset — why it is safe here" section below.

On the vGPU (sandbox) workload path, a MIG-mode change does not take effect until the GPU is reset, and neither the MIG mode nor the SR-IOV VFs survive a node reboot. The MIG manager can set MIG mode to Enabled, but committing that change requires a GPU reset it does not perform — its only reset mechanism reboots the whole node (WITH_REBOOT). So after a reboot of a MIG-backed vGPU node, MIG mode is left requested-but-not-committed and MIG-backed vGPU devices cannot be created until the state is re-established out-of-band (host systemd units that enable MIG, run nvidia-smi --gpu-reset, then sriov-manage -e).

This adds the MIG-mode-commit half of the reboot-recovery seam discussed in #2600. During vGPU Manager validation, right before the wait for VFs, VGPUManager.validate() commits a pending-but-uncommitted MIG-mode enable via a targeted nvidia-smi --gpu-reset. It composes with the SR-IOV VF re-enable seam in #2601: the reset must run before VF re-enablement, because a GPU reset requires SR-IOV to be disabled first. Together they let MIG-backed vGPU recover after a reboot without a hand-rolled host unit.

Destructive GPU reset — why it is safe here

nvidia-smi --gpu-reset is destructive: it resets the GPU and tears down anything running on it. The step is therefore off by default and tightly gated. It performs a reset on a GPU only when all of the following hold:

• Opt-in. A new flag --commit-mig-mode-with-gpu-reset (env COMMIT_MIG_MODE_WITH_GPU_RESET), default false. When unset, behavior is identical to today — no reset, validate() unchanged.
• vGPU path only. commitMIGMode is reached only from VGPUManager.validate(), which early-returns for non-vm-vgpu workloads.
• MIG-mode requested but not committed. mig.mode.pending == Enabled and mig.mode.current != Enabled — i.e. a change something else already requested that a reset would apply. The enable is never decided here; only committed.
• No SR-IOV VFs enabled. VFs do not survive a reboot, so a GPU that still has VFs has a vGPU VM attached; a GPU reset also requires SR-IOV to be disabled first.
• No running compute processes. A fast-path guard; nvidia-smi --gpu-reset itself independently refuses to reset a GPU that is in use, which is the authoritative safety net.

In the steady state (MIG already committed, or VFs/VMs present) it is a no-op. It is best-effort: reset failures are logged and validation continues, preserving current behavior for setups that commit MIG mode and create VFs out-of-band.

On NVSwitch / Fabric Manager systems, a single-GPU reset can interact with fabric partition state; this change only commits MIG mode and does not manage fabric partitions, which is one of the open design questions for the RFC.

Checklist

• No secrets, sensitive information, or unrelated changes
• Lint checks passing (make lint) — the changed package is clean: GOOS=linux golangci-lint run ./cmd/nvidia-validator/ reports 0 issues. Repo-wide make lint was not run locally.
• Generated assets in-sync (make validate-generated-assets) — this change adds no API/CRD changes, so no generated assets change; the full target was not run.
• Go mod artifacts in-sync (make validate-modules)
• Test cases are added for new code paths

Testing

• gofmt, go build, go vet, and golangci-lint run on the changed package, cross-compiled GOOS=linux (the validator is a Linux-only binary): clean, 0 issues. No asset or go.mod/go.sum changes.
• New unit tests run green in a Linux container (go test ./cmd/nvidia-validator/): TestNormalizePCIAddress (nvidia-smi 8-digit vs go-nvlib 4-digit PCI domains normalize to the same key), TestParseMIGModes (CSV parsing, [N/A], malformed rows, the pending-vs-current guard), TestMIGModeNeedsCommit (the enable-only commit predicate across every current/pending combination, including case-insensitivity), and TestShouldResetForMIGCommit (the full reset-decision guard matrix over uncommitted-enable × VFs-present × workload-running).
• Not yet exercised on GPU hardware; the runtime chroot / nvidia-smi --gpu-reset path relies on CI and cluster testing. This is why the PR is opened as a draft/RFC.

Refs #2600, #2601.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.