Integrate vendor VFIO vGPUs into the instance lifecycle

[yummybomb]

Summary

Top layer of the vendor VFIO vGPU stack (generalize-vgpu-device #322 ← hypervisor-liveness #363 ← vendor-vfio-backend #364 ← this). The backend itself is in #364; this layer wires it into the instance lifecycle and adds the release guards that make reusable-VF assignments safe:

• Live-instance claim guard on release — an assignment goes stale when its release succeeds but the metadata save does not (or start fails between the release and its first save). Since vendor VFIO reuses the same VF path across assignments, honoring a stale claim could clear a VF out from under the live instance that owns it now. Every release consults live instance metadata (liveness verified via Harden hypervisor process liveness checks #363's socket-ownership check) and drops stale metadata without touching the device.
• Assignment durability — assignments are tagged with the owning instance ID, persisted before booting a started instance without carrying over the previous VMM identity, and retained in metadata when rollback release fails in create/start so later release paths can still find the device.
• Fail-closed startup reconciliation — the protected set of live VFs is built from ListInstancesForReconcile, which fails on any unreadable metadata instead of silently skipping it. Recent assignments receive bounded startup protection when their PID is absent or stale; when the inventory is unavailable, vendor VFIO reconciliation is skipped entirely while mdev reconciliation still runs.
• Hypervisor scope — hypervisor selection remains a caller policy, preserving the framework layer's existing mdev behavior. The production rollout continues to place vGPU instances on QEMU; the Cloud Hypervisor vendor-VFIO limitation (upstream cloud-hypervisor#7572) is documented in lib/devices/GPU.md (in the base PR Add the vendor VFIO vGPU device backend #364).
• Failure reporting — a failed create or start whose vGPU release also failed surfaces as vgpu_cleanup_pending with retained/unretained guidance; the retention record keeps identity fields (name, image, hypervisor) so it lists as a recognizable, deletable instance. The instance API exposes gpu.device_path, the assignment identity on vendor VFIO hosts where mdev_uuid is empty.

Testing

• go build ./..., go vet ./... clean
• go test -race passes for lib/instances targeted suites, lib/devices, lib/resources, lib/builds (TestCreateInstanceWithNetwork needs image pulls + iptables and fails in this environment on the unmodified base as well)

---

Note

High Risk
Changes core GPU assignment, release, reconciliation, and hypervisor teardown paths; mistakes can strand VFs, clear assignments under live VMs, or wedge hardware until SR-IOV reset.

Overview
Vendor VFIO vGPUs are wired through create, start, stop, and delete (the create path no longer rejects that framework). Assignments persist GPUDevicePath, GPUAssignedAt, and instance ID; the API exposes gpu.device_path and maps failed rollback to vgpu_cleanup_pending with delete vs startup-reconcile guidance.

Release safety for reusable VF paths: vendor VFIO releases scan live instance metadata and fail closed on ambiguous claims; stale claims drop without destroying a device another instance owns. Failed create/start rollback can leave a delete-only retention stub (GPURetainedForCleanup); start, fork, and snapshot refuse those records. Delete schedules background orphaned-VF retries after a failed release.

Startup reconciliation replaces mdev-only cleanup with ReconcileVGPUs, building a protected device set from ListInstancesForReconcile (strict on bad metadata) with grace for recent assignments without a live PID, and optional delayed re-reconcile.

VF wedge mitigation: VFIO QEMU start-failure cleanup uses SIGTERM before SIGKILL; initializing vGPU hypervisors get the same on force-kill, with a warning if SIGTERM fails.

^{Reviewed by Cursor Bugbot for commit ecf86b6. Bugbot is set up for automated code reviews on this repo. Configure here.}

[yummybomb]

Trimmed over-engineering after an architectural review against the parent task (host kernel 6.8 / Ubuntu 24.04 support). Most of the removed weight came from earlier review-round additions rather than the original design:

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 032e74c. Configure here.}