Add the vendor VFIO vGPU device backend - #364
Conversation
2bb8e86 to
7fc3b49
Compare
7fc3b49 to
f661e63
Compare
f661e63 to
d1207d0
Compare
d1207d0 to
5b47670
Compare
5b47670 to
ffbf8a0
Compare
ffbf8a0 to
78801e7
Compare
78801e7 to
b4651f3
Compare
b4651f3 to
d2bf3fa
Compare
d2bf3fa to
495ab62
Compare
|
added 7f5233f: report |
7f5233f to
0baacb3
Compare
-->
✱ stlc build✅ go code · compare
✅ python code · compare
✅ typescript code · compare
Diagnostics: ❗ 0 new / 1 total error, 💡 0 new / 5 total note
Build metadata
This comment is auto-generated by stlc and is kept up to date as you push. |
58539a3 to
ed5a990
Compare
Linux 6.8 hosts with NVIDIA R580 drop the mdev interface: vGPUs are assigned by writing a type ID to a VF's nvidia/current_vgpu_type and passed to QEMU as a plain VFIO PCI device. Add a vendor VFIO backend behind the existing framework dispatch: profile discovery from the capacity-dependent creatable catalogs, least-loaded VF placement, create/verify/rollback, and release. Because the same VF path is reused across assignments (unlike mdev UUIDs), release is guarded: an in-process owner map covers the window before QEMU opens the device, and an open-VFIO-handle scan refuses to clear a VF a running VM still holds. Reconciliation clears orphaned assignments on startup, skipping VFs protected by the caller and failing closed when the protected set is unavailable. Branch the vGPU integration test by discovered framework and extend it to cover release on stop and reacquisition on start.
Sort GPUs with unaccountable load last instead of rejecting placement, and stop reporting passthrough capacity when vGPU discovery fails.
…nership through releases
The instance lifecycle already routes create/start/stop/delete through CreateVGPU/DestroyVGPU, so dispatching vendor VFIO creates here would activate the backend before assignment durability and release guards exist. Reject vendor VFIO creates for now; destroy stays wired so existing assignments remain releasable. The integration test skips on vendor VFIO hosts at this layer and no longer asserts the transitional stop-retention behavior.
Counting every free VF advertising a type overreports concurrent capacity: sibling VFs share their parent GPU's framebuffer, so one 48Q assignment revokes the type from every other VF on that GPU. Bound each GPU's contribution by both its free VFs and how many times the profile framebuffer fits into the GPU's remaining framebuffer, using the largest still-creatable profile as a lower bound on what remains.
This reverts commit d6adc4c.
A single unreadable current_vgpu_type failed discoverVFs wholesale, and GetGPUStatus turns a discovery error into a host with no GPU, so one flaky sysfs read blanked out the host's entire GPU capacity for admission and monitoring. Skip unreadable VFs with a warning and keep the readable inventory: a skipped VF is never selected for placement and never reconciled, both safe directions. When no VF is readable, discovery still fails so a wholesale sysfs outage cannot demote a vGPU host to passthrough while assignments exist. Also document that vendor VFIO vGPUs are known broken on Cloud Hypervisor upstream and QEMU is the required hypervisor for GPU instances.
listProfiles failed wholesale when one VF's creatable_vgpu_types read failed, blanking every advertised profile while discoverVFs directly above it already skips unreadable VFs for exactly that reason. Skip and warn instead; underreporting is the safe direction for status and admission. Also document why openVFIOPaths stays strict where mdev's scan is lax (it authorizes clearing a reused VF path), and the 0Q/0B parsing caveat in framebufferFromProfileName.
listProfiles skips an unreadable VF but create still failed placement wholesale when profileMetadata or selectLeastLoadedVF hit the same VF, so /resources could advertise capacity a create then failed to use. Skip the VF in both loops; it simply stops being a placement candidate.
A process that exits between the /proc listing and its fd walk surfaces ENOENT or ESRCH; it holds nothing open, so skipping it cannot produce a false "not in use" answer. Everything else still fails the scan closed.
An mdev assignment carrying neither a UUID nor a device path would
resolve to DestroyMdev("."); release nothing instead.
ed5a990 to
345db47
Compare
Summary
Layer 2 of the vendor VFIO vGPU stack (
generalize-vgpu-device←hypervisor-liveness← this ←vendor-vfio-vgpu). Self-contained inlib/devices+lib/resources; nothing in the instance lifecycle calls it yet (that's the top layer).Linux 6.8 hosts with NVIDIA R580 drop the mdev interface: vGPUs are assigned by writing a type ID to a VF's
nvidia/current_vgpu_typeand passed to QEMU as a plain VFIO PCI device. This adds that backend behind the framework dispatch introduced in #322:creatable_vgpu_typescatalogs, least-loaded-GPU VF selection, create/verify/rollback. Profile availability counts free VFs currently advertising each type as a best-effort snapshot; creating one assignment may change sibling catalogs.Testing
go build ./...,go vetcleango test -race ./lib/devices/ ./lib/resources/passNote
Medium Risk
Changes host GPU discovery,
/resourcesadmission signals, and sysfs assignment/release paths; vendor create remains blocked but incorrect discovery or release logic could misreport capacity or clear the wrong VF.Overview
Adds NVIDIA vendor VFIO vGPU support alongside mdev: VFs are discovered via
creatable_vgpu_types/current_vgpu_type, profiles are listed from sysfs, assignments are created by writing a type ID and released by writing0, with least-loaded-GPU placement, in-process owner checks, and a strict/procVFIO open-handle scan before release.Unified vGPU stack:
DiscoverVGPU()picks mdev when usable VFs exist, otherwise vendor VFIO;DetectHostGPUMode/ publicDiscoverVFsgo away.DestroyVGPU/ReconcileVGPUsdispatch by framework;CreateVGPUstill errors on vendor VFIO until instance lifecycle integration lands. Instance cleanup passesInstanceIDonVGPUAssignmentfor vendor release ownership.Safer capacity reporting: mdev and vendor discovery skip individual unreadable VFs (warn) instead of reporting zero capacity;
/resourcesusesDiscoverVGPUwith context and avoids falling through to passthrough when vGPU discovery fails. API/docs clarify best-effort profile availability and ephemeral stop/start lifecycle.Tests & docs: large sysfs unit tests for vendor VFIO; integration test asserts framework-specific sysfs, stop/release, and start/reacquire (skips vendor VFIO hosts until create is wired).
Reviewed by Cursor Bugbot for commit 345db47. Bugbot is set up for automated code reviews on this repo. Configure here.