uffd: add support for UFFD_EVENT_REMOVE events

[bchalios]

Enabling Firecracker free-page-reporting feature requires us to handle remove events (UFFD_EVENT_REMOVE) in our userfaultfd handler. These events are triggered whenever Firecracker calls madvise(MADV_DONTNEED) (or similar) on a range of guest memory addresses.

The main thing that changes on our logic is that page faults in a page that has previously been removed need to be served with a zero page rather than a page from the snapshot file.

This commit changes the page fault serving logic to:

1. Introduce tracking of the state of every page in the guest's memory mappings.
2. Add logic to handle the new UFFD_EVENT_REMOVE event
3. Modify existing logic to take into account current state when deciding how to handle each page fault

This is dependent on the part of #1858 that enables free page reporting on the Firecracker side.

---

Note

High Risk
High risk because it changes low-level userfaultfd event handling and page population semantics (including new ioctls and concurrency/defer logic), which can affect VM stability and memory correctness. It also introduces a new template/sandbox configuration flag that changes runtime behavior based on Firecracker version and feature flags.

Overview
This PR upgrades the default Firecracker version to v1.14.1 and introduces opt-in free page reporting, wiring a new freePageReporting setting from template creation through template build/sandbox startup to Firecracker balloon configuration. To support this, the userfaultfd handler is reworked to read and process UFFD_EVENT_REMOVE events, track per-page state, zero-fill pages after removal, and defer/retry faults that race with removes, with tests expanded to validate remove, write-protect, and gated ordering scenarios.

^{Written by Cursor Bugbot for commit b35c9f7. This will update automatically on new commits. Configure here.}

[cursor]

Prefault can overwrite removed state causing thread hangs

Medium Severity

Prefault runs concurrently with the Serve loop but is not part of the wg errgroup. The Serve loop calls wg.Wait() before processing REMOVE events, but this does not wait for in-flight Prefault calls. A race exists: Prefault reads state as unfaulted, then Serve processes a REMOVE (setting state to removed), then Prefault's faultPage succeeds and setState(faulted) overwrites the removed state. Subsequent page faults on that address see faulted, skip handling, and the faulting thread hangs indefinitely.

Additional Locations (1)

• packages/orchestrator/pkg/sandbox/uffd/userfaultfd/userfaultfd.go#L260-L272

 

[bchalios]

This is actually a valid point for concurrent removes. I think in practice pre-faulting finishes way before we start receiving on-demand events, but I think the safest thing to do here would be to only resume vCPUs after pre-faulting has completed.

[bchalios]

WDYT @ValentaTomas ?

[bchalios]

Ok,I think I've found another work-around. We can reuse the settleRequests RW mutex. If we take it the write-lock when we 're handling remove events, I think we're safe. Both page fault handling and prefaulter Go routines take the read lock. So we can never have page/pre-faulting racing with remove event handling.

[cursor]

Partial zero+WP+wake failure loses write-protection permanently

Medium Severity

In the 4K read zero-fill path, if zero() succeeds but writeProtect() fails, the page is populated but not write-protected. The writeErr propagates to the EAGAIN/error checks. On retry via deferred, zero() returns EEXIST, and faultPage returns (true, nil), so setState(faulted) runs without the WP bit ever being set. This silently breaks dirty page tracking for that page.

 

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

There are 3 total unresolved issues (including 2 from previous reviews).

^{Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

[cursor]

Version checks will break for major version > 1

Low Severity

Both HasHugePages and HasFreePageReporting use Major() == 1 which will return false for any future Firecracker version with major > 1 (e.g., v2.0). The correct check for "version >= X.Y" is Major() > X || (Major() == X && Minor() >= Y). While Firecracker is currently at v1.14, this pattern is easy to copy-paste incorrectly for future version gates.

Additional Locations (1)

• packages/api/internal/sandbox/sandbox_features.go#L48-L54