test(podman): close CI coverage gaps in Podman compute driver testing

[politerealism]

User Story

As an OpenShell maintainer relying on the Podman compute driver, I want CI to actually exercise the driver's core guarantees (resource limits, failure handling, network policy enforcement, rootful parity, GPU support) against a real Podman daemon, so that regressions in these areas are caught before merge instead of relying on unit tests against mocked clients.

Problem Statement

An audit of Podman CI coverage found that almost all of it either runs against mocked/fake Podman clients (unit tests) or exercises only generic driver-agnostic conformance scenarios plus a single userns-mapping check. Several behaviors the driver is expected to guarantee as a ComputeDriver backend are not verified against a real Podman daemon in CI at all — and in one case, a real test already exists but is silently skipped.

Impact / Why This Matters

Today, a regression in resource-limit enforcement, daemon-failure handling, or rootful-specific behavior could land without any CI job catching it, because the only real-daemon coverage is default_userns.rs plus the shared conformance suite. This is inconsistent with Kubernetes, which recently gained dedicated HA and credential-driver e2e suites (#3626) — Podman has no equivalent depth of driver-specific real-daemon testing. Podman GPU support additionally has no CI wiring at all, so it can silently break with no signal until a user hits it manually. Separately, an existing Podman network-policy test has apparently never been exercised in CI due to a wiring bug, so it may already be bit-rotted.

Findings

1. IMPLEMENTED. Rootful userns coverage was missing, and closing it needed new playbook logic, not a one-line CI matrix change. tests/suites/drivers/podman/tests/default_userns.rs (a single test, configured_userns_matches_podman_reference) checks /proc/self/uid_map against a reference captured by shelling out to podman directly. The driver-specific-integration job in .github/workflows/branch-e2e.yml (lines 250-264) runs the driver-podman testsuite only on fedora-podman-rootless — unlike conformance-integration (lines 213-229), which runs both rootful and rootless. Reading /proc/self/uid_map happens from inside the sandboxed process itself, not a host-side cross-domain /proc/<pid>/exe read, so this specific test carries no SELinux/AppArmor concern if widened.

   Correction to an earlier pass of this issue: tests/suites/drivers/podman/fixtures/userns-auto.toml, userns-keep-id.toml, and userns-private.toml are not orphaned. They're wired up through the Nix-based tmachine test harness (tests/config.nix:130-150), which drives four sequential Ansible-playbook rounds for the driver-podman testsuite: default-userns-baseline.yaml (default), then userns-auto.yaml, userns-keep-id.yaml, and userns-private.yaml (each importing the shared userns-profile.yaml, tests/ansible/playbooks/drivers/podman/), each rewriting /etc/openshell/gateway.toml, restarting the gateway, capturing a fresh direct-Podman reference, and rerunning default_userns.rs against it. So multi-userns-mode coverage (auto/keep-id/private) already exists in CI today — just rootless-only. This was missed by an earlier grep-based pass because the wiring lives in tests/config.nix/tests/ansible/, not in .rs/.yml files a naive search would associate with the fixtures.

   The real blocker to rootful coverage is that default-userns-baseline.yaml and userns-profile.yaml both hard-assert the gateway's systemd service user is tmachine (rootless) and fail otherwise ("Require the rootless Podman gateway user"), and their reference-capture tasks use rootless-specific execution (become_user: tmachine, HOME=/home/tmachine, XDG_RUNTIME_DIR=/run/user/<uid>). Pointing the existing testsuite at fedora-podman-rootful would just fail that assertion immediately, not exercise anything.

   This is not blocked by Podman itself: verified directly against the installed Podman 5.8.4 podman-create(1) man page, --userns=auto and --userns=keep-id are both valid under rootful mode (keep-id under rootful maps root's own UID:GID into a fresh namespace instead of erroring; auto under rootful just requires /etc/subuid//etc/subgid entries for the containers user instead of the calling user's). Only --userns=nomap is documented as rootless-only, and none of the three fixtures use it. So a rootful equivalent of all four scenarios (default/auto/keep-id/private) is achievable; it requires a rootful branch in the playbooks (different service-user assertion, direct root capture instead of become_user/XDG_RUNTIME_DIR), not new Podman-side capability.

   Fix: default-userns-baseline.yaml, userns-profile.yaml, and tests.yaml (failure diagnostics) now detect rootful vs. rootless via the existing tmachine_container_runtime role (the same role openshell_gateway's own install logic already uses to pick the gateway's service user) instead of hard-asserting tmachine. branch-e2e.yml's driver-specific-integration matrix now runs driver-podman on both fedora-podman-rootful and fedora-podman-rootless — no new tests/config.nix testsuite entry was needed since the same playbooks adapt to either environment. One real bug was caught and fixed during implementation: the captured reference file must stay owned by tmachine regardless of rootful/rootless, because the archived test binary that reads it back always runs unprivileged as tmachine (the inventory's ansible_user) — only the podman command used to produce the reference needs to match the daemon's mode. Ansible syntax-validated (ansible-playbook --syntax-check); not run end-to-end against a live rootful VM (no such environment available outside CI).

   Scope note confirmed during this work: rootful is a real, first-class supported configuration of the compute driver, not a hypothetical one — crates/openshell-driver-podman/src/driver.rs:447-477 auto-detects rootless/rootful from Podman's own system info and never errors on rootful, and container.rs has dedicated, unit-tested isolation-spec logic for the rootful case (workload.user == "0:0"). This contradicts docs/reference/sandbox-compute-drivers.mdx:275 and docs/sandboxes/manage-gateways.mdx:30, which describe the Podman driver as rootless-only ("avoid a rootful Docker daemon") with no mention of rootful support — that's a separate documentation gap, not fixed here.

2. IMPLEMENTED. No integration test exercised real resource-limit enforcement (CPU/memory/cgroup) against a live Podman daemon. admit_container_resources (crates/openshell-driver-podman/src/driver.rs:712) and reconcile_resource_admission (crates/openshell-driver-podman/src/driver.rs:822) are only covered by unit tests (crates/openshell-driver-podman/src/driver.rs:1974 onward) that drive a hyper-based Unix-socket stub (crates/openshell-driver-podman/src/test_utils.rs:70, spawn_podman_stub) returning canned JSON — not a real daemon.

   Narrower than it first looked: e2e/rust/tests/sandbox_templates.rs (#![cfg(feature = "e2e")], driver-agnostic, real gateway) already exercises --cpu 500m --memory 512Mi template creation, but only asserts the API/template JSON echoes the requested values back (template_json["resources"]["cpu"], ["resources"]["memory"]) — it never inspects the resulting container's actual cgroup limits. So the gap was specifically "nothing verifies the declared limit is enforced at runtime," not "nothing touches resources at all."

   Fix: new e2e/rust/tests/podman_resource_limits.rs creates a sandbox with --cpu 500m --memory 512Mi and reads /sys/fs/cgroup/memory.max/cpu.max from inside the sandbox itself — verifying the actual enforcement boundary the workload experiences, not just driver-reported metadata. Expected values (536870912 bytes; 50000 100000 quota/period) were cross-checked two ways: against crates/openshell-driver-podman/src/container.rs's own parse_cpu_to_microseconds/parse_memory_to_bytes (and their existing unit test container_spec_applies_cpu_and_memory_limits), and empirically against a real local podman run --cpus=0.5 --memory=512m container's actual cpu.max/memory.max output. Compiles cleanly (cargo check --features e2e-podman) and cargo fmt clean; not run end-to-end against a live gateway+sandbox (would require standing up the full Podman e2e harness, image builds included).

3. PARTIALLY IMPLEMENTED (daemon-unavailable sub-case only). No negative-path integration tests against a real daemon existed — daemon-unavailable, container-creation failure, OOM-kill, and exit-code propagation via watcher.rs's WatchStream (crates/openshell-driver-podman/src/watcher.rs:551-572) were only unit-tested with synthetic structs (crates/openshell-driver-podman/src/watcher.rs:608-988), not a real daemon.

   Correction from an earlier pass: e2e/rust/tests/docker_preflight.rs looked like a reusable pattern ("mirror it as podman_preflight.rs"), but that test exercises openshell doctor check — a CLI diagnostic subcommand that is hardcoded Docker-only (crates/openshell-cli/src/run.rs:215-251 unconditionally shells out to docker info, no driver detection at all). There is no Podman-equivalent CLI behavior to test, so mirroring it directly wasn't applicable. The doctor check Docker-only limitation is a separate, small product gap this issue does not address.

   Fix: instead, new e2e/rust/tests/podman_preflight.rs tests the actual driver-level failure path this finding is about — it spawns the standalone openshell-driver-podman binary directly (no gateway/sandbox needed) pointed at a guaranteed-nonexistent Podman socket, and asserts it exits non-zero within its bounded retry window (~10s, from the 5-retry/2s-delay loop in driver.rs:396-397) with an error naming the unreachable socket path (connection error: <path>: ..., from PodmanApiError::Connection in client.rs:38). This one was actually built and run end-to-end (not just compiled) — cargo build -p openshell-driver-podman + cargo test --features e2e-podman --test podman_preflight, both assertions pass in ~10s, confirming the exact error format against the real binary.

   OOM-kill and exit-code propagation still have zero precedent for any driver and need new test infrastructure — not addressed here.

   LSM note for implementation: if new tests verify container process identity or exit state, prefer Podman's own API/inspect output over host-side /proc/<pid>/exe or /proc/<pid>/fd reads for the container's PID — once the container transitions into container_t, that read returns ENOENT (not EACCES) on SELinux-enforcing hosts, which can produce misleading negative-path test failures.

4. Corrected finding, now largely in flight via fix(podman): restore host gateway alias mediation #3606 — a Podman-specific network-policy/L7 enforcement test already exists, but nothing runs it in CI today. e2e/rust/tests/podman_corporate_proxy.rs (898 lines, #![cfg(feature = "e2e-podman")]) implements TLS proxy interception and allow/deny egress assertions against a real Podman-backed gateway. Three driver-agnostic L7 tests (forward_proxy_l7_bypass.rs, forward_proxy_jsonrpc_l7.rs, forward_proxy_graphql_l7.rs) are also reachable under the same feature set, since e2e-podman = ["e2e", "e2e-host-gateway", "e2e-local-container-driver"] (e2e/rust/Cargo.toml:36).

   On main today, none of these execute in CI. The only Podman e2e job wired into branch-e2e.yml, podman-external-driver-e2e (lines 307-321), runs mise run e2e:podman:external-driver, which sets OPENSHELL_E2E_PODMAN_FEATURES = "" (tasks/test.toml:253-257). In e2e/rust/e2e-podman.sh:13,25-27, E2E_FEATURES="${OPENSHELL_E2E_PODMAN_FEATURES-e2e-podman}" only falls back to the e2e-podman default when the variable is unset — an explicit empty string bypasses that default, so E2E_FEATURES ends up empty and the script only runs the openshell-conformance CLI scenarios before exiting; the entire cargo test invocation (and therefore podman_corporate_proxy.rs and the other e2e-podman-gated tests) is skipped.

   Unlike Podman, Docker has both a bundled-driver job (docker-e2e, branch-e2e.yml:266-277, running the full e2e-docker feature set including the equivalent forward-proxy L7 tests) and an external-driver conformance-only job (docker-external-driver-e2e, lines 290-306). Podman only has the latter today — there is no bundled-driver podman-e2e job.

   This is already being addressed by open PR fix(podman): restore host gateway alias mediation #3606 ("fix(podman): restore host gateway alias mediation", fixes Podman sandboxes cannot reach host.openshell.internal #3605), as a side effect of its host-gateway fix rather than a dedicated CI-coverage change. That PR adds a podman-e2e job to branch-e2e.yml (mirroring docker-e2e) plus a new OPENSHELL_E2E_PODMAN_TEST_SET=ci mechanism (e2e:podman:ci task) that runs a curated list of Podman e2e tests including podman_corporate_proxy, forward_proxy_l7_bypass, forward_proxy_jsonrpc_l7, forward_proxy_graphql_l7, host_gateway_alias, and podman_host_gateway. It does not touch podman-external-driver-e2e's empty-features override, which now reads as intentional rather than an oversight — it mirrors the Docker split between a bundled-driver behavioral job and an external-driver wiring-only job.

   However, the new podman-e2e job is rootless-only, same as podman-external-driver-e2e: both use .github/actions/setup-e2e-podman, which explicitly asserts Host.Security.Rootless == true and configures pasta networking. So even once fix(podman): restore host gateway alias mediation #3606 merges, none of the newly-running L7/host-gateway tests exercise rootful Podman — finding 1's rootful/rootless gap extends to this new coverage too.

   Whoever picks up this issue should wait for fix(podman): restore host gateway alias mediation #3606 to merge (or coordinate with it) and then verify the curated e2e:podman:ci list still passes, rather than duplicating the CI-wiring work it already does. Remaining work after fix(podman): restore host gateway alias mediation #3606 lands: confirm podman_corporate_proxy.rs and the other now-enabled tests actually pass against current code (they may have bit-rotted if never run against a live daemon before), and decide whether rootful coverage for these specific tests is needed.

5. Podman GPU e2e has zero CI wiring. The mise task e2e:podman:gpu (tasks/test.toml:178-182, sets OPENSHELL_E2E_PODMAN_GPU=1 / OPENSHELL_E2E_PODMAN_FEATURES=e2e-podman-gpu, runs e2e/rust/e2e-podman.sh) is not referenced by any .github/workflows/*.yml/*.yaml file — no PR gate, label trigger, schedule, or manual dispatch invokes it. The only GPU e2e workflow actually wired into CI, .github/workflows/e2e-gpu-test.yaml (invoked by branch-e2e.yml's gpu-e2e job, gated on the test:e2e-gpu label), is Docker-only (CONTAINER_ENGINE: docker at line 57, mise run --no-deps --skip-deps e2e:docker:gpu at line 87).

   Cost context for the open question below: e2e-gpu-test.yaml already runs on NVIDIA self-hosted GPU runners (linux-arm64-gpu-l4-latest-1, linux-amd64-gpu-rtxpro6000-latest-1, wsl-amd64-gpu-rtxpro6000-latest-1), gated behind the opt-in test:e2e-gpu label rather than running on every PR. Adding a Podman leg would reuse this same shared runner pool and only add GPU-minutes on PRs that already opt into GPU testing — it would not require provisioning new hardware. This lowers the bar for a "yes" on wiring it in, though the actual go/no-go is still a maintainer call.

   DELIBERATELY NOT ADDRESSED. The wiring path is understood and is low effort in isolation (extend e2e-gpu-test.yaml with a Podman leg analogous to the Docker one, running mise run e2e:podman:gpu), but unlike the other findings it can't be validated without real GPU hardware — there's no way to confirm Podman's CDI-based GPU injection actually works end-to-end through the driver without running it on a GPU-equipped runner, so writing this blind carries real risk of landing broken, unvalidated CI. It also adds a permanent ongoing CI cost (another GPU job on every PR that applies test:e2e-gpu, forever) that's a genuine judgment call, not a mechanical fix. Leaving this as an open, unimplemented finding for a maintainer to decide whether the coverage is worth it — see Acceptance Criteria.

Proposed Design

For each finding:

• (1) Add rootful variants of the default-userns-baseline.yaml/userns-profile.yaml playbooks (branching on gateway service user instead of hard-asserting tmachine, and using a direct-root reference capture instead of become_user/XDG_RUNTIME_DIR), a corresponding driver-podman-rootful (or similar) testsuite entry in tests/config.nix, and a fedora-podman-rootful leg in the driver-specific-integration matrix — or explicitly document why rootful is intentionally out of scope for this suite.
• (2) Extend e2e/rust/tests/sandbox_templates.rs (or a driver-agnostic conformance scenario built on the same pattern) with an assertion step that inspects the created container's actual cgroup/resource state via the driver's own API, not just the template JSON response.
• (3) Add podman_preflight.rs mirroring docker_preflight.rs for the daemon-unavailable case (cheap, existing pattern to copy). OOM-kill and exit-code propagation need new test infrastructure — assess whether these are better added as new driver-agnostic conformance scenarios (crates/openshell-conformance/src/scenarios/, currently just smoke.rs and sandbox_lifecycle.rs), reusable across Docker/Podman/Kubernetes, versus Podman-specific tests in e2e/rust/tests/ (not tests/suites/drivers/podman/, which uses a different, conformance-runner-only harness). These are ComputeDriver-contract guarantees Docker and Kubernetes also need to prove, so the conformance-scenario route is likely the better fit; only Podman-specific error shapes (e.g., PodmanApiError::NotFound) would need a driver-specific test.
• (4) Superseded by pending PR fix(podman): restore host gateway alias mediation #3606, which adds the bundled-driver podman-e2e job and e2e:podman:ci task this finding calls for. Once fix(podman): restore host gateway alias mediation #3606 merges, verify the curated test list actually passes rather than re-doing the wiring fix. Rootful coverage for these tests remains an open gap either way (see finding 1).
• (5) Wire e2e:podman:gpu into a CI workflow analogous to e2e-gpu-test.yaml, reusing the existing shared GPU runner pool and the test:e2e-gpu label (or a Podman-specific label) — this is additive runner-minutes on already opt-in PRs, not new infrastructure, so the main remaining question is whether a maintainer wants the coverage at all.

Note on precedent: the Kubernetes HA/credential-driver suites referenced above are not in a tests/suites/drivers/kubernetes/ directory (no such directory exists) — they live in e2e/rust/tests/kubernetes_ha_rebalancing.rs and e2e/rust/tests/credential_drivers.rs, wired via dedicated label-gated jobs in .github/workflows/branch-e2e.yml:416-447. That's the structural pattern to follow for any new Podman-specific e2e test, not the tests/suites/drivers/podman/ conformance-runner harness.

Acceptance Criteria

• The driver-podman testsuite (or a rootful-specific variant) runs against both rootful and rootless Podman, or the rootless-only scope is explicitly documented as intentional. (Implemented — pending real CI validation once merged/run.)
• At least one CI job exercises resource-limit enforcement (CPU/memory/cgroup) against a real Podman daemon. (New test written and math-verified; not yet run end-to-end against a live gateway.)
• At least one CI job exercises negative paths against a real Podman daemon — daemon-unavailable only, built and run for real. Container-creation failure, OOM-kill, and exit-code propagation remain unaddressed and need new test infrastructure.
• podman_corporate_proxy.rs and the other e2e-podman-gated L7 tests actually execute in some CI job, and pass (or are fixed to pass) against current code. (Tracked to land via fix(podman): restore host gateway alias mediation #3606; verify after merge rather than duplicating.)
• e2e:podman:gpu either runs in some CI workflow (label-gated or scheduled), or a decision to not run it in CI is explicitly recorded here. Deliberately left unaddressed — path is understood, but wiring it blind without GPU hardware to validate against is unacceptable risk, and it carries an ongoing CI-cost tradeoff a maintainer should decide on, not an agent.

Alternatives Considered

Leaving coverage as-is and relying on the driver-agnostic conformance suite plus unit tests. Rejected because the conformance suite doesn't target Podman-specific failure/resource/network paths, and unit tests against mocked clients don't catch real-daemon behavior changes (e.g., cgroup delegation differences, pasta networking changes, Podman version upgrades).

Notes

This issue documents CI coverage gaps identified by an ad hoc audit, then verified/corrected via a deeper follow-up investigation (not a full create-spike investigation) for tracking and follow-up. I intend to address this myself.

Implementation status (2026-09-24): findings 1–3 implemented on branch 3663-podman-driver-ci-gaps, mise run pre-commit clean. Finding 4 intentionally left to PR #3606. Finding 5 intentionally left unimplemented pending a maintainer decision on GPU CI cost. PR forthcoming.

Cross-reference: PR #3606 ("fix(podman): restore host gateway alias mediation", fixes #3605, currently open/gator:blocked) substantially overlaps with finding 4 — it adds the missing bundled-driver podman-e2e job and wires up podman_corporate_proxy.rs plus the forward-proxy L7 tests, as a side effect of its host-gateway fix. That also answers the open question below: the answer appears to be "intentional," matching the Docker bundled/external split. Coordinate with #3606 rather than duplicating its CI-wiring changes; it does not address rootful coverage (finding 1) for the tests it enables.

[grs]

#3606 includes enablement of more of the e2e tests for podman (had hoped to get them all passing but ran out of time last night)

[politerealism]

@grs PR #3606 fixes host gateway networking (host.openshell.internal resolution) for Podman sandboxes. The new CI job it adds (podman-e2e) only tests this under rootless Podman and it never runs against rootful Podman. I want to know if that is a deliberate scoping decision (rootful isn't a target for this fix, or is covered some other way), or is it just inheriting the existing rootless only test setup without anyone having thought about rootful specifically?

[grs]

@politerealism Sorry, I missed that. I didn't make a conscious choice, just went with what was already there.

[politerealism]

No worries! Just planning out and researching more so was curious. Thanks :D