Skip to content

bug(supervisor): SPIFFE-enabled sandboxes crash because identity mount namespace is never prepared #2571

Description

@ewhauser

Agent Diagnostic

  • Investigated a live Kubernetes deployment with OpenShell gateway and supervisor v0.0.92.
  • Confirmed SPIRE server, agent, CSI driver, and OIDC provider were healthy before reproducing.
  • Confirmed the sandbox Pod received the read-only csi.spiffe.io volume and OPENSHELL_PROVIDER_SPIFFE_WORKLOAD_API_SOCKET=/spiffe-workload-api/spire-agent.sock.
  • Confirmed SPIRE created an entry bound to the sandbox Pod UID with the expected per-sandbox SPIFFE ID and five-minute JWT-SVID TTL.
  • Traced the failure to crates/openshell-supervisor-process/src/process.rs: prepare_supervisor_identity_mount_namespace_from_env() is defined, while supervisor_identity_mount_from_env() requires its OnceLock, but no caller exists.
  • Checked the latest published release, v0.0.92. The missing call is also still absent from the v0.0.95 source tag and current main at the time of filing.
  • Searched open and closed issues and pull requests. No open issue matches the exact failure. PR fix(supervisor): wire prepare_supervisor_identity_mount_namespace_from_env #2012 contains the exact diagnosis and fix but was automatically closed by the first-time-contributor vouch/DCO workflow before technical review. PR fix(sandbox): undo tmpfs overlay when setns fails during SPIFFE mount namespace setup #2184 is related but addresses cleanup after setns() failure, not the missing initializer call.
  • The upstream OpenShell agent skills were not exposed in the current harness; equivalent live-cluster, source, release, and duplicate diagnostics were performed manually.

Description

When provider_spiffe_workload_api_socket_path is configured for the Kubernetes driver, every sandbox supervisor crashes during startup with:

Failed to prepare supervisor identity isolation: supervisor identity mount namespace was not prepared before startup hardening

Expected behavior: the supervisor prepares the private mount namespace while privileged, retains access to the SPIFFE Workload API socket, hides that socket from sandbox child processes, and starts normally.

Actual behavior: the initializer is never called. The sandbox Pod enters a restart loop before a command can run, making SPIFFE-backed dynamic token grants unusable.

Reproduction Steps

  1. Deploy SPIRE and the SPIFFE CSI driver.
  2. Configure the Kubernetes driver with:
provider_spiffe_workload_api_socket_path = "/spiffe-workload-api/spire-agent.sock"
  1. Configure a ClusterSPIFFEID matching gateway-managed sandbox Pods.
  2. Create a sandbox using combined topology and the upstream v0.0.92 supervisor.
  3. Observe that:
    • the Pod is scheduled and both images pull successfully;
    • the CSI Workload API volume is mounted read-only;
    • SPIRE creates the Pod-UID-bound registration entry;
    • the supervisor container exits with code 1 and the error above.
  4. Inspect the tagged source and observe that prepare_supervisor_identity_mount_namespace_from_env() has no caller.

The sandbox container had CAP_SYS_ADMIN, AppArmor: Unconfined, and valid node placement, so this is not the EPERM path addressed by #2184.

Environment

  • OpenShell gateway: 0.0.92 with a downstream gateway-only patch; supervisor is the unmodified upstream ghcr.io/nvidia/openshell/supervisor:0.0.92
  • Local CLI used to initiate the API request: 0.0.69
  • Kubernetes API: EKS v1.36.2
  • Node: Bottlerocket 1.62.1, Linux arm64, kubelet v1.36.0
  • SPIRE chart: 0.29.0
  • SPIFFE CSI driver: 0.2.7
  • Latest published OpenShell release checked: v0.0.92
  • Newer source tag checked: v0.0.95
  • Existing issues and PRs checked: yes

Logs

Error: Failed to prepare supervisor identity isolation: supervisor identity mount namespace was not prepared before startup hardening

SPIRE successfully registered the same sandbox immediately before the supervisor failure:

Created entry ... spiffeID=spiffe://<trust-domain>/openshell/sandbox/<sandbox-uuid> ... selectors=[k8s:pod-uid:<pod-uid>] ... jwtSVIDTTL=5m0s

No credentials or token values were present in the sandbox specification or collected logs.

Likely Fix

PR #2012 proposes the minimal fix: call prepare_supervisor_identity_mount_namespace_from_env() in run_process() after privileged filesystem preparation and before startup hardening or child process creation. The fix should include a regression test proving both successful initialization and socket hiding from child processes.

Agent-First Checklist

  • I pointed my agent at the repository and had it investigate this issue
  • I loaded relevant upstream skills; they were unavailable in this harness, so equivalent diagnostics are documented above
  • I checked the latest published OpenShell release and newer source tags
  • I searched existing issues and pull requests for possible duplicates
  • The issue could not be resolved without patching the upstream supervisor startup sequence

Metadata

Metadata

Assignees

No one assigned

    Labels

    state:triage-neededOpened without agent diagnostics and needs triage

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions