You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Investigated a live Kubernetes deployment with OpenShell gateway and supervisor v0.0.92.
Confirmed SPIRE server, agent, CSI driver, and OIDC provider were healthy before reproducing.
Confirmed the sandbox Pod received the read-only csi.spiffe.io volume and OPENSHELL_PROVIDER_SPIFFE_WORKLOAD_API_SOCKET=/spiffe-workload-api/spire-agent.sock.
Confirmed SPIRE created an entry bound to the sandbox Pod UID with the expected per-sandbox SPIFFE ID and five-minute JWT-SVID TTL.
Traced the failure to crates/openshell-supervisor-process/src/process.rs: prepare_supervisor_identity_mount_namespace_from_env() is defined, while supervisor_identity_mount_from_env() requires its OnceLock, but no caller exists.
Checked the latest published release, v0.0.92. The missing call is also still absent from the v0.0.95 source tag and current main at the time of filing.
The upstream OpenShell agent skills were not exposed in the current harness; equivalent live-cluster, source, release, and duplicate diagnostics were performed manually.
Description
When provider_spiffe_workload_api_socket_path is configured for the Kubernetes driver, every sandbox supervisor crashes during startup with:
Failed to prepare supervisor identity isolation: supervisor identity mount namespace was not prepared before startup hardening
Expected behavior: the supervisor prepares the private mount namespace while privileged, retains access to the SPIFFE Workload API socket, hides that socket from sandbox child processes, and starts normally.
Actual behavior: the initializer is never called. The sandbox Pod enters a restart loop before a command can run, making SPIFFE-backed dynamic token grants unusable.
Configure a ClusterSPIFFEID matching gateway-managed sandbox Pods.
Create a sandbox using combined topology and the upstream v0.0.92 supervisor.
Observe that:
the Pod is scheduled and both images pull successfully;
the CSI Workload API volume is mounted read-only;
SPIRE creates the Pod-UID-bound registration entry;
the supervisor container exits with code 1 and the error above.
Inspect the tagged source and observe that prepare_supervisor_identity_mount_namespace_from_env() has no caller.
The sandbox container had CAP_SYS_ADMIN, AppArmor: Unconfined, and valid node placement, so this is not the EPERM path addressed by #2184.
Environment
OpenShell gateway: 0.0.92 with a downstream gateway-only patch; supervisor is the unmodified upstream ghcr.io/nvidia/openshell/supervisor:0.0.92
Local CLI used to initiate the API request: 0.0.69
Kubernetes API: EKS v1.36.2
Node: Bottlerocket 1.62.1, Linux arm64, kubelet v1.36.0
SPIRE chart: 0.29.0
SPIFFE CSI driver: 0.2.7
Latest published OpenShell release checked: v0.0.92
Newer source tag checked: v0.0.95
Existing issues and PRs checked: yes
Logs
Error: Failed to prepare supervisor identity isolation: supervisor identity mount namespace was not prepared before startup hardening
SPIRE successfully registered the same sandbox immediately before the supervisor failure:
Created entry ... spiffeID=spiffe://<trust-domain>/openshell/sandbox/<sandbox-uuid> ... selectors=[k8s:pod-uid:<pod-uid>] ... jwtSVIDTTL=5m0s
No credentials or token values were present in the sandbox specification or collected logs.
Likely Fix
PR #2012 proposes the minimal fix: call prepare_supervisor_identity_mount_namespace_from_env() in run_process() after privileged filesystem preparation and before startup hardening or child process creation. The fix should include a regression test proving both successful initialization and socket hiding from child processes.
Agent-First Checklist
I pointed my agent at the repository and had it investigate this issue
I loaded relevant upstream skills; they were unavailable in this harness, so equivalent diagnostics are documented above
I checked the latest published OpenShell release and newer source tags
I searched existing issues and pull requests for possible duplicates
The issue could not be resolved without patching the upstream supervisor startup sequence
Agent Diagnostic
csi.spiffe.iovolume andOPENSHELL_PROVIDER_SPIFFE_WORKLOAD_API_SOCKET=/spiffe-workload-api/spire-agent.sock.crates/openshell-supervisor-process/src/process.rs:prepare_supervisor_identity_mount_namespace_from_env()is defined, whilesupervisor_identity_mount_from_env()requires itsOnceLock, but no caller exists.mainat the time of filing.setns()failure, not the missing initializer call.Description
When
provider_spiffe_workload_api_socket_pathis configured for the Kubernetes driver, every sandbox supervisor crashes during startup with:Expected behavior: the supervisor prepares the private mount namespace while privileged, retains access to the SPIFFE Workload API socket, hides that socket from sandbox child processes, and starts normally.
Actual behavior: the initializer is never called. The sandbox Pod enters a restart loop before a command can run, making SPIFFE-backed dynamic token grants unusable.
Reproduction Steps
ClusterSPIFFEIDmatching gateway-managed sandbox Pods.prepare_supervisor_identity_mount_namespace_from_env()has no caller.The sandbox container had
CAP_SYS_ADMIN,AppArmor: Unconfined, and valid node placement, so this is not theEPERMpath addressed by #2184.Environment
0.0.92with a downstream gateway-only patch; supervisor is the unmodified upstreamghcr.io/nvidia/openshell/supervisor:0.0.920.0.69v1.36.21.62.1, Linux arm64, kubeletv1.36.00.29.00.2.7v0.0.92v0.0.95Logs
SPIRE successfully registered the same sandbox immediately before the supervisor failure:
No credentials or token values were present in the sandbox specification or collected logs.
Likely Fix
PR #2012 proposes the minimal fix: call
prepare_supervisor_identity_mount_namespace_from_env()inrun_process()after privileged filesystem preparation and before startup hardening or child process creation. The fix should include a regression test proving both successful initialization and socket hiding from child processes.Agent-First Checklist