bug: enableUserNamespaces causes sandbox creation to fail with "setgroups: Invalid argument"

[2000krysztof]

Agent Diagnostic

Skills loaded: debug-openshell-cluster create-github-issue

Investigation findings:

• Confirmed the user namespace feature exists (enableUserNamespaces in Helm values, added in commit 1f35abbe)
• Traced configuration flow: Helm → gateway TOML → Kubernetes driver → pod spec (hostUsers: false)
• Located root cause in crates/openshell-supervisor-process/src/process.rs:1431
• Verified no fix exists in current main branch
• Checked latest release: v0.0.92 (2026-07-27) - bug present
• Searched existing issues: No duplicate found
  • Issue bug: Kubernetes user namespace E2E uses incompatible kubectl setup #1597: Different issue (E2E test setup)
  • Issue Kubernetes User Namespace Isolation for Sandbox Pods #982: Original feature request (closed, implemented in 1f35abbe)
  • Issue bug: drop_privileges_succeeds_for_current_user test fails as non-root due to initgroups requiring CAP_SETGID #622: Related initgroups issue for tests only, not runtime

Testing confirmation:

• Tested on OpenShift with enableUserNamespaces=false: ✅ SUCCESS
• Tested on OpenShift with enableUserNamespaces=true: ❌ FAILS with "setgroups: Invalid argument"

Root cause analysis:

When server.enableUserNamespaces=true, the Kubernetes driver correctly sets hostUsers: false and adds SETUID/SETGID capabilities. However, the supervisor's drop_privileges() function unconditionally calls initgroups() when dropping to a named user identity.

In Kubernetes user namespaces (hostUsers: false):

1. Container runtime creates user namespace and writes /proc/<pid>/gid_map
2. Kernel sets /proc/<pid>/setgroups to "deny" (privilege escalation prevention)
3. Once gid_map is written, /proc/self/setgroups cannot be changed to "allow"
4. Supervisor calls initgroups() → internally calls setgroups() → fails with EINVAL
5. Container exits during startup

Technical details:

• File: crates/openshell-supervisor-process/src/process.rs
• Line: 1431 - nix::unistd::initgroups(user_cstr.as_c_str(), target_gid).into_diagnostic()?;
• Call is only skipped on macOS/iOS/Haiku/Redox, not for Linux user namespaces
• No detection of /proc/self/setgroups restriction exists

Proposed fix:

Before calling initgroups(), check if /proc/self/setgroups contains "deny". If so, skip the initgroups() call. In user namespace context, supplementary groups are already appropriately isolated, so skipping is safe.

Note: Original feature commit 1f35abbe claimed validation on OCP 4.22, but current testing shows the feature is broken on OpenShift.

Description

Actual behavior: When deploying OpenShell with --set server.enableUserNamespaces=true, all sandbox creation attempts fail during container startup with Error: container create failed: setgroups: Invalid argument.

Expected behavior: Sandboxes should start successfully with user namespace isolation enabled, providing defense-in-depth by mapping container UID 0 to an unprivileged host UID.

Reproduction Steps

1. Deploy with user namespaces enabled:

   helm install openshell deploy/helm/openshell \
     --namespace openshell \
     --create-namespace \
     --set server.enableUserNamespaces=true \
     --wait

2. Attempt to create a sandbox:

   openshell sandbox create --name test

3. Observe failure:

   Created sandbox: test
   ✓ Sandbox allocated (3s)
   ✓ Image pulled (8s)
   ⠄ Starting sandbox... Error: container create failed: setgroups: Invalid argument (5s)
   

Environment

• OS: OpenShift / Kubernetes with user namespace support (1.33+)
• Container runtime: containerd 2.0+ or cri-o 1.25+
• OpenShell: v0.0.92 (latest release)
• Latest release checked: yes (v0.0.92)
• Possible duplicates checked: yes (issues bug: Kubernetes user namespace E2E uses incompatible kubectl setup #1597, Kubernetes User Namespace Isolation for Sandbox Pods #982, bug: drop_privileges_succeeds_for_current_user test fails as non-root due to initgroups requiring CAP_SETGID #622 reviewed - none match)
• Tested platforms: OpenShift (confirmed broken with flag enabled, works with flag disabled)

Logs

Expected container logs when inspecting the failed sandbox pod:

Error: setgroups: Invalid argument

Kernel behavior:

# Inside a user namespace pod (hostUsers: false)
$ cat /proc/self/setgroups
deny

# Attempting setgroups() in this state returns EINVAL

[matthewgrossman]

📋 triage-agent

Triage Assessment

Classification: bug-confirmed

Summary

Kubernetes user-namespace pods deny setgroups, but the supervisor’s privilege-drop path still calls initgroups/setgroups, preventing sandbox startup.

Investigation

The behavior remains in v0.0.92 and current origin/main. The suggested unconditional skip is insufficient: inherited supplementary groups must not silently survive privilege dropping.

Recommendation

Handle /proc/self/setgroups=deny explicitly: continue only after verifying supplementary groups are already empty or exactly safe; otherwise fail closed. Add a Kubernetes user-namespace E2E regression test.