bug(boundary): CA bundle JSON serialization inflates control frame past 1MB limit on macOS

[benoitf]

User Story

As a developer running the VM compute driver on macOS,
I want sandbox creation to succeed without hitting boundary protocol frame size limits,
so that I can use VM-backed sandboxes on machines with standard macOS system CA bundles.

Problem Statement

The StartAgent boundary control request fails with control frame is too large when the host system CA bundle exceeds ~300KB. The ca_bundle field in Request::StartAgent is typed as Option<Vec<u8>> and serialized via serde's default JSON encoding, which represents each byte as a decimal number in a JSON array ([45, 45, 45, 10, ...]). This inflates the payload ~3.3× compared to the raw bytes. A typical macOS system CA bundle is
~334KB, which serializes to ~1.1MB of JSON — exceeding the MAX_CONTROL_FRAME_BYTES limit of 1MB (1,048,576 bytes).

The supervisor attaches and confirms the isolation boundary successfully, but crashes when encoding the StartAgent request that carries the CA bundle to the guest.

Impact / Why This Matters

When this happens, every openshell sandbox create on the affected machine fails after completing image pull and boundary attachment. The supervisor exits with status 1 and the sandbox enters an error phase. Users see:

Error: process error: encode control request: control frame is too large: 1099704 bytes

There is no workaround short of patching the frame size limit or trimming the system CA store. This blocks all VM-backed sandbox usage on macOS systems with default certificate stores. The issue is latent on Linux as well / any system with a CA bundle above ~300KB will hit it.

Acceptance Criteria

• openshell sandbox create succeeds on macOS with the default system CA bundle (300–400KB on my computer)
• The ca_bundle and ca_cert fields in Request::StartAgent use a space-efficient serialization (e.g., base64) instead of JSON byte arrays
• Existing supervisor and sandbox binaries using the old serialization format are handled gracefully (protocol versioning or backwards-compatible encoding)
• The MAX_CONTROL_FRAME_BYTES limit remains a meaningful safety bound (not raised to paper over the encoding inefficiency)

Reproduction Steps

1. Run the gateway with VM compute driver on macOS: openshell-gateway --config gateway.toml --compute-driver vm
2. Create a sandbox: openshell sandbox create
3. Observe the supervisor error in the sandbox error output:
   Error: process error: encode control request: control frame is too large: 1099704 bytes
4. Verify the system CA bundle size: ls -la ~/.local/state/openshell/vm-driver/sandboxes/<id>/proxy-tls/ca-bundle.pem ~334KB on my macOS

Environment

• OpenShell: (development build from main)
• OS: macOS 26, Apple Silicon (arm64)
• Runtime: VM compute driver (libkrun, Apple Hypervisor.framework)
• CA bundle size: 334,127 bytes (default macOS system trust store)

Logs

2026-09-17T17:01:55.036Z INFO openshell_supervisor: Starting sandbox supervision
2026-09-17T17:01:55.107Z INFO openshell_supervisor: Isolation boundary attached
2026-09-17T17:01:55.108Z INFO openshell_supervisor: Isolation boundary enforcement confirmed
2026-09-17T17:01:55.121Z OCSF CONFIG:ENABLED [INFO] TLS termination enabled: ephemeral CA generated
2026-09-17T17:01:55.122Z OCSF CONFIG:READY [INFO] Policy DNS connected to isolation boundary
  2026-09-17T17:01:55.122Z OCSF NET:LISTEN [INFO] 127.0.0.1:3128
  2026-09-17T17:01:55.130Z OCSF CONFIG:LOADED [INFO] Acknowledged initial policy revision as loaded
  Error:   × process error: encode control request: control frame is too large: 1099704 bytes

  Relevant code path: `openshell-sandbox-backend/src/boundary_protocol.rs:35` defines `MAX_CONTROL_FRAME_BYTES = 1024 * 1024`. The `encode_frame` function at line 1092–1096 rejects any serialized payload exceeding this limit. The `ca_bundle: Option<Vec<u8>>` field at `boundary_protocol.rs:534` uses serde's default `Vec<u8>` → JSON array encoding.

 Relevant code path: `openshell-sandbox-backend/src/boundary_protocol.rs:35` defines `MAX_CONTROL_FRAME_BYTES = 1024 * 1024`. The `encode_frame` function at line 1092–1096 rejects any serialized payload exceeding this limit. The `ca_bundle: Option<Vec<u8>>` field at `boundary_protocol.rs:534` uses serde's default `Vec<u8>` → JSON array encoding.