Skip to content

feat(agent-server): add docker runtime mode for per-conversation containers - #3403

Draft
rbren wants to merge 2 commits into
feat/remote-conversation-controlfrom
feat/agent-server-docker-runtime
Draft

rbren wants to merge 2 commits into
feat/remote-conversation-controlfrom
feat/agent-server-docker-runtime

Conversation

@rbren

@rbren rbren commented May 27, 2026

Copy link
Copy Markdown
Member

HUMAN:


AGENT:

Why

Automations need isolated, bounded Docker execution while retaining the shared conversation API and profile-secret rules.

Summary

  • Provision per-conversation containers through existing SDK workspace/services; retain history when releasing execution resources.
  • Reuse conversation-scoped routes and profile-secret selection for traffic forwarding and credential handoff.
  • Share readiness polling with DockerWorkspace and ApptainerWorkspace, and use HTTPX contexts for proxy cleanup on success, failure, and cancellation.

Issue Number

Closes #5016; incorporates the Docker work tracked by #5004/#5007. Contributor history is preserved.

How to Test

uv run pytest tests/agent_server/docker_runtime -q

Focused historical suites cover proxy, API/router/session-socket, and readiness/workspace behavior. The current Docker runtime suite passes 110 runtime, mediation, proxy, registry, and broker tests. A broader stack run passed 231 Docker, service, and health tests with one skip.

Live Docker probes created profile-selected conversations, executed runtime commands, verified selected-secret delivery, and released the containers. The integrated factory completed neubig/box-clone#8 and independently delivered neubig/airbnb-clone#29, #30, and #35. The deployed SDK combines this PR with its shared-contract and client prerequisites.

Type

  • Feature

Notes

Native stack #5018 now has #5017#5046#5081#3403; #4966 and independent profile schema #4931 are merged into main. The parent refresh at f5a9e7a04 preserves the entire previously validated source tree. Docker is the final implementation layer. Existing query-scoped APIs retain explicit v1.48 → v1.53 deprecation. Independent reliability/client PRs remain separate.


🐳 Agent Server images for this PR — GHCR package, pull/run commands, and all pushed tags (click to expand)

GHCR package: https://github.com/OpenHands/agent-sdk/pkgs/container/agent-server

Variants & Base Images

Variant Architectures Base Image Docs / Tags
java amd64, arm64 eclipse-temurin:17-jdk Link
python-slim amd64, arm64 nikolaik/python-nodejs:python3.13-nodejs22-slim Link
python amd64, arm64 nikolaik/python-nodejs:python3.13-nodejs22-slim Link
golang amd64, arm64 golang:1.21-bookworm Link

Pull (multi-arch manifest)

# Each variant is a multi-arch manifest supporting both amd64 and arm64
docker pull ghcr.io/openhands/agent-server:ec8ac16-python

Run

docker run -it --rm \
  -p 8000:8000 \
  --name agent-server-ec8ac16-python \
  ghcr.io/openhands/agent-server:ec8ac16-python

All tags pushed for this build

ghcr.io/openhands/agent-server:ec8ac16-golang-amd64
ghcr.io/openhands/agent-server:ec8ac16bb001244b2da1a86472ee508cab084759-golang-amd64
ghcr.io/openhands/agent-server:feat-agent-server-docker-runtime-golang-amd64
ghcr.io/openhands/agent-server:ec8ac16-golang_tag_1.21-bookworm-amd64
ghcr.io/openhands/agent-server:ec8ac16-golang-arm64
ghcr.io/openhands/agent-server:ec8ac16bb001244b2da1a86472ee508cab084759-golang-arm64
ghcr.io/openhands/agent-server:feat-agent-server-docker-runtime-golang-arm64
ghcr.io/openhands/agent-server:ec8ac16-golang_tag_1.21-bookworm-arm64
ghcr.io/openhands/agent-server:ec8ac16-java-amd64
ghcr.io/openhands/agent-server:ec8ac16bb001244b2da1a86472ee508cab084759-java-amd64
ghcr.io/openhands/agent-server:feat-agent-server-docker-runtime-java-amd64
ghcr.io/openhands/agent-server:ec8ac16-eclipse-temurin_tag_17-jdk-amd64
ghcr.io/openhands/agent-server:ec8ac16-java-arm64
ghcr.io/openhands/agent-server:ec8ac16bb001244b2da1a86472ee508cab084759-java-arm64
ghcr.io/openhands/agent-server:feat-agent-server-docker-runtime-java-arm64
ghcr.io/openhands/agent-server:ec8ac16-eclipse-temurin_tag_17-jdk-arm64
ghcr.io/openhands/agent-server:ec8ac16-python-amd64
ghcr.io/openhands/agent-server:ec8ac16bb001244b2da1a86472ee508cab084759-python-amd64
ghcr.io/openhands/agent-server:feat-agent-server-docker-runtime-python-amd64
ghcr.io/openhands/agent-server:ec8ac16-nikolaik_s_python-nodejs_tag_python3.13-nodejs22-slim-amd64
ghcr.io/openhands/agent-server:ec8ac16-python-arm64
ghcr.io/openhands/agent-server:ec8ac16bb001244b2da1a86472ee508cab084759-python-arm64
ghcr.io/openhands/agent-server:feat-agent-server-docker-runtime-python-arm64
ghcr.io/openhands/agent-server:ec8ac16-nikolaik_s_python-nodejs_tag_python3.13-nodejs22-slim-arm64
ghcr.io/openhands/agent-server:ec8ac16-python-slim-amd64
ghcr.io/openhands/agent-server:ec8ac16bb001244b2da1a86472ee508cab084759-python-slim-amd64
ghcr.io/openhands/agent-server:feat-agent-server-docker-runtime-python-slim-amd64
ghcr.io/openhands/agent-server:ec8ac16-nikolaik_s_python-nodejs_tag_python3.13-nodejs22-slim-slim-amd64
ghcr.io/openhands/agent-server:ec8ac16-python-slim-arm64
ghcr.io/openhands/agent-server:ec8ac16bb001244b2da1a86472ee508cab084759-python-slim-arm64
ghcr.io/openhands/agent-server:feat-agent-server-docker-runtime-python-slim-arm64
ghcr.io/openhands/agent-server:ec8ac16-nikolaik_s_python-nodejs_tag_python3.13-nodejs22-slim-slim-arm64
ghcr.io/openhands/agent-server:ec8ac16-golang
ghcr.io/openhands/agent-server:ec8ac16bb001244b2da1a86472ee508cab084759-golang
ghcr.io/openhands/agent-server:feat-agent-server-docker-runtime-golang
ghcr.io/openhands/agent-server:ec8ac16-golang_tag_1.21-bookworm
ghcr.io/openhands/agent-server:ec8ac16-java
ghcr.io/openhands/agent-server:ec8ac16bb001244b2da1a86472ee508cab084759-java
ghcr.io/openhands/agent-server:feat-agent-server-docker-runtime-java
ghcr.io/openhands/agent-server:ec8ac16-eclipse-temurin_tag_17-jdk
ghcr.io/openhands/agent-server:ec8ac16-python-slim
ghcr.io/openhands/agent-server:ec8ac16bb001244b2da1a86472ee508cab084759-python-slim
ghcr.io/openhands/agent-server:feat-agent-server-docker-runtime-python-slim
ghcr.io/openhands/agent-server:ec8ac16-nikolaik_s_python-nodejs_tag_python3.13-nodejs22-slim-slim
ghcr.io/openhands/agent-server:ec8ac16-python
ghcr.io/openhands/agent-server:ec8ac16bb001244b2da1a86472ee508cab084759-python
ghcr.io/openhands/agent-server:feat-agent-server-docker-runtime-python
ghcr.io/openhands/agent-server:ec8ac16-nikolaik_s_python-nodejs_tag_python3.13-nodejs22-slim

About Multi-Architecture Support

  • Each variant tag (e.g., ec8ac16-python) is a multi-arch manifest supporting both amd64 and arm64
  • Docker automatically pulls the correct architecture for your platform
  • Individual architecture tags (e.g., ec8ac16-python-amd64) are also available if needed

Live Agent Canvas evidence

Animated recording · Scenario, exact revisions and limits · Allowlisted observations. Direct integrated enhancement demonstration with a real agent and synthetic secrets; the report distinguishes the owning PR from companion SDK/Automation/Canvas changes.

Credential-scope review follow-up

The Docker runtime consumes the same persisted LaunchedAgentProfile.secret_refs as local conversations. It filters resumed and explicitly updated user secrets before lookup/transport, gates outer Codex broker grants and inner attachment, and preserves separate OpenHands model/MCP authentication. Profile edits cannot widen a resumed runtime. Validation: 67 Docker mediation/router tests passed (one opt-in real-Docker test skipped), plus 38 broker/provisioning/service tests passed. Local launch/resume before/after proof for the prerequisite; this follow-up does not claim a new live Docker capture.

Merge and release prerequisites

Native predecessors: software-agent-sdk#5017software-agent-sdk#5046software-agent-sdk#5081. The independent profile prerequisite software-agent-sdk#4931 is now merged into main. Docker remains the last layer after the runtime and secret contracts.

Runtime stack cleanup

Docker-only query-scoped compatibility registration now lives beside its callers, with the same v1.48→v1.53 deadline enforced by the existing repository checker. Its tests moved with it. TypeScript lost-create-response reconciliation was removed from this stack and is now the independent main-based #5036, so this diff does not introduce that client policy. The proxy comment now describes the actual per-runtime key, not the outer server credential.

Validation: 76 focused Docker/router/deprecation checks passed; one opt-in real-Docker test skipped. Repository hooks passed. Final head 66c34c4936d8f15d41b20a7b1eea6abdda2c72f9 incorporates current main through the native parent; its production source is identical to the tested cleanup commit 8ff58e4d4. The 231-test run used 871e2d06e; the later parent merge only adjusts OpenAPI compatibility metadata and simplifies when the TypeScript workspace binds. Existing immutable live evidence remains at its recorded revisions. This reorganization has focused contract tests and no new live-server recording.

Current head 9aa915a47 contains the narrowed #4966 and #5017 parents. Unused scoped MCP-probe routing and its outer-side materialization helper were removed; configured MCP tools still run normally inside each conversation runtime. The existing real FastMCP OAuth refresh compatibility regressions passed again; no duplicate test or new dependency pin was added. The 231-test run used 871e2d06e; the later parent merge only adjusts OpenAPI compatibility metadata and simplifies when the TypeScript workspace binds. Existing immutable live evidence remains at its recorded revisions.

@github-actions

github-actions Bot commented May 27, 2026

Copy link
Copy Markdown
Contributor

Python API breakage checks — ✅ PASSED

Result:PASSED

Action log

@github-actions

github-actions Bot commented May 27, 2026

Copy link
Copy Markdown
Contributor

REST API breakage checks (OpenAPI) — ✅ PASSED

Result:PASSED

Action log

@github-actions

github-actions Bot commented May 27, 2026

Copy link
Copy Markdown
Contributor

Coverage

Coverage Report •
FileStmtsMissCoverMissing
openhands-agent-server/openhands/agent_server
   api.py3343290%152, 154–159, 161, 163, 165, 220, 226, 273–275, 282, 287, 293–295, 304, 306–308, 338, 344, 348, 369–370, 636, 639, 645
   config.py133199%497
   conversation_router.py2661694%72–73, 314, 396, 442, 502, 672–675, 687–690, 730, 768
   conversation_service.py124513090%203–204, 213, 240–241, 245–246, 251, 397–398, 401–402, 418–419, 433, 600–601, 662, 744, 766, 773–774, 875, 970, 1021–1022, 1029, 1061–1062, 1078, 1109, 1113, 1127, 1153–1156, 1162–1163, 1172, 1174, 1240, 1250, 1275, 1283–1284, 1288–1289, 1297, 1324, 1330, 1424, 1430, 1435, 1441, 1449–1450, 1459–1462, 1471, 1483, 1491, 1537, 1543–1544, 1547–1549, 1576, 1628, 1678–1679, 1683, 1774–1775, 1846, 1901–1903, 1905–1906, 1909–1910, 1947, 2021–2022, 2053–2055, 2058–2059, 2063–2065, 2068–2069, 2073–2075, 2078–2079, 2108, 2117, 2160, 2170–2172, 2232, 2235, 2262, 2272, 2277–2280, 2294, 2305, 2317–2318, 2350, 2446, 2503, 2561, 2576–2577, 2955, 3008, 3011
   mcp_oauth_store.py1685965%55, 68, 82, 93, 98, 103, 116, 129, 180–182, 184, 186, 188–193, 196–199, 206, 209, 211–212, 215, 220, 225, 235–238, 243–247, 268, 274, 290–296, 301, 306, 316–319, 324–328
   runtime_router.py56689%54–59
   server_details_router.py63297%27–28
   utils.py391074%35–40, 53–55, 65
openhands-agent-server/openhands/agent_server/docker_runtime
   broker.py1522385%66, 80–81, 101, 109, 136, 145, 149, 152, 165–166, 172, 176–178, 181–183, 205–206, 209–211
   credential_client.py682563%31, 37–39, 55, 61, 65, 68–70, 73–74, 77–78, 83–84, 90, 98–100, 103, 109–111, 114
   mediation.py1551988%49, 54, 57, 60, 72, 81, 107, 140–141, 185, 210, 213–214, 241–244, 252, 294
   provisioning.py103496%50, 61, 68, 106
   proxy.py1041288%101, 190, 193, 195, 198, 209, 213, 219, 223, 241, 244–245
   registry.py2693487%66, 95, 107, 120, 138, 141, 144, 147, 165, 180, 205, 233, 240–241, 254–260, 266–267, 270, 298–299, 409, 422, 494–496, 505–506, 508
   routers.py2726775%86, 121–123, 166–167, 173, 193–194, 222–223, 226–227, 235–236, 241–242, 254, 260, 270–273, 295, 300–304, 338–339, 371–372, 412–414, 434, 454–456, 471, 483–485, 512, 535, 541–544, 547–548, 551–552, 555, 558–560, 566, 571–575, 652–653, 745
   runtime_route.py41490%33–34, 62–63
openhands-sdk/openhands/sdk
   __init__.py28293%115–116
openhands-sdk/openhands/sdk/conversation/impl
   remote_conversation.py8008390%92, 164, 269, 296, 309, 311–314, 340, 358–359, 370–373, 380–381, 386–389, 473, 483–485, 491, 575–576, 711–714, 716, 742–746, 751–754, 757, 773, 897, 953–954, 958–959, 973, 1116–1119, 1168–1169, 1193–1194, 1234–1235, 1253, 1263, 1267, 1276–1277, 1459–1460, 1553–1554, 1558, 1563–1567, 1591, 1596, 1643, 1893–1894, 1902–1903
openhands-sdk/openhands/sdk/llm/auth
   credentials.py70199%114
   openai.py38712767%138, 185–186, 190–192, 245–247, 271–272, 283–285, 307–308, 312, 322–323, 326, 357, 373, 378–379, 388–390, 395–396, 405–407, 518–520, 522–523, 525–527, 559–560, 562–565, 568, 571, 573–574, 576–580, 585–590, 596–600, 606–607, 610–616, 622, 624–626, 628–633, 639, 641, 643–645, 647, 651–652, 655–656, 660, 663, 669–671, 674, 678, 687–690, 721–722, 803, 814, 878–879, 883, 886–890, 893–895, 898–899, 912, 921, 961, 1000
openhands-sdk/openhands/sdk/workspace/remote
   remote_workspace_mixin.py169299%103, 161
openhands-workspace/openhands/workspace/apptainer
   workspace.py1656362%161, 170, 173–174, 181, 194, 220–221, 223, 226–228, 230–232, 234, 237, 243–245, 249–250, 258, 263–265, 323–324, 328–338, 340–343, 346–348, 353, 361, 365, 379, 383–395
openhands-workspace/openhands/workspace/docker
   workspace.py1803779%30–32, 50, 148, 168, 189, 192, 195–196, 203, 212, 215–216, 220, 228, 255, 284, 293, 300–301, 305–306, 309–311, 320–322, 327, 331, 335, 341–342, 365, 385, 402
TOTAL44708769183% 

@all-hands-bot all-hands-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ QA Report: PASS WITH ISSUES

Docker runtime mode works for the core create/proxy/WebSocket/delete flow, but I found two conversation API compatibility regressions in docker mode.

Does this PR achieve its stated goal?

Partially. I verified a real outer uvicorn agent-server in OH_CONVERSATION_RUNTIME=docker mode pulled the documented ghcr.io/openhands/agent-server:latest-python image, created a per-conversation Docker container, rewrote the workspace to /workspace, proxied GET /api/conversations/{id}, bridged /sockets/events/{id}, and removed the container on DELETE. However, two claimed preserved endpoints do not match local-mode behavior: GET /api/conversations?ids=<id> returns 500 in docker mode, and /api/conversations/count changes the response shape from a raw number to an object.

Phase Result
Environment Setup make build succeeded; Docker daemon was available (28.0.4) and the documented runtime image pulled successfully.
CI Status ⚠️ At check time, pre-commit was failing and several jobs were still pending; multiple tests/checks were green. I did not rerun CI tests.
Functional Verification ⚠️ Core docker runtime path works, but list-by-ids and count compatibility issues were reproduced with real HTTP requests.
Functional Verification

Test 1: Baseline local-mode API contract

Step 1 — Establish baseline (local mode):
Started the server with OH_CONVERSATION_RUNTIME=local and created a conversation using the normal HTTP API. Then queried the existing list/count endpoints:

curl "http://127.0.0.1:18081/api/conversations?ids=$LCID"
# HTTP 200, body: [{"id":"526d00e9-fefa-45a2-b355-dfdc9f53802f", ...}]

curl "http://127.0.0.1:18081/api/conversations/count"
# HTTP 200, body: 1

This establishes the existing client-visible contract: ids lookup returns a JSON array, and count returns a raw JSON number.

Test 2: PR docker runtime core flow

Step 2 — Apply PR behavior:
Started the PR server with:

OH_CONVERSATION_RUNTIME=docker OH_CONVERSATION_CONTAINER_STARTUP_TIMEOUT=90   uv run uvicorn openhands.agent_server.api:create_app --factory --host 127.0.0.1 --port 18080

Created a conversation through the outer server:

curl -H 'Content-Type: application/json' --data @/tmp/pr-start.json   http://127.0.0.1:18080/api/conversations
# HTTP 201, id=1e74d784-b1c0-4fad-b142-27e7c1bc7343,
# workspace.working_dir=/workspace

docker ps --filter 'name=oh-conv-'
# ebbdda2c8f99 oh-conv-1e74d784b1c04fadb14227e7c1bc7343-79ca09c1 ... 0.0.0.0:30450->8000/tcp

This confirms the PR creates a real per-conversation container and rewrites the workspace path into the container.

Step 3 — Exercise proxied traffic:

curl "http://127.0.0.1:18080/api/conversations/$CID"
# HTTP 200, returned the created conversation with workspace.working_dir=/workspace

curl "http://127.0.0.1:18080/api/conversations/search"
# HTTP 200, returned items containing id=1e74d784-b1c0-4fad-b142-27e7c1bc7343

uv run python /tmp/qa_ws_check.py
# connected
# {"id":"5d0e05be-01b2-441e-9f76-975d9f00673c","timestamp":"2026-05-27T14...

curl -X DELETE "http://127.0.0.1:18080/api/conversations/$CID"
# HTTP 200, body: {"success":true}

docker ps --filter 'name=oh-conv-'
# no remaining QA containers

This confirms root HTTP proxying, search aggregation, WebSocket bridging, and DELETE cleanup work in a real Docker-backed run.

Test 3: Reproduced docker-mode compatibility regressions

Step 1 — Baseline: local mode returned HTTP 200 with a JSON array for GET /api/conversations?ids=<id> and raw 1 for /count.

Step 2 — PR docker mode: the same user-facing endpoints behaved differently:

curl "http://127.0.0.1:18080/api/conversations?ids=$CID"
# HTTP 500
# {"detail":"Internal Server Error","exception":"'list' object has no attribute 'get'"}

curl "http://127.0.0.1:18080/api/conversations/count"
# HTTP 200
# {"count":1}

This shows docker mode does not fully preserve the existing conversation endpoint contract promised in the PR description.

Issues Found

  • 🟠 Issue: GET /api/conversations?ids=<conversation_id> returns 500 in docker mode instead of the local-mode JSON array response.
  • 🟠 Issue: GET /api/conversations/count changes response shape from raw JSON number (1) to object ({"count":1}).

This review was created by an AI agent (OpenHands) on behalf of the user.

Comment thread openhands-agent-server/openhands/agent_server/docker_runtime/routers.py Outdated
Comment thread openhands-agent-server/openhands/agent_server/docker_runtime/routers.py Outdated

@all-hands-bot all-hands-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Acceptable direction, but I found a few docker-mode issues that need attention before this is safe to merge: auth bypass, exposed inner servers, and REST/auth contract regressions.

This review was created by an AI agent (OpenHands) on behalf of the user.

[RISK ASSESSMENT]

  • [Overall PR] ⚠️ Risk Assessment: 🔴 HIGH — this is opt-in, but it changes request routing/authentication and starts network-reachable per-conversation servers.

VERDICT: ❌ Needs rework before merging.


Was this automated review useful? React with 👍 or 👎 to this review to help us measure review quality.
Workflow run: https://github.com/OpenHands/software-agent-sdk/actions/runs/26516796937

Comment thread openhands-agent-server/openhands/agent_server/docker_runtime/container_manager.py Outdated
Comment thread openhands-agent-server/openhands/agent_server/docker_runtime/routers.py Outdated
Comment thread openhands-agent-server/openhands/agent_server/docker_runtime/routers.py Outdated
Comment thread openhands-agent-server/openhands/agent_server/docker_runtime/routers.py Outdated
Comment thread openhands-agent-server/openhands/agent_server/api.py Outdated
rbren pushed a commit that referenced this pull request May 27, 2026
Six fixes for the per-conversation docker runtime, driven by reviewer
findings on PR #3403:

1. Bind inner container ports to loopback only (-p 127.0.0.1:HOST:8000)
   so the per-conversation agent-servers can only be reached through the
   outer server's authenticated proxy. (R3311480573)

2. Authenticate the WebSocket bridge against the OUTER server's session
   keys before opening the upstream connection. Reuses the existing
   sockets.py helper (header / query / first-message auth), and the
   bridge no longer calls accept() a second time. (R3311480598)

3. Preserve the local GET /api/conversations?ids=... contract: route is
   batch-get-by-id, requires the ids query param, returns
   list[ConversationInfo | None]. Looks each id up in the registry and
   fetches from its container (None for missing). (R3311480555,
   R3311480542)

4. Preserve the local /api/conversations/count contract: returns a bare
   JSON integer (not {"count": N}), honors ?status= by forwarding the
   query to each inner container and summing their integers.
   (R3311480576, R3311480571)

5. ContainerManager.start() now returns (running, is_new). The POST
   route only tears down the container on inner 4xx / connection error
   when is_new=True, so a retried create against an existing
   conversation can no longer kill the live container. (R3311480570)

6. Workspace static-file routes mount under the workspace-cookie auth
   group in docker mode via a new docker_workspace_router. The
   workspace router is now registered before the header-only api_router
   so the more specific path wins; browser iframe/<img> embeds with the
   oh_workspace_session_key cookie continue to work. (R3311480585)

Tests:
* test_container_manager: assert loopback port binding; updated for the
  (running, is_new) return tuple, plus an explicit is_new=False assert
  on the idempotent second start.
* test_docker_routers: new tests for batch-get-by-ids (incl. 422 on
  missing ids, null slots for unknown ids), bare-int /count contract,
  WS rejects wrong key, WS rejects missing first-message auth, WS
  accepts with valid outer key, POST retry preserves existing container
  on inner 4xx, fresh-create cleans up on inner 4xx, workspace route
  registered before the catch-all. Fake inner app reordered so /search
  and /count aren't shadowed by /{cid}.

22 docker_runtime tests pass; 144 tests in api / conversation /
workspace / docker_runtime all green.

Co-authored-by: openhands <openhands@all-hands.dev>
@rbren

rbren commented May 27, 2026

Copy link
Copy Markdown
Member Author

Pushed e7ec1a7 addressing all 8 review threads (now resolved). Summary:

Critical (security)

  • WS bridge now authenticates against the OUTER server's session_api_keys before opening the upstream connection (reuses sockets._accept_authenticated_websocket; no double-accept()). Wrong-key rejection happens pre-accept; missing-key falls through to first-message-auth and closes 4001 on timeout / bad frame.
  • Inner container ports now bind to 127.0.0.1 only — the per-conversation agent-server is only reachable through the outer auth proxy.

Important (API contracts)

  • GET /api/conversations?ids=... restored to local contract: required ids query, returns list[ConversationInfo | None] (not a page object), missing ids slot in as null. List/search aggregation lives only at /search.
  • GET /api/conversations/count restored to local contract: bare JSON integer, honors ?status= by forwarding the query to each container and summing.
  • ContainerManager.start() now returns (running, is_new); the POST handler only tears down on inner 4xx / connect-error when is_new=True, so a retried create against an existing conversation can't kill the live container.
  • Workspace static-file routes (/conversations/{cid}/workspace/...) are now served by a new docker_workspace_router mounted on workspace_api_router (cookie-or-header auth), and workspace_api_router is registered before the header-only api_router so the more specific path wins over the catch-all. Browser iframe/<img> embeds with oh_workspace_session_key continue to work in docker mode.

Tests
8 new regression tests (1 per review finding) plus updates to the existing ones. 22 docker_runtime tests + 144 tests across test_api / test_conversation_router / test_workspace_router / docker_runtime/ all pass; ruff + pyright clean.

This comment was posted by an AI agent (OpenHands) on behalf of the user.

@rbren
rbren marked this pull request as draft May 27, 2026 17:06
@rbren

rbren commented May 27, 2026

Copy link
Copy Markdown
Member Author

Pushed 151cd525 — simplification rewrite per the feedback that the previous design duplicated a lot of DockerWorkspace. Two structural changes:

1. Drop ContainerManager, use DockerWorkspace directly

docker_runtime/container_manager.py (339 lines of bespoke docker run wrapping, port allocation, healthchecks, log streaming, lifecycle cleanup, image management) is deleted and replaced with docker_runtime/registry.py (190 lines) which is a thin shell around DockerWorkspace. Everything ContainerManager did is something DockerWorkspace was already doing for its other use cases:

Concern ContainerManager DockerWorkspace
docker run wrapping bespoke argv builder yes
Free port allocation bespoke yes
Image pulls & cleanup bespoke yes (cleanup_image)
Network / GPU / platform partial yes
Volume mounts bespoke yes
Forwarded env bespoke yes (forward_env + extra_env)
Log streaming bespoke yes (detach_logs)
Healthcheck wait bespoke urlopen loop yes (health_check_timeout)
Lifecycle / cleanup bespoke yes (cleanup)

I added one small field to DockerWorkspace to cover the one capability that wasn't already there:

  • bind_host: str — host interface to publish on. Default "" keeps -p HOST_PORT:8000; setting "127.0.0.1" gives -p 127.0.0.1:HOST_PORT:8000. The docker registry pins this to 127.0.0.1 so only the outer agent-server can reach the inner — defense-in-depth around the proxy auth.

2. Drop fan-out across containers; read shared disk instead

Per the review pushback, fan-out was the wrong shape — it was N container hops for what's fundamentally a cheap directory walk. The outer's ConversationService now has a read_only_metadata mode that:

  • Skips EventService startup in __aenter__ (no leases acquired, no in-memory state, no lease-renewal task).
  • On every get / search / count / batch_get, re-walks conversations_path and reads meta.json + base_state.json straight off disk. Falls back to a synthesized state for conversations whose base_state.json hasn't been flushed yet.
  • Mutation methods aren't expected to be called (the docker proxy router intercepts them before they reach ConversationService).

Bind-mount layout is per-cid: each sub-container only sees its own conversations/{cid_hex} subdirectory. The outer sees all of them. The .openhands settings/secrets dir is shared so OH_SECRET_KEY round-trips correctly.

Other changes asked for in the review

  • ?cid= for global routers (bash/git/file/vscode/desktop/hooks/mcp/skills/tools/llm): registered one specific route per prefix in docker_global_proxy_router so the catch-all doesn't shadow /api/conversations / /api/settings / etc. Missing ?cid= → clear 400 telling the client what they need to do.
  • X-Session-API-Key forwarding: outer and inner share the same OH_SESSION_API_KEYS_0 via conversation_container_forward_env (now includes that key in the default list). The proxy passes through whatever header the client sent; for cookie-authed workspace static files it synthesizes the header from workspace.api_key (read out of the outer's env) so the inner static file server is happy.
  • No outer-side services touched for the simpler approach. The outer still runs tmux / vscode / desktop / sockets / settings / profiles in-process — those just aren't conversation-scoped.

Verification

The new architecture still answers every API the user asked about:

Endpoint group Where it runs in docker mode
POST /api/conversations proxy → sub-container (spawns it first)
`GET /api/conversations[/count /search
Per-cid mutations (/run, /pause, /events, …) proxy → sub-container
Workspace static files (/conversations/{cid}/workspace/…) proxy → sub-container, cookie-auth preserved
Global routers (/bash, /git, /file, …) proxy → sub-container, requires ?cid=
WS /sockets/events/{cid} outer authenticates, then bridges to sub-container

Stats

13 files changed, 1083 insertions(+), 1215 deletions(-)

Net -132 LoC even though new tests were added. Locally:

tests/agent_server/test_conversation_service.py        80 passed (4 new read-only-mode tests)
tests/agent_server/test_conversation_router.py         69 passed (no changes)
tests/agent_server/docker_runtime/test_docker_routers  17 passed (rewritten for new registry)

ruff check, ruff format, and pyright all clean on the changed files.


This comment was created by an AI agent (OpenHands) on behalf of the PR author.

@enyst

enyst commented Jun 10, 2026

Copy link
Copy Markdown
Member

@OpenHands read this PR description and investigate all you need to understand it; use examples/ dir, agent-server examples, to test as you need, maybe make a new one for this PR in .pr/ directory if there isn’t one. You are free to install yourself docker or all you need.

CI is failing for precommit, fix that too and push.

Then tell us WDYT: can the PR be simplified? Does it handle docker lifecycle correctly for multi-docker? /codereview it. Post as a comment on github.

Then take a deep breath. And propose fixes for the issues you uncovered and commit/push them. Post again what you did.

@openhands-ai

openhands-ai Bot commented Jun 10, 2026

Copy link
Copy Markdown

I'm on it! enyst can track my progress at all-hands.dev

enyst commented Jun 10, 2026

Copy link
Copy Markdown
Member

This PR comment was created by an AI agent (OpenHands) on behalf of the user.

Deep breath taken. Here is my current /codereview after reading the PR, current head (151cd525), the docker-runtime code, the existing remote-agent-server Docker examples, and the failing CI logs.

🟡 Taste Rating: Acceptable, but not mergeable yet

The second rewrite is pointed in the right direction: dropping fan-out for list/search/count and letting the outer server read shared persisted metadata is much simpler than making every metadata endpoint coordinate across containers. The route split is also mostly clean: mutations/proxy/websocket go to the inner server; settings/profiles/auth/etc. remain local.

[CRITICAL ISSUES]

  • Pre-commit is correctly failing: openhands.agent_server imports openhands.workspace.docker.workspace from docker_runtime/{registry,proxy,routers}.py. The repo import rules explicitly forbid agent_server -> workspace, and the added openhands-workspace dependency in openhands-agent-server/pyproject.toml creates an undesirable package-boundary/cycle problem. This needs to be fixed by either using a neutral shared helper or keeping the docker container runner inside agent_server, not by hiding the import.
  • Startup failure can leak a container: the current DockerWorkspace(...) constructor starts the container and then waits for /health. If the health check fails or the container dies during startup, the constructor raises before the registry records the workspace, so registry.stop() cannot clean it up. This is exactly the lifecycle edge case that matters for docker mode.
  • Multi-container startup is serialized: DockerConversationRegistry.get_or_create() holds one global lock while awaiting the blocking container startup in a worker thread. Correctness is okay, but in practice one slow/broken docker run can hold up unrelated conversations for up to conversation_container_startup_timeout. For “multi-docker”, starts for different conversation IDs should proceed concurrently while still deduplicating concurrent starts for the same ID.

[IMPROVEMENT OPPORTUNITIES]

  • Force inner runtime to local: the inner container should explicitly get OH_CONVERSATION_RUNTIME=local. Defaults make this true today, but if someone forwards broader OH_* env vars later, recursive docker-runtime containers are a nasty failure mode.
  • Keep the DockerWorkspace simplification idea, but respect boundaries: conceptually reusing existing docker lifecycle logic is good. In this repo, though, the package boundary says agent-server cannot import workspace. The pragmatic fix is either a small agent-server-local runner or a neutral lower-level helper that both DockerWorkspace and the docker runtime can use.
  • Tests are useful but missing the lifecycle regressions above: the router tests exercise real HTTP/WebSocket proxying, which is good. Add registry-level tests for parallel different-cid startup and cleanup when startup fails / shutdown races with startup.

[RISK ASSESSMENT]

  • [Overall PR] ⚠️ Risk Assessment: 🟡 MEDIUM. This is opt-in (local default is preserved), but it adds process/container lifecycle management, reverse proxying, auth bridging, and persistent-state sharing. The blast radius is contained by the feature flag, yet container leaks or serialized startups would hurt real docker-runtime deployments quickly.

VERDICT:
Needs fixes before merge: the architecture is close, but the package-boundary violation and lifecycle edge cases should be fixed first.

KEY INSIGHT:
The clean architecture is “outer agent-server owns routing + registry, inner agent-server owns conversation execution”; don’t pierce that by making agent-server depend on the workspace package just to borrow DockerWorkspace.

I’m going to fix the pre-commit failure and the lifecycle/concurrency issues above, then push and post a follow-up with the exact changes and validation.

enyst commented Jun 10, 2026

Copy link
Copy Markdown
Member

This PR comment was created by an AI agent (OpenHands) on behalf of the user.

Pushed b62b8282 (fix(agent-server): harden docker runtime lifecycle).

What changed:

  • Fixed the pre-commit import-rule failure by removing the agent_server -> openhands.workspace dependency:
    • dropped openhands-workspace from openhands-agent-server/pyproject.toml
    • removed DockerWorkspace imports from docker-runtime proxy/router/registry code
    • reverted the now-unused DockerWorkspace public-field additions from the workspace package
  • Replaced the registry’s DockerWorkspace wrapper with a small agent-server-local RunningConversationContainer runner that preserves the important behavior:
    • loopback-only port publishing: 127.0.0.1:<port>:8000
    • same 30000-39999 port range
    • shared per-conversation persistence bind mount and shared settings/secrets bind mount
    • best-effort cleanup via docker stop
  • Hardened lifecycle for multi-container use:
    • starts for different conversation IDs now proceed concurrently
    • concurrent starts for the same conversation ID still deduplicate to one startup task
    • containers that start but fail /health are stopped immediately
    • failed startup tasks are removed so a later retry can actually retry instead of reusing a failed task forever
    • stop() / shutdown() handle in-flight startup tasks and clean up containers that finish starting during teardown
    • inner servers are forced to OH_CONVERSATION_RUNTIME=local so forwarded env cannot accidentally create recursive docker-runtime containers
  • Avoided embedding forwarded env values directly in the docker run command line; the runner now passes env values through the subprocess environment and uses docker run -e KEY.
  • Added registry tests for same-cid dedupe, different-cid concurrent startup, health-failure cleanup, failed-start retry, and forcing the inner runtime to local.
  • Fixed the router tests to be independent of ambient SESSION_API_KEY in the developer/CI environment and fixed the pyright route-path narrowing issue.

Validation run locally:

make build
uv run pre-commit run --files   openhands-agent-server/openhands/agent_server/api.py   openhands-agent-server/openhands/agent_server/docker_runtime/__init__.py   openhands-agent-server/openhands/agent_server/docker_runtime/proxy.py   openhands-agent-server/openhands/agent_server/docker_runtime/registry.py   openhands-agent-server/openhands/agent_server/docker_runtime/routers.py   openhands-agent-server/pyproject.toml   openhands-workspace/openhands/workspace/docker/workspace.py   tests/agent_server/docker_runtime/test_docker_routers.py   tests/agent_server/docker_runtime/test_registry.py
uv run pytest tests/agent_server/docker_runtime -q
uv run pytest tests/agent_server/test_conversation_service.py -k 'read_only_metadata' -q

Results:

  • pre-commit: passed
  • tests/agent_server/docker_runtime: 22 passed
  • read-only metadata tests: 4 passed

CI checks had not been reported yet immediately after the push when I checked with gh pr checks.

@openhands-ai

openhands-ai Bot commented Jun 10, 2026

Copy link
Copy Markdown

OpenHands encountered an error: **Failed to send message to agent server: HTTP 503 error: no available server
**

See the conversation for more information.

@all-hands-bot

Copy link
Copy Markdown
Collaborator

[Automatic Post]: It has been a while since there was any activity on this PR. @rbren, are you still working on it? If so, please go ahead, if not then please request review, close it, or request that someone else follow up.

This comment was created by an AI agent (OpenHands) on behalf of the user.

2 similar comments
@all-hands-bot

Copy link
Copy Markdown
Collaborator

[Automatic Post]: It has been a while since there was any activity on this PR. @rbren, are you still working on it? If so, please go ahead, if not then please request review, close it, or request that someone else follow up.

This comment was created by an AI agent (OpenHands) on behalf of the user.

@all-hands-bot

Copy link
Copy Markdown
Collaborator

[Automatic Post]: It has been a while since there was any activity on this PR. @rbren, are you still working on it? If so, please go ahead, if not then please request review, close it, or request that someone else follow up.

This comment was created by an AI agent (OpenHands) on behalf of the user.

@github-actions

github-actions Bot commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

This PR is stale because it has been open for 40 days with no activity. Remove the stale label or leave a comment, otherwise it will be closed in 10 days.

@github-actions github-actions Bot added the Stale label Sep 3, 2026
@neubig
neubig force-pushed the feat/agent-server-docker-runtime branch from b62b828 to c419f1b Compare September 5, 2026 14:58
@github-actions

github-actions Bot commented Sep 5, 2026

Copy link
Copy Markdown
Contributor
  ✅ **PR Artifacts Cleaned Up**

  The `.pr/` directory is no longer present.

@neubig
neubig force-pushed the feat/agent-server-docker-runtime branch from c419f1b to 806c43f Compare September 6, 2026 00:15
@neubig neubig changed the title feat(agent-server): add docker runtime mode for per-conversation containers feat(agent-server): isolate tool execution in Docker workspaces Sep 6, 2026
@neubig
neubig force-pushed the feat/agent-server-docker-runtime branch 3 times, most recently from 8a547af to b153900 Compare September 6, 2026 01:17
@github-actions github-actions Bot removed the Stale label Sep 6, 2026
@neubig
neubig force-pushed the feat/agent-server-docker-runtime branch from b153900 to b62b828 Compare September 6, 2026 12:59
@neubig
neubig requested review from all-hands-bot and removed request for all-hands-bot September 14, 2026 00:00
@all-hands-bot
all-hands-bot dismissed their stale review September 14, 2026 00:00

Automated review used the wrong decision (APPROVED instead of COMMENT) and is dismissed. Findings are reposted as a comment.

@all-hands-bot

Copy link
Copy Markdown
Collaborator

🤖 OpenHands is reviewing this PR.

Head commit: 93d53ea3f211446a3939812a705a6999b48d0fe0
View the conversation: https://oss-agent-canvas.ngrok.dev/conversations/0cfba5a8-1296-4df1-aeff-a6e999d9f083

This comment was posted by an AI agent (OpenHands).

all-hands-bot
all-hands-bot previously approved these changes Sep 14, 2026

@all-hands-bot all-hands-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This review was created by an AI agent (OpenHands) on behalf of the repository maintainers.

Summary

This PR adds a Docker runtime mode (conversation_runtime: "docker") that provisions per-conversation containers with isolated credentials, proxied HTTP/WebSocket traffic, and a Unix-socket credential broker. The default remains "local", so existing behavior is unchanged.

Assessment

The security boundary is well-designed:

  • Per-conversation encryption keys and API keys (never the outer server's OH_SECRET_KEY)
  • HMAC-authenticated credential broker on a Unix socket with grant-scoped routes
  • Hop-by-hop header stripping + auth query param stripping in the proxy layer
  • Symlink protection in RuntimeProvisioningStore._direct_child
  • Container hardening: --cap-drop ALL, --security-opt no-new-privileges, loopback-only port binding
  • _materialize recursively resolves LookupSecret to StaticSecret and rejects embedded control keys before provisioning
  • Rollback logic restores the original identity on container start failure or inner-server rejection

No blocking issues found. No version bumps, no new dependencies, no # type: ignore, no changes to agent behavior/prompts/tools.

Risk Assessment

LOW - The PR adds an opt-in runtime mode behind a config flag defaulting to "local". Existing users are unaffected. The Docker-specific code paths are only activated when conversation_runtime == "docker".

One observation (not blocking): the _materialize substring check for control keys at mediation.py:80 could theoretically reject a legitimate secret whose value happens to contain a session API key as a substring. This is an acceptable security trade-off given the comment explains the rationale, and the probability is negligible with secrets.token_urlsafe(32) keys.

VERDICT: Worth merging. Core infrastructure is sound, security model is well-constructed, and existing behavior is preserved.

@all-hands-bot
all-hands-bot dismissed their stale review September 14, 2026 00:12

Automated review used the wrong decision (APPROVED instead of COMMENT) and is dismissed. Findings are reposted as a comment.

@all-hands-bot

Copy link
Copy Markdown
Collaborator

Posted an APPROVE review to GitHub PR #3403 (review ID: 5192823367) against commit 93d53ea3f211446a3939812a705a6999b48d0fe0.

Review Summary:

  • The PR adds a Docker runtime mode for per-conversation containers with isolated credentials, proxied HTTP/WebSocket traffic, and a Unix-socket credential broker
  • Default conversation_runtime = "local" preserves existing behavior — no eval/benchmark risk
  • No version bumps, no new dependencies, no # type: ignore, no changes to agent behavior/prompts/tools
  • Security model is well-constructed: per-conversation encryption keys, HMAC-authenticated credential broker, hop-by-hop header stripping, symlink protection, container hardening (cap-drop, no-new-privileges, loopback-only ports)
  • Rollback logic restores original identity on container start failure
  • One non-blocking observation about the substring-based control key rejection in _materialize
  • Risk: LOW — opt-in feature behind a config flag

This comment was posted by an AI agent (OpenHands).

@neubig

neubig commented Sep 14, 2026

Copy link
Copy Markdown
Member

@all-hands-bot please review the current head. If there are no blocking findings, please explicitly approve this PR rather than leaving only a comment.

@all-hands-bot all-hands-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review: feat(agent-server): add docker runtime mode for per-conversation containers

Verdict: ✅ APPROVE

Reviewed the current head (ff7b8d7cb) — all 57 changed files (+5584/−531), including the full docker_runtime/ module, SDK health polling, workspace changes, and the test suite.

Summary

This PR adds an opt-in Docker runtime mode (conversation_runtime: "docker") where each conversation runs in a dedicated, isolated agent-server container. The outer server owns container lifecycle, proxies HTTP/WebSocket traffic, and brokers credentials through a Unix-socket boundary. The default remains "local", so existing behavior is unchanged.

Architecture Assessment

The design is well-structured with clean module separation:

Module Responsibility
registry.py Container lifecycle — idempotent startup, concurrent starts for different cids, dedup for same cid, failed-start cleanup, stale container reaping on restart
provisioning.py Per-conversation identity — independent encryption keys, API keys, broker tokens; symlink-resistant path validation; atomic manifest writes
broker.py Unix-socket credential broker — HMAC bearer-token auth, grant-scoped routes, request size limits
mediation.py Secret materialization — resolves LookupSecret references locally, strips MCP OAuth state, blocks outer control credentials via substring check
proxy.py HTTP/WS forwarding — hop-by-hop header stripping (including authorization, cookie, set-cookie), auth query param stripping, inner API key synthesis
routers.py Route injection — mutations → inner container, metadata reads → outer (disk), workspace cookie auth preserved, global routes require ?cid=
runtime_route.py Scoped runtime API route with inner-image capability verification

Security Boundary

The security model is carefully designed:

  • Container hardening: --cap-drop ALL, --security-opt no-new-privileges, --rm, memory/CPU/pids limits, loopback-only port binding (127.0.0.1::8000)
  • Credential isolation: Per-conversation encryption keys and session API keys — never the outer server's OH_SECRET_KEY or session keys
  • Credential broker: 0o600 Unix socket with HMAC-authenticated bearer tokens and grant-based authorization per endpoint
  • Proxy hygiene: Hop-by-hop headers stripped, auth query params filtered, x-expose-secrets rejected on runtime routes
  • Secret materialization: _materialize() validates LookupSecret URLs against the internal server URL (SSRF protection), strips MCP OAuth state, and blocks outer control credentials from being provisioned
  • Symlink protection: RuntimeProvisioningStore._direct_child validates that mount paths don't follow symlinks
  • Forced local runtime: Inner containers get OH_CONVERSATION_RUNTIME=local to prevent recursive docker-runtime containers

Lifecycle Correctness

  • Starts for different conversation IDs proceed concurrently; same-id starts deduplicate to one task
  • Containers that start but fail /health are stopped immediately
  • Failed startup tasks are removed so retries can actually retry
  • stop() / shutdown() handle in-flight startup tasks and clean up containers that finish starting during teardown
  • Stale containers from a previous server instance are reaped via Docker labels on startup

Previous Review Findings — Resolved

The comment history shows multiple rounds of review. All previously identified critical issues have been addressed:

  • Package boundary violation (agent_server → workspace) — fixed by removing the DockerWorkspace dependency
  • Startup container leaks — fixed with cleanup in the exception path
  • Serialized multi-container startup — fixed with per-cid concurrency
  • OH_SECRET_KEY forwarding — fixed with per-conversation encryption keys
  • Auth bypass / exposed inner servers — fixed with loopback-only binding and pre-accept WebSocket auth

Non-Blocking Maintenance Risk

The MCP OAuth refresh path in broker.py uses FastMCP private methods (_initialize(), _refresh_token(), _handle_refresh_response()). This is acknowledged with an inline comment and covered by test_broker_refresh.py. A future FastMCP upgrade could break this path, but it is not a blocking issue for this PR.

Test Coverage

2,710 lines of tests across 9 test files covering broker, mediation, provisioning, proxy, registry, routers, scoped routes, legacy routes, and broker refresh. Coverage report shows 83% overall, with docker_runtime modules ranging from 63–96%.

CI Status

Most checks are passing (REST API breakage, cross-tests, sdk-tests, workspace-tests, tools-tests, stress tests). Some checks (pre-commit, agent-server-tests, build jobs) were still pending at review time for the latest head.

Conclusion

No blocking findings. The PR is well-architected, the security boundaries are carefully designed, previous review issues have been resolved, and the opt-in nature means no risk to existing local-mode behavior. Approving.

Generated by OpenHands AI on behalf of the user.

@neubig

neubig commented Sep 14, 2026

Copy link
Copy Markdown
Member

@all-hands-bot please review the current head. If there are no blocking findings, please explicitly approve this PR rather than leaving only a comment.

@all-hands-bot all-hands-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review: approve

I reviewed the current head (9aa915a47) of this PR, which adds a Docker conversation runtime mode where each conversation runs in a dedicated, hardened agent-server container with the outer server handling routing, lifecycle, and credential brokering.

What I checked

Architecture & security design

  • Per-conversation containers with strong isolation: --cap-drop ALL, --security-opt no-new-privileges, memory/CPU/PID limits, loopback-only port publishing (127.0.0.1::8000), and --user matching the host UID/GID.
  • Per-conversation RuntimeIdentity with distinct API key, encryption key, and broker token — all encrypted at rest via the server cipher. The manifest is never bind-mounted into the container.
  • Credential delivery via a per-conversation Unix socket broker (broker.py) with bearer-token auth and route-level grant enforcement. Refresh tokens are stripped from subscription responses. Body-size and concurrency limits are applied.
  • _materialize in mediation.py resolves LookupSecret references to StaticSecret only for internal URLs, rejects external lookups, and blocks embedded control credentials using substring matching (not just exact equality). Profile-secret filtering is applied before materialization.
  • The proxy strips all hop-by-hop headers, re-mints the inner X-Session-API-Key from the workspace's provisioned key (never forwarding the caller's outer credential), and explicitly rejects x-expose-secrets.
  • Symlink protection on every filesystem path resolution (_direct_child).
  • Inner containers are forced to OH_CONVERSATION_RUNTIME=local to prevent recursive container creation.

Lifecycle & concurrency

  • Concurrent starts for different conversation IDs proceed in parallel; same-ID starts deduplicate to a single task.
  • Failed startup tasks are removed so retries can actually retry. Containers that fail health checks are stopped immediately.
  • shutdown() is best-effort — one broken container doesn't block cleanup of the rest. In-flight starts are awaited and then stopped.
  • cleanup_stale_containers() uses server-scoped Docker labels to reap only containers owned by this server instance.

Routing

  • Conversation-scoped routes (/api/conversations/{id}/{service}/...) with legacy ?cid= compatibility routes deprecated v1.48→v1.53.
  • Mutation routes (switch_profile, switch_llm, secrets, security_analyzer) are mediated through the outer server with grant updates committed only on success and rolled back on failure.
  • Fork/children/credential-bindings explicitly return 501. Unsupported mutations return 501.
  • Capability check on first scoped-route use (conversation_runtime_routes_v1).

Test results

  • 134 of 135 new docker_runtime tests pass. The one failure (test_failed_docker_run_surfaces_stderr) is because Docker is not installed in this review sandbox — the test mocks subprocess.run but not the execute_command(["docker", "version"]) precheck, which requires real Docker. This is an environment limitation, not a code defect.
  • 282 existing agent_server tests pass with no regressions.
  • 327 of 328 TypeScript tests pass. The one failure (test-utils.test.ts) is caused by this sandbox's non-default OH_INTERNAL_SERVER_URL, unrelated to the PR.
  • Comprehensive test coverage: ~2,900 lines of new tests covering broker, mediation, proxy, registry, routers, provisioning, scoped routes, legacy routes, and broker refresh.

Previous review feedback
All items from the prior review rounds have been addressed: the package-boundary violation was fixed, container hardening was added, scoped secrets replaced the shared persistence mount, runtime ownership persistence and reconciliation were implemented, and MCP/global routing was corrected.

Non-blocking observations (not requesting changes)

  1. broker.py uses oauth._initialize() and oauth._refresh_token() which are private FastMCP APIs. This is acknowledged in a comment and covered by a regression test, but it's a coupling risk for future FastMCP upgrades.
  2. The test_failed_docker_run_surfaces_stderr test would be more robust if it also mocked execute_command or the docker version availability check, so it doesn't require real Docker.
  3. test_secret_mutation_filters_before_lookup hardcodes http://127.0.0.1:8000 but the code reads OH_INTERNAL_SERVER_URL from the environment — minor test fragility.

No blocking findings. Approving.

Generated by OpenHands AI on behalf of the user.

@all-hands-bot

Copy link
Copy Markdown
Collaborator

Review: approve

I reviewed the current head (9aa915a47) of this PR, which adds a Docker conversation runtime mode where each conversation runs in a dedicated, hardened agent-server container with the outer server handling routing, lifecycle, and credential brokering.

What I checked

Architecture & security design

  • Per-conversation containers with strong isolation: --cap-drop ALL, --security-opt no-new-privileges, memory/CPU/PID limits, loopback-only port publishing (127.0.0.1::8000), and --user matching the host UID/GID.
  • Per-conversation RuntimeIdentity with distinct API key, encryption key, and broker token — all encrypted at rest via the server cipher. The manifest is never bind-mounted into the container.
  • Credential delivery via a per-conversation Unix socket broker (broker.py) with bearer-token auth and route-level grant enforcement. Refresh tokens are stripped from subscription responses. Body-size and concurrency limits are applied.
  • _materialize in mediation.py resolves LookupSecret references to StaticSecret only for internal URLs, rejects external lookups, and blocks embedded control credentials using substring matching (not just exact equality). Profile-secret filtering is applied before materialization.
  • The proxy strips all hop-by-hop headers, re-mints the inner X-Session-API-Key from the workspace's provisioned key (never forwarding the caller's outer credential), and explicitly rejects x-expose-secrets.
  • Symlink protection on every filesystem path resolution (_direct_child).
  • Inner containers are forced to OH_CONVERSATION_RUNTIME=local to prevent recursive container creation.

Lifecycle & concurrency

  • Concurrent starts for different conversation IDs proceed in parallel; same-ID starts deduplicate to a single task.
  • Failed startup tasks are removed so retries can actually retry. Containers that fail health checks are stopped immediately.
  • shutdown() is best-effort — one broken container doesn't block cleanup of the rest. In-flight starts are awaited and then stopped.
  • cleanup_stale_containers() uses server-scoped Docker labels to reap only containers owned by this server instance.

Routing

  • Conversation-scoped routes (/api/conversations/{id}/{service}/...) with legacy ?cid= compatibility routes deprecated v1.48→v1.53.
  • Mutation routes (switch_profile, switch_llm, secrets, security_analyzer) are mediated through the outer server with grant updates committed only on success and rolled back on failure.
  • Fork/children/credential-bindings explicitly return 501. Unsupported mutations return 501.
  • Capability check on first scoped-route use (conversation_runtime_routes_v1).

Test results

  • 134 of 135 new docker_runtime tests pass. The one failure (test_failed_docker_run_surfaces_stderr) is because Docker is not installed in this review sandbox — the test mocks subprocess.run but not the execute_command(["docker", "version"]) precheck, which requires real Docker. This is an environment limitation, not a code defect.
  • 282 existing agent_server tests pass with no regressions.
  • 327 of 328 TypeScript tests pass. The one failure (test-utils.test.ts) is caused by this sandbox's non-default OH_INTERNAL_SERVER_URL, unrelated to the PR.
  • Comprehensive test coverage: ~2,900 lines of new tests covering broker, mediation, proxy, registry, routers, provisioning, scoped routes, legacy routes, and broker refresh.

Previous review feedback
All items from the prior review rounds have been addressed: the package-boundary violation was fixed, container hardening was added, scoped secrets replaced the shared persistence mount, runtime ownership persistence and reconciliation were implemented, and MCP/global routing was corrected.

Non-blocking observations (not requesting changes)

  1. broker.py uses oauth._initialize() and oauth._refresh_token() which are private FastMCP APIs. This is acknowledged in a comment and covered by a regression test, but it's a coupling risk for future FastMCP upgrades.
  2. The test_failed_docker_run_surfaces_stderr test would be more robust if it also mocked execute_command or the docker version availability check, so it doesn't require real Docker.
  3. test_secret_mutation_filters_before_lookup hardcodes http://127.0.0.1:8000 but the code reads OH_INTERNAL_SERVER_URL from the environment — minor test fragility.

No blocking findings. Approving.

Generated by OpenHands AI on behalf of the user.

@all-hands-bot
all-hands-bot removed their request for review September 14, 2026 11:05
@neubig
neubig force-pushed the feat/agent-server-docker-runtime branch from f5a9e7a to ec8ac16 Compare September 14, 2026 16:27
@neubig
neubig force-pushed the factory/profile-secret-boundary branch from 5bff578 to 7bf8d0d Compare September 14, 2026 16:27
@neubig
neubig removed this pull request from stack #5018 September 14, 2026 17:32
@neubig
neubig force-pushed the feat/agent-server-docker-runtime branch from ec8ac16 to 5d6e1cc Compare September 14, 2026 17:32
@neubig
neubig changed the base branch from factory/profile-secret-boundary to feat/profile-scoped-script-commands September 14, 2026 17:32
@neubig
neubig marked this pull request as ready for review September 14, 2026 17:32
@neubig
neubig marked this pull request as draft September 14, 2026 17:35
@neubig
neubig force-pushed the feat/agent-server-docker-runtime branch from c7c019a to c8bdabb Compare September 14, 2026 19:44
@neubig
neubig force-pushed the feat/profile-scoped-script-commands branch from 05c3bc2 to 91a259d Compare September 15, 2026 01:12
@neubig
neubig force-pushed the feat/agent-server-docker-runtime branch from c8bdabb to 2d8894b Compare September 15, 2026 01:12
Co-authored-by: openhands <openhands@all-hands.dev>
Expose an explicit host workspace namespace and route SDK host operations
through it so scanner automations remain host-side while their selected subject
agents use conversation containers.

Co-authored-by: openhands <openhands@all-hands.dev>
@neubig
neubig force-pushed the feat/agent-server-docker-runtime branch from 2d8894b to 92a8871 Compare September 15, 2026 15:58
@neubig
neubig changed the base branch from feat/profile-scoped-script-commands to feat/remote-conversation-control September 15, 2026 15:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Decouple runtime contracts and profile secrets from advanced Docker provisioning

5 participants