feat(v1): add prime-agent harness over native ACP - #2254
Conversation
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
![macroscopeapp[bot]](https://avatars.githubusercontent.com/in/900172?s=60&v=4){kind=small}
ApprovabilityVerdict: Needs human review This PR introduces a new prime-agent harness with substantial new runtime logic (~450+ lines of new code). New features of this scope require human review. Additionally, an unresolved review comment identifies a potential bug in the resume path where the system prompt may be incorrectly dropped. You can customize Macroscope's approvability policy. Learn more. |
|
Pushed b59b806 with four fixes found by running a real eval in the docker runtime. Each was verified in a container, not inferred. The eval now completes end to endFrom the trace: What was brokenProgress was measurable at each step, which is how I knew each fix was real:
Two are worth calling out because they come from copying the Env prefix. The published release derives its prefix from its own package The Now inlined. DependencyThe silent- I verified this harness against a locally patched 0.6.0 tarball containing that fix, using the
|
Drives prime-agent's own ACP mode through the existing ACP helper rather than the pi harness's third-party pi-acp adapter. That adapter spawns `pi --mode rpc` and hard-codes pi's RPC command and event union, so prime-agent's IPython-only tool model, subagents, autonomous gates, goals, and heartbeats either degrade to a generic tool call or disappear. prime-agent speaks ACP natively as of its ACP mode, and carries the concepts ACP has no field for in a namespaced `ai.primeintellect.prime-agent` `_meta` envelope, so a rollout can observe them without the harness parsing a private protocol. Targets the current launch() contract. HOME is pinned per trace because prime-agent writes session and kernel state beneath it and concurrent rollouts must not share either.
0.6.0 is the first Prime Agent release that ships native ACP mode, so the harness can now install a published tarball that actually has --mode acp. Verified the derived tarball URL returns 200.
… key Four fixes found by running an actual eval in the docker runtime, each verified in a container rather than inferred: - Containers ship no Node and prime-agent requires >=22.8, so install failed with "npm: not found". Bootstraps Node the way the pi harness does, and exports it on PATH for the launch wrapper too, since the bundled Node is not on the container PATH. - The published release derives its env prefix from its own package piConfig, so it reads PRIME_AGENT_CODING_AGENT_DIR, not the upstream PI_ prefix. With the wrong name models.json was ignored and the run failed with "Unknown provider intercept". - prime-agent does not expand "$VAR" in models.json: it sends the literal string as the bearer token, which produced "401 unauthorized". Confirmed against a capturing HTTP server, which received `Bearer $PRIME_AGENT_INTERCEPT_KEY`. Inlines the secret instead of the pi-style indirection. With these, an eval completes end to end: 4 model calls, err 0.00, and a scored alphabet_sort reward.
…he prompt Review follow-up on two real problems. The bearer token was written in plaintext to the per-trace models.json. Concurrent rollouts share a runtime, so model-executed code in one rollout could read another rollout's credential and issue authenticated requests against its interception endpoint. This was a consequence of inlining the secret to work around prime-agent not expanding "$VAR": the pi harness's indirection had kept it out of the file. models.json now carries a placeholder, and the launch wrapper substitutes the real value from the environment at exec time under a 0700 dir and a 0600 file. The system prompt was applied twice: once via --append-system-prompt and again by the ACP runner, which seeds it into the conversation for a new session. Dropped the flag and left a note so it does not come back. Verified by rerunning the eval: ok=true, 3 model calls, 0 errors, and a scored alphabet_sort reward.
…Alpine The install guard short-circuited on the binary alone, so a runtime that already held one build reused it after `version` or `tarball_url` changed. Key the guard on the requested tarball like the pi harness keys on its versions, and give the Alpine branch the same repo-bump retry, since the official Node build is glibc-only and an older Alpine's own nodejs-current is below the 22.8 prime-agent needs.
…have prime-agent's ACP mode ignores the `mcpServers` of `session/new` entirely, and its own MCP integrations are authored Python skills the model imports in its kernel, so tool servers handed to the harness never reached the model: the repo's own echo-acp-resume-v1 fixture ran to completion with the agent reporting the tool did not exist and the reward at 0. Declare SUPPORTS_MCP false so `validate_pairing` rejects that pairing instead of degrading it.
…t segment A resumed segment replays the accreted conversation, and the ACP runner renders that transcript into the prompt with the `[system]` block the first segment already rendered into it, so passing `system_prompt` again delivered the task instructions twice. Observed on echo-user-sim-v1: the second segment's prompt carried two copies of the system prompt, one after this change.
…arly
Three install/launch edges from review, none reachable on the images this runs
on today but all of them failing obscurely when they are hit:
- the wrapper substituted the bearer token with `sed`, so a token containing
`|`, `&`, or a backslash would corrupt the key or abort the wrapper before
exec; node now rewrites the parsed models.json instead. Today's secret is
`secrets.token_urlsafe(16)`, which cannot contain those, so this is about not
depending on the generator's alphabet.
- a curl-less non-Debian image ran `apt-get` regardless and failed three steps
later inside `tar` ("tar: invalid magic"); it now says what it needs.
- an unrecognized machine fell back to the x64 Node archive and failed as
"prime-agent requires Node.js 22.8 or newer"; it is now rejected by name,
like the OS check already does.
The bucket URL reads like a stale internal endpoint next to the user-facing installer at app.primeintellect.ai/prime-agent/install.sh, so note why the harness uses it. That script is a thin front end: it defaults its own prime_agent_base_url to this bucket and downloads $base/releases/v$version/$package-$version.tgz, the same shape _tarball() builds. The value is also the prime-agent repo's R2_PUBLIC_BASE_URL variable, which its release workflow publishes to. This harness installs the npm tarball directly instead of running the script, so it needs the artifact base rather than the installer URL. Comment only; no behavior change. Verified the derived URL for the pinned version returns 200, and the docker eval still reports ok=true with 4 model calls, no errors, and a scored reward.
a0681ae to
f9513af
Compare
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit f9513af. Configure here.
| # it rendered on the first segment. Re-emitting the system prompt here | ||
| # would hand the model the same instructions twice. | ||
| if trace.branches and trace.branches[-1].messages: | ||
| system_prompt = None |
There was a problem hiding this comment.
Resume drops system prompt
Medium Severity
On the non-live fallback path, launch clears system_prompt whenever the branch already has messages. Default resume already strips system messages from the replayed transcript so resolve_prompt can re-emit them, so those instructions never reach the new ACP session.
Reviewed by Cursor Bugbot for commit f9513af. Configure here.
2 participants
Adds a
prime-agentharness for v1, driving Prime Agent's native ACP mode through the existingverifiers/v1/acp/helper.Previously closed while waiting on the dependency; that is now resolved. prime-agent#613 merged and shipped in v0.6.0, which is the first release containing
--mode acp. The harness pins that version, and I verified the tarball URL it derives returns HTTP 200 and that the published package actually contains the ACP mode code.Why not the
piharnessThe
piharness reaches Prime Agent's upstream through third-partypi-acp@0.0.31, which spawnspi --mode rpcand hard-codes pi's RPC command and event union. Prime Agent has diverged, and the differences are exactly what a rollout wants to observe:othertool call with no cell semanticsPrime Agent now speaks ACP natively, so this harness talks to it directly. Nothing here parses a private protocol.
Design
Targets the current
launch()contract onmain. I read #2116 and #2141 first: both are open, unapproved, andCONFLICTING, and #2141's own review objects to its dual-API shape. #2141 keepslaunch()/resume()as an adapter, so this harness keeps working if that lands, and can later opt into persistence by overridingopen_session()and returningACP.session(...).endpoint+secret, registered as an ordinary OpenAI-compatible providerSUPPORTS_MCP,SUPPORTS_RESUME,SUPPORTS_SKILLS,APPENDS_SYSTEM_PROMPTHOMEis pinned per trace, because Prime Agent writes session and IPython kernel state beneath it and concurrent rollouts must not share eitherautonomous+gatesmap to--autonomous/--autonomous-gate; gates withoutautonomousis rejected rather than silently ignoredVerification
harness_class("prime-agent")→PrimeAgentHarness, instantiated viaharness_config_type+load_harnessmode: "acp", provider/model,--autonomous-gate, and--append-system-promptall land--gateand--exclude-tools, neither of which exists.disabled_toolsnow raises instead of emitting a flag that would be silently ignoreduv run ruff check,ruff format --check, anduv run ty check verifiersall passPer AGENTS.md ("prefer e2e tests over unit tests; extra unit tests clog the repo") I added no test file; the checks above were run as throwaway scripts.
Note
Committed with
--no-verify: the hooks runuv run --locked, and my local uv (0.11.25) writes lockfile revision 3 against this repo's revision 4, so the hook fails on an unrelated lockfile rewrite.uv.lockis untouched in this branch, and I ran each hook's underlying command directly instead.Note
Medium Risk
New harness depends on external npm tarballs and Node bootstrap in containers; API key handling and concurrent install locking are security-sensitive but scoped per trace with explicit MCP rejection for tool tasksets.
Overview
Adds a
prime-agentv1 harness that runs Prime Agent in native ACP mode via the sharedACPhelper, instead of going through thepiadapter.PrimeAgentHarnessbootstraps Node (≥22.8) and installs a pinned npm release (default 0.6.0) under a locked shared path, wires model calls through the interception endpoint as an OpenAI-compatible provider, and launchesprime-agent --mode acpwith per-traceHOME(session + IPython kernel isolation). It setsSUPPORTS_MCP = falseso tool-bearing tasksets fail pairing validation, supports resume/skills/autonomous gates, injects API keys at exec time through a wrapper script, and removes.vf-prime-agent-{trace.id}on cleanup.Tests: new
prime-agent-persistence-v1fixture and docker e2etest_prime_agent_persists_native_acp_sessioncheck IPython state across two interaction turns, rollout dir cleanup, and native (non–transcript-replay) message shape; pytestprime_agentmark and harness list docs are updated.Reviewed by Cursor Bugbot for commit f9513af. Bugbot is set up for automated code reviews on this repo. Configure here.
Note
Add prime-agent harness over native ACP for v1 e2e evaluation
PrimeAgentHarnessin harness.py, which installs the prime-agent npm tarball into/tmp/vf-prime-agenton first use and launches it in ACP mode with per-trace isolated working directories._preparewrites amodels.jsonrouting through an interception endpoint and generates a wrapper script that injects the API key at launch time, with restricted file permissions (700/600) to limit secret exposure.SUPPORTS_RESUME=True), skipping the system prompt on resumed segments, and optional autonomous mode with configurable gates.PrimeAgentPersistenceEnvand matching taskset in prime_agent_persistence_v1.py that validates IPython kernel state persists across two interaction turns without reloading from conversation text.flock/lockfand a Node one-liner during setup and launch; failures in those steps will surface as non-zero exit codes with no structured error wrapping.Macroscope summarized f9513af.