Connect harness tool interception hooks - #2229
Conversation
![macroscopeapp[bot]](https://avatars.githubusercontent.com/in/900172?s=60&v=4){kind=small}
10ce46b to
4ffec10
Compare
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 10ce46b2e3
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
4ffec10 to
36dd49b
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 36dd49b67d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
ApprovabilityVerdict: Needs human review 2 blocking correctness issues found. This PR adds a new tool interception feature with significant complexity across multiple files and harnesses. Additionally, there are multiple unresolved HIGH severity review comments identifying potential bugs in trace management, protocol handling, and race conditions that require human attention. You can customize Macroscope's approvability policy. Learn more. |
36dd49b to
c1084d9
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c1084d94da
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
c1084d9 to
55ea9c6
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 55ea9c6db4
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
55ea9c6 to
3b50905
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3b50905efb
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
3b50905 to
142f853
Compare
e1e1512 to
990a6c0
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 990a6c02eb
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
990a6c0 to
b74aaba
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b74aaba6fc
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
b74aaba to
7a4870d
Compare
7a4870d to
79d1156
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 79d115619f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
79d1156 to
b984ce9
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b984ce973c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
b984ce9 to
19c94aa
Compare
19c94aa to
35e7e4e
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9bb042cfcb
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| rewritten = message | ||
| if records and not isinstance(rewritten.content, str): | ||
| raise TaskError("harness tool hooks can only rewrite plain text") | ||
| graph.prepare_turn(session.trace, [*branch, rewritten]).commit_prompt() |
There was a problem hiding this comment.
When an @on_request handler rebuilds a ToolMessage for a harness hook and accidentally changes tool_call_id, this commits the changed identity to the trace even though the hook response can only return replacement content to the harness. Bash/Claude/Codex will continue the conversation with the native original tool_call_id, and graph.message_hash() includes that id, so the next model request won't reuse this node and can re-run the same tool-result interceptor or hit the non-hook rewrite error; reject identity changes or commit the original id with the rewritten content.
Useful? React with 👍 / 👎.
| graph.prepare_turn(session.trace, [*branch, rewritten]).commit_prompt() | ||
| session.trace.interceptions.extend(records) | ||
| if termination is not None: | ||
| session.terminate(termination) |
There was a problem hiding this comment.
Sealed trace mutated after release
Medium Severity
handle_tool only checks session.released before waiting on tool_lock, then commits after await intercept_request with no re-check. Cancellation is delivered at await points, so the sync commit_prompt / terminate path can still run after release() seals the trace. handle_request already re-checks released before committing for this reason.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit 9bb042c. Configure here.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
There are 2 total unresolved issues (including 1 from previous review).
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 134ed86. Configure here.
| ToolInterceptionDecision( | ||
| action="rewrite", content=content_text(rewritten.content) | ||
| ).model_dump() | ||
| ) |
There was a problem hiding this comment.
Pending tool rewrite skips commit
High Severity
When a response hook supplies a pending ToolMessage, the before-phase path returns rewrite and records interceptions but never commit_prompts that result. Later tools in the same turn still commit against a branch that omits the synthetic message, so the harness conversation and graph diverge and the next model request can fork.
Reviewed by Cursor Bugbot for commit 134ed86. Configure here.
| content=native | ||
| if native is None or isinstance(native, str) | ||
| else json.dumps(native), |
There was a problem hiding this comment.
🟠 High harnesses/tool_hook.py:131
When a PostToolUse event has tool_response set to JSON null, main sends content: null in the ToolInterceptionRequest instead of serializing it to the string "null". The /tool handler rejects after-execution requests with null content ("tool result content is required after execution"), so the hook fails and interrupts the rollout even though the tool result is valid. The content field is set to native directly when native is None, bypassing json.dumps. Consider serializing None to "null" so the callback receives a string payload.
| content=native | |
| if native is None or isinstance(native, str) | |
| else json.dumps(native), | |
| content=json.dumps(native) | |
| if native is not None and not isinstance(native, str) | |
| else None | |
| if native is None | |
| else native, |
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @verifiers/v1/harnesses/tool_hook.py around lines 131-133:
When a `PostToolUse` event has `tool_response` set to JSON `null`, `main` sends `content: null` in the `ToolInterceptionRequest` instead of serializing it to the string `"null"`. The `/tool` handler rejects after-execution requests with null content ("tool result content is required after execution"), so the hook fails and interrupts the rollout even though the tool result is valid. The `content` field is set to `native` directly when `native is None`, bypassing `json.dumps`. Consider serializing `None` to `"null"` so the callback receives a string payload.
| event: [ | ||
| {"hooks": [{"type": "command", "command": command}]} | ||
| ] | ||
| for event in ("PreToolUse", "PostToolUse") |
There was a problem hiding this comment.
🟠 High codex/harness.py:271
When tool_interception_url is set, the same hook command is registered for both PreToolUse and PostToolUse. If the interception server returns action="terminate" during PreToolUse, tool_hook.py emits {"continue": false, "stopReason": ...}, but Codex's PreToolUse hook contract does not recognize those stop fields. Codex treats the hook as failed and proceeds with the tool call anyway, so a termination decision made before execution does not prevent the tool and its side effects from running. Consider emitting a Codex-compatible PreToolUse deny response (or not registering the terminate path for PreToolUse) so the tool call is actually blocked.
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @verifiers/v1/harnesses/codex/harness.py around line 271:
When `tool_interception_url` is set, the same hook command is registered for both `PreToolUse` and `PostToolUse`. If the interception server returns `action="terminate"` during `PreToolUse`, `tool_hook.py` emits `{"continue": false, "stopReason": ...}`, but Codex's `PreToolUse` hook contract does not recognize those stop fields. Codex treats the hook as failed and proceeds with the tool call anyway, so a termination decision made before execution does not prevent the tool and its side effects from running. Consider emitting a Codex-compatible `PreToolUse` deny response (or not registering the terminate path for `PreToolUse`) so the tool call is actually blocked.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a2b9c78c5a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| elif decision.action == "rewrite" and before: | ||
| output: BaseModel = PreToolUseOutput( | ||
| hook_specific_output=PreToolUseDecision( | ||
| permission_decision_reason=cast(str, decision.content) | ||
| ) | ||
| ) |
There was a problem hiding this comment.
When a response interceptor returns a ToolMessage for Codex, /tool reports that as a rewrite during PreToolUse, but this branch translates it into permissionDecision: deny. I checked the Codex hooks reference (https://learn.chatgpt.com/docs/hooks#tool-calls-from-code-mode): a blocked PreToolUse rejects code-mode tool promises and is surfaced as a denial, so Codex sees a failed tool instead of the synthetic output promised by @on_response. This affects Codex tasks that try to skip/synthesize a tool call; only advertise pre-tool rewrites for harnesses that can deliver a real replacement result, or use a native synthetic-output mechanism.
Useful? React with 👍 / 👎.
1 participant
Overview
Connects harness-owned tool calls and results to Verifiers interception.
A response hook can return a
vf.ToolMessagefor a tool call the model just proposed. The harness then blocks that call before execution and gives the synthetic result back to the agent as a normal tool message.What this PR adds
/toolcallback for pre-execution calls and post-execution results, keyed by the native tool-call ID.PreToolUse/ post-tool hooks for Claude Code and Codex.Trace.The original assistant tool call stays in the trace, followed by the synthetic tool result and the agent's continuation. Codex decorates denied commands with its native blocked-command text, while Bash and Claude return the synthetic text directly.
Stack
@vf.on_requestand@vf.on_responsehooks.vf.Terminatefinal for rollout and environment scoring.Note
Medium Risk
Touches interception, trace graph commits, and agent execution paths; Codex interception enables hook trust bypass, and incorrect
/toolgraph alignment could mis-attribute training data.Overview
Harness-owned tools now flow through Verifiers interception when a task defines
@on_response/@on_requestinterceptors. Response hooks can return aToolMessageto skip execution and supply a synthetic result; the harness never runs that call and the agent sees a normal tool reply.A new
POST /toolendpoint on the interception server handles before (pre-execution) and after (post-result) phases with allow, rewrite, or terminate decisions. Callbacks are serialized with a per-session lock so parallel tool calls extend one graph branch correctly. Tool messages committed via/toolget canonical graph updates, including backfilling token ids for nodes created before the renderer ran.Bash posts to
/toolaround each tool invocation (--tool-interception-url). Claude Code and Codex install a sharedtool_hook.pybridge for native PreToolUse/PostToolUse events; Codex enables hooks with--dangerously-bypass-hook-trustwhen interception is on.Rollouts only enable hook setup, expose
/toolin the runtime network allowlist, and pass the URL when interceptors exist.SUPPORTS_TOOL_INTERCEPTIONgates harness capability; graph MoE routing attribution now keys nodes by explicit(node_id, token_start)pairs to support/tool-committed messages.Reviewed by Cursor Bugbot for commit a2b9c78. Bugbot is set up for automated code reviews on this repo. Configure here.
Note
Connect harness tool interception hooks for bash, Claude Code, and Codex harnesses
/toolendpoint toInterceptionServerthat authenticates, validates, and responds toToolInterceptionRequestevents with allow, rewrite, or terminate decisions.setup_tool_interceptionandSUPPORTS_TOOL_INTERCEPTIONto the baseHarnessclass; bash, Claude Code, and Codex harnesses implement this to configure native hook mechanisms at launch.Rolloutconstructs the/toolURL when interceptors are present, adds it to the egress allowlist, and passes it intoharness.run/launch/resume.RolloutSessioncan now synthesizeToolMessageresults or terminate when the harness supports tool interception.--dangerously-bypass-hook-trustis set for Codex when hooks are enabled; requires Python ≥3.11 in the bash program.Macroscope summarized a2b9c78.