Add typed model interception - #2165
Conversation
![macroscopeapp[bot]](https://avatars.githubusercontent.com/in/900172?s=60&v=4){kind=small}
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
ApprovabilityVerdict: Needs human review Unable to check for correctness in 9485382. This PR adds a new interception feature with new public API. Two high-severity bugs have been identified in review comments: tool rewrites are incorrectly rejected, and response termination references an undefined attribute. These issues require resolution before approval. You can customize Macroscope's approvability policy. Learn more. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1811fdfbf3
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a9826c5dc5
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
a9826c5 to
3bec3e8
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 076bad00ca
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
076bad0 to
8600d36
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c06a7847c3
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
c06a784 to
7c185b3
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7c185b3723
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
7c185b3 to
02c134e
Compare
4659dc4 to
da40b40
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 587c514b2e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
587c514 to
d9bad21
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d9bad21229
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
d9bad21 to
68e440d
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 00137f6f41
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 93882b287d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 538591e8cb
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b40cbb0dfc
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
Macroscope skipped reviewing this pull request. Per-PR cost limit exceeded (workspace setting). Reviews on this PR have cost $99.61 so far. This review would add an estimated $2.26, bringing the total to $101.87 — above your per-PR limit of $100.00. Tip To get this pull request reviewed, you can:
|
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7116fc75fb
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e56ed059f2
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a6a224bfd9
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 126529f58e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2dbce12916
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8d0e76c03c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0701022efb
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6f617c247f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: cb37d77911
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 644c594eb8
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4e2352ea0f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| except BaseException: | ||
| abandon() | ||
| raise | ||
| if messages[-1] != original: |
There was a problem hiding this comment.
When an @on_request hook returns a sanitized ToolMessage for a tool result, this newly added check turns the same-type rewrite into a TaskError instead of sending the redacted content to the model. The public hook contract allows replacing request[-1] with the same message type, so tool-output sanitizers either crash the rollout here or cannot be written for the server-side tool-result path; update the provider request body for the rewritten tool result or avoid dispatching rewrite-capable request hooks on ToolMessages.
Useful? React with 👍 / 👎.
| TaskError( | ||
| "tool results can only be rewritten by a harness tool hook" | ||
| ), | ||
| ) |
There was a problem hiding this comment.
Tool rewrites rejected after accept
High Severity
@on_request is documented to replace request[-1] with a same-type message, and intercept_request applies that rewrite for tool results. The interception server then fails the turn with a TaskError, so a documented tool-result replacement aborts the rollout instead of replacing the message or ending cleanly via Terminate.
Additional Locations (2)
Reviewed by Cursor Bugbot for commit 4e2352e. Configure here.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
There are 25 total unresolved issues (including 24 from previous reviews).
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 9485382. Configure here.
| ) | ||
| stream_owner: tuple[str, asyncio.Future[StreamReplay | None]] | None = None | ||
| if stream_key is not None: | ||
| request_id, identity = stream_key |
There was a problem hiding this comment.
Broken response termination path
High Severity
When an @on_response hook returns Terminate on a non-streaming turn, the server reads session.trace.terminated_by_intercept, which is not defined on Trace. That raises before commit/terminate run, so the rollout never records the interception and the failure is surfaced as a retryable 502 instead of ending the exchange.
Reviewed by Cursor Bugbot for commit 9485382. Configure here.
3 participants
Overview
Adds two small hooks for inspecting and changing model exchanges.
@vf.on_requestreceives one newvf.Messagesentry before it reaches the model.@vf.on_responsereceives avf.Responsebefore it reaches the harness. A hook returnsNoneto allow the exchange, a typedvf.Messageto replace it, orvf.Terminateto end the rollout.What this PR adds
@vf.on_requestand@vf.on_response, with priorities and sequential replacements.request[-1].Traceinjection with the full branch ending at the intercepted message.A request replacement must keep the same message type. A response replacement must be a text-only
vf.AssistantMessage.Stack
vf.Terminatefinal for rollout and environment scoring.Note
Medium Risk
Changes rollout and interception-server control flow (retries, streaming, termination) and narrows intercept return types, which can break existing task interceptors.
Overview
Adds typed model interception for tasks via
@vf.on_requestand@vf.on_response, exported alongsideInterceptRecordandTerminate.Request hooks run on the last message before inference (including user turns via
intercept_usersinRollout.step, and tool results in the interception server). Response hooks run after the model returns, before the harness sees the answer. Hooks may return a same-type replacement,Terminate, orNone; decisions are recorded on the trace. Breaking: interceptors no longer accept plainstr;InterceptDecisionno longer carries a replacement message;intercept_responsereturns(decision, replacement_text).Introduces
ProviderToolEventonAssistantMessage.provider_tools, parsed from Anthropic and OpenAI Responses dialects so provider-hosted tools (e.g. web search) are visible in traces.The graph gains
commit_promptso a request-sideTerminatecan commit the conversation prefix without a model call. The interception server reorders refusal checks, in-flight coalescing, and stream idempotency, and merges request/response interception records on each turn.Reviewed by Cursor Bugbot for commit 9485382. Bugbot is set up for automated code reviews on this repo. Configure here.
Note
Add typed model interception with
@on_requestand@on_responsetask hooks@on_requestand@on_responsedecorator factories so tasks can declare hooks that run before and after model inference respectively, with support for message rewriting and rollout termination.ProviderToolEventto represent provider-hosted tool activity (e.g. Anthropic/OpenAI built-in tools), now attached toAssistantMessage.provider_toolsand rendered in trace output.InterceptRecordentries, and support pre-inference termination viaPendingTurn.commit_prompt.intercept_responsenow mutates theResponsein place and returns(InterceptDecision, replacement_text | None); both request and response records are appended totrace.interceptions.InterceptResultno longer acceptsstr; hooks must returnMessage,Terminate, orNone, andInterceptDecisionno longer carries a rewritten message directly.Macroscope summarized 9485382.