Add interception example environments - #2178
Conversation
![macroscopeapp[bot]](https://avatars.githubusercontent.com/in/900172?s=60&v=4){kind=small}
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d41b94349e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
ApprovabilityVerdict: Needs human review 1 blocking correctness issue found. This PR adds new example environments, but there are multiple unresolved review comments identifying bugs: incorrect domain matching in web_search_interception, unguarded JSON parsing in bash_interception that can crash rollouts, and auth configuration issues. These substantive issues should be addressed. You can customize Macroscope's approvability policy. Learn more. |
295cec1 to
7aae47f
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7aae47f960
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
7aae47f to
b1f77a4
Compare
b1f77a4 to
3113425
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 311342505e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
3113425 to
4b92fe4
Compare
8dad0d9 to
b12342b
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b12342b8be
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
b12342b to
ecb3b4b
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6939c63202
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8d623180c4
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1496a4d8f6
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 51e83618cc
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a8d62aad59
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2176426ac4
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| def load(self) -> list[WebSearchInterceptionTask]: | ||
| prompt = ( | ||
| "Use your native web search to find the official OpenAI Responses API " | ||
| "documentation. Cite the source and include the search query." |
There was a problem hiding this comment.
Wrong blocked docs domain
Medium Severity
BLOCKED_WORD is platform.openai.com, but the task asks for the official Responses API docs, which now live on developers.openai.com. Termination with reward is the only scoring path, so runs that correctly search and cite the current docs never match and stay at reward 0.
Reviewed by Cursor Bugbot for commit 30cfbc5. Configure here.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5c5d47976c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if ( | ||
| call | ||
| and call.name == "bash" | ||
| and json.loads(call.arguments).get("command") == GIT_COMMAND |
There was a problem hiding this comment.
When the model emits a bash tool call whose arguments are invalid JSON or a valid non-object, the Bash harness turns that into an error tool result so the model can retry, but this request hook reparses call.arguments without the defensive handling used above in block_bash. In that scenario json.loads(...).get(...) raises inside @on_request, converting a recoverable tool-argument mistake in the new bash_interception_v1 eval into a TaskError instead of forwarding the tool error result.
Useful? React with 👍 / 👎.
| message = request[-1] | ||
| if not isinstance(message, vf.ToolMessage): | ||
| return None | ||
| call = next( |
There was a problem hiding this comment.
🟠 High bash_interception_v1/taskset.py:35
rewrite_git_result calls json.loads(call.arguments).get("command") without the try/except and isinstance(..., dict) guard used in block_bash. When a model emits malformed JSON arguments, json.loads raises ValueError; when it emits valid non-object JSON like [], .get() raises AttributeError. A tool harness can return an error ToolMessage for such malformed arguments, so this interceptor crashes the rollout instead of passing the error result through unchanged. Consider wrapping the parse in the same try/except and checking isinstance(arguments, dict) before calling .get().
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @environments/bash_interception_v1/bash_interception_v1/taskset.py around line 35:
`rewrite_git_result` calls `json.loads(call.arguments).get("command")` without the `try`/`except` and `isinstance(..., dict)` guard used in `block_bash`. When a model emits malformed JSON arguments, `json.loads` raises `ValueError`; when it emits valid non-object JSON like `[]`, `.get()` raises `AttributeError`. A tool harness can return an error `ToolMessage` for such malformed arguments, so this interceptor crashes the rollout instead of passing the error result through unchanged. Consider wrapping the parse in the same `try`/`except` and checking `isinstance(arguments, dict)` before calling `.get()`.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 2 potential issues.
There are 3 total unresolved issues (including 1 from previous review).
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 48b142a. Configure here.
| == GIT_COMMAND | ||
| ), | ||
| None, | ||
| ) |
There was a problem hiding this comment.
Unguarded tool argument parsing
Medium Severity
block_git parses tool arguments with bare json.loads and then calls .get without checking the value is a dict. Invalid JSON or a non-object payload raises into @on_response and fails the rollout as a TaskError, instead of skipping the call. The sibling block_bash handler already guards both cases.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit 48b142a. Configure here.
|
|
||
|
|
||
| class InterceptionTaskConfig(vf.TaskConfig): | ||
| judge: vf.JudgeConfig = vf.JudgeConfig() |
There was a problem hiding this comment.
Judge ignores OpenAI-only credentials
Medium Severity
InterceptionTaskConfig.judge always constructs a default JudgeConfig, which targets Prime. The smoke runner can evaluate with only OPENAI_API_KEY, but judge_guard still calls that Prime-default judge, so the JUDGE_BLOCK example fails auth in OpenAI-only setups even when the main model client is configured correctly.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit 48b142a. Configure here.
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
1 similar comment
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
2 participants
Overview
Adds four small environments that demonstrate
@vf.on_requestand@vf.on_response.What this PR adds
interception-v1: deterministic replacement, judge-backed replacement, termination, and metric-only observation.grayscale-interception-v1: returns a new user message with inline images converted to grayscale.bash-interception-v1: replaces one Bash response outright and blocks another proposed command before execution by returning a syntheticvf.ToolMessage.web-search-interception-v1: uses the Codex harness, inspects native web-search results, and terminates when a blocked word appears.The replacement examples return typed
vf.Messageobjects. The metric-only example callstrace.record_metric()and returnsNone, leaving the response unchanged. The judge example also receivesTraceso its model usage is recorded.Stack
@vf.on_requestand@vf.on_responsehooks.vf.Terminatefinal for rollout and environment scoring.Note
Low Risk
Adds isolated example environments and lockfile entries only; no changes to core verifiers runtime or harness logic.
Overview
Adds four new example packages under
environments/that demonstrate@vf.on_requestand@vf.on_response, and wires them into the rootexamplesdependency group anduveditable sources.interception-v1shows four response-side patterns on marker prompts: high-priorityvf.Terminate, deterministicAssistantMessagereplacement, async judge replacement, and metric-only observation viatrace.record_metricwith rewards keyed ontrace.interceptions/ metrics.bash-interception-v1blocks specific bash tool commands by returning a replacement assistant message or a syntheticToolMessage, with rewards that verify sentinel files never appeared on the runtime filesystem.grayscale-interception-v1intercepts the user request to rewrite inlinedata:image/parts to grayscale (Pillow) before the model sees them.web-search-interception-v1usesCodexHarnessand terminates when a nativeweb_searchprovider event’s serialized payload contains a blocked domain string.Reviewed by Cursor Bugbot for commit 27b838c. Bugbot is set up for automated code reviews on this repo. Configure here.
Note
Add interception example environments for bash, grayscale, web search, and response blocking
Adds four new example environment packages under
environments/that demonstrate different interception patterns in the verifiers framework:interception_v1shows four response interception modes: deterministic blocking, LLM judge-based blocking, early rollout termination, and metric-only recording without modifying delivery.bash_interception_v1intercepts bash tool calls matching sentinel commands, either replacing them with an assistant message or injecting a synthetic tool result to let the agent continue without executing.grayscale_interception_v1converts inline base64 images in user messages to grayscale via PIL before they reach the model.web_search_interception_v1terminates rollouts when a native web search result contains a blocked word, preventing the result from reaching Codex.examplesgroup.Macroscope summarized 27b838c.