feat(eval): add plan-mode and HITL contract evals (D2 follow-up)#236
Open
zjshen14 wants to merge 1 commit into
Open
feat(eval): add plan-mode and HITL contract evals (D2 follow-up)#236zjshen14 wants to merge 1 commit into
zjshen14 wants to merge 1 commit into
Conversation
Extends the replay harness with two surfaces from the D2 contract-eval backlog (#234): **Plan-mode contract eval** — a synthesised JSONL tape whose single turn is prefixed with `/plan`, causing agent.run() to run with readOnly=true. The tape's write tool call is blocked by the executor's readOnly guard before reaching TapeRegistry, emitting tool_denied(plan_mode). Asserts: - tool_denied fires exactly once with reason="plan_mode" - tool_exec_start / tool_exec_end counts are both 0 - tape exhausted cleanly, 0 unconsumed results **HITL contract eval** — three programmatic tapes exercising the three confirmation paths: - confirmFn present + returns "deny" → tool_denied(user_denied) - confirmFn absent (non-interactive context) → tool_denied(non_interactive) - confirmFn present + returns "allow" → tool executes normally, no denial RunTapeOptions gains confirmFn? and forcesConfirmation? to wire these paths; runner.ts calls agent.setConfirmFn() and agent.setForcesConfirmationFn() after construction. Closes #234 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
4 tasks
Owner
Author
Review: feat(eval): add plan-mode and HITL contract evals (D2 follow-up)What's good: Solid approach — the synthesized JSONL tape for plan-mode, the dedicated Overlap with #237This PR and #237 were opened one day apart and both address the same
Merging both would produce a Design trade-offs:
Recommendation: Pick one and close the other — no code-quality blocker in either. Generated by Claude Code |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
RunTapeOptionsin the replay harness withconfirmFn?andforcesConfirmation?so contract evals can drive HITL decisions without an interactive terminal.plan-mode-write-block) whose single/plan-prefixed turn contains awritecall, proving the executor'sreadOnlyguard fires before the call reaches the registry.src/eval/replay/contract.test.tswith 9 tests covering two D2 contract surfaces: plan-mode write-blocking and the three HITL paths (user_denied, non_interactive, allow).Related issue
Closes #234
Test plan
npm run typecheck && npm run lint && npm run format:check && npm test— all pass (713 tests, 9 new)tool_denied(plan_mode)fires,tool_exec_startcount is 0, tape exhausted cleanlytool_denied(user_denied)fires whenconfirmFnreturns"deny"tool_denied(non_interactive)fires when noconfirmFnis providedtool_exec_startfires once whenconfirmFnreturns"allow"https://claude.ai/code/session_01HRHNetsbTUajSxK33fby1u
Generated by Claude Code