Persist and resume model-free task validation

[rasdani]

Summary

• persist resolved validate config, incremental per-task JSONL results, logs, and an aggregate summary in a fresh default run directory or an explicit -o/--output-dir
• add --resume RUN_DIR config replay and canonical resume selection: retain completed valid/invalid tasks; rerun missing, error, timeout, malformed, or torn records without duplicate final rows
• match eval resume identity exactly: hash canonical task content with task_key, count target/result multiplicity by key, and use distribute to assign owed work across content-identical tasks
• factor format-agnostic task_key, distribute, and split_resume into shared cli/resume.py; keep eval episode loading and validate result-row loading in their respective callers
• apply the same persistence and resume behavior to the existing --only-gold and --only-setup modes while preserving fresh per-check runtimes and teardown

Validation

• uv run pytest tests/ -m 'not e2e' — 910 passed, 67 deselected
• uv run ruff check --fix .
• uv run ruff format --check .
• uv run ty check verifiers
• uv run pre-commit run --all-files
• temporary uncommitted assertions covering shared resume primitives, eval-style duplicate-content distribution, torn rows, valid retention, timeout retry, canonicalization, and config replay
• eval and validate CLI help smoke
• local subprocess smoke: setup-only 2-task run plus no-op resume
• local subprocess smoke: gold-only 1-task run plus no-op resume

Live e2e note

The earlier full uv run pytest tests/ attempt reached 857 passing tests before the live e2e tail failed on expired/unauthorized model and Prime credentials and a missing Modal extra; it was stopped after the shared failure cause was established.

---

Note

Medium Risk
Validate’s default behavior now always creates on-disk output and resume rewrites results.jsonl; mistakes in row canonicalization could drop or duplicate work, though the logic mirrors proven eval resume patterns.

Overview
Validate now writes durable run artifacts (config.toml, incremental results.jsonl, summary.json, validate.log) under a default outputs/<taskset>--validate/<uuid> path or -o/--output-dir, and supports --resume <dir> that replays the saved config and only re-runs tasks that are missing or ended in error/timeout (valid/invalid finals are kept). Task identity and owed-work spreading match eval: content hashes via shared task_key and distribute.

Eval is refactored to import task_key, distribute, and split_resume from new verifiers/v1/cli/resume.py instead of cli/eval/resume.py; eval-specific episode loading stays in cli/eval/resume.py. split_resume now takes the CLI name for clearer usage errors.

ValidateConfig gains run uuid, output_dir, and internal resume fields. The validate runner plans work from resume state, persists each finished task under a lock (with shielded writes), updates the live dashboard from kept rows, and prints the final summary on success; interrupt exits 130 with a pointer to partial results.

^{Reviewed by Cursor Bugbot for commit 717a550. Bugbot is set up for automated code reviews on this repo. Configure here.}

Note

Add persist and resume support to the validate CLI command

• The validate command now writes per-task results to results.jsonl and a summary.json incrementally as each task completes, using an output directory derived from the taskset name and a per-run UUID or an explicit -o <output-dir> flag.
• A --resume <output-dir> flag reloads the prior run config and skips tasks that already have terminal outcomes, computing remaining work via content-based task keys (SHA-256 of sorted JSON).
• Resume utilities (task_key, distribute, split_resume) are extracted from the eval CLI into a shared resume.py module and reused by both eval and validate.
• On KeyboardInterrupt, the CLI exits with code 130 and prints the partial results directory.
• Behavioral Change: validate now always writes output files to disk; runs that previously produced no on-disk artifacts will now create a structured output directory.

^{Macroscope summarized 717a550.}

[macroscopeapp]

Approvability

Verdict: Needs human review

This PR introduces a new persistence and resume feature for task validation, adding significant new functionality including file I/O operations, new CLI arguments, and workflow changes. The scope and complexity of the new capability warrants human review.

^{You can customize Macroscope's approvability policy. Learn more.}