Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
dc442bb
feat(stage_router): corroborative tanh-scored routing with windowed s…
sabhatinas Jul 14, 2026
ad67a9c
feat(stage_router): efficient_first escalates to strong on any wrong …
sabhatinas Jul 15, 2026
0f0091f
feat(stage-router): window severity and no_progress over the recent w…
sabhatinas Jul 16, 2026
d132e2a
feat(dimension-collector): add trace-mined "file does not exist" erro…
sabhatinas Jul 17, 2026
e5348de
refactor(stage-router): two-axis scorer with exclusive spinning/explo…
sabhatinas Jul 17, 2026
7bddaa8
refactor(stage-router): drop no_error_streak; make exploring a full e…
sabhatinas Jul 17, 2026
060a8a6
feat(stage_router): escalate on context compaction + repeated-command…
sabhatinas Jul 20, 2026
7d68caa
feat(stage_router): optional tier-transition handoff notes (off by de…
sabhatinas Jul 20, 2026
5b78ff5
fix(stage_router): drop repeated_cmd_ratio from scorer (0/71 escalati…
sabhatinas Jul 20, 2026
08e33c8
feat(stage_router): stateless handoff notes + strong_system_prompt in…
sabhatinas Jul 24, 2026
d6bcb58
feat(stage_router): add weak_system_prompt injection for efficient tier
sabhatinas Jul 24, 2026
cee8419
style(stage-router): rustfmt tool_signals test assertions
sabhatinas Jul 22, 2026
5493bbc
fix(stage-router): restore configurable stateless de-escalation note
sabhatinas Jul 22, 2026
84534b5
docs(stage-router): update signals/calibration for corroborative scor…
sabhatinas Jul 23, 2026
18052fb
refactor(stage_router): move scorer+picker into a shared Rust core; p…
sabhatinas Jul 23, 2026
a67b94b
chore(benchmark): remove stray stage-router routing-profile configs f…
sabhatinas Jul 23, 2026
5fbe40b
chore: transfer changes from tool_signals to libsy
messiaen Jul 24, 2026
2f7ba88
chore: formating
messiaen Jul 24, 2026
d90b832
fix(ci): SPDX on score_staged_run, sync .pyi stub, narrow picker type…
sabhatinas Jul 24, 2026
f6aa522
docs(stage-router): sync docs + scorer skill to the Rust core (score_…
sabhatinas Jul 24, 2026
96fc5e3
refactor(stage_router): score via direct field math (drop string-keye…
sabhatinas Jul 24, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
94 changes: 94 additions & 0 deletions .agents/skills/switchyard-stage-router-scorer/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,94 @@
# Skill: switchyard-stage-router-scorer

**description**: Score benchmark run trajectories through the stage-router Rust scorer and picker, then visualise score distributions. Use when you want to replay trajectories through the picker, analyse routing splits, or compare score distributions across configs.

## Scripts

| Script | Purpose | Input | Output |
|--------|---------|-------|--------|
| `benchmark/score_staged_run.py` | Score a live run dir via real picker | run dir path | per-turn JSONL + per-task CSV |

## Quick Reference

```bash
# Score a run
uv run python benchmark/score_staged_run.py --run benchmark/tb_runs/<run-name>
# → /tmp/<run-name>-scores.jsonl (per turn)
# → /tmp/<run-name>-per-task.csv (per task)

# Custom threshold or window
uv run python benchmark/score_staged_run.py \
--run benchmark/tb_runs/<run-name> \
--threshold 0.15 --window 3
```

## Scoring pipeline (what score_staged_run.py does per turn)

```
trajectory step (tool_use + tool_result)
→ append to cumulative Anthropic messages list
→ ChatRequest.anthropic({"model": ..., "messages": messages}) # Rust binding
→ dc.process(ctx, request) # DimensionCollector — one per task, accumulates state
→ get_tool_result_signal(ctx) # read signal from ctx
→ stage_score_signal(signal) # raw (score, confidence) from the Rust scorer (for analysis)
→ pick_capable_first(ctx, threshold) # actual cf decision (same ctx, no re-process)
→ pick_efficient_first(ctx, threshold) # actual ef decision (same ctx, no re-process)
```

**Key:** `dc.process()` is called **once per turn** on a single ctx. Both pickers read the signal
already stored in that ctx — no duplicate processing.

**What the picker does beyond raw score:**
- **escalate** (`should_escalate`): `compacted` OR `severity >= 1.0` → force CAPABLE
- **de-escalate** (`should_deescalate`): `tests_passed AND recent_write+edit >= 1 AND severity <= 0` → force EFFICIENT
- `confidence < threshold` → fall_open to default tier (CAPABLE for cf, EFFICIENT for ef)
- Only when `confidence >= threshold`: route by score direction

## Per-turn JSONL schema

```json
{
"task_name": "terminal-bench/...", "trial_name": "...", "run_id": "...",
"reward": 1.0, "turn_depth": 5,
"score": -0.83, "confidence": 0.95,
"tool_name": "Bash", "is_error": false,
"write_count": 2, "edit_count": 1, "read_count": 3,
"no_error_streak": 4, "pure_bash_streak": 2, "tests_passed": false,
"pick_cf": 1,
"pick_ef": 0
}
```

`pick_cf` / `pick_ef`: `1` = CAPABLE (Opus), `0` = EFFICIENT (Nemotron)

## Per-task CSV columns

`run_id, task_name, reward, n_turns, mean_score, mean_confidence,
pct_strong_clear, pct_strong_uncertain, pct_weak_uncertain, pct_weak_clear,
opus_pct_cf, nemotron_pct_cf, opus_pct_ef, nemotron_pct_ef`

`opus_pct_cf` / `opus_pct_ef` are derived from actual `pick_cf` / `pick_ef` decisions, not score bands.

## Band definitions (for histogram colouring, threshold T)

| Band | Score range | cf default | ef default |
|------|-------------|------------|------------|
| strong_clear | ≥ T | Opus | Opus |
| strong_uncertain | (0, T) | Opus (fall_open) | Nemotron (fall_open) |
| weak_uncertain | (-T, 0) | Nemotron (fall_open) | Nemotron (fall_open) |
| weak_clear | ≤ -T | Nemotron | Nemotron |

Overrides can change any band. Use `pick_cf`/`pick_ef` for the true decision.

## Key findings (v0.2.0 baseline tarballs, T=0.20)

- Distribution is **highly bimodal** — most turns land in strong_clear or weak_clear
- `cf, t=0.20`: ~34% Opus on partial run; ~42% on full baseline
- `ef, t=0.20`: ~20% Opus on partial run; ~41% on full baseline
- Live routing split differs from shadow score — Nemotron trajectories are shorter and reshape the distribution

## Anti-patterns

- Don't call `dc.process()` multiple times per turn for different pickers — both pickers read from the same ctx. One `dc.process()` call per turn is correct.
- Don't infer routing split from score bands alone — overrides and fall_open change the actual decision. Always use `pick_cf` / `pick_ef`.
- Don't compare cost directly across configs: Nemotron has ~39% cache hit rate vs ~92% for Opus.
146 changes: 0 additions & 146 deletions benchmark/calibration/stage_router/calibrate.py

This file was deleted.

Loading
Loading