docs: tune skill eval cases and document how to write them

[cliffhall]

Closes #2202

npm run skills:eval was at 38/46, with several model-invoked skills measuring 0–33% on situations they exist to cover. The low scores turned out to be mostly bad eval cases rather than bad descriptions — and the reasons why were nowhere written down, so the next person to add a skill would have rediscovered them.

The two root causes

1. The harness runs each case with --max-turns 1, so the skill has to be the model's first tool call. A prompt phrased as "how does X work here", or one naming a concrete file or helper, sends the model to Read/Grep first — a better answer to the question, scored as a miss. Every 0% case in the suite was one of these. Probing the same questions reshaped confirms it:

Prompt	First tool call
"How do TUI components get mounted and keypresses driven in this repo's tests?"	Bash
"How do I write a test for a TUI component in this repo?"	Skill ✅
"Which part of this repo owns the browser's HTTP transport and the backend it talks to?"	Bash
"Does browser-only transport code belong in core or in the web client?"	Skill ✅
"Which parts of the gate only ever run locally and never in CI?"	Read
"The gate failed at the lint stage. How do I work out what is wrong?"	Skill ✅

2. AGENTS.md is loaded in full on every turn, so a case it already answers can never pass. Three committed cases were in that category — including "Which package owns the OAuth secret storage backends?", whose answer is a verbatim line of the directory tree, and which had measured 0% across every run. I then wrote a fourth one myself mid-investigation ("What should I run before I push?", 40%), which is fair evidence the trap is easy to fall into even while documenting it.

Changes

• Eval cases — every model-invoked skill goes from 3 to 5 positives (negatives kept), so a single flake can no longer cross the 0.8 threshold. Cases that measured AGENTS.md or invited a Read are replaced with "how do I…", "does X belong in A or B", "<stage> failed, how do I diagnose it" shapes.
• local-dev description reshaped to the action-first Use when …; when … form. project-structure's reshape was reverted: it measured no improvement, so it was 110 characters of listing budget for nothing.
• docs/skill-authoring.md (new) — how the eval measures, the first-move rule, the shapes that fire and the ones that cannot, why an AGENTS.md-answerable case is unpassable, the probe-then-measure loop, and a checklist. Linked from the README table and from AGENTS.md's skill-maintenance rules.
• AGENTS.md — listing-cost figure refreshed (2,834 → 3,144 of 4,000, after the test-servers reshape below) and a pointer to the new page.

Two smaller findings, also captured

• Probe a marginal case more than once. A prompt can fire on a single probe and still measure 60% over five runs.
• When a neighbouring skill fires and is arguably right, the case is wrong, not the description. "How do I get an issue I already filed onto the board?" fired board-ops 3/3 as an issue-create case — board-ops is the correct answer, so the case was replaced rather than the description broadened onto another skill's ground.

Verification

RUNS=5 npm run skills:eval — 63/63 at or above threshold, 60 of them at 100%, all 18 negative cases clean, script exits 0 (was 38/46).

The three that sat at 80% — local-dev's dependency-resync case, pr-flow's Copilot-review case, and project-structure's Node-backend placement case — are revisited in the follow-up below.

npm run local:gate passes (GATE_EXIT=0).

No UI change, so no screenshots.

---

Follow-up (commit a344225e)

Chasing the three cases that had been sitting at 80% turned up a fourth failure mode, plus one gap in the harness.

Partial overlap with AGENTS.md reads as flakiness, not as a miss

A case AGENTS.md answers outright sits at 0% and is easy to spot. A case it answers half of oscillates — and looks like eval noise rather than a defect in the case:

Case	Scores across runs	The overlap
"I added a dependency and the TUI bundle broke at import time. Where should it have gone?"	33 / 67 / 100 / 100 / 60%	Dependency placement is stated in full in AGENTS.md; only the diagnosis is the skill's
"I just pulled and a client's dependencies look out of sync. What do I need to run?"	80 / 80 / 60%	AGENTS.md says a single root npm install is all you need

Both stabilised at 100% once re-aimed at what only the skill holds — "what do I check" instead of "where should it have gone", and the worktree install trap instead of the install command. docs/skill-authoring.md now records this as its own section, with the rule: if a case keeps moving between runs while its neighbours hold steady, suspect the case before you suspect the noise.

test-servers reshaped, and testing pointed at it

test-servers kept its noun-phrase description ("Covers the stale-build hazard"). Reshaped to the action-first Use when …; when … form that worked elsewhere, and its stale-build case — which AGENTS.md partly answers — re-aimed at "I need a real server to exercise this change against by hand."

Since integration and smoke tests here drive a real server rather than a mock, testing now carries an explicit pointer to /test-servers for anything needing one — that is a procedure testing does not hold.

This is what moves the listing from 2,921 to 3,144 of 4,000 characters, and AGENTS.md's recorded figure is updated in the same commit per its own maintenance rule.

skills:eval accepts several skill names

Iterating on two related skills meant either one skill at a time or the full suite. npm run skills:eval -- testing test-servers now works.

An unknown name is a hard error rather than an empty run — a typo would otherwise enqueue zero cases and report a green 0/0, which reads exactly like a clean pass of the skill you meant. Two new tests in scripts/skill-eval.test.mjs cover the multi-name form and that error; the single-name form is unchanged.

Narrowing never narrows what a negative case is scored against — a focused run still fails a negative that fires any model-invoked skill, since "no skill of ours fired" is the property being asserted. Documented alongside a warning that a focused run cannot see displacement, so the full suite is still what you take before pushing.

Verification (this round)

RUNS=5 npm run skills:eval — 63/63 at 100%, including all 18 negatives clean and all three previously-80% cases. No displacement of untouched skills.

npm run local:gate — passes, all 34 stages (GATE_EXIT=0).

One note for anyone re-running the gate locally: with FORCE_COLOR set in the environment, six TUI tests fail on toContain assertions because Ink emits accelerator-key underline escapes inside the matched word (Info renders as \e[4mI\e[24mnfo). Unrelated to this PR — the same six pass with FORCE_COLOR=0, and CI has no TTY — but it costs a confusing half hour if you hit it.

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.