npm run skills:eval was passing 38 of 46 cases, with several model-invoked skills sitting at 0–33% trigger rates. Investigating showed the low scores were mostly bad eval cases rather than bad descriptions, and that the reasons why are not written down anywhere — so the next person to add a skill will rediscover them.
Two root causes, both invisible from the eval output alone:
- The harness runs each case with
--max-turns 1. The skill has to be the model's first tool call. A prompt phrased as "how does X work here" or naming a concrete file sends the model to Read/Grep first — a better answer to the question, scored as a miss. Every 0% case in the suite was one of these.
AGENTS.md is loaded in full on every turn. A prompt it already answers gets answered without any skill, so the case can never pass. Three committed cases were in this category, including one that had been at 0% across every run.
What this issue covers
- Replace the cases that measure the wrong thing, and expand every model-invoked skill from 3 to 5 positive cases (negatives kept), so a single flake cannot cross the threshold.
- Reshape the
local-dev description to the action-first "Use when …; when …" form that measurement keeps rewarding.
- Add a
docs/ page describing how to write a skill description that fires and eval cases that measure it, so this is a reference instead of a re-derivation.
Acceptance
RUNS=5 npm run skills:eval — the whole suite, not just the edited skills — at ≥80% on every case, with all negative cases clean.- Every model-invoked skill carries 5 positive cases plus its negatives, and no committed case is answerable from
AGENTS.md alone.
- A new
docs/ page captures everything this investigation established, linked from the README documentation table and from AGENTS.md's skill-maintenance rules, and covering at minimum:
- How the eval actually measures — fresh session,
--max-turns 1, negatives scored against every model-invoked skill in the repo, positives restricted to their own skill.
- The first-move rule: the skill must be the model's first tool call, so "how does X work here" and any prompt naming a concrete file lose the turn to
Read/Grep. With the measured before/after prompt pairs.
- The case shapes that reliably fire, and the ones that reliably do not.
- Why a case
AGENTS.md already answers can never pass, with the retired cases as worked examples.
- When a neighbouring skill fires and is arguably right, the case is wrong — not the description.
- Description shape: action-first, then
Use when …; when …; when ….
- The tuning loop — probe first (and probe a marginal prompt more than once), then measure; how much noise to expect at
RUNS=3 vs RUNS=5; and why this can never be a CI gate.
- A checklist for adding or editing a skill.
npm run skills:evalwas passing 38 of 46 cases, with several model-invoked skills sitting at 0–33% trigger rates. Investigating showed the low scores were mostly bad eval cases rather than bad descriptions, and that the reasons why are not written down anywhere — so the next person to add a skill will rediscover them.Two root causes, both invisible from the eval output alone:
--max-turns 1. The skill has to be the model's first tool call. A prompt phrased as "how does X work here" or naming a concrete file sends the model toRead/Grepfirst — a better answer to the question, scored as a miss. Every 0% case in the suite was one of these.AGENTS.mdis loaded in full on every turn. A prompt it already answers gets answered without any skill, so the case can never pass. Three committed cases were in this category, including one that had been at 0% across every run.What this issue covers
local-devdescription to the action-first "Use when …; when …" form that measurement keeps rewarding.docs/page describing how to write a skill description that fires and eval cases that measure it, so this is a reference instead of a re-derivation.Acceptance
RUNS=5 npm run skills:eval— the whole suite, not just the edited skills — at ≥80% on every case, with all negative cases clean.AGENTS.mdalone.docs/page captures everything this investigation established, linked from the README documentation table and fromAGENTS.md's skill-maintenance rules, and covering at minimum:--max-turns 1, negatives scored against every model-invoked skill in the repo, positives restricted to their own skill.Read/Grep. With the measured before/after prompt pairs.AGENTS.mdalready answers can never pass, with the retired cases as worked examples.Use when …; when …; when ….RUNS=3vsRUNS=5; and why this can never be a CI gate.