This issue is the roadmap. It carries the order and the reasoning; the milestones
carry the contents. Nothing is listed twice, so nothing can disagree with itself.
Where we are, 20 August
The page is live. /evals and /evals.md publish from a release rather than from
whatever ran last, so an automated run can no longer move the public numbers.
Two releases cut. v0.1.0 the baseline, v0.2.0 a correction: eight of the twenty
failures published in v0.1.0 turned out to be our own scoring, in two scenarios, both
cases of a scorer asserting how an outcome was reached rather than whether it was.
Corrected, frontier agents came out level at 3 failures of 30 each — which closed the
"nearly every failure is Codex" finding as not confirmed.
Milestones 1 and 3 are done. Credentials contained; the page shipped.
The order, and why it is this order
Reordered on 20 August. The page is live and the numbers behind it are trustworthy enough
to publish, so the work that matters now is what the benchmark is for: covering the
product properly, and settling whether the skills earn their place. Trust and publishing
work continues behind those.
1. Cover the product
Outpost carries a whole product on one scenario. Console has none. Two journey stages hold
two scenarios each against eleven for build.
First because the page is live: the Product pivot publishes 33% for Outpost on the
strength of a single scenario, which is the number most likely to be wrong in a way that
matters. A product score drawn from one test is not a product score, and a reader has no
way to know that from the page.
2. Prove the skills earn their place
Skills must improve the product or there is no point having them.
Currently measured at +1 for both frontier agents and −2 for the deliberately weak model —
so they help where the agent is strong and hurt where it is not, and nobody knows why. But
every one of those figures comes from single-attempt runs taken before 14 August, and Loop
1 showed that a single attempt cannot separate a result from variance. Two of the three
numbers already moved once when scenarios were found to be mis-scoring.
Settling it means both arms at three attempts on the current scorers — about $64, and the
run this milestone exists for. Worth saying in advance: if the answer is that skills are
neutral for frontier agents and negative for weaker ones, that is a finding and it gets
published. The alternative is shipping something because we built it.
3. Finish publishing
The page is live. The release helper is half built, and two gaps shipped knowingly: the
page never says what the paired columns mean (#29), and the deliberately weak model is
published as a low score with nothing marking it deliberate (#30). Both matter because the
page names vendors — and #29 is a prerequisite for anyone reading the milestone-2 result.
4. Trust the numbers
Substantially done. Three independent causes of moving results were found and handled:
fixed sleeps against asynchronous ingestion, a rubric that contradicted its own scenario,
and configuration readable before it is enforced.
What remains will not change today's figures but will change later ones — a submodule pin
that drifts silently (#26), and a project reset that adopts leftovers as pristine (#28).
5. Sustain
Cadence and budget, durable transcripts, cost reporting, and the product findings this
repository cannot close itself (#25).
What this order is not
It is not a schedule and has no dates. Milestones 4 through 6 can move in parallel where
they do not collide.
It is also not a commitment to do everything. An issue that stops being worth doing should
be closed with the reason — #3 was closed as a duplicate of #19, and #4 as not confirmed.
Keeping this current
Update this issue when the reasoning changes: a phase reordered, dropped, or finished.
Do not list individual issues here; that is what milestones are for, and a list kept in
two places goes stale in one of them. This repository has learned that three times.
This issue is the roadmap. It carries the order and the reasoning; the milestones
carry the contents. Nothing is listed twice, so nothing can disagree with itself.
.plans/delivery-plan.md— why the project isshaped this way
LOOPS.md— what the benchmark found and whether fixing itworked
Where we are, 20 August
The page is live.
/evalsand/evals.mdpublish from a release rather than fromwhatever ran last, so an automated run can no longer move the public numbers.
Two releases cut. v0.1.0 the baseline, v0.2.0 a correction: eight of the twenty
failures published in v0.1.0 turned out to be our own scoring, in two scenarios, both
cases of a scorer asserting how an outcome was reached rather than whether it was.
Corrected, frontier agents came out level at 3 failures of 30 each — which closed the
"nearly every failure is Codex" finding as not confirmed.
Milestones 1 and 3 are done. Credentials contained; the page shipped.
The order, and why it is this order
Reordered on 20 August. The page is live and the numbers behind it are trustworthy enough
to publish, so the work that matters now is what the benchmark is for: covering the
product properly, and settling whether the skills earn their place. Trust and publishing
work continues behind those.
1. Cover the product
Outpost carries a whole product on one scenario. Console has none. Two journey stages hold
two scenarios each against eleven for build.
First because the page is live: the Product pivot publishes 33% for Outpost on the
strength of a single scenario, which is the number most likely to be wrong in a way that
matters. A product score drawn from one test is not a product score, and a reader has no
way to know that from the page.
2. Prove the skills earn their place
Skills must improve the product or there is no point having them.
Currently measured at +1 for both frontier agents and −2 for the deliberately weak model —
so they help where the agent is strong and hurt where it is not, and nobody knows why. But
every one of those figures comes from single-attempt runs taken before 14 August, and Loop
1 showed that a single attempt cannot separate a result from variance. Two of the three
numbers already moved once when scenarios were found to be mis-scoring.
Settling it means both arms at three attempts on the current scorers — about $64, and the
run this milestone exists for. Worth saying in advance: if the answer is that skills are
neutral for frontier agents and negative for weaker ones, that is a finding and it gets
published. The alternative is shipping something because we built it.
3. Finish publishing
The page is live. The release helper is half built, and two gaps shipped knowingly: the
page never says what the paired columns mean (#29), and the deliberately weak model is
published as a low score with nothing marking it deliberate (#30). Both matter because the
page names vendors — and #29 is a prerequisite for anyone reading the milestone-2 result.
4. Trust the numbers
Substantially done. Three independent causes of moving results were found and handled:
fixed sleeps against asynchronous ingestion, a rubric that contradicted its own scenario,
and configuration readable before it is enforced.
What remains will not change today's figures but will change later ones — a submodule pin
that drifts silently (#26), and a project reset that adopts leftovers as pristine (#28).
5. Sustain
Cadence and budget, durable transcripts, cost reporting, and the product findings this
repository cannot close itself (#25).
What this order is not
It is not a schedule and has no dates. Milestones 4 through 6 can move in parallel where
they do not collide.
It is also not a commitment to do everything. An issue that stops being worth doing should
be closed with the reason — #3 was closed as a duplicate of #19, and #4 as not confirmed.
Keeping this current
Update this issue when the reasoning changes: a phase reordered, dropped, or finished.
Do not list individual issues here; that is what milestones are for, and a list kept in
two places goes stale in one of them. This repository has learned that three times.