A leftover from a crashed run becomes permanent project state

[leggetter]

benchmark-alerting-001-delivery-alerts has moved more than any other scenario, in a
pattern that does not look like agent variance, and none of the three flake mechanisms
found so far can account for it.

The history

Snapshot	Score	Failing
13 Aug	4/6	mini-no-skills, 5.6-no-skills
17 Aug	0/4	claude, mini, mini-no-skills, 5.6
17 Aug	1/6	the above plus 5.6-no-skills
current	1/6	everything except claude-no-skills

Note the 13 August set and the 17 August set are almost inverted: the two arms failing
first are among those passing later, and vice versa. A scenario that gets harder for
everyone at once is not agents behaving differently.

Why the known mechanisms do not explain it

• Not the ingestion race (Scenarios flake, and single-attempt runs publish the flake as a result #14). This scorer contains no setTimeout and no
  waitFor — it never had a fixed sleep, because it reads configuration rather than
  sending traffic and reading it back. The polling conversion did not touch it.
• Not the judge (The LLM judge samples at default temperature, so judge checks disagree with themselves #22). Every check here is deterministic; there is no judged check to
  flip.
• Not rule propagation (No signal for when a configuration change is in force #25). Nothing is sent through the gateway.

What the failure actually says

The failing check is the first one, and its note is specific:

created an alert for delivery failures
  -> only the project default triggers are present

So the scorer finds no agent-created alert at all — not a misconfigured one. Five of six
agents are apparently creating nothing, having previously created something.

That points away from the agents and towards one of:

• the project's alert configuration is shared state that a reset does not restore, so an
  earlier run's teardown leaves later runs unable to create or detect one
• what counts as a "project default" changed, so agent-created alerts are now being
  classified as defaults and skipped
• alerts are per-project rather than per-connection in a way the scenario assumes they
  are not, making concurrent or sequential runs interfere

The first is the most likely and the most concerning: FixedProjectSource resets to
pristine, and a reset that does not fully restore alert state would produce exactly this —
fine on an early run, progressively worse afterwards, and inverted between arms depending
on run order.

Why this matters beyond one scenario

alerting-001 is currently 1/6 on the published page. If that is a reset defect rather
than agent capability, we are publishing a near-total failure that agents did not earn,
against named vendors.

It is also the scenario that made #4 look like a Codex problem — it fails for both
frontier agents, and a scenario both frontier agents fail is a statement about the
scenario or the product, not a vendor.

How to investigate cheaply

score-only --eval benchmark-alerting-001-delivery-alerts --repeat 5 with a
SOLUTION.ts that creates the alert. If the solution's alert is not visible to the
scorer, the defect is in the scorer or the reset and no agent run is needed to prove it.

Follows from #4, which is closed as not confirmed.

[leggetter]

Cause found, and it is two defects

1. The scorer asserted implementation rather than outcome

It counted only delivery triggers whose created_at fell after the project lease. An
agent that repaired an existing trigger scored zero, because updating does not change
created_at.

The project held an orphaned delivery trigger, it_N3Rae22hh3Ngyh, created 7 August:
enabled, email channel, final_attempt strategy, scoped to connection
web_D9LvYE3SWoVr — which had since been deleted. So alerting looked configured in the
dashboard and could never fire. Claude Code found it and said so:

Found the actual root cause: a delivery issue trigger named
fulfilment-delivery-failures already existed with email notifications enabled and the
right strategy (final_attempt). But it was pointed at connection web_jpM7PJIrA9CX,
which had been deleted. The trigger was silently pointing at nothing — that's why
nobody was told.

I updated the trigger (it_N3Rae22hh3Ngyh) to point at the current connection.

That is the best answer any agent has given in this scenario, and it failed. It
repaired silent alerting instead of leaving a broken trigger in place and adding a
duplicate beside it — which is what the scorer was rewarding.

This also explains the shape of the collapse. The scenario gets harder as agents get
better at inspecting existing state before acting. 4/6 to 1/6 was not agents degrading;
it was more of them finding the orphan and doing the right thing with it.

Fixed: the scorer now asks whether an alert exists, is enabled, covers the connection and
can notify, regardless of who created it or when.

2. The pristine snapshot canonises leftovers

issue-triggers is in RESOURCE_KINDS, so the reset does collect them. The orphan
survived anyway, because loadOrCaptureSnapshot captures whatever exists the first time
it runs as pristine, and CI has no persisted snapshot so it recaptures every run.

The orphan existed on 7 August. Every capture since has listed it as pristine, so every
reset has preserved it. Anything left behind by a crashed or interrupted run becomes
permanent project furniture the next time a snapshot is captured, and nothing will ever
collect it.

The orphan is now deleted, so the next capture is clean. The mechanism is not fixed, and
it will do this again: a run that dies between creating a resource and releasing the
project leaves something that the following run adopts as pristine.

Worth its own issue. A snapshot committed to the repository, or captured once against a
known-good project and pinned, would remove the self-reinforcing part.

Re-running

alerting-001 at three attempts across all six experiments to establish the corrected
figure. The 1/6 currently published is not a measurement of agent capability.

[leggetter]

Retitled and relabelled, because the original title is now false.

It read "alerting-001 collapsed from 4/6 to 1/6 and no scenario change explains it". A scenario change did explain it: the scorer counted only triggers created during the run, so an agent that repaired an orphaned one scored zero. That is fixed, the orphan is deleted, and the scenario re-ran 6/6.

What remains is the mechanism that let the orphan survive, which is a different thing and the reason this stays open: loadOrCaptureSnapshot captures whatever exists the first time it runs as pristine, and CI has no persisted snapshot so it recaptures every run. Anything a crashed run leaves behind is adopted as project furniture by the next one, and nothing will ever collect it.

Relabelled harness from finding. It is a defect in our own instrument, not something found about Hookdeck — and finding feeds the page's counters, where it would have inflated the findings count with one of our bugs.