fix(api,k6): ride out chain read-after-write races that flake the staging deploy gate

[GTC6244]

HOLD

I think I may have merged a second branch into main before the first one completed - this also matches the results of the investigation... and by Occum's Razor ...

Why

Three consecutive Deploy Staging runs on main were rolled back by k6-correctness failures, leaving staging stuck on 2c972ff3 (v1.1.10-58):

Run	Commit	Failure
29880089230	6e98d2b	POST /new_account → 500 Simulation failed: NoAccountAccess (1 of 4 iterations)
29880104213	d40b01d	POST /update_action_metadata → 400 Cannot update action with hash 0x0
29882923105	e79f93a	every request EOF — cold box unreachable ~6 min after boot

The Jul 20 run (pre-dating all three commits) failed with the identical NoAccountAccess error, so failures 1–2 are chronic flakes, not regressions from those commits. Both reduce to the same root cause: reads racing just-mined writes on the load-balanced RPC.

What

new_account 500 (real user-facing bug, not just a gate flake): the newAccount receipt proves the account exists, but send_transaction dry-runs the follow-up registerWalletDerivation via eth_call, which the provider can serve from a node that hasn't executed that block yet — reverting NoAccountAccess for an account that was just mined. newAccount itself never reverts NoAccountAccess (checked WritesFacet.sol), so at this call site the revert can only be the stale-read window. Retry it (4 attempts, 250 ms exponential backoff); any other error still fails immediately.

hash 0x0 400: integration.spec.ts took hashed_cid from a list_actions read issued immediately after add_action/add_action_to_group; a stale read returned the action without a usable id and the spec forwarded 0x0. Both list reads now poll until the action is visible with a non-zero id (up to 5 attempts, 2 s apart — worst case ~8 s per list, paid only while the read is actually stale; the fast path costs nothing), and the spec fails explicitly if it never appears.

Not addressed: the EOF in run 29882923105

The cold box (chipotle-next-rep-sa6xj, prod2) passed wait-for-api/attestation/openapi/smoke at minute ~2 after boot and was fully unreachable by minute ~7. Runs 1–2 stopped their cold box before minute 5, so they can't rule a dies-after-boot defect in or out. The ping-pong rollback worked as designed (domain restored to ...-rep-w4hdl, bad box stopped). The next deploy is the second data point — if it EOFs again ~6 min after boot, the new build has a genuine post-boot crash and the box's phala logs need a look before it's stopped.

Testing

• cargo +1.91 check / cargo +1.91 test --lib account_management / cargo +1.91 fmt on lit-api-server
• k6 inspect k6/correctness/integration.spec.ts parses clean

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This PR mitigates staging deploy flakes (and a user-facing POST /new_account failure mode) caused by RPC read-after-write propagation lag by adding bounded retries/polling around immediately-following reads/simulations after on-chain writes.

Changes:

• Add bounded retry + exponential backoff in new_account for registerWalletDerivation when the dry-run hits a transient NoAccountAccess due to stale RPC reads.
• Update k6 correctness integration to poll list_actions until the newly-added action is visible with a non-zero id before proceeding.
• Document the fix in CHANGELOG.md.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.

File	Description
lit-api-server/src/core/account_management.rs	Retries registerWalletDerivation on transient NoAccountAccess to ride out RPC stale-read windows after newAccount.
k6/correctness/integration.spec.ts	Polls list_actions until the newly added action appears with a usable (non-zero) id to avoid forwarding 0x0.
CHANGELOG.md	Adds a release note describing the POST /new_account mitigation.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[GTC6244]

Good catch on the arithmetic — the description said "5 attempts / 2 s" which read as a ~2 s total bound; the real worst case is 4 retries × 2 s ≈ 8 s per list. I've corrected the PR description rather than the code: the sleep only happens while the read is actually stale (the fast path does zero sleeps), 2 s matches Base's block cadence so polling faster would just double the requests without converging sooner, and the alternative to waiting up to ~8 s here is failing the gate and paying a full ~30-minute deploy → rollback cycle.

[GTC6244]

Update after merging origin/main (7d5f769): main's #593 landed a complementary fix for the same flake, so the conflict resolution combines both — this account-level poll is retained and is now also the source of hashedCid (per #593 it only depends on metadata propagation, not group membership), still guarded by the non-zero-id check. Worst-case bound unchanged: ≤4 retries × 2 s, paid only while the read is stale.

[GTC6244]

Same resolution as the account-level comment above: the code's ~8 s worst case is intentional and only paid while the read is stale; the PR description's "~2 s" framing was the error and has been corrected. In practice 10a's polling absorbs most of the propagation lag, so this group-level loop almost always hits on the first try.

[GTC6244]

Update after merging origin/main (7d5f769): with #593 merged, this group-level list is no longer the source of hashedCid (that now comes from the account-level list above) — this poll remains purely as the visibility assertion that the action shows up in the group. Same bound as 10a; in practice 10a's polling absorbs the lag so this loop hits on the first try.

[GTC6244]

Agreed — reworded in c3e9745. The entry now describes the retry behavior ("now retries (with backoff) the intermittent 500 … riding out the typical propagation window") instead of promising the 500 can never recur.

[GTC6244]

Done in c3e9745: the retry decision + backoff is factored into register_wallet_derivation_retry_backoff(attempt, error_msg) -> Option, and two unit tests pin the policy — the doubling 250/500/1000 ms schedule with stop-at-budget (4 = REGISTER_WALLET_DERIVATION_ATTEMPTS), and no-retry for non-stale-read errors (contract InvalidRequest, nonce failures, on-chain reverts, empty message).