Repository navigation
fix(engine): cap static-frame dedup verification at 60 s so it cannot outlast the first-frame stall - #7532
Conversation
… outlast the first-frame stall Long caption overlays plan thousands of verification screenshots, so the verification budget clamped at its 300 s ceiling in every render worker. That equals the producer's 300 s first-frame and session-init stall floor, so the stall always won and the render failed before its first frame. Verification fails closed on exhaustion (dedup is disabled for unverified runs and normal capture proceeds), so a lower ceiling costs only some frame reuse. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
bf7448b to
8c616e8
Compare
![heygengenesis[bot]](https://avatars.githubusercontent.com/in/2913576?s=60&v=4){kind=small}
There was a problem hiding this comment.
Verdict: APPROVE · 0 blocker · 0 important · 1 nit · head 8c616e80
- nit (
frameCapture.ts:3933): the logs showverification infrastructure failure, nottime_budget. The stall killed the page mid-verify before the 300 s budget fired, because the stall clock starts beforestartedAt(page load,computeStaticFrameSet,createStaticVerificationPage). That backs the fix: any ceiling near 300 s loses. Maybe say "well under" the 300 s stall in the comment.
Scope and the two open questions
Audited: verifyStaticFramesSafe, planStaticVerification, armStaticDedup, both test edits; producer FIRST_FRAME_STALL_MS / SESSION_INIT_STALL_MS.
Dedup savings at 60 s: planStaticVerification sorts runs by netSavings descending (frameCapture.ts:4047), and a budget exit keeps every fully verified run (:4252, :4417-4424). So 60 s drops only the lowest-savings tail. The budget only hits 60 s once planned shots × first shot > 40 s. Each worker also re-verifies the whole composition, so time past 60 s is paid N times per pod.
In-flight shot overrun: this predates the PR and doesn't block it. The budget is checked before each shot (:4237), and nothing times out a single seek+capture except CDP's protocolTimeout (300 s, config.ts:304). One hung shot can still reach the stall. A per-shot Promise.race against the remaining budget would close that, as a follow-up.
CI: the required workflows pass at this head. Windows render verification was still running when I checked. BLOCKED = review gate.
— genesis-v2
What
Lower
STATIC_VERIFY_CEILING_MSinframeCapture.tsfrom 300 s to 60 s.Why
Static-frame dedup verification runs inside
initializeSessionin every render worker. Its wall budget ismax(15 s, min(ceiling, 1.5 × plannedScreenshots × firstShotMs)). A long caption overlay plans thousands of verification screenshots, so the budget clamps at the 300 s ceiling, which equals the producer's 300 s first-frame and session-init stall floor. The stall always wins and the render fails before its first frame.Evidence (prod producer logs): a 38-minute one-word caption video failed twice, on two hosts, with
capture_disk stalled: no progress for 300000ms (stuck at 0)and all 6 workers insession_init. Each worker loggedstatic-frame dedup: disabled (verification infrastructure failure)after 298.7–299.7 s, at the moment the stall fired. Normal renders finish the pass in 1–2 s.Related work
Refs #7517, which widens the first-frame window to
max(300 s, min(900 s, 2 × probe start-up)). That may unblock these renders once deployed. This change is still worth making: the verifier would still spend 300 s in each of N workers on one pod, and the streaming path'sSESSION_INIT_STALL_MSis still a fixed 300 s.How
Only the ceiling and its comment change. Verification already fails closed on exhaustion (
time_budget: dedup is disabled for unverified runs and normal capture proceeds), so a lower ceiling costs only some frame reuse on very large compositions.Two test edits follow from the new ceiling: the wall-clock test now expects 8 shots (8 s each) instead of 38, and the "scales with planned work" test uses 500 ms shots so its planned work stays between the 15 s floor and the new ceiling.
Open question for the engine owners: the old comment says nothing was gained past 600 s on a 4K job, and the 300 s cap was a choice rather than a measurement. Is there data showing dedup savings that need 60–300 s of verification? If so, a ceiling tied to the stall floor (for example 120 s) is the fallback.
Release path: the producer image is built from hyperframes-internal, which pins
@hyperframes/producer. This needs an OSS release and an hfi bump before it reaches the render workers.Test plan
frameCapture-staticDedupVerifyDensity.test.ts: the edited wall-clock test failed on the unchanged source (expected 8 shots, got 38) and passes after; with the second test's edit reverted alone it fails (time_budgetinstead ofverified). 13 passed. oxlint, oxfmt and the engine typecheck are clean.🤖 Generated with Claude Code