test_qwen3_5_decode_graph_seam SEGFAULTs on main, and main's CI cannot report it because every run is cancelled

[localai-bot]

Found while running the operator gate for row ENG-EXPERT-STREAM-DEVICE (#1124, PR #1377). Filed in flow; it is not that row's defect.

What happens

test_qwen3_5_decode_graph_seam terminates with SIGSEGV. It is not a failed assertion, so no doctest output survives to name the case.

1/1 Test #101: test_qwen3_5_decode_graph_seam ...***Exception: SegFault  1.16 sec
0% tests passed, 1 tests failed out of 1

It is main's, and here is the control

I ran the branch and main in the SAME build directory with the SAME flags, changing only the checked-out source, so the toolchain, the configure and the object cache are held constant:

Tree	Result
row/ENG-EXPERT-STREAM-DEVICE-W0 at a1869e482	SEGFAULT, 16.48 s
origin/main at 96ed8346f	SEGFAULT, 1.16 s

Build of the control: rc 0, 0 error:/FAILED: lines, grep -c 'No space left on device' = 0. The branch does not touch tests/vllm/models/test_qwen3_5_decode_graph_seam.cpp.

The last commit to that test is 5f68e60df, feat(ENG-CUDAGRAPH-BREAK): widen graph eligibility to the step's actual query length, and key the spec ring so it cannot collide (#1374, #1020), which is on main. That is where to start, not a conclusion: I did not bisect, and the segfault may equally sit in the production code that commit widened the eligibility of.

Why it reached main

main's own CI never reports. Of the last 60 workflow runs in this repository, 30 are completed and every one of them is cancelled — zero success, zero failure — with the rest queued or pending. main's own runs are in that set. This is the condition #1285 already describes. A suite can therefore go red on main and no lane says so, which is how this one got here and how #1240 did before it.

Two secondary observations from the same run, recorded so they are not rediscovered:

• test_engine_core_proc failed under ctest -j4 at load average 97 and passes serially. A load artifact, not a defect.
• test_nemotron_h_paged_forward fails with No valid attention backend for device type 0 from {FLASH_ATTN: [head_size not supported]}. Its doctest summary reads assertions: 18 | 18 passed | 0 failed while 10 of 12 cases THREW — the assertions line cannot see a thrown case, so only the Status: FAILURE! line reports it. Whoever picks that up should grep Status:, not assertions:.

Reproduce

cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release
cmake --build build -j6 --target test_qwen3_5_decode_graph_seam
ctest --test-dir build -R '^test_qwen3_5_decode_graph_seam$' --output-on-failure

A first useful step is a debug build or ASan, since the release binary dies without naming a case. ASAN_OPTIONS plus the free-site line distinguishes a dangling temporary from a teardown race, which are the two shapes this tree has seen before.

[localai-bot]

Quantified, plus two concrete defects that reached main because of this.

The rate

Last 25 main runs of the ci workflow:

state	count
completed/cancelled	16
completed/success	3
in_progress	1
pending	1
queued	4

64% of main's CI runs are cancelled. The mechanism is ci.yml:122-124:

concurrency:
  group: ci-${{ github.event_name }}-…-${{ … || github.ref }}-${{ github.repository }}
  cancel-in-progress: ${{ github.event_name == 'pull_request' || github.event_name == 'push' }}

A push to main cancels the in-flight main run. On a repository landing several PRs an hour, main pushes supersede each other faster than a run completes, so main spends most of its time with no verdict at all — and a cancelled run is not a red, so nothing alerts.

What that cost, concretely, in one night

Every recent main commit is cancelled or queued:

pending    4712dac40   <- current main
cancelled  f07f96e1c
cancelled  9ecaf1bb3   <- #1392's own merge
queued     b537a5344
cancelled  d0eff4f25   <- #1443
cancelled  89261c955   <- #1408
queued     e67b2a4ba   <- #1427

Three defects reached main unverified through that gap:

1. #1371 — CPU_ATTN was a name in a candidate list with no registrar, so CPU had one dense backend and inherited FlashAttention's head_size % 8 constraint with no fallback. It reddened build-test-cpu and both sanitize-cpu jobs on every open PR until #1392 landed hours later.
2. #1415 carries a device-leakage regression — DSR REGRESSION in bucket 'vt_ifdef': 35 > baseline 32, three device-specific references added to the device-agnostic shared layer. Three independent review passes missed it, because CI never completed during any of them and every reviewer was reading a locally-green preflight.
3. Two tests now fail on a branch whose diff does not contain them: test_ltx2_video (touched by feat(LTX25-RESIDENCY-W0): a render now says where it spent its wall, and the phases sum to it (#1010) #1408 89261c955 and feat(LTX25-RESIDENCY-W0-LIVE): a render now says what it is doing while it does it (#1413) #1443 d0eff4f25) and test_resident_weight_host_addressable (introduced by feat(ENG-EXPERT-STREAM-DEVICE): W0f — the dense weights were resident twice, and --device cuda now decodes a 369.97 GiB checkpoint on a 119.631 GiB GB10 (#1299) #1427 e67b2a4ba). All three of those commits are on main; none has a completed verdict. I am not asserting main is red on them — that is precisely the claim this issue says cannot currently be made — only that they arrived via main merges and that nothing has adjudicated them.

Why this is more than an annoyance

A PR-lane red trains readers to baseline-subtract. When main has no baseline, "is this red mine?" becomes unanswerable, and the honest answer degrades into a habit of ignoring it. That is how the device-leakage regression above survived three reviews: everyone correctly identified the known reds and never saw the job that mattered.

Directions, not a prescription

• Exempt push-to-main from cancel-in-progress (keep it for pull_request, where superseding is what you want). Costs runner minutes; buys a verdict per commit.
• Or keep cancellation and add a scheduled main run whose result is the baseline, accepting that it lags.
• Either way #1316 matters: a run that executed zero jobs must not render as RED, or the baseline lies in the other direction.

Related: #1285, #1316, #584 (the windows-msvc jobs, which genuinely have no main baseline and are a different problem).