main is RED on build-test-cpu (test_qwen27_dense_forward), and main's own CI never reports because every run is CANCELLED

[localai-bot]

Two defects, and the second is why the first went unnoticed.

1. main fails build-test-cpu

test_qwen27_dense_forward throws, 1 of 9 cases, on build-test-cpu, build-newest-gcc and both sanitize-cpu legs:

tests/vllm/models/test_qwen27_dense_forward.cpp:280: ERROR: test case THREW exception:
vt: qwen3_5 dense: `layer.linear_attn.in_proj_qkv.weight_scale_inv` is present, which is
the block-wise (fine-grained) FP8 scale, but the checkpoint`s quantization_config declares
no weight_block_size. The tensors and the config disagree and there is no block geometry
to read the scale with; refusing rather than guessing 128x128
  at src/vllm/model_executor/models/qwen3_5_dense_weights.cpp:428
[doctest] test cases: 9 | 8 passed | 1 failed | 0 skipped

The refusal itself looks correct — a fixture appears to present weight_scale_inv without a weight_block_size, and the loader is right to refuse rather than guess 128x128. What is wrong is that it fires in a test on main.

Attribution, not assumption: 4ee5f4a69 (fix(FIX-PROBE-CANNOT-SAY-NO), #1258/#1267) is the last commit to touch both qwen3_5_dense_weights.cpp and test_qwen27_dense_forward.cpp, and it is an ancestor of main. Preceding block-FP8 work: 281b4bc76 (#1189/#1256) and 09597106e (#1189).

How it surfaced: PR #1081 is records-only. git diff origin/main...HEAD --name-only | grep -cE "^(src|include|tests|scripts|CMakeLists)" returns 0 — it touches no build input at all, yet inherits this failure. A records PR cannot break a C++ test, so the failure is mains.

2. main CI never reports, so nobody sees it

Every recent main run is cancelled:

6e99e4540 cancelled   32b32bb14 cancelled   836c13c35 cancelled   45a4a71d3 cancelled

ci.ymls concurrency group cancels in-progress runs on push, and main is receiving pushes faster than a run completes (cuda-fat-build alone is 45-90 minutes). So mains own status is never established, and a red like this is only discovered when some unrelated PR inherits it.

That is the more expensive defect. It means main can be red for an unbounded time with no signal, and every PR author then has to prove the failure is not theirs — which costs a full investigation each time.

Worth considering: make the push-to-main lane non-cancellable (the schedule/workflow_dispatch events already keep their own partition through the event_name token), or add a periodic baseline run whose job is to answer "is main green right now". Either change is a CI-semantics change and owes its own spec and red-before per AGENTS.md.

Not fixed in flow

The repair belongs to the row that owns the block-wise FP8 loader path, and diagnosing whether the fixture or the loader is wrong needs that context. Filing rather than guessing at someone elses fixture.

[localai-bot]

Sharper diagnosis, static only — I have not built it, and the fix belongs to the row that owns this path.

The test is not failing where it looks

The throw is reported at test_qwen27_dense_forward.cpp:280, but the enclosing case starts at :189, TEST_CASE("qwen27 loader packs GDN in_proj_ba in exact b,a row order"). Line 280 is its last assertion:

// A per-output-channel scale read as per-tensor would be silently wrong, so
// any other element count is REJECTED rather than reinterpreted.
const float bad_scale[2] = {2.0F, 2.0F};
add("a_scale", "F32", {2}, bad_scale, sizeof(bad_scale));
CHECK_THROWS_WITH_AS(LoadMergedBf16RawNK(get, {"b", "a"}),
                     doctest::Contains("per-tensor or one value per output row"),
                     std::runtime_error);

The case expects a throw, and gets the WRONG one. It wants "per-tensor or one value per output row"; it receives the block-FP8 refusal from qwen3_5_dense_weights.cpp:428. CHECK_THROWS_WITH_AS fails on the message mismatch, and doctest reports it as the case throwing.

So this is not "a fixture is missing weight_block_size". It is an ordering change: the block-FP8 check now fires before the per-output-channel scale-shape check that this assertion is aiming at.

Why the ordering moved

4ee5f4a69 deleted the inline probe and moved it to a shared seam:

-  const TensorExists has = [&get](const std::string& name) {
-    try { (void)get(name); return true; }
-    catch (const std::exception&) { return false; }
-  };
+  // FIX-PROBE-CANNOT-SAY-NO (#1258): the `try`/`catch` this used to spell out
+  // now lives ONCE in `dense_loaders`, because the sibling seam below needs the
+  // identical probe and two copies of a subtle thing is the next defect.

Consolidating it was right, and the commit's own title says the point was a probe that can prove it says no. The question for whoever picks this up is whether the shared probe answers has("...weight_scale_inv") the same way the inline one did for this fixture — the fixture adds a_scale, not weight_scale_inv, so a probe that returns true too readily would make IsFp8BlockProjection refuse on a checkpoint that has no block quant and no weight_scale_inv at all.

What that means for the repair

Two candidate shapes, and they are not equivalent:

1. The probe over-answers — then the defect is in the shared seam and every caller inherits it, which is more serious than one test.
2. The check order changed — then IsFp8BlockProjection is being consulted earlier than the scale-shape validation, and the fix is ordering, not the probe.

Distinguishing them is one build: assert what has("layer.linear_attn.in_proj_qkv.weight_scale_inv") returns for this fixture. I have not run it, so I am not asserting which.

The main-CI-never-reports half of this issue stands unchanged and is the more expensive of the two.

[localai-bot]

Defect 1 (main RED on test_qwen27_dense_forward) does not reproduce on current main 8f64ff2ad:

[doctest] test cases:  12 |  12 passed | 0 failed | 0 skipped
[doctest] assertions: 606 | 606 passed | 0 failed |
[doctest] Status: SUCCESS!

The issue reported 9 cases with 1 throwing; the suite now has 12 and asserts 606 times, so it changed after filing. The likely fix is 281b4bc76 (feat(MODEL-FP8-BLOCK-LINEAR): the block-wise FP8 linear method and the dense forward that reaches it, #1189/#1256), which implements the path the refusal was guarding — verified as an ancestor of HEAD. The test reads no CHECKPOINT_ROOT, so this is not a skip wearing a pass.

Scope of what I checked, so nobody over-reads it: one Release CPU build on x86-64, no sanitizers. The issue also reported the failure on build-newest-gcc and both sanitize-cpu legs, and I have verified neither. Those remain unverified rather than clear.

Defect 2 is unchanged and is why this stays open. Every ci run on main is still cancelled — 8f64ff2ad, 6e99e4540, 32b32bb14, 836c13c35, 45a4a71d3 all completed/cancelled, with no completed run among them. The mechanism is .github/workflows/ci.yml:83:

cancel-in-progress: ${{ github.event_name == 'pull_request' || github.event_name == 'push' }}

Every merge to main is a push against the same github.ref, so each merge cancels its predecessor's run. On a busy day nothing on main ever completes. Keeping cancel-in-progress for pull_request and dropping it for push is the narrow fix.

Two consequences worth recording: gh pr checks renders a cancelled run as fail, so this also manufactures phantom failures (7 of them on #1238 — attest, drift-check, manifest, plan, promote, publish, verify — all cancellations, no real failure). And it means defect 1 could have been fixed at any point since filing without anyone being told.

[localai-bot]

Defect 1 is already fixed on main, and the attribution in the comment above is inverted. Measured, not inherited.

Defect 1 does not reproduce

Built origin/main at 5c8671c50 in a clean worktree, CI's own configure line, and ran the two ctest entries bare:

./build/tests/test_qwen27_dense_forward                 -> RC=0
[doctest] test cases:  12 |  12 passed | 0 failed | 0 skipped
[doctest] assertions: 606 | 606 passed | 0 failed |
[doctest] Status: SUCCESS!

VT_GLUE_FUSE=0 ./build/tests/test_qwen27_dense_forward  -> RC=0
[doctest] test cases:  12 |  12 passed | 0 failed | 0 skipped
[doctest] assertions: 606 | 606 passed | 0 failed |

606 assertions over 12 cases, so this is not assertions: 0 wearing a pass, and the suite reads no CHECKPOINT_ROOT.

The report line was read as an assertion, and it is a declaration

:280 is not "the last assertion of the case at :189". At the RED SHA ae581da3e437, line 280 is the TEST_CASE line:

159:TEST_CASE("qwen27 loader routing: IsQwen27QuantizedLinear encodes 3.6")
187:TEST_CASE("qwen27 loader packs GDN in_proj_ba in exact b,a row order")
280:TEST_CASE("qwen27 GDN loader retains one merged BA owner and no split copies")

doctest reports an uncaught throw at the case's declaration line. The CI log agrees: the line above the ERROR is TEST CASE: qwen27 GDN loader retains one merged BA owner and no split copies. So there was never a CHECK_THROWS_WITH_AS message mismatch — the case threw outright, and the "wrong exception" reading is an artifact of mapping a declaration line onto the nearest assertion.

4ee5f4a69 is the cure, not the cause

It postdates both the RED and the GREEN:

SHA	authored	role
39692b2bb	08-18 10:47	baseline GREEN
09597106e	08-18 15:32	introduced it (#1189 M3)
ae581da3e	08-18 18:11	baseline RED, build-test-cpu + both sanitize-cpu
281b4bc76	08-18 21:25	fixed it (#1189 M4 / #1256)
74a0ae420	08-18 22:31	baseline GREEN again
4ee5f4a69	08-18 22:49	hardening (#1258), after the lane was already green

git log ae581da3e437..74a0ae4204eb over the three implicated paths returns exactly one commit: 281b4bc76.

Of the two candidate shapes, it was the probe over-answering, never an ordering change. LoadQwen3_5DenseGdn built its probe as [](const std::string&) { return true; }, so IsFp8BlockProjection asked has(proj + ".weight_scale_inv"), was told yes, and refused a disagreement it had invented. 281b4bc76 replaced it with a resolver-backed probe; 4ee5f4a69 then moved that to dense_loaders::ProbeThroughResolver and added CheckProbeCanAnswerNo at three seams so a probe that cannot say NO is refused by name.

The refusal is intact and still covered

qwen3_5_dense_weights.cpp:429 still refuses, and tests/vllm/model_executor/models/test_fp8_block_weight_load.cpp:520 reaches it through a real checkpoint load (TempCheckpoint -> LoadDense), not a hand-built object.

Four of the five named jobs were never still red

Per-job conclusions for the newest completed baseline run (32183400023, 74a0ae420):

success  build-test-cpu
success  sanitize-cpu (address,undefined)
success  sanitize-cpu (thread)
failure  build-newest-gcc          <- #1296, PR #1297 open
failure  windows-msvc-cpu          <- #584/#968
failure  windows-msvc-vulkan       <- #584/#968
skipped  documentation-checkpoint

documentation-checkpoint was not red; ci.yml:436 excludes it from the schedule/workflow_dispatch lane by design, because those payloads have no github.event.before. Reading main-baseline.py's not run: line as a failure is a trap worth naming.

Defect 2 stands, and this changes nothing about it

Half 2 is why this took a full investigation to answer: no run has completed on main since 4ee5f4a69, so the recovery was as invisible as the breakage. Do not read this comment as closing defect 2. It still owns its own spec and red-before, and it currently has no owner.

Full suite, and the mutations that make the above causal rather than correlational

ctest over the whole tree at 5c8671c50, CI's configure line, clean build:

100% tests passed, 0 tests failed out of 485
 86/485 Test #89: test_qwen27_dense_forward ............... Passed
 87/485 Test #90: test_qwen27_dense_forward_glue_fuse_off . Passed

Three mutations in the worktree, each printing compile_err and git diff --stat beside its result so a mutation that failed to build or never applied could not read as a pass. Tree restored byte-for-byte after each; git status --porcelain empty at the end, and both suites re-verified green (606|606, 102|102).

A — restore the always-true stub and drop the CheckProbeCanAnswerNo guard. Reproduces the reported failure exactly:

compile_err=0  test_rc=1
[doctest] test cases: 12 | 11 passed | 1 failed
tests/vllm/models/test_qwen27_dense_forward.cpp:282: ERROR: test case THREW exception:
vt: qwen3_5 dense: 'layer.linear_attn.in_proj_qkv.weight_scale_inv' is present, ...

That is the causal proof of shape 1: the probe over-answered.

B — restore the stub but keep 4ee5f4a69's guard. The failure changes from blaming a tensor that does not exist to naming the real defect:

vt: dense loader: LoadQwen3_5DenseGdn was given a tensor-presence probe that
answered YES for '__vllm_cpp__a_tensor_no_checkpoint_carries__'

C — delete the :429 refusal (VT_CHECK(!has_scale_inv, -> VT_CHECK(true,). test_fp8_block_weight_load goes red at :529-530, so the refusal this issue worried about is still reachable and still gated:

compile_err=0  test_rc=1
[doctest] test cases: 7 | 6 passed | 1 failed | assertions: 102 | 100 passed | 2 failed

Scope of what I did not verify

x86-64 Linux only, no sanitizer build, no Windows and no gcc 16. build-newest-gcc is #1296 and PR #1297 is still open; the two windows-msvc-* failures are #584/#968. Those three are unrelated to this issue and remain red on their own tickets.

[localai-bot]

Retraction: my earlier analysis in this thread is WRONG on three counts

A fresh implementer was dispatched on it, found there was nothing to fix, and disproved my reasoning. Recording that here because a wrong diagnosis left in an issue is worse than no diagnosis — the next reader has no way to tell it from the parts that held.

1. I named the wrong commit, and got the direction backwards. I wrote that 4ee5f4a69 (FIX-PROBE-CANNOT-SAY-NO) was the likely cause. It is the cure, and it postdates both the RED and the GREEN:

09597106e  15:32  introduced
ae581da3e  18:11  RED baseline
281b4bc76  21:25  FIXED here
74a0ae420  22:31  GREEN baseline
4ee5f4a69  22:49  hardening, after the fact

git log ae581da3e..74a0ae420 over the three paths returns exactly one commit: 281b4bc76.

2. My "the test expects one exception and gets the wrong one" reading was an artifact. I claimed :280 was the last assertion of the case beginning at :189. At the RED SHA, :280 is itself a TEST_CASE declaration — "qwen27 GDN loader retains one merged BA owner and no split copies". doctest reports an uncaught throw at the case declaration line, not at the assertion. There was no CHECK_THROWS_WITH_AS message mismatch. The whole "ordering changed" hypothesis followed from mapping a declaration line onto the nearest assertion above it.

3. documentation-checkpoint was never red. ci.yml:436 excludes it from the schedule lane by design. I read main-baseline.pys not run: line as a failure.

What the defect actually was

The probe over-answered — my candidate shape 1, proven causally rather than by correlation. LoadQwen3_5DenseGdn built its probe as [](const std::string&) { return true; }, so IsFp8BlockProjection asked has(proj + ".weight_scale_inv"), was told yes, and refused a disagreement it had invented. Already fixed by 281b4bc76.

Current state, measured

At origin/main 5c8671c50, clean build, CI configure line, run bare:

100% tests passed, 0 tests failed out of 485
test_qwen27_dense_forward: 12 cases | 606 assertions | SUCCESS

Non-zero assertion count, reads no CHECKPOINT_ROOT — not a skip wearing a pass. The :428 refusal is still reachable, proven by mutation: VT_CHECK(!has_scale_inv, -> VT_CHECK(true, turns test_fp8_block_weight_load.cpp:529-530 red through a real checkpoint load.

The remaining red on main is build-newest-gcc only (#1296, PR #1297 open), plus the inherited windows-msvc-* (#584/#968).

The half of this issue that still stands, and is still unowned

mains CI never reports — every push cancels its predecessor, so no run completes. That is why the breakage AND the recovery were both invisible: no CI run has completed on main since 4ee5f4a69. My three errors above are downstream of the same thing — I was reasoning from a PR that inherited main, because main itself had no verdict to read.

That half is untouched, needs its own spec as a CI-semantics change, and is the more expensive defect of the two.