main is RED on five CPU test suites, and a SPEC-ONLY PR is the control that proves it

[localai-bot]

build-test-cpu fails on every open PR with the same five tests, and the cleanest evidence is a pull request that changes no code at all.

The following tests FAILED:
	 60 - test_minimax_music3_ar (Failed)
	 71 - test_ltx2_video (Failed)
	 78 - test_ltx2_text_encoder (Failed)
	166 - test_muse_glimmer_text (Failed)
	167 - test_muse_glimmer_text_fallback (Failed)

sanitize-cpu (thread) adds a sixth: 257 - test_resident_weight_host_addressable.

Why this is main, not a branch

#1409 is spec-only. Its three-dot diff against the merge base is two files, both under .agents/, and zero files under src/, include/, tests/ or scripts/. It cannot break a C++ test. #1360 is an unrelated sampler fix. Both fail on exactly the same five tests, in the same order.

The before/after is sharper still. Before merging current main, #1409 failed on one test — test_nemotron_h_paged_forward, which is #1371 and is now fixed by #1392. After merging main, that one is gone and these five appeared. The only change on the branch was the merge.

Stated honestly: I have not built these suites on a pristine main worktree. This box is at load 120+ on 20 cores and a five-suite build there would be slow and itself unreliable. The spec-only PR is the control that carries the claim, not a direct run. Anyone with a quiet host should confirm by running them at origin/main directly — that is the first task here.

Candidate causes, by subsystem

The merge that surfaced them brought in ten main commits. Mapping tests to plausible owners:

failing test	plausible commit
test_ltx2_video, test_ltx2_text_encoder	d0eff4f25 (#1445), 89261c955 (#1408), 7f9c6802e — three LTX-2.5 commits
test_resident_weight_host_addressable	e67b2a4ba (#1427) — "the dense weights were resident twice", which introduced this test
test_minimax_music3_ar, test_muse_glimmer_text{,_fallback}	unattributed — these need bisecting

This is a mapping, not an attribution. Do not close any of those PRs' rows on the strength of this table; bisect.

How they got in

Every one of those commits landed without a completed CI verdict. Of the last 25 main runs of the ci workflow: 16 cancelled, 3 success. ci.yml:122-124 sets cancel-in-progress for push as well as pull_request, so on a repository landing several PRs an hour, main pushes supersede each other faster than a run finishes. A cancelled run is not a red, so nothing alerts. That mechanism is #1403, and this issue is its consequence rather than a separate cause — see also #1285 and #1316.

Why it matters beyond five tests

A red that every PR inherits trains reviewers to baseline-subtract, and a reviewer who has learned to ignore build-test-cpu will ignore it on the day it is genuinely theirs. That already happened this week: #1415 carries a real device-leakage regression — DSR REGRESSION in bucket 'vt_ifdef': 35 > baseline 32 — which three independent review passes missed, because CI never completed during any of them and everyone was reading a locally-green preflight.

What would close this

1. Run the five suites at origin/main on a quiet host and record the result — turning the inference above into a measurement.
2. Bisect each to its commit.
3. Fix or revert, per subsystem owner.

Until then no PR can be gated on build-test-cpu, because no one can answer "is this red mine?"

[localai-bot]

Measured, and this no longer holds. The five suites are GREEN on main.

Primary evidence — a workflow_dispatch CI run at main = 483cd3198, run 32464632413:

build-test-cpu: success -- 100% tests passed, 0 tests failed out of 584
  Test  #60: test_minimax_music3_ar .......... Passed   0.28 sec
  Test  #71: test_ltx2_video ................. Passed 201.56 sec
  Test  #78: test_ltx2_text_encoder .......... Passed   0.24 sec
  Test #170: test_muse_glimmer_text .......... Passed   0.07 sec
  Test #171: test_muse_glimmer_text_fallback . Passed   0.04 sec

workflow_dispatch was chosen deliberately: this box never offered a quiet window (load 16 to 138 all session, disk to 100%), and it is the one lane whose long jobs are not cancelled by the next push — the very problem in #1403 that let these land unverified.

The attribution in this issue was WRONG

The mapping table above guessed the three LTX-2.5 commits and e67b2a4ba. Verified, the fixes were:

suite	actually fixed by
music3_ar, ltx2_text_encoder, muse_glimmer_text{,_fallback}	5ba99ac4e VT-ACT-ROUND-POLARITY (#1458/#1461) — the bounds were wrong, not the kernel
ltx2_video, resident_weight_host_addressable	6b48edb2c GATE-CI-RED-REPAIR (#1499/#1494) — an 8-byte overrun in the test

An intermediate baseline at 5702d8f83 confirms the sequence: build-test-cpu was down to 1 failure of 583 before 6b48edb2c landed. I am recording the wrong guess rather than deleting it — the issue said explicitly it was a mapping and not an attribution, and this is what checking one costs.

What stands, and what does not

The diagnosis in this issue was right: a spec-only PR failing identically to an unrelated code PR did prove the cause was main. The specific failures are fixed. main moved 25+ commits while this sat behind the runner queue.

Local corroboration at 6b48edb2c: 5 of 6 passed, with test_resident_weight_host_addressable passing in 0.04s. Only test_ltx2_video failed locally, and that is this box rather than the code — it passes on CI in 201s against 2863s locally (14x), passes in isolation, and its failing assertions are wall-clock (span_slack 0.00322 <= 0.00025), swinging 171x between two samples on the same binary.

Closing as resolved by 5ba99ac4e and 6b48edb2c. The mechanism that let them land unverified — #1403 — is still open and is the thing worth fixing.