record(MODEL-DIFFUSION-LTX25): the first render on the full 21.004 B DiT, and the 1024x576 rung it could not fit (#1375)

.agents/benchmark-record.md

@@ -24755,3 +24755,86 @@ CUDA/ROCm/Vulkan is not covered. The counter sees every global `operator new`
 and nothing below it, so a raw `std::malloc` in the kernel is outside it (M-E
 above). No full-model render has been rerun, so nothing here says what such a
 run would now do — only that #1252 is not what stopped the last one.
+
+## MODEL-DIFFUSION-LTX25 FULL MODEL, the first render on the 21.004 B DiT, and the geometry ceiling beside it (2026-08-19, `row/MODEL-DIFFUSION-LTX25-FULLMODEL-RENDER`, binary `0a43a750` built from `7b9e207b1`, GB10, #1375 / #1252 / #1092)
+
+Every previous LTX-2.5 artifact was rendered on the DISTILLED arm. This is the
+first on the full `ltx-2.5-22b-dev-transformer-bf16.safetensors`: 42,018,190,584
+bytes, 4349 tensors, 21,004,025,600 elements, BF16 4059 / F32 290, zero
+quantisation scales. The checkpoint was verified semantically before use (header
+parses, tensor count matches, last tensor's `data_end` lands exactly on the file
+size), because a byte-count match alone does not prove a copy is whole.
+
+### The accepted measurement
+
+| | |
+|---|---|
+| Geometry | 768x448, 25 frames, 24 fps (1344 latent tokens) |
+| Wall | 2990 s (49.8 min), 2987 samples at 1 Hz |
+| Output | 25 frames + `audio.wav`, 1.01 s, 48 kHz stereo, plus `video.mp4` |
+| Own demand | 74.247 GiB above a 5.397 GiB baseline; peak system 79.644 GiB |
+| Min MemAvailable | 39.99 GiB against a 12.0 GiB hard floor |
+| GPU busy | 1010 of 2987 samples above 50% |
+
+Verifier verdict `PASS` on both arms, with the numbers rather than the verdict:
+25 distinct frame hashes of 25, zero near-uniform and zero near-black frames,
+adjacent-frame MAD 2.01 against far-frame (lag 12) MAD 9.34, a ratio of 0.215
+where uncorrelated noise gives ~1.0. Audio RMS -15.63 dBFS, envelope CV 0.112,
+spectral crest 52.0, active window fraction 1.0.
+
+### The rung that was refused, on the same lease
+
+1024x576/25f (2304 tokens) ran 3251 s and was refused by the governor rather than
+run to the wall: `first_dit = 481.5 s`, `per_forward ~ 162.0 s` from 7 resolved
+forward starts, so the recipe's fixed 60 forwards plus a 600 s tail project
+10803 s against that rung's 7153 s budget. The forward count is not tunable:
+`one_stage` at model version 2.5 resolves to 30 inference steps
+(`ltx2_pipeline.cpp:968` via `Ltx2Params24`), and `cfg_scale` 3.0 != 1.0 forces
+an unconditional forward per step (`ltx2_pipeline.cpp:521-523`), giving exactly
+2 forwards per step. So the ceiling here is geometry against lease length.
+
+### What this render is NOT
+
+It is not a full-fidelity reference render. `Ltx2DitForwardDevice` takes no
+`perturbations` argument, so the device-resident arm refuses the perturbed and
+isolated-modality passes by name (`ltx2_video.cpp:2738-2749`, #1092), while
+LTX-2.5's own params carry `stg_scale` 1.0 and `modality_scale` 3.0. The run used
+the engine's own sanctioned escape, `--video-stg-guidance-scale 0
+--audio-stg-guidance-scale 0 --a2v-guidance-scale 1.0 --v2a-guidance-scale 1.0`,
+which keeps real classifier-free guidance (video 3.0, audio 7.0) and gives up the
+STG term and the isolated-modality term. No oracle ran against it, so no parity
+claim is made and the speed axis stays `PENDING`.
+
+### Two instrument facts, recorded because both have already caused a wrong reading
+
+`gpu_edges = 0` means SATURATED, not idle. The counter looks for a GPU busy to
+idle transition, and the 1024x576 rung sat above 50% utilisation in 2704 of 3191
+samples, so there was no idle gap to sample. Reading that counter as "no work
+ran" is the available reading and the wrong one.
+
+`eu-stack` attaches inside the rc worker container and resolves ZERO frames, so
+every phase label came back `OTHER`. Phase attribution in this record is derived
+from the cpu%/rss signature instead: a single-threaded phase pins ~100% of one
+core of twenty with rss flat, weight loading ramps rss, and device staging shows
+above 800%. Do not budget on stack sampling in that container.
+
+### Why this was reachable at all
+
+Before #1252 the caption projection was a scalar triple loop with a `double`
+accumulator (`ltx2_text_encoder.cpp:61-70`) that could not reach an f32 FMA.
+Runs on the pre-fix binary measured single-core stretches of 1740 s and 1738 s,
+0.11% apart, reproducing the ~1731 s figure in #1087 twice in one render. After
+#1252 routed it through `vt::MatmulBT`, the same phase appears as `first_dit`
+481 s. That is the difference between a render that cannot fit a 4 h lease at any
+useful geometry and one that fits at 768x448.
+
+### Reproduce
+
+    rc run -d dgx:gpu0 --max-runtime 4h -- \
+      bash -lc 'BUDGET_S=14400 bash /workspace/ltx25-fullmodel/job/build_and_render.sh'
+
+The job builds `ltx2-gen` from `7b9e207b1` in-lease, refusing to proceed unless
+`vt::MatmulBT` is present in the unpacked source, then renders. Weights,
+`bin/` and `job/` are staged under `/workspace/ltx25-fullmodel` on the house NAS.
+Run evidence is `out/20260819T150230Z/` with `PROVENANCE`, `memguard.tsv`,
+`phases.tsv`, `verify.txt` and `render.log` per rung.

.agents/issue-index.md

@@ -451,3 +451,4 @@ rather than merged. `scripts/check-agent-record.py` gates both.
 | [#1370](https://github.com/mudler/vllm.cpp/issues/1370) | `ENG-HF-MODEL-DOWNLOAD` | `HubListRepoFiles`'s tree-listing size rule was evadable by LETTER CASE. `IsHexString` (`src/vllm/transformers_utils/hf_hub.cpp:30`) accepts `A`-`F` as well as `a`-`f`, mirroring llama.cpp's `is_valid_oid` (`common/hf-cache.cpp:161 @ b10451`), and `oid_owner` is a `std::map<std::string, OidOwner>` keyed on the RAW identifier with nothing folding case anywhere in the file. Two spellings of one identifier landed TWO keys and the rule compared nothing. MEASURED at `db0af37b1`: `a.bin` at 4096 bytes under `ab234567...` beside `b.bin` at 2048 bytes under `AB234567...` is **ACCEPTED**, where the same pair in one case is refused. This is [#1339](https://github.com/mudler/vllm.cpp/issues/1339) through a different door: `HF_ENDPOINT` is user configurable, so the listing is input the hub does not have to answer truthfully, and where one omitted `size` field bought an unconditional pass there, one changed letter case bought the same pass here. The raw-identifier assumption reaches PAST the owner map: `HfFile::oid` left the function in the listing's case and `HfBlobPath` (`src/vllm/transformers_utils/hf_cache.cpp:145`) makes that value a cache FILE NAME, so one listing would populate a different cache on a case-sensitive file system than on a case-insensitive one such as this project's CIFS checkpoint mounts. That half is latent, because `HfBlobPath` has no production caller until W3. `IsDegenerateOid` is unaffected, since folding a repeated character leaves it repeated. FIXED IN FLOW during the fourth fresh review of [#1282](https://github.com/mudler/vllm.cpp/pull/1282): the identifier is folded once, on the identifier ITSELF rather than only on the map key, after the hexadecimal form is validated and before either rule or `HfFile::oid` reads it. FOLDED rather than REFUSED, because hexadecimal is case-insensitive by definition and both git and the hub emit lower case, so an upper-case spelling is another spelling of one value and refusing it would reject a mirror over a difference that means nothing. Four cases RED first: the mixed-case pair at disagreeing sizes, the fold reaching `HfFile::oid`, the mixed-case pair at an agreed size still accepted, and an upper-case degenerate identifier still refused. Spec [`hf-model-download.md`](specs/hf-model-download.md) | bug |
 | [#1353](https://github.com/mudler/vllm.cpp/issues/1353) | `ENG-RECORD-ANCHOR-RATCHET` | A full disk makes `scripts/agent-preflight.sh` report record and policy defects that do not exist. Measured at `63ff58272` with 896M free of 447G: ten suites go red together — `test_check_release_binary_contract`, `test_release_postpublish_audit`, `test_check_container_matrix`, `test_release_index`, `test_release_metadata`, `test_release_accelerator_metadata`, `test_release_macos_metadata`, `test_release_windows_metadata`, `test_agent_role`, `test_agent_onboard` — and the trailing summary names all ten without mentioning the disk. The cause appears only inside one suite's captured output several screens up (`error: copy-fd: write returned: No space left on device`), interleaved with fixture failures that read exactly like findings (`ERROR: x: see (#157) and #174`, `ERROR: x: landed without a row PR`). All ten build a scratch git repository in a temporary directory, so all ten fail together and all ten fail toward a verdict about RECORDS rather than about the environment — the `.agents/verification.md` broken-instrument shape, and an agent reading it has every reason to start repairing records that were never broken. Reclaiming 2.7G of one row's own build tree and re-running the identical command turned all ten green with no tree change (`86 gates, 86 ok, 0 FAIL, 0 SKIP`, exit 0). The fix is a precondition, not a suppression: check free space before the suites that need a scratch repository and refuse naming the disk, exactly as `test_cpu_x86_llamacpp_floor` refuses to measure under contention ([#618](https://github.com/mudler/vllm.cpp/issues/618)). A gate that cannot run must say so rather than return a verdict. Found while landing [#1332](https://github.com/mudler/vllm.cpp/issues/1332) M0+M1 and NOT fixed in that flow, because it changes preflight semantics and adds a refusal path, which `AGENTS.md` routes to the normal row, spec and fresh-review path | bug |
 | [#1376](https://github.com/mudler/vllm.cpp/issues/1376) | `ENG-CUDAGRAPH-BREAK` | `main` was red on `tests/scripts/test_check_gate_commands.py`, measured at `601b576c6` in a detached worktree of `origin/main`: 8 failures of 44 tests, every one a comparison between the computed runnable population and `RUNNABLE_BASELINE`. `ENG-CUDAGRAPH-BREAK` was in the first and absent from the second. Cause: W5 of that row ([#1361](https://github.com/mudler/vllm.cpp/issues/1361)) filled its spec's `## Gates` section with runnable evidence, including a named test binary with its case and assertion counts and an exit status, which is exactly what moves a row into the runnable population. The ratchet's own error text instructs a re-pin in the SAME change, and the re-pin was not made. This is the growth case the ratchet exists to force a decision about, not a defect in that row's work. **It landed with no remote verdict**: the continuous integration lane that would have caught it independently has not executed for this repository since roughly 07:43Z on 19 August 2026, with runs queueing and none starting while GitHub reports Actions operational. FIXED IN FLOW while merging `origin/main` into `row/ENG-HF-MODEL-DOWNLOAD` for [#1280](https://github.com/mudler/vllm.cpp/issues/1280), because the fix is small and clear and a red `main` blocks every other row's gate. The entry is added with a justifying comment in the form the neighbouring entries use, no checker semantics change, and no test is weakened. After the re-pin the suite reports 45 tests OK and the audit reads 39 runnable of 119 gated rows | bug |
+| [#1375](https://github.com/mudler/vllm.cpp/issues/1375) | `MODEL-DIFFUSION-LTX25` | First end-to-end per-forward cost for the FULL 21.004 B LTX-2.5 DiT on GB10, measured on run `20260819T150230Z` with binary `0a43a750` built from [`7b9e207b1`](https://github.com/mudler/vllm.cpp/commit/7b9e207b1) (#1252). At 1024x576/25f (2304 latent tokens) the governor resolved **7 forward starts from the GPU busy/idle edge counter** and measured `per_forward ~162.0 s` with `first_dit = 481.5 s`, so the recipe's fixed 60 forwards (30 steps x 2 CFG legs, `ltx2_pipeline.cpp:521-529`) project **10 803 s against the rung's 7 153 s budget** and the rung was refused rather than run to the wall. The same lease then COMPLETED 768x448/25f (1344 tokens) in 2990 s, so the ceiling is geometry against lease length, not a defect. TWO instrument facts belong with the number, because both have already caused a wrong reading: `gpu_edges=0` means the GPU never went idle long enough to sample an edge (SATURATED), not that no work ran — this rung sampled 85% of 3191 samples above 50% utilisation; and `eu-stack` resolves no frames in the rc worker container, so phase attribution came from the cpu%/rss signature rather than from symbols. Owned by the LTX-2.5 row; spec [`ltx-2-5.md`](specs/ltx-2-5.md) | measurement |

docs/BENCHMARKS.md

@@ -519,6 +519,7 @@ built on it rather than keeping the flattering one.
 | vLLM 0.26 re-benchmark | Pending | Re-run the binding grids on the advanced pin |
 | MiniMax-H3 FP4 speed (W-FP4a) | **Measured GB10 (`row/H3-FP4-GPU-E2E`).** Marlin W4A16 byte-exact vs bf16; fp4 a memory win, 0.8x bf16/forward. Real-ckpt fp4-resident e2e RUNS (mp4/wav) | fp4 speed CLOSED. bf16-vs-quant A/B: ENCODER half MEASURED (§8.15), DiT half NOT (no bf16 render exists). Detail: benchmark-record + spec §8 |
 | LTX-2.5 axes | Speed `PENDING` (vllm-omni#6066 has no native 2.5), binding oracle too. **SIZE: 704x448/25f and 448x256/25f both COMPLETE on GB10 (4231 s, 3085 s)**; one run each, contended box, no oracle, no ceiling (#1088) | NOT the VAE decode (#1041/#1009): 39-100% of the ~1731 s phase is the caption projection (#1208), x86 671.8→78.4 s; LoRA fusion 17.78→0.124 s (#1202), add-back 59% (#1254). **#1286 REFUTED (#1317)**; alloc bound 464 MiB |
+| LTX-2.5 FULL 21.004B (`one_stage`, bf16 dev DiT) | **FIRST render on the full model: 768x448/25f on GB10 in 2990 s**, 25 frames + 1.01 s 48 kHz stereo, verifier PASS both arms. Binary `0a43a750` from `7b9e207b1`, run `20260819T150230Z` | Same lease REFUSED 1024x576: 162.0 s/forward x 60 projects 10803 s vs a 7153 s budget (#1375). NOT full fidelity: STG and isolated-modality OFF (#1092), real CFG only. Detail: benchmark-record |
 | MiniMax-Music3 (`MiniMaxMusic3ForConditionalGeneration`) | **Every axis vs the reference stays `PENDING`.** A PARTIAL device arm now exists (#672): the 8.6B LM and the 2.4B fp32 DiT run on the accelerator, so the rows below are internal two-arm numbers and NOT parity ratios | Denominator: SGLang-Omni `748a0b43` in its production configuration (both CUDA graphs, compiled DIT and DAV, batched seeded sampling) |
 | MiniMax-Music3 device arm, Jetson Thor sm_110 (#672) | `--device 1` vs `--device 0`, same request/seed, idle box: 2 AR frames **846.6 vs 835.1 s (1.014x SLOWER)**; 10 frames **1430.4 vs 1512.1 s (0.946x)**. Fit: **-11.65 s/frame, +34.8 s fixed** | A third duration (the fit has no residual), and moving the depth decoder + DiT + vocoder, which are 5 of 6 stages and still host scalar loops |
 | MiniMax-Music3 DiT device arm, `thor:gpu0` sm_110 (#672) | Per DiT forward at the capture's geometry, same binary/weights/inputs, idle box: **204.955 s host vs 0.186-0.187 s device, 1094-1102x** (1201x fitted). Staged ONCE (0.61 s; loop intercept 0.063 s). Whole process 3.5-4.5x | e2e song pair NOT runnable (host DiT alone ~37.6 h at 30 steps). Depth decoder/condition mix (bf16-storage), vocoder (no `ConvTranspose1d`) still host. Detail: benchmark-record |