fix(LTX25-TEXT-LINEAR-SEAM): the caption projection is one core for 671 s per pass, so route it through vt::MatmulBT - #1252
Merged
Conversation
…e routing the caption projection through vt #1208's own second comment withdrew the attribution its first comment implied: two single-core stretches 0.11% apart do not fit a projection whose two calls have a 2:1 size ratio. This spec settles that from the tree rather than from the trace. The pair is the guided render's two text-conditioning passes, both of which run on a hard-coded CPU queue; each pass contains both projections, so the 2:1 ratio lives inside a stretch and cannot split one. The row count is the constant 1024, not a prompt length, which collapses the comment's 0.34-5.4 GMAC/s band to one number, and the Gemma-4 tower runs on the vt threadpool and so cannot be single-core time at all. It also records the accumulator decision the issue asks for, and says why the file header's existing argument for f64 is true and still not decisive. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
…71 s per pass, so route it through vt::MatmulBT `Linear` in the LTX-2.5 text tower was a scalar triple loop that widened both operands to `double` per multiply. It could not reach an f32 FMA, and no thread but the caller's ever entered it. The attribution came first, because #1208's own second comment withdrew it: two single-core stretches 0.11% apart do not fit a projection whose two calls sit in a 2:1 size ratio. They do not have to. The pair is the guided render's two text-conditioning passes -- ltx2_video.cpp:2085 and :2799 each build a CPU queue and call Ltx2EncodePromptToConditioning, so the tower runs on the host whatever --device asked for -- and each pass contains both projections, which puts the 2:1 ratio inside a stretch where it cannot split one. The row count is not a prompt length either: out.seq is max_length and that is the constant 1024, so the projection costs 1.1838e12 MACs on every render, which collapses the comment's 0.34-5.4 GMAC/s band to 0.681 GMAC/s. Measured here at the shipped geometry the old loop takes 671.777 s at 99% of one CPU, and that is ONE WHOLE pass containing BOTH project() calls, so it must not be doubled. Converting it to a GB10 wall needs a per-core ratio between the two boxes that this row could not measure, so it is bounded rather than assumed: at least 1, because a GB10 Arm core does not out-run a 5.7 GHz-class Zen 5 on a scalar double-widening loop, and at most 2.587, because the projection runs inside the stretch and cannot exceed it. The projection is therefore 39% to 100% of each stretch and 0 to 1066 s per stretch is still unattributed. The buffer-work hypothesis for that residual is refuted by measurement rather than left open: at the shipped geometry the stack is 0.839-0.910 s, the V2 norm 0.632-0.688 s and both scaled copies 0.390-0.468 s, so 1.861-2.066 s in total -- 0.28% of the pass, with the two GEMMs at 99.7%. The only remaining candidate of the right magnitude is the U8/NVFP4 caption weights becoming the f32 4.6 GB, and this row does not claim it. Through the seam the same call takes 78.421 s on about fourteen cores of twenty, 8.57x. The full-size scaled copy inside project() is kept unchanged on both arms; it is 0.06% of the before arm and 0.5-0.6% of the after arm, so removing it belongs to whoever takes the memory format. The accumulator becomes f32 because that is what F.linear does and what the goldens beside this file were produced by, not because f32 is better: vLLM never registers LTX-2.5 and vLLM-Omni is unpinned and ungateable, so the oracle is Lightricks ltx_core itself, executed in torch.float32 by scripts/gen-ltx2-text-goldens.py at fd4ded7f2, and F.linear on f32 tensors is an addmm onto an f32 sgemm with no f64 anywhere. The header's old argument that f64 lands nearer torch's blocked sum is true, and it is rewritten rather than deleted, because closer to exact is not closer to what upstream computes. Measured at K = 188160, f32 sequential costs 2.12e-05 max absolute error where f64 costs 5.88e-08 -- about 330x below one bf16 ulp of the value it produces. Ten of twelve golden numbers are byte-unchanged; the two that moved sit 84x and 134x inside the file's 1e-5 bound and no tolerance was widened. The new case runs at flat = 8192 and requires the projection to equal vt::MatmulBT plus bias byte for byte. It carries its own discrimination proof, so it cannot pass vacuously: 96/96 elements differed before the change, 0/96 after, and an f64 reference sits 4.11e-06 away where it sat 5.28e-08 away before. Still owed, and named rather than left to be discovered: the per-core ratio and with it 0-1066 s of each stretch, which one rc lease on dgx:gpu0 would close; no GB10 number at all, so the LTX-2.5 speed axis stays PENDING; the seam is memory-bound at ~1.0 GMAC/s per busy core, with K-blocking named as the next hypothesis and no ceiling declared; the trace's third single-core stretch; and the text tower's hard-coded CPU queue, found while attributing this row. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
… in the spec rather than in the reader's head `## Now` still described the defect in the present tense on a row that has landed, and `## 8. Stop conditions` listed two conditions without saying whether either fired. Both are answered here. The rate condition did NOT trigger: 1.7622 GMAC/s locally is consistent with the trace's implied 0.681 for any per-core ratio in [1.0, 2.587], and both ends of that interval are closed by argument rather than assumed, so the fix belongs here and the residual is carried as owed. The goldens condition DID trigger and is decided here: two of twelve numbers moved, both on the V1 arm, 84x and 134x inside the file's 1e-5 bound. No tolerance was changed. Writing that down is the difference between a decision and a number a later reader has to re-derive. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
…as, which the first guard dropped The early return added with the seam covered `in_features == 0` alongside `rows == 0` and `out_features == 0`, and those are not the same case. With no rows and no output columns there is nothing to write, so returning the empty vector is what the old loop did. With a zero-width reduction there IS something to write: the loop seeded its accumulator with the bias and skipped the inner loop, so every output was the bias alone, and the guard returned zeros instead. Skipping only the GEMM and still running the bias add reproduces the old result exactly. Nothing reaches this: RequireDeclaredProjection pins `in_features` to `embedding_dim * num_layers` and both are refused at zero. It is corrected anyway, because a replacement that is behaviour-preserving only where the tests happen to look is the kind of difference that surfaces later as a shape nobody predicted, and the reason it cannot be reached now belongs beside the code rather than in a reviewer's head. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
…nal gate `origin/main` advanced while this branch was under review preparation. The branch is already pushed, so a rebase would need a force-push and this merge is the correct shape instead. `git merge-tree` reports the merge clean, and this branch appends no `.agents/issue-index.md` row, so the union-driver hazard that makes an append read CONFLICTING on GitHub does not apply here. The full gate is rerun on the merge result rather than inherited from either parent. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
localai-bot
pushed a commit
that referenced
this pull request
Aug 18, 2026
`origin/main` moved from `ae581da3e` to `1dac4f9a7` while this row was in review, and the five new commits touch four of the same record files. None of them touches the depth schedule, the music3 sources, or this row's spec section, so this merge carries records only and no code. `docs/STATUS.md` conflicted on two adjacent rows, because #1214 and #1252 rewrote the LTX-2.5 and MiniMax-Music3 rows while this branch held older text for both. Main's complete version is taken here, byte for byte, including its new music3 wording. This row's own claim is added back in the commit that owns it rather than inside a merge resolution, so the record edit and the measurement it describes stay in one reviewable commit. `.agents/benchmark-record.md` and `.agents/issue-index.md` were rebuilt the way both files require: main's version verbatim, then this row's entry and its two index rows appended at the end. Main's text is an exact prefix of each result, verified by comparison rather than by reading. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
This was referenced Aug 18, 2026
localai-bot
pushed a commit
that referenced
this pull request
Aug 19, 2026
…gressions, and its review proved all three (#1286) The fresh review of `39b34c0c8` returned nine findings and no PASS. The refutation of #1286 is confirmed and unchanged; what failed review was the gate that landed beside it. Three of the reviewer's mutations wrote an obvious memory regression into `cpu_ops.cpp` and passed **13 of 13 assertions**. **The bound admitted a 2x per-worker regression while its comment said it could not.** `Bound(n)` was `8 MiB + n * 2 * kTileBytes` — a per-thread term of literally two tiles, under a comment claiming it was "not wide enough to admit a second tile-sized buffer per worker". Mutation M-B added exactly that and doubled the growth at every worker count, green. At the shipped geometry that is 232 -> 464 MiB shipping silently. The bound is now `1 MiB + n * kTileBytes`, which is the property the file actually claims, over a fixed term measured at **64 bytes**. The comment now names the worker counts that detect a doubling (4 and up) instead of claiming all of them do. **The counter was bypassed by an over-aligned `operator new`.** Only the non-aligned overloads were replaced, so the C++17 `std::align_val_t` family fell through to the library. M-D delivered the whole-activation defect through a 64-byte-aligned `operator new` — the natural shape for a SIMD scratch buffer in this kernel — and read byte-identical growth, green. All eight aligned `new`/`delete` overloads are now replaced. **`std::malloc` is seen too, through the LINKER rather than a symbol definition.** M-E delivered the same defect through `std::malloc` and the first pass of this repair left it green, narrowing the claim instead. That was the wrong call. `-Wl,--wrap=malloc` and four siblings, scoped to this one target, redirect the calls made by the objects in this link — `libvllm.a`, and so `cpu_ops.cpp` — without defining `malloc`, so AddressSanitizer's interceptor keeps its symbol and there is no second strong definition to collide with. **M-E now reds at 1,086,324,800 bytes.** `free` is deliberately not wrapped, so the C-allocator figure is bytes REQUESTED cumulatively: it errs toward red, and it reads exactly zero on a clean run. A new COVERAGE case makes the whole claim executable — it allocates through plain `new`, array `new`, over-aligned `new` at two alignments, and `std::malloc`, and requires the counter to move by the amount asked for. Two failure modes, two mechanisms: dropping a `--wrap` flag fails the LINK (`undefined reference to '__real_malloc'`), and the subtler "route present but folded away" mode is what the assertion catches. That case calls through a `volatile` function pointer for a measured reason — a direct `std::malloc` in the test's own translation unit is folded by the compiler and never becomes a symbol reference, so it is never wrapped. Measured in a standalone probe: same-TU calls gave `malloc=0 calloc=0 posix_memalign=1`, the identical calls from a separate TU gave `malloc=1 calloc=1 posix_memalign=1`. Product code is always the separate-TU case. **`MatmulOneChunk<false>` had no assertion behind it.** `<false>` and `<true>` are separate instantiations with separate `thread_local` buffers, and the gate called only `vt::MatmulBT` on a non-repacked weight. `vt::Matmul` and `MatmulBTKernel`'s `elem_kn_repacked` lever both route to `<false>`, and `vt::Matmul` is #1259's `Ltx2FuseLoraIntoTensor` path — merged and unexercised on the full model. Both arms are measured now, and the measurement confirms the two buffers are distinct: 8 fresh workers read 33,554,496 bytes on `MatmulBT` and 33,554,496 again on the `vt::Matmul` immediately after, eight tiles both times rather than eight then seven. The retention figure is therefore **per instantiation** — up to 464 MiB, not 232 MiB. **The chunk-row-span defect was invisible at every geometry the gate ran.** M-A (`nrows = ir1_end - iir1`) produced byte-identical growth, green. Replaying `MatmulChunked`'s grid arithmetic over all twelve shapes the sweeps use gives `dr1` = 16 at eleven and 8 at the twelfth — never above 16, which is a coincidence of the chosen shapes rather than a property of the kernel. The collapse to one chunk per thread is LIVE and the shipped default, so a case now runs at `rows = 128, n = 16, nth = 4`, a live non-NUMA shape where it gives `dr1 = 32`. One correction to that finding as written. The review called it "a real ~3x regression the gate cannot see on a NUMA host", citing `dr1 = 52` at `rows = 1024, nth = 20`. `IsNuma()` is `constexpr false` here (`cpu_threadpool.h:74`, NUMA unported per `:25`), so that branch is dead today and the gap is reachable through the live collapse instead. And at the shipped caption projection the weight is the longer axis (`n = 4096 > rows = 1024`), so the collapse takes `nchunk1 = 1` and `dr1 = 1024`, not 52 — 770 MB per worker and 15.4 GB across 20. The finding was right that the gate was blind; the magnitude is larger and the trigger is nearer. The mutation table. Every row rebuilt and re-run; `sha256(cpu_ops.cpp) = dc39ecc...4f1f3` before and after each. | mutation | `BUILT`/`compile_err` | `39b34c0c8` | after | |---|---|---|---| | M-A, chunk row span | `YES`/0 | green 13/13 | exit 1, 2 of 36 failed | | M-A2, whole activation | `YES`/0 | red | exit 1, 14 of 36 failed | | M-B, second tile per worker | `YES`/0 | green 13/13 | exit 1, 10 of 36 failed | | M-D, whole activation via aligned `new` | `YES`/0 | green 13/13 | exit 1, 14 of 36 failed | | M-E, whole activation via `std::malloc` | `YES`/0 | green 13/13 | exit 1, 14 of 38 failed | | M-W, the `--wrap=malloc` flag removed | **`NO`**/1 | n/a | link fails, `undefined reference to '__real_malloc'` | Green after the repair: **5 cases, 38 assertions, 0 failed, exit 0**, against 2 cases and 13 assertions before it. The other mutation rows are M-A (2 of 38 failed), M-A2 (14), M-B (10) and M-D (14), each `BUILT=YES compile_err=0` with `git diff --stat` recorded, and `sha256(cpu_ops.cpp)` restored to `dc39ecc...4f1f3` after every one. The `sanitize-cpu` risk was exercised in two parts, and the second was blocked. The aligned-`operator new` widening ran in the lane's own configuration (`VLLM_CPP_SANITIZE='address,undefined'`, `VLLM_CPP_CUDA=OFF`, `UBSAN_OPTIONS=print_stacktrace=1`): 5 cases, 36 assertions, 0 failed, exit 0, no diagnostic; `thread` likewise, with ASLR disabled to work around this box's `FATAL: unexpected memory mapping`. Rebuilding that lane after `--wrap` was added hit `No space left on device` — other sessions filled the shared disk to 100% — so it was NOT re-run with `--wrap` in place, and this says so rather than implying otherwise. In its place, a standalone probe reproducing the whole mechanism (replaced plain and aligned `operator new`/`delete`, the five `--wrap` redirections, and a separate TU standing in for `cpu_ops.cpp`) built and ran clean under `-fsanitize=address,undefined`: all three C-allocator routes intercepted, alignment preserved, no diagnostic, exit 0. That establishes the mechanism against ASan; it is not a full-lane run. Records repaired in the same change. `cpu_ops.cpp:125` is cited for `static thread_local std::vector<float> af`, which is at **line 130** — `:125` is `const int64_t blck_1 = 16;`. Fixed in the spec, the benchmark-record and the test header. It cannot be fixed in `.agents/issue-index.md:423`, which is append-only and keyed, so the correction is recorded in the spec's `## Owed` and in this body instead; a corrective row would be a duplicate key. The uncovered class behind it is recorded too: `check-symbol-anchors.py:87-90` requires a literal `::`, so of 613 `path::Symbol` and **14,448 `path:line`** citations in this tree's markdown, the second number is checked by nothing. `tests/CMakeLists.txt` justified the Linux-only guard with "the measurement is /proc/self/{status,clear_refs}". The shipped test opens nothing under `/proc`. The real reason is now stated: the gate replaces the global allocation functions for the whole program, so it needs every `delete` in the binary to reach the matching replaced `delete` and its fixed term calibrated against the library doing the allocating — both established for glibc/libstdc++ and nothing else. Three record corrections beside them. The spec's `## Tests to port` and `## Risks/decisions` describe the refuted `VmHWM` instrument, so both now point forward to `## Outcome` W4. `## Outcome` W2's first table was headed "at the shipped geometry" while running `rows = 64`, breaking the spec's own rule at `:158-162`; the reduction is beside the number now. And the "threaded stretch sits at `anon = 34.30`" triple was that stretch's minima at a single sample: recomputed from the retained `memguard.tsv`, the stretch is a **ramp** — `anon` 34.280 -> 42.593, `used` 97.350 -> 105.853 over 236 samples — while the pre-#1252 run genuinely is flat (`anon` 34.283-34.441 over 1676). The two coincide only at the ramp's first sample. W1's argument is peak-minus-own-`t=0` and does not rest on it. `docs/BENCHMARKS.md` and `docs/STATUS.md` carry the new keyed projection, which `check-doc-checkpoint.py` requires of a benchmark-record change: the seam's allocation is bounded on both instantiations, and the process-lifetime retention is 464 MiB rather than 232. No product code changed. `src/vt/cpu/cpu_ops.cpp` is byte-identical to `39b34c0c8`; every mutation above was applied to a scratch tree and restored to the same sha256. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
localai-bot
added a commit
that referenced
this pull request
Aug 19, 2026
…e, and the seam had no memory bound at all (#1286) (#1331) fix(LTX25-TEXT-LINEAR-MEM): #1252's +26 GiB is the box, not the change, and the seam had no memory bound at all [#1286](#1286) reported that routing the LTX-2.5 caption projection through `vt::MatmulBT` ([#1252](#1252)) raised peak host memory from ~79 GiB to 105.85 GiB on a 119 GiB GB10 and aborted a full-model render before any denoise step. It named three candidates — the shared threadpool's per-worker tiles, the call site's full-size `scaled` copy, or an interaction — and said plainly that the attribution was not measured. **It is none of the three.** The attribution came first, as the issue asked, and it changed what this row lands. ## The +26 GiB is the machine, measured on the retained evidence `runguard.py:236-237,260` fixes what the compared column is: `used_gib` is the **system-wide** `MemTotal - MemAvailable`, `anon_gib` is system-wide `AnonPages`, and only `rss_gib` belongs to the child. Both runs are retained under `/mnt/nas_share/rc/ltx25-fullmodel/out/`, which is what makes this answerable at all. | | pre-#1252, `1024x576-25f/` | #1252, `20260818T220620Z/1024x576-25f/` | |---|---:|---:| | `used_gib` at `t=0` | **4.741** | **31.553** | | `avail_gib` at `t=0` | 114.890 | 88.078 | | peak `used_gib` | 79.206 at t=1867.4 s, `cpu=1885.5%` | 105.853 at t=199.2 s, `cpu=1925.2%` | The box carried **26.812 GiB** before `ltx2-gen` was started in the second run. The reported regression is `105.853 - 79.206 = ` **26.647 GiB**. They agree to **0.165 GiB**. Subtract each run's own baseline and the two are the same binary's worth of memory, on three independent columns: | axis | pre-#1252 | #1252 | delta | |---|---:|---:|---:| | `used_gib` peak minus own `t=0` | **74.465** | **74.300** | **-0.165** | | `anon_gib` peak minus own `t=0` | 38.012 | 38.217 | +0.205 | | child `rss_gib` at the peak sample | 41.952 | 42.090 | +0.138 | Both peaks were sampled inside a ~1900% CPU stretch, so this is the same phase class rather than two different ones. On a box as clean as the first run's, the #1252 binary's own 74.300 GiB would have left **40.59 GiB** available — above the 12 GiB hard floor and the 8 GiB projection floor, and in line with the "never below ~40 GiB" the pre-#1252 runs showed. **The guard fired correctly on a real condition; the condition was not the binary.** Two readings from the same evidence, recorded because each on its own would have been read as supporting #1286. During its single-core projection (`cpu ~ 100%`, t=124-1801 s, 1676 samples) the pre-#1252 run is **dead flat** — `anon` 34.283-34.441, `rss` 33.484-33.570, `used` 74.405-74.573 — while the #1252 run's threaded stretch (`cpu > 500%`, t=137.1-385.5 s, 236 samples) is a **ramp** rather than a plateau: `anon` climbs 34.280 → 42.593, `rss` 30.322 → 42.120, `used` 97.350 → 105.853. The two coincide on `anon` at the ramp's FIRST SAMPLE (34.280 against 34.283) and nowhere after it, so that near-identity is where the ramp starts rather than a matching steady state; `used` is 23 GiB apart there and 26.6 GiB apart at the peak. The argument above is peak-minus-own-`t=0` and does not rest on any single sample. And **neither run ever loaded the DiT** (`dit_runs=0` in both; the first was stopped from outside at t=3017.3 s, `# TERMINAL signal=15`), so neither peak is a whole-render peak and neither is presented as one. Filed as [#1317](#1317), with both halves named: the harness compares absolute peaks of a system-wide column without recording the starting occupancy, and something held 26.8 GiB on `dgx:gpu0` before the job began. Neither is in this repository, so neither is fixed in flow. ## Measured locally too, before and after, with the throughput beside it Same probe source and flags on both arms, only the `Linear` body differing; peak RSS is `VmHWM` read before and after the call with every operand already allocated and touched. 20-core Zen 5 under KVM, Release, `-ffp-contract=off`, shipped `K = 188160` and `out_features = 4096`, `rows = 64` (**reduced** from the shipped 1024 to keep the single-threaded arm inside a sensible wall; the tile term the sweep isolates is independent of rows by construction). Box **not idle**, loadavg 5.4-13.3, three replicates, median: | arm (`rows = 64`, reduced) | wall | rate | peak-RSS growth | |---|---:|---:|---:| | before, scalar `double` loop | **28.488 s** | 1.732 GMAC/s | **1.0 MiB** | | after, `vt::MatmulBT` | **3.378 s** | 14.60 GMAC/s | **232 MiB** | **8.43x, and +231 MiB** — the speedup reproduces #1252's 8.57x, and the memory cost is **0.85% of the 26.6 GiB #1286 attributes to the change**, 117x too small to be it. Swept over threadpool width at the same reduced `rows = 64` geometry: | workers (`rows = 64`, reduced) | 1 | 2 | 4 | 8 | 16 | 20 | |---|---:|---:|---:|---:|---:|---:| | peak-RSS growth (MiB) | 12.9 | 24.7 | 47.8 | 93.8 | 186.1 | **232.1** | | per worker (MiB) | 12.9 | 12.3 | 12.0 | 11.7 | 11.6 | 11.6 | That is `cpu_ops.cpp:130`'s `static thread_local std::vector<float> af` and nothing else: sized by ggml's 16-row `blck_1`, so `16 x 188160 x 4 = 11.48 MiB` per worker. The measurement lands on the model, and the output checksum is **byte-identical across all six widths**, which is the dispatch determinism contract holding. At the **full shipped geometry**, `rows = 1024`, both projections, default 20 workers — the box was heavily loaded (loadavg 15.9 then 28.5), so **the wall times are not a speed claim** and only the memory column is: | projection | wall | rate | peak-RSS growth | |---|---:|---:|---:| | `1024 x 188160 x 4096` | 57.34 s | 13.76 GMAC/s | **247 MiB** | | `1024 x 188160 x 2048` | 28.95 s | 13.63 GMAC/s | **239 MiB** | 16x the rows moved the growth by 15 MiB, which is the output buffer (`1024 x 4096 x 4 = 16.8 MB`) allocated inside the timed region. The tile does not scale with rows. **So: not the threaded kernel** (232 MiB at 20 workers). **Not the call site** — its `scaled` copy is 770 MB and is present on **both** arms, so it cannot be a regression. **Not an interaction.** Nothing needed capping, blocking or an arena, and the byte-exactness question the row was told to stop on never arose, because no summation order was touched. ## #1259 has the same profile and needs nothing `Ltx2FuseLoraIntoTensor` takes `vt::Matmul`, the other member of the same seam. `MatmulOneChunk` is one template *source* for both orientations but two separate INSTANTIATIONS with two separate buffers (see the review section below) — so the per-worker buffer is `16 x K x 4` there too, with `K = rank`. At the shipped `4096 x 450 x 4096`, measured with the same allocation counter the new gate uses: | workers | 1 | 2 | 4 | 8 | 16 | 20 | |---|---:|---:|---:|---:|---:|---:| | bytes requested | 28,864 | 28,864 | 86,464 | 201,664 | 432,064 | **547,264** | **0.52 MiB at 20 workers**, exactly `19 x 28,800 + 64`. `bs` and `agg` are unchanged by that row. It cannot hit the wall #1286 describes when a LoRA-bearing pipeline first runs on full weights, so nothing is fixed and nothing is filed for it. ## What the attribution exposed, and the one thing this lands **Nothing in this tree bounds the seam's memory.** Every GEMM gate here asserts values or byte equality, so a kernel that allocated a whole intermediate per worker would have passed all of them, on every model, silently, until a box ran out. The reported defect would have been invisible to every gate in the repository had it been real. That is worth closing while the correct number is known rather than after the next report, so `tests/vt/test_ops_matmul_mem.cpp` lands with the refutation. **Peak RSS could not carry the gate, and finding that out was the useful part.** The first draft measured `VmHWM` around `/proc/self/clear_refs`. It read `growth_bytes = 0` for **every** thread count and **every** row count while passing every bound — the operands are freed between measurements, glibc keeps the arena, and the next tile is served from pages that are already resident. A mute switch reporting green over a kernel doing anything at all. The shipped gate counts what the seam **asks for** through replaced global `operator new` overloads, a figure the allocator's RETENTION policy cannot silence, with a liveness case beside it requiring the counter to see at least six of eight fresh workers take a tile. What that counter does and does not see is stated exactly in the review section below — the first form of the claim was broader than the instrument. Green, at `K = 65536` so one tile is exactly 4 MiB: | workers | 1 | 2 | 4 | 8 | |---|---:|---:|---:|---:| | bytes requested | 64 | 4,194,368 | 12,582,976 | 29,360,192 | `(nthreads - 1)` tiles, because the caller thread's `thread_local` is already sized. Rows 64 vs 1024 at 4 workers: **12,582,976 both times**, difference 0. ### Red first, by mutation, and the first attempt did not build M1 widens `af` from the 16-row tile to the whole activation — #1286's own hypothesis, written into the kernel. - **First attempt: `BUILT=NO`, `compile_err=2`** — `cpu_ops.cpp:134:19: error: unused variable 'nrows' [-Werror=unused-variable]`. Recorded because a mutation that fails to build reads exactly like a passing test. - **Second attempt: `BUILT=YES`, `compile_err=0`**, `git diff --stat` = `src/vt/cpu/cpu_ops.cpp | 2 +-, 1 file changed, 1 insertion(+), 1 deletion(-)`. `test_ops_matmul_mem` **exit 1**, `Status: FAILURE!`, `2 test cases | 1 passed | 1 failed`, `13 assertions | 7 passed | 6 failed`. Failing by name: `CHECK(growth <= Bound(nthreads))` at three worker counts — `67,108,928 <= 25,165,824`, `201,326,656 <= 41,943,040`, `469,762,112 <= 75,497,472` — and all three row-scaling assertions, with `rows=1024` reading **1,073,741,888 bytes**, a gibibyte of exactly the shape the issue supposed. - Restored byte-for-byte: `sha256(cpu_ops.cpp) = dc39eccdece48879e82be7209e95d182c6ed624eb04389de689fa4b58fe4f1f3` before and after. Rebuilt (binary mtime moved `07:15:48` → `07:16:27`, so the green is not a stale binary) and green again: **2 cases, 13 assertions, 0 failed, exit 0**. M1 mutates `src/vt/cpu/cpu_ops.cpp`, product code reached from `ModelRegistry::Forward` and from the LTX-2.5 text tower, which is what makes the red evidence that the gate measures shipped behaviour rather than a test-local copy. This row adds **no product code**, so there is no production call site to delete; `reachability.md` answers that case directly. ## The review of that bound, and the three regressions it did NOT catch A fresh review of `39b34c0c8` returned nine findings and no `PASS`. **The refutation above was confirmed and is unchanged.** What failed review was the gate that lands with it, which is the irony worth stating plainly: a bound written to catch a memory regression did not catch three obvious shapes of one. **The bound admitted a 2x per-worker regression while its comment said it could not.** `Bound(n)` was `8 MiB + n * 2 * kTileBytes` — a per-thread term of literally two tiles, under a comment claiming it was "[not] wide enough to admit a second tile-sized buffer per worker, which is what keeps it a bound rather than a formality". Mutation **M-B** added exactly that second buffer and doubled the growth at every worker count, **13/13 green**. At the shipped geometry that is 232 → 464 MiB shipping silently. The bound is now `1 MiB + n * kTileBytes`, which is the property the file actually claims, over a fixed term **measured at 64 bytes**. The comment now names the worker counts that detect a doubling (4 and up) instead of claiming all of them do. **The counter was bypassed by an over-aligned `operator new`.** Only the non-aligned overloads were replaced, so the C++17 `std::align_val_t` family fell through to the library. **M-D** delivered the whole-activation defect through a 64-byte-aligned `operator new` — the natural shape for a SIMD scratch buffer in this kernel — and read byte-identical growth, **13/13 green**. All eight aligned `new`/`delete` overloads are replaced now. **`std::malloc` is seen too, through the LINKER rather than a symbol definition.** **M-E** delivers the same defect through `std::malloc`, and the first pass of this repair left it green and narrowed the claim instead. That was the weaker resolution, and the argument against widening turned out not to hold. The objection was that an `extern "C" void* malloc()` in the test would be a second strong definition beside AddressSanitizer's own interceptor and would break `sanitize-cpu`. `-Wl,--wrap=malloc` is not a definition: scoped to this one target, it redirects the CALLS made by the objects in this link — `libvllm.a`, and so `cpu_ops.cpp` — and leaves every symbol where it was. **M-E now reds at 1,086,324,800 bytes.** `free` is deliberately not wrapped: releasing a block gives no size without a header, and a header would mean applying an offset to pointers libc allocated before this file was reached, which is heap corruption rather than a failed assertion. So the C-allocator figure is bytes REQUESTED, cumulatively — it errs toward red, never toward a silent green — and it reads exactly **zero** on a clean run, so every per-thread figure is byte-identical to what it was without the wrapping. A new **coverage case** makes the whole claim executable: it allocates through plain `new`, array `new`, over-aligned `new` at two alignments, and `std::malloc`, and requires the counter to move by the amount asked for. Two failure modes, two mechanisms. Dropping a `--wrap` flag fails the **link**, because the file references `__real_malloc` directly (mutation **M-W**: `BUILT=NO`, `undefined reference to '__real_malloc'`), so the coverage cannot silently vanish from the build; the subtler "route present but folded away" mode is what the assertion catches. **That case calls through a `volatile` function pointer for a measured reason.** A direct `std::malloc(n)` written in the test's own translation unit is never wrapped — the compiler treats it as a builtin and folds it before the linker sees a symbol reference. In a standalone probe, same-TU calls gave `malloc=0 calloc=0 posix_memalign=1`; the identical calls from a **separate** translation unit gave `malloc=1 calloc=1 posix_memalign=1` and the full 3 MiB. Product code is always the separate-TU case, which is why M-E reds — but a coverage probe written the obvious way would have asserted nothing. **`MatmulOneChunk<false>` had no assertion behind it.** `<false>` and `<true>` are separate instantiations with separate `thread_local` buffers, and the gate called only `vt::MatmulBT` on a non-repacked weight. `MatmulKernel` (`cpu_ops.cpp:292-294`) routes `vt::Matmul` to `<false>` and `MatmulBTKernel`'s `elem_kn_repacked` lever (`cpu_ops.cpp:306-314`) routes there too; `vt::Matmul` is #1259's `Ltx2FuseLoraIntoTensor` path — merged and unexercised on the full model. Both arms are measured now. The two `af` symbols are **separately confirmed with `nm -C`** on the gate's own binary, at distinct addresses, and the measurement shows the same thing: 8 fresh workers read 33,554,496 bytes on `MatmulBT` and 33,554,496 again on the `vt::Matmul` immediately after — eight tiles both times rather than eight then seven, which one shared buffer could not produce. So the retention figure is **per instantiation**: up to 464 MiB, not 232 MiB, for a process running both orientations at `K = 188160` on 20 workers. **The chunk-row-span defect was invisible at every geometry the gate ran.** **M-A** (`nrows = ir1_end - iir1`) produced byte-identical growth and **13/13 green**. Replaying `MatmulChunked`'s grid arithmetic (`cpu_ops.cpp:230-264`) over all twelve shapes the sweeps use gives `dr1` = 16 at eleven of them and 8 at the twelfth — never above 16, which is the entire reason M-A could not be seen, and a coincidence of the chosen shapes rather than a property of the kernel. The collapse to one chunk per thread (`nchunk0 * nchunk1 < nth * 4`) is **live** and is the shipped default, since `VT_CPU_MATMUL_STEAL` is off. Cheap to close rather than to file, so a case now runs at `rows = 128, n = 16, nth = 4` — a live non-NUMA shape where the collapse gives `dr1 = 32` and a chunk-sized buffer costs two tiles per worker instead of one. **One correction to that finding as it was written.** The review called it "a real ~3x regression the gate cannot see on a NUMA host", citing `dr1 = ceil(rows/nth) = 52` at `rows = 1024, nth = 20`. Two things are off. `IsNuma()` is `constexpr false` in this tree (`cpu_threadpool.h:74`; NUMA is recorded as unported at `cpu_threadpool.h:25`), so that branch is dead today and the gap is reachable through the live collapse instead. And at the shipped LTX-2.5 caption projection the weight is the longer axis (`n = 4096 > rows = 1024`), so the collapse takes `nchunk1 = 1` and `dr1 = 1024`, not 52 — a chunk-sized `af` there would be **770 MB per worker and 15.4 GB across 20**, which is #1286's hypothesised shape almost exactly. The finding was right that the gate was blind; the magnitude is larger than it said and the trigger is nearer. ### The mutation table Release, 20-core Zen 5 under KVM. Every row rebuilt and re-run; `sha256(cpu_ops.cpp) = dc39eccdece48879e82be7209e95d182c6ed624eb04389de689fa4b58fe4f1f3` before and after every one of them. | mutation | defect written into `cpu_ops.cpp` | `BUILT` / `compile_err` | `git diff --stat` on `src/` | at `39b34c0c8` | after | |---|---|---|---|---|---| | **M-A** | `nrows` = the chunk's row span | `YES` / 0 | `cpu_ops.cpp \| 2 +-` | green, 13/13 | **exit 1**, 2 of 38 failed — `33,554,496 <= 17,825,792` on both arms of the collapsed case | | **M-A2** | `nrows` = the whole activation | `YES` / 0 | `cpu_ops.cpp \| 2 +-` | red | **exit 1**, 14 of 38 failed | | **M-B** | a second tile-sized buffer per worker | `YES` / 0 | `cpu_ops.cpp \| 3 +` | green, 13/13 | **exit 1**, 10 of 38 failed — `25,165,888 <= 17,825,792` at 4 workers, `58,720,320 <= 34,603,008` at 8 | | **M-D** | whole activation via 64-byte-aligned `operator new` | `YES` / 0 | `cpu_ops.cpp \| 12 +` | green, 13/13 | **exit 1**, 14 of 38 failed — `1,019,215,936` bytes at `rows=1024`, 4 workers | | **M-E** | whole activation via `std::malloc` | `YES` / 0 | `cpu_ops.cpp \| 12 +` | green, 13/13 | **exit 1**, 14 of 38 failed — `71,303,232 <= 9,437,184` at 2 workers, `1,086,324,800` bytes at `rows=1024`, 4 workers | | **M-W** | the `--wrap=malloc` link flag removed | **`NO`** / 1 | `CMakeLists.txt \| 1 -` | n/a | **link fails**: `undefined reference to '__real_malloc'` — the coverage cannot be dropped silently | **The `sanitize-cpu (address,undefined)` risk was exercised, and the record is in two parts because the second attempt was blocked.** The FIRST widening — the aligned `operator new` family, before `--wrap` existed — ran in the CI lane's own configuration (`-DVLLM_CPP_SANITIZE='address,undefined'`, `-DVLLM_CPP_CUDA=OFF`, `UBSAN_OPTIONS=print_stacktrace=1`): **5 cases, 36 assertions, 0 failed, exit 0**, no ASan or UBSan diagnostic. The `thread` lane was green too, once ASLR was disabled to work around this box's `FATAL: ThreadSanitizer: unexpected memory mapping` — a local kernel/TSan incompatibility, not a finding. **Rebuilding that lane after `--wrap` was added hit `No space left on device`** — other sessions filled the shared disk to 100%, and the build's first output was five `fatal error: error writing to /tmp/...: No space left on device` lines, which is an infrastructure failure wearing the costume of a code verdict. **So the full lane was NOT re-run with `--wrap` in place**, and this says so rather than implying otherwise. What ran in its place is a standalone probe reproducing the gate's entire allocator mechanism in miniature — replaced plain and aligned `operator new`/`delete`, the five `--wrap` redirections, and a separate translation unit standing in for `cpu_ops.cpp` — compiled and linked with `-fsanitize=address,undefined`. It builds clean and runs clean: all three C-allocator routes intercepted (3 MiB accounted), 64-byte alignment preserved through the replaced aligned `new`, replaced `operator new` accounting correct, no diagnostic, exit 0. That establishes the **mechanism** against ASan, which is precisely the collision the reviewer named. It does not stand in for a full-lane run and is not offered as one. **The bound is deliberately tight now, and that is the point.** At 8 workers the honest worst case is 8 tiles — every worker fresh, which the liveness case reaches — at 33,554,496 bytes against a bound of 34,603,008. The 1 MiB of headroom is 16,000x the measured incidental. Widening it again buys nothing except the 2x hole M-B just walked through. ### Records repaired in the same change **A wrong line anchor, cited six times.** `cpu_ops.cpp:125` is cited for `static thread_local std::vector<float> af` — it is at **line 130**; `:125` is `const int64_t blck_1 = 16;`, and the comment block cited as `122-125` is at `127-129`. Wrong at the named baseline, at the implementation commit and at `HEAD`. Fixed in the spec, the benchmark-record and the test header. **It CANNOT be fixed in `.agents/issue-index.md:423`**, which cites it twice. That file is append-only and keyed, its checker reads commits rather than the working tree, and a corrective row would be a duplicate key its checker refuses. So the correction is recorded where a reader following the row arrives — the spec's `## Owed`, and this body. The same wrong anchor is frozen in the bodies of `0377cde70` and `8d46223fd` for the same reason. **The uncovered class behind it is recorded, not fixed here.** `check-symbol-anchors.py:87-90` requires a literal `::` between path and identifier, so it cannot match a `path:line` anchor at all. Counted over every `.md` in this tree: **613 `path::Symbol` citations and 14,448 `path:line` citations**, the second checked by nothing. A 24:1 ratio deserves its own row and its own red-first mutation, not a follow-up line here. **`tests/CMakeLists.txt` gave a false rationale for a live conditional.** It called the test "the seam's PEAK RESIDENT bound" and justified the `Linux`-only guard with "because the measurement is /proc/self/{status,clear_refs}". The shipped test measures neither and opens nothing under `/proc`; it landed in the same commit as the shipped instrument, so it was wrong rather than historical. The real reason is stated now: the gate replaces the global allocation functions for the whole program, so it needs every `delete` in the binary to reach the matching replaced `delete`, and its fixed term calibrated against the library doing the allocating — both established for glibc/libstdc++ and nothing else. **Three record corrections beside them.** The spec's `## Tests to port` and `## Risks/decisions` describe the refuted `VmHWM` instrument while `## Outcome` W4 records that it was replaced, so both now carry a forward pointer. `## Outcome` W2's first table was headed "at the shipped geometry" while running `rows = 64`, breaking the spec's own rule at `:158-162` that a scaled probe is never presented as a full-geometry one; the reduction sits beside the number now. And the "threaded stretch sits at `anon = 34.30`" triple was that stretch's minima at a single sample — recomputed above from the retained `memguard.tsv`. **`runguard.py`'s line citations are unstable and were not chased.** The file lives outside this repository on a mutable NAS path and was rewritten after the implementation commit, so `runguard.py:236-237,260` no longer resolves reproducibly. The claim does not rest on it: in the retained `memguard.tsv` the #1252 run's first sample reads `avail = 88.078`, `used = 31.553` — summing to 119.631 GiB, the box's whole `MemTotal` — beside `rss = 0.001` for a child that had just started. A column reading 31.553 GiB before the process has allocated anything is system-wide by construction, whatever line computes it. ## Gates - `test_ops_matmul_mem`: **5 cases, 38 assertions, 0 failed, exit 0**, `Status: SUCCESS!` — up from 2 cases and 13 assertions. - `test_ops_matmul_mem` under `VLLM_CPP_SANITIZE='address,undefined'` (before `--wrap`; see above for why it could not be re-run after): **5 cases, 36 assertions, 0 failed, exit 0**, no sanitizer diagnostic. - `test_ops_matmul_mem` under `VLLM_CPP_SANITIZE='thread'` (ASLR disabled to work around this box's `FATAL: ThreadSanitizer: unexpected memory mapping`): **5 cases, 36 assertions, 0 failed, exit 0**. - Full `ctest -j 6` after merging `origin/main` `edbc47ce0`, on this exact head: **100% tests passed, 0 tests failed out of 554**, 3 skipped by design, 995.03 s. The box was heavily contended by other sessions throughout (loadavg 180-333 on 20 cores, 179 competing processes), so **no wall time in that run is a measurement of anything** — only the pass/fail column is. Nothing was re-run to make it green. - `scripts/check-agent-record.py`: `agent record OK: ENGINE=165 MODEL=377 QUANT=84 KERNEL=52 BACKEND=85 ANCHOR-ROT=38`. - `scripts/check-doc-checkpoint.py --base origin/main --head HEAD`: `OK: public documents match the claims this change makes.` It failed first — a benchmark-record change owes `docs/BENCHMARKS.md` and `docs/STATUS.md`, and the projection had to ride in the SAME commit as the record edit. - `scripts/check-public-doc-tables.py`: `OK`. This one failed first too, and correctly: the projection as first written was a 496-char table cell against a 220 limit, and pushed `docs/STATUS.md` to 45 oversized cells against its 44 ratchet. Both were shortened into budget rather than the ratchet being raised. - `scripts/check-commit-trailers.py` and `scripts/check-commit-style.py` over `origin/main..HEAD`: both `OK`. - `scripts/check-issue-index-append-only.py --base origin/main`: `OK: issue index append-only`. **413 rows, zero duplicate keys**, #1286 and #1317 each present exactly once. - Mutation evidence for every claim tightened or added: the table above, with `BUILT`, `compile_err`, `git diff --stat` and the failing assertion by name for each. Byte-exactness is untouched — **`src/vt/cpu/cpu_ops.cpp` is byte-identical to `39b34c0c8`**, sha256 `dc39ecc...4f1f3`, no tolerance was widened anywhere, and no summation order was altered. Every mutation above was applied to a scratch tree and restored. ## Records, and commit order `.agents/specs/ltx25-text-linear-mem.md` is committed **before** the implementation (`0377cde70`, then `8d46223fd`). `docs/STATUS.md` and `docs/BENCHMARKS.md` carry the keyed projection; the forensics are in `.agents/benchmark-record.md`. Two index rows appended, #1286 owned by this row and #1317 owned by this spec's `## Owed`. The review repair rides in the same pull request rather than a new one: it is the record edit and the gate repair the reviewed change made necessary, which `AGENTS.md` puts in the pull request whose change made them stale. It touches four files — the test, its registration, the spec's `## Outcome` W5 and `## Owed`, and the benchmark-record entry — and **no product code**. ## Owed, named rather than left to be found - [#1317](#1317) — the harness comparison and the 26.8 GiB that was already on the box. **This is what actually aborted the render**, and both halves are outside this repository. - **No GB10 number of any kind.** Every measurement here is from a 20-core Zen 5. Allocation sizes are architecture-independent so the attribution carries; a throughput ratio does not, and none is claimed for GB10. #1252's unmeasured per-core ratio `R` is untouched. - **The tile is retained for the process lifetime, PER INSTANTIATION.** `af` grows to the largest `K` any GEMM on that worker ever saw and never shrinks — 232 MiB held until exit at `K = 188160` on 20 workers, and there are **two** such buffers, one in each of `MatmulOneChunk<false>` and `<true>`, so a process running both orientations at that `K` retains up to **464 MiB**. Bounded, deliberate, and now measured; not repaired here, because the buffer exists precisely so a worker allocates once, and trading that away is a performance decision with its own spec. - **The counter does not see `mmap`/`sbrk`, nor an allocator reached inside a shared library's own internal calls.** `--wrap` binds at THIS link, so a `malloc` that `libstdc++.so` calls internally is outside it; every global `operator new` is covered regardless. A kernel taking pages straight from `mmap` would be invisible to this gate. Nothing in the seam does, and no mutation attempts it, so this is a stated boundary rather than a measured gap. - **`path:line` citations are checked by nothing.** `check-symbol-anchors.py:87-90` requires a literal `::`, so of **613 `path::Symbol` and 14,448 `path:line`** citations in this tree's markdown, only the first population is covered. That is how the wrong `cpu_ops.cpp:125` here survived the implementation commit, the merge and the gate. Extending the checker needs its own spec and its own red-first mutation. - **The `cpu_ops.cpp:125` anchor cannot be corrected in `.agents/issue-index.md:423`.** The index is append-only and keyed, so the row cannot be edited and a corrective row would be a duplicate key its checker refuses. The correction is recorded in the spec's `## Owed` and in this body instead. The same wrong anchor is frozen in the bodies of `0377cde70` and `8d46223fd`. - **Linux-only registration is a calibration scope, not a platform dependency.** Extending the gate to the macOS and Windows lanes is owed work; the counter itself is portable. - **`runguard.py:236-237,260` is an unstable citation** — the file is outside this repository on a mutable NAS path and was rewritten after the implementation commit. The claim it supported was verified independently against the retained `memguard.tsv` columns and does not depend on it. - **The bound is a CPU bound.** `vt::MatmulBT` on CUDA/ROCm/Vulkan is not covered. - **`docs/BENCHMARKS.md` and `docs/STATUS.md`** carry the new keyed projection, which `check-doc-checkpoint.py` requires of a benchmark-record change: the allocation is bounded on both instantiations, and the retention figure is 464 MiB rather than 232. - **No full-model render has been rerun**, so nothing here says what one would now do — only that #1252 is not what stopped the last one. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
localai-bot
added a commit
that referenced
this pull request
Aug 19, 2026
…DiT, and the 1024x576 rung it could not fit (#1375) (#1379) Every LTX-2.5 artifact until now was rendered on the distilled arm. This records the first on the full `ltx-2.5-22b-dev-transformer-bf16.safetensors` (42,018,190,584 bytes, 4349 tensors, 21,004,025,600 elements): 768x448/25f on GB10 in 2990 s, 25 frames plus 1.01 s of 48 kHz stereo, verifier PASS on both the pixel and audio arms. Binary `0a43a750` built from `7b9e207b1`, run `20260819T150230Z`. The verifier numbers rather than its verdict: 25 distinct frame hashes of 25, zero near-uniform and zero near-black frames, adjacent-frame MAD 2.01 against far-frame MAD 9.34 (ratio 0.215, where uncorrelated noise gives ~1.0), audio RMS -15.63 dBFS, envelope CV 0.112, spectral crest 52.0. The same lease REFUSED 1024x576/25f, and that contrast is the result rather than a failure beside it. The governor measured `first_dit` 481.5 s and 162.0 s per forward from 7 resolved forward starts, so the recipe's fixed 60 forwards plus a 600 s tail project 10803 s against that rung's 7153 s budget. The 60 is not tunable: `one_stage` at model version 2.5 resolves to 30 inference steps (`ltx2_pipeline.cpp:968`) and a non-unit `cfg_scale` forces an unconditional forward per step (`ltx2_pipeline.cpp:521-523`), so the ceiling is geometry against lease length. That per-forward measurement is issue #1375. What it is not is recorded with it. `Ltx2DitForwardDevice` takes no `perturbations` argument, so the device-resident arm refuses the perturbed and isolated-modality passes by name (`ltx2_video.cpp:2738-2749`, issue #1092), while LTX-2.5's own params carry `stg_scale` 1.0 and `modality_scale` 3.0. The run used the engine's own sanctioned escape and kept real classifier-free guidance (video 3.0, audio 7.0) while giving up the STG and isolated-modality terms. No oracle ran against it, so the speed axis stays PENDING and no parity claim is made. Two instrument facts ride with the number because both have already produced a wrong reading elsewhere. `gpu_edges = 0` means saturated rather than idle: the counter needs a busy-to-idle transition, and that rung sat above 50 per cent utilisation in 2704 of 3191 samples. And `eu-stack` resolves no frames in the rc worker container, so phase attribution came from the cpu and rss signature instead of from symbols. Reachable at all only because of issue #1252. Before it, the caption projection was a scalar triple loop with a `double` accumulator that could not reach an f32 FMA; runs on the pre-fix binary measured single-core stretches of 1740 s and 1738 s, 0.11 per cent apart, reproducing the figure in issue #1087 twice in one render. After it, the same phase appears as `first_dit` 481 s. Three files change and all three are additions only: the entry in `docs/BENCHMARKS.md`, its forensics and reproduce recipe in `.agents/benchmark-record.md` where the public table's own checker directs detail that does not fit an entry, and the append-only index row for issue #1375. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [Claude Code] Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
fix(LTX25-TEXT-LINEAR-SEAM): the caption projection is one core for 671 s per pass, so route it through vt::MatmulBT
Linearin the LTX-2.5 text tower was a scalar triple loop that widened bothoperands to
doubleper multiply. It could not reach an f32 FMA, and no threadbut the caller's ever entered it.
The attribution came first, because the issue's own record withdrew it
The second comment on #1208
argued that two single-core stretches 0.11% apart cannot be a projection whose
two
project()calls sit in a 2:1 size ratio. Correct, and it does not apply.Four facts, each read out of the tree:
ltx2_video.cpp:2085,:2799and:4479each buildvt::Queue{vt::Device{kCPU, 0}, nullptr}immediately before calling
Ltx2EncodePromptToConditioning.--device cudanever moved this path.
unconditional branch, and each pass contains both projections — so the
2:1 ratio lives inside a stretch and cannot split one.
out.seq = max_lengthandkLtx2GemmaTokenizerMaxLength = 1024, so the projection is1024 x 1,156,055,040 = 1.1838e12MACs on every render. That collapses thecomment's 0.34-5.4 GMAC/s band to 0.681 GMAC/s.
hardware_concurrency()threads.What was measured, and what stays open
At the shipped geometry the old loop takes 671.777 s at 99% of one CPU, and
that number is one whole pass containing both
project()calls — it mustnot be doubled. Turning it into a GB10 wall needs
R, the per-core ratiobetween the two boxes on this loop, and
Rwas not measured. It is boundedinstead:
R >= 1because a GB10 Arm core does not out-run a 5.7 GHz-classZen 5 on a scalar
double-widening loop, andR <= 2.587because theprojection runs inside the stretch and cannot exceed it.
RSo this addresses 39-100% of the dominant phase, and 0-1066 s per stretch is
still unattributed. That is stated rather than rounded into the projection,
and it is carried under
## Owed.One residual hypothesis was offered and is refuted by measurement. The
candidate was the surrounding single-threaded buffer work. Timed at the shipped
geometry, two runs:
Ltx2StackHiddenStates0.839/0.910 s,Ltx2NormAndConcatPerTokenRms0.632/0.688 s, thescaledcopy x20.390/0.468 s — 1.861/2.066 s in total, 0.28% of the pass, with the two GEMMs
at 99.7%. A core ten times slower still puts it at ~20 s. The only remaining
candidate of the right magnitude is the U8/NVFP4 caption weights becoming the
f32 4.6 GB, and this branch does not claim it.
The A/B
doubledoublevt::MatmulBTvt::MatmulBT671.777 s -> 78.421 s, 8.57x, one core of twenty becoming about fourteen.
Recipe: 20-core Zen 5 under KVM, Release CPU tier,
-ffp-contract=off, sameharness source and flags on both arms, box not idle (load 4.1-10.1).
The full-size
scaledcopy insideproject()is kept unchanged on botharms: 0.06% of the before arm and 0.5-0.6% of the after arm.
The accumulator, decided against a named oracle
f32, because that is what upstream computes. vLLM never registers LTX-2.5, and
vLLM-Omni is
UNPINNED/gateable = no, so the oracle is Lightricksltx_coreitself, executed intorch.float32byscripts/gen-ltx2-text-goldens.pyatfd4ded7f2. The module istorch.nn.functional.linear— the reference the C++ comment already cited —and on f32 tensors that is
addmmonto an f32sgemm, with no f64 anywhere.The header's old argument that f64 lands nearer torch's blocked sum is true and
is rewritten rather than deleted, because closer to exact is not closer to
what upstream computes.
Cost measured at
K = 188160against along doublereference: f32 sequential2.12e-05 max absolute error, f64-then-store 5.88e-08 — the f32 arm's worst
error is ~330x below one bf16 ulp of the value it produces. Ten of twelve
golden numbers are byte-unchanged; the two that moved are the V1 arm at
1.19e-07 and 7.45e-08 against the file's 1e-5 bound, 84x and 134x inside it.
No tolerance was widened.
Red first, and the instrument proves it discriminates
The new case runs at
flat = 8192and requires the projection to equalvt::MatmulBT+ bias byte for byte, with a second assertion that anf64-accumulating reference is measurably further away — so it cannot pass
vacuously.
Mutation M1, reverting
Linearto the pre-row scalar loop:BUILT=YES,compile_err=0,git diff --stat1 file / 19 insertions / 56 deletions,CHECK(mismatched == 0)96/96 elements differed,CHECK(worst_f64 > 1e-6)read 5.28e-08,
Status: FAILURE!, 1 case ran and 26 skipped. Restoredbyte-for-byte. On the fixed tree the same case reads 0/96 and 4.11e-06,
Status: SUCCESS!.Gates
test_ltx2_text_encoder: 27/27 cases, 4118 assertions,Status: SUCCESS!(26/4115 before, so the case count moved).
ctest -j 6: 529 tests, 3 failed.test_engine_core_procpasses on serial rerun (feat(muse-glimmer): trace Meta's Muse Glimmer in vLLM and land the W0 scaffold #279 starvation).test_qwen27_dense_forward{,_glue_fuse_off}fail serially and arepre-existing on
main— already tracked as#1240, introduced by
09597106etoday. Proven not mine two ways: this branch is byte-identical tomainon that path, and reverting onlyltx2_text_encoder.cppto itspre-row version reproduces the same failure.
scripts/agent-preflight.sh --staged: All gates green.Reachability
Ltx2EncodePromptToConditioning(ltx2_video.cpp:2086,:2800,:4480) ->Ltx2TextEncoderConditioning->Ltx2TextFeatureExtractorForward->Linear->
vt::MatmulBT. The production-entry case "ltx2 prompt -> conditioning: theVALUES, against the left-padded oracle" runs unskipped in CI and goes through
that entry point.
Records
Spec
.agents/specs/ltx25-text-linear-seam.mdis committed before theimplementation (
b70755a18then9b7952b1a). No issue-index row is added:839d24313appended one for #1208 while this row was in flight, and the file isappend-only, so a second row would be the duplicate its own checker refuses.
#1208 moves out of
ltx25-decode-speed.md's## Owedand into this row's,following that spec's own #1008 precedent.
Owed, named rather than left to be discovered
R, and with it 0-1066 s of each stretch. Onerclease ondgx:gpu0running this harness measures it and collapses the table above to one row.
PENDING.arm's 1.84 on one core.
MatmulOneChunkreduces over the wholeK = 188160in one micro-kernel call, so a 752 KB weight row and a 752 KB activation row
each exceed the 1 MB L2 — ~1.25 bytes per MAC at
mr = 4. Next hypothesis:K-blocking. No ceiling is declared.
signature.
Out of scope and deliberately untouched:
#1202 and
#1210. Neither becomes
trivially reachable from this change.
Closes #1208
FOLLOWING_AGENTS_PROTOCOL
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]