Skip to content

fix(LTX25-TEXT-LINEAR-MEM): #1252's +26 GiB is the box, not the change, and the seam had no memory bound at all (#1286) - #1331

Merged
localai-bot merged 6 commits into
mainfrom
row/LTX25-TEXT-LINEAR-MEM
Aug 19, 2026
Merged

fix(LTX25-TEXT-LINEAR-MEM): #1252's +26 GiB is the box, not the change, and the seam had no memory bound at all (#1286)#1331
localai-bot merged 6 commits into
mainfrom
row/LTX25-TEXT-LINEAR-MEM

Conversation

@localai-bot

@localai-bot localai-bot commented Aug 19, 2026

Copy link
Copy Markdown
Collaborator

fix(LTX25-TEXT-LINEAR-MEM): #1252's +26 GiB is the box, not the change, and the seam had no memory bound at all

#1286 reported that routing the LTX-2.5 caption projection through vt::MatmulBT (#1252) raised peak host memory from ~79 GiB to 105.85 GiB on a 119 GiB GB10 and aborted a full-model render before any denoise step. It named three candidates — the shared threadpool's per-worker tiles, the call site's full-size scaled copy, or an interaction — and said plainly that the attribution was not measured.

It is none of the three. The attribution came first, as the issue asked, and it changed what this row lands.

The +26 GiB is the machine, measured on the retained evidence

runguard.py:236-237,260 fixes what the compared column is: used_gib is the system-wide MemTotal - MemAvailable, anon_gib is system-wide AnonPages, and only rss_gib belongs to the child. Both runs are retained under /mnt/nas_share/rc/ltx25-fullmodel/out/, which is what makes this answerable at all.

pre-#1252, 1024x576-25f/ #1252, 20260818T220620Z/1024x576-25f/
used_gib at t=0 4.741 31.553
avail_gib at t=0 114.890 88.078
peak used_gib 79.206 at t=1867.4 s, cpu=1885.5% 105.853 at t=199.2 s, cpu=1925.2%

The box carried 26.812 GiB before ltx2-gen was started in the second run. The reported regression is 105.853 - 79.206 = 26.647 GiB. They agree to 0.165 GiB.

Subtract each run's own baseline and the two are the same binary's worth of memory, on three independent columns:

axis pre-#1252 #1252 delta
used_gib peak minus own t=0 74.465 74.300 -0.165
anon_gib peak minus own t=0 38.012 38.217 +0.205
child rss_gib at the peak sample 41.952 42.090 +0.138

Both peaks were sampled inside a ~1900% CPU stretch, so this is the same phase class rather than two different ones. On a box as clean as the first run's, the #1252 binary's own 74.300 GiB would have left 40.59 GiB available — above the 12 GiB hard floor and the 8 GiB projection floor, and in line with the "never below ~40 GiB" the pre-#1252 runs showed. The guard fired correctly on a real condition; the condition was not the binary.

Two readings from the same evidence, recorded because each on its own would have been read as supporting #1286. During its single-core projection (cpu ~ 100%, t=124-1801 s, 1676 samples) the pre-#1252 run is dead flatanon 34.283-34.441, rss 33.484-33.570, used 74.405-74.573 — while the #1252 run's threaded stretch (cpu > 500%, t=137.1-385.5 s, 236 samples) is a ramp rather than a plateau: anon climbs 34.280 → 42.593, rss 30.322 → 42.120, used 97.350 → 105.853. The two coincide on anon at the ramp's FIRST SAMPLE (34.280 against 34.283) and nowhere after it, so that near-identity is where the ramp starts rather than a matching steady state; used is 23 GiB apart there and 26.6 GiB apart at the peak. The argument above is peak-minus-own-t=0 and does not rest on any single sample. And neither run ever loaded the DiT (dit_runs=0 in both; the first was stopped from outside at t=3017.3 s, # TERMINAL signal=15), so neither peak is a whole-render peak and neither is presented as one.

Filed as #1317, with both halves named: the harness compares absolute peaks of a system-wide column without recording the starting occupancy, and something held 26.8 GiB on dgx:gpu0 before the job began. Neither is in this repository, so neither is fixed in flow.

Measured locally too, before and after, with the throughput beside it

Same probe source and flags on both arms, only the Linear body differing; peak RSS is VmHWM read before and after the call with every operand already allocated and touched. 20-core Zen 5 under KVM, Release, -ffp-contract=off, shipped K = 188160 and out_features = 4096, rows = 64 (reduced from the shipped 1024 to keep the single-threaded arm inside a sensible wall; the tile term the sweep isolates is independent of rows by construction). Box not idle, loadavg 5.4-13.3, three replicates, median:

arm (rows = 64, reduced) wall rate peak-RSS growth
before, scalar double loop 28.488 s 1.732 GMAC/s 1.0 MiB
after, vt::MatmulBT 3.378 s 14.60 GMAC/s 232 MiB

8.43x, and +231 MiB — the speedup reproduces #1252's 8.57x, and the memory cost is 0.85% of the 26.6 GiB #1286 attributes to the change, 117x too small to be it.

Swept over threadpool width at the same reduced rows = 64 geometry:

workers (rows = 64, reduced) 1 2 4 8 16 20
peak-RSS growth (MiB) 12.9 24.7 47.8 93.8 186.1 232.1
per worker (MiB) 12.9 12.3 12.0 11.7 11.6 11.6

That is cpu_ops.cpp:130's static thread_local std::vector<float> af and nothing else: sized by ggml's 16-row blck_1, so 16 x 188160 x 4 = 11.48 MiB per worker. The measurement lands on the model, and the output checksum is byte-identical across all six widths, which is the dispatch determinism contract holding.

At the full shipped geometry, rows = 1024, both projections, default 20 workers — the box was heavily loaded (loadavg 15.9 then 28.5), so the wall times are not a speed claim and only the memory column is:

projection wall rate peak-RSS growth
1024 x 188160 x 4096 57.34 s 13.76 GMAC/s 247 MiB
1024 x 188160 x 2048 28.95 s 13.63 GMAC/s 239 MiB

16x the rows moved the growth by 15 MiB, which is the output buffer (1024 x 4096 x 4 = 16.8 MB) allocated inside the timed region. The tile does not scale with rows.

So: not the threaded kernel (232 MiB at 20 workers). Not the call site — its scaled copy is 770 MB and is present on both arms, so it cannot be a regression. Not an interaction. Nothing needed capping, blocking or an arena, and the byte-exactness question the row was told to stop on never arose, because no summation order was touched.

#1259 has the same profile and needs nothing

Ltx2FuseLoraIntoTensor takes vt::Matmul, the other member of the same seam. MatmulOneChunk is one template source for both orientations but two separate INSTANTIATIONS with two separate buffers (see the review section below) — so the per-worker buffer is 16 x K x 4 there too, with K = rank. At the shipped 4096 x 450 x 4096, measured with the same allocation counter the new gate uses:

workers 1 2 4 8 16 20
bytes requested 28,864 28,864 86,464 201,664 432,064 547,264

0.52 MiB at 20 workers, exactly 19 x 28,800 + 64. bs and agg are unchanged by that row. It cannot hit the wall #1286 describes when a LoRA-bearing pipeline first runs on full weights, so nothing is fixed and nothing is filed for it.

What the attribution exposed, and the one thing this lands

Nothing in this tree bounds the seam's memory. Every GEMM gate here asserts values or byte equality, so a kernel that allocated a whole intermediate per worker would have passed all of them, on every model, silently, until a box ran out. The reported defect would have been invisible to every gate in the repository had it been real. That is worth closing while the correct number is known rather than after the next report, so tests/vt/test_ops_matmul_mem.cpp lands with the refutation.

Peak RSS could not carry the gate, and finding that out was the useful part. The first draft measured VmHWM around /proc/self/clear_refs. It read growth_bytes = 0 for every thread count and every row count while passing every bound — the operands are freed between measurements, glibc keeps the arena, and the next tile is served from pages that are already resident. A mute switch reporting green over a kernel doing anything at all. The shipped gate counts what the seam asks for through replaced global operator new overloads, a figure the allocator's RETENTION policy cannot silence, with a liveness case beside it requiring the counter to see at least six of eight fresh workers take a tile. What that counter does and does not see is stated exactly in the review section below — the first form of the claim was broader than the instrument.

Green, at K = 65536 so one tile is exactly 4 MiB:

workers 1 2 4 8
bytes requested 64 4,194,368 12,582,976 29,360,192

(nthreads - 1) tiles, because the caller thread's thread_local is already sized. Rows 64 vs 1024 at 4 workers: 12,582,976 both times, difference 0.

Red first, by mutation, and the first attempt did not build

M1 widens af from the 16-row tile to the whole activation — #1286's own hypothesis, written into the kernel.

  • First attempt: BUILT=NO, compile_err=2cpu_ops.cpp:134:19: error: unused variable 'nrows' [-Werror=unused-variable]. Recorded because a mutation that fails to build reads exactly like a passing test.
  • Second attempt: BUILT=YES, compile_err=0, git diff --stat = src/vt/cpu/cpu_ops.cpp | 2 +-, 1 file changed, 1 insertion(+), 1 deletion(-). test_ops_matmul_mem exit 1, Status: FAILURE!, 2 test cases | 1 passed | 1 failed, 13 assertions | 7 passed | 6 failed. Failing by name: CHECK(growth <= Bound(nthreads)) at three worker counts — 67,108,928 <= 25,165,824, 201,326,656 <= 41,943,040, 469,762,112 <= 75,497,472 — and all three row-scaling assertions, with rows=1024 reading 1,073,741,888 bytes, a gibibyte of exactly the shape the issue supposed.
  • Restored byte-for-byte: sha256(cpu_ops.cpp) = dc39eccdece48879e82be7209e95d182c6ed624eb04389de689fa4b58fe4f1f3 before and after. Rebuilt (binary mtime moved 07:15:4807:16:27, so the green is not a stale binary) and green again: 2 cases, 13 assertions, 0 failed, exit 0.

M1 mutates src/vt/cpu/cpu_ops.cpp, product code reached from ModelRegistry::Forward and from the LTX-2.5 text tower, which is what makes the red evidence that the gate measures shipped behaviour rather than a test-local copy. This row adds no product code, so there is no production call site to delete; reachability.md answers that case directly.

The review of that bound, and the three regressions it did NOT catch

A fresh review of 39b34c0c8 returned nine findings and no PASS. The refutation above was confirmed and is unchanged. What failed review was the gate that lands with it, which is the irony worth stating plainly: a bound written to catch a memory regression did not catch three obvious shapes of one.

The bound admitted a 2x per-worker regression while its comment said it could not. Bound(n) was 8 MiB + n * 2 * kTileBytes — a per-thread term of literally two tiles, under a comment claiming it was "[not] wide enough to admit a second tile-sized buffer per worker, which is what keeps it a bound rather than a formality". Mutation M-B added exactly that second buffer and doubled the growth at every worker count, 13/13 green. At the shipped geometry that is 232 → 464 MiB shipping silently. The bound is now 1 MiB + n * kTileBytes, which is the property the file actually claims, over a fixed term measured at 64 bytes. The comment now names the worker counts that detect a doubling (4 and up) instead of claiming all of them do.

The counter was bypassed by an over-aligned operator new. Only the non-aligned overloads were replaced, so the C++17 std::align_val_t family fell through to the library. M-D delivered the whole-activation defect through a 64-byte-aligned operator new — the natural shape for a SIMD scratch buffer in this kernel — and read byte-identical growth, 13/13 green. All eight aligned new/delete overloads are replaced now.

std::malloc is seen too, through the LINKER rather than a symbol definition. M-E delivers the same defect through std::malloc, and the first pass of this repair left it green and narrowed the claim instead. That was the weaker resolution, and the argument against widening turned out not to hold. The objection was that an extern "C" void* malloc() in the test would be a second strong definition beside AddressSanitizer's own interceptor and would break sanitize-cpu. -Wl,--wrap=malloc is not a definition: scoped to this one target, it redirects the CALLS made by the objects in this link — libvllm.a, and so cpu_ops.cpp — and leaves every symbol where it was. M-E now reds at 1,086,324,800 bytes.

free is deliberately not wrapped: releasing a block gives no size without a header, and a header would mean applying an offset to pointers libc allocated before this file was reached, which is heap corruption rather than a failed assertion. So the C-allocator figure is bytes REQUESTED, cumulatively — it errs toward red, never toward a silent green — and it reads exactly zero on a clean run, so every per-thread figure is byte-identical to what it was without the wrapping.

A new coverage case makes the whole claim executable: it allocates through plain new, array new, over-aligned new at two alignments, and std::malloc, and requires the counter to move by the amount asked for. Two failure modes, two mechanisms. Dropping a --wrap flag fails the link, because the file references __real_malloc directly (mutation M-W: BUILT=NO, undefined reference to '__real_malloc'), so the coverage cannot silently vanish from the build; the subtler "route present but folded away" mode is what the assertion catches.

That case calls through a volatile function pointer for a measured reason. A direct std::malloc(n) written in the test's own translation unit is never wrapped — the compiler treats it as a builtin and folds it before the linker sees a symbol reference. In a standalone probe, same-TU calls gave malloc=0 calloc=0 posix_memalign=1; the identical calls from a separate translation unit gave malloc=1 calloc=1 posix_memalign=1 and the full 3 MiB. Product code is always the separate-TU case, which is why M-E reds — but a coverage probe written the obvious way would have asserted nothing.

MatmulOneChunk<false> had no assertion behind it. <false> and <true> are separate instantiations with separate thread_local buffers, and the gate called only vt::MatmulBT on a non-repacked weight. MatmulKernel (cpu_ops.cpp:292-294) routes vt::Matmul to <false> and MatmulBTKernel's elem_kn_repacked lever (cpu_ops.cpp:306-314) routes there too; vt::Matmul is #1259's Ltx2FuseLoraIntoTensor path — merged and unexercised on the full model. Both arms are measured now. The two af symbols are separately confirmed with nm -C on the gate's own binary, at distinct addresses, and the measurement shows the same thing: 8 fresh workers read 33,554,496 bytes on MatmulBT and 33,554,496 again on the vt::Matmul immediately after — eight tiles both times rather than eight then seven, which one shared buffer could not produce. So the retention figure is per instantiation: up to 464 MiB, not 232 MiB, for a process running both orientations at K = 188160 on 20 workers.

The chunk-row-span defect was invisible at every geometry the gate ran. M-A (nrows = ir1_end - iir1) produced byte-identical growth and 13/13 green. Replaying MatmulChunked's grid arithmetic (cpu_ops.cpp:230-264) over all twelve shapes the sweeps use gives dr1 = 16 at eleven of them and 8 at the twelfth — never above 16, which is the entire reason M-A could not be seen, and a coincidence of the chosen shapes rather than a property of the kernel. The collapse to one chunk per thread (nchunk0 * nchunk1 < nth * 4) is live and is the shipped default, since VT_CPU_MATMUL_STEAL is off. Cheap to close rather than to file, so a case now runs at rows = 128, n = 16, nth = 4 — a live non-NUMA shape where the collapse gives dr1 = 32 and a chunk-sized buffer costs two tiles per worker instead of one.

One correction to that finding as it was written. The review called it "a real ~3x regression the gate cannot see on a NUMA host", citing dr1 = ceil(rows/nth) = 52 at rows = 1024, nth = 20. Two things are off. IsNuma() is constexpr false in this tree (cpu_threadpool.h:74; NUMA is recorded as unported at cpu_threadpool.h:25), so that branch is dead today and the gap is reachable through the live collapse instead. And at the shipped LTX-2.5 caption projection the weight is the longer axis (n = 4096 > rows = 1024), so the collapse takes nchunk1 = 1 and dr1 = 1024, not 52 — a chunk-sized af there would be 770 MB per worker and 15.4 GB across 20, which is #1286's hypothesised shape almost exactly. The finding was right that the gate was blind; the magnitude is larger than it said and the trigger is nearer.

The mutation table

Release, 20-core Zen 5 under KVM. Every row rebuilt and re-run; sha256(cpu_ops.cpp) = dc39eccdece48879e82be7209e95d182c6ed624eb04389de689fa4b58fe4f1f3 before and after every one of them.

mutation defect written into cpu_ops.cpp BUILT / compile_err git diff --stat on src/ at 39b34c0c8 after
M-A nrows = the chunk's row span YES / 0 cpu_ops.cpp | 2 +- green, 13/13 exit 1, 2 of 38 failed — 33,554,496 <= 17,825,792 on both arms of the collapsed case
M-A2 nrows = the whole activation YES / 0 cpu_ops.cpp | 2 +- red exit 1, 14 of 38 failed
M-B a second tile-sized buffer per worker YES / 0 cpu_ops.cpp | 3 + green, 13/13 exit 1, 10 of 38 failed — 25,165,888 <= 17,825,792 at 4 workers, 58,720,320 <= 34,603,008 at 8
M-D whole activation via 64-byte-aligned operator new YES / 0 cpu_ops.cpp | 12 + green, 13/13 exit 1, 14 of 38 failed — 1,019,215,936 bytes at rows=1024, 4 workers
M-E whole activation via std::malloc YES / 0 cpu_ops.cpp | 12 + green, 13/13 exit 1, 14 of 38 failed — 71,303,232 <= 9,437,184 at 2 workers, 1,086,324,800 bytes at rows=1024, 4 workers
M-W the --wrap=malloc link flag removed NO / 1 CMakeLists.txt | 1 - n/a link fails: undefined reference to '__real_malloc' — the coverage cannot be dropped silently

The sanitize-cpu (address,undefined) risk was exercised, and the record is in two parts because the second attempt was blocked. The FIRST widening — the aligned operator new family, before --wrap existed — ran in the CI lane's own configuration (-DVLLM_CPP_SANITIZE='address,undefined', -DVLLM_CPP_CUDA=OFF, UBSAN_OPTIONS=print_stacktrace=1): 5 cases, 36 assertions, 0 failed, exit 0, no ASan or UBSan diagnostic. The thread lane was green too, once ASLR was disabled to work around this box's FATAL: ThreadSanitizer: unexpected memory mapping — a local kernel/TSan incompatibility, not a finding.

Rebuilding that lane after --wrap was added hit No space left on device — other sessions filled the shared disk to 100%, and the build's first output was five fatal error: error writing to /tmp/...: No space left on device lines, which is an infrastructure failure wearing the costume of a code verdict. So the full lane was NOT re-run with --wrap in place, and this says so rather than implying otherwise.

What ran in its place is a standalone probe reproducing the gate's entire allocator mechanism in miniature — replaced plain and aligned operator new/delete, the five --wrap redirections, and a separate translation unit standing in for cpu_ops.cpp — compiled and linked with -fsanitize=address,undefined. It builds clean and runs clean: all three C-allocator routes intercepted (3 MiB accounted), 64-byte alignment preserved through the replaced aligned new, replaced operator new accounting correct, no diagnostic, exit 0. That establishes the mechanism against ASan, which is precisely the collision the reviewer named. It does not stand in for a full-lane run and is not offered as one.

The bound is deliberately tight now, and that is the point. At 8 workers the honest worst case is 8 tiles — every worker fresh, which the liveness case reaches — at 33,554,496 bytes against a bound of 34,603,008. The 1 MiB of headroom is 16,000x the measured incidental. Widening it again buys nothing except the 2x hole M-B just walked through.

Records repaired in the same change

A wrong line anchor, cited six times. cpu_ops.cpp:125 is cited for static thread_local std::vector<float> af — it is at line 130; :125 is const int64_t blck_1 = 16;, and the comment block cited as 122-125 is at 127-129. Wrong at the named baseline, at the implementation commit and at HEAD. Fixed in the spec, the benchmark-record and the test header.

It CANNOT be fixed in .agents/issue-index.md:423, which cites it twice. That file is append-only and keyed, its checker reads commits rather than the working tree, and a corrective row would be a duplicate key its checker refuses. So the correction is recorded where a reader following the row arrives — the spec's ## Owed, and this body. The same wrong anchor is frozen in the bodies of 0377cde70 and 8d46223fd for the same reason.

The uncovered class behind it is recorded, not fixed here. check-symbol-anchors.py:87-90 requires a literal :: between path and identifier, so it cannot match a path:line anchor at all. Counted over every .md in this tree: 613 path::Symbol citations and 14,448 path:line citations, the second checked by nothing. A 24:1 ratio deserves its own row and its own red-first mutation, not a follow-up line here.

tests/CMakeLists.txt gave a false rationale for a live conditional. It called the test "the seam's PEAK RESIDENT bound" and justified the Linux-only guard with "because the measurement is /proc/self/{status,clear_refs}". The shipped test measures neither and opens nothing under /proc; it landed in the same commit as the shipped instrument, so it was wrong rather than historical. The real reason is stated now: the gate replaces the global allocation functions for the whole program, so it needs every delete in the binary to reach the matching replaced delete, and its fixed term calibrated against the library doing the allocating — both established for glibc/libstdc++ and nothing else.

Three record corrections beside them. The spec's ## Tests to port and ## Risks/decisions describe the refuted VmHWM instrument while ## Outcome W4 records that it was replaced, so both now carry a forward pointer. ## Outcome W2's first table was headed "at the shipped geometry" while running rows = 64, breaking the spec's own rule at :158-162 that a scaled probe is never presented as a full-geometry one; the reduction sits beside the number now. And the "threaded stretch sits at anon = 34.30" triple was that stretch's minima at a single sample — recomputed above from the retained memguard.tsv.

runguard.py's line citations are unstable and were not chased. The file lives outside this repository on a mutable NAS path and was rewritten after the implementation commit, so runguard.py:236-237,260 no longer resolves reproducibly. The claim does not rest on it: in the retained memguard.tsv the #1252 run's first sample reads avail = 88.078, used = 31.553 — summing to 119.631 GiB, the box's whole MemTotal — beside rss = 0.001 for a child that had just started. A column reading 31.553 GiB before the process has allocated anything is system-wide by construction, whatever line computes it.

Gates

  • test_ops_matmul_mem: 5 cases, 38 assertions, 0 failed, exit 0, Status: SUCCESS! — up from 2 cases and 13 assertions.
  • test_ops_matmul_mem under VLLM_CPP_SANITIZE='address,undefined' (before --wrap; see above for why it could not be re-run after): 5 cases, 36 assertions, 0 failed, exit 0, no sanitizer diagnostic.
  • test_ops_matmul_mem under VLLM_CPP_SANITIZE='thread' (ASLR disabled to work around this box's FATAL: ThreadSanitizer: unexpected memory mapping): 5 cases, 36 assertions, 0 failed, exit 0.
  • Full ctest -j 6 after merging origin/main edbc47ce0, on this exact head: 100% tests passed, 0 tests failed out of 554, 3 skipped by design, 995.03 s. The box was heavily contended by other sessions throughout (loadavg 180-333 on 20 cores, 179 competing processes), so no wall time in that run is a measurement of anything — only the pass/fail column is. Nothing was re-run to make it green.
  • scripts/check-agent-record.py: agent record OK: ENGINE=165 MODEL=377 QUANT=84 KERNEL=52 BACKEND=85 ANCHOR-ROT=38.
  • scripts/check-doc-checkpoint.py --base origin/main --head HEAD: OK: public documents match the claims this change makes. It failed first — a benchmark-record change owes docs/BENCHMARKS.md and docs/STATUS.md, and the projection had to ride in the SAME commit as the record edit.
  • scripts/check-public-doc-tables.py: OK. This one failed first too, and correctly: the projection as first written was a 496-char table cell against a 220 limit, and pushed docs/STATUS.md to 45 oversized cells against its 44 ratchet. Both were shortened into budget rather than the ratchet being raised.
  • scripts/check-commit-trailers.py and scripts/check-commit-style.py over origin/main..HEAD: both OK.
  • scripts/check-issue-index-append-only.py --base origin/main: OK: issue index append-only. 413 rows, zero duplicate keys, #1252's threaded path raises peak host memory ~79 -> 105.85 GiB and aborts the full-model render on GB10 #1286 and The full-model render harness compares ABSOLUTE MemAvailable peaks across runs that started 26.8 GiB apart, and #1286's whole regression is that offset #1317 each present exactly once.
  • Mutation evidence for every claim tightened or added: the table above, with BUILT, compile_err, git diff --stat and the failing assertion by name for each.

Byte-exactness is untouched — src/vt/cpu/cpu_ops.cpp is byte-identical to 39b34c0c8, sha256 dc39ecc...4f1f3, no tolerance was widened anywhere, and no summation order was altered. Every mutation above was applied to a scratch tree and restored.

Records, and commit order

.agents/specs/ltx25-text-linear-mem.md is committed before the implementation (0377cde70, then 8d46223fd). docs/STATUS.md and docs/BENCHMARKS.md carry the keyed projection; the forensics are in .agents/benchmark-record.md. Two index rows appended, #1286 owned by this row and #1317 owned by this spec's ## Owed.

The review repair rides in the same pull request rather than a new one: it is the record edit and the gate repair the reviewed change made necessary, which AGENTS.md puts in the pull request whose change made them stale. It touches four files — the test, its registration, the spec's ## Outcome W5 and ## Owed, and the benchmark-record entry — and no product code.

Owed, named rather than left to be found

  • #1317 — the harness comparison and the 26.8 GiB that was already on the box. This is what actually aborted the render, and both halves are outside this repository.
  • No GB10 number of any kind. Every measurement here is from a 20-core Zen 5. Allocation sizes are architecture-independent so the attribution carries; a throughput ratio does not, and none is claimed for GB10. fix(LTX25-TEXT-LINEAR-SEAM): the caption projection is one core for 671 s per pass, so route it through vt::MatmulBT #1252's unmeasured per-core ratio R is untouched.
  • The tile is retained for the process lifetime, PER INSTANTIATION. af grows to the largest K any GEMM on that worker ever saw and never shrinks — 232 MiB held until exit at K = 188160 on 20 workers, and there are two such buffers, one in each of MatmulOneChunk<false> and <true>, so a process running both orientations at that K retains up to 464 MiB. Bounded, deliberate, and now measured; not repaired here, because the buffer exists precisely so a worker allocates once, and trading that away is a performance decision with its own spec.
  • The counter does not see mmap/sbrk, nor an allocator reached inside a shared library's own internal calls. --wrap binds at THIS link, so a malloc that libstdc++.so calls internally is outside it; every global operator new is covered regardless. A kernel taking pages straight from mmap would be invisible to this gate. Nothing in the seam does, and no mutation attempts it, so this is a stated boundary rather than a measured gap.
  • path:line citations are checked by nothing. check-symbol-anchors.py:87-90 requires a literal ::, so of 613 path::Symbol and 14,448 path:line citations in this tree's markdown, only the first population is covered. That is how the wrong cpu_ops.cpp:125 here survived the implementation commit, the merge and the gate. Extending the checker needs its own spec and its own red-first mutation.
  • The cpu_ops.cpp:125 anchor cannot be corrected in .agents/issue-index.md:423. The index is append-only and keyed, so the row cannot be edited and a corrective row would be a duplicate key its checker refuses. The correction is recorded in the spec's ## Owed and in this body instead. The same wrong anchor is frozen in the bodies of 0377cde70 and 8d46223fd.
  • Linux-only registration is a calibration scope, not a platform dependency. Extending the gate to the macOS and Windows lanes is owed work; the counter itself is portable.
  • runguard.py:236-237,260 is an unstable citation — the file is outside this repository on a mutable NAS path and was rewritten after the implementation commit. The claim it supported was verified independently against the retained memguard.tsv columns and does not depend on it.
  • The bound is a CPU bound. vt::MatmulBT on CUDA/ROCm/Vulkan is not covered.
  • docs/BENCHMARKS.md and docs/STATUS.md carry the new keyed projection, which check-doc-checkpoint.py requires of a benchmark-record change: the allocation is bounded on both instantiations, and the retention figure is 464 MiB rather than 232.
  • No full-model render has been rerun, so nothing here says what one would now do — only that fix(LTX25-TEXT-LINEAR-SEAM): the caption projection is one core for 671 s per pass, so route it through vt::MatmulBT #1252 is not what stopped the last one.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]

mudler added 3 commits August 19, 2026 07:07
… it accuses

#1286 reports that #1252's threaded caption projection raises peak host memory
from ~79 GiB to 105.85 GiB and aborts the full-model LTX-2.5 render on GB10. It
names three candidate causes and says plainly that the attribution is not
measured. This spec takes the row and commits the attribution plan before any
code, because the three causes have different fixes and different blast radii:
one is a shared-seam defect affecting every GEMM caller, one is an LTX call
site, and one is neither.

The baseline is read out of the tree rather than assumed. `vt::MatmulBT`'s CPU
path allocates exactly one thing per call, `cpu_ops.cpp:125`'s `static
thread_local` widened-activation tile, sized by ggml's 16-row `blck_1` and not
by the chunk's row span; the call site's full-size `scaled` copy is present on
BOTH arms and so cannot explain a regression; and no test anywhere in this tree
asserts a peak-RSS or allocation bound, so the reported defect would have been
invisible to every gate had it been real.

The row therefore carries a fourth item that is worth having whatever the
attribution says: a memory bound for the shared GEMM seam, red by mutating the
tile to the shape #1286 hypothesised.

Refs #1286

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]
…e, and the seam had no memory bound at all (#1286)

#1286 reported that routing the LTX-2.5 caption projection through
`vt::MatmulBT` raised peak host memory from ~79 GiB to 105.85 GiB and aborted a
full-model render on GB10, and named three candidates. It is none of them.

`runguard.py:236-237,260` fixes the compared column as the SYSTEM-WIDE
`MemTotal - MemAvailable`. The pre-#1252 run began at `used = 4.741 GiB`; the
#1252 run began at `used = 31.553 GiB`. That is 26.812 GiB against a claimed
regression of 26.647 GiB. Each run's own demand -- peak minus its own t=0 -- is
74.465 GiB before and 74.300 GiB after, with system AnonPages deltas of 38.012
vs 38.217 and child VmRSS at each peak sample of 41.952 vs 42.090. Both peaks
were sampled inside a ~1900% CPU stretch, so they are the same phase class. On a
box as clean as the first run's, the second binary's own demand would have left
40.59 GiB available, above both floors.

Measured locally as well, since the issue asked for attribution rather than
inference: at the shipped K = 188160 the seam costs 232 MiB of peak RSS at 20
workers against the scalar loop's 1.0 MiB, and is 8.43x faster -- 0.85% of the
memory #1286 attributes to it, with the speedup intact. Swept over threadpool
width the growth is 11.6 MiB per worker, which is `cpu_ops.cpp:125`'s
`static thread_local` tile at 16 x K x 4 and nothing else, and the output
checksum is byte-identical across all six widths. #1259 measures 0.52 MiB at 20
workers and needs nothing.

What the attribution found is that NOTHING IN THIS TREE BOUNDS THE SEAM'S
MEMORY. Every GEMM gate asserts values or byte equality, so a kernel that
allocated a whole intermediate per worker would have passed all of them until a
box ran out. `tests/vt/test_ops_matmul_mem.cpp` closes that. It counts what the
seam asks for through a replaced global `operator new` rather than measuring
peak RSS, because the peak-RSS draft read `growth_bytes = 0` at every thread
count and every row count while passing every bound -- the operands are freed
between measurements and glibc serves the next tile from resident pages. A
liveness case beside it requires the counter to see six of eight fresh workers
take a tile, so the bound cannot pass on a silent instrument.

No product code changes. The two record-only edits and the issue this filed are
in the pull request body.

Refs #1286, #1252, #1259, #1317

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]
Five commits landed while this row measured #1286. The append to
.agents/issue-index.md is the only surface both sides touch, and it is
append-only under merge=union, so both sets of rows are kept and no key is
duplicated.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]
mudler added 3 commits August 19, 2026 12:00
Keeps the branch fast-forwardable onto main and reruns the row's gate on the
merged tree rather than on a stale base.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]
…gressions, and its review proved all three (#1286)

The fresh review of `39b34c0c8` returned nine findings and no PASS. The
refutation of #1286 is confirmed and unchanged; what failed review was the gate
that landed beside it. Three of the reviewer's mutations wrote an obvious memory
regression into `cpu_ops.cpp` and passed **13 of 13 assertions**.

**The bound admitted a 2x per-worker regression while its comment said it could
not.** `Bound(n)` was `8 MiB + n * 2 * kTileBytes` — a per-thread term of
literally two tiles, under a comment claiming it was "not wide enough to admit a
second tile-sized buffer per worker". Mutation M-B added exactly that and
doubled the growth at every worker count, green. At the shipped geometry that is
232 -> 464 MiB shipping silently. The bound is now `1 MiB + n * kTileBytes`,
which is the property the file actually claims, over a fixed term measured at
**64 bytes**. The comment now names the worker counts that detect a doubling
(4 and up) instead of claiming all of them do.

**The counter was bypassed by an over-aligned `operator new`.** Only the
non-aligned overloads were replaced, so the C++17 `std::align_val_t` family fell
through to the library. M-D delivered the whole-activation defect through a
64-byte-aligned `operator new` — the natural shape for a SIMD scratch buffer in
this kernel — and read byte-identical growth, green. All eight aligned
`new`/`delete` overloads are now replaced.

**`std::malloc` is seen too, through the LINKER rather than a symbol
definition.** M-E delivered the same defect through `std::malloc` and the first
pass of this repair left it green, narrowing the claim instead. That was the
wrong call. `-Wl,--wrap=malloc` and four siblings, scoped to this one target,
redirect the calls made by the objects in this link — `libvllm.a`, and so
`cpu_ops.cpp` — without defining `malloc`, so AddressSanitizer's interceptor
keeps its symbol and there is no second strong definition to collide with.
**M-E now reds at 1,086,324,800 bytes.** `free` is deliberately not wrapped, so
the C-allocator figure is bytes REQUESTED cumulatively: it errs toward red, and
it reads exactly zero on a clean run.

A new COVERAGE case makes the whole claim executable — it allocates through
plain `new`, array `new`, over-aligned `new` at two alignments, and
`std::malloc`, and requires the counter to move by the amount asked for. Two
failure modes, two mechanisms: dropping a `--wrap` flag fails the LINK
(`undefined reference to '__real_malloc'`), and the subtler "route present but
folded away" mode is what the assertion catches. That case calls through a
`volatile` function pointer for a measured reason — a direct `std::malloc` in
the test's own translation unit is folded by the compiler and never becomes a
symbol reference, so it is never wrapped. Measured in a standalone probe:
same-TU calls gave `malloc=0 calloc=0 posix_memalign=1`, the identical calls
from a separate TU gave `malloc=1 calloc=1 posix_memalign=1`. Product code is
always the separate-TU case.

**`MatmulOneChunk<false>` had no assertion behind it.** `<false>` and `<true>`
are separate instantiations with separate `thread_local` buffers, and the gate
called only `vt::MatmulBT` on a non-repacked weight. `vt::Matmul` and
`MatmulBTKernel`'s `elem_kn_repacked` lever both route to `<false>`, and
`vt::Matmul` is #1259's `Ltx2FuseLoraIntoTensor` path — merged and unexercised
on the full model. Both arms are measured now, and the measurement confirms the
two buffers are distinct: 8 fresh workers read 33,554,496 bytes on `MatmulBT`
and 33,554,496 again on the `vt::Matmul` immediately after, eight tiles both
times rather than eight then seven. The retention figure is therefore **per
instantiation** — up to 464 MiB, not 232 MiB.

**The chunk-row-span defect was invisible at every geometry the gate ran.** M-A
(`nrows = ir1_end - iir1`) produced byte-identical growth, green. Replaying
`MatmulChunked`'s grid arithmetic over all twelve shapes the sweeps use gives
`dr1` = 16 at eleven and 8 at the twelfth — never above 16, which is a
coincidence of the chosen shapes rather than a property of the kernel. The
collapse to one chunk per thread is LIVE and the shipped default, so a case now
runs at `rows = 128, n = 16, nth = 4`, a live non-NUMA shape where it gives
`dr1 = 32`.

One correction to that finding as written. The review called it "a real ~3x
regression the gate cannot see on a NUMA host", citing `dr1 = 52` at
`rows = 1024, nth = 20`. `IsNuma()` is `constexpr false` here
(`cpu_threadpool.h:74`, NUMA unported per `:25`), so that branch is dead today
and the gap is reachable through the live collapse instead. And at the shipped
caption projection the weight is the longer axis (`n = 4096 > rows = 1024`), so
the collapse takes `nchunk1 = 1` and `dr1 = 1024`, not 52 — 770 MB per worker
and 15.4 GB across 20. The finding was right that the gate was blind; the
magnitude is larger and the trigger is nearer.

The mutation table. Every row rebuilt and re-run;
`sha256(cpu_ops.cpp) = dc39ecc...4f1f3` before and after each.

| mutation | `BUILT`/`compile_err` | `39b34c0c8` | after |
|---|---|---|---|
| M-A, chunk row span | `YES`/0 | green 13/13 | exit 1, 2 of 36 failed |
| M-A2, whole activation | `YES`/0 | red | exit 1, 14 of 36 failed |
| M-B, second tile per worker | `YES`/0 | green 13/13 | exit 1, 10 of 36 failed |
| M-D, whole activation via aligned `new` | `YES`/0 | green 13/13 | exit 1, 14 of 36 failed |
| M-E, whole activation via `std::malloc` | `YES`/0 | green 13/13 | exit 1, 14 of 38 failed |
| M-W, the `--wrap=malloc` flag removed | **`NO`**/1 | n/a | link fails, `undefined reference to '__real_malloc'` |

Green after the repair: **5 cases, 38 assertions, 0 failed, exit 0**, against 2
cases and 13 assertions before it. The other mutation rows are M-A (2 of 38
failed), M-A2 (14), M-B (10) and M-D (14), each `BUILT=YES compile_err=0` with
`git diff --stat` recorded, and `sha256(cpu_ops.cpp)` restored to
`dc39ecc...4f1f3` after every one.

The `sanitize-cpu` risk was exercised in two parts, and the second was blocked.
The aligned-`operator new` widening ran in the lane's own configuration
(`VLLM_CPP_SANITIZE='address,undefined'`, `VLLM_CPP_CUDA=OFF`,
`UBSAN_OPTIONS=print_stacktrace=1`): 5 cases, 36 assertions, 0 failed, exit 0,
no diagnostic; `thread` likewise, with ASLR disabled to work around this box's
`FATAL: unexpected memory mapping`. Rebuilding that lane after `--wrap` was
added hit `No space left on device` — other sessions filled the shared disk to
100% — so it was NOT re-run with `--wrap` in place, and this says so rather than
implying otherwise. In its place, a standalone probe reproducing the whole
mechanism (replaced plain and aligned `operator new`/`delete`, the five `--wrap`
redirections, and a separate TU standing in for `cpu_ops.cpp`) built and ran
clean under `-fsanitize=address,undefined`: all three C-allocator routes
intercepted, alignment preserved, no diagnostic, exit 0. That establishes the
mechanism against ASan; it is not a full-lane run.

Records repaired in the same change. `cpu_ops.cpp:125` is cited for
`static thread_local std::vector<float> af`, which is at **line 130** — `:125`
is `const int64_t blck_1 = 16;`. Fixed in the spec, the benchmark-record and the
test header. It cannot be fixed in `.agents/issue-index.md:423`, which is
append-only and keyed, so the correction is recorded in the spec's `## Owed` and
in this body instead; a corrective row would be a duplicate key. The uncovered
class behind it is recorded too: `check-symbol-anchors.py:87-90` requires a
literal `::`, so of 613 `path::Symbol` and **14,448 `path:line`** citations in
this tree's markdown, the second number is checked by nothing.

`tests/CMakeLists.txt` justified the Linux-only guard with "the measurement is
/proc/self/{status,clear_refs}". The shipped test opens nothing under `/proc`.
The real reason is now stated: the gate replaces the global allocation functions
for the whole program, so it needs every `delete` in the binary to reach the
matching replaced `delete` and its fixed term calibrated against the library
doing the allocating — both established for glibc/libstdc++ and nothing else.

Three record corrections beside them. The spec's `## Tests to port` and
`## Risks/decisions` describe the refuted `VmHWM` instrument, so both now point
forward to `## Outcome` W4. `## Outcome` W2's first table was headed "at the
shipped geometry" while running `rows = 64`, breaking the spec's own rule at
`:158-162`; the reduction is beside the number now. And the "threaded stretch
sits at `anon = 34.30`" triple was that stretch's minima at a single sample:
recomputed from the retained `memguard.tsv`, the stretch is a **ramp** —
`anon` 34.280 -> 42.593, `used` 97.350 -> 105.853 over 236 samples — while the
pre-#1252 run genuinely is flat (`anon` 34.283-34.441 over 1676). The two
coincide only at the ramp's first sample. W1's argument is peak-minus-own-`t=0`
and does not rest on it.

`docs/BENCHMARKS.md` and `docs/STATUS.md` carry the new keyed projection, which
`check-doc-checkpoint.py` requires of a benchmark-record change: the seam's
allocation is bounded on both instantiations, and the process-lifetime retention
is 464 MiB rather than 232.

No product code changed. `src/vt/cpu/cpu_ops.cpp` is byte-identical to
`39b34c0c8`; every mutation above was applied to a scratch tree and restored to
the same sha256.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]
…d appends

GitHub refused the squash while the local merge is clean on the issue index,
which is the usual `merge=union` blind spot on the forge. This merge also
carries one real conflict the union driver does not cover.

`.agents/benchmark-record.md` took an append from each side into the same
section: `BENCH-QWEN38-27B-BF16` on main and `LTX25-TEXT-LINEAR-MEM` here. They
are independent entries about different rows, so both are kept, main's first,
leaving this branch's as the newest append. No existing entry was edited, which
is what AGENTS.md requires of a keyed record: take the target-branch version and
re-apply the scoped edit rather than accept a three-way merge.

The issue index is verified rather than assumed, because the union driver can
interleave rows and reinstate a deleted one.

This branch carries no product code, so the merge cannot change a build result.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]
@localai-bot
localai-bot merged commit 2f67c93 into main Aug 19, 2026
1 check failed
@localai-bot
localai-bot deleted the row/LTX25-TEXT-LINEAR-MEM branch August 19, 2026 13:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants