fix(LTX25-TEXT-LINEAR-MEM): #1252's +26 GiB is the box, not the change, and the seam had no memory bound at all (#1286) - #1331
Merged
Conversation
… it accuses #1286 reports that #1252's threaded caption projection raises peak host memory from ~79 GiB to 105.85 GiB and aborts the full-model LTX-2.5 render on GB10. It names three candidate causes and says plainly that the attribution is not measured. This spec takes the row and commits the attribution plan before any code, because the three causes have different fixes and different blast radii: one is a shared-seam defect affecting every GEMM caller, one is an LTX call site, and one is neither. The baseline is read out of the tree rather than assumed. `vt::MatmulBT`'s CPU path allocates exactly one thing per call, `cpu_ops.cpp:125`'s `static thread_local` widened-activation tile, sized by ggml's 16-row `blck_1` and not by the chunk's row span; the call site's full-size `scaled` copy is present on BOTH arms and so cannot explain a regression; and no test anywhere in this tree asserts a peak-RSS or allocation bound, so the reported defect would have been invisible to every gate had it been real. The row therefore carries a fourth item that is worth having whatever the attribution says: a memory bound for the shared GEMM seam, red by mutating the tile to the shape #1286 hypothesised. Refs #1286 FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
…e, and the seam had no memory bound at all (#1286) #1286 reported that routing the LTX-2.5 caption projection through `vt::MatmulBT` raised peak host memory from ~79 GiB to 105.85 GiB and aborted a full-model render on GB10, and named three candidates. It is none of them. `runguard.py:236-237,260` fixes the compared column as the SYSTEM-WIDE `MemTotal - MemAvailable`. The pre-#1252 run began at `used = 4.741 GiB`; the #1252 run began at `used = 31.553 GiB`. That is 26.812 GiB against a claimed regression of 26.647 GiB. Each run's own demand -- peak minus its own t=0 -- is 74.465 GiB before and 74.300 GiB after, with system AnonPages deltas of 38.012 vs 38.217 and child VmRSS at each peak sample of 41.952 vs 42.090. Both peaks were sampled inside a ~1900% CPU stretch, so they are the same phase class. On a box as clean as the first run's, the second binary's own demand would have left 40.59 GiB available, above both floors. Measured locally as well, since the issue asked for attribution rather than inference: at the shipped K = 188160 the seam costs 232 MiB of peak RSS at 20 workers against the scalar loop's 1.0 MiB, and is 8.43x faster -- 0.85% of the memory #1286 attributes to it, with the speedup intact. Swept over threadpool width the growth is 11.6 MiB per worker, which is `cpu_ops.cpp:125`'s `static thread_local` tile at 16 x K x 4 and nothing else, and the output checksum is byte-identical across all six widths. #1259 measures 0.52 MiB at 20 workers and needs nothing. What the attribution found is that NOTHING IN THIS TREE BOUNDS THE SEAM'S MEMORY. Every GEMM gate asserts values or byte equality, so a kernel that allocated a whole intermediate per worker would have passed all of them until a box ran out. `tests/vt/test_ops_matmul_mem.cpp` closes that. It counts what the seam asks for through a replaced global `operator new` rather than measuring peak RSS, because the peak-RSS draft read `growth_bytes = 0` at every thread count and every row count while passing every bound -- the operands are freed between measurements and glibc serves the next tile from resident pages. A liveness case beside it requires the counter to see six of eight fresh workers take a tile, so the bound cannot pass on a silent instrument. No product code changes. The two record-only edits and the issue this filed are in the pull request body. Refs #1286, #1252, #1259, #1317 FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
Five commits landed while this row measured #1286. The append to .agents/issue-index.md is the only surface both sides touch, and it is append-only under merge=union, so both sets of rows are kept and no key is duplicated. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
Keeps the branch fast-forwardable onto main and reruns the row's gate on the merged tree rather than on a stale base. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
…gressions, and its review proved all three (#1286) The fresh review of `39b34c0c8` returned nine findings and no PASS. The refutation of #1286 is confirmed and unchanged; what failed review was the gate that landed beside it. Three of the reviewer's mutations wrote an obvious memory regression into `cpu_ops.cpp` and passed **13 of 13 assertions**. **The bound admitted a 2x per-worker regression while its comment said it could not.** `Bound(n)` was `8 MiB + n * 2 * kTileBytes` — a per-thread term of literally two tiles, under a comment claiming it was "not wide enough to admit a second tile-sized buffer per worker". Mutation M-B added exactly that and doubled the growth at every worker count, green. At the shipped geometry that is 232 -> 464 MiB shipping silently. The bound is now `1 MiB + n * kTileBytes`, which is the property the file actually claims, over a fixed term measured at **64 bytes**. The comment now names the worker counts that detect a doubling (4 and up) instead of claiming all of them do. **The counter was bypassed by an over-aligned `operator new`.** Only the non-aligned overloads were replaced, so the C++17 `std::align_val_t` family fell through to the library. M-D delivered the whole-activation defect through a 64-byte-aligned `operator new` — the natural shape for a SIMD scratch buffer in this kernel — and read byte-identical growth, green. All eight aligned `new`/`delete` overloads are now replaced. **`std::malloc` is seen too, through the LINKER rather than a symbol definition.** M-E delivered the same defect through `std::malloc` and the first pass of this repair left it green, narrowing the claim instead. That was the wrong call. `-Wl,--wrap=malloc` and four siblings, scoped to this one target, redirect the calls made by the objects in this link — `libvllm.a`, and so `cpu_ops.cpp` — without defining `malloc`, so AddressSanitizer's interceptor keeps its symbol and there is no second strong definition to collide with. **M-E now reds at 1,086,324,800 bytes.** `free` is deliberately not wrapped, so the C-allocator figure is bytes REQUESTED cumulatively: it errs toward red, and it reads exactly zero on a clean run. A new COVERAGE case makes the whole claim executable — it allocates through plain `new`, array `new`, over-aligned `new` at two alignments, and `std::malloc`, and requires the counter to move by the amount asked for. Two failure modes, two mechanisms: dropping a `--wrap` flag fails the LINK (`undefined reference to '__real_malloc'`), and the subtler "route present but folded away" mode is what the assertion catches. That case calls through a `volatile` function pointer for a measured reason — a direct `std::malloc` in the test's own translation unit is folded by the compiler and never becomes a symbol reference, so it is never wrapped. Measured in a standalone probe: same-TU calls gave `malloc=0 calloc=0 posix_memalign=1`, the identical calls from a separate TU gave `malloc=1 calloc=1 posix_memalign=1`. Product code is always the separate-TU case. **`MatmulOneChunk<false>` had no assertion behind it.** `<false>` and `<true>` are separate instantiations with separate `thread_local` buffers, and the gate called only `vt::MatmulBT` on a non-repacked weight. `vt::Matmul` and `MatmulBTKernel`'s `elem_kn_repacked` lever both route to `<false>`, and `vt::Matmul` is #1259's `Ltx2FuseLoraIntoTensor` path — merged and unexercised on the full model. Both arms are measured now, and the measurement confirms the two buffers are distinct: 8 fresh workers read 33,554,496 bytes on `MatmulBT` and 33,554,496 again on the `vt::Matmul` immediately after, eight tiles both times rather than eight then seven. The retention figure is therefore **per instantiation** — up to 464 MiB, not 232 MiB. **The chunk-row-span defect was invisible at every geometry the gate ran.** M-A (`nrows = ir1_end - iir1`) produced byte-identical growth, green. Replaying `MatmulChunked`'s grid arithmetic over all twelve shapes the sweeps use gives `dr1` = 16 at eleven and 8 at the twelfth — never above 16, which is a coincidence of the chosen shapes rather than a property of the kernel. The collapse to one chunk per thread is LIVE and the shipped default, so a case now runs at `rows = 128, n = 16, nth = 4`, a live non-NUMA shape where it gives `dr1 = 32`. One correction to that finding as written. The review called it "a real ~3x regression the gate cannot see on a NUMA host", citing `dr1 = 52` at `rows = 1024, nth = 20`. `IsNuma()` is `constexpr false` here (`cpu_threadpool.h:74`, NUMA unported per `:25`), so that branch is dead today and the gap is reachable through the live collapse instead. And at the shipped caption projection the weight is the longer axis (`n = 4096 > rows = 1024`), so the collapse takes `nchunk1 = 1` and `dr1 = 1024`, not 52 — 770 MB per worker and 15.4 GB across 20. The finding was right that the gate was blind; the magnitude is larger and the trigger is nearer. The mutation table. Every row rebuilt and re-run; `sha256(cpu_ops.cpp) = dc39ecc...4f1f3` before and after each. | mutation | `BUILT`/`compile_err` | `39b34c0c8` | after | |---|---|---|---| | M-A, chunk row span | `YES`/0 | green 13/13 | exit 1, 2 of 36 failed | | M-A2, whole activation | `YES`/0 | red | exit 1, 14 of 36 failed | | M-B, second tile per worker | `YES`/0 | green 13/13 | exit 1, 10 of 36 failed | | M-D, whole activation via aligned `new` | `YES`/0 | green 13/13 | exit 1, 14 of 36 failed | | M-E, whole activation via `std::malloc` | `YES`/0 | green 13/13 | exit 1, 14 of 38 failed | | M-W, the `--wrap=malloc` flag removed | **`NO`**/1 | n/a | link fails, `undefined reference to '__real_malloc'` | Green after the repair: **5 cases, 38 assertions, 0 failed, exit 0**, against 2 cases and 13 assertions before it. The other mutation rows are M-A (2 of 38 failed), M-A2 (14), M-B (10) and M-D (14), each `BUILT=YES compile_err=0` with `git diff --stat` recorded, and `sha256(cpu_ops.cpp)` restored to `dc39ecc...4f1f3` after every one. The `sanitize-cpu` risk was exercised in two parts, and the second was blocked. The aligned-`operator new` widening ran in the lane's own configuration (`VLLM_CPP_SANITIZE='address,undefined'`, `VLLM_CPP_CUDA=OFF`, `UBSAN_OPTIONS=print_stacktrace=1`): 5 cases, 36 assertions, 0 failed, exit 0, no diagnostic; `thread` likewise, with ASLR disabled to work around this box's `FATAL: unexpected memory mapping`. Rebuilding that lane after `--wrap` was added hit `No space left on device` — other sessions filled the shared disk to 100% — so it was NOT re-run with `--wrap` in place, and this says so rather than implying otherwise. In its place, a standalone probe reproducing the whole mechanism (replaced plain and aligned `operator new`/`delete`, the five `--wrap` redirections, and a separate TU standing in for `cpu_ops.cpp`) built and ran clean under `-fsanitize=address,undefined`: all three C-allocator routes intercepted, alignment preserved, no diagnostic, exit 0. That establishes the mechanism against ASan; it is not a full-lane run. Records repaired in the same change. `cpu_ops.cpp:125` is cited for `static thread_local std::vector<float> af`, which is at **line 130** — `:125` is `const int64_t blck_1 = 16;`. Fixed in the spec, the benchmark-record and the test header. It cannot be fixed in `.agents/issue-index.md:423`, which is append-only and keyed, so the correction is recorded in the spec's `## Owed` and in this body instead; a corrective row would be a duplicate key. The uncovered class behind it is recorded too: `check-symbol-anchors.py:87-90` requires a literal `::`, so of 613 `path::Symbol` and **14,448 `path:line`** citations in this tree's markdown, the second number is checked by nothing. `tests/CMakeLists.txt` justified the Linux-only guard with "the measurement is /proc/self/{status,clear_refs}". The shipped test opens nothing under `/proc`. The real reason is now stated: the gate replaces the global allocation functions for the whole program, so it needs every `delete` in the binary to reach the matching replaced `delete` and its fixed term calibrated against the library doing the allocating — both established for glibc/libstdc++ and nothing else. Three record corrections beside them. The spec's `## Tests to port` and `## Risks/decisions` describe the refuted `VmHWM` instrument, so both now point forward to `## Outcome` W4. `## Outcome` W2's first table was headed "at the shipped geometry" while running `rows = 64`, breaking the spec's own rule at `:158-162`; the reduction is beside the number now. And the "threaded stretch sits at `anon = 34.30`" triple was that stretch's minima at a single sample: recomputed from the retained `memguard.tsv`, the stretch is a **ramp** — `anon` 34.280 -> 42.593, `used` 97.350 -> 105.853 over 236 samples — while the pre-#1252 run genuinely is flat (`anon` 34.283-34.441 over 1676). The two coincide only at the ramp's first sample. W1's argument is peak-minus-own-`t=0` and does not rest on it. `docs/BENCHMARKS.md` and `docs/STATUS.md` carry the new keyed projection, which `check-doc-checkpoint.py` requires of a benchmark-record change: the seam's allocation is bounded on both instantiations, and the process-lifetime retention is 464 MiB rather than 232. No product code changed. `src/vt/cpu/cpu_ops.cpp` is byte-identical to `39b34c0c8`; every mutation above was applied to a scratch tree and restored to the same sha256. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
…d appends GitHub refused the squash while the local merge is clean on the issue index, which is the usual `merge=union` blind spot on the forge. This merge also carries one real conflict the union driver does not cover. `.agents/benchmark-record.md` took an append from each side into the same section: `BENCH-QWEN38-27B-BF16` on main and `LTX25-TEXT-LINEAR-MEM` here. They are independent entries about different rows, so both are kept, main's first, leaving this branch's as the newest append. No existing entry was edited, which is what AGENTS.md requires of a keyed record: take the target-branch version and re-apply the scoped edit rather than accept a three-way merge. The issue index is verified rather than assumed, because the union driver can interleave rows and reinstate a deleted one. This branch carries no product code, so the merge cannot change a build result. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
fix(LTX25-TEXT-LINEAR-MEM): #1252's +26 GiB is the box, not the change, and the seam had no memory bound at all
#1286 reported that routing the LTX-2.5 caption projection through
vt::MatmulBT(#1252) raised peak host memory from ~79 GiB to 105.85 GiB on a 119 GiB GB10 and aborted a full-model render before any denoise step. It named three candidates — the shared threadpool's per-worker tiles, the call site's full-sizescaledcopy, or an interaction — and said plainly that the attribution was not measured.It is none of the three. The attribution came first, as the issue asked, and it changed what this row lands.
The +26 GiB is the machine, measured on the retained evidence
runguard.py:236-237,260fixes what the compared column is:used_gibis the system-wideMemTotal - MemAvailable,anon_gibis system-wideAnonPages, and onlyrss_gibbelongs to the child. Both runs are retained under/mnt/nas_share/rc/ltx25-fullmodel/out/, which is what makes this answerable at all.1024x576-25f/20260818T220620Z/1024x576-25f/used_gibatt=0avail_gibatt=0used_gibcpu=1885.5%cpu=1925.2%The box carried 26.812 GiB before
ltx2-genwas started in the second run. The reported regression is105.853 - 79.206 =26.647 GiB. They agree to 0.165 GiB.Subtract each run's own baseline and the two are the same binary's worth of memory, on three independent columns:
used_gibpeak minus ownt=0anon_gibpeak minus ownt=0rss_gibat the peak sampleBoth peaks were sampled inside a ~1900% CPU stretch, so this is the same phase class rather than two different ones. On a box as clean as the first run's, the #1252 binary's own 74.300 GiB would have left 40.59 GiB available — above the 12 GiB hard floor and the 8 GiB projection floor, and in line with the "never below ~40 GiB" the pre-#1252 runs showed. The guard fired correctly on a real condition; the condition was not the binary.
Two readings from the same evidence, recorded because each on its own would have been read as supporting #1286. During its single-core projection (
cpu ~ 100%, t=124-1801 s, 1676 samples) the pre-#1252 run is dead flat —anon34.283-34.441,rss33.484-33.570,used74.405-74.573 — while the #1252 run's threaded stretch (cpu > 500%, t=137.1-385.5 s, 236 samples) is a ramp rather than a plateau:anonclimbs 34.280 → 42.593,rss30.322 → 42.120,used97.350 → 105.853. The two coincide onanonat the ramp's FIRST SAMPLE (34.280 against 34.283) and nowhere after it, so that near-identity is where the ramp starts rather than a matching steady state;usedis 23 GiB apart there and 26.6 GiB apart at the peak. The argument above is peak-minus-own-t=0and does not rest on any single sample. And neither run ever loaded the DiT (dit_runs=0in both; the first was stopped from outside at t=3017.3 s,# TERMINAL signal=15), so neither peak is a whole-render peak and neither is presented as one.Filed as #1317, with both halves named: the harness compares absolute peaks of a system-wide column without recording the starting occupancy, and something held 26.8 GiB on
dgx:gpu0before the job began. Neither is in this repository, so neither is fixed in flow.Measured locally too, before and after, with the throughput beside it
Same probe source and flags on both arms, only the
Linearbody differing; peak RSS isVmHWMread before and after the call with every operand already allocated and touched. 20-core Zen 5 under KVM, Release,-ffp-contract=off, shippedK = 188160andout_features = 4096,rows = 64(reduced from the shipped 1024 to keep the single-threaded arm inside a sensible wall; the tile term the sweep isolates is independent of rows by construction). Box not idle, loadavg 5.4-13.3, three replicates, median:rows = 64, reduced)doubleloopvt::MatmulBT8.43x, and +231 MiB — the speedup reproduces #1252's 8.57x, and the memory cost is 0.85% of the 26.6 GiB #1286 attributes to the change, 117x too small to be it.
Swept over threadpool width at the same reduced
rows = 64geometry:rows = 64, reduced)That is
cpu_ops.cpp:130'sstatic thread_local std::vector<float> afand nothing else: sized by ggml's 16-rowblck_1, so16 x 188160 x 4 = 11.48 MiBper worker. The measurement lands on the model, and the output checksum is byte-identical across all six widths, which is the dispatch determinism contract holding.At the full shipped geometry,
rows = 1024, both projections, default 20 workers — the box was heavily loaded (loadavg 15.9 then 28.5), so the wall times are not a speed claim and only the memory column is:1024 x 188160 x 40961024 x 188160 x 204816x the rows moved the growth by 15 MiB, which is the output buffer (
1024 x 4096 x 4 = 16.8 MB) allocated inside the timed region. The tile does not scale with rows.So: not the threaded kernel (232 MiB at 20 workers). Not the call site — its
scaledcopy is 770 MB and is present on both arms, so it cannot be a regression. Not an interaction. Nothing needed capping, blocking or an arena, and the byte-exactness question the row was told to stop on never arose, because no summation order was touched.#1259 has the same profile and needs nothing
Ltx2FuseLoraIntoTensortakesvt::Matmul, the other member of the same seam.MatmulOneChunkis one template source for both orientations but two separate INSTANTIATIONS with two separate buffers (see the review section below) — so the per-worker buffer is16 x K x 4there too, withK = rank. At the shipped4096 x 450 x 4096, measured with the same allocation counter the new gate uses:0.52 MiB at 20 workers, exactly
19 x 28,800 + 64.bsandaggare unchanged by that row. It cannot hit the wall #1286 describes when a LoRA-bearing pipeline first runs on full weights, so nothing is fixed and nothing is filed for it.What the attribution exposed, and the one thing this lands
Nothing in this tree bounds the seam's memory. Every GEMM gate here asserts values or byte equality, so a kernel that allocated a whole intermediate per worker would have passed all of them, on every model, silently, until a box ran out. The reported defect would have been invisible to every gate in the repository had it been real. That is worth closing while the correct number is known rather than after the next report, so
tests/vt/test_ops_matmul_mem.cpplands with the refutation.Peak RSS could not carry the gate, and finding that out was the useful part. The first draft measured
VmHWMaround/proc/self/clear_refs. It readgrowth_bytes = 0for every thread count and every row count while passing every bound — the operands are freed between measurements, glibc keeps the arena, and the next tile is served from pages that are already resident. A mute switch reporting green over a kernel doing anything at all. The shipped gate counts what the seam asks for through replaced globaloperator newoverloads, a figure the allocator's RETENTION policy cannot silence, with a liveness case beside it requiring the counter to see at least six of eight fresh workers take a tile. What that counter does and does not see is stated exactly in the review section below — the first form of the claim was broader than the instrument.Green, at
K = 65536so one tile is exactly 4 MiB:(nthreads - 1)tiles, because the caller thread'sthread_localis already sized. Rows 64 vs 1024 at 4 workers: 12,582,976 both times, difference 0.Red first, by mutation, and the first attempt did not build
M1 widens
affrom the 16-row tile to the whole activation — #1286's own hypothesis, written into the kernel.BUILT=NO,compile_err=2—cpu_ops.cpp:134:19: error: unused variable 'nrows' [-Werror=unused-variable]. Recorded because a mutation that fails to build reads exactly like a passing test.BUILT=YES,compile_err=0,git diff --stat=src/vt/cpu/cpu_ops.cpp | 2 +-, 1 file changed, 1 insertion(+), 1 deletion(-).test_ops_matmul_memexit 1,Status: FAILURE!,2 test cases | 1 passed | 1 failed,13 assertions | 7 passed | 6 failed. Failing by name:CHECK(growth <= Bound(nthreads))at three worker counts —67,108,928 <= 25,165,824,201,326,656 <= 41,943,040,469,762,112 <= 75,497,472— and all three row-scaling assertions, withrows=1024reading 1,073,741,888 bytes, a gibibyte of exactly the shape the issue supposed.sha256(cpu_ops.cpp) = dc39eccdece48879e82be7209e95d182c6ed624eb04389de689fa4b58fe4f1f3before and after. Rebuilt (binary mtime moved07:15:48→07:16:27, so the green is not a stale binary) and green again: 2 cases, 13 assertions, 0 failed, exit 0.M1 mutates
src/vt/cpu/cpu_ops.cpp, product code reached fromModelRegistry::Forwardand from the LTX-2.5 text tower, which is what makes the red evidence that the gate measures shipped behaviour rather than a test-local copy. This row adds no product code, so there is no production call site to delete;reachability.mdanswers that case directly.The review of that bound, and the three regressions it did NOT catch
A fresh review of
39b34c0c8returned nine findings and noPASS. The refutation above was confirmed and is unchanged. What failed review was the gate that lands with it, which is the irony worth stating plainly: a bound written to catch a memory regression did not catch three obvious shapes of one.The bound admitted a 2x per-worker regression while its comment said it could not.
Bound(n)was8 MiB + n * 2 * kTileBytes— a per-thread term of literally two tiles, under a comment claiming it was "[not] wide enough to admit a second tile-sized buffer per worker, which is what keeps it a bound rather than a formality". Mutation M-B added exactly that second buffer and doubled the growth at every worker count, 13/13 green. At the shipped geometry that is 232 → 464 MiB shipping silently. The bound is now1 MiB + n * kTileBytes, which is the property the file actually claims, over a fixed term measured at 64 bytes. The comment now names the worker counts that detect a doubling (4 and up) instead of claiming all of them do.The counter was bypassed by an over-aligned
operator new. Only the non-aligned overloads were replaced, so the C++17std::align_val_tfamily fell through to the library. M-D delivered the whole-activation defect through a 64-byte-alignedoperator new— the natural shape for a SIMD scratch buffer in this kernel — and read byte-identical growth, 13/13 green. All eight alignednew/deleteoverloads are replaced now.std::mallocis seen too, through the LINKER rather than a symbol definition. M-E delivers the same defect throughstd::malloc, and the first pass of this repair left it green and narrowed the claim instead. That was the weaker resolution, and the argument against widening turned out not to hold. The objection was that anextern "C" void* malloc()in the test would be a second strong definition beside AddressSanitizer's own interceptor and would breaksanitize-cpu.-Wl,--wrap=mallocis not a definition: scoped to this one target, it redirects the CALLS made by the objects in this link —libvllm.a, and socpu_ops.cpp— and leaves every symbol where it was. M-E now reds at 1,086,324,800 bytes.freeis deliberately not wrapped: releasing a block gives no size without a header, and a header would mean applying an offset to pointers libc allocated before this file was reached, which is heap corruption rather than a failed assertion. So the C-allocator figure is bytes REQUESTED, cumulatively — it errs toward red, never toward a silent green — and it reads exactly zero on a clean run, so every per-thread figure is byte-identical to what it was without the wrapping.A new coverage case makes the whole claim executable: it allocates through plain
new, arraynew, over-alignednewat two alignments, andstd::malloc, and requires the counter to move by the amount asked for. Two failure modes, two mechanisms. Dropping a--wrapflag fails the link, because the file references__real_mallocdirectly (mutation M-W:BUILT=NO,undefined reference to '__real_malloc'), so the coverage cannot silently vanish from the build; the subtler "route present but folded away" mode is what the assertion catches.That case calls through a
volatilefunction pointer for a measured reason. A directstd::malloc(n)written in the test's own translation unit is never wrapped — the compiler treats it as a builtin and folds it before the linker sees a symbol reference. In a standalone probe, same-TU calls gavemalloc=0 calloc=0 posix_memalign=1; the identical calls from a separate translation unit gavemalloc=1 calloc=1 posix_memalign=1and the full 3 MiB. Product code is always the separate-TU case, which is why M-E reds — but a coverage probe written the obvious way would have asserted nothing.MatmulOneChunk<false>had no assertion behind it.<false>and<true>are separate instantiations with separatethread_localbuffers, and the gate called onlyvt::MatmulBTon a non-repacked weight.MatmulKernel(cpu_ops.cpp:292-294) routesvt::Matmulto<false>andMatmulBTKernel'selem_kn_repackedlever (cpu_ops.cpp:306-314) routes there too;vt::Matmulis #1259'sLtx2FuseLoraIntoTensorpath — merged and unexercised on the full model. Both arms are measured now. The twoafsymbols are separately confirmed withnm -Con the gate's own binary, at distinct addresses, and the measurement shows the same thing: 8 fresh workers read 33,554,496 bytes onMatmulBTand 33,554,496 again on thevt::Matmulimmediately after — eight tiles both times rather than eight then seven, which one shared buffer could not produce. So the retention figure is per instantiation: up to 464 MiB, not 232 MiB, for a process running both orientations atK = 188160on 20 workers.The chunk-row-span defect was invisible at every geometry the gate ran. M-A (
nrows = ir1_end - iir1) produced byte-identical growth and 13/13 green. ReplayingMatmulChunked's grid arithmetic (cpu_ops.cpp:230-264) over all twelve shapes the sweeps use givesdr1= 16 at eleven of them and 8 at the twelfth — never above 16, which is the entire reason M-A could not be seen, and a coincidence of the chosen shapes rather than a property of the kernel. The collapse to one chunk per thread (nchunk0 * nchunk1 < nth * 4) is live and is the shipped default, sinceVT_CPU_MATMUL_STEALis off. Cheap to close rather than to file, so a case now runs atrows = 128, n = 16, nth = 4— a live non-NUMA shape where the collapse givesdr1 = 32and a chunk-sized buffer costs two tiles per worker instead of one.One correction to that finding as it was written. The review called it "a real ~3x regression the gate cannot see on a NUMA host", citing
dr1 = ceil(rows/nth) = 52atrows = 1024, nth = 20. Two things are off.IsNuma()isconstexpr falsein this tree (cpu_threadpool.h:74; NUMA is recorded as unported atcpu_threadpool.h:25), so that branch is dead today and the gap is reachable through the live collapse instead. And at the shipped LTX-2.5 caption projection the weight is the longer axis (n = 4096 > rows = 1024), so the collapse takesnchunk1 = 1anddr1 = 1024, not 52 — a chunk-sizedafthere would be 770 MB per worker and 15.4 GB across 20, which is #1286's hypothesised shape almost exactly. The finding was right that the gate was blind; the magnitude is larger than it said and the trigger is nearer.The mutation table
Release, 20-core Zen 5 under KVM. Every row rebuilt and re-run;
sha256(cpu_ops.cpp) = dc39eccdece48879e82be7209e95d182c6ed624eb04389de689fa4b58fe4f1f3before and after every one of them.cpu_ops.cppBUILT/compile_errgit diff --statonsrc/39b34c0c8nrows= the chunk's row spanYES/ 0cpu_ops.cpp | 2 +-33,554,496 <= 17,825,792on both arms of the collapsed casenrows= the whole activationYES/ 0cpu_ops.cpp | 2 +-YES/ 0cpu_ops.cpp | 3 +25,165,888 <= 17,825,792at 4 workers,58,720,320 <= 34,603,008at 8operator newYES/ 0cpu_ops.cpp | 12 +1,019,215,936bytes atrows=1024, 4 workersstd::mallocYES/ 0cpu_ops.cpp | 12 +71,303,232 <= 9,437,184at 2 workers,1,086,324,800bytes atrows=1024, 4 workers--wrap=malloclink flag removedNO/ 1CMakeLists.txt | 1 -undefined reference to '__real_malloc'— the coverage cannot be dropped silentlyThe
sanitize-cpu (address,undefined)risk was exercised, and the record is in two parts because the second attempt was blocked. The FIRST widening — the alignedoperator newfamily, before--wrapexisted — ran in the CI lane's own configuration (-DVLLM_CPP_SANITIZE='address,undefined',-DVLLM_CPP_CUDA=OFF,UBSAN_OPTIONS=print_stacktrace=1): 5 cases, 36 assertions, 0 failed, exit 0, no ASan or UBSan diagnostic. Thethreadlane was green too, once ASLR was disabled to work around this box'sFATAL: ThreadSanitizer: unexpected memory mapping— a local kernel/TSan incompatibility, not a finding.Rebuilding that lane after
--wrapwas added hitNo space left on device— other sessions filled the shared disk to 100%, and the build's first output was fivefatal error: error writing to /tmp/...: No space left on devicelines, which is an infrastructure failure wearing the costume of a code verdict. So the full lane was NOT re-run with--wrapin place, and this says so rather than implying otherwise.What ran in its place is a standalone probe reproducing the gate's entire allocator mechanism in miniature — replaced plain and aligned
operator new/delete, the five--wrapredirections, and a separate translation unit standing in forcpu_ops.cpp— compiled and linked with-fsanitize=address,undefined. It builds clean and runs clean: all three C-allocator routes intercepted (3 MiB accounted), 64-byte alignment preserved through the replaced alignednew, replacedoperator newaccounting correct, no diagnostic, exit 0. That establishes the mechanism against ASan, which is precisely the collision the reviewer named. It does not stand in for a full-lane run and is not offered as one.The bound is deliberately tight now, and that is the point. At 8 workers the honest worst case is 8 tiles — every worker fresh, which the liveness case reaches — at 33,554,496 bytes against a bound of 34,603,008. The 1 MiB of headroom is 16,000x the measured incidental. Widening it again buys nothing except the 2x hole M-B just walked through.
Records repaired in the same change
A wrong line anchor, cited six times.
cpu_ops.cpp:125is cited forstatic thread_local std::vector<float> af— it is at line 130;:125isconst int64_t blck_1 = 16;, and the comment block cited as122-125is at127-129. Wrong at the named baseline, at the implementation commit and atHEAD. Fixed in the spec, the benchmark-record and the test header.It CANNOT be fixed in
.agents/issue-index.md:423, which cites it twice. That file is append-only and keyed, its checker reads commits rather than the working tree, and a corrective row would be a duplicate key its checker refuses. So the correction is recorded where a reader following the row arrives — the spec's## Owed, and this body. The same wrong anchor is frozen in the bodies of0377cde70and8d46223fdfor the same reason.The uncovered class behind it is recorded, not fixed here.
check-symbol-anchors.py:87-90requires a literal::between path and identifier, so it cannot match apath:lineanchor at all. Counted over every.mdin this tree: 613path::Symbolcitations and 14,448path:linecitations, the second checked by nothing. A 24:1 ratio deserves its own row and its own red-first mutation, not a follow-up line here.tests/CMakeLists.txtgave a false rationale for a live conditional. It called the test "the seam's PEAK RESIDENT bound" and justified theLinux-only guard with "because the measurement is /proc/self/{status,clear_refs}". The shipped test measures neither and opens nothing under/proc; it landed in the same commit as the shipped instrument, so it was wrong rather than historical. The real reason is stated now: the gate replaces the global allocation functions for the whole program, so it needs everydeletein the binary to reach the matching replaceddelete, and its fixed term calibrated against the library doing the allocating — both established for glibc/libstdc++ and nothing else.Three record corrections beside them. The spec's
## Tests to portand## Risks/decisionsdescribe the refutedVmHWMinstrument while## OutcomeW4 records that it was replaced, so both now carry a forward pointer.## OutcomeW2's first table was headed "at the shipped geometry" while runningrows = 64, breaking the spec's own rule at:158-162that a scaled probe is never presented as a full-geometry one; the reduction sits beside the number now. And the "threaded stretch sits atanon = 34.30" triple was that stretch's minima at a single sample — recomputed above from the retainedmemguard.tsv.runguard.py's line citations are unstable and were not chased. The file lives outside this repository on a mutable NAS path and was rewritten after the implementation commit, sorunguard.py:236-237,260no longer resolves reproducibly. The claim does not rest on it: in the retainedmemguard.tsvthe #1252 run's first sample readsavail = 88.078,used = 31.553— summing to 119.631 GiB, the box's wholeMemTotal— besiderss = 0.001for a child that had just started. A column reading 31.553 GiB before the process has allocated anything is system-wide by construction, whatever line computes it.Gates
test_ops_matmul_mem: 5 cases, 38 assertions, 0 failed, exit 0,Status: SUCCESS!— up from 2 cases and 13 assertions.test_ops_matmul_memunderVLLM_CPP_SANITIZE='address,undefined'(before--wrap; see above for why it could not be re-run after): 5 cases, 36 assertions, 0 failed, exit 0, no sanitizer diagnostic.test_ops_matmul_memunderVLLM_CPP_SANITIZE='thread'(ASLR disabled to work around this box'sFATAL: ThreadSanitizer: unexpected memory mapping): 5 cases, 36 assertions, 0 failed, exit 0.ctest -j 6after mergingorigin/mainedbc47ce0, on this exact head: 100% tests passed, 0 tests failed out of 554, 3 skipped by design, 995.03 s. The box was heavily contended by other sessions throughout (loadavg 180-333 on 20 cores, 179 competing processes), so no wall time in that run is a measurement of anything — only the pass/fail column is. Nothing was re-run to make it green.scripts/check-agent-record.py:agent record OK: ENGINE=165 MODEL=377 QUANT=84 KERNEL=52 BACKEND=85 ANCHOR-ROT=38.scripts/check-doc-checkpoint.py --base origin/main --head HEAD:OK: public documents match the claims this change makes.It failed first — a benchmark-record change owesdocs/BENCHMARKS.mdanddocs/STATUS.md, and the projection had to ride in the SAME commit as the record edit.scripts/check-public-doc-tables.py:OK. This one failed first too, and correctly: the projection as first written was a 496-char table cell against a 220 limit, and pusheddocs/STATUS.mdto 45 oversized cells against its 44 ratchet. Both were shortened into budget rather than the ratchet being raised.scripts/check-commit-trailers.pyandscripts/check-commit-style.pyoverorigin/main..HEAD: bothOK.scripts/check-issue-index-append-only.py --base origin/main:OK: issue index append-only. 413 rows, zero duplicate keys, #1252's threaded path raises peak host memory ~79 -> 105.85 GiB and aborts the full-model render on GB10 #1286 and The full-model render harness compares ABSOLUTE MemAvailable peaks across runs that started 26.8 GiB apart, and #1286's whole regression is that offset #1317 each present exactly once.BUILT,compile_err,git diff --statand the failing assertion by name for each.Byte-exactness is untouched —
src/vt/cpu/cpu_ops.cppis byte-identical to39b34c0c8, sha256dc39ecc...4f1f3, no tolerance was widened anywhere, and no summation order was altered. Every mutation above was applied to a scratch tree and restored.Records, and commit order
.agents/specs/ltx25-text-linear-mem.mdis committed before the implementation (0377cde70, then8d46223fd).docs/STATUS.mdanddocs/BENCHMARKS.mdcarry the keyed projection; the forensics are in.agents/benchmark-record.md. Two index rows appended, #1286 owned by this row and #1317 owned by this spec's## Owed.The review repair rides in the same pull request rather than a new one: it is the record edit and the gate repair the reviewed change made necessary, which
AGENTS.mdputs in the pull request whose change made them stale. It touches four files — the test, its registration, the spec's## OutcomeW5 and## Owed, and the benchmark-record entry — and no product code.Owed, named rather than left to be found
Ris untouched.afgrows to the largestKany GEMM on that worker ever saw and never shrinks — 232 MiB held until exit atK = 188160on 20 workers, and there are two such buffers, one in each ofMatmulOneChunk<false>and<true>, so a process running both orientations at thatKretains up to 464 MiB. Bounded, deliberate, and now measured; not repaired here, because the buffer exists precisely so a worker allocates once, and trading that away is a performance decision with its own spec.mmap/sbrk, nor an allocator reached inside a shared library's own internal calls.--wrapbinds at THIS link, so amallocthatlibstdc++.socalls internally is outside it; every globaloperator newis covered regardless. A kernel taking pages straight frommmapwould be invisible to this gate. Nothing in the seam does, and no mutation attempts it, so this is a stated boundary rather than a measured gap.path:linecitations are checked by nothing.check-symbol-anchors.py:87-90requires a literal::, so of 613path::Symboland 14,448path:linecitations in this tree's markdown, only the first population is covered. That is how the wrongcpu_ops.cpp:125here survived the implementation commit, the merge and the gate. Extending the checker needs its own spec and its own red-first mutation.cpu_ops.cpp:125anchor cannot be corrected in.agents/issue-index.md:423. The index is append-only and keyed, so the row cannot be edited and a corrective row would be a duplicate key its checker refuses. The correction is recorded in the spec's## Owedand in this body instead. The same wrong anchor is frozen in the bodies of0377cde70and8d46223fd.runguard.py:236-237,260is an unstable citation — the file is outside this repository on a mutable NAS path and was rewritten after the implementation commit. The claim it supported was verified independently against the retainedmemguard.tsvcolumns and does not depend on it.vt::MatmulBTon CUDA/ROCm/Vulkan is not covered.docs/BENCHMARKS.mdanddocs/STATUS.mdcarry the new keyed projection, whichcheck-doc-checkpoint.pyrequires of a benchmark-record change: the allocation is bounded on both instantiations, and the retention figure is 464 MiB rather than 232.FOLLOWING_AGENTS_PROTOCOL
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]