dasLLAMA/dasLLVM: generated-GEMM tune family + dispatch overhaul — per-box self-tuning kernels by borisbat · Pull Request #3385 · GaijinEntertainment/daScript

borisbat · 2026-07-06T00:15:51Z

Generated-GEMM tune family + dispatch overhaul: the whole kernel path for dasLLAMA is now a per-box self-tuning code generator, and the jobque dispatcher is work-proportional. Hand-written per-ISA kernels are deleted; four ISAs were validated on silicon with zero per-ISA kernel code written after the generator landed.

The tune framework (dasLLVM)

[tune] / [tune_perm] / [tune_companion] on a reference function turn one das body into a permutation grid of [llvm_code]-stamped variants (daslib/llvm_tune.das). Modes: normal (stamp one winner), test (lockstep all rows vs the reference), tune (bench the grid).
Per-ISA fallback chains: fallback="a;b;c" + requires="feat,feat|feat" eligibility hints — one grid serves every ISA; cpuid (OR DAS_JIT_X64_FORCE_FEATURES) picks the first emittable row. Manifests (DAS_TUNE_MANIFEST) bypass and pin exact rows.
e2e confirm pass: a divergent 1-core tune crown must beat the per-ISA fallback in a real-model prefill A/B before it ships to the manifest (harness/tune_confirm_prefill.das); rejected crowns pin the fallback explicitly. (First SPR outing rejected a 2x-losing AMX crown exactly as designed.)
JIT fastpath: SimFunction::jitFunction mirror lets das_invoke_* call JIT'd functions directly, skipping the interpreter prologue — empty team-op round-trip 1308 -> 631 ns/op on M1.

The GEMM generator (dasLLAMA)

dasllama_gemm_gen.das emits the whole Q8xQ8 kernel family from one schema: tile/batch/gemv/group3/groupN/mx4 cores across kstep/nrsplit/mr/gkstep/width permutation axes, with 9 companion generators (layout, repack, wbias, kgroup, tokstep, amx-cfg, gemv, mx4-gemv, witness).
x64 legs: maddubs (AVX2), vpdpbusd 256/512 (VNNI), vpdpbssd; bias128 repack (w^0x80 plane + acc-init −128·Σx) eliminates the sign trick — biased busd512 = +49% over plain on SPR metal, the SPR manifest winner.
arm64 legs: sdot (kstep/nrsplit/mr grid), smmla/kgroup (i8mm row-pair plane; silicon-adjudicated on M3 — sdot keeps that box, leg stays grid-resident for Neoverse).
AMX legs (slice I+K): 2x2 TMUL macro over the grp16 plane, ts=16·nrsplit token walk, plus the "fair shot" arms — software-pipelined fold (double-buffered C spill), raw immediate-tmm with a per-chunk LDTILECFG companion (undoes LLVM's serialized tile RA), TDPBSUD bias128, 2-D cell walk. Emission-proven via llc cross-compile gates (100/100) and silicon-validated on SPR; verdict: 0.36-0.55x the VNNI champion e2e on 4 models (the kstep=1 fold spills 4x the weight traffic — full decomposition in gemm_generator_plan.md "Slice K SILICON"), so the family stays grid-resident and the SPR manifest stays biased busd512.
The deletion: dasllama_math_x64_avx.das (2236 lines, the whole hand AVX matrix), 6 probes, and the arm64-laneq hand wrappers are gone (−4086 lines); portable floor + generated family is THE path. Registered A/B on silicon: generated 4.0-4.3x the best hand tiers on SPR, 1.65-1.7x on Zen2, and the per-ISA fallback IS each box's tune winner.

Dispatch overhaul (jobque + dasLLAMA)

Work-proportional dispatch: lanes_for_work(work, cap) sizes every matmul dispatch by measured per-box constants instead of a global parallel threshold; 25 sites migrated; decode_attn_par_threshold and g_matmul_par_threshold machinery deleted.
Per-box profile (box_profile.json): target_chunk_work / gemv_lane_cap / batch_lane_cap / team_rank_gate / batch_grid_2d knobs, minted by harness/team_probe.das (empty-dispatch ladder, per-chunk invoke slope, streaming GEMV lane ladder).
Worker-pool limit (set_jobque_worker_limit): dormant workers park outside the publish wake gate, woken only by a limit raise — with a dedicated limit condvar (fixes notify-theft and a notify_all storm found in the wake audit).
Per-op rank gate (set_jobque_team_rank_gate / box knob): every published team op carries its admit set; over-limit workers skip without touching claim words. 135M @T48: +48% SPR, +10-12% zen2, 1B neutral.
Team-prof counters (set/get_jobque_team_prof*): per-op publish/serve/join-tail attribution, exposed in prefill_perf via DASLLAMA_TEAM_PROF.
2-D batch chunk grid (batch_grid_2d): starved shapes (1-D grain-capped chunks < admitted lanes) go rows x token-blocks; wave-aligned strategy pinned on zen2 (+8-15% 135M pp@T48), off elsewhere.
Flash-decode: past 256 ctx, decode attention publishes (KV-head x ctx-slice) partials with a per-head combine stage; KV rows stream once for all GQA heads. +12% M1 / +11.6% zen2 / +57% SPR at ctx 2048; exact-recompute witness gates correctness (set_flash_decode_check).
Gemma pre/post-norm fused chains: deferred-residual tail lets Gemma2/3 ride the fused decode chains (5 -> 2 rendezvous/layer, +2-3.6%).

Validation

Suites: dasLLAMA 213/213, jobque, jit_tests — all -jit, on M1 + zen2 + SPR; oracle parity (simple_ids) 40/40 token-for-token on every box and stamp combination tested, including AMX + 2-D grid.
Emission gates: gen_x64_emission_check.sh (llc cross, 100 gates) + gen_arm64_emission_check.sh run without silicon; semantic grid (gen_tune_probe test mode) runs lockstep-vs-reference on each box (51 rows).
Fleet numbers on the boxes: zen2 small-model decode at parity with llama.cpp (24-cell mean ~1.01, was 0.24x on 135M pre-arc); SPR 75-cell fleet emit mean 0.87 vs llama.cpp-AMX with the 135M T-collapse cured (0.24 -> 0.70-0.92); gpt-oss pp 1.38 @t8 beats the AMX build outright; das-on-VNNI ~= lcpp-on-AMX pp within 8-15% on 3 of 4 arch-ladder models.
Docs: jobque externs grouped + described (das2rst clean, sphinx -W green both builders); design/evidence docs in modules/dasLLAMA/gemm_generator_plan.md, tune_for_this_box.md.

🤖 Generated with Claude Code

…e ([llvm_code] hook, generalized tune framework + manifest, GEMM as client) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…or registry, JIT dispatch, selftest [llvm_code(name="key", ...)] on a function asks the LLVM JIT to emit its IR via the das generator registered under "key"; the body stays the reference implementation (interp/AOT/ declined targets compile it verbatim). Generator modules wire in via llvm_user_modules.das (direct-call registration in the reading context — [init]/global-init don't cross macro-context copies). Annotation args fold into the JIT DLL cache key; generator-code changes still need an LLVM_JIT_CODEGEN_VERSION bump. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…onstant-arg calls no longer const-fold away before reaching the generator Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…orm-independent-reduce experiment M0: kernel_bench.cpp extended to lcpp's arm repack tier (gemm/gemv_q8_0_4x4 direct calls, hand repack mirroring repack_q8_0_to_q8_0_4_bl, quantize_mat activations, correctness-gated vs vec_dot; builds straight against libggml-cpu dylibs). gemm_1core_probe.das made platform-aware (enumerates registered backends, per-backend repack copies, gemv column, neon require). M1 Max ceiling table: our laneq batch tier BEATS lcpp gemm_4x4 on every slot shape (128-132 vs 118-122 GMAC/s, 1.06-1.11x); disasm on both sides: identical 32-sdot lane-indexed core, our edge = repack-time f32 scales + lane-fmla fold + u2 unroll. M2 bar set at ~130 GMAC/s. M1: reduce_experiment.ll — generic vector IR (sext-mul + partial.reduce.add, no target intrinsics) lowers to hand-shaped sdot loops on arm64 (LLVM 22.1.5); only the indexed sdot lane form needs the target intrinsic (~3-op per-ISA emitter); on x64 generic partial.reduce is unusable, per-ISA dot emitters stay mandatory. Conclusion recorded in gemm_generator_plan.md: dot is the only per-ISA knob, decline path is the fallback tier. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…eats the hand kernel dasllama_gemm_gen.das: the generator (llvm-jit context, wired via llvm_user_modules) — emits the tile as IR: byte-GEP loads, phi loop with kstep=2 paired blocks + odd tail, indexed sdot via aarch64.neon.sdot + lane shuffle (the M1 finding), f32-scale block fold, v4f32 stores; declines off-aarch64/no-dotprod to the reference body. dasllama_math_gen.das: the [llvm_code] stub (reference body = 4 laneq4 dots, same fold order) + arm64-laneq-gen backend (laneq with only the batch tile swapped; 3 laneq slot fns flipped public to be borrowed). gen_parity_probe.das: exactness gate. Gates: bit-exact maxdiff 0 (nb-even, nb-odd tail, production slice); generator firing proven via named blocks in --jit-dump; machine code 96 sdot / 0 dup (indexed fold held); iso gemm_1core_probe generated vs hand laneq batch: kv 130.4/131.2, qo 129.4/126.8, w13 129.9/128.8, w2 133.7/131.7 GMAC/s — matches-or-beats on all four, all above lcpp's 118-122. Suites: dasLLAMA 177/177, jit_tests 300/300, llvm_code 1/1. llvm_boost: *Aligned build wrappers now return non-const handles (internal let constified every caller's value chain); unused-arg lint fix. llvm_user_modules: the dasllama_gemm_gen require is branch-local wiring pending the require ?-gate (plan doc). gemm_1core_probe: platform-aware backend enumeration picked the new tier up unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…JSON manifest llvm_tune.das: [tune_perm(...)] rows + trailing [tune(gen=..., fallback=...)] stamp [llvm_code] perms onto a reference function. DAS_TUNE_MODE=normal stamps exactly one winner (manifest authoritative when present, else fallback, else reference body); tune/test stamp the full grid as <name>__<suffix> clones plus a <name>_variants() registry. Manifest APIs: tune_manifest_get (macro reads at stamp time) and tune_manifest_set (harness writes winners); DAS_TUNE_MANIFEST overrides the <das_root>/tune_manifest.json default. Rows route through a macro-context registry because a function's annotations only attach after every apply has run — [tune] cannot see its siblings on func.annotations, so ordering (rows first) is load-bearing. llvm_code_selftest.das grows the perm-parameterized worked example (add_plus_k: a+b+k from the annotation arg, decline=true rail); tests/jit_tests/llvm_tune.das pins the normal-mode contract in both lanes. CLI-gated (probes): grid rows k0/k1/k2 -> 5/6/7 with declined -> 5, set-API write -> normal-run consume, authoritative-absence. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

dasllama_gemm_gen: the M2 hand generator becomes dasllama_gemm_gen::q8q8_tile with knobs from the [llvm_code] annotation args — kstep 1|2|4 (remainder emitted as a conditional-block chain, so kstep=2 reproduces the M2 CFG exactly), nrsplit 4|2 (k-loop per token slice, fixed 4-token contract), mr, dot. Strides derive from q8q8_repack_type(mr) in the new dasllama_gemm_schema.das — one source read by the generator at emit time and by the runtime repack side, no desync possible. Decline rails in order: per-ISA dot emitter, perm legality, q-reg budget (2mr + nrsplit*mr/4 + 2nrsplit + 2 > 32), grp4 layout contract (mr!=4 waits for M4 repack generation). dasllama_math_gen: stub renamed q8q8_tile_gen, now a [tune] client — grid kstep1/2/4 + kstep2_nrsplit2 + three deliberate decline rows (mr8_budget, mr2_layout, dot_vpdpbusd), fallback kstep2 = the M2 perm. Gates (M1 Max): kstep2 disasm == the M2 baseline 291/291 instructions (96 sdot / 0 dup); DAS_TUNE_MODE=test — all 8 grid rows bit-exact vs the hand-laneq oracle, with per-variant disasm proving distinct generation (32/96/224 sdots for kstep1/2/4, decline rows instruction-identical to the reference body); tune run — kstep2 wins 133.9 GMAC/s @2048x512x64, written via tune_manifest_set, and the normal run stamps it from the manifest with parity green (harness/gen_tune_probe.das). Suites: jit 301/301, dasLLAMA 177/177; CI lint 8 files clean. M3 results section added to gemm_generator_plan.md. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…d from dasLLAMA A guarded require now also loads when the guard module is unregistered but the target's own file resolves (a mounted pure-das package — nothing C++ to guard on). Both walkers changed in lockstep: the textual collector (ast_parse.cpp) and the parser statement (parser_impl.cpp). typeinfo builtin_module_exists additionally sees promoted (shared das) modules, so the static_if pairing works for das-module contributors. The original "app requires the implementation first" idea cannot work on the -jit rail: the LLVM chain is injected as an extra module and is collected AND compiled before the app root's requires are even scanned — the opt-in unit is the mount, not the app. llvm_user_modules now does `require ?dasllama_gemm_gen dasllama/dasllama_gemm_register` (new registration shim) + static_if around the call — dasLLVM builds without dasLLAMA skip both cleanly, and non-dasllama apps on a mounted checkout get name-keyed inert registration. The math_gen side-effect require is gone. Tests: tests/language/optional_require.das + fixture pin the resolvability fallback, guard-absent silence, and das-module builtin_module_exists; tests/jit_tests/llvm_tune_modes.das spawns a child daslang with DAS_TUNE_MODE=test and asserts every grid row's runtime value (5/5/6/7/5 — per-perm generation + decline fallback); cant_tune_bad_grids.das pins all six [tune] validation errors (expect 20800:6). Suites: language 1085/1085, jit_tests both lanes, dasLLAMA 177/177; GEN PARITY OK with fresh codegen on both the mounted and non-dasllama paths. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

[tune_manifest(name = "myapp", dir = "...")] on the app's main opts into a manifest at <dir>/<name>.tune.json, resolved against the declaring file's directory (the get_this_module_dir pattern; distinct names disambiguate apps sharing a folder). The shared <das_root> default dies — no declaration and no DAS_TUNE_MANIFEST env means no manifest (fallback perms), so unrelated apps can never stomp each other's winners. The root module compiles after every library [tune], so the declaration cannot feed their applies; instead its own apply re-stamps every [tune]-annotated function in the program from the declared manifest (grid + gen key re-read from the by-then-attached annotations, fallback stamp dropped, manifest authoritative while present) and injects an [init] pointing tune_manifest_set at the same file. Env override wins outright; grid modes skip the re-stamp. Known edges documented in the plan: constant-arg call sites compiled before a late re-stamp can stay on the (bit-exact) reference tier, and shared library modules keep the first program's stamps in multi-program processes. tests/jit_tests/llvm_tune_manifest.das: declaration-driven round-trip — the client template is copied to a temp dir (manifest lands there by construction), one spawn writes k2 via the injected-init path, the next spawn re-stamps from the manifest and reports the k2 tier (RESULT 7 + the consumption log line). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

[tune_companion(fn=, gen=)] rows in a [tune] bracket stamp sibling functions with the SAME perm from the same manifest entry, in all three modes (normal winner, tune/test grid clones + <sibling>_variants() registries, [tune_manifest] re-stamp included). One manifest entry = one perm decision across N functions, so a kernel and its repack-layout query cannot desync — and when both generators share a decline predicate, JIT-time declines stay coherent too (both fall back to reference together). Selftest grows k_value (emits `return k` for a `() : int` companion, lockstep decline rail); llvm_tune/llvm_tune_modes/llvm_tune_manifest assert the companion follows the stamp on each rail; cant_tune_bad_grids pins the 6 new validation branches (12 total). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…guard Two coherence fixes surfaced by the M4 grp<mr> work: apply_fast_math_to_module now fills ONLY instructions with unset flags — an [llvm_code] generator that authors explicit FMF keeps them. The GEMM fold authors contract-without-reassoc (0x7e): its fold order is the bit-exactness contract, and the blanket 0x7f was licensing the AArch64 machine combiner to regroup (af*s)*q into af*(s*q) in some loop shapes (mr8 x kstep1/kstep4: last-ulp drift vs the oracle). das-frontend code is unaffected (nothing else authors flags). [tune] now forces sideEffectFlags.userScenario on every companion at apply time, not just at stamp time: an unstamped pure zero-arg companion (no fallback, no env manifest) could const-fold at its module's compile, leaving stale reference values at call sites when a root [tune_manifest] re-stamps the family later — a layout/kernel desync, the exact thing the two-function stamp exists to prevent. LLVM_JIT_CODEGEN_VERSION 0x37 -> 0x38 (generator emission changed for fixed args — the mr generalization lands in dasLLAMA next). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

The tile generator is generalized over row quads (rq = mr/4: 2*mr weight vectors per block at kg*4*mr + qd*16, token x quad accumulators, per-quad scale folds and stores) — mr=4 emission is instruction- identical to M2 (289/289 disasm gate). The layout companion (q8q8_layout_gen, generator dasllama_gemm_gen::q8q8_layout, stamped via [tune_companion] from the same manifest entry) reports the stamped tile's repack interleave; both generators share ONE decline predicate, so kernel and repack fall back to the grp4 reference pair together on every rail. The runtime asks the companion everywhere layout matters: repack_q8q8_grp(mr) (one executor, byte-identical to repack_q8q8_neon at 4), batch traversal strides, mm/group3 slot choice at [init] (generic scalar grp<mr> forms off the grp4 layout — the GEMV perm sub-family stays future work). GEMM_GEN_MR -> GEMM_REFERENCE_MR. Grid: kstep 1|2|4 x nrsplit 4|2 at mr4, kstep 1|2|4 at nrsplit2+mr8, three decline rows (mr8_budget 34 q-regs, mr2_lanes mr%4, dot_vpdpbusd). Probes are per-variant: the layout variants registry gives each tile variant its mr, fresh row-major copies are repacked per variant, and the oracle is a grp4-copy laneq4 pass — layout-independent, and never the generic scalar dot (full fast-math may reassociate ITS fold; the first draft tripped exactly that). Gates: test mode 13/13 bit-exact (mr8 rows on their own grp8 repack, decline rows lockstep); parity exact under both the kstep2 fallback and a manifest mr8 stamp (the production two-function round-trip); kstep2_nrsplit2_mr8 = 192 sdot / 0 dup; suites jit 305/305, dasLLAMA 177/177. The sweep (kv iso 2048x512x64, interleaved best-of-6): kstep2_nrsplit2_ mr8 WINS at 135.0 GMAC/s over the hand-tier-equivalent kstep2 at 132.4 — an 8-row x 2-slice tile the hand tier never had. Manifest round-trip green end to end. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…ins at 137.5 Boris: llama.cpp holds no cross-kernel bit-parity, and _jit_fast_math is already declared non-bit-exact program-wide — bit-exactness across variants is not the standard. Measured cost of the reassoc pin: -2.5% on the kstep4-mr8 shape (the machine combiner's (af*s)*q regroup is a critical-path optimization). So the generated fold keeps blanket fast-math (no authored-FMF pinning; the dasLLVM authored-flags-wins semantics and the companion const-fold guard stay — correct framework behavior regardless), and the probes gate on a tolerance vs the grp4-laneq oracle (CLOSE_REL 1e-4 / CLOSE_ABS 1e-5; legal drift ~1e-6 relative, real generator bugs are orders louder; 11/13 grid rows still land bit-exact). Re-sweep (kv iso 2048x512x64, interleaved best-of-6): kstep4_nrsplit2_mr8 WINS at 137.5 GMAC/s vs 133.2 for the hand- equivalent kstep2 (+3.3%) — the reassociated 8-row x 4-block tile overtakes slice B's pinned winner (kstep2_nrsplit2_mr8, 135.8). Manifest round-trip green; the kstep2-mr4 fallback machine code stays 289/289 identical to the M2 baseline; suites jit + dasLLAMA green. LLVM_JIT_CODEGEN_VERSION 0x38 -> 0x39. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…est cliff A manifest with no entry for a function now falls through to the annotation's fallback= perm instead of forcing the reference body, on both the env path and the [tune_manifest] re-stamp (which leaves no-entry functions' [tune]-time stamps untouched instead of stripping them). Rationale (Boris): a manifest written before a new kernel family landed must not silently drop that family to the reference tier (~85 vs ~133 GMAC/s on the GEMM tile); an explicit "reference" entry still forces the original body, so nothing expressible is lost. The stamp log reports the true provenance (manifest path vs "fallback"). Pinned by llvm_tune_manifest's new OTHER (no entry -> fallback tier survives the re-stamp) and REF (explicit "reference" -> original body despite fallback) assertions; jit suite green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…lice B) Hand-written kernels go away apart from the default tiers; the generated [tune] family is the path, worked until strong on all supported architectures. Ordered breakdown: promote the gen backend on arm64 → GEMV/group/mx4/tails coverage → x64 dot emitters → delete laneq + the AVX matrix + the [tuned]/box_profile rail. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

A guard containing '/' is resolved as a FILE (require ?llvm/daslib/llvm_tune <target>): the require proceeds only when that file resolves, with NO target-resolvability fallback. This is the shape the plain-name guard cannot express — a target living in an always-present package while depending on an optional one (dasllama_math_gen is in dasLLAMA's own mount, so its file always resolves, but it hard-requires dasLLVM's llvm_tune). Plain-name guards keep the existing semantics (registered C++ module, else target-file fallback). Both walkers updated in lockstep (ast_parse.cpp collector + parser_impl.cpp); grammar reuses require_module_name for the guard. Pinned by two new optional_require rows: path guard present -> loaded; path guard absent -> skipped even though the target file resolves. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

The generated GEMM family stops being probe-only. dasllama_common pulls dasllama_math_gen in behind the new PATH guard (require ?llvm/daslib/llvm_tune — loads exactly when dasLLVM is mounted, skips cleanly without it), and the backend registers at priority 25 with FULL slots, so select_matmul_backend_for_load now picks it over arm64-laneq (20) on arm64+jit: every Q8 model load runs the generated batch tile (kstep2 fallback = M2-perm machine code, 289/289 identical to the hand tile; laneq stays registered for by-name pins/A-B). Slot coverage by stamped layout: mr=4 borrows the full laneq set (mm/group3/ groupn/rows/mx4 interleaved family — the byte-identical grp4 layout; six neon kernels flip public for the cross-module borrow). mr!=4 gets the generic grp<mr> Q8 forms (new q8q8_groupn_grp_generic: per-region generic GEMV + bias post-add), a row-major mx4 family (interleaved planes + a grp<mr> Q8 scratch can't share the laneq expand map -> identity repack + sdot-tier kernels + row-major expand repacked through the backend's own Q8 repack), and the per-op decode path (no rows core). emission_bench grows the DASLLAMA_PIN_BACKEND rail its siblings already had plus a backend log line — the slice C fleet-sweep evidence. Gates: language 1087/1087 (+2 path-guard rows), jit 305/305, dasLLAMA 177/177, gen_parity_probe exact, DAS_TUNE_MODE=test grid 13/13 (mr8 rows within tolerance, declines lockstep), model smoke: default stamps kstep2 + selects arm64-laneq-gen; DAS_TUNE_MANIFEST mr8 stamps kstep4_nrsplit2_mr8 and runs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…JIT stamps The slice C promotion branched slot choice on q8q8_layout_gen() inside the registration [init] — and [init]s run before the JIT installs generated code, so the layout companion answered with its reference grp4 there while the load-time repack (a runtime call) used the stamped mr8: laneq grp4 readers on grp8 weights. Every kernel-level gate passed (they sit below the slot wiring); the mr8 model run generated fluent-looking garbage that even benched fast. Caught by a parity.das GEN_IDS token print; the general lesson is the dastest -jit reference-tier wrinkle writ large: NOTHING may consume a JIT-stamped value at [init]-time. Fix: KernelBackend grows `available : function<():bool>`, evaluated by select_matmul_backend_for_load and pin_kernel_backend at runtime (the registration-time auto-activate deliberately does not evaluate it), and dasllama_math_gen registers TWO statically-correct flavors — arm64-laneq-gen (grp4: laneq slots + the generated tile, priority 25) and arm64-grp-gen (any other stamped mr: generic grp<mr> slots + row-major mx4 family, priority 24) — whose predicates read the stamped layout when it is truthful. Pinned three ways: NEW harness/gen_slot_parity_probe.das (portable reference vs the gen backend through the real rail — pin, backend repack, mm/batch/ group3 wrappers; green at BOTH layouts, and its pre-fix run isolated the bug: batch exact, mm/group3 wrong = grp4-readers-on-grp8), parity.das mr8 GEN_IDS token-for-token identical to the default stream, and a portable unit test (test_backend_availability_predicate: priority-99 unavailable backend skipped by for-load select, refused by pin). dasLLAMA suite 179/179. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Fleet re-sweep table (M1, 17 models): B/A pp median 1.01 / emit 1.00 — the gen backend supersedes arm64-laneq at zero cost. Honest mr8 column (post-fix): the batch-tile prefill win is real in the 1.5B-4B dense band (+12-25%), decode on >=1.5B models rides the auto-vectorized generic GEMV at 0.88-1.05 of hand laneq, and the two loser shapes (vocab-heavy cls GEMV, gpt-oss row-major mx4) are exactly the GEMV/mx4 coverage items that gate stamping mr8 by default. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…rated family The nr=1 rows-range core (mm_rows slot signature, group walk inside the generated function) joins the [tune] bracket as a second companion — the three-function stamp: one manifest entry decides layout + batch tile + GEMV, with the one shared decline predicate keeping all three in lockstep. New gkstep knob (GEMV blocks per K-iteration = independent sdot chains); rows carrying it spell tile knobs explicitly so the tune probe can key rows back to their tile family. mm/group3/groupn become das traversals over the core, mm_rows is the core itself, and the grp<mr> generic stopgaps are deleted — the first hand-kernel deletion of the mandate; arm64-grp-gen gains a rows core (fused decode chains now run at mr != 4). Flavors differ only in the mx4 family. The sweep grew stream + hot decode shapes: streamed GEMV is DRAM-bound and cannot tell kernels apart; on the hot shape mr8 GEMV beats the mr4 hand walk ~+8% (shared activation loads) and gkstep2 adds ~2%. Tune mode writes the merged winner (batch bench picks tile knobs + layout, hot decode picks gkstep): kstep4_nrsplit2_mr8_gkstep2 on this box. Gates: grid test 17/17 (tile AND gemv, both layouts), gen_parity_probe + gen_slot_parity_probe green at both stamps, parity.das token-for-token default-vs-mr8 on Llama-3.2-1B and Qwen2.5-0.5B, dasLLAMA suite 179/179. Fleet re-check: the slice C mr8 losers (SmolLM2-135M 0.64, Qwen2.5-0.5B 0.54 emit / 0.71 pp, Qwen3-0.6B 0.59) are all 1.00 now — only the mx4 family (gpt-oss) remains before mr8 is the unconditional daily driver. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

… one arm64-gen MXFP4 joins the stamp as the fourth companion: mx4q8_gemv_gen is the Q8 rows core with the weight-load stage swapped for tbl LUT-dequant (constant doubled-e2m1 vector, one nibble load per quad per dword-group feeding tbl-lo/hi + the same indexed-sdot lattice) and the scale loads for an in-register vectorized e8m0_half. One layout companion drives BOTH repacks (repack_mx4_grp = the mr-generalized plane interleave); the mx4 expand's byte-for-byte positional map generalizes to any mr off a new activation-time layout cache (KernelBackend.q8_layout), so the expand-row-major-then-repack double pass is gone on arm64. perm_declines grew the tbl rail — four generators, one lockstep predicate. With every slot now layout-driven, arm64-laneq-gen and arm64-grp-gen differed in nothing: merged into arm64-gen, the laneq mx4 borrow and the row-major mx4 stopgap wiring deleted. Gates: grid test 17/17 x three kernels (mx4 bit-exact vs the hand interleaved walk at mr4), slot parity + new mm_mx4/groupn_mx4 legs green at both stamps, gpt-oss tokens identical default-vs-mr8, suite 179/179. gpt-oss fleet: mr8 pp 0.49 -> 1.06, emit 0.94 -> 1.08 — no model loses at mr8 anymore; the mr8 manifest is a legitimate daily driver on this box. Also repro'd (not fixed): ternary string into an extern's `string implicit` param SIGSEGVs the compiler; hoisted the one hit site. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…n-proven) The generated GEMM family grows its x64 legs inside the SAME generator and grid: dot=maddubs (VPMADDUBSW+VPMADDWD sign-trick pair, the Zen2 leg), dot=vpdpbusd (VNNI, sign trick, one dot op; ymm+zmm), dot=vpdpbssd (AVX-VNNI-INT8, native s8*s8, new avxvnniint8 cpuid tier), width=256|512 — TileEmit generalized over rows-per-vector, ONE dot_lane dispatch, mx4 tbl->pshufb, all four companions declining in lockstep through the one ISA-aware predicate. Emission-only rail (no x64 silicon needed): --jit-compile-only (build + verify + optimize + dump, write/load/install nothing) + cross x64 triples read DAS_JIT_X64_FORCE_FEATURES as the ONLY tier truth; jit_dll_basename folds the target triple (latent cross-DLL cache-poison fix). Gates: arm64 machine code byte-identical pre/post at BOTH stamps (full-DLL neutralized stream diff -> no LLVM_JIT_CODEGEN_VERSION bump); grid 29/29 on arm64 (12 x64 rows decline to reference exactly); gen_x64_emission_check.sh 21/21 instruction-count gates on the first run (96 vpdpbusd / 96 vpsignb / 24 vpabsb tile, exact maddubs pairs, bssd zero sign ops, zmm legs, 8 vpshufb mx4, single-vpbroadcastd splats, no scalarization); slot parity both stamps; dasLLAMA 179/179; jit_tests 305/305 + new llvm_compile_only test. Semantics/perf on silicon + witness companion/x64-gen registration = the queued Zen2/EPYC session (plan doc slice F). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…ice-F test/tune mode drive the variant registries directly, so no gen backend is needed: on x64 the grid rows light per cpuid with nothing registered (x64-gen is queued behind the witness companion). Keep selecting arm64-gen where it exists. First silicon result (Zen2 3990X, native Windows/MSVC build): the two maddubs-256 rows run live at mr=8 and are BIT-EXACT vs the scalar oracle — tile, Q8 gemv, and pshufb mx4 gemv; every VNNI/512/vnniint8 row declines per cpuid. Slice F semantics gate closed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

The dispatch sites' chunk-count args read get_dispatch_lanes() eagerly even when maybe_parallel_for stays inline; production always has a que, the probe had none, and the first que-less box (Zen2 Windows) threw. Nothing dispatches at the infinite threshold, so the measurement stays 1-core. Zen2 3990X day-1 scoreboard (1-core, this probe + gen_tune_probe tune mode): hand tiers batch 18.6 (portable) / 22 (x64-avx2) / 25.3 (repack) / 24.4-29.5 (acc8) GMAC/s; the generated dot_maddubs_width256_mr8_kstep2 tile = 47.6 GMAC/s on kv — ~1.6x the best hand tier, manifest round-trip green (zen2_manifest.json). Registered-backend apples-to-apples lands with the witness companion + x64-gen registration. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…registered backend q8q8_family_live() : bool joins the [tune] bracket as the fifth companion: reference body false, generated body 'return true', declining through the ONE shared predicate — witness true <=> every slot is generated code. Read at backend-SELECTION time only (the slice C rule). x64-gen registers over the same ISA-agnostic slot traversals as arm64-gen (priority 25, needs_repack, available = witness, avx2 minimum): on x64 a declined stamp's reference bodies are the NEON scalar fallbacks, so an un-emitted family must never load-select; arm64-gen keeps no predicate (declined = the fast grp4 pair). gen_tune_probe asserts the lockstep invariants (mr != reference => live; reference row => not live); gen_slot_parity_probe picks the gen backend by arch and SKIPs loudly when the witness refuses the pin. Zen2 3990X (native Windows/MSVC) results, all through the registered rail: slot parity ALL slots maxdiff 0 at the maddubs-mr8 stamp; registered A/B (gemm_1core_probe): x64-gen batch 52.6-53.2 GMAC/s vs acc8 30.9-32.1 / repack ~26 = 1.65-1.7x the best hand tier on every shape, gemv 1.3-1.5x; parity.das Llama-1B GEN_IDS token-for-token identical hand-rail vs x64-gen; emission_bench logs 'backend: x64-gen', ~994 t/s pp512 / 36.1 t/s emit; dasLLAMA suite 179/179 on the box AND on arm64 (witness column: sdot rows live, x64 rows ref; slot parity green at both stamps). The slot probe also caught its own fixture bug: activations were raw random int8 including -128, violating the documented sign-trick precondition (PSIGNB of -128 wraps; the Q8 quantizer guarantees [-127,127] by construction — verified in dasllama_quant). arm64 never noticed (sdot has no precondition). Activations now use the quantizer's range; weights stay full-range (w = -128 must stay covered). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…l on Zen2 x64-gen (maddubs-mr8) vs llama.cpp CPU (fdb1db877), 4 small models, T=16/32: prefill WINS 3 of 4 (Qwen3-0.6B 1.16-1.36x, gemma-3-1b 1.13-1.38x, Llama-1B 1.41-1.94x; SmolLM-135M 0.89-0.95); decode 0.47-0.93, losses shrinking with model size — the known tiny-model dispatch/par-threshold issue, not kernels. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

c7i.4xlarge scout (Xeon 8488C): ALL ELEVEN x64 grid rows live and bit-exact (maxdiff 0) — first zmm execution; tune sweep answers the Genoa question: native-512 vpdpbusd zmm WINS the tile 88.3 vs 71.5 GMAC/s ymm (+23%), winner dot_vpdpbusd_width512_mr16_kstep2_gkstep2 -> spr_manifest.json. GEMV is width-insensitive (bandwidth-bound). bssd correctly declines (no avx_vnni_int8). Scout terminated after ~1 hour — the no-parking discipline. Ledger grows the tighten-emission agenda (Boris: 'the biggest thing', the follow-up sessions): bias128, the kstep4@512 collapse (88->63, fresh scout data), broadcast amortization via an 8-byte kgroup, mx4 biased-LUT fusion, per-slot perms, and the new-silicon dot families (smmla/SME/AMX/bssd). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…dict kstep4@512 collapse CLOSED from the SPR scout dump: no spills, no scheduling — nrsplit2 halves weight-vector token reuse, so the busd sign-trick preamble doubles per dot (32/32/32 loads/abs/masks vs kstep2's 16/16/16 for the same 64 dots, weight plane streamed twice). The budget rail is correct; the fix is deleting the preamble. bias128 (ledger item 1): the repack bakes w^0x80 into grp<mr> group rows (tails stay plain s8), dots become plain vpdpbusd(acc, w_biased, x), and the exactness correction -128*sum(x) per (token, block) is the ACCUMULATOR INIT — zero ALU (replaces the free vpxor with one embedded-broadcast load). Machinery: sixth companion q8q8_wbias_gen (same shared decline predicate), Q8RepackType.wbias, tile stub grows a 10th param xbsp (bsums plane, built by the batch wrapper once per call, O(ntok*n)); the gemv computes the block bsum inline (mm_rows is a shared slot typedef — concat chunks, vpdpbusd(0, splat128u, x), vector.reduce.add, negate); the generic token tail un-biases via the int8 +128 involution; KernelBackend.q8_wbias + active_q8_wbias() feed the mx4->Q8 expand a biased LUT copy. +6 grid rows incl. a bias-on-maddubs decline pin (word-saturating pair sums) and a biased kstep4_nrsplit2@512 diagnostic twin. Gates: emission 29/29 first run (biased tiles 96 vpdpbusd / zero sign-trick ops; biased gemv gk2 = 24+3 bsum dots); biased busd512 hot loop 138 insns per 64-dot iteration vs 289 unbiased (2.09x) — bonus: activation splats fold INTO the dots as EVEX embedded broadcasts ({1to16}) once the sign chain stops separating them; per-iteration SIMD-ALU 192 -> 96. das rail bit-exact interpreted (plane ^0x80, tails/scales untouched, tail un-bias exact). arm64: grid 35/35 (new rows decline, wbias lockstep asserted), gen parity + slot parity maxdiff 0, suite 179/179. Silicon semantics + perf = next SPR/EPYC session (Zen2 declines biased rows — no VNNI). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…rid max mr q8q8_token_grp_generic's f scratch was float[8]; the first-ever mr16 production stamp (SPR biased busd512 winner) crashed the batch token tail at f[8]. Cap is now 32 = the grid's max stampable mr (busd512 mr32 row). Proven on SPR metal: slot parity 7/7 at the biased mr16 stamp, GEN_IDS identical default-vs-stamp on Llama-1B + gpt-oss. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…r16 stamp fix, 4x hand tiers) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Copilot

Pull request overview

Copilot reviewed 114 out of 116 changed files in this pull request and generated 2 comments.

…ability fallback The fallback (0ef74f9) collided with master's #3382: 'require ?sqlite sqlite/sqlite_provider' means 'loaded only when sqlite is LINKED', but module source dirs exist in every checkout, so the fallback pulled the provider into module-less CI lanes and its hard 'require sqlite' failed the darwin/wasm aotlib step. Split cleanly: plain-name guard = registered module only; path guard = the guard's own file resolves (the rail pure-das packages should use). llvm_user_modules switches its dasllama wiring to a path guard (?dasllama/dasllama_gemm_gen) — same availability, honest spelling. Both walkers (parser + textual collector) in lockstep; optional_require.das re-pins the strict case. Also per Copilot: DASLLAMA_WORKER_LIMIT env passes 0 through (caller-only is A/B-able; negative = unlimited), gemm_register comment matches the new wiring. Suites: optional_require 4/4, jit_tests 312/312, kernel_backend, dasSQLITE + sql_conformance — all green on a full local build. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Copilot

Pull request overview

Copilot reviewed 114 out of 116 changed files in this pull request and generated 2 comments.

…ard semantics (Copilot) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Copilot

Pull request overview

Copilot reviewed 114 out of 116 changed files in this pull request and generated 1 comment.

… (Copilot) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Copilot

Pull request overview

Copilot reviewed 114 out of 116 changed files in this pull request and generated no new comments.

…harness-dependent The wasm suite runs from a root where %/ doesn't resolve, so the pctguard require self-skips there; unconditional assertions broke that lane. The scanner-parity regression stays loud regardless (a mangled guard+target fails 20605 at require time on native lanes, before static_if). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Copilot

Pull request overview

Copilot reviewed 114 out of 116 changed files in this pull request and generated 3 comments.

…lution, arm64 script note maybe_parallel_for spliced num_jobs twice on the taken path (profile + dispatch) — and callers pass shaper CALLS, so every dispatch ran its chunk shaper twice. Fixed via count_parallel_dispatch_thru pass-through (a let-bind trips the no-shadowing rule across expansions). ast_requireModule now resolves the target only after the guard admits the require (matches the collector; skipped requires probe nothing). gen_arm64_emission_check documents why it shares the arch-neutral x64-named probe. dasLLAMA suite 213/213 -jit + arm64 emission gate green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Copilot

Pull request overview

Copilot reviewed 114 out of 116 changed files in this pull request and generated no new comments.

…h is frame-less like AOT The invoke fastpath (this PR) dispatches via jitFunction with no interpreter frames, so the deep-recursion probe SUCCEEDS under -jit — the old else-branch pinned interpreter overflow for every non-AOT tier and failed CI's darwin JIT step. Expectations now key on use_aot || has_jit_fastpath(fn) (the witness extern), overflow only on the true interpreter path. Verified all three tiers. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Copilot

Pull request overview

Copilot reviewed 115 out of 117 changed files in this pull request and generated no new comments.

…nifest Both tests (new in #3385) fail deterministically on the windows Release lane - on master's own merge-commit run as well as on this PR - and reproduce locally. Their popen command lines start with a quoted exe path; cmd.exe /c strips the FIRST and LAST quote of such a line when more than one quoted region is present, mangling the child command (child exits 1, no marker lines). llvm_tune_modes.das dodges it only because its line starts with `set VAR=...&&`. Fix per the established pattern (daspkg popen incident): wrap the full command in a sacrificial outer quote pair on windows only. Both tests now pass locally under -jit; tests/jit_tests sweep 309/309. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

borisbat and others added 30 commits July 5, 2026 17:14

dasLLAMA: rewrite gemm_generator_plan.md — settled 3-part architectur…

f2e4430

…e ([llvm_code] hook, generalized tune framework + manifest, GEMM as client) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

dasLLVM: [llvm_code] forces sideeffects on the annotated function — c…

eeab0b4

…onstant-arg calls no longer const-fold away before reaching the generator Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

dasLLAMA: plan — slice H SPR-silicon results (bias128 157.9 GMAC/s, m…

0339b68

…r16 stamp fix, 4x hand tiers) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

borisbat requested a review from Copilot July 6, 2026 00:57

Copilot started reviewing on behalf of borisbat July 6, 2026 00:57 View session