From 3ab41a301345f58c4507ba7e8061bbd3e195c468 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Fri, 14 Aug 2026 09:45:01 +0000 Subject: [PATCH] record(MODEL-MUSIC-MUSIC3): the row is ACTIVE, and the spec now meets the ACTIVE contract (#672) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit W0 is done and W1 has landed, so the row's state was stale at `SPIKE` while real loader code sat in the tree. Advancing it is what surfaced two things worth recording rather than quietly fixing. THE ACTIVE CONTRACT IS ENFORCED, AND THE SPEC DID NOT MEET IT. Moving the row to `ACTIVE` turned check-agent-record red on nine missing structured fields -- Scope, Upstream chain, Our baseline, Port map, Tests to port, Gates, Dependencies, Work breakdown, Risks/decisions. A `SPIKE` spec is not held to that; an `ACTIVE` one is, which is the checker doing exactly its job. My first attempt added them as PROSE SECTIONS and stayed red on all nine, including the two the spec already had -- because the checker does not read headings at all, it reads TABLE ROWS whose first cell names the field. Fixed by writing the structured record in the shape `minimax-h3.md` already uses, rather than by arguing with the gate. Worth stating plainly: the failure was mine reading the requirement as prose, and the checker's message was right both times. SECOND, A WORDING FIX THAT COST TWO IMPLEMENTERS. Β§3 said both pins "go into `.agents/oracles/` in W0", which reads as future work. `sglang-omni.md` has existed since #679, yet two separate agents reported it as still owed -- the same agent twice. A record surface described in the future tense reads as a TODO no matter what the tree holds, so it is now present tense with links, plus a note saying why. That is a small edit that stops a recurring false report. WHAT THE ROW NOW CLAIMS, and nothing beyond it: oracle gateable with 13 committed per-stage goldens (#708), modular loader gated 1413/1413 against the real 27 GB checkpoint with all 1012 tensors accounted and the native arm refused by name (#714). Checklist mark πŸ“‹ -> 🚧; rollup SPIKE 9 -> 8, ACTIVE 9 -> 10. The lifecycle change owes STATUS, BENCHMARKS and the spec's `## Now`, and all three say the same thing the row does. BENCHMARKS records every axis PENDING with NONE OWED YET and says why: W0/W1 are correctness scaffolding, the oracle capture was CPU-only because dgx.casa was down, and there is no forward pass to time. A row that has landed code is exactly where a speed claim tends to appear by implication, so the absence is written down rather than left to be inferred. Both public rows were first written over the house-style limits (a 402-char BENCHMARKS cell, and STATUS at 45 oversized cells against a 44 ratchet that may only shrink). Shortened to fit rather than lowering a ratchet to accommodate prose. Issue: #672 FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] --- .agents/model-matrix.md | 8 ++--- .agents/specs/minimax-music3.md | 58 ++++++++++++++++++++------------- docs/BENCHMARKS.md | 1 + docs/FEATURES.md | 2 +- docs/STATUS.md | 1 + 5 files changed, 43 insertions(+), 27 deletions(-) diff --git a/.agents/model-matrix.md b/.agents/model-matrix.md index 2e7846ffd..8e2a97071 100644 --- a/.agents/model-matrix.md +++ b/.agents/model-matrix.md @@ -71,8 +71,8 @@ Rollup by lifecycle state (must equal the detailed per-state row counts): |---|---| | INVENTORIED | 323 | | PARTIAL | 20 | -| ACTIVE | 9 | -| SPIKE | 9 | +| ACTIVE | 10 | +| SPIKE | 8 | | BLOCKED | 5 | | DONE | 3 | | READY | 3 | @@ -112,7 +112,7 @@ Engaged architectures (the 50 non-`INVENTORIED` rows): | πŸ“‹ | `MuseGlimmerForConditionalGeneration` | Muse Glimmer 30B (Meta, agentic multimodal; iRoPE + gated attn + perception encoder + DFlash) | **W0-W7 ON `row/MODEL-MUSE-GLIMMER` (PR #279, NOT merged): 52-layer text forward, 50-layer perception encoder, mm wiring, ATEM reasoning + tool parsers β€” the forward no longer refuses.** The evidence behind that is NARROW and says so: text agrees with a torch transcription of #51655 and with HF's own `muse_glimmer` on REAL 30B tensors at **reduced depth 4/52 only** (5 prefill argmax positions, no generated tokens); full depth never ran, multi-step decode is untested, the perception encoder has NO reference check, nothing has run through the server, and the ATEM parsers' channel scoping does not work at server defaults (the seam has no `adjust_request` dispatch site and `skip_special_tokens` defaults true, so the framing is stripped first). The lifecycle token stays `SPIKE`/πŸ“‹ deliberately: advancing it owes `docs/STATUS.md` and `.agents/NOW.md` rows, and both are byte-exactly at their shrink-only ratchet/budget, so the advance belongs to the landing commit that can pay for them. Released 2026-08-08, **BEYOND-PIN**: no Muse code at `555967922` nor on vLLM `main`; the only upstream impl is the OPEN PR [#51655](https://github.com/vllm-project/vllm/pull/51655) head `075d645af` (approved, 3/20 CI red), ported from on explicit developer direction as a tracked exception (porting-inventory Β§9 deviation 16). **No oracle β‡’ no speed axis is claimable**; correctness gates against the HF reference. Text tower fully traced (Gemma-style sandwich norms w/ baked +1 offset + split eps, Llama-4-style iRoPE NoPE/full vs RoPE/sliding, weightless pre-RoPE QK-norm, ~3.87 query pre-scale w/ dual config schema, Qwen3.5-style attn output gate reading the LAYER INPUT); 50-layer perception encoder (linear patchify, bilinear pos-emb interp, width-first 2D RoPE, block-windowed attn, pixel-shuffle merge) reuses the Qwen3-VL vision seam; DFlash is recognition + `is_neox_style` threading on the existing speculator, not a new drafter | `MODEL-MM-muse-glimmer-muse-glimmer-for-conditional-generation` | | πŸ“‹ | `KimiK3ForConditionalGeneration` | Kimi K3 (2.8T MoE + MoonViT-V2, DERIVE-AND-SHIP) | **W2/W5 CPU scaffolding landed** (registry stub + nested text/vision/quant config descent + text-backbone structural name-map + REFUSE-by-name forward + MXFP4-refuse loader; clean CPU build, scaffold gate 6/6). text backbone IS `KimiLinearForCausalLM` (KDA+MLA+MoE hybrid, HEAVY reuse); **does NOT fit GB10 (~1.56 TB MXFP4, ~12Γ—)** and NOT in the pinned oracle β‡’ no on-box golden β€” DERIVED, proxy-gated on Kimi-Linear-48B; forward + MXFP4 + KDA delta + MoonViT-V2 not implemented (NOT-YET-BUILDABLE) | `MODEL-MM-kimi-k3-kimi-k3-for-conditional-generation` | | 🚧 | `MiniMaxH3DiTModel` | MiniMax-H3 (33.1B omni-modal video+audio DiT, DERIVE-AND-SHIP) | **W1/W2 landed**: packed layout (fl2va + ref2va, fp64 position grid BIT-EXACT), latent packing, euler-ancestral eta0 scheduler, and the full DiT forward all parity-gated against the UPSTREAM vLLM-Omni modules executed at reduced dimensions (**max abs diff 1.6e-7**, 10/10 cases / 2539 assertions). NOT autoregressive (no KV cache, no sampler, no logits) and **e2e HW-BLOCKED** (~354 GB checkpoint, ~133 GB/rank on 4x B300 vs 119 GiB unified); bf16 production stream + request planning + the ComfyUI-GGUF arm also landed (535 REAL tensors resolve onto our contract, geometry from shapes alone). **HW verdict CORRECTED: quantized arms FIT (~41 GB in 119 GiB)**, so e2e + speed are reachable; encoder/VAEs/audio VAE DONE (4.2e-9 vs the checkpoint's remote code); NVFP4 layout GATED as identical to ours (speed path is loader wiring); BOTH VAE DECODERS done (audio 4.2e-9, video ViT3D 8.9e-8); video tiling + 3D-CNN encoder (conditioning only) pending; encoder TEXT tower done (1.2e-7); **serving `/v1/videos` DONE and the DEVICE-RESIDENT forward (W2b, f32) LANDED + GPU-VERIFIED on Thor sm_110 at video 1.49e-7 / audio 8.94e-8**; bf16 stream + fusion folds + the FP4 path (needs sm_121a) + a real-checkpoint run pending. **2026-08-05: the AUDIO-VAE ENCODER is ported** (DAC analysis stack + `pre_block` AttnProjection + `mean_proj`, gated stage by stage vs the checkpoint's own remote code at 2.98e-8 / 1.64e-7 / 1.86e-8) with its own checkpoint loader gated on the real 1087-tensor manifest β€” so **ref2va AUDIO and VIDEO+AUDIO references are now WIRED** (audio rows move by 0.51 / 0.71; a different waveform still moves them by 7.1e-4). Both VAEs are now complete in both directions. **bf16 13-SHARD RELEASE INDEXES 2026-08-07 (`row/H3-BF16-SHARDED-DIT`)**: `MiniMaxH3ShardedCheckpoint` resolves the ORIGINAL 66.3 GB release through its own `model.safetensors.index.json` (a tensor named in the index but missing from its shard throws BY NAME), `EnumerateMiniMaxH3ShardedTensors` feeds the shared shapes-only geometry parser, and `LoadMiniMaxH3DitFromShards` is the host-f32 reference loader. Gated CPU-only at 72/72/54497 (post-rebase): every tensor resolves to the shard the index named AND to the bytes written there, the derived geometry equals the single-file path field for field, and a SPARSE 13-shard release with the REAL 535 tensors at REAL shapes (66.3 GB declared, 144 KB on disk) derives the SHIPPED geometry (50/5376/56/128/14336/24/32/1x2x2/5120). **STREAMS 2026-08-07 (`row/H3-BF16-SHARDED-STREAM`)**: `StreamMiniMaxH3ShardedToDeviceBf16` uploads it one tensor at a time β€” a BF16 tensor bound for a bf16 device slot goes straight from the mmap with ZERO host buffer, so peak host is bounded by ONE tensor (observed `host_peak=8192`, `direct=37 converted=9`); bit-exact vs the non-streamed `StageMiniMaxH3DitWeights` reference over all 46 views with identical logits, rope.inv_freq host-resident, 73/73/55203. Spec Β§8.14. **bf16 TEXT ENCODER + THE CONDITIONING NUMBER 2026-08-07 (`row/H3-ENC-BF16-COND-DIFF`)**: the 14-shard 63 GB bf16 Qwen3-VL-32B encoder streams to device too (`StreamMiniMaxH3EncoderShardsToDevice`, q/k/v and gate/up fused ON DEVICE), `--encoder-only` runs the tower alone (peak ~96 -> ~49 GiB by not loading the DiT first), and the widening is gated BIT-IDENTICAL vs an f32-staged tower so the A/B cannot be confounded. MEASURED on Thor over 233 tokens: Q4_K_M vs bf16 conditioning is cos 0.99745 mean / 0.909 min, rel RMS 6.85% excluding the attention sink, median rotation 3.5 deg β€” same energy as a ONE-WORD prompt edit but DIFFUSE (232/233 tokens rotate vs 172/233). Whether the RENDER changes is NOT established. 75/75/55609. Spec Β§8.15. This UNBLOCKS the quantization-quality question; no bf16-vs-quant render or speed number is claimed. Spec Β§8.13. **W-FP4a LANDED (CPU) 2026-08-06 (`row/H3-FP4-SPEED`)**: the device DiT forward now routes the NVFP4 projections through the shared Marlin W4A16 dispatcher (fp4 kept packed; no new quant code), fp4-vs-bf16 wiring gate GREEN (62/62Β·30039). **W-FP4a GB10 leg LANDED 2026-08-06 (`row/H3-FP4-GPU-E2E`, PR #64):** on sm_121a the Marlin W4A16 path RAN for all 11 projections (`dense_gemms==11` default β€” VT_MARLIN_DENSE is default-ON β†’ vLLM's own DENSE Marlin GEMM, not the grouped route; `marlin_gemms==11` under VT_MARLIN_DENSE=0; `fallback_gemms==0`), fp4-vs-bf16 BYTE-EXACT (max\|diff\|=0), and the fp4 arm is a MEMORY win not a diffusion-forward speed win (per-forward bf16/fp4 3.47Γ— @seq64 β†’ 0.79–0.83Γ— @seq4224–7040; ~16 vs ~66 GB device). Real-checkpoint fp4-resident t2va e2e RUNS (real 18.75 GB NVFP4 DiT + VAEs + GGUF Qwen3-VL-32B encoder β†’ valid mp4/wav; DiT s/step 5.45/20.0/209 s @512/768/REF-209f) but frames are a non-scene patch-grid at 12/20/50 steps β†’ OPEN render bug (device VAE/denoise). vLLM-Omni has no quantized H3 arm (BF16-only) so any comparison is HW/loader-forced-indirect β€” spec Β§8. **ONE-SURFACE ROW 2 LANDED 2026-08-08 (`row/H3-VIDEO-ABI`)**: the whole assembly pipeline is library-owned (`vllm::multimodal::MiniMaxH3VideoEngine`, `minimax_h3_video.cpp`) behind the ABI v12 `vllm_video_*` entry points; `/v1/videos` routes through the SAME seam; `minimax_h3_gen`+`minimax_h3_mux` are thin `vllm.h` clients, frames+WAV byte-identical to the pre-fold binary on the committed fold fixture (`test_minimax_h3_video_fold` 3-arm gate + the v12 `test_capi` section); GB10 real-video re-verify via the v12 ABI = named residual. **ROW 2 DEVICE-SEAM FOLLOW-UP (#135; replaces #134):** ABI 0/1 maps once to `vt::DeviceType`; shared code dispatches through `GetBackend(device_type)`, restoring DSR 34β†’32 without a baseline/allowlist change; CPU compile/fold test pending in CI due shared-disk pressure **PRUNED CHECKPOINTS LOAD AND RUN 2026-08-10 (`row/H3-PRUNED`, #241, spec Β§8.21):** the community `pruned` variants (`unsloth/MiniMax-H3-GGUF` Q2_K..Q8_0, `lilcheaty/MiniMax-H3-NVFP4` `*_pruned_nvfp4`) are not lossily pruned but ComfyUI's AdaLN timestep-CURVE refactor (`comfy/ldm/minimax/model.py:419-432,610-615`): an `adaln_t_table` [1025, 8] lerp replaces the sinusoidal+MLP time embedder, no SiLU before the AdaLN linear, and its in_features drop 2688 -> 8, collapsing `adaln_proj` from 13.04B of 33.12B to 0.04B and the DiT to 20.11B β€” so a pruned Q8_0 (21.4 GB) costs what our unpruned Q4_K_M (19.9 GB) costs. Gated on the REAL 532-tensor pruned GGUF manifest (532 = 535 - 4 `time_embedder.*` + 1 table, header-only), the clamped-lerp golden, and a CONSTRUCTED-curve test proving the pruned forward is numerically identical to the unpruned one; 79/79/57299. | `MODEL-DIFFUSION-minimax-h3-mini-max-h3-dit` | -| πŸ“‹ | `MiniMaxMusic3ForConditionalGeneration` | MiniMax-Music3 (text-to-music: ~8.6B Qwen3 global LLM + 0.646B 8-codebook RVQ depth decoder + 2.4B fp32 flow-matching DiT + 0.054B DAC Flow-VAE) | **W0 spec committed 2026-08-13 (#672, [spec](specs/minimax-music3.md)).** Geometry MEASURED from every component `config.json` plus safetensors headers by range request. First row to exercise the fallback-oracle rule (#647): absent from vLLM AND vLLM-Omni, so its primary oracle is the OPEN diffusers PR #14456 head `c6da9936` with SGLang-Omni `748a0b43` as the e2e cross-check. Not implemented | `MODEL-MUSIC-minimax-music3-mini-max-music3-for-conditional-generation` | +| 🚧 | `MiniMaxMusic3ForConditionalGeneration` | MiniMax-Music3 (text-to-music: ~8.6B Qwen3 global LLM + 0.646B 8-codebook RVQ depth decoder + 2.4B fp32 flow-matching DiT + 0.054B DAC Flow-VAE) | **W0 DONE + W1 LANDED (#672, [spec](specs/minimax-music3.md)).** Diffusers oracle gateable with committed per-stage goldens (#708); modular loader in the tree, 1413/1413 assertions against the real checkpoint (#714). Geometry MEASURED from every component `config.json` plus safetensors headers by range request. First row to exercise the fallback-oracle rule (#647): absent from vLLM AND vLLM-Omni, so its primary oracle is the OPEN diffusers PR #14456 head `c6da9936` with SGLang-Omni `748a0b43` as the e2e cross-check. W2/W3 in flight | `MODEL-MUSIC-minimax-music3-mini-max-music3-for-conditional-generation` | | πŸ“‹ | `LTX2VideoTransformer3DModel` | LTX-2.5 (21.00B joint video+audio flow-matching DiT, Lightricks) | **L0 spec committed 2026-08-11 (#435, [spec](specs/ltx-2-5.md)).** Geometry MEASURED from the FP8 checkpoint's own safetensors header by HTTP range request (6124 tensors, 881,048-byte header, no payload downloaded): **21.00B** params β€” 48 blocks @ 386.7M = 18.560B, audio embeddings connector 2.016B, global 0.427B. The filename says `22b` and the Diffusers card says ~19B; the MEASURED count is what this row uses. Video stream 4096 (32 heads x 128), audio stream 2048 (32 heads x 64), in/out channels 128 both. Per block SIX attentions β€” `attn1` (video self), `attn2` (video<->text, cross 4096), `audio_attn1`, `audio_attn2` (cross 2048), and the two CROSS-MODAL `audio_to_video_attn` / `video_to_audio_attn` β€” which is the structural break from MiniMax-H3: H3 packs every modality into ONE sequence with per-row token tags, LTX runs TWO streams coupled by explicit cross-attention. **Per-head gated attention on every attention** (`to_gate_logits` = `Linear(query_dim, heads, bias=True)`, `attention.py:513-514`, applied AFTER the attention output at `:577`) β€” H3 has no analogue and getting it wrong renders plausibly-wrong rather than erroring. FFN is gelu-approximate 4096->16384->4096 with **NO bias** while `audio_ff` 2048->8192->2048 **HAS** bias, which independently confirms `ff_bias=false` / `audio_ff_bias=true` from `model_configurator.py:78-80` β€” checkpoint and source agreeing, not either alone. **RETRACTED 2026-08-12, was billed as a FREE WIN.** The spec claimed 2.5 sets `use_prompt_adaln_single=false` so the cross-attention K/V are timestep-free and cacheable. The shipped checkpoint DISPROVES it: it carries 12 `prompt_adaln_single`/`audio_prompt_adaln_single` tensors including a `timestep_embedder.linear_1 [4096, 256]` (256 = the sinusoidal timestep width), and `model.py:223-227` builds that module ONLY when the flag is TRUE. `transformer.py:441` was quoted as proof of no timestep term, but `:442-443` add one whenever `prompt_timestep` is not None, and the comment above them says exactly that. NO SHIPPED DEFECT: `ltx2_dit.cpp:672` refuses the cache by name when the flag is on, so the feature is correct-and-inapplicable rather than silently wrong, and stays gated bit-identical and prompt-bound for any checkpoint that does set it false. **ORACLE:** vLLM-Omni does NOT support 2.5 β€” `resolve_ltx_pipeline_recipe` keys on 2 / 2.3 only and RAISES otherwise (`ltx2_recipes.py:162-166`), with upstream [vllm-omni#6066](https://github.com/vllm-project/vllm-omni/issues/6066) filed 2026-08-11 β€” but its `DiffusersAdapterPipeline` is generic (`DiffusionPipeline.from_pretrained`, `pipeline_diffusers_adapter.py:116`), so vLLM-Omni CAN execute 2.5 via `--load-format diffusers`. Binding oracle = that adapter; immediate cross-check = Lightricks `ltx-pipelines`. **HW: FITS ONE GB10** at ~29 GB (NVFP4 DiT 18.72 + NVFP4 Gemma-4 TE 7.40 + VAEs 1.83 + upscaler 1.00) vs H3's ~41 GB. **OWED UP FRONT, not to be discovered later:** the speed axis lands `PENDING` because the adapter is a black box (`supports_step_execution=False`, `supports_request_batch=False`) and therefore NOT vLLM's production configuration, which AGENTS.md requires as the denominator; DiffVAE (`NADiffusionDecoder`, neighborhood attention) is REFUSED BY NAME until its own row rather than silently downgraded to the Conv VAE; and no render-quality claim is made from structural e2e. | `MODEL-DIFFUSION-ltx-2-5-ltx2-video-transformer-3d-model` | | βœ… | `LagunaForCausalLM` | Poolside Laguna-S-2.1 (118B/8B MoE) | **LONG-CTX DECODE LEVERS LANDED + MEASURED (2026-08-03, `CLAIM-LAGUNA-LONGCTX-LEVERS`): window-bounded SWA reads (`VT_LAGUNA_SWA_WINDOW`, default-ON, BYTE-EXACT) bound the four `DecodeAttnGqa*` kernels' read to the ~512 sliding window (vLLM `laguna.py:412`) β€” GB10 A/B token-IDENTICAL `=1` vs `=0` at 520-token context (truncation active), MEASURED βˆ’0.30 ms/step at ~2k (~0 at ≀512, grows linearly). bf16 paged KV (`VT_LAGUNA_KV_BF16`, default-OFF opt-in) a distributional near-tie left UNRATIFIED. See BENCHMARKS `CLAIM-LAGUNA-LONGCTX-LEVERS`.** β€” **NVFP4 W4A4 ARM RAN on GB10 (N4, 2026-08-01, `CLAIM-LAGUNA-NVFP4-N4`): the additive safetensors NVFP4 arm (N1a/N1b/N2/N3 β€” `Nvfp4Weight` expert fields + `LoadLagunaForCausalLMWeights` + `LqGemmNvfp4Fp4` per-expert TRUE-W4A4 + `LagunaFfnBlock` `fp4` branch + `laguna_gen` dir-autodetect; CPU-gated `test_laguna_nvfp4_loader` 3/3Β·61, GGUF path byte-identical) generates COHERENTLY on the real 67 GiB `poolside/Laguna-S-2.1-NVFP4`. vs the vLLM MARLIN golden (vLLM's exact prompt ids injected): FIRST 2 TOKENS MATCH exactly, then near-tie divergence (our TRUE-W4A4 fp4-activations vs the MARLIN golden's W4A16 bf16-activations β€” different precision, EXPECTED; shares golden vocab). SPEED (N5, trace-driven, 2026-08-01): 0.16 β†’ ~4.5 tok/s (~28Γ— THIS SESSION), now ~4Γ— from vLLM 18.8. **Lever #2** (nsys found the bf16 tower running host `MatmulNK` on the CUDA queue): route it to the GPU (`LqGemm` bf16 β†’ `CastBf16` + `MatmulBT`, weight stays bf16) β†’ 6.34 β†’ 0.39 s/tok (16Γ—). **Lever #1** (nsys found the emulation expert GEMM at 92%, GPU 87% busy): the engine's native sm120a fp4 tensor-core MMA (`MatmulNvfp4Fp4Native`) reads the SAME linear scales β€” it was gated OFF behind `VT_NVFP4_FP4_NATIVE`; default it ON in the driver β†’ 0.39 β†’ ~0.20-0.24 s/tok (~2Γ—). Both coherent + near-tie (byte-identical ids to emulation; first token matches golden). Two GB10 memory fixes landed to run (shard-release + context-before-load). OPEN #234 (remaining ~4Γ—): grouped W4A4 MoE (top_kΓ—3 launches β†’ 3), `ResidentNvfp4`, decode CUDA-graph + on-GPU sampling (the host-orchestration tail). Spec `.agents/specs/laguna-nvfp4-arm-2026-07-31.md` Β§N4/Β§N5. The GGUF-Q4_K track (below) is the separate keep-quant vehicle.** Prior **FASTER DECODE (W9, 2026-07-31, `CLAIM-LAGUNA-W9-GROUPED`): the 30 un-grouped per-expert keep-quant GEMV launches/step (top_k Γ— {gate,up,down} `LqGemmRowSlice`) fold onto the SHARED `vt::MatmulBTQuantGrouped` op β€” per token, Pk experts' gate/up/down each collapse to ONE grouped launch over the already-stacked `[E*N,H]` tower (no loader change). Same-binary A/B on real UD-Q4_K_XL (GB10, `--gpu`, drop_caches cold, 24 tok): grouped (`VT_LAGUNA_GROUPED_MOE=1`, default) == per-expert (`=0`) BYTE-IDENTICAL (md5 `754728c6`, both == W6 golden) + decode 0.18 β†’ 0.13 s/tok (1.38Γ—). Routes through the shared vt op (fold policy). Cumulative with W8: decode 0.66 β†’ 0.13 s/tok (5.1Γ—; 1.5 β†’ 7.7 tok/s; 18Γ— β†’ 3.6Γ— vs llama.cpp 27.8). Next lever: device-resident decode (#1). See spec Β§W9.** Prior **FASTER DECODE (W8, 2026-07-31, `CLAIM-LAGUNA-W8-EMBED`): `LagunaEmbed` no longer converts the whole 1.23 GB embed table to f32 every token (it gathered T rows out of the whole [Vsz,H] table via `ReadF32` β€” ~311M host element-converts/token, the DOMINANT decode cost the W7 profile under-filed as "#5"); now gathers only the T needed rows directly (BIT-IDENTICAL β€” same per-element conversion, same rows). GATED on the real 3-shard UD-Q4_K_XL GGUF (GB10, `--gpu`, W6 cached, drop_caches cold, 24 tok): TOKEN-IDENTICAL to the W5/W6 golden (`22345 83 350 785 …`, coherent " Paris.") + decode 0.66 β†’ 0.17 s/tok = 3.9Γ— (1.5 β†’ 5.9 tok/s; 18Γ— β†’ 4.7Γ— vs llama.cpp 27.8). See `.agents/specs/laguna-s21-w7-speed-2026-07-31.md` Β§W8. Next: grouped-expert GEMM (=A3) then device-resident decode.** Prior **DECODE-SPEED ATTRIBUTED (W7 profile-only, 2026-07-31, `CLAIM-LAGUNA-W7-SPEED`): `nsys` of the W6 decode (real UD-Q4_K_XL GGUF, GB10) attributes the 0.66 s/tok (~1.5 tok/s vs llama.cpp 27.8 on identical bytes, ~15-18x) to HOST-ORCHESTRATION, not kernel compute β€” GPU active only 32.7% of the step, 67.3% host/idle; 22,115 `cudaStreamSynchronize` (~2,764/step, zero GPU overlap) from the ~1,795 per-GEMM `DrainQueue` in `LagunaForwardGgufCached` + scalar host glue; 39.4% of GPU time is `QuantizeQ8K` activation-quant (per-GEMM), weight GEMVs un-grouped at ~22% of the 240 GB/s peak (llama.cpp ~76%); no H2D/D2H (unified memory). Ranked levers (all in-tree from ds4): device-resident decode 1.5->~5-7 tok/s, grouped-expert GEMM (`MatmulBTQuantGrouped`) +1.5-2x + dedupes the activation-quant, decode CUDA-graph, tuned MMVQ; + free host cleanups (`LagunaEmbed` copies the whole 1.23 GB embed table/token, per-token RoPE-cache rebuild). Honest reachable ~13-20 tok/s, 27.8 a stretch. NO code changed. See `.agents/specs/laguna-s21-w7-speed-2026-07-31.md`. Prior RUNNABLE + FAST DECODE (W6, 2026-07-31): a per-layer K/V cache + single-token incremental decode replaces W5's O(nΒ²) STATELESS recompute β€” TOKEN-IDENTICAL (byte-equal ids, md5 `754728c6…` match, == the W5 golden) and 5.05Γ— faster per token: decode 3.33 β†’ 0.66 s/tok on the real UD-Q4_K_XL GGUF (GB10, `--gpu`, keep-quant), same " Paris.…" text. `LagunaKvCache` (mirrors `DeepseekV4KvCache`, MLA-latent β†’ GQA multi-head K/V; caches post-QK-RMSNorm/post-RoPE K + raw V at f32 β€” bit-exact since RoPE/QK-norm are position-only and attention is causal), MIXED attention per-layer: 12 GLOBAL layers grow unbounded + 36 SLIDING-WINDOW-512 layers EVICT rows beyond the 512 window (gemma2/3 `is_sliding`); `LagunaForwardGgufCached` + shared `LagunaAttention`/`LagunaFfnBlock` helpers used by BOTH forwards (identical float ops; recompute ids unchanged after refactor), `examples/laguna_gen --stateless` A/B flag. No cache bug (bit-exact first run). Next speed = grouped-expert GEMM + device-resident decode (both in-tree from ds4). See `.agents/specs/laguna-s21-w6-2026-07-31.md`. Prior RUNNABLE (W5, 2026-07-31): our engine greedy-generates COHERENT text on the REAL 3-shard UD-Q4_K_XL GGUF (GB10 keep-quant) β€” "The capital of France is" β†’ " Paris. …", first token "Paris." matches the llama.cpp-Poolside reference. Multi-shard GGUF reader + keep-quant tower (`LoadLagunaFromGgufShards`) + `LagunaForwardGguf` (ds4 keep-quant Gemm/GemmRowSlice) + `examples/laguna_gen`; load 20.6s, peak 71 GiB, 3.27 s/tok stateless recompute (speed=W6).** Prior W3: **W3 REAL forward + 3 new ops landed** (`laguna_ops.cpp`: per-head softplus attn out-gate + ungrouped sigmoid-noaux router + dual per-layer RoPE cos/sin builders; `LagunaModel::Forward` now a REAL runnable host-reference composition β€” variable-Q-head GQA + dual RoPE + sliding-window mask + softplus gate + dense L0 / ungrouped-MoE L1..47 + untied lm_head β€” replacing the W1/W2 `VT_CHECK(false)` stub; CPU `-Werror` full-library build clean; `test_laguna_scaffold` **8/8Β·166** incl. softplus math, router selection+tie-break RED-first, dual-RoPE cos/sin bit-match, variable-Q-head shapes, forward composition on synthetic weights; `test_model_registry` 24/24). W1 oracle DECISION: vLLM native `laguna.py` in pin β‡’ config constructs; dual-oracle = vLLM-NVFP4/-FP8 (fits GB10, BF16 235 GiB does NOT) + llama.cpp-Q4_K token-exact. DEFERRED to W4 (needs 73 GB checkpoint): GGUF keep-quant tower materialization + device/paged production forward + strict dual-oracle greedy gate. ~85-90% reuse (ds4-MoE + gemma-sliding + olmo3-dual-rope + landed Q4_K keep-quant); NEW = the 3 landed host ops + name-map + variable-Q-head device runner. **W4 (2026-07-31, `CLAIM-LAGUNA-W4`, in progress):** the UD-Q4_K_XL GGUF (73.4 GiB, 3 shards) FETCHED to dgx + its metadata/tensor-map READ AUTHORITATIVELY (814 tensors, arch `laguna`, `expert_gating_func=2` sigmoid, `leading_dense_block_count=1`, `expert_weights_scale=2.5`). Three CPU-verified FIDELITY corrections the W1-W3 scaffold got wrong, each grounded in the real GGUF + llama.cpp: (1) **per-head QK-RMSNorm** (`attn_q_norm`/`attn_k_norm` F32[128]) added to params+forward β€” the scope MISSED it (surfaces only in the tensor map); (2) **dual-RoPE mscale** now uses llama.cpp's `yarn_attn_factorΒ·(1+0.1Β·ln(factor))` off the GGUF-authoritative `factor=32`/`yarn_attn_factor=1.0` (256K-ctx build, NOT HF's factor-128/1.4852 1M-ctx scalar) β€” resolves the numerics-delicate residual; (3) **separate** `ffn_gate_exps`/`ffn_up_exps` (Q4_K) + `ffn_down_exps` (Q5_K) + Q8_0 shared/attn (the scaffold assumed merged gate_up). GGUF keep-quant tower materialization (`Mw`/`Sew` mirror of ds4) + keep-quant `ForwardGguf` (vt::MatmulBT/GemmRowSlice) + the real-model greedy run vs the llama.cpp-laguna same-quant oracle remain the W5 close (73 GB single-GB10, host-orchestrated) | `MODEL-TEXT-laguna-laguna-for-causal-lm` | | 🚫 | `DeepseekV3ForCausalLM` / `DeepseekV32ForCausalLM` | DeepSeek-V3 / V3.2 | HW-blocked (671B, ~642 GiB fp8 vs 119 GiB unified memory); V3.2 additionally DEP-blocked (DSA indexer) | `MODEL-TEXT-deepseek-v2-deepseek-v3-for-causal-lm` | @@ -450,7 +450,7 @@ Transformers compatibility is capability-driven and excluded from finite counts. | `MODEL-MM-voxtral-realtime-voxtral-realtime-for-conditional-generation` | `VoxtralRealtimeForConditionalGeneration` (target-pending) | **TARGET-PENDING β€” the architecture string is registered NOWHERE we can find.** Recipe `mistralai/Voxtral-Mini-4B-Realtime-2602` (one of the 157 recipes at `vllm-project/recipes` `86c7777a`) declares `architectures: ["VoxtralRealtimeForConditionalGeneration"]` with an inner `audio_config.model_type: "voxtral_realtime_encoder"`, but that exact string appears in none of: (a) the pinned registry `555967922`; (b) any of the ten dictionaries of `vllm/model_executor/models/registry.py` on vLLM `main`; (c) `_OMNI_MODELS` in `vllm_omni/model_executor/models/registry.py` @ `bbe6ccc512a404a2df8c977ea29003002f2683e8`; (d) `vllm-omni` `docs/models/supported_models.md`. **Related but NOT equal, recorded so nobody re-derives it:** `VoxtralRealtimeGeneration` β€” a DIFFERENT string β€” is registered at the pin (`registry.py:585`, `main` `:603`) and already has its own row here, `MODEL-MM-voxtral-realtime-voxtral-realtime-generation`; `VoxtralForConditionalGeneration` (pin `registry.py:584`) is the audioβ†’TEXT model; and `vllm-omni` `registry.py:305-309` registers `VoxtralTTSForConditionalGeneration` for the SEPARATE `mistralai/Voxtral-4B-TTS-2603` recipe. The most likely reading is that this checkpoint's `config.json` names a renamed or not-yet-landed spelling of the existing `VoxtralRealtimeGeneration` target, but that is a hypothesis: no anchor is asserted until one of those is shown to be what actually resolves. Tracked by [#610](https://github.com/mudler/vllm.cpp/issues/610) | conditional generation / audio in, AUDIO OUT (realtime speech-to-speech) β€” modality inferred from the recipe and config, NOT from an upstream implementation | unresolved until the target is located | ☐ required | `INVENTORIED` | none | unassigned | | `MODEL-MM-bailing-mm-native-bailing-mm-native-for-conditional-generation` | `BailingMMNativeForConditionalGeneration` (target-pending) | **TARGET-PENDING β€” the architecture string is registered NOWHERE we can find.** Recipe `inclusionAI/Ming-omni-tts-0.5B` (one of the 157 recipes at `vllm-project/recipes` `86c7777a`) declares `architectures: ["BailingMMNativeForConditionalGeneration"]`, but that exact string appears in none of: (a) the pinned registry `555967922`; (b) any dictionary of `vllm/model_executor/models/registry.py` on vLLM `main`; (c) `_OMNI_MODELS` in `vllm_omni/model_executor/models/registry.py` @ `bbe6ccc512a404a2df8c977ea29003002f2683e8`; (d) `vllm-omni` `docs/models/supported_models.md`. **Related but NOT equal:** `vllm-omni` `docs/models/supported_models.md:75` lists that SAME checkpoint under a different architecture name, `MingTTSForConditionalGeneration` (`registry.py:250-254` β†’ module `ming_tts/ming_tts.py`, class `MingTTSForConditionalGeneration`); and `vllm-omni` `registry.py:417-421` carries a near-miss alias `BailingMM2NativeForConditionalGeneration` β€” note the `2` β€” commented "HF repo currently ships this architecture name in config.json", pointing at module `ming_flash_omni/ming_flash_omni.py`, class `MingFlashOmniForConditionalGeneration`, for the DIFFERENT Ming-flash-omni-2.0 checkpoint. So the shape of the gap is a missing config-string alias upstream, but that is an observation, not an anchor, and none is asserted here. Tracked by [#610](https://github.com/mudler/vllm.cpp/issues/610) | conditional generation / text+image+audio in, AUDIO OUT (TTS) β€” modality from the recipe and config, NOT from a resolved upstream target | unresolved until the target is located | ☐ required | `INVENTORIED` | none | unassigned | | `MODEL-DIFFUSION-ltx-2-5-ltx2-video-transformer-3d-model` | `LTX2VideoTransformer3DModel` | **BEYOND-PIN AND OUT-OF-REPO** β€” not in `555967922` (LTX-2.5 released 2026-08) and not in the vLLM repository at all. Architecture reference is Lightricks' own `LTX-2` (`packages/ltx-core/src/ltx_core/`: `model/transformer/{transformer,model,attention,rope,model_configurator}.py`, `model/audio_vae/`, `components/`, `guidance/`). vLLM-Omni carries an `ltx2` module but its recipes stop at 2.3 (`vllm_omni/diffusion/models/ltx2/ltx2_recipes.py:162-166`); 2.5 is upstream-OPEN at [vllm-omni#6066](https://github.com/vllm-project/vllm-omni/issues/6066) and [#4985](https://github.com/vllm-project/vllm-omni/issues/4985). Serving oracle reaches 2.5 through the GENERIC `DiffusersAdapterPipeline` (`pipeline_diffusers_adapter.py:116`) against `Lightricks/LTX-2.5-Diffusers`. | diffusion generation / video + audio (text/image/video in, video+audio out) | flow-matching denoise loop; TWO coupled modality streams with audio<->video cross-attention; per-head gated attention; AdaLN-single with 9-vector per-block tables; timestep-INDEPENDENT prompt K/V (cacheable once per request); split/interleaved 3D RoPE; gelu-approximate FFN; Gemma-4 12B text encoder with dual caption projections (4096 video / 2048 audio); Conv video VAE + DiffVAE; audio VAE + vocoder; latent spatial/temporal x2 upsamplers; duration head; distilled two-stage recipe | [ltx-2.5](specs/ltx-2-5.md) | `SPIKE` | **L0 LANDED 2026-08-11 (#435).** Spec committed BEFORE implementation per AGENTS.md. Geometry measured from the ungated `vonkaiser/LTX-2.5-FP8-NVFP4` FP8 checkpoint header by range request β€” the same no-payload technique used for H3's manifests: 21.00B params, 48 blocks, 1775 F8_E4M3 tensors each with a per-tensor F32 `weight_scale`, biases/norms BF16. Cross-modal projection shapes are ASYMMETRIC and are a named test trap: `audio_to_video_attn.to_q` is [2048, 4096] while `to_k`/`to_v` are [2048, 2048] and `to_out` is [4096, 2048], so a transposition still type-checks against a square assumption. **CHECKPOINT ACCESS (verified against the HF API 2026-08-11):** `Lightricks/LTX-2.5` is `gated: auto` (accepting the license opens it; holds the first-party NVFP4 DiT at 18.72 GB), `Lightricks/LTX-2.5-Diffusers` is RESTRICTED behind manual approval (needed for the binding oracle), and `vonkaiser/LTX-2.5-FP8-NVFP4` is UNGATED and unblocks L1-L2 today. All artifacts land under `$CHECKPOINT_ROOT` (`/mnt/nas_share/checkpoints`) so dgx.casa and the cluster nodes share one copy. **PHASES (one PR, developer-directed):** L1 `vllm::multimodal::VideoEngine` interface + checkpoint-detected registry with MiniMax-H3 moved behind it UNCHANGED (gated on frames+WAV byte-identical to the pre-refactor fold fixture) and ABI **v13 by ADDING fields only**; L2 DiT forward; L3 Gemma-4 TE reusing `gemma4.cpp`; L4 Conv video VAE + audio VAE + vocoder; L5 pipeline/recipes/upsampler/duration head; L6 NVFP4 arms + GB10 load-time residency; L7 e2e on dgx.casa under `flock $HOME/gpu.lock`. **OPEN, same as H3:** there is no vllm-omni parity PIN β€” `.agents/upstream-sync.md` covers the vLLM repo only, so each golden records the vllm-omni SHA inline. | `CLAIM-LTX25-L0-L7` | -| `MODEL-MUSIC-minimax-music3-mini-max-music3-for-conditional-generation` | `MiniMaxMusic3ForConditionalGeneration` | **BEYOND-PIN AND OUT-OF-REPO** β€” not in `555967922` and not in the vLLM repository at all; absent from `vllm-omni` too. Its reference implementation is the OPEN `diffusers` PR [#14456](https://github.com/huggingface/diffusers/pull/14456) head `c6da9936` (`src/diffusers/models/transformers/transformer_minimax_music3.py`, `.../minimax_music3_rvq_depth_decoder.py`, `src/diffusers/models/autoencoders/minimax_music3_vocoder.py`, `src/diffusers/models/condition_embedders/condition_embedder_minimax_music3.py`, `src/diffusers/modular_pipelines/minimax_music3/`), with SGLang-Omni `748a0b43` (`sglang_omni/models/minimax_music3/`) serving the NATIVE layout as the e2e cross-check. **The first row to exercise the fallback-oracle rule (#647)** | music generation / text -> audio (lyrics + structured description in, multi-minute stereo song out) | Qwen3 dense forward + paged KV (LANDED); flow-matching denoise loop (H3/LTX-2.5 precedent); DAC-style audio VAE decode + WAV writer (H3/LTX-2 precedent); NEW: 8-codebook RVQ frame path, 4-layer depth decoder, learned 8-layer condition mix, snake activations with weight-norm folding, and an LLM->diffusion handoff on CONTINUOUS hidden states rather than discrete tokens | [minimax-music3](specs/minimax-music3.md) | `SPIKE` | **W0 spec committed 2026-08-13 (#672).** Spec committed BEFORE implementation per AGENTS.md. Geometry MEASURED, not inferred: every component `config.json` read, plus safetensors headers by HTTP range request β€” `transformer` 231 tensors/1.240B per shard **all F32** (so the card's 2.4B is right and 9.73 GB is fp32 storage, NOT a 4.9B bf16 model), `rvq_depth_decoder` 47/0.646B BF16, `vocoder` 121/0.054B F32 with `weight_g`/`weight_v` weight-norm, `condition_encoder` **4 tensors** (`layer_scale`, `layer_weight_logits`, `proj`) which makes it a learned weighted mix over 8 LLM layers rather than an encoder tower. `language_model` is our LANDED `Qwen3ForCausalLM` architecture exactly (36L/4096/32-8/head_dim 128) at `vocab_size 200000`. The repo ships TWO packagings of ONE set of weights (native `qwen_7B/`+`flowmatching_vae.pth`+`dav.pth` vs the diffusers six); `scripts/convert_minimax_music3_to_diffusers.py@c6da9936` `:29-38,47-189` proves the conversion is a pure re-layout, so the diffusers arm is the port target at **~28.5 GB resident** (no pickle, no `trust_remote_code`) and SGLang-Omni stays a valid e2e cross-check. **Sample rate RESOLVED 2026-08-13** (spec Β§1.1): not a contradiction but a stage boundary. The vocoder natively emits 44100 stereo (86.133 Hz latent frames x 512 upsampling, derived from `minimax_music3_vocoder.py:84,92-95` and confirmed by `dav.py:94,115`); diffusers returns that unresampled, while SGLang-Omni's SERVER resamples 44100 -> 32000 on delivery (`constants.py:18-19`, `acoustic.py:55-58,422-431`). Goldens are captured at 44100 stereo, the native resample-free rate; 32 kHz is a downstream delivery transform gated separately, and NOT free β€” reproducing its bytes needs torchaudio's default sinc filter, and a latent-parity check sits upstream of that call and cannot see it | `CLAIM-MODEL-MUSIC3-W0` | +| `MODEL-MUSIC-minimax-music3-mini-max-music3-for-conditional-generation` | `MiniMaxMusic3ForConditionalGeneration` | **BEYOND-PIN AND OUT-OF-REPO** β€” not in `555967922` and not in the vLLM repository at all; absent from `vllm-omni` too. Its reference implementation is the OPEN `diffusers` PR [#14456](https://github.com/huggingface/diffusers/pull/14456) head `c6da9936` (`src/diffusers/models/transformers/transformer_minimax_music3.py`, `.../minimax_music3_rvq_depth_decoder.py`, `src/diffusers/models/autoencoders/minimax_music3_vocoder.py`, `src/diffusers/models/condition_embedders/condition_embedder_minimax_music3.py`, `src/diffusers/modular_pipelines/minimax_music3/`), with SGLang-Omni `748a0b43` (`sglang_omni/models/minimax_music3/`) serving the NATIVE layout as the e2e cross-check. **The first row to exercise the fallback-oracle rule (#647)** | music generation / text -> audio (lyrics + structured description in, multi-minute stereo song out) | Qwen3 dense forward + paged KV (LANDED); flow-matching denoise loop (H3/LTX-2.5 precedent); DAC-style audio VAE decode + WAV writer (H3/LTX-2 precedent); NEW: 8-codebook RVQ frame path, 4-layer depth decoder, learned 8-layer condition mix, snake activations with weight-norm folding, and an LLM->diffusion handoff on CONTINUOUS hidden states rather than discrete tokens | [minimax-music3](specs/minimax-music3.md) | `ACTIVE` | **W0 DONE + W1 LANDED (#672).** Oracle GATEABLE: `tools/oracle/music3_oracle.py` generated 0.998 s of 44100 Hz stereo at diffusers `c6da9936` (identity asserted at runtime, proved red first), with 13 per-stage goldens under `tests/parity/goldens/minimax_music3_oracle/` (#708). W1 modular loader `minimax_music3_loader.{h,cpp}`: 1413/1413 assertions against the real 27 GB tree, all 1012 tensors accounted, 50 mutations proven to fire, native arm refused by name; `MaterializeWeightNorm` relocated into the shared `vocoder1d` primitives (H3 79/79 cases, 57,395 assertions unchanged) rather than forked (#714). Spec committed BEFORE implementation per AGENTS.md. Geometry MEASURED, not inferred: every component `config.json` read, plus safetensors headers by HTTP range request β€” `transformer` 231 tensors/1.240B per shard **all F32** (so the card's 2.4B is right and 9.73 GB is fp32 storage, NOT a 4.9B bf16 model), `rvq_depth_decoder` 47/0.646B BF16, `vocoder` 121/0.054B F32 with `weight_g`/`weight_v` weight-norm, `condition_encoder` **4 tensors** (`layer_scale`, `layer_weight_logits`, `proj`) which makes it a learned weighted mix over 8 LLM layers rather than an encoder tower. `language_model` is our LANDED `Qwen3ForCausalLM` architecture exactly (36L/4096/32-8/head_dim 128) at `vocab_size 200000`. The repo ships TWO packagings of ONE set of weights (native `qwen_7B/`+`flowmatching_vae.pth`+`dav.pth` vs the diffusers six); `scripts/convert_minimax_music3_to_diffusers.py@c6da9936` `:29-38,47-189` proves the conversion is a pure re-layout, so the diffusers arm is the port target at **~28.5 GB resident** (no pickle, no `trust_remote_code`) and SGLang-Omni stays a valid e2e cross-check. **Sample rate RESOLVED 2026-08-13** (spec Β§1.1): not a contradiction but a stage boundary. The vocoder natively emits 44100 stereo (86.133 Hz latent frames x 512 upsampling, derived from `minimax_music3_vocoder.py:84,92-95` and confirmed by `dav.py:94,115`); diffusers returns that unresampled, while SGLang-Omni's SERVER resamples 44100 -> 32000 on delivery (`constants.py:18-19`, `acoustic.py:55-58,422-431`). Goldens are captured at 44100 stereo, the native resample-free rate; 32 kHz is a downstream delivery transform gated separately, and NOT free β€” reproducing its bytes needs torchaudio's default sinc filter, and a latent-parity check sits upstream of that call and cannot see it | `CLAIM-MODEL-MUSIC3-W0` | | `MODEL-MM-moss-transcribe-diarize-moss-transcribe-diarize-for-conditional-generation` | `MossTranscribeDiarizeForConditionalGeneration` (v0.25.0 target-pending) | v0.25.0 target `registry.py:450-453`; `vllm/model_executor/models/moss_transcribe_diarize.py::MossTranscribeDiarizeForConditionalGeneration` @ `702f481` | conditional generation / audio | MM processor; Whisper encoder; VQ adaptor; Qwen3 decoder; speech-to-text/diarization frontend | ☐ required | `INVENTORIED` | none | unassigned | | `MODEL-MM-lightonocr-light-on-ocrfor-conditional-generation` | `LightOnOCRForConditionalGeneration` | `registry.py:450-453`; `vllm/model_executor/models/lightonocr.py::LightOnOCRForConditionalGeneration` | conditional generation / image | MM processor; encoder/merge; vision encoder | ☐ required | `INVENTORIED` | none | unassigned | | `MODEL-MM-lfm2-vl-lfm2-vlfor-conditional-generation` | `Lfm2VlForConditionalGeneration` | `registry.py:454`; `vllm/model_executor/models/lfm2_vl.py::Lfm2VLForConditionalGeneration` | conditional generation / image | MM processor; encoder/merge; Mamba/SSM state; vision encoder; video path | ☐ required | `INVENTORIED` | none | unassigned | diff --git a/.agents/specs/minimax-music3.md b/.agents/specs/minimax-music3.md index ba8b50cea..8281c71c8 100644 --- a/.agents/specs/minimax-music3.md +++ b/.agents/specs/minimax-music3.md @@ -10,7 +10,7 @@ [#14456](https://github.com/huggingface/diffusers/pull/14456), head `c6da9936e4bda83107943a16eb8682e9a37d8527` β€” **OPEN, not merged**. **Cross-check:** SGLang-Omni `748a0b437e4a8faad44d7bbfd5a0ae55d1fef830`. -**Status:** W0 β€” spec committed, diffusers oracle gateable, no engine code. +**Status:** **W0 DONE, W1 LANDED.** Spec committed, both oracles pinned, Β§1.1 resolved and confirmed at runtime, the diffusers oracle gateable against committed goldens, and the modular loader in the tree. W2/W3 dispatched. **Developer directive (2026-08-13):** "land minimax music 3 support complete, to vllm.cpp, wired to the ABI and to the example http server, merge to main, tested e2e." That fixes W6's shape (the ABI surface and the example server are in scope, @@ -216,9 +216,14 @@ force-pushed under us, and a comparison against "whatever the branch was that da is not reproducible. If the PR merges, advancing to the merge commit is a pin advance with its own reconciliation, not a silent follow. -Both pins go into `.agents/oracles/` in W0. SGLang-Omni gets its **own record** -rather than riding on `sglang.md`: it is a third repository with its own cadence, -and this row binds to it directly. +**Both pins are recorded** β€” [`../oracles/diffusers.md`](../oracles/diffusers.md) +and [`../oracles/sglang-omni.md`](../oracles/sglang-omni.md), landed in #679 and +advanced in #708. SGLang-Omni has its **own record** rather than riding on +`sglang.md`: it is a third repository with its own cadence, and this row binds to +it directly. (An earlier revision said the pins "go into `.agents/oracles/` in +W0", which read as future work and misled two implementers into reporting the +SGLang-Omni record as owed after it existed. Present tense, because the record +is a fact and not a plan.) --- @@ -271,6 +276,20 @@ LLMβ†’diffusion handoff on *continuous hidden states* rather than discrete token --- +## 4G. The row's structured record + +| Field | Value | +|---|---| +| Scope | IN: the diffusers-arm six-component checkpoint, all five modules, load through waveform; lyrics + structured description in, 44100 Hz stereo out; registration as a `SpeechRegistry` family with the `vllm_speech_*` ABI and the example HTTP server as a thin client (Β§4.1); quantized arms incl. GGUF k-quants (W7). OUT: the native `AbabForCausalLM` + `.pth` arm, refused by name (Β§2); streaming, which upstream does not support and which is refused rather than faked; a 32 kHz delivery arm, which is a downstream resample gated separately (Β§1.1); any change to `SpeechEngine` behaviour for IndexTTS-2.5. | +| Upstream chain | `minimax_music3` is ABSENT from the pinned vLLM, from vLLM `main` and from `vllm-omni` β€” this row is why AGENTS.md Β§"When vLLM has no implementation" exists. Primary oracle `diffusers` PR [#14456](https://github.com/huggingface/diffusers/pull/14456) head `c6da9936` (OPEN), [`../oracles/diffusers.md`](../oracles/diffusers.md) `gateable = yes`. Cross-check SGLang-Omni `748a0b43`, [`../oracles/sglang-omni.md`](../oracles/sglang-omni.md) `gateable = no`, which serves the NATIVE layout (Β§2). `transformers` 5.14.1 for the `Qwen3ForCausalLM` half. Checkpoint `MiniMaxAI/MiniMax-Music3` diffusers arm, 27 GB, at `/mnt/nas_share/checkpoints/minimax-music3`. | +| Our baseline | LANDED for this row: the modular loader `minimax_music3_loader.{h,cpp}` (#714, 1413/1413 assertions against the real tree, all 1012 tensors accounted, native arm refused by name) and the gateable oracle `tools/oracle/music3_oracle.py` with 13 per-stage goldens (#708). REUSED rather than rebuilt: the token-exact Qwen3 dense forward and paged KV, the `vocoder1d` primitives, `multimodal::SpeechEngine`, and the H3 / LTX-2.5 flow-matching and audio-VAE precedent (Β§4, Β§4.1). Before this row there was no music generation and no text-to-audio path of any kind. | +| Port map | loader -> `src/vllm/model_executor/models/minimax_music3_loader.cpp` (LANDED, from `scripts/convert_minimax_music3_to_diffusers.py`). `language_model` -> the landed Qwen3 dense path (W2). `condition_embedder_minimax_music3.py` -> W3. `minimax_music3_rvq_depth_decoder.py` -> W3. `transformer_minimax_music3.py` + `FlowMatchEulerDiscreteScheduler` -> W4. `minimax_music3_vocoder.py` -> W5, over the shared `vocoder1d` primitives. `modular_pipelines/minimax_music3/{encoders,before_denoise,denoise,decoders}.py` -> W6. | +| Tests to port | Upstream ships NO unit tests for this model at the pinned SHA β€” the PR carries docs, a conversion script and the modules, and nothing test-shaped was found in the seven PR files fetched. So the references are CAPTURED, not ported, and this spec says so rather than implying a port that never happened: `tests/parity/goldens/minimax_music3_oracle/` holds per-stage tensors with a manifest recording shape, dtype, sha256 and min/max/mean per entry. Each phase gates against its own stage's entry. If upstream later adds tests, they are ported in the same change that touches the corresponding module. | +| Gates | Split by half, and conflating them is the failure mode Β§0 warns about. LLM half: TOKEN-EXACT against `rvq_codes.npy` `[26,8]` int32, where row 0 is the priming decode that emits no frame so `rows[1:]` align with the 25 frames. Acoustic half: per-stage tensor parity at fixed seed and reduced dimensions against `condition_chunk0`, `denoise_{first,last}_*`, `vocoder_input_chunk0`, `waveform` β€” no logits exist, so no token gate does either. A correlation coefficient is NOT a gate here: Pearson is scale-invariant and cannot see a uniformly scaled latent. Speed is measured against SGLang-Omni in its production configuration (both CUDA graphs, compiled DIT and DAV, batched seeded sampling), never with those disabled. | +| Dependencies | `multimodal::SpeechEngine` + `SpeechRegistry` for W6, extended additively per Β§4.1 with IndexTTS-2.5 left byte-identical. The landed Qwen3 dense forward and paged KV for W2. The `vocoder1d` primitives for W5. The diffusers oracle staying gateable at its pin, for every phase. NO dependency on vLLM-Omni, and none on `dgx.casa`, which was down throughout W0 β€” the correctness gate runs on CPU by design. | +| Work breakdown | Β§6. W0 spec + both oracle pins + Β§1.1 + oracle stand-up (DONE). W1 modular loader, weight-norm folding, dtype invariant, native-arm refusal (DONE). W2 global LLM. W3 condition mix + RVQ depth decoder. W4 flow-matching DiT + scheduler. W5 vocoder over `vocoder1d`. W6 speech-family registration + `vllm_speech_*` ABI + example HTTP server. W7 quantized arms, anything unimplemented refused by name. | +| Risks/decisions | The primary oracle is an OPEN PR: it may be rebased or refactored in review, so the pin is the head SHA and the W1 tensor mapping is re-checked at merge. The on-disk dtype set is NOT runnable (Β§2.1) β€” an early revision of this spec asserted the opposite, and the correction is why the loader enforces `dtype(LM) == dtype(rvq) == dtype(cond)` and refuses violations by name. fp32 on the acoustic half is upstream's choice, mirrored, and sets a speed baseline in a regime this project has not optimised for β€” W7 is where that becomes interesting. The 5000-token prompt and 9000-frame ceilings are enforced, not discovered. Non-streaming is refused by name rather than buffered and called streaming. | + ## 5. Gates **LLM half β€” token-exact.** The global LLM and the depth decoder emit discrete RVQ @@ -295,7 +314,7 @@ against it with those off would be a dishonest denominator. --- -## 6. Phases +## 6. Phases (work breakdown) Each phase is dispatched to a **fresh implementer** from this spec, reviewed by a **fresh reviewer** who mutates the claimed guarantees, and its gate is rerun by @@ -353,20 +372,15 @@ pinned SHA. ## Now -W0 β€” spec committed, the diffusers oracle **stood up and running**, no engine -code. `.agents/oracles/diffusers.md` is `gateable = yes`: the oracle generated a -44100 Hz stereo waveform from a fixed seed, Β§1's geometry and dtypes were -re-derived from the loaded modules and agree, and the per-stage reference tensors -W3–W5 gate against are committed under -`tests/parity/goldens/minimax_music3_oracle/` with -[`tools/oracle/music3_oracle.py`](../../tools/oracle/music3_oracle.py) as the -reproducible recipe. Β§1.1 is confirmed at runtime, not merely from source: the -pipeline returns 44100 Hz stereo with no resample. - -Two things W0 hands forward rather than closes. The on-disk dtype set of Β§2.1 is -**not runnable through upstream's own pipeline** β€” the condition encoder and the -depth decoder consume the language model's hidden states uncast, so they must -share its dtype β€” which W1's loader has to mirror deliberately rather than -inherit by accident. And the capture ran on CPU, so it is a correctness -reference and no part of the speed axis. Still owed in W0: the SGLang-Omni -oracle record. W1 is unblocked. +**W0 DONE, W1 LANDED, row `ACTIVE`.** The diffusers oracle generates audio and is +`gateable = yes` against 13 committed per-stage goldens; both oracles are pinned; +Β§1.1 is resolved and confirmed at runtime; the modular loader is in the tree with +the dtype invariant Β§2.1 enforced and the native arm refused by name. + +W2/W3 (the autoregressive half) are dispatched. Then W4/W5 acoustic, W6 the +speech-family registration plus the `vllm_speech_*` ABI and the example HTTP +server, W7 the quantized arms. + +Two things are owed and neither is W0's to close: **no speed number exists** β€” the +oracle capture was CPU-only because `dgx.casa` was down, so nothing here touches +the speed axis β€” and SGLang-Omni remains `gateable = no`, read but never executed. diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md index 8ce7e32ae..b24c5da5d 100644 --- a/docs/BENCHMARKS.md +++ b/docs/BENCHMARKS.md @@ -436,6 +436,7 @@ built on it rather than keeping the flattering one. | vLLM 0.26 re-benchmark | Pending | Re-run the binding grids on the advanced pin | | MiniMax-H3 FP4 speed (W-FP4a) | **Measured GB10 (`row/H3-FP4-GPU-E2E`).** Marlin W4A16 byte-exact vs bf16; fp4 a memory win, 0.8x bf16/forward. Real-ckpt fp4-resident e2e RUNS (mp4/wav) | fp4 speed CLOSED. bf16-vs-quant A/B: ENCODER half MEASURED (Β§8.15), DiT half NOT (no bf16 render exists). Detail: benchmark-record + spec Β§8 | | LTX-2.5 axes | Speed `PENDING` (vllm-omni#6066 has no native 2.5), binding oracle too. **SIZE: 320x192/25f completes on GB10, 448x256 does not**; that render was REGISTER-conditioned, not prompted | Wall is the HOST VAE decode, not the pool: drain returns 0.11 GiB, byte-inert. 2 baselines UNRESOLVED (lock). A real-checkpoint PROMPTED render is OWED | +| MiniMax-Music3 (`MiniMaxMusic3ForConditionalGeneration`) | **Every axis `PENDING`; none owed yet.** W0/W1 are correctness scaffolding; no forward pass exists to time | Denominator when it does: SGLang-Omni `748a0b43` in its production configuration. First number owed at W6 | | MiniMax-H3 render coherence (`row/H3-RENDER-CLOSE` #77) | **CLOSED: a COHERENT scene on GB10.** #70/#74 white was wrong-PARTITION usage (t2va on the ref2va ckpt); t2va on the FL2VA GGUF renders a prompt-matched orange cat (adj-cos 0.95 vs 0.06, no patch-grid) | Verified first: t2va inputs byte-exact vs upstream; CUDA device==host at seq 1920. Follow-up `H3-TASK-PARTITION-GUARD`: the task/partition mismatch now RAISES 1:1 with `_resolve_task` (spec Β§8.6-8.7) | | MiniMax-H3 image conditioning (`row/H3-CONDITIONED-E2E`, `row/H3-VISION-SCATTER`, `row/H3-REF2VA-ASSEMBLY`) | **fl2va COHERENT; ref2va assembly bug FIXED+gated.** visionβ†’cond scatter gated; ref2va block-dim double-division fixed + RED-first gated (128 vs 512) + a permanent ref2va DiT-forward rung (Β§8.10) | grid RE-ATTRIBUTED: with the fix ref2va grids in fp4 AND bf16, and t2va with no refs on the ref2va NVFP4 also grids while FL2VA-GGUF renders, so it is the **NVFP4 checkpoint/loader**, NOT assembly/fp4 (Β§8.10) | | MiniMax-H3 Thor render speed (sm_110, no FA2) | **34.6 s/step** at 864x480/124f/50 steps on Q4_K_M, **16.6x** off 574.5 (render ~28 min, was ~8 h). Landed: warp-per-query, chunked warp reduce-scatter (1.76x), bf16 `mma.sync` (9.82x) | Shared-memory K/V tiling (23% SLOWER) and register Q-blocking (-0.8%) both measured and REVERTED: memory traffic is not the bound (one head's K+V is 3.9 MB against 32 MB of L2) | diff --git a/docs/FEATURES.md b/docs/FEATURES.md index 2feb062f2..39d635481 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -162,7 +162,7 @@ in `ltx2_text_encoder.cpp` is the call that would have to change. | Whisper audio encoder | openai/whisper-small; whisper-large-v3 (Voxtral cfg) | encoder tower 77/77; large-v3 tower 203/203 | pending | | MiniMax-H3 DiT (`MiniMaxH3DiTModel`, vllm-omni lane) | MiniMax-H3 (33.1B video+audio) | portable 79/79; all three modalities COHERENT on Q4_K_M (Β§8.20); PRUNED ckpts run, Q8_0 seam 0.9941 (Β§8.21); ref2va grid was NVFP4 quant error, Β§8.9 REFUTED; GGUF/NVFP4/bf16 shards stream | FP4/Marlin landed; speed pending; no bf16 render yet. Render from the Q4_K_M GGUF, not the NVFP4 arm. Krea 2 text-to-image (roadmap C11) is scoped to reuse these DiT seams | | LTX-2.5 DiT (`LTX2VideoTransformer3DModel`, Lightricks lane) | LTX-2.5 (21.00B video+audio) | `SPIKE`. DiT, VAEs+ENCODERS, conditioning, pipeline, quant loaders gated at reduced dims. Prompt-side AdaLN ported, host+device. Typed prompt to Gemma-4 to cross-attn, FIXTURE-gated. A prompted render is OWED | Family `ltx-2.5`, `ltx2-gen`. ~29 GB NVFP4/GB10, FP8 ~44 GB, +~24 GB tower. FP8/torchao/1st-party NVFP4 load; `keyframes_abs_pos_embedding` alone needs `allow_unported`. DiffVAE, LoRA, image cond refused. Speed PENDING | -| MiniMax-Music3 (`MiniMaxMusic3ForConditionalGeneration`, diffusers lane) | MiniMax-Music3 (8.6B Qwen3 LLM + 0.646B RVQ decoder + 2.4B fp32 DiT + DAC Flow-VAE); diffusers arm, ~28.5 GB | `SPIKE`. W1 LOADER ONLY: it loads, it does not generate. 1012 tensors accounted, vocoder weight-norm folded, native arm refused by name. On-disk dtypes are NOT the runtime set; the AR half must share one dtype | Not measured. The denominator will be SGLang-Omni in its production configuration (both CUDA graphs, compiled DIT and DAV, batched seeded sampling) | +| MiniMax-Music3 (`MiniMaxMusic3ForConditionalGeneration`, diffusers lane) | MiniMax-Music3 (8.6B Qwen3 LLM + 0.646B RVQ decoder + 2.4B fp32 DiT + DAC Flow-VAE); diffusers arm, ~28.5 GB | `ACTIVE`. Oracle gateable with 13 per-stage goldens (#708); modular loader gated 1413/1413 on the real 27 GB checkpoint, all 1012 tensors accounted (#714). Output 44100 Hz stereo. No stage runs yet: W2-W7 owed | Not measured. The denominator will be SGLang-Omni in its production configuration (both CUDA graphs, compiled DIT and DAV, batched seeded sampling) | | MTP speculator | Qwen3.6-27B, Qwen3.6-35B-A3B | token-identical to vLLM `mtp` at c1 | ~4% faster c1; +16% output tput (MoE) | | DFlash block-diffusion | Qwen3 (DFlash draft) | near-tie e2e 27/27 vs vLLM | 2.9x over spec-off, 1.003x vs vLLM DFlash-on | | DeepSeek-V4 MTP | DeepSeek-V4-Flash (nextn head) | lossless 5/5; real-model weight-blocked | pending | diff --git a/docs/STATUS.md b/docs/STATUS.md index 1b79a1dd3..ce2d935cd 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -140,6 +140,7 @@ token-for-token correctness against the pinned oracle. | InternLM2 dense (fused-`wqkv` interleaved split) | Correctness-complete, speed-pending | Token-exact 16/16 (internlm2-chat-1_8b): 12/16 strict + 4/16 bf16 near-tie (max gap 0.0 nats), 0 divergent; first InternLM model; ZERO new compute kernel (reuses the Llama dense forward; the only delta is a loader-side de-interleave of the fused `wqkv`, which packs q/k/v interleaved by KV-group) | | MiniMax-H3 (`MiniMaxH3DiTModel`, video+audio DIFFUSION) | **ABI v12 ONE SURFACE; device selector uses generic `DeviceType`; DSR 32.** t2va+fl2va COHERENT; bf16 shards STREAM | ref2va ckpt fidelity Β§8.12; encoder A/B Β§8.15; GB10 re-verify residual; CPU fold 6/137 (one queue + device provenance mutation-gated) | | LTX-2.5 (`LTX2VideoTransformer3DModel`, video+audio DIFFUSION) | **L1-L9c landed (#435).** 21.00B / 48 blocks. `VideoEngine` seam + ABI **v18**, DiT forward (CPU f32 parity, bf16 device-resident), Gemma-4 TE, both VAEs, the embeddings connector, pipeline, NVFP4/FP8 arms, `/v1/videos` | A shipped 21.00B FP8 DiT runs device-resident on GB10. The 320x192/25f frames ARE a scene, register-conditioned. L13 encodes a typed prompt, FIXTURE-gated; a prompted render is OWED. Speed and oracle parity `PENDING` | +| MiniMax-Music3 (`MiniMaxMusic3ForConditionalGeneration`, text-to-MUSIC) | **`ACTIVE`: W0 + W1 landed (#672).** First row whose oracle is not vLLM: the OPEN diffusers PR #14456 `c6da9936`, which generates audio here | W2-W7 owed. Loader gated 1413/1413 on the real checkpoint; no speed number exists or is claimed | | Command-R / Cohere dense (`CohereForCausalLM`) | Implemented, gate-blocked | ZERO-new-kernel port grounded in vLLM `commandr.py`: weight-only Cohere LayerNorm + GPT-J full-width RoPE + PARALLEL residual + `logit_scale` + tied embeddings, all reuse; compiles, links, self-registers. No SACRED gate yet (real checkpoints HF-gated, ungated ones tiny-random, GPU box disk-full); oracle run-verified at W0. See docs/BENCHMARKS.md | | Phi-1 / Phi-2 dense (`PhiForCausalLM`, parallel residual) | Correctness-complete, speed-pending | Token-exact 16/16 (microsoft/phi-2): 9/16 strict + 7/16 bf16 near-ties (max gap 0.25 nats), 0 forward-divergent; the OLDER Microsoft Phi arch, DISTINCT from Phi-3/Phi-4; ZERO new compute kernel (GPT-J parallel residual, LayerNorm-with-bias, biased qkv/dense, partial NeoX rope 32/80, non-gated NewGELU MLP reusing `vt::GeluTanh`, untied biased lm_head); F16 dtype-aware loader | | MiniCPM dense (`MiniCPMForCausalLM`, three scalars) | Correctness-complete, speed-pending | Token-exact 16/16 (openbmb/MiniCPM-2B-sft-bf16): 10/16 strict + 6/16 bf16 near-ties (max gap 0.0 nats), 0 forward-divergent; first OpenBMB MiniCPM model; ZERO new compute kernel (the Llama/Granite dense forward plus three scalars: scale_emb, scale_depth/sqrt(layers) residual, dim_model_base logit scaling), tied lm_head; `.bin`-only weights converted to safetensors via trusted torch |