diff --git a/.agents/issue-index.md b/.agents/issue-index.md index 8c12b5df8..e07c6ac6b 100644 --- a/.agents/issue-index.md +++ b/.agents/issue-index.md @@ -474,6 +474,8 @@ rather than merged. `scripts/check-agent-record.py` gates both. | [#1394](https://github.com/mudler/vllm.cpp/issues/1394) | `ENG-CUDAGRAPH-BREAK` | The CPU paged attention reads `btab[r * bt_row + (j / block_size) * bt_col]` for every `j < seq_lens[r]` without checking that the block table has that many columns, so a caller with a short table gets an out-of-bounds read, a plausible block index out of it, and attention over the WRONG page — silently. `tests/vllm/models/test_qwen3_5_decode_graph_seam.cpp`'s `SpecAttnMeta` supplies one: hardcoded `block_table_num_cols = 1` against shape C's `seq_lens = 24` at `block_size = 16`. Found while fixing [#1380](https://github.com/mudler/vllm.cpp/issues/1380), whose `DevicePool` change moved the bytes after the table and turned the same read into a SIGSEGV on `thor:gpu0` (`gdb` at `src/vt/cpu/cpu_paged_attn.cpp:224` under `FullAttnBlockPaged`). PRE-EXISTING: the case passes at `origin/main` only because the read landed on bytes that decoded to an in-range index. FIXED IN FLOW in both halves — the kernel refuses a short table with one compare per request outside the token loop, and the helper sizes its table for the sequence length it declares. Owned by row `ENG-CUDAGRAPH-BREAK` | bug | | [#1387](https://github.com/mudler/vllm.cpp/issues/1387) | `ENG-EXPERT-STREAM-DEVICE` | `docs/FEATURES.md`'s routed-expert-streaming row still read "CPU keep-quant towers only" after W0c made the seam take the slot arm on `is_cpu()` OR `host_memory_is_device_addressable()` and W0d made the load-time fit refusal drop those towers from its bound. `AGENTS.md` routes a feature-surface change to that page, and the change that moved the surface did not write it. `scripts/check-doc-checkpoint.py` said so, on commit `939755f99` of `row/ENG-EXPERT-STREAM-DEVICE-W0`: a measurement was appended to `.agents/benchmark-record.md` with no `docs/FEATURES.md` edit beside it. The PAGE is FIXED IN FLOW while repairing the fresh review of [#1377](https://github.com/mudler/vllm.cpp/pull/1377): the row now names both the device arm ([#1124](https://github.com/mudler/vllm.cpp/issues/1124)) and the residency condition ([#1378](https://github.com/mudler/vllm.cpp/issues/1378)), and the observability detail it displaced to stay inside the 220-character cell budget is stated at `docs/USAGE.md:4598-4620`. The GATE is NOT fixed and needs a decision: the checker walks a range one COMMIT at a time, so once a commit is published on a branch that may not be force-pushed, no later commit can make it green, and `scripts/agent-preflight.sh` keeps reporting `doc-checkpoint range` red on this branch until it merges. The squashed commit that lands on `main` carries both paths and passes. Changing the walk is checker semantics and needs its own row, spec and red-first evidence per `AGENTS.md` "Changing the rules or a checker", so it is not folded in here. Spec [`expert-stream-device-slots.md`](specs/expert-stream-device-slots.md) | bug | | [#1386](https://github.com/mudler/vllm.cpp/issues/1386) | — | `tools/bench/gpu_clock_state.py`'s `QUERY_FIELDS` collects nine fields and **none of them is thermal and none is electrical**, so the driver's own `SwThermalSlowdown` label can never be checked against a die reading on any window this helper has ever recorded. The measured consequence is that the nine windows of 2026-08-19 cannot distinguish a load transition from a thermal excursion. The concrete evidence is `clock-c1-r1.samples.json` in `/mnt/nas_share/rc/q38bf16/out/bench-20260819T035148Z/`: ours c1 r1 dips five times on the same period at the same `utilization.gpu = 96` — 48.83 s / 2177 MHz, 80.60 s / 2320 MHz, 109.28 s / 2210 MHz, 137.98 s / 2359 MHz, 166.07 s / 2268 MHz — and **two of those five carry `0x0000000000000000`**, no throttle bit at all (2210 and 2359), while three carry `0x20`. The 2210 MHz unlabelled dip is deeper than two of the three labelled ones, so the driver labels comparable excursions inconsistently and the bit alone cannot decide it. What would settle it: add `temperature.gpu` and `power.draw` to `QUERY_FIELDS`. That changes the clock-record schema, so it owes its own row and spec. Split out of [#1354](https://github.com/mudler/vllm.cpp/issues/1354) and owed under `## Owed` in [lease-clock-pinning.md](specs/lease-clock-pinning.md) | gap | +| [#1074](https://github.com/mudler/vllm.cpp/issues/1074) | `MODEL-TEXT-nemotron-h-nemotron-hfor-causal-lm` | The NemotronH model-matrix row described `main` as of 2026-08-12 and stayed `INVENTORIED` while A2-R, A2-P, A2-Q2a and the A3 driver landed on top of it. Reconciled against `main` at `b626be75a`, every claim re-verified rather than inherited: the non-gated `relu²` MoE EXISTS (`4d0c399e1`, `vt::MoeRelu2` called from `nemotron_h.cpp:354`), ModelOpt `MIXED_PRECISION` per-module loading EXISTS (`1bc5ef82c`), the MTP head is STILL OWED (W5, 270 tensors deferred by name), and the `KERNEL-SSM-MAMBA` block is FALSE — [#496](https://github.com/mudler/vllm.cpp/issues/496) W1 landed the host arm at `47960a009` and W2 the CUDA arm at `43a6c5518`, in `src/vt/cuda/cuda_mamba2_ssd.cuh`, a `.cuh` included by `cuda_gdn.cu` rather than a translation unit of its own, which is why a `src/vt/*mamba*` FILE GLOB reports absence; `nemotron_h.cpp:597` calls that op today. Row moved `INVENTORIED` -> `PARTIAL` with the rollup, the checklist entry and the projections it owes. `PARTIAL` and not `ACTIVE` deliberately: `check-agent-record.py` requires an `ACTIVE` row to name a `CLAIM-*` row a claim source carries, no file under `.agents/claims/` claims this row, and authoring one for another session's in-flight work would be a fabricated record. NO end-to-end token gate has passed and no throughput, latency or memory number is claimed; the A3 gate stays PENDING and [#1157](https://github.com/mudler/vllm.cpp/issues/1157) is the open decode divergence | bug | +| [#1080](https://github.com/mudler/vllm.cpp/issues/1080) | — | `scripts/check-doc-checkpoint.py:153` matches a spec's live-position section with `^##\s+Now\s*$`, and specs in this tree write it as `## N. Now`, so `spec_now_errors` reports "has no `## Now` section" about a section that is present and current. It fires only when a row moves lifecycle state. `nemotron-h-model.md` was one of them and is repaired in flow by [#1074](https://github.com/mudler/vllm.cpp/issues/1074), which is the change that made it the spec a moving row links. Re-measured at `b626be75a` after that repair: 15 specs still write the numbered spelling — `gate-audit-branch-evidence`, `ltx25-a2v-audio-input`, `ltx25-image-conditioning`, `ltx25-t2a-one-stage`, `ltx25-token-append`, `ltx2-device-staged-view-uaf`, `mamba2-ssd`, `nas-mount-path`, `nemotron-h-a2p-paged-forward`, `nemotron-h-a2q1-fp8-mamba`, `nemotron-h-a2q2b-realckpt-lmhead`, `nemotron-h-a2q2-nvfp4-moe-lmhead`, `nemotron-h-abi-e2e`, `offload-docs-refusal`, `registry-downcast-sweep`. The population GREW from the twelve this was filed against, which is the argument for the checker-semantics fix over a rename sweep. Either close needs its own spec and a red-before test, so neither rides in a records reconcile. `tests/scripts/` covers `NOW_SECTION` nowhere, which is how the mismatch survived. Listed under `## Owed` in [`nemotron-h-model.md`](specs/nemotron-h-model.md) | bug | | [#1410](https://github.com/mudler/vllm.cpp/issues/1410) | `MODEL-TEXT-nemotron-h-nemotron-hfor-causal-lm` | `scripts/check-runner-routing-consistency.py` resolves the registry hook's delegate across translation units only for the `Class::ForwardDevice` shape (`_DELEGATE`, `:142-145`) or a helper defined in the registry TU itself (`classify_with_helpers`, `:246-271`). NemotronH's production device forward is a FREE function in another TU — `ForwardNemotronHForCausalLM` (`nemotron_h_registry.cpp`) calls `NemotronHPagedForward` (`nemotron_h_device.cpp`) — so the hop finds nothing and the hook's own host-reference fall-through, which A2-P deliberately keeps below the paged fold as the numeric gate's operand, classifies the model HOST. Measured on `row/A2-Q2b-lmhead-nvfp4` after A2-Q2b put `lm_head` on the device: `NemotronHPagedForward` assigns both `fl.device_tensor` and `fl.device_storage = dlogits.ReleaseShared()`, which IS `_DEVICE_SEAM` (`:125-128`), and the checker still names the model. A FALSE RED, so the safe direction — but it holds an allowlist entry open for a clause that is MET, and the allowlist is what a reader trusts to know what is still unrouted; it is latent the other way for any future model whose device logits come from a cross-TU free function. The checker already builds the `free_fn_file` map the fix needs (invariant (b) uses it). NOT fixed in flow: it CHANGES CHECKER SEMANTICS, which `AGENTS.md` `## Changing the rules or a checker` routes to the normal row, spec and fresh-review path, and widening a classification to turn a red gate green is exactly the move that section slows down — it needs its own red-before in `tests/scripts/test_check_runner_routing_consistency.py`. Owned by row `MODEL-TEXT-nemotron-h-nemotron-hfor-causal-lm`, listed under `## 5. Owed` in [`nemotron-h-a2q2b-realckpt-lmhead.md`](specs/nemotron-h-a2q2b-realckpt-lmhead.md) | bug | | [#1421](https://github.com/mudler/vllm.cpp/issues/1421) | `ENG-RECORD-ANCHOR-RATCHET` | Two NVFP4 W4A16 upstream anchors are WRONG AT THE PIN `5559679229bc961848b121ccdeaa8fa5d79bec98`, verified against a local checkout of `vllm-project/vllm` at that exact commit (`origin` is upstream, not a fork). The quoted `elif linear_backend == "auto" and use_a16: force_kernel = MarlinNvFp4LinearKernel` is at `kernels/linear/__init__.py:922-924`, not `:879-881`; `:879-881` is the tail of `init_wfp8_a16_linear_kernel` (`:848`), an FP8 helper sharing the `a16` vocabulary, and `init_nvfp4_linear_kernel` does not start until `:883`. And `packed_modules_mapping` is at `models/qwen3.py:275-278`, not `:271-274`, which is the `class Qwen3ForCausalLM(...)` line. Both are the worst shape of anchor rot: the cited line EXISTS and holds plausible neighbouring code, so a reader who checks the anchor is confirmed in a wrong belief. Nine code and test sites over SEVEN files (`dense_nvfp4_gemm.h:18,627,783`, `nvfp4.h:10`, `gguf_keep_quant.h:201`, `dense_weight_loaders.h:674`, `minimax_h3_device.cpp:153`, `test_qwen3_32b_nvfp4a16_paged_engine.cpp:63`, `test_qwen3_32b_nvfp4a16_load.cpp:19` -- count the parenthetical, the earlier `six files` did not) plus EIGHT record sites over FIVE record files, `parity-ledger.md:599,915,916` carrying three of them and `quantization-matrix.md:124`, `specs/sweep-qwen3-32b-nvfp4a16.md:86,465`, `specs/minimax-h3.md:332` and `completed/state-events/2026-08/STATE-20260806T141900-001.md:14` the rest. PRE-EXISTING and NOT fixed in flow: it is not small or confined, it spans paths several rows own, it would turn `parity-ledger.md` into a lock, and it needs the whole citation set RE-DERIVED at the pin rather than a sed -- range-checking cannot find it, because every one of these is in range. Found while repairing fresh-review findings on [#1415](https://github.com/mudler/vllm.cpp/pull/1415) (A2-Q2b, [#810](https://github.com/mudler/vllm.cpp/issues/810)), which relocates the `:783` citation from `:725` and deliberately leaves its VALUE unchanged so this issue owns every site uniformly instead of half of them looking reviewed. Owned by row `ENG-RECORD-ANCHOR-RATCHET`, the anchor-rot ratchet row | bug | | [#1327](https://github.com/mudler/vllm.cpp/issues/1327) | `SPEC-DFLASH2` | `.agents/specs/dflash2-spec-decode.md` `## Upstream chain` said the three output scalars `input_embedding_scale`, `output_multiplier` and `final_logit_softcapping` are "ABSENT from this config" and that "no published checkpoint exercises them, so the port implements them and gates them synthetically". That was measured on `z-lab/Qwen3.8-27B-DFlash2` alone. `z-lab/Muse-Glimmer-30B-DFlash2` — the SECOND published DFlash2 checkpoint, `config.json` sha256 `cb684d6f688a22619a63ea1debe7d30c139c195bf3141fd86a763763ab34b5d9`, read 2026-08-19 — sets `output_multiplier` to `0.19611613513818404` and `final_logit_softcapping` to `20.0`, and ships `block_size` 16 against the 27B's 8, hidden 6656 (416 groups, a 1664-wide `kernel_projection`) and `rope_theta` 500000.0. Both scalars are applied to the candidate VALUES in `compute_candidates` BEFORE the selector scores them, so a wrong value reorders the top-K and moves acceptance without raising — the `is_causal` failure class one layer up, and the class no token gate here can see. A port reading all three with `.get(key, default)` would pass every gate built from the 27B draft and be measuring the default path. The same file also falsifies `## Scope`'s exclusion of "a second DFlash2 target family": upstream registers ONE architecture and both checkpoints declare `model_type` `qwen3`, so what the second adds is values rather than a class. FIXED IN FLOW by `SPEC-DFLASH2` W2, which is the wave that had to read both configs anyway: `## Scope` drops the exclusion, `## Upstream chain` records both values with their source, `## Gates` G1 now requires BOTH published block shapes (upstream's own reference test parametrises 5 and 8 and never reaches 16), and `## Risks/decisions` D9 binds W3 to gate the scalars against the checkpoint that sets them | gap | diff --git a/.agents/model-matrix.md b/.agents/model-matrix.md index 06e6fdcb7..0ef1a8f67 100644 --- a/.agents/model-matrix.md +++ b/.agents/model-matrix.md @@ -282,7 +282,7 @@ Transformers compatibility is capability-driven and excluded from finite counts. | `MODEL-TEXT-mimo-v2-mi-mo-v2-flash-for-causal-lm` | `MiMoV2FlashForCausalLM` | `registry.py:166`; `vllm/model_executor/models/mimo_v2.py::MiMoV2FlashForCausalLM` | causal generation / text | model loader/forward; paged attention/KV; FusedMoE/grouped GEMM; sliding-window attention | ☐ required | `INVENTORIED` | none | unassigned | | `MODEL-TEXT-mimo-v2-mi-mo-v2-for-causal-lm` | `MiMoV2ForCausalLM` | `registry.py:167`; `vllm/model_executor/models/mimo_v2.py::MiMoV2ForCausalLM` | causal generation / text | model loader/forward; paged attention/KV; FusedMoE/grouped GEMM; sliding-window attention | ☐ required | `INVENTORIED` | none | unassigned | | `MODEL-TEXT-nemotron-nemotron-for-causal-lm` | `NemotronForCausalLM` | `registry.py:168`; `vllm/model_executor/models/nemotron.py::NemotronForCausalLM` | causal generation / text | model loader/forward; paged attention/KV | ☐ required | `INVENTORIED` | none | unassigned | -| `MODEL-TEXT-nemotron-h-nemotron-hfor-causal-lm` | `NemotronHForCausalLM`, `NemotronHPuzzleForCausalLM` | `registry.py:169-170`; `vllm/model_executor/models/nemotron_h.py::NemotronHForCausalLM` | causal generation / text | model loader/forward; paged attention/KV; FusedMoE/grouped GEMM; Mamba/SSM state; sliding-window attention. **2026-08-12: row spec committed ([#517](https://github.com/mudler/vllm.cpp/issues/517), [spec](specs/nemotron-h-model.md)); BLOCKED on `KERNEL-SSM-MAMBA` [#496](https://github.com/mudler/vllm.cpp/issues/496)** ([spec](specs/mamba2-ssd.md)) — the Mamba2 SSD core is unported. Beyond that kernel this arch owes three further things, none of which exist locally: non-gated `relu²` MoE (every grouped-MoE op we have is SwiGLU-shaped), ModelOpt `MIXED_PRECISION` per-module loading (`quantization/modelopt.py:2280` — NVFP4 W4A16 g16 experts and FP8 W8A8 mamba projections in one checkpoint), and the DeepSeek-style MTP head. Driver checkpoint `nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4` (20.1 GiB, fits one GB10). **2026-08-17: A2-P lands the PAGED forward** ([#810](https://github.com/mudler/vllm.cpp/issues/810), [spec](specs/nemotron-h-a2p-paged-forward.md)): `ForwardNemotronHForCausalLM` selects `NemotronHPagedForward` whenever the runner supplies paged KV and recurrent state, so K/V go into the runner's pages at `attn_meta.slot_mapping` and the conv + SSM rows are gathered from and scattered to `GdnStateCache` at the metadata's state indices — the first time this architecture carries state between decode steps. G-SAFE narrows from three clauses to `num_reqs <= 1` (batching is A2-B). The row stays `INVENTORIED`: the A3 end-to-end token gate against the released checkpoint is PENDING on `dgx.casa`, and `lm_head` (A2-Q2b), the FP8 mamba projections (A2-Q1, [#940](https://github.com/mudler/vllm.cpp/issues/940)), MTP (W5) and GGUF (W7) are all still owed | ☐ required | `INVENTORIED` | none | unassigned | +| `MODEL-TEXT-nemotron-h-nemotron-hfor-causal-lm` | `NemotronHForCausalLM`, `NemotronHPuzzleForCausalLM` | `registry.py:169-170`; `vllm/model_executor/models/nemotron_h.py::NemotronHForCausalLM` | causal generation / text | model loader/forward; paged attention/KV; FusedMoE/grouped GEMM; Mamba/SSM state; sliding-window attention. Driver checkpoint `nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4` (20.1 GiB, fits one GB10). **The `KERNEL-SSM-MAMBA` block this cell carried from 2026-08-12 is FALSE, and a file glob is why it survived six days.** [#496](https://github.com/mudler/vllm.cpp/issues/496) landed the host arm at `47960a009` (W1) and the CUDA arm at `43a6c5518` (W2). The kernel is [cuda_mamba2_ssd.cuh](../src/vt/cuda/cuda_mamba2_ssd.cuh) — a 692-line `.cuh` INCLUDED at [cuda_gdn.cu:48](../src/vt/cuda/cuda_gdn.cu#L48) and registered as `kMamba2ChunkScan` at [cuda_gdn.cu:6669](../src/vt/cuda/cuda_gdn.cu#L6669), not a translation unit of its own — so a `src/vt/*mamba*` FILE GLOB finds nothing and reads as absence. That glob is the search this cell and [#1074](https://github.com/mudler/vllm.cpp/issues/1074) both rested on, and `src/vt/ops.cpp::Mamba2ChunkScan` plus [nemotron_h.cpp:597](../src/vllm/model_executor/models/nemotron_h.cpp#L597) refute it: this model calls the op today. [#496](https://github.com/mudler/vllm.cpp/issues/496) stays open for its GENERIC `MambaSpec` producer, which this architecture does not need, because A1 made the runner read the model's OWN KV spec. **Two of the three things this cell called nonexistent now exist.** The non-gated `relu²` MoE landed at `4d0c399e1` — `src/vt/ops.cpp::MoeRelu2`, called from [nemotron_h.cpp:354](../src/vllm/model_executor/models/nemotron_h.cpp#L354) — and ModelOpt `MIXED_PRECISION` per-module loading landed at `1bc5ef82c` ([modelopt_mixed_precision.h](../src/vllm/model_executor/layers/quantization/modelopt_mixed_precision.h)). The third, the DeepSeek-style MTP head, is genuinely OWED as W5: the loader DEFERS its 270 tensors BY NAME at [nemotron_h_weights.cpp:1123](../src/vllm/model_executor/models/nemotron_h_weights.cpp#L1123) | [nemotron-h-model](specs/nemotron-h-model.md) (row spec, W1-W7); [nemotron-h-abi-e2e](specs/nemotron-h-abi-e2e.md) (A1/A2/A3); [nemotron-h-a2p-paged-forward](specs/nemotron-h-a2p-paged-forward.md); [nemotron-h-a2q1-fp8-mamba](specs/nemotron-h-a2q1-fp8-mamba.md); [nemotron-h-a2q2-nvfp4-moe-lmhead](specs/nemotron-h-a2q2-nvfp4-moe-lmhead.md); [nemotron-h-a2q2b-realckpt-lmhead](specs/nemotron-h-a2q2b-realckpt-lmhead.md); [mamba2-ssd](specs/mamba2-ssd.md) | `INVENTORIED` | **`INVENTORIED` still, because the END-TO-END TOKEN GATE HAS NOT BEEN RUN AGAINST `main` — this cell claims nothing beyond that.** The code and its unit gates below are real, and they are what a state move will rest on once a gate result measured on `main` exists. Code: the hybrid host forward [nemotron_h.cpp:597](../src/vllm/model_executor/models/nemotron_h.cpp#L597) (`vt::Mamba2ChunkScan`) and [nemotron_h.cpp:354](../src/vllm/model_executor/models/nemotron_h.cpp#L354) (`vt::MoeRelu2`); the PAGED forward [nemotron_h_device.cpp:1474](../src/vllm/model_executor/models/nemotron_h_device.cpp#L1474) (`NemotronHPagedForward`, A2-P `a6df72777`), which puts K/V into the runner's pages at `attn_meta.slot_mapping` and gathers and scatters the conv and SSM rows at the metadata's state indices — the first time this architecture carries state between decode steps; the device MoE arm (A2-Q2a, 23 blocks, NVFP4 g16 Marlin) and the A2-R dense arm (embeddings, 52 norms, 6 GQA blocks); the weight loader [nemotron_h_weights.cpp:1123](../src/vllm/model_executor/models/nemotron_h_weights.cpp#L1123) (18487/18487 accounted, 18217 materialized in their SHIPPED formats, 270 MTP deferred by name); and the ABI driver [examples/nemotron_h_gen/main.cpp](../examples/nemotron_h_gen/main.cpp) (A3 `c83b96934`), a thin `include/vllm.h` client. Tests: [test_nemotron_h_paged_forward.cpp:595](../tests/vllm/models/test_nemotron_h_paged_forward.cpp#L595) (a multi-step runner decode matches the host reference token for token) and `:685` (the recurrent pages carry state across steps and are INDEXED); [test_nemotron_h_forward.cpp:990](../tests/vllm/models/test_nemotron_h_forward.cpp#L990) (the Mamba2 mixer vs an independent sequential recurrence) and `:1190` (the MoE vs an independent non-gated relu² reference); [test_nemotron_h_scaffold.cpp:204](../tests/vllm/models/test_nemotron_h_scaffold.cpp#L204) (the REAL released `config.json` parses); [test_nemotron_h_loader.cpp:150](../tests/vllm/models/test_nemotron_h_loader.cpp#L150) (the REAL checkpoint loads and the forward produces logits); [test_nemotron_h_quantized_forms.cpp:297](../tests/vllm/models/test_nemotron_h_quantized_forms.cpp#L297) (a shipped quantized weight is REFUSED, never silently dequantized); [test_nemotron_h_moe_device.cpp:221](../tests/vllm/models/test_nemotron_h_moe_device.cpp#L221) (A2-Q2a vs the host reference); [test_ops_mamba2_ssd.cpp:470](../tests/vt/test_ops_mamba2_ssd.cpp#L470) (chunk-size invariance). **GATED END TO END ON A MEASURED TREE, NOT ON `main` TODAY, and no number exists.** The A3 gate against the committed goldens `tests/parity/goldens/nemotron_35_lightning_greedy` **PASSES on GB10: `TOKEN MATCH: 96/96 over 3 prompt(s) (full rows=3, short rows=0, mode=decode)`, `STRICT PASS`**, against the pinned oracle `vllm=0.23.1rc1.dev1511+g555967922` on `nemotron-3.5-lightning-30b-nvfp4` revision `29f2d1746d8f41e316523194b19018707749b1b1`. It is the DEVICE leg and not the host queue: the binary is `libvllm 0.0.3+cuda`, `cfg.log` records `fp4-mma`, `cutlass-nvfp4` and `cutlass-fp8` `ENABLED for [121a]`, and the run logs `Asynchronous scheduling is enabled (max_concurrent_batches=2)` — the path on which `ModelForwardInput::device_token_ids` is non-null, and the only one on which the [#1157](https://github.com/mudler/vllm.cpp/issues/1157) defect can occur. The proof is the DELTA, not the pass: the same binary and the same checkpoint with ONLY `nemotron_h_device.cpp` reverted to the fix's parent scores `4/24 (full rows=0, short rows=3)` and bails at 8 generated tokens. Evidence `/usr/local/nas_share/rc/nh1157/` — `gate_fixed.out`, `gate_red.out`, `cfg.log`, `build.log`. **That pass was measured on the tree that landed as `0ea5d249f` ([#1221](https://github.com/mudler/vllm.cpp/pull/1221), MERGED 2026-08-18), which is now `main`'s last touch of `nemotron_h_device.cpp`. The fix is therefore on `main` and nothing is pending a merge; but `main` has advanced many commits since that tree, NO run against current `main` exists, and this cell does not claim one.** One config caveat travels with the run: `--gpu-memory-utilization 0.92` did NOT size the KV pool, which fell back to 256 blocks ([#83](https://github.com/mudler/vllm.cpp/issues/83)). The step still refuses batched decode by name at [nemotron_h_registry.cpp:157](../src/vllm/model_executor/models/nemotron_h_registry.cpp#L157) (`input.num_reqs <= 1`, A2-B owns it), and `scripts/runner-routing-allowlist.txt:26` still carries `nemotron_h` because `lm_head` is NVFP4 g16 and refuses on a non-CPU queue at [nemotron_h.cpp:1032](../src/vllm/model_executor/models/nemotron_h.cpp#L1032), so the last step is the HOST projection. **OWED:** A2-Q2b the device `lm_head`, A2-Q1 the 46 FP8 W8A8 mamba projections ([#940](https://github.com/mudler/vllm.cpp/issues/940)), A2-B batched decode, W5 the MTP head, W7 the GGUF k-quant arm, and a RE-RUN of the A3/S5 gate against `main`, which is now the only thing that can move this row. **No throughput, latency or memory number exists for this architecture and none is claimed.** The wall times in `gate_fixed.out` (264.4s load, 327-343s per 32-token prompt) are a CORRECTNESS run whose `lm_head` and 46 FP8 mamba projections still execute host-side; they are not benchmarks and nothing may read them as one. Rows: [#517](https://github.com/mudler/vllm.cpp/issues/517) (model), [#810](https://github.com/mudler/vllm.cpp/issues/810) (ABI e2e); record reconciled by [#1074](https://github.com/mudler/vllm.cpp/issues/1074) | unassigned | | `MODEL-TEXT-olmo-olmo-for-causal-lm` | `OlmoForCausalLM` | `registry.py:171`; `vllm/model_executor/models/olmo.py::OlmoForCausalLM` | causal generation / text | model loader/forward; paged attention/KV | ☐ required | `INVENTORIED` | none | unassigned | | `MODEL-TEXT-olmo2-olmo2-for-causal-lm` | `Olmo2ForCausalLM`, `Olmo3ForCausalLM` | `registry.py:172-173`; `vllm/model_executor/models/olmo2.py::Olmo2ForCausalLM` | causal generation / text | OLMo-2-0425-1B dense LANDED (W0-W4). Impl: `include/vllm/model_executor/models/olmo2.h` + `src/vllm/model_executor/models/{olmo2,olmo2_weights,olmo2_registry}.cpp` (TWO `REGISTER_VLLM_MODEL` for `Olmo2ForCausalLM`+`Olmo3ForCausalLM`, one shared factory; reuses the shared dense glue). ZERO new compute kernel — the two "distinctives" both reduce to WIRING over landed ops: (1) PURE POST-NORM (`norm_after`, `olmo2.py:261-277`) = standalone `vt::RmsNorm` on each sublayer OUTPUT + plain `vt::Add` residual re-join, NO pre-norm (the GLM-4 standalone-output-norm op, pre-norms dropped); (2) FULL-WIDTH QK-norm (`olmo2.py:113-117,160-172`) = two standalone `vt::RmsNorm` over `[T,q_size]`/`[T,kv_size]` before NeoX rope (NOT the per-head `kAttnQkNormRope`). Reuse: SiLU SwiGLU (`kSiluAndMul`), NeoX RoPE (`RopeFromCache`/`RopeNeox`), GQA/MHA paged glue (`dense_attn_block.h`), merged qkv/gate_up loader. Checkpoint is F32 on-disk → loader downcasts f32→bf16 (round-to-nearest-even, `vt::F32ToBF16`) to match vLLM-bf16. UNTIED lm_head (`tie_word_embeddings:false`). Loader 179 tensors, zero missing/unmapped. Tokenizer: ONE shared-TU touch — `tokenizer.cpp` accepts the OLMo-2 Split `behavior=Removed,invert=true` encoding (equiv. to `Isolated` for the full-cover cl100k regex → `kLlama3`); guarded additive branch, no other checkpoint uses it. NO-BOS-verified real ByteLevel gate. Runs EAGER (bf16, no decode graph). Gate: 16/16 vs vLLM 0.25.0 (STRICT 13/16 + near-tie 3/16, max gap 0.094 nats, 0 forward-divergent) — speed pending. **OLMo-3 W5 (batch3) IMPLEMENTED** (guarded additive edits to `olmo2.{h,cpp,weights}`, diff-inert for OLMo-2 — re-run OLMo-2 gate 16/16 UNCHANGED): per-layer interleaved routing off `config.layer_types` — sliding_attention layers use plain NeoX rope (theta 500000) + finite window (masked at the FA kernel, inert for short contexts), full_attention layers use a precomputed YaRN cos/sin cache (get_rope yarn: factor 8, original 8192, mscale=yarn_get_mscale(8)=1.2079=config attention_factor, indexed by real positions); dtype-aware loader (OLMo-3 is BF16 on-disk vs OLMo-2 F32). **BUT the pinned vLLM 0.25.0 oracle CANNOT run `allenai/OLMo-3-1025-7B`** — its transformers version predates OLMo-3's nested per-layer-type rope schema: `olmo2.py:143` does `rope_parameters["rope_theta"]` → `KeyError: 'rope_theta'` (top-level rope_theta not folded into standardized rope_parameters), and forcing it surfaces `Unrecognized keys {'sliding_attention','full_attention'}` → `TypeError: unhashable type: 'dict'`. So there is NO pinned-oracle SACRED bar for OLMo-3 (DEP-blocked, spec D5); our engine LOADS + RUNS it (bootstrap loaded clean). W5 SACRED gate pending an oracle that can construct OLMo-3's rope config. | ✅ [sweep-olmo2](specs/sweep-olmo2.md) | `PARTIAL` | `test_olmo2_paged_engine` 16/16 (dgx, 92 assertions, batch3 re-run UNCHANGED); `test_olmo3_paged_engine` present (oracle-blocked, skips); loader 179 tensors zero-unmapped; registry resolves both arch strings; registration `src/vllm/model_executor/models/olmo2_registry.cpp:126`; test `tests/vllm/models/test_model_registry.cpp:118` | `CLAIM-SWEEP-OLMO2` (Claude Code opus-4-8) | | `MODEL-TEXT-olmo-hybrid-olmo-hybrid-for-causal-lm` | `OlmoHybridForCausalLM` | `registry.py:174`; `vllm/model_executor/models/olmo_hybrid.py::OlmoHybridForCausalLM` | causal generation / text | model loader/forward; paged attention/KV; Mamba/SSM state; GDN/linear-attention state | ☐ required | `INVENTORIED` | none | unassigned | diff --git a/.agents/specs/nemotron-h-model.md b/.agents/specs/nemotron-h-model.md index b5723fa23..98b02f000 100644 --- a/.agents/specs/nemotron-h-model.md +++ b/.agents/specs/nemotron-h-model.md @@ -1972,61 +1972,89 @@ inside a red-lane repair would have made the repair unreviewable. This follows how the unaligned-read class was handled — named site in #674/PR #688, residue filed as #772 — rather than widening silently. -## 7. Now - -**State at this commit:** **W1 and W3 have LANDED on `main`; W2 is in -re-review.** The `MIXED_PRECISION` resolver landed at `1bc5ef82c` (#561) and the -W3 scaffold at `c6b240edd` (#576); both are merged into this branch, and their -spec sections (§4's three W1 subsections, §5c/§5d/§5e) are main's, carried here -byte for byte. The Mamba2 SSD kernel work W1 landed earlier at `47960a009` -(#496), so `include/vt/ops.h` carries main's -`kMamba2ChunkScan`/`kMamba2StateUpdate`/`kRmsNormGatedGroup` first and appends -`kMoeRelu2` after them; no existing op id shifted. W2 (the non-gated `relu²` -expert, §6a) was reviewed PASS at `e2d68404`, repaired for that review's six -findings at `dd7a6477d`, and this branch is its land-prep: re-merged onto -`origin/main` and fully re-gated. A second fresh review (PR #586 @ `f6a7f8709`) -returned **PASS** — it established that the repair delta changes zero executable -lines — with four RECORD findings, all repaired on -`row/MODEL-NEMOTRON-H-W2-RECORDS`: two stale `activation.py:33` anchors -(`include/vt/ops.h`, this spec), M7's assertion count (§6a), §4's W2 seam text -contradicting §6a, and an overstated bit-identity comment in `cuda_moe.cu` -(#591). That repair touches comments and this spec only; no executable line -moved. `tests/vt/test_ops_moe_nongated_relu2.cpp:12` already carried the -corrected anchors and needed no change. - -The row stays `INVENTORIED`; this commit changes no lifecycle state, so it owes -no `STATUS`/`BENCHMARKS` write. **Oracle gateability is CLOSED** — §5a records -the pinned oracle loading and running the checkpoint on GB10 with three greedy -goldens committed, so W6 has a denominator whenever it is reached. - -**W4 has LANDED** (`ce8c8bf67`, #718), and the WEIGHT LOADER §7 named as "the -next brick" is built and gated on `row/MODEL-NEMOTRON-H-LOADER` — **§6d is the -authority on it**. The forward no longer refuses on a checkpoint load: 18487 of -18487 tensors are accounted (18217 materialized in their shipped formats, 270 -deferred by name to W5), the real 20.1 GiB checkpoint runs at **17.70 GiB peak -RSS**, and its first greedy token matches the pinned oracle's committed golden on -**3 of 3** prompts. - -**Next action:** the loader needs a **FRESH REVIEW** — never the agent that wrote -it. The two claims it changes rather than adds, and which a review should mutate, -are (a) `NemotronHOwned::View`'s refusal of a non-dense weight, which is the only -thing standing between a packed NVFP4 buffer and a plausible-garbage GEMM operand, -and (b) the expert-major reorder in `NemotronHMoeMixer`, whose result-neutrality -is claimed from the disjointness of the output slots rather than measured. - -Then W5 (the MTP head, whose 270 tensors the loader already names as owed), W6 -(the e2e token gate against the committed goldens, now unblocked — it has weights), -and W7 (GGUF). Carried forward, not resolved: the two OWED GPU items in §6a -(`kMoeGroupedGemmNvfp4Marlin` on the real g16 tensors, and the end-to-end -NemotronH MoE block on GB10) — the loader now produces exactly those g16 tensors, -so the first of them is reachable — and the OWED GGUF k-quant arm (§5b). - -**Reported, outside this task's authority to fix.** `test_op_parity` is RED on -this row's base for a reason belonging to MODEL-MUSIC-MUSIC3 (#672); §6d's gate -evidence has the diagnosis. - -The row stays `INVENTORIED`: the loader changes no lifecycle state, because the -forward is still the HOST reference and nothing runs on the paged runner (W6). +## Now + +**State at this commit: the row STAYS `INVENTORIED`, and what changes is that +its text stops being false.** Nothing in this change touches `src/`, `include/`, +`tests/` or `scripts/`. The matrix row still described `main` as of 2026-08-12, +having gone untouched while A2-R (`598226e96`), A2-P (`a6df72777`), A2-Q2a and +the A3 driver (`c83b96934`) all landed on top of it. This reconcile corrects +that description and records the A3 measurement against the tree it was taken +on. It deliberately does NOT move the lifecycle state: see the gate paragraph +below, which is why the move is not this change's to make. + +**The `KERNEL-SSM-MAMBA` block the row carried is FALSE and was measured so.** +[#496](https://github.com/mudler/vllm.cpp/issues/496) landed its host arm at +`47960a009` and its CUDA arm at `43a6c5518`. The kernel is +`src/vt/cuda/cuda_mamba2_ssd.cuh`, a 692-line header included at +`cuda_gdn.cu:48` and registered as `kMamba2ChunkScan` at `cuda_gdn.cu:6669` +rather than a translation unit of its own, so a `src/vt/*mamba*` FILE GLOB finds +nothing and reads as absence. That glob is the search the row's text rested on. +`src/vllm/model_executor/models/nemotron_h.cpp::NemotronHMamba2Mixer` calls +`vt::Mamba2ChunkScan` today. Of the three things the row said "exist nowhere +locally", the non-gated `relu²` MoE (`4d0c399e1`) and ModelOpt +`MIXED_PRECISION` loading (`1bc5ef82c`) both exist; only the MTP head (W5) is +still genuinely owed. + +**Why no state move rides here.** The earlier draft of this section moved the +row to `PARTIAL` on the strength of the A3 gate. That argument rested on the +gate's pass being a property of a branch that was about to land, and the premise +expired when it landed: the pass belongs to the tree measured, not to `main` +today. A state move needs a gate result measured on `main`, and none exists. Two +things remain true and are recorded rather than acted on: no file under +`.agents/claims/` claims `MODEL-TEXT-nemotron-h-nemotron-hfor-causal-lm`, so +`scripts/check-agent-record.py` would refuse `ACTIVE` regardless; and the row's +code and test anchors are now exact, so whoever re-runs the gate has the +contract `PARTIAL` requires already written down. + +**The end-to-end token gate PASSES, and it passes on a branch that is not +`main`.** On GB10 the A3 96-token gate reads `TOKEN MATCH: 96/96 over 3 +prompt(s) (full rows=3, short rows=0, mode=decode)` and `STRICT PASS`, against +the pinned oracle `vllm=0.23.1rc1.dev1511+g555967922` on +`nemotron-3.5-lightning-30b-nvfp4` at revision +`29f2d1746d8f41e316523194b19018707749b1b1`. + +It is the DEVICE leg. Three things say so, and the third is the one that +matters. The binary is `libvllm 0.0.3+cuda`, and `cfg.log` records `fp4-mma`, +`cutlass-nvfp4` and `cutlass-fp8` as `ENABLED for [121a]`. The run logs +`Asynchronous scheduling is enabled (max_concurrent_batches=2)`, which is +precisely the path where `ModelForwardInput::device_token_ids` is non-null; on +the host queue it is always null and the [#1157](https://github.com/mudler/vllm.cpp/issues/1157) +defect cannot occur at all. And the same binary on the same checkpoint, with +ONLY `nemotron_h_device.cpp` reverted to the fix's parent, scores `4/24 (full +rows=0, short rows=3)` and bails at 8 generated tokens. A host-leg run would +have been unmoved by that revert. **The delta is the proof; the pass on its own +is not.** Evidence: `/usr/local/nas_share/rc/nh1157/` — `gate_fixed.out`, +`gate_red.out`, `cfg.log`, `build.log`. + +**The pass belongs to the tree it was measured on, which is now on `main` -- +and that is still not a pass on `main`.** +[#1221](https://github.com/mudler/vllm.cpp/pull/1221) MERGED on 2026-08-18 as +`0ea5d249f`, which is `main`'s last touch of `nemotron_h_device.cpp`, so the +`device_token_ids` repair the 96/96 depended on is no longer pending. The 96/96 +was taken on that branch tree, `main` has advanced many commits since, and no +run against current `main` exists. Naming the SHA a measurement belongs to is +the point: an evidence line that names a tree it was not measured on has cost +this repository before. This spec therefore records a measured result and its +tree, and it does NOT record `main` as gated. One config caveat travels with the +run: `--gpu-memory-utilization 0.92` did not size the KV pool, which fell back +to 256 blocks ([#83](https://github.com/mudler/vllm.cpp/issues/83)). + +**No throughput, latency or memory number exists for this architecture and none +is claimed.** The wall times in `gate_fixed.out` — 264.4s to load, 327-343s per +32-token prompt — are a correctness run on a path whose `lm_head` and 46 FP8 +mamba projections still execute host-side. They are not benchmarks, they are not +a denominator, and nothing may carry them into `docs/BENCHMARKS.md` as a +performance figure. + +**Next action:** re-run the A3 gate against `main`. #1221 has landed, so +nothing is blocked on a merge any more; what is missing is a gate result +measured on a `main` tree, which is the only thing that can move this row off +`INVENTORIED`. Then W5 (the MTP head, whose 270 tensors the loader already names +as owed) and W7 (GGUF k-quants). A2-Q2b (the device `lm_head`) is what removes +`nemotron_h` from `scripts/runner-routing-allowlist.txt`; A2-B is what removes +the `input.num_reqs <= 1` refusal. + ## 8. Stop conditions @@ -2055,3 +2083,44 @@ forward is still the HOST reference and nothing runs on the paged runner (W6). The entry stays rather than being deleted because #847's row in the append-only `.agents/issue-index.md` names no owning row, so a spec must keep claiming it; GitHub holds the closed state. Do not read it as open work. + +- [#1080](https://github.com/mudler/vllm.cpp/issues/1080) — `scripts/check-doc-checkpoint.py:153` + matches a spec's live-position section with `^##\s+Now\s*$`, and specs in this + tree write it as `## N. Now`, so `spec_now_errors` reports "has no `## Now` + section" about a section that is present and current. It fires only when a row + moves lifecycle state. THIS spec was one of them and is repaired in flow here — + `## 7. Now` becomes `## Now` — because this change is what makes it the spec a + moving row links, and the next lifecycle move on this row would otherwise red on + a section it is looking straight at. Re-measured at `b626be75a` AFTER that + repair: **15 specs still write the numbered spelling** — `gate-audit-branch-evidence`, + `ltx25-a2v-audio-input`, `ltx25-image-conditioning`, `ltx25-t2a-one-stage`, + `ltx25-token-append`, `ltx2-device-staged-view-uaf`, `mamba2-ssd`, + `nas-mount-path`, `nemotron-h-a2p-paged-forward`, `nemotron-h-a2q1-fp8-mamba`, + `nemotron-h-a2q2b-realckpt-lmhead`, `nemotron-h-a2q2-nvfp4-moe-lmhead`, + `nemotron-h-abi-e2e`, `offload-docs-refusal`, `registry-downcast-sweep`. The + population GREW since #1080 was filed against twelve, which is the argument for + the checker-semantics fix over a rename sweep: a rename repairs today's files and + the next spec written to the numbered spelling reintroduces it. Either close needs + its own spec and a red-before test, so neither rides in a records reconcile. + `tests/scripts/` covers `NOW_SECTION` nowhere, which is how the mismatch survived. + +- [#1217](https://github.com/mudler/vllm.cpp/issues/1217) — the runner hands + `ModelForwardInput::device_token_ids` to whatever model a step routes to, and + nothing enforces that a forward which ignores the field is never given one. This + architecture's paged forward was the SECOND model cut from that divergence + (Kimi-Linear was the first), which is what [#1157](https://github.com/mudler/vllm.cpp/issues/1157) + turned out to be. It is owed to `MODEL-NEMOTRON-H-ABI-A2P`. **The record for it + landed on `main` on 2026-08-18, in + [#1221](https://github.com/mudler/vllm.cpp/pull/1221) as `0ea5d249f`.** Both + writes are in this tree at this head. `.agents/issue-index.md:404` carries the + index row and names `MODEL-NEMOTRON-H-ABI-A2P` as its owning row. + [`nemotron-h-a2p-paged-forward.md:901`](nemotron-h-a2p-paged-forward.md) lists + the issue as the first bullet under that spec's `## 11. Owed`. The earlier + reason to withhold the index row was that #1221 was still open. Two branches + would then append the same key, and the union driver would merge them into a + DUPLICATE. That reason expired with the merge, and appending a row now would + create the duplicate it was written to avoid. **This reconcile therefore owes + nothing for #1217.** The issue stays open as a seam defect owned by + `MODEL-NEMOTRON-H-ABI-A2P`, and it is not why this row's end-to-end gate is + unrecorded on `main`. `## Now` gives that reason: the `STRICT PASS` 96/96 was + measured on the #1221 branch tree, and no run against current `main` exists. diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md index a3ae193c8..aea3101ca 100644 --- a/docs/BENCHMARKS.md +++ b/docs/BENCHMARKS.md @@ -12,9 +12,9 @@ | **Binary release (ACTIVE; Windows pre-alpha pending)** | v0.0.2 shipped eight primary archive/checksum/provenance triplets + two indexes (26 assets) from source SHA `7020de93652ca920424a10ac5255b34810dd2f24`, run `31466516224` | Windows W14-W16 implemented. **PENDING:** native hosted gates, merged-SHA ten-tuple dry run, matching-hardware evidence, v0.0.3-pre.1 publication, 32-asset audit | W12 optional/non-primary | | **Container images (ACTIVE; arm64 cuda verified on GB10 + Orin 2026-08-11)** | `ENG-RELEASE-CONTAINERS` ([spec](../.agents/specs/container-images.md)) | cpu amd64 783 MB; cuda arm64 **1.71 GB**. GB10 `sm_121a`: `/health`+`/version`+SIGTERM on `--gpus all`. Orin `sm_87` (Tegra): Qwen3-0.6B **generates**, GPU **GR3D 95-97%** | n/a | | **Developer/row protocol** | Contribution entry point; `ENG-NOW-DERIVED` #374 @`dbd0d51c` | Entry-point gates retained. #374 W1-W5 DONE; benchmark/runtime/parity `VOID`; row specs now carry `## Now` | n/a | -| **NemotronH paged forward** (`MODEL-NEMOTRON-H-ABI-A2P`, [#810](https://github.com/mudler/vllm.cpp/issues/810)) | **No speed number, by the unit's own rule** ([spec](../.agents/specs/nemotron-h-a2p-paged-forward.md) §5) | **A3 host gate PASSES 96/96 `STRICT PASS`.** GB10 read 4/24; cause and fix [#1157](https://github.com/mudler/vllm.cpp/issues/1157), sm_121a re-run pending a lease | CPU gate 12/12. Load 209.0 s, peak RSS 20 142 392 KB | | **NemotronH host re-expansion / decode token** (`A2-Q2b`, [#810](https://github.com/mudler/vllm.cpp/issues/810)) | Real Nemotron-3.5-Lightning-30B NVFP4 (rev `29f2d174`), T=1 decode step, counted at the `DenseBf16` seam via `examples/nemotron_h_gen` | Full host arm **3 078 512 640 elements (6.157 GB bf16)**, 369 calls. `lm_head` **352 321 536 (11.44%)**, **28.35%** of the post-A2-Q2a residue, **100%** once the mamba arm lands | Per-group table in [the record](../.agents/benchmark-record.md). Attribution only, no speed number. Device `lm_head` gate PENDING a `dgx:gpu0` window | | **Record-anchor ratchet** (`ENG-RECORD-ANCHOR-RATCHET`, #632) | **No number owed:** a record checker. At `8daa67b39`, **832 of 867** in-scope citations (**96.0%**) were already parsed; no symbol test and no report ran. Rot **38** (32 stale, 6 broken), **32 in range** | +| **NemotronH paged forward** (`MODEL-NEMOTRON-H-ABI-A2P`, [#810](https://github.com/mudler/vllm.cpp/issues/810)) | **No speed number, by the unit's own rule** ([spec](../.agents/specs/nemotron-h-a2p-paged-forward.md) §5) | **A3 96/96 `STRICT PASS`, mode=decode, GB10**, on the tree landed as `0ea5d249f` ([#1221](https://github.com/mudler/vllm.cpp/pull/1221)); NO run against current `main`. Fix reverted: 4/24 | CPU gate 12/12. Load 264.4 s, peak host 43 405 MB. NOT a benchmark | | **LoRA runtime W2** (`LORA-RUNTIME`, #278) | **No number owed:** correctness-only; a grid PENDS the W7 model gate | | **ARCH audit: ABI is text-only** | 4 capabilities (H3 video, Laguna, Kimi-Linear, DeepSeek-V4) reachable only from `examples/`, none registry-backed. No gate asks whether a CONSUMER can reach a capability. Documentation only | | **CUDA-graph break seam W1** (`ENG-CUDAGRAPH-BREAK`, [#1192](https://github.com/mudler/vllm.cpp/issues/1192)) | **No number owed, and none taken:** coverage and correctness row, no throughput gate declared | Capability, not a rate: mid-forward capture re-begin holds on a leased GPU; scoped forward matches eager, 500 logits, 0 differing. Probe committed, recipe and sha256 in the [record](../.agents/benchmark-record.md) | @@ -506,6 +506,7 @@ built on it rather than keeping the flattering one. | CPU keep-quant MoE decode | **No number owed**: correctness-only P0. The grouped keep-quant GEMM read activations as f32 whatever their dtype, so CPU MoE decode emitted token-0 garbage from `b4f5610a` (2026-07-31) | Speed unmeasured and unclaimed; `test_ops_quant_dot` GREEN (150224 assertions) | | Accepted-and-inert serve args (`SERVE-RECIPE-ARGS`, #606) | **No number owed**: argument parsing only, so nothing to time and no oracle leg. Correctness gate 4 cases / 58 asserts GREEN, RED-first, mutation-proven | None. A speed axis would be fabricated; closes on review plus the operator gate rerun | | GDN core/z at the model dtype, MoE arms (`GDN-MOE-BF16-OUT`, [#1168](https://github.com/mudler/vllm.cpp/issues/1168)) | **No number, none claimed.** `GATING`, CPU tier only. A token gate cannot see this axis: f32 is the MORE precise deviation ([spec](../.agents/specs/gdn-moe-bf16-out.md)) | 35B correctness first, then 315/315, 235/235 dense inertness, the `VT_GDN_OUT_BF16` 0-versus-1 A/B per leaf, and `nsys` for the memory format. All `PENDING` a GPU host | +| NemotronH model row (`MODEL-TEXT-nemotron-h-...`, [#517](https://github.com/mudler/vllm.cpp/issues/517)) | **No number, none claimed.** The row's FALSE `KERNEL-SSM-MAMBA` block is corrected (#1074); it STAYS `INVENTORIED`, because the A3 96/96 belongs to `0ea5d249f`, not to current `main` | A re-run of the A3 gate against `main`. Only a pass measured there moves this row, or gives a grid a denominator | | DeepSeek-V2-Lite MLA | Attributed miss, `ACTIVE` | Throughput at every concurrency | | Qwen3.5 text-only arms (#490) | **No number; run gates OWED**, both `PARTIAL`. The loader half is CLOSED (#740, #864 `DONE`), so what blocks these is hardware, not a refusal | No fitting ckpt for either causal-LM arm: no denominator. `Qwen3.8-2.4T-A95B` is ~4.8 TB vs 128 GB; its load plan resolves, which is not a token | | Darwin Qwen3.5 build repair (#1054, 2026-08-16) | **NOT APPLICABLE.** Removing a redundant namespace-scope lambda capture that Apple Clang rejects under `-Werror` changes no generated refusal text, no model math and no runtime path | None. The binding gate is the Apple Clang build | diff --git a/docs/FEATURES.md b/docs/FEATURES.md index 26ff27229..4afc081da 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -148,7 +148,7 @@ speed-pending, which [BENCHMARKS.md](BENCHMARKS.md) tracks. | `LagunaForCausalLM` | poolside/Laguna-S-2.1-NVFP4, GGUF-Q4_K, Laguna-XS | byte-exact near-tie (distributional vs vLLM) | vLLM parity+ 1.03x, default on, via the `laguna-gen` CLI; the registered engine forward VT_CHECKs non-bf16 (`ARCH-ONE-SURFACE` fold) | | `KimiLinearForCausalLM` | Kimi-Linear-48B-A3B (KDA + NoPE-MLA + MoE) | **Folded onto the shared paged runner (ROW 7 §21, #122): engine==CLI 128/128 byte-identical; vs golden 122/128 (the intrinsic near-tie profile); FA2 paged MLA default-ON; SACRED post-fold green** | Served via `vllm_engine_load` + `vllm_complete_tokens` (ABI v13); server 19.0 tok/s wall vs vLLM ~21 (~0.90×), speed residual open | | `KimiK3ForConditionalGeneration` | Kimi-K3 (2.8T MoE) | scaffold: registry+config+enumeration gated, forward refuses | HW-infeasible (~1.56 TB); no run | -| `NemotronHForCausalLM` | Nemotron-3.5-Lightning-30B-A3B-NVFP4 (`nvidia` @`29f2d174`) | config+enumeration+KV-shape gated; hybrid forward COMPUTES; loader materializes 18487/18487 as SHIPPED | **PAGED (#810 A2-P): K/V go to the runner's pages; conv+SSM rows carry at the metadata's state indices.** G-SAFE: `num_reqs <= 1`. Device `lm_head` (A2-Q2b), UNMEASURED. Owed: FP8 mamba (A2-Q1), MTP, GGUF | +| `NemotronHForCausalLM` | Nemotron-3.5-Lightning-30B-A3B-NVFP4 (`nvidia` @`29f2d174`) | config+enumeration+KV-shape gated; hybrid forward COMPUTES; loader materializes 18487/18487 as SHIPPED; A3 e2e gate 96/96 `STRICT PASS` on GB10 at `0ea5d249f` (#1221); NO run against current `main` | **PAGED (#810 A2-P): K/V go to the runner's pages; conv+SSM rows carry at the metadata's state indices.** G-SAFE: `num_reqs <= 1`. Device `lm_head` (A2-Q2b), UNMEASURED. Owed: FP8 mamba (A2-Q1), MTP, GGUF | | `MuseGlimmerForCausalLM` | real tensors, **bf16 depth 4/52 only**: 5 prefill argmax positions match a torch transcription of vllm#51655 and HF. GGUF full depth generates coherently (#347, #359) but is **NOT token-exact** | text forward + loader vs an fp32 reference, per-mechanism property tests, scaffold 11/11, GGUF gate 17/17. An ABSENT config key now takes the architecture's constant (#412): GGUF post-norms ran at 1e-5, not 1e-8 | no vLLM denominator (pin cannot load it); SECONDARY llama.cpp, same GGUF, GB10 CPU: prefill tie **0.997x**, decode 0.232x, RSS 1.92x (#333) | | `MuseGlimmerForConditionalGeneration` | vision: **no reference run of any kind**; enumeration gated vs the released 30B index (1436/1436). Image/video need bf16 safetensors: `mmproj-kquant.gguf` is refused by name | perception encoder loaded and wired, so an image or video prompt runs; `perception_emb_norm` now armed by default (#405). Reachability plus placeholder scatter only, no image or video correctness | not measurable; anchored to open vllm#51655 | | `LlamaModel` | landed tiny synthetic embedding fixture (engine path == direct pooler path, identical vectors; f64 LAST+normalize reference); real checkpoint (e5-mistral class) is a NAMED residual | pooling/embed only, text paths refuse by task; `vllm_embed` + `/v1/embeddings` | n/a (CPU correctness-grade embeddings) | diff --git a/docs/STATUS.md b/docs/STATUS.md index 526b312cf..09f8ad05d 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -45,7 +45,7 @@ The project uses these lifecycle terms: | Surface | Current state | Open gate or limitation | |---|---|---| -| Text generation | Correctness-complete on the gated model paths | Performance remains checkpoint-, model-, backend-, and concurrency-dependent | +| Text generation | Correctness-complete on the gated model paths | Performance remains checkpoint-, model-, backend-, and concurrency-dependent. Nemotron-3.5-Lightning-30B-A3B read 96/96 `STRICT PASS` (mode=decode) on GB10, measured on the tree that landed as `0ea5d249f` ([#1221](https://github.com/mudler/vllm.cpp/pull/1221), merged 2026-08-18); that fix is on `main`, no run against current `main` exists, and the row stays inventoried until one does ([model guide](models/nemotron-3-5-lightning.md), [#517](https://github.com/mudler/vllm.cpp/issues/517)) | | OpenAI server | Subset; v0.0.2 publishes eight server bundles; Windows v0.0.3-pre.1 pending | Some vLLM endpoints, pooling paths, and multimodal server paths remain incomplete | | C ABI and C++ library | Available | The C ABI is the stable public embedding surface; internal C++ headers are not ABI-stable | | Continuous batching, chunked prefill, prefix caching, and recompute | Available on the documented engine paths | Some hybrid-cache modes and scheduling policies still need broader gates |