diff --git a/.agents/claims/CLAIM-SPEC-DSPARK-QWEN3-ROUTING.md b/.agents/claims/CLAIM-SPEC-DSPARK-QWEN3-ROUTING.md new file mode 100644 index 000000000..80d753034 --- /dev/null +++ b/.agents/claims/CLAIM-SPEC-DSPARK-QWEN3-ROUTING.md @@ -0,0 +1,5 @@ +# CLAIM-SPEC-DSPARK-QWEN3-ROUTING + +| Claim | Row IDs | Agent | Worktree / remote dir | Branch | Owned scope | State | Last update | +|---|---|---|---|---|---|---|---| +| `CLAIM-SPEC-DSPARK-QWEN3-ROUTING` | `SPEC-DSPARK-QWEN3-ROUTING` (`ACTIVE`) | Claude Code (claude-opus-5), fresh implementer repairing a fresh review | worktree `dspark-qwen3-routing`, CPU only (no `nvcc`, no GPU lease) | `row/DSPARK-QWEN3-ROUTING-IMPL`, issue [#1193](https://github.com/mudler/vllm.cpp/issues/1193), spec [dspark-qwen3-routing.md](../specs/dspark-qwen3-routing.md). The branch name predates the row ID and does NOT match the `row/` form; it is recorded in the spec's `Role / claim` field rather than renamed, because renaming mid-flight strands the review already anchored to it | Owns ONLY: `SpeculativeConfig::IsDsparkDraft` and `SpeculativeConfig::ResolveDsparkArchitecture` in `include/vllm/config/speculative.h`; `ReadDsparkDraftIdentity` and the dspark branch of `LoadedEngine::ResolveSpecConfig` in `src/vllm/entrypoints/model_loader.cpp`; `tests/vllm/entrypoints/test_dspark_draft_routing.cpp` and the DSpark-routing cases of `tests/vllm/config/test_speculative_dspark.cpp`; this row, this claim file, the spec, and the `DSparkDraftModel` model-matrix row's routing note. **NON-COLLISION:** disjoint from `row/DSPARK-BLOCK-SIZE-GUARD` ([#1225](https://github.com/mudler/vllm.cpp/issues/1225)), which a parallel agent owns and which works on `n_predict` and `dspark_block_size` inside `ResolveDspark`; this claim touches neither key and neither `ResolveDspark` call site. EXCLUDED: the DSpark speculator, the draft weight path, the `--speculative-config` parser, and the second classification call site inside `LoadDsparkDraft` | `ACTIVE` | 2026-08-18 — W1-W4 landed, reviewed once, repaired once. G1 green (`test_speculative_dspark` 12/40, `test_dspark_draft_routing` 7/19), G2/G3 mutation-proven in a scratch copy, G4 green on the branch. Merged `origin/main` `b626be75a` on 2026-08-18, whose hoist of the DSpark resolution (#1225) makes this classification RUN FIRST on the `FromModelDir` path and retires the second call site this claim had excluded. Owed and named in the spec's `## Owed`: G5 (token-exact run gate, needs the 2.53 GiB draft and a GPU lease), G6 (spec-off byte identity), and the reading that would settle the empty-`architectures` narrowing | diff --git a/.agents/engine-matrix.md b/.agents/engine-matrix.md index a74ec0d6c..13b1cd9f2 100644 --- a/.agents/engine-matrix.md +++ b/.agents/engine-matrix.md @@ -43,12 +43,12 @@ forensics: roadmap_v1.md and the parity ledger. | Parallelism | 6 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 5 | | Sampling and generation | 15 | 4 | 2 | 0 | 0 | 4 | 0 | 1 | 4 | | Structured output and tools | 7 | 0 | 4 | 0 | 0 | 2 | 0 | 0 | 1 | -| Speculative decoding | 24 | 0 | 0 | 1 | 1 | 7 | 0 | 4 | 10 | +| Speculative decoding | 24 | 0 | 0 | 1 | 0 | 8 | 0 | 4 | 10 | | Serving, API, CLI, library | 35 | 10 | 2 | 1 | 3 | 10 | 2 | 3 | 4 | | LoRA and adapters | 2 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 1 | | Long context and attention | 10 | 5 | 0 | 0 | 1 | 0 | 1 | 0 | 3 | | Loading, tokenizer, config | 10 | 3 | 3 | 0 | 0 | 1 | 1 | 1 | 1 | -| **Total** | **163** | **35** | **17** | **5** | **12** | **36** | **8** | **11** | **38** | +| **Total** | **163** | **35** | **17** | **5** | **11** | **37** | **8** | **11** | **38** | ## Engine core and scheduling @@ -197,7 +197,7 @@ claims it. | `SPEC-CUSTOM-CLASS` | Pluggable custom proposer (`method="custom_class"`): the proposer class path is read from `speculative_config.model` and must expose a `propose` method — serves the extensibility-first priority (drop-in third-party speculators). | T2 | `vllm/v1/spec_decode/custom_class_proposer.py:12`; dispatch `vllm/v1/worker/gpu_model_runner.py:596-599` | - | - | [specs/spec-decode-inventory.md](specs/spec-decode-inventory.md) | `INVENTORIED` | - | | `SPEC-EXTRACT-HIDDEN` | Hidden-state extraction proposer (`method="extract_hidden_states"`): an offline / data-collection path that runs the target with aux hidden-state outputs (for training EAGLE/MTP drafts) rather than accelerating decode; model row `MODEL-SPEC-extract-hidden-states-extract-hidden-states-model`. | T2 | `vllm/v1/spec_decode/extract_hidden_states.py:29`; dispatch `vllm/v1/worker/gpu_model_runner.py:646-650` | - | - | [specs/spec-decode-inventory.md](specs/spec-decode-inventory.md) | `INVENTORIED` | - | | `SPEC-MLP-SPECULATOR` | MLP speculator (`method="mlp_speculator"`): a small MLP head predicting K tokens. UPSTREAM-DEPRECATED at the pin — the enum still resolves it but the V1 dispatch has NO branch, so it raises `ValueError` (`gpu_model_runner.py:651-654`); V0-only. Rowed for enumeration completeness, lowest priority. | T2 | `vllm/model_executor/models/mlp_speculator.py`; enum `vllm/config/speculative.py:69,890` | - | - | [specs/spec-decode-inventory.md](specs/spec-decode-inventory.md) | `INVENTORIED` | - | -| `SPEC-DSPARK-QWEN3-ROUTING` | DSpark draft-architecture routing: a draft whose config declares `architectures=["DSparkDraftModel"]` together with `model_type` `qwen3` must resolve to the landed Qwen3 DSpark lane, and the predicate that decides it must be reached from the loader. At the pin, upstream forces EVERY DSpark draft that is not `Qwen3DSparkModel` or `Gemma4DSparkModel` onto `model_type` `deepseek_v4` plus `architectures` `["DSparkDraftModel"]` (`speculative.py:934-944`); **BEYOND-PIN** vLLM PR vllm#52197, merged 2026-08-17 at `7075ddac`, puts a leading branch in front of that catch-all which normalizes the pair to `Qwen3DSparkModel`, and adds the same pair to the `method = "dspark"` auto-detection at `:882-887`. We diverge from BOTH ends: the forced rewrite was NEVER ported, so no code reads a draft config's `architectures` key (the three reads in `model_loader.cpp` at `:1247`, `:1249`, `:1692` are the TARGET model), and `SpeculativeConfig::IsDsparkDraft` (`speculative.h:120-136`, arch list `:131`) has NO production caller — every reference outside its header is in `tests/vllm/config/test_speculative_dspark.cpp:132-140`, while `ResolveSpecConfig` branches on `cli.method` alone (`model_loader.cpp:844`, `:862`, `:875`, `:886`). Net effect today: the checkpoint loads, by never having ported the pinned behavior rather than by decision, and the DeepSeek-V4 arm we do not implement has no named refusal. GATEABLE ON THIS HOST: `RadixArk/Qwen3.8-27B-DSpark` @ `85ef153be924f17ce4bf62726954eeaa4a73e854` carries exactly the shape (one 2718576122-byte shard, sha256 `9d26d5e637551c244d543c67c790bd0947f360e005c569e5851a185ffe692786`, 5 layers against a 64-layer Qwen3.8-27B target); the 2.4T lane named by the upstream PR stays memory-infeasible and is NOT claimed. Scoped to classification only: the speculator, the weight path and the `--speculative-config` parser are excluded | T1 | `vllm/config/speculative.py:882-887,934-944` @ `555967922`; **BEYOND-PIN** PR vllm#52197 @ `7075ddac28c25d4fd2b84bc2a9a6c5ffde0345c8` | - | - | [dspark-qwen3-routing.md](specs/dspark-qwen3-routing.md) ([#1193](https://github.com/mudler/vllm.cpp/issues/1193)) | `READY` | - | +| `SPEC-DSPARK-QWEN3-ROUTING` | DSpark draft-architecture routing: a draft whose config declares `architectures=["DSparkDraftModel"]` together with `model_type` `qwen3` must resolve to the landed Qwen3 DSpark lane, and the predicate that decides it must be reached from the loader. At the pin, upstream forces EVERY DSpark draft that is not `Qwen3DSparkModel` or `Gemma4DSparkModel` onto `model_type` `deepseek_v4` plus `architectures` `["DSparkDraftModel"]` (`speculative.py:934-944`); **BEYOND-PIN** vLLM PR vllm#52197, merged 2026-08-17 at `7075ddac`, puts a leading branch in front of that catch-all which normalizes the pair to `Qwen3DSparkModel`, and adds the same pair to the `method = "dspark"` auto-detection at `:882-887`. We diverged from BOTH ends until W1-W4 landed. **Before:** the forced rewrite was NEVER ported, so no code read a draft config's `architectures` key (the three reads in `src/vllm/entrypoints/model_loader.cpp` were the TARGET model's), and `include/vllm/config/speculative.h::IsDsparkDraft` had NO production caller — every reference outside its header was in `tests/vllm/config/test_speculative_dspark.cpp`, while `LoadedEngine::ResolveSpecConfig` branched on `cli.method` alone. So the checkpoint loaded by never having ported the pinned behavior rather than by decision, and the DeepSeek-V4 arm we do not implement had no named refusal. **Now (W1-W4 landed):** the dspark branch of `src/vllm/entrypoints/model_loader.cpp::ResolveSpecConfig` reads the draft's own `config.json` through `src/vllm/entrypoints/model_loader.cpp::ReadDsparkDraftIdentity` — which translates the Speculators layout first — and classifies it with `include/vllm/config/speculative.h::IsDsparkDraft` and then `include/vllm/config/speculative.h::ResolveDsparkArchitecture`. The predicate therefore HAS a production caller, the `DSparkDraftModel` + `qwen3` pair routes to the landed Qwen3 lane, and a draft that resolves to the DeepSeek-V4 lane is refused BY NAME rather than rewritten into the `DeepseekV4Model` stub (TWO tracked divergences from upstream: that refusal, and the Gemma4 collapse onto one implemented lane). A draft that DECLARES no architecture is deliberately NOT classified, which is narrower than upstream's catch-all and is recorded under `## Owed`. REACHED, as of the merge of `SPEC-DSPARK-BLOCK-SIZE-GUARD` ([#1225](https://github.com/mudler/vllm.cpp/issues/1225)): that row hoists the DSpark resolution to the top of `LoadedEngine::FromModelDir`, above the target-directory check and far above `maybe_load_dflash`, so `ResolveSpecConfig` and therefore this classification now run BEFORE `LoadDsparkDraft`. A user arriving through `include/vllm.h` or the server at a DeepSeek-V4 DSpark draft gets the NAMED refusal, not the draft loader's missing-key message. The second call site inside `LoadDsparkDraft` that this row previously owed is therefore no longer owed; only two of the resolution's own earlier messages still precede the classification (#1225's missing-`k` refusal, and the GGUF branch's named refusal). Still owed: the G5 token-exact run gate (2.53 GiB draft + GPU authority) and the G6 spec-off gate. GATEABLE ON THIS HOST: `RadixArk/Qwen3.8-27B-DSpark` @ `85ef153be924f17ce4bf62726954eeaa4a73e854` carries exactly the shape (one 2718576122-byte shard, sha256 `9d26d5e637551c244d543c67c790bd0947f360e005c569e5851a185ffe692786`, 5 layers against a 64-layer Qwen3.8-27B target); the 2.4T lane named by the upstream PR stays memory-infeasible and is NOT claimed. Scoped to classification only: the speculator, the weight path and the `--speculative-config` parser are excluded | T1 | `vllm/config/speculative.py:882-887,934-944` @ `555967922`; **BEYOND-PIN** PR vllm#52197 @ `7075ddac28c25d4fd2b84bc2a9a6c5ffde0345c8` | [speculative.h:134-232](../include/vllm/config/speculative.h#L134-L232) (`IsDsparkDraft` + `ResolveDsparkArchitecture`); [model_loader.cpp:466-502](../src/vllm/entrypoints/model_loader.cpp#L466-L502) (`ReadDsparkDraftIdentity`) called from [model_loader.cpp:1017-1086](../src/vllm/entrypoints/model_loader.cpp#L1017-L1086), the dspark branch of `LoadedEngine::ResolveSpecConfig` | [test_speculative_dspark.cpp:130-216](../tests/vllm/config/test_speculative_dspark.cpp#L130-L216) 12 cases / 40 assertions (the predicate); [test_dspark_draft_routing.cpp:368-460](../tests/vllm/entrypoints/test_dspark_draft_routing.cpp#L368-L460) 7 cases / 19 assertions (REACHABILITY, entered through the `LoadedEngine` constructor, both published draft layouts). Mutation-proven 2026-08-18 in a scratch copy: deleting the `IsDsparkDraft` call, making it return constant true, and deleting the `ResolveDsparkArchitecture` call each turn the routing suite RED (rc=1). G5 run gate and G6 spec-off OWED, GPU/authority-blocked | [dspark-qwen3-routing.md](specs/dspark-qwen3-routing.md) ([#1193](https://github.com/mudler/vllm.cpp/issues/1193)) | `ACTIVE` | `CLAIM-SPEC-DSPARK-QWEN3-ROUTING` | | `SPEC-DSPARK-BLOCK-SIZE-GUARD` | DSpark block floor, reached. `SpeculativeConfig::ResolveDspark` implements upstream's `k >= dspark_block_size` hard error, and BOTH production resolutions USED TO pass `std::nullopt` for `n_predict` AND for `dspark_block_size`, so the floor could not fire from any user path — the unpassed-parameter shape of `.agents/reachability.md`, exercised only by `tests/vllm/config/test_speculative_dspark.cpp` with hand-written values. The failure was SILENT: our draft step is sized only by `k` (`DsparkBlockLayout::num_speculative_steps` in `include/vllm/v1/worker/gpu/spec_decode/dspark/speculator.h`), nothing under `src/vllm/v1/worker/gpu/spec_decode/dspark/` reads the draft's block key, and no weight is shaped by the block, so a short `k` tripped no `VT_CHECK` and drafted a structurally wrong block while the tokens kept flowing. **A literal port would still not fire:** upstream reads `dspark_block_size`, an identifier that appears nowhere in the pinned checkout except `speculative.py`, and NEITHER published Qwen3 draft carries it — `deepseek-ai/dspark_qwen3_4b_block7` (`Qwen3DSparkModel`) and `RadixArk/Qwen3.8-27B-DSpark` @ `85ef153b` (`DSparkDraftModel`, `model_type` `qwen3`) both carry `block_size: 7` and no `n_predict`, and upstream's `block_size` normalization at `:945-961` is Gemma4-only, so `k=6` is accepted upstream on both sides of vllm#52197. LANDED: a file-local `ReadDsparkDraftKeys` threads the draft `config.json`'s real values into the ONE resolution that remains. `LoadedEngine::FromModelDir` no longer resolves a second time with its own argument list — it delegates to `LoadedEngine::ResolveSpecConfig` before it touches the model directory and hands the result to the draft load — so one resolution applies the floor, defaults `k` from `n_predict`, and owns the messages. TWO recorded divergences, both argued in the spec: `block_size` supplies the floor when `dspark_block_size` is absent (§2), and the refusal runs BEFORE the model-directory error where upstream builds `ModelConfig` first (§2a). Upstream's threshold, wording and `n_predict` rules are otherwise unchanged, and the refusal now names the key the value was read from. `test_dspark_block_size_guard` is 14/14 with 39 assertions; the reachability mutations red it 8/14 and 2/14, and the spec's §6c records the one claim the tests do NOT carry. G3 (run) and G4 (spec-off) owed, both needing a GPU lease. Excludes the speculator, `IsDsparkDraft` and the architecture classification (`SPEC-DSPARK-QWEN3-ROUTING`) | T1 | `vllm/config/speculative.py:945-961,973-994,1003-1027` @ `555967922` | `src/vllm/entrypoints/model_loader.cpp` `ReadDsparkDraftKeys` + the `ResolveDspark` call in `LoadedEngine::ResolveSpecConfig`, which `FromModelDir` now delegates to | `tests/vllm/entrypoints/test_dspark_block_size_guard.cpp` | [dspark-block-size-guard.md](specs/dspark-block-size-guard.md) ([#1225](https://github.com/mudler/vllm.cpp/issues/1225)) | `ACTIVE` | `CLAIM-SPEC-DSPARK-BLOCK-SIZE-GUARD` | ## Serving surface, CLI, and library diff --git a/.agents/model-matrix.md b/.agents/model-matrix.md index 5833fad2e..dc01be5be 100644 --- a/.agents/model-matrix.md +++ b/.agents/model-matrix.md @@ -546,7 +546,7 @@ of their own and are NOT part of the 366 architecture rows counted above. | `MODEL-SPEC-llama4-eagle-eagle-llama4-for-causal-lm` | `EagleLlama4ForCausalLM` | `registry.py:589`; `vllm/model_executor/models/llama4_eagle.py::EagleLlama4ForCausalLM` | speculative draft / target-dependent | draft runner; acceptance/sampling; EAGLE/EAGLE3 | ☐ required | `INVENTORIED` | none | unassigned | | `MODEL-SPEC-minicpm-eagle-eagle-mini-cpmfor-causal-lm` | `EagleMiniCPMForCausalLM` | `registry.py:590`; `vllm/model_executor/models/minicpm_eagle.py::EagleMiniCPMForCausalLM` | speculative draft / target-dependent | draft runner; acceptance/sampling; EAGLE/EAGLE3 | ☐ required | `INVENTORIED` | none | unassigned | | `MODEL-SPEC-qwen3-dflash-dflash-qwen3-for-causal-lm` | `DFlashDraftModel` | `registry.py:591`; `vllm/model_executor/models/qwen3_dflash.py::DFlashQwen3ForCausalLM` | speculative draft / target-dependent | draft runner; acceptance/sampling; sliding-window attention; DFlash | ✅ [DFlash spec](specs/dflash-spec-decode.md) §0 D14 | `DONE` | D0-redo (`CLAIM-DFLASH-D0D1`, 2026-07-26) UNBLOCKED on the advanced pin `555967922`/vLLM 0.26.0.dev0: the mixed-SWA/full z-lab 27B draft CONSTRUCTS + drafter ALIVE under `VLLM_USE_V2_MODEL_RUNNER=1` (acceptance > 1); D0+D1 (`DF-AUX-TAPS` target-side multi-tap) landed. **D2 (`DF-DRAFT-MODEL`, `CLAIM-DFLASH-D2`, 2026-07-26) CODE LANDED + CPU-GATED:** drafter model `qwen3_dflash.{h,cpp}` (plain 5-layer Qwen3-dense reusing the landed dense block ops, routing attention through the NEW `vt::DFlashBlockAttention` non-causal in-block primitive), the fc aux-combine, mask-embed, per-layer SWA/full resolution (`ResolveQwen3DFlashAttnModes`), and the z-lab loader. CPU gate GREEN: `test_ops_dflash_block_attn` 5/12 (hand-checked non-causal + RED causal-vs-non-causal + block isolation + SWA window + GQA), `test_qwen3_dflash_forward` 5/95 (forward runs + RED full-layer-causal-flip + block isolation + fc-combine ref + RED reversed-tap-order + attn-mode resolution). Existing causal `test_ops_attention` 9/9·23 + `test_qwen3_forward` 5/1028 UNCHANGED (new op is SEPARATE; causal path byte-identical). **`ACTIVE` — GPU promotion GREEN on dgx 2026-07-26 (GB10 sm_121a):** CUDA `-Werror` build clean (kernel compiles as-written); CUDA==CPU parity `test_ops_dflash_block_attn` 198412/198412 + compute-sanitizer 0; draft-forward parity `test_qwen3_dflash_draft_parity` vs the REAL vLLM DFlash draft (via `collective_rpc`) — fc rel-L2 0.46%, per-layer hidden ≤1.30%, final 0.88%, 11 deterministic rows STRICT top-1 + 5 bf16-near-tie cluster-matched; 27B SACRED 235/235 + MTP 9/9 byte-identical. Loader now tolerates the draft ckpt's omitted embed_tokens/lm_head (target-shared). Capture harness `scripts/spec/d2_dflash_draft_ref.py`. **D3 `DF-DRAFT-KV-PREP` CODE LANDED + CPU-GATED 2026-07-26 (`CLAIM-DFLASH-D3`):** `PrepareDflashInputs` (pure-integer HOST port of `_prepare_dflash_inputs_kernel`) + `PrecomputeContextKV` + `ForwardBlockLogitsWithContext` in `qwen3_dflash.{h,cpp}` reuse the landed MatmulBT/RmsNorm/RopeNeox + the UNCHANGED D2 `vt::DFlashBlockAttention` via a `[context;block]` combined sequence (NO new kernel); CPU gate `tests/vllm/v1/spec_decode/test_dflash_kvprep.cpp` 6/114 (prepare INTEGER bit-exact + RED valid_ctx_end; context-KV envelope + RED hidden_norm/k_norm/pos; context forward degenerates to D2 at empty ctx + diverges with ctx + block isolation); additive-only diff so D2/MTP/SACRED byte-identical. **D3 DONE 2026-07-26 — GPU numeric-parity GREEN on dgx (GB10 sm_121a):** CUDA `-Werror` 0 warnings; `test_qwen3_dflash_kvprep_parity` 61/61 vs the REAL vLLM draft — `prepare_dflash_inputs` INTEGER bit-exact vs vLLM's ACTUAL Triton `_prepare_dflash_inputs_kernel` (`kernel_matches_numpy`), context-KV worst K/V rel-L2 0.31%/0.26% across all 5 layers (proves per-layer `qkv_proj` slicing == vLLM's fused KV weight), block-proposal 13 STRICT + 3 near-tie = 16/16; CPU `test_dflash_kvprep` 114/114 re-passed (RED-proven); 27B SACRED 235/235 + MTP 9/9 + D2 parity 37/37 byte-identical. Capture harness `scripts/spec/d3_dflash_kvprep_ref.py`. **D4 `DF-ENGINE-INTEGRATION` propose brick + `dflash` config-select CODE LANDED + CPU-GATED 2026-07-26 (`CLAIM-DFLASH-D4D5`):** NEW `src/vllm/v1/worker/gpu/spec_decode/dflash/speculator.{h,cpp}` `DflashProposeBlock`/`SampleDflashBlockDrafts` — the non-autoregressive whole-block propose swapping in for the MTP k=1 `MtpProposePrefill`, composing D3 `ForwardBlockLogitsWithContext` + greedy per-mask argmax (anchor not sampled, `dflash/speculator.py:300-413`); `ParseSpeculativeConfigJson`/`ResolveDflash` accept `method:"dflash"` (lookahead k+1). CPU gate `tests/vllm/v1/spec_decode/test_dflash_propose.cpp` 5/19 GREEN (RED-first: sampler reading the anchor row fails 4/5 cases; brick == forward+sampler; empty-ctx degenerates to D2 context-free argmax; config parses; dspark throws), `-Werror`-clean TU. Additive + config-gated ⇒ MTP/non-spec byte-identical BY CONSTRUCTION (no runner/model/loader/scheduler edit). **D5 `DF-ENGINE-INTEGRATION` runner-loop LANDED + e2e RUNS on dgx 2026-07-26 (`CLAIM-DFLASH-D5`):** the verify/propose loop is wired — loader loads the SEPARATE z-lab draft (`LoadDflashDraft`, host bf16 + target-SHARED bf16 embed/lm_head) via a `--speculative-config` `model` key + `ResolveSpecConfig` dflash branch + `runner.set_dflash_draft`; the verify forward captures the D1 multi-tap (`aux_tap`→`ForwardDeviceMultiTap`); `propose_drafts_dflash` ACCUMULATES the per-request combined-feature context (`CombineAuxFeatures(aux_tap)`) and honors the `num_rejected` rollback by appending only the accepted-prefix features, then runs `DflashProposeBlock` (k=16). **e2e `test_qwen27_dflash_spec_decode` (4×32 tok vs the committed vLLM-DFlash-ON golden): 2/4 STRICT token-exact (fibonacci, three-laws) + acceptance ~ vLLM on ALL 4 (accepted 19/39/29/25 vs golden 17/39/30/25 — the MANDATORY dead-drafter-trap condition MET).** The 2 divergences (France tok11, 17*23 tok12) are SINGLE bf16 near-tie flips (17*23 RE-CONVERGES = proven near-tie; France cascades from one flip) — the ratified near-tie ROOT, rooted in the D3-documented inline bf16 context-KV recompute envelope (~0.3-1.3% rel-L2), NOT a wiring bug (proven by the 2 exact prompts + near-exact acceptance). Inertness GREEN (SACRED 235/235 + MTP 9/9 byte-identical); CUDA `-Werror` clean; NO new CUDA kernel (host orchestration reusing D1/D2/D3-sanitized ops). NOT a clean strict-4/4 pass — STRICT 4/4 token-identity + speed A/B = D6 (persistent paged draft-KV bit-matching vLLM's fused context-KV projections). **D6 `CLAIM-DFLASH-D6` 2026-07-27 (records-only, NO code) — c1 SPEED A/B DONE + STRICT proven bf16-IRREDUCIBLE + CG feasibility:** our DFlash-ON = **2.50x TPOT (40.4 vs 101.2 ms) / 2.48x output-tput (24.4 vs 9.86 tok/s)** over our OFF at c1 (8 prose+code prompts×256 tok greedy, 2 reps, acceptance 0.22 = 3.56/16, rep-stable <1.5%), `benchmark_binding=true`; vs vLLM-DFlash-ON graphed vLLM-DFlash-ON graphed = 28.5 tok/s / 35.1 ms TPOT / acceptance_len 4.30 (same 8 prompts, `VLLM_USE_V2_MODEL_RUNNER=1`, mm-off, gpu_util 0.30), so OURS IS ~14% BELOW vLLM-DFlash-ON on output throughput (24.4 vs 28.5 tok/s) - both ~on-par at spec-OFF (9.86 vs 9.83 tok/s), but vLLM extracts a larger DFlash speedup (2.90x vs our 2.47x) because its draft step is fully device-resident + CUDA-graphed (ours host-orchestrates 13 downloads/step) + slightly higher acceptance (~4.3 vs ~3.6 draft tokens/step). The DONE speed bar (ours >= vLLM) is NOT met; closing it = the device-resident draft rewrite + FULL CG (D6 part 2). STRICT-4/4 is bf16-IRREDUCIBLE — the draft KV cache is bf16 not fp8 (`torch_utils.py:398` `auto`→model dtype; the D0 "fp8-KV" was the backend name), the D3 golden already compares pre-storage bf16 (residual = sub-ULP kernel noise), and a fused multi-layer KV GEMM is per-element invariant to our per-layer GEMMs (bit-exact needs vLLM's exact kernels) ⇒ the ratified near-tie gate is the FINAL correctness form (no fused-KV code landed). FULL CG + persistent-paged-KV BLOCKED on a device-resident draft-path rewrite (the D5 path does 13 device→host downloads/step) — the remaining throughput-parity increment. Inertness by construction (D5 binary; SACRED 235/235 + MTP 9/9 stand). Row STAYS `ACTIVE` (correctness-final + c1-speed-measured). Evidence tool `scripts/spec/vllm_dflash_timing.py`. **D7 `CLAIM-DFLASH-D7` 2026-07-27 (source-owning) — within-step draft forward DEVICE-RESIDENT:** `PrecomputeContextKVDevice` keeps per-layer K/V on device + `ForwardBlockLogitsWithContext` builds `[context;block]` with `vt::IndexCopy`/`IndexSelect` (removes ~30 D→H `Download`s/step), bit-identical (identity `bf16↔f32` round-trips replaced, no float op changed) — e2e `test_qwen27_dflash_spec_decode` 27/27 SAME tokens (2/4 STRICT + 2/4 near-tie at identical divergence points, acceptance 19/39/29/25), SACRED 235/235 + MTP 9/9, CUDA `-Werror` clean, compute-sanitizer 0 (198412; no new kernel). The direct old-vs-new A/B = **+2.0% output-tput (IN-NOISE)** ⇒ **D6's "downloads = the ~14% gap" REFUTED by measurement**; ours 19.68 tok/s STILL **~33% BELOW** vLLM-DFlash-ON 29.2 (reconstructed 8-prompt set; OFF parity our 9.97 ≥ vLLM 9.66); residual re-attributed to **acceptance** (ours 2.49 vs vLLM ~3.13 accepted draft-tok/step, bf16-irreducible) + per-step **context-KV RECOMPUTE** (O(context²); needs the cross-step persistent paged draft-KV store) + eager-vs-graphed. SPEED BAR NOT met; STAYS `ACTIVE` **D9 `CLAIM-DFLASH-D9` 2026-07-27 (source-owning) — PERSISTENT PAGED DRAFT-KV LANDED (bit-identical, +22.7%):** `qwen3_dflash.{h,cpp}` `AppendContextKVHost` (project ONLY newly-accepted rows → per-layer bf16 K/V) + `ForwardBlockLogitsWithPrecomputedKV` (upload the persistent store, NO re-projection) share the core `ForwardWithCtxKVDev` with the old recompute; `runner.cpp::propose_drafts_dflash` swaps the O(context²) per-step recompute (`dflash_ctx_feats_`) for an append-only per-request `dflash_kv_store_` (rollback=don't-append). NO new CUDA kernel; config-gated. Bit-identity: CPU `test_dflash_propose` 2 new D9 cases = exact float equality vs full recompute; GPU e2e `test_qwen27_dflash_spec_decode` **27/27 SAME tokens** (acceptance 19/39/29/25, same divergences); SACRED 235/235 + MTP 9/9 byte-identical; CUDA `-Werror` clean. **A/B (c1, 8 prose+code×256 tok, `benchmark_binding=true`):** ours-ON **25.75 tok/s** (was 20.99, +22.7%) / 38.40 ms / acc 3.68/step vs vLLM-ON graphed **28.09** / 35.60 / acc 3.31 = **0.917×** (was 0.69×). **Part 1:** same-trajectory per-step acceptance == vLLM EXACTLY (ratio 1.00 on the 2 token-identical prompts) AND ours realized acceptance (3.68) > vLLM (3.31) ⇒ D8's "bf16 acceptance ceiling" REFUTED (confound). Residual ~8% = eager-vs-graphed ONLY; the FULL uniform-(1+k) CG (device paged-KV store + paged attn, new-CUDA) is the SOLE un-landed increment. **D12/D13 `CLAIM-DFLASH-D12/D13` 2026-07-27 — fixed-capacity paged draft-KV store (`vt::DFlashPagedBlockAttention`) + the draft-step CUDA graph landed (capture-correctness PROVEN replayed==eager bit-identical); c1 0.917×→0.978×.** **D14 `CLAIM-DFLASH-D14` 2026-07-27 — SPEED GATE MET → `DONE`:** nsys attributed the D13 ~2% residual to the from-scratch `DFlashPagedBlockAttentionKernel` draft attention (1.8% of GPU time, ~460 us/call; the draft bf16 GEMMs run `cutlass_80_wmma` in BOTH engines, NOT the gap); ported it to a WARP-scoped `__shfl_xor` online-softmax variant (mirrors the shipped `AttentionWarpKernel`; `src/vt/cuda/cuda_ops.cu` `DFlashPagedBlockAttentionWarpKernel`, default ON, `VT_DFLASH_ATTN_BLOCK=1` = the bit-identical D12/D13 block kernel) → draft attn 242.9→77.9 ms (3.1×), our-ON c1 28.60→29.32 tok/s. FINAL same-session 3-rep A/B: our-ON 29.42/29.27/29.32 vs vLLM-ON 29.240/29.247/29.233 — our WORST > vLLM's BEST, 1.003× ⇒ **≥vLLM MET.** Correctness UNCHANGED (e2e 27/27 graph==eager, acceptance 19/39/29/25 identical, 1629 draft accepted identical warp-vs-block; CUDA==CPU 795648/795648 + compute-sanitizer 0), inertness SACRED 235/235 + MTP 9/9. Correctness-complete (ratified near-tie) AND at/above vLLM throughput. Anchors: [cuda_ops.cu](../src/vt/cuda/cuda_ops.cu#L1433) + [test_ops_dflash_paged_block_attn](../tests/vt/test_ops_dflash_paged_block_attn.cpp#L79) + [ledger](parity-ledger.md#L738); closing commit `164453a2` (claims CLAIM-DFLASH-D0D1..D14 recorded in coordination.md + ledger). | `489a7544` | -| `MODEL-SPEC-deepseek-v4-dspark-deepseek-v4-for-causal-lm` | `DSparkDraftModel` | `registry.py:592`; `vllm/models/deepseek_v4/__init__.py::DSparkDeepseekV4ForCausalLM` | speculative draft / target-dependent | draft runner; acceptance/sampling; FusedMoE/grouped GEMM; MLA/latent KV; sliding-window attention; DSpark | ☐ required | `INVENTORIED` | none | unassigned | +| `MODEL-SPEC-deepseek-v4-dspark-deepseek-v4-for-causal-lm` | `DSparkDraftModel` | `registry.py:592`; `vllm/models/deepseek_v4/__init__.py::DSparkDeepseekV4ForCausalLM` | speculative draft / target-dependent | draft runner; acceptance/sampling; FusedMoE/grouped GEMM; MLA/latent KV; sliding-window attention; DSpark | ☐ required | `INVENTORIED` | The DeepSeek-V4 destination stays `INVENTORIED` and unimplemented, but the STRING now has two: **BEYOND-PIN** vllm#52197 (merged 2026-08-17 at `7075ddac`) makes `DSparkDraftModel` name a QWEN3 draft whenever the draft's `model_type` is `qwen3`, which is what `RadixArk/Qwen3.8-27B-DSpark` declares. `SPEC-DSPARK-QWEN3-ROUTING` ([#1193](https://github.com/mudler/vllm.cpp/issues/1193), [spec](specs/dspark-qwen3-routing.md)) routes that pair to the landed `Qwen3DSparkModel` lane in `SpeculativeConfig::ResolveDsparkArchitecture` and REFUSES this DeepSeek-V4 destination by name from `LoadedEngine::ResolveSpecConfig`, rather than mirroring upstream's silent rewrite into a stub (§7 R2). Evidence: [test_speculative_dspark.cpp](../tests/vllm/config/test_speculative_dspark.cpp), [test_dspark_draft_routing.cpp](../tests/vllm/entrypoints/test_dspark_draft_routing.cpp) | unassigned | | `MODEL-SPEC-qwen3-dspark-qwen3-dspark-for-causal-lm` | `Qwen3DSparkModel` | `registry.py:593`; `vllm/model_executor/models/qwen3_dspark.py::Qwen3DSparkForCausalLM` | speculative draft / target-dependent | draft runner; acceptance/sampling; DSpark | ☐ required | `INVENTORIED` | none | unassigned | | `MODEL-SPEC-laguna-dflash-dflash-laguna-for-causal-lm` | `DFlashLagunaForCausalLM` (v0.25.0 target-pending) | v0.25.0 target `registry.py:598`; `vllm/model_executor/models/laguna_dflash.py::DFlashLagunaForCausalLM` @ `702f481` | speculative draft / Laguna targets | draft runner; acceptance/sampling; full/sliding attention; DFlash | ☐ required | `INVENTORIED` | none | unassigned | | `MODEL-SPEC-llama-eagle3-eagle3-llama-for-causal-lm` | `PEagleDraftModel`, `PeagleLlamaForCausalLM`, `Eagle3LlamaForCausalLM`, `Eagle3MiniMaxM2ForCausalLM`, `LlamaForCausalLMEagle3`, `Eagle3Qwen2_5vlForCausalLM`, `Eagle3Qwen3vlForCausalLM` | `registry.py:594-600`; `vllm/model_executor/models/llama_eagle3.py::Eagle3LlamaForCausalLM` | speculative draft / target-dependent | draft runner; acceptance/sampling; EAGLE/EAGLE3 | ☐ required | `INVENTORIED` | none | unassigned | diff --git a/.agents/specs/dspark-qwen3-routing.md b/.agents/specs/dspark-qwen3-routing.md index dc4ecef22..cb74a3e0d 100644 --- a/.agents/specs/dspark-qwen3-routing.md +++ b/.agents/specs/dspark-qwen3-routing.md @@ -14,7 +14,7 @@ | Work breakdown | §4. | | Risks/decisions | §7. | | Pin policy | Mirror ahead of the pin, marked `BEYOND-PIN`. §2 gives the reasoning and the precedents. | -| Role / claim | helper, branch `row/SPEC-DSPARK-QWEN3-ROUTING` | +| Role / claim | helper. The implementation branch is `row/DSPARK-QWEN3-ROUTING-IMPL`, which does NOT match the `row/` form this row's ID (`SPEC-DSPARK-QWEN3-ROUTING`) requires. Recorded rather than renamed: renaming mid-flight strands the review that is already anchored to the branch, and the deviation costs a reader one lookup, not a wrong answer. The next branch for this row uses `row/SPEC-DSPARK-QWEN3-ROUTING` | | Base | `fd64c76ee45ba49b070ea83024f6678ddd7f64a6` (origin/main, 2026-08-18) | | Parity pin | vLLM `555967922` (0.26.0.dev0) at `$VLLM_SOURCE` | @@ -204,14 +204,37 @@ the predicate. `AGENTS.md` "Nothing lands dead" requires the smallest failing test to enter the new code through a production entry point, and a unit test on `IsDsparkDraft` is exactly the unit test that rule excludes. +**Each refusal is asserted by its OWN wording.** Both refusals on this path name +DeepSeek-V4 and say "not implemented", so a case that asserts only those two +substrings cannot tell the identity check from the architecture normalization — +and a test that cannot tell them apart cannot see either one deleted. The +identity case asserts "does not identify as"; the normalization case asserts +"routes to the DeepSeek-V4 DSpark lane". Both mutations in §6 G2 depend on this. + +**Branch 3 needs a loader case too.** Added on the `b626be75a` merge rather +than at the original landing. `ResolveDsparkArchitecture`'s Gemma4 arm — the +COLLAPSE onto `Qwen3DSparkModel` that `## Outcome` records as the second +divergence — was exercised only by a hand-called case in +`tests/vllm/config/test_speculative_dspark.cpp`. That proves the function +answers; it does not prove the loader reaches the arm, which is exactly the +distinction `AGENTS.md` `## Nothing lands dead` draws. The loader case asserts +the Gemma4 name is ADMITTED, from a directory whose name carries no `dspark`, so +only the architecture arm can answer for it. + +**Both published draft layouts, not one.** The Speculators layout declares no +top-level `architectures` and is translated to `["Qwen3DSparkModel"]` before the +classification reads it, so it is the one shape where the no-architecture +narrowing and the classification disagree and the ORDER decides. It gets its own +loader case. + ## 6. Gates | Gate | Content | State | |---|---|---| -| G1, focused | The §5 cases green, the pinned `:132-140` cases unchanged. | owed | -| G2, mutation | Delete the `ResolveSpecConfig` call site in a scratch copy and rerun G1. A green G1 without the call site measures a class, not a capability, and fails this row. | owed | -| G3, mutation | Revert the `:131` architecture list to the two pinned names and rerun G1. The first §5 case must go red. | owed | -| G4, full | `scripts/agent-preflight.sh` and the repository gate on the branch. | owed | +| G1, focused | The §5 cases green, the pinned `IsDsparkDraft` cases unchanged. | GREEN 2026-08-18 on the `b626be75a` merge: `test_speculative_dspark` 12 cases / 40 assertions, `test_dspark_draft_routing` 7 cases / 19 assertions, both `Status: SUCCESS!`, rc=0. The routing suite grew from 6/17 at the merge: the Gemma4 collapse (branch 3 of `ResolveDsparkArchitecture`) had only a hand-called unit case, which is the shape `AGENTS.md` `## Nothing lands dead` excludes, so it gained a loader-level case beside the other `DraftDir` fixtures | +| G2, mutation | Delete the `ResolveSpecConfig` call site in a scratch copy and rerun G1. A green G1 without the call site measures a class, not a capability, and fails this row. Three mutations, because the call site has three separable parts and a refusal that names only the LANE cannot tell them apart. | RED as required 2026-08-18, in a scratch copy at `$SCRATCH/mut`, each with `compile_rc=0`: (a) delete the whole `IsDsparkDraft` call → routing rc=1; (b) make `IsDsparkDraft` return constant `true` → routing rc=1 and the predicate suite rc=1; (c) delete the `ResolveDsparkArchitecture` call → routing rc=1. The scratch tree was restored byte-for-byte and re-verified green after each. A FOURTH mutation was added on the `b626be75a` merge, for the Gemma4 case above: narrowing branch 3's guard from `!has_qwen3 && !has_gemma4` to `!has_qwen3` reds the routing suite at rc=1, 7 cases run, 6 passed, 1 failed, and the one failure is the new case. Its FIRST writing failed to build — `-Werror` on the now-unused `has_gemma4` — and the stale binary reported 7/19 SUCCESS, so the mutation was rewritten with a `(void)has_gemma4;` to compile cleanly and only then measured (`compile_rc=0`). The header was restored from a byte copy and all three suites re-verified green | +| G3, mutation | Revert the architecture list in `IsDsparkDraft` to the two pinned names and rerun G1. The first §5 case must go red. | proven by G2(b), which is the stronger form: a predicate that admits EVERYTHING already fails the case that only the `DSparkDraftModel` + `qwen3` arm can pass | +| G4, full | `scripts/agent-preflight.sh` and the repository gate on the branch. | GREEN 2026-08-18 on the tree merged up to `origin/main` `aba8d5ffb`, with ONE environmental red: `test_cpu_x86_llamacpp_floor` refuses to measure while the box is loaded (`NO_QUIET_WINDOW`, exit 4 where the case expects 2, at load average 41-107 driven by other sessions). It is not this row's: the file's last change is `0305b909f` on `main`, this branch touches no benchmark harness, and the same suite reported `ok` on the same sources in the preceding preflight run at a quieter moment. Every other gate `ok`, including `check-agent-record`, `check-symbol-anchors`, `check-doc-checkpoint`, `check-public-doc-tables`, and both trailer gates | | G5, run | Token-exact decode against the pinned oracle with #52197 applied, same target, same draft, same k, greedy. `Qwen/Qwen3.8-27B` + `RadixArk/Qwen3.8-27B-DSpark` at revision `85ef153be924f17ce4bf62726954eeaa4a73e854`. | owed, GPU-blocked, NOT hardware-blocked | | G6, spec-off | Decode with speculation off byte-identical before and after. The change touches classification only, so a difference here is a defect in the change. | owed | @@ -272,7 +295,13 @@ that lands it. **R3. Two draft-config reads.** §3 leaves the choice to the implementer and requires the reason under `## Outcome`. -**R4. `block_size` is read by nobody.** Both `ResolveDspark` call sites +**R4. `block_size` is read by nobody.** CLOSED by +[#1225](https://github.com/mudler/vllm.cpp/issues/1225) +(`SPEC-DSPARK-BLOCK-SIZE-GUARD`, landed `b626be75a`), which took the issue and +the row this risk asked for. The paragraph below records the state at the time +this spec was written; on the merged tree there is ONE `ResolveDspark` call +site and it is passed the draft's real `n_predict` and block floor. Both +`ResolveDspark` call sites (`model_loader.cpp:881-883` and `:1675-1677`) pass `std::nullopt` for `n_predict` and for `dspark_block_size`, so the `k >= dspark_block_size` hard error at `speculative.py:1003-1027` cannot fire. The 27B draft carries @@ -301,9 +330,79 @@ in §2 do. `.agents/model-matrix.md:549`. The comment stays correct once the amendment above lands. No engine-matrix arch row is created, for the keyed-record reason in §0. -- R4's `block_size` gap needs its own issue and row. It is a correctness hole in - the landed `SPEC-DSPARK` lane, not in this routing change, and folding it in - here would bundle unrelated work into one branch. +- ~~R4's `block_size` gap needs its own issue and row.~~ DISCHARGED: it became + [#1225](https://github.com/mudler/vllm.cpp/issues/1225) and row + `SPEC-DSPARK-BLOCK-SIZE-GUARD`, which landed on `main` as `b626be75a` and is + merged into this branch. Keeping it out of this branch was the right call for + the reason given — it is a correctness hole in the landed `SPEC-DSPARK` lane, + not in this routing change — and the two rows met at the merge rather than in + one branch. +- **The empty `architectures` list is NOT classified.** Upstream reads the key + off a HuggingFace `ModelConfig`, where an absent key is `[]`, so its catch-all + sends that list to DeepSeek-V4. The loader here skips classification instead, + because refusing on the ABSENCE of evidence would refuse the native + `deepseek-ai/dspark_qwen3_*_block7` drafts if they declare no architecture, and + no copy of one has been read on this host to settle it. Owed: read a published + native draft's `config.json`, and tighten the guard to upstream's catch-all if + it declares the key. Gated today by + `test_dspark_draft_routing.cpp`'s no-architecture case, which pins the + narrowing so it cannot change silently. +- **The refusal LEADS, as of the merge of `SPEC-DSPARK-BLOCK-SIZE-GUARD` + ([#1225](https://github.com/mudler/vllm.cpp/issues/1225), `b626be75a`).** The + two entries this replaces were both true when they were written and are both + false now, so they are re-derived here rather than deleted. `AGENTS.md` + `## Nothing lands dead` judges reachability at the MERGE commit, which is why + this could not be left to the next reader. + + #1225 hoists the DSpark resolution to the top of + `src/vllm/entrypoints/model_loader.cpp::LoadedEngine::FromModelDir`: a + `dspark` method now calls `LoadedEngine::ResolveSpecConfig(params, + vllm::HfConfig{})` there and hands the result down to the draft load. That + call is ABOVE the target-directory existence check and far above + `maybe_load_dflash`, so `ResolveSpecConfig` — and therefore + `ReadDsparkDraftIdentity`, `IsDsparkDraft` and `ResolveDsparkArchitecture` — + runs BEFORE `LoadDsparkDraft` rather than after it. Re-derived from the merged + control flow: + + - `FromModelDir` is the one production engine constructor, reached from + `include/vllm.h` through `src/capi/vllm_c.cpp` and from + `src/vllm/entrypoints/openai/server_main.cpp`. A user arriving through the C + ABI or the server at a DeepSeek-V4 DSpark draft now gets the NAMED refusal. + `LoadDsparkDraft`'s "the draft config must carry target_layer_ids and + mask_token_id" no longer wins, because the load it comes from no longer + runs first. + - Two of the resolution's OWN messages still precede the classification, and + both are correct where they are. #1225's "requires num_speculative_tokens" + check sits ahead of the classification inside the same `cli.method == + "dspark"` branch, so a run that names no `k` against a draft carrying no + `n_predict` is refused for the missing `k`. And a `.gguf` target takes the + GGUF branch above the hoist, whose own named refusal ("needs a safetensors + target at this pin") fires first. + - The in-memory constructors + `LoadedEngine(HfConfig, Qwen3_5DenseWeights, Tokenizer, EngineParams)` and + its `Qwen3_5MoeWeights` sibling — the ones whose member-init runs + `ResolveSpecConfig`, and the entry point the reachability suite uses — have + exactly ONE caller across `src/`, `include/` and `examples/`, and the + earlier writing of this entry said there was none. Re-run on the merge, a + grep for `LoadedEngine` over those three trees hits 23 files; the three that + CONSTRUCT one are `src/capi/vllm_c.cpp:776` and + `src/vllm/entrypoints/openai/server_main.cpp:974,1155`, both through + `LoadedEngine::FromModelDir`, and `examples/bench/bench_core.h:570`, which + calls the in-memory `Qwen3_5MoeWeights` constructor directly. The + CONCLUSION is unchanged and now rests on the right reason: an example's + internals are not a production entry point under `AGENTS.md` + `## Nothing lands dead`, and that call sets no `params.speculative_config` + at all (`bench_core.h` assigns it only in its `else` arm at `:587-589`, + which loads through `FromModelDir` at `:591`), so `ResolveSpecConfig` + returns `std::nullopt` at its first line and never reaches the dspark + branch. + + So the second classification call site inside `LoadDsparkDraft`, which the + earlier entry owed, is NOT owed: the ordering it existed to fix is fixed, and + adding it would put a second copy of the classification behind the one that + already leads. What remains owed on this item is nothing. The row's + remaining debt is §6's G5 run gate and G6 spec-off gate, both above. + - A pin advance past `555967922` inherits from this row: remove the `BEYOND-PIN` marks, re-point the `speculative.py` anchors at the new line numbers, and re-check whether the `K3DSparkModel` arm visible in #52197's @@ -328,20 +427,114 @@ Stop and report `NEEDS_DECISION` when any of these holds. Stop and report `NEEDS_CONTEXT` when the draft download authority or the GPU authority is unrecorded. Do not assume either. -## Now - -`READY`. The spec is committed, [#1193](https://github.com/mudler/vllm.cpp/issues/1193) -is open, and no implementation has landed. - -The pin decision is made and recorded: **mirror #52197 ahead of the pin, marked -`BEYOND-PIN`**, on the precedent of `.agents/model-matrix.md:318-319`, -`:100` and `:211`. §2 carries the reasoning. A pin advance is not scoped by this -row and is not blocked by it. +## Outcome + +Recorded at the W1-W4 landing, ahead of `DONE`, because §3 and §7 R3 name a +decision this row had to make and neither the code nor the Git history states +the reason on its own. + +**R3, THREE draft-config reads, not one.** #1225 has now landed +(`b626be75a`), so the count this entry anticipated is the count that exists. The +dspark path reads the draft's `config.json` three times: in +`src/vllm/entrypoints/model_loader.cpp::ReadDsparkDraftIdentity` for the two +classification keys, in +`src/vllm/entrypoints/model_loader.cpp::ReadDsparkDraftKeys` for `n_predict` and +the block floor, and in +`src/vllm/entrypoints/model_loader.cpp::LoadDsparkDraft` for the weights. All +three bodies open the file, parse it, and run the same Speculators translation. +**Decision: unchanged — do not hoist here.** + +The REASON first given for it has expired and is corrected. It was that the +reads "sit on opposite sides of the engine constructor", so a shared read had +nowhere to live that both callers reach. That was true while `LoadDsparkDraft` +ran inside `FromModelDir`'s `maybe_load_dflash` and `ResolveSpecConfig` ran in +the constructor's member-init. #1225's hoist moved the resolution to the top of +`FromModelDir`, so `ReadDsparkDraftIdentity` and `ReadDsparkDraftKeys` are now +ADJACENT — consecutive statements in the same branch of the same function — and +a shared read between those two would need no new member and no new parameter on +any public seam. The structural obstacle is gone. + +What survives is the cost argument and the scope argument, and they still carry +the decision. The cost is a five-layer draft's `config.json` parsed three times +at load time, on a path that then reads a 2.53 GiB shard; it is not measurable, +and this row measured nothing that would justify changing it. The three readers +want different keys and answer to different rows, and each names the keys it +wants, so neither can silently change what the others see. And the third reader, +`LoadDsparkDraft`, is still on the far side of the whole target load, so a hoist +that unified only the two adjacent readers would leave the duplication in a +shape harder to reason about than three symmetric readers. ONE hoist serving all +three callers is worth its own row; it is not this row's, whose §9 lists growth +into the loader's weight path as a stop condition. + +**The DeepSeek-V4 refusal, and what it reaches.** §7 R2's decision stands and is +argued in three places (the predicate's header, the loader, and this row). Its +REACH has been restated twice. The first writing overclaimed it; a review +corrected it to "leads on no production path today", which was right while +`FromModelDir` ran `LoadDsparkDraft` before `ResolveSpecConfig`; and #1225's +hoist (`b626be75a`) made that correction false in turn. The refusal now LEADS +from `include/vllm.h` and from the server, which is this row's own goal being +met by a change that landed beside it. The re-derivation, the two resolution +messages that still precede it, and the second call site that is consequently NO +LONGER OWED are under `## Owed`. + +**TWO divergences from upstream, not one.** The first writing named only the +DeepSeek-V4 refusal. The second is at branch 3: upstream leaves a Gemma4 draft's +`Gemma4DSparkModel` architecture in place and normalizes only its keys, because +upstream has a Gemma4 DSpark class to dispatch to. This engine has one DSpark +draft lane, so branch 3 COLLAPSES onto `Qwen3DSparkModel`, and that collapse is +what makes `ResolveDsparkArchitecture` total over a single lane. + +**Why the loader dispatches on nothing.** The first shape of the call site +guarded on `lane != "Qwen3DSparkModel"` and threw. Because of the collapse +above, no input can enter that branch: the function answers `Qwen3DSparkModel` +or throws. A reviewer's mutation deleted the branch and both suites stayed +green, which is the definition of dead code under `AGENTS.md` `## Nothing lands +dead`. It was deleted rather than disclosed, because the staged-slice exception +is for work a named row will wire, and no row will wire this one: the branch +becomes live only when a SECOND lane exists, and the change that adds that lane +is the change that should add its dispatch. `ResolveDsparkArchitecture` is still +called, for its refusal — a mutation deleting the call turns the routing suite +red. + +**Why an empty `architectures` list is not classified.** Under `## Owed`. +Upstream can send an absent key to its DeepSeek-V4 path; refusing here would +refuse a checkpoint whose contents nobody on this host has read, and the native +`deepseek-ai/dspark_qwen3_*_block7` drafts load today. -The reality check is answered: `RadixArk/Qwen3.8-27B-DSpark` exists at revision -`85ef153be924f17ce4bf62726954eeaa4a73e854` with the exact config shape, so this -row is gateable on this host. The 2.4T lane named by the upstream PR stays -memory-infeasible here and is not claimed. +## Now -Next: dispatch a fresh implementer for W1-W3 against this spec. W1 captures the -two reds of §5 before any production edit. +`ACTIVE`. W1-W4 have landed on the row's branch, been reviewed once and repaired +once. [#1193](https://github.com/mudler/vllm.cpp/issues/1193) is open and stays +open until G5 runs. + +Landed: `IsDsparkDraft` carries the `DSparkDraftModel` + `qwen3` pair; +`ResolveDsparkArchitecture` normalizes the three accepted shapes and refuses the +DeepSeek-V4 lane by name; and the dspark branch of `ResolveSpecConfig` reaches +both from the draft's own `config.json`. G1 green (`test_speculative_dspark` +12 cases / 40 assertions, `test_dspark_draft_routing` 7 cases / 19 assertions), +G2 and G3 proven by mutation, G4 by the branch gate. + +The pin decision stands as recorded in §2: **mirror #52197 ahead of the pin, +marked `BEYOND-PIN`**, on the precedent of the three existing `BEYOND-PIN` rows +in `.agents/model-matrix.md`. A pin advance is not scoped by this row and is not +blocked by it. + +Next, in order: G5, the token-exact run gate against +`Qwen/Qwen3.8-27B` + `RadixArk/Qwen3.8-27B-DSpark` @ +`85ef153be924f17ce4bf62726954eeaa4a73e854`, which needs the 2.53 GiB draft +download and a GPU lease and has neither; then G6, the spec-off byte-identity +check. Both are `PENDING` against a named external authority, not failing, and +the row does not reach `DONE` until they run. They are the row's only remaining +debt. + +Merged `origin/main` `b626be75a` on 2026-08-18. `SPEC-DSPARK-BLOCK-SIZE-GUARD` +([#1225](https://github.com/mudler/vllm.cpp/issues/1225)) landed there and +hoisted the DSpark resolution to the top of `LoadedEngine::FromModelDir`, which +makes this row's classification and its named DeepSeek-V4 refusal RUN FIRST on +the production path instead of behind `LoadDsparkDraft`. Two statements this +spec carried became false at that merge and are re-derived under `## Owed`, and +`## Outcome`'s R3 reason expired with them and is corrected there. The focused +suites on the merge: `test_speculative_dspark` 12 cases / 40 assertions, +`test_dspark_draft_routing` 7 cases / 19 assertions, and #1225's own +`test_dspark_block_size_guard` 14 cases / 39 assertions, all `Status: SUCCESS!`, +rc=0. diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md index 6e3976b8b..a560f0af4 100644 --- a/docs/BENCHMARKS.md +++ b/docs/BENCHMARKS.md @@ -397,6 +397,7 @@ in the tree, default-OFF, for reproducibility; detail in the benchmark record. | n-gram | Qwen3.6-27B NVFP4 | draft-free (`SPEC-NGRAM`); 27B 5/5 STRICT our-ngram-ON == vLLM-ngram-ON, 180/180 drafts accepted (correctness only, no speed row yet) | `DONE` | | DSpark | 27B NVFP4 dense k=15; 35B-A3B MoE k=8 | MoE 35B-A3B: **0.835x** paired on kairos-17dd (matched 89 tokens, warm oracle cache). Prior 0.957-0.989 came from a different machine with a cold oracle (#442) | `ACTIVE` | | DSpark block floor | Qwen3.8-27B + `RadixArk/Qwen3.8-27B-DSpark` @ `85ef153b` | a `k` below the draft's block is refused instead of drafted; the run gate that exhibits the garbling is **owed** and needs a GPU lease (#1225) | `ACTIVE` | +| DSpark draft routing | Qwen3.8-27B + `RadixArk/Qwen3.8-27B-DSpark` @ `85ef153b` | **PENDING**, no number. The token-exact run gate needs the 2.53 GiB draft and GPU time, and neither authority is recorded; only the CPU classification gate has run (`.agents/specs/dspark-qwen3-routing.md` §6) | `ACTIVE` | | Breadth (EAGLE1/3, suffix, ngram-gpu, dynamic-k, ...) | n/a | enumerated from vLLM source + `INVENTORIED` 2026-08-06 (`.agents/specs/spec-decode-inventory.md`), unmeasured | `INVENTORIED` | ## How we measure diff --git a/docs/FEATURES.md b/docs/FEATURES.md index da1f014a3..d9aeb453b 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -240,6 +240,7 @@ both refuse, naming what is missing. | DFlash block diffusion | ✅ 2.9x over spec-off, at/above vLLM DFlash-on | ✅ | ☐ | | n-gram / prompt lookup | ✅ 27B 5/5 strict vs vLLM | ✅ | ✅ | | DSpark (semi-autoregressive block drafter) | ◐ **both gate models** ([spec](../.agents/specs/dspark-spec-decode.md)): token-identical to spec-off; T=1+k verify CAPTURED. Cross-engine ratio UNSETTLED (**0.834x** matched-and-warm); Marlin MoE CLEARED as the residual | ✅ | ◐ | +| DSpark draft routing (which draft the loader takes) | ◐ `Qwen3DSparkModel`, `Gemma4DSparkModel` and (BEYOND-PIN, vllm#52197) `DSparkDraftModel` + `qwen3` take the Qwen3 lane; DeepSeek-V4 DSpark is REFUSED by name ([spec](../.agents/specs/dspark-qwen3-routing.md)) | ◐ at the pinned `555967922` that pair routes to DeepSeek-V4; ✅ only since vllm#52197, merged 2026-08-17 | not assessed | | Other methods (ngram-gpu, suffix, custom-class, dynamic-k, mlp-speculator) | ☐ inventoried | ✅ | ◐ | ## Structured output and tool calling diff --git a/docs/SPECULATIVE-DECODING.md b/docs/SPECULATIVE-DECODING.md index 656b3c09e..b6685d76b 100644 --- a/docs/SPECULATIVE-DECODING.md +++ b/docs/SPECULATIVE-DECODING.md @@ -172,6 +172,33 @@ carries no `n_predict`), and it must be at least the checkpoint's block size — a smaller value produces incorrect output rather than merely lower acceptance, so it is rejected. +### Which DSpark draft the loader will take + +The lane is decided by the DRAFT's own `config.json`, not by the method string +you typed. Three architecture spellings route to the Qwen3 DSpark lane: + +| The draft declares | Routes to | +|---|---| +| `architectures: ["Qwen3DSparkModel"]` | the Qwen3 DSpark lane | +| `architectures: ["Gemma4DSparkModel"]` | the Qwen3 DSpark lane (same loader) | +| `architectures: ["DSparkDraftModel"]` with `model_type: "qwen3"` | the Qwen3 DSpark lane | + +The third row is vLLM PR +[52197](https://github.com/vllm-project/vllm/pull/52197), merged 2026-08-17, +which is AHEAD of the pinned oracle; it is mirrored here because the pinned +behavior is wrong for a checkpoint that is already published +(`RadixArk/Qwen3.8-27B-DSpark` declares exactly that pair). + +A draft config that carries no `architectures` key at all is not classified. It +loads exactly as it did before, because an absent key is not evidence of a lane. + +Anything else that names an architecture is the DeepSeek-V4 DSpark draft, whose weights ship inside the +DeepSeek-V4 target rather than in a separate checkpoint. vLLM rewrites such a +config onto `model_type: "deepseek_v4"` and loads it; **this engine refuses it by +name** instead, because it carries only a stub for that model and the lane needs +two DGX Sparks. A refusal that names the missing arm is what you get, rather than +a load that fails later on a missing key. + ## The flag On the OpenAI server: diff --git a/docs/STATUS.md b/docs/STATUS.md index 22f0df59a..42afdb933 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -278,6 +278,8 @@ a spike while its user-facing serving surface is finalized. **DSpark block floor** (`SPEC-DSPARK-BLOCK-SIZE-GUARD`, ACTIVE, [#1225](https://github.com/mudler/vllm.cpp/issues/1225)). A speculative length below the draft's block was accepted silently: both `ResolveDspark` call sites passed `std::nullopt`, so the `k >= block` floor reached no user path, and our draft block is sized by `k` alone. It is refused now, with `block_size` supplying the floor when upstream's `dspark_block_size` is absent — one recorded divergence, because neither published Qwen3 draft sets that key. The GPU run gate that exhibits the garbling is owed. +**DSpark draft routing** (`SPEC-DSPARK-QWEN3-ROUTING`, ACTIVE) makes the loader classify a DSpark draft from the draft's own `config.json` before it resolves anything else. `Qwen3DSparkModel`, `Gemma4DSparkModel` and — ahead of the pin, mirroring vllm#52197 — `DSparkDraftModel` with `model_type` `qwen3` take the landed Qwen3 lane; a draft that resolves to the DeepSeek-V4 DSpark lane is refused BY NAME instead of being rewritten into a stub. CPU-gated only: the token-exact run gate against the pinned oracle waits on a draft download and GPU time that are not authorized, so it stays owed (#1193). + **DeepSeek-V4 native MTP** (`DeepSeekV4MTPModel`, ACTIVE — W1 self-spec wiring, 2026-07-30) has its nextn draft head wired to the same lossless spec-decode path. Unlike V3's fused `eh_proj`, the V4 nextn layer keeps separate `e_proj`/`h_proj`, diff --git a/docs/USAGE.md b/docs/USAGE.md index 045467a78..86641e3ad 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -2350,7 +2350,7 @@ a stop token early. | `--reasoning-parser ` | `none` | Reasoning parser (`think_auto`, `deepseek_r1`, `deepseek_v3`, `holo2`, `mistral`, `minimax_m2`, `minimax_m2_append_think`, `step3`, `olmo3`, `muse_glimmer`, `qwen3`, `mimo`). `auto` detects, `none` disables. `qwen3` and its `mimo` alias are the engine-backed adapter (one upstream class, two registry names): thinking is ON, so a marker-less stream is reasoning and a `` ends reasoning with no ``. `auto` never selects it — a generic `` template resolves to `think_auto`, which is the right default for hybrid-thinking models that may answer with no think block at all | | `--kv-transfer-config ''` | (unset) | External KV connector, same JSON as vLLM's flag. See [docs/KV-OFFLOAD.md](KV-OFFLOAD.md) | | `--offload-config ''` | (unset) | Weight offload, the same JSON vLLM's `OffloadConfig` takes (distinct from `--kv-transfer-config`, which offloads KV blocks). Parsed and validated at startup, so a malformed document, an unknown backend, an unknown TOP-LEVEL key (the four legal ones are `offload_backend`, `uva`, `prefetch` and `vllm_cpp`) or a validator violation is refused before any model I/O; a backend/field mismatch is a warning, as upstream. **Enabling it fails startup on every model today**: no loader consults the offloader, so the engine refuses the configuration by architecture name rather than accept a budget that frees nothing. A config that leaves offloading disabled still parses and reports normally. On unified memory such as GB10 offload cannot help at all, because host and device share one pool. See [docs/WEIGHT-OFFLOAD.md](WEIGHT-OFFLOAD.md). The same document also carries the **`vllm_cpp` key**, which governs the tier BELOW this one — weights borrowed out of the file mapping rather than moved to host RAM — and which is live rather than refused: see [Streaming routed experts from disk](#streaming-routed-experts-from-disk-capacity-mode). A `vllm_cpp`-only document does not enable vLLM's offload backends and is not subject to the refusal above. The flag is accepted by `vllm-server` (the generate/chat and the pooling/embedding paths), by `vllm-cli`, and by the C ABI; the server's transcription-only path REFUSES it by name, because that path builds no engine and could only accept the document and ignore it ([#1195](https://github.com/mudler/vllm.cpp/issues/1195)) | -| `--speculative-config ''` | (unset) | Speculative decoding (`mtp`, `dflash`, `ngram`), same JSON as vLLM's flag. For `mtp`, `num_speculative_tokens` sets the draft DEPTH and defaults to the checkpoint's `mtp_num_hidden_layers`, which is 1 on both gate checkpoints, so the default is unchanged. A value above it must be a multiple of it, mirroring vLLM. Depth cannot move the emitted tokens under greedy decoding, and no speed number is claimed above k=1 yet ([#81](https://github.com/mudler/vllm.cpp/issues/81)). What is gated on CPU at k=1..4 is that the propose runs `k-1` draft decode forwards per propose call, that k drafts reach the verify path, and that the drafts DELIVERED to the verify path vary with depth rather than repeating the first one. That last one is counted over a RUN and never per call, because a correct drafter may resample the same token and this fixture does. Two things are NOT gated there. A draft is never accepted at depth, because acceptance is zero at every depth on the synthetic gate model. And nothing here proves the draft at depth j came from the j-th forward. Both are owed to the GPU gate, which must close the second by comparing the per-depth acceptance RATE against a PADDED control rather than by asserting a non-zero acceptance count, because a padded drafter earns acceptance at depth whenever the target's own greedy continuation repeats a token. `dspark` speculates on the Qwen3.6 gate models (native + Speculators drafts), token-identically to speculative-off, but is not gated on speed: the cross-engine ratio is UNSETTLED, with a matched-and-warm paired measurement of 0.834x against the pinned oracle and the earlier 0.957x-0.989x figures taken against a single COLD oracle invocation on a machine that has since been reimaged. A GGUF target, or a target with no aux multi-tap, is refused by name (`SPEC-DSPARK`). Its sequential Markov sampling runs on device by default; `VT_DSPARK_DEVICE_SAMPLE=0` restores the host loop (token-identical, cost only). The speculative verify runs from a captured CUDA graph, worth +12.2%/+3.5% on the 35B cells; `VT_SPEC_DECODE_GRAPH=0` restores the eager verify (also token-identical). The object is admitted key by key and NOTHING is dropped ([#1160](https://github.com/mudler/vllm.cpp/issues/1160)): the honoured keys are `method`, `num_speculative_tokens`, `model`, `prompt_lookup_min` and `prompt_lookup_max`, plus `draft_sample_method` and `rejection_sample_method` at their upstream defaults `greedy` and `standard`, which are what this engine implements. Any other value of those two names row `SPEC-ACCEPT-VARIANTS` and is refused. A name vLLM's `SpeculativeConfig` declares but this engine does not implement, such as `quantization`, is refused as exactly that, and any other name is refused as unknown with the accepted list. Before this the extra key was discarded, so `draft_sample_method=probabilistic` ran GREEDY and a misspelled `num_speculatve_tokens` took the default, both silently and both at exit 0. For `dspark`, `num_speculative_tokens` may no longer sit BELOW the draft checkpoint's block: DSpark drafts a block, our block is sized from this value alone, and a shorter one drafted a structurally wrong block in silence. It is refused now, before any weight is loaded, naming the block, the config key the block was read from, and the value given ([#1225](https://github.com/mudler/vllm.cpp/issues/1225)). The block is read from the draft config's `dspark_block_size`, or from `block_size` when that key is absent, which is the case on every published Qwen3 draft (`deepseek-ai/dspark_qwen3_4b_block7` and `RadixArk/Qwen3.8-27B-DSpark` both carry `block_size: 7`, so k must be at least 7). vLLM reads only the first key and accepts the shorter value. vLLM also builds its model config BEFORE its speculative config, so a command that names both a target directory it cannot open and a short `k` hears about the target there and about the `k` here. Those are the two recorded divergences, both argued in `.agents/specs/dspark-block-size-guard.md`. A k at or above the block behaves exactly as before. See [docs/SPECULATIVE-DECODING.md](SPECULATIVE-DECODING.md) | +| `--speculative-config ''` | (unset) | Speculative decoding (`mtp`, `dflash`, `ngram`), same JSON as vLLM's flag. For `mtp`, `num_speculative_tokens` sets the draft DEPTH and defaults to the checkpoint's `mtp_num_hidden_layers`, which is 1 on both gate checkpoints, so the default is unchanged. A value above it must be a multiple of it, mirroring vLLM. Depth cannot move the emitted tokens under greedy decoding, and no speed number is claimed above k=1 yet ([#81](https://github.com/mudler/vllm.cpp/issues/81)). What is gated on CPU at k=1..4 is that the propose runs `k-1` draft decode forwards per propose call, that k drafts reach the verify path, and that the drafts DELIVERED to the verify path vary with depth rather than repeating the first one. That last one is counted over a RUN and never per call, because a correct drafter may resample the same token and this fixture does. Two things are NOT gated there. A draft is never accepted at depth, because acceptance is zero at every depth on the synthetic gate model. And nothing here proves the draft at depth j came from the j-th forward. Both are owed to the GPU gate, which must close the second by comparing the per-depth acceptance RATE against a PADDED control rather than by asserting a non-zero acceptance count, because a padded drafter earns acceptance at depth whenever the target's own greedy continuation repeats a token. `dspark` speculates on the Qwen3.6 gate models (native + Speculators drafts), token-identically to speculative-off, but is not gated on speed: the cross-engine ratio is UNSETTLED, with a matched-and-warm paired measurement of 0.834x against the pinned oracle and the earlier 0.957x-0.989x figures taken against a single COLD oracle invocation on a machine that has since been reimaged. A GGUF target, or a target with no aux multi-tap, is refused by name (`SPEC-DSPARK`). The DRAFT is classified from its own `config.json` rather than from the method string: `Qwen3DSparkModel`, `Gemma4DSparkModel`, and — BEYOND-PIN, mirroring [vllm#52197](https://github.com/vllm-project/vllm/pull/52197) merged 2026-08-17 — `DSparkDraftModel` together with `model_type` `qwen3` all route to the Qwen3 DSpark lane, and every other DSpark draft that DECLARES an architecture is the DeepSeek-V4 variant, which is refused by name because this engine carries only a stub for it (`SPEC-DSPARK-QWEN3-ROUTING`, [#1193](https://github.com/mudler/vllm.cpp/issues/1193)). A draft config carrying no `architectures` key at all is not classified and loads as before, because an absent key is not evidence of a lane. Its sequential Markov sampling runs on device by default; `VT_DSPARK_DEVICE_SAMPLE=0` restores the host loop (token-identical, cost only). The speculative verify runs from a captured CUDA graph, worth +12.2%/+3.5% on the 35B cells; `VT_SPEC_DECODE_GRAPH=0` restores the eager verify (also token-identical). The object is admitted key by key and NOTHING is dropped ([#1160](https://github.com/mudler/vllm.cpp/issues/1160)): the honoured keys are `method`, `num_speculative_tokens`, `model`, `prompt_lookup_min` and `prompt_lookup_max`, plus `draft_sample_method` and `rejection_sample_method` at their upstream defaults `greedy` and `standard`, which are what this engine implements. Any other value of those two names row `SPEC-ACCEPT-VARIANTS` and is refused. A name vLLM's `SpeculativeConfig` declares but this engine does not implement, such as `quantization`, is refused as exactly that, and any other name is refused as unknown with the accepted list. Before this the extra key was discarded, so `draft_sample_method=probabilistic` ran GREEDY and a misspelled `num_speculatve_tokens` took the default, both silently and both at exit 0. For `dspark`, `num_speculative_tokens` may no longer sit BELOW the draft checkpoint's block: DSpark drafts a block, our block is sized from this value alone, and a shorter one drafted a structurally wrong block in silence. It is refused now, before any weight is loaded, naming the block, the config key the block was read from, and the value given ([#1225](https://github.com/mudler/vllm.cpp/issues/1225)). The block is read from the draft config's `dspark_block_size`, or from `block_size` when that key is absent, which is the case on every published Qwen3 draft (`deepseek-ai/dspark_qwen3_4b_block7` and `RadixArk/Qwen3.8-27B-DSpark` both carry `block_size: 7`, so k must be at least 7). vLLM reads only the first key and accepts the shorter value. vLLM also builds its model config BEFORE its speculative config, so a command that names both a target directory it cannot open and a short `k` hears about the target there and about the `k` here. Those are the two recorded divergences, both argued in `.agents/specs/dspark-block-size-guard.md`. A k at or above the block behaves exactly as before. See [docs/SPECULATIVE-DECODING.md](SPECULATIVE-DECODING.md) | | `--language-model-only` / `--no-language-model-only` | off | Disable all multimodal input by setting **every** modality limit to 0, mirroring vLLM's flag of the same name. It is not a "skip the encoder" switch: the server then **refuses** a multimodal request with ``400 At most 0 image(s) may be provided in one prompt. Set `--limit-mm-per-prompt` to increase this limit.`` It does **not** free VRAM yet — nothing gates tower construction on it ([#607](https://github.com/mudler/vllm.cpp/issues/607) wave L3) | | `--limit-mm-per-prompt ''` | (unset ⇒ 999 per modality) | Maximum multimodal input items per prompt, per modality, as the same JSON object vLLM's flag takes: `'{"image": 2, "video": 0}'`, or with profiling options `'{"video": {"count": 1, "num_frames": 32}}'` (the options are validated and ignored — they size dummy inputs for memory profiling, which this engine does not do). A limit can only **lower** what the model/seam supports, never raise it. Malformed JSON, a negative count, or an unknown option on `image` / `video` / `audio` is refused at startup rather than defaulted. An unknown option on any other modality name is dropped rather than refused, mirroring upstream, whose fallback `BaseDummyOptions` is the one such dataclass without `extra="forbid"`. Upstream's dotted spelling (`--limit-mm-per-prompt.image 2`) is not accepted here, as for `--kv-transfer-config` and `--speculative-config` | | `--enable-log-requests` / `--disable-log-requests` | on | Log each incoming request. Mirrors vLLM's flag of the same name | @@ -2460,6 +2460,51 @@ For a production deployment, use [LocalAI](https://localai.io), which can embed engines like this behind a model gallery, multi-model serving, the full OpenAI API surface, auth, and metrics. +## DSpark drafts: the exact checkpoints + +A DSpark draft is a SEPARATE checkpoint named by the `model` key of +`--speculative-config`. A repo id alone is not a pin, because a checkpoint can be +re-quantized in place under an unchanged name, so the revision is part of the +identity. + +| Draft | Repo and revision | File | Bytes | sha256 | +|---|---|---|---|---| +| Qwen3.8-27B, 5 layers against a 64-layer target | `RadixArk/Qwen3.8-27B-DSpark` @ `85ef153be924f17ce4bf62726954eeaa4a73e854` | `model.safetensors` | 2 718 576 122 | `9d26d5e637551c244d543c67c790bd0947f360e005c569e5851a185ffe692786` | + +That draft declares `architectures: ["DSparkDraftModel"]` with `model_type: +"qwen3"` and `block_size: 7`, which is the pair +[vllm#52197](https://github.com/vllm-project/vllm/pull/52197) routes to the Qwen3 +DSpark lane and which this engine mirrors ahead of its pinned oracle +(`SPEC-DSPARK-QWEN3-ROUTING`, +[#1193](https://github.com/mudler/vllm.cpp/issues/1193)). **It has not been run +here yet**: the token-exact gate against the pinned oracle needs the 2.53 GiB +download and GPU time, and both are pending developer authority, so the routing +is gated on CPU and the decode is not. + +The two layouts that already run are the native +`deepseek-ai/dspark_qwen3_*_block7` drafts and the Speculators-format +`RedHatAI/*.dspark` drafts; the DeepSeek-V4 DSpark draft, whose weights ship +inside the DeepSeek-V4 target, is refused by name. + +**Which refusal you actually get today.** Point the server or the C API at a +DeepSeek-V4 DSpark draft and the message is the named DeepSeek-V4 refusal, the +one the classification produces. `LoadedEngine::FromModelDir` resolves a +`dspark` speculative config ONCE, at the top of the function +([#1225](https://github.com/mudler/vllm.cpp/issues/1225)), before it opens the +target directory and long before it loads the draft, so the classification is now +the FIRST thing a DSpark run meets. An earlier writing of this paragraph said the +draft loader's "the draft config must carry target_layer_ids and mask_token_id" +won instead; that was true while the draft load ran ahead of the resolution, and +it stopped being true when the resolution was hoisted. + +Two messages still come out in front of it, and both are the resolution's own. +A `dspark` run that names no `num_speculative_tokens` against a draft whose +config carries no `n_predict` is refused for the missing `k` first, because that +check sits ahead of the classification in the same branch. And a `.gguf` target +takes the GGUF branch above the hoist, which carries its own named refusal for a +GGUF DSpark target (`SPEC-DSPARK`). Either way the draft is refused and nothing +loads it as a Qwen3 draft. + ## Muse Glimmer 30B from a GGUF k-quant The text tower loads from a `muse-glimmer`-architecture GGUF, so the 30B model diff --git a/include/vllm/config/speculative.h b/include/vllm/config/speculative.h index 3c6af4217..86a692fa8 100644 --- a/include/vllm/config/speculative.h +++ b/include/vllm/config/speculative.h @@ -118,8 +118,22 @@ struct SpeculativeConfig { // RedHatAI/Qwen3.6-35B-A3B-speculator.dspark) OR whose architectures name // "Qwen3DSparkModel" / "Gemma4DSparkModel". Kept separate from ResolveDspark so // the loader can classify a checkpoint before building any config. + // + // BEYOND-PIN (SPEC-DSPARK-QWEN3-ROUTING, #1193): the fourth arm — the + // "DSparkDraftModel" + model_type "qwen3" PAIR — is vllm-project/vllm#52197 + // hunk 1, merged 2026-08-17 at 7075ddac28c25d4fd2b84bc2a9a6c5ffde0345c8 and + // absent from the pin 555967922. It is mirrored AHEAD of the pin because the + // pinned behavior is wrong for a checkpoint that is published and loads here + // today (RadixArk/Qwen3.8-27B-DSpark declares exactly that pair); + // .agents/specs/dspark-qwen3-routing.md §2 carries the decision and the three + // model-matrix precedents for the mark. + // + // `model_type` defaults to the empty string so a caller holding only the + // architecture list asks exactly the pinned question and gets the pinned + // answer. static bool IsDsparkDraft(const std::string& draft_model_id, - const std::vector& architectures) { + const std::vector& architectures, + const std::string& model_type = "") { std::string lowered = draft_model_id; for (char& c : lowered) { c = static_cast( @@ -132,10 +146,91 @@ struct SpeculativeConfig { if (arch == "Qwen3DSparkModel" || arch == "Gemma4DSparkModel") { return true; } + // vllm#52197 hunk 1 (BEYOND-PIN): the same architecture string names a + // DeepSeek-V4 draft when model_type is "deepseek_v4", so the PAIR is the + // condition and the architecture alone is not. + if (arch == "DSparkDraftModel" && model_type == "qwen3") { + return true; + } } return false; } + // ResolveDsparkArchitecture: the architecture normalization upstream performs + // before a DSpark draft loads (speculative.py:934-944 @ 555967922, plus + // vllm-project/vllm#52197 hunk 2, BEYOND-PIN). Upstream writes the result back + // onto the draft's hf_config and calls update_arch_(); this returns it, + // because our loader dispatches on the value instead of mutating a config + // object it will read again. + // + // Upstream's three branches, in upstream's own order: + // 1. "DSparkDraftModel" + model_type "qwen3" -> ["Qwen3DSparkModel"] (#52197) + // 2. neither "Qwen3DSparkModel" nor "Gemma4DSparkModel" -> the DeepSeek-V4 + // rewrite (model_type "deepseek_v4", architectures ["DSparkDraftModel"]) + // 3. "Gemma4DSparkModel" -> key normalization only; the architecture stands + // + // The order is the whole of #52197: branch 1 is a guard in front of a + // catch-all, so the specific case is claimed before the general one rewrites + // it. + // + // TWO TRACKED DIVERGENCES, and both are deliberate. + // + // DIVERGENCE 1, at branch 2. Upstream rewrites the config and lets the + // DeepSeek-V4 model path take it. That path does not exist here: + // DeepseekV4Model is a stub that fails a VT_CHECK + // (src/vllm/model_executor/models/deepseek_v4_registry.cpp) and the lane needs + // two Sparks. Mirroring the rewrite would send the draft into the stub and + // fail on an internal check instead of naming the missing arm, so branch 2 + // REFUSES by name — which is what AGENTS.md requires of an unimplemented arm. + // .agents/specs/dspark-qwen3-routing.md §7 R2 records the decision. + // + // DIVERGENCE 2, at branch 3. Upstream leaves a Gemma4 draft's architecture + // ALONE — "Gemma4DSparkModel" stands and only its keys are normalized + // (speculative.py:945-961) — because upstream has a Gemma4 DSpark model class + // to dispatch to. This engine does not: LoadQwen3DSpark is the one DSpark + // draft lane, and both published layouts load through it + // (src/vllm/entrypoints/model_loader.cpp::LoadDsparkDraft). So branch 3 + // COLLAPSES onto "Qwen3DSparkModel" together with branch 1, and this function + // answers exactly one lane or throws. The collapse is what makes a + // lane-dispatch guard at the call site dead code today; it is undone by the + // change that lands a second lane, not before. + // .agents/specs/dspark-qwen3-routing.md §3 designs it this way. + static std::string ResolveDsparkArchitecture( + const std::vector& architectures, + const std::string& model_type) { + bool has_draft_model = false; + bool has_qwen3 = false; + bool has_gemma4 = false; + for (const std::string& arch : architectures) { + if (arch == "DSparkDraftModel") has_draft_model = true; + if (arch == "Qwen3DSparkModel") has_qwen3 = true; + if (arch == "Gemma4DSparkModel") has_gemma4 = true; + } + if (has_draft_model && model_type == "qwen3") { + return "Qwen3DSparkModel"; // #52197 hunk 2, the leading branch + } + if (!has_qwen3 && !has_gemma4) { + std::string listed; + for (const std::string& arch : architectures) { + if (!listed.empty()) listed += ", "; + listed += "\"" + arch + "\""; + } + throw std::invalid_argument( + "speculative-config: this DSpark draft routes to the DeepSeek-V4 " + "DSpark lane, which is not implemented here. Upstream rewrites a " + "draft whose architectures name neither \"Qwen3DSparkModel\" nor " + "\"Gemma4DSparkModel\" onto model_type \"deepseek_v4\" and " + "architectures [\"DSparkDraftModel\"] " + "(vllm/config/speculative.py:934-944 @ 555967922), and this engine " + "carries only a DeepseekV4Model stub for that lane, which also needs " + "two Sparks. Got architectures [" + + listed + "] with model_type \"" + model_type + + "\". Owed by row SPEC-DSPARK-QWEN3-ROUTING " + "(.agents/specs/dspark-qwen3-routing.md)."); + } + return "Qwen3DSparkModel"; + } + // ResolveDspark: build the scheduler-facing SpeculativeConfig for a DSpark // semi-autoregressive BLOCK draft (SPEC-DSPARK W1). Mirrors the DSpark path of // speculative.py __post_init__ exactly: diff --git a/src/vllm/entrypoints/model_loader.cpp b/src/vllm/entrypoints/model_loader.cpp index 377cd93a6..c57c85291 100644 --- a/src/vllm/entrypoints/model_loader.cpp +++ b/src/vllm/entrypoints/model_loader.cpp @@ -450,6 +450,57 @@ vllm::HfConfig MakeDsparkDraftConfig(const nlohmann::json& c) { return cfg; } +// SPEC-DSPARK-QWEN3-ROUTING (#1193): the two keys upstream classifies a DSpark +// draft by — `architectures` and `model_type` (speculative.py:882-887 and +// :934-944 @ 555967922, plus vllm-project/vllm#52197). Nothing else is read. +struct DsparkDraftIdentity { + std::vector architectures; + std::string model_type; +}; + +// Read them off the draft's config.json, or nullopt when there is no config.json +// to read. A GGUF draft (ResolveDflashDraftDir hands back the .gguf file itself) +// and an HF repo id that is not in the local cache both land there, and both +// already have their own precise error further down the load; refusing them HERE +// would replace "draft checkpoint not found" with a classification failure. +std::optional ReadDsparkDraftIdentity(const std::string& path) { + std::error_code ec; + const fs::path cfg = fs::path(ResolveDflashDraftDir(path)) / "config.json"; + if (!fs::exists(cfg, ec)) return std::nullopt; + std::ifstream f(cfg.string()); + nlohmann::json doc; + try { + f >> doc; + } catch (const nlohmann::json::exception&) { + return std::nullopt; // LoadDsparkDraft parses it again and reports this + } + if (!doc.is_object()) return std::nullopt; + // Classify the SAME document LoadDsparkDraft will load: the speculators layout + // carries no top-level `architectures`, and its translation writes + // ["Qwen3DSparkModel"] (qwen3_dspark.cpp, update_dspark). + if (vllm::Qwen3DSparkModel::IsSpeculatorsDsparkConfig(doc)) { + doc = vllm::Qwen3DSparkModel::TranslateSpeculatorsDsparkConfig(doc); + } + DsparkDraftIdentity id; + if (doc.contains("architectures") && doc.at("architectures").is_array()) { + for (const nlohmann::json& a : doc.at("architectures")) { + if (a.is_string()) id.architectures.push_back(a.get()); + } + } + // A config that DECLARES no architecture is not classified at all. Upstream + // reads the key off a HuggingFace `ModelConfig`, where an absent key is `[]`, + // and its catch-all would send that empty list to DeepSeek-V4. Refusing on it + // here would refuse a draft on the ABSENCE of evidence, and the native + // `deepseek-ai/dspark_qwen3_*_block7` layouts have not been read on this host + // to confirm they declare it. The narrowing is deliberate and is recorded + // under `## Owed` in .agents/specs/dspark-qwen3-routing.md. + if (id.architectures.empty()) return std::nullopt; + if (doc.contains("model_type") && doc.at("model_type").is_string()) { + id.model_type = doc.at("model_type").get(); + } + return id; +} + // The two DSpark resolution keys, read off the draft checkpoint's own // config.json (SPEC-DSPARK-BLOCK-SIZE-GUARD, #1225). struct DsparkDraftKeys { @@ -974,6 +1025,59 @@ std::optional LoadedEngine::ResolveSpecConfig( "speculative-config: method \"dspark\" requires num_speculative_tokens " "(a DSpark draft config carries no n_predict)"); } + // SPEC-DSPARK-QWEN3-ROUTING (#1193): classify the draft by its OWN config + // before resolving anything else. Upstream picks the DSpark lane from the + // draft's architectures and model_type (speculative.py:882-887 and :934-944 + // @ 555967922, plus vllm#52197); this engine picked it from the CLI method + // string alone, so a `DSparkDraftModel` checkpoint loaded as a Qwen3 draft by + // OMISSION rather than by decision, and a DeepSeek-V4 one loaded far enough + // to fail on a missing key. This is the production caller + // `SpeculativeConfig::IsDsparkDraft` lacked. + if (cli.draft_model_path.has_value()) { + const std::optional ident = + ReadDsparkDraftIdentity(*cli.draft_model_path); + if (ident.has_value()) { + std::string listed; + for (const std::string& arch : ident->architectures) { + if (!listed.empty()) listed += ", "; + listed += "\"" + arch + "\""; + } + // Upstream's detection (speculative.py:882-887 + #52197 hunk 1). A draft + // that fails it is precisely the set upstream's fallback rewrites into + // the DeepSeek-V4 lane, so it is refused with that lane named. + if (!vllm::SpeculativeConfig::IsDsparkDraft( + *cli.draft_model_path, ident->architectures, ident->model_type)) { + throw std::invalid_argument( + "speculative-config: the draft checkpoint at \"" + + *cli.draft_model_path + + "\" does not identify as a Qwen3 or Gemma4 DSpark draft: its " + "model id carries no \"dspark\", and its architectures [" + + listed + "] with model_type \"" + ident->model_type + + "\" name none of \"Qwen3DSparkModel\", \"Gemma4DSparkModel\" or " + "the \"DSparkDraftModel\" + \"qwen3\" pair " + "(vllm/config/speculative.py:882-887 @ 555967922 + " + "vllm-project/vllm#52197). Upstream routes exactly this set into " + "the DeepSeek-V4 DSpark lane (:934-944), and that lane is not " + "implemented here: DeepseekV4Model is a stub and it needs two " + "Sparks. Owed by row SPEC-DSPARK-QWEN3-ROUTING " + "(.agents/specs/dspark-qwen3-routing.md)."); + } + // Upstream's normalization (#52197 hunk 2), called for its REFUSAL. At + // this pin `SpeculativeConfig::ResolveDsparkArchitecture` is TOTAL over + // its two outcomes: it answers "Qwen3DSparkModel" or it throws the + // DeepSeek-V4 refusal by name. So the returned lane is always the one + // `LoadDsparkDraft` implements, and a `lane != "Qwen3DSparkModel"` guard + // here would be a branch nothing can enter — dead code, which the earlier + // shape of this call site carried and a mutation caught. When a further + // upstream lane arrives (#52197's own context already carries a + // `K3DSparkModel` arm absent from this pin), the returned value becomes a + // decision and the dispatch lands WITH the lane that needs it; the + // pin-advance item under `## Owed` in + // .agents/specs/dspark-qwen3-routing.md carries that. + vllm::SpeculativeConfig::ResolveDsparkArchitecture(ident->architectures, + ident->model_type); + } + } vllm::SpeculativeConfig resolved = vllm::SpeculativeConfig::ResolveDspark( keys.n_predict, keys.block_floor, cli.num_speculative_tokens, keys.block_floor_key); diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index 36555d942..2bbecde9f 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -1139,6 +1139,11 @@ vllm_cpp_add_test(test_speculative_draft_max_position_embeddings # SPEC-DSPARK W1: the DSpark config slice (method acceptance, block-size floor, # lookahead slots, draft identity). vllm_cpp_add_test(test_speculative_dspark vllm/config/test_speculative_dspark.cpp) +# SPEC-DSPARK-QWEN3-ROUTING (#1193): REACHABILITY of that classification — the +# loader's own ResolveSpecConfig reads the draft's config.json and refuses the +# DeepSeek-V4 DSpark lane by name, instead of trusting the CLI method string. +vllm_cpp_add_test(test_dspark_draft_routing + vllm/entrypoints/test_dspark_draft_routing.cpp) # #1160: `--speculative-config` key ADMISSION — an unknown or misspelled key is # refused by name, and an upstream key this engine does not implement is refused # with the missing part named, instead of being silently dropped. diff --git a/tests/vllm/config/test_speculative_dspark.cpp b/tests/vllm/config/test_speculative_dspark.cpp index cb69f1616..9bcb730e1 100644 --- a/tests/vllm/config/test_speculative_dspark.cpp +++ b/tests/vllm/config/test_speculative_dspark.cpp @@ -140,3 +140,77 @@ TEST_CASE("DSpark method resolution from the draft checkpoint identity") { CHECK_FALSE(SpeculativeConfig::IsDsparkDraft("z-lab/Qwen3.6-27B-DFlash", {"DFlashDraftModel"})); } + +// ─── SPEC-DSPARK-QWEN3-ROUTING (#1193) ─────────────────────────────────────── +// +// BEYOND-PIN. Both cases below mirror vllm-project/vllm#52197, merged +// 2026-08-17 at 7075ddac28c25d4fd2b84bc2a9a6c5ffde0345c8, which is AHEAD of the +// pin 555967922. The PR ships no test of its own — its "Test Plan" is empty and +// its "Test Result" is a pasted gsm8k serve log — so there is nothing to port +// one-for-one and these are written here. .agents/porting.md requires the +// adaptation to be recorded, and this paragraph is that record; +// .agents/specs/dspark-qwen3-routing.md §2 carries the pin decision. +// +// The REACHABILITY of both functions — that the loader calls them at all — is +// tests/vllm/entrypoints/test_dspark_draft_routing.cpp, because a case here +// proves the predicate works and never that anything reaches it. + +TEST_CASE("a DSparkDraftModel draft with model_type qwen3 IS a DSpark draft") { + // vllm#52197 hunk 1. The model id carries no "dspark" substring, so only the + // architecture arm can answer: with the id spelled the other way the case + // would pass on speculative.py:882 alone and prove nothing about the + // architecture list — the mute switch this test set has to avoid. + CHECK(SpeculativeConfig::IsDsparkDraft("RadixArk/Qwen3.8-27B-Draft", + {"DSparkDraftModel"}, "qwen3")); + // The PAIR is the condition. The same architecture string names a DeepSeek-V4 + // draft under any other model_type, and upstream's leading branch does not + // claim it. + CHECK_FALSE(SpeculativeConfig::IsDsparkDraft("RadixArk/Qwen3.8-27B-Draft", + {"DSparkDraftModel"}, + "deepseek_v4")); + // And the pinned two-argument question keeps the pinned answer. + CHECK_FALSE( + SpeculativeConfig::IsDsparkDraft("RadixArk/Qwen3.8-27B-Draft", + {"DSparkDraftModel"})); +} + +TEST_CASE("DSpark architecture normalization routes the three implemented names") { + // vllm#52197 hunk 2 branch 1, and speculative.py:934-944's two survivors. All + // three answer with the one draft lane this engine implements + // (LoadQwen3DSpark), which is what upstream's update_arch_() writes back onto + // the draft config. + CHECK(SpeculativeConfig::ResolveDsparkArchitecture({"DSparkDraftModel"}, + "qwen3") == + "Qwen3DSparkModel"); + CHECK(SpeculativeConfig::ResolveDsparkArchitecture({"Qwen3DSparkModel"}, + "qwen3") == + "Qwen3DSparkModel"); + CHECK(SpeculativeConfig::ResolveDsparkArchitecture({"Gemma4DSparkModel"}, + "gemma4") == + "Qwen3DSparkModel"); +} + +TEST_CASE("DSpark architecture normalization REFUSES the DeepSeek-V4 fallback") { + // The one tracked divergence from speculative.py:940-944 (§7 R2): upstream + // rewrites the config onto model_type "deepseek_v4" and lets the DeepSeek-V4 + // path take it; we have only a stub for that path, so the arm is refused with + // the missing part NAMED, as AGENTS.md requires. + std::string message; + try { + SpeculativeConfig::ResolveDsparkArchitecture({"DSparkDraftModel"}, + "deepseek_v4"); + } catch (const std::invalid_argument& e) { + message = e.what(); + } + REQUIRE_FALSE(message.empty()); + CHECK(message.find("DeepSeek-V4") != std::string::npos); + CHECK(message.find("not implemented") != std::string::npos); + + // The catch-all is upstream's, so an architecture list that names nothing + // DSpark at all lands in the same refusal rather than loading as a Qwen3 + // draft. + CHECK_THROWS_AS( + SpeculativeConfig::ResolveDsparkArchitecture({"DeepseekV4ForCausalLM"}, + "deepseek_v4"), + std::invalid_argument); +} diff --git a/tests/vllm/entrypoints/test_dspark_draft_routing.cpp b/tests/vllm/entrypoints/test_dspark_draft_routing.cpp new file mode 100644 index 000000000..c9e229a5e --- /dev/null +++ b/tests/vllm/entrypoints/test_dspark_draft_routing.cpp @@ -0,0 +1,460 @@ +// `SPEC-DSPARK-QWEN3-ROUTING` (issue #1193) — REACHABILITY of the DSpark draft +// classification, not the predicate that performs it. +// +// WHY THIS FILE EXISTS SEPARATELY. `tests/vllm/config/test_speculative_dspark.cpp` +// proves `SpeculativeConfig::IsDsparkDraft` and +// `SpeculativeConfig::ResolveDsparkArchitecture` answer correctly. Before this row +// every reference to `IsDsparkDraft` outside its own header was in that file: +// `ResolveSpecConfig` branched on the CLI method string alone, so nothing shipped +// classified a draft by its own config, and a `DSparkDraftModel` checkpoint loaded +// as a Qwen3 DSpark draft by OMISSION rather than by decision +// (.agents/specs/dspark-qwen3-routing.md §0). AGENTS.md `## Nothing lands dead` +// requires the smallest failing test to enter through a production entry point, and +// a unit test on the predicate is exactly the unit test that rule excludes. +// +// THE PRODUCTION ENTRY POINT is the `LoadedEngine` CONSTRUCTOR, which resolves the +// speculative config the caller passed in `EngineParams` +// (`resolved_spec_config_(ResolveSpecConfig(params, config_))`, +// src/vllm/entrypoints/model_loader.cpp). `ResolveSpecConfig` is private, and a +// test that called it would prove the function works rather than that anything +// reaches it — the same argument `test_loaded_engine_dense.cpp` makes for +// `ResolveNumBlocks`. The chain under test is therefore +// `EngineParams::speculative_config` -> LoadedEngine ctor -> ResolveSpecConfig -> +// IsDsparkDraft / ResolveDsparkArchitecture, entered with a real draft DIRECTORY on +// disk, which is what the classification reads. The synthetic dense model is the +// TARGET and is irrelevant to the decision: the dspark branch reads the draft's +// config.json and never the target's. +// +// THE REACHABILITY MUTATION for this row deletes the classification block in +// `ResolveSpecConfig` and requires this suite RED. +#include "vllm/entrypoints/model_loader.h" + +#include + +#include +#include +#include +#include +#include +#include +#include +#include + +#include + +#include "support/process_id.h" +#include "vllm/config/speculative.h" +#include "vllm/model_executor/models/qwen3_5_dense.h" +#include "vllm/tokenizer/bpe.h" +#include "vllm/tokenizer/tokenizer.h" +#include "vllm/transformers_utils/hf_config.h" +#include "vt/dtype.h" + +using nlohmann::json; +using vllm::DenseMlpWeights; +using vllm::HfConfig; +using vllm::OwnedTensor; +using vllm::Qwen3_5DenseLayerWeights; +using vllm::Qwen3_5DenseWeights; +using vllm::entrypoints::EngineParams; +using vllm::entrypoints::LoadedEngine; +using vllm::tok::MapBytesToUnicode; +using vllm::tok::Tokenizer; +using vt::DType; + +namespace fs = std::filesystem; + +namespace { + +// ─── Synthetic weights (mirrors test_qwen27_paged_forward.cpp) ─────────────── +uint64_t Mix(uint64_t x) { + x += 0x9E3779B97F4A7C15ULL; + x = (x ^ (x >> 30)) * 0xBF58476D1CE4E5B9ULL; + x = (x ^ (x >> 27)) * 0x94D049BB133111EBULL; + return x ^ (x >> 31); +} +float RandV(uint64_t seed) { + const double u = + static_cast(Mix(seed) >> 40) / static_cast(1 << 24); + return static_cast(u * 0.16 - 0.08); +} +OwnedTensor MakeOwned(DType dt, std::vector shape, uint64_t seed) { + OwnedTensor t; + t.dtype = dt; + t.rank = static_cast(shape.size()); + int64_t n = 1; + for (int i = 0; i < t.rank; ++i) { + t.shape[i] = shape[static_cast(i)]; + n *= shape[static_cast(i)]; + } + if (dt == DType::kBF16) { + t.bytes.resize(static_cast(n) * 2); + auto* p = reinterpret_cast(t.bytes.data()); + for (int64_t i = 0; i < n; ++i) + p[i] = vt::F32ToBF16(RandV(seed + static_cast(i))); + } else { + t.bytes.resize(static_cast(n) * 4); + auto* p = reinterpret_cast(t.bytes.data()); + for (int64_t i = 0; i < n; ++i) p[i] = RandV(seed + static_cast(i)); + } + return t; +} + +// Vocab == the tiny BPE fixture's assigned ids (0..23), block_size == +// max_model_len == hash_block_size (hybrid coordinator constraint; prompts far +// shorter than a block keep prefix caching inert), matching test_llm_engine.cpp. +constexpr int kVocab = 24; +constexpr int kMaxModelLen = 32; + +// 27B-shaped small DENSE config: layer_types [LA, LA, LA, FA], no experts, +// GQA ratio 3 (Hv/Hk = 6/2), attn_output_gate. num_experts==0 => dense arch. +HfConfig MakeDenseConfig() { + HfConfig c; + c.model_type = "qwen3_5_text"; + c.architectures = {"Qwen3_5ForConditionalGeneration"}; + c.hidden_size = 32; + c.num_hidden_layers = 4; + c.vocab_size = kVocab; + c.num_attention_heads = 6; + c.num_key_value_heads = 2; + c.head_dim = 8; + c.layer_types = {"linear_attention", "linear_attention", "linear_attention", + "full_attention"}; + c.intermediate_size = 16; + c.num_experts = 0; + c.linear_num_key_heads = 2; + c.linear_num_value_heads = 6; // GQA ratio 3 + c.linear_key_head_dim = 8; + c.linear_value_head_dim = 8; + c.linear_conv_kernel_dim = 4; + c.rope_theta = 10000.0; + c.rotary_dim = 4; + c.rms_norm_eps = 1e-6; + c.max_position_embeddings = kMaxModelLen; + c.raw = json::object(); // no eos_token_id -> generation runs to max_tokens. + return c; +} + +DenseMlpWeights MakeMlp(const HfConfig& c, uint64_t s) { + DenseMlpWeights m; + const int64_t H = c.hidden_size, I = c.intermediate_size; + m.gate_proj = MakeOwned(DType::kBF16, {H, I}, s + 1); + m.up_proj = MakeOwned(DType::kBF16, {H, I}, s + 2); + m.down_proj = MakeOwned(DType::kBF16, {I, H}, s + 3); + return m; +} + +Qwen3_5DenseWeights MakeDenseWeights(const HfConfig& c) { + Qwen3_5DenseWeights w; + const int64_t H = c.hidden_size, V = c.vocab_size; + const int64_t Hq = c.num_attention_heads, Hkv = c.num_key_value_heads, + Dh = c.head_dim; + const int64_t Hk = c.linear_num_key_heads, Hv = c.linear_num_value_heads, + Dk = c.linear_key_head_dim, Dv = c.linear_value_head_dim, + Kw = c.linear_conv_kernel_dim; + const int64_t key_dim = Hk * Dk, value_dim = Hv * Dv, + conv_dim = 2 * key_dim + value_dim; + w.embed_tokens = MakeOwned(DType::kBF16, {V, H}, 11); + w.final_norm = MakeOwned(DType::kBF16, {H}, 12); + w.lm_head = MakeOwned(DType::kBF16, {H, V}, 13); + for (int64_t l = 0; l < c.num_hidden_layers; ++l) { + const uint64_t s = 1000 + static_cast(l) * 5000; + Qwen3_5DenseLayerWeights lw; + lw.is_linear_attention = + (c.layer_types[static_cast(l)] == "linear_attention"); + lw.input_layernorm = MakeOwned(DType::kBF16, {H}, s + 1); + lw.post_attention_layernorm = MakeOwned(DType::kBF16, {H}, s + 2); + if (lw.is_linear_attention) { + lw.gdn.in_proj_qkv = MakeOwned(DType::kBF16, {H, conv_dim}, s + 10); + lw.gdn.in_proj_z = MakeOwned(DType::kBF16, {H, value_dim}, s + 20); + lw.gdn.in_proj_b = MakeOwned(DType::kBF16, {H, Hv}, s + 30); + lw.gdn.in_proj_a = MakeOwned(DType::kBF16, {H, Hv}, s + 40); + lw.gdn.conv1d_weight = MakeOwned(DType::kBF16, {conv_dim, Kw}, s + 50); + lw.gdn.a_log = MakeOwned(DType::kF32, {Hv}, s + 60); + lw.gdn.dt_bias = MakeOwned(DType::kF32, {Hv}, s + 70); + lw.gdn.norm_weight = MakeOwned(DType::kBF16, {Dv}, s + 80); + lw.gdn.out_proj = MakeOwned(DType::kBF16, {value_dim, H}, s + 90); + } else { + lw.attn.q_proj = MakeOwned(DType::kBF16, {H, 2 * Hq * Dh}, s + 10); + lw.attn.k_proj = MakeOwned(DType::kBF16, {H, Hkv * Dh}, s + 20); + lw.attn.v_proj = MakeOwned(DType::kBF16, {H, Hkv * Dh}, s + 30); + lw.attn.o_proj = MakeOwned(DType::kBF16, {Hq * Dh, H}, s + 40); + lw.attn.q_norm = MakeOwned(DType::kBF16, {Dh}, s + 50); + lw.attn.k_norm = MakeOwned(DType::kBF16, {Dh}, s + 60); + } + lw.mlp = MakeMlp(c, s + 500); + w.layers.push_back(std::move(lw)); + } + return w; +} + +// The tiny oracle-verified BPE fixture (ids 0..23, no holes) from test_llm_engine. +Tokenizer BuildFixture() { + static int counter = 0; + const std::string path = + (std::filesystem::temp_directory_path() / + ("vllm_dense_engine_tok_" + std::to_string(counter++) + ".json")) + .string(); + json doc; + doc["version"] = "1.0"; + doc["added_tokens"] = json::array( + {{{"id", 19}, {"content", "<|end|>"}, {"special", true}}, + {{"id", 20}, {"content", ""}, {"special", false}}, + {{"id", 21}, {"content", "<|end|>of"}, {"special", true}}}); + doc["normalizer"] = nullptr; + doc["pre_tokenizer"] = { + {"type", "Sequence"}, + {"pretokenizers", + json::array( + {{{"type", "Split"}, + {"pattern", + {{"Regex", + R"((?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\r\n\p{L}\p{N}]?[\p{L}\p{M}]+|\p{N}| ?[^\s\p{L}\p{M}\p{N}]+[\r\n]*|\s*[\r\n]+|\s+(?!\S)|\s+)"}}}, + {"behavior", "Isolated"}, + {"invert", false}}, + {{"type", "ByteLevel"}, + {"add_prefix_space", false}, + {"trim_offsets", false}, + {"use_regex", false}}})}}; + json vocab = {{"h", 0}, {"e", 1}, {"l", 2}, {"o", 3}, {"w", 4}, + {"r", 5}, {"d", 6}, {"Ġ", 7}, {"1", 8}, {"2", 9}, + {"ll", 10}, {"he", 11}, {"llo", 12}, {"hello", 13}, + {"Ġw", 14}, {"or", 15}, {"orld", 16}, {"Ġworld", 17}, + {"ld", 18}}; + vocab[MapBytesToUnicode("\xF0\x9F")] = 22; + vocab[MapBytesToUnicode("\x8C\x8D")] = 23; + doc["model"] = { + {"type", "BPE"}, + {"ignore_merges", false}, + {"vocab", vocab}, + {"merges", + json::array({json::array({"l", "l"}), json::array({"h", "e"}), + json::array({"ll", "o"}), json::array({"he", "llo"}), + json::array({"Ġ", "w"}), json::array({"o", "r"}), + json::array({"l", "d"}), json::array({"or", "ld"}), + json::array({"Ġw", "orld"})})}}; + std::ofstream(path, std::ios::binary) << doc.dump(); + Tokenizer tok = Tokenizer::FromHfJson(path); + std::remove(path.c_str()); + return tok; +} + +Tokenizer FreshFixture() { return BuildFixture(); } + +// ─── The draft checkpoints the classification reads ───────────────────────── + +// A draft checkpoint directory carrying just `config.json`. The classification +// reads nothing else, and no draft weight is ever opened on this path. +class DraftDir { + public: + DraftDir(const std::string& leaf, const std::string& config_json) { + // The shared portable process id, so two concurrent runs cannot collide and + // the file does not reintroduce the ::getpid() include class this tree just + // fixed (tests/support/process_id.h). + dir_ = fs::temp_directory_path() / + ("vllm-cpp-" + leaf + "-" + std::to_string(vllm_test::ProcessId())); + fs::remove_all(dir_); + fs::create_directories(dir_); + std::ofstream(dir_ / "config.json") << config_json; + } + ~DraftDir() { + std::error_code ec; + fs::remove_all(dir_, ec); + } + DraftDir(const DraftDir&) = delete; + DraftDir& operator=(const DraftDir&) = delete; + std::string path() const { return dir_.string(); } + + private: + fs::path dir_; +}; + +// The instrument's own precondition. `IsDsparkDraft` answers TRUE for any model +// id whose lowercase form contains "dspark", so a temporary directory that +// happened to carry the substring would make the ARCHITECTURE arm unobservable — +// the mute-switch shape .agents/specs/dspark-qwen3-routing.md §5 names as this +// test set's one trap. +bool CarriesDsparkSubstring(const std::string& path) { + std::string lowered = path; + for (char& c : lowered) { + c = static_cast(std::tolower(static_cast(c))); + } + return lowered.find("dspark") != std::string::npos; +} + +// RadixArk/Qwen3.8-27B-DSpark @ 85ef153be924f17ce4bf62726954eeaa4a73e854, the +// checkpoint .agents/specs/dspark-qwen3-routing.md §7 R1 pins, reduced to the two +// keys the classification reads. `architectures` is the post-vllm#52197 pair. +constexpr const char* kQwen3DraftModelConfig = + R"({"architectures":["DSparkDraftModel"],"model_type":"qwen3",)" + R"("block_size":7,"num_hidden_layers":5})"; + +// The same architecture string with the DeepSeek-V4 model_type. Upstream's +// fallback (vllm/config/speculative.py:934-944 @ 555967922) rewrites exactly this +// config into the DeepSeek-V4 DSpark lane, which this engine does not implement. +constexpr const char* kDeepseekV4DraftConfig = + R"({"architectures":["DSparkDraftModel"],"model_type":"deepseek_v4",)" + R"("num_hidden_layers":5})"; + +// A draft that DECLARES no architecture at all. Upstream reads the key off a +// HuggingFace ModelConfig, where an absent key is `[]`, and its catch-all would +// send that to DeepSeek-V4. This engine does not classify on the ABSENCE of +// evidence, because the native `deepseek-ai/dspark_qwen3_*_block7` layouts have +// not been read here to confirm they declare it, and refusing them would break a +// lane that loads today. +constexpr const char* kNoArchitecturesDraftConfig = + R"({"model_type":"qwen3","block_size":7,"num_hidden_layers":5})"; + +// The native name, routed since SPEC-DSPARK W5. The regression guard: the +// classification must not refuse the lane that already loads. +constexpr const char* kNativeQwen3DSparkConfig = + R"({"architectures":["Qwen3DSparkModel"],"model_type":"qwen3",)" + R"("block_size":7,"num_hidden_layers":5})"; + +// The GEMMA4 name, with its own model_type. This is BRANCH 3 of +// `SpeculativeConfig::ResolveDsparkArchitecture`: upstream leaves +// `Gemma4DSparkModel` in place and normalizes only its keys, because upstream has +// a Gemma4 DSpark class to dispatch to; this engine has one DSpark draft lane, so +// the branch COLLAPSES onto "Qwen3DSparkModel". That collapse is the SECOND +// tracked divergence recorded in .agents/specs/dspark-qwen3-routing.md +// `## Outcome`, and until this case it was exercised only by the hand-called unit +// case in tests/vllm/config/test_speculative_dspark.cpp — the shape AGENTS.md +// `## Nothing lands dead` excludes, because it proves the function works rather +// than that anything reaches it. +constexpr const char* kGemma4DSparkConfig = + R"({"architectures":["Gemma4DSparkModel"],"model_type":"gemma4",)" + R"("block_size":7,"num_hidden_layers":5})"; + +// The SPECULATORS layout, the second published shape `LoadDsparkDraft` accepts. +// It declares NO top-level `architectures` at all — `speculators_model_type` +// identifies it, and `Qwen3DSparkModel::TranslateSpeculatorsDsparkConfig` writes +// `["Qwen3DSparkModel"]` while translating (algos.py update_dspark). So it is the +// one layout for which the no-architecture narrowing above and the classification +// disagree, and which of the two answers it gets depends on the translation +// running FIRST. Keys reduced to what the translation asserts on. +constexpr const char* kSpeculatorsDsparkConfig = + R"({"speculators_model_type":"dspark","aux_hidden_state_layer_ids":[5,17,29],)" + R"("speculators_config":{"proposal_methods":[{"speculative_tokens":7}]},)" + R"("block_size":7,"mask_token_id":151665,)" + R"("transformer_layer_config":{"model_type":"qwen3","num_hidden_layers":5}})"; + +EngineParams DsparkParams(const std::string& draft_dir, int k) { + EngineParams params; + vllm::SpeculativeConfig cli; + cli.method = "dspark"; + cli.draft_model_path = draft_dir; + cli.num_speculative_tokens = k; + params.speculative_config = cli; + return params; +} + +// Build one engine through the public constructor and return the refusal text it +// produced, or "" when construction got PAST the classification. Returning the +// empty string rather than asserting keeps the RED legible: before this row the +// loader classified nothing, so the refusal is ABSENT, not wrong. +std::string RefusalForDraft(const std::string& draft_dir) { + const HfConfig c = MakeDenseConfig(); + try { + LoadedEngine eng(c, MakeDenseWeights(c), FreshFixture(), + DsparkParams(draft_dir, 7)); + } catch (const std::invalid_argument& e) { + return e.what(); + } + return ""; +} + +} // namespace + +TEST_CASE("loader refuses a DeepSeek-V4 DSpark draft by name") { + const DraftDir draft("draft-v4", kDeepseekV4DraftConfig); + REQUIRE_FALSE(CarriesDsparkSubstring(draft.path())); + + const std::string message = RefusalForDraft(draft.path()); + REQUIRE_FALSE(message.empty()); // RED before this row: nothing refuses. + // AGENTS.md: an unimplemented arm refuses with a message that NAMES the missing + // part. "DeepSeek-V4" is that name; without it the user is sent hunting. + CHECK(message.find("DeepSeek-V4") != std::string::npos); + CHECK(message.find("not implemented") != std::string::npos); + // The ARM, not just the lane. Both refusals on this path name DeepSeek-V4 and + // say "not implemented", so those two substrings alone cannot tell them apart — + // and a test that cannot tell them apart cannot see the identity check + // disappear. This case must reach `SpeculativeConfig::IsDsparkDraft`'s refusal, + // whose wording is its own. + CHECK(message.find("does not identify as") != std::string::npos); +} + +TEST_CASE("loader refuses a DeepSeek-V4 draft whose id DOES carry \"dspark\"") { + // The other arm of the same refusal. Here the model id alone makes + // `IsDsparkDraft` answer true (the "dspark" substring, + // vllm/config/speculative.py:882), so the DeepSeek-V4 lane has to be caught by + // the ARCHITECTURE normalization rather than by the identity check. Both arms + // must name the same missing part. + const DraftDir draft("dspark-v4", kDeepseekV4DraftConfig); + REQUIRE(CarriesDsparkSubstring(draft.path())); + + const std::string message = RefusalForDraft(draft.path()); + REQUIRE_FALSE(message.empty()); // RED before this row: nothing refuses. + CHECK(message.find("DeepSeek-V4") != std::string::npos); + CHECK(message.find("not implemented") != std::string::npos); + // The other arm's own wording, for the reason given in the case above: this + // draft must be refused by `SpeculativeConfig::ResolveDsparkArchitecture`, + // having PASSED the identity check on its model id. + CHECK(message.find("routes to the DeepSeek-V4 DSpark lane") != std::string::npos); +} + +TEST_CASE("loader admits a DSparkDraftModel + qwen3 draft") { + // The routing this row exists for, and the case that goes RED if the + // `DSparkDraftModel` + "qwen3" pair is reverted out of `IsDsparkDraft` (gate + // G3). The model id deliberately carries no "dspark" substring, so ONLY the + // architecture arm can answer here. + const DraftDir draft("draft-qwen3", kQwen3DraftModelConfig); + REQUIRE_FALSE(CarriesDsparkSubstring(draft.path())); + + CHECK(RefusalForDraft(draft.path()).empty()); +} + +TEST_CASE("loader still admits the native Qwen3DSparkModel draft") { + const DraftDir draft("draft-native", kNativeQwen3DSparkConfig); + CHECK(RefusalForDraft(draft.path()).empty()); +} + +TEST_CASE("loader admits a Gemma4DSparkModel draft through the collapsed lane") { + // Branch 3 of `ResolveDsparkArchitecture`, reached from the production entry + // point rather than by hand. Two things are asserted at once: the Gemma4 name + // is ADMITTED (it must not fall into the DeepSeek-V4 refusal, which is what a + // `!has_qwen3 && !has_gemma4` guard reverted to `!has_qwen3` would do), and it + // is admitted onto the ONE lane this engine implements. The directory name + // carries no "dspark", so `IsDsparkDraft`'s model-id arm cannot answer for it + // and only the architecture arm can — the mute switch this file's + // `CarriesDsparkSubstring` precondition exists to keep shut. + const DraftDir draft("draft-gemma4", kGemma4DSparkConfig); + REQUIRE_FALSE(CarriesDsparkSubstring(draft.path())); + + CHECK(RefusalForDraft(draft.path()).empty()); +} + +TEST_CASE("loader does not classify a draft that declares no architecture") { + // The narrowing recorded beside `ReadDsparkDraftIdentity`. The directory name + // carries "dspark", exactly as the published native drafts' repo ids do, so + // `IsDsparkDraft` answers true and the ONLY thing standing between this draft + // and the DeepSeek-V4 refusal is the empty-list guard. + const DraftDir draft("dspark-noarch", kNoArchitecturesDraftConfig); + REQUIRE(CarriesDsparkSubstring(draft.path())); + + CHECK(RefusalForDraft(draft.path()).empty()); +} + +TEST_CASE("loader classifies a Speculators-layout DSpark draft as the Qwen3 lane") { + // The gap this case closes: nothing drove the second published layout through + // the new classification. It matters because that layout reaches + // `ReadDsparkDraftIdentity` with no `architectures` key of its own, so the + // empty-list narrowing would skip it — unless the speculators translation runs + // before the key is read, which is what + // `src/vllm/entrypoints/model_loader.cpp::ReadDsparkDraftIdentity` does. The + // directory name deliberately carries no "dspark", so the ONLY thing that can + // admit this draft is the translated architecture. + const DraftDir draft("draft-spec-layout", kSpeculatorsDsparkConfig); + REQUIRE_FALSE(CarriesDsparkSubstring(draft.path())); + + CHECK(RefusalForDraft(draft.path()).empty()); +}