From 062b5500358116c50b95ad9c65ddb1ac2e36e622 Mon Sep 17 00:00:00 2001 From: Don Mirror Date: Thu, 13 Aug 2026 16:35:35 -0500 Subject: [PATCH 1/2] record(GEMMA4-ROCM-KEEP): measured SharedK-WMMA plateau on 2x R9700 Contributor closeout (PREFIX_CACHE=0, unique pads, 2026-08-13): 2014 t/s @~11k, 1099 t/s @~42k, decode ~55 t/s. Quality Paris/63/tool_calls. Vulkan Q8 on the same box is still ahead; this is the reliable recipe, not that bar. Speculative/ngram/FMHA/layer-split stay off. HIP cm1 spill is a residual, not a named LLVM defect. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Hermes:grok-4.6 [Hermes] --- .agents/specs/gemma4-rocm-fp8-moe.md | 31 ++++++++++++++++++++++++++++ docs/ENVIRONMENT.md | 2 +- docs/FEATURES.md | 2 +- docs/ROCM.md | 7 +++++-- docs/USAGE.md | 10 ++++++++- 5 files changed, 47 insertions(+), 5 deletions(-) diff --git a/.agents/specs/gemma4-rocm-fp8-moe.md b/.agents/specs/gemma4-rocm-fp8-moe.md index deb6d0a08..351fbe963 100644 --- a/.agents/specs/gemma4-rocm-fp8-moe.md +++ b/.agents/specs/gemma4-rocm-fp8-moe.md @@ -58,3 +58,34 @@ and env-gated MoE paths do not. Revisit as a **separate** PR with CUDA golden. `VT_GEMMA4_MLP_MOE_PARALLEL` is documented but **not** wired in this tip (was only in the deferred layer-loop path). + +## Measured KEEP (2026-08-13, contributor lab) + +Hardware: 2x R9700 gfx1201, ROCm 7.2.4, kernel 7.0.0-29-generic. +Model: Gemma-4-26B-A4B-it FP8. Fair protocol: `PREFIX_CACHE=0` + unique pads. + +Recipe (product): SharedK-WMMA on, FLASH/FMHA/scoreless off, `PREFILL_GEMM_M=2048`, +`PEER_ACT=1`, batch MoE `T>=64`, decode `KV_SPLITS=16` / `SLIDE_SPLITS=8` / +`SLIDE_WARPS=16` / `SPLIT_WARPS=12`. Speculative, ngram, layer-split: **off**. + +| Depth | median prefill t/s | notes | +|------:|-------------------:|-------| +| ~3k | 2112 | 3 reps, Paris | +| ~11k | 2014 | first rep 1170 outlier dropped from claim; median 2014 | +| ~18k | 1705 | 3 reps | +| ~42k | 1099 | 2 reps | + +Decode stream 64 tok: **55.5 t/s** temp=0, **49.1 t/s** temp=0.7. +Quality: Paris, arith `63`, `gemma4` parser `tool_calls`, `/health` `/metrics`. +No engine-fatal / hipError in the closeout log. + +Same-box Vulkan Q8 unique-pad bar: **3503 @11k / 2714 @42k**. KEEP gap ~1.74x / 2.47x. +This row ships the reliable ROCm plateau. It does **not** close that bar. + +### Rejected (do not default-on) + +FMHA_WMMA2 (quality fail + 0.67x @11k). Isolated P1 cm1 wg256 (~1.13x KEEP; need ~3.35x isolated). Layer-split FIFO (~0.60x vs the Q8 bar). Head-TP peer-read (peer BW). hipBLASLt dual-GPU (fatal or slower). + +### Residual compiler / ACO + +Faithful HIP cm1 hsaco on gfx1201: 192 VGPR, 339 spills, 940 B private, vs KEEP SharedK 35 spills / 120 B. Mechanism of the isolated 1.13x, not a named LLVM defect. Avoidability (regalloc vs live-range vs fundamental) is open. Phase-2 ROCm/LLVM rebuild stays blocked until a discriminator names a smallest component and a measurable prediction. A general codegen fix, if one exists, is a separate claim from the 3.35x bar. diff --git a/docs/ENVIRONMENT.md b/docs/ENVIRONMENT.md index 395d1f779..20841ee3a 100644 --- a/docs/ENVIRONMENT.md +++ b/docs/ENVIRONMENT.md @@ -195,7 +195,7 @@ on CUDA/CPU builds beyond the documented behavior. | `VT_GEMMA4_PREFILL_BATCH_MOE` | auto / `1` in lab recipe | `=1` group-by-expert prefill GEMM for `T>=64`; `=0` serial M=1 (slow). Unset = auto | | `VT_GEMMA4_MLP_MOE_PARALLEL` | off | `=1` run Gemma4 MLP and MoE on two HIP streams (lab; wall ~flat on R9700). Not wired in this PR tip (decode-graph-free split) | | `VT_ATTN_PREFILL_FLASH` | off | `=1` SGLang-style BM×BN GQA flash prefill (lab A/B) | -| `VT_GEMMA4_PREFILL_GEMM_M` | `256` | Tokens per expert in prefill-batch GEMM chunks (`16..2048`). Larger M → fewer launches; lab `512` ~+37% prefill vs `64` | +| `VT_GEMMA4_PREFILL_GEMM_M` | `256` | Tokens per expert in prefill-batch GEMM chunks (`16..2048`). Larger M → fewer launches. Lab KEEP on dual R9700 uses `2048` | | `VT_GEMMA4_HOST_EXPERT_MB` | `512` | Host-side expert staging budget (MiB) for non-resident paths | | `VT_GEMMA4_LAYER_TRACE` | off | `=1` layer GPU-synced phase timers; `=2` per-layer heartbeats | | `VLLM_CPP_HTTP_FIXED_POOL` | `1` (fixed) | `=0` reverts the HTTP worker pool to the legacy dynamic mode. Production uses the capacity-derived fixed pool; the opt-out exists for same-binary A/B attribution | diff --git a/docs/FEATURES.md b/docs/FEATURES.md index d835bbd30..ccb875fb4 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -332,7 +332,7 @@ CPU elementwise GEMM (f32/f16/bf16) runs AVX2 and AVX-512 tiers on x86 where the | XPU, TPU | Not started | CUDA, CPU, Metal and Vulkan are the built backends | | Custom logits processors on CUDA | Open, not root-caused | Segfaults in a CUDA build, 232/232 green on CPU | | Memory budgeting (`ROAD-V1-MEM`, #83) | M1+M2 landed (absolute bytes) | `--kv-cache-memory` sizes the KV pool from an absolute byte budget (ABI v16, group-aware divisor); `--num-blocks` overrides; `--gpu-memory-utilization` needs the M3 profile run (dgx-gated). See `specs/kv-sizing.md` | -| Gemma4 MoE ROCm FP8 + SharedK-WMMA | Partial | Dual-GPU FP8 resident experts, SharedK-WMMA prefill (RDNA4); decode-graph and forward extract deferred. Env `VT_GEMMA4_*`/`VT_ATTN_*`, seam `test_gemma4_rocm_fp8_seams`. [spec](../.agents/specs/gemma4-rocm-fp8-moe.md) | +| Gemma4 MoE ROCm FP8 + SharedK-WMMA | Partial | Dual-GPU FP8 resident + SharedK-WMMA. Lab KEEP fair ~2014 t/s@11k / ~1099@42k (2xR9700). Decode-graph/forward extract deferred. [spec](../.agents/specs/gemma4-rocm-fp8-moe.md) | ## How to read this page diff --git a/docs/ROCM.md b/docs/ROCM.md index cc2f708c2..f12d0b258 100644 --- a/docs/ROCM.md +++ b/docs/ROCM.md @@ -9,8 +9,11 @@ Gemma-3-1B-it is 48/48 token-identical to two independent vLLM-ROCm oracles; Qwen3-0.6B exposed a deterministic cross-version near-tie. Qwen3.5-0.8B also runs all-native through the GDN stack, but its CPU/ROCm divergence is still an open correctness gap. The Gemma-4 FP8 MoE and SharedK-WMMA path has contributor -runtime evidence on 2x R9700; this repository has only CPU link coverage for -that path, and no matched oracle performance result. +runtime evidence on 2x R9700 (KEEP fair ~2014 t/s @~11k / ~1099 @~42k, +`PREFIX_CACHE=0`; decode ~55 t/s). This repository has only CPU link coverage +for that path, and no matched vLLM-ROCm oracle performance result. Vulkan Q8 +prefill on the same box is faster (~3503 / ~2714); the ROCm path is the +reliable recipe, not that bar. This page exists because several people offered hardware in [issue #41](https://github.com/mudler/vllm.cpp/issues/41), and it answers the diff --git a/docs/USAGE.md b/docs/USAGE.md index 4f80a43ef..9bce1ed10 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -2118,9 +2118,17 @@ so deltas are not collapsed by a poll loop. Dual-GPU resident FP8 MoE and SharedK-WMMA prefill are controlled via ENVIRONMENT.md (`VT_GEMMA4_RESIDENT_*`, `VT_ATTN_*`). Defaults stay safe off RDNA4. -This PR does **not** restructure the Gemma-4 layer loop or enable decode hipGraph +This path does **not** restructure the Gemma-4 layer loop or enable decode hipGraph (those stay lab-only until a CUDA token-exact gate can land them). +Contributor KEEP recipe (2x R9700 gfx1201, ROCm 7.2.4, `PREFIX_CACHE=0`, unique +pads, 2026-08-13): SharedK-WMMA on, FLASH/FMHA off, `VT_GEMMA4_PREFILL_GEMM_M=2048`, +`PEER_ACT=1`, batch MoE `T>=64`, decode KV-splits 16 / slide-splits 8. Fair median +prefill **2014 t/s @~11k** and **1099 t/s @~42k**; stream decode **55 t/s** temp=0. +Paris / arith `63` / `gemma4` tool_calls held. Speculative, ngram, FMHA, and +layer-split are **out of this recipe**. This is a reliable plateau, not Vulkan +prefill parity. Details: [spec](../.agents/specs/gemma4-rocm-fp8-moe.md). + ## LTX-2.5 text conditioning This documents **one brick of the shipped render path** — the text conditioning From c08af5271ef0e81c9ff2a5a9a4da39c957a51683 Mon Sep 17 00:00:00 2001 From: Don Mirror Date: Thu, 13 Aug 2026 16:49:24 -0500 Subject: [PATCH 2/2] record(GEMMA4-ROCM-KEEP): ACO cm1 is 0-spill vs HIP 235-339 Matched RADV pipeline stats on gfx1201 (Mesa 26.0.3): 0 spilled VGPR, 0 scratch at VGPR=256 for llama.cpp coopmat1 d=512. HIP cm1 still 235-339 spills. (a)-lean only; no component name, no 3.35x claim. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Hermes:grok-4.6 [Hermes] --- .agents/specs/gemma4-rocm-fp8-moe.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/.agents/specs/gemma4-rocm-fp8-moe.md b/.agents/specs/gemma4-rocm-fp8-moe.md index 351fbe963..8e848d6f4 100644 --- a/.agents/specs/gemma4-rocm-fp8-moe.md +++ b/.agents/specs/gemma4-rocm-fp8-moe.md @@ -88,4 +88,6 @@ FMHA_WMMA2 (quality fail + 0.67x @11k). Isolated P1 cm1 wg256 (~1.13x KEEP; need ### Residual compiler / ACO -Faithful HIP cm1 hsaco on gfx1201: 192 VGPR, 339 spills, 940 B private, vs KEEP SharedK 35 spills / 120 B. Mechanism of the isolated 1.13x, not a named LLVM defect. Avoidability (regalloc vs live-range vs fundamental) is open. Phase-2 ROCm/LLVM rebuild stays blocked until a discriminator names a smallest component and a measurable prediction. A general codegen fix, if one exists, is a separate claim from the 3.35x bar. +Faithful HIP cm1 hsaco on gfx1201: 192 VGPR, 339 spills, 940 B private, vs KEEP SharedK 35 spills / 120 B. Output-slicing bottoms out at 235 spills / 736 B (D_PT=1). Mechanism of the isolated 1.13x, not by itself a named LLVM defect. + +Matched RADV/ACO (Mesa 26.0.3, discrete R9700, llama.cpp coopmat1 spec Br=16 Bc=64 wg=128 sg=32 row_split=4): official pipeline stats are **0 spilled VGPR, 0 scratch** at VGPR=256 (f16 and q8_0 d=512). That is an (a)-leaning HIP-LLVM vs ACO disparity on equivalent ownership. Smallest LLVM/rocWMMA component is not named. Phase-2 rebuild stays blocked until that component and a measurable prediction exist. A general codegen fix, if one exists, is a separate claim from the 3.35x bar.