record(GEMMA4-ROCM-KEEP): measured SharedK-WMMA plateau on 2x R9700

.agents/specs/gemma4-rocm-fp8-moe.md

@@ -58,3 +58,36 @@ and env-gated MoE paths do not. Revisit as a **separate** PR with CUDA golden.
 
 `VT_GEMMA4_MLP_MOE_PARALLEL` is documented but **not** wired in this tip (was only
 in the deferred layer-loop path).
+
+## Measured KEEP (2026-08-13, contributor lab)
+
+Hardware: 2x R9700 gfx1201, ROCm 7.2.4, kernel 7.0.0-29-generic.
+Model: Gemma-4-26B-A4B-it FP8. Fair protocol: `PREFIX_CACHE=0` + unique pads.
+
+Recipe (product): SharedK-WMMA on, FLASH/FMHA/scoreless off, `PREFILL_GEMM_M=2048`,
+`PEER_ACT=1`, batch MoE `T>=64`, decode `KV_SPLITS=16` / `SLIDE_SPLITS=8` /
+`SLIDE_WARPS=16` / `SPLIT_WARPS=12`. Speculative, ngram, layer-split: **off**.
+
+| Depth | median prefill t/s | notes |
+|------:|-------------------:|-------|
+| ~3k | 2112 | 3 reps, Paris |
+| ~11k | 2014 | first rep 1170 outlier dropped from claim; median 2014 |
+| ~18k | 1705 | 3 reps |
+| ~42k | 1099 | 2 reps |
+
+Decode stream 64 tok: **55.5 t/s** temp=0, **49.1 t/s** temp=0.7.
+Quality: Paris, arith `63`, `gemma4` parser `tool_calls`, `/health` `/metrics`.
+No engine-fatal / hipError in the closeout log.
+
+Same-box Vulkan Q8 unique-pad bar: **3503 @11k / 2714 @42k**. KEEP gap ~1.74x / 2.47x.
+This row ships the reliable ROCm plateau. It does **not** close that bar.
+
+### Rejected (do not default-on)
+
+FMHA_WMMA2 (quality fail + 0.67x @11k). Isolated P1 cm1 wg256 (~1.13x KEEP; need ~3.35x isolated). Layer-split FIFO (~0.60x vs the Q8 bar). Head-TP peer-read (peer BW). hipBLASLt dual-GPU (fatal or slower).
+
+### Residual compiler / ACO
+
+Faithful HIP cm1 hsaco on gfx1201: 192 VGPR, 339 spills, 940 B private, vs KEEP SharedK 35 spills / 120 B. Output-slicing bottoms out at 235 spills / 736 B (D_PT=1). Mechanism of the isolated 1.13x, not by itself a named LLVM defect.
+
+Matched RADV/ACO (Mesa 26.0.3, discrete R9700, llama.cpp coopmat1 spec Br=16 Bc=64 wg=128 sg=32 row_split=4): official pipeline stats are **0 spilled VGPR, 0 scratch** at VGPR=256 (f16 and q8_0 d=512). That is an (a)-leaning HIP-LLVM vs ACO disparity on equivalent ownership. Smallest LLVM/rocWMMA component is not named. Phase-2 rebuild stays blocked until that component and a measurable prediction exist. A general codegen fix, if one exists, is a separate claim from the 3.35x bar.

docs/ENVIRONMENT.md

@@ -195,7 +195,7 @@ on CUDA/CPU builds beyond the documented behavior.
 | `VT_GEMMA4_PREFILL_BATCH_MOE` | auto / `1` in lab recipe | `=1` group-by-expert prefill GEMM for `T>=64`; `=0` serial M=1 (slow). Unset = auto |
 | `VT_GEMMA4_MLP_MOE_PARALLEL` | off | `=1` run Gemma4 MLP and MoE on two HIP streams (lab; wall ~flat on R9700). Not wired in this PR tip (decode-graph-free split) |
 | `VT_ATTN_PREFILL_FLASH` | off | `=1` SGLang-style BM×BN GQA flash prefill (lab A/B) |
-| `VT_GEMMA4_PREFILL_GEMM_M` | `256` | Tokens per expert in prefill-batch GEMM chunks (`16..2048`). Larger M → fewer launches; lab `512` ~+37% prefill vs `64` |
+| `VT_GEMMA4_PREFILL_GEMM_M` | `256` | Tokens per expert in prefill-batch GEMM chunks (`16..2048`). Larger M → fewer launches. Lab KEEP on dual R9700 uses `2048` |
 | `VT_GEMMA4_HOST_EXPERT_MB` | `512` | Host-side expert staging budget (MiB) for non-resident paths |
 | `VT_GEMMA4_LAYER_TRACE` | off | `=1` layer GPU-synced phase timers; `=2` per-layer heartbeats |
 | `VLLM_CPP_HTTP_FIXED_POOL` | `1` (fixed) | `=0` reverts the HTTP worker pool to the legacy dynamic mode. Production uses the capacity-derived fixed pool; the opt-out exists for same-binary A/B attribution |

docs/FEATURES.md

@@ -332,7 +332,7 @@ CPU elementwise GEMM (f32/f16/bf16) runs AVX2 and AVX-512 tiers on x86 where the
 | XPU, TPU | Not started | CUDA, CPU, Metal and Vulkan are the built backends |
 | Custom logits processors on CUDA | Open, not root-caused | Segfaults in a CUDA build, 232/232 green on CPU |
 | Memory budgeting (`ROAD-V1-MEM`, #83) | M1+M2 landed (absolute bytes) | `--kv-cache-memory` sizes the KV pool from an absolute byte budget (ABI v16, group-aware divisor); `--num-blocks` overrides; `--gpu-memory-utilization` needs the M3 profile run (dgx-gated). See `specs/kv-sizing.md` |
-| Gemma4 MoE ROCm FP8 + SharedK-WMMA | Partial | Dual-GPU FP8 resident experts, SharedK-WMMA prefill (RDNA4); decode-graph and forward extract deferred. Env `VT_GEMMA4_*`/`VT_ATTN_*`, seam `test_gemma4_rocm_fp8_seams`. [spec](../.agents/specs/gemma4-rocm-fp8-moe.md) |
+| Gemma4 MoE ROCm FP8 + SharedK-WMMA | Partial | Dual-GPU FP8 resident + SharedK-WMMA. Lab KEEP fair ~2014 t/s@11k / ~1099@42k (2xR9700). Decode-graph/forward extract deferred. [spec](../.agents/specs/gemma4-rocm-fp8-moe.md) |
 
 ## How to read this page
 

docs/ROCM.md

@@ -9,8 +9,11 @@ Gemma-3-1B-it is 48/48 token-identical to two independent vLLM-ROCm oracles;
 Qwen3-0.6B exposed a deterministic cross-version near-tie. Qwen3.5-0.8B also
 runs all-native through the GDN stack, but its CPU/ROCm divergence is still an
 open correctness gap. The Gemma-4 FP8 MoE and SharedK-WMMA path has contributor
-runtime evidence on 2x R9700; this repository has only CPU link coverage for
-that path, and no matched oracle performance result.
+runtime evidence on 2x R9700 (KEEP fair ~2014 t/s @~11k / ~1099 @~42k,
+`PREFIX_CACHE=0`; decode ~55 t/s). This repository has only CPU link coverage
+for that path, and no matched vLLM-ROCm oracle performance result. Vulkan Q8
+prefill on the same box is faster (~3503 / ~2714); the ROCm path is the
+reliable recipe, not that bar.
 
 This page exists because several people offered hardware in
 [issue #41](https://github.com/mudler/vllm.cpp/issues/41), and it answers the

docs/USAGE.md

@@ -2118,9 +2118,17 @@ so deltas are not collapsed by a poll loop.
 
 Dual-GPU resident FP8 MoE and SharedK-WMMA prefill are controlled via
 ENVIRONMENT.md (`VT_GEMMA4_RESIDENT_*`, `VT_ATTN_*`). Defaults stay safe off RDNA4.
-This PR does **not** restructure the Gemma-4 layer loop or enable decode hipGraph
+This path does **not** restructure the Gemma-4 layer loop or enable decode hipGraph
 (those stay lab-only until a CUDA token-exact gate can land them).
 
+Contributor KEEP recipe (2x R9700 gfx1201, ROCm 7.2.4, `PREFIX_CACHE=0`, unique
+pads, 2026-08-13): SharedK-WMMA on, FLASH/FMHA off, `VT_GEMMA4_PREFILL_GEMM_M=2048`,
+`PEER_ACT=1`, batch MoE `T>=64`, decode KV-splits 16 / slide-splits 8. Fair median
+prefill **2014 t/s @~11k** and **1099 t/s @~42k**; stream decode **55 t/s** temp=0.
+Paris / arith `63` / `gemma4` tool_calls held. Speculative, ngram, FMHA, and
+layer-split are **out of this recipe**. This is a reliable plateau, not Vulkan
+prefill parity. Details: [spec](../.agents/specs/gemma4-rocm-fp8-moe.md).
+
 ## LTX-2.5 text conditioning
 
 This documents **one brick of the shipped render path** — the text conditioning