feat(qwen36): complete concurrent serving implementation with speculation

[Graffioh]

Summary

Depends on #595 at a90ffe45. This PR completes concurrent Qwen3.6-27B serving for the requested Strix Halo configuration:

--target-device hip:0
--draft-device hip:0
--ddtree --ddtree-budget 22
--draft-residency persistent
--prefill-compression auto
--prefill-drafter <path>
--kvflash auto

Area	Implementation
DDTree	Per-slot draft state, packed multi-sequence verification, accepted-path replay, ordered burst/stop handling
PFlash	Prepare once before admission, persistent residency, effective-token accounting
KVFlash	Bounded shared residency, pinned host backing, fair eviction, pending-write protection, scorer/LRU reselection
Scheduling	Concurrent feature gates, adaptive low-yield fallback to AR, request-correlated telemetry
Benchmarks	Fail-closed activation matrix plus canonical HumanEval, GSM8K, and Math500 workloads

Concurrent speculation uses packed verify-and-replay; standalone fast rollback remains a single-sequence path.

Strix Halo measurements

AMD Ryzen AI MAX+ 395 / Radeon 8060S (gfx1151), ROCm 7.2.4, Release HIP. Full commands, hashes, prompt ranges, and caveats are in STRIX_HALO_RESULTS.md and #596.

Recommended routing

Blog-aligned setup: Qwen3.6-27B Q4_K_M target, Q8_0 draft, SWA 2048, DDTree budget 22, fast rollback flag, paged attention, and Q4_0 paged KV.

Workload	C	Repeats	AR goodput	DDTree goodput	DDTree vs AR	Acceptance
HumanEval	1	1	11.59	15.17	+30.9%	33.8%
HumanEval	2	1	16.66	22.32	+34.0%	35.0%
HumanEval	3	1	18.88	25.47	+34.9%	38.2%
GSM8K	3	5	18.88	24.46	+29.4%	37.0%
Math500	3	5	19.03	28.71	+50.9%	43.8%
HumanEval	4	1	31.43	28.44	-9.5%	36.4%
GSM8K	4	5	31.73	27.84	-12.3%	34.8%
Math500	4	5	31.81	32.92	+3.1%	44.4%
HumanEval	5	1	37.50	25.05	-33.2%	34.9%
HumanEval	10	1	56.13	24.48	-56.4%	34.9%

Default policy: use fixed blog-aligned DDTree for C=1–3; use AR for C>3 and mixed or unknown traffic. Math500's small C=4 DDTree win is a workload-specific exception.

The earlier adaptive-DDTree screen used a weak-yield prompt cohort and measures fallback overhead; it does not contradict the canonical workload crossover above.

AR prefill tuning at C=16

Three paired observations per band, 16/16 successful requests per case, 64 output tokens. The selected policy keeps packed width at 8 and raises the mixed prefill budget from 2,048 to 4,096 only while an admitted prompt has at least 768 tokens.

Prompt band	Policy	Median goodput	vs baseline	Median max TTFT
977–1,147 tokens	Baseline 2K mixed budget	17.40 tok/s	—	49.10s
977–1,147 tokens	Adaptive 4K mixed budget	17.83 tok/s	+2.5%	47.57s
1,873–2,043 tokens	Baseline 2K mixed budget	10.24 tok/s	—	90.23s
1,873–2,043 tokens	Adaptive 4K mixed budget	10.43 tok/s	+2.1%	88.14s

Widths 12/16, an 8K idle budget, and a 1,024-token allocation quantum did not survive repeated testing. Short prompts retain the 2K mixed budget.

Complete DDTree + PFlash + KVFlash screen

One activation repeat per level, 64 output tokens, 38K–45K raw prompts compressed to roughly 4.9%, and an 8,192-token resident KV pool:

C	Output-window tok/s	Effective prompt tokens	Max TTFT	KV page in/out
1	12.39	2,021	19.3s	0/0
4	31.61	1,870–2,235	77.5s	1/18
8	52.60	1,869–2,237	153.8s	245/792
16	18.73	1,867–2,238	307.4s	293/1,880

All rows proved DDTree, PFlash, and KVFlash active. C8 was the peak; C16 became paging-bound.

Validation

Check	Result
Release HIP/gfx1151 server build	Passed
Server unit tests	353/353
Feature gates	180 assertions
Slot manager / batch plan / engine contract	204 / 72 / 15
Paged KV pool / residency / transfer layout	15 / 10 / 3
Packed DDTree and HIP tree attention	Passed
Benchmark tooling	34/34
Final measured requests/tokens	211/211, 13,504/13,504
Independent policy and measurement audits	No P0/P1 findings

Constraints

• PFlash concurrency requires persistent draft residency.
• Concurrent paged DDTree requires a local target and draft on the same device.
• A transient NoEvictableBlock during selected prefill still fails instead of deferring.
• Greedy output hashes varied across concurrent repeats for both AR and DDTree; token accounting and feature activation remained exact.
• The controlled 8K KV pool is paging-bound at C16.

[cubic-dev-ai]

@cubic can you review this PR

@Graffioh I have started the AI code review. It will take a few minutes to complete.

[cubic-dev-ai]

All reported issues were addressed across 109 files

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[Graffioh]

@cubic-dev-ai review this PR

[cubic-dev-ai]

@cubic-dev-ai review this PR

@Graffioh I have started the AI code review. It will take a few minutes to complete.

[cubic-dev-ai]

All reported issues were addressed across 110 files

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

All reported issues were addressed across 110 files

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

6 issues found across 110 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="harness/benchmarks/concurrency/generate_feature_prompts.py">

<violation number="1" location="harness/benchmarks/concurrency/generate_feature_prompts.py:17">
P2: `--profile tiny` is accepted here, but the feature-matrix runner has no `tiny` workload limit and fails after generating the prompts. Restrict this map to runner-supported base profiles or add a `tiny` workload case before exposing it.</violation>
</file>

<file name="server/src/qwen35/concurrency/qwen35_slot_manager.h">

<violation number="1" location="server/src/qwen35/concurrency/qwen35_slot_manager.h:88">
P2: When `max_ctx` is negative, admission can succeed but the slot cannot decode: the prompt check converts `max_ctx` to an unsigned size, while decode rejects the resulting slot. Validate `max_ctx > 0` at construction (or reject it before any unsigned conversion) so invalid configuration cannot create an admitted request that can never progress.</violation>
</file>

<file name="server/src/qwen35/qwen35_backend.cpp">

<violation number="1" location="server/src/qwen35/qwen35_backend.cpp:112">
P2: Initializing one concurrent Qwen backend permanently changes the process-wide rocBLAS setting for every later backend. Keep this default scoped to the intended backend or configure it outside the server process instead of mutating the global environment here.</violation>
</file>

<file name="server/src/common/paged_attention_config.h">

<violation number="1" location="server/src/common/paged_attention_config.h:51">
P2: Negative cache or reserve fields increase `usable` instead of reducing capacity, which can over-allocate the physical pool. Validate both fields as nonnegative and perform bounded subtraction before sizing tokens.</violation>

<violation number="2" location="server/src/common/paged_attention_config.h:57">
P2: When a caller supplies an oversized `max_ctx`, `paged_token_capacity(max_ctx)` overflows its signed `int` arithmetic before this cast. Reject values above `INT32_MAX - PAGED_BLOCK_SIZE + 1` before calculating the logical cap.</violation>
</file>

<file name="server/src/server/scheduler.cpp">

<violation number="1" location="server/src/server/scheduler.cpp:833">
P2: During shutdown, pending final responses are discarded as soon as the first drain pass returns. Continue servicing drains until they flush or reach their existing deadline before calling `finish_job()` on them.</violation>
</file>

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>