measure(MODEL-NEMOTRON-H-ABI-A2P): the first Nemotron ratios -- decode 0.001392x, load 2.12x faster, and the GPU idle for 93.7% of our decode (#1250)

[localai-bot]

There was no Nemotron number on any axis. docs/BENCHMARKS.md carried
MODEL-NEMOTRON-H-ABI-A2P as "No speed number, by the unit's own rule". This
branch takes the first ones, against the pinned vLLM oracle, which completed a
model run inside an rc lease for the first time.

FOLLOWING_AGENTS_PROTOCOL

The numbers

Both legs on dgx:gpu0 through rc run, same box, same boot id, same
checkpoint directory, same three pre-tokenized prompts, same 32 greedy tokens
with ignore_eos, batch 1 sequential, two legs over one engine load on each
side. Tree 5325b7b970b67f97a77834e907fc34fb2990b71e, which contains
0ea5d249f (#1221). enforce_eager=False throughout: CUDA graphs were on and
were never disabled.

axis	ours	pinned vLLM	ratio
per output token, warm (n=5)	10.3194 s	0.014369 s	718.2x slower
output throughput, batch 1	0.09691 tok/s	69.595 tok/s	0.001392x
engine load	280.9 s	596.3 s	0.4711x — we are 2.12x FASTER
peak host memory	44,616 MB	70,974 MB	0.629x raw, not like-for-like
KV pool	8192 tokens	644,096 tokens	78.6x, could NOT be matched

Every timing leg is a gated leg. Ours read TOKEN MATCH: 96/96 over 3 prompt(s) (full rows=3, short rows=0, mode=decode) on BOTH legs, STRICT PASS,
192 of 192 tokens, so no number here comes from a configuration whose tokens were
not compared in the same process. The same-binary A/B over one load is 1.0016
warm and the five warm prompts spread 0.245%.

The build was not degraded: CUDA 13.3.73, CFG_RC=0, BUILD_RC=0, zero compile
errors, and fp4-mma / cutlass-nvfp4 / cutlass-fp8 / marlin-nvfp4 / fa2
each ENABLED for [121a], counted as FEATURE_LINES_SEEN=5 DEGRADED_FEATURE_LINES=0 rather than eyeballed. The box was idle:
--query-compute-apps empty, 4,892 of 122,502 MB used.

Two refusals that travel with the ratios

They are recorded beside the numbers rather than netted out of them.

The clock gate refuses this pair. gpu_clock_state compare exits 1 with six
reasons: ours 6.31% busy and vLLM 31.05% busy against a 50% floor, vLLM spread
5.14% against a 5.0% ceiling, ours throttled SwPowerCap, persistence
Disabled on both. Same boot id, both medians 2411 MHz, median_offset_pct
0.0. What the refusal cannot do is explain the result: the tool's own recorded
basis is 0.7548 points of kernel time per point of clock, and the gap is 718x.

The KV pools could not be matched. vLLM's hybrid allocator resolves its own
block size — 512 here, 4192 at other settings — against our 32, so a token
capacity match is the most that was ever available and this run did not get one.
The memory axis is therefore reported raw, and the straggler still resident in
the oracle's window (PID 40514, 22,986 MiB, the EngineCore this row's own
watchdog killed) is named rather than subtracted.

What names the bottleneck, and why it is not a ceiling

nvidia-smi reported GPU utilization 0% in 2,019 of 2,155 samples of our
window. The GPU was busy in 6.31% of it. That is the driver's own answer, not
a reading of the source, and the clock helper's refusal to attribute such a
window IS the finding: this decode is not GPU work.

It matches the mechanism exactly. Of 52 layers, 6 are attention and stay on the
device, 23 MoE layers run on the device through the NVFP4 Marlin arm, and 23
Mamba2 layers bounce — the normed hidden is downloaded, the mixer runs on the
CPU queue, the result is uploaded — once per layer per token; then
NemotronHHostLmHead projects on the host because nemotron_h.cpp:1031-1034
refuses the NVFP4 lm_head on a non-CPU queue.

So the next traceable hypothesis is A2-Q1 (#940), the 46 FP8 W8A8 mamba
projections that are 36.6% of decode bytes, then A2-Q2b for the lm_head. Each
now has a prediction it must answer rather than a story: it has to RAISE that
6.31% busy fraction, measured with the same battery on the same binary so the
delta is attributable.

The oracle does not reproduce its own golden here

ORACLE TOKEN MATCH: 180/192 — prompt 2 matched 26/32, identically in both legs,
so it is deterministic within the configuration. Ours matched 96/96 on the same
golden. The golden was captured at the oracle's default memory configuration,
and the two knobs the box forced (gpu_memory_utilization=0.30 because 0.9 took
104,992 MB of a 122,502 MB unified pool and was killed, and
max_num_batched_tokens=512) moved its resolved block_size to 512 and with it
the reduction order. The throughput comparison survives — max_tokens is fixed
with ignore_eos, so both sides took the same number of decode steps — but a
token-exact gate against this golden does not survive a change to the oracle's
memory configuration, and that now bounds how the gate may be re-run.

Also here

--num-blocks, --kv-cache-memory and --repeat on nemotron-h-gen. The
driver could not state its own KV budget and inherited the 256-block fallback
the engine takes when gpu_memory_utilization is accepted and does not size the
pool (#83); both legs of the A3 gate logged that warning, and this run does not.
--repeat runs the battery again over one load, and the verdict reads grand
totals, so a battery that passed once and diverged on the second leg cannot
report a pass.

#1253, filed and fixed in this flow. The oracle could not run a model in a
lease for want of python3-dev — Triton compiles cuda_utils.c at runtime and
the failure surfaces four frames up as Engine core initialization failed ... Failed core proc(s): {}, an empty proc set naming nothing. And nvcc --version
is not the toolkit postcondition: the worker ships a partial CUDA 13.0 whose
compiler answers while cuBLAS dev is absent, so CMake printed all five feature
lines and then failed with Target "vllm" links to CUDA::cublasLt but the target was not found. #1185's recorded hazard for the oracle — a reboot of the box in
the step after torch.compile — did NOT occur; the watchdog held.

Closes #1250.

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]