fix(#1157): NemotronH decoded with STALE input ids, and the recurrent carry was never the cause

[localai-bot]

The A3 token gate diverged on real weights and the cause on record was the
recurrent carry. It was not. NemotronHPagedForward embedded the HOST
input.token_ids while ModelForwardInput::device_token_ids was non-null, and
that field's contract is that the host vector is STALE for decode rows
(model_registry.h:314-324) — not materialising it on the host is the
synchronize ENG-ASYNC-SCHED W4 exists to remove. So on the default CUDA path
every decode step embedded the same placeholder id.

FOLLOWING_AGENTS_PROTOCOL

What the measurement says

One binary, one checkpoint (released nemotron-3.5-lightning-30b-nvfp4,
revision 29f2d174), one committed oracle golden, driven through
include/vllm.h alone:

arm	takes a decode step?	result
host, decode	yes	96/96, STRICT PASS, full rows 3
GB10, decode	yes	4/24
GB10, fresh-prefill	no — one token per completion	24/24

The GB10 got streams reproduce the earlier recorded run byte for byte, so this
is the same defect on a fresh build and not drift. The build was not degraded:
CUDA 13.x from the ubuntu2404/sbsa lane, CFG_RC=0, cutlass-nvfp4 /
cutlass-fp8 / marlin-nvfp4 / fa2 all ENABLED for [121a], BUILD_RC=0,
compile_errors=0, binary sha256 b4677cdb7cf5….

The per-layer trace puts it at the first operation of the decode step rather
than inferring it. At the prefill step host and GB10 agree to six digits on all
52 layers. At the first decode step the gathered conv/SSM state is IDENTICAL on
the two — |conv|=310.374, |ssm|=3985.8 on both, so the recurrent carry is
exact — while layer 0's embedding row differs, and it reads 0.228135 on GB10
at BOTH decode steps although they consumed different tokens. A constant
embedding is a constant input id.

What this refutes

#1157 reasoned that
gm.num_decodes might classify a decode as a prefill so the gather would hand
the mixer zeros. On real weights the trace reports nd=1 np=0 init=[1] on every
decode step, and mutating that mask to 0 turns the A2-P CPU gate RED (1 case,
6 assertions) — that gate was never blind to that defect. It was blind to this
one structurally: the runner sets device_token_ids only under
VLLM_CPP_CUDA with a live device mirror, so no CPU gate can reach the branch
at all. That is why the red-before here is a device measurement rather than a
unit test.

What else is in the change

Instruments, all off by default. VT_NEMOTRON_H_DIAG prints the
decode/prefill split, the state indices, the has-initial mask, and per Mamba2
layer the L2 of the state gathered in and written out — it is what separated the
carry from the input id, and it reads a healthy carry on the CPU fixture, which
is what makes it usable as a negative control. nemotron-h-gen --fresh-prefill
/ --both-modes generate the same stream without ever taking a decode step,
over one engine load. VT_NEMOTRON_H_DEVICE_MOE=0 was the bisect lever for the
device MoE. Both env vars are documented in docs/ENVIRONMENT.md.

Two coverage gaps found while bisecting, both at the decode shape. The
device MoE gate measured T=4 and T=2, two prefill widths, never the T=1 the
model spends its whole decode in. The FA-2 d128 decode op gate measured GQA
ratios 2 and 4 (16/8, 32/8), and NemotronH is 32/2, ngroups 16 — the path
is default-ON for any bf16 causal pure-decode at head_dim 128, so it is not
Qwen3-dense-only as its comment claims. Neither new case has run on a GPU
yet; a lease is queued to measure them, and they skip loudly everywhere else.

#1217 is filed and owed,
not fixed here. The runner hands device_token_ids to whatever model the step
routes to; the field's comment claims a model that ignores it is never given
one, and nothing enforces that. Kimi-Linear was cut from this same divergence
(kimi_linear_device.cpp:2270-2280) and NemotronH is the second. Both closes —
an explicit ModelFactory::honors_device_token_ids with a runner fallback, or a
checker over the registered .forward entry points — change a shared seam or
checker semantics, which AGENTS.md sends through its own spec.

Still owed

The sm_121a green-after re-run of the full 96-token gate under this fix. The
fleet lease for it is queued; docs/BENCHMARKS.md records the device leg as
pending rather than as a pass, and the host leg as the pass it is.

Note for the merger: #1158 is
open against the same benchmark-record.md, STATUS.md, BENCHMARKS.md and
both specs, and records the failure this change explains. Whichever lands second
should take the target-branch version whole and re-apply its own scoped edit.

Closes #1157

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]