measure(#810 A3): the token gate ran on real weights and FAILED 6/96, and the failure names the mechanism

[localai-bot]

measure(#810 A3): the token gate ran on real weights and FAILED 6/96, and the failure names the mechanism

The A3 end-to-end gate has run for the first time on the released
nemotron-3.5-lightning-30b-nvfp4 checkpoint, on GB10 sm_121a, through
include/vllm.h and nothing else. GATE_RC=1. Six of ninety-six tokens match.

Token 0 is correct on all three prompts, and decode then decays and collapses
onto the repeated id 1050 by about step five. That shape is the finding. Token 0
comes from prefill; every token after it comes from a decode step that must read
what the previous step wrote. Numeric drift wanders, it does not converge on one
id and sit there, so this is the recurrent carry rather than a noisy kernel, and
#1157 records where to look plus a cheap discriminator between the two halves.

This row's own documents predicted it. The parent spec matched 3/3 FIRST tokens
against a forward carrying no state at all and warned how little that proves;
this run reproduces 3/3 first tokens and then fails. A2-P's section 10 already
recorded that the synthetic fixture could not see the recurrent carry, and the
two numeric cases added to compensate turn out to be insufficient: the CPU gate
is 12/12 green at the very commit that produces this. A green class and a wrong
capability is the reachability failure this tree keeps naming, and it now has a
measurement instead of a warning.

Nothing about the build can explain it. CUDA 13.3.73, CFG_RC=0, VOID_FLAG=0 with
cutlass-nvfp4, cutlass-fp8, marlin-nvfp4 and fa2 all ENABLED for [121a] rather
than [121] or DISABLED, BUILD_RC=0, compile_errors=0. Nor was the instrument
mute: compared=96 over three prompts of width 32, full rows 3, short rows 0. A
run that had compared fewer would have exited 4 instead of 1.

Recorded as a result rather than as silence, and no band was widened to make it
read better. Repair is owed to a fresh implementer, because this row wrote the
driver that found it.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]

[localai-bot]

Closing as superseded — and it would now REGRESS main rather than merely duplicate it.

Its content is already on main, more completely

#1221 landed as 0ea5d249f and recorded the same run in .agents/benchmark-record.md with the full arm table:

| decode         | the shipped path                              | 4/24   |
| fresh-prefill  | one token per completion, no decode step      | 24/24  |
| host           | CPU queue, where device_token_ids is null     | 96/96  |

main carries 36 matching entries for this run. The only line this PR would still add is "against the committed oracle golden."

Two ways merging it would now make main worse

1. It would add a claim that is false. Its nemotron-h-a2p-paged-forward.md edit says:

The §5.4 A3 end-to-end token gate HAS RUN and FAILED: 6/96 tokens, GATE_RC=1

The gate has since PASSED on GB10: TOKEN MATCH 96/96 over 3 prompts, full rows=3, short rows=0, mode=decode, STRICT PASS, against vllm=0.23.1rc1.dev1511+g555967922 on nemotron-3.5-lightning-30b-nvfp4 rev 29f2d1746d8f41e316523194b19018707749b1b1. Red-before with only nemotron_h_device.cpp reverted: 4/24 on the same binary and checkpoint. Artefacts at /usr/local/nas_share/rc/nh1157/{gate_fixed,gate_red,cfg}.out; cfg.log shows fp4-mma/cutlass-nvfp4/cutlass-fp8/marlin-nvfp4/fa2 all ENABLED for [121a].

2. It would DELETE text main now has. Its nemotron-h-abi-e2e.md diff removes the passage recording that "nothing can build a gate binary" was a host measurement reported as a container measurement — that the dgx host genuinely lacks nvcc/cmake since the 14 Aug reimage (#1019), but the host is not where work runs, and inside rc run the worker is Ubuntu 24.04 as uid 0 with the full toolchain. That lesson cost this row hours and is worth more than the record it sits beside.

Where the corrected state is landing

#1081 carries it, resolved against post-#1221 main: it corrects the one claim main is now stale on ("GB10 read 4/24 ... sm_121a re-run pending"), since that re-run has happened and passed. It keeps the discipline this PR established — still not a benchmark, no ratio claimed on any axis, with host-side NVFP4 lm_head (A2-Q2b) and the 46 FP8 W8A8 mamba projections (A2-Q1, #940) named as why.

Nothing measured here is lost. The work this PR did — running the first e2e gate and recording the divergence honestly rather than as a pass — is what produced #1157 and, through it, the fix.