[executorch][cuda] Recapture off-graph KV decode graphs

[Gasoonjia]

Stack from ghstack (oldest at bottom):

• [executorch][cuda] Isolate off-graph KV cache by session #22820
• -> [executorch][cuda] Recapture off-graph KV decode graphs #22819
• [executorch][muse] Wire dynamic KV cache into the runtime #22818
• [executorch][muse] Add dynamic off-graph KV export #22817
• [executorch][cuda] Add dynamic off-graph KV runtime manager #22816
• [executorch][cuda] Lower off-graph KV storage through AOTI #22815
• [executorch][cuda] Add off-graph KV Triton primitives #22814

Allow the Muse Glimmer off-graph KV path to use the existing decode-only CUDA Graph backend option. When a flat cache grows and its address changes, release the stale graph resources, rebind the new cache allocation, warm up, and capture a replacement graph. Prefill remains outside CUDA Graph capture.

Authored with Codex.

Differential Revision: D120091513

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/22819

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure, 1 Unrelated Failure, 9 Unclassified Failures

As of commit 511ec00 with merge base ccecb93 ():

NEW FAILURE - The following job has failed:

• Lint / lintrunner (gh)
  >>> Lint for examples/models/muse-glimmer/runtime/engine/muse_glimmer_engine.cpp:

UNCLASSIFIED FAILURES - DrCI could not classify the following jobs because the workflow did not run on the merge base. The failures may be pre-existing on trunk or introduced by this PR:

• cuda-perf / benchmark-cuda (SocialLocalMobile/Qwen3.5-35B-A3B-HQQ-INT4, quantized-int4-tile-packed, SocialLoc... / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• MLX / test-mlx-muse-glimmer-build / test-mlx-muse-glimmer-build (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  /Users/runner/work/executorch/executorch/pytorch/executorch/examples/models/muse-glimmer/runtime/engine/muse_glimmer_engine.cpp:1508:22: error: no member named 'with_load_scope' in 'executorch::extension::llm::MuseGlimmerOffGraphKVCacheContextOwner'
• Test CUDA Builds / merge-and-test-cuda-multi-arch-a10g / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 2
• Test CUDA Builds / test-model-cuda-e2e-Qwen3.5-35B-A3B-HQQ-INT4-quantized-int4-tile-packed / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• Test CUDA Builds / test-models-cuda / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  RuntimeError: Command docker exec -t add9434ec7fdd95ee3af464072d2cff22eeb2d502326cdca9caf17971943baea /exec failed with exit code 2
• Test CUDA Builds / test-muse-glimmer-cuda-e2e-kquant-17gb-solo-text-offgraph / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• Test ROCm AOTI / test-voxtral-realtime-rocm-gfx950-rocm7.2 / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  /pytorch/executorch/backends/cuda/runtime/cuda_kv_cache.cpp:241:37: error: ‘cudaMemcpy2DAsync’ was not declared in this scope; did you mean ‘cudaMemcpyAsync’?
• Test ROCm AOTI / unittest-rocm-gfx950-rocm7.2 / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  /pytorch/executorch/backends/cuda/runtime/cuda_kv_cache.cpp:241:37: error: ‘cudaMemcpy2DAsync’ was not declared in this scope; did you mean ‘cudaMemcpyAsync’?
• trunk / test-models-macos-coreml (emformer_transcribe) / macos-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  ##[error]The operation was canceled.

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• trunk / test-samsung-models-linux / linux-job (gh) (trunk failure)
  test_mobilebert_qat_a8w8

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[github-actions]

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.