fish-speech: make compile:true usable on Blackwell sm_121 by honouring the CUDA toolkit's ptxas

[tpnet3]

Is your feature request related to a problem? Please describe.

The fish-speech backend exposes compile as a model option, which turns on
torch.compile inside launch_thread_safe_queue:

compile_model = self.options.get("compile", False)
llama_queue = launch_thread_safe_queue(
    checkpoint_path=model_path, device=device, precision=precision, compile=compile_model
)

On Blackwell sm_121 hardware (NVIDIA DGX Spark, GB10) this option cannot be used
at all. Enabling it makes every TTS request fail with HTTP 500:

error during TTS: TTS inference error: PTXAS error: Internal Triton PTX codegen error
`ptxas` stderr:
ptxas fatal   : Value 'sm_121a' is not defined for option 'gpu-name'

Repro command: /backends/cuda13-nvidia-l4t-arm64-fish-speech/venv/lib/python3.10/site-packages/triton/backends/nvidia/bin/ptxas -lineinfo -v --gpu-name=sm_121a /tmp/tmphozi1i5i.ptx -o /tmp/tmphozi1i5i.ptx.o

The ptxas that ships inside the Triton wheel predates sm_121, while the CUDA
13.0 toolkit already present in the same image understands the target natively.
This is a known Triton/PyTorch packaging gap rather than a LocalAI bug, but
LocalAI is where it surfaces, and LocalAI is where it can be worked around
cheaply. See triton-lang/triton#10331 and pytorch/pytorch#163801.

Two things make this worse than a plain unsupported-option error:

1. The failure is deferred. The model loads successfully and reports healthy;
   /system lists it under loaded_models. The 500 only appears on the first
   inference, because torch.compile warms up lazily. A user who enables the
   option and restarts sees a working server that 500s on every request.
2. The cost of leaving it off is large. On this hardware compile:true is
   not a marginal tuning knob — it is worth 5–7x. Autoregressive decode at batch
   size 1 is bandwidth- and kernel-launch-bound on GB10's unified LPDDR5X
   (~273 GB/s), which is exactly the case CUDA graphs address.

Measured on a DGX Spark against fish-speech-s2-pro over the HTTP API, same
input text, warm model:

workload	compile off	compile on	speedup
short sentence (1.5 s audio)	29.6 s	5.7 s	5.2x
chunked narration (22.0 s audio)	397.3 s	57.4 s	6.9x

Expressed as a realtime factor, that is 18.7x slower than realtime dropping to
3.8x for short input and 17.8x dropping to 2.6x for long input. GPU utilization
sat at 96% in both cases, so the slow path was never a CPU fallback — it was
kernel-launch overhead that CUDA graphs remove.

Describe the solution you'd like

Have backend/python/fish-speech/run.sh point Triton at the toolkit ptxas when
one is available, in the same place the PYTHONPATH export goes:

if [ -z "${TRITON_PTXAS_PATH:-}" ] && [ -x /usr/local/cuda/bin/ptxas ]; then
    export TRITON_PTXAS_PATH=/usr/local/cuda/bin/ptxas
fi

Properties that make this cheap:

• It is a no-op when the CUDA toolkit is absent, so CPU and non-CUDA images are
  unaffected.
• It is a no-op when the caller already set TRITON_PTXAS_PATH.
• compile still defaults to False, so nothing changes for users who don't opt in.
• It needs no new dependency and no image rebuild beyond the shell script.

Two optional improvements, either of which would have saved the debugging time
here — happy to split them into separate issues if preferred:

• Fail at load, not at first inference. When compile is requested, either
  run a trivial warmup compile inside LoadModel or catch the PTXAS error and
  fall back to eager with a warning, rather than 500-ing every request.
• Document the option. compile isn't mentioned in the TTS docs, so its
  existence and its cost/benefit on bandwidth-limited hardware are not
  discoverable.

The same TRITON_PTXAS_PATH treatment likely applies to every Python backend
that can reach torch.compile on this hardware, not just fish-speech.

Describe alternatives you've considered

• Leave compile off. Works, but gives up 5–7x on hardware LocalAI already
  ships a dedicated nvidia-l4t-arm64-cuda-13 image for.
• Patch the running container. What I did to gather the numbers above. It
  does not survive a redeploy, so it has to be re-applied from a post-deployment
  hook alongside the three workarounds from my earlier arm64 report. Fragile,
  and the failure mode when it silently doesn't run is a fully broken TTS
  endpoint rather than a slow one.
• Ship a newer Triton in the backend image. Fixes the root cause rather than
  routing around it, but it is a much heavier change with its own compatibility
  risk against the pinned torch 2.9.1+cu130, and it has to be redone every
  time a new architecture lands.
• Set TRITON_PTXAS_PATH from the model YAML. Model options are read after
  the process has started, so this cannot reliably affect Triton's toolchain
  discovery. It belongs in run.sh.

Additional context

Environment:

• NVIDIA DGX Spark (GB10, compute capability 12.1 / sm_121), aarch64, Docker
• localai/localai:latest-nvidia-l4t-arm64-cuda-13, backend
  cuda13-nvidia-l4t-arm64-fish-speech
• Driver 580.95.05, CUDA 13.0; backend venv Python 3.10.18, torch 2.9.1+cu130
• Model fish-speech-s2-pro, voice cloning via a Voice Library profile

Reproduce:

1. Bring up the image on an sm_121 host with the workarounds from my earlier
   arm64 report applied, so the backend loads on GPU
2. Add to /models/fish-speech-s2-pro.yaml:

   options:
       - compile:true

3. Restart, or POST /backend/shutdown with {"model":"fish-speech-s2-pro"} —
   editing the YAML alone is not enough, since compile is only read at load
4. Issue any /v1/audio/speech request → HTTP 500 with the PTXAS error above
5. Export TRITON_PTXAS_PATH=/usr/local/cuda/bin/ptxas in the backend's
   run.sh, reload the model, and the same request succeeds

Note for anyone reproducing the timings: the first inference after a restart
costs ~120 s of torch.compile warmup and is easy to mistake for a hang.
Requests are also serialized by the single launch_thread_safe_queue worker —
two concurrent requests took 52.7 s against 29.6 s for one — so client-side
parallelism does not hide the latency.

Caveat: everything above was verified by patching a running container and
driving the HTTP API. I have not rebuilt the arm64 CUDA 13 backend image, so the
proposed run.sh change is unverified at image-build time. I have GB10 hardware
available and am happy to test any build, or to open the PR for the run.sh
change if that's useful.

Related: my earlier report on this backend covering the PYTHONPATH, CPU torch
wheel and partial cuDNN bundle defects on the same platform. #11344

[localai-org-maint-bot]

I’m picking this up. I’ll keep the change scoped to fish-speech startup: preserve an explicit TRITON_PTXAS_PATH, otherwise select /usr/local/cuda/bin/ptxas when it is executable, with focused regression coverage and documentation for compile:true.