diff --git a/.agents/skills/deployment/references/support-matrix.md b/.agents/skills/deployment/references/support-matrix.md index 265f1b58560..bbef86efe24 100644 --- a/.agents/skills/deployment/references/support-matrix.md +++ b/.agents/skills/deployment/references/support-matrix.md @@ -2,26 +2,14 @@ ## Unified HF Checkpoint — Framework Compatibility -| Model | Quant Format | TRT-LLM | vLLM | SGLang | -|-------|-------------|---------|------|--------| -| Llama 3.x | FP8 | yes | yes | yes | -| Llama 3.x | FP4 | yes | yes | yes | -| Llama 4 | FP8 | yes | — | yes | -| Llama 4 | FP4 | yes | — | — | -| DeepSeek R1 | FP8 | yes | yes | yes | -| DeepSeek R1 | FP4 | yes | yes | yes | -| DeepSeek V3 | FP8 | yes | yes | yes | -| DeepSeek V3 | FP4 | yes | yes | yes | -| Qwen 3 | FP8 | yes | yes | yes | -| Qwen 3 | FP4 | yes | yes | — | -| Qwen 3 MoE | FP8 | yes | yes | yes | -| Qwen 3 MoE | FP4 | yes | — | — | -| Qwen 2.5 | FP8 | yes | yes | yes | -| Qwen 2.5 | FP4 | yes | yes | — | -| QwQ-32B | FP8 | yes | yes | yes | -| QwQ-32B | FP4 | yes | yes | — | -| Mixtral 8x7B | FP8 | yes | yes | yes | -| Mixtral 8x7B | FP4 | yes | — | — | +**Do not maintain a copy of the matrix here.** The single source of truth is +`docs/source/deployment/3_unified_hf.rst` ("Model Support Matrix"), and every validated entry in it +is backed by `tests/examples/hf_ptq/test_deploy.py`. + +To answer "is model X supported on framework Y", read one of those two files — `test_deploy.py` is +the more precise answer, since it also carries the exact checkpoint, tensor-parallel size, and +minimum SM version per entry. It covers language models, VLMs (Qwen2.5-VL, Qwen3-VL, +Phi-4-multimodal, Nemotron Omni), EAGLE3/Medusa drafters, and diffusion models. ## Supported Quantization Formats @@ -50,9 +38,9 @@ | SGLang | `quantization="modelopt"` | `quantization="modelopt_fp4"` | | TRT-LLM | auto-detected from checkpoint | auto-detected from checkpoint | -## Models not in this list +## Models not in the matrix -This matrix covers officially validated combinations. For unlisted models: +The matrix covers officially validated combinations, not the full set of what will run. For unlisted models: 1. **Check the framework's own docs** — vLLM and SGLang support many HuggingFace models natively. Use WebSearch to check `vllm supported models` or `sglang supported models`. 2. **Try it** — if the model uses standard `nn.Linear` layers and has `hf_quant_config.json`, vLLM/SGLang will likely work with `--quantization modelopt`. @@ -64,4 +52,5 @@ This matrix covers officially validated combinations. For unlisted models: - **B300/GB300 are `sm_103`** and need a **CUDA-13** serving image — from v0.20.0 the unsuffixed tag is CUDA-13 (`-cu129` opts back to CUDA 12); `cu12` images lack the `sm_103` FP4 kernel and serve NVFP4 as gibberish or error out. See the CUDA-13 note in the deployment `SKILL.md`. - **Verify the GPU with `nvidia-smi`** before choosing the image — cluster GPU labels can be stale. - INT4_AWQ and W4A8_AWQ are only supported by TRT-LLM (not vLLM or SGLang). -- Source: `examples/hf_ptq/README.md` and `docs/source/deployment/3_unified_hf.rst` +- For VLMs, only the language model is quantized; the vision encoder stays in high precision, so multimodal serving depends on the framework's own support for that architecture. +- Source: `docs/source/deployment/3_unified_hf.rst` and `tests/examples/hf_ptq/test_deploy.py` diff --git a/docs/source/deployment/3_unified_hf.rst b/docs/source/deployment/3_unified_hf.rst index 59a2782c5e7..a564530666a 100644 --- a/docs/source/deployment/3_unified_hf.rst +++ b/docs/source/deployment/3_unified_hf.rst @@ -51,48 +51,140 @@ The unified HF export API supports the following quantization formats: 5. INT4_AWQ - 4-bit integer with AWQ optimization 6. W4A8_AWQ - 4-bit weights and 8-bit activations with AWQ optimization -Framework-Specific Support +Minimum Framework Versions -------------------------- -TensorRT-LLM -~~~~~~~~~~~~ +=============== ================= +Framework Minimum version +=============== ================= +TensorRT-LLM v1.2.0 +vLLM v0.10.1 +SGLang v0.4.10 +=============== ================= + +These are the oldest versions expected to load a unified HF checkpoint. The matrix below is +validated against the versions CI currently runs, which are newer — see the container tags in +``.github/workflows/``. + +.. _unified-hf-support-matrix: + +Model Support Matrix +-------------------- + +Legend: + +* ✅ — covered by the release deployment test suite + (`tests/examples/hf_ptq/test_deploy.py `_), + which loads the exported checkpoint in the framework and runs generation. +* ⚠ — documented as working previously but not in the current test suite; expected to work, unvalidated. +* ``-`` — not currently covered. It may still work; see `Models not listed here`_. + +Language models +~~~~~~~~~~~~~~~ + +============================================ ============== ============ ====== ======== +Model Quant format TensorRT-LLM vLLM SGLang +============================================ ============== ============ ====== ======== +Llama 3.1, 3.3 FP8, NVFP4 ✅ ✅ ✅ +Llama 4 Scout, Maverick FP8 ✅ ✅ ✅ +Llama 4 Scout NVFP4 ✅ ✅ ✅ +Llama Nemotron Super 49B v1, v1.5 FP8 ✅ ✅ ✅ +Llama Nemotron Ultra 253B v1 FP8 ✅ ✅ ✅ +Nemotron 3 Nano 30B-A3B FP8, NVFP4 ✅ ✅ ✅ +Nemotron 3 Super 120B-A12B FP8, NVFP4 ✅ ✅ ✅ +Nemotron 3 Ultra 550B-A55B NVFP4 ✅ ✅ ✅ +DeepSeek R1, R1-0528 NVFP4 ✅ ✅ ✅ +DeepSeek V3, V3.1, V3.2 NVFP4 ✅ ✅ ✅ +DeepSeek V4 Flash NVFP4 ✅ ✅ ✅ +DeepSeek V4 Pro NVFP4 \- ✅ ✅ +Qwen 3 (8B, 14B, 32B) FP8, NVFP4 ✅ ✅ ✅ +Qwen 3 MoE 235B-A22B FP8, NVFP4 ✅ ✅ ✅ +Qwen 3 MoE 30B-A3B NVFP4 ✅ ✅ ✅ +Qwen 3 Coder 480B-A35B NVFP4 ✅ ✅ ✅ +Qwen 3-Next 80B-A3B NVFP4 ✅ ✅ ✅ +Qwen 3.5 397B-A17B NVFP4 ✅ ✅ ✅ +Qwen 3.5 122B-A10B, Qwen 3.6 35B-A3B NVFP4 \- ✅ \- +Qwen 2.5 FP8 ⚠ ⚠ ⚠ +Qwen 2.5 NVFP4 ⚠ ⚠ \- +QwQ-32B FP8 ⚠ ⚠ ⚠ +QwQ-32B NVFP4 ⚠ ⚠ \- +Phi-4 reasoning-plus FP8, NVFP4 ✅ ✅ ✅ +Gemma 4 31B NVFP4 ✅ ✅ ✅ +Gemma 4 26B-A4B NVFP4 \- ✅ \- +GLM-4.7, GLM-5, GLM-5.2 NVFP4 ✅ ✅ ✅ +GLM-5.1 NVFP4 \- ✅ ✅ +Kimi K2-Thinking, K2.5 NVFP4 ✅ ✅ ✅ +Kimi K2.6 NVFP4 \- ✅ \- +MiniMax M2.5, M3 NVFP4 ✅ ✅ ✅ +Mixtral 8x7B FP8 ⚠ ⚠ ⚠ +Mixtral 8x7B NVFP4 ⚠ \- \- +============================================ ============== ============ ====== ======== + +Vision-language and multimodal models +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +For VLMs, modelopt quantizes the language model only; the vision encoder is kept in high precision. +The exported checkpoint therefore relies on the serving framework's own multimodal support for that +architecture — see the +`TensorRT-LLM multimodal support matrix `_. + +============================================ ============== ============ ====== ======== +Model Quant format TensorRT-LLM vLLM SGLang +============================================ ============== ============ ====== ======== +Qwen 2.5-VL 7B FP8, NVFP4 ✅ ✅ ✅ +Qwen 3-VL 235B-A22B NVFP4 ✅ ✅ ✅ +Phi-4-multimodal FP8, NVFP4 ✅ ✅ ✅ +Nemotron 3 Nano Omni 30B-A3B FP8, NVFP4 ✅ ✅ ✅ +============================================ ============== ============ ====== ======== + +Speculative decoding drafters +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Drafters are deployed on top of their base checkpoint. vLLM is not currently covered for these. + +Unlike the tables above, drafter coverage is conditional: the EAGLE3 tests run only when +``MODELOPT_LOCAL_EAGLE_MODEL`` points at a directory containing the drafter, and skip otherwise. +✅ below means "covered when that drafter is available locally". + +============================================================ ============ ============ ====== ======== +Drafter Quant format TensorRT-LLM vLLM SGLang +============================================================ ============ ============ ====== ======== +EAGLE3 for Llama 3.3 70B, Llama 4 Maverick FP8 ✅ \- ✅ +EAGLE3 for Qwen 3 235B-A22B (incl. Thinking-2507, FP4) BF16, NVFP4 ✅ \- ✅ +EAGLE3 for Qwen 3 30B-A3B-Thinking-2507 BF16 ✅ \- ✅ +EAGLE3 for Kimi K2-Thinking, K2.5, K2.6 NVFP4 ✅ \- ✅ +EAGLE3 for gpt-oss-120b BF16 ✅ \- ✅ +Medusa for Llama 3.1 8B FP8 ✅ \- ✅ +============================================================ ============ ============ ====== ======== + +Diffusion models +~~~~~~~~~~~~~~~~ + +============================================ ============== ============ ====== ======== +Model Quant format TensorRT-LLM vLLM SGLang +============================================ ============== ============ ====== ======== +Wan 2.2 T2V A14B FP8, NVFP4 ✅ \- ✅ +DiffusionGemma 26B-A4B NVFP4 ✅ ✅ ✅ +============================================ ============== ============ ====== ======== -Models: - * Llama 4, 3.x (FP8, NVFP4) - * Qwen 3, 2.5 (FP8, NVFP4) - * Qwen 3 MoE (FP8, NVFP4) - * Qwen 3-VL (FP8, NVFP4) - * Deepseek R1/V3 (NVFP4) - * Mixtral 8x7B (FP8, NVFP4) - * Medusa (FP8) - * Eagle (FP8) - -Requirements: TensorRT-LLM v0.17.0 or later - -vLLM -~~~~ - -Models: - * Llama 4, 3.x (FP8, NVFP4) - * Qwen 3, 2.5 (FP8, NVFP4) - * Qwen 3 MoE (FP8, NVFP4) - * Mixtral 8x7B (FP8) - * Deepseek R1/V3 (NVFP4) - -Requirements: vLLM v0.10.1 or later - -SGLang -~~~~~~ +.. note:: + NVFP4 inference requires Blackwell GPUs. Hopper can produce an NVFP4 checkpoint but cannot serve + it. On B300/GB300 (``sm_103``) use a CUDA-13 build of the serving framework; CUDA-12 builds lack + the ``sm_103`` FP4 kernels. -Models: - * Llama 4, 3.x (FP8, NVFP4) - * Qwen 3, 2.5 (FP8, NVFP4) - * Qwen 3 MoE (FP8, NVFP4) - * Deepseek R1/V3 (NVFP4) +Models not listed here +~~~~~~~~~~~~~~~~~~~~~~ -Requirements: SGLang v0.4.10 or later +This matrix records the combinations modelopt validates. It is not an exhaustive list of what will +run: vLLM, SGLang, and TensorRT-LLM load unified HF checkpoints generically, so a model built from +standard ``nn.Linear`` layers with an ``hf_quant_config.json`` will often deploy without any modelopt +change. Check the serving framework's own model support list first, then try it. -Note: While other models and quantization formats may work, they have not been thoroughly tested and validated. +The exact checkpoints behind every ``Y`` above, including tensor-parallel size and minimum SM +version, are listed in +`tests/examples/hf_ptq/test_deploy.py `__; +most are published under the +`NVIDIA Hugging Face organization `_. Deployment with Selected Inference Frameworks @@ -102,7 +194,7 @@ Deployment with Selected Inference Frameworks Follow the `TensorRT-LLM installation instructions. `_ - Currently we support fp8 and nvfp4 quantized models for TensorRT-LLM deployment, you need v0.17.0 or later version of TensorRT-LLM. + FP8 and NVFP4 quantized models are supported; you need v1.2.0 or later version of TensorRT-LLM. To run modelopt quantized model from Huggingface model hub, e.g., `nvidia/Llama-3.1-8B-Instruct-FP8`_, refer to the sample code below: @@ -136,7 +228,8 @@ Deployment with Selected Inference Frameworks Follow `vLLM installation instructions. `_ - Currently we support fp8 quantized models (without fp8 kv cache) for vLLM deployment, you need v0.6.5 or later version of vLLM. + FP8 and NVFP4 quantized models are supported; you need v0.10.1 or later version of vLLM. Pass + ``quantization="modelopt"`` for FP8 and ``quantization="modelopt_fp4"`` for NVFP4. To run modelopt quantized model from Huggingface model hub, e.g., `nvidia/Llama-3.1-8B-Instruct-FP8`_, refer to the sample code below: @@ -171,7 +264,8 @@ Deployment with Selected Inference Frameworks Follow the `SGLang installation instructions. `_ - Currently we support fp8 quantized models (without fp8 kv cache) for SGLang deployment, you need to use the main branch of SGLang (since Jan 6, 2025) and build it from source. + FP8 and NVFP4 quantized models are supported; you need v0.4.10 or later version of SGLang. Pass + ``quantization="modelopt"`` for FP8 and ``quantization="modelopt_fp4"`` for NVFP4. To run modelopt quantized model from Huggingface model hub, e.g., `nvidia/Llama-3.1-8B-Instruct-FP8`_, refer to the sample code below: diff --git a/examples/diffusers/README.md b/examples/diffusers/README.md index a9efb5fc3a3..3c8e5c80876 100644 --- a/examples/diffusers/README.md +++ b/examples/diffusers/README.md @@ -78,7 +78,7 @@ mtq.quantize(model=transformer, config=quant_config, forward_func=forward_pass) > *1.The w4a8_awq is an experimental quantization scheme that may result in a higher accuracy penalty.* -> *2.A selective set of the popular models are internally tested. The actual model support list may be longer. NVFP4 inference requires Blackwell GPUs and TensorRT-LLM v0.17 or later* +> *2.A selective set of the popular models are internally tested. The actual model support list may be longer. NVFP4 inference requires Blackwell GPUs and TensorRT-LLM v1.2 or later* > *3.The SVDQuant Perf in TRT might not good as the [Nunchaku: MIT-Nvidia](https://github.com/nunchaku-tech/nunchaku) at this moment.* diff --git a/examples/hf_ptq/README.md b/examples/hf_ptq/README.md index e80926bea91..573c79cd49a 100755 --- a/examples/hf_ptq/README.md +++ b/examples/hf_ptq/README.md @@ -130,7 +130,7 @@ Please reference our [framework scripts](#framework-scripts) and our [docs](http > *2.For some models, there is only support for exporting quantized checkpoints.* \ > *3.W4A8_AWQ is only available on some models but not all* \ > *4.For some models, KV cache quantization may result in a higher accuracy penalty.* \ -> *5.A selective set of the popular models are internally tested. The actual model support list may be longer. NVFP4 inference requires Blackwell GPUs and TensorRT-LLM v0.17 or later* \ +> *5.A selective set of the popular models are internally tested. The actual model support list may be longer. NVFP4 inference requires Blackwell GPUs and TensorRT-LLM v1.2 or later* \ > *6.Some models currently support export to HF format only.* \ > *7.[PTQ for DeepSeek](../deepseek/README.md)* \ > *8.GLM-4.7 has MTP (Multi-Token Prediction) layers that are automatically loaded and excluded from quantization.* \ @@ -588,27 +588,20 @@ print(llm_fp8.generate(["What's the age of the earth? "])) ### Unified HF Checkpoint Deployment Model Support Matrix -| Model | Quant format | TRT-LLM | vLLM | SGLang | -| :---: | :---: | :---: | :---: | :---: | -| LLAMA 3.x | FP8 | ✅ | ✅ | ✅ | -| LLAMA 3.x | FP4 | ✅ | ✅ | ✅ | -| LLAMA 4 | FP8 | ✅ | - | ✅ | -| LLAMA 4 | FP4 | ✅ | - | - | -| DS-R1 | FP8 | ✅ | ✅ | ✅ | -| DS-R1 | FP4 | ✅ | ✅ | ✅ | -| DS-V3 | FP8 | ✅ | ✅ | ✅ | -| DS-V3 | FP4 | ✅ | ✅ | ✅ | -| QWen3 | FP8 | ✅ | ✅ | ✅ | -| QWen3 | FP4 | ✅ | ✅ | - | -| QWen3 MoE | FP8 | ✅ | ✅ | ✅ | -| QWen3 MoE | FP4 | ✅ | - | - | -| QWen3.5 MoE | FP4 | - | - | ✅ | -| QWen2.5 | FP8 | ✅ | ✅ | ✅ | -| QWen2.5 | FP4 | ✅ | ✅ | - | -| QwQ-32B | FP8 | ✅ | ✅ | ✅ | -| QwQ-32B | FP4 | ✅ | ✅ | - | -| Mixtral 8x7B | FP8 | ✅ | ✅ | ✅ | -| Mixtral 8x7B | FP4 | ✅ | - | - | +The deployment support matrix — which model families and quantization formats are validated on +TRT-LLM, vLLM, and SGLang, including vision-language models, speculative decoding drafters, and +diffusion models — lives in the documentation so there is a single copy to keep current: + +**[Unified HF Checkpoint → Model Support Matrix](https://nvidia.github.io/Model-Optimizer/deployment/3_unified_hf.html#model-support-matrix)** + +Every validated entry there is backed by [`tests/examples/hf_ptq/test_deploy.py`](../../tests/examples/hf_ptq/test_deploy.py), +which loads the exported checkpoint in each framework and runs generation. That file is also the +place to look for the exact checkpoint, tensor-parallel size, and minimum SM version behind each +entry. + +> *Note: the matrix records what modelopt validates, not the full set of what will run. vLLM, SGLang, +> and TRT-LLM load unified HF checkpoints generically, so unlisted models frequently deploy without +> any modelopt change — check the serving framework's own model support list and try it.* ### (Legacy) TensorRT-LLM Checkpoints