[TRTLLM-14822][feat] deprecate WIDEEP MoE backend - #17119
Open
xxi-nv wants to merge 1 commit into
Open
Conversation
Raise an explicit error for the WIDEEP MoE backend in both get_moe_cls and create_moe_backend, so the backend cannot be selected through create_moe or by calling create_moe_backend directly. Wide expert parallelism and EPLB remain available through the other backends. TestDeepSeekV4FlashBase FP8 EPLB cases previously required WIDEEP because CutlassFp8BlockScaleGemmRunner takes a Hopper-only wgmma path with no SM100/SM103 implementation. DEEPGEMM routes the same block-scale weights through DeepGEMM, so those cases now use DEEPGEMM. Examples, docs and test lists that referenced WIDEEP are updated to CUTEDSL or DEEPGEMM accordingly. The layer_wise_benchmarks --scaled-from test is skipped because that feature is implemented by monkeypatching WideEPMoE.select_alltoall_method_type. Signed-off-by: xxi <xxi@nvidia.com>
xxi-nv
requested review from
Shixiaowei02,
arysef,
aswinvisva,
dc3671,
hyukn,
reasonsolo,
rosong11,
ruodil,
sunnyqgg and
xinhe-nv
July 31, 2026 09:12
xxi-nv
requested review from
JunyiXu-nv,
QiJune,
allisonlim-nv,
yingguo-trt and
yufeiwu-nv
July 31, 2026 09:12
Contributor
WalkthroughWIDEEP is deprecated and rejected during MoE backend selection and construction. CLI choices, deployment examples, runtime validation, DeepSeek accuracy coverage, QA manifests, and benchmark tests now use supported backends or remove WIDEEP cases. ChangesWIDEEP backend retirement
Estimated code review effort: 3 (Moderate) | ~20 minutes Possibly related PRs
Suggested labels: Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1🛠️ Fix failing CI checks 💡
🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tests/integration/defs/accuracy/test_llm_api_pytorch.py`:
- Line 4045: Add the exact DEEPGEMM parameterized node for
TestDeepSeekV4FlashBase::test_fp8_4gpus_online_eplb to the appropriate active
CI, test-db, and QA lists before recording its waiver, then add the matching
waiver for the known failure. Ensure the test is scheduled and gated rather than
only waived.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: aaf1a23f-c0ed-4f44-a653-8ef6f2ca34f4
📒 Files selected for processing (24)
docs/source/deployment-guide/deployment-guide-for-deepseek-r1-on-trtllm.mdexamples/layer_wise_benchmarks/run.pyexamples/llm-api/llm_sparse_attention.pyexamples/llm-api/quickstart_advanced.pyexamples/longbench/eval_longbench_v1.pyexamples/models/core/exaone/README.mdexamples/wide_ep/README.mdexamples/wide_ep/slurm_scripts/config.yamlexamples/wide_ep/slurm_scripts/kimi-k2-thinking.yamltensorrt_llm/_torch/models/modeling_deepseekv4.pytensorrt_llm/_torch/modules/fused_moe/create_moe.pytensorrt_llm/_torch/peft/lora/validation.pytensorrt_llm/tools/layer_wise_benchmarks/runner.pytests/integration/defs/accuracy/test_disaggregated_serving.pytests/integration/defs/accuracy/test_llm_api_pytorch.pytests/integration/defs/test_e2e.pytests/integration/test_lists/qa/llm_function_core.txttests/integration/test_lists/qa/llm_function_rtx6k.txttests/integration/test_lists/qa/llm_perf_disagg.ymltests/integration/test_lists/qa/llm_perf_multinode.txttests/integration/test_lists/test-db/l0_dgx_b200.ymltests/integration/test_lists/waives.txttests/unittest/_torch/lora/test_moe_lora_validator.pytests/unittest/tools/test_layer_wise_benchmarks.py
💤 Files with no reviewable changes (7)
- tests/integration/test_lists/qa/llm_perf_multinode.txt
- tests/integration/test_lists/waives.txt
- tests/integration/test_lists/qa/llm_function_core.txt
- tests/integration/test_lists/test-db/l0_dgx_b200.yml
- tests/integration/test_lists/qa/llm_function_rtx6k.txt
- tensorrt_llm/tools/layer_wise_benchmarks/runner.py
- tests/integration/test_lists/qa/llm_perf_disagg.yml
|
|
||
| @pytest.mark.skip_less_mpi_world_size(4) | ||
| @parametrize_with_ids("moe_backend", ["WIDEEP"]) | ||
| @parametrize_with_ids("moe_backend", ["DEEPGEMM"]) |
Contributor
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
rg -n -C 5 \
-e 'test_fp8_4gpus_online_eplb' \
-e 'TestDeepSeekV4FlashBase' \
tests/integration/test_lists/waives.txt \
tests/integration/test_lists/test-db \
tests/integration/test_lists/qa || trueRepository: NVIDIA/TensorRT-LLM
Length of output: 157
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo '--- source files ---'
git ls-files 'tests/integration/defs/accuracy/test_llm_api_pytorch.py' 'tests/integration/test_lists/waives.txt' 'tests/integration/test_lists/test-db/**' 'tests/integration/test_lists/qa/**'
echo '--- test definition and parametrization ---'
sed -n '3985,4085p' tests/integration/defs/accuracy/test_llm_api_pytorch.py
echo '--- matching entries in active list files ---'
rg -n -i -C 4 \
-e 'test_fp8_4gpus_online_eplb' \
-e 'TestDeepSeekV4FlashBase' \
-e 'DeepSeekV4FlashBase' \
-e 'DEEPGEMM' \
tests/integration/test_lists/waives.txt \
tests/integration/test_lists/test-db \
tests/integration/test_lists/qa || true
echo '--- all list entries containing the test name ---'
find tests/integration/test_lists/test-db tests/integration/test_lists/qa -type f -print0 |
xargs -0 grep -n -i -C 2 'test_fp8_4gpus_online_eplb' || trueRepository: NVIDIA/TensorRT-LLM
Length of output: 34383
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo '--- focused diff for the changed test file ---'
git diff --unified=12 -- tests/integration/defs/accuracy/test_llm_api_pytorch.py
echo '--- test-list and waiver semantics ---'
sed -n '1,220p' tests/integration/test_lists/qa/README.md
sed -n '1,220p' tests/integration/test_lists/test-db/README.md
sed -n '1,80p' tests/integration/test_lists/waives.txt
echo '--- exact and broader list matches ---'
rg -n -i \
-e 'TestDeepSeekV4FlashBase' \
-e 'test_fp8_4gpus_(static|online)_eplb' \
-e 'DeepSeekV4Flash-Base' \
tests/integration/test_lists/waives.txt \
tests/integration/test_lists/test-db \
tests/integration/test_lists/qa || true
echo '--- all references to the class and methods in repository metadata ---'
rg -n -i \
-e 'TestDeepSeekV4FlashBase' \
-e 'test_fp8_4gpus_(static|online)_eplb' \
tests/integration .github 2>/dev/null || trueRepository: NVIDIA/TensorRT-LLM
Length of output: 21636
Add the test to the active lists before adding a waiver.
TestDeepSeekV4FlashBase::test_fp8_4gpus_online_eplb[moe_backend=DEEPGEMM] has no local skip and is absent from waives.txt, test-db, and qa. It is not scheduled by active CI or QA, so a waiver alone will not gate it. Add the exact node to the intended lists, then add the matching waiver for the known failure. Test coverage is insufficient.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tests/integration/defs/accuracy/test_llm_api_pytorch.py` at line 4045, Add
the exact DEEPGEMM parameterized node for
TestDeepSeekV4FlashBase::test_fp8_4gpus_online_eplb to the appropriate active
CI, test-db, and QA lists before recording its waiver, then add the matching
waiver for the known failure. Ensure the test is scheduled and gated rather than
only waived.
Source: Path instructions
Collaborator
Author
|
/bot run --disable-fail-fast |
Collaborator
|
PR_Github #63033 [ run ] triggered by Bot. Commit: |
Collaborator
|
PR_Github #63033 [ run ] completed with state
|
Collaborator
Author
|
/bot run --disable-fail-fast |
Collaborator
|
PR_Github #63064 [ run ] triggered by Bot. Commit: |
Collaborator
|
PR_Github #63064 [ run ] completed with state
|
Collaborator
Author
|
/bot run --disable-fail-fast |
Collaborator
|
PR_Github #63132 [ run ] triggered by Bot. Commit: |
Collaborator
|
PR_Github #63132 [ run ] completed with state
|
Collaborator
Author
|
/bot run --disable-fail-fast |
Collaborator
|
PR_Github #63153 [ run ] triggered by Bot. Commit: |
Collaborator
|
PR_Github #63153 [ run ] completed with state |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Deprecate the
WIDEEPMoE backend by raising an explicit error in bothget_moe_clsandcreate_moe_backend, so it cannot be selected viacreate_moenor by callingcreate_moe_backenddirectly. Wide expert parallelism and EPLB stay available through the remaining backends (DEEPGEMMfor FP8 block-scale checkpoints,TRTLLM/CUTEDSL/CUTLASSotherwise).AUTOnever resolves toWIDEEP, so no implicit user path is affected.Why the FP8 EPLB tests can move off WIDEEP
TestDeepSeekV4FlashBaseFP8 EPLB cases used to requireWIDEEPbecause the default CUTLASS FP8 block-scale path (CutlassFp8BlockScaleGemmRunner::moeGemm) takes a Hopper-onlywgmmaroute with no SM100/SM103 implementation.WIDEEPavoided this via anis_sm_100f()check that switches to DeepGEMM.DEEPGEMMreaches the same kernel directly, so those cases now useDEEPGEMM.Other updates
WIDEEPnow point atCUTEDSL/DEEPGEMM.WIDEEPare removed.WideEPMoEis dropped from the mirroredswiglu_limitcapability tuple inmodeling_deepseekv4.pyto keep it consistent withcreate_moe.py.--scaled-fromtest is skipped: that feature is implemented by monkeypatchingWideEPMoE.select_alltoall_method_type.Test plan
Ran the full
TestDeepSeekV4FlashBaseclass on 4x B300 (SM103), wheel built with-a "100-real;103-real", usingTRTLLM_ACCURACY_NO_REFERENCE=1(this model has no registered MMLU/GSM8K reference entries):test_fp8_4gpus_static_eplb[moe_backend=DEEPGEMM]- PASSED (15m09s)test_auto_dtype[moe_backend=DEEPGEMM]- PASSED (7m46s)test_auto_dtype[moe_backend=TRTLLM]- PASSED (7m58s)test_fp8_chunked_prefill- PASSED (7m18s)test_fp8_4gpus_online_eplb[moe_backend=DEEPGEMM]- fails on a pre-existing incompatibility, see belowNote on the online EPLB case
test_fp8_4gpus_online_eplbfails withAssertionError: t.is_contiguous() must be Trueatmoe_load_balancer.py:151. This is not introduced by this PR:WideEPMoE._get_quant_methodreturns the sameDeepSeekFP8BlockScalesFusedMoEMethodDeepGemmthatDeepGemmFusedMoEreturns, so switching the backend does not change this code path._needs_e8m0_resmooth()is true for all Blackwell architectures, and the resulting scale tensor fromtransform_sf_into_required_layoutis non-contiguous, which the online EPLB host tensor sharer rejects.staticEPLB passeslayer_updates_per_iter=0and short-circuits that requirement.@skip_pre_blackwell, so every platform that can run this case hits the same assertion, including the previousWIDEEPparametrization.Fixing it requires deciding whether forcing contiguity is safe for the DeepGEMM scale layout, which is a separate concern from this deprecation.
Dev Engineer Review
WIDEEPnow fails explicitly inget_moe_clsandcreate_moe_backend.WIDEEPis removed from CLI choices, examples, documentation, validation, load balancing, and benchmark routing.CUTEDSLfor NVFP4 andDEEPGEMMorTRTLLMfor FP8.WideEPMoEremains available through supported backends and is removed only from theswiglu_limitcapability tuple.--scaled-fromrequires the removedWIDEEPall-to-all hook.WIDEEPcoverage and waivers without adding invalid entries.QA Engineer Review
Modified test code includes:
test_llm_api_pytorch.py.test_llm_api_pytorch.py.test_deepseek_r1_gen_scaled_from_16_dep, now skipped.test_moe_lora_validator.py.test_disaggregated_serving.py.test_e2e.py.The related CI and QA manifests remove obsolete
WIDEEPcases fromllm_function_core.txt,llm_function_rtx6k.txt,llm_perf_disagg.yml,llm_perf_multinode.txt,l0_dgx_b200.yml, andwaives.txt.Coverage uses supported backends, including
DEEPGEMM,TRTLLM, andCUTLASS. The static FP8 EPLB, automatic-dtype, and chunked-prefill tests passed on 4x B300. The online FP8 EPLB test still fails for a pre-existing issue.Verdict: needs follow-up because the online FP8 EPLB test remains failing.