Skip to content

fix(profiles): gate DeepSeek thinking-off default on reasoning-hint compatibility#122

Open
himorishige wants to merge 1 commit into
NVIDIA-NeMo:mainfrom
himorishige:fix/deepseek-overrides-reasoning-hint
Open

fix(profiles): gate DeepSeek thinking-off default on reasoning-hint compatibility#122
himorishige wants to merge 1 commit into
NVIDIA-NeMo:mainfrom
himorishige:fix/deepseek-overrides-reasoning-hint

Conversation

@himorishige

@himorishige himorishige commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Fixes the tier-call half of #121.

What

apply_deepseek_overrides() unconditionally injects the vLLM-only chat_template_kwargs.enable_thinking=false hint for any target whose model id contains deepseek-v4. Providers with strict request validation (observed: Fireworks AI serverless) reject the unknown field with HTTP 400 Extra inputs are not permitted, breaking every routed call for such tiers.

This PR gates the body default on model_accepts_reasoning_hint() — the same model-id compatibility check LLMClassifierConfig.disable_reasoning auto-detect already applies before sending the identical hint on classifier calls. The provider-neutral X-Inference-Priority header default is unchanged, and explicitly provided extra_body still wins as before.

Pairs with #123 (adds fireworks to the deny list); together they stop the 400 for Fireworks-served deepseek-v4 targets.

Validation

  • New unit test: deny-listed model id keeps extra_body untouched while still receiving the batch-priority header; existing TestDeepSeekOverrides cases unchanged and passing.
  • tests/test_deterministic_routing_profile.py + tests/test_reasoning_hint.py: 47 passed. Two failures (TestProfileStructure::test_shared_stats_accumulator, test_overflow_reroutes_to_custom_strong_id_through_full_profile) are pre-existing on unmodified main in my environment (same class of stale-build failure noted in feat(serve): route-log per-request routing stats for profile-config serve #118's validation notes).

Summary by CodeRabbit

  • Bug Fixes

    • Improved compatibility for DeepSeek models that do not support reasoning hints.
    • Prevented unsupported “thinking off” request parameters that could cause HTTP 400 errors.
    • Preserved batch inference prioritization for affected requests.
  • Tests

    • Added coverage to verify incompatible DeepSeek models receive valid request settings.

…ompatibility

apply_deepseek_overrides() unconditionally injects the vLLM-only
chat_template_kwargs.enable_thinking=false hint for any target whose
model id contains "deepseek-v4". OpenAI-compatible providers with
strict request validation reject the unknown field with HTTP 400
("Extra inputs are not permitted"), which breaks every routed call
for deepseek-v4 tiers served outside the NVIDIA benchmark gateway
(observed on Fireworks AI serverless).

Gate the body default on model_accepts_reasoning_hint(), the same
model-id compatibility check the LLM classifier already applies before
sending the hint on its own calls. The provider-neutral
X-Inference-Priority header default is unchanged, and explicitly
provided extra_body still wins as before.

Signed-off-by: Hiroshi Morishige <hiroshi.morishige@gmail.com>
@coderabbitai

coderabbitai Bot commented Jul 23, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: b363c6d1-1f92-4b9a-a786-38228865ac43

📥 Commits

Reviewing files that changed from the base of the PR and between 060ad75 and 15e8ec1.

📒 Files selected for processing (2)
  • switchyard/lib/profiles/tier_target_builders.py
  • tests/test_deterministic_routing_profile.py

Walkthrough

DeepSeek v4 thinking-disabled overrides are now gated by model compatibility with reasoning hints, and a regression test covers incompatible model IDs while preserving batch-priority routing.

Changes

DeepSeek override compatibility

Layer / File(s) Summary
Reasoning-hint gating and regression coverage
switchyard/lib/profiles/tier_target_builders.py, tests/test_deterministic_routing_profile.py
apply_deepseek_overrides checks model_accepts_reasoning_hint before adding the DeepSeek v4 thinking override; the test verifies incompatible models omit extra_body while retaining the batch-priority header.

Estimated code review effort: 2 (Simple) | ~10 minutes

Poem

I’m a rabbit with code in my paws,
Thinking stays off when hints break the laws.
Batch headers hop neatly in line,
No extra body slips through the design.
Sniff, test, and celebrate—fine!

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly matches the main change: gating DeepSeek's thinking-off default on reasoning-hint compatibility.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

Comment @coderabbitai help to get the list of available commands.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant