fix(profiles): gate DeepSeek thinking-off default on reasoning-hint compatibility#122
Open
himorishige wants to merge 1 commit into
Open
Conversation
…ompatibility
apply_deepseek_overrides() unconditionally injects the vLLM-only
chat_template_kwargs.enable_thinking=false hint for any target whose
model id contains "deepseek-v4". OpenAI-compatible providers with
strict request validation reject the unknown field with HTTP 400
("Extra inputs are not permitted"), which breaks every routed call
for deepseek-v4 tiers served outside the NVIDIA benchmark gateway
(observed on Fireworks AI serverless).
Gate the body default on model_accepts_reasoning_hint(), the same
model-id compatibility check the LLM classifier already applies before
sending the hint on its own calls. The provider-neutral
X-Inference-Priority header default is unchanged, and explicitly
provided extra_body still wins as before.
Signed-off-by: Hiroshi Morishige <hiroshi.morishige@gmail.com>
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: 📒 Files selected for processing (2)
WalkthroughDeepSeek v4 thinking-disabled overrides are now gated by model compatibility with reasoning hints, and a regression test covers incompatible model IDs while preserving batch-priority routing. ChangesDeepSeek override compatibility
Estimated code review effort: 2 (Simple) | ~10 minutes Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the tier-call half of #121.
What
apply_deepseek_overrides()unconditionally injects the vLLM-onlychat_template_kwargs.enable_thinking=falsehint for any target whose model id containsdeepseek-v4. Providers with strict request validation (observed: Fireworks AI serverless) reject the unknown field with HTTP 400Extra inputs are not permitted, breaking every routed call for such tiers.This PR gates the body default on
model_accepts_reasoning_hint()— the same model-id compatibility checkLLMClassifierConfig.disable_reasoningauto-detect already applies before sending the identical hint on classifier calls. The provider-neutralX-Inference-Priorityheader default is unchanged, and explicitly providedextra_bodystill wins as before.Pairs with #123 (adds
fireworksto the deny list); together they stop the 400 for Fireworks-served deepseek-v4 targets.Validation
extra_bodyuntouched while still receiving the batch-priority header; existingTestDeepSeekOverridescases unchanged and passing.tests/test_deterministic_routing_profile.py+tests/test_reasoning_hint.py: 47 passed. Two failures (TestProfileStructure::test_shared_stats_accumulator,test_overflow_reroutes_to_custom_strong_id_through_full_profile) are pre-existing on unmodifiedmainin my environment (same class of stale-build failure noted in feat(serve): route-log per-request routing stats for profile-config serve #118's validation notes).Summary by CodeRabbit
Bug Fixes
Tests