Skip to content

Conversation

@roomote
Copy link
Contributor

@roomote roomote bot commented Jan 31, 2026

Summary

This PR attempts to address Issue #11132.

Problem

LiteLLM's auto_router feature passes user content to embedding models for semantic routing. However, embedding models expect plain strings, not arrays of content blocks. When Roo Code sends messages with array content like:

{"role": "user", "content": [{"type": "text", "text": "..."}]}

The auto_router fails because it passes this array directly to embeddings, which is incompatible.

Solution

This PR adds a new litellmFlattenContent option (enabled by default) that flattens array content to simple string format before sending to LiteLLM:

{"role": "user", "content": "..."}

Changes

  • packages/types/src/provider-settings.ts: Add litellmFlattenContent option to the LiteLLM schema
  • src/api/providers/lite-llm.ts: Add flattenMessageContent() method and apply it in createMessage() when the option is enabled (default: true)
  • webview-ui/src/components/settings/providers/LiteLLM.tsx: Add UI checkbox for the option
  • webview-ui/src/i18n/locales/*/settings.json: Add translations for all 17 supported languages
  • src/api/providers/tests/lite-llm.spec.ts: Add 5 comprehensive tests for the flattening logic

Notes

  • The option is enabled by default for maximum compatibility with LiteLLM's auto_router
  • Users who need multimodal content (images) can disable this option
  • Content containing non-text blocks (images) is preserved as arrays even with the option enabled

Feedback and guidance are welcome!


Important

Adds litellmFlattenContent option to flatten message content for LiteLLM's auto_router, with schema, UI, and test updates.

  • Behavior:
    • Adds litellmFlattenContent option to flatten array content to strings for LiteLLM's auto_router compatibility.
    • Default behavior is to flatten content unless litellmFlattenContent is set to false.
    • Content with non-text blocks (e.g., images) remains as arrays.
  • Implementation:
    • Adds litellmFlattenContent to provider-settings.ts schema.
    • Implements flattenMessageContent() in lite-llm.ts and applies it in createMessage().
    • Adds UI checkbox for the option in LiteLLM.tsx.
  • Testing:
    • Adds tests in lite-llm.spec.ts to verify flattening logic under different configurations.
  • Localization:
    • Updates settings.json in 17 locales to include translations for the new option.

This description was created by Ellipsis for 6fe9a9e. You can customize this summary. It will automatically update as commits are pushed.

LiteLLM's auto_router feature passes user content to embedding models for
semantic routing. However, embedding models expect plain strings, not arrays
of content blocks. This causes errors when Roo Code sends messages with array
content format.

This commit adds a new `litellmFlattenContent` option (enabled by default)
that flattens array content to simple string format before sending to LiteLLM.

Changes:
- Add litellmFlattenContent option to provider-settings.ts schema
- Add flattenMessageContent method to LiteLLMHandler
- Apply flattening in createMessage when option is enabled (default: true)
- Add UI checkbox in LiteLLM settings component
- Add translations for all supported languages
- Add comprehensive tests for the flattening logic

Users who need multimodal content (images) can disable this option.

Fixes #11132
@roomote
Copy link
Contributor Author

roomote bot commented Jan 31, 2026

Rooviewer Clock   See task on Roo Cloud

Reviewed the implementation of the litellmFlattenContent option. Found one issue that could cause the feature to not work as intended in certain configurations.

  • Content flattening is undone by prompt caching logic when both features are enabled (comment)

Mention @roomote in a comment to request specific changes to this pull request or fix all unresolved issues.

Comment on lines +162 to +166
// Flatten array content to string for compatibility with LiteLLM's auto_router
// This is enabled by default (when litellmFlattenContent is undefined or true)
if (this.options.litellmFlattenContent !== false) {
openAiMessages = this.flattenMessageContent(openAiMessages)
}
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When both litellmFlattenContent (enabled by default) and litellmUsePromptCache are enabled with a model that supports prompt caching, the flattening is undone. The flattening converts array content to strings here, but lines 194-207 below wrap string content back into arrays to add cache_control. This means auto_router users who also enable prompt caching will still encounter the embedding model failure this PR intends to fix. Consider either moving the flattening after the prompt caching logic, or having the prompt caching logic skip re-wrapping when flattening is enabled.

Fix it with Roo Code or mention @roomote and request a fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant