v3:stt and tts models #4603
v3:stt and tts models #4603
Conversation
📝 WalkthroughWalkthroughAdds mode-aware STT support for Changes
Sequence Diagram(s)sequenceDiagram
participant Client as "Client"
participant STT as "STT"
participant SpeechStream as "SpeechStream"
participant SarvamWS as "Sarvam WS"
Client->>STT: start stream(mode)
STT->>SpeechStream: create(options including mode)
SpeechStream->>SarvamWS: open websocket (URL includes mode when model=="saaras:v3")
SarvamWS-->>SpeechStream: transcription events
SpeechStream-->>STT: forward transcripts/events
Note over SpeechStream,SarvamWS: SpeechStream.update_options(mode) -> validate -> reconnect with new mode
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 3✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing touches
🧹 Recent nitpick comments
📜 Recent review detailsConfiguration used: Organization UI Review profile: CHILL Plan: Pro 📒 Files selected for processing (1)
🧰 Additional context used📓 Path-based instructions (1)**/*.py📄 CodeRabbit inference engine (AGENTS.md)
Files:
🧬 Code graph analysis (1)livekit-plugins/livekit-plugins-sarvam/livekit/plugins/sarvam/stt.py (2)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (3)
🔇 Additional comments (5)
✏️ Tip: You can disable this entire section by setting Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
livekit-plugins/livekit-plugins-sarvam/livekit/plugins/sarvam/tts.py (1)
628-638: Gate pitch/loudness by model in streaming config.The HTTP path explicitly omits
pitchandloudnessfor v3-beta (see lines 489-492: "not supported in v3-beta"), but the streaming config sends them unconditionally. This inconsistency will cause v3-beta streaming sessions to fail with API rejection. Apply the same model check as the HTTP path.♻️ Suggested adjustment
- config_msg = { - "type": "config", - "data": { - "target_language_code": self._opts.target_language_code, - "speaker": self._opts.speaker, - "pitch": self._opts.pitch, - "pace": self._opts.pace, - "loudness": self._opts.loudness, - "enable_preprocessing": self._opts.enable_preprocessing, - "model": self._opts.model, - }, - } + config_data = { + "target_language_code": self._opts.target_language_code, + "speaker": self._opts.speaker, + "pace": self._opts.pace, + "enable_preprocessing": self._opts.enable_preprocessing, + "model": self._opts.model, + } + if self._opts.model == "bulbul:v2": + config_data["pitch"] = self._opts.pitch + config_data["loudness"] = self._opts.loudness + config_msg = {"type": "config", "data": config_data}
🤖 Fix all issues with AI agents
In `@livekit-plugins/livekit-plugins-sarvam/livekit/plugins/sarvam/tts.py`:
- Around line 399-405: When update_options sets a new model (the block that
assigns self._opts.model), also revalidate the currently set speaker
(self._opts.speaker) if no new speaker is passed: check compatibility of the
existing speaker with the requested model and raise a ValueError if
incompatible. Implement this by adding a compatibility check (e.g., call a
helper like is_speaker_supported(model, self._opts.speaker) or inline logic)
immediately after setting model in update_options (the same scope where model,
speaker and self._opts.model are handled) so switching to "bulbul:v3-beta" with
an incompatible current speaker (e.g., "anushka") fails early rather than
causing runtime API errors.
📜 Review details
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (2)
livekit-plugins/livekit-plugins-sarvam/livekit/plugins/sarvam/stt.pylivekit-plugins/livekit-plugins-sarvam/livekit/plugins/sarvam/tts.py
🧰 Additional context used
📓 Path-based instructions (1)
**/*.py
📄 CodeRabbit inference engine (AGENTS.md)
**/*.py: Format code with ruff
Run ruff linter and auto-fix issues
Run mypy type checker in strict mode
Maintain line length of 100 characters maximum
Ensure Python 3.9+ compatibility
Use Google-style docstrings
Files:
livekit-plugins/livekit-plugins-sarvam/livekit/plugins/sarvam/stt.pylivekit-plugins/livekit-plugins-sarvam/livekit/plugins/sarvam/tts.py
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (3)
- GitHub Check: unit-tests
- GitHub Check: type-check (3.9)
- GitHub Check: type-check (3.13)
🔇 Additional comments (4)
livekit-plugins/livekit-plugins-sarvam/livekit/plugins/sarvam/stt.py (1)
56-57: LGTM — model literal updated cleanly.livekit-plugins/livekit-plugins-sarvam/livekit/plugins/sarvam/tts.py (3)
50-50: LGTM — model enum expanded for v3-beta.
77-105: LGTM — speaker list expansion aligns with lowercased validation.
108-171: LGTM — compatibility mapping looks consistent with new speakers.
✏️ Tip: You can disable this entire section by setting review_details to false in your review settings.
| if model is not None: | ||
| if not model.strip(): | ||
| raise ValueError("Model cannot be empty") | ||
| if model not in ["bulbul:v2"]: | ||
| if model not in ["bulbul:v2", "bulbul:v3-beta"]: | ||
| raise ValueError(f"Unsupported model: {model}") | ||
| self._opts.model = model | ||
|
|
There was a problem hiding this comment.
Revalidate the existing speaker when switching models.
update_options(model=...) can set bulbul:v3-beta while keeping an incompatible current speaker (e.g., "anushka"), and the mismatch isn’t checked unless speaker is also passed. This can surface as runtime API errors later.
🐛 Proposed fix
if model is not None:
if not model.strip():
raise ValueError("Model cannot be empty")
if model not in ["bulbul:v2", "bulbul:v3-beta"]:
raise ValueError(f"Unsupported model: {model}")
self._opts.model = model
+ if speaker is None and not validate_model_speaker_compatibility(
+ model, self._opts.speaker
+ ):
+ compatible = MODEL_SPEAKER_COMPATIBILITY.get(model, {}).get("all", [])
+ raise ValueError(
+ f"Speaker '{self._opts.speaker}' is not compatible with model '{model}'. "
+ "Please choose a compatible speaker from: "
+ f"{', '.join(compatible)}"
+ )🤖 Prompt for AI Agents
In `@livekit-plugins/livekit-plugins-sarvam/livekit/plugins/sarvam/tts.py` around
lines 399 - 405, When update_options sets a new model (the block that assigns
self._opts.model), also revalidate the currently set speaker
(self._opts.speaker) if no new speaker is passed: check compatibility of the
existing speaker with the requested model and raise a ValueError if
incompatible. Implement this by adding a compatibility check (e.g., call a
helper like is_speaker_supported(model, self._opts.speaker) or inline logic)
immediately after setting model in update_options (the same scope where model,
speaker and self._opts.model are handled) so switching to "bulbul:v3-beta" with
an incompatible current speaker (e.g., "anushka") fails early rather than
causing runtime API errors.
There was a problem hiding this comment.
Actionable comments posted: 0
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
livekit-plugins/livekit-plugins-sarvam/livekit/plugins/sarvam/stt.py (1)
70-95: Duplicatemodefield definition will cause unexpected behavior.The
modefield is defined twice in this dataclass:
- Line 87:
mode: SarvamSTTModes | str = "transcribe"- Line 95:
mode: Literal["translate", "transcribe", "verbatim", "translit", "codemix"] = "transcribe"In Python dataclasses, duplicate field names are not allowed and will cause issues. Additionally, the docstring has duplicate entries for
modeat lines 77 and 81.🐛 Proposed fix: Remove duplicate field and docstring entry
`@dataclass` class SarvamSTTOptions: """Options for the Sarvam.ai STT service. Args: language: BCP-47 language code, e.g., "hi-IN", "en-IN" model: The Sarvam STT model to use mode: Mode for saaras:v3 (transcribe/translate/verbatim/translit/codemix) base_url: API endpoint URL (auto-determined from model if not provided) streaming_url: WebSocket streaming URL (auto-determined from model if not provided) prompt: Optional prompt for STT translate (saaras models only) - mode: Mode for saaras:v3 (transcribe/translate/verbatim/translit/codemix) """ language: str # BCP-47 language code, e.g., "hi-IN", "en-IN" api_key: str model: SarvamSTTModels | str = "saarika:v2.5" mode: SarvamSTTModes | str = "transcribe" base_url: str | None = None streaming_url: str | None = None prompt: str | None = None # Optional prompt for STT translate (saaras models only) high_vad_sensitivity: bool | None = None sample_rate: int = 16000 flush_signal: bool | None = None input_audio_codec: str | None = None - mode: Literal["translate", "transcribe", "verbatim", "translit", "codemix"] = "transcribe"
🧹 Nitpick comments (3)
livekit-plugins/livekit-plugins-sarvam/livekit/plugins/sarvam/stt.py (3)
282-284: Inconsistent pattern for checkingNotGivenvalues.Lines 282-284 use
isinstance(x, type(NOT_GIVEN))while thestream()method (lines 391-393) usesis_given(). For consistency and clarity, prefer theis_given()helper which is already imported.♻️ Proposed fix: Use is_given() consistently
- opts_language = self._opts.language if isinstance(language, type(NOT_GIVEN)) else language - opts_model = self._opts.model if isinstance(model, type(NOT_GIVEN)) else model - opts_mode = self._opts.mode if isinstance(mode, type(NOT_GIVEN)) else mode + opts_language = self._opts.language if not is_given(language) else language + opts_model = self._opts.model if not is_given(model) else model + opts_mode = self._opts.mode if not is_given(mode) else modeOr alternatively, to match the pattern in
stream():- opts_language = self._opts.language if isinstance(language, type(NOT_GIVEN)) else language - opts_model = self._opts.model if isinstance(model, type(NOT_GIVEN)) else model - opts_mode = self._opts.mode if isinstance(mode, type(NOT_GIVEN)) else mode + opts_language = language if is_given(language) else self._opts.language + opts_model = model if is_given(model) else self._opts.model + opts_mode = mode if is_given(mode) else self._opts.mode
560-602:update_optionssignature differs from the standard plugin pattern.The current signature requires
languageandmodelas mandatory parameters:def update_options(self, *, language: str, model: str, prompt: str | None = None, mode: str | None = None)The standard pattern (e.g., in
baseten/stt.py) usesNotGivenOrwithNOT_GIVENdefaults, enabling partial updates without requiring all parameters:def update_options(self, *, language: NotGivenOr[str] = NOT_GIVEN, ...)If partial updates are intended to be supported, consider aligning with the standard pattern. If mandatory parameters are intentional (to always reconnect with full config), this is acceptable but worth documenting.
573-584: Consider extracting the allowed modes set to reduce duplication.The same
allowed_modesset is defined in bothSarvamSTTOptions.__post_init__(lines 106-112) and here. Extracting to a module-level constant would improve maintainability.♻️ Proposed refactor
Add at module level (near line 57):
SARVAM_V3_ALLOWED_MODES: set[str] = {"transcribe", "translate", "verbatim", "translit", "codemix"}Then use in both validation locations:
- allowed_modes: set[str] = { - "transcribe", - "translate", - "verbatim", - "translit", - "codemix", - } - if mode not in allowed_modes: + if mode not in SARVAM_V3_ALLOWED_MODES:
📜 Review details
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
livekit-plugins/livekit-plugins-sarvam/livekit/plugins/sarvam/stt.py
🧰 Additional context used
📓 Path-based instructions (1)
**/*.py
📄 CodeRabbit inference engine (AGENTS.md)
**/*.py: Format code with ruff
Run ruff linter and auto-fix issues
Run mypy type checker in strict mode
Maintain line length of 100 characters maximum
Ensure Python 3.9+ compatibility
Use Google-style docstrings
Files:
livekit-plugins/livekit-plugins-sarvam/livekit/plugins/sarvam/stt.py
🧬 Code graph analysis (1)
livekit-plugins/livekit-plugins-sarvam/livekit/plugins/sarvam/stt.py (3)
livekit-plugins/livekit-plugins-baseten/livekit/plugins/baseten/stt.py (3)
model(128-129)update_options(168-195)update_options(224-244)livekit-plugins/livekit-plugins-mistralai/livekit/plugins/mistralai/stt.py (2)
model(83-84)update_options(90-107)livekit-agents/livekit/agents/utils/misc.py (1)
is_given(25-26)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (3)
- GitHub Check: type-check (3.9)
- GitHub Check: type-check (3.13)
- GitHub Check: unit-tests
🔇 Additional comments (6)
livekit-plugins/livekit-plugins-sarvam/livekit/plugins/sarvam/stt.py (6)
56-57: LGTM!The new type definitions for
SarvamSTTModelsandSarvamSTTModesare well-defined and consistent with the validation logic used throughout the file.
105-118: LGTM!The mode validation logic appropriately restricts mode options to
saaras:v3only, with a sensible default for other models.
173-174: LGTM!The mode parameter is correctly added to the WebSocket URL only for the
saaras:v3model.
197-239: LGTM!The
modeparameter is properly integrated into theSTT.__init__method and correctly passed through toSarvamSTTOptions.
298-299: LGTM!The mode is correctly added to the form data only for
saaras:v3model, consistent with the WebSocket URL building logic.
377-431: LGTM!The
stream()method correctly integrates themodeparameter with proper use ofis_given()and appropriate type validation.
✏️ Tip: You can disable this entire section by setting review_details to false in your review settings.
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Fix all issues with AI agents
In `@livekit-plugins/livekit-plugins-sarvam/livekit/plugins/sarvam/stt.py`:
- Around line 572-590: The internal mode state becomes inconsistent because
__post_init__ forces non-v3 models to "transcribe" but update_options allows
arbitrary modes; update update_options to mirror __post_init__'s normalization:
when model != "saaras:v3" ensure self._opts.mode is set to "transcribe" (or
ignore any provided mode) and only accept other modes when model == "saaras:v3";
apply this logic in the update_options method that sets self._opts.model and
self._opts.mode so the internal state stays consistent with API gating.
🧹 Nitpick comments (2)
livekit-plugins/livekit-plugins-sarvam/livekit/plugins/sarvam/stt.py (2)
94-94: Remove commented-out dead code.This line is a leftover from development and duplicates the
modefield already defined on line 86.🧹 Proposed fix
- `#mode`: Literal["translate", "transcribe", "verbatim", "translit", "codemix"] = "transcribe"
104-117: Consider extractingallowed_modesto a module-level constant.The same set of allowed modes is duplicated in
update_options(lines 573-579). Extracting to a constant improves maintainability and ensures consistency.♻️ Suggested refactor
Add a module-level constant near the type definitions:
# After line 57 SAARAS_V3_ALLOWED_MODES: set[str] = { "transcribe", "translate", "verbatim", "translit", "codemix", }Then use it in both
__post_init__andupdate_options:if self.model == "saaras:v3": - allowed_modes: set[str] = { - "transcribe", - "translate", - "verbatim", - "translit", - "codemix", - } - if self.mode not in allowed_modes: + if self.mode not in SAARAS_V3_ALLOWED_MODES: raise ValueError( "mode must be one of transcribe, translate, verbatim, translit, codemix" )
📜 Review details
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
livekit-plugins/livekit-plugins-sarvam/livekit/plugins/sarvam/stt.py
🧰 Additional context used
📓 Path-based instructions (1)
**/*.py
📄 CodeRabbit inference engine (AGENTS.md)
**/*.py: Format code with ruff
Run ruff linter and auto-fix issues
Run mypy type checker in strict mode
Maintain line length of 100 characters maximum
Ensure Python 3.9+ compatibility
Use Google-style docstrings
Files:
livekit-plugins/livekit-plugins-sarvam/livekit/plugins/sarvam/stt.py
🧬 Code graph analysis (1)
livekit-plugins/livekit-plugins-sarvam/livekit/plugins/sarvam/stt.py (1)
livekit-agents/livekit/agents/utils/misc.py (1)
is_given(25-26)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (3)
- GitHub Check: type-check (3.13)
- GitHub Check: type-check (3.9)
- GitHub Check: unit-tests
🔇 Additional comments (4)
livekit-plugins/livekit-plugins-sarvam/livekit/plugins/sarvam/stt.py (4)
56-57: LGTM!Type definitions for the new model and modes are well-defined using
Literaltypes for proper type safety.
172-173: LGTM!Mode parameter is correctly added to WebSocket URL only for
saaras:v3model.
281-284: LGTM!Mode parameter propagation follows the same pattern as language and model, with proper fallback to instance defaults.
390-399: LGTM!Mode handling in
stream()follows the established pattern for language and model parameters, with proper type guards.
✏️ Tip: You can disable this entire section by setting review_details to false in your review settings.
livekit-plugins/livekit-plugins-sarvam/livekit/plugins/sarvam/stt.py
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
livekit-plugins/livekit-plugins-sarvam/livekit/plugins/sarvam/stt.py (1)
255-300: Add mode value validation in non-streaming recognize.
_recognize_implaccepts any string formodewhenmodel == "saaras:v3", allowing invalid values to reach the API instead of failing fast. The same validation used inSarvamSTTOptions.__post_init__()and streamingupdate_options()should be applied here before sending the request.🛠️ Proposed fix
- opts_mode = self._opts.mode if not is_given(mode) else mode - if is_given(mode) and opts_model != "saaras:v3": - raise ValueError("mode is only supported when model is saaras:v3") + opts_mode = self._opts.mode if not is_given(mode) else mode + if opts_model != "saaras:v3": + if is_given(mode): + raise ValueError("mode is only supported when model is saaras:v3") + else: + allowed_modes: set[str] = { + "transcribe", + "translate", + "verbatim", + "translit", + "codemix", + } + if opts_mode not in allowed_modes: + raise ValueError( + "mode must be one of transcribe, translate, verbatim, translit, codemix" + )
🤖 Fix all issues with AI agents
In `@livekit-plugins/livekit-plugins-sarvam/livekit/plugins/sarvam/stt.py`:
- Around line 562-606: update_options currently changes self._opts.model but
doesn't refresh endpoint fields, so reconnection uses stale URLs; modify
update_options (around the code that sets self._opts.model) to also set
self._opts.base_url and self._opts.streaming_url based on the new model value:
if model == "saaras:v3" set base_url to
"https://api.sarvam.ai/speech-to-text-translate" and streaming_url to
"wss://api.sarvam.ai/speech-to-text-translate/ws", otherwise set base_url to
"https://api.sarvam.ai/speech-to-text" and streaming_url to
"wss://api.sarvam.ai/speech-to-text/ws"; ensure these assignments occur before
triggering the reconnection/logging so the reconnect uses the updated endpoints.
📜 Review details
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
livekit-plugins/livekit-plugins-sarvam/livekit/plugins/sarvam/stt.py
🧰 Additional context used
📓 Path-based instructions (1)
**/*.py
📄 CodeRabbit inference engine (AGENTS.md)
**/*.py: Format code with ruff
Run ruff linter and auto-fix issues
Run mypy type checker in strict mode
Maintain line length of 100 characters maximum
Ensure Python 3.9+ compatibility
Use Google-style docstrings
Files:
livekit-plugins/livekit-plugins-sarvam/livekit/plugins/sarvam/stt.py
🧠 Learnings (1)
📓 Common learnings
Learnt from: CR
Repo: livekit/agents PR: 0
File: AGENTS.md:0-0
Timestamp: 2026-01-16T07:44:56.353Z
Learning: Implement Model Interface Pattern for STT, TTS, LLM, and Realtime models with provider-agnostic interfaces, fallback adapters for resilience, and stream adapters for different streaming patterns
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (3)
- GitHub Check: unit-tests
- GitHub Check: type-check (3.9)
- GitHub Check: type-check (3.13)
🔇 Additional comments (4)
livekit-plugins/livekit-plugins-sarvam/livekit/plugins/sarvam/stt.py (4)
55-116: LGTM — mode validation/defaulting is clear and consistent.
157-176: Good: mode is only appended for saaras:v3.
195-237: LGTM — mode is properly plumbed into options at construction.
377-433: LGTM — stream path correctly carries mode into per-stream options.
✏️ Tip: You can disable this entire section by setting review_details to false in your review settings.
1 participant
Added:
Summary by CodeRabbit
New Features
Behavior Changes
✏️ Tip: You can customize this high-level summary in your review settings.