Add agent stt sdk by giorgosHadji · Pull Request #127 · speechmatics/speechmatics-python-sdk

giorgosHadji · 2026-08-18T11:14:21Z

Add agent stt sdk which will communicate with our AgentSTT offering - add docs here.

This MR doesn't remove/deprecate the voice sdk - that will happen in follow up MRs.

How to test this:
Get an api key, and point to rt preview (SPEECHMATICS_RT_URL=wss://preview.rt.speechmatics.com/v2)

Install locally the agent stt sdk - > python -m pip install -e sdk/agent_stt[dev]

Then run for a file based test : python examples/agent_stt/realtime_file/main.py
or you can use an interactive script (using microphone):
python .\examples\agent_stt\microphone\main.py

speechmatics-agent-stt is an extension of the RT SDK for the Voice Agent Service. The service works in segments rather than word groups and reports speech and turn events, so the client consumes AddSegment/AddPartialSegment and accumulates the session transcript from them. The SDK runs no VAD or turn detection of its own. Either the service's VAD closes turns (VADMode.SERVER), or the host framework's does (VADMode.CLIENT) by calling finalize(), which sends ForceEndOfUtterance stamped with the audio position at the moment of the call. This is what Pipecat and LiveKit need, and replaces the voice SDK's in-process VAD and smart-turn models. - endpoint is the RT URL plus /agent, optionally plus a service profile - TranscriptionConfig adds vad_mode, vad_config and emit_sentences, and leaves model unset so it cannot conflict with the profile's locked operating_point - every server message is kept in client.events, including unmodelled ones - 16 kHz raw PCM defaults, as the service requires Verified against the real service with a stub transcriber behind it. Nothing in the voice SDK or Pipecat is touched; see sdk/agent_stt/PLAN.md for the Pipecat migration plan. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Engine silence-based end of utterance is off for the Agent STT service, so TurnDetectionMode.FIXED has nothing to map onto and is removed rather than aliased. Records the settings that go with it, and notes in the SDK README that a turn ends only via the service VAD or finalize(). SMART_TURN stays open: it ran an in-process model, which this SDK does not. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The service has no smart-turn endpoint yet and this SDK loads no models, so the mode cannot be honoured. It costs nothing: any host-side endpointing, including Pipecat's own turn analyzer, reaches the service through finalize(). Makes VADMode.CLIENT explicit that it is agnostic about what produced the end-of-speech signal - VAD, turn model, or push-to-talk. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

A typed entry for transcription_config.additional_vocab, so callers can pass words with pronunciation hints instead of raw dicts. Both forms are accepted. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The SDK previously left `model` unset, on the reasoning that the service's default profile pins and locks `operating_point: enhanced`, so sending a model name alongside it would put both keys in the merged StartRecognition. That reasoning held for a direct connection to the service websocket, but it is not how requests actually reach it: they go through a proxy first, which resolves the Agent STT model name onto the engine's operating point. The transcriber therefore never sees a name it has no notion of, and sending `linden-1` is safe. Replace the re-exported RT `Model` with an Agent STT one. The RT models (`enhanced`, `standard`) are not Agent STT models, so leaving them reachable from this package only invited configs the service would reject. Default resolution uses the same `_UNSET` sentinel pattern as `rt.TranscriptionConfig` rather than a plain default, because the deprecated `operating_point` has to suppress the model rather than collide with it: passing `operating_point` leaves `model` unsent, and passing both still raises. `DEFAULT_MODEL` is a separate constant and the docstrings name it rather than the value, so linden-2 is one enum member plus, if it becomes the default, one line. The end-to-end verification recorded in PLAN.md predates this and ran against the service with a stub transcriber and no proxy, so it exercised neither the model default nor its resolution. Noted there as still needing a run against the real proxy. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…_trigger option

Smiljanic19A · 2026-08-26T09:32:12Z

+        result.pop("turn_detection_mode", None)
+        vad_config = result.pop("vad_config", None) or {}
+        vad_config["enabled"] = self.turn_detection_mode is not TurnDetectionMode.EXTERNAL
+        result["vad_config"] = vad_config


QQ:
The spec for Agent STT will reject vad_config - so why do we inject it here?

Not sure whether the problem is in the specs or in the SDK

@LArmstrongDev Isn't having external (so an option to turn vad on/off a requirement for integrations?

I am pretty certain you are right @giorgosHadji we do need a switch to enable/disable vad - i'm just pointing out our switch does not work atm.

Will discuss this with Lorna tomorrow at our sync - but loosening the specs makes the most sense for me here

The spec updates have not yet been made for turn_detection_mode because it is in review with Andrew on the naming front. This will be the configuration field that is used to dictate which turn detection mode is used for the session.

Cool, I will leave it as is, if people are happy with the code and, and once this (and its name is finalized and spec is updated) I can change it here as the last step I guess)

Co-authored-by: Lorna Armstrong <lorna.armstrong@speechmatics.com>

… external)