STT & TTS Providers

RoomKit's voice pipeline uses pluggable Speech-to-Text (STT) and Text-to-Speech (TTS) providers. This guide covers all built-in providers, their configuration, and streaming capabilities.

STT Provider ABC

from __future__ import annotations

from roomkit.voice.stt import STTProvider, TranscriptionResult


class STTProvider(ABC):
    @property
    def name(self) -> str: ...

    @property
    def supports_streaming(self) -> bool: ...

    @property
    def supports_language_override(self) -> bool: ...
        """Whether transcribe/transcribe_stream honour a per-call language."""

    @property
    def supports_diarization(self) -> bool: ...
        """Whether final results say who spoke (TranscriptionResult.segments)."""

    async def transcribe(self, audio, *, language=None) -> TranscriptionResult:
        """Batch transcription — send all audio, get full text."""

    async def transcribe_stream(
        self, audio_stream, *, language=None
    ) -> AsyncIterator[TranscriptionResult]:
        """Streaming transcription — get partial results in real time."""

    async def warmup(self) -> None:
        """Pre-load models (optional)."""

    async def close(self) -> None:
        """Release resources."""

language on a call overrides the provider's configured language for that call only. A provider that cannot honour it answers False to supports_language_override, and VoiceChannel never passes a language to such a provider — an implementation written against the older signature keeps working unchanged.

TranscriptionResult

@dataclass
class TranscriptionResult:
    text: str
    is_final: bool = True
    confidence: float | None = None
    language: str | None = None
    words: list[dict[str, Any]] = []
    is_speech_start: bool = False
    segments: list[SpeakerSegment] = []   # who said what (RFC §12.2.3)

    @property
    def speaker(self) -> str | None: ...  # the label all segments share


@dataclass(frozen=True)
class SpeakerSegment:
    speaker: str | None      # provider label, e.g. "A"; None = unattributed
    text: str
    start_ms: int | None = None
    end_ms: int | None = None

language is what the provider reports — what it detected when it was asked to detect — never an echo of the language it was configured with. A Deepgram stream pinned to fr-CA reports nothing; one opened in multi reports the language most words carried.

Speaker labels from the STT

A recogniser that diarizes itself reports supports_diarization = True and gives every final the segments its text is made of, each with one speaker. A label is the provider's ("A", "B"…), opaque, and holds within one stream only: another stream may give the same voice another label.

async for result in stt.transcribe_stream(audio):
    if result.is_final:
        print(f"{result.speaker}: {result.text}")   # A: Bonjour Julie…

Through a VoiceChannel (RFC §12.2.3), in continuous mode — a streaming STT and no VAD in the pipeline:

• The channel keeps one stream across turns, so the labels compare. It keeps the stream's audio level with the clock: whenever the audio pauses and the stream is behind — a microphone muted during playback, packets lost over a long call — it fills the gap with silence, since Meta Muse ends a stream whose audio falls behind real time (1008 after ~15 s). An echo transcript it discards does not end the stream either.
• Each segment is its own room message. The sender stays the session's participant, the owner of the audio stream; the speaker is metadata: speaker_label ("A"), speaker_epoch and sender_name, which the AI channel uses to attribute turns (see Multi-Speaker Rooms). sender_name is "Speaker A", "Speaker A#1" once a new stream (epoch) has started, and "Unknown speaker" for words nobody was attributed.
• ON_TRANSCRIPTION fires per segment, its event carrying speaker, speaker_epoch and sender_name. A hook returning the event with another sender_name names that segment; every segment starts from the default name, so the hook names the label each time it sees it:

@kit.hook(HookTrigger.ON_TRANSCRIPTION)
async def name_known_voices(event, ctx):
    if event.speaker == "A" and event.speaker_epoch == 0:
        return HookResult.modify(dataclasses.replace(event, sender_name="Sylvain"))
    return HookResult.allow()

• ON_SPEAKER_CHANGE fires with source="stt" when a segment's label differs from the last one routed, and on the first label of each epoch; the event carries the label's speaker_epoch, and confidence is None. A turn detector never joins two speakers' segments: a change of voice routes the pending turn first. A session's segments reach the room in the order they were said, and the reply to one never holds back the next.
• A new stream — the provider ended it, an error, a language change — starts a new epoch: the same letter afterwards is not assumed to be the same person.

A VoiceChannel behind a VAD (a stream per utterance) or in batch mode refuses a diarizing provider at construction, and so does a ConferenceChannel, which knows who spoke from each participant's track and transcribes utterance by utterance.

Three providers fill segments: MetaSTTProvider in DIARIZATION mode (labels per turn), DeepgramSTTProvider with diarize_model (labels per word) and GeminiSTTProvider with speaker_segments (batch only). Labels are normalised by roomkit.voice.base.speaker_label: a string, None for an unattributed speaker, and without a leading "speaker" word ("Speaker 1" → "1"). Deepgram's older diarize=True keeps its ids in words and does not report supports_diarization.

This is the STT's own diarization, aligned to its words. The pipeline's DiarizationProvider (see Audio Pipeline Stages) is the other source: it labels audio frames and fires ON_SPEAKER_CHANGE with source="pipeline". With both, the STT's label is the one on the transcript.

Speakers from the pipeline stage

When the STT labels nobody, VoiceChannel(pipeline_speakers=True) names each transcript from the pipeline's DiarizationProvider instead (RFC §12.2.3):

voice = VoiceChannel(
    "voice",
    stt=stt,                       # any STT; a diarizing one keeps its own labels
    backend=backend,
    pipeline=AudioPipelineConfig(vad=vad, diarization=diarization),
    pipeline_speakers=True,
)

• The transcript's speaker is the one the stage heard the longest over it: the utterance behind a VAD, the audio since the last final in continuous mode. Each result counts for the audio since the stage's previous one, so a verdict sherpa-onnx gives over 2 s of speech outweighs one over a short tail. The room message carries speaker_label, speaker_epoch (always 0: the stage keeps its labels for the session) and sender_name, and ON_TRANSCRIPTION carries the speaker, so the hook above renames it the same way.
• A voice the stage matched to nobody (sherpa-onnx says "unknown" below its search_threshold) is "Unknown speaker". An utterance the stage gave no result for carries no speaker, as without the option.
• ON_SPEAKER_CHANGE still comes from the stage alone, with source="pipeline", and never for a voice it matched to nobody.
• Refused without a diarization stage, and in batch mode, where one flush may hold several voices.

SherpaOnnxDiarizationProvider identifies enrolled voices, and only behind a VAD: it extracts an embedding when an utterance ends (and every 2 s of speech), so it names nobody in continuous mode. Its label is the enrolled name: enroll "Sylvain" and the message reads "Speaker Sylvain" until a hook names it. On a French two-voice dialogue (TitaNet embeddings, TEN VAD, pauses lengthened so the VAD cut at every turn), 9 utterances of 9 went to the right voice at search_threshold=0.4; at the default 0.5 one short utterance scored 0.48 and came out as "Unknown speaker". On a laptop microphone, the wrong voice never scored above 0.21 while the right one scored 0.36 to 0.66: a search_threshold around 0.35 suits it. Sub-second utterances score low whoever speaks.

---

Deepgram (Cloud API)

The most feature-rich cloud STT provider. Real-time streaming with interim results, keyword boosting, and entity detection.

from __future__ import annotations

from roomkit.voice.stt.deepgram import DeepgramConfig, DeepgramSTTProvider

stt = DeepgramSTTProvider(
    config=DeepgramConfig(
        api_key="your-api-key",
        model="nova-3",
        language="en",
        punctuate=True,
        smart_format=True,
        interim_results=True,
        endpointing=300,         # Silence duration (ms) before endpoint
        vad_events=True,         # Emit speech_start events
    )
)

Parameter	Default	Description
model	"nova-2"	Model name ("nova-3" for multi and keyterms)
language	"en"	Language code — "multi" detects and code-switches (Nova-3)
punctuate	True	Add punctuation
smart_format	True	Smart formatting (dates, numbers)
numerals	False	Convert numbers to digits
interim_results	True	Include partial results while speaking
endpointing	300	Silence ms before utterance end, or False to disable
utterance_end_ms	None	Additional utterance end signal
vad_events	True	Emit VAD events
diarize	False	Deepgram's older diarizer; the ids stay in words
diarize_model	None	Speaker segments ("latest"): every final carries segments; exclusive with diarize
filler_words	False	Include "um", "uh"
keywords	[]	Keywords to boost recognition
keyterm	[]	Key terms (Nova-3)
profanity_filter	False	Filter profanity
redact	[]	Redaction rules (e.g., ["pci", "ssn"])
detect_entities	False	Detect named entities

Streaming events: SpeechStarted, Results (partial + final), UtteranceEnd

Batch mode: HTTP POST to /listen endpoint.

Streaming mode: WebSocket with real-time partials and finals.

Speaker segments (diarize_model)

diarize_model="latest" makes the provider report supports_diarization and fill TranscriptionResult.segments on every final, in batch and in streaming (see Speaker labels from the STT). Deepgram labels each word, so a final spanning a change of voice becomes two segments; labels are "0", "1"…, and the offsets are Deepgram's word times.

stt = DeepgramSTTProvider(
    DeepgramConfig(api_key="...", model="nova-3", language="fr", diarize_model="latest")
)
result = await stt.transcribe(recording)
for segment in result.segments:
    print(f"{segment.speaker}: {segment.text}")

Measured on a two-voice French dialogue (2026-09-27):

• Batch attributed the 5 turns correctly, with their offsets.
• Streaming labelled every word 0 for about the first 30 seconds, then told the voices apart, mostly correctly. Through a continuous VoiceChannel the same dialogue played three times gave one stream, "Speaker 0" throughout the first pass and alternating speakers after. For live attribution from the first sentence, Meta Muse's DIARIZATION did better on the same audio.
• diarize=True, the older diarizer, labelled every word 0 in both modes. It is kept as it was — ids in words, no segments, no supports_diarization — so an existing configuration behaves the same, a VoiceChannel behind a VAD included.

diarize_model travels as a query parameter, so it works with every SDK the deepgram extra allows.

Language: detect, then lock

Nova-3 transcribes code-switched speech with language="multi", and reports what it heard — TranscriptionResult.language carries the language most words were tagged with. A stream pinned to the speaker's language (fr-CA) is measurably better than multi, but Deepgram fixes the language in the WebSocket URL: it holds for the life of a stream. RoomKit opens one stream per utterance (VAD mode) or per turn (continuous mode), so the language can change between them — never inside one.

VoiceChannel.set_stt_language chooses the language for one session from its next stream on; None returns to the provider's configuration. The typical caller is an ON_TRANSCRIPTION hook reading event.language:

stt = DeepgramSTTProvider(DeepgramConfig(api_key="...", model="nova-3", language="multi"))
voice = VoiceChannel("voice", stt=stt, backend=backend, pipeline=AudioPipelineConfig())

@kit.hook(HookTrigger.ON_TRANSCRIPTION)
async def pin_language(event, ctx):
    # A code while the stream detects, None once it is pinned
    if event.language == "fr":
        voice.set_stt_language(event.session, "fr-CA")
    return HookResult.allow()

STTLanguageLock packages that loop, with a way back:

from roomkit import STTLanguageLock

voice = VoiceChannel(
    "voice",
    stt=stt,
    backend=backend,
    pipeline=AudioPipelineConfig(),
    stt_language_lock=STTLanguageLock(
        detect_language="multi",     # every session starts here
        prefer={"fr": "fr-CA"},      # reported -> locked
        lock_after=1,                # agreeing finals before locking
        release_after=2,             # consecutive misses before detecting again
        min_confidence=0.5,          # a final below this is a miss
    ),
)

Parameter	Default	Description
detect_language	"multi"	Language every session starts in, and returns to
prefer	{}	Reported code → locked code (Deepgram reports fr, you want fr-CA)
lock_after	1	Consecutive finals reporting the same language before locking
release_after	2	Consecutive misses before releasing back to detect_language
min_confidence	0.5	A final below this confidence is a miss

A miss is a final with no text, a confidence below min_confidence, or a reported language other than the lock; a fitting final resets the count. A locked stream reports no language, so the way back is noise: a caller who switches to English on a fr-CA stream produces empties and low-confidence finals, and two of those in a row reopen the session in multi.

Where the change lands:

• VAD mode — the next utterance. A stream already open stays open, so the utterance in progress is not cut in two.
• Continuous mode — right away: the current cycle is ended and the loop reconnects with the new language. Audio arriving in the gap is kept.
• Batch mode — the next flush_stt().

Runnable: examples/voice_deepgram_language_lock.py.

---

SherpaOnnx (Local/Offline)

Run STT locally without API calls using ONNX models. Supports transducer (streaming) and Whisper (batch) modes.

from __future__ import annotations

from roomkit.voice.stt.sherpa_onnx import SherpaOnnxSTTConfig, SherpaOnnxSTTProvider

# Streaming transducer mode
stt = SherpaOnnxSTTProvider(
    config=SherpaOnnxSTTConfig(
        mode="transducer",
        tokens="path/to/tokens.txt",
        encoder="path/to/encoder.onnx",
        decoder="path/to/decoder.onnx",
        joiner="path/to/joiner.onnx",
        sample_rate=16000,
        num_threads=2,
        provider="cpu",                        # or "cuda"
        enable_endpoint_detection=True,
        rule1_min_trailing_silence=2.4,        # Seconds
        rule2_min_trailing_silence=1.2,
        rule3_min_utterance_length=20.0,
    )
)

# Batch Whisper mode (no streaming)
stt_whisper = SherpaOnnxSTTProvider(
    config=SherpaOnnxSTTConfig(
        mode="whisper",
        tokens="path/to/tokens.txt",
        encoder="path/to/encoder.onnx",
        decoder="path/to/decoder.onnx",
        language="en",
        task="transcribe",                     # or "translate" for English translation
    )
)

Parameter	Default	Description
mode	required	"transducer" (streaming) or "whisper" (batch only)
tokens	required	Path to tokens.txt
encoder	required	Path to encoder ONNX model
decoder	required	Path to decoder ONNX model
joiner	None	Path to joiner ONNX (transducer only)
sample_rate	16000	Expected input sample rate
num_threads	2	CPU threads for inference
provider	"cpu"	ONNX runtime: "cpu" or "cuda"
enable_endpoint_detection	True	Detect utterance endpoints
rule1_min_trailing_silence	2.4	Silence threshold (seconds) for rule 1
rule2_min_trailing_silence	1.2	Silence with text threshold
rule3_min_utterance_length	20.0	Min utterance length for rule 3

---

Gradium (Cloud API)

Cloud STT with built-in server-side VAD and pre-connect buffering to avoid lost first words.

from __future__ import annotations

from roomkit.voice.stt.gradium import GradiumSTTConfig, GradiumSTTProvider

stt = GradiumSTTProvider(
    config=GradiumSTTConfig(
        api_key="your-api-key",
        region="us",
        model_name="default",
        input_format="pcm",
        language="en",
        connect_buffer_ms=300,    # Buffer audio before WebSocket opens
        delay_in_frames=7,        # Processing delay (7-48, each = 80ms)
        vad_threshold=0.9,        # VAD inactivity threshold
        vad_steps=10,             # Steps above threshold to confirm end
        timeout_s=3.0,            # Server inactivity timeout
    )
)

Streaming events: text (partial), end_text (segment done), step (VAD heartbeat)

Pre-connect buffering: Accumulates real audio before opening the WebSocket, then sends a burst — avoids losing the first few words.

---

Qwen3 ASR (Local/GPU)

HuggingFace-based ASR with optional vLLM backend for streaming.

from __future__ import annotations

from roomkit.voice.stt.qwen3 import Qwen3ASRConfig, Qwen3ASRProvider

stt = Qwen3ASRProvider(
    config=Qwen3ASRConfig(
        model_id="Qwen/Qwen3-ASR-0.6B",
        backend="vllm",             # "vllm" for streaming, "transformers" for batch
        device_map="auto",
        dtype="bfloat16",
        language=None,               # None = auto-detect
        chunk_size_sec=2.0,          # Streaming chunk duration
        gpu_memory_utilization=0.3,
        max_new_tokens=2048,
    )
)

Parameter	Default	Description
model_id	"Qwen/Qwen3-ASR-0.6B"	HuggingFace model ID
backend	"transformers"	"transformers" (batch) or "vllm" (batch + streaming)
dtype	"bfloat16"	Model precision
language	None	Language code or None for auto-detect
chunk_size_sec	2.0	Streaming chunk duration (vLLM only)
gpu_memory_utilization	0.3	GPU memory fraction (vLLM only)

---

Gemini (Cloud API, batch only)

The API takes a complete recording and answers in one pass, in seconds, so this provider is batch by construction. For live turn-taking use Gemini Transcribe below, which drives Google's dedicated recogniser over a WebSocket.

What the batch shape buys is what a streaming recogniser structurally cannot give. The model sees the whole recording before it answers, so one pass returns the transcript, the speaker turns and the timestamps together — no diarization stage, no merge. This is the provider for meeting recordings, voicemail and imported audio files.

from __future__ import annotations

from roomkit.voice.stt.gemini import GeminiSTTConfig, GeminiSTTProvider

stt = GeminiSTTProvider(
    config=GeminiSTTConfig(
        api_key="your-gemini-api-key",
        model="gemini-3.8-flash",     # any multimodal model that accepts audio
        language="fr-CA",              # optional hint; detected otherwise
        diarize=True,                  # ask for speaker labels
        prompt="The product is spelled RoomKit.",   # optional vocabulary/format
    )
)

transcript = await stt.transcribe_recording("meeting.wav")
print(transcript.language)                    # "fr-CA"
for turn in transcript.segments:
    print(f"[{turn.start}-{turn.end}] {turn.speaker}: {turn.text}")

Parameter	Default	Description
api_key	(required)	Gemini API key (GEMINI_API_KEY)
model	"gemini-3.8-flash"	A multimodal model that accepts audio input
language	None	BCP-47 hint; unset, the model identifies and reports it
diarize	True	Ask for Speaker 1, Speaker 2, … labels
prompt	None	Extra instruction: formatting rules. Refused on the recogniser
timeout	600.0	Per-request timeout in seconds
max_inline_bytes	15 MiB	Above this, the recording is uploaded via the Files API
mode	"verbatim"	"smart" for the recogniser's cleaned-up transcript (recogniser only)
custom_vocabulary	[]	Terms to spell as written, up to 1000; in the prompt of a multimodal model
word_timestamps	True	Time every word (recogniser only)
speaker_segments	False	Put the turns on transcribe()'s result as segments; needs diarize

Speaker turns in the shared contract

transcribe_recording() returns Gemini's own Transcript, whose turns carry MM:SS strings. transcript.speaker_segments() gives the same turns as the shared SpeakerSegment (label "1", "2"…, offsets in milliseconds — to the 100 ms word time on the recogniser, to the second otherwise), the type Meta and Deepgram fill too. With speaker_segments=True, transcribe() returns them directly in TranscriptionResult.segments and the provider reports supports_diarization:

stt = GeminiSTTProvider(GeminiSTTConfig(api_key="...", speaker_segments=True))
result = await stt.transcribe(recording)
for segment in result.segments:
    print(f"{segment.speaker}: {segment.text}")

It is off by default because diarize has always been on: a diarizing STT is refused by a VoiceChannel behind a VAD, where labels would not compare across utterances (see Speaker labels from the STT). Both kinds of model attributed the two-voice French dialogue's 5 turns correctly (2026-09-27).

Two kinds of model

model picks how the transcript is obtained:

• A multimodal model (the default, gemini-3.8-flash) is instructed: a prompt asks for the transcript and a JSON schema shapes it. Turns are timed to the second, and long recordings are fine.
• The dedicated recogniser, gemini-3.5-transcribe, is configured: it refuses a prompt and a response schema, and takes a transcription_config instead. It answers about twice as fast and times every word to 100 ms, in transcript.words.

Measured against the live API on 2026-09-27, on a 12-second, two-speaker French dialogue:

	gemini-3.8-flash	gemini-3.5-transcribe
Time	2.9–4.0 s	1.7–2.7 s
Speaker turns	4, correct	4, correct
Timing	per turn, MM:SS	per word, 100 ms
Longest recording	long files	1 hour; 30 minutes with speakers or word timing

stt = GeminiSTTProvider(
    GeminiSTTConfig(api_key="...", model="gemini-3.5-transcribe", language="fr-CA")
)
transcript = await stt.transcribe_recording("meeting.wav")
for word in transcript.words:
    print(f"{word.start:.1f}-{word.end:.1f}s {word.speaker}: {word.text}")

The recogniser's options exclude each other the way the service does, and the config refuses a forbidden pair at construction rather than on the first call:

Option	Needs	Excludes
diarize=True	word_timestamps=True — speakers ride on the words
mode="smart"		diarize, word_timestamps
custom_vocabulary	language — without one the service answers the first sentence alone	diarize, word_timestamps
prompt		always refused

Two more differences to know. The recogniser detects the language but never reports it, so transcript.language is empty unless you set language (short codes such as "fr" are accepted). And raw PCM is sent to every model inside a WAV header, because the recogniser refuses bare audio/l16.

Two methods, two shapes

transcribe() is the STTProvider contract and returns flat text, dropping the structure. transcribe_recording() returns the whole Transcript — the language plus one TranscriptSegment per speaker turn, with .text (labelled) and .plain_text (words only) helpers, and, from the recogniser, the timed words. It also accepts a file path, which transcribe() does not.

Input paths

Raw AudioChunk/AudioFrame audio is sent inline as WAV; a data: URL is sent inline as-is; a local path is inlined below max_inline_bytes and uploaded through the Files API above it (and deleted afterwards, rather than left to expire). Arbitrary http(s) URLs are refused, not fetched: dereferencing a caller-supplied URL would make the provider an SSRF vector. Upload the file or pass the bytes.

Limits worth knowing

supports_streaming is False, so a VoiceChannel transcribes on SPEECH_END instead of streaming partials — mechanically fine, but seconds of model latency after every utterance is not a conversation. Use batch_mode=True or transcribe a finished recording.

Speaker labels are the model's judgement, not an acoustic decision. On a live run against a four-turn recording with two voices, the model labelled the fourth turn Speaker 3 although it was the second voice again (2026-08-07). Where the speakers are already separated — a conference records one track per participant — transcribe each track with diarize=False and merge on the timestamps instead. The labels earn their keep on a single mixed file.

A multimodal model's timestamps are its reading, not a forced alignment: good for navigating a recording, not for syncing against anything. The recogniser's word offsets are measured, at 100 ms.

See examples/meeting_transcription.py — a recording becomes speaker turns, enters a room, and an AI channel writes the minutes. Run it with GEMINI_STT_MODEL=gemini-3.5-transcribe for the recogniser and its word timing.

---

Gemini Transcribe (Cloud API, streaming)

gemini-3.5-transcribe-live is a dedicated recogniser reached over the Live API, not a chat model asked to write down what it hears. It takes a PCM stream over a WebSocket and answers with interim transcripts while the speaker is still talking, then a final one per utterance — the shape a VoiceChannel wants for live turn-taking, and the opposite of the batch provider above.

from __future__ import annotations

from roomkit.voice.stt.gemini_transcribe import (
    GeminiTranscribeConfig,
    GeminiTranscribeProvider,
)

stt = GeminiTranscribeProvider(
    GeminiTranscribeConfig(
        api_key="your-gemini-key",
        # Empty is not a missing value: it asks the model to identify the
        # language itself, across the 85+ locales it covers.
        language_codes=[],
        custom_vocabulary=["RoomKit", "Aubervilliers"],
        mode="VERBATIM",
    )
)

Option	Default	Meaning
api_key	(required)	Gemini API key (GEMINI_API_KEY)
model	gemini-3.5-transcribe-live	Live transcription model id
language_codes	[]	BCP-47 hints; empty means automatic detection
custom_vocabulary	[]	Up to 1000 terms to bias towards; best below ~100
mode	VERBATIM	VERBATIM keeps what was said, SMART removes disfluencies and formats
timeout	60.0	Read budget in seconds
connect_timeout	5.0	TCP connect timeout in seconds

Install with pip install roomkit[realtime-gemini] — the Live API extra, the same one Gemini Live uses.

Limits worth knowing

Audio must be raw 16-bit PCM, 16 kHz, mono, little-endian. Another rate is sent as it is, with its true rate in the MIME type and one warning: guessing a resample inside the provider would hide a pipeline that is misconfigured upstream. Resample before you get here — examples/stt_gemini_transcribe_live.py shows it with RoomKit's own resampler.

Speaker diarization and word-level timestamps are not available over the Live API, whatever the batch model offers, so the config carries no knob for either. A session is capped at ten minutes. Both are reasons a long recording belongs to the batch provider above, which also returns the speaker turns.

transcribe() works and runs the audio through the same socket, which keeps one client, one auth path and one config; the ten-minute cap applies to it too.

Two examples. examples/stt_gemini_transcribe_mic.py is the one to run first: speak into your microphone and the caption line rewrites itself while you talk, committing on a pause. That is what the streaming shape buys, and real speech exercises the hesitation, accent and language detection a synthesized sentence cannot. examples/stt_gemini_transcribe_live.py does the same from a file or a synthesized sentence, for a machine with no audio device.

Meta Muse Voice Transcribe (Cloud API, streaming + batch)

muse-voice-transcribe-1.0 is Meta's recogniser on the Meta Model API. One provider speaks both of its endpoints: a realtime WebSocket that answers interim transcripts while the speaker talks and, when asked, finds where each turn ends by itself; and a REST endpoint for one finished clip.

from __future__ import annotations

from roomkit.voice.stt.meta import MetaSTTConfig, MetaSTTProvider

stt = MetaSTTProvider(
    MetaSTTConfig(
        api_key="your-meta-model-api-key",
        mode="ENDPOINTING",  # no pipeline VAD: the model ends each turn
        language_bias=["French"],
        keywords=["RoomKit", "Tremblay"],
    )
)

Option	Default	Meaning
api_key	(required)	Meta Model API key
model	muse-voice-transcribe-1.0	Transcription model id
mode	ENDPOINTING	How a stream ends — see below
keywords	[]	Terms to bias recognition toward; a bias, not a guaranteed spelling
language_bias	[]	Language names to bias toward ("French", not "fr")
base_url	https://api.meta.ai/v1	REST endpoint root
realtime_url	wss://api.meta.ai/v1/asr/realtime	Realtime WebSocket endpoint
timeout_s	60.0	REST transcription timeout
handshake_timeout_s	10.0	How long a stream waits for the service to accept its configuration

Install with pip install roomkit[meta-stt] (websockets for streams, httpx for transcribe()).

Pick the mode from the channel

The provider cannot see how the VoiceChannel is set up, and the right mode depends on it:

Channel	Mode	What the model does
No pipeline VAD — continuous STT	ENDPOINTING	Signals speech start (ON_SPEECH_START fires), sends interims, and ends each turn itself after about 550 ms of silence
Pipeline VAD delimits utterances	PUSH_TO_TALK	Sends interims, then one final when the channel closes the utterance's stream
No pipeline VAD, speakers wanted	DIARIZATION	ENDPOINTING plus a speaker per turn; a change of voice also ends a turn

ENDPOINTING behind a pipeline VAD would hand the channel several finals for one utterance, of which it keeps the last. The endpointing silence is not configurable, and two voices separated by less than that silence make one turn.

Limits worth knowing

• Audio: mono 16-bit PCM at 16 or 24 kHz goes through as it is; any other rate is resampled to 24 kHz (the model's native rate) by the provider, so an 8 kHz telephony leg works without a pipeline resampler.
• Language: language_bias biases, it does not pin, so the provider reports supports_language_override = False and the channel never passes it a per-call language. The names are checked at construction against the 25 Meta documents (SUPPORTED_LANGUAGES): the service itself accepts a misspelt one silently, and the bias then does nothing. Meta reports no detected language and no confidence.
• Sessions: a stream lasts at most 60 minutes, then the service closes it with code 1011. Every failure raises MetaSTTError, which carries Meta's code and error_type (e.g. billing_not_configured / billing_error) and retryable: true for a server fault (1011), a rate limit (1013, HTTP 429), a 5xx and a dropped connection, false for a refused request (1008, HTTP 4xx).
• Backlog: the service refuses a stream sent 7 s or more ahead of real time at once (Audio processing backlog too large; 6 s passes, measured 2026-10-07). That happens after a reconnect: a handshake the service does not answer holds the stream for handshake_timeout_s, and the microphone audio queues up meanwhile. The provider paces what it sends (at most 3 s at once, then twice real time until it has caught up), and a continuous VoiceChannel carries at most the last 5 s of audio into the next stream, logging a warning with the seconds it dropped (RFC §12.2).
• REST (transcribe()): one WAV of at most 10 minutes and 32 MB, sent in PUSH_TO_TALK mode; the provider wraps the PCM in the WAV itself. An AudioContent carrying a WAV data: URI is decoded locally; an http(s) URL is refused rather than fetched.
• Speakers: in DIARIZATION mode every final carries its turn as one SpeakerSegment labelled "A", "B"…, with the turn's offsets; a turn Meta could not attribute has speaker=None. transcribe() answers the clip's turns as segments the same way. On a two-voice French dialogue (2026-09-27) both paths attributed 5 turns out of 5. Meta documents the mode as not tuned for low latency. A VoiceChannel in continuous mode carries the labels to the room (see Speaker labels from the STT); through one on the live service, the same dialogue gave 5 room messages on one stream, and a 20 s silence did not end it.

Two examples. examples/stt_meta_mic.py is the one to run first: speak into your microphone and the caption line reacts as the model hears you start, rewrites itself while you talk, and commits once you pause — the model's own endpointing, with no VAD on RoomKit's side. Add --diarize and talk in turn with someone: each committed line says who spoke (> A: …, > B: …). examples/voice_meta_diarization.py does the same through a VoiceChannel: each turn becomes a room message with its sender_name, and SPEAKER_NAMES ("A=Sylvain,B=Julie") names the voices you know. examples/stt_meta_live.py streams a WAV file in real time in either mode and then sends it over REST, for a machine with no audio device.

---

Microsoft MAI-Transcribe (Cloud API, streaming)

MAI-Transcribe-2-Streaming is Microsoft's realtime recogniser, served by a Microsoft Foundry resource over a WebSocket modelled on the OpenAI Realtime transcription protocol. It returns a transcript that firms up while the speaker talks, in 57 languages including French, and detects the language by itself unless told one.

from __future__ import annotations

import os

from roomkit.voice.pipeline import AudioPipelineConfig
from roomkit.voice.pipeline.vad.energy import EnergyVADProvider
from roomkit.voice.stt.azure_mai import AzureMAISTTConfig, AzureMAISTTProvider

stt = AzureMAISTTProvider(
    AzureMAISTTConfig(
        endpoint="https://<resource>.services.ai.azure.com",
        api_key=os.environ["AZURE_SPEECH_KEY"],
        language="fr-CA",  # sent as "fr"; None detects
    )
)
pipeline = AudioPipelineConfig(vad=EnergyVADProvider())  # required, see below

Option	Default	Meaning
endpoint	(required)	The Foundry resource URL, as the portal shows it
api_key	(required)	The resource's key
deployment	MAI-Transcribe-2-Streaming	The model's deployment name on the resource
language	None	BCP-47 tag or service code; None lets the service detect it
handshake_timeout_s	10.0	How long a stream waits for the session to open and accept its settings
final_timeout_s	10.0	How long a stream waits for the final once its audio is committed

Install with pip install roomkit[azure-speech] (websockets, no Azure SDK).

A pipeline VAD is required

The service detects no turns: it finalises the audio it has received when the client commits it. Behind a pipeline VAD, a VoiceChannel opens one stream per utterance and the provider commits when the utterance ends, so the final follows the end of speech. Without a VAD the channel runs continuous STT, the stream never ends, and no final comes. This also keeps silence off the wire: the service bills per hour of audio sent.

Limits worth knowing

• Preview: the model is in public preview (October 2026), without an SLA.
• Measured (2026-10-08, from Québec to Sweden Central): an English and a French sentence, rendered by MAI-Voice and played back in real time, came back word for word, the French one with its language detected; the final arrived 125 to 146 ms after the end of speech, and partials every 250 ms or so while it spoke. The service writes numbers as figures ("quatorze heures trente" comes back as "14h30"). French accuracy on real speech is not measured yet.
• Regions: Microsoft serves the model from Sweden Central, Central US, East US 2 and Southeast Asia, and routes requests to them; there is no Canadian region.
• Audio: mono 16-bit PCM at 16 or 24 kHz goes through as it is; any other rate is resampled to 16 kHz by the provider. Audio leaves in appends of at most 100 ms.
• Language: the service takes language codes, not regions, so fr-CA is sent as fr; the provider reports supports_language_override = True. The provider takes the 57 codes in SUPPORTED_LANGUAGES: Microsoft lists 60, and the endpoint refuses three of them (as, or, pa). A code outside the list is refused at construction or before the stream opens, because the endpoint takes some the model does not know (no, tl, cy) and the model then detects instead. Detection is asked by leaving the language out: the endpoint refuses an explicit null. The service reports no detected language, no confidence and no speaker.
• Sessions: one stream is one session, and the service caps a session at one hour; a stream per utterance never comes near it. Every failure raises AzureMAISTTError, with the service's code and error_type and retryable: true for a server error, a rate limit, a 5xx or a 429 at the upgrade, a timeout and a dropped connection.
• Batch (transcribe()): there is no batch endpoint for this model, so a clip goes through one session, committed once. It takes an AudioChunk or an AudioFrame; an AudioContent is refused.

Two examples. examples/voice_azure_mai_agent.py is the one to run first: a voice agent through your microphone, MAI-Transcribe hearing you behind a VAD, Claude Haiku 5.5 answering in the language you spoke, MAI-Voice speaking (VOICE=fr-FR-Soleil:MAI-Voice-2.1-Flash for French) at +15%. Its VAD waits 800 ms of silence instead of 500: at 500, a pause in the middle of a sentence ended the turn, and a word cut by it came back as two halves transcribed out of context. Its barge-in is semantic, so an "okay" does not stop the agent. examples/voice_azure_mai.py needs no audio device: MAI-Voice speaks a sentence and plays it to MAI-Transcribe in real time, logging the TTS's first audio and the STT's final after the end of speech.

---

TTS Provider ABC

from __future__ import annotations

from roomkit.voice.tts import AudioChunk, AudioContent, TTSProvider


class TTSProvider(ABC):
    @property
    def name(self) -> str: ...

    @property
    def default_voice(self) -> str | None: ...

    @property
    def supports_streaming_input(self) -> bool: ...

    async def synthesize(self, text, *, voice=None) -> AudioContent:
        """Batch synthesis — full text in, complete audio out."""

    async def synthesize_stream(self, text, *, voice=None) -> AsyncIterator[AudioChunk]:
        """Streaming output — yields audio chunks as they're generated."""

    async def synthesize_stream_input(self, text_stream, *, voice=None) -> AsyncIterator[AudioChunk]:
        """Streaming input — accepts async text stream, yields audio."""

    async def warmup(self) -> None:
        """Pre-load models (optional)."""

    async def close(self) -> None:
        """Release resources."""

AudioChunk

@dataclass
class AudioChunk:
    data: bytes
    sample_rate: int = 16000
    channels: int = 1
    format: str = "pcm_s16le"
    timestamp_ms: int | None = None
    is_final: bool = False

---

ElevenLabs (Cloud API)

High-quality cloud TTS with streaming output, and request stitching from the conversation context.

from __future__ import annotations

from roomkit.voice.tts.elevenlabs import ElevenLabsConfig, ElevenLabsTTSProvider

tts = ElevenLabsTTSProvider(
    config=ElevenLabsConfig(
        api_key="your-api-key",
        voice_id="21m00Tcm4TlvDq8ikWAM",    # Rachel
        model_id="eleven_multilingual_v2",
        stability=0.5,
        similarity_boost=0.75,
        style=0.0,
        use_speaker_boost=True,
        output_format="mp3_44100_128",
        optimize_streaming_latency=3,         # 0-4, higher = faster; v2 / v2.5 models only
        use_context=True,                     # request stitching (see below)
    )
)

# List the account's voices, its own clones included, as VoiceInfo
for v in await tts.list_voices(language="fr"):
    print(f"{v.id}: {v.name} ({v.attributes.get('category')})")

Parameter	Default	Description
voice_id	"21m00Tcm4TlvDq8ikWAM"	Voice ID (Rachel)
model_id	"eleven_multilingual_v2"	TTS model
stability	0.5	Voice stability (0–1)
similarity_boost	0.75	Voice similarity (0–1)
style	0.0	Style exaggeration (0–1)
output_format	"mp3_44100_128"	<codec>_<rate>[_<bitrate>]: mp3_44100_128, pcm_16000, ulaw_8000, opus_48000_64... Chunks declare the codec and the rate it names. wav_* is for synthesize() only: ElevenLabs refuses it on a streamed request. On a VoiceChannel, use pcm_<rate>: it plays PCM only and refuses MP3, Opus or G.711 chunks
optimize_streaming_latency	None	Latency optimization level (0–4, higher is faster at some cost of quality). Sent to the v2 / v2.5 models only: v3 and v4 refuse it, and a value set for them is left out with a warning. None sends nothing
expressive	False	Eleven v4 Turbo with inline audio tags (see below)
use_context	True	Request stitching from the conversation context
stream_input	False	Streaming text input over WebSocket for streaming AI responses (see below)

Models. model_id takes any ElevenLabs text-to-speech model id; the module exports the common ones as constants:

Constant	Model id	Use
MODEL_V4_TURBO	eleven_v4_turbo	Voice agents: audio tags, 90+ languages, ~150 ms to first speech (ElevenLabs' figure)
MODEL_V4	eleven_v4	Highest quality for pre-generated audio, 10,000 characters per request
MODEL_FLASH_V2_5	eleven_flash_v2_5	Lowest latency (~75 ms), 32 languages, no audio tags
MODEL_MULTILINGUAL_V2	eleven_multilingual_v2	The default
MODEL_V3	eleven_v3	Previous expressive model, without request stitching

MODEL_TURBO_V2_5 is kept for existing code; ElevenLabs deprecated it in favour of Flash v2.5.

Expressive mode. expressive=True selects eleven_v4_turbo, unless model_id already names a v3 or v4 model, which is kept. The text may then carry audio tags ([laughs], [whispers], [sighs], [excited], [pause]), alone or stacked; ElevenLabs renders them instead of reading them out. v3 and v4 do not take SSML <break> tags: use [pause] or an ellipsis. Do not set StripBrackets as the channel's TTS filter, which would remove the tags. examples/voice_expressive.py runs a voice assistant in this mode.

Three synthesis modes:

• synthesize(): batch, returns complete audio as a base64 data URL
• synthesize_stream(): streaming output, yields audio chunks over HTTP
• synthesize_stream_input(): streaming input and output over a WebSocket

Streaming input. With stream_input=True (off by default) the provider declares supports_streaming_input, so a Voice Channel speaks a streaming AI response from its first sentence instead of waiting for the whole text. Each sentence is sent and flushed as the LLM produces it, over one WebSocket per response. Measured with an LLM writing a sentence every 0.8 s, first audio came after 140 ms on v4 Turbo and flash v2.5 and 350 ms on multilingual v2, against 2.6 to 3.9 s on the HTTP path. The model picks the socket:

Models	Socket	Voice settings
eleven_v4_turbo, eleven_v4 (expressive mode included)	Text to Dialogue	Not applied: the socket takes none. The provider warns at construction when stability, similarity_boost, style or use_speaker_boost differ from their defaults
eleven_multilingual_v2, eleven_flash_v2*, eleven_turbo_v2*	Text to Speech	Applied, with optimize_streaming_latency when set
eleven_v3*	None: supports_streaming_input is False

It is opt-in because a streamed response is handled differently from one synthesized whole:

• the Voice Channel runs BEFORE_TTS on each sentence before the TTS reads it, not once on the whole text: a hook judges one sentence at a time (a BLOCK drops that sentence, a MODIFY replaces it, a failing hook drops it) and sees it after the text filter; AFTER_TTS receives the text as spoken;
• a TTS failure (a 401, a 429, a network error) stops that session as a barge-in does: tts_error names the session once its playback is released, and once every session has stopped the response is stored as it stood (with metadata.cancelled when the generation was still running, which then ends) and nothing is replayed; ON_ERROR does not fire, the AI did not fail, and AFTER_TTS fires only if a session was served. The HTTP path reports a failed session the same way;
• it is not stitched to the previous responses (below): neither socket takes a request id or a previous text.

say() and non-streaming AI responses keep the HTTP path either way.

Request stitching. With use_context=True (the default) the provider declares TTSContextLevel.SELF and receives its own previous turns in the voice session (TTS Conversation Context). Stitching applies to the HTTP path, not to a response spoken over the socket. Each HTTP request then carries the request_id of the up to three previous responses the user heard to the end, in the same voice (previous_request_ids, ids younger than two hours), so ElevenLabs continues the voice from one response to the next. When no id is usable, the last response's text is passed as previous_text. After a response the user cut off with a barge-in, nothing is sent and the next response starts afresh. The user's words are never sent. v3 models (an eleven_v3* model_id) do not support stitching and receive no context; v4 models, expressive mode included, are stitched. examples/voice_elevenlabs_context.py writes the same conversation with and without stitching to two WAV files for comparison.

---

SherpaOnnx TTS (Local/Offline)

Local TTS using VITS/Piper ONNX models. No API calls required.

from __future__ import annotations

from roomkit.voice.tts.sherpa_onnx import SherpaOnnxTTSConfig, SherpaOnnxTTSProvider

tts = SherpaOnnxTTSProvider(
    config=SherpaOnnxTTSConfig(
        model="path/to/model.onnx",
        tokens="path/to/tokens.txt",
        data_dir="path/to/espeak-ng-data",     # For Piper models
        speaker_id=0,                           # Multi-speaker models
        speed=1.0,                              # < 1.0 = faster, > 1.0 = slower
        sample_rate=22050,
        num_threads=2,
        provider="cpu",                         # or "cuda"
    )
)

Text splitting: Automatic sentence-based chunking (max 300 chars per chunk) with short-fragment merging.

---

Qwen3 TTS (Local/GPU, Voice Cloning)

LLM-based TTS with zero-shot voice cloning from reference audio.

from __future__ import annotations

from roomkit.voice.tts.qwen3 import Qwen3TTSConfig, Qwen3TTSProvider, VoiceCloneConfig

tts = Qwen3TTSProvider(
    config=Qwen3TTSConfig(
        model_id="Qwen/Qwen3-TTS-12Hz-1.7B-Base",
        device_map="auto",
        dtype="bfloat16",
        language="English",
        voices={
            "default": VoiceCloneConfig(
                ref_audio="reference.wav",       # 3s+ clean speech
                ref_text="Transcript of the reference audio.",
            ),
            "french": VoiceCloneConfig(
                ref_audio="french_ref.wav",
                ref_text="Transcription de l'audio de reference.",
            ),
        },
        temperature=0.6,
        top_p=0.8,
        repetition_penalty=1.05,
        max_new_tokens=4096,
    )
)

# Pre-load model and encode reference audio
await tts.warmup()

Parameter	Default	Description
model_id	"Qwen/Qwen3-TTS-12Hz-1.7B-Base"	HuggingFace model
voices	{}	Voice name → VoiceCloneConfig mapping
language	"English"	Default synthesis language
temperature	0.6	Sampling temperature
top_p	0.8	Nucleus sampling probability
max_new_tokens	4096	Max output tokens

Voice cloning: Provide a 3+ second reference WAV and its transcript. The model learns the voice characteristics at warmup time.

Output: Fixed 24kHz PCM.

---

Grok TTS (Cloud API)

xAI Grok TTS with REST and bidirectional WebSocket streaming. 5 voices, 20 languages, expressive speech tags.

from __future__ import annotations

from roomkit.voice.tts.grok import GrokTTSConfig, GrokTTSProvider

tts = GrokTTSProvider(
    config=GrokTTSConfig(
        api_key="your-xai-api-key",
        voice_id="eve",             # eve, ara, rex, sal, leo
        language="en",              # BCP-47 code or "auto"
        codec="pcm",               # pcm, wav, mp3, mulaw, alaw
        sample_rate=24000,          # 8000–48000
    )
)

Configuration

Parameter	Default	Description
api_key	(required)	xAI API key
voice_id	"eve"	Voice — eve, ara, rex, sal, leo
language	"en"	BCP-47 language code or "auto"
codec	"pcm"	Output codec: pcm, wav, mp3, mulaw, alaw. wav applies to synthesize(); a streamed request asks for pcm, since a WAV stream opens with a header that would be played as audio
sample_rate	24000	Sample rate in Hz
bit_rate	128000	MP3 bit rate (only used with codec="mp3")
base_url	"https://api.x.ai/v1"	REST API base URL
ws_url	"wss://api.x.ai/v1/tts"	WebSocket streaming URL
timeout	60.0	HTTP request timeout in seconds

Streaming modes

• synthesize_stream() — HTTP chunked streaming from the REST endpoint
• synthesize_stream_input() — Bidirectional WebSocket: send text.delta/text.done, receive audio.delta/audio.done. Starts speaking while the AI is still generating text.

Expressive speech tags

Grok TTS supports inline tags like [pause], [laugh], [sigh] and wrapping tags like <whisper>, <soft>, <loud>, <slow>, <fast>, <high-pitch>, <low-pitch>.

---

Gradium TTS (Cloud API)

Cloud TTS with streaming input support and fine-grained voice control.

from __future__ import annotations

from roomkit.voice.tts.gradium import GradiumTTSConfig, GradiumTTSProvider

tts = GradiumTTSProvider(
    config=GradiumTTSConfig(
        api_key="your-api-key",
        voice_id="default",
        region="us",
        model_name="default",
        output_format="pcm_16000",  # "wav" applies to synthesize(); streams ask for "pcm"
        temperature=0.7,
        cfg_coef=2.0,               # Voice similarity (1.0–4.0)
        padding_bonus=0.0,          # Speed: negative = faster, positive = slower
        rewrite_rules="en",         # Language-specific text rewriting
    )
)

---

Gemini TTS (Cloud API)

Google's generative speech models. They perform the text rather than read it, so a natural-language direction steers delivery — that is what style_prompt is for. 30 prebuilt voices and more than 70 documented languages.

from __future__ import annotations

from roomkit.voice.tts.gemini import GeminiTTSConfig, GeminiTTSProvider

tts = GeminiTTSProvider(
    config=GeminiTTSConfig(
        api_key="your-gemini-api-key",
        model="gemini-3.8-flash-tts",                    # the default
        voice="Kore",                                    # 30 prebuilt voices
        language="fr-CA",                                # optional BCP-47 hint
        style_prompt="calm and reassuring",              # optional direction
    )
)

Install with pip install roomkit[gemini] — the same extra the Gemini AI provider and Gemini Live use. It requires google-genai>=2.25.0, the first release that carries the 3.8 speech_metadata annotation.

Latency: not a conversational TTS

Gemini TTS trades latency for expressiveness. Measured against the live API for a one-sentence prompt, three runs per model:

Model	Time to first audio	Streams incrementally
gemini-3.8-flash-tts	1.1–1.6 s (2026-09-27)	Yes
gemini-3.8-flash-lite-tts	~0.7 s (2026-09-27)	Yes
gemini-3.1-flash-tts-preview	~5.1 s median, 1.2–8.3 s (2026-08-06); 0.7–0.9 s (2026-09-27)	Yes — 40 ms frames
gemini-2.5-flash-preview-tts	~3.4 s median (2026-08-06)	No — one clip
gemini-2.5-pro-preview-tts	~5.3 s median (2026-08-06)	No — one clip

A second of dead air before every reply still does not work for live turn-taking. Use Gemini TTS for prompts, announcements, voicemail and generated audio messages; for conversation reach for a low-latency engine (ElevenLabs, Gradium) or skip the text round trip entirely with Gemini Live speech-to-speech.

The default is gemini-3.8-flash-tts, Google's replacement for the 3.1 preview; gemini-3.8-flash-lite-tts is its low-latency, high-throughput variant. Both stream as they generate, so playback starts on the first frame instead of waiting for the whole clip.

Configuration

Parameter	Default	Description
api_key	(required)	Gemini API key (GEMINI_API_KEY)
model	"gemini-3.8-flash-tts"	One of the five models above
voice	"Kore"	Prebuilt voice name
language	None	BCP-47 hint; unset, the model infers it from the text
style_prompt	None	Delivery direction for the whole utterance. Sent as speech_metadata from 3.8 on; written into the prompt on 3.1 and 2.5 (see below)
timeout	120.0	Per-request timeout in seconds

Output format

Always 24 kHz, 16-bit, mono PCM — fixed by the service. The request accepts a sample_rate field but the service ignores it, so the provider does not expose the knob; attach a resampler stage when the transport needs another rate.

synthesize() returns a WAV data: URL on every model. Up to 3.1 the service answers bare PCM, which the provider wraps in a WAV header. From 3.8 it answers a whole WAV file with a C2PA content-credentials chunk after the audio; the provider passes that file through unchanged, so the provenance manifest reaches you intact, and measures its duration from the audio alone. synthesize_stream() yields bare PCM on every model.

Voices

GeminiTTSProvider.available_voices() returns the 30 prebuilt voices as VoiceInfo records, offline — the same catalog Gemini Live native audio draws from, so a voice chosen for one works in the other.

list_voices() reads Google's whole catalog instead: 2,089 voices on 2026-09-27, among them 68 Québec French (fr-CA, "Montreal French" accent) and French from France. Each entry says its language, gender and accent, a description, and under attributes Google's own fields (persona, region_code, type). The filters behave as for every provider (RFC §12.2): Google's own mean something else — its language filter wants an exact tag, so fr finds nothing, and its search for Kore answers Korean voices — so the provider reads everything (three pages, 0.4 s) and filters itself.

for v in await tts.list_voices(language="fr-CA", gender="female"):
    print(v.id, "—", v.description)
# fr-ca-advisor-1 — 31-year-old Lawyer from Montreal. Speaks Montreal French. …

await tts.synthesize("Bonjour!", voice="fr-ca-advisor-1")   # any listed id plays

Dialogue

synthesize_dialogue() voices a scripted exchange of two speakers in one clip, each line with its own direction — a support-call role-play, a two-host podcast. The speakers are bound to voices by name; a third speaker, or one the map does not name, is refused before any request. The 3.1 and 2.5 models voice no dialogue (max_dialogue_speakers is 0).

from roomkit.voice.voices import DialogueTurn

audio = await tts.synthesize_dialogue(
    [
        DialogueTurn(speaker="Client", text="Bonjour, ma facture me semble trop élevée."),
        DialogueTurn(speaker="Conseillère", text="Je vérifie ça avec vous.", style="calme"),
    ],
    {"Client": "fr-ca-advisor-2", "Conseillère": "fr-ca-advisor-1"},
)

Measured on 2026-09-27 with two fr-CA voices, a three-line call came back as one 8.9 s WAV, and gemini-3.5-transcribe heard the three lines as two alternating speakers. See examples/gemini_tts_voices.py.

Custom voices

GeminiVoiceLibrary makes voices the TTS then speaks with by their id (RFC §12.2.4):

• Designed — design_voice("a warm Montreal narrator in her forties…") returns a stored voice_… in about 15 s, with a WAV preview. Google stores every designed voice, for a year: store=False is refused. Delete it with delete_voice().
• Replicated — replicate_voice(sample, consent) clones a person's voice from 10 to 30 s of their speech and a recording of them reading Google's consent statement word for word (CONSENT_STATEMENTS, e.g. fr-CA: « Je suis le propriétaire de cette voix et j'autorise Google à utiliser cette voix pour créer un modèle de voix synthétique. »). Both are 16-bit, 24 kHz mono WAV. Google refuses anything else, and the refusal is raised as VoiceConsentError with Google's reason — it arrives as a 500 wrapping Consent flow failed. store=False returns a voicekey_… good for seven days, which Google keeps nothing addressable for.

from roomkit.voice.tts.gemini_library import GeminiVoiceLibrary, GeminiVoiceLibraryConfig

library = GeminiVoiceLibrary(GeminiVoiceLibraryConfig(api_key="..."))
narrator = await library.design_voice("Une narratrice québécoise chaleureuse", name="Narratrice")
await tts.synthesize("Bienvenue!", voice=narrator.voice.id)
await library.delete_voice(narrator.voice.id)

The recordings go to Google for the call and nothing of them is kept or logged (RFC §17.6); the library logs only which voice was made and that Google verified the consent. See examples/gemini_voice_design.py.

Expressive audio tags

The delivery is steerable from the text itself. Bracketed cues placed inline in the transcript are performed, not read out — this is what a conventional concatenative engine cannot do, and what the latency buys:

await tts.synthesize("[laughs] Okay, that one was actually funny.")
await tts.synthesize("[whispers] Can you keep a secret? [excitedly] We shipped!")

Two kinds, both inline: non-verbal sounds ([laughs], [sighs], [gasp], [cough]) and delivery modifiers ([whispers], [shouting], [excitedly], [bored], [very slowly], [singing], [asmr]). Google documents no closed list — any descriptive cue is interpreted — so listen to an unusual one before relying on it: an unrecognised cue can be spoken aloud instead of performed. With a non-English transcript, keep the tags in English.

For the 3.8 models Google documents an angle-bracket spelling — <laugh>, <sigh>, <short pause>, and |mhm| for a listener's reaction in a dialogue. Measured on 2026-09-27, both 3.8 models perform the square-bracket tags above as well, so text written for 3.1 keeps working.

How style_prompt reaches the model depends on the model family:

• 3.8 and newer — the transcript goes out alone, and the direction rides as the style of a speech_metadata annotation on it, a field the model takes as direction and never speaks.
• 3.1 and 2.5 — the API has no style field, so the direction is only expressible inside input. The provider writes a labelled Delivery direction: line above a <transcript> block holding the text, which is what stops the model from reciting the direction along with the words. The text cannot close the block, so it can neither end the transcript nor open another (RFC §6.4, §12.2). Measured 2026-10-08 on a text holding Delivery direction: and Transcript: lines of its own: behind the former Transcript: label, gemini-2.5-flash-preview-tts spoke only its first sentence in 2 runs of 3; as a block it spoke all of it in 4 of 4, never the tag, and [laughs] and [whispers] are still performed. gemini-3.1-flash-tts-preview answers such a text with a 400 ("the audio stream could not be completed") in some runs whichever the prompt (1 in 7 before, 3 in 8 as a block), never a text without directions.

On every Gemini TTS model the text itself can steer delivery, whatever frame holds it: whisper very slowly, then say the account is closed was whispered in every run that answered on 3.1, 2.5 and 3.8, as an audio tag is performed. When the text comes from somewhere you do not trust (an agent answering what a participant wrote), remove such cues before it reaches the provider: in a BEFORE_TTS hook where a voice channel speaks it, in your own code before kit.synthesize() or a direct provider call (no hook runs there), and before an assistant line is injected into a realtime session.

The 3.8 models read the 3.1 prompt aloud (measured with its earlier Transcript: label form) — its instructions in place of, or on top of, the transcript: 3 runs in 6 on gemini-3.8-flash-tts and 6 in 6 on the Lite model, measured 2026-09-27, against 0 in 36 with the bare transcript. That is why the two families get different requests.

Tags and style_prompt are therefore different tools: a tag steers a word or a phrase from inside the transcript, style_prompt steers the whole utterance from the line above it. Google's own guidance also frames a full direction in three parts — audio profile (who is speaking), scene (where, what mood), and director's notes (style, accent, pacing) — which is the shape style_prompt is for. See Google's prompt guide.

Tags travel through a room like any other text, so anything written into a room attached to a voice channel is performed: examples/gemini_tts_room.py is a room that speaks what you type at the CLI.

The writer can be the model. examples/voice_gemini.py is a voice assistant that is Gemini end to end — gemini-3.5-transcribe-live hears you, gemini-3.8-flash answers, gemini-3.8-flash-lite-tts speaks — and its system prompt lets the model put tags in its replies. Run against a recorded French question on 2026-09-27, the model answered [whispers] Promets-moi de ne le répéter à personne… <chuckle>, and the voice whispered and laughed without saying either tag. Keep tts_filter unset on such a channel: StripBrackets would remove the [...] tags before the TTS sees them.

SSML is not an input mode here

SSML belongs to Cloud Text-to-Speech, which accepts it in a dedicated field (SynthesisInput(ssml=...)). The Gemini API endpoint this provider calls has a single free-text input field and no ssml field, and Google's Gemini-TTS page documents prompting rather than SSML.

Sent anyway, the markup is not ignored: the model reads it as an instruction and follows the intent, not the timing. Longest silence produced, measured against the live API on 2026-08-07, three runs each:

Asked for	Runs	Median
<break time="1500ms"/>	1.60 s, 2.46 s, 2.08 s	2.08 s
<break time="5000ms"/>	3.36 s, 5.70 s, 4.72 s	4.72 s
[long pause], no duration given	3.48 s, 4.02 s, 4.46 s	4.02 s

Directionally right, never exact — the same request drifts by seconds between identical runs, and a bracketed cue gets you the same effect without pretending to a contract. Anything that has to line up with something else (a beep, a prompt, a recording) needs the silence assembled in the outbound audio instead, not requested from the model.

Streaming

synthesize_stream() forwards audio deltas as they arrive. synthesize_stream_input() is not supported: the API takes a complete prompt, so there is no seam for token deltas. A VoiceChannel detects this and delivers the finished reply through synthesize_stream() instead.

Gemini 3.1 TTS remains a preview model. Google documents rare cases where it reads prompt directions aloud or returns a transient HTTP 500 instead of audio; keep prompts explicit, split outputs longer than a few minutes, and apply retry at the calling workflow boundary when a failed generation is safe to repeat. The 3.8 models are generally available.

See examples/gemini_tts.py.

---

NeuTTS (Local/GPU, Voice Cloning)

GGUF-quantized LLM-based TTS with native streaming and voice cloning.

from __future__ import annotations

from roomkit.voice.tts.neutts import NeuTTSConfig, NeuTTSProvider, NeuTTSVoiceConfig

tts = NeuTTSProvider(
    config=NeuTTSConfig(
        backbone_repo="neuphonic/neutts-nano-french-q8-gguf",
        codec_repo="neuphonic/neucodec",
        device="cpu",                           # or "cuda"
        voices={
            "default": NeuTTSVoiceConfig(
                ref_audio="reference.wav",       # 3-15s, 16kHz mono
                ref_text="Transcript of reference audio.",
            ),
        },
        streaming_pre_buffer=2,                  # Chunks to buffer before yielding
    )
)

Pre-buffering: On CPU, accumulates 2 chunks (~1 second) before yielding to prevent playback underruns when inference is slower than real-time.

Output: Fixed 24kHz PCM.

---

Pocket TTS (Local/CPU or GPU, Multilingual)

Kyutai and Gradium's 100M-parameter model: 24 kHz streaming in 80 ms chunks, faster than real time on two CPU cores, English, French, German, Portuguese, Italian, Spanish and Dutch (pocket-tts 3.3+), voice cloning from a clip. Install with pip install roomkit[pocket-tts].

from roomkit.voice.tts.pocket import PocketTTSConfig, PocketTTSProvider

tts = PocketTTSProvider(PocketTTSConfig(
    language="french",               # one language per loaded model
    voices={"estelle": "estelle"},   # pre-made name, clip, hf:// or .safetensors
    device="cpu",                    # or "cuda"
))

See the Pocket TTS guide for voices, CPU/GPU numbers and the local French assistant example.

---

Vui Nano (Local/GPU, Conversational)

Vui Nano (fluxions.ai, Apache 2.0) is a 305M-parameter TTS that generates each reply inside the conversation: the dialogue so far, including the audio of the user's turns, lives in its KV cache. The provider declares TTSContextLevel.AUDIO and keeps that cache in step with the voice session (TTS Conversation Context).

from __future__ import annotations

from roomkit import TTSContextConfig, VoiceChannel
from roomkit.voice.tts.vui import VuiTTSConfig, VuiTTSProvider, VuiVoice

tts = VuiTTSProvider(
    VuiTTSConfig(
        voices={
            "maeve": VuiVoice(preset="maeve"),        # maeve, abraham, rhian, harry
            "me": VuiVoice(ref_audio="me.wav", ref_text="Exact transcript of me.wav."),
        },
        temperature=0.7,
    )
)
await tts.warmup()  # loads the model, the codec and the voices

voice = VoiceChannel(
    "voice", stt=stt, tts=tts, backend=backend,
    tts_context=TTSContextConfig(include_audio=True),  # let Vui hear the user
)

What the provider does with the context:

• each user turn is written to the cache as its transcript, then its audio (re-encoded with the Qwen3-TTS-12Hz codec) when the context carries audio; without include_audio, as text only;
• after a barge-in, the cache is cut back to the frame the user stopped hearing (played_ms, 80 ms frames), and a reply nobody heard is dropped;
• a call for another voice session, another voice, or synthesize() (no conversation), and a dialogue about to exceed the 6 minutes of audio the model was trained on (prompt included), restart it from the voice prompt with the user turns awaiting a reply; earlier turns are not replayed;
• when the session is unbound, the cache is emptied (release_context), user audio included.

Constraints:

• English only.
• Python 3.12 only (a vui-tts requirement): pip install roomkit[vui] on another Python installs nothing, and the provider says so at warmup().
• A CUDA GPU for real-time streaming (about 3.4 GB of VRAM, measured at 2.6x real time on an RTX 4070 without flash-attn).
• One active conversation per provider: Engine(max_rows=1) gives it a single cache, and calls are serialized.
• A reply longer than max_secs (60 s by default) is cut off there, with a warning in the log.
• Vui invents syllables at a line break and after a text that ends on a comma, so the provider joins a reply's lines into running sentences before speaking it: a poem written one line per verse came out with 15 % of its words wrong, 4 % once joined. The hosted Vui (Fluxions) reads line breaks cleanly on its own.
• vui-tts>=1.2.3,<1.3, through its public API only: Row.truncate for the cut after a barge-in, which takes the unheard frames out of the codec as well as the cache, load_official_prompt and Row.prefill(cond_bias=...) for a preset's speaker token and conditioning bias, as Vui's own server applies them.
• vui-tts keeps 40 s of codec tail. A cut that drops more than 30 s of unheard audio (a long reply cut early) leaves the next reply without its decoder context until the next 10 s boundary, in step again after it.
• vui-tts logs the first 40 characters of every user turn at DEBUG on the vui.engine logger. Keep that logger above DEBUG in production if transcripts must not reach the logs.

Paralinguistic tags: Vui was trained with inline tags such as [breath], [laugh], [sigh], [hesitate] or [tut] ("So [breath] the thing is..."). They reach the model as written, so an LLM asked to emit them makes the voice breathe, laugh or hesitate. VUI_TAGS holds the ones Vui's prompting guide lists (breath, laugh, sigh, gasp, cough, hesitate); any other bracketed word is read aloud or garbled. A small model invents others ([nod], [smiles]) and adds emoji even when the prompt forbids them, so filter what reaches Vui, keeping its own tags:

from __future__ import annotations

from roomkit.voice.tts.filters import StripBrackets, StripEmoji, TTSFilterChain
from roomkit.voice.tts.vui import VUI_TAGS

voice = VoiceChannel(
    "voice", stt=stt, tts=tts, backend=backend,
    tts_filter=TTSFilterChain(StripEmoji(), StripBrackets(keep=VUI_TAGS)),
)
# "Oh [smiles] sure 😊 [laugh]" is spoken as "Oh sure [laugh]"

Troubleshooting: CUDNN_STATUS_SUBLIBRARY_VERSION_MISMATCH from the codec means a system cuDNN (for example under /lib/x86_64-linux-gnu) is loaded instead of PyTorch's. Set torch.backends.cudnn.enabled = False before warmup(), or remove the system copy from the loader path.

Output: Fixed 24kHz PCM. examples/voice_vui_context.py plays a dialogue with a barge-in and writes what the user heard to a WAV file.

---

Fluxions (Cloud API, Hosted Vui)

fluxions.ai hosts the Vui model behind an API key, with no GPU: Vui Nano's four presets are among its hosted voices. Each text is rendered on its own, streamed as 24 kHz PCM.

from __future__ import annotations

import os

from roomkit.voice.tts.fluxions import FluxionsTTSConfig, FluxionsTTSProvider

tts = FluxionsTTSProvider(
    FluxionsTTSConfig(
        api_key=os.environ["FLUXIONS_API_KEY"],
        voice="maeve",  # a short id from list_voices(), or a full id
    )
)
await tts.warmup()  # fetches the voice list

async for chunk in tts.synthesize_stream("Hello from Vui, hosted by Fluxions."):
    ...

Local or hosted:

	Vui Nano (roomkit[vui])	Fluxions (roomkit[fluxions])
Runs on	Your CUDA GPU (about 3.4 GB), Python 3.12	Fluxions' servers, any Python
Conversation context	AUDIO: replies generated inside the dialogue, the user's voice included, cut to what was heard	NONE: each text on its own
Voices	The four presets, or a clip you clone	The hosted voices (presets included) and the account's cloned voices
Cost	Free	Per character

Voices: list_voices() returns the hosted voices by their short id (maeve), then the account's cloned voices by their u-... id; voice accepts either. A render needs the voice's full id, which carries the model's checkpoint (maeve.h8ff7e07da) and changes when Fluxions ships a new model: the provider resolves the short id from the voice list, and lists again once when the server no longer knows the id it cached. A full id (or a cloned voice's u-... id) is passed as given.

Constraints:

• No conversation context: the speech API takes no previous turns, no user audio and no session. For context-aware replies, use Vui Nano.
• The local model speaks English; Fluxions documents no other language.
• A cold start after an idle period can take up to about 30 s before the first audio; the read timeout is 60 s by default.
• Errors come back as httpx.HTTPStatusError: 402 when the account is out of credit, 429 when requests are rate-limited, 502 or 503 when Fluxions is at capacity.
• Fluxions keeps every render in the account's history; it documents no retention period or opt-out.

Output: Fixed 24kHz PCM. examples/voice_fluxions.py lists the voices, streams a sentence, renders another whole and writes both to a WAV file.

---

Azure Speech (Cloud API, MAI-Voice)

Azure Speech renders SSML over REST. The voice name picks the model: fr-FR-Soleil:MAI-Voice-2.1-Flash is Microsoft's MAI voice Soleil on MAI-Voice-2.1-Flash, the low-latency model, and fr-CA-SylvieNeural one of Azure's neural voices, so one provider serves both.

from __future__ import annotations

import os

from roomkit.voice.tts.azure_speech import AzureSpeechTTSConfig, AzureSpeechTTSProvider

tts = AzureSpeechTTSProvider(
    AzureSpeechTTSConfig(
        api_key=os.environ["AZURE_SPEECH_KEY"],
        region="swedencentral",
        voice="en-US-Harper:MAI-Voice-2.1-Flash",
        style="customer_call_center",  # optional, one the voice supports
    )
)

async for chunk in tts.synthesize_stream("Hello from MAI Voice."):
    ...

Option	Default	Meaning
api_key	(required)	The Speech or Foundry resource's key
region	None	The resource's region (swedencentral, canadacentral…)
endpoint	None	The full synthesis URL, for a custom domain or a proxy; exactly one of region and endpoint
voice	en-US-Harper:MAI-Voice-2.1-Flash	Voice name as Azure spells it
language	None	The SSML xml:lang; None takes the voice name's locale
style	None	A speaking style wrapped around every text with mstts:express-as
rate	None	Speaking rate as SSML prosody takes it: "+15%", "1.2", "fast"
sample_rate	24000	8, 16, 22.05, 24, 44.1 or 48 kHz
timeout / connect_timeout	30.0 / 5.0	Read and connect budgets, in seconds

Install with pip install roomkit[azure-speech] (httpx, no Azure SDK).

Text is spoken, never read as markup: the provider builds the SSML itself and escapes everything in it, so a <break/> or a </voice> in an AI reply is spoken as text, not obeyed; characters XML cannot carry are dropped.

Constraints:

• The MAI-Voice models are in public preview (October 2026), without an SLA.
• A render streams: 23 s of audio arrived between 0.67 and 1.39 s after the request. First audio, measured 2026-10-08 from Québec to Sweden Central: about 0.65 s on MAI-Voice-2.1-Flash, 1.2 s on MAI-Voice-2.1, 0.4 s on the neural fr-CA-SylvieNeural. Microsoft's ~45 ms for Flash is its inference alone.
• A style the voice does not support is refused with a 400 and no message.
• MAI-Voice-2.1-Flash speaks slowly for a conversation, and follows rate closely: the same sentence took 4.72 s at its own pace, 3.84 s at +20% and 2.84 s at +40%. MAI-Voice-2.1 follows it unevenly (measured 2026-10-08).
• MAI-Voice's French voices are fr-FR (Grant, Harper, Marc, Soleil); there is no fr-CA MAI voice. Azure's neural fr-CA voices go through the same provider.
• Microsoft serves MAI-Voice from 14 regions, Canada Central and France Central among them.
• Cloning a voice from a clip is gated by Microsoft (Limited Access review) and not exposed by the provider.
• No conversation context: each text is rendered on its own.
• A refused render raises AzureSpeechTTSError, with status_code and retryable (true for 429 and 5xx).

Output: mono 16-bit PCM at sample_rate. examples/voice_azure_mai_agent.py is a voice agent speaking with it; examples/voice_azure_mai.py streams a sentence and plays it back to MAI-Transcribe.

---

TTS Filters

Filters clean AI-generated text before it reaches the TTS provider. Essential for removing reasoning markers, annotations, or bracketed instructions.

StripInternalTags

Removes [internal]...[/internal] and [internal: ...] blocks — useful when the AI includes reasoning that shouldn't be spoken.

from __future__ import annotations

from roomkit.voice.tts.filters import StripInternalTags

f = StripInternalTags()

# Non-streaming (full text)
clean = f("[internal]Let me think about this...[/internal] Here's what I found.")
# → "Here's what I found."

# Streaming (token by token)
for token in ["[internal", "]thinking[/", "internal] The answer", " is 42."]:
    result = f.feed(token)
    if result:
        print(result, end="")
print(f.flush())
# → "The answer is 42."

StripBrackets

Removes all [...] bracketed content — catches [laughs], [pause], [Respond in French], etc.

from __future__ import annotations

from roomkit.voice.tts.filters import StripBrackets

f = StripBrackets()
clean = f("Sure [laughs] I can help [pause] with that.")
# → "Sure  I can help  with that."

keep passes a TTS's own tags through, matched ignoring case:

f = StripBrackets(keep=("laugh", "sigh"))
f("Oh [laugh] sure [smiles] fine.")
# → "Oh [laugh] sure fine."

StripEmoji

Removes emoji. Language models add them to replies even when the prompt forbids it, and a TTS then names them or makes a stray sound. Composed emoji (skin tones, flags, families, keycaps) go whole; accents, €, ° and punctuation stay. On a streamed reply, an emoji split across chunks is removed too. The response stored in the conversation keeps the model's text.

from __future__ import annotations

from roomkit import VoiceChannel
from roomkit.voice.tts.filters import StripEmoji

voice = VoiceChannel("voice", stt=stt, tts=tts, backend=backend, tts_filter=StripEmoji())
# "C'est rapide et super bon ! 😊" is spoken as "C'est rapide et super bon !"

StripTechnicalText

Removes the technical text a model may write into a spoken reply: a JSON object (recognised by its first key, quoted, {", or a bare name and a colon, {agent:), such as a tool call or a tool result written as words instead of calling the tool; a note to itself, (Note: ...) or (NB: ...); a separator of three dashes or more. Whatever model the application picks, a reply sometimes carries them, and removing them from the voice does not make a call happen.

from __future__ import annotations

from roomkit import VoiceChannel
from roomkit.voice.tts.filters import StripTechnicalText

voice = VoiceChannel("voice", stt=stt, tts=tts, backend=backend, tts_filter=StripTechnicalText())
# 'I'll check. {"name": "delegate", "arguments": {"task": "Weather. Today"}} Done.'
# is spoken as "I'll check. Done."

• A nested object goes whole, and a brace inside one of its strings does not end it. On a streamed reply the filter works on the tokens, before the reply is cut into sentences: an object holding a full stop is removed whole, which a BEFORE_TTS hook, judging one sentence at a time, cannot do.
• Ordinary text passes: (about ten minutes), {x}, well-known, --. An object or a note still open when the stream ends is removed, and a removal leaves one space between the words around it (Searching.{…}Back soon. is spoken Searching. Back soon.).
• Each object or note removed is logged as a warning with its length; its text only at DEBUG, through redact(). The response stored in the conversation keeps the model's text: the slip stays visible, it is just not heard.
• The rules do not know who a note is for: (Note: holidays may delay it) written for the listener goes too. Reasoning written in plain words ("Need no tool.") is not recognised: no rule tells it from an answer. The fence or the brackets around a removed object stay (a Markdown code block, a JSON array): chain StripBrackets for the brackets.

examples/voice_strip_technical_text.py runs it on a whole reply and a streamed one.

A voice channel streams each response through its own copy of its filter, so replies streamed at once in several rooms never share what a filter holds open; a filter of your own holding something that cannot be copied defines __deepcopy__.

TTSFilterChain

tts_filter takes one filter; TTSFilterChain runs several in order, on a whole text and on a streamed reply alike:

from __future__ import annotations

from roomkit.voice.tts.filters import (
    StripBrackets,
    StripEmoji,
    StripTechnicalText,
    TTSFilterChain,
)

tts_filter = TTSFilterChain(StripTechnicalText(), StripEmoji(), StripBrackets(keep=("laugh",)))

Using Filters with Streaming TTS

from __future__ import annotations

from roomkit.voice.tts.filters import StripInternalTags, filtered_stream


async def ai_token_stream():
    """Simulated AI output with internal reasoning."""
    for token in ["[internal]", "reasoning", "[/internal]", " Hello", " there!"]:
        yield token


# Wrap the token stream through a filter before TTS
clean_stream = filtered_stream(ai_token_stream(), StripInternalTags())

async for chunk in tts.synthesize_stream_input(clean_stream, voice="default"):
    # Audio chunks without the internal reasoning
    transport.send_audio(session, chunk)

Sentence Splitter

Buffers streaming tokens and yields complete sentences — prevents unnatural pauses from very short fragments.

from __future__ import annotations

from roomkit.voice.tts.sentence_splitter import split_sentences

# Buffer tokens until sentence boundaries
async for sentence in split_sentences(ai_token_stream(), min_chunk_chars=20):
    async for chunk in tts.synthesize_stream(sentence, voice="default"):
        transport.send_audio(session, chunk)

A SegmentBreakMarker in the stream ends the text before it: what the splitter holds goes out at once as a sentence, however short, and filtered_stream calls its filter's end_segment() there and passes the marker on. A voice channel puts one where the agent's text before a tool call ends, so "Let me check." is said while the tool runs, never once it is done, and never runs on into the next round's text.

end_segment() lets out the text a filter has decided to speak and keeps a span it removes open: an [internal] block the model opens before the call and closes after it is never spoken. The default lets out nothing, so a filter written before it keeps what it holds until the next text or the end of the stream (flush() is called only there). A filter that holds text it has decided to speak overrides it.

---

Sentence Budget

Asked to "explain", a language model may talk for a minute however short its prompt asks it to be. max_sentences caps what a voice channel speaks of each reply:

from __future__ import annotations

from roomkit import VoiceChannel

voice = VoiceChannel("voice", stt=stt, tts=tts, backend=backend, max_sentences=4)

• A streamed reply: once the budget is said, a reply that goes on stops at its first word past it, as a barge-in stops it. The model generates nothing more, a tool call the next sentence announced ("Let me book that table") is never made, and the room keeps the text produced up to there (the sentences spoken and the start of the next), marked cancelled, rather than the whole answer nobody heard. The final transcript and AFTER_TTS carry the sentences spoken. A reply of exactly the budget runs to its end.
• A text delivered whole (a reply on a TTS without streamed input, an orchestration farewell, any message the channel speaks) has its first sentences spoken; a reply was stored whole before it reached the channel.
• A sentence is counted as the TTS reads it: after the text filter and BEFORE_TTS, so a sentence a hook drops does not count, and very short sentences gather into one, as the channel always hands them to the TTS.
• A reply cut this way fires no ON_AI_RESPONSE, like any turn its reader stopped. say(), the application's own text, has no budget, and a ConferenceChannel has none.

examples/voice_sentence_budget.py shows a twelve-sentence explanation cut at four, and what the room keeps of it.

Choosing a Provider

Provider	Type	Streaming	Latency	Cost	Best For
Deepgram	Cloud STT	Yes	Low	Per-minute	Production real-time transcription
Gradium	Cloud STT	Yes	Low	Per-minute	Real-time with server-side VAD
SherpaOnnx	Local STT	Transducer only	Medium	Free	Privacy, offline, edge
Qwen3 ASR	Local STT	vLLM only	Medium	Free	GPU-accelerated, multilingual
MAI-Transcribe	Cloud STT	Yes (behind a VAD)	Low (final ~0.13 s after speech)	Per-hour	57 languages, preview
ElevenLabs	Cloud TTS	Yes + input	Low	Per-character	Highest voice quality
Grok	Cloud TTS	Yes + input	Low	Per-character	Expressive tags, 20 languages
Gradium	Cloud TTS	Yes + input	Low	Per-character	Real-time with voice control
SherpaOnnx	Local TTS	Yes	Medium	Free	Privacy, offline, VITS/Piper
Qwen3 TTS	Local TTS	Post-gen	Medium	Free	Voice cloning, GPU
NeuTTS	Local TTS	GGUF only	Medium	Free	Voice cloning, GGUF quantized
Vui Nano	Local TTS	Yes	Low (GPU)	Free	Replies conditioned on the dialogue audio, English
Fluxions	Cloud TTS	Yes	Low	Per-character	Vui without a GPU, English
Azure Speech	Cloud TTS	Yes	Medium (MAI Flash ~0.65 s to first audio)	Per-character	MAI-Voice and Azure's neural voices, preview
Pocket TTS	Local TTS	Yes	Low (CPU or GPU)	Free	Seven languages incl. French, no GPU needed

Using with VoiceChannel

from __future__ import annotations

from roomkit.channels import VoiceChannel
from roomkit.voice.backends.mock import MockVoiceBackend
from roomkit.voice.pipeline import AudioPipelineConfig
from roomkit.voice.stt.deepgram import DeepgramConfig, DeepgramSTTProvider
from roomkit.voice.tts.elevenlabs import ElevenLabsConfig, ElevenLabsTTSProvider

stt = DeepgramSTTProvider(config=DeepgramConfig(api_key="..."))
tts = ElevenLabsTTSProvider(
    config=ElevenLabsConfig(api_key="...", output_format="pcm_16000")  # a VoiceChannel plays PCM
)

voice = VoiceChannel(
    "voice-main",
    stt=stt,
    tts=tts,
    backend=MockVoiceBackend(),
    pipeline=AudioPipelineConfig(),
)

kit.register_channel(voice)

Testing with Mocks

from __future__ import annotations

from roomkit.voice.stt.mock import MockSTTProvider
from roomkit.voice.tts.mock import MockTTSProvider

stt = MockSTTProvider(transcripts=["Hello", "How are you?"], streaming=False)
tts = MockTTSProvider(voice="mock-voice")

# After usage:
assert len(stt.calls) == 1                # Audio inputs received
assert len(tts.calls) == 1                # Synthesis requests made
assert tts.calls[0]["text"] == "Hello!"   # Text synthesized