Voice Providers¶

Voice Backend¶

VoiceBackend ¶

Bases: ABC

Abstract base class for voice transport backends.

VoiceBackend handles the transport layer for real-time audio: - Managing voice session connections - Streaming audio to/from clients - Delivering raw inbound audio frames via on_audio_received

The backend is framework-agnostic and a pure transport — all audio intelligence (VAD, denoising, diarization) is handled by the AudioPipeline.

Example usage

backend = WebRTCVoiceBackend()

Register raw audio callback¶

backend.on_audio_received(handle_audio_frame)

Connect a participant¶

session = await backend.connect("room-1", "user-1", "voice-channel")

Stream audio to the client¶

await backend.send_audio(session, audio_chunks)

Disconnect¶

await backend.disconnect(session)

name `abstractmethod` `property` ¶

name

Backend name (e.g., 'webrtc', 'websocket', 'livekit').

auto_connect `property` ¶

auto_connect

Whether attach_channel should auto-create a session.

Returns True for single-user backends (local mic, single-user FastRTC) where the developer IS the participant. Returns False for server backends (SIP, RTP, WebRTC) that accept external connections.

capabilities `property` ¶

capabilities

Declare supported capabilities.

Override to enable features like interruption, barge-in, etc. By default, no optional capabilities are supported.

Returns:

Type	Description
`VoiceCapability`	Flags indicating supported capabilities.

feeds_aec_reference `property` ¶

feeds_aec_reference

Whether this backend feeds AEC reference at the transport level.

When True, the pipeline skips aec.feed_reference() in the outbound path to avoid double-feeding. Transport-level feeding (from the speaker callback) is preferred because it is time-aligned with actual speaker output.

audio_received_callback `property` ¶

audio_received_callback

Return the currently registered audio-received callback, if any.

supports_playback_callback `property` ¶

supports_playback_callback

Whether this backend fires :meth:on_audio_played callbacks.

When True, the pipeline can rely on playback-time AEC reference instead of generation-time feeding from process_outbound.

connect `async` ¶

connect(room_id, participant_id, channel_id, *, metadata=None)

Create a new voice session for a participant.

Backends that initiate connections (VoiceChannel path) override this. Backends that receive external connections (realtime transport path) override :meth:accept instead.

Parameters:

Name	Type	Description	Default
`room_id`	`str`	The room to join.	required
`participant_id`	`str`	The participant's ID.	required
`channel_id`	`str`	The voice channel ID.	required
`metadata`	`dict[str, Any] \| None`	Optional session metadata.	`None`

Returns:

Type	Description
`VoiceSession`	A VoiceSession representing the connection.

disconnect `abstractmethod` `async` ¶

disconnect(session)

End a voice session.

Parameters:

Name	Type	Description	Default
`session`	`VoiceSession`	The session to disconnect.	required

send_audio `abstractmethod` `async` ¶

send_audio(session, audio)

Send audio to a voice session.

The audio is decoded 16-bit signed PCM: a VoiceChannel refuses a TTS chunk in another encoding before it gets here (RFC section 12.2), and encoding for the wire (G.711, Opus) is the backend's own work.

An exception raised by the chunk stream (a TTS failing mid-sentence) is raised to the caller once the backend has stopped playing; the backend may log and absorb errors of its own transport. Reading the stream inside :class:PlaybackErrors does both.

Parameters:

Name	Type	Description	Default
`session`	`VoiceSession`	The target session.	required
`audio`	`bytes \| AsyncIterator[AudioChunk]`	Raw audio bytes or an async iterator of AudioChunks for streaming.	required

send_audio_sync ¶

send_audio_sync(session, chunk)

Synchronously send a single audio chunk to a session.

Used by the audio bridge for frame-by-frame forwarding from audio callback threads where await is not available. Backends that support bridging SHOULD override this with an efficient implementation. The default schedules the async send_audio() on the event loop, which adds latency.

Parameters:

Name	Type	Description	Default
`session`	`VoiceSession`	The target session.	required
`chunk`	`AudioChunk`	A single audio chunk to send.	required

get_session ¶

get_session(session_id)

Get a session by ID.

Override for backends that track sessions internally.

Parameters:

Name	Type	Description	Default
`session_id`	`str`	The session ID to look up.	required

Returns:

Type	Description
`VoiceSession \| None`	The VoiceSession if found, None otherwise.

list_sessions ¶

list_sessions(room_id)

List all active sessions in a room.

Override for backends that track sessions internally.

Parameters:

Name	Type	Description	Default
`room_id`	`str`	The room to list sessions for.	required

Returns:

Type	Description
`list[VoiceSession]`	List of active VoiceSessions in the room.

close `async` ¶

close()

Release backend resources.

Override in subclasses that need cleanup.

on_audio_received ¶

on_audio_received(callback)

Register a callback for raw inbound audio frames.

Backends supporting shared ownership return an idempotent unsubscribe function. Legacy implementations may return None. The same convention applies to playback and disconnect registrations.

The pipeline or channel calls this to receive every audio frame produced by the transport.

Parameters:

Name	Type	Description	Default
`callback`	`AudioReceivedCallback`	Function called with (session, audio_frame).	required

on_session_ready ¶

on_session_ready(callback)

Register callback for when a session's audio path becomes live.

Fired when the transport is ready to send/receive audio for a session (e.g. WebSocket connected, RTP socket active, mic stream started). The VoiceChannel uses this together with bind_session to implement the dual-signal ON_SESSION_STARTED hook.

Parameters:

Name	Type	Description	Default
`callback`	`SessionReadyCallback`	Function called with `(session)`.	required

on_barge_in ¶

on_barge_in(callback)

Register callback for barge-in detection.

Only called if capabilities includes BARGE_IN. Backends should call this when user starts speaking while audio is being played (TTS interruption).

Parameters:

Name	Type	Description	Default
`callback`	`BargeInCallback`	Function called with (session).	required

send_dtmf ¶

send_dtmf(session, digit, duration_ms=160)

Send a DTMF digit to the remote party.

Sends an RFC 4733 telephone-event via RTP (out-of-band signaling). Only backends with DTMF_SIGNALING capability support this.

Parameters:

Name	Type	Description	Default
`session`	`VoiceSession`	The voice session to send DTMF on.	required
`digit`	`str`	The DTMF digit to send ('0'-'9', '*', '#', 'A'-'D').	required
`duration_ms`	`int`	Duration of the tone in milliseconds (default 160).	`160`

Raises:

Type	Description
`NotImplementedError`	If the backend does not support sending DTMF.

cancel_audio `async` ¶

cancel_audio(session)

Cancel ongoing audio playback for a session.

Delegates to :meth:interrupt and returns True. Subclasses may override for more nuanced behaviour.

Parameters:

Name	Type	Description	Default
`session`	`VoiceSession`	The session to cancel audio for.	required

Returns:

Type	Description
`bool`	True if audio was cancelled, False if nothing was playing.

accept `async` ¶

accept(session, connection)

Bind an external connection to a session.

Backends that receive connections from external sources (e.g. WebSocket, WebRTC, SIP) override this. Backends that create their own connections override :meth:connect instead.

Parameters:

Name	Type	Description	Default
`session`	`VoiceSession`	The voice session to bind.	required
`connection`	`Any`	Protocol-specific connection object.	required

interrupt ¶

interrupt(session)

Signal interruption — flush outbound queue, stop playback.

set_input_muted ¶

set_input_muted(session, muted)

Mute or unmute the input (microphone) for a session.

Parameters:

Name	Type	Description	Default
`session`	`VoiceSession`	The session to mute/unmute.	required
`muted`	`bool`	`True` to mute, `False` to unmute.	required

set_input_gated ¶

set_input_gated(session, gated)

Gate or un-gate audio input for primary speaker mode.

When gated, audio is not forwarded to provider callbacks but may still be fed to a pipeline for diarization analysis.

Parameters:

Name	Type	Description	Default
`session`	`VoiceSession`	The session to gate/un-gate.	required
`gated`	`bool`	`True` to gate, `False` to un-gate.	required

on_client_disconnected ¶

on_client_disconnected(callback)

Register callback for client disconnection.

Parameters:

Name	Type	Description	Default
`callback`	`TransportDisconnectCallback`	Called with (session) when the client disconnects.	required

on_speaker_change ¶

on_speaker_change(callback)

Register callback for speaker change events.

Parameters:

Name	Type	Description	Default
`callback`	`SpeakerChangeCallback`	Called with (session, diarization_result).	required

end_of_response ¶

end_of_response(session)

Signal end of an AI response for outbound pacing.

wait_playback `async` ¶

wait_playback(session)

Wait for queued response audio to play; return False if discarded.

Call after the channel is idle (its EOR has reached the transport). Queued transports may override this with a response boundary so that continuous silence after the response does not keep the wait open.

is_playing ¶

is_playing(session)

Check if audio is currently being sent to the session.

Used for barge-in detection to know if interruption is possible.

Parameters:

Name	Type	Description	Default
`session`	`VoiceSession`	The session to check.	required

Returns:

Type	Description
`bool`	True if audio is currently playing, False otherwise.

on_audio_played ¶

on_audio_played(callback)

Register a callback for audio frames as they are played.

Called with each audio frame at the moment it is output by the speaker, providing time-aligned reference for echo cancellation. The pipeline uses this to feed AEC reference at the correct time.

Note

Callbacks may be invoked from the audio I/O thread — implementations must be thread-safe. A backend with a persistent output stream may mark the final drained frame with frame.metadata["playback_ended"] = True: the physical end of the response. A pipeline AEC is bypassed there unless the backend also sets frame.metadata["reference_continues"] = True, saying it reports every played block between responses too (silence included); the canceller then runs on that silent reference until the stream ends (RFC §12.3.4).

Parameters:

Name	Type	Description	Default
`callback`	`AudioPlayedCallback`	Function called with (session, audio_frame).	required

on_aec_tap ¶

on_aec_tap(callback)

Register a callback for this transport's own echo cancellation, for debug taps (RFC §12.3.15).

A backend that cancels echo itself (feeds_aec_reference) calls it for every captured frame with "transport_raw" (the frame before its AEC) and "aec_reference" (the reference its AEC received meanwhile, silence where none), both the captured frame's length, so they align with the pipeline's raw sample for sample. It calls it on the event loop, never on an audio I/O thread. Default: no such frames.

Parameters:

Name	Type	Description	Default
`callback`	`AECTapCallback`	Function called with (session, stage, audio_frame).	required

set_trace_emitter ¶

set_trace_emitter(emitter)

Set a callback for emitting protocol traces.

Called by the owning channel when trace observers are registered. Implementations should store the emitter and call it at key protocol points (e.g. INVITE, BYE for SIP).

Parameters:

Name	Type	Description	Default
`emitter`	`Callable[..., Any] \| None`	The channel's :meth:`emit_trace` method, or `None` to disable.	required

send_transcription `async` ¶

send_transcription(session, text, role='user')

Send transcription text to the client for UI display.

Optional method for backends that support sending text updates.

Parameters:

Name	Type	Description	Default
`session`	`VoiceSession`	The voice session to send to.	required
`text`	`str`	The transcribed or response text.	required
`role`	`str`	Either "user" (transcription) or "assistant" (AI response).	`'user'`

PlaybackErrors ¶

PlaybackErrors(log, message, *args)

Log and absorb a backend's own errors while it plays, never its stream's.

An exception raised by the stream send_audio reads (the TTS provider's, or a chunk the VoiceChannel refused) belongs to the caller (RFC section 12.2): it leaves the block even when the backend caught it on the way. The backend's own errors, its transport's, are logged with message and end the block quietly. A cancellation is never touched::

with PlaybackErrors(logger, "Error sending audio to %s", session.id) as playback:
    async for chunk in playback.watch(audio):
        ...

watch ¶

watch(chunks)

The stream to read, so its failure is told apart from the backend's.

VoiceCapability ¶

Bases: Flag

Capabilities a VoiceBackend can support.

Backends declare their capabilities via the capabilities property. This allows RoomKit to know which features are available and enables integrators to choose backends based on their needs.

Example

class MyBackend(VoiceBackend): @property def capabilities(self) -> VoiceCapability: return ( VoiceCapability.INTERRUPTION | VoiceCapability.BARGE_IN )

NONE `class-attribute` `instance-attribute` ¶

NONE = 0

No optional capabilities (default).

INTERRUPTION `class-attribute` `instance-attribute` ¶

INTERRUPTION = auto()

Backend can cancel ongoing audio playback (cancel_audio).

BARGE_IN `class-attribute` `instance-attribute` ¶

BARGE_IN = auto()

Backend detects and handles barge-in (user interrupts TTS).

NATIVE_AEC `class-attribute` `instance-attribute` ¶

NATIVE_AEC = auto()

Backend provides its own Acoustic Echo Cancellation.

NATIVE_AGC `class-attribute` `instance-attribute` ¶

NATIVE_AGC = auto()

Backend provides its own Automatic Gain Control.

DTMF_INBAND `class-attribute` `instance-attribute` ¶

DTMF_INBAND = auto()

Backend can detect DTMF tones from the audio stream.

DTMF_SIGNALING `class-attribute` `instance-attribute` ¶

DTMF_SIGNALING = auto()

Backend receives DTMF via out-of-band signaling (e.g. SIP INFO).

NATIVE_BRIDGE `class-attribute` `instance-attribute` ¶

NATIVE_BRIDGE = auto()

Backend can bridge audio at the transport level (RTP relay).

VoiceSession `dataclass` ¶

VoiceSession(id, room_id, participant_id, channel_id, state=CONNECTING, provider_session_id=None, created_at=_utcnow(), metadata=dict(), _last_usage=dict())

Active voice connection for a participant.

state is guarded (RFC §12.1). Leaving ENDED is refused outright — it is the one transition the RFC forbids, and letting a torn-down session go back to ACTIVE resurrects audio paths the framework has already released. Any other move outside the table is logged and allowed: the table does not model every provider's reality (a realtime provider renegotiating goes ACTIVE → CONNECTING), and turning an unmodelled transition into a crash would trade a documentation gap for an outage.

last_usage `property` ¶

last_usage

What the provider recorded last for this session (RFC §12.4.2).

A snapshot, empty until the first report. input_tokens and output_tokens are the two totals a token-billed provider reports; beside them sits whatever breakdown its API sends — per modality, the cached share, reasoning or tool use — under that API's own names, so an absent key means unreported, not zero. A provider billed by session duration reports its seconds here instead, and a hosted backend's tokens ride under their own key.

The next report replaces it and the realtime channel clears it at the end of each turn, so a reader that polls can miss a turn. To bill a call, register :meth:RealtimeVoiceProvider.on_usage instead, which fires on every report. Mutating the returned dict changes nothing.

renegotiate ¶

renegotiate()

Return the session to CONNECTING for a provider renegotiation.

A reconfigure — swapping an agent's personality, voice or tools during a handoff — tears the upstream connection down and builds a new one while the participant's session continues. Nobody hung up. The default provider implements that as disconnect + connect, which leaves the session ENDED in between, and reconnecting from there is the resurrection §12.1 forbids.

This is the only sanctioned way out of ENDED, and it is narrow by design: it says "the framework itself just tore this down to rebuild it". It does not make ENDED non-terminal for anyone else — a participant who really hung up still gets a new session.

VoiceSessionState ¶

Bases: StrEnum

State of a voice session.

AudioChunk `dataclass` ¶

AudioChunk(data, sample_rate=16000, channels=1, format='pcm_s16le', timestamp_ms=None, is_final=False)

A chunk of audio data for streaming (used for outbound TTS).

TranscriptionResult `dataclass` ¶

TranscriptionResult(text, is_final=True, confidence=None, language=None, words=list(), is_speech_start=False, segments=list())

Result from speech-to-text transcription.

is_speech_start `class-attribute` `instance-attribute` ¶

is_speech_start = False

Set by providers with server-side VAD to signal speech detected.

segments `class-attribute` `instance-attribute` ¶

segments = field(default_factory=list)

Who said which part of text, in order (RFC §12.2.3). Empty when the provider attributes nothing; always given on a final by a provider that reports supports_diarization.

speaker `property` ¶

speaker

The label every segment shares, None when there is not exactly one.

SpeakerSegment `dataclass` ¶

SpeakerSegment(speaker, text, start_ms=None, end_ms=None)

One speaker's part of a transcription (RFC §12.2.3).

Attributes:

Name	Type	Description
`speaker`	`str \| None`	The provider's label for the voice, as a string. Opaque and stable within one STT stream only: another stream may give the same voice another label. `None` when the provider could not attribute the words.
`text`	`str`	The words that speaker said.
`start_ms`	`int \| None`	Where the segment starts, from the start of the stream, when the provider says.
`end_ms`	`int \| None`	Where it ends, likewise.

STT (Speech-to-Text)¶

STTProvider ¶

Bases: ABC

Speech-to-text provider.

name `property` ¶

name

Provider name (e.g. 'whisper', 'deepgram').

supports_streaming `property` ¶

supports_streaming

Whether this provider supports streaming transcription.

supports_language_override `property` ¶

supports_language_override

Whether transcribe and transcribe_stream honour language.

A provider that answers False is never handed a per-call language: :class:~roomkit.channels.VoiceChannel keeps calling it with audio only, so an implementation written against the older signature keeps working unchanged.

supports_diarization `property` ¶

supports_diarization

Whether final results say who spoke, in TranscriptionResult.segments.

Labels are stable within one stream only (RFC §12.2.3), so a channel that opens a stream per utterance cannot use them across turns and refuses such a provider.

transcribe `abstractmethod` `async` ¶

transcribe(audio, *, language=None)

Transcribe complete audio to text.

Parameters:

Name	Type	Description	Default
`audio`	`AudioContent \| AudioChunk \| AudioFrame`	Audio content (URL), raw audio chunk, or audio frame.	required
`language`	`str \| None`	Language for this call only, overriding the provider's configuration. Honoured when :attr:`supports_language_override` is true; a caller must not pass it otherwise.	`None`

Returns:

Type	Description
`TranscriptionResult`	TranscriptionResult with text and metadata. `language` on the
`TranscriptionResult`	result is what the provider reports, never an echo of the request.

transcribe_stream `async` ¶

transcribe_stream(audio_stream, *, language=None)

Stream transcription with partial results.

Override for providers that support streaming. Default: buffers all audio and returns single result.

Parameters:

Name	Type	Description	Default
`audio_stream`	`AsyncIterator[AudioChunk]`	Audio chunks; the stream ends when it is exhausted.	required
`language`	`str \| None`	Language for this stream only — see :meth:`transcribe`. A language is fixed for the life of a stream: a change takes effect on the next stream the caller opens.	`None`

warmup `async` ¶

warmup()

Pre-load models so the first call is fast. Override in subclasses.

close `async` ¶

close()

Release resources. Override in subclasses if needed.

MockSTTProvider ¶

MockSTTProvider(transcripts=None, *, streaming=False, languages=None)

Bases: STTProvider

Mock speech-to-text for testing.

Parameters:

Name	Type	Description	Default
`transcripts`	`list[str] \| None`	Texts returned in turn, cycling.	`None`
`streaming`	`bool`	Whether to advertise streaming support.	`False`
`languages`	`list[str \| None] \| None`	The language each result reports, in step with `transcripts` and cycling the same way — what a detecting provider would have heard. `None` entries report nothing.	`None`

languages_requested `instance-attribute` ¶

languages_requested = []

The language handed to each call, in order (None = the default).

STTLanguageLock `dataclass` ¶

STTLanguageLock(detect_language='multi', prefer=dict(), lock_after=1, release_after=2, min_confidence=0.5)

Start detecting, pin the language the speaker uses, let go when it stops fitting.

A streaming STT does better with its language set than in a detecting mode (Deepgram Nova-3 multi), but nobody knows the language before the caller speaks. This policy watches the final results a session produces and tells :class:~roomkit.channels.VoiceChannel which language its next stream should use:

Every session starts in detect_language.
Once lock_after consecutive finals report the same language, the session is locked to it — mapped through prefer first, so a reported "fr" can become "fr-CA".
A locked session counts a miss for every final with no text, a confidence below min_confidence, or a reported language other than the lock. release_after consecutive misses send it back to detect_language; a fitting final resets the count.

The object is shared by the channel and keeps one small state per session id; the channel calls :meth:forget when a session goes away. Everything it does is reachable from a hook with event.language and VoiceChannel.set_stt_language — it is the packaged version of that loop, not the only way to run it.

language_for ¶

language_for(session_id)

The language the session should be using right now.

observe ¶

observe(session_id, result)

Account for a final result and return the language for the next stream.

forget ¶

forget(session_id)

Drop the state kept for a session.

Sherpa-ONNX STT¶

SherpaOnnxSTTProvider ¶

SherpaOnnxSTTProvider(config)

Bases: STTProvider

Speech-to-text provider using sherpa-onnx.

Supports transducer models for both streaming and batch recognition, and Whisper models for batch recognition only.

warmup `async` ¶

warmup()

Pre-load the recognizer model (CUDA init can be slow).

transcribe `async` ¶

transcribe(audio, *, language=None)

Transcribe complete audio.

For transducer mode, uses the OnlineRecognizer (feeds all audio then reads the result). For whisper mode, uses the OfflineRecognizer.

Parameters:

Name	Type	Description	Default
`audio`	`AudioContent \| AudioChunk \| AudioFrame`	Audio content or raw audio chunk (PCM S16LE expected).	required

Returns:

Type	Description
`TranscriptionResult`	TranscriptionResult with text.

transcribe_stream `async` ¶

transcribe_stream(audio_stream, *, language=None)

Stream transcription with partial results using OnlineRecognizer.

Only supported for transducer mode. Whisper mode raises ValueError.

Parameters:

Name	Type	Description	Default
`audio_stream`	`AsyncIterator[AudioChunk]`	Async iterator of audio chunks.	required

Yields:

Type	Description
`AsyncIterator[TranscriptionResult]`	TranscriptionResult with partial and final transcripts.

SherpaOnnxSTTConfig `dataclass` ¶

SherpaOnnxSTTConfig(mode='transducer', tokens='', encoder='', decoder='', joiner='', model_type='', language='en', task='transcribe', sample_rate=16000, num_threads=2, provider='cpu', enable_endpoint_detection=True, rule1_min_trailing_silence=2.4, rule2_min_trailing_silence=1.2, rule3_min_utterance_length=20.0, tail_padding_s=0.66)

Configuration for the sherpa-onnx STT provider.

Attributes:

Name	Type	Description
`mode`	`str`	Recognition mode — `"transducer"` or `"whisper"`.
`tokens`	`str`	Path to `tokens.txt`.
`encoder`	`str`	Path to encoder `.onnx` model.
`decoder`	`str`	Path to decoder `.onnx` model.
`joiner`	`str`	Path to joiner `.onnx` model (transducer only).
`model_type`	`str`	Model type hint for sherpa-onnx (e.g. `"nemo_transducer"` for NeMo TDT/transducer models). When set, the model is treated as offline-only (no streaming).
`language`	`str`	Language code (Whisper only).
`task`	`str`	Whisper task — `"transcribe"` (default) or `"translate"` (translates to English).
`sample_rate`	`int`	Expected audio sample rate.
`num_threads`	`int`	Number of CPU threads for inference.
`provider`	`str`	ONNX execution provider (`"cpu"` or `"cuda"`).
`enable_endpoint_detection`	`bool`	Enable sherpa-onnx endpoint detection. Enabled by default. When VAD drives the stream lifecycle the VAD fires first (its silence threshold is shorter), so this is harmless in a pipeline and useful for standalone use.
`rule1_min_trailing_silence`	`float`	Endpoint rule 1 — minimum trailing silence (seconds) after speech to trigger endpoint.
`rule2_min_trailing_silence`	`float`	Endpoint rule 2 — minimum trailing silence (seconds) after speech with decoded text.
`rule3_min_utterance_length`	`float`	Endpoint rule 3 — minimum utterance length (seconds) to trigger endpoint regardless of silence.
`tail_padding_s`	`float`	Silence (seconds) fed to a streaming transducer before `input_finished()`. The model holds its last frames in its lookahead and decodes them only with audio behind them, so without it the last words are cut ("Hello" -> "Hell"). Must be a finite number >= 0; `0` disables it.

Usage¶

from roomkit.voice.stt.sherpa_onnx import SherpaOnnxSTTProvider, SherpaOnnxSTTConfig

# Transducer model (streaming + batch)
stt = SherpaOnnxSTTProvider(SherpaOnnxSTTConfig(
    mode="transducer",
    tokens="path/to/tokens.txt",
    encoder="path/to/encoder.onnx",
    decoder="path/to/decoder.onnx",
    joiner="path/to/joiner.onnx",
))

# Whisper model (batch only)
stt = SherpaOnnxSTTProvider(SherpaOnnxSTTConfig(
    mode="whisper",
    tokens="path/to/tokens.txt",
    encoder="path/to/encoder.onnx",
    decoder="path/to/decoder.onnx",
    language="en",
))

Install with: pip install roomkit[sherpa-onnx]

Gemini STT (Google, batch)¶

GeminiSTTProvider ¶

GeminiSTTProvider(config)

Bases: STTProvider

Google Gemini speech-to-text provider.

Batch only — :attr:supports_streaming is False, so a :class:~roomkit.channels.voice.VoiceChannel transcribes on SPEECH_END rather than streaming partials. Given seconds of model latency, the honest placement is batch_mode=True (dictation, voicemail) or transcription of a finished recording, not live turn-taking.

Two entry points:

:meth:transcribe — the ABC contract, returning flat text.
:meth:transcribe_recording — the whole :class:Transcript, with speaker turns and timestamps, and the one that accepts a file path.

supports_streaming `property` ¶

supports_streaming

The API takes a complete recording; there is no stream to open.

supports_diarization `property` ¶

supports_diarization

True with speaker_segments (and diarize): results carry the turns.

transcribe `async` ¶

transcribe(audio, *, language=None)

Transcribe a complete recording.

Parameters:

Name	Type	Description	Default
`audio`	`AudioContent \| AudioChunk \| AudioFrame`	Audio content (`data:` URL, local path or Files API URI), raw audio chunk, or audio frame.	required

Returns:

Type	Description
`TranscriptionResult`	TranscriptionResult whose `text` carries the spoken words and
`TranscriptionResult`	whose `language` carries what the model identified. With
`TranscriptionResult`	`speaker_segments` its `segments` carry the speaker turns;
`TranscriptionResult`	meth:`transcribe_recording` returns the full :class:`Transcript`.

transcribe_recording `async` ¶

transcribe_recording(source)

Transcribe a recording into speaker turns.

Parameters:

Name	Type	Description	Default
`source`	`Any`	A path to a recording (`str` or `Path`), an `AudioContent`, an `AudioChunk` or an `AudioFrame`. Files larger than `max_inline_bytes` are uploaded through the Files API and deleted afterwards.	required

Returns:

Name	Type	Description
`The`	`Transcript`	class:`Transcript` — language, one segment per speaker turn,
	`Transcript`	and, from the dedicated recogniser, the timed words.

Raises:

Type	Description
`RuntimeError`	The model answered without a usable transcript.

close `async` ¶

close()

Close the genai client's connection pool and drop the reference.

GeminiSTTConfig `dataclass` ¶

GeminiSTTConfig(api_key, model='gemini-3.8-flash', language=None, diarize=True, prompt=None, timeout=600.0, max_inline_bytes=_MAX_INLINE_BYTES, connect_timeout=5.0, mode='verbatim', custom_vocabulary=list(), word_timestamps=True, speaker_segments=False)

Configuration for the Gemini batch STT provider.

Parameters:

Name	Type	Description	Default
`api_key`	`str`	Gemini API key (`GEMINI_API_KEY`).	required
`model`	`str`	A text/multimodal Gemini model that accepts audio input — see :meth:`~roomkit.providers.gemini.ai.GeminiAIProvider.available_models` for the catalog roomkit keeps — or a dedicated recogniser from :data:`RECOGNISER_MODELS`. The default is the current flash model: transcription is not a reasoning task, flash is the cheapest way to buy the audio context window, and it takes recordings longer than the recogniser's 30 minutes with speakers.	`'gemini-3.8-flash'`
`language`	`str \| None`	Optional BCP-47 hint (e.g. `"fr-CA"`). Left unset, a multimodal model identifies the language itself and reports it on the transcript; the recogniser detects it too but reports nothing, so the transcript's `language` is then empty.	`None`
`diarize`	`bool`	Ask for speaker labels. Worth turning off for a single-speaker recording, where labelling costs tokens and invents distinctions. A conference recorded per participant track needs no diarization at all — transcribe each track and merge on the timestamps. On the recogniser it needs `word_timestamps`, since the labels ride on the words.	`True`
`prompt`	`str \| None`	Extra instruction appended to the transcription request — formatting rules, anything the model should know before it listens. The recogniser takes no prompt: setting one there is refused.	`None`
`timeout`	`float`	Per-request timeout in seconds. Generous by design: a model answering on an hour of audio is not answering in milliseconds.	`600.0`
`max_inline_bytes`	`int`	Recordings larger than this are uploaded through the Files API instead of being inlined in the request.	`_MAX_INLINE_BYTES`
`connect_timeout`	`float`	TCP connect timeout in seconds, apart from `timeout`.	`5.0`
`mode`	`str`	`"verbatim"` (the default) or `"smart"`, the recogniser's cleaned-up transcript: fillers dropped, self-corrections resolved, formatting applied. `"smart"` needs the recogniser and excludes `diarize` and `word_timestamps`.	`'verbatim'`
`custom_vocabulary`	`list[str]`	Terms to spell as written ("RoomKit"), up to 1000. The recogniser takes them natively and then excludes `diarize` and `word_timestamps`; a multimodal model reads them in its prompt.	`list()`
`word_timestamps`	`bool`	Time every word (:attr:`Transcript.words`) and the turns built from them. Recogniser only — a multimodal model always times its turns to the second and never its words. Turning it off lifts the recogniser's limit from 30 minutes to an hour.	`True`
`speaker_segments`	`bool`	Put the speaker turns on :meth:`transcribe`'s result, as `TranscriptionResult.segments` (RFC §12.2.3), and report :attr:`supports_diarization`. Needs `diarize`. Off by default: a diarizing STT is refused by a `VoiceChannel` behind a VAD, where labels would not compare across utterances, and `diarize` has always been on.	`False`

Transcript `dataclass` ¶

Transcript(language, segments, words=list())

A whole recording, as speaker turns.

language `instance-attribute` ¶

language

BCP-47 code of the language spoken. Empty when the model reported none: the dedicated recogniser only knows the language it was told.

words `class-attribute` `instance-attribute` ¶

words = field(default_factory=list)

Word-level timing, when the model returned it (the dedicated recogniser with word_timestamps=True); empty otherwise.

text `property` ¶

text

The turns joined into a readable transcript, one line per speaker.

plain_text `property` ¶

plain_text

The spoken words alone, without speaker labels.

speaker_segments ¶

speaker_segments()

The turns as RoomKit's shared :class:SpeakerSegment (RFC §12.2.3).

Labels lose their "Speaker" word ("Speaker 1" → "1"). With word timing (the recogniser), a turn is a run of one speaker's words timed to 100 ms; otherwise the turns keep the model's MM:SS reading, to the second, or no offsets when it gave none.

TranscriptSegment `dataclass` ¶

TranscriptSegment(speaker, start, end, text)

One speaker turn.

Timestamps are MM:SS strings, as the model returns them. They are the model's reading of the recording, not a forced alignment: treat them as navigation, not as sync marks. They are empty when the model returned no timing at all (the dedicated recogniser with word_timestamps=False).

TranscriptWord `dataclass` ¶

TranscriptWord(text, start, end, speaker=None)

One recognised word with its timing, from the dedicated recogniser.

Offsets are seconds from the start of the recording, at the 100 ms resolution the service reports.

speaker `class-attribute` `instance-attribute` ¶

speaker = None

Label of the turn the word belongs to, or None without diarization.

Usage¶

from roomkit.voice.stt.gemini import GeminiSTTConfig, GeminiSTTProvider

stt = GeminiSTTProvider(GeminiSTTConfig(api_key="your-gemini-api-key"))

transcript = await stt.transcribe_recording("meeting.wav")
for turn in transcript.segments:
    print(f"[{turn.start}-{turn.end}] {turn.speaker}: {turn.text}")

Batch only: the model takes a complete recording and answers in seconds, so this fits meetings, voicemail and imported files rather than live turn-taking. Pass model="gemini-3.5-transcribe" for Google's dedicated recogniser and its word-level timing in transcript.words — see the STT & TTS Providers guide.

Install with: pip install roomkit[gemini]

Meta Muse Voice Transcribe STT (streaming + batch)¶

MetaSTTProvider ¶

MetaSTTProvider(config)

Bases: STTProvider

Speech-to-text on Meta's Muse Voice Transcribe, streaming and batch.

supports_diarization `property` ¶

supports_diarization

True in DIARIZATION mode: every final then names its speaker.

transcribe `async` ¶

transcribe(audio, *, language=None)

Transcribe one finished clip over the REST endpoint.

The clip is sent as a WAV in PUSH_TO_TALK mode — one clip, one transcript — or in DIARIZATION mode when the provider is configured for it, and the service's turns then come back as speaker segments. Meta reports no confidence and no language, so neither is set.

Raises:

Type	Description
`ValueError`	`audio` is an http(s) URL, or a `data:` URI that does not carry a WAV. The endpoint takes an upload, and fetching a caller's URL from here would make RoomKit the client of whatever it names; a `data:` URI needs no fetch.
`MetaSTTError`	The service refused or failed the request.

transcribe_stream `async` ¶

transcribe_stream(audio_stream, *, language=None)

Stream audio to the realtime endpoint and yield its transcripts.

Yields is_speech_start when the model hears speech begin (ENDPOINTING and DIARIZATION), cumulative partials while it listens, and a final per turn, with its speaker in DIARIZATION. The input rate is read off the first chunk: 16 and 24 kHz go through as they are, any other rate is resampled to 24 kHz.

Raises:

Type	Description
`MetaSTTError`	The service refused the stream or closed it abnormally; `retryable` says whether a new stream may work.

close `async` ¶

close()

Release the REST client.

MetaSTTConfig `dataclass` ¶

MetaSTTConfig(api_key, model=DEFAULT_MODEL, mode='ENDPOINTING', keywords=list(), language_bias=list(), base_url='https://api.meta.ai/v1', realtime_url='wss://api.meta.ai/v1/asr/realtime', timeout_s=60.0, handshake_timeout_s=10.0)

Configuration for :class:MetaSTTProvider.

Attributes:

Name	Type	Description
`api_key`	`str`	Meta Model API key (sent as `Bearer`).
`model`	`str`	Transcription model id.
`mode`	`MetaSTTMode`	How a stream ends — see the module docstring. `ENDPOINTING` for a channel without a pipeline VAD, `PUSH_TO_TALK` behind one, `DIARIZATION` to have every final name its speaker.
`keywords`	`list[str]`	Terms to bias recognition toward (product names, people, places). A bias, not a guaranteed spelling.
`language_bias`	`list[str]`	Language names to bias toward, from :data:`SUPPORTED_LANGUAGES` (`["French"]`, not `["fr"]`). A bias, not a pin: the model still transcribes what it hears, which is why the provider reports `supports_language_override = False`.
`base_url`	`str`	REST endpoint root. Override only to point at a proxy.
`realtime_url`	`str`	Realtime WebSocket endpoint.
`timeout_s`	`float`	Timeout of a REST transcription, in seconds.
`handshake_timeout_s`	`float`	How long a stream waits for the service to acknowledge its configuration.

MetaSTTError ¶

MetaSTTError(message, *, code=None, error_type=None, retryable=False, status_code=None)

Bases: ProviderError

A request the Meta Model API refused or failed.

Attributes:

Name	Type	Description
`code`		Meta's `error.code` / `errorCode` (e.g. `"billing_not_configured"`), or the WebSocket close code as a string when the service closed without an error frame.
`error_type`		Meta's `error.type` / `errorType` (e.g. `"billing_error"`), when it sent one.

Usage¶

from roomkit.voice.stt.meta import MetaSTTConfig, MetaSTTProvider

# No pipeline VAD: the model finds where each turn ends
stt = MetaSTTProvider(MetaSTTConfig(api_key="your-meta-model-api-key"))

# Behind a pipeline VAD: one final per utterance
stt = MetaSTTProvider(MetaSTTConfig(api_key="...", mode="PUSH_TO_TALK"))

# Who spoke: every final names its speaker (read the provider directly)
stt = MetaSTTProvider(MetaSTTConfig(api_key="...", mode="DIARIZATION"))
async for result in stt.transcribe_stream(audio):
    if result.is_final:
        print(f"{result.speaker}: {result.text}")

See the STT & TTS Providers guide for the mode choice and the limits.

Install with: pip install roomkit[meta-stt]

Microsoft MAI-Transcribe STT (streaming)¶

AzureMAISTTProvider ¶

AzureMAISTTProvider(config)

Bases: STTProvider

Speech-to-text on Microsoft's MAI-Transcribe, one session per stream.

transcribe `async` ¶

transcribe(audio, *, language=None)

Transcribe one finished clip: a stream of the whole clip, committed once.

The service has no batch endpoint for this model, so the clip goes through the same session a stream opens.

Raises:

Type	Description
`ValueError`	`audio` is an :class:`~roomkit.models.event.AudioContent`; pass the PCM as an `AudioChunk` or an `AudioFrame`.
`AzureMAISTTError`	The service refused or failed the stream.

transcribe_stream `async` ¶

transcribe_stream(audio_stream, *, language=None)

Stream audio to the service, commit it when the stream ends, yield the transcript.

Yields partials while the audio arrives, then one final once the stream ends, unless the service heard no words. The input rate is read off the first chunk: 16 and 24 kHz go through as they are, any other rate is resampled to 16 kHz.

Parameters:

Name	Type	Description	Default
`audio_stream`	`AsyncIterator[AudioChunk]`	The utterance; it ends when exhausted or on a chunk marked `is_final`.	required
`language`	`str \| None`	Language for this stream, overriding the configured one.	`None`

Raises:

Type	Description
`ValueError`	`language` is not one the service supports.
`AzureMAISTTError`	The service refused or failed the stream; `retryable` says whether a new stream may work.

AzureMAISTTConfig `dataclass` ¶

AzureMAISTTConfig(endpoint, api_key, deployment=DEFAULT_DEPLOYMENT, language=None, handshake_timeout_s=10.0, final_timeout_s=10.0)

Configuration for :class:AzureMAISTTProvider.

Attributes:

Name	Type	Description
`endpoint`	`str`	The Foundry resource URL, as the portal shows it (`https://<resource>.services.ai.azure.com`).
`api_key`	`str`	The resource's key (sent as the `api-key` header).
`deployment`	`str`	The name of the model's deployment on the resource, which Foundry sets to the model name unless renamed.
`language`	`str \| None`	Language to transcribe, as a BCP-47 tag or a code from :data:`SUPPORTED_LANGUAGES` (`"fr-CA"` is sent as `"fr"`). `None` lets the service detect it, utterance by utterance.
`handshake_timeout_s`	`float`	How long a stream waits for the service to open the session and accept its configuration.
`final_timeout_s`	`float`	How long a stream waits, once its audio is committed, for the final transcript.

AzureMAISTTError ¶

AzureMAISTTError(message, *, code=None, error_type=None, retryable=False, status_code=None)

Bases: ProviderError

A stream the MAI-Transcribe service refused or failed.

Attributes:

Name	Type	Description
`code`		The service's `error.code`, or the WebSocket close code as a string when it closed without an error event.
`error_type`		The service's `error.type` (e.g. `"invalid_request_error"`), when it sent one.

Usage¶

from roomkit.voice.stt.azure_mai import AzureMAISTTConfig, AzureMAISTTProvider

# Behind a pipeline VAD: one final per utterance
stt = AzureMAISTTProvider(
    AzureMAISTTConfig(endpoint="https://<resource>.services.ai.azure.com", api_key="...")
)

See the STT & TTS Providers guide for why it needs a pipeline VAD and the limits.

Install with: pip install roomkit[azure-speech]

TTS (Text-to-Speech)¶

TTSProvider ¶

Bases: ABC

Text-to-speech provider.

name `property` ¶

name

Provider name (e.g. 'elevenlabs', 'openai').

default_voice `property` ¶

default_voice

Default voice ID. Override in subclasses.

max_dialogue_speakers `property` ¶

max_dialogue_speakers

Distinct speakers one :meth:synthesize_dialogue call can voice.

0, the default, means no dialogue at all (RFC §12.2).

supports_streaming_input `property` ¶

supports_streaming_input

Whether this TTS accepts streaming text input.

context_level `property` ¶

context_level

What conversation context this TTS consumes (RFC §12.2.2).

A provider left at NONE never receives a context. Override to receive the dialogue of the session on each streaming call.

available_voices `classmethod` ¶

available_voices()

Curated, offline catalog of the voices this provider offers.

No credentials or network: call it on the class. The base returns an empty list; a provider with a fixed or curated set overrides it.

list_voices `async` ¶

list_voices(*, language=None, gender=None, query=None)

Voices the vendor reports, custom voices the caller owns included.

Every provider applies the filters the same way, server side or on the results (:func:~roomkit.voice.voices.filter_voices). The base filters :meth:available_voices, for a vendor with no voices endpoint.

Parameters:

Name	Type	Description	Default
`language`	`str \| None`	BCP-47 tag or prefix (`"fr"` matches `fr-CA`).	`None`
`gender`	`str \| None`	`"male"`, `"female"` or `"neutral"`.	`None`
`query`	`str \| None`	Case-insensitive text found in the name or description.	`None`

synthesize_dialogue `async` ¶

synthesize_dialogue(turns, voices)

Voice a scripted exchange in one clip (RFC §12.2).

Parameters:

Name	Type	Description	Default
`turns`	`Sequence[DialogueTurn]`	The lines, in order, each naming its speaker.	required
`voices`	`Mapping[str, str]`	Voice id per speaker.	required

Returns:

Type	Description
`AudioContent`	AudioContent whose `transcript` holds each turn as
`AudioContent`	`"<speaker>: <text>"` on its own line.

Raises:

Type	Description
`NotImplementedError`	The provider voices no dialogue (:attr:`max_dialogue_speakers` is 0).
`ValueError`	A turn names an unmapped speaker, or more speakers than :attr:`max_dialogue_speakers`.

synthesize `abstractmethod` `async` ¶

synthesize(text, *, voice=None)

Synthesize text to audio.

Parameters:

Name	Type	Description	Default
`text`	`str`	Text to synthesize.	required
`voice`	`str \| None`	Voice ID (uses default_voice if not specified).	`None`

Returns:

Type	Description
`AudioContent`	AudioContent with URL to generated audio.

release_context ¶

release_context(context_id)

Drop any state held for context_id (its voice session ended).

Called by the Voice Channel when the session is unbound. A provider that keeps per-context state (a dialogue KV cache, request handles) overrides it; an unknown context is a no-op.

synthesize_stream_input `async` ¶

synthesize_stream_input(text_stream, *, voice=None, context=None)

Stream audio from streaming text chunks.

Override for providers that accept an async text stream as input. context is the session's dialogue so far, passed only when context_level is not NONE.

synthesize_stream `async` ¶

synthesize_stream(text, *, voice=None, context=None)

Stream audio chunks as they're generated.

Override for providers that support streaming. context is the session's dialogue so far, passed only when context_level is not NONE. Default: synthesizes full audio and yields single chunk.

warmup `async` ¶

warmup()

Pre-load models so the first call is fast. Override in subclasses.

close `async` ¶

close()

Release resources. Override in subclasses if needed.

MockTTSProvider ¶

MockTTSProvider(voice='mock-voice', *, context_level=NONE)

Bases: TTSProvider

Mock text-to-speech for testing.

context_level makes it declare a conversation context level; the contexts it receives land in contexts (one entry per streaming call) and the released ones in released.

Voices and Dialogue¶

VoiceInfo ¶

Bases: BaseModel

Metadata describing a single voice a provider offers.

Both the curated catalog (available_voices) and the live query (list_voices) return these. Only id is guaranteed; the other fields are what the vendor reports, and are None when it reports nothing — never inferred from another voice (RFC §12.2).

Attributes:

Name	Type	Description
`id`	`str`	Exact voice identifier passed as `voice` (e.g. `"alloy"`, `"Puck"`, `"fr-ca-advisor-1"`, an ElevenLabs `voice_id`).
`name`	`str \| None`	Human-friendly display name.
`language`	`str \| None`	BCP-47 tag if voice-specific (e.g. `"en-US"`), or `"multilingual"`, else `None`.
`gender`	`str \| None`	`"male"`/`"female"`/`"neutral"` if known.
`accent`	`str \| None`	The accent the vendor names (e.g. `"Montreal French"`).
`description`	`str \| None`	Short characterization (e.g. `"Upbeat"`) if known.
`deprecated`	`bool`	Whether the provider marks the voice deprecated.
`attributes`	`dict[str, str]`	What the vendor reports beyond the common fields, under its own names (persona, age, use case, category).

DialogueTurn ¶

Bases: BaseModel

One line of a scripted exchange for synthesize_dialogue (RFC §12.2).

Attributes:

Name	Type	Description
`speaker`	`str`	Key into the call's `voices` map.
`text`	`str`	What the speaker says.
`style`	`str \| None`	Delivery direction for this turn only (e.g. `"whispering"`).

filter_voices ¶

filter_voices(voices, *, language=None, gender=None, query=None)

Keep the voices every given filter matches, the way RFC §12.2 defines them.

language matches a tag equal to it or starting with it and a hyphen ("fr" matches fr-CA; fr-CA does not match fr-FR); gender matches exactly; query is a case-insensitive substring of the name or the description. A voice with no language or gender does not match a filter on it.

Custom Voices¶

VoiceLibrary ¶

Bases: ABC

Designs, replicates and deletes custom voices (RFC §12.2.4).

Replicating a person's voice takes their recorded consent as a required argument, and the recordings are handed to the vendor for the call only: nothing of them is kept or logged (RFC §17.6).

supports_design `property` ¶

supports_design

Whether :meth:design_voice is offered.

supports_replication `property` ¶

supports_replication

Whether :meth:replicate_voice is offered.

design_voice `async` ¶

design_voice(description, *, store=True, name=None)

A voice written from a natural-language description.

Parameters:

Name	Type	Description	Default
`description`	`str`	Who speaks and how (e.g. `"a warm Montreal narrator in her forties, unhurried, a smile in the voice"`).	required
`store`	`bool`	Ask the vendor to keep the voice for the account.	`True`
`name`	`str \| None`	Display name for a stored voice.	`None`

Raises:

Type	Description
`NotImplementedError`	The library does not design voices.

replicate_voice `async` ¶

replicate_voice(sample, consent, *, store=True, name=None)

A person's voice, from a recording of it and their recorded consent.

Parameters:

Name	Type	Description	Default
`sample`	`VoiceAudio`	The recording of the voice to replicate.	required
`consent`	`VoiceAudio`	A recording, by the same person, agreeing to it.	required
`store`	`bool`	Ask the vendor to keep the voice for the account.	`True`
`name`	`str \| None`	Display name for a stored voice.	`None`

Raises:

Type	Description
`NotImplementedError`	The library does not replicate voices.
`VoiceConsentError`	The vendor refused the consent.

get_voice `abstractmethod` `async` ¶

get_voice(voice_id)

The stored voice voice_id, or None if the vendor holds none.

delete_voice `abstractmethod` `async` ¶

delete_voice(voice_id)

Delete a stored voice; an id the vendor does not hold is not an error.

close `async` ¶

close()

Release resources. Override in subclasses if needed.

CustomVoice ¶

Bases: BaseModel

A voice a :class:VoiceLibrary made.

Attributes:

Name	Type	Description
`voice`	`VoiceInfo`	The voice; `voice.id` is what the vendor's TTS accepts.
`stored`	`bool`	Held by the vendor for the caller's account, listed by its TTS and readable with :meth:`VoiceLibrary.get_voice`. An unstored voice's id is the only handle on it.
`expires_at`	`datetime \| None`	When the vendor stops honouring the id, if it says.
`sample`	`AudioContent \| None`	The vendor's preview of the voice, if it sends one.

VoiceConsentError ¶

Bases: Exception

The vendor refused a replication because the consent did not hold.

Raised in place of the vendor's own error so a caller can tell "record the consent again" from any other failure; no voice exists afterwards.

Conversation Context¶

TTSContextLevel ¶

Bases: StrEnum

What conversation context a TTS provider consumes.

Each level includes the ones above it.

NONE `class-attribute` `instance-attribute` ¶

NONE = 'none'

Nothing: every call is a sentence in isolation (the default).

SELF `class-attribute` `instance-attribute` ¶

SELF = 'self'

Which of its own previous turns were played, and how far.

TEXT `class-attribute` `instance-attribute` ¶

TEXT = 'text'

The text of every turn, both roles.

AUDIO `class-attribute` `instance-attribute` ¶

AUDIO = 'audio'

The text and the audio of every turn.

TTSContext `dataclass` ¶

TTSContext(context_id, turns, next_turn_id)

The snapshot of a session's dialogue passed to one synthesis call.

context_id `instance-attribute` ¶

context_id

The voice session the context belongs to.

turns `instance-attribute` ¶

turns

Oldest first.

next_turn_id `instance-attribute` ¶

next_turn_id

The turn_id the assistant turn of this call will get, if it is played.

ConversationTurn `dataclass` ¶

ConversationTurn(turn_id, role, participant_id, text, audio=None, played_ms=None, interrupted=False)

One turn of the dialogue heard on a voice session.

text `instance-attribute` ¶

text

As synthesized, or as transcribed after ON_TRANSCRIPTION.

audio `class-attribute` `instance-attribute` ¶

audio = None

The whole turn, only when the provider's level is AUDIO and audio is kept.

played_ms `class-attribute` `instance-attribute` ¶

played_ms = None

Assistant turns: how much of the audio was actually played.

interrupted `class-attribute` `instance-attribute` ¶

interrupted = False

Assistant turns: playback was cancelled before the end.

TTSContextConfig `dataclass` ¶

TTSContextConfig(enabled=True, include_audio=False, max_turns=20, max_audio_seconds=120.0)

How a Voice Channel keeps the TTS conversation context.

Attributes:

Name	Type	Description
`enabled`	`bool`	Keep a context at all (only for a provider whose `context_level` is not NONE).
`include_audio`	`bool`	Keep the audio of each turn. Off by default: audio is a copy of the user's voice, kept only when asked for.
`max_turns`	`int`	Oldest turns are dropped past this.
`max_audio_seconds`	`float`	Oldest audio is dropped past this; its text stays.

Vui Nano TTS¶

VuiTTSProvider ¶

VuiTTSProvider(config=None)

Bases: TTSProvider

Vui Nano: each reply is generated inside the conversation it answers.

Call :meth:close when done: it closes Vui on the provider's own thread and stops that thread. A provider dropped without it leaves torch's inference mode on in whichever thread collects it.

release_context ¶

release_context(context_id)

Empty the cache of context_id's dialogue, on the Vui thread, after any reply.

warmup `async` ¶

warmup()

Load the model, the codec and every voice prompt.

synthesize_stream `async` ¶

synthesize_stream(text, *, voice=None, context=None)

Speak text as the next turn of context's conversation.

synthesize `async` ¶

synthesize(text, *, voice=None)

Synthesize text on its own (no conversation) as a WAV data URL.

The provider has one cache: this empties it, and the conversation it held restarts from the prompt at its next call.

VuiTTSConfig `dataclass` ¶

VuiTTSConfig(voices=(lambda: {'maeve': VuiVoice('maeve')})(), checkpoint='vui-nano-1.1', temperature=0.7, max_secs=60.0)

Configuration for :class:VuiTTSProvider.

Attributes:

Name	Type	Description
`voices`	`dict[str, VuiVoice]`	Named voices; the first one is the default.
`checkpoint`	`str`	Vui checkpoint name or local path.
`temperature`	`float`	Sampling temperature.
`max_secs`	`float`	Longest reply, in seconds; a longer text is cut off there, with a warning. A minute holds most spoken replies, a short poem included.

VuiVoice `dataclass` ¶

VuiVoice(preset=None, ref_audio=None, ref_text=None)

A voice prompt: a preset from the Hub, or a clip and its transcript.

Attributes:

Name	Type	Description
`preset`	`str \| None`	One of `maeve`, `abraham`, `rhian`, `harry`.
`ref_audio`	`str \| None`	Path to a clean speech clip (under ~15 s) to clone.
`ref_text`	`str \| None`	Exact transcript of `ref_audio`.

Install with: pip install roomkit[vui] (Python 3.12, CUDA GPU).

Fluxions TTS (Hosted Vui)¶

FluxionsTTSProvider ¶

FluxionsTTSProvider(config)

Bases: TTSProvider

Vui hosted by fluxions.ai: each text rendered on its own, streamed.

warmup `async` ¶

warmup()

Fetch the voice lists and resolve the configured voice.

It renders nothing: after an idle period, the first render can still wait for Fluxions to start a worker (up to about 30 s).

list_voices `async` ¶

list_voices(*, language=None, gender=None, query=None)

The voices Fluxions hosts, then the account's cloned voices.

Each id is one voice accepts: a hosted voice's short id, a cloned voice's id.

synthesize_stream `async` ¶

synthesize_stream(text, *, voice=None, context=None)

Stream text as 24 kHz PCM; context is ignored (no context level).

synthesize `async` ¶

synthesize(text, *, voice=None)

Render text whole, as a WAV data URL.

close `async` ¶

close()

Release the HTTP client.

FluxionsTTSConfig `dataclass` ¶

FluxionsTTSConfig(api_key, voice='maeve', temperature=None, max_secs=None, verify_chunks=False, base_url='https://api.fluxions.ai', timeout=60.0, connect_timeout=5.0)

Configuration for :class:FluxionsTTSProvider.

Attributes:

Name	Type	Description
`api_key`	`str`	Fluxions API key.
`voice`	`str`	A voice's short id (`maeve`) or its full id (`maeve.h…`, `u-…` for a cloned voice). A short id follows the model Fluxions currently serves.
`temperature`	`float \| None`	Sampling temperature; `None` keeps the server's default.
`max_secs`	`float \| None`	Longest render, in seconds; `None` lets the server size it from the text.
`verify_chunks`	`bool`	Let the server check each rendered chunk, and render it again when it fails (slower first audio).
`base_url`	`str`	API root.
`timeout`	`float`	Read budget, in seconds: a cold start takes up to about 30 s.
`connect_timeout`	`float`	TCP connect budget, in seconds, apart from `timeout`.

Install with: pip install roomkit[fluxions].

Azure Speech TTS (MAI-Voice)¶

AzureSpeechTTSProvider ¶

AzureSpeechTTSProvider(config)

Bases: TTSProvider

Azure Speech voices, MAI-Voice included: each text rendered on its own, streamed.

synthesize_stream `async` ¶

synthesize_stream(text, *, voice=None, context=None)

Stream text as PCM at the configured rate; context is ignored.

A text with nothing to speak makes no request.

Raises:

Type	Description
`ValueError`	voice names no locale and no language is configured.
`AzureSpeechTTSError`	Azure Speech refused or failed the render.

synthesize `async` ¶

synthesize(text, *, voice=None)

Render text whole, as a WAV data URL.

close `async` ¶

close()

Release the HTTP client.

AzureSpeechTTSConfig `dataclass` ¶

AzureSpeechTTSConfig(api_key, region=None, endpoint=None, voice=DEFAULT_VOICE, language=None, style=None, rate=None, sample_rate=24000, timeout=30.0, connect_timeout=5.0)

Configuration for :class:AzureSpeechTTSProvider.

Attributes:

Name	Type	Description
`api_key`	`str`	The Speech (or Foundry) resource's key.
`region`	`str \| None`	The resource's region identifier (`swedencentral`, `canadacentral`…). Exactly one of `region` and `endpoint`.
`endpoint`	`str \| None`	The full synthesis URL, for a custom domain or a proxy, in place of `region`.
`voice`	`str`	Voice name as Azure spells it: `<locale>-<name>:<model>` for a MAI voice, `<locale>-<name>Neural` for a neural one.
`language`	`str \| None`	The SSML `xml:lang`. `None` takes the locale the voice name starts with; a voice name without one needs it set.
`style`	`str \| None`	A speaking style the voice supports (`joyful`, `customer_call_center`…), wrapped around every text with `mstts:express-as`. `None` speaks in the voice's default style.
`rate`	`str \| None`	Speaking rate, as SSML `prosody` takes it: a relative change (`"+20%"`), a multiplier (`"1.2"`) or a keyword (`"fast"`). `None` speaks at the voice's own pace. MAI-Voice-2.1-Flash follows it closely (measured 2026-10-08); MAI-Voice-2.1 less so.
`sample_rate`	`int`	Output rate, one of :data:`OUTPUT_FORMATS`.
`timeout`	`float`	Read budget of a render, in seconds.
`connect_timeout`	`float`	TCP connect budget, in seconds, apart from `timeout`.

AzureSpeechTTSError ¶

AzureSpeechTTSError(message, *, status_code)

Bases: ProviderError

A render Azure Speech refused or failed.

Install with: pip install roomkit[azure-speech].

Pocket TTS¶

PocketTTSProvider ¶

PocketTTSProvider(config=None)

Bases: TTSProvider

Kyutai Pocket TTS, streaming from a model held in this process.

warmup `async` ¶

warmup()

Load the model and every voice state.

synthesize_stream `async` ¶

synthesize_stream(text, *, voice=None, context=None)

Stream text as 16-bit PCM chunks at 24 kHz, then a final marker.

synthesize `async` ¶

synthesize(text, *, voice=None)

Synthesize text as a WAV data URL.

PocketTTSConfig `dataclass` ¶

PocketTTSConfig(language='english', voices=(lambda: {'alba': 'alba'})(), device='cpu', quantize=False, temperature=None)

Configuration for :class:PocketTTSProvider.

Attributes:

Name	Type	Description
`language`	`str`	Pocket TTS model to load: `english`, `french`, `german`, `portuguese`, `italian`, `spanish`, `dutch` (pocket-tts 3.3+), each but `english` also as a larger `_24l` variant (`french_24l`) that sounds better and is about three times slower.
`voices`	`dict[str, str]`	Named voices; the first one is the default. Each value is a pre-made voice name (`alba`, `estelle`...), a local audio clip to clone, an `hf://` path, or a `.safetensors` voice state exported by `pocket-tts export-voice --language <same language>` (the fastest to load).
`device`	`str`	`"cpu"` or a CUDA device (`"cuda"`, `"cuda:1"`).
`quantize`	`bool`	int8 dynamic quantization, CPU only.
`temperature`	`float \| None`	Sampling temperature; `None` keeps the model's default.

Install with: pip install roomkit[pocket-tts] (CPU or CUDA GPU).

Sherpa-ONNX TTS¶

SherpaOnnxTTSProvider ¶

SherpaOnnxTTSProvider(config)

Bases: TTSProvider

Text-to-speech provider using sherpa-onnx with VITS/Piper or Kokoro models.

warmup `async` ¶

warmup()

Pre-load the TTS model (CUDA init can be slow).

synthesize `async` ¶

synthesize(text, *, voice=None)

Synthesize text to audio.

Parameters:

Name	Type	Description	Default
`text`	`str`	Text to synthesize.	required
`voice`	`str \| None`	Speaker ID as string (uses default if not specified).	`None`

Returns:

Type	Description
`AudioContent`	AudioContent with a `data:` URL containing WAV audio.

synthesize_stream `async` ¶

synthesize_stream(text, *, voice=None, context=None)

Stream audio chunks using a callback bridge.

Parameters:

Name	Type	Description	Default
`text`	`str`	Text to synthesize.	required
`voice`	`str \| None`	Speaker ID as string (uses default if not specified).	`None`

Yields:

Type	Description
`AsyncIterator[AudioChunk]`	AudioChunk with PCM S16LE audio data.

Closing the stream early (a barge-in) stops the synthesis at the sentence under way; a synthesis error is raised to the reader.

SherpaOnnxTTSConfig `dataclass` ¶

SherpaOnnxTTSConfig(model='', tokens='', data_dir='', lexicon='', speaker_id=0, speed=1.0, sample_rate=22050, num_threads=2, provider='cpu', model_type='vits', voices='', lang='')

Configuration for the sherpa-onnx TTS provider.

Attributes:

Name	Type	Description
`model`	`str`	Path to the `.onnx` model.
`tokens`	`str`	Path to `tokens.txt`.
`data_dir`	`str`	Path to espeak-ng data directory (Piper and Kokoro models).
`lexicon`	`str`	Path to optional lexicon file. A multi-lang Kokoro takes several, comma-separated (`lexicon-us-en.txt,lexicon-zh.txt`).
`speaker_id`	`int`	Speaker ID for multi-speaker models; for Kokoro, the index of a voice in `voices.bin` (`ff_siwis` is 30 in v1.0).
`speed`	`float`	Speech speed multiplier (1.0 = normal).
`sample_rate`	`int`	The rate a caller sets its output up for (Kokoro: 24000). Audio always carries the model's own rate.
`num_threads`	`int`	Number of CPU threads for inference.
`provider`	`str`	ONNX execution provider (`"cpu"` or `"cuda"`).
`model_type`	`str`	`"vits"` (VITS/Piper) or `"kokoro"`.
`voices`	`str`	Path to Kokoro's `voices.bin`, required with `"kokoro"`.
`lang`	`str`	Language espeak-ng reads the text in for a multi-lang Kokoro (`"fr"`, `"es"`, `"it"`...). Empty, any text that is not Chinese is read as English: French comes out mispronounced.

Usage¶

from roomkit.voice.tts.sherpa_onnx import SherpaOnnxTTSProvider, SherpaOnnxTTSConfig

# VITS/Piper model with multi-speaker support
tts = SherpaOnnxTTSProvider(SherpaOnnxTTSConfig(
    model="path/to/vits-model.onnx",
    tokens="path/to/tokens.txt",
    data_dir="path/to/espeak-ng-data",  # for Piper models
    speaker_id=0,
    speed=1.0,
))

Install with: pip install roomkit[sherpa-onnx]

Grok TTS (xAI)¶

GrokTTSProvider ¶

GrokTTSProvider(config)

Bases: TTSProvider

xAI Grok text-to-speech provider with WebSocket streaming support.

Supports: * synthesize() — REST endpoint, returns full audio. * synthesize_stream() — REST with chunked reading. * synthesize_stream_input() — bidirectional WebSocket for real-time text-to-speech (send text deltas, receive audio deltas).

synthesize `async` ¶

synthesize(text, *, voice=None)

Synthesize text to audio via the REST endpoint.

Parameters:

Name	Type	Description	Default
`text`	`str`	Text to synthesize (max 15 000 characters).	required
`voice`	`str \| None`	Voice ID override.	`None`

Returns:

Type	Description
`AudioContent`	AudioContent with a data-URL of the generated audio.

synthesize_stream `async` ¶

synthesize_stream(text, *, voice=None, context=None)

Stream audio chunks from the REST endpoint.

The xAI API returns the audio as a single response, so we chunk it ourselves for pipeline compatibility.

synthesize_stream_input `async` ¶

synthesize_stream_input(text_stream, *, voice=None, context=None)

Stream audio from streaming text input via WebSocket.

Protocol (xAI): * Client → {"type": "text.delta", "delta": "..."} * Client → {"type": "text.done"} * Server → {"type": "audio.delta", "delta": "<base64>"} * Server → {"type": "audio.done", "trace_id": "..."}

The connection stays open after audio.done allowing multiple turns, but we close after the first turn completes.

Parameters:

Name	Type	Description	Default
`text_stream`	`AsyncIterator[str]`	Async iterator yielding text chunks.	required
`voice`	`str \| None`	Voice ID override.	`None`

Yields:

Type	Description
`AsyncIterator[AudioChunk]`	AudioChunk with decoded audio data.

close `async` ¶

close()

Release the HTTP client.

GrokTTSConfig `dataclass` ¶

GrokTTSConfig(api_key, voice_id='eve', language='en', codec='pcm', sample_rate=24000, bit_rate=128000, base_url='https://api.x.ai/v1', ws_url='wss://api.x.ai/v1/tts', timeout=60.0, connect_timeout=5.0)

Configuration for xAI Grok TTS provider.

Parameters:

Name	Type	Description	Default
`api_key`	`str`	xAI API key (or set `XAI_API_KEY` env var).	required
`voice_id`	`str`	One of `eve`, `ara`, `rex`, `sal`, `leo`.	`'eve'`
`language`	`str`	BCP-47 language code or `auto`.	`'en'`
`codec`	`str`	Output codec — `pcm`, `wav`, `mp3`, `mulaw`, `alaw`. `wav` applies to `synthesize()`; a streamed request asks for `pcm` instead. A VoiceChannel plays `pcm` only.	`'pcm'`
`sample_rate`	`int`	Output sample rate in Hz.	`24000`
`bit_rate`	`int`	MP3 bit rate (only used when codec is `mp3`).	`128000`
`base_url`	`str`	Override the REST API base URL.	`'https://api.x.ai/v1'`
`ws_url`	`str`	Override the WebSocket streaming URL.	`'wss://api.x.ai/v1/tts'`
`timeout`	`float`	HTTP request timeout in seconds.	`60.0`
`connect_timeout`	`float`	TCP connect timeout in seconds, apart from `timeout`.	`5.0`

Usage¶

from roomkit.voice.tts.grok import GrokTTSProvider, GrokTTSConfig

tts = GrokTTSProvider(GrokTTSConfig(
    api_key="your-xai-api-key",
    voice_id="eve",
    codec="pcm",
    sample_rate=24000,
))

Install with: pip install httpx websockets

Gemini TTS (Google)¶

GeminiTTSProvider ¶

GeminiTTSProvider(config)

Bases: TTSProvider

Google Gemini text-to-speech provider.

Supports:

:meth:synthesize — one request, full clip as WAV.
:meth:synthesize_stream — audio deltas forwarded as they arrive.

Streaming text input is not supported: the API takes a complete prompt, so there is no seam to feed token deltas into. A voice channel therefore delivers through :meth:synthesize_stream on the complete reply.

max_dialogue_speakers `property` ¶

max_dialogue_speakers

Two from 3.8 on; none on the 3.1 and 2.5 models, whose prompt-only contract this provider does not extend to several speakers.

available_voices `classmethod` ¶

available_voices()

The 30 prebuilt voices, shared with Gemini Live native audio.

list_voices `async` ¶

list_voices(*, language=None, gender=None, query=None)

Google's voice catalog, the account's custom voices first.

The whole catalog is read — 2,089 voices in three pages, 0.4 s on 2026-09-27 — and filtered here: the service's own filters mean something else (its language_code wants an exact tag, fr finds nothing, and its search for Kore answers Korean voices), and RFC §12.2 wants every provider to filter alike. Any id returned is accepted as voice.

synthesize_dialogue `async` ¶

synthesize_dialogue(turns, voices)

Voice a scripted exchange of up to two speakers in one clip.

Each turn goes out as its own text item, its speaker (and its style, when set) riding as speech_metadata, and each speaker is bound to its voice in speech_config. style_prompt does not apply: a dialogue directs each turn on its own.

Raises:

Type	Description
`NotImplementedError`	The configured model voices no dialogue.
`ValueError`	More than two speakers, or a speaker `voices` does not map (both before any request).
`RuntimeError`	The interaction completed without audio.

synthesize `async` ¶

synthesize(text, *, voice=None)

Synthesize the whole text in one request.

Parameters:

Name	Type	Description	Default
`text`	`str`	Text to speak. Must not be blank.	required
`voice`	`str \| None`	Prebuilt voice name overriding the configured one.	`None`

Returns:

Type	Description
`AudioContent`	AudioContent holding a WAV `data:` URL. A WAV file the service
`AudioContent`	answered (3.8 on) is passed through unchanged, its C2PA
`AudioContent`	content-credentials chunk included; bare PCM is wrapped in one.

Raises:

Type	Description
`ValueError`	text is empty or whitespace.
`RuntimeError`	The interaction completed without audio, or with audio that does not decode.

synthesize_stream `async` ¶

synthesize_stream(text, *, voice=None, context=None)

Stream audio deltas as the service emits them.

Blank text yields nothing but the terminating chunk — TTS filters can strip a reply down to whitespace, and that is not worth a round trip the service would reject.

close `async` ¶

close()

Close the genai client's connection pool and drop the reference.

GeminiTTSConfig `dataclass` ¶

GeminiTTSConfig(api_key, model='gemini-3.8-flash-tts', voice='Kore', language=None, style_prompt=None, timeout=120.0, connect_timeout=5.0)

Configuration for the Gemini TTS provider.

Parameters:

Name	Type	Description	Default
`api_key`	`str`	Gemini API key (`GEMINI_API_KEY`).	required
`model`	`str`	One of :data:`GEMINI_TTS_MODELS`. The default is Google's replacement for `gemini-3.1-flash-tts-preview`; `gemini-3.8-flash-lite-tts` is the lower-latency, higher-throughput variant. Both stream incrementally, which is what lets playback start before the whole clip is generated.	`'gemini-3.8-flash-tts'`
`voice`	`str`	Prebuilt voice name — see :meth:`GeminiTTSProvider.available_voices`.	`'Kore'`
`language`	`str \| None`	Optional BCP-47 hint (e.g. `"fr-CA"`). Left unset, the model infers the language from the text.	`None`
`style_prompt`	`str \| None`	Natural-language direction for the whole utterance (e.g. `"calm and reassuring"`). From 3.8 on it is sent as the `style` of the text's `speech_metadata`, a field the model takes as direction, never as words. The 3.1 and 2.5 models have no such field, so there it is written as a labelled `Delivery direction:` line above the `<transcript>` block in the same `input` string, which is what keeps the model reciting the transcript instead of the direction; the 3.1 preview can occasionally read it aloud anyway. For cues that steer a word or a phrase rather than the whole utterance, put audio tags inline in the text itself. Google documents `<laugh>`, `<sigh>` and `<short pause>` for 3.8 and `[laughs]`, `[whispers]` for 3.1; the 3.8 models perform both spellings (measured 2026-09-27).	`None`
`timeout`	`float`	Per-request timeout in seconds. Generous by design: measured time-to-first-audio for a one-sentence prompt is ~1.1 to ~1.6 s on the default model and ~0.7 s on the Lite one (2026-09-27), was up to ~8 s on 3.1, and long text is slower still.	`120.0`
`connect_timeout`	`float`	TCP connect timeout in seconds, apart from `timeout`.	`5.0`

GeminiVoiceLibrary ¶

GeminiVoiceLibrary(config)

Bases: VoiceLibrary

Custom voices on the Gemini API's voice store (RFC §12.2.4).

design_voice `async` ¶

design_voice(description, *, store=True, name=None)

A voice written from description, stored for the account.

Raises:

Type	Description
`ValueError`	A blank description, or `store=False`: Google stores every designed voice, and returning a stored one to a caller who asked for none would leave a voice they do not know of.

replicate_voice `async` ¶

replicate_voice(sample, consent, *, store=True, name=None)

A person's voice from sample, with their recorded consent.

Both recordings are 16-bit, 24 kHz mono WAV. Google checks that the consent is spoken by the person in the sample and refuses the voice otherwise; that refusal is raised, and no voice is returned.

Raises:

Type	Description
`ValueError`	A recording that is not 16-bit 24 kHz mono WAV.

close `async` ¶

close()

Close the genai client's connection pool and drop the reference.

GeminiVoiceLibraryConfig `dataclass` ¶

GeminiVoiceLibraryConfig(api_key, timeout=120.0, connect_timeout=5.0)

Configuration for :class:GeminiVoiceLibrary.

Parameters:

Name	Type	Description	Default
`api_key`	`str`	Gemini API key (`GEMINI_API_KEY`).	required
`timeout`	`float`	Per-request timeout in seconds. Designing a voice took about 15 s on 2026-09-27.	`120.0`
`connect_timeout`	`float`	TCP connect timeout in seconds, apart from `timeout`.	`5.0`

Usage¶

from roomkit.voice.tts.gemini import GeminiTTSConfig, GeminiTTSProvider

tts = GeminiTTSProvider(GeminiTTSConfig(
    api_key="your-gemini-api-key",
    model="gemini-3.8-flash-tts",
    voice="Kore",
    style_prompt="calm and clear",                 # delivery guidance
))

list_voices() reads Google's catalog of about 2,000 voices, synthesize_dialogue() voices two speakers in one clip, and GeminiVoiceLibrary designs and replicates custom voices.

Time to first audio is measured in seconds, so this fits prompts and generated audio messages rather than live turn-taking — see the STT & TTS Providers guide.

Install with: pip install roomkit[gemini]

RTP Backend¶

RTPVoiceBackend ¶

RTPVoiceBackend(*, local_addr=('0.0.0.0', 0), remote_addr=None, payload_type=0, clock_rate=8000, dtmf_payload_type=101, rtcp_interval=5.0, port_allocator=None)

Bases: VoiceBackend

VoiceBackend that sends and receives audio over RTP.

Each :meth:connect call creates a new aiortp.RTPSession bound to the configured local address and sending to the remote address.

Inbound audio is decoded by aiortp (G.711, L16, Opus) and delivered as AudioFrame objects via the on_audio_received callback. DTMF digits are received out-of-band via RFC 4733 and delivered via on_dtmf_received callbacks.

Parameters:

Name	Type	Description	Default
`local_addr`	`tuple[str, int]`	`(host, port)` to bind RTP. Use port `0` for OS-assigned.	`('0.0.0.0', 0)`
`remote_addr`	`tuple[str, int] \| None`	`(host, port)` to send RTP to. May be `None` if supplied per-session via `metadata["remote_addr"]` in :meth:`connect`.	`None`
`payload_type`	`int`	RTP payload type number (default `0` = PCMU).	`0`
`clock_rate`	`int`	Clock rate in Hz (default `8000` for G.711).	`8000`
`dtmf_payload_type`	`int`	RTP payload type for RFC 4733 DTMF events (default `101`).	`101`
`rtcp_interval`	`float`	Seconds between RTCP sender reports.	`5.0`

send_audio_sync ¶

send_audio_sync(session, chunk)

Synchronously send a single audio chunk via RTP.

Used by the audio bridge for low-latency frame forwarding.

send_transcription `async` ¶

send_transcription(session, text, role='user')

Log transcription text (no UI channel in RTP mode).

on_dtmf_received ¶

on_dtmf_received(callback)

Register a callback for inbound DTMF digits (RFC 4733).

Parameters:

Name	Type	Description	Default
`callback`	`DTMFReceivedCallback`	Function called with `(session, dtmf_event)`.	required

Install with: pip install roomkit[rtp]

Mock Voice Backend¶

MockVoiceBackend ¶

MockVoiceBackend(*, capabilities=NONE)

Bases: VoiceBackend

Mock voice backend for testing.

Tracks all method calls and provides helpers to simulate events. The backend is a pure transport — no VAD or audio intelligence.

Example

backend = MockVoiceBackend()

Track calls¶

session = await backend.connect("room-1", "user-1", "voice-1") assert backend.calls[-1].method == "connect"

Simulate raw audio received¶

frame = AudioFrame(data=b"audio-data") await backend.simulate_audio_received(session, frame)

Simulate barge-in¶

await backend.simulate_barge_in(session)

simulate_audio_received `async` ¶

simulate_audio_received(session, frame)

Simulate the backend receiving a raw audio frame.

Fires all registered on_audio_received callbacks.

simulate_barge_in `async` ¶

simulate_barge_in(session)

Simulate user speaking while TTS is playing (barge-in).

Fires all registered on_barge_in callbacks.

start_playing ¶

start_playing(session)

Mark a session as playing audio (for barge-in testing).

stop_playing ¶

stop_playing(session)

Mark a session as no longer playing audio.

simulate_session_ready `async` ¶

simulate_session_ready(session)

Simulate the backend signalling that a session's audio path is live.

Fires all registered on_session_ready callbacks.

simulate_client_disconnected `async` ¶

simulate_client_disconnected(session)

Simulate the backend signalling that a client has disconnected.

Fires all registered on_client_disconnected callbacks.

MockVoiceCall `dataclass` ¶

MockVoiceCall(method, args=dict())

Record of a call made to MockVoiceBackend.

Voice Events¶

BargeInEvent `dataclass` ¶

BargeInEvent(session, interrupted_text, audio_position_ms, timestamp=_utcnow())

User started speaking while TTS was playing.

This event is fired when the VAD detects speech starting while audio is being sent to the user. This allows the system to: - Cancel the current TTS playback - Adjust response strategy (e.g., acknowledge interruption) - Track conversation dynamics

session `instance-attribute` ¶

session

The voice session where barge-in occurred.

interrupted_text `instance-attribute` ¶

interrupted_text

The text that was being spoken when interrupted.

audio_position_ms `instance-attribute` ¶

audio_position_ms

How much of the TTS audio the user heard (ms), synthesis latency excluded: the played_ms the timeline records for the same cut.

timestamp `class-attribute` `instance-attribute` ¶

timestamp = field(default_factory=_utcnow)

When the barge-in was detected.

TTSCancelledEvent `dataclass` ¶

TTSCancelledEvent(session, reason, text, audio_position_ms, timestamp=_utcnow())

TTS playback was cancelled.

This event is fired when TTS synthesis or playback is stopped before completion. Reasons include: - barge_in: User started speaking - explicit: Application called interrupt() - disconnect: Session ended - error: TTS or playback error

session `instance-attribute` ¶

session

The voice session where TTS was cancelled.

reason `instance-attribute` ¶

reason

Why the TTS was cancelled.

text `instance-attribute` ¶

text

The text that was being synthesized.

audio_position_ms `instance-attribute` ¶

audio_position_ms

How much of the audio the user heard (ms), 0 if none went out: the played_ms the timeline records for the same cut.

timestamp `class-attribute` `instance-attribute` ¶

timestamp = field(default_factory=_utcnow)

When the cancellation occurred.

PartialTranscriptionEvent `dataclass` ¶

PartialTranscriptionEvent(session, text, confidence, is_stable, role='user', language=None, timestamp=_utcnow())

Interim transcription result during speech.

This event is fired by backends that support streaming STT, providing real-time transcription updates before the final result. Use cases include: - Live captions/subtitles - Early intent detection - Visual feedback during speech

session `instance-attribute` ¶

session

The voice session being transcribed.

text `instance-attribute` ¶

text

The current transcription (may change in subsequent events).

confidence `instance-attribute` ¶

confidence

Confidence score (0.0 to 1.0).

is_stable `instance-attribute` ¶

is_stable

True if this portion is unlikely to change significantly.

role `class-attribute` `instance-attribute` ¶

role = 'user'

Who spoke: 'user' (input) or 'assistant' (AI output).

language `class-attribute` `instance-attribute` ¶

language = None

The language the STT provider reports for this partial, if any.

timestamp `class-attribute` `instance-attribute` ¶

timestamp = field(default_factory=_utcnow)

When this transcription was received.

VADSilenceEvent `dataclass` ¶

VADSilenceEvent(session, silence_duration_ms, timestamp=_utcnow())

Silence detected after speech.

This event is fired when the VAD detects a period of silence following speech. It can be used for: - Early end-of-utterance detection (before full speech_end) - Adaptive silence thresholds - Turn-taking management

session `instance-attribute` ¶

session

The voice session where silence was detected.

silence_duration_ms `instance-attribute` ¶

silence_duration_ms

Duration of silence in milliseconds.

timestamp `class-attribute` `instance-attribute` ¶

timestamp = field(default_factory=_utcnow)

When the silence was detected.

VADAudioLevelEvent `dataclass` ¶

VADAudioLevelEvent(session, level_db, is_speech, timestamp=_utcnow())

Periodic audio level update for UI feedback.

This event is fired periodically (typically 10Hz) to provide audio level information for UI visualization. Use cases include: - Audio level meters - Speaking indicators - Noise detection

session `instance-attribute` ¶

session

The voice session.

level_db `instance-attribute` ¶

level_db

Audio level in dB (typically -60 to 0, where 0 is max).

is_speech `instance-attribute` ¶

is_speech

VAD's determination if this audio contains speech.

timestamp `class-attribute` `instance-attribute` ¶

timestamp = field(default_factory=_utcnow)

When this measurement was taken.

Callback Types¶

Callback	Signature
`SpeechStartCallback`	`(VoiceSession) -> Any`
`SpeechEndCallback`	`(VoiceSession, bytes) -> Any`
`PartialTranscriptionCallback`	`(VoiceSession, str, float, bool) -> Any`
`VADSilenceCallback`	`(VoiceSession, int) -> Any`
`VADAudioLevelCallback`	`(VoiceSession, float, bool) -> Any`
`BargeInCallback`	`(VoiceSession) -> Any`

Voice Providers¶

Voice Backend¶

VoiceBackend ¶

Register raw audio callback¶

Connect a participant¶

Stream audio to the client¶

Disconnect¶

name abstractmethod property ¶

auto_connect property ¶

capabilities property ¶

feeds_aec_reference property ¶

audio_received_callback property ¶

supports_playback_callback property ¶

connect async ¶

disconnect abstractmethod async ¶

send_audio abstractmethod async ¶

send_audio_sync ¶

get_session ¶

list_sessions ¶

close async ¶

on_audio_received ¶

on_session_ready ¶

on_barge_in ¶

send_dtmf ¶

cancel_audio async ¶

accept async ¶

interrupt ¶

set_input_muted ¶

set_input_gated ¶

on_client_disconnected ¶

on_speaker_change ¶

end_of_response ¶

wait_playback async ¶

is_playing ¶

on_audio_played ¶

on_aec_tap ¶

set_trace_emitter ¶

send_transcription async ¶

PlaybackErrors ¶

watch ¶

VoiceCapability ¶

NONE class-attribute instance-attribute ¶

INTERRUPTION class-attribute instance-attribute ¶

BARGE_IN class-attribute instance-attribute ¶

NATIVE_AEC class-attribute instance-attribute ¶

NATIVE_AGC class-attribute instance-attribute ¶

DTMF_INBAND class-attribute instance-attribute ¶

DTMF_SIGNALING class-attribute instance-attribute ¶

NATIVE_BRIDGE class-attribute instance-attribute ¶

VoiceSession dataclass ¶

last_usage property ¶

renegotiate ¶

VoiceSessionState ¶

AudioChunk dataclass ¶

TranscriptionResult dataclass ¶

is_speech_start class-attribute instance-attribute ¶

segments class-attribute instance-attribute ¶

speaker property ¶

SpeakerSegment dataclass ¶

STT (Speech-to-Text)¶

STTProvider ¶

name property ¶

supports_streaming property ¶

supports_language_override property ¶

supports_diarization property ¶

transcribe abstractmethod async ¶

transcribe_stream async ¶

warmup async ¶

close async ¶

MockSTTProvider ¶

languages_requested instance-attribute ¶

STTLanguageLock dataclass ¶

language_for ¶

observe ¶

forget ¶

Sherpa-ONNX STT¶

SherpaOnnxSTTProvider ¶

warmup async ¶

transcribe async ¶

transcribe_stream async ¶

name `abstractmethod` `property` ¶

auto_connect `property` ¶

capabilities `property` ¶

feeds_aec_reference `property` ¶

audio_received_callback `property` ¶

supports_playback_callback `property` ¶

connect `async` ¶

disconnect `abstractmethod` `async` ¶

send_audio `abstractmethod` `async` ¶

close `async` ¶

cancel_audio `async` ¶

accept `async` ¶

wait_playback `async` ¶

send_transcription `async` ¶

NONE `class-attribute` `instance-attribute` ¶

INTERRUPTION `class-attribute` `instance-attribute` ¶

BARGE_IN `class-attribute` `instance-attribute` ¶

NATIVE_AEC `class-attribute` `instance-attribute` ¶

NATIVE_AGC `class-attribute` `instance-attribute` ¶

DTMF_INBAND `class-attribute` `instance-attribute` ¶

DTMF_SIGNALING `class-attribute` `instance-attribute` ¶

NATIVE_BRIDGE `class-attribute` `instance-attribute` ¶

VoiceSession `dataclass` ¶

last_usage `property` ¶

AudioChunk `dataclass` ¶

TranscriptionResult `dataclass` ¶

is_speech_start `class-attribute` `instance-attribute` ¶

segments `class-attribute` `instance-attribute` ¶

speaker `property` ¶

SpeakerSegment `dataclass` ¶

name `property` ¶

supports_streaming `property` ¶

supports_language_override `property` ¶

supports_diarization `property` ¶

transcribe `abstractmethod` `async` ¶

transcribe_stream `async` ¶

warmup `async` ¶

close `async` ¶

languages_requested `instance-attribute` ¶

STTLanguageLock `dataclass` ¶

warmup `async` ¶

transcribe `async` ¶

transcribe_stream `async` ¶

SherpaOnnxSTTConfig `dataclass` ¶

supports_streaming `property` ¶

supports_diarization `property` ¶

transcribe `async` ¶

transcribe_recording `async` ¶

close `async` ¶

GeminiSTTConfig `dataclass` ¶

Transcript `dataclass` ¶

language `instance-attribute` ¶

words `class-attribute` `instance-attribute` ¶

text `property` ¶

plain_text `property` ¶

TranscriptSegment `dataclass` ¶

TranscriptWord `dataclass` ¶

speaker `class-attribute` `instance-attribute` ¶

supports_diarization `property` ¶

transcribe `async` ¶

transcribe_stream `async` ¶

close `async` ¶

MetaSTTConfig `dataclass` ¶

transcribe `async` ¶

transcribe_stream `async` ¶

AzureMAISTTConfig `dataclass` ¶

name `property` ¶

default_voice `property` ¶

max_dialogue_speakers `property` ¶

supports_streaming_input `property` ¶

context_level `property` ¶

available_voices `classmethod` ¶

list_voices `async` ¶

synthesize_dialogue `async` ¶

synthesize `abstractmethod` `async` ¶

synthesize_stream_input `async` ¶

synthesize_stream `async` ¶

warmup `async` ¶

close `async` ¶

supports_design `property` ¶