Voice Providers

Voice Backend

VoiceBackend

Bases: ABC

Abstract base class for voice transport backends.

VoiceBackend handles the transport layer for real-time audio: - Managing voice session connections - Streaming audio to/from clients - Delivering raw inbound audio frames via on_audio_received

The backend is framework-agnostic and a pure transport — all audio intelligence (VAD, denoising, diarization) is handled by the AudioPipeline.

Example usage

backend = WebRTCVoiceBackend()

Register raw audio callback

backend.on_audio_received(handle_audio_frame)

Connect a participant

session = await backend.connect("room-1", "user-1", "voice-channel")

Stream audio to the client

await backend.send_audio(session, audio_chunks)

Disconnect

await backend.disconnect(session)

name abstractmethod property

name

Backend name (e.g., 'webrtc', 'websocket', 'livekit').

auto_connect property

auto_connect

Whether attach_channel should auto-create a session.

Returns True for single-user backends (local mic, single-user FastRTC) where the developer IS the participant. Returns False for server backends (SIP, RTP, WebRTC) that accept external connections.

capabilities property

capabilities

Declare supported capabilities.

Override to enable features like interruption, barge-in, etc. By default, no optional capabilities are supported.

Returns:

Type	Description
VoiceCapability	Flags indicating supported capabilities.

feeds_aec_reference property

feeds_aec_reference

Whether this backend feeds AEC reference at the transport level.

When True, the pipeline skips aec.feed_reference() in the outbound path to avoid double-feeding. Transport-level feeding (from the speaker callback) is preferred because it is time-aligned with actual speaker output.

audio_received_callback property

audio_received_callback

Return the currently registered audio-received callback, if any.

supports_playback_callback property

supports_playback_callback

Whether this backend fires :meth:on_audio_played callbacks.

When True, the pipeline can rely on playback-time AEC reference instead of generation-time feeding from process_outbound.

connect async

connect(room_id, participant_id, channel_id, *, metadata=None)

Create a new voice session for a participant.

Backends that initiate connections (VoiceChannel path) override this. Backends that receive external connections (realtime transport path) override :meth:accept instead.

Parameters:

Name	Type	Description	Default
room_id	str	The room to join.	required
participant_id	str	The participant's ID.	required
channel_id	str	The voice channel ID.	required
metadata	dict[str, Any] | None	Optional session metadata.	None

Returns:

Type	Description
VoiceSession	A VoiceSession representing the connection.

disconnect abstractmethod async

disconnect(session)

End a voice session.

Parameters:

Name	Type	Description	Default
session	VoiceSession	The session to disconnect.	required

send_audio abstractmethod async

send_audio(session, audio)

Send audio to a voice session.

Parameters:

Name	Type	Description	Default
session	VoiceSession	The target session.	required
audio	bytes | AsyncIterator[AudioChunk]	Raw audio bytes or an async iterator of AudioChunks for streaming.	required

send_audio_sync

send_audio_sync(session, chunk)

Synchronously send a single audio chunk to a session.

Used by the audio bridge for frame-by-frame forwarding from audio callback threads where await is not available. Backends that support bridging SHOULD override this with an efficient implementation. The default schedules the async send_audio() on the event loop, which adds latency.

Parameters:

Name	Type	Description	Default
session	VoiceSession	The target session.	required
chunk	AudioChunk	A single audio chunk to send.	required

get_session

get_session(session_id)

Get a session by ID.

Override for backends that track sessions internally.

Parameters:

Name	Type	Description	Default
session_id	str	The session ID to look up.	required

Returns:

Type	Description
VoiceSession | None	The VoiceSession if found, None otherwise.

list_sessions

list_sessions(room_id)

List all active sessions in a room.

Override for backends that track sessions internally.

Parameters:

Name	Type	Description	Default
room_id	str	The room to list sessions for.	required

Returns:

Type	Description
list[VoiceSession]	List of active VoiceSessions in the room.

close async

close()

Release backend resources.

Override in subclasses that need cleanup.

on_audio_received

on_audio_received(callback)

Register a callback for raw inbound audio frames.

Backends supporting shared ownership return an idempotent unsubscribe function. Legacy implementations may return None. The same convention applies to playback and disconnect registrations.

The pipeline or channel calls this to receive every audio frame produced by the transport.

Parameters:

Name	Type	Description	Default
callback	AudioReceivedCallback	Function called with (session, audio_frame).	required

on_session_ready

on_session_ready(callback)

Register callback for when a session's audio path becomes live.

Fired when the transport is ready to send/receive audio for a session (e.g. WebSocket connected, RTP socket active, mic stream started). The VoiceChannel uses this together with bind_session to implement the dual-signal ON_SESSION_STARTED hook.

Parameters:

Name	Type	Description	Default
callback	SessionReadyCallback	Function called with (session).	required

on_barge_in

on_barge_in(callback)

Register callback for barge-in detection.

Only called if capabilities includes BARGE_IN. Backends should call this when user starts speaking while audio is being played (TTS interruption).

Parameters:

Name	Type	Description	Default
callback	BargeInCallback	Function called with (session).	required

send_dtmf

send_dtmf(session, digit, duration_ms=160)

Send a DTMF digit to the remote party.

Sends an RFC 4733 telephone-event via RTP (out-of-band signaling). Only backends with DTMF_SIGNALING capability support this.

Parameters:

Name	Type	Description	Default
session	VoiceSession	The voice session to send DTMF on.	required
digit	str	The DTMF digit to send ('0'-'9', '*', '#', 'A'-'D').	required
duration_ms	int	Duration of the tone in milliseconds (default 160).	160

Raises:

Type	Description
NotImplementedError	If the backend does not support sending DTMF.

cancel_audio async

cancel_audio(session)

Cancel ongoing audio playback for a session.

Delegates to :meth:interrupt and returns True. Subclasses may override for more nuanced behaviour.

Parameters:

Name	Type	Description	Default
session	VoiceSession	The session to cancel audio for.	required

Returns:

Type	Description
bool	True if audio was cancelled, False if nothing was playing.

accept async

accept(session, connection)

Bind an external connection to a session.

Backends that receive connections from external sources (e.g. WebSocket, WebRTC, SIP) override this. Backends that create their own connections override :meth:connect instead.

Parameters:

Name	Type	Description	Default
session	VoiceSession	The voice session to bind.	required
connection	Any	Protocol-specific connection object.	required

interrupt

interrupt(session)

Signal interruption — flush outbound queue, stop playback.

set_input_muted

set_input_muted(session, muted)

Mute or unmute the input (microphone) for a session.

Parameters:

Name	Type	Description	Default
session	VoiceSession	The session to mute/unmute.	required
muted	bool	True to mute, False to unmute.	required

set_input_gated

set_input_gated(session, gated)

Gate or un-gate audio input for primary speaker mode.

When gated, audio is not forwarded to provider callbacks but may still be fed to a pipeline for diarization analysis.

Parameters:

Name	Type	Description	Default
session	VoiceSession	The session to gate/un-gate.	required
gated	bool	True to gate, False to un-gate.	required

on_client_disconnected

on_client_disconnected(callback)

Register callback for client disconnection.

Parameters:

Name	Type	Description	Default
callback	TransportDisconnectCallback	Called with (session) when the client disconnects.	required

on_speaker_change

on_speaker_change(callback)

Register callback for speaker change events.

Parameters:

Name	Type	Description	Default
callback	SpeakerChangeCallback	Called with (session, diarization_result).	required

end_of_response

end_of_response(session)

Signal end of an AI response for outbound pacing.

is_playing

is_playing(session)

Check if audio is currently being sent to the session.

Used for barge-in detection to know if interruption is possible.

Parameters:

Name	Type	Description	Default
session	VoiceSession	The session to check.	required

Returns:

Type	Description
bool	True if audio is currently playing, False otherwise.

on_audio_played

on_audio_played(callback)

Register a callback for audio frames as they are played.

Called with each audio frame at the moment it is output by the speaker, providing time-aligned reference for echo cancellation. The pipeline uses this to feed AEC reference at the correct time.

Note

Callbacks may be invoked from the audio I/O thread — implementations must be thread-safe. A backend with a persistent output stream may mark the final drained frame with frame.metadata["playback_ended"] = True so pipeline AEC can leave bypass mode at the physical boundary.

Parameters:

Name	Type	Description	Default
callback	AudioPlayedCallback	Function called with (session, audio_frame).	required

set_trace_emitter

set_trace_emitter(emitter)

Set a callback for emitting protocol traces.

Called by the owning channel when trace observers are registered. Implementations should store the emitter and call it at key protocol points (e.g. INVITE, BYE for SIP).

Parameters:

Name	Type	Description	Default
emitter	Callable[..., Any] | None	The channel's :meth:emit_trace method, or None to disable.	required

send_transcription async

send_transcription(session, text, role='user')

Send transcription text to the client for UI display.

Optional method for backends that support sending text updates.

Parameters:

Name	Type	Description	Default
session	VoiceSession	The voice session to send to.	required
text	str	The transcribed or response text.	required
role	str	Either "user" (transcription) or "assistant" (AI response).	'user'

VoiceCapability

Bases: Flag

Capabilities a VoiceBackend can support.

Backends declare their capabilities via the capabilities property. This allows RoomKit to know which features are available and enables integrators to choose backends based on their needs.

Example

class MyBackend(VoiceBackend): @property def capabilities(self) -> VoiceCapability: return ( VoiceCapability.INTERRUPTION | VoiceCapability.BARGE_IN )

NONE class-attribute instance-attribute

NONE = 0

No optional capabilities (default).

INTERRUPTION class-attribute instance-attribute

INTERRUPTION = auto()

Backend can cancel ongoing audio playback (cancel_audio).

BARGE_IN class-attribute instance-attribute

BARGE_IN = auto()

Backend detects and handles barge-in (user interrupts TTS).

NATIVE_AEC class-attribute instance-attribute

NATIVE_AEC = auto()

Backend provides its own Acoustic Echo Cancellation.

NATIVE_AGC class-attribute instance-attribute

NATIVE_AGC = auto()

Backend provides its own Automatic Gain Control.

DTMF_INBAND class-attribute instance-attribute

DTMF_INBAND = auto()

Backend can detect DTMF tones from the audio stream.

DTMF_SIGNALING class-attribute instance-attribute

DTMF_SIGNALING = auto()

Backend receives DTMF via out-of-band signaling (e.g. SIP INFO).

NATIVE_BRIDGE class-attribute instance-attribute

NATIVE_BRIDGE = auto()

Backend can bridge audio at the transport level (RTP relay).

VoiceSession dataclass

VoiceSession(id, room_id, participant_id, channel_id, state=CONNECTING, provider_session_id=None, created_at=_utcnow(), metadata=dict(), _last_usage=dict())

Active voice connection for a participant.

state is guarded (RFC §12.1). Leaving ENDED is refused outright — it is the one transition the RFC forbids, and letting a torn-down session go back to ACTIVE resurrects audio paths the framework has already released. Any other move outside the table is logged and allowed: the table does not model every provider's reality (a realtime provider renegotiating goes ACTIVE → CONNECTING), and turning an unmodelled transition into a crash would trade a documentation gap for an outage.

renegotiate

renegotiate()

Return the session to CONNECTING for a provider renegotiation.

A reconfigure — swapping an agent's personality, voice or tools during a handoff — tears the upstream connection down and builds a new one while the participant's session continues. Nobody hung up. The default provider implements that as disconnect + connect, which leaves the session ENDED in between, and reconnecting from there is the resurrection §12.1 forbids.

This is the only sanctioned way out of ENDED, and it is narrow by design: it says "the framework itself just tore this down to rebuild it". It does not make ENDED non-terminal for anyone else — a participant who really hung up still gets a new session.

VoiceSessionState

Bases: StrEnum

State of a voice session.

AudioChunk dataclass

AudioChunk(data, sample_rate=16000, channels=1, format='pcm_s16le', timestamp_ms=None, is_final=False)

A chunk of audio data for streaming (used for outbound TTS).

TranscriptionResult dataclass

TranscriptionResult(text, is_final=True, confidence=None, language=None, words=list(), is_speech_start=False)

Result from speech-to-text transcription.

is_speech_start class-attribute instance-attribute

is_speech_start = False

Set by providers with server-side VAD to signal speech detected.

STT (Speech-to-Text)

STTProvider

Bases: ABC

Speech-to-text provider.

name property

name

Provider name (e.g. 'whisper', 'deepgram').

supports_streaming property

supports_streaming

Whether this provider supports streaming transcription.

supports_language_override property

supports_language_override

Whether transcribe and transcribe_stream honour language.

A provider that answers False is never handed a per-call language: :class:~roomkit.channels.VoiceChannel keeps calling it with audio only, so an implementation written against the older signature keeps working unchanged.

transcribe abstractmethod async

transcribe(audio, *, language=None)

Transcribe complete audio to text.

Parameters:

Name	Type	Description	Default
audio	AudioContent | AudioChunk | AudioFrame	Audio content (URL), raw audio chunk, or audio frame.	required
language	str | None	Language for this call only, overriding the provider's configuration. Honoured when :attr:supports_language_override is true; a caller must not pass it otherwise.	None

Returns:

Type	Description
TranscriptionResult	TranscriptionResult with text and metadata. language on the
TranscriptionResult	result is what the provider reports, never an echo of the request.

transcribe_stream async

transcribe_stream(audio_stream, *, language=None)

Stream transcription with partial results.

Override for providers that support streaming. Default: buffers all audio and returns single result.

Parameters:

Name	Type	Description	Default
audio_stream	AsyncIterator[AudioChunk]	Audio chunks; the stream ends when it is exhausted.	required
language	str | None	Language for this stream only — see :meth:transcribe. A language is fixed for the life of a stream: a change takes effect on the next stream the caller opens.	None

warmup async

warmup()

Pre-load models so the first call is fast. Override in subclasses.

close async

close()

Release resources. Override in subclasses if needed.

MockSTTProvider

MockSTTProvider(transcripts=None, *, streaming=False, languages=None)

Bases: STTProvider

Mock speech-to-text for testing.

Parameters:

Name	Type	Description	Default
transcripts	list[str] | None	Texts returned in turn, cycling.	None
streaming	bool	Whether to advertise streaming support.	False
languages	list[str | None] | None	The language each result reports, in step with transcripts and cycling the same way — what a detecting provider would have heard. None entries report nothing.	None

languages_requested instance-attribute

languages_requested = []

The language handed to each call, in order (None = the default).

STTLanguageLock dataclass

STTLanguageLock(detect_language='multi', prefer=dict(), lock_after=1, release_after=2, min_confidence=0.5)

Start detecting, pin the language the speaker uses, let go when it stops fitting.

A streaming STT does better with its language set than in a detecting mode (Deepgram Nova-3 multi), but nobody knows the language before the caller speaks. This policy watches the final results a session produces and tells :class:~roomkit.channels.VoiceChannel which language its next stream should use:

• Every session starts in detect_language.
• Once lock_after consecutive finals report the same language, the session is locked to it — mapped through prefer first, so a reported "fr" can become "fr-CA".
• A locked session counts a miss for every final with no text, a confidence below min_confidence, or a reported language other than the lock. release_after consecutive misses send it back to detect_language; a fitting final resets the count.

The object is shared by the channel and keeps one small state per session id; the channel calls :meth:forget when a session goes away. Everything it does is reachable from a hook with event.language and VoiceChannel.set_stt_language — it is the packaged version of that loop, not the only way to run it.

language_for

language_for(session_id)

The language the session should be using right now.

observe

observe(session_id, result)

Account for a final result and return the language for the next stream.

forget

forget(session_id)

Drop the state kept for a session.

Sherpa-ONNX STT

SherpaOnnxSTTProvider

SherpaOnnxSTTProvider(config)

Bases: STTProvider

Speech-to-text provider using sherpa-onnx.

Supports transducer models for both streaming and batch recognition, and Whisper models for batch recognition only.

warmup async

warmup()

Pre-load the recognizer model (CUDA init can be slow).

transcribe async

transcribe(audio, *, language=None)

Transcribe complete audio.

For transducer mode, uses the OnlineRecognizer (feeds all audio then reads the result). For whisper mode, uses the OfflineRecognizer.

Parameters:

Name	Type	Description	Default
audio	AudioContent | AudioChunk | AudioFrame	Audio content or raw audio chunk (PCM S16LE expected).	required

Returns:

Type	Description
TranscriptionResult	TranscriptionResult with text.

transcribe_stream async

transcribe_stream(audio_stream, *, language=None)

Stream transcription with partial results using OnlineRecognizer.

Only supported for transducer mode. Whisper mode raises ValueError.

Parameters:

Name	Type	Description	Default
audio_stream	AsyncIterator[AudioChunk]	Async iterator of audio chunks.	required

Yields:

Type	Description
AsyncIterator[TranscriptionResult]	TranscriptionResult with partial and final transcripts.

SherpaOnnxSTTConfig dataclass

SherpaOnnxSTTConfig(mode='transducer', tokens='', encoder='', decoder='', joiner='', model_type='', language='en', task='transcribe', sample_rate=16000, num_threads=2, provider='cpu', enable_endpoint_detection=True, rule1_min_trailing_silence=2.4, rule2_min_trailing_silence=1.2, rule3_min_utterance_length=20.0)

Configuration for the sherpa-onnx STT provider.

Attributes:

Name	Type	Description
mode	str	Recognition mode — "transducer" or "whisper".
tokens	str	Path to tokens.txt.
encoder	str	Path to encoder .onnx model.
decoder	str	Path to decoder .onnx model.
joiner	str	Path to joiner .onnx model (transducer only).
model_type	str	Model type hint for sherpa-onnx (e.g. "nemo_transducer" for NeMo TDT/transducer models). When set, the model is treated as offline-only (no streaming).
language	str	Language code (Whisper only).
task	str	Whisper task — "transcribe" (default) or "translate" (translates to English).
sample_rate	int	Expected audio sample rate.
num_threads	int	Number of CPU threads for inference.
provider	str	ONNX execution provider ("cpu" or "cuda").
enable_endpoint_detection	bool	Enable sherpa-onnx endpoint detection. Enabled by default. When VAD drives the stream lifecycle the VAD fires first (its silence threshold is shorter), so this is harmless in a pipeline and useful for standalone use.
rule1_min_trailing_silence	float	Endpoint rule 1 — minimum trailing silence (seconds) after speech to trigger endpoint.
rule2_min_trailing_silence	float	Endpoint rule 2 — minimum trailing silence (seconds) after speech with decoded text.
rule3_min_utterance_length	float	Endpoint rule 3 — minimum utterance length (seconds) to trigger endpoint regardless of silence.

Usage

from roomkit.voice.stt.sherpa_onnx import SherpaOnnxSTTProvider, SherpaOnnxSTTConfig

# Transducer model (streaming + batch)
stt = SherpaOnnxSTTProvider(SherpaOnnxSTTConfig(
    mode="transducer",
    tokens="path/to/tokens.txt",
    encoder="path/to/encoder.onnx",
    decoder="path/to/decoder.onnx",
    joiner="path/to/joiner.onnx",
))

# Whisper model (batch only)
stt = SherpaOnnxSTTProvider(SherpaOnnxSTTConfig(
    mode="whisper",
    tokens="path/to/tokens.txt",
    encoder="path/to/encoder.onnx",
    decoder="path/to/decoder.onnx",
    language="en",
))

Install with: pip install roomkit[sherpa-onnx]

Gemini STT (Google, batch)

GeminiSTTProvider

GeminiSTTProvider(config)

Bases: STTProvider

Google Gemini speech-to-text provider.

Batch only — :attr:supports_streaming is False, so a :class:~roomkit.channels.voice.VoiceChannel transcribes on SPEECH_END rather than streaming partials. Given seconds of model latency, the honest placement is batch_mode=True (dictation, voicemail) or transcription of a finished recording, not live turn-taking.

Two entry points:

• :meth:transcribe — the ABC contract, returning flat text.
• :meth:transcribe_recording — the whole :class:Transcript, with speaker turns and timestamps, and the one that accepts a file path.

supports_streaming property

supports_streaming

The API takes a complete recording; there is no stream to open.

transcribe async

transcribe(audio, *, language=None)

Transcribe a complete recording.

Parameters:

Name	Type	Description	Default
audio	AudioContent | AudioChunk | AudioFrame	Audio content (data: URL, local path or Files API URI), raw audio chunk, or audio frame.	required

Returns:

Type	Description
TranscriptionResult	TranscriptionResult whose text carries the spoken words and
TranscriptionResult	whose language carries what the model identified. Speaker turns
TranscriptionResult	and timestamps are dropped by this shape — call
TranscriptionResult	meth:transcribe_recording for those.

transcribe_recording async

transcribe_recording(source)

Transcribe a recording into speaker turns.

Parameters:

Name	Type	Description	Default
source	Any	A path to a recording (str or Path), an AudioContent, an AudioChunk or an AudioFrame. Files larger than max_inline_bytes are uploaded through the Files API and deleted afterwards.	required

Returns:

Name	Type	Description
The	Transcript	class:Transcript — detected language, and one segment per
	Transcript	speaker turn.

Raises:

Type	Description
RuntimeError	The model answered without a usable transcript.

close async

close()

Close the genai client's connection pool and drop the reference.

GeminiSTTConfig dataclass

GeminiSTTConfig(api_key, model='gemini-3.6-flash', language=None, diarize=True, prompt=None, timeout=600.0, max_inline_bytes=_MAX_INLINE_BYTES, connect_timeout=5.0)

Configuration for the Gemini batch STT provider.

Parameters:

Name	Type	Description	Default
api_key	str	Gemini API key (GEMINI_API_KEY).	required
model	str	A text/multimodal Gemini model that accepts audio input — see :meth:~roomkit.providers.gemini.ai.GeminiAIProvider.available_models for the catalog roomkit keeps. The default is the current flash model: transcription is not a reasoning task, and flash is the cheapest way to buy the audio context window.	'gemini-3.6-flash'
language	str | None	Optional BCP-47 hint (e.g. "fr-CA"). Left unset, the model identifies the language itself and reports it on the transcript.	None
diarize	bool	Ask for speaker labels. Worth turning off for a single-speaker recording, where labelling costs tokens and invents distinctions. A conference recorded per participant track needs no diarization at all — transcribe each track and merge on the timestamps.	True
prompt	str | None	Extra instruction appended to the transcription request — vocabulary that matters ("the product is spelled RoomKit"), formatting rules, anything the model should know before it listens.	None
timeout	float	Per-request timeout in seconds. Generous by design: a model answering on an hour of audio is not answering in milliseconds.	600.0
max_inline_bytes	int	Recordings larger than this are uploaded through the Files API instead of being inlined in the request.	_MAX_INLINE_BYTES
connect_timeout	float	TCP connect timeout in seconds, apart from timeout.	5.0

Transcript dataclass

Transcript(language, segments)

A whole recording, as speaker turns.

text property

text

The turns joined into a readable transcript, one line per speaker.

plain_text property

plain_text

The spoken words alone, without speaker labels.

TranscriptSegment dataclass

TranscriptSegment(speaker, start, end, text)

One speaker turn.

Timestamps are MM:SS strings, as the model returns them. They are the model's reading of the recording, not a forced alignment: treat them as navigation, not as sync marks.

Usage

from roomkit.voice.stt.gemini import GeminiSTTConfig, GeminiSTTProvider

stt = GeminiSTTProvider(GeminiSTTConfig(api_key="your-gemini-api-key"))

transcript = await stt.transcribe_recording("meeting.wav")
for turn in transcript.segments:
    print(f"[{turn.start}-{turn.end}] {turn.speaker}: {turn.text}")

Batch only: the model takes a complete recording and answers in seconds, so this fits meetings, voicemail and imported files rather than live turn-taking — see the STT & TTS Providers guide.

Install with: pip install roomkit[gemini]

TTS (Text-to-Speech)

TTSProvider

Bases: ABC

Text-to-speech provider.

name property

name

Provider name (e.g. 'elevenlabs', 'openai').

default_voice property

default_voice

Default voice ID. Override in subclasses.

supports_streaming_input property

supports_streaming_input

Whether this TTS accepts streaming text input.

synthesize abstractmethod async

synthesize(text, *, voice=None)

Synthesize text to audio.

Parameters:

Name	Type	Description	Default
text	str	Text to synthesize.	required
voice	str | None	Voice ID (uses default_voice if not specified).	None

Returns:

Type	Description
AudioContent	AudioContent with URL to generated audio.

synthesize_stream_input async

synthesize_stream_input(text_stream, *, voice=None)

Stream audio from streaming text chunks.

Override for providers that accept an async text stream as input.

synthesize_stream async

synthesize_stream(text, *, voice=None)

Stream audio chunks as they're generated.

Override for providers that support streaming. Default: synthesizes full audio and yields single chunk.

warmup async

warmup()

Pre-load models so the first call is fast. Override in subclasses.

close async

close()

Release resources. Override in subclasses if needed.

MockTTSProvider

MockTTSProvider(voice='mock-voice')

Bases: TTSProvider

Mock text-to-speech for testing.

Sherpa-ONNX TTS

SherpaOnnxTTSProvider

SherpaOnnxTTSProvider(config)

Bases: TTSProvider

Text-to-speech provider using sherpa-onnx with VITS/Piper models.

warmup async

warmup()

Pre-load the TTS model (CUDA init can be slow).

synthesize async

synthesize(text, *, voice=None)

Synthesize text to audio.

Parameters:

Name	Type	Description	Default
text	str	Text to synthesize.	required
voice	str | None	Speaker ID as string (uses default if not specified).	None

Returns:

Type	Description
AudioContent	AudioContent with a data: URL containing WAV audio.

synthesize_stream async

synthesize_stream(text, *, voice=None)

Stream audio chunks using a callback bridge.

Parameters:

Name	Type	Description	Default
text	str	Text to synthesize.	required
voice	str | None	Speaker ID as string (uses default if not specified).	None

Yields:

Type	Description
AsyncIterator[AudioChunk]	AudioChunk with PCM S16LE audio data.

SherpaOnnxTTSConfig dataclass

SherpaOnnxTTSConfig(model='', tokens='', data_dir='', lexicon='', speaker_id=0, speed=1.0, sample_rate=22050, num_threads=2, provider='cpu')

Configuration for the sherpa-onnx TTS provider.

Attributes:

Name	Type	Description
model	str	Path to VITS/Piper .onnx model.
tokens	str	Path to tokens.txt.
data_dir	str	Path to espeak-ng data directory (Piper models).
lexicon	str	Path to optional lexicon file.
speaker_id	int	Speaker ID for multi-speaker models.
speed	float	Speech speed multiplier (1.0 = normal).
sample_rate	int	Output sample rate (usually determined by the model).
num_threads	int	Number of CPU threads for inference.
provider	str	ONNX execution provider ("cpu" or "cuda").

Usage

from roomkit.voice.tts.sherpa_onnx import SherpaOnnxTTSProvider, SherpaOnnxTTSConfig

# VITS/Piper model with multi-speaker support
tts = SherpaOnnxTTSProvider(SherpaOnnxTTSConfig(
    model="path/to/vits-model.onnx",
    tokens="path/to/tokens.txt",
    data_dir="path/to/espeak-ng-data",  # for Piper models
    speaker_id=0,
    speed=1.0,
))

Install with: pip install roomkit[sherpa-onnx]

Grok TTS (xAI)

GrokTTSProvider

GrokTTSProvider(config)

Bases: TTSProvider

xAI Grok text-to-speech provider with WebSocket streaming support.

Supports: * synthesize() — REST endpoint, returns full audio. * synthesize_stream() — REST with chunked reading. * synthesize_stream_input() — bidirectional WebSocket for real-time text-to-speech (send text deltas, receive audio deltas).

synthesize async

synthesize(text, *, voice=None)

Synthesize text to audio via the REST endpoint.

Parameters:

Name	Type	Description	Default
text	str	Text to synthesize (max 15 000 characters).	required
voice	str | None	Voice ID override.	None

Returns:

Type	Description
AudioContent	AudioContent with a data-URL of the generated audio.

synthesize_stream async

synthesize_stream(text, *, voice=None)

Stream audio chunks from the REST endpoint.

The xAI API returns the audio as a single response, so we chunk it ourselves for pipeline compatibility.

synthesize_stream_input async

synthesize_stream_input(text_stream, *, voice=None)

Stream audio from streaming text input via WebSocket.

Protocol (xAI): * Client → {"type": "text.delta", "delta": "..."} * Client → {"type": "text.done"} * Server → {"type": "audio.delta", "delta": "<base64>"} * Server → {"type": "audio.done", "trace_id": "..."}

The connection stays open after audio.done allowing multiple turns, but we close after the first turn completes.

Parameters:

Name	Type	Description	Default
text_stream	AsyncIterator[str]	Async iterator yielding text chunks.	required
voice	str | None	Voice ID override.	None

Yields:

Type	Description
AsyncIterator[AudioChunk]	AudioChunk with decoded audio data.

close async

close()

Release the HTTP client.

GrokTTSConfig dataclass

GrokTTSConfig(api_key, voice_id='eve', language='en', codec='pcm', sample_rate=24000, bit_rate=128000, base_url='https://api.x.ai/v1', ws_url='wss://api.x.ai/v1/tts', timeout=60.0, connect_timeout=5.0)

Configuration for xAI Grok TTS provider.

Parameters:

Name	Type	Description	Default
api_key	str	xAI API key (or set XAI_API_KEY env var).	required
voice_id	str	One of eve, ara, rex, sal, leo.	'eve'
language	str	BCP-47 language code or auto.	'en'
codec	str	Output codec — pcm, wav, mp3, mulaw, alaw.	'pcm'
sample_rate	int	Output sample rate in Hz.	24000
bit_rate	int	MP3 bit rate (only used when codec is mp3).	128000
base_url	str	Override the REST API base URL.	'https://api.x.ai/v1'
ws_url	str	Override the WebSocket streaming URL.	'wss://api.x.ai/v1/tts'
timeout	float	HTTP request timeout in seconds.	60.0
connect_timeout	float	TCP connect timeout in seconds, apart from timeout.	5.0

Usage

from roomkit.voice.tts.grok import GrokTTSProvider, GrokTTSConfig

tts = GrokTTSProvider(GrokTTSConfig(
    api_key="your-xai-api-key",
    voice_id="eve",
    codec="pcm",
    sample_rate=24000,
))

Install with: pip install httpx websockets

Gemini TTS (Google)

GeminiTTSProvider

GeminiTTSProvider(config)

Bases: TTSProvider

Google Gemini text-to-speech provider.

Supports:

• :meth:synthesize — one request, full clip as WAV.
• :meth:synthesize_stream — audio deltas forwarded as they arrive.

Streaming text input is not supported: the API takes a complete prompt, so there is no seam to feed token deltas into. A voice channel therefore delivers through :meth:synthesize_stream on the complete reply.

available_voices classmethod

available_voices()

The 30 prebuilt voices, shared with Gemini Live native audio.

synthesize async

synthesize(text, *, voice=None)

Synthesize the whole text in one request.

Parameters:

Name	Type	Description	Default
text	str	Text to speak. Must not be blank.	required
voice	str | None	Prebuilt voice name overriding the configured one.	None

Returns:

Type	Description
AudioContent	AudioContent holding a WAV data: URL.

Raises:

Type	Description
ValueError	text is empty or whitespace.
RuntimeError	The interaction completed without audio.

synthesize_stream async

synthesize_stream(text, *, voice=None)

Stream audio deltas as the service emits them.

Blank text yields nothing but the terminating chunk — TTS filters can strip a reply down to whitespace, and that is not worth a round trip the service would reject.

close async

close()

Close the genai client's connection pool and drop the reference.

GeminiTTSConfig dataclass

GeminiTTSConfig(api_key, model='gemini-3.1-flash-tts-preview', voice='Kore', language=None, style_prompt=None, timeout=120.0, connect_timeout=5.0)

Configuration for the Gemini TTS provider.

Parameters:

Name	Type	Description	Default
api_key	str	Gemini API key (GEMINI_API_KEY).	required
model	str	One of :data:GEMINI_TTS_MODELS. The default is the only model that streams incrementally, which is what lets playback start before the whole clip is generated.	'gemini-3.1-flash-tts-preview'
voice	str	Prebuilt voice name — see :meth:GeminiTTSProvider.available_voices.	'Kore'
language	str | None	Optional BCP-47 hint (e.g. "fr-CA"). Left unset, the model infers the language from the text.	None
style_prompt	str | None	Natural-language delivery direction (e.g. "Read this in a calm, reassuring voice"). The API has no style field — style is only expressible inside the prompt — so this is written as a labelled Delivery direction: line above the Transcript: label in the same input string, which is what keeps the model reciting the transcript instead of the direction. Gemini 3.1 is a preview model and can occasionally read directions aloud anyway. For cues that steer a word or a phrase rather than the whole utterance, put audio tags such as [laughs] or [whispers] inline in the text itself.	None
timeout	float	Per-request timeout in seconds. Generous by design: measured time-to-first-audio for a one-sentence prompt ranges from ~1.2 s to ~8 s on the default model, and long text is slower still.	120.0
connect_timeout	float	TCP connect timeout in seconds, apart from timeout.	5.0

Usage

from roomkit.voice.tts.gemini import GeminiTTSConfig, GeminiTTSProvider

tts = GeminiTTSProvider(GeminiTTSConfig(
    api_key="your-gemini-api-key",
    model="gemini-3.1-flash-tts-preview",
    voice="Kore",
    style_prompt="Read this calmly and clearly",   # delivery guidance
))

Time to first audio is measured in seconds, so this fits prompts and generated audio messages rather than live turn-taking — see the STT & TTS Providers guide.

Install with: pip install roomkit[gemini]

RTP Backend

RTPVoiceBackend

RTPVoiceBackend(*, local_addr=('0.0.0.0', 0), remote_addr=None, payload_type=0, clock_rate=8000, dtmf_payload_type=101, rtcp_interval=5.0, port_allocator=None)

Bases: VoiceBackend

VoiceBackend that sends and receives audio over RTP.

Each :meth:connect call creates a new aiortp.RTPSession bound to the configured local address and sending to the remote address.

Inbound audio is decoded by aiortp (G.711, L16, Opus) and delivered as AudioFrame objects via the on_audio_received callback. DTMF digits are received out-of-band via RFC 4733 and delivered via on_dtmf_received callbacks.

Parameters:

Name	Type	Description	Default
local_addr	tuple[str, int]	(host, port) to bind RTP. Use port 0 for OS-assigned.	('0.0.0.0', 0)
remote_addr	tuple[str, int] | None	(host, port) to send RTP to. May be None if supplied per-session via metadata["remote_addr"] in :meth:connect.	None
payload_type	int	RTP payload type number (default 0 = PCMU).	0
clock_rate	int	Clock rate in Hz (default 8000 for G.711).	8000
dtmf_payload_type	int	RTP payload type for RFC 4733 DTMF events (default 101).	101
rtcp_interval	float	Seconds between RTCP sender reports.	5.0

send_audio_sync

send_audio_sync(session, chunk)

Synchronously send a single audio chunk via RTP.

Used by the audio bridge for low-latency frame forwarding.

send_transcription async

send_transcription(session, text, role='user')

Log transcription text (no UI channel in RTP mode).

on_dtmf_received

on_dtmf_received(callback)

Register a callback for inbound DTMF digits (RFC 4733).

Parameters:

Name	Type	Description	Default
callback	DTMFReceivedCallback	Function called with (session, dtmf_event).	required

Install with: pip install roomkit[rtp]

Mock Voice Backend

MockVoiceBackend

MockVoiceBackend(*, capabilities=NONE)

Bases: VoiceBackend

Mock voice backend for testing.

Tracks all method calls and provides helpers to simulate events. The backend is a pure transport — no VAD or audio intelligence.

Example

backend = MockVoiceBackend()

Track calls

session = await backend.connect("room-1", "user-1", "voice-1") assert backend.calls[-1].method == "connect"

Simulate raw audio received

frame = AudioFrame(data=b"audio-data") await backend.simulate_audio_received(session, frame)

Simulate barge-in

await backend.simulate_barge_in(session)

simulate_audio_received async

simulate_audio_received(session, frame)

Simulate the backend receiving a raw audio frame.

Fires all registered on_audio_received callbacks.

simulate_barge_in async

simulate_barge_in(session)

Simulate user speaking while TTS is playing (barge-in).

Fires all registered on_barge_in callbacks.

start_playing

start_playing(session)

Mark a session as playing audio (for barge-in testing).

stop_playing

stop_playing(session)

Mark a session as no longer playing audio.

simulate_session_ready async

simulate_session_ready(session)

Simulate the backend signalling that a session's audio path is live.

Fires all registered on_session_ready callbacks.

simulate_client_disconnected async

simulate_client_disconnected(session)

Simulate the backend signalling that a client has disconnected.

Fires all registered on_client_disconnected callbacks.

MockVoiceCall dataclass

MockVoiceCall(method, args=dict())

Record of a call made to MockVoiceBackend.

Voice Events

BargeInEvent dataclass

BargeInEvent(session, interrupted_text, audio_position_ms, timestamp=_utcnow())

User started speaking while TTS was playing.

This event is fired when the VAD detects speech starting while audio is being sent to the user. This allows the system to: - Cancel the current TTS playback - Adjust response strategy (e.g., acknowledge interruption) - Track conversation dynamics

session instance-attribute

session

The voice session where barge-in occurred.

interrupted_text instance-attribute

interrupted_text

The text that was being spoken when interrupted.

audio_position_ms instance-attribute

audio_position_ms

How far into the TTS audio playback (in milliseconds).

timestamp class-attribute instance-attribute

timestamp = field(default_factory=_utcnow)

When the barge-in was detected.

TTSCancelledEvent dataclass

TTSCancelledEvent(session, reason, text, audio_position_ms, timestamp=_utcnow())

TTS playback was cancelled.

This event is fired when TTS synthesis or playback is stopped before completion. Reasons include: - barge_in: User started speaking - explicit: Application called interrupt() - disconnect: Session ended - error: TTS or playback error

session instance-attribute

session

The voice session where TTS was cancelled.

reason instance-attribute

reason

Why the TTS was cancelled.

text instance-attribute

text

The text that was being synthesized.

audio_position_ms instance-attribute

audio_position_ms

How far into playback (0 if not started).

timestamp class-attribute instance-attribute

timestamp = field(default_factory=_utcnow)

When the cancellation occurred.

PartialTranscriptionEvent dataclass

PartialTranscriptionEvent(session, text, confidence, is_stable, role='user', language=None, timestamp=_utcnow())

Interim transcription result during speech.

This event is fired by backends that support streaming STT, providing real-time transcription updates before the final result. Use cases include: - Live captions/subtitles - Early intent detection - Visual feedback during speech

session instance-attribute

session

The voice session being transcribed.

text instance-attribute

text

The current transcription (may change in subsequent events).

confidence instance-attribute

confidence

Confidence score (0.0 to 1.0).

is_stable instance-attribute

is_stable

True if this portion is unlikely to change significantly.

role class-attribute instance-attribute

role = 'user'

Who spoke: 'user' (input) or 'assistant' (AI output).

language class-attribute instance-attribute

language = None

The language the STT provider reports for this partial, if any.

timestamp class-attribute instance-attribute

timestamp = field(default_factory=_utcnow)

When this transcription was received.

VADSilenceEvent dataclass

VADSilenceEvent(session, silence_duration_ms, timestamp=_utcnow())

Silence detected after speech.

This event is fired when the VAD detects a period of silence following speech. It can be used for: - Early end-of-utterance detection (before full speech_end) - Adaptive silence thresholds - Turn-taking management

session instance-attribute

session

The voice session where silence was detected.

silence_duration_ms instance-attribute

silence_duration_ms

Duration of silence in milliseconds.

timestamp class-attribute instance-attribute

timestamp = field(default_factory=_utcnow)

When the silence was detected.

VADAudioLevelEvent dataclass

VADAudioLevelEvent(session, level_db, is_speech, timestamp=_utcnow())

Periodic audio level update for UI feedback.

This event is fired periodically (typically 10Hz) to provide audio level information for UI visualization. Use cases include: - Audio level meters - Speaking indicators - Noise detection

session instance-attribute

session

The voice session.

level_db instance-attribute

level_db

Audio level in dB (typically -60 to 0, where 0 is max).

is_speech instance-attribute

is_speech

VAD's determination if this audio contains speech.

timestamp class-attribute instance-attribute

timestamp = field(default_factory=_utcnow)

When this measurement was taken.

Callback Types

Callback	Signature
SpeechStartCallback	(VoiceSession) -> Any
SpeechEndCallback	(VoiceSession, bytes) -> Any
PartialTranscriptionCallback	(VoiceSession, str, float, bool) -> Any
VADSilenceCallback	(VoiceSession, int) -> Any
VADAudioLevelCallback	(VoiceSession, float, bool) -> Any
BargeInCallback	(VoiceSession) -> Any