Audio Pipeline¶
Audio processing pipeline for voice channels. See the Audio Pipeline Stages guide for usage examples.
Pipeline¶
AudioPipeline ¶
AudioPipeline(config, *, backend_capabilities=NONE, backend_feeds_aec_reference=False)
Orchestrates audio frame processing through pipeline stages.
Inbound processing order
[Resampler] -> [Recorder tap] -> [DTMF] -> [AEC] -> [AGC] -> [Denoiser] -> [VAD] -> [Diarization]
Outbound processing order
[PostProcessors] -> [Recorder tap] -> AEC.feed_reference -> [Resampler]
AEC and AGC stages are skipped when the backend declares NATIVE_AEC / NATIVE_AGC capabilities.
on_speech_frame ¶
Register callback for processed audio frames during speech.
on_processed_frame ¶
Register callback for every processed inbound frame.
Fires after all pipeline stages (AEC, denoiser, VAD, etc.) for every frame, regardless of speech state. Used by continuous STT streaming when no local VAD is configured.
set_parent_span ¶
Set the parent span (VOICE_SESSION) for pipeline spans.
process_frame ¶
Process a single inbound audio frame through the pipeline.
Backwards-compatible alias for process_inbound().
process_inbound ¶
Process a single inbound audio frame through the pipeline.
[Resampler] -> [Recorder tap] -> [DTMF] -> [AEC] -> [AGC] ->
[Denoiser] -> [VAD] -> [Diarization]
process_inbound_stream ¶
Process a frame for a stream that has no VoiceSession behind it.
Same stages in the same order as :meth:process_inbound — there is one
implementation of the ordering and this shares it — but the result is
returned instead of fanned out to the registered callbacks, which are
typed on a VoiceSession.
This is what a conference lane calls. The alternative is to fabricate a VoiceSession per track, which keying the stages on a stream identity exists to make unnecessary.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
stream
|
str
|
Identity of the audio stream — the key the stages hold
their state under, and the one |
required |
frame
|
AudioFrame
|
The frame to process. |
required |
Returns:
| Type | Description |
|---|---|
InboundResult
|
The processed frame and the VAD event it produced, if any. |
release_stream ¶
Release everything a stream held, in the engine and in the stages.
A lane calls this when its track goes away. Stage state is keyed by stream and some of it is native memory, so a stream that is never released leaks for as long as the pipeline lives.
process_outbound ¶
Process a single outbound audio frame through the pipeline.
Thread-safe per session: concurrent calls for the same session (e.g. bridge forwarding + TTS) are serialized via a per-session lock to protect the recorder handle and other per-session state.
[PostProcessors] -> [Recorder tap] -> AEC.feed_reference ->
[Resampler]
enable_playback_aec_feed ¶
Mark that AEC reference is fed at playback time.
When called, process_outbound() skips its own
aec.feed_reference() to avoid double-feeding with
misaligned timing.
feed_aec_reference ¶
Feed an AEC reference frame directly (from speaker output).
Called by the backend's speaker callback at playback time so the AEC has time-aligned reference for echo cancellation.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
frame
|
AudioFrame
|
The audio frame the speaker is playing. |
required |
stream
|
str
|
Identity of the stream this playback belongs to — the session whose canceller should model this echo. |
required |
Thread-safety: may be called from the audio I/O thread. Uses
a separate resampler instance from process_outbound to
avoid thread-safety issues.
set_aec_active ¶
Track one playback source and update per-stream AEC activity.
Multiple sources can play into the same session concurrently. AEC is bypassed only after the final source stops. Its converged adaptive filter is preserved for the next playback turn; session teardown owns the destructive reset.
on_session_active ¶
Called when a voice session becomes active.
Cleans up stale state for this session and starts recording if configured.
on_session_ended ¶
Called when a voice session ends.
Releases the stages' state for this stream, then stops recording and debug taps if active.
close ¶
Release all pipeline resources.
Every provider is closed whatever became of the ones before it — the
providers are independent, and stopping at the first failure left
every provider after it open for good. What failed is raised together,
as an ExceptionGroup, once everything has been asked to close.
Raises:
| Type | Description |
|---|---|
ExceptionGroup
|
if any provider's |
AudioPipelineConfig
dataclass
¶
AudioPipelineConfig(vad=None, denoiser=None, diarization=None, postprocessors=list(), vad_config=None, aec=None, agc=None, agc_config=None, dtmf=None, dtmf_redaction=None, turn_detector=None, backchannel_detector=None, recorder=None, recording_config=None, interruption=None, resampler=None, contract=None, debug_taps=None, telemetry=None, inbound_dsp_threads=None)
Configuration for the audio processing pipeline.
All stages are optional. At least one provider should be set for the pipeline to be useful.
Typical combinations:
- VoiceChannel (STT path):
vad(+ optional denoiser/diarization) - RealtimeVoiceChannel (speech-to-speech):
denoiserand/ordiarization(VAD not needed — the AI provider handles turn detection)
diarization
class-attribute
instance-attribute
¶
Optional speaker diarization applied after VAD.
postprocessors
class-attribute
instance-attribute
¶
Optional postprocessors applied on the outbound path.
vad_config
class-attribute
instance-attribute
¶
Optional VAD-specific configuration override.
agc_config
class-attribute
instance-attribute
¶
Create the built-in SimpleAGCProvider when no explicit AGC is set.
dtmf
class-attribute
instance-attribute
¶
Optional DTMF tone detector (runs in parallel with main chain).
dtmf_redaction
class-attribute
instance-attribute
¶
Optional DTMF masking (RFC §17.6). When set, the digits the framework
itself exposes — frame metadata, and the redacted_digit carried to
ON_DTMF hooks — are masked. None leaves digits in the clear.
turn_detector
class-attribute
instance-attribute
¶
Optional post-STT turn completion detector.
backchannel_detector
class-attribute
instance-attribute
¶
Optional backchannel detector for semantic interruption strategy.
recording_config
class-attribute
instance-attribute
¶
Optional recording configuration.
interruption
class-attribute
instance-attribute
¶
Optional interruption (barge-in) configuration.
resampler
class-attribute
instance-attribute
¶
Optional resampler provider for format conversion.
contract
class-attribute
instance-attribute
¶
Optional input/output format contract.
debug_taps
class-attribute
instance-attribute
¶
Optional diagnostic audio capture at pipeline stage boundaries.
telemetry
class-attribute
instance-attribute
¶
Optional telemetry provider for pipeline metrics.
inbound_dsp_threads
class-attribute
instance-attribute
¶
Run the inbound stage chain on a thread pool of this size.
None (default) processes each frame on the caller's thread —
usually the event loop, which caps concurrent sessions at one core
and lets one slow stage delay every session. With a pool, frames of
one session stay strictly FIFO while sessions spread across the
workers; the native stages release the GIL, so the ceiling scales
with cores. See :class:roomkit.voice.pipeline.offload.InboundFrameOffload.
AudioFrame
dataclass
¶
A single frame of inbound audio for pipeline processing.
AudioFrame flows through the audio pipeline stages: denoiser -> VAD -> diarization. Each stage may annotate the metadata dict with its results.
This is distinct from AudioChunk, which is used for outbound TTS audio streaming.
VAD (Voice Activity Detection)¶
VADProvider ¶
Bases: ABC
Abstract base class for Voice Activity Detection providers.
process
abstractmethod
¶
Process an audio frame and optionally return a VAD event.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
frame
|
AudioFrame
|
The audio frame to analyse. |
required |
stream
|
str
|
Identity of the audio stream this frame belongs to. A provider keeps its state per stream: a voice session and a conference track are separate speakers, and letting one advance the other's detection state makes silence from one close the other's utterance. |
required |
Returns:
| Type | Description |
|---|---|
VADEvent | None
|
A VADEvent if a state transition occurred, else None. |
reset ¶
Drop a stream's state.
Called when the stream ends, so a long-running room does not accumulate the state of every speaker that ever joined.
VADConfig
dataclass
¶
Configuration for VAD processing.
silence_threshold_ms
class-attribute
instance-attribute
¶
Milliseconds of silence before triggering SPEECH_END.
speech_pad_ms
class-attribute
instance-attribute
¶
Padding added around detected speech segments.
min_speech_duration_ms
class-attribute
instance-attribute
¶
Minimum speech duration to trigger events.
extra
class-attribute
instance-attribute
¶
Provider-specific configuration.
VADEvent
dataclass
¶
Event produced by a VAD provider.
audio_bytes
class-attribute
instance-attribute
¶
Speech audio. Set on SPEECH_START (pre-roll buffer) and SPEECH_END (full accumulated speech including pre-roll).
duration_ms
class-attribute
instance-attribute
¶
Duration in milliseconds (speech or silence).
level_db
class-attribute
instance-attribute
¶
Audio level in dB (set on AUDIO_LEVEL).
VADEventType ¶
Bases: StrEnum
Types of VAD events.
MockVADProvider ¶
Bases: VADProvider
Mock VAD provider that returns a preconfigured sequence of events.
Example
from roomkit.voice.pipeline.vad.base import VADEvent, VADEventType
events = [ VADEvent(type=VADEventType.SPEECH_START), None, VADEvent(type=VADEventType.SPEECH_END, audio_bytes=b"audio"), ] vad = MockVADProvider(events=events)
Acoustic Echo Cancellation¶
AECProvider ¶
Bases: ABC
Abstract base class for Acoustic Echo Cancellation providers.
process
abstractmethod
¶
Remove echo from an audio frame.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
frame
|
AudioFrame
|
The captured audio frame (may contain echo). |
required |
stream
|
str
|
Identity of the audio stream this frame belongs to. A provider keeps its state per stream: a voice session and a conference track are separate speakers, and letting one advance the other's detection state makes silence from one close the other's utterance. |
required |
Returns:
| Type | Description |
|---|---|
AudioFrame
|
A new or modified AudioFrame with echo removed. |
feed_reference
abstractmethod
¶
Feed a reference (playback) frame for echo estimation.
Called on the outbound path so the AEC can model the echo.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
frame
|
AudioFrame
|
The outbound audio frame being played to speakers. |
required |
stream
|
str
|
Identity of the audio stream this reference belongs to —
the same key |
required |
set_active ¶
Enable or disable AEC processing (bypass mode).
When active is False, process() should pass audio
through without echo cancellation. Default is no-op (always
active).
set_stream_active ¶
Enable or disable AEC processing for one stream.
Providers with stream-local bypass state should override this method.
The default preserves compatibility with providers whose activation is
global by delegating to :meth:set_active.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
stream
|
str
|
Identity passed to :meth: |
required |
active
|
bool
|
Whether echo cancellation should run for the stream. |
required |
reset ¶
Drop a stream's state.
Called when the stream ends, so a long-running room does not accumulate the state of every speaker that ever joined.
WebRTCAECProvider ¶
WebRTCAECProvider(sample_rate=16000, channels=1, stream_delay_ms=0, enable_ns=False, enable_agc=False)
Bases: AECProvider
AEC provider backed by WebRTC AEC3.
WebRTC AEC3 is significantly more effective than Speex for real-world speaker+mic echo cancellation. It includes nonlinear echo suppression, double-talk detection, and comfort noise generation.
WebRTC requires exactly 10 ms audio frames. This provider handles chunking transparently — callers can pass any frame size and the provider will buffer and process in 10 ms increments.
When enable_ns or enable_agc is set, those capture effects use a
separate WebRTC processor after AEC and remain active while echo
cancellation is bypassed between playback turns.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
sample_rate
|
int
|
Audio sample rate in Hz (default 16000). |
16000
|
channels
|
int
|
Number of audio channels (default 1, mono). |
1
|
stream_delay_ms
|
int
|
Estimated delay between speaker output and mic capture in milliseconds. Helps the AEC align reference and capture for better cancellation. Default 0. |
0
|
enable_ns
|
bool
|
Also enable WebRTC noise suppression. Default False. |
False
|
enable_agc
|
bool
|
Also enable WebRTC automatic gain control. Default False. |
False
|
stream_delay_ms
property
¶
Configured render-to-capture delay in milliseconds.
A value of zero means that no platform delay has been supplied yet.
LocalAudioBackend uses this to seed AEC3 from the actual PortAudio
input and output latencies before playback begins.
set_stream_delay_ms ¶
Update the platform render-to-capture delay without resetting AEC.
set_active ¶
Enable or disable AEC processing.
When active is False, process() passes audio through
without echo cancellation (bypass mode). Call with True
when TTS playback starts, and False when it ends.
This compatibility method changes the default and every stream known
to the provider. Channel integrations should prefer
:meth:set_stream_active so concurrent sessions remain independent.
set_stream_active ¶
Enable or bypass echo cancellation for one playback stream.
feed_reference ¶
Feed a reference (playback / TTS) frame for echo modelling.
reset ¶
Drop this stream's processor, discarding its adaptive filter.
Playback boundaries use :meth:set_stream_active instead so the
learned hardware echo path survives. Reset is reserved for stream
teardown or a real format/device change.
SpeexAECProvider ¶
Bases: AECProvider
AEC provider backed by SpeexDSP's adaptive echo canceller.
Uses the split (asynchronous) API — speex_echo_playback() feeds
reference audio from the speaker, speex_echo_capture() processes
mic audio and returns echo-cancelled output. The split API maintains
an internal ring buffer that handles temporal misalignment between
when reference audio is played and when the echo arrives at the mic,
which is critical for real hardware with output latency.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
frame_size
|
int
|
Number of samples per frame. Must match the frames delivered by the pipeline (e.g. 320 for 20 ms at 16 kHz). |
320
|
filter_length
|
int
|
Echo-tail length in samples. Longer values can cancel more reverberation but use more CPU. A good default is 10× the frame size (e.g. 3200 samples = 200 ms at 16 kHz). |
3200
|
sample_rate
|
int
|
Audio sample rate in Hz. |
16000
|
feed_reference ¶
Feed a reference (playback / TTS) frame for echo modelling.
Calls speex_echo_playback() directly so the internal ring
buffer tracks the speaker output timing.
Automatic Gain Control¶
AGCConfig
dataclass
¶
AGCConfig(target_level_dbfs=-3.0, max_gain_db=30.0, attack_ms=10.0, release_ms=100.0, metadata=dict())
Configuration for Automatic Gain Control.
target_level_dbfs
class-attribute
instance-attribute
¶
Target output level in dBFS.
attack_ms
class-attribute
instance-attribute
¶
Attack time in milliseconds (how quickly gain increases).
release_ms
class-attribute
instance-attribute
¶
Release time in milliseconds (how quickly gain decreases).
metadata
class-attribute
instance-attribute
¶
Provider-specific configuration.
AGCProvider ¶
Bases: ABC
Abstract base class for Automatic Gain Control providers.
process
abstractmethod
¶
Apply gain control to an audio frame.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
frame
|
AudioFrame
|
The audio frame to normalise. |
required |
stream
|
str
|
Identity of the audio stream this frame belongs to. A provider keeps its state per stream: a voice session and a conference track are separate speakers, and letting one advance the other's detection state makes silence from one close the other's utterance. |
required |
Returns:
| Type | Description |
|---|---|
AudioFrame
|
A new or modified AudioFrame with gain applied. |
reset ¶
Drop a stream's state.
Called when the stream ends, so a long-running room does not accumulate the state of every speaker that ever joined.
SimpleAGCProvider ¶
Bases: AGCProvider
Adaptive RMS-based gain control for PCM16 audio.
The controller measures each frame's RMS level, moves a stream-local gain
toward AGCConfig.target_level_dbfs using the configured attack/release
time constants, and applies a peak limiter before converting back to PCM.
Near-silence is never amplified, which avoids turning an idle microphone's
noise floor into apparent speech.
Provider-specific settings may be supplied through AGCConfig.metadata:
silence_threshold_dbfs(default-60): frames below this RMS level pass through at unity gain.min_gain_db(default-30): maximum attenuation.
MockAGCProvider ¶
Denoiser¶
DenoiserProvider ¶
Bases: ABC
Abstract base class for audio denoising providers.
process
abstractmethod
¶
Denoise an audio frame.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
frame
|
AudioFrame
|
The noisy audio frame. |
required |
stream
|
str
|
Identity of the audio stream this frame belongs to. A provider keeps its state per stream: a voice session and a conference track are separate speakers, and letting one advance the other's detection state makes silence from one close the other's utterance. |
required |
Returns:
| Type | Description |
|---|---|
AudioFrame
|
A new or modified AudioFrame with reduced noise. |
reset ¶
Drop a stream's state.
Called when the stream ends, so a long-running room does not accumulate the state of every speaker that ever joined.
AICousticsDenoiserConfig
dataclass
¶
AICousticsDenoiserConfig(model='quail-vf-2.0-l-16khz', model_dir='./models', license_key='', enhancement_level=0.8, num_channels=1, sample_rate=16000)
Configuration for the ai|coustics Quail denoiser.
Attributes:
| Name | Type | Description |
|---|---|---|
model |
str
|
Model identifier for download (e.g. |
model_dir |
str
|
Local cache directory for downloaded models. |
license_key |
str
|
SDK license key. Defaults to the |
enhancement_level |
float
|
Enhancement strength from 0.0 (off) to 1.0 (maximum). 0.8 gives the best WER for voice AI workloads. |
num_channels |
int
|
Number of audio channels (1 = mono, 2 = stereo). |
sample_rate |
int
|
PCM sample rate expected by the selected model. |
AICousticsDenoiserProvider ¶
Bases: DenoiserProvider
Denoiser provider using ai|coustics Quail speech enhancement.
The processor is created lazily on the first call to :meth:process.
aic-sdk must be installed (pip install roomkit[aicoustics]).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
config
|
AICousticsDenoiserConfig | None
|
Provider configuration. |
None
|
process ¶
Denoise an audio frame using Quail speech enhancement.
Buffers incoming PCM to match the SDK's expected frame size, then processes complete chunks. Any remainder is held for the next call.
reset ¶
Drop this stream's processor and its buffer.
The next frame for this stream builds a fresh processor, which is what clears the model's recurrent state.
RNNoiseDenoiserProvider ¶
Bases: DenoiserProvider
Denoiser provider backed by RNNoise (Mozilla/Xiph).
RNNoise is a recurrent neural network that suppresses stationary and non-stationary noise in real time. Internally it operates at 48 kHz with 480-sample float32 frames. When the pipeline delivers 16 kHz audio the provider up-samples before processing and down-samples afterward (exact 1:3 ratio).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
sample_rate
|
int
|
Expected input sample rate. Supports 16000, 24000, and 48000 Hz. |
16000
|
process ¶
Denoise an audio frame.
Exact chunks are processed without added delay. Once an irregular chunk is observed, a fixed 10 ms output delay preserves byte-for-byte timeline continuity while complete native chunks are accumulated.
SherpaOnnxDenoiserProvider ¶
Bases: DenoiserProvider
Denoiser provider using sherpa-onnx GTCRN speech enhancement.
The denoiser is created lazily on the first call to :meth:process.
sherpa-onnx must be installed (pip install roomkit[sherpa-onnx]).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
config
|
SherpaOnnxDenoiserConfig
|
Provider configuration. |
required |
WebRTCNoiseSuppressorProvider ¶
Bases: DenoiserProvider
WebRTC noise suppression that runs continuously on the inbound path.
WebRTC consumes exact 10 ms PCM16 blocks. Arbitrary caller chunk sizes are converted to a fixed one-block-delay stream so every input byte has exactly one output byte and buffered audio is never emitted twice.
MockDenoiserProvider ¶
Bases: DenoiserProvider
Mock denoiser that passes frames through unchanged.
Tracks processed frames for test assertions.
Diarization¶
DiarizationProvider ¶
Bases: ABC
Abstract base class for speaker diarization providers.
process
abstractmethod
¶
Analyse an audio frame for speaker identity.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
frame
|
AudioFrame
|
The audio frame to analyse. |
required |
stream
|
str
|
Identity of the audio stream this frame belongs to. A provider keeps its state per stream: a voice session and a conference track are separate speakers, and letting one advance the other's detection state makes silence from one close the other's utterance. |
required |
Returns:
| Type | Description |
|---|---|
DiarizationResult | None
|
A DiarizationResult if a speaker was identified, else None. |
reset ¶
Drop a stream's state.
Called when the stream ends, so a long-running room does not accumulate the state of every speaker that ever joined.
clear_speakers ¶
Forget every enrolled speaker.
Unlike :meth:reset (which clears transient clustering state), this
drops the enrollment set so a provider reused across sessions does not
carry speakers from a previous conversation into the next one.
Providers with no enrollment concept may leave this as a no-op.
DiarizationResult
dataclass
¶
Result from a diarization provider.
MockDiarizationProvider ¶
Bases: DiarizationProvider
Mock diarization provider that returns a preconfigured sequence of results.
Example
results = [ DiarizationResult(speaker_id="speaker_0", confidence=0.9, is_new_speaker=True), DiarizationResult(speaker_id="speaker_0", confidence=0.95, is_new_speaker=False), ] diarizer = MockDiarizationProvider(results=results)
TTS Stream Filters¶
TTSStreamFilter ¶
Bases: ABC
Base class for TTS text filters.
Supports both streaming (chunk-by-chunk via :meth:feed/:meth:flush)
and non-streaming (full text via :meth:__call__) usage.
Subclasses must implement :meth:feed, :meth:flush, and
:meth:reset. The default :meth:__call__ delegates to feed+flush
but subclasses may override it with a more efficient implementation
(e.g. a single regex pass).
StripBrackets ¶
Bases: TTSStreamFilter
Strip all [...] bracketed content from TTS text.
A simpler variant that catches markers like [Respond in French],
[laughs], [thinking], etc.
StripInternalTags ¶
Bases: TTSStreamFilter
Strip [internal]...[/internal] and [internal: ...] blocks.
Handles two formats that AI models commonly produce:
- Paired tags:
[internal]reasoning here[/internal] spoken text - Single bracket:
[internal: reasoning here] spoken text
In streaming mode, buffers text when [internal is detected and
discards everything up to the matching close. Text outside tags is
passed through immediately.
In non-streaming mode (__call__), a single regex removes all
tagged blocks.
Events & Callbacks¶
SpeakerChangeEvent
dataclass
¶
Speaker change detected by diarization.
This event is fired when the audio pipeline's diarization stage detects a different speaker than the previous frame.
BargeInCallback
module-attribute
¶
BargeInCallback = Callable[[VoiceSession], Any]
Callback for barge-in detection: (session).