RFC: RoomKit — Multi-Channel Conversation Framework

Status	Draft
Author	Sylvain Boily
Contributions	TchatNSign, Angany AI
Version	v18 Draft
Created	2026-01-27
Last Updated	2026-10-10
Supersedes	v17 Draft

---

Abstract

RoomKit is a specification for a multi-channel conversation framework that unifies humans, AI agents, and programs in shared conversation spaces called Rooms.

This document defines the core conversation machinery: data models, channel abstractions, processing pipelines, permission model, hook engine, and identity resolution. It specifies the realtime media architecture — the voice pipeline, video, speech-to-speech, SFU-based conferencing, and recording — and the surfaces an AI agent is built on: multi-agent orchestration, memory, tool access control, delivery strategies, task delegation, skills, and image generation. Storage, observability, integration surfaces (REST, MCP), resilience patterns, and conformance levels complete what constitutes a conforming implementation.

The specification is language-agnostic. Implementations MAY be written in any programming language. All examples use pseudocode or structured notation.

---

Table of Contents

1. Conventions and Terminology
2. Introduction
3. Architecture Overview
4. Core Concepts
5. Data Models
6. Channel System
7. Permission Model
8. Event System
9. Hook System
10. Processing Pipelines
11. Identity Resolution
12. Voice and Realtime Media
13. Resilience and Error Handling
14. Storage Interface
15. Observability
16. Integration Surfaces
17. Security Considerations
18. Design Principles
19. Multi-Agent Orchestration
20. Memory System
21. Tool Access Control
22. Delivery Strategies
23. Task Delegation
24. Skills Framework
25. Image Generation
26. Conformance Levels
27. Appendix A: Channel Reference
28. Appendix B: Complete Event Flow Examples

---

1. Conventions and Terminology

1.1 Key Words

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119.

1.2 Definitions

Term	Definition
Room	A conversation space where channels connect and events flow. The unit of state.
Channel	Any entity that interacts with a Room — transports messages, generates AI responses, or observes events.
Event	A record of something that happened in a Room (message, system event, status change). Once committed, its index and status never change (§4.3).
Participant	A human (or system identity) taking part in a Room conversation.
Identity	A cross-channel representation of a person, linking addresses across channel types.
Provider	An interchangeable implementation behind a channel (e.g., Twilio behind SMS).
Source	A persistent connection that pushes inbound events (as opposed to per-message webhook callbacks).
Hook	A pluggable function that intercepts, blocks, modifies, or reacts to events.
Binding	The attachment of a channel to a room, including permissions and metadata.
Side Effect	A task, observation, or room-metadata update produced by a channel or hook, not subject to permission restrictions.
Chain Depth	The number of response hops between an event and the external inbound event that started its chain (AI responds to AI responds to...).
Integrator	The developer building an application on top of a RoomKit implementation.
Broadcast	The act of routing an event to all eligible channels in a Room.
Transcoding	Converting event content from one format to another for cross-channel delivery.
Audio Pipeline	A configurable chain of audio processing stages (resampler, AEC, AGC, denoiser, VAD, diarization, DTMF, recording) between the transport and the conversation engine.
ACP	Agent Client Protocol — a protocol for communication between code editors or conversation hosts (clients) and coding agents (servers).
AEC	Acoustic Echo Cancellation — removes the bot's own audio from the inbound stream to prevent self-triggering.
AGC	Automatic Gain Control — normalizes audio volume to a consistent level regardless of input device or distance.
DTMF	Dual-Tone Multi-Frequency — telephone keypad tones used for IVR navigation and call control.
PLC	Packet Loss Concealment — synthesizing replacement audio for packets confirmed lost in transit, preserving the temporal continuity of the inbound stream.
Turn Detection	The process of determining whether a speaker has finished their conversational turn, using acoustic and/or semantic signals.
Backchannel	Short verbal acknowledgments ("mmhmm", "ok", "yes") that signal attention without requesting a turn change.
Full-Duplex Provider	A speech-to-speech provider whose model listens and speaks at the same time, manages overlap itself, and exposes no response or speech boundaries on the wire (Section 12.4.1).
Reasoning Delegation	A full-duplex provider's model handing reasoning and tool use to a backend model — hosted by the provider or supplied by the integrator — while it keeps the conversation going. Distinct from Task Delegation (Section 23).
Protocol Trace	An immutable record of a transport-level protocol exchange (e.g., SIP INVITE, 200 OK, BYE) emitted by a channel for observability and debugging.
Agent	An AIChannel subclass with structured identity metadata (role, description, scope, voice) for multi-agent orchestration.
Orchestration	The system that routes events to the correct agent, manages conversation phases, and handles agent-to-agent handoffs.
Handoff	The transfer of conversation responsibility from one agent to another, with context preservation.
ConversationState	Per-room state tracking the current phase, active agent, and orchestration metadata.
Memory Provider	A pluggable strategy for constructing AI conversation history (sliding window, summarization, retrieval, etc.).
Tool Policy	Access control rules (allow/deny glob patterns with role overrides) governing which tools an AI agent can invoke.
Steering Directive	A mid-generation instruction (Cancel, UpdateSystemPrompt, InjectMessage) injected into an active AI tool loop.
Delivery Strategy	A strategy controlling when proactive content (task results, notifications) is delivered into a conversation (Immediate, WaitForIdle, Queued).
Delegation	Dispatching background work to a specialized agent in an isolated child room, with result collection and delivery.
Skill	A self-contained, discoverable package of agent instructions, reference materials, and scripts defined by a SKILL.md file.
Video Pipeline	A configurable chain of video processing stages (decoder, resizer, transform, filter, overlay) between the transport and the vision/recording engine.
Vision Provider	A pluggable service that analyzes video frames and produces structured results (descriptions, labels, faces, OCR text).
Avatar Provider	A service that generates lip-synced video frames from TTS audio for visual agent embodiment.
Video Bridge	A component that forwards video frames between sessions in the same room, with keyframe buffering and PLI requests.
StatusBus	A shared event log for inter-agent coordination, allowing agents to post and subscribe to status updates without sending room events.
Tool Auditor	A pluggable recorder that captures every tool call (input, output, timing, status) for debugging and monitoring.
Session Auditor	A pluggable recorder that captures the full conversation timeline (speech, tool calls, vision, interruptions) for compliance and replay.
SFU	Selective Forwarding Unit — a media server that routes encoded media streams between conference participants without decoding or mixing them.
Conference	A multi-party real-time media session (audio, video, screen share) attached to a Room, whose media plane is owned by an external SFU.
Track	A single media stream (audio, video, or screen share) published by a conference participant. The unit of media identity in a conference.
Bot Participant	The framework's own server-side connection to a conference, used to subscribe to tracks (STT, vision, recording) and publish media (TTS, avatar).
Egress	Server-side media export performed by the SFU (e.g., conference recording), as opposed to framework-side recording.

1.3 Notation

Data models are described using structured notation:

ModelName
├── field_name: Type                    # Required field
├── field_name: Type = default_value    # Field with default
├── field_name: Type | null             # Nullable field
└── field_name: map<string, any>        # Dictionary/map type

Enumeration values are written in UPPER_SNAKE_CASE. Field names use snake_case.

---

2. Introduction

2.1 Problem Statement

Modern conversations span multiple channels — a customer starts on SMS, continues on WhatsApp, while an AI assistant and human advisor collaborate behind the scenes. Each channel has different capabilities, protocols, and constraints. Building applications that manage these multi-channel conversations requires solving the same set of problems repeatedly: message routing, permission management, event ordering, identity resolution, and channel abstraction.

Once an AI agent is one of the participants rather than a feature bolted onto the side, a second set of problems appears and is solved just as repeatedly: what the agent is allowed to see of a conversation it shares with humans, which tools it may reach and when, which instructions bind it and for how long, and how its turn interleaves with everyone else's.

2.2 Scope

RoomKit provides primitives for multi-channel conversations, not business logic.

Where an AI agent is one of the participants, RoomKit is the harness around that agent: it defines where the agent sits in a conversation, what it may see, what it may call, what rules bind it, and how its output reaches the humans on every other channel. It does not reason, and it does not decide what the agent is for.

RoomKit IS:

• A room-based conversation manager
• A unified channel abstraction (SMS, Email, AI, Voice, Video — same interface)
• A permission system (access, mute, visibility)
• A hook engine (intercept, block, modify, enrich)
• A provider abstraction layer (channel type ≠ provider)
• An identity resolution pipeline
• A realtime media architecture (voice pipeline, speech-to-speech, SFU conferencing, recording)
• A harness for AI participation (orchestration, memory, tool access control, task delegation, skills) — the surfaces an agent runs on, not its reasoning

RoomKit is NOT:

• A CPaaS provider (Twilio, Sinch, etc. own the transport)
• An LLM runtime (inference, prompting, and reasoning belong to the model library)
• A unified communications platform (no PBX, presence, directory, or end-user client — RoomKit is a component that complements such a platform, not a replacement for it and not the product)
• A chat application (RoomKit provides primitives; integrators build apps)
• Opinionated about when or why to use its primitives

2.3 Design Philosophy

The framework provides primitives. The integrator provides business logic.

FRAMEWORK provides                 INTEGRATOR decides
─────────────────────              ────────────────────
Channel access primitives          When to set each access level
Mute / unmute operations           When to mute or unmute
Visibility rules                   What visibility each channel gets
Attach / detach channels           Which channels to attach when
Hook pipeline (sync/async)         What hook handlers to register
Hook block + inject                What to block and what to inject
Event routing and permissions      Room setup and configuration
Two output paths                   What tasks and observations to create
Provider abstraction               Which provider to use
Identity resolution interface      Resolution strategy
Storage interface                  Storage implementation
Event chain depth limit            Turn budgets and orchestration
Audio pipeline stages              Which resampler, AEC, AGC, denoiser, VAD, etc. to use
Tool policy and skill gating       Which tools and skills an agent gets
Skill activation lifecycle         What a skill's instructions actually say
Memory interface                   What is worth remembering

---

3. Architecture Overview

A conforming RoomKit implementation consists of the following layers:

┌─────────────────────────────────────────────────────────────────────┐
│                     Integration Surfaces                            │
│                                                                     │
│   ┌───────────────┐    ┌───────────────┐                           │
│   │   REST API    │    │  MCP Server   │                           │
│   │  (humans,     │    │  (AI agents,  │                           │
│   │   systems)    │    │   tools)      │                           │
│   └───────┬───────┘    └───────┬───────┘                           │
└───────────┼────────────────────┼───────────────────────────────────┘
            │                    │
            ▼                    ▼
┌─────────────────────────────────────────────────────────────────────┐
│                          RoomKit Core                               │
│                                                                     │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐              │
│  │    Room       │  │    Event     │  │   Identity   │              │
│  │   Manager     │  │   Router     │  │   Resolver   │              │
│  └──────────────┘  └──────┬───────┘  └──────────────┘              │
│                           │                                         │
│                    ┌──────┴───────┐                                  │
│                    │  Hook Engine │                                  │
│                    └──────┬───────┘                                  │
│                           │                                         │
│  ┌──────────────────────────────────────┐                           │
│  │       Conversation Store (ABC)       │                           │
│  │    (Rooms, Events, Identities)       │                           │
│  └──────────────────────────────────────┘                           │
└────────────────────────┬────────────────────────────────────────────┘
                         │
              Channel Interface (unified for ALL)
                         │
      ┌──────────┬───────┴──┬──────────┬──────────┬──────────┐
      ▼          ▼          ▼          ▼          ▼          ▼
 ┌────────┐┌────────┐┌─────────┐┌──────────┐┌────────┐┌─────────┐
 │  SMS   ││ Email  ││  HTTP/  ││    AI    ││ Voice  ││ Custom  │
 │Channel ││Channel ││  WS     ││ Channel  ││Channel ││ Channel │
 └───┬────┘└───┬────┘└─────────┘└────┬─────┘└───┬────┘└─────────┘
     │         │                     │           │
  Provider  Provider              Provider    Provider
  Layer     Layer                 Layer       Layer
     │         │                     │           │
     ▼         ▼                     ▼           ▼
 ┌────────┐┌────────┐          ┌─────────┐┌──────────┐
 │Twilio  ││Elastic │          │Anthropic││ Deepgram │
 │Sinch   ││  Email │          │OpenAI   ││ElevenLabs│
 │Telnyx  ││SendGrid│          │Gemini   ││SherpaONNX│
 │VoiceMeUp│         │          │Mistral  ││ Google   │
 └────────┘└────────┘          │XAI/Grok ││ OpenAI   │
                               │Azure    ││ Gradium  │
                               │vLLM     ││ Qwen     │
                               └─────────┘└──────────┘

3.1 Layer Responsibilities

Layer	Responsibility
Integration Surfaces	REST API for humans/systems, MCP for AI agents; real-time clients connect through the WebSocket channel (Section 6.3, Appendix A.8) on a socket the host serves
RoomKit Core	Room lifecycle, event routing, hooks, permissions, identity, store
Channel Interface	Unified abstraction — every channel implements the same interface
Provider Layer	Interchangeable implementations behind channels

3.2 Key Separations

1. Channel type and Provider are separate. SMS is a channel type. Twilio is a provider. Swap providers without changing room logic.
2. Core and Integration Surfaces are separate. Core has no web framework dependency. REST/MCP are thin wrappers; real-time clients reach the core through the WebSocket channel.
3. Framework and Business Logic are separate. Framework provides primitives. Integrator registers hooks and configures channels.
4. Transport and Audio Processing are separate. The transport delivers raw audio frames. The audio pipeline (resampler, AEC, AGC, denoiser, VAD, diarization, DTMF, recording) is a distinct layer with pluggable providers, independent of the transport choice.

---

4. Core Concepts

4.1 The Room

A Room is a conversation space where channels connect and events flow through.

Properties:

• Channel-agnostic — A Room does not know whether it carries SMS or AI traffic.
• Multi-channel — SMS + WebSocket + AI + Observer simultaneously.
• Dynamic — Channels can be attached, detached, muted, or reconfigured at any time.
• Observable — Hooks see everything in the Room; a channel sees what its access and each event's visibility allow (§7.5).
• Persistent — Rooms survive session boundaries, channel switches, and escalations.

A Room holds no message content directly. Content lives in Events stored in the Room's timeline.

4.2 The Channel

Everything that interacts with a Room is a Channel. SMS, Email, WebSocket, AI, Voice, Observer — all implement the same interface.

Channels are classified along two dimensions:

Category — what the channel does:

Category	Purpose	Examples
TRANSPORT	Carries messages to/from external systems or humans	SMS, Email, WhatsApp, WebSocket, Voice
INTELLIGENCE	Processes events and produces responses or insights	AI agent, sentiment analyzer

Direction — what the channel can do physically:

Direction	Meaning
INBOUND	Can receive from outside only
OUTBOUND	Can send to outside only
BIDIRECTIONAL	Both receive and send

Three methods, three concerns:

Method	When Called	Direction
handle_inbound()	External payload arrives (webhook, WebSocket message, source push)	INBOUND
deliver()	Framework needs to push an event to the channel's external recipient	OUTBOUND
on_event()	A room event occurs and this channel has read access	READ

Direction declares capability. Permissions restrict per room.

4.3 The Event

Everything in a Room is a RoomEvent — messages, system notifications, typing indicators, channel state changes, participant joins/leaves. Once stored, an event keeps its index and status. Its content and source change only through §10.3 (an EDIT event or a direct update_event()); its metadata, delivery_results and consumption can also be written afterwards, by §10.3 and by the framework's own records of what became of it (a delivery that failed, the end of a streamed turn, a response cancelled as superseded, a response read or heard, §8.5); a direct delete_event() removes it (§10.3). Events are sequentially indexed within their room.

4.4 The Participant

A Participant is a human (or system identity) in the Room. AI channels and integration channels are NOT participants — they are infrastructure. Participants have roles, identity status, and are connected via one or more channels.

4.5 The Identity

An Identity is a cross-channel representation of a person. One person may be reachable via SMS (+15551234), Email (john@example.com), and WhatsApp simultaneously. Identity resolution maps channel addresses to a unified identity.

4.6 Two Output Paths

When a channel or hook processes an event, it produces output along two distinct paths:

Channel/Hook Output
├── Room Events (messages, responses)
│   ├── Subject to permissions (access, mute, visibility)
│   ├── Broadcast according to the binding's visibility
│   └── Stored in timeline
│
└── Side Effects (ALWAYS allowed, even when muted)
    ├── Tasks (actionable work items)
    ├── Observations (passive insights)
    └── Metadata updates (enrich room data)

Muting silences the voice, not the brain. A muted AI channel cannot produce room events but CAN produce tasks ("customer at risk of churning") and observations ("sentiment: negative").

---

5. Data Models

5.1 Room

Room
├── id: string                              # Unique identifier
├── organization_id: string | null          # Multi-tenant isolation
├── status: RoomStatus                      # Current lifecycle state
├── created_at: datetime                    # When the room was created
├── updated_at: datetime                    # Last modification time
├── closed_at: datetime | null              # When the room was closed
├── timers: RoomTimers                      # Auto-transition configuration
├── agent_response_policy: AgentResponsePolicy = AGENT_CHAIN  # What an agent's own output solicits (Section 19.3.1)
├── metadata: map<string, any>              # Integrator-defined data
├── event_count: int                        # Total events stored
└── latest_index: int                       # Highest event index (for read tracking)

RoomStatus enumeration:

Value	Meaning	Accepts new events
ACTIVE	Room is active; channels can send and receive	Yes
PAUSED	Room is paused (e.g., after inactivity timer); can be resumed	Yes
CLOSED	Room is closed	MUST be refused
ARCHIVED	Room is archived; read-only historical access	MUST be refused

A room's status governs whether it will accept another event, and implementations MUST enforce that at every point where the timeline can grow — an inbound message, direct injection (§10.5), a hook's injected event, the framework's own re-injection, and a record committed outside the pipeline (§10.5) alike. Enforcing it only where an inbound router happens to look is not conformant: a caller that names the room explicitly bypasses the router entirely, and so does the framework when it re-injects into a room it already knows.

A streamed response writes many rows, and reading the room's status before each one costs a store read per row. A stream MAY instead read the status once, before its first row, and read it again only once the framework instance running it has changed a room's status since; it checks the status again after a row's BEFORE_BROADCAST hooks, which run without the room lock. A status changed elsewhere (another process, another framework instance sharing the store, a direct write to the store) is then seen by the next response, not mid-stream.

A refused event MUST NOT be appended to the timeline, MUST NOT be broadcast, and MUST NOT be stored as a BLOCKED event: a closed room accepts nothing, and an audit record written into it would be the very thing the status forbids. The framework MUST return a blocked result naming the refusal (reason = room_closed) and, when what it turned away is an inbound event, a response re-entering the room or a regeneration, MUST emit the room_refused_event framework event (§8.2) so the condition is observable — a room that has gone quiet because it closed is otherwise indistinguishable from one whose integration has broken. The framework's own lifecycle records (SystemContent, §5.3) and the events an identity challenge injects (§11) MAY be refused without an event: neither is a message a subscriber could be waiting for.

One exception, and only one: the framework's own record of the transition — the lifecycle hook, the framework event, and any timeline entry an implementation chooses to write to mark it — is produced by the transition itself rather than through the pipeline, and is not subject to this refusal. Closing a room MUST remain observable: a room that simply goes quiet, with no signal of why, is indistinguishable from one whose integration has broken.

CLOSED and ARCHIVED refuse identically; they differ in intent. CLOSED is a conversation that has ended and that the integrator MAY reopen by returning the room to ACTIVE. ARCHIVED is a terminal storage state.

Reading is unaffected by status: history, participants and bindings remain readable at every status, and an implementation MUST NOT make a closed room unreadable.

Closing a room does not, by itself, sever what is attached to it. Whether close_room() detaches channels, ends voice sessions or leaves them in place is an implementation's choice; what it MUST NOT do is leave a path by which a still-attached channel can append to the closed room.

RoomTimers:

RoomTimers
├── inactive_after_seconds: int | null      # Seconds of inactivity before ACTIVE → PAUSED
├── closed_after_seconds: int | null        # Seconds in PAUSED before PAUSED → CLOSED
└── last_activity_at: datetime | null       # Timestamp of last event

Implementations MUST enforce timer transitions when configured. When inactive_after_seconds elapses without a new event, the room MUST transition to PAUSED. When closed_after_seconds elapses in PAUSED without resumption, the room MUST transition to CLOSED.

5.2 RoomEvent

RoomEvent
├── id: string                              # Unique event identifier
├── room_id: string                         # Owning room
├── type: EventType                         # What kind of event
├── source: EventSource                     # Where it came from
├── content: EventContent                   # Normalized payload (discriminated union)
├── status: EventStatus                     # Delivery outcome
├── blocked_by: string | null               # Hook name if blocked
├── visibility: string                      # Who can see this event
├── addressed_to: list<string> | null       # Intelligence channels asked to act (§19.3)
├── index: int >= 0                         # Sequential position in room timeline
├── chain_depth: int >= 0                   # Response chain depth (loop prevention)
├── parent_event_id: string | null          # In-app thread: the root of its thread
├── responds_to: string | null              # The event this response answers (§8.5)
├── consumption: map<string, Consumption>   # Per delivering channel: taken in? (§8.5)
├── correlation_id: string | null           # Integrator's external reference
├── idempotency_key: string | null          # Duplicate prevention key
├── created_at: datetime                    # When the event was created
├── metadata: map<string, any>              # Integrator-defined data
├── channel_data: ChannelData               # Provider-specific structured metadata
└── delivery_results: map<string, any>      # Per-channel delivery outcomes

EventType enumeration:

Value	Category	Description
MESSAGE	Core	Text, media, or rich content message
SYSTEM	Core	Framework-generated notification
INSTRUCTION	Directive	The application directs an agent; delivered to it, never stored (§10.1.1)
TYPING	Ephemeral	User is typing
READ_RECEIPT	Status	User has read up to an index
DELIVERY_RECEIPT	Status	Provider confirms delivery
PRESENCE	Ephemeral	User online/offline/away
REACTION	Content	Emoji reaction to another event
EDIT	Content	Edit of a previous message
DELETE	Content	Deletion of a previous message
PARTICIPANT_JOINED	Lifecycle	Participant entered the room
PARTICIPANT_LEFT	Lifecycle	Participant left the room
PARTICIPANT_IDENTIFIED	Lifecycle	Pending participant was identified
PARTICIPANT_UPDATED	Lifecycle	Participant presentation changed (display name)
CHANNEL_ATTACHED	Lifecycle	Channel was attached to room
CHANNEL_DETACHED	Lifecycle	Channel was detached from room
CHANNEL_MUTED	Lifecycle	Channel was muted
CHANNEL_UNMUTED	Lifecycle	Channel was unmuted
CHANNEL_UPDATED	Lifecycle	Channel binding was modified (access/visibility)
DTMF	Voice	DTMF tone detected (keypad digit)
RECORDING_STARTED	Voice	Audio recording started for a session
RECORDING_STOPPED	Voice	Audio recording stopped, result available
TASK_CREATED	Side effect	A task was created
TASK_DELEGATED	Side effect	A task was delegated to a child room; payload of ON_TASK_DELEGATED, never stored (Section 23.5)
TASK_COMPLETED	Side effect	A delegated task ended; payload of ON_TASK_COMPLETED, never stored (Section 23.5)
OBSERVATION	Side effect	An observation was recorded
TOOL_CALL_START	Activity	A tool call started (ToolCallContent, status pending)
TOOL_CALL_END	Activity	A tool call ended (ToolCallContent, with its status and outcome)

EventStatus enumeration:

Value	Meaning
PENDING	Event created, not yet processed
DELIVERED	Event successfully stored and broadcast
READ	Event read by recipient (from read receipt)
FAILED	Delivery failed after all retries
BLOCKED	Event refused: by a sync hook, a source that cannot write (§7.5 rule 2), or the chain-depth limit (§8.3)

EventSource:

EventSource
├── channel_id: string                      # Which channel produced this
├── channel_type: ChannelType               # Channel type enum
├── direction: ChannelDirection             # INBOUND or OUTBOUND
├── participant_id: string | null           # Which human, if applicable
├── verification: VerificationStatus | null # The sender's verification when the
│                                           # event was received (§11.7); null where
│                                           # none applies
├── external_id: string | null              # External system reference
├── provider: string | null                 # Provider/backend name for event attribution
├── raw_payload: map<string, any>           # Original provider payload — never lost
└── provider_message_id: string | null      # Provider's message identifier

Implementations MUST preserve raw_payload unmodified. This is the audit trail and the source of truth for provider-specific data.

EventSource.provider population: Every channel MUST populate the provider field when constructing an EventSource. The value SHOULD be the name of the underlying provider or backend — for example, "SIP" for a SIP voice backend, "SIPRealtimeTransport" for a SIP realtime transport, "TwilioSMS" for an SMS provider. The channel SHOULD expose a provider_name property that returns the appropriate name. System-generated events (channel_id="system") MAY leave provider as null.

5.3 Event Content

Event content is a discriminated union — each event carries exactly one content type. Implementations MUST support all content types defined here. Each content type carries a type field that names it and discriminates the union: text, rich, media, location, audio, video, composite, system, template, edit, delete or tool_call.

TextContent — Plain text message:

TextContent
├── body: string                            # Message text
└── language: string | null                 # ISO 639-1 language code

RichContent — Formatted text with interactive elements:

RichContent
├── body: string                            # Primary text, in `format`
├── format: "html" | "markdown" = "markdown" # Markup of body
├── plain_text: string | null               # Plain text fallback
├── buttons: list<map<string, any>>         # Interactive buttons
├── cards: list<map<string, any>>           # Structured card elements
└── quick_replies: list<string>             # Suggested quick responses

MediaContent — File, image, or document:

MediaContent
├── url: string                             # Media URL (or data: URI)
├── mime_type: string                       # MIME type (image/jpeg, application/pdf, etc.)
├── filename: string | null                 # Original filename
├── caption: string | null                  # Text caption
└── size_bytes: int | null                  # File size

LocationContent — Geographic coordinates:

LocationContent
├── latitude: float                         # Latitude, -90 to 90
├── longitude: float                        # Longitude, -180 to 180
├── label: string | null                    # Location name
└── address: string | null                  # Street address

AudioContent — Audio message or voice note:

AudioContent
├── url: string                             # Audio URL (or data: URI)
├── duration_seconds: float | null          # Duration
├── mime_type: string = "audio/ogg"         # Audio MIME type
└── transcript: string | null               # STT transcript (if available)

VideoContent — Video message:

VideoContent
├── url: string                             # Video URL (or data: URI)
├── duration_seconds: float | null          # Duration
├── mime_type: string                       # Video MIME type
└── thumbnail_url: string | null            # Preview image URL

CompositeContent — Multi-part message (e.g., text + image + audio):

CompositeContent
└── parts: list<EventContent>               # Ordered list of content parts, at least one

Implementations MUST enforce a maximum nesting depth of 5 levels for CompositeContent.

SystemContent — Framework-generated message:

SystemContent
├── body: string                            # Human-readable description
├── code: string | null                     # Machine-readable code
└── data: map<string, any>                  # Structured payload

TemplateContent — Pre-approved template (WhatsApp, RCS):

TemplateContent
├── template_id: string                     # Template identifier
├── language: string = "en"                 # Template language
├── parameters: map<string, string>         # Variable substitutions
└── body: string | null                     # Text for channels without template support

EditContent — Edit of a previously sent message:

EditContent
├── target_event_id: string                # The event being edited
├── new_content: EventContent              # The replacement content
└── edit_source: string | null             # "sender" or "system" (e.g., auto-moderation)

DeleteContent — Deletion of a previously sent message:

DeleteContent
├── target_event_id: string                # The event being deleted
├── delete_type: DeleteType = SENDER       # SENDER, SYSTEM, or ADMIN
└── reason: string | null                  # Optional reason

DeleteType enumeration: SENDER | SYSTEM | ADMIN

Value	Description
SENDER	The original message author deleted their own message
SYSTEM	Automated deletion (e.g., auto-moderation, policy enforcement)
ADMIN	Administrative deletion by a room administrator or operator

ToolCallContent — A tool call, carried by TOOL_CALL_START and TOOL_CALL_END events:

ToolCallContent
├── tool_name: string                       # Tool called
├── tool_id: string                         # The call's identifier
├── arguments: map<string, any>             # Call arguments
├── result: any | null                      # Call result (end row)
├── status: "pending" | "completed" | "failed" = "pending"
│       # pending on the start row, completed or failed on the end row
├── duration_ms: int | null                 # Execution time (end row)
├── error: string | null                    # Error of a failed call
├── structured_content: map<string, any> | null
│       # Structured copy of the result for UI surfaces (Section 9.3)
├── outcome: "served" | "refused" | "failed" | "blocked" | "unserved"
│       | "cancelled" | null
│       # How the call ended, on an end row (Section 6.4); null on a row
│       # written without it, which a reader takes by its status
└── refused_but_ran: bool = false
        # The call ran although RoomKit refused it (Section 9.3)

5.4 Channel Data (Typed, Per-Channel)

Each channel type MAY define a typed ChannelData structure to carry channel-specific metadata on events. Common examples:

SMSChannelData
├── from_number: string
├── to_number: string
├── segments: int
└── encoding: string | null

EmailChannelData
├── from_address: string
├── to_addresses: list<string>
├── cc: list<string>
├── subject: string | null
├── thread_id: string | null
└── html_body: string | null

WhatsAppChannelData
├── wa_id: string
├── template_name: string | null
├── buttons: list<object> | null
├── context: object | null
└── is_group: bool

AIChannelData
├── model: string
├── agent_name: string | null
├── tokens_used: int | null
└── latency_ms: float | null

Implementations SHOULD define typed ChannelData for each supported channel type. Unknown channel data MUST be preserved in a generic map structure.

5.5 Participant

Participant
├── id: string                              # Unique identifier
├── room_id: string                         # Owning room
├── channel_id: string                      # Primary channel used to join
├── display_name: string | null             # Human-readable name
├── role: ParticipantRole                   # Role in the room
├── status: ParticipantStatus               # Current status
├── identification: IdentificationStatus    # Identity resolution state
├── identity_id: string | null              # Resolved Identity reference
├── candidates: list<string> | null         # Candidate identity IDs (when ambiguous)
├── connected_via: list<string>             # Channel IDs this participant uses
├── external_id: string | null              # Integrator's external reference
├── resolved_at: datetime | null            # When identity was confirmed
├── resolved_by: string | null              # What resolved it (hook name, "auto", "manual")
├── joined_at: datetime                     # When participant entered the room
└── metadata: map<string, any>              # Integrator-defined data

ParticipantRole enumeration:

Value	Description
OWNER	Room creator or primary responsible party
AGENT	Human agent (advisor, support representative)
MEMBER	Regular participant
OBSERVER	Read-only participant (supervisor, auditor)
BOT	Automated system participant

ParticipantStatus enumeration:

Value	Description
ACTIVE	Currently participating
INACTIVE	No recent activity
LEFT	Explicitly left the room
BANNED	Removed and blocked

IdentificationStatus enumeration:

Value	Description
IDENTIFIED	Identity confirmed — identity_id is set
PENDING	Awaiting resolution — may have candidates
AMBIGUOUS	Multiple candidates found — requires disambiguation
UNKNOWN	No matching identity — may create new
CHALLENGE_SENT	Identification challenge sent to the sender (§11.2); not a verification (§11.7)
REJECTED	Identity challenge failed or was rejected

VerificationStatus enumeration (Section 11.7): whether an identified sender has proven who they are.

Value	Description
UNVERIFIED	Identified, not proven: no verification in force
PENDING	A verification request is open for the identity
VERIFIED	Proven by the integrator's method; in force until it expires
EXPIRED	Was verified; its time ran out
REVOKED	Ended by the integrator before its time

Joining and leaving: a deliberate join is the operation add_member. It creates the participant ACTIVE, or reactivates an existing record that is not ACTIVE (a re-join keeps joined_at), then emits PARTICIPANT_JOINED and fires ON_PARTICIPANT_JOINED; on a member already ACTIVE it only records a newly reached channel, with no event. With an identity_id the participant is IDENTIFIED, otherwise PENDING. Leaving is remove_member: it sets the participant's status to LEFT, or to BANNED when asked, keeping the record, then emits PARTICIPANT_LEFT and fires ON_PARTICIPANT_LEFT; an unknown participant is an error. Both hooks are ASYNC; their payload is a system event whose data carries participant_id and, on a join, identity_id, on a leave, the new status.

Renaming (normative): a member's presentation can change while they are present — their display name above all. Implementations SHOULD provide an update operation (rename_member) that changes display_name in place, emits PARTICIPANT_UPDATED and fires ON_PARTICIPANT_UPDATED, so an interface reflecting the room learns of the change the same way it learned of the join. id and identity_id are not presentation and MUST NOT be changed through it: they are what attribution, correlation (Section 12.10.2) and moderation stand on. A rename that changes nothing is a no-op — no write, no event.

One record, several channels (normative): a Participant is one record per (room_id, id). Section 4.4 says a participant is connected via one or more channels, and this is the record all of them share — the same person reached by SMS and then by email is one participant, not two. channel_id is their primary channel; connected_via is every channel the room has reached them through, the primary included.

• A lazy materialisation — the get-or-create that materialises a sender the first time they speak — MUST NOT re-home an existing record: the channel_id it is given applies to a record being created, and to no other. A deliberate join MAY set channel_id to the channel joined through, which is what "primary channel used to join" means; the channel it replaces MUST remain in connected_via.
• Either way, the channel named by the caller MUST be added to connected_via when absent from it, and implementations MUST persist connected_via. A channel reached once and never recorded is a channel the room cannot afterwards say it reached.
• Being reached through a channel that is not the primary one is legitimate and MUST NOT fail. It is, however, invisible in the record handed back, whose channel_id still names another channel: implementations SHOULD log it at warning level naming both channels, because a caller that goes on to keep per-channel state on that record — a lifecycle, a presence, a status — is driving another channel's record without having been told. That silence is what lets one channel's departure erase another channel's membership.
• status follows the primary channel's lifecycle. A channel that observes presence on a record it does not home — a conference seeing a member connect and disconnect — MUST NOT write that record's status: an arrival there is not a join, a departure there is not a leave, and a room member who hangs up on its call is still a member of the room. What such a channel owes the record is the bookkeeping above (connected_via); what it owes everyone else is its own events and hooks, which report the arrival and the departure whether or not the status moved.
• Recording a channel is bookkeeping, not presentation: it does not emit PARTICIPANT_UPDATED and does not fire ON_PARTICIPANT_UPDATED.

5.6 Identity

Identity
├── id: string                              # Unique identifier
├── organization_id: string | null          # Multi-tenant scope
├── display_name: string | null             # Human-readable name
├── channel_addresses: map<ChannelType, list<string>>
│       # Cross-channel addresses, e.g., {SMS: ["+15551234"], EMAIL: ["john@example.com"]}
├── external_id: string | null              # CRM or external system reference
└── metadata: map<string, any>              # Integrator-defined data

5.7 ChannelBinding

When a channel is attached to a Room, a ChannelBinding is created:

ChannelBinding
├── channel_id: string                      # Which channel
├── room_id: string                         # Which room
├── channel_type: ChannelType               # Channel type enum
├── category: ChannelCategory               # TRANSPORT or INTELLIGENCE
├── direction: ChannelDirection             # INBOUND, OUTBOUND, BIDIRECTIONAL
├── access: Access                          # Permission level
├── muted: bool                             # Temporarily silenced (suppress response events)
├── output_muted: bool                      # Speech output muted: events still arrive,
│                                           # a speech-to-speech provider's audio is not
│                                           # forwarded, injected text is not spoken
├── visibility: string                      # Write visibility rule
├── participant_id: string | null           # The binding's correspondent (§10.4): the
│                                           # sender a room was created for, or the
│                                           # first one routed to it; on a chat
│                                           # channel, the chat
├── group: bool                             # Several senders share this binding's
│                                           # conversation (§10.4); default false
├── last_read_index: int | null             # Read horizon for unread tracking
├── attached_at: datetime                   # When attached
├── capabilities: ChannelCapabilities       # What this channel supports
├── rate_limit: RateLimit | null            # Per-channel rate limiting
├── retry_policy: RetryPolicy | null        # Per-channel retry configuration
└── metadata: map<string, any>              # Binding-specific data (recipient_id, persona, etc.)

A binding's category is the channel's own unless the attachment names one: an intelligence channel attached without a category takes part as an intelligence channel, a transport as a transport.

A binding carries one correspondent's conversation unless it is declared group. The framework records on it the sender a room was created for, or the first sender routed to it (participant_id, §10.1 step 2), and the default router admits no one else through it (§10.4). On a transport channel whose replies go to the address the correspondent writes from (a phone number, an email address: SMS, RCS, WhatsApp, email, Messenger), the binding's recipient and its correspondent are one address. When the framework records the sender, or finds a binding naming the sender by address without a recipient, it writes the address as the binding's recipient, under the metadata key the channel delivers to (phone_number, email_address, ...; Appendix A); a recipient the binding already has is kept. Conversely, a binding the integrator gives a recipient and no correspondent names that recipient as its correspondent: the room is that person's conversation, and the router admits no one else through it (§10.4). On a transport that delivers to a chat or a conversation (Telegram, Teams, Discord), the chat a message was posted in is its conversation: a private chat is its user's, a group chat the group's, shared by all its members. The framework routes the message, records the binding's correspondent and admits a sender by the chat, and a room opened for the message replies there; the message's author stays its sender. The chat a binding delivers to names its conversation as a phone number names its correspondent: a room the integrator opens for a chat admits no one else. A user who writes to the bot privately and in a group holds two conversations, and a room found by the sender would answer one in the other. A channel that carries both private chats and groups (WhatsApp Personal) takes a group message's group as its conversation and a private message's sender. On a transport that delivers to a URL (an HTTP webhook), nothing of the message is written as the recipient.

A channel addressed by phone number compares and stores numbers in one form, E.164 (+15550000001). The inbound sender, a binding's correspondent and recipient, and a participant's id given on that channel are normalized before they are compared or recorded, so whatsapp:+15550000001, 15550000001 and +1 (555) 000-0001 are one person. A number written without its country code takes a country code the integrator configures on the channel, or is kept as written when none is; a short code, an email address or any other id is kept as written. The message's raw payload keeps the provider's own spelling. An email channel compares and stores an address lower-case and without a display name (Alice <Alice@Example.com> is alice@example.com).

A binding declared group holds a conversation of several senders, such as a group chat on a channel dedicated to its room: every sender writing on it belongs to the room, and none is recorded on it, as correspondent or as recipient. The declaration is the integrator's to make, because nothing the store holds tells a group channel from a number shared by many correspondents.

5.8 ChannelCapabilities

Each channel declares what it supports:

ChannelCapabilities
├── media_types: list<ChannelMediaType>     # What content categories are supported
├── max_length: int | null                  # Maximum text length (null = unlimited)
│
│   # Text features:
├── supports_rich_text: bool
├── supports_buttons: bool
├── max_buttons: int | null
├── supports_cards: bool
├── supports_quick_replies: bool
├── supports_templates: bool
│
│   # Media features:
├── supports_media: bool
├── supported_media_types: list<string>     # MIME types
├── max_media_size_bytes: int | null
│
│   # Audio/Video features:
├── supports_audio: bool
├── supports_video: bool
│
│   # Delivery features:
├── supports_threading: bool
├── supports_typing: bool
├── supports_read_receipts: bool
├── supports_reactions: bool
├── supports_edit: bool
├── supports_delete: bool
├── delivery_mode: DeliveryMode
├── rate_limit: RateLimit | null
│
│   # Extensible:
└── custom: map<string, any>

ChannelMediaType enumeration:

Value	Description
TEXT	Plain text messages
RICH	Formatted text with buttons, cards, quick replies
MEDIA	Images, documents, files
AUDIO	Audio messages, voice notes
VIDEO	Video messages
LOCATION	Geographic coordinates
TEMPLATE	Pre-approved message templates

5.9 Three Levels of Channel Metadata

Channel-specific information lives at three levels. Implementations MUST maintain all three and MUST NOT conflate them:

Level	Where	Scope	Example
Channel Instance	Channel.info	Global, static	{provider: "twilio", from_number: "+15551234"}
Channel Binding	ChannelBinding.metadata	Per-room	{persona: "formal", language: "fr"}
Event Source	RoomEvent.channel_data	Per-event	SMSChannelData{from_number: "+15559876", segments: 1}

5.10 Task

Task
├── id: string                              # Unique identifier
├── room_id: string                         # Originating room
├── title: string                           # Human-readable title
├── description: string | null              # Detailed description
├── assigned_to: string | null              # Who is responsible
├── status: TaskStatus = PENDING            # Current state
├── created_at: datetime                    # When created (default: now)
└── metadata: map<string, any>              # Integrator-defined data

TaskStatus enumeration: PENDING, IN_PROGRESS, COMPLETED, FAILED, CANCELLED

5.11 Observation

Observation
├── id: string                              # Unique identifier
├── room_id: string                         # Originating room
├── channel_id: string                      # Which channel produced this
├── content: string                         # The observation, as text
├── category: string | null                 # Category (e.g., "sentiment", "compliance_violation")
├── confidence: float = 1.0                 # From 0.0 to 1.0
├── created_at: datetime                    # When created (default: now)
└── metadata: map<string, any>              # Integrator-defined data

5.12 Inbound Message

The normalized representation of a message arriving from outside the framework:

InboundMessage
├── channel_id: string                      # Which registered channel
├── sender_id: string                       # Sender identifier (phone number, email, user ID)
├── content: EventContent                   # Parsed content
├── event_type: EventType = MESSAGE         # SYSTEM for a voice session start;
│                                           # INSTRUCTION (Section 10.1.1)
├── external_id: string | null              # Provider-side reference
├── thread_id: string | null                # Provider-native thread reference,
│                                           # passed through to the provider
├── parent_event_id: string | null          # In-app thread: the event this replies to
├── idempotency_key: string | null          # Duplicate prevention
├── raw_payload: map<string, any>           # Original provider payload
├── provider_message_id: string | null      # Provider's message ID
├── metadata: map<string, any>              # Extra data
├── session: VoiceSession | null            # Voice session to connect after processing
├── visibility: string = "all"              # Event visibility (Section 7.3);
│                                           # "transport" reaches no intelligence channel
├── addressed_to: list<string> | null       # Intelligence channels asked to act
│                                           # (Section 19.3); null = any eligible
├── response_visibility: string | null      # Where the answer may go; null = unrestricted
├── chain_depth: int = 0                    # The chain the message continues (Section 8.3)
└── standalone: bool = false                # INSTRUCTION only: the turn reads nothing
                                            # of the room (Section 10.1.1)

The room, when the caller knows it, is passed to process_inbound() (Section 10.1), not carried by the message; the channel type is the registered channel's.

Unified voice inbound: When the session field is set, process_inbound() connects the voice session to the channel after successful hook processing. This allows voice calls and text messages to flow through the same entry point, same hooks, and same pipeline. See Section 10.1 for the additional step.

A convenience helper parse_voice_session(session, channel_id) SHOULD be provided to convert a VoiceSession into an InboundMessage with a SystemContent(code="session_started") content and the session pre-attached:

parse_voice_session(session: VoiceSession, channel_id: string) → InboundMessage
    # Returns InboundMessage with:
    #   channel_id = channel_id
    #   sender_id = session.participant_id
    #   event_type = SYSTEM
    #   content = SystemContent(body="Voice session started",
    #             code="session_started", data={session_id, channel_id, caller})
    #   session = session
    #   metadata = session.metadata, merged with the caller's metadata

5.13 Delivery Result

DeliveryResult
├── channel_id: string                      # Target channel
├── status: string                          # "sent", "queued", "failed"
├── provider_message_id: string | null      # Provider's message ID
├── error: DeliveryError | null             # Error details if failed
├── retry_after: datetime | null            # When to retry (if rate limited)
└── provider_result: ProviderResult | null  # The provider's own answer, if any

DeliveryError
├── code: string                            # Machine-readable error code
├── message: string                         # Human-readable description
└── retryable: bool                         # Whether a retry may succeed

ProviderResult                              # What a provider's send() returns
├── success: bool                           # Whether the provider accepted it
├── provider_message_id: string | null      # Provider's message ID
├── error: string | null                    # Provider's error if not accepted
└── metadata: map<string, any>              # Provider-specific data

5.14 Protocol Trace

An immutable record of a transport-level protocol exchange, used for observability and debugging. Channels that interact with signaling protocols (SIP, RTP, SMTP, etc.) SHOULD emit traces for significant protocol events.

ProtocolTrace (frozen)
├── channel_id: string                      # Which channel emitted this trace
├── direction: string                       # "inbound" or "outbound"
├── protocol: string                        # Protocol name ("sip", "rtp", "smtp", etc.)
├── summary: string                         # Human-readable summary (e.g., "INVITE from +1555...")
├── raw: bytes | string | null              # Full serialized protocol message (e.g., raw SIP request)
├── metadata: map<string, any>              # Protocol-specific data (call_id, codec, etc.)
├── session_id: string | null               # Voice session identifier (if applicable)
├── room_id: string | null                  # Room identifier (if known at emission time)
└── timestamp: datetime                     # When the trace was emitted (UTC, auto-populated)

Immutability: ProtocolTrace MUST be frozen (immutable) after construction.

Raw payloads: When available, the raw field SHOULD contain the complete serialized protocol message. For SIP, this is the full SIP request/response as returned by the SIP library's serialize() method. For protocols where the raw payload is not available (e.g., outbound requests where the library does not expose the serialized form), raw MAY be null.

Trace emission and routing: See Section 15.6 for the trace emission infrastructure, including channel-level APIs, framework hook wiring, and pre-room buffering.

---

6. Channel System

6.1 Channel Interface

All channels MUST implement the following interface:

Channel (interface)
├── id: string                              # Unique channel identifier
├── channel_type: ChannelType               # SMS, EMAIL, WHATSAPP, AI, VOICE, etc.
├── category: ChannelCategory               # TRANSPORT or INTELLIGENCE
├── direction: ChannelDirection             # INBOUND, OUTBOUND, BIDIRECTIONAL
│
├── handle_inbound(message: InboundMessage, context: RoomContext) → RoomEvent
│       # Convert external payload to a RoomEvent
│
├── deliver(event: RoomEvent, binding: ChannelBinding, context: RoomContext) → ChannelOutput
│       # Push event to external recipient
│
├── on_event(event: RoomEvent, binding: ChannelBinding, context: RoomContext) → ChannelOutput
│       # React to a room event (default: no-op)
│
├── supports_streaming_delivery: bool (default false)
│       # Whether this channel accepts streaming text delivery
│
├── deliver_stream(text_stream, event, binding, context) → ChannelOutput
│       # Deliver streaming text to this channel
│       # Default: accumulate text, deliver as complete event via deliver()
│       # A stream that raises: deliver what was accumulated, then propagate
│       # the failure (Section 12.2 step 13s)
│
├── connect_session(session: VoiceSession, room_id: string, binding: ChannelBinding) → void
│       # Connect a voice session to this channel (voice/realtime channels only)
│       # Default: no-op for non-voice channels
│
├── provider_name: string | null
│       # Provider or backend name for event attribution (see EventSource.provider)
│       # Default: provider.name if provider has a name attribute, else null
│
├── capabilities() → ChannelCapabilities
│       # Declare supported features
│
├── info: map<string, any>
│       # Channel instance information (Section 5.9), read as an attribute
│       # like the other properties: code reading it on any channel gets
│       # the map
│
└── close() → void
        # Release resources

ChannelOutput:

ChannelOutput
├── responded: bool = false                 # The channel answered the event; only then
│                                           # do its response_events re-enter
├── response_events: list<RoomEvent>        # Response events (subject to permissions)
├── response_stream: async_iterator<str>    # Streaming response (mutually exclusive with response_events)
│       # When set, framework pipes stream to streaming-capable channels,
│       # and stores each segment as it is produced, delivering it to the others.
├── response_metadata: ResponseMetadata     # The turn's response-metadata record (Section 6.7),
│       # merged into each event stored from response_stream as that segment
│       # is stored; a buffered reply carries its final state in response_events
├── provider_result: ProviderResult | null  # A transport delivery's provider answer (Section 5.13)
├── tasks: list<Task>                       # Side effects (always allowed)
├── observations: list<Observation>         # Side effects (always allowed)
├── metadata_updates: map<string, any>      # Room metadata to update
└── error: Exception | null                 # An error met while producing the output; the
        # output is still used, and the error surfaces as for a channel that
        # raised: ON_ERROR fires and InboundResult.error carries it

6.2 ChannelType Enumeration

Value	Category	Description
SMS	TRANSPORT	Short Message Service
MMS	TRANSPORT	Multimedia Message Service
RCS	TRANSPORT	Rich Communication Services
EMAIL	TRANSPORT	Electronic mail
WHATSAPP	TRANSPORT	WhatsApp Business Cloud API
WHATSAPP_PERSONAL	TRANSPORT	WhatsApp Web multidevice protocol
WEBSOCKET	TRANSPORT	WebSocket real-time connection
MESSENGER	TRANSPORT	Facebook Messenger
TEAMS	TRANSPORT	Microsoft Teams
WEBHOOK	TRANSPORT	Generic HTTP webhook
VOICE	TRANSPORT	Voice channel (STT/TTS pipeline)
REALTIME_VOICE	TRANSPORT	Speech-to-speech API (e.g., OpenAI Realtime, Gemini Live)
AUDIO_VIDEO	TRANSPORT	Combined voice + video channel (STT/TTS pipeline + video)
REALTIME_AUDIO_VIDEO	TRANSPORT	Combined speech-to-speech + video channel
VIDEO	TRANSPORT	Video-only channel
CONFERENCE	TRANSPORT	Multi-party SFU conference (audio + video + screen share)
TELEGRAM	TRANSPORT	Telegram Bot API
CLI	TRANSPORT	Command-line interface (development/testing)
PUSH	TRANSPORT	Push notification channel
AI	INTELLIGENCE	AI/LLM agent
SYSTEM	—	Framework-generated events

Implementations MAY define additional channel types. A type added by the implementation itself — a new transport it ships — is an ordinary member of this table and needs no marking; the list above is expected to grow.

This document does not define a registration mechanism for channel types added by a third party, and a naming convention alone would not be one: a closed enumeration is a legitimate implementation of the table, and a third party cannot add to it whatever it calls the value. An implementation that does want to admit external types owns both the mechanism and its naming.

6.3 Transport Channels

Transport channels carry messages between the framework and external systems or humans. They MUST implement deliver(). Common transport channels and their reference capabilities:

Channel	Media Types	Max Length	Key Features
SMS	TEXT, MEDIA	1,600	Read receipts
Email	TEXT, RICH, MEDIA	unlimited	Threading, rich HTML
WhatsApp	TEXT, RICH, MEDIA, LOCATION, TEMPLATE	4,096	Reactions, templates, buttons, edit, delete
WhatsApp Personal	TEXT, RICH, MEDIA, AUDIO, VIDEO, LOCATION	4,096	Typing, reactions, audio/video, edit, delete
Messenger	TEXT, RICH, MEDIA, TEMPLATE	2,000	Buttons, quick replies, delete
Teams	TEXT, RICH	28,000	Threading, reactions, rich text, edit, delete
RCS	TEXT, RICH, MEDIA	8,000	Buttons, cards, SMS fallback
WebSocket	TEXT, RICH, MEDIA, AUDIO, VIDEO, LOCATION	unlimited	All features, real-time, edit, delete
Voice	AUDIO	—	Streaming audio, STT/TTS
Realtime Voice	AUDIO	—	Speech-to-speech, tool calling
Telegram	TEXT, RICH, MEDIA, LOCATION	4,096	Buttons, inline keyboards, edit, delete
Audio/Video	AUDIO, VIDEO	—	Combined voice + video, STT/TTS + vision
Realtime Audio/Video	AUDIO, VIDEO	—	Speech-to-speech + video, tool calling
Video	VIDEO	—	Video-only, vision provider integration
Conference	AUDIO, VIDEO	—	Multi-party SFU, per-track STT, AI bot participant
Webhook	TEXT, RICH	unlimited	Generic HTTP POST

6.4 Intelligence Channels

Intelligence channels process events and produce responses or insights. They MUST implement on_event(). They do NOT deliver to external systems — the framework routes their output through the normal event pipeline.

AI Channel:

An AI channel wraps an AI Provider (see Section 6.7). When on_event() is called:

1. Build conversation history from room timeline.
2. Determine target transport channel's capabilities and media types.
3. Construct AI context with capabilities, system instructions, and room metadata.
4. Run the turn's tool loop over the provider's structured stream (One loop, streamed or not, below).
5. Return ChannelOutput with a response stream (Appendix A.9).

Anatomy of an AI turn. Each step of a turn has its own contract; no stage abstraction sits on top of them:

Step	Carried by
Whether the channel answers at all	Addressing and the agent-response policy (Section 19.3); BEFORE_AI_CONTEXT_BUILD would block here (Planned, Section 9.2)
The conversation history	The memory provider (Section 20), over the channel's view of the room (Section 7.5 rule 8)
The tools the turn may call	The tool registry, tool policy and skill gating (Sections 21 and 24)
The system prompt	The channel's and the binding's instructions, the instructions of the MCP servers the host names (Section 21.2), the skills' preambles (Section 24)
The gate before generation	BEFORE_AI_GENERATION (Section 9.2)
Generation and tool calls	The provider's structured stream and the one tool loop (below, Section 6.7); BEFORE_TOOL_USE, ON_TOOL_CALL, AFTER_TOOL_ROUND (Section 9.3)
What goes out	BEFORE_BROADCAST on each stored segment of a streamed response (Section 9.5)
Observing the turn	ON_AI_RESPONSE (Section 9.2)

A check of the whole answer before anything is sent is only possible on a turn that is not streamed: a streamed segment, once delivered, is not taken back.

The AI channel MUST skip events originating from itself to prevent infinite loops.

Capability-aware generation: The framework MUST provide the target transport channel's capabilities to the AI at generation time, NOT post-process the output. This allows the AI to tailor its response (e.g., short text for SMS, rich for Email).

Tool-call composition events: A provider whose structured stream exposes tool-call argument fragments MAY yield StreamToolCallDelta before the complete StreamToolCall. AIChannel SHOULD project those fragments onto the realtime bus as TOOL_CALL_DELTA, carrying the call id, name and cumulative argument character count, but MUST NOT carry the argument content. Each composition attempt that published a non-terminal frame MUST publish one terminal frame with an empty tool_calls list before that attempt ends. This includes successful completion, cancellation, provider failure, retry and fallback: a retry closes the abandoned attempt before the replacement attempt starts, so cumulative counts MUST NOT span two attempts. The complete StreamToolCall remains the unit of execution and persistence; composition events are ephemeral and MUST NOT be persisted. The ephemeral TOOL_CALL_END MUST carry each call's status, completed or failed, as the stored TOOL_CALL_END does: a call refused, failed, blocked, served by nothing or cancelled ends failed, so a live surface does not read the outcome out of the result preview. The stored TOOL_CALL_END also carries the call's outcome, the way it ended as the channel determined it (Section 9.3): served, refused, failed, blocked, unserved or cancelled. A reader that rebuilds what earlier turns did from the store, as a tool-usage digest reseeded after a restart does, reads that outcome rather than inferring it from the body: such a digest keeps the calls the live one keeps (served, failed, blocked; never a refusal, a call nothing served or one cancelled). A row stored without it is read by its status.

What a provider hands the loop of its reasoning (normative). A provider whose reasoning comes in blocks, each with its own signature (Anthropic), hands the loop every block as it received it, each StreamThinkingDelta naming its block: an AIThinkingPart carries the block's thinking and signature, or, for a block the vendor redacted, its opaque redacted data. The vendor wants the blocks back as it sent them (it refuses a round whose blocks changed in number, and a block without its signature), so the tool loop replays a round's blocks one by one, in the order and at the place they came in relative to the round's calls and stretches of text. A block the response cut before its signature is not replayed, nor is a block without one at all: that is another vendor's reasoning, a round a fallback_provider (Appendix A.9) receives from the primary, which the vendor refuses and the round does without. A vendor that refuses, in the current turn, a call without its own signature (Gemini 3, which signs each call it makes) receives such a round of that turn, none of whose calls carries one, as text: what it said, each call, then each result with the images it carries, the one form it takes for calls it did not make. An earlier turn's round, which it does not check, keeps its form. A provider whose reasoning has no blocks hands one block per round, replayed first. Two readers cannot know where blocks came, and replay them first, then the text, then the calls: a caller of generate(), since a response does not say where its blocks came, and a turn under a response schema, whose text is held until it is checked.

A provider MUST render a replayed round's reasoning where its vendor reads it. DeepSeek, in thinking mode, refuses a round of the turn in progress that called tools without its reasoning_content, unless it issued the round's call ids itself a moment ago; the DeepSeek provider therefore sends that field on every round that called tools, empty when the round did not reason, and an answer's reasoning in it too.

What a provider hands the loop for a call (normative). Every provider, streamed or not, hands the loop the same thing for the same call. Each call of a response carries an id no other call of that response carries: a provider whose server issues none, or the same one twice, MUST mint one, since results, eviction and cancellation all find a call by its id. Two calls stay two calls, whatever stream index or arguments they share; a call the provider receives twice (a re-emission carrying what the first lacked) stays one, and only a call the wire cannot tell from such a re-emission (Gemini, an identical call in a later chunk, one copy or both without an id) is folded into the first, which keeps what either copy carries (its id, its thought signature). A call's arguments reach the loop as a mapping and never as an error: no arguments (nothing, or JSON null) are {}, and arguments that do not parse to an object (invalid JSON, an array, a fragment) are kept whole under raw. A response is cut short when it ran out of room (the output cap, or the context window filling up mid-answer), when a content filter or a refusal stopped it, or when its stream ended without a stop reason, under any provider's word for each (length, max_tokens, MAX_TOKENS, model_context_window_exceeded, model_length, content_filter, refusal, and Mistral's generation error): every provider reads one such set, so none runs a call another provider refuses. Only the last call of a response can be cut: a call another call followed was closed by it, and runs when its arguments read, the response cut over its successor or not. Under a response cut short, only argument text that arrived and reads shows the last call's arguments whole: a call with no argument text yet, or known only through a parse of what arrived (a vendor SDK reads a fragment leniently), does not read either. Nothing under a cut is no evidence of no arguments (Anthropic sends no stop for the block it cuts, often before its first argument chunk, measured). Such a call is marked partial, whatever the provider and whatever stop reason the response gave, and a tool loop, the AI channel's or a reasoning backend's, MUST NOT run it, nor may a realtime channel, whose provider hands such arguments as the model's text rather than a mapping (Section 12.4): no tool does anything useful with raw. A realtime provider applies the same rule where its wire tells a cut: a call whose item the response cut (OpenAI Realtime's and GPT-Live's status: "incomplete", whether the output cap or a cancel ended the response) runs only when its argument text arrived and reads, and is otherwise handed on marked cut, so the model reads that its call was cut off; such a provider hands a call on once its item says whether it was cut (OpenAI Realtime's function_call_arguments.done comes before the item's status, measured). A provider whose wire carries no sign of a cut (Deepgram's FunctionCallRequest) reads none. The model reads that nothing ran, and why: its call was cut when the response was cut short, so it can call again, with less if the response ran out of room; its arguments could not be read otherwise, so it can call again with valid ones. A response the provider ended because it could not parse the model's call (Gemini's MALFORMED_FUNCTION_CALL), or would not take it (Gemini's UNEXPECTED_TOOL_CALL, a call to a tool the request did not enable), carries no call at all: the tool loop MUST tell the model that its call did not run and why (its call could not be read, or the tool is not in the request: calling it again would end the same way), within the bound it gives an empty round, on any round and whatever text the round said, and a turn whose bound runs out ends cut short (empty_response), never completed. A round that ran out of room before any answer after a tool round ends the turn truncated and is not tried again, since the same room runs out again: the output cap and the context window alike, read by the same rule as a response schema's truncated (Section 6.7).

An answer that did not act. An AI channel MAY carry a continuation policy: given the text a round ended on, the instruction that makes the model go on, or nothing when the answer stands (a host's recognizer of an announced action, "I will check that", say). The loop consults it only for a round that ended naturally, with text and without a call: not cut short (Section 6.4 above), not a call it could not parse, not a refusal, and with a tool declared, since going on means acting. Every provider's word for a natural stop counts alike (stop, end_turn, stop_sequence, STOP), and only those: a truncation, a filter, a refusal or a stream that ended without saying why is never one. The policy shares the bound of an empty round, and the loop keeps its guards: no continuation after a cancellation or a force-stop, nor past the turn's deadline or budget. A continuation hands the next round the text as the assistant's message, then the instruction as the user's; the text already reached the room as the round's segment and is neither repeated nor withdrawn. Any round the loop tries again (an empty one, one whose call could not be parsed, one the policy continues) ends its segment there, as a call's start does, so the next round's text never runs on from it. A turn whose policy still asks once the bound has run out ends unfinished, never completed: the loop reaches its end (its end marker, ON_AI_RESPONSE), and running out of the bound never raises out of the loop.

Whether the agent speaks. An AI channel MAY carry a speak policy, which decides on every event the channel would answer whether the agent speaks now, offers to, or stays silent: an agent in a conversation with several people, or listening to one who thinks aloud, does not answer every turn. An instruction (Section 10.1.1), a delegated task's hand-back (Section 23.3) among them, is not submitted to it: the application asked for that turn. The channel consults the policy once per event, after building the turn's context and before BEFORE_AI_GENERATION. The recent events a policy reads are those the channel may know, as its context has them (Section 7.5 rule 8): a policy may hand them to a classifier outside, and an event withheld from the channel at delivery MUST NOT reach it that way.

SpeakPolicy (interface)
├── decide(turn: SpeakTurn) → SpeakDecision
└── forget_room(room_id: string)        # Optional: drop what it keeps of a room the channel joins or leaves

SpeakTurn
├── event: RoomEvent                    # The event the turn would answer (its trigger)
├── recent: list<RoomEvent>             # The conversation the turn reads, `event` excluded, oldest first, the agent's answers included:
│                                       #   what came before `event`, and in a discussion what came after it up to the turn's start (§19.7.5)
├── people: list<string>                # Who takes part besides the agent, named as its context names them
├── channel_id: string                  # The agent's channel: its own answers in `recent` come from it
├── speakers: map<string, string>       # Who said `event` and each of `recent`, by event id, where the room names them
├── thought: Thought | null = null      # What the agent has in mind, when the channel has a thinker (below)
└── cut: CutReply | null = null         # The agent's answer cut off just before the event, when it has not spoken since (below)

SpeakDecision
├── mode: "speak" | "offer" | "silent"
├── reason: string = ""                 # Why, in a few words, for logs and hooks
├── judgments: map<string, float> = {}  # What the policy weighed, by name, each in [0, 1] or its own scale
├── notes: list<string> = []            # Blocks the turn's notes carry when the agent speaks or offers
└── final: bool = false                 # A silence the agent's thought will not change (below); only on `silent`

• speak runs the turn as without a policy; the decision's notes join the turn's notes (below) before BEFORE_AI_GENERATION, which sees them.
• offer runs the turn with one more block: offer, in one short sentence, what the agent could add, without giving it.
• silent runs no turn: no generation, BEFORE_AI_GENERATION does not fire, nothing is delivered and no answer names the event (Section 8.5). The event is stored as any event is: the conversation is the agent's memory whether it spoke or not.

The channel bounds the wait for a decision (2 s by default). A policy that fails or does not decide in time does not silence the agent: the channel logs it and runs the turn as without a policy, and the decision it reports is speak with the reason fallback. Every decision fires ON_SPEAK_DECISION with the event, the channel and the decision, how long the policy took to decide (the bound, when it did not decide in time) and whether it was asked again once the agent thought (below), so that what an agent left unanswered, why, and what deciding cost can be followed and measured without wrapping the policy. A channel without a policy answers every event, as before, and decides nothing; a policy that always speaks reports each decision, a baseline to measure another against. A turn the channel's strategy takes in a room (Section 19.7) is not submitted to it either, save a turn a discussion gives in answer to a message (Section 19.7.5): that one is the agent's own answer to the room, and is submitted to it. The agent's identity is the policy's own: a policy that judges whether the agent was addressed is told its name when it is made.

The people are the room's active participants that are neither agents nor bots nor on the agent's channel, or, when more, the distinct speakers named in the recent events: one microphone is one participant but may carry several diarized voices. A speaker is named as the turn's context names speakers: the name the sender's transport stamped on the event, else the participant's display name.

RoomKit provides a policy built on a classifier (Section 6.8). It asks narrow questions about the turn, in one call, over the turn's text, the recent turns named by speaker (the agent's own under its name) and the people, and composes the answers in code, in this order:

1. silent when the speaker has not finished, postpones or declines asking, or asks the agent to keep quiet on this turn;
2. speak when the turn answers a question the agent just asked;
3. speak when the agent is asked directly or indirectly, or asked for something tentatively, or asked anything when one person talks with it;
4. offer when the speaker only wonders whether the agent could help;
5. silent otherwise.

Each judgment is reported with the decision. Its questions can be replaced by name, and its composition by an implementation of its own. Told the languages the agent answers in, it also judges the language the speaker speaks, over their recent turns, so that one misheard word does not switch it, and its decision's notes say which language to answer in. A classifier that fails falls back as any policy does: the agent speaks, unless the room listens (below).

Staying quiet when asked. A request to stay quiet or only listen from now on is a state of the room, not a judgment remade on every turn: re-judged from the recent turns, it faded as they passed, and was lost once it left them. The classifier policy keeps it per room, in memory as the thought is (rule 6 below): the channel has the policy forget a room it joins or leaves (forget_room), so a room that reuses an id inherits no request. It is set when a turn asks the agent to stay quiet or only listen until told otherwise: that turn is silent, and final. The classifier reads the request with every turn, with who made it and bounded as any text from outside, so that each judgment is made knowing it, and is asked, instead of that request, whether the turn puts a question or a request to the agent itself and whether it lets the agent talk again. While the room listens:

• the agent is silent, and the decision is final (below);
• a question or a request put to the agent and addressing it directly (its name, or "you") is answered, and the room goes on listening: answering one question does not lift a request to stay quiet; a speaker who has not finished, postpones it or asks for quiet is not answered;
• a turn that lets the agent talk again ends the state, and is decided as without it;
• a turn the policy cannot judge (no text, a classifier that fails) is silent: falling back to speaking would break the request.

A turn is decided as it was judged: when another turn of the room starts or ends the state during the classifier call, the answers to the questions asked decide it. A conversation with one person does not make every request the agent's while the room listens: a question said aside and one put to the agent read alike without the address. A cut answer is not resumed. Each decision made while the room listens reports it among its judgments (listening).

RoomKit also provides a policy that answers only some people, around any other: an agent that listens to everyone in the room (a television, a meeting it assists one person in) but answers only them. An event whose speaker, as speakers names them, is not one of them is decided silent with the reason only listened to, final, without asking the policy it wraps, and so is an event whose speaker the room does not name. The people it passes on are only those it answers, the speaker among them even when no participant record names them so (a microphone's record and the voice its transport stamps), so that a voice only listened to never makes a conversation with one person a group one. An event so left silent is stored, and thought about, as any other: the agent hears it. A speaker is matched by the name the room gives them, kept to a name's characters as the room keeps it and ignoring case: this chooses whom the agent answers, it is not an access control.

What the agent thinks. An AI channel that has a speak policy MAY also carry a thinker: what the agent thinks while it listens, its answer, when it is silent, to "what are you thinking about?". The thought is neither memory (the room's history is, Section 20) nor a summary: what the agent makes of what is said, what it would say if given the turn, and whether that cannot wait. It is about what the agent hears: what is being talked about, what the speaker is doing (asking, telling, thinking aloud, talking to someone else, reading something), and what the agent makes of it, the people named; never about the agent itself (what it said or did, whether it should speak, how it is seen). What it would say holds only what it would say to the people about what is being discussed: never a rule for itself, nor what it does not know. When the topic changes, the thought starts again from the new one, keeping of the previous only what still concerns it: a thought rewritten from itself on every call drifts into its own concerns.

Thought
├── text: string = ""                   # What the agent thinks, in the first person
├── want_to_say: list<string> = []      # What it would say if given the turn, most important first, three at most
└── urgent: bool = false                # What it wants to say cannot wait

Thinker (interface)
└── think(previous: Thought, context: AIContext) → Thought

1. The thinker runs only on events the policy left silent: a turn the agent answers waits for no thought. The channel builds the event's context as for an answer (the agent's own prompt, the conversation it may know), every speaker named, the one person of a one-to-one conversation included (an answer's context leaves that conversation unlabelled), passes it through BEFORE_AI_GENERATION with the purpose thought, and hands what the hooks left to the thinker with the previous thought. A hook that blocks it keeps the previous thought. A thinker MUST NOT reach a model with a context that hook did not see: whatever an application keeps from a model there (consent, redaction, a budget) holds for the thought too.
2. One thinker call runs at a time per room. Events that arrive during a call are thought about in the next one, from the latest context; none is queued.
3. The channel waits for that thought up to a bound (1.5 s by default). When it comes back in time with something to say, the policy is asked again on the same event, with the thought: the agent may raise its hand on a turn it first listened to. Each decision fires ON_SPEAK_DECISION. A decision the policy marks final is a silence no thought changes (a room that listens, a voice only listened to): the thinker thinks, and the channel neither waits for it nor asks again, so the next turn is not held back.
4. When the agent speaks or offers on an event submitted to the policy, the turn's notes carry its thought, and what it wanted to say is then empty: it is said, or offered. A thinker call running meanwhile does not bring it back. An instruction carries no thought and empties nothing. The thought is a model's reading of what people said, so whatever they said can reach it: the notes MUST carry it quoted (as any text from outside, below), bounded, and named as information to weigh, not as the runtime's instructions. Asked what it is thinking, the agent answers with it.
5. A thinker that fails, or the end of its wait, keeps the previous thought.
6. The thought is the channel's, per room, in memory: it is not stored, and a restart starts from an empty thought, as does a room the channel is attached to or detached from, so that a room reusing an id never inherits another conversation's thought.

Every new thought fires ON_THOUGHT with the room, the channel and the thought, the one it replaces, and how long the thinker call that brought it took; a thought emptied because the agent spoke (rule 4) came from no call and carries no duration. A call that fails or brings back the same thought fires nothing. A policy reads it in SpeakTurn.thought. The classifier policy then also judges whether what the agent wants to say answers what the turn asks, or corrects or warns about what it says. Only wondered about and knowing the answer, the agent speaks rather than offers; not addressed, it offers when that judgment reaches the policy's proactivity (0.5 by default, lower is more eager), half of it when the thought is urgent, never on urgency alone. RoomKit provides a thinker on any AI provider that answers under a JSON schema (Section 6.7), whose instructions the application can replace, and a mock.

A cut answer. When the agent's answer is cut off by a barge-in (Section 12.3.13), the room holds a record of it that names the answer. The AI channel uses it twice:

CutReply
├── text: string                        # What the agent was saying: the text handed to speech when it was cut
├── played_ms: int                      # How much of it played before the cut
└── at: datetime                        # When it was cut

• In the context the agent's next turn reads, that answer is marked as interrupted: the model reads that the person may not have heard its end, rather than an answer heard whole.
• A speak policy reads it in SpeakTurn.cut when the event came after the cut and the agent has not answered since. The classifier policy then also judges whether the turn leaves the agent free to go on (an acknowledgement, a thanks, a short reaction, talking over it by accident) or wants the turn (a question, a request, a correction, asking it to stop). Free to go on, and not asked for quiet nor talking over an unfinished speaker, the agent speaks with the reason resume after cut, its notes asking it to go on from where it was cut without repeating what was heard.

Which part of the answer was heard is not guessed: the record keeps the text handed to speech and how long it played.

Only a record a voice channel of the room wrote counts: internal, with no participant, its source a channel bound to the room as voice, and interrupted exactly true. Event metadata is anyone's to write, and a record taken at its word would put a sender's text in the agent's mouth as its own. A record whose answer_responds_to is neither a string nor absent is skipped, and a played_ms that is not a non-negative number reads as 0: a malformed record never fails the turns of the room it stays in.

What the generation hooks see: BEFORE_AI_GENERATION fires once per turn (purpose answer), and once per thinker call (purpose thought, above, with the context the thinker would read, its tools never called), so a hook tells the agent's turn from its thinking. For a turn it fires with the context the turn starts from and the event the turn answers (trigger, Section 8.5), and its tools are the turn's toolset as the tool policy and skill gating leave it. Without Tool Search that is the first round's declaration, the tool that reads back a large result aside (below). Under Tool Search it is the whole catalogue the turn may reach, before Tool Search collapses it: the first round declares the pinned floor, the tools orchestration injected (Section 21.1), the tools the hook added and the discovery tools, all taken from what the hook left, and a tool find_tools reveals enters the declaration of the next round. A hook that sees only the first round's declaration cannot withdraw a deferred tool, which find_tools would then reveal. The toolset the hook leaves is the turn's: a tool it removes MUST NOT be declared at any later round of the turn, MUST NOT be named by find_tools or list_tools nor recovered at call time, and a call naming it MUST be refused, a tool the channel provides itself included; a tool it adds or edits stays as the hook left it, subject to the tool policy and skill gating, and a tool it adds is declared at every round of the turn, never deferred by Tool Search. A hook that withdraws a tool from the first round only has withdrawn nothing. A tool the hook adds is never named by find_tools, being declared already. Once the turn's toolset is resolved, a call MUST name a tool the round declared, an empty declaration included, or one Tool Search recovers from the turn's catalogue at call time: a round that declares nothing admits no other call. The channel's own tools are no exception: one the turn does not offer (find_tools while Tool Search hides nothing, run_skill_script with no executor) is refused as any undeclared tool is, on every path (Section 12.4). Likewise a skill activation (Section 24.4) counts only once its activate_skill call is served: an ON_TOOL_CALL BLOCK or a failure activates nothing, whatever the model read. So does a find_tools reveal, on every path: a BLOCK or a failure reveals nothing, not for the next round nor for a later turn, and a search that finds nothing reveals nothing, the reveal window left as it was. A served call reveals its matches whatever an ON_TOOL_CALL hook replaced its result with, as a served activation opens its gates. A realtime session declares what a served find_tools call revealed once its result went out, its call id belonging to the connection a reconfiguration can replace, and declares nothing of it when a reconfiguration gave the session another catalogue while the call was judged: its matches were found in the catalogue that left. So for the tools an activation's hint names. An activation whose answer carries no hint (it finds no skill and no tool matches the name, or the skill exists but is unavailable) reveals nothing and is refused, as a reference or a script asked of a skill the registry does not offer is. A tool Tool Search recovers at call time joins the reveal window only once the tool answered the call (served, failed, or its result withheld by an ON_TOOL_CALL hook), as the room's tool memory keeps any tool used: a call refused before it ran (a gate, its arguments, BEFORE_TOOL_USE, its handler's refusal) or that nothing served reveals nothing, not for the next round nor for a later turn, and leaves every other reveal of its round as it was. ON_AI_RESPONSE MUST therefore carry the turn's whole declaration as declared_tools: the union, over every generation round of the turn, of the tools the channel handed the provider, each with the name, description and parameters as declared and the reason it was visible (always when Tool Search did not gate it, pinned, sticky or revealed when it did, the earliest reason winning). The union is keyed by name in first-declaration order, one entry per name: the reveal window slides from one find_tools to the next, and a per-round rewrite would lose a tool revealed earlier in the same turn. A turn that runs without Tool Search MUST report its single declaration through the same field, so a host has one reading whatever the turn's mode. The field reports the declaration RoomKit made, not the provider's wire form of it. An intelligence channel whose toolset is not RoomKit's to declare (an external ACP agent, below) leaves it empty.

The declaration from round to round. A provider that caches requests reads each one as a prefix, in the order tools, system prompt, messages, and a declaration that gains, loses or reorders a tool invalidates everything cached after the tools, the history included: the next round is billed as if nothing were cached. Within a turn the declaration SHOULD therefore change only where a tool must appear, one find_tools reveals or one a skill activation opens (Section 24.4), and its order is the toolset's, the same from one turn to the next. The tool that reads back a large result (Section 21.5) is declared from the first round of any turn that declares a tool, after the others, so the declaration does not change when a result is stored; the generation hook sees it once the room holds a stored result, and may withdraw it then. A round the implementation forces to answer, when an anti-loop guard stops a model that keeps repeating a refused call, keeps the round's declaration and tells the model that no further call will run: none of that round's calls runs, and the turn ends force_stopped.

Between two rounds. AFTER_TOOL_ROUND fires after each round in which the channel ran at least one call, once that round's results are in and before the next round is built: not after a round cancelled before its calls ran, nor after one whose calls were all served outside the channel. Its event carries the round's index, its calls, the results the channel produced and the ones the provider served, each in the order of the calls, and the names of the turn's toolset as BEFORE_AI_GENERATION is shown it: what the tool policy and skill gating let the turn reach, Tool Search's whole catalogue included, nothing withdrawn. A hook reads the round whole and names what it withdraws: the calls of one round run concurrently, and a rule about them (one success among failures, say) is a rule about the round. A hook MAY withdraw tools for the rest of the turn, any name whether the event lists it or not, with every guarantee of a withdrawal by BEFORE_AI_GENERATION above (never declared again, never named by find_tools or list_tools nor recovered, a call naming it refused, a tool the channel provides itself included, never handed to an external handler), and MAY add messages the next round reads after the round's results. A withdrawal changes the declaration of the next round, the one exception this hook makes to the declaration staying the same, and costs what a changed declaration costs. A hook acts on the event it receives, in place: the event a MODIFY returns is not read. A BLOCK stops the hooks after it, as on any SYNC trigger, and changes nothing of the round, which has run; a hook that raises or times out is skipped, or, registered fail-closed, stops the hooks after it. The hook belongs to a registered AI channel's tool loop: a realtime session does not fire it, nor does a reasoning backend's agent (not a registered channel), nor an external agent that runs its own loop (Section 6.4, ACP).

The prompt from turn to turn. For the same reason, the system prompt SHOULD stay the same from one turn to the next: a system prompt that changes invalidates everything cached after it, the whole history included. What changes between turns as the conversation works travels with the turn's input instead, as notes: what the channel recalls of the tools used in the room (and what they returned), the room's plan, the room's background tasks and how far each got (Section 23.4), how speakers are named when the history window holds several (which speakers it holds changes as it slides), what a vision provider last saw (Section 12.8.7), and what a memory provider retrieved for this turn alone (Section 20.2), each passage set apart as data. The notes follow the turn's input (the participant's words, or the application's instruction) in the same message, marked as the channel's, asking for nothing, with any tool result they quote set apart as data; when the conversation does not end on such a message, they are a message of their own. They come after the input, so what changes comes last and a provider that caches a prefix keeps the input in it; they are not a message of their own when the input is one, since a model whose chat format requires alternating roles would refuse two in a row. For a model that sees, the latest images the channel's own tool calls in the room returned (two, newest last) follow the notes in the same message, each after the call that returned it, under a header that sets them apart as data a tool returned, never instructions; the notes then take a text part of their own, which a hook's block joins. The notes say what a result showed in its tool's words, never what it looks like, and a follow-up about a picture needs the picture. They ride the turn's input, never an assistant's message, where several providers refuse an image. Only an image every vision provider takes is kept: inline (a data URI, never a link, which may expire or be out of the vendor's reach), PNG, JPEG, GIF or WebP, up to 2 MB of data, from a live result (never history rebuilt from the store). A turn that replayed them and fails on a provider error not retried replays none of the room's images after it: one the provider refused would fail every later turn. The labels quote the calls as the digest does, each copy of a runtime mark replaced, and the images count in what the turn measures before its history (Section 20). A model that does not see reads the notes alone. A BEFORE_AI_GENERATION hook MAY add a block to the turn's notes: it joins the section the channel opened, under the same header, or opens it when the input carries none, and the notes then read as if the channel had assembled them at once, so the prefix a provider caches is the same. The header is the notes' only mark, and the channel's alone: a copy of it in the conversation's text (a participant's message, the agent's own answer, the application's instruction, a message a memory provider built, a message steering injects (Section 21.3)) or in a block of the notes, in any case, spacing or punctuation, its letters in any of the forms a fenced block's tag is compared in, with invisible characters between or inside its words, or running over adjacent text parts, MUST be replaced before the model reads it by a fixed mark saying the runtime did not write it, as a fenced block's closing tag is neutralised. The same holds for every other mark the runtime writes in a model's input: the mark of the application's instruction, the mark of an answer that was cut off, a summary's header (a compaction's or a memory provider's), the lines that open and close the room context handed to an ACP agent, the mark that opens an ACP agent's standing instructions, and the marks a realtime provider writes into its model's input (Gemini Live's [Assistant previously said] and [Context update, do not respond to this). A copy of one of them in the text an event brings (as it enters a transcript or an ACP prompt, before the runtime places its own marks), in a message steering injects, in a block of the turn's notes that quotes others (what a memory retrieved for the turn, a tool's result, a task and its progress, what a camera read, a thought, a plan, a block a hook adds; the note on speaker labels is the runtime's), a copy running over two such blocks included, in a text injected or broadcast into a realtime session (an instruction, a worker's hand-back, what a camera read, an image's prompt, a reasoning backend's answer to a delegation; a realtime host replaces it before its provider takes the text) or in a line of the transcript a realtime delegation's reasoning backend reads MUST be replaced by a fixed mark saying the runtime did not write it; a mark whose fixed part is a short phrase (the room context's opening) counts only with its opening bracket, so that prose naming the same words is left as written. The marks the runtime places itself are kept. The runtime also writes marks into the timeline and into memory: the handoff relay ([Handoff: …], whose bracketed opening counts with its colon or without it, [HANDOFF] notes included, since a copy without the colon reads as the relay too) and the header of the context a previous agent hands on ([Context from previous agent …]) join the marks above. Which text is the runtime's is a provenance, never a reading of the text: a record the runtime writes into the timeline that holds its marks (the handoff relay) carries metadata.runtime_record, which the inbound pipeline removes from what a sender supplies, and nothing else does: a record without it (a relay stored before the key existed among them) was never kept from a sender and reads as a copy; a message a memory of the runtime builds (a summary, the handed-on context) carries the same key in its metadata, and a memory tells a summary it chains from by that key alone, never by its header; the text a model wrote into either (a handoff's reason, a summary) is cleaned as it is written. A copy of a mark is replaced in every record and every memory message without it, the messages a host's memory provider builds included, and so is a copy that runs over consecutive user messages the runtime has not labelled (a memory's last message and the turn after it included), which an API may merge into one; a labelled message is strings after its author's label, which no copy runs out of, and is never cut, since a cut would take its label off. The replacement is the same on every turn, so the prefix a provider caches is the same, and the header the model reads is the one the channel placed, a paragraph of its own with a block after it: the block a hook adds joins the notes it marks, and a reader that separates the input from its notes cuts there (an input with images carries its notes in a text part of their own, which such a reader leaves out). The turn's own rounds are not rewritten: what the agent writes between its tool calls, and a tool's result, which the tool role frames. Only a copy a hook writes into the messages itself is what both misread. An active skill's instructions stay in the system prompt (Section 24.4): they change only when a skill is activated, and they are instructions, not notes. Those of a skill an MCP server serves ride there as data, in a block of their own (Section 24.7). A standalone turn carries none of the room's memories (Section 10.1.1).

A turn compacted. When a provider refuses a turn's context as too long, the channel MAY compact the context once and replay the call. A compaction keeps the turn's input and its notes whole. When the input falls in the older half of the messages, it summarizes the history that precedes the input and shortens the long results of the turn's older tool rounds by storing each as a large result is (Section 21.5), so the model keeps a preview and can read the rest back; a skill's instructions (Section 24.4) and a page already read back stay whole. Otherwise it summarizes the older half. It keeps every tool call with its result, never quotes part of a delimited block, and fails the call rather than cut the input when nothing before the input can be shortened. Text the runtime places next to a user message (the turn's notes, a compaction's summary, a summary a memory provider returns) joins that message rather than forming a second user message in a row.

Text from outside in a model's context (normative). Text the runtime places in a model's context without having written it (a person's words or name, a worker's output or progress, a tool's result, a passage a memory retrieved, a skill an MCP server serves, the instructions an MCP server publishes, a model's output such as a thought, a summary, or the task a tool call asked for) can hold anything, a frame of the runtime's own included. Every rendering that places such text, present and future, MUST keep it in a frame it cannot leave, so that nothing it holds reads as written by the runtime:

• a block, set apart in a tag of its own (<tool_result>, <worker_output>, <knowledge>, <conversation_summary>, <context> for content a realtime provider adds to its prompt, <task> for the goal or task an orchestration strategy copies into another model's prompt, <vision> for what a vision provider saw, <transcript> for the text a speech model's prompt asks it to speak, <skill> for a skill an MCP server serves, Section 24.7, <server_instructions> for what an MCP server publishes on using its tools, Section 21.2), where any closing tag of that name the text holds is neutralised where it starts, its end bracket there or not and what follows kept: compared in any case and under NFKC folding (fullwidth and mathematical letters, ＜／tag＞) and canonical decomposition (an accented letter), with the forms Unicode lists as confusable with its letters (UTS #39 confusables: Cyrillic, Greek, Armenian, Coptic, Cherokee letters, digits such as 0 for o) and small capitals, a character that reads as several of its letters (ⅵ for vi) included, a form of another case that reads as another letter (i for the I that reads as l) left out, with spacing around its brackets and slash, a character Unicode marks as ignorable by default, a control or line-break character or a combining mark between its letters, one slash or more (an escaped one included) and the common look-alikes of the bracket and the slash, and with a mark after the name (</tag.>): a closing tag errs toward what a model could read as one. An opening tag of that name, in the same forms and spacing, its name ended by anything that does not continue it (the text's end included; <tag-list> and <tag.v2> are other tags), is neutralised as well, the rest of it kept as written, so no reader tracking nesting reads the runtime's text after the block as data. A provider that rewrites a framed text before sending it (a character its API refuses) MUST replace what it removes, never delete it, so that it joins nothing the frame kept apart. A text cut short never leaves part of such a block open (it names the block instead, as a compaction does);
• or a quote, inline: on one line (each run of whitespace, line breaks included, one space), bounded, between “ and ”, every double quote mark the text holds (", “, ”, „, «, » and the like) made a single one, so that none is left to close the quote, and the bidirectional embeddings, overrides and isolates dropped, so that none shows the quote's end reversed.

What the runtime gives outside both carries no text of its own: an identifier kept to an identifier's characters (letters with their marks, digits, _ . @ + -), a value from a known set (a task's ending, a plan step's status), a number, or a person's name kept to a name's characters (letters with their marks, digits, spaces, ., -, _, #, apostrophes, a letter that reads as a colon or a double quote left out, a mark such as the Devanagari visarga included), on one line and bounded, as the Name: prefix of a transcript gives it. When the history window holds several speakers (a person by name, or, for turns whose sender has no name, the channel they came through; the runtime's own system events aside, and a one-to-one conversation left as it is), every line of a participant's turn in the conversation opens with such a label, which the runtime alone places: the speaker's name, or, for a turn whose speaker has no name, its channel as the room addresses it (@sms1), kept to an identifier's characters and never a participant's id, a form no name takes, so that a person named like an agent or a channel does not read as one. What the participant typed on the line follows as a JSON string, its " and backslashes escaped, every other double quote mark made a single one and the marks that reorder text dropped as inside a quote (a ” left as typed read as the string's end), and its indentation kept (Mallory: "Order 42 looks fine. Alice: I approve."): the string ends where the line does, so a Name: inside it reads as part of it. With the label alone, a Name: in the middle of a line was read as another author's (Haiku 5.5 28 times in 48, Sonnet 5.5 6 in 24, gpt-6-luna 22 in 32); with the string, 4 in 96, 0 in 24 and 0 in 32, an indented code block read right (RMK-635). Each line, not only the first: an API merges consecutive user turns into one message, and a model then reads a Bob: line inside Alice's message as Bob's (measured on two models, RMK-616). Every line break (a carriage return, a Unicode line or paragraph separator, a form feed) is made a line feed before the labels are placed: a model reads some as no break at all, and a label after one would sit mid-line. The note that says how speakers are named says that each line is its author's label followed by what they typed on it as a JSON string, and that a Name: inside the string is what its author wrote; it names no attack (a note that named one made a model fall for it more often, RMK-635). The request an ACP agent is prompted with is labelled the same way. One resolver gives that label wherever a model reads a participant's turn (the conversation, the room context handed to an ACP agent and the request that follows it, a line broadcast into a realtime session, the lines a memory summarizer reads), at one threshold: a reader labels once the turns it reads hold several speakers (a turn with no text, the runtime's own and the application's instruction aside), and a one-to-one conversation reads as before. A conversation a memory summarizes counts the turns the memory retrieved, summarized or kept, and the turn being answered: the summarizer's lines then give each turn's label out of its quote, [assistant] naming only the agent the summary is for, and the conversation labels its turns when the summary named several speakers. An ACP agent's session keeps what it was sent: once a request reached it labelled, after the note, every later request in that session is labelled, the note given once per session. The note is one of the runtime's marks: a copy of it in a participant's text is replaced as any other. A turn a memory's summary is joined to keeps its label out of the quote in every rendering of it (a thinker's transcript, a compaction's), the summary rendered before it, and a speak policy reads each turn under the label the conversation gives it. The label is named from the event's sender name (a transport's stamp, or the voice a channel's diarization names on a shared microphone, Section 12.2.3) and from the room's participant record otherwise (a participant's id names them on a channel they are reached through, an identity the identity pipeline resolved names them on any, Section 5.5): when two distinct sources of the room (a participant, whichever channel reached them, or a sender) have names that read alike (in case or in Unicode's confusables, Alice and Аlice), a participant the room registered keeps the name, then the first source the room saw, and each later one carries it with its rank (Alice (2)), a form no name takes; a name that reads like two others (Ian, like Lan and ian) ranks after both and renumbers neither. The name and the rank are fixed when the turn enters the room (Section 10.1, step 12) and recorded on the event, so every surface and every later prompt reads the same, and a participant renamed later keeps on their earlier turns the name they spoke under. A tool call the channel recalls in the turn's notes names its tool and its arguments' keys as identifiers and quotes each text value. The conversation itself (a participant's message, the agent's own answers, the application's instruction) keeps its author's role and is not quoted beyond the line labels above, a copy of the turn's notes' header it holds replaced as above; the runtime's own renderings of it (the transcript a thinker, a summarizer, a compaction, an external agent or a realtime delegation's reasoning backend reads) quote each message, after the name the runtime gave its speaker, never a name read from the message's text. The rule holds for the turn's notes, a hand-back, a speaker's name, the room context handed to an ACP agent and the summary a memory provider builds, and for any text a realtime session, an orchestration strategy or a vision provider places in a model's context.

A tool that has to appear. A provider that can hold a tool declared but unseen (Anthropic's deferred loading keeps such a definition out of the cached prefix) SHOULD receive in that form, from the first round of the turn, what Tool Search hides and what a skill's gating keeps closed (Sections 21.1, 24.2), and the result that opens such a tool (find_tools naming it, activate_skill unlocking it) makes it callable by reference instead of by a new declaration: the declaration then does not change within the turn at all. A result references exactly the tools the declaration described above would show after the call it answers, and a result that fails references none. A call to a held tool no result has referenced yet is recovered as a call to a tool Tool Search hides (Section 21.1), and its result references the tool itself. The tool policy still decides what is declared, hidden or not: a tool it denies is never declared; a gated tool so declared is still refused until its skill is active. A tool held unseen is reported in declared_tools once a result has made it callable, with the reason it is visible then (revealed for one find_tools named). A provider without that capability keeps the declaration described above; a fallback_provider without it receives the tools the turn made callable declared plainly and the held ones not at all. A compaction that summarizes a referencing result away declares the tools it referenced from then on, since the reference was what made them callable.

The declaration is the room's on such a provider, kept from one turn to the next: the tools shown at the first round of the room's first turn stay the tools shown, with the tools the channel never holds (its own, the tools orchestration injected, a tool a hook added; a tool the host pins is held while a skill's gating closes it), and every other tool the turn declares is held. A tool the turn would show beyond that declaration (one an earlier turn made callable, revealed, used or unlocked, or one added since) stays held, and the turn reopens it with an exchange placed after the history and before its input: a find_tools call where Tool Search is active, otherwise an activate_skill call for each active skill that gates such a tool, answered as activate_skill answers a skill already active, each with a result that references the tools it reopens. A tool neither reaches is shown instead. The exchange is the turn's context, never an event of the room: it is not stored, not delivered, and not counted as a tool call, and a fallback_provider that cannot hold a tool receives the tools it reopens declared plainly, without the exchange. A standalone turn (Section 10.1.1) neither reads nor keeps the room's declaration, and reopens nothing. A tool the turn no longer shows (the policy denies it, its skill closed) leaves the declaration, and a room whose declaration is not known (a new room, a restarted process) takes the one its turn would show and keeps it from then on.

One loop, streamed or not. An AI channel runs every turn through one tool loop, whatever its provider streams and whether the turn carries tools: a turn without tools is a loop whose first round asks for none. A provider that does not stream its structured events is read through generate_structured_stream's default, which wraps generate() and MUST hand the loop everything generate() returned: each thinking block with its signature, each call with its metadata (a thought signature among them), and the usage and metadata of the done event. A provider that streams text alone streams a turn without tools through generate_stream. Every rule this section gives a turn therefore holds once, for every provider.

A turn cut short. A tool loop that ends before its answer (the provider failed after at least one round, or the turn was cancelled between rounds) keeps what the room already has. The text each round said before its calls was delivered as its own message and MUST NOT be delivered again, and nothing from an earlier turn of the room MUST be delivered either: the history the model was given is context, not this turn's output. A turn the provider interrupted after a round is an error, surfaced as when the loop raises (ON_ERROR fires and the caller learns the provider's error), and its rounds are kept, the calls that ran included. No message is added to say so: the turn's record (below) says how it ended, and the provider's error never reaches the room. A cancellation adds nothing a round already said.

A stored interruption marker ([Response interrupted], a message carrying metadata["interruption_marker"] = true, a key distinct from the interrupted of a spoken reply a barge-in cut, Section 12.3.13) is not an answer. It MUST NOT solicit any intelligence channel (Section 19.3), and nothing that reads an agent's answer (an orchestration strategy, a delegated task's result) may take it for one. A turn the provider interrupted after a round has no answer: a task delegated to it fails with the provider's error, and carries the turn's end (error) and narration as a cut turn does (Section 23.3 step 6), while what reads the task tells the error from a cut; a strategy that reads an agent's answer reads none from an output that carries an error, its rounds' text included. A turn constrained to a schema is the exception: interrupted after a round, it fails with truncated, and ON_AI_RESPONSE does not fire (Appendix A.9).

A turn's budget. A channel MAY bound what a turn spends: a token budget over every token its provider bills for the turn (input, cache reads and writes, output; Section 6.7), a cost budget at the model's catalogue price, each generation priced as one response, or both, set for the channel, a room or a turn; a room or a turn that leaves a budget null keeps the next level's (Appendix A.9, per-turn settings). The loop checks them at each round boundary, on the usage the provider reported for the generations so far: a turn that has reached a budget ends there, budget_exceeded, running none of the calls its last generation asked for and asking for no further generation, a retry of an empty answer included; a final round that failed to answer past its budget ends budget_exceeded too. A budget that is not a positive number, or a cost budget for a model with no known price, is a configuration error, reported when the channel is built for a channel budget and by the turn that reads it otherwise. A generation a fallback provider serves is priced at the primary provider's rate.

A turn that did not complete. ON_AI_RESPONSE reports a turn whose loop reached its end, with the reason it ended for: cancelled for a cancellation between rounds, error for a turn the provider interrupted after a round (a turn constrained to a schema aside, Appendix A.9). The hook carries what its rounds used, then the provider's error surfaces as when the loop raises (ON_ERROR fires), through the turn's response stream once the hook has fired. A turn whose loop did not reach its end MUST NOT fire it: one that raised (the provider's interruption after a round aside), or one whose response stream was closed before the loop's end (a barge-in, a transport that stopped reading, a consumer that refused the answer, a task cancelled from outside). Its response never reached the end the hook reports, and a closed stream is never reported as completed; what its rounds delivered stays in the room. Its llm.generate span MUST end cancelled or error, never ok, and never stay open, and it carries what the rounds used, since no hook reports the turn. The span of a turn that reached its end cancelled (a steering Cancel) or error ends with that status too, never ok.

The turn's record. A turn that reached its end records it on its last MESSAGE: loop_end_reason (the reason above) and ai_usage (what its rounds used) in the message's metadata. That message is the one of the turn's final text or, when the turn has none (a cancellation between rounds, an interruption), the last message the turn wrote. A response without tools records it too; a provider that streams text alone reports no usage, and its record carries none. A turn learns its end after its segments are written, so the implementation updates the stored message, and ON_EVENT_UPDATED fires as for any change to a stored event; it does not deliver the message again. Writing that update is best effort: a failure is logged, and the turn's outcome stands. A host reads how any turn ended from its reply.

ACP Agent Channel:

An ACP agent channel connects a Room to an external Agent Client Protocol agent. In this integration RoomKit is the ACP client and the coding agent is the ACP server. Exposing a RoomKit-built agent as an ACP server is a separate integration surface and is outside this channel's contract.

An ACP agent channel:

1. MUST use ChannelType.AI, category INTELLIGENCE, and direction BIDIRECTIONAL;
2. MUST initialize one ACP connection per channel instance and create a distinct ACP session for each Room, opened once: callers that ask for it at the same time (a first prompt, a rebuild, a host setting a session value) find the session the first of them opened, and a host call made from inside a turn MUST NOT wait for that turn to end to get it;
3. MUST serialize prompts within an ACP session while allowing different Room sessions to operate concurrently;
4. MUST convert Room message content into ACP prompt content and map ACP agent message chunks back to the normal RoomKit streaming output path;
5. MUST map ACP tool-call lifecycle updates to RoomKit tool-call stream markers and MUST map ACP plan updates to RoomKit realtime events;
6. MUST skip events originating from itself and persisted tool-call activity events, preventing duplicate prompts and agent loops;
7. MUST support cancellation of an active Room turn and MUST release ACP sessions and transport resources when the channel closes. A turn is active from the moment it holds its Room, while the agent is started and its session is opened, set up or rebuilt: cancelled then, it MUST end without prompting, its response metadata marked interrupted, and a session opened for it alone (a standalone turn's) MUST be closed;
8. MUST NOT bypass the standard RoomKit persistence, visibility, chain-depth, or re-broadcast behavior for generated text and tool activity; and
9. MUST fire ON_AI_RESPONSE when a Room turn completes, carrying the text the turn produced, the number of tool calls it made, its duration, and how it ended as its loop_end_reason: the stop reason the agent returned, completed for end_turn. A turn that ended in an error, or whose response stream was closed before its terminal update, MUST NOT fire it: no response was delivered to the Room. Token counters the agent reports MUST be relayed unaltered. ACP annotates them as running session figures while agents are observed to report them per prompt; a client cannot distinguish the two from a single reading, and reinterpreting them silently corrupts an accounting figure. An implementation MAY carry the session's context occupancy and running cost alongside them under distinct keys; and
10. MUST expose an unclean prompt outcome on the turn's response-metadata record: a returned ACP stop reason other than end_turn is written as response_metadata["acp"]["stop_reason"], while a prompt that exits before returning a response is written as response_metadata["acp"]["interrupted"] = true. A clean end_turn writes neither marker. The same final record MUST reach the caller through InboundResult.response_metadata, including when the turn produced no final MESSAGE event.

The stable ACP protocol version MUST be the default. Experimental protocol versions MAY be supported only behind explicit opt-in and MUST NOT silently replace the stable default. The channel MUST negotiate the protocol version during initialization and MUST fail clearly when no compatible version exists.

The reference transport is ACP over stdio. A stdio implementation MUST launch the configured command as an argument vector without a shell. Alternate ACP transports MAY be implemented when standardized, but MUST preserve the lifecycle and permission requirements above.

ACP sessions are process-local by default. An implementation MAY add persistent session resumption, but MUST scope resumed session identifiers to the associated Room and agent endpoint.

Session setup. What a session is opened with is the integrator's, per Room. The channel's mcp_servers (default none) are the MCP servers given to each session in session/new: a list, the same for every Room, or a function of the Room identifier, called each time the channel opens a session for that Room, so that a server can be told which Room it acts for (a stdio server's environment, a URL). The channel's session_config (default none) maps ACP session configuration option identifiers to values (mode, model, an agent's own options). The channel MUST set each of them on every session it opens for a Room before the session is prompted: the Room's first session, one opened again after the previous one was invalidated or after a new connection to the agent, and a standalone turn's session (Section 10.1.1 step 7). A value the session already announces is not set again. A value the host sets on a Room's session while it lives (set_config_option) belongs to the Room as well: the channel MUST set it again on the Room's next session, over session_config, until the host closes the Room's session, which forgets it. A value the host sets while the Room has no open session is set on the session that call opens in place of the Room's earlier value for that option, so a value the agent has come to refuse is replaced by setting another one. An agent's mode decides what it asks the client before acting, and an agent that acts unasked never reaches the external tool handler: a session left on another value than the one set up is not degraded but ungated. So when the agent refuses a value, the channel MUST NOT prompt that session, MUST release it, and MUST fail the turn, or the call that was opening the session, with an error naming the option.

ACP agents can request permission to execute tools and can request client-side filesystem or terminal operations. An ACP agent channel:

• MUST advertise filesystem and terminal capabilities as unavailable unless the integrator explicitly supplies conforming implementations;
• MUST deny or cancel permission requests when no external tool handler is configured;
• MUST pass tool name, input, call identifier, ACP session identifier, and Room identifier to the configured external tool handler;
• MUST select only a permission option offered by the ACP agent; and
• SHOULD expose tool and plan progress through the realtime backend without persisting transient progress updates.

Identity and instructions. An ACP agent channel MAY carry an identity as an Agent does (Section 19.1): a name, a role and a description, which a discussion's notes, a dispatch policy and a host's view present it by (Section 19.7.5). Its instructions (default null) are the integrator's standing direction for the agent (a persona, how the room works), since the agent's own system prompt is not RoomKit's to set. The channel sends them in the first prompt of each session it opens for a Room, a session opened again after its transport died included, as the runtime's block ahead of the catch-up and the request, never as a participant's words, and not again while the session keeps them.

In a discussion. An ACP agent channel takes a discussion's turns as an AI channel does (Section 19.7.5). The turn's notes (rule 14) ride its prompt as the runtime's own block, under the notes' header, and every copy of that header in the prompt's other text (the catch-up, the request, a host's block) is replaced as Section 6.4 replaces it in a turn's input, so a participant cannot pass their words off as the runtime's notes. Its session keeps its own history, so the turn sends what the agent has not read yet, up to the turn's start, and the event the turn answers as its request; the notes say which event that is and who asked for it. A hold that cuts its turn (rule 12) cancels it (item 7), its session's opening included. Its answer streams like any agent's, so a silent one is held back and blocked as rule 13 says.

6.5 Source Providers

Source providers maintain persistent connections that push inbound events to the framework. This is distinct from webhook-based channels where the framework receives HTTP callbacks.

SourceProvider (interface)
├── id: string                              # Source identifier
├── status: SourceStatus                    # STOPPED, CONNECTING, CONNECTED, RECONNECTING, ERROR
│
├── start(emit: function(InboundMessage)) → void
│       # Connect and begin listening. Call `emit` for each inbound message.
│
├── stop() → void
│       # Disconnect and release resources
│
└── healthcheck() → SourceHealth
        # Report connection health

SourceStatus enumeration: STOPPED, CONNECTING, CONNECTED, RECONNECTING, ERROR

Examples of source providers:

Source	Description
WhatsApp Personal (neonize)	Persistent WhatsApp Web multidevice connection
WebSocket	Client connection to an external WebSocket server
SSE	Server-Sent Events stream from an HTTP endpoint

Sources are attached to the framework and MUST call the emit callback for each inbound message. The framework routes emitted messages through the standard inbound pipeline (Section 10.1).

6.6 Content Transcoding

When an event is broadcast to channels with different capabilities, the framework MUST transcode content to match each target channel's supported media types.

Default transcoding rules:

Source Content	Target Supports	Transcoded To
RichContent	TEXT only	TextContent (plain_text, else body unchanged)
MediaContent	TEXT only	TextContent ("[Media: {caption or filename or url}]")
AudioContent	TEXT only	TextContent (the transcript, or "[Voice message: {url}]")
VideoContent	TEXT only	TextContent ("[Video: {url}]")
LocationContent	TEXT only	TextContent ("[Location: {label or address} ({latitude}, {longitude})]")
CompositeContent	varies	Filter parts to target's supported types
TemplateContent	no templates	TextContent (use body or "[Template: {template_id}]")
EditContent	no edit support	TextContent ("Correction: {new text}")
DeleteContent	no delete support	TextContent ("[Message deleted]") or SystemContent

When the target channel supports edits or deletes natively (i.e., capabilities.supports_edit or capabilities.supports_delete is true), the framework MUST deliver the EDIT or DELETE event without transcoding. When the target channel does NOT support the operation, the framework MUST transcode to the fallback representation shown above.

Max length enforcement: After transcoding, if the target channel declares a max_length, the framework MUST truncate TextContent to that limit.

Implementations SHOULD allow integrators to provide a custom transcoding strategy.

6.7 Provider Abstraction

Channel type and Provider are separate concepts. This applies to ALL channels, including AI.

AI Provider interface:

AIProvider (interface)
├── name: string                            # Provider name (e.g., "anthropic", "openai")
├── model_name: string                      # Model identifier (e.g., "claude-sonnet-4-5")
├── supports_vision: bool (default false)   # Whether image parts reach the model
├── supports_streaming: bool (default false) # Whether generate_stream is implemented
├── supports_structured_streaming: bool (default false)
│       # Whether generate_structured_stream streams natively (else it wraps generate)
├── supports_response_schema: bool (default false)
│       # Whether generate honours AIContext.response_schema (see below)
├── supports_response_schema_with_tools: bool (default false)
│       # Whether a schema may ride a turn that also carries tools
├── context_window: int | null              # Input window of the active model, when known
│
├── generate(context: AIContext) → AIResponse
│       # Answer the conversation the context carries
│
├── generate_stream(context: AIContext) → async_iterator<string>
│       # Yield text deltas as they arrive (requires supports_streaming = true)
│       # Streams a turn without tools when the provider streams text alone
│       # (Section 6.4, One loop)
│
├── generate_structured_stream(context: AIContext) → async_iterator<StreamEvent>
│       # Yield thinking deltas, text deltas and tool calls, then one done event
│
├── available_models() → list<ModelInfo>
│       # Curated offline catalogue (MAY be empty)
├── list_models() → list<ModelInfo>
│       # Live catalogue; default: available_models() (see Model listings below)
│
└── close() → void
        # Release connections

AIMessage
├── role: string                            # "user", "assistant", "system", "tool"
├── content: string | list<AIContentPart>   # Text, image, tool call, tool result, thinking parts
└── metadata: map<string, any>

AIContentPart                               # One of these, told apart by type
├── AITextPart
│   ├── type: string = "text"
│   └── text: string
├── AIImagePart
│   ├── type: string = "image"
│   ├── url: string
│   └── mime_type: string | null
├── AIToolCallPart
│   ├── type: string = "tool_call"
│   ├── id: string
│   ├── name: string
│   ├── arguments: map<string, any>
│   └── metadata: map<string, any>
├── AIToolResultPart
│   ├── type: string = "tool_result"
│   ├── tool_call_id: string
│   ├── name: string
│   ├── result: string | list<AITextPart | AIImagePart>
│   ├── is_error: bool (default false)      # A failure or a refusal, carried, never inferred
│   ├── structured_content: map<string, any> | null  # MCP structuredContent; never rendered to providers
│   ├── outcome: string | null              # served, refused, failed, blocked, unserved or cancelled; never rendered to providers
│   └── references: list<string>            # Tools the result makes callable (Section 6.4)
└── AIThinkingPart
    ├── type: string = "thinking"
    ├── thinking: string
    ├── signature: string | null
    └── redacted: string | null             # A redacted block's opaque data; thinking is then empty

AIContext
├── messages: list<AIMessage>               # The conversation to answer
├── system_prompt: string | null
├── temperature: float                      # Default 0.7
├── max_tokens: int | null                  # null defers to the provider's configured cap
├── thinking_budget: int | null
├── enable_thinking: bool | null
├── reasoning_effort: string | null
├── tools: list<AITool>                     # Functions the model may call
├── response_schema: map<string, any> | null  # JSON Schema of the answer, see below
├── room: RoomContext | null                # Current room state
├── target_capabilities: ChannelCapabilities | null
├── target_media_types: list<ChannelMediaType>
├── metadata: map<string, any>
└── response_metadata: ResponseMetadata     # The turn's one record, see below

AIResponse
├── content: string                         # Generated text
├── thinking: string | null                 # Reasoning the provider exposed
├── thinking_signature: string | null       # The last block's signature, for a reader of one
├── thinking_parts: list<AIThinkingPart> | null  # The blocks to replay, in order; null for reasoning without blocks
├── finish_reason: string | null            # The provider's own stop reason
├── usage: map<string, int>                 # input_tokens, output_tokens, cache counters
├── tool_calls: list<AIToolCall>            # Calls for the tool loop to run
├── tasks: list<Task>                       # Tasks to create
├── observations: list<Observation>         # Observations to record
└── metadata: map<string, any>              # Provider-specific data (model, latency)

Model listings. A live listing backfills from the curated catalogue what the vendor's endpoint leaves blank (display name, context window, vision, price), but not its capabilities: a catalogue's tags MAY be internal routing flags, and an empty list means "unknown", never "none". A provider whose endpoint reports no model type at all MAY take the tags from its catalogue when that catalogue holds public tags only. A conversational provider's listing that surfaces speech models SHOULD tag them, as the image catalogue is tagged (Section 25.6): transcription for a speech-to-text model, speech for a text-to-speech one, from what the vendor reports or, failing that, from the model's name. A model that converses in audio is neither.

The model that answered. metadata.model names the model that answered, as the response names it (a dated snapshot, a router's pick), and the model asked for only when the response names none; a stream's done event names the same, so both modes report it alike.

Usage counters. input_tokens, output_tokens and the cache counters are disjoint, as in Section 25.5. A cache counter the vendor reports at zero is omitted, as one it does not report: on a text turn the two bill the same, and every provider reports them alike (an image's usage keeps a reported zero, Section 25.5). output_tokens counts every output token the vendor bills, a model's thinking included, whether the vendor reports that thinking inside its completion count or beside it. reasoning_tokens, when a provider reports it, is the thinking share of output_tokens: a detail, never added to the other counters nor priced a second time.

Reasoning settings. thinking_budget, enable_thinking and reasoning_effort are the turn's. Where a provider's configuration carries one of them under the same name, the provider MUST prefer the turn's value, false included. A vendor setting of the provider's own (Gemini's thinking_level, Ollama's think, PolarGrid's thinking) yields to the turn on what the turn states and supplies what it leaves unstated. Whether the model reasons is stated by thinking_budget (0 off, above 0 on), else by enable_thinking, and a reasoning_effort of none states off; how much it reasons is stated by reasoning_effort. Where the turn's settings come from several levels (Appendix A.9, per-turn settings), an enable_thinking: false set at one level states off over a thinking_budget set at a less specific one. Where such a vendor setting names levels, a level the model does not take is sent as the nearest one it takes, and left out where the provider cannot know the levels the model takes; and a model that cannot stop reasoning is sent its lowest level where the turn states off. The provider knows both from what its model catalogue declares of the model, or the levels from one its configuration already sets, never from the model's name. A provider MUST apply them on a turn that declares tools as on one that does not, save for the cases below. Where the vendor restricts what a turn with tools accepts (OpenAI's Chat Completions, for one, takes function tools only with reasoning_effort none on its recent models), the provider sends the value accepted, read from what its model catalogue declares of the model, never guessed from the model's name. A provider MAY leave a setting out of a turn with tools where it cannot know the model behind its endpoint (a base_url, a deployment name, a proxy's alias) and its configuration does not state that the server takes the setting there, or where turning reasoning on would require passing the model's reasoning back from one tool round to the next in a form it does not produce; it documents that it does, and MUST NOT leave it out otherwise. Such a statement covers only what the catalogue does not declare: a model the catalogue declares is sent the value the catalogue says it accepts, and a provider that knows its vendor's rule refuses the statement in its configuration rather than ignore it.

Response metadata is one record per turn. AIContext.response_metadata is a dict-like mapping created with the turn and shared by identity by every extension point of that turn: a memory provider MAY write it while the context is built, a BEFORE_AI_GENERATION hook MAY write it through event.ai_context.response_metadata, and a tool handler or a BEFORE_TOOL_USE hook MAY write it through current_response_metadata() (the hook runs under the call's turn, so a host counts the calls a turn started there, before any runs). The record a caller reads on InboundResult.response_metadata is the same one, whether the turn answered or failed. The caller's copy alone also names how each channel that replied to the caller's own event ended, streamed or not, under turns[channel_id]: its loop_end_reason (an AI channel's; an ACP agent's stop reason, interrupted when its prompt never returned or failed after, completed once it returned on end_turn) and the ai_usage its record carries (an ACP agent's carries none), so a turn that produced no MESSAGE still tells its caller how it ended and two agents replying to one inbound never overwrite each other's end. An answer to an answer is no reply to the caller and has no entry, nor has an ACP turn never prompted (a transport that read nothing). turns is the framework's key: a value a turn's hooks or tools write under it is not carried to the caller, and never on a MESSAGE. The framework MUST merge the record into the metadata of every MESSAGE event the turn produces, as the record stands when that event is created — a streamed segment persisted before a tool round carries what was known then, the final answer carries everything the turn learned. It is how a host attributes to the reply what the turn read (cited sources, say) without a post-hoc rewrite of persisted events.

Response schema. A caller that needs the answer in a known shape sets AIContext.response_schema to a JSON Schema. A provider whose supports_response_schema is true MUST translate it into its native constrained output, so that the content returned by generate() is one JSON document satisfying the schema. The implementation MUST check the document against the schema before returning it, whatever the server claims: a proxy may accept the constraint and route to an upstream that ignores it. The subset below is small enough to check without a JSON Schema library. content stays a string all the same, and turning it into a typed value is the caller's work.

The schema MUST stay within the portable subset below, the one every provider that supports response schemas accepts. An implementation MUST refuse a schema outside it when the context is built or the field assigned, before any provider is called.

Keyword	Rule
type	One of object, array, string, number, integer, boolean, as a single string. The root MUST be object.
properties	REQUIRED on an object: a map from names to subschemas, possibly empty.
required	REQUIRED on an object: every key of properties, each once.
additionalProperties	REQUIRED on an object, and false.
items	REQUIRED on an array: one subschema.
enum	On a string only: a non-empty list of distinct strings.
title, description	Allowed on any subschema, as strings: guidance for the model.

No other keyword is portable. null and optional fields, anyOf, $ref, and numeric or string bounds each break on at least one provider. An answer that may be absent is a required field with an agreed empty value (an empty string, an empty list).

A provider MUST NOT ignore a schema it cannot honour, and MUST NOT return text that does not answer it. generate() and the streaming methods raise a ResponseSchemaError, a ProviderError that is never retryable, with one of these reasons:

Reason	When
unsupported	supports_response_schema is false, or the context also carries tools and supports_response_schema_with_tools is false. Raised before any request is sent.
refusal	The model declined to answer: a refusal field, a refusal or safety stop reason, or the prompt itself blocked before any answer.
truncated	The answer was cut: a length or max-tokens stop reason, or the context window filling up mid-answer.
invalid_json	The text is not a JSON document satisfying the schema: a server that accepted the constraint and did not apply it.

Streaming a constrained answer. generate_stream() and generate_structured_stream() carry the schema like generate(). The text deltas are the document as it is written: partial JSON, provisional until the stream ends. The implementation MUST check the whole document, as generate() does, before it yields the final done event, and MUST raise the error above instead of that event when the check fails. A consumer MUST NOT act on the streamed text before the done event, which is what marks it as a document satisfying the schema.

Tools in the same turn. A context may carry tools and a schema together where supports_response_schema_with_tools is true. The model then either calls tools or answers in the schema: a response that carries tool calls is a step of the tool loop and is not checked, and the check applies to the answer that carries none, the final one. A provider whose constraint is a decoding grammar (it forces the output into the schema token by token) MUST leave that capability false: the grammar stops the model from calling a tool, and it writes a schema-valid answer in its place, invented rather than looked up, which no check can tell from a true one. A response that ended on a call the provider would not hand over (Gemini's MALFORMED_FUNCTION_CALL, UNEXPECTED_TOOL_CALL) is a step of the tool loop as well, which tells the model and asks again (Section 6.4): it is not checked, carries no document, and says so by that finish reason, which a caller of generate() outside the loop reads.

Support depends on the model and, behind an OpenAI-compatible base URL, on the server, so supports_response_schema is a provider default that an implementation SHOULD let the host override in the provider's configuration. A consumer that must run on any provider reads the property, and where it is false asks for JSON in the prompt and parses the answer itself.

Declaring tools (normative). A provider declares each tool of the context in its vendor's format. A tool without parameters (an absent or empty schema) is declared as an object schema with no properties, which every vendor accepts, never as an empty map, and a schema whose root has no type is declared an object's, for every vendor: Anthropic and OpenAI refuse it untyped. Vendors accept different tool names, so the names are checked in two places:

• An implementation refuses, when a tool is defined, a name no vendor accepts: an empty one, or one with a character other than a letter, a digit, _, ., : or -. A loader of tools defined elsewhere (MCP) skips such a tool with a warning and keeps the others.
• Each provider checks its own vendor's rule when it declares tools, an AI provider a turn's, a realtime provider a session's (at connection and at every reconfiguration), and for a name its vendor refuses it raises, before the request, an error naming the tool and the rule, rather than letting the vendor reject the request mid-turn, or a realtime endpoint accept the declaration and fail the call later, opaquely. Where a realtime endpoint hands the tools to another model (a hosted reasoning backend, an agent's think provider), the rule is that model's vendor's. A provider that cannot know its server's rule (any server behind a custom base URL, a vendor-compatible gateway included) checks none: the server decides.

A tool result on the wire. A provider renders each tool result as its vendor's format carries one. Where that format has an error flag on a result (Anthropic's is_error, Gemini's error key), a result whose call was refused, failed, blocked, served by nothing or cancelled carries it; the text the model reads is the same either way.

A request the model cannot serve. Where a provider knows, from its model catalogue, that the model refuses function tools on the endpoint it calls, or that the endpoint does not serve the model at all, the request fails before it is sent, with a non-retryable error naming the model. Behind a base URL the server decides.

What a failed generation reads as (normative). A provider MUST raise a failure as a ProviderError whose retryable flag does not depend on the provider or on the form the server gave the failure. Its status is the one the server described: the HTTP status of an error response, else the status or error type written in the error's body, as when a stream already answered 200 carries an error event (Anthropic's overloaded_error reads as the 529 it stands for). A status of 408, 409, 429 or any 5xx is retryable; any other status is final. Without a status, a lost connection (refused, reset or timed out) is retryable, and an answer that could not be read is final: a body that is not the provider's format, such as a gateway's HTML page in place of JSON, including a stream that ends without one event, never reads as an empty success. A 200 whose body is an error object reads as the status it names. A provider whose SDK drops a status it does not classify cannot read it, and its failure is final; any other failure without a status is retryable only when its message names a retryable status, a rate limit or an overload, as whole words. One exception is kept: Ollama reports a stream its server aborted (its template failed to parse the model's own output) with no status, a generation defect that a new sample may not repeat, and it reads as a lost connection.

SMS Provider interface:

SMSProvider (interface)
├── send(event: RoomEvent, to: string, from: string) → ProviderResult
├── parse_webhook(payload: map) → InboundMessage
└── verify_signature(payload, signature, timestamp) → bool

Email Provider interface:

EmailProvider (interface)
├── send(event: RoomEvent, to: string, from: string, subject: string | null) → ProviderResult
└── parse_inbound(payload: map) → InboundMessage

Implementations SHOULD define similar provider interfaces for WhatsApp, Messenger, Teams, RCS, HTTP, Voice (STT, TTS), and any custom channel types.

6.8 Classifiers

A classifier answers narrow, typed questions about a state (a text, or structured data such as a turn with the conversation around it) with probabilities, not generated text. It is how a component asks for judgment where code needs understanding (was the agent addressed, did the person finish) without a generation: the component asks its questions together, in one call, and composes the answers in code, where each one stays visible and measurable. Support is OPTIONAL.

Classifier (interface)
├── classify(state: string | JSON, questions: map<string, Question>) → Answers
└── close() → void

YesNoQuestion
├── instructions: string                # The question, and what it means
├── yes: string | null = null           # What a yes covers, when that needs saying
└── no: string | null = null            # What a no covers

ChoiceQuestion
├── instructions: string
└── options: map<string, string>        # Each option and what it covers; two at least

ScoreQuestion
├── instructions: string
└── levels: list<string>                # Ordered levels, lowest first, each described; two at least

YesNoAnswer
└── probability: float                  # Of yes, in [0, 1]

ChoiceAnswer
├── choice: string                      # The most probable option
└── probabilities: map<string, float>   # Every option's

ScoreAnswer
├── score: float                        # The expected level: 0 for the first, a fraction between two
└── probabilities: list<float>          # Each level's, in the levels' order

Answers: map<string, Answer>            # By question name, every question answered

A classifier answers every question it was asked, or raises ClassifierError for whatever kept it from answering (a refused request, an answer it cannot read, the end of its wait, which it bounds). It sends the questions' wording as the caller wrote it. An implementation says how far its probabilities can be trusted: a trained classifier's are calibrated; a generative model asked for one answer gives the answer it chose probability 1, and is documented as such. close() releases what the classifier opened itself; a provider or a client it was handed stays its owner's.

RoomKit provides a mock (scripted answers), a classifier on any AI provider that answers under a JSON schema (Section 6.7), uncalibrated, Jev (TypeSafe's System One model), calibrated, behind an optional extra, and OpenAI's Decisions API (gpt-6-luna), behind the OpenAI extra. The Decisions API answers the same three kinds of question with probabilities (predicate for a yes/no question, choice, score over labelled levels), over a text: a structured state is sent as its JSON. OpenAI advises calibrating thresholds on labelled data, and the classifier is documented as such. A question the API declines to answer (a refusal) fails the call with ClassifierError, since a classifier answers every question or none. A speak policy built on judgments (Section 6.4) is a classifier's first consumer; a discussion's dispatch policy (Section 19.7.5) is another.

---

7. Permission Model

The permission model consists of three orthogonal primitives. The framework provides the primitives; the integrator decides when and how to use them.

7.1 Access

Controls whether a channel can read events, write events, or both within a room.

Value	Can Read	Can Write	Description
READ_WRITE	Yes	Yes	Full participation
READ_ONLY	Yes	No	Observe only (events via on_event())
WRITE_ONLY	No	Yes	Blind sender (unusual, but valid)
NONE	No	No	Fully disconnected (binding exists but inactive)

7.2 Muting

A boolean flag on the binding. When muted = true:

• The channel STILL receives events via on_event() (reading is preserved).
• The channel's response events are not broadcast (writing is blocked): each re-enters and its commit pass stores it BLOCKED with source_muted, after its hooks (§7.5 rule 2); a streamed one MAY be closed unread instead.
• Side effects (tasks, observations) are STILL collected.

Muting is temporary. The integrator calls mute() and unmute() as needed.

7.3 Visibility

Controls which channels see events produced by the source channel. Set on the binding as a string value:

Value	Meaning
"all"	All channels in the room see the event
"none"	No channels see the event (stored in timeline only)
"transport"	Only transport channels see the event
"intelligence"	Only intelligence channels see the event
"channel_a,channel_b"	Comma-separated list of specific channel IDs
"channel_a"	Single channel ID

"See" is not "be delivered to". "none" reads stored in timeline only, and it means it: an event a channel was not allowed to see at delivery MUST NOT reach that channel later through history the framework rebuilds for it either. Rule 8 in §7.5 states that half.

7.4 Named Patterns

These are vocabulary for common configurations, NOT framework concepts. The framework only knows access, muted, and visibility.

Pattern	Configuration	Use Case
Direct	access=READ_WRITE, visibility="all"	AI speaks to everyone in the room
Assistant	access=READ_WRITE, visibility="ws_advisor"	AI whispers only to the advisor
Observer	access=READ_ONLY	Sentiment analyzer watches, produces side effects only
Muted	muted=true	AI temporarily silenced but still tracking
Internal	access=READ_WRITE, visibility="none"	Writes stored in timeline but never broadcast

7.5 Permission Rules

Implementations MUST enforce these rules:

1. Reading: A channel receives events via on_event() if and only if its binding has access ∈ {READ_WRITE, READ_ONLY} AND the event's visibility includes this channel.
2. Writing: A channel's response events are broadcast if and only if its binding has access ∈ {READ_WRITE, WRITE_ONLY} AND muted = false. Likewise, an inbound or directly-injected event whose source binding cannot write (access ∉ {READ_WRITE, WRITE_ONLY} OR muted = true) MUST be stored with status BLOCKED (blocked_by = source_read_only or source_muted) rather than DELIVERED, and MUST NOT be broadcast. A response re-enters like any other event (§10.1 step 14) and meets this check after its BEFORE_BROADCAST hooks (step 11 after step 9), on each path that commits a response: a buffered response, a streamed row, a regenerated answer. The check reads the source's binding as the hooks left it, so a hook that mutes the source of the event it reads blocks that event. The greeting, and the trace a delegation writes into a child room no transport is shared into (§23.3), and a record an integrator commits outside the pipeline (§10.5), are committed by paths of their own and are not covered here; a child room with a shared transport commits its agent's response as a room does, and is covered. A muted or read-only source's response is therefore stored BLOCKED with what its hooks decided collected (rule 3), and a hook that blocks it names the block. A streamed response from a source that cannot write is piped live to no channel: a read-only source's stream is read to its end and each row stored BLOCKED; a muted source's MAY be closed unread, the reply never generated.
3. Side effects: Tasks and observations are ALWAYS collected regardless of access or mute status — including for events blocked by rule 2.
4. Visibility filtering: When broadcasting an event, the framework MUST check the source binding's visibility and deliver only to channels included in the visibility rule.
5. Self-skip: A channel MUST NOT receive its own events via on_event().
6. A binding is never widened implicitly. Wherever the framework creates a binding from an existing one — sharing a channel into a delegated room (§23.3), copying a template, re-attaching a channel it detached earlier — the new binding MUST NOT grant more than the one it derives from. Carrying over a binding's category and metadata while letting access, visibility and muted fall back to defaults silently promotes a read-only observer into a full participant, in a room the integrator never configured. Widening a binding is an explicit act: it MUST come from an integrator call that says so, never from a default filling a gap.
7. Attaching a channel is a decision, not a repair. An implementation MAY attach a channel automatically when a message arrives for a room the channel is not yet bound to. It MUST NOT do so when the channel was previously detached from that room, and MUST NOT use such an attach to replace a binding that already exists — detaching is how an integrator revokes a channel's access, and an automatic re-attach at default permissions undoes exactly that.
8. Reconstructed context obeys the same rule as delivery. Rule 1 governs on_event(); this one governs everything the implementation rebuilds from the timeline for a channel — the history handed to a memory provider, the turns assembled into a model prompt, any per-channel replay of what was said. A channel MUST NOT obtain there an event visibility withheld from it at delivery. Withholding an event at broadcast and returning it one turn later as context is not a partial enforcement of visibility; it is none.

The same holds for an event delivered to nobody. An event stored BLOCKED — refused by a hook (§10.1 step 10), written by a source that could not write, read-only or muted (step 11, rule 2), stopped by the chain-depth cap or the reentry cap (§8.3) — MUST NOT reach any channel's reconstructed context, its own source's included: the room refused that turn, and a model that reads its own refused answer as history continues from a turn nobody received. The record stays in the timeline for host code and audit (the point on host code below applies to it unchanged), behind an explicit request: a timeline read serves the rows the room received by default and the BLOCKED rows only when asked (§14.1), a read by id is unconditional, and the history rebuilt for hooks stays whole.

The filter is per reader, so it MUST be applied where the reader is known — a single RoomContext shared by every channel of a broadcast cannot be filtered once for all of them. An implementation MUST apply it before the history reaches a component that can transform it (a summarizing or compacting memory provider re-emits hidden content as a summary otherwise), and SHOULD expose the per-reader view so host code can ask for it.

Three points the rule does NOT cover:

• A channel always sees its own accepted events. Rule 5 skips a channel's own event at delivery because it produced it, not because it may not know it. A rule that dropped an agent's own turns from its own prompt would erase the assistant pattern of §7.4 (visibility="ws_advisor") from the inside. Its own refused turns are the exception stated above: an answer nobody received, a muted channel's included, is not history the channel may continue from.
• Host code reads the timeline whole. Hooks, scorers and framework machinery run in the integrator's process and hold the store already; filtering their recent_events would forbid nothing and would break legitimate readers. Visibility is a rule between the room and its channels.
• Visibility is a live property of the binding. An implementation that resolves a source's visibility from its binding at read time MUST accept that widening a binding widens its past too, and MUST document it. One that stamps the effective visibility onto the event at commit gets a replay-stable answer instead. Either is conformant; the choice MUST be stated. When the source binding no longer exists, the event's own visibility field is the whole answer — an implementation MUST NOT treat an unresolvable source as a reason to hide ordinary history, which would empty a room's context the moment a transport detaches.

---

8. Event System

8.1 Room Events

Room events are stored in the room's timeline. They have a sequential index that is monotonically increasing within each room.

Sequential indexing requirements:

• The index MUST start at 0 for the first event in a room.
• Each subsequent event MUST have index = previous_index + 1.
• The index MUST be assigned atomically per room. A room-level lock (§13.5) serializes assignment within a single process; it is NOT sufficient when a persistent store is shared across processes (e.g. a load-balanced deployment), because per-process locks do not coordinate. Such a store MUST make index assignment authoritative at the storage layer: a UNIQUE(room_id, index) constraint, with the index computed and the event inserted in a single storage transaction, so concurrent writers serialize on the store rather than on an in-process lock.
• A duplicate (room_id, index) MUST be rejected by the store (the constraint violation surfaced to the caller), never silently persisted.
• The index enables pagination, read horizon tracking, and gap detection.

8.2 Framework Events

Framework events are global lifecycle notifications published to subscribers. They are NOT stored in any room timeline.

Event	When	Data
room_created	New room created	room_id
room_paused	Room transitioned to PAUSED by its inactivity timer	room_id, reason (timer)
room_closed	Room transitioned to CLOSED	room_id; reason (timer) when its inactivity timer closed it
room_archived	Room transitioned to ARCHIVED	room_id
room_refused_event	A room whose status refuses new events turned one away (§5.1)	room_id, event_id, status, operation, event_type
channel_registered	Channel registered with framework	channel_id, channel_type
channel_unregistered	Channel unregistered	channel_id, channel_type
room_channel_attached	Channel attached to a room	room_id, channel_id, access, visibility
room_channel_detached	Channel detached from a room	room_id, channel_id
channel_connected	WebSocket connection registered for a room	channel_id, room_id, connection_id
channel_disconnected	WebSocket connection unregistered	channel_id, connection_id
source_connected	Source provider connected	channel_id, source_name
source_disconnected	Source provider disconnected	channel_id, source_name
source_attached	Source provider attached to its channel	channel_id, source_name
source_detached	Source provider detached from its channel	channel_id, source_name
source_error	A source provider's run failed	channel_id, source_name, error, attempt
source_exhausted	Source provider given up after its last restart attempt	channel_id, source_name, attempts, last_error
event_processed	An event that entered through the inbound pipeline or direct injection (§10.5) finished its delivery set and its AFTER_BROADCAST hooks; not emitted for the events its processing produced (responses, streamed rows, hook-injected events)	room_id, event_id
event_blocked	Event blocked by a hook or by a source that cannot write (§7.5 rule 2); a chain-depth record emits chain_depth_exceeded instead	room_id, event_id, channel_id (the event's source), reason, blocked_by
process_timeout	The pre-commit phase exceeded process_timeout (§13.6)	room_id, channel_id (expired before the room lock) or event_id (under it), timeout; operation = regenerate for a regeneration, which carries neither channel_id nor event_id
delivery_succeeded	A transport channel's deliver() succeeded, for an event that entered through the inbound pipeline or direct injection (§10.5); a response's deliveries are not reported	room_id, event_id, channel_id
delivery_failed	A target's on_event() or deliver() failed (deliver() after its retries), for the same events as delivery_succeeded	room_id, event_id, channel_id, error
broadcast_partial_failure	delivery_failed was emitted for at least one of an event's targets; the event stays DELIVERED (§13.6)	room_id, event_id, failed, total, errors (channel_id → error)
delivery_skipped	A room's delivery lane passed over committed indexes whose delivery never came (left, for example, by a process that crashed before delivering them) after waiting gap_timeout (default 30 s)	room_id, from_index, to_index
identity_resolved	The inbound pipeline ended with an identity for the sender, found by the resolver or supplied by a hook	room_id, channel_id, event_id, identity_id, display_name, status (the resolution's status)
identity_timeout	Identity resolution timed out, for an inbound sender or a conference participant	room_id, channel_id, timeout (seconds)
verification_requested	A verification request was opened (§11.7.2)	identity_id, request_id, room_id, channel_id, method, expires_at
verification_completed	A verification request completed (§11.7.2)	identity_id, request_id, room_id, channel_id, method, expires_at
verification_ended	A verification or a request ended: expired, revoked, failed, cancelled (§11.7.5)	identity_id, request_id (for a request), reason
chain_depth_exceeded	Event blocked by chain depth limit	room_id, event_id, channel_id, data.chain_depth, data.max_chain_depth
session_started	ON_SESSION_STARTED fired (§9.2): a voice session started, or the inbound pipeline auto-created a room for a text channel	room_id; session_id, channel_id (voice) or channel_id, channel_type (text)
before_ai_generation	BEFORE_AI_GENERATION decided on an AI turn (allowed when no hook ran)	room_id, channel_id, allowed, blocked_by
ai_response	An AI channel's turn completed (ON_AI_RESPONSE)	room_id, channel_id, tool_calls_count, latency_ms, streaming
before_tool_use	BEFORE_TOOL_USE decided on a tool call	room_id, channel_id, tool_name, tool_call_id, allowed, reason
tool_call	A tool call was reported (§9.3), whatever its outcome	room_id, channel_id, tool_name, tool_call_id, channel_type; is_error, cancelled, refused, refused_but_ran when true
user_input_required	A tool call asked for the user's input; ON_USER_INPUT_REQUIRED decided (allowed when no hook ran)	room_id, channel_id, pending_id, tool_name, tool_call_id, allowed, reason
feedback	Feedback submitted for a room or a response (ON_FEEDBACK)	room_id, channel_id, event_id, dimension, rating
status_posted	An entry was posted to the StatusBus	ts, agent_id, action, status, detail, metadata
hook_error	A BEFORE_BROADCAST SYNC hook failed at an event's commit gate (§10.1 step 9): it raised, timed out or returned an unusable result	room_id, event_id, hook, error
hook_timeout	A hook, SYNC or ASYNC, exceeded its timeout	room_id, hook_name, trigger, timeout (seconds)
circuit_breaker_opened	Channel circuit breaker tripped	channel_id, state (open)
circuit_breaker_closed	Channel circuit breaker recovered: a call succeeded after it tripped	channel_id, state (closed)
voice_session_started	Voice session transitioned to ACTIVE	session_id, room_id, channel_id
voice_session_ended	Voice session transitioned to ENDED	session_id, room_id, channel_id
recording_started	A recording started: a voice session's audio, a room's media (Section 12.11) or a conference track (Section 12.10)	room_id, id; session_id (voice); scope = room (room); channel_id, track_id, participant_id (conference track)
recording_stopped	A recording stopped, its result available	room_id, id, duration_seconds; session_id (voice); scope = room, url (room); channel_id, track_id, participant_id, url, size_bytes (conference track)
stt_error	STT transcription failed	session_id, provider, error
tts_error	A session's TTS synthesis or its playback failed	session_id, provider, error
voice_session_ready	Voice session audio path is live and ready	session_id, room_id, channel_id, participant_id
video_session_started	Video session started	room_id, session_id, channel_id
video_session_ended	Video session ended	room_id, session_id, channel_id
video_vision_result	A vision provider analysed a video session's frame	room_id, session_id, channel_id, description, labels, confidence, text, faces (count), elapsed_ms
conference_started	Bot connection to the conference is live	room_id, channel_id, bot_session_id
conference_ended	Bot left the conference	room_id, channel_id, bot_session_id, duration_ms
conference_participant_joined	Participant joined the media session	room_id, channel_id, participant_id
conference_participant_left	Participant left the media session	room_id, channel_id, participant_id
conference_bot_grants_changed	A connected bot session's effective grants changed (Section 12.10.4)	room_id, channel_id, bot_session_id, hidden

An implementation MUST emit each event of this table for the features it provides: the rows of a feature (voice, recording, video, conference, AI channel, tool calls, event sources, WebSocket connections, StatusBus) are owed only by an implementation that provides it. Integrators subscribe to framework events for monitoring and integration purposes.

room_refused_event carries one data shape whichever path refused, so a subscriber reads it without knowing which one did:

• status — the room status that refused (closed or archived); null only when the room no longer exists.
• operation — the path that met the gate: inbound for a message or a direct injection refused at §10.1 step 6; reentry for the framework's own re-injection of a response; regenerate for a regeneration of the last answer.
• event_type — the type of the refused event; null when a regenerate had nothing to replay.

event_id is the refused event's id — an id that names nothing in the store, since a refused event is never appended — or, for a regenerate, the message it would have replayed, null when nothing qualified.

8.3 Event Chain Depth

When a channel produces a response event (e.g., AI responds to a message), that response may trigger further responses from other channels. This creates an event chain.

Chain depth tracking:

• Events from external inbound (human messages, webhooks) have chain_depth = 0.
• An event a channel produces in answer to another carries that event's chain_depth + 1, whichever path produces it: a buffered response, each row of a streamed one (text segments and tool-call rows), the assistant transcription of a speech-to-speech provider (Section 12.4), the result an orchestration strategy returns for a turn (Section 19.7), the turn a discussion gives in answer to the event it answers (Section 19.7.5), and a result delivered back to the room for a turn that delegated (Section 23.3). A path that restarts the count at 0 opens a chain the limit never reaches.

At the limit. When an event's chain_depth + 1 >= max_chain_depth, an intelligence channel it solicits (Section 19.3) MUST NOT be asked to act: its on_event() is not called, so no model is called and no tool runs, whether the channel streams or not. The limit exists to stop what a runaway chain does, not only what it says. In place of the response, the implementation MUST store one record per channel not asked:

• status = BLOCKED and blocked_by = "event_chain_depth_limit";
• the channel not asked as its source, and empty text as its content;
• as chain_depth, the depth the response would have had;
• its own observation, and a chain_depth_exceeded framework event.

A tool-call row (TOOL_CALL_START, TOOL_CALL_END) is an activity record that no agent answers, and gets no record. Response events a channel other than an intelligence one returns at the limit are stored BLOCKED with the same blocked_by.

Below the limit, a response that was started is read. A buffered response's events re-enter (§10.1 step 14), and a streamed response is consumed to its end, whichever pass started it: the trigger's delivery set, a reentry pass, or the delivery of a streamed segment. A response generated and then discarded costs its model call and runs its tools, with nothing of it in the room.

Reentry cap. The responses one cascade re-enters (the answers to one inbound or injected event, one regeneration or one delegated run) share a reentry budget of 10 × max_chain_depth passes, buffered responses and chained streams alike (§10.1 step 14). A buffered response past the budget does not re-enter, and a chained stream past it is closed unread, so nothing is generated. Either is stored BLOCKED with blocked_by = "reentry_loop_cap"; neither emits chain_depth_exceeded nor event_blocked.

Requirements:

• Implementations MUST support a configurable max_chain_depth (default: 5). A discussion's max_depth takes its place for the turns the discussion gives and the events they produce (Section 19.7.5), and cannot be disabled either (Section 17.5).
• Blocked records MUST still be stored in the timeline (for audit), whichever path broadcast the trigger, a regeneration included.
• A channel that was not asked has no side effects to collect. Side effects of a channel whose response events were blocked MUST still be collected.
• A framework event chain_depth_exceeded MUST be emitted for each record.

8.4 Realtime / Ephemeral Events

Some events (typing indicators, presence changes) are ephemeral — they are published in real-time but not stored in the timeline.

RealtimeBackend (interface)
├── publish(room_id: string, event_type: string, data: map) → void
├── subscribe(room_id: string, callback: function) → subscription
└── unsubscribe(subscription) → void

Implementations SHOULD provide at least an in-memory realtime backend for single-process deployments.

8.5 Responses: What They Answer, and Whether They Were Consumed

An intelligence channel answers events: a participant's message, an instruction (Section 10.1.1), a delegated task's hand-back (Section 23.3). Several answers can be under way in one room at once: the room lock does not cover generation (Section 13.5). An answer therefore says which event it answers, and the room records whether each answer reached the person it was delivered to. With both, a host can tell each turn what it answers and where the room's other requests stand: answered, being answered, taken in or not.

What a response answers. Every event an intelligence channel produces for a turn (each message, streamed segment and tool row of it) MUST carry responds_to: the id of the event that triggered the turn. An instruction is never stored, but it has an id, which its hooks see; an answer to it carries that id. An event that answers no event (an inbound message, a system event, a greeting) has responds_to = null. A turn the channel runs again for the same event carries the same responds_to. A buffered answer a channel already named keeps its name. The pair (source.channel_id, responds_to) names one channel's answer to one event. parent_event_id is unrelated: it places an event in an in-app thread, and always names the thread's root. A speech-to-speech channel (Section 12.4) is not bound by this yet: its answer to the user's speech starts before that speech's transcript is committed, so there is no event to name, and its answers carry responds_to = null (Planned).

What the turn is told. BEFORE_AI_GENERATION receives, with the turn's context, the event the turn answers (trigger), so a hook can tell the model what this turn answers, and that another answer in the room is under way. A stand-in an orchestration strategy hands a turn in place of the event it answers (a supervisor's workers' results, Section 19.7) names that event in its own responds_to, which is what the turn's answer names.

Consumption. A response is delivered to channels (Section 10.2). Whether the people it was delivered to took it in is its consumption, recorded per delivering channel and, where the channel can tell them apart, per participant:

Consumption                                 # One delivering channel's record
├── state: "consumed" | "partial" | "unconsumed"
├── at: datetime                            # When the state was last recorded
├── detail: map<string, any>                # How far, by channel: played_ms, played_percentage
└── participants: map<string, ParticipantConsumption>  # By participant id; empty when
                                                       # the channel cannot tell them apart

ParticipantConsumption
├── state: "consumed" | "partial" | "unconsumed"
├── at: datetime
└── detail: map<string, any>

A channel that serves its participants separately (a voice session per caller, a read receipt per reader), or knows who was present while it played a shared stream (a conference room's subscribers), records each participant. Every participant served by the channel has an entry. The channel's own state is then the least consumed of its participants (unconsumed before partial before consumed), so consumed says everyone took it in, and the entry of the participant who sent the triggering event says whether the asker did. A channel that cannot tell its participants apart (one microphone shared by several people) records its own state alone.

• consumed: taken in whole. A voice response played to its end, or a response a READ_RECEIPT of the delivering channel covers (its index at or below the receipt's).
• partial: taken in part. A voice response cut by a barge-in (Section 12.3.13), with detail.played_ms, and detail.played_percentage when the total duration is known.
• unconsumed: delivered, but nothing of it reached the person. A voice response cancelled as superseded before any of its audio played (Section 12.2).

The record is the same on every event of the answer, which is every event that has the same source.channel_id and responds_to. A channel that cannot tell records nothing: a missing entry, the channel's or a participant's, means unknown, not unconsumed. For the channel and for each participant, a later state replaces an earlier one; consumed is final, so a response read after it was cut becomes consumed, and a consumed response stays consumed. Each change fires ON_CONSUMPTION (ASYNC) in the room, with the answer's events, its responds_to, the delivering channel, the participant when one changed, and the new state. Consumption says nothing of delivery: an event that failed to deliver has its delivery_results, and no consumption.

---

9. Hook System

Hooks are the primary extensibility mechanism. They allow integrators to intercept, block, modify, and react to events in the pipeline.

9.1 Hook Registration

HookRegistration
├── trigger: HookTrigger                    # When this hook fires
├── execution: HookExecution                # SYNC or ASYNC, on any trigger
├── handler: function                       # The hook function
├── priority: int = 0                       # Execution order (lower = first)
├── name: string                            # Human-readable identifier
├── timeout: float = 30.0                   # Maximum execution time (seconds)
├── channel_types: set<ChannelType> | null  # Filter: only fire for these channel types
├── channel_ids: set<string> | null         # Filter: only fire for these channel IDs
├── directions: set<ChannelDirection> | null # Filter: only fire for these directions
├── fail_closed: bool = false               # SYNC only: an unusable result blocks (§9.3)
└── needs_lock: bool = true                 # BEFORE_BROADCAST SYNC only: false runs
                                            # the hook off the room lock (§9.5.1)

Hooks MAY be registered globally (apply to all rooms) or per-room.

needs_lock = false is only meaningful on a SYNC BEFORE_BROADCAST hook; implementations MUST reject it on any other trigger or execution mode at registration. Likewise fail_closed = true MUST be rejected on an ASYNC hook, which cannot block. A room's BEFORE_BROADCAST SYNC hooks MUST satisfy one ordering rule: no hook that needs the lock may have a lower priority than a hook that does not. Off-lock hooks run first (§9.5.1), so a locked hook ordered before an off-lock one could not run in its declared place; implementations MUST reject the registration that would break the rule, naming both hooks, rather than silently reorder.

9.2 Hook Triggers

HookTrigger enumeration. The Status column distinguishes triggers the reference implementation emits today (Implemented — 83 as of this revision) from those specified for a forthcoming capability but not yet emitted (Planned), and from a trigger kept for historical reference whose behaviour has moved elsewhere (Superseded). Conformance targets the Implemented set; Planned rows are normative design intent for the named capability.

The Execution column says how the framework runs the trigger; it does not restrict registration, and a hook may be registered SYNC or ASYNC on any trigger. On a SYNC trigger the hooks registered SYNC run as the chain of Section 9.3, which may block or modify; then, unless the chain blocked, the hooks registered ASYNC run as observers on the payload it left (ON_TOOL_CALL tells its observers of a block too, Section 9.3). On an ASYNC trigger every registered hook runs, concurrently, whatever its declared mode, and what it returns is discarded. ON_IDENTITY_AMBIGUOUS and ON_IDENTITY_UNKNOWN are decided by the identity hooks of Section 11.2, registered apart; a hook registered on them as above runs after those as an observer.

Trigger	Execution	Status	When It Fires
BEFORE_BROADCAST	SYNC	Implemented	Before event reaches channels — can block/modify
AFTER_BROADCAST	ASYNC	Implemented	After all channels have processed the event
ON_EVENT_UPDATED	ASYNC	Implemented	A persisted event's stored state was mutated (inbound EDIT, §10.3, or direct update_event())
ON_EVENT_DELETED	ASYNC	Implemented	A persisted event was deleted (inbound soft DELETE, §10.3, or direct hard delete_event())
ON_CHANNEL_ATTACHED	ASYNC	Implemented	Channel attached to a room
ON_CHANNEL_DETACHED	ASYNC	Implemented	Channel detached from a room
ON_CHANNEL_MUTED	ASYNC	Implemented	Channel muted in a room
ON_CHANNEL_UNMUTED	ASYNC	Implemented	Channel unmuted in a room
ON_ROOM_CREATED	ASYNC	Implemented	New room created
ON_ROOM_PAUSED	ASYNC	Implemented	Room transitioned to PAUSED
ON_ROOM_CLOSED	ASYNC	Implemented	Room transitioned to CLOSED
ON_IDENTITY_AMBIGUOUS	SYNC	Implemented	Multiple identity candidates found
ON_IDENTITY_UNKNOWN	SYNC	Implemented	No identity match found
ON_PARTICIPANT_IDENTIFIED	ASYNC	Implemented	Participant identity resolved
ON_VERIFICATION_REQUIRED	SYNC	Implemented	An identified sender's message arrives while the identity is not verified — the hook holds, passes or blocks it (§11.7.3)
ON_VERIFICATION_COMPLETED	ASYNC	Implemented	A verification request completed (§11.7.2)
ON_VERIFICATION_ENDED	ASYNC	Implemented	A verification or a request ended: expired, revoked, failed (§11.7.5)
ON_PARTICIPANT_JOINED	ASYNC	Implemented	Explicit member join via add_member
ON_PARTICIPANT_LEFT	ASYNC	Implemented	Explicit member leave via remove_member
ON_PARTICIPANT_UPDATED	ASYNC	Implemented	Member presentation changed via rename_member
ON_TASK_CREATED	ASYNC	Implemented	A task was created
ON_DELIVERY_STATUS	ASYNC	Implemented	Delivery status webhook from provider
ON_ERROR	ASYNC	Implemented	A turn failed, or a speech-to-speech provider reported an error; payload and categories below the table
ON_SPEECH_START	ASYNC	Implemented	Audio pipeline detected speech start (voice; per conference lane)
ON_SPEECH_END	ASYNC	Implemented	Audio pipeline detected speech end (voice; per conference lane)
ON_TRANSCRIPTION	SYNC	Implemented	After STT transcription (voice) — can modify
BEFORE_TTS	SYNC	Implemented	Before TTS synthesis (voice) — can block or modify the text; once per sentence on a streamed response
AFTER_TTS	ASYNC	Implemented	After TTS synthesis (voice)
ON_BARGE_IN	ASYNC	Implemented	User interrupted TTS playback (voice)
ON_TTS_CANCELLED	ASYNC	Implemented	TTS was cancelled (voice)
ON_PARTIAL_TRANSCRIPTION	ASYNC	Implemented	Streaming partial STT result (voice)
ON_VAD_SILENCE	ASYNC	Implemented	Audio pipeline detected silence (voice)
ON_VAD_AUDIO_LEVEL	ASYNC	Implemented	Audio pipeline audio level update (voice)
ON_INPUT_AUDIO_LEVEL	ASYNC	Implemented	Per-frame inbound audio level, throttled to ~10/sec (voice)
ON_OUTPUT_AUDIO_LEVEL	ASYNC	Implemented	Per-frame outbound audio level, throttled to ~10/sec (voice)
ON_SPEAKER_CHANGE	ASYNC	Implemented	Audio pipeline or a diarizing STT detected a speaker change (Section 12.2.3)
ON_DTMF	ASYNC	Implemented	Audio pipeline detected a DTMF tone
ON_TURN_COMPLETE	ASYNC	Implemented	Turn detector determined user turn is complete
ON_TURN_INCOMPLETE	ASYNC	Implemented	Turn detector determined user is still speaking (for logging)
ON_BACKCHANNEL	ASYNC	Implemented	Backchannel detector classified speech as backchannel
ON_SESSION_STARTED	ASYNC	Implemented	Session started on any channel type (voice: audio path live; text: room auto-created)
ON_RECORDING_STARTED	ASYNC	Implemented	Recording started for a voice session, a conference track (Section 12.10.8) or a room (Section 12.11)
ON_RECORDING_STOPPED	ASYNC	Implemented	Recording stopped for a voice session, a conference track or a room, result available
ON_REALTIME_TOOL_CALL	SYNC	Superseded	Speech-to-speech tool call — superseded by ON_TOOL_CALL (unified across AI and realtime channels)
ON_REALTIME_TEXT_INJECTED	ASYNC	Implemented	Text injected into realtime session
ON_REALTIME_DELEGATION	ASYNC	Implemented	Speech-to-speech model handed reasoning or tool use to a backend, hosted or integrator-side (Section 12.4.1); carries the delegation id and its target
ON_PROTOCOL_TRACE	ASYNC	Implemented	Transport-level protocol trace emitted (SIP, RTP, etc.)
BEFORE_BRIDGE_AUDIO	SYNC	Implemented	Before an audio frame is forwarded via bridge — can block/modify (voice)
Delivery:
BEFORE_DELIVER	SYNC	Implemented	Before proactive delivery strategy executes — can block/modify
AFTER_DELIVER	ASYNC	Implemented	After proactive delivery completes
ON_CONSUMPTION	ASYNC	Planned	A response's consumption changed on a delivering channel: consumed, partial or unconsumed (Section 8.5)
AI Generation:
BEFORE_AI_CONTEXT_BUILD	SYNC	Planned	Before an AI channel builds a turn's context (before memory retrieval and tool resolution) — can block the turn cheaply (Section 6.4, Anatomy of an AI turn); required by nothing today
ON_SPEAK_DECISION	ASYNC	Implemented	An AI channel's speak policy decided whether the agent speaks, offers or stays silent on an event (Section 6.4)
ON_THOUGHT	ASYNC	Implemented	An AI channel's thinker came back with what the agent has in mind in a room (Section 6.4)
ON_SPEAK_QUEUE	ASYNC	Implemented	A discussion's speak queue changed: an agent queued, a turn given or ended, an agent listening only or talking again, the discussion waiting for a person or over (Section 19.7.5)
ON_DISPATCH_DECISION	ASYNC	Implemented	A discussion's dispatch policy decided which agents take a person's message that names none (Section 19.7.5 rule 18)
BEFORE_AI_GENERATION	SYNC	Implemented	Before AI provider generate() — can modify context; a turn's (answer) or a thinker's (thought, Section 6.4)
ON_AI_THINKING	ASYNC	Implemented	AI model began extended thinking/reasoning
ON_AI_RESPONSE	ASYNC	Implemented	A turn of intelligence completed (observability). Fired by any channel of category INTELLIGENCE, whether the turn ran in-process or in an external agent (Section 6.4)
BEFORE_TOOL_USE	SYNC	Implemented	Before a tool executes — can block or override the call
AFTER_TOOL_ROUND	SYNC	Implemented	After a round of an AI channel's tool loop ran its calls, before the next round is built — can withdraw tools for the rest of the turn and add messages for the next round (Section 6.4)
ON_TOOL_CALL	SYNC	Implemented	A tool call reached its outcome — served, refused before execution, failed during it, or abandoned by the model before its result (unified across AI and realtime channels; Section 9.3)
ON_USER_INPUT_REQUIRED	SYNC	Implemented	Human-in-the-loop: a tool paused, waiting for user input
Orchestration:
ON_PHASE_TRANSITION	ASYNC	Implemented	Conversation phase changed (e.g., triage → specialist)
ON_HANDOFF	ASYNC	Implemented	Agent handoff accepted and executed
ON_HANDOFF_REJECTED	ASYNC	Implemented	Agent handoff was rejected
ON_STATUS_POSTED	ASYNC	Implemented	Status posted to the inter-agent StatusBus
Delegation:
ON_TASK_DELEGATED	ASYNC	Implemented	Background task delegated to a child room
ON_TASK_COMPLETED	ASYNC	Implemented	Delegated task ended: completed with its result, failed with its error, or cancelled
Video:
BEFORE_BRIDGE_VIDEO	SYNC	Implemented	Before video frame is forwarded via bridge — can block/modify
ON_VIDEO_SESSION_STARTED	ASYNC	Implemented	Video session became active
ON_VIDEO_SESSION_ENDED	ASYNC	Implemented	Video session ended
ON_VIDEO_TRACK_ADDED	ASYNC	Planned	Video track added to session
ON_VIDEO_TRACK_REMOVED	ASYNC	Planned	Video track removed from session
ON_VISION_RESULT	SYNC	Implemented	VisionProvider returned an analysis result — can block it, or modify its description, labels, confidence, text or faces
ON_SCREEN_SHARE_STARTED	ASYNC	Implemented	Screen sharing started: a conference participant published a screen-share track (Section 12.10.4)
ON_SCREEN_SHARE_STOPPED	ASYNC	Implemented	Screen sharing stopped: a conference screen-share track was unpublished (Section 12.10.4)
ON_VIDEO_DETECTION	ASYNC	Implemented	Video detection event (object, face, etc.)
Conference: (SFU)
ON_CONFERENCE_PARTICIPANT_JOINED	ASYNC	Implemented	Participant joined the conference media session
ON_CONFERENCE_PARTICIPANT_LEFT	ASYNC	Implemented	Participant left the conference media session
ON_CONFERENCE_TRACK_PUBLISHED	ASYNC	Implemented	Conference participant published a track (audio, video, screen share)
ON_CONFERENCE_TRACK_UNPUBLISHED	ASYNC	Implemented	Conference track unpublished
ON_CONFERENCE_TRACK_MUTED	ASYNC	Implemented	Publisher muted their track — "camera off" included
ON_CONFERENCE_TRACK_UNMUTED	ASYNC	Implemented	Publisher unmuted their track
ON_ACTIVE_SPEAKER_CHANGED	ASYNC	Implemented	SFU reported a dominant-speaker change
ON_CONNECTION_QUALITY_CHANGED	ASYNC	Implemented	SFU reported a participant's connection quality
Other:
ON_PLAN_UPDATED	ASYNC	Implemented	An AI channel's plan_tasks tool stored a new task plan for the room; the payload carries the room, the channel, the plan's tasks (title, status) and a timestamp
ON_FEEDBACK	ASYNC	Implemented	submit_feedback recorded a rating (0.0–1.0, clamped) as an Observation of the room; the payload is that Observation, carrying the rating, its dimension (default overall), the comment and the rated event

ON_ERROR payload. ON_ERROR receives a RoomEvent that is never stored. Its source is the channel that failed (on a realtime channel, with the session's participant), its content is a text whose body is the error message, and its metadata carries error (the same message), error_type (the error's type name, or the provider's error code) and error_category:

error_category	Fired when
generation	An intelligence channel failed to respond to a broadcast event
streaming	A streamed response raised, a delegated task's turn included
realtime_provider	A speech-to-speech provider reported an error, fatal or not, or a conference's lazy connect to it failed (Section 12.10.12)
reasoning	A reasoning backend's turn raised (Section 12.4.1)

On a room turn the event's chain_depth is the triggering event's plus one, and its visibility is the triggering event's response visibility. A transport's delivery failure does not fire ON_ERROR.

9.3 Hook Execution Modes

SYNC hooks:

• Run sequentially, ordered by priority (lower number = first).
• Each hook receives the event and room context.
• Each hook MUST return a HookResult, except the identity hooks that decide ON_IDENTITY_AMBIGUOUS and ON_IDENTITY_UNKNOWN, which are registered apart and return an IdentityHookResult or null (Section 11.2).
• A BLOCK result stops the pipeline — no further hooks run.
• A hook that does not produce a usable result MUST be treated as ALLOW with an error logged, so that a broken hook cannot take a room down. This covers a hook that raises, one that exceeds its timeout, one that returns something other than a HookResult, and a MODIFY whose payload is not of the type the trigger passed in.
• Except on three triggers, which fail closed: BEFORE_TTS, ON_TRANSCRIPTION and BEFORE_TOOL_USE, where every one of those outcomes MUST block instead. Every other trigger, BEFORE_BROADCAST and ON_TOOL_CALL included, is fail-open unless a hook declares itself fail-closed (below). BEFORE_TTS and ON_TRANSCRIPTION carry content a hook may exist to withhold: a redaction hook that fails, times out, or returns something unusable must not let the original through, and allowing would publish exactly what the hook was there to suppress. On a streamed TTS response BEFORE_TTS runs on each sentence (Section 12.2, step 12s.b), so this applies to every sentence: a streamed response is never a way around the hook. BEFORE_TOOL_USE carries an action a hook may exist to prevent: it is the gate of a tool call, where an approval hook sits, and an approval hook that cannot answer must not let the call run. A partial rule is no rule — an implementation that blocks on exceptions but allows on timeouts leaks through the timeout.
• A BEFORE_TOOL_USE that failed closed refuses the call before execution, as its BLOCK does, on every channel. A BLOCK's reason is the hook's own words for the model, and the model reads them ({"error": "<reason>"}) on every channel and every entry of one, as it reads the reason of an ON_TOOL_CALL block. A BLOCK with no reason, and a hook that failed closed, give the fixed refusal ({"error": "Tool 'x' denied by pre-execution hook."}), never the hook's error, whose message can hold anything the hook held. The hook's name and error go to the log and to ON_TOOL_CALL's observers, on the refused call's error_detail. An external tool handler decides and reports its calls itself: the refusal reaches it as a denied decision that carries the hook's error for it alone, and what the agent reads, and what it reports, are the handler's; a handler that hands that error on with its decision has it reported on the refusal's error_detail, as on the channel's own doors.
• A hook MAY declare itself fail-closed (fail_closed = true), on any trigger whose SYNC hooks run as a chain (Section 9.2): a content check on BEFORE_BROADCAST (PII, moderation) is exactly such a hook, while the trigger as a whole stays fail-open so that a broken logging hook cannot take every room down. Every unusable outcome of a fail-closed hook MUST block, and the block MUST name the hook and the outcome so the sender can be told why: blocked_by is the hook name and the reason is hook_timeout:<name>, hook_error:<name> or hook_invalid_result:<name>.
• A MODIFY result replaces the payload whenever one is supplied, including a falsy one. Redacting a string to empty is a modification; treating it as absent would pass the original to the next hook.

ASYNC hooks:

• Run concurrently after the triggering operation completes.
• Cannot block or modify events.
• Exceptions MUST be caught and logged, never propagated.
• Used for observability, logging, side effects.

Who serves a call. An AI channel decides it call by call, not turn by turn. A call the provider already ran is reported: the provider says so on the call, with its result and whether it failed, never in its arguments, which are the model's text: a key a model writes there (_result) is an argument like any other, and its call is judged as any call. A call an external tool handler is configured to decide is that handler's; any other call is the channel's own, through the pre-execution gate, then its handler, and is a call nothing served when no handler takes it. A call the provider ran therefore never reaches the channel's handler because the channel has tools of its own, and a BEFORE_TOOL_USE BLOCK refuses every call the channel serves, a channel without a handler included. A tool a BEFORE_AI_GENERATION hook withdrew is gone from the turn: a call to it is the channel's to refuse, never an external handler's to decide. A round that mixes both kinds hands the next round every call it made, each with its result: the model reads the calls served outside the channel beside its own. A call the turn cut before its report (the turn's cancellation, a transport that stopped reading before the call was decided) is reported once, cancelled, to ON_TOOL_CALL's observers alone, whoever was to serve it: an external handler hears of it through on_tool_cancelled, which reports it so by default, and never through on_tool_result, which reports an outcome. The handler may never have been asked about the call; a decision of its still pending is its own to withdraw. The channel's close is such a cut: an AI channel that closes cancels the calls its running turns execute, each reported once, cancelled, its end row too, asks no further round, and each such turn ends cancelled, as a speech-to-speech channel's close interrupts its calls (Section 12.4). A handler that raises while it decides fails the call, as a tool handler that raises does. A call whose outcome the model already read (the provider ran it, an external handler decided it, its realtime result went out) is still owed its report when a cut interrupts that report: it is reported once, to ON_TOOL_CALL's observers alone, with that outcome. A report is claimed where the observers hear it, past the SYNC chain and once their context is built, so a report a cut interrupted is still owed and one the observers heard is never made twice; an ACP call's end, once closed, reaches its report even when the task carrying it is cut. A call issued once its session ended runs no gate and is reported once, cancelled. A call cut after its report keeps its stored end row cancelled when the turn ended before the model read its result: the row says what the turn delivered, the report what ran.

ON_TOOL_CALL, where the call was served:

These hold where the channel's own tool loop serves the call. An external handler's firing is a report on a call that ran outside the channel, and so is a firing on any call whose outcome the model already read (a call the provider ran itself): its override is discarded, and the display payload an external agent attaches (ACP's tool content) is that agent's. No hook can rewrite or withhold what the agent already read, so its observers see the provider's outcome and the provider's own failure marker, whatever a SYNC hook returned, a BLOCK included.

• A SYNC hook sees the call's result and its structured copy (the payload a tool publishes for UI surfaces beside its result, such as MCP structuredContent). It MAY replace the result, and MAY replace or clear the structured copy. A result replaced alone keeps the copy: a hook that rewrites text, such as PII re-tokenisation, is not withholding the payload.
• A BLOCK withholds the result: the model reads the block's reason as a failed call, and the tool-call event carries the failure marker and no structured copy. An event that kept the payload while the text was withheld would publish, to every surface that renders it, what the hook was there to withhold.
• A call that failed or was refused carries no structured copy either.
• SYNC hooks apply in turn to one outcome: each sees the result and the structured copy as the previous hook left them, whichever form its replacement took (a MODIFY whose event carries a new result, or the result override). The model reads what the last one left. A second hook handed the original would undo the first: "redact", then "cite the source", would publish what the redaction removed. A MODIFY whose payload is not a tool-call event replaces nothing, as the general rule above has it: the chain carries on from the outcome the previous hook left, that hook's own override included.
• A replacement counts whatever its value: a hook that clears a served result (replaces it with no value) has the model read that empty value, serialized as any other result (JSON null), on every channel, never the original.
• ASYNC observers run after the SYNC hooks, on the final outcome: the result the model reads, as it reads it (a replacement serialized), never one a hook replaced. A BLOCK does not spare them, nor does the failure of a fail-closed hook that withheld the result: they fire with the failure marker and the reason the model reads, since every call MUST fire ON_TOOL_CALL with its outcome (below).

ON_TOOL_CALL, where the call was refused or failed:

• Every tool call MUST fire ON_TOOL_CALL with its outcome, whether it was served, refused before execution (undeclared tool, invalid arguments, tool policy, skill gating, a BLOCK from BEFORE_TOOL_USE), failed during execution (a handler that raised), or reached no servant at all. A trigger that fires only for the calls that succeeded cannot support an audit trail: a refused agent and an idle one produce the same record, which is none.
• Every report of a call, on every door, carries the arguments the call ran with, or those the gate had when it stopped it (repaired, rewritten by BEFORE_TOOL_USE); the model's own only for a call stopped before its arguments were read. A call its turn or its session cut while it ran is reported with the arguments it ran with.
• Such a firing MUST NOT be able to serve the call. Where the trigger's SYNC hooks do serve tools — the AI tool loop, the realtime channel — that means dispatching to the hooks registered ASYNC only: a refused call reaching a servant would let the refusal hide the side effect instead of preventing it, and the pre-execution gate of Section 12.4 would hold under some configurations and not others. Where the tool already ran outside the channel and the firing is a report by construction — an external tool handler relaying its provider's outcome — it MAY reach every hook, and any result such a hook offers MUST be discarded. A call that never ran is no such report: a refusal on every door (the pre-execution gate's, an external handler's, a call the external door refuses itself, a rejected ACP permission, a refusal ACP imposes on an approval it cannot apply), a handler that raised while it decided, and a cancellation reach the ASYNC observers only, marked refused or cancelled beside is_error, so an audit tells a refusal from a failure and from a cancellation. An external handler hears its own refusal through on_tool_refused, which reports it so by default, and never through on_tool_result; a handler that raises while it decides fails the call, which the channel reports itself, with what failed (error_detail), as it reports a call it refused itself. A handler that raises while it reports a call (its outcome, its refusal, its cancellation) is logged and does not fail the turn: the channel reports the call itself, once, as it ended, unless the handler's report had already reached ON_TOOL_CALL's observers, on every door that hands a call's report to a handler (an AI channel's external door, ACP); an override that cannot take the arguments the door hands it raises the same way. An ACP call reports the same body with an external handler as without one: a cancellation's envelope, a failure's bounded error. An ACP call whose permission RoomKit rejected (it refused the call, or its handler raised deciding it) that the agent ran anyway and closed completed is reported as it ended, served, its markers those of a served call, and MUST carry a marker of its own (refused_but_ran) on its report and its end row, so an audit sees the rejection the agent went past; that report carries what failed in the rejection (error_detail) when something did, whether the handler or the channel makes it, but for a handler whose override cannot take it, which reports the call without it and is logged, as a refusal's detail is.
• The outcome MUST be carried on the event as a discrete marker, not left to be inferred from the result body. A refusal's body is a body like any other: one implementation's refusals are JSON error envelopes, another's are the prose the model is meant to read, and an external provider's are whatever it printed. An observer that has to recognise a failure in the text will read a refused call as a completed one, and a tool whose own output resembles a refusal as a failure.
• That obligation covers every outcome the channel itself determines: a refusal by the pre-execution gate, a tool not available in the turn, a guard the channel applies to the call (a repeat of the same call with the same arguments), a handler that raised, a call nothing served, and the verdict an external tool handler received from its provider. A handler that instead returns a refusal has stated it in its body, where only its own convention can be read — Section 15.8.1 governs that reading. A producer that can state the outcome structurally SHOULD do so rather than leave it to be recognised.
• The body MUST travel verbatim beside that marker: it is what the model is told, and its wording answers to that reader alone. An oversized body reaches the model through the bound of Section 21.5 like any tool result; the event carries it as the tool produced it.
• A call whose handler raised MUST read the same on every channel: the tool's failure, named by the exception's class and nothing more ({"error": "Tool 'x' failed (<ExceptionClass>)"}), with the failure marker. The exception's message can hold anything the failing code held (a connection string with its password, a path, a record), and neither the model nor its provider's logs are a place for it. The message goes to the log and to ON_TOOL_CALL's observers, on the event's error_detail, never to the model nor to the stored TOOL_CALL_END. A handler that wants the model to read its words says which outcome they carry: a refusal (ToolRefusedError, nothing ran) or a failure (ToolFailedError, the tool ran and could not do it), whose text passes as it is, the failure's also as the event's error_detail. An MCP tool whose result says isError ran and failed: it is a failure, never a refusal, on every channel. A tool the channel serves itself (a skill script, a sandbox command) reads the same when what it runs raises, and so does any work a tool hands off whose failure the model reads (a delegated task, a reasoning backend's tool). A call nothing served whose hooks raised reads as any call nothing served: the hooks' messages go to error_detail.
• A result override supplied on such a firing MUST be ignored. Nothing ran, so there is no result to correct.
• A call that no handler and no hook served MUST report a failure, not a success. Reporting one — {"status": "ok"} and its kin — tells the model work was done that nobody did, and records a completed call for a tool that was never reached. A declared tool that no handler serves reaches the SYNC hooks with no result: a hook MAY serve it by supplying one, and that result is then the call's. If none does, the call is reported once, as failed; the firing that offered it to the hooks was a chance to serve it, not a report.
• A hook that empties a served call's result (an override of null) leaves the call served, with a null result: the next hook in the chain reads a served call whose result is null, never a call nothing served, and does not get to serve it. Every channel applies a verdict through the same reading (a block, an override, a bare override, an emptied result).
• When the context the hooks need cannot be built, the hooks do not run and the call keeps the outcome its handler gave it, on every channel: a context that fails is not the tool's failure. The ASYNC observers need that context and do not run either; the call is still reported once, by the tool_call framework event for a served call, and as the failure it is for a call nothing served. So for a refused, failed or cancelled call whose observers' context fails, and for a call an external handler ran: its tool_call framework event still reports it once.

ON_TOOL_CALL, where the model abandoned the call:

• A speech-to-speech model interrupted while a call is outstanding MAY discard it (Gemini Live's tool_call_cancellation; a reconnect that orphans every call the previous connection issued, blocking or not, since call ids are connection-scoped and the new socket never issued them). A provider that learns of it MUST tell the channel (on_tool_call_cancelled, Section 12.4). A provider whose protocol carries no such event fires nothing: its calls stay in the conversation and their results are still read.
• The channel MUST interrupt the handler still running for such a call and MUST NOT send its result. The model will not read it, and a result sent for an id the provider no longer knows is an error the application never asked for.
• The call MUST fire ON_TOOL_CALL's observers with a discrete cancelled marker beside the failure marker. An abandoned call is neither a refusal nor a failure, and an audit counting refusals must not count it as one; it is still not a served call, so a consumer reading the failure marker alone reads it correctly as work that did not complete. The firing is observational, as above.
• A call the channel no longer holds when the cancellation arrives (its result already left) has no outcome left to report: the provider drops the stale result and the channel reports nothing.
• A reconnect the call's own handler caused (it reconfigured its session and the provider reconnected to apply it) is not the model abandoning that call. The channel MUST NOT interrupt that handler, which still has work to do on the new connection (a handoff greets as the new agent next), and MUST NOT send its result, since the new socket never issued the id. The call's outcome is reported as it would be otherwise, served when the handler returns a result, and never as cancelled. The reconnect is the call's own when it runs within the handler's execution, the handler itself or a task it started while the call was outstanding; a reconfiguration handed to work that does not descend from the handler is not, and abandons the call. Every other call the same reconnect orphaned is abandoned as above.

ON_USER_INPUT_REQUIRED, where the other party is a human:

• A human-input request MUST be answerable from the moment it is created. The notification MUST NOT gate whether an answer can be recorded, and an answer arriving while the notification is still running MUST be honoured. The human is not inside the implementation's timing: they answer the question they were shown, and a notification that fans out slowly, or a hook that spends its whole timeout, is not a reason to lose their answer.
• These hooks keep every SYNC property that matters to their authors — priority order, and a BLOCK that rejects the request. What a BLOCK does not do is retract an answer already recorded; a request that has been answered is settled.
• Implementations MUST NOT report an answer that was recorded as a failure to the caller that asked for it. An outcome already delivered SHOULD stay readable for long enough that a second read, or a caller that lost track of the request, reads the outcome rather than an error.
• A human-input tool call follows the line between a refusal and a failure: a request a human or a BLOCK rejected is a refusal, the model reading why; one nobody answered in time is a failure, the tool having run without an answer; one the handler gave up on before an answer (its close, a release), whether before the call or while it waits, and any other error take the generic failure path, the message withheld from the model. A request whose waiting call is cut (a turn cancelled, a session ended, a call abandoned) is withdrawn as a release withdraws it: no answer is taken for it after.
• Human-input tools have these rules on every channel that serves them (an AI channel, a speech-to-speech channel, a conference) and on every door of that channel (a speech-to-speech session's provider calls, the calls it recovers from speech, a reasoning backend's): the channel declares and serves them itself, before the host's handler, which replacing that handler leaves alone; their own timeout bounds the call (Section 21.6); each request fires ON_USER_INPUT_REQUIRED and names the channel type of the door it was asked on; the requests still open are settled when the channel closes, or when what asked them is taken out (a conference's speech-to-speech provider unplugged). Tool Search never hides them, and a human-input definition given twice, or under a name the channel serves itself, is refused when the handler is given, as a host tool is (Section 21.1). A human-input handler given as a plain tool handler is a host handler, bound by none of these rules. A realtime pipeline's agent never carries human-input tools of its own: the session serves the channel's (Section 19.5).

9.4 HookResult

Sync hooks MUST return a HookResult, the identity hooks of Section 11.2 aside:

HookResult
├── action: "allow" | "block" | "modify"
├── event: any | null                       # The replacement payload (for "modify" action)
├── reason: string | null                   # Why blocked or modified
├── injected_events: list<InjectedEvent>    # Events to inject when blocking
├── tasks: list<Task>                       # Tasks to create (side effects)
├── observations: list<Observation>         # Observations to record (side effects)
└── metadata: map<string, any>              # Additional hook metadata

event carries the replacement payload for a "modify" result, and its type is whatever the trigger passed in. Only BEFORE_BROADCAST and BEFORE_DELIVER (Section 22.3) pass a RoomEvent: the TTS trigger receives a string, the bridge triggers receive media frames, and the tool and generation triggers receive their own event types. Typing this field as a RoomEvent would make "modify" unusable for every trigger but one, so consumers MUST check the type they expect before substituting it.

InjectedEvent:

InjectedEvent
├── event: RoomEvent                        # The event to inject
└── target_channel_ids: list<string> | null # Deliver to specific channels (null = store only)

9.5 Hook Pipeline (BEFORE_BROADCAST)

Inbound Event
      │
      ▼
┌──────────────────────────────────────┐
│ Sync Hooks (ordered by priority)     │
│                                      │
│ [0] Hook A → allow / block / modify  │
│ [1] Hook B → allow / block / modify  │
│ [2] Hook C → allow / block / modify  │
└──────────────┬───────────────────────┘
               │
          blocked? ──yes──→ Store event (status=BLOCKED, blocked_by=hook_name)
               │              Deliver InjectedEvents to target channels
               │              Persist tasks and observations
               │
               ▼ (allowed, possibly modified)
         Event Router
         Broadcasts to channels
               │
               ▼
┌──────────────────────────────────────┐
│ Async Hooks (fire and forget)        │
│                                      │
│ [·] Audit Logger                     │
│ [·] Analytics                        │
│ [·] Webhook Notifier                 │
└──────────────────────────────────────┘

9.5.1 Off-lock checks (needs_lock = false)

A check that only reads the event — a PII scan, a moderation call — does not need the room lock, and running its I/O under the lock makes every other event of the room wait behind it (§13.5). A SYNC BEFORE_BROADCAST hook registered with needs_lock = false runs before the room lock is taken, on the inbound pipeline (§10.1 step 5a) and on direct injection (§10.5):

5a. OFF-LOCK CHECK (only when an off-lock hook matches the event)
    ├── Take the room's admission ticket (arrival order, per process)
    ├── Run the off-lock hooks in priority order, on the pre-lock context
    │   # Each bounded by its own timeout; checks of different events of
    │   # the same room overlap
    ├── Wait for every earlier ticket of the room to be released
    ├── Acquire the room lock (step 6) and continue; step 9 reuses the
    │   off-lock outcome and runs only the hooks that need the lock
    └── Release the ticket once the locked section has decided and committed

• Order. Events that took a ticket commit in ticket order, so two events from one sender keep their arrival order even when the second one's check finishes first. The ticket is held in process memory: the guarantee holds within one process. Events that match no off-lock hook take no ticket and are not ordered against ticketed ones — a system notice does not wait for a scan it has nothing to do with.
• Release. A ticket MUST be released on every path: committed, blocked by a hook or a gate, refused, timed out, failed, or cancelled. A ticket left held stalls every later ticketed event of the room.
• Bound. Waiting for a turn adds nothing to process_timeout (§13.6): it counts against the same pre-commit budget. The wait is bounded by the earlier holders' hook timeouts and locked sections, never by a separate setting.
• Decision. An off-lock BLOCK or MODIFY is applied at step 9, after the status, idempotency and EDIT/DELETE gates, which keep their order and still decide first. The side effects (injected events, tasks, observations) of both phases are collected together.
• Context. An off-lock hook receives the context built before the lock. Its history can be stale by the time the event commits; a hook that reads the room's state rather than the event MUST keep needs_lock = true.
• Reentrance. An event injected (§10.5) by a caller that already stands in the room's admission order takes no ticket in that room: from inside an off-lock hook's body, or while holding the room lock (a locked hook, code under the lock). It would otherwise wait for the ticket its own caller holds. It runs every hook under the lock and commits ahead of the event whose processing emitted it.
• Other commit paths (reentry passes, regeneration, hook-injected events) run every BEFORE_BROADCAST SYNC hook under the lock, the off-lock ones included. needs_lock = false permits a hook to run off the lock; it never lets an event skip it. A streamed row is the exception (§4): it runs every hook, the needs_lock = true ones included, and commits without the room lock; the store assigns its index atomically (§8.1), and the room's status is checked again after its hooks.

9.6 When to Use What

	Sync Hook	Async Hook	Read-Only Channel
Can block events	Yes	No	No
Can modify events	Yes	No	No
Can inject targeted events	Yes (on block)	No	No
Can produce tasks/observations	Yes	No	Yes
Can produce response messages	No	No	No (read-only)
Runs	Before broadcast	After broadcast	During broadcast
Typical use	Rule-based filtering	Logging, analytics	AI-powered analysis

---

10. Processing Pipelines

10.1 Inbound Pipeline

The inbound pipeline processes messages arriving from external sources.

process_inbound(message: InboundMessage, room_id: string | null) → InboundResult

Step-by-step:

1. RESOLVE CHANNEL
   ├── Look up registered channel by message.channel_id
   ├── Fail if channel not registered
   └── Write sender_id in the channel's form (E.164 on a channel
       addressed by phone number, §5.7) before anything reads it

2. ROUTE TO ROOM
   ├── If room_id provided → use it
   ├── Otherwise → call InboundRoomRouter.route(channel_id, channel_type, sender_id, metadata)
   │   ├── Router returns existing room → use it
   │   └── Router returns null → create new room
   ├── For a room id in hand (provided or routed):
   │   ├── A room of another organization than the one the caller acts for
   │   │   → fail, nothing written (§17.2)
   │   ├── No room with that id → create a new room with that id
   │   ├── Room exists, channel not bound to it → attach the channel, unless
   │   │   it was detached from that room (§7.5 rule 7)
   │   └── Routed (not provided), and the channel's binding names no
   │       participant and is not `group` → record the sender on it
   │       (`participant_id`, §10.4) and, on a transport, address the
   │       binding's replies where the message came from (the sender's
   │       address, or its chat) where it has no recipient (§5.7), under
   │       the room lock and before step 3;
   │       when the binding names another sender by then (a concurrent
   │       message was recorded first), route the message once more
   └── If new room created → fire ON_ROOM_CREATED hook and emit
       `room_created`, then attach channel, its binding naming the sender
       (`participant_id = sender_id`, §5.7) and, on a transport, addressing
       replies where the message came from (the sender's address, or the
       chat it was posted in, §5.7); then, for a
       channel other than
       VOICE and REALTIME_VOICE, fire ON_SESSION_STARTED (§9.2) and emit
       `session_started` (§8.2)

3. BUILD CONTEXT
   ├── Fetch room state
   ├── Fetch all bindings
   ├── Fetch all participants
   └── Fetch recent events (for AI context)

4. CHANNEL PROCESSES INBOUND
   └── channel.handle_inbound(message, context) → RoomEvent

5. IDENTITY RESOLUTION (if resolver configured and the sender has an address
   left to resolve — §11.6 skips a channel that names its own senders, and a
   sender the room has already identified)
   ├── Call resolver.resolve(message, context) with timeout
   ├── Handle result:
   │   ├── IDENTIFIED → create/update identified participant
   │   ├── AMBIGUOUS → fire ON_IDENTITY_AMBIGUOUS hook
   │   ├── UNKNOWN → fire ON_IDENTITY_UNKNOWN hook
   │   └── CHALLENGE_SENT → deliver challenge, block processing
   └── Stamp participant_id on event

5b. VERIFICATION (§11.7; when verification is configured and the sender is
    IDENTIFIED)
   ├── A reply that is the code of the identity's open code request in this
   │   room (§11.7.4) → attempt it; the message is consumed: its text is
   │   never stored nor broadcast, and no agent reads it
   ├── Stamp source.verification
   └── Not VERIFIED → fire ON_VERIFICATION_REQUIRED (SYNC), which holds the
       event (the default), passes it or blocks it (§11.7.3)

5a. OFF-LOCK CHECK (only when a `needs_lock = false` hook matches, §9.5.1)
    ├── Take the room's admission ticket
    ├── Run the off-lock BEFORE_BROADCAST SYNC hooks, off the lock
    └── Wait for the earlier tickets of the room; release ours after step 12

6. ACQUIRE ROOM LOCK, THEN CHECK THE ROOM STILL ACCEPTS EVENTS
   ├── Re-read the room's status under the lock (§5.1)
   ├── If the status refuses new events (CLOSED, ARCHIVED):
   │   └── Return InboundResult(blocked=true, reason=room_closed)
   │       # Nothing is stored: a refused event is not appended to the
   │       # timeline, not even as BLOCKED (§5.1)
   └── Otherwise → continue
       # Under the lock because close_room() takes the same one. Checked
       # earlier, the answer can be stale by the time the event commits.

7. IDEMPOTENCY CHECK
   ├── If idempotency_key exists and was seen → return the result the first
   │   delivery produced, without reprocessing (§13.4). A store that cannot
   │   resolve the key back to its committed event MAY return a blocked
   │   result instead; neither answer reprocesses.
   └── Otherwise → continue

8. ASSIGN EVENT INDEX
   └── event.index = room.event_count

8a. VALIDATE EDIT/DELETE (EDIT and DELETE events only — §10.3 Validation)
    ├── If validation fails → Return InboundResult(blocked=true, reason=
    │   target_event_not_found | identity_required_for_edit |
    │   not_authorized | not_original_author)
    │   # The sender and the target's author must both be identified.
    │   # Nothing is stored; the target's state update waits for step 12,
    │   # once the hooks have allowed the event.
    └── Otherwise → continue

9. RUN BEFORE_BROADCAST SYNC HOOKS
   ├── Apply the step 5a outcome first, when there was one (§9.5.1)
   ├── Execute the remaining hooks in priority order
   └── Collect result: allow / block / modify

10. IF BLOCKED BY HOOK:
    ├── Commit event with status=BLOCKED, blocked_by=hook_name, in the
    │   same atomic transaction as step 12: the event takes its index,
    │   event_count += 1, latest_index = event.index (§14.1, §14.3)
    │   # An audit record: it reaches no channel, at delivery or as
    │   # reconstructed context one turn later (§7.5 rule 8), and a
    │   # default timeline read skips it (§14.1)
    ├── Deliver injected events from hook result
    ├── Persist tasks and observations from hook result
    └── Return InboundResult(blocked=true)

11. CHECK SOURCE CAN WRITE
    ├── If source_binding.access ∉ {READ_WRITE, WRITE_ONLY} OR source_binding.muted:
    │   ├── Commit event with status=BLOCKED,
    │   │   blocked_by = source_read_only | source_muted, as at step 10
    │   ├── Deliver injected events from hook result
    │   ├── Persist tasks and observations (side effects ALWAYS collected, §7.5)
    │   └── Return InboundResult(blocked=true)
    └── Otherwise → continue
        # A source that cannot write MUST NOT inject a DELIVERED event.

12. COMMIT EVENT — the commit point (§13.6, §14.3):
    ├── For EDIT/DELETE: apply the target state update now (§10.3) — deferred
    │   to this point so a hook that blocks the edit/delete leaves the target
    │   unmutated
    ├── Record the author (§6.4): when the event's author has a name, the
    │   │   room's register of the sources and names it has seen fixes the
    │   │   rank of this source among those whose names read like its own,
    │   │   and the event carries its name, its rank and its source
    │   │   (metadata.author, any value the event came with dropped); a
    │   │   reader reads that record while the event's source is the one
    │   │   recorded. The room's named participants take their seats first,
    │   │   in the order they joined, each the lowest rank no source whose
    │   │   name reads like theirs holds; a source new to the names it uses
    │   │   takes one more than the highest rank those sources hold; a source
    │   │   keeps its rank while no source whose name reads like the one it
    │   │   uses holds it, and takes a new one otherwise. A source new to the room joins
    │   │   the register whatever its event's visibility: a restricted turn
    │   │   ranked without joining it would leave its rank to the next
    │   │   source, and a reader who sees both would read two sources under
    │   │   one label. The rank is the room's, so a reader may learn from it
    │   │   that a source it does not see has a name that reads alike, never
    │   │   the source nor its turn (§7.5 rule 8). A room with no register (a
    │   │   room from before it) has it rebuilt once from its timeline: the
    │   │   ranks its turns' records hold, then its named participants'
    │   │   seats, then the authors of its other received turns in index
    │   │   order; a turn with no record of its author reads its rank from
    │   │   the register. The register is read and written under the room lock,
    │   │   so a store shared across processes needs the distributed lock of
    │   │   §13.5. Before the commit, so the rank of a turn never changes once
    │   │   stored: a commit that fails after it leaves a rank unused, never
    │   │   one held twice. A BLOCKED record (steps 10, 11) takes none
    ├── Commit atomically, as ONE logical transaction:
    │   ├── Store event with status=DELIVERED
    │   └── Update room state: latest_index = event.index; event_count += 1;
    │       timers.last_activity_at = now
    │   # After this transaction returns, the event is durably part of the
    │   # timeline. It MUST NOT be retroactively re-marked BLOCKED or FAILED
    │   # (e.g. by process_timeout, §13.6) — the timeline is authoritative.
    │   # No separate, non-atomic room-state write ever follows this commit:
    │   # the timeline and the counters can never diverge (§14.3).
    ├── Deliver any injected events
    ├── IF message.session is set:
    │   └── Call channel.connect_session(session, room_id, binding)
    │       # Connects the voice session (starts realtime AI, etc.)
    └── PLAN BROADCAST — resolve the event's delivery set under the lock:
        └── Eligible target bindings (access, mute, visibility — §7) and
            their capability snapshot, appended to the room's delivery
            lane (§10.2). Planning reads bindings under the lock, so the
            target set is consistent with the committed timeline;
            EXECUTION does not require the lock.

13. RELEASE ROOM LOCK
    # The lock covers steps 6–12: room-status gate, idempotency, index
    # assignment, hooks, permission, commit, broadcast planning. External
    # delivery executes outside it (§10.2 delivery lanes) — a slow
    # provider MUST NOT extend the room's critical section (§13.5).

14. EXECUTE DELIVERY LANE (§10.2) — per-room FIFO, after lock release:
    ├── An event's delivery set MUST complete before the next event's
    │   begins — per room, in index order. Executing delivery under the
    │   room lock (the pre-lane behavior) trivially satisfies this
    │   ordering and remains conformant.
    ├── Each target: transcode → rate limit → on_event() / deliver()
    │   # The framework sets no timeout of its own: each external call is
    │   # bounded by its channel's or its provider's own timeout.
    └── REENTRY: response events emitted here (intelligence channels)
        re-enter the pipeline's locked section (steps 6–12) as new
        passes — each response takes the room lock for ITS OWN commit
        (index assignment, chain depth §8.3, the same atomic shape as
        step 12), and its delivery set joins the same lane in FIFO order.
        Every pass re-enters its responses: a reentry pass, a streamed
        segment, a greeting and a regenerated answer are an agent's
        output like the trigger's response (§19.3.1). A streamed
        response started by any pass is read by the caller that owns
        the chain, after the trigger's own, and counts against the
        same reentry budget as a buffered response. Every response that
        re-enters counts against that budget, a muted or read-only
        source's included, whose pass stores it BLOCKED (§7.5 rule 2).
        # Relaxation vs pre-lane implementations: a concurrent inbound
        # event MAY commit between a trigger and its response. Ordering
        # guarantees are per-room index monotonicity and parent linkage —
        # never timeline adjacency.

15. PERSIST SIDE EFFECTS
    └── Store all tasks and observations from hooks + channels

16. RUN AFTER_BROADCAST ASYNC HOOKS
    └── After the event's delivery set completes — "all channels have
        processed the event" (§9.2) is unchanged; only the timing may be
        later than under pre-lane implementations. Non-blocking,
        exceptions swallowed (§9.3) — a slow observer hook MUST NOT hold
        the room lock.

17. EMIT FRAMEWORK EVENTS
    └── event_processed, delivery_succeeded/failed, etc.

18. RETURN InboundResult
    └── The caller observes its event's delivery-set completion, so
        delivery_results reports executed deliveries. An implementation
        MAY additionally offer detached completion; the outbox model
        (§13.6) then governs crash recovery. A detached-completion call
        returns at the commit — blocking decisions (steps 9–11) stay
        synchronous, the committed event is on the result — and the
        result SHOULD then carry a completion handle covering the whole
        deferred tail: the delivery set, the reentry passes it spawns
        and any streamed responses. Waiting on the handle backfills
        delivery_results, after which the result reports what step 18
        reports on the waiting path. A call refused before the locked
        region reports the refusal alone — there is no deferred tail to
        hand back. Waiting on the handle from a context the delivery
        lane cannot progress past (§10.2) MUST NOT deadlock: it returns
        unwaited, the same short-circuit the waiting path applies.

Explicit delivery cancellation: An implementation MAY expose cancellation of a call's delivery cascade. Cancellation MUST NOT change the committed timeline event's status (§13.6). It MUST stop subsequent generation/tool rounds and reentry belonging to the cancelled cascade, join its running generation, tool handlers and stream consumers (including their finalizers), and report a terminal cancellation outcome before claiming cleanup has completed. Other cascades and shared providers MUST remain usable. The next call MAY resume after that cleanup. Cancellation cannot revoke external effects already committed by a handler, nor implicitly own tasks detached by application code.

The awaited path SHOULD propagate caller cancellation after draining its own cascade. A detached-completion handle SHOULD provide explicit cancellation; cancelling a mere completion waiter SHOULD leave detached work running. Repeated cancellation MUST NOT interrupt cleanup already in progress. A bounded cleanup that expires MUST report incomplete drainage, not success; resources still in use MUST be retained. An implementation MUST refuse a drain from a context that its own delivery needs to progress, such as its room lock or delivery lane. These guarantees cover execution owned by the local process; cross-worker cancellation requires an explicit distributed contract.

InboundResult:

InboundResult
├── event: RoomEvent | null                 # The processed event (null if blocked)
├── blocked: bool                           # Whether the event was blocked
├── reason: string | null                   # Block reason
├── error: Exception | null                 # Generation/stream failure, when observed
├── response_metadata: ResponseMetadata     # Final records of root response turns
├── delivery: CompletionHandle | null       # Detached completion only (step 18)
└── delivery_results: map<string, DeliveryResult>

response_metadata MUST merge the response-metadata records produced by the intelligence channels in the caller's root delivery set. It is the final state of each record: the non-streaming path collects it after generation, and the streaming path after consuming the response stream. A deferred call MAY return with the record still empty, but delivery.wait() MUST backfill it together with delivery_results and error, so the waited result reads like the ordinary waiting path. A call that ran no response turn returns an empty record. When several channels contribute metadata they SHOULD namespace their top-level keys; the merge order for colliding keys is unspecified.

10.1.1 Instructions

An application sometimes needs an agent to speak because of something nobody in the room said: a handoff asking the new agent to introduce itself, a scheduled nudge. Committed as an inbound MESSAGE, that direction is stored as a participant's words, shown in the transcript as theirs, and read by the model as a user turn. An INSTRUCTION event carries it through the same pipeline — the same hooks, lane and addressing — without either. It is the system intent of §12.4, for every kind of intelligence channel.

process_inbound() with event_type = INSTRUCTION (normative):

1. It is addressed. addressed_to MUST name at least one intelligence channel (§19.3). An unaddressed or empty-addressed instruction is refused, before anything is written, with InboundResult(blocked=true, reason=instruction_unaddressed): a direction meant for one agent does not make every agent in the room speak.
2. It carries no idempotency key. Step 7 finds a duplicate through the stored event, and an instruction is never stored. One that carries a key is refused, before anything is written, with reason=instruction_not_idempotent.

Direct injection (§10.5) takes the same event, and an API whose contract is the committed event cannot report either refusal by returning one: it MUST raise instead, before anything is written. 3. Intelligence channels only see it. The pipeline sets its visibility to "intelligence" whatever the caller asked: no transport delivers it, and a voice channel never speaks it. 4. Steps 1–11 apply unchanged. Identity, the room-status gate and the source write check hold, and BEFORE_BROADCAST hooks run on it and MAY block or modify it. The caller's metadata rides the instruction, whatever the transport's parser keeps: the pipeline applies it to the event (a key the parser set keeps its value, and standalone is the typed field's word, step 7), so a hook reads what the caller attached to it, a delegation's hand-back its task (§23.3 step 8). A hook — and an AFTER_BROADCAST one — MUST NOT treat it as something a participant said: its author is the application. A blocked instruction is not stored (the BLOCKED commit of §10.1 steps 10–11 does not apply) and the result reports the block; event_blocked is still emitted, and the injected events, tasks and observations of its hooks are still processed. 5. It is not committed. At step 12 nothing is stored, no index is consumed (the one step 8 assigned is provisional, for the hooks), and no room counter moves (§14.3 holds trivially). Its delivery set joins the room's delivery lane without an index, behind the room's latest committed event, so it keeps its place in the room's order (§10.2). InboundResult.event is the uncommitted instruction. 6. An intelligence channel takes it as a directive for one turn. The text is the turn's input after the rebuilt history (unless it stands alone, step 7), marked as the application's instruction and never attributed to a speaker, so the model cannot read it as something a participant said; it is absent from the history rebuilt for any later turn, and a memory provider does not ingest it. (A system-role message after the history is not portable: several model APIs refuse one, others silently turn it into a user turn, and a request that ends on the assistant's last reply reads as a continuation of it.) The response is an ordinary response event: committed, broadcast, subject to reentry and chain depth. It MUST record the instruction that produced it in its metadata as a fingerprint, instruction = {sha256, length} (the hex SHA-256 of the UTF-8 text and its length in Unicode code points), and MUST NOT copy the text: the metadata rides on every response event and every segment of it, so a copy would store an instruction the room never stores — a summary's transcript, a long prompt — once per reply, and deliver it to every transport. The timeline still records that an instruction produced the reply and which one; an application that shows its text keeps it. 7. It MAY stand alone. standalone = true asks that the turn it opens read nothing of the room: its input is the instruction alone, with no rebuilt history, and the channel MUST NOT call the memory provider for it — an empty view is not enough, since a provider MAY return messages of its own (a summary, a minimum it always keeps). Nor does it read the room's working memories, which are its past in another form: skills activated in the room, the room's plan, the room's background tasks, and what the channel recalls of tools used there (a digest that may quote their results). The channel's own system prompt, tools and skill catalogue are unchanged. It is for a pass that must start from a blank page, such as a summary re-run that would otherwise read, and copy, its previous answer.

A channel whose conversation lives in a session it does not rebuild (an ACP agent, §19.3.2) cannot empty that session, so it MUST run a standalone turn in a session opened for that turn and retired after it: no catch-up is sent, the room's session is neither prompted nor told, and its catch-up cursor does not move. The reply is the room's like any other, and the room's session did not produce it: its next catch-up MUST carry it, as the channel's own words from a separate turn, or the agent would answer later questions about a message it appears to have posted from a history that lacks it (§19.3.2). The channel MUST declare the turn session as the turn's, not the room's, to whatever carries it (for ACP, the session/new metadata): a transport that files sessions by room would otherwise hand the turn the room's session and close that one after it. Retired means the channel MUST NOT prompt the turn session again after the turn, and closes it where the agent announces it can: closing is optional in ACP (session/close is announced in the agent's capabilities), and an agent that does not announce it keeps the session until its connection ends. The turn session MUST take the Room's session setup as the Room's next session would (the ACP agent channel's Session setup, a refused value failing the turn), then the room session's current configuration (model, mode) where the agent accepts it; host-contributed context is the channel's own and still applies. Such a channel also keeps an ordinary instruction in its session once prompted with it — nothing is rebuilt there — which is why the step 6 marker matters on it as much as the history rule does elsewhere.

standalone is a property of an instruction only: set on any other event type it MUST be refused before anything is written (raised, since the event would otherwise land as a participant's line without the requested isolation). 8. A realtime session is not reached this way. A channel hosting a realtime model (a realtime voice or audio-video channel, a conference with a realtime model plugged in) is a transport and never sees an intelligence-only event; its instruction is inject_text(role="system") on the model's session (§12.4).

10.2 Broadcast Pipeline

The broadcast pipeline routes an event to all eligible channels in a room.

broadcast(event, source_binding, context) → BroadcastResult

Planning vs execution — delivery lanes. Broadcast splits at the room lock boundary (§10.1 steps 12–14). Planning — resolving the delivery set (steps 1–2 below) — MUST run under the room lock, against binding state consistent with the committed timeline. Execution — the per-target work of step 3 — is NOT required to hold the room lock; it MUST instead preserve per-room order: events execute their delivery sets in index order, one event's set completing before the next event's begins (a delivery lane per room). Executing under the room lock trivially satisfies this ordering and remains conformant — the lane model exists so that a slow provider or a long AI generation does not extend the room's critical section (§13.5), which is what serializes multi-process deployments. Response events re-enter the pipeline as their own commit passes (§10.1 step 14) rather than being drained inside the trigger's lock tenure.

Step-by-step:

1. CHECK SOURCE CAN WRITE
   ├── source_binding.access must be READ_WRITE or WRITE_ONLY
   └── source_binding.muted must be false
       (if muted: suppress events but collect side effects)

2. DETERMINE TARGET CHANNELS
   ├── Get all bindings in room
   ├── Exclude source channel
   ├── Exclude channels with access = WRITE_ONLY or NONE
   ├── Exclude OUTBOUND-only channels
   └── Apply visibility filter from source_binding.visibility

3. FOR EACH TARGET CHANNEL (concurrently):
   │
   ├── a. TRANSCODE CONTENT
   │      └── Convert event content to target's supported media types
   │
   ├── b. ENFORCE MAX LENGTH
   │      └── Truncate text if target.capabilities.max_length exceeded
   │
   ├── c. CALL on_event() (all readable channels)
   │      ├── Intelligence channels: only those solicited (§19.3), and
   │      │   none when the response would reach max_chain_depth (§8.3)
   │      └── Collect ChannelOutput (response events, tasks, observations)
   │
   ├── d. CALL deliver() (transport channels only)
   │      ├── Apply rate limiter
   │      ├── Check circuit breaker
   │      ├── No recipient in the binding → refuse before any send:
   │      │   delivery_failed (error `no_recipient`), not retried, not
   │      │   counted by the circuit breaker (§13.1). A channel whose
   │      │   provider delivers to a destination of its own (a webhook's
   │      │   configured URL) needs no recipient and is not refused
   │      ├── Call provider
   │      ├── On failure → apply retry policy
   │      └── Record delivery result
   │
   └── e. COLLECT RESULTS
          ├── Response events → queue for reentry
          ├── Tasks and observations → accumulate
          └── Errors → record per-channel

4. HANDLE MUTED CHANNELS
   ├── Muted channels STILL receive on_event()
   ├── Response events from muted channels re-enter like any other
   │   (§10.1 step 14); their commit pass stores them BLOCKED source_muted
   │   after their hooks (§7.5 rule 2), and none is broadcast
   └── Tasks and observations from muted channels are COLLECTED

5. RETURN BroadcastResult

BroadcastResult:

BroadcastResult
├── outputs: map<string, ChannelOutput>         # on_event() results per channel
├── delivery_outputs: map<string, ChannelOutput> # deliver() results per channel
├── reentry_events: list<RoomEvent>             # Response events for reentry loop
├── tasks: list<Task>                           # Accumulated tasks
├── observations: list<Observation>             # Accumulated observations
├── metadata_updates: map<string, any>          # Room metadata to update
├── blocked_events: list<RoomEvent>             # Chain-depth-blocked events
└── errors: map<string, string>                 # Per-channel error messages

Solicitation. Step 3c asks a channel to act. For transport channels that is unconditional; for intelligence channels the set is narrowed by the event's address and by the room's agent-response policy (§19.3). Narrowing solicitation MUST NOT narrow visibility: the two filters are independent, and an event hidden from a channel by §7.3 is hidden whether or not it was addressed to it. An unsolicited intelligence channel is not called at step 3c at all — not asked, and not told — and rebuilds what it missed from the timeline when it is next solicited (§19.3.2). In a room a discussion holds (Section 19.7.5), no agent is solicited at step 3c: the discussion reads the event's names from the text it commits with, once its BEFORE_BROADCAST hooks have run, queues the agents the event asks for, and gives each its turn later, a rerun planned for that agent alone.

10.3 Edit and Delete Processing

When an inbound event has type EDIT or DELETE, the framework MUST perform additional validation and state updates before broadcasting.

Validation:

1. The framework MUST verify that target_event_id (from EditContent or DeleteContent) references an existing event in the same room.
2. For DeleteContent with delete_type = SENDER or EditContent with edit_source = "sender", the framework MUST verify that the inbound sender is the original author of the target event.
3. For DeleteContent with delete_type = ADMIN, the framework MUST verify that the sender has administrative authority (e.g., via permissions or role).
4. For DeleteContent with delete_type = SYSTEM, the event SHOULD originate from a SYSTEM channel or a hook.
5. If validation fails, the framework MUST reject the event and SHOULD return an error to the source channel.

State updates (applied only after BEFORE_BROADCAST hooks allow the event — a hook that blocks the EDIT/DELETE MUST leave the target event unmutated):

1. On successful EDIT: the framework SHOULD call the store's update_event() (§14.1) to replace the original event's content with EditContent.new_content and set metadata.edited = true. The EDIT event itself MUST be stored in the timeline.
2. On successful DELETE: the framework SHOULD call the store's update_event() (§14.1) to set metadata.deleted = true on the original event. The DELETE event itself MUST be stored in the timeline.

Mutation triggers: After a successful EDIT the framework MUST fire ON_EVENT_UPDATED with the mutated target event; after a successful DELETE it MUST fire ON_EVENT_DELETED. Firings follow the async-hook locking rule of §10.1 (deferred until the room lock is released). A blocked EDIT/DELETE MUST NOT fire either trigger. These triggers let observers (e.g. maintainers of denormalized projections over the timeline) see every stored-state change regardless of origin.

Direct mutation (host-owned): The framework SHOULD also expose direct APIs — update_event(room_id, event_id, content|source|metadata) and delete_event(room_id, event_id, cascade_replies) — for host applications that own authorization on their own surfaces. Direct update_event() replaces the provided fields wholesale and adds no edited marker; direct delete_event() is a HARD delete that MUST cascade to thread replies by default (no self-referential FK protects them) and MUST return the deleted event IDs. Both MUST serialise against inbound processing via the room lock and MUST fire the corresponding mutation trigger after the lock is released; a target that does not exist in the room MUST be a no-op that fires nothing.

Broadcast behavior:

During broadcast, for each target channel:

1. If the target channel's capabilities.supports_edit is true (for EDIT events) or capabilities.supports_delete is true (for DELETE events), the framework SHOULD deliver the event natively.
2. If the target channel does NOT support the operation, the framework MUST apply transcoding fallback (see Section 6.6).

10.4 Inbound Room Routing

When an inbound message arrives without a pre-determined room_id, the framework MUST route it to an appropriate room.

InboundRoomRouter (interface)
└── route(channel_id, channel_type, sender_id, metadata) → Room | null

Default routing strategy:

1. Find the latest ACTIVE room whose binding of this channel names the sender (§5.7); otherwise, the latest ACTIVE room where a participant with the same sender address is connected via the same channel type. The sender is named by their address, then by the identity the store resolves it to (resolve_identity, §14.1), so a member the integrator added under their identity is found by an address linked to it. An identity reaches every channel its person uses: a room found through it as a participant is the sender's only when the identity joined that room through the same channel, so a message is not carried into a room of another channel, whose messages would then go out on this one. On a chat channel the address routed by is the chat the message was posted in, not its sender (§5.7), here and in step 3.
2. If found → return that room.
3. Otherwise, if the channel is bound to exactly one ACTIVE room and that room admits the sender (below) → return that room.
4. If not found → return null (framework creates a new room).

Step 3 admits one conversation. Step 3 is what routes a channel dedicated to a single conversation. It is deliberately narrower than "the channel is bound to some room": on a number shared by many correspondents, the one ACTIVE room bound to it is someone's conversation, and a second correspondent routed into it would be stored there, answered with that conversation as context, and answered at that conversation's address. The room admits the sender only when its binding of the channel is declared group (§5.7), or when all of the following hold:

• the binding names no participant, or names the sender;
• on a channel whose replies go to the sender's address (§5.7), the binding has no recipient, or its recipient is the sender's address: a room that delivers to someone else is their conversation;
• no participant of the room other than the sender joined it through this channel: a member the integrator added on the channel says whose conversation the room is before they have written;
• the room has received nothing on this channel from another sender: no stored event of the channel, a BLOCKED one excepted, whose source names a participant other than the sender.

The sender is known to the room by their address, and by the identity the store resolves that address to (resolve_identity, §14.1): a participant or an event named by either is the sender's own. A message the framework writes itself, under the sender system (a deliver() to the room, an orchestration cue), names no correspondent: it is never recorded on a binding, and a message routed under that sender is never admitted by step 3 into a room whose binding is not group. The framework names the room it writes to, and a sender borrowing the name, on a channel whose senders choose their id, gets a room of its own. A member added under an id that is neither the address nor a resolvable identity cannot be told from a stranger, so the room is closed to step 3: its correspondent's messages are routed by room_id, or the address is linked to the identity (link_address, §14.1), unless the binding's recipient names them (§5.7), in which case step 1 finds them through it.

A message with no sender address is held to the same conditions, so it is admitted only where no one is named, joined or heard on the channel.

When step 3 admits a sender through a binding that names no participant and is not group, the framework records the sender on it (§10.1 step 2): step 1 then finds them on their next message, and step 3 admits no one else. A room the framework creates for an inbound message is bound to its sender from the start. The sender is recorded when the message is routed, under the room lock, before anything of the message is processed: two first messages arriving together cannot both be admitted, since the second finds the binding naming the first and is routed again. The wait for that lock is bounded by process_timeout (§13.6); past it the message is refused as a process timeout, never let in unrecorded. A message a hook later refuses has been routed all the same: routing decides which conversation a message belongs to, a hook what becomes of it.

A delivery status follows its message. A provider's delivery status that names no room is routed by the same rule: the room whose binding of the status's channel names its recipient, else the one ACTIVE room bound to the channel unless its binding names another correspondent, else none. It is never dispatched with the context of one room among several.

A binding speaks for its own channel. The sender a binding names is found through it on that channel only. A correspondent recorded on one number's binding, writing to another number of the same type, is not taken to the first number's room: on a kit serving a bank and a clinic, each on its own SMS number, a customer the bank's number recorded writing to the clinic lands in the clinic's conversation. A participant record is another matter: a member the integrator added to the bank's room is found by step 1 whatever number they write to. Merging or separating a person's conversations across channels is decided for participants, not by bindings.

Example, on one SMS number with no room yet: alice writes first, and the room created for her has a binding naming her. Bob's first message finds no room of his at step 1; at step 3 the binding names alice, so null, and bob gets a room of his own. Alice's next message is found at step 1 through her binding, and lands in hers. Had the integrator opened alice's room itself, addressed to her number or with her added as a member on the SMS channel, bob would be refused the same way before alice ever wrote, and alice found at her first message; had it opened the room naming no one and no recipient, the first sender would be admitted and recorded, and the second refused.

A conversation of several senders on one channel, such as a group chat bound to its room, is declared group on its binding. Without the declaration, the default router routes its second speaker to a room of their own, which is the safe answer: nothing in the store tells a group channel from a shared number.

A router MUST NOT guess. When the inputs it was given match more than one ACTIVE room and nothing in them tells which conversation the message belongs to — for the default strategy, a channel bound to several ACTIVE rooms (step 3) — it MUST return null rather than choose among them. Step 1 is not a guess: it returns the most recently created of the ACTIVE rooms where the sender is connected, a conversation the sender is in. Returning any one of several candidates delivers a message into a conversation it does not belong to, and — because the room it lands in is where the message is stored, broadcast to that room's channels, and read back as context by that room's agent — the disclosure is durable, not transient. Null is the safe answer: the framework creates a new room, which is recoverable, whereas a wrong room is not.

This is not a hypothetical configuration. An implementation that lets a channel be shared across rooms (§19, delegation) creates it as a matter of course, and a channel re-attached after its room closed accumulates the same shape over time. An integrator whose deployment genuinely needs a shared channel routed by something else MUST name the room explicitly (§10.1 step 2) or supply a custom strategy.

Routing MUST be deterministic. The same inputs against the same stored state MUST select the same room, whatever the storage backend. An implementation whose answer depends on insertion order in one store and on the query planner in another does not satisfy this, and its behaviour cannot be reasoned about from the specification.

Implementations MUST allow integrators to provide a custom routing strategy. The framework records the sender on the binding of the room any strategy returns (§10.1 step 2): a custom strategy that sends several senders to one room declares that room's binding group.

10.5 Direct Event Injection

Integrators MAY inject events into a room directly, without an inbound channel message (e.g., from a REST API or MCP tool call):

send_event(room_id, channel_id, content, event_type, ...) → RoomEvent

The injected event traverses the SAME pipeline as an inbound message (Section 10.1): index assignment, BEFORE_BROADCAST hooks, the source write-permission gate, EDIT/DELETE handling, persistence and broadcast planning under the room lock, then its delivery lane, reentry passes, and AFTER_BROADCAST hooks. A blocking hook therefore yields a BLOCKED event and suppresses delivery, exactly as for an inbound message. That includes the off-lock check (§9.5.1, step 5a): an injected event matched by a needs_lock = false hook takes an admission ticket like an inbound one — unless it is injected from inside an off-lock hook's body or under the room lock, in which case it takes none and commits ahead of the event being processed (§9.5.1, Reentrance).

Its result is the committed event, so a refusal before the commit point is raised, never returned as an event that was not written: a room whose status refuses new events (§5.1) raises a room-closed error, and process_timeout (§13.6), which bounds its wait for the off-lock check and the room lock as it bounds its pre-commit phase, raises a process-timeout error once it has emitted the process_timeout framework event. Nothing of the event is written.

A record outside the pipeline. An integrator MAY commit an event that no member is meant to receive (a trace, a display snapshot, a copy a branched conversation starts from) without the pipeline:

commit_event(room_id, event, organization_id) → RoomEvent

The record names the room it is committed to: one whose room_id is another room's MUST be refused before anything is read or written, so the room checked is the room written. The room is read scoped to organization_id (§17.2): a missing room, or another organization's, is not found. A room whose status refuses new events (§5.1) MUST refuse the record before anything is written. The record is committed under the room lock with the next index, which the room's delivery lane MUST count as delivered at once: no delivery set, no hook, and no broadcast follow it, and the next event of the room MUST NOT wait on its index. It is stored as the caller gives it, whatever the persistence policy, and rule 2 of §7.5 does not apply to it. An event the members must receive is injected with send_event instead.

---

11. Identity Resolution

11.1 Identity Resolver Interface

IdentityResolver (interface)
└── resolve(message: InboundMessage, context: RoomContext) → IdentityResult

IdentityResult:

IdentityResult
├── status: IdentificationStatus            # IDENTIFIED, AMBIGUOUS, UNKNOWN, PENDING, CHALLENGE_SENT, REJECTED
├── identity: Identity | null               # Resolved identity (if IDENTIFIED)
├── candidates: list<Identity>              # Candidate identities (if AMBIGUOUS)
└── metadata: map<string, any>              # Resolution metadata

11.2 Resolution Pipeline

Inbound arrives (channel_type, sender_address)
      │
      ▼
IdentityResolver.resolve(message, context)
      │
      ▼
Returns IdentityResult
      │
      ├── IDENTIFIED (1 match)
      │   └── Create participant with identity_id set
      │
      ├── AMBIGUOUS (N matches)
      │   ├── Fire ON_IDENTITY_AMBIGUOUS hook
      │   └── Hook returns:
      │       ├── resolved(identity) → use that identity
      │       ├── pending() or null → create pending participant
      │       ├── challenge(inject) → send challenge, block processing
      │       └── reject() → reject the message
      │
      └── UNKNOWN (0 matches)
          ├── Fire ON_IDENTITY_UNKNOWN hook
          └── Hook returns:
              ├── resolved(identity) → use that identity
              ├── pending(candidates) → create pending participant
              ├── challenge(inject) → send challenge, block processing
              ├── reject() → reject the message
              └── null → process the message, its sender unresolved

Identity hooks receive the event, the room context and the IdentityResult, and run in registration order. The first one that returns a result decides; one that raises is logged and the next one runs. A result whose status the trigger does not act on is read as null. A hook registered on these two triggers through HookRegistration (Section 9.1) runs after them as an observer and cannot decide. The pending participant carries the resolver's candidates on ON_IDENTITY_AMBIGUOUS, and on ON_IDENTITY_UNKNOWN the hook's, else the resolver's.

11.3 Identity Hook Result

IdentityHookResult
├── status: IdentificationStatus            # The outcome (below)
├── identity: Identity | null               # For IDENTIFIED: the sender
├── display_name: string | null             # Set by pending(); not read
├── candidates: list<Identity> | null       # For PENDING, on UNKNOWN only
├── inject: InjectedEvent | null            # For CHALLENGE_SENT: the challenge
├── reason: string | null                   # For REJECTED: the block's reason
├── challenge_type: string | null           # Not read by the framework
└── message: string | null                  # Set by challenge(); not read

resolved(identity), pending(display_name, candidates), challenge(inject, message) and reject(reason) set status to IDENTIFIED, PENDING, CHALLENGE_SENT and REJECTED; reject defaults its reason to "Unknown sender". A challenge blocks the message with the reason identity_challenge_sent; a REJECTED result with no reason blocks it with identity_rejected on ON_IDENTITY_AMBIGUOUS and unknown_sender on ON_IDENTITY_UNKNOWN.

11.4 Channel Type Filtering

Implementations SHOULD support configuring which channel types trigger identity resolution. Not all channels carry meaningful identity information (e.g., an internal WebSocket may not need identity resolution).

11.5 Timeout Handling

If identity resolution exceeds the configured timeout, the implementation MUST:

1. Treat the result as UNKNOWN.
2. Emit an identity_timeout framework event.
3. Continue processing the inbound message (do not block).

11.6 Senders With Nothing To Resolve

A resolver maps an address — a number, an email, a handle — to an Identity. Two senders carry no such question, and an implementation MUST NOT run resolution for them:

1. The channel names its own senders. A channel MAY declare that its inbound sender_id is a room Participant.id rather than an address. A conference is the case this exists for (Section 12.10.4): its utterances carry the identity a track was published under, and resolving that is the "resolve on the opaque identity" Section 12.10.2 rule 3 rules out.
2. The room has already identified the sender. Where the event's participant_id names a Participant of the room whose identification is IDENTIFIED, the question has been answered and the answer is on the roster.

Case 1 is not the conference alone. The test is where the sender_id comes from. A channel that reads it off the wire carries whatever the remote network put there, and is not this case. A channel the implementation itself names the sender of — an interactive terminal, whose sender is a Participant the implementation chose rather than a party that reached it from outside — is the same case as the conference, and declares it for the same reason. A channel whose sender_id is supplied by the integrator is neither: it may carry an address or it may not, which is why Section 11.4 leaves that one to configuration rather than to the channel.

PENDING and AMBIGUOUS participants are deliberately not in case 2: a participant the room has is not a participant the room has identified, a resolver may still be what settles it, and the identity hooks of Section 11.2 may still want to challenge or refuse it.

This is not an optimisation. Re-resolving a sender the room has already identified produces UNKNOWN — no resolver matches a framework identifier — so ON_IDENTITY_UNKNOWN fires per message, and the refusal pattern Section 11.2 provides for (reject) then blocks every message from a participant the implementation itself identified. Skipping it is what keeps that pattern composable with channels that name their own participants.

When resolution is skipped, the event's participant_id MUST be left as the channel set it, and no participant record is created or modified: the sender is either already a participant or deliberately unattributed.

11.7 Identity Verification

Identification says whose an address is; verification says the person writing has proven it. A number that resolves to Alice says the phone is Alice's, not that Alice is typing. A service that answers account questions over a channel needs both. An implementation that supports verification carries the state, its lifecycle, a label on every message and the hooks; the integrator keeps the method (how a person proves who they are: a page behind a link, a code, a login) and the policy (what a message from an unverified sender may reach). Verification applies to an IDENTIFIED sender only; an identification challenge (§11.2) is another matter.

11.7.1 State

Verification
├── identity_id: string
├── organization_id: string | null         # Its scope (§11.7.7)
├── status: VerificationStatus             # §5
├── method: string | null                  # The integrator's name for it ("link", "code", "sso")
├── verified_via: string | null            # Channel the completing request was opened on
├── verified_at: datetime | null
├── expires_at: datetime | null
├── ended_reason: string | null            # "expired", "revoked", "failed" once ended
├── request_id: string | null              # The request open for the identity
└── completed_request_id: string | null    # The request that completed it: its end is
                                           # announced in that request's room

A verification belongs to the identity: once verified, the person is verified on every channel and in every room their addresses reach, and verified_via tells where they proved it, for an integrator that requires a channel of its own. An identity with no record is UNVERIFIED.

11.7.2 Requests

VerificationRequest
├── id: string                             # Public reference
├── identity_id, room_id, channel_id: string
├── organization_id: string | null         # The room's
├── method: string                         # "link", "code", or the integrator's
├── secret_hash: string                     # The secret is never stored in clear
├── window_start: datetime                 # Reception of the message that opened it
├── expires_at: datetime
├── attempts: int
└── status: OPEN | COMPLETED | FAILED | EXPIRED | CANCELLED

request_verification(identity_id, room_id, channel_id, method, window_start) opens a request and returns it with its secret, once: the integrator puts the secret in its link or sends it as a code, and the implementation keeps only its hash. Its lifetime and attempts come from the configured policy. A request takes its room's organization, the scope whose messages it verifies; a caller acting for another organization is refused as for the room itself (§17.2). An identity that is not verified becomes PENDING; one already verified stays VERIFIED meanwhile (asking again does not end a verification in force). Opening a request cancels the identity's earlier open one. complete_verification(request_id, secret, method) is called by the integrator once its own check passed (the page's PIN, the login); it compares the secret in constant time and refuses a request that is not OPEN or has expired. A mismatch counts as an attempt, and so does a failure of the integrator's own check, which it reports with record_failed_attempt(request_id) (a wrong PIN on its page): the bound is the request's, so a PIN cannot be guessed through the integrator's page. At the policy's bound the request FAILS, and a request that fails while the identity is VERIFIED ends that verification too: whoever could not prove it is not let on as the person (fail closed). A request that expires unanswered changes nothing. A completion makes the identity VERIFIED until now + ttl, emits verification_completed and fires ON_VERIFICATION_COMPLETED (ASYNC).

11.7.3 Labels and holding

The inbound pipeline stamps source.verification on every event of an identified sender (§10.1 step 5b), so a hook, a router, a user interface and a context builder read one fact. While the identity is not VERIFIED, every such event fires ON_VERIFICATION_REQUIRED (SYNC), whose hooks decide per message:

• hold (the default when no hook passes it): the event is stored and delivered marked held, its scope narrowed to the transport channels it reached — its own visibility, or its source binding's — so the room's people and transports (an advisor's console) see it and no intelligence channel reads it, in delivery or in context. Holding never widens what a message reached;
• pass: the event goes on unchanged, labelled — where an integrator lets a general question through. A hook passes a message explicitly; a hook that merely allows it (as one that only watches does) does not;
• block: as any blocked event (§9.4).

An edit or a deletion (§10.3) from an unverified sender is refused unless a hook passes it: held, it would still change a message the agents read.

Labelling and holding apply where the integrator configured verification; a configuration MAY label without holding. Opening a request and telling the sender (sending the link, the code) is the hook's: the method is the integrator's. The hook reads the sender's verification with sender_verification(event, context): whose it is, in the room's organization, and whether a request is open. It answers on the sender's first message too, when the sender is not a participant of the room yet. A hook that opens a request only when none is open leaves a sender who writes again meanwhile in silence; reminding them means opening a new request, since a secret is never kept to be sent again.

The mark of a held event is the runtime's: the inbound pipeline removes one that a sender's message carries, and only an event the pipeline held and labelled not VERIFIED is ever released (§11.7.6). A mark copied by any other way releases nothing.

Labelling and holding act on messages. In a speech-to-speech session (§12.4) the model hears the person before any message exists, and their transcript is recorded afterwards: nothing there is labelled or held, and holding a transcript would not keep the words from the model. An integrator that requires verification gates the session itself: it starts the session, or gives it the instructions and tools that reach an account, only once verification_of reads VERIFIED for the identity in the room's organization, and on ON_VERIFICATION_ENDED it ends or reconfigures the session. The implementation does not claim to hold anything there.

11.7.4 Codes in the conversation

With the method code, the secret is a short code the person replies with in the conversation. A message from the request's identity, in its room, whose whole text has the code's form is an attempt and is consumed (§10.1 step 5b): its text is never stored nor broadcast, no agent reads it, and the attempt is recorded by the framework events alone. Any other message is an ordinary one. A code is short, so its request's time and attempts MUST be bounded tightly.

11.7.5 The end of a verification

A verification ends when its time runs out (EXPIRED) or when the integrator revokes it (revoke_verification(identity_id, reason), REVOKED); a request ends when it fails, expires or is cancelled. Each end emits verification_ended and fires ON_VERIFICATION_ENDED (ASYNC) with the reason and what ended (ended: verification when the verification in force ended — expired, revoked, or a request that failed while it held — request when a request alone did, the identity's verification as it was), when it happens, not on the sender's next message: the implementation arms a timer at expires_at, and when it starts it ends the verifications whose time passed while nothing ran and re-arms the timers of the others, from the store (check_verifications()). A policy MAY open a new request at once; otherwise the sender's next message meets ON_VERIFICATION_REQUIRED.

11.7.6 Releasing held messages

When a request completes, the messages its sender wrote while it was open are released to the room's intelligence channels, unless the policy says otherwise: the held events of that room whose sender is the identity, received from window_start to the completion, never released before, within the policy's bounds of number and age. Releasing an event restores the visibility it had before it was held (update_event, ON_EVENT_UPDATED), and the room's intelligence channels act on it as on any inbound message of its author: nothing is copied, and the last released event alone solicits an answer, the earlier ones being its context.

A released message MUST reach an agent as its author's own turn, never as an instruction or anything else carrying the application's authority: a message an unverified sender wrote, quoted or not, gains none by its author proving who they are. Nothing outside the window is released: what was written before the request — before a verification was revoked, during a request that failed — may not be the verified person's.

11.7.7 Security floor

• A secret carried in a link has at least 128 bits of entropy; a code is short, so its request is short-lived and allows few attempts.
• A secret is stored only as its hash, compared in constant time, used once, and never logged.
• A completion is refused after the request expired, failed, was cancelled or completed, and after the verification was revoked.
• Every transition emits its framework event (§8), the audit trail.
• A link is a bearer secret: forwarded, it verifies whoever opens it and passes the method's own check. That check (a PIN, a login) is the method's; a method with none SHOULD NOT guard anything that matters.

Verification state is scoped to the organization as identities are (§17.2). A message is labelled by the verification of its room's organization: one proven to an organization never verifies the person in another organization's conversation, whatever the organization the resolver's identity names.

---

12. Voice and Realtime Media

12.1 Architecture Overview

RoomKit supports four real-time media architectures:

Architecture 1: STT/TTS Pipeline (VoiceChannel)
┌──────────┐     ┌──────────────┐     ┌──────────┐     ┌─────┐     ┌──────────┐
│ Client   │────→│Audio Pipeline│────→│   STT    │────→│Room │────→│   TTS    │────→ Client
│ (audio)  │     │(denoise/VAD/ │     │Provider  │     │Kit  │     │Provider  │
└──────────┘     │ diarization) │     └──────────┘     └─────┘     └──────────┘
                 └──────────────┘

Architecture 2: Speech-to-Speech (RealtimeVoiceChannel)
┌──────────┐     ┌──────────────┐     ┌──────────────────────────┐     ┌──────────┐
│ Client   │────→│Audio Pipeline│────→│ Speech-to-Speech API     │────→│ Client   │
│ (audio)  │     │  (optional)  │     │ (OpenAI Realtime, etc.)  │     │ (audio)  │
└──────────┘     └──────────────┘     └────────────┬─────────────┘     └──────────┘
                  denoise, diarize                  │
                                              transcriptions
                                              tool calls
                                                    │
                                                    ▼
                                               ┌─────────┐
                                               │ RoomKit │
                                               │ (events)│
                                               └─────────┘

Choosing between architectures:

Criterion	VoiceChannel (STT/TTS)	RealtimeVoiceChannel (Speech-to-Speech)	VoiceChannel (Bridge)	ConferenceChannel (SFU)
Latency	Moderate (~500-800ms TTFA with streaming)	Lower — end-to-end streaming	Lowest — direct audio forwarding	Low — SFU forwards media directly between clients
Control	Full — choose STT, LLM, TTS	Limited — provider bundles all; with reasoning delegation (Section 12.4.1) the reasoning model is the integrator's	Full pipeline, no AI required	Control plane only — media plane is external
Text access	Always — utterances become RoomEvents	Optional — if transcription configured	Optional — if STT configured	Per-track STT, attributed per participant
Multi-channel	Native — text routes to any channel	Requires transcription	Requires STT for text routing	Native — transcripts are RoomEvents
AI involvement	Required (generates responses)	Required (speech-to-speech)	Optional (observer/monitor)	Optional (bot participant)
Voice quality	Depends on TTS provider	Native speech generation	Original voice preserved	Original voice preserved
Pipeline control	Full audio pipeline (all stages)	Pipeline optional	Full pipeline (all stages)	Per-track inbound lanes (no AEC/diarization)
Use when	AI voice agent with text in the loop	Lowest latency AI with native voice	Human-to-human calls, conferencing, call center	Multi-party video conferences, AI in meetings

All four architectures MAY coexist in the same room. For example, a room could have a VoiceChannel for a human participant (PSTN/SIP) and a RealtimeVoiceChannel for the AI agent, with text events bridging them. Or two VoiceChannels in bridge mode for a human-to-human call with optional AI observation.

Architecture 3: Audio Bridge (VoiceChannel with bridge=True)
┌──────────┐     ┌──────────────┐     ┌──────────────┐     ┌──────────────┐     ┌──────────┐
│ Client A │────→│Audio Pipeline│────→│ Audio Bridge  │────→│Audio Pipeline│────→│ Client B │
│ (audio)  │     │  (inbound)   │     │ (forward/mix) │     │  (outbound)  │     │ (audio)  │
└──────────┘     └──────────────┘     └───────┬───────┘     └──────────────┘     └──────────┘
                                              │
                                         (optional)
                                       STT transcription
                                              │
                                              ▼
                                         ┌─────────┐
                                         │ RoomKit │
                                         │ (events)│
                                         └─────────┘

Architecture 4: SFU Conference (ConferenceChannel — Section 12.10)
┌──────────┐           ┌─────────────────┐           ┌──────────┐
│ Client A │◄─────────►│                 │◄─────────►│ Client B │
│ (browser)│   media   │  External SFU   │   media   │ (browser)│
└──────────┘           │  (media plane)  │           └──────────┘
                       └────────┬────────┘
                                │ bot participant
                                │ (subscribed tracks, published TTS)
                                ▼
                       ┌─────────────────┐
                       │     RoomKit     │
                       │ per-track STT,  │
                       │ vision, TTS bot │
                       │ (control plane) │
                       └─────────────────┘

12.2 Voice Channel (STT/TTS Pipeline)

The Voice Channel processes audio through a pipeline of STT and TTS providers, converting speech to text events and text responses back to speech.

Required components:

Component	Interface	Purpose
VoiceBackend	See below	Real-time audio transport (WebRTC, WebSocket)
STTProvider	See below	Speech-to-text conversion
TTSProvider	See below	Text-to-speech synthesis

VoiceBackend interface:

VoiceBackend (interface)
├── connect(room_id, participant_id, channel_id) → VoiceSession
├── disconnect(session) → void
├── send_audio(session, audio_chunks) → void
│       # Decoded 16-bit PCM only; encoding for the wire is the backend's
│       # An exception raised by audio_chunks MUST reach the caller once the
│       # backend has stopped playing; the backend MAY log and absorb errors of
│       # its own transport
├── send_audio_sync(session, audio_chunk) → void
│       # Send one chunk without awaiting, from an audio callback thread; the
│       # audio bridge sends through it (Section 12.7). Backends that support
│       # bridging SHOULD override the default, which schedules send_audio()
│       # on the calling thread's running event loop (adding latency) and
│       # drops the chunk with a warning when that thread runs no loop
├── cancel_audio(session) → void            # Cancel current playback (if supported)
├── send_dtmf(session, digit, duration_ms) → void  # Send outbound DTMF (RFC 4733)
│
│   # Raw audio callback:
├── on_audio_received(callback) → void      # Raw inbound audio frames from client
│
│   # Barge-in (transport-level detection):
├── on_barge_in(callback) → void            # Client audio detected during playback
│
│   # Session lifecycle:
├── on_session_ready(callback) → void       # Audio path is live, ready for send/receive
├── on_client_disconnected(callback) → void  # Client or transport disconnected
├── get_session(session_id) → VoiceSession | null  # null if sessions are not tracked
├── list_sessions(room_id) → list<VoiceSession>    # Empty if sessions are not tracked
│
│   # Server-side session acceptance:
├── accept(request) → VoiceSession          # Accept an inbound session (SIP/PSTN/WebRTC offer)
│
│   # Protocol tracing:
├── set_trace_emitter(emitter | null) → void  # Set callback for emitting ProtocolTraces
│
│   # Properties:
├── name: string                            # Backend identifier for attribution and logging
├── auto_connect: bool                      # Auto-create session when channel is attached to a room
│
├── capabilities() → VoiceCapability        # What the backend supports
└── close() → void

Note: on_speech_start, on_speech_end, on_vad_silence, and on_vad_audio_level are removed from VoiceBackend. These events are now emitted by the audio pipeline's VAD provider (Section 12.3). on_partial_transcription is emitted by the STT provider. on_barge_in remains on VoiceBackend as it is a transport-level concern (detecting client audio during active playback).

VoiceCapability flags:

Flag	Description
INTERRUPTION	Can cancel TTS playback
BARGE_IN	Detect user audio during TTS playback
NATIVE_AEC	Backend provides built-in echo cancellation (skip pipeline AEC)
NATIVE_AGC	Backend provides built-in gain control (skip pipeline AGC)
DTMF_INBAND	Backend receives in-band DTMF tones in audio stream
DTMF_SIGNALING	Backend sends and receives DTMF via signaling (RFC 4733)
NATIVE_BRIDGE	Backend can bridge audio at the transport level (RTP relay)

When a VoiceBackend declares NATIVE_AEC or NATIVE_AGC, the pipeline MUST skip the corresponding stage automatically, even if a provider is configured in AudioPipelineConfig. This prevents double processing.

When a VoiceBackend declares DTMF_SIGNALING, DTMF events arrive via the signaling layer (not the audio stream). The pipeline's DTMFDetector handles in-band detection only. Implementations MUST merge both sources into the same ON_DTMF hook.

VoiceSession:

VoiceSession
├── id: string                              # Session identifier
├── room_id: string                         # Associated room
├── participant_id: string                  # Associated participant
├── channel_id: string                      # Associated voice channel
├── state: VoiceSessionState                # Current session state
├── created_at: datetime                    # Session start time
├── metadata: map<string, any>              # Session-specific data
└── last_usage: map<string, any>            # Read-only: what the provider recorded last (Section 12.4.2)

VoiceSessionState enumeration:

Value	Meaning
CONNECTING	Session is being established (transport handshake, provider init)
ACTIVE	Audio is flowing bidirectionally
PAUSED	Session temporarily suspended (e.g., hold, mute)
ENDED	Session terminated — no further audio

Transitions:

• CONNECTING → ACTIVE: Transport connected and provider ready.
• ACTIVE → PAUSED: Integrator pauses session (e.g., call hold).
• PAUSED → ACTIVE: Integrator resumes session.
• ACTIVE → ENDED: Session terminated (user hangup, timeout, integrator call).
• CONNECTING → ENDED: Connection failed.
• PAUSED → ENDED: Session terminated while paused.

Implementations MUST NOT allow transitions from ENDED to any other state. A new session MUST be created if the participant reconnects.

STTProvider interface:

STTProvider (interface)
├── name: string
├── supports_language_override: bool (default false)
│       # Whether transcribe/transcribe_stream honour a per-call language
├── supports_diarization: bool (default false)
│       # Whether final results say who spoke (Section 12.2.3)
├── transcribe(audio_chunk, language: string | null) → TranscriptionResult
└── transcribe_stream(audio_stream, language: string | null) → async_iterator<TranscriptionResult>

TranscriptionResult
├── text: string
├── is_final: bool
├── confidence: float | null
├── language: string | null                 # The language the provider reports for the text
└── segments: list<SpeakerSegment>          # Who spoke which part of text (Section 12.2.3);
                                            # empty when the provider attributes nothing

SpeakerSegment
├── speaker: string | null                  # Provider label, stable within one stream; null = unattributed
├── text: string
├── start_ms: int | null                    # Offsets from the start of the stream
└── end_ms: int | null

language on a call overrides the provider's configured language for that call only. A provider that cannot honour it MUST report supports_language_override = false, and a Voice Channel MUST NOT pass a language to such a provider. TranscriptionResult.language is what the provider reports — what it detected when it was asked to detect — and MUST NOT be back-filled from the request: null means the provider said nothing.

A Voice Channel SHOULD let the STT language be chosen per session at runtime. The choice applies from the next stream opened for that session: a stream is opened per utterance (VAD mode) or per turn (continuous mode), and every streaming STT API RoomKit targets fixes the language for the life of a stream. The typical flow starts a session in a detecting mode (Deepgram multi), reads the reported language, and pins the following streams to it. An implementation MAY ship that flow as a policy; it MUST leave the primitive reachable from hooks.

Audio between streams. In continuous mode, audio that arrives while no stream is open (between two turns, during a reconnect after an error) is carried into the next stream, so a reconnect loses no speech. What is carried MUST be bounded: a reconnect that takes long (a service that does not answer its handshake) would otherwise hand the next stream a backlog the service refuses, or a lag behind the conversation that only grows. A Voice Channel keeps the most recent audio and drops the oldest, and MUST expose how much it dropped (RoomKit keeps 5 s and logs a warning naming the seconds dropped). A provider whose service limits how far ahead of real time audio may be sent MUST pace what it sends rather than send a backlog at once. Meta's realtime service refuses a stream sent 7 s or more ahead at once ("Audio processing backlog too large", measured 2026-10-07); RoomKit's Meta provider sends at most 3 s at once, then at most twice real time until it has caught up.

TTSProvider interface:

TTSProvider (interface)
├── name: string
├── supports_streaming_input: bool (default false)
│       # Whether this TTS accepts streaming text input
├── context_level: TTSContextLevel (default NONE)
│       # What conversation context this TTS consumes (Section 12.2.2)
├── max_dialogue_speakers: int (default 0)
│       # Distinct speakers one synthesize_dialogue call can voice; 0 = no dialogue
├── available_voices() → VoiceInfo[]
│       # Curated offline catalog (MAY be empty); no API call
├── list_voices(language: string | null, gender: string | null,
│               query: string | null) → VoiceInfo[]
│       # Live catalog; default: available_voices(), filtered
├── synthesize(text, voice: string | null) → AudioContent
│       # Returns AudioContent (Section 5) with url, mime_type, transcript, duration_seconds
├── synthesize_stream(text, voice: string | null,
│                     context: TTSContext | null) → async_iterator<AudioChunk>
│       # Stream audio from a complete text string
├── synthesize_stream_input(text_stream, voice: string | null,
│                           context: TTSContext | null) → async_iterator<AudioChunk>
│       # Stream audio from streaming text chunks (requires supports_streaming_input = true)
│       # Accepts an async iterator of text strings (e.g., sentences from a sentence splitter)
│       # Used by VoiceChannel for streaming AI → TTS
├── synthesize_dialogue(turns: list<DialogueTurn>,
│                       voices: map<string, string>) → AudioContent
│       # One clip of several speakers (requires max_dialogue_speakers > 0)
└── release_context(context_id: string) → void
        # Drop any state held for this context (default: no-op; Section 12.2.2)

VoiceInfo
├── id: string                              # What synthesize() and connect() accept as voice
├── name: string | null
├── language: string | null                 # BCP-47 tag, or "multilingual"
├── gender: string | null                   # "male" | "female" | "neutral"
├── accent: string | null
├── description: string | null
├── deprecated: bool (default false)
└── attributes: map<string, string>         # Provider-specific (persona, age, use case, category)

DialogueTurn
├── speaker: string                         # Key into the call's voices map
├── text: string
└── style: string | null                    # Delivery direction for this turn only

A TTS provider that builds a speech model's prompt around the text to speak (a model with no field of its own for the text and its delivery) MUST set the text apart as a <transcript> block (Section 6.4), so that the text can neither end the transcript nor open another; a line a realtime session is asked to say keeps the quote of Section 12.4. A generative speech model may perform a delivery cue written in the text it speaks (an audio tag, a sentence such as "whisper this"), whatever frame holds the text: text the application does not trust has such cues removed before it reaches the model, in a BEFORE_TTS hook where a channel speaks it, and by the application itself before it calls synthesize() directly or injects an assistant line into a realtime session.

synthesize() returns an AudioContent (Section 5) rather than raw bytes. This allows the result to carry metadata (transcript, duration, MIME type) and supports both inline data URLs and remote storage URLs. For streaming, synthesize_stream() yields AudioChunk objects as they become available. synthesize_stream_input() is the streaming-input variant: it accepts an async iterator of text chunks (typically sentences) and yields audio as each chunk is synthesized. This enables the streaming AI → TTS pipeline where LLM tokens are buffered into sentences and fed to TTS incrementally.

Encoding. A streamed AudioChunk names its encoding in format (Section 12.3.11), and a vendor offering several output encodings may stream any of them (MP3, Opus, G.711). A Voice Channel plays decoded 16-bit PCM only: it MUST refuse a TTS chunk whose format is not pcm_s16le (or pcm) before the chunk reaches the outbound pipeline or the transport, with an error naming the format, rather than play encoded bytes as samples. Encoding for the wire belongs to the backend, as it does for a conference (Section 12.10.3); a TTS that speaks on a Voice Channel is configured for PCM output.

context carries the conversation so far to a TTS that can use it (Section 12.2.2). A provider whose context_level is NONE never receives one, so an existing provider needs no change.

Voices. A voice is named by the id of a VoiceInfo: the string a provider returns in its catalog is the string synthesize() accepts, with no translation between the two. available_voices() is the curated catalog a provider ships, readable without credentials or network; list_voices() asks the vendor, and includes the custom voices the caller owns (Section 12.2.4). A provider whose vendor has no voices endpoint keeps the default, which filters its curated catalog. The filters apply wherever the catalog comes from, server side or on the results, so a caller gets the same answer from every provider: language matches a voice whose tag equals it or starts with it and a hyphen ("fr" matches fr-CA, and fr-CA does not match fr-FR); gender matches exactly; query matches, without regard to case, a substring of the name or the description. A voice whose language or gender is unknown does not match a filter on it. VoiceInfo fields a vendor does not report are null or absent, and MUST NOT be inferred from another voice; attributes carries what a vendor reports beyond the common fields, under the vendor's own names. The RealtimeVoiceProvider catalog (Section 12.4) uses the same VoiceInfo.

Dialogue. synthesize_dialogue() voices a scripted exchange in one clip, each turn in the voice its speaker is mapped to, the way a podcast or a training role-play is produced. A provider whose max_dialogue_speakers is 0 MUST raise, and every provider MUST raise before any call when the turns name more distinct speakers than its maximum or a speaker the voices map does not hold. style directs one turn's delivery, and a provider that cannot direct a single turn MUST raise rather than drop it. The returned AudioContent carries the turns, each as "<speaker>: <text>" on its own line, as its transcript. A Voice Channel speaks with one voice and never calls synthesize_dialogue(): dialogue is produced by the integrator, and enters a room as any other audio.

Audio processing flow:

1. Client sends audio stream → VoiceBackend
2. VoiceBackend emits raw frames → Audio Pipeline (Section 12.3)
3. Pipeline: [Resampler] → [AEC] → [AGC] → [Denoiser] → [VAD] → [Diarization]
   (DTMF detection runs in parallel from resampled stream)
   (Recorder captures inbound after resampler)
4. VAD detects speech end → ON_SPEECH_END hook fires
5. STT transcribes captured audio → TranscriptionResult
6. Fire ON_TRANSCRIPTION hook (can modify transcript)
6a. If the TTS consumes context → record the user turn (Section 12.2.2)
7. If TurnDetector configured → evaluate turn completion
   ├── Complete → Create RoomEvent
   └── Incomplete → accumulate, wait for next speech

--- Standard path (non-streaming) ---
8. Route through normal inbound pipeline
9. Room broadcasts → AI or other channels respond
10. Response event arrives at Voice channel via deliver()
11. If TextContent → Fire BEFORE_TTS hook → TTS synthesizes → AudioChunk stream
    (with a sentence budget, 12s.e, the first sentences of a text delivered
     whole, a response or any text the channel speaks of a room event, are
     spoken up to it, cut by the same sentence rule as the streamed path; the
     response was stored whole before it reached the channel)
    (each session plays in parallel; a session whose synthesis or playback fails
     is reported as `tts_error` once its playback is released, and AFTER_TTS fires
     only when a session was served)

--- Streaming AI → TTS path (framework-native, when AIProvider supports streaming) ---
8s. Route through normal inbound pipeline → Room broadcasts to AIChannel
9s. AIChannel.on_event() returns ChannelOutput(response_stream=…), the text of the
    turn's tool loop (Section 6.4, One loop)
10s. Framework detects response_stream in broadcast result
11s. Framework finds streaming delivery targets (channels with supports_streaming_delivery)
12s. Framework pipes stream → VoiceChannel.deliver_stream():
     a. Sentence splitter buffers tokens → yields at sentence boundaries (.!?)
        (min_chunk_chars threshold prevents very short TTS fragments), and at
        the end of each text segment (Section 6.4: a call's start ends one, as
        does a round the loop tries again). What the segment still holds goes
        out then as its last sentence, however short, before the call runs: a
        reply's "Let me check." is said while the tool works, not once it is
        done, and never runs on into the next round's first word. A streaming
        text filter lets out there the text it has decided to speak; a span it
        removes (an `[internal]` block), opened before the call, stays open
        across it, so its text is never spoken, whichever segment closes it.
     b. BEFORE_TTS runs on each sentence before the TTS reads it, once for all
        sessions: the sentence is the hook's payload, a MODIFY replaces it, a
        BLOCK drops it, and the next sentence is judged on its own. The
        fail-closed rule of Section 9.3 applies sentence by sentence: a hook
        that raises, times out or returns something unusable drops that
        sentence, which is never synthesized. A sentence a hook redacts to an
        empty string is not synthesized either. The hook sees the sentence as
        the TTS would read it, after any text filter (a streaming filter works
        on the tokens, before the splitter) — where the standard path runs the
        hook before the filter.
     c. tts.synthesize_stream_input(sentences) → AudioChunk stream (~200ms TTS TTFB)
        First audio reaches speaker ~500-800ms after speech end (vs ~2-3s standard)
     d. Several sessions on the binding: the stream is read once and every session
        receives every sentence, in parallel. The stream is pulled at the pace of
        the fastest session; a session that stops early (barge-in, transport error,
        its TTS failing) does not cut the others off. A session whose TTS fails is
        reported as `tts_error`; its failure is not the response's, which takes
        step 13s once every session has stopped (no `ON_ERROR`, no replay).
     e. With a sentence budget (the channel's `max_sentences`), the sentences
        BEFORE_TTS left are counted. Once the budget is said, a reply that
        goes on (a word, a tool call, another round) stops there, as when
        every session stops early (step 13s): the channel reads no further,
        before the turn asks for a call its next text would announce; what it
        held of the sentence over the budget is not synthesized; the stream is
        closed (no further token, no tool call that has not started), and the
        text produced so far is stored marked cancelled, so the room keeps
        about what was heard rather than what was never said, and the turn
        reports no response (`ON_AI_RESPONSE`), as any turn its reader
        stopped. A reply of exactly the budget is not cut. The application's
        own text (`say()`) has no budget.
13s. Framework accumulates full text from stream → stores AI response event
     Interrupted: when deliver_stream() returns before the stream is exhausted,
     for any reason (typically every session of 12s.d stopped early), the
     framework MUST close the response stream (no further token is generated and
     no tool call starts after that point) and MUST store the text already
     produced as the AI response event with `metadata.cancelled = true`. A turn
     its reader stopped once it began records it `cancelled`, as its caller and
     a delegation read it (Section 6.4, Section 23.3), unless its record already
     names how it was cut (an ACP agent's `interrupted`, a stop reason): an
     agent that finished while its reader had stopped (an ACP prompt that
     returned) is cut all the same. A delegated turn so stopped fails, never
     handing on its partial text as an answer. A turn begins with the first
     thing its reader is handed, a thinking delta included; a response never
     read names no end. A tool
     call already executing when the stream stops is let finish: its
     TOOL_CALL_END is stored with its real result, and the stream is closed
     before the model's next round. A call announced but not yet executing never
     runs; its TOOL_CALL_END is stored with `status = failed`, so no start row
     stays pending. A turn cancelled from outside (e.g. an operator aborting it)
     stores its text the same way, but aborts a running tool, whose
     TOOL_CALL_END is stored with `status = failed` as well, and so does a turn
     that fails while a tool runs. A TOOL_CALL_END written once the stream no
     longer carries rows goes to every channel, the streaming one included. With
     `flush_partial_tts = false` the sessions keep reading, so the stream runs
     to its end and nothing is cancelled.
     Failed: when the response stream raises (its provider fails mid-response),
     the framework MUST store the text already produced as the AI response event,
     not marked cancelled, close every call still open with `status = failed`,
     fire `ON_ERROR` and return the failure to the caller, even when the streaming
     channel swallows it. That event reaches the streaming channel through the
     stream, before the failure, as a completed response's last event does: the
     channel was handed every token of it, so it MUST NOT receive that text again
     as an ordinary delivery (no second synthesis, no second message). A channel
     that buffers the stream instead of rendering it delivers what it buffered,
     then the failure propagates. A failure of the streaming channel itself is not
     the response's: what it was handed may not all have been rendered, so the
     text goes out as an ordinary event to every channel, the streaming one
     included.
14s. Framework re-broadcasts complete event to non-streaming channels (exclude_delivery
     skips channels that already received streaming content, whether the
     response completed or failed)
15s. Fire AFTER_TTS hook with the text the sessions were sent: the sentences as
     BEFORE_TTS left them (12s.b) when a hook changed or dropped one, or as many
     as the budget let through when it ended the reply (12s.e), the whole
     streamed text otherwise, its segments (12s.a) apart as `ON_AI_RESPONSE`
     sets them. The final assistant transcript carries the same text. AFTER_TTS does not fire when every sentence was dropped, nor when no
     session was served (each stopped on a failed synthesis or playback). A
     response that fails (step 13s) is spoken to the end of the text it
     produced, its last partial sentence included: the sessions that heard it
     get that text as their final transcript and AFTER_TTS fires with it, then
     the failure propagates.
16s. When a served session's playback of the response reaches its end, the
     response is recorded `consumed` on the voice channel (Section 8.5); a cut
     records `partial` (Section 12.3.13), and a response superseded before any
     audio records `unconsumed` (Section 12.2).

One playback at a time: a response that reaches a session while another plays
there (a background task's result relayed as the agent finishes a sentence,
Section 23.4) waits until that playback has been sent whole, then plays. The
agent's words are cut by the person speaking (Section 12.3.13), never by its
own next response; several responses waiting play in turn, each whole: the
session is one response's from the moment it is found free, so of several
waiting one takes it and the others wait on that one. A playback sent whole
whose echo is still draining (no AEC) is taken over, not cancelled, by
whatever takes the session (a response, `say()` or `play()`): it was heard
whole, so no `ON_TTS_CANCELLED` fires and no buffered audio is flushed; its
drain, or the drain of a playback cut short, ends no later playback. The
application's own text (`say()`) does not wait: it cuts the playback under way
(`ON_TTS_CANCELLED`, reason `new_tts`), and the responses waiting play after
it. Audio the application plays from a file (`play()`) is not the agent's
words: a response cuts it (reason `new_tts`). Like `say()`, `play()` does not
wait: it cuts the playback under way (`ON_TTS_CANCELLED`, reason `new_tts`),
and its playback is measured as a response's is, so a cut reports, and the
session's played audio counts, what was heard of it.

--- Common outbound path ---
12. AudioChunk stream → [PostProcessors] → [Recorder] → [Resampler] → Transport
    (TTS emits variable-size AudioChunks; pipeline stages that require fixed-size
     AudioFrames MUST buffer and re-chunk the stream internally)
13. AEC reference: Transport feeds AEC.feed_reference() at playback time (local hw)
    or pipeline feeds it in the outbound path (network transports) — see 12.3.4
14. If barge-in detected → InterruptionConfig determines response
15. If the TTS consumes context → record the assistant turn once playback ends
    or is cancelled (Section 12.2.2)

The streaming path flows through the framework's normal routing infrastructure. AIChannel returns a streaming handle, the framework pipes it to streaming-capable transport channels, accumulates the full text, stores the complete event, and re-broadcasts to non-streaming channels. This preserves multi-channel delivery, hooks, tool calling (falls back to non-streaming), and keeps VoiceChannel as pure transport with no AIProvider reference.

12.2.1 Packet Loss Handling (Lossy Transports)

Backends carrying audio over lossy transports (RTP/SIP, datagram-based) are the only layer that can observe packet loss: sequence numbers exist at the transport, and the pipeline above sees only PCM frames. A backend that silently drops lost packets compresses the delivered timeline — recordings shorten, AEC reference alignment drifts — and no downstream stage can detect or repair it.

Packet loss concealment (PLC) is OPTIONAL for Level 3 conformance and RECOMMENDED for RTP-based backends.

Detection. A backend implementing PLC:

• MUST distinguish packets that have not yet arrived (reordering) from packets confirmed lost, using sequence-number analysis with a bounded reordering window. Packets the jitter buffer can still deliver MUST NOT trigger concealment.
• MUST NOT treat sender-side transmission pauses as loss. Discontinuous transmission (VAD suppression, comfort noise) advances the RTP timestamp without consuming sequence numbers; only sequence-number gaps indicate loss.
• MUST account for non-media packets filtered out of the audio path before loss detection. RFC 4733 telephone-events consume sequence numbers from the same RTP stream: a receiver that filters them upstream of its jitter buffer MUST mark those sequence numbers as received, so the filtered slots are neither confirmed as loss nor concealed.

Concealment. On confirmed loss, the backend:

• MUST deliver a temporally continuous stream: the lost duration is replaced with concealment audio so the frames emitted to the pipeline preserve timeline alignment. Downstream stages (AEC, recorder, STT) require no loss awareness.
• SHOULD use codec-native concealment when the negotiated codec provides it (e.g., libopus PLC). For codecs without native concealment (G.711, G.722, L16), the backend MUST fall back to a generic concealer that at minimum repeats the last received frame with progressive attenuation.
• MUST bound synthetic audio: a generic concealer MUST decay to silence within max_conceal_ms (RECOMMENDED 60 ms) of consecutive concealment and fill the remaining lost duration with silence. Codec-native concealment with built-in energy decay (e.g., libopus PLC) satisfies this requirement as-is. Either way, alignment is preserved without generating artifacts on long bursts.

Observability. A backend implementing PLC MUST expose, per session:

• packets_lost — count of packets confirmed lost.
• concealed_frames — count of frames synthesized by concealment.

Emitted concealed frames SHOULD be annotated (metadata.concealed = true on the AudioFrame) so that pipeline stages MAY treat them differently — e.g., the recorder journaling concealed ranges, or STT adjusting confidence.

12.2.2 Conversation Context for TTS

A TTS that synthesizes each sentence in isolation cannot carry prosody across a conversation. Some can, when they are told what came before: the voice's own previous generations, the text of the dialogue, or the dialogue's audio — including how the user spoke. The Voice Channel already holds all of it: the captured utterance at speech end, its final transcript after ON_TRANSCRIPTION, and what it synthesized and played. This section defines how it hands that to the TTS.

TTSContextLevel enumeration — what a provider consumes. Each level includes the ones above it:

Value	The provider consumes	Typical use
NONE	Nothing (default)	Most TTS engines
SELF	Which of its own previous turns were played, and how far	Continuity handles returned by a cloud API (a request id, a generation id)
TEXT	The text of every turn, both roles	Previous / next text conditioning
AUDIO	The text and the audio of every turn	Dialogue-conditioned models that decode inside the conversation

TTSContext
├── context_id: string                      # The VoiceSession.id the context belongs to
├── turns: list<ConversationTurn>           # Oldest first, bounded by TTSContextConfig
└── next_turn_id: string                    # The turn_id the assistant turn of this call will get

ConversationTurn
├── turn_id: string                         # Assigned by the channel, unique within the context
├── role: "user" | "assistant"
├── participant_id: string
├── text: string                            # As synthesized, or as transcribed after ON_TRANSCRIPTION
├── audio: AudioFrame | null                # The whole turn; only when the provider's level is AUDIO
├── played_ms: int | null                   # Assistant turns: audio actually played
└── interrupted: bool                       # Assistant turns: playback was cancelled

TTSContextConfig
├── enabled: bool = true                    # Whether the channel keeps a context at all
├── include_audio: bool = false             # Whether user and assistant audio is kept
├── max_turns: int = 20                     # Oldest turns are dropped past this
└── max_audio_seconds: float = 120.0        # Oldest audio is dropped past this; its text stays

Passing rules. A Voice Channel MUST NOT pass a context to a provider whose context_level is NONE, and MUST NOT put audio in a context unless the provider's level is AUDIO and include_audio is true. When include_audio is false, an AUDIO provider receives the context at TEXT level: every audio is null. The context passed to a call is a snapshot: the provider MUST NOT expect it to change during the call, and MUST NOT write to it.

One context per session. A context belongs to one VoiceSession and holds the dialogue heard on that session only — never other channels of the room, and never other sessions. The room timeline is not its source: the timeline mixes channels and holds no audio. A response delivered to several sessions is therefore synthesized once per session, each call with its own context.

Recording the user turn. After ON_TRANSCRIPTION (step 6a of the audio processing flow), the channel appends a user turn with the transcript as the hook left it and, when audio is kept, the utterance captured at speech end. If the hook changed the transcript — a redaction, typically — the channel MUST NOT keep the audio of that turn: the audio would still hold what the hook removed. A speech segment classified as a backchannel (Section 12.3.13) is not recorded.

Recording the assistant turn. When playback ends or is cancelled, the channel appends an assistant turn with the text actually sent to the TTS — after BEFORE_TTS and any text filter — under the next_turn_id the call was given, and played_ms set to the duration played. When audio is kept, the turn carries the synthesized audio. A call cancelled before any of its audio was played leaves no turn: the user heard nothing of it. On cancellation, the turn records interrupted = true and holds only what was played: the audio is cut at played_ms, and the text is cut to the part the provider or the channel can attribute to the played audio, or kept whole with played_ms as the only bound when no such attribution exists. A provider consuming the context MUST treat played_ms as the end of what the user heard.

Stateful providers. A provider that keeps its own state per context (a model that holds the dialogue in a KV cache, say) MAY rely on the turns it already saw and read only the new ones on each call, keyed by context_id and turn_id. A provider that needs its own handle for a turn — the request id a cloud API returned, at level SELF — keeps it under the call's next_turn_id, and finds it again when that turn appears in a later context, with its played_ms. A call cancelled before any audio was played leaves no turn (see above): the provider MUST NOT assume every next_turn_id it saw is recorded. The channel MUST call release_context(context_id) when the session is unbound or ends, and the provider MUST drop everything it holds for that context. release_context on an unknown context is a no-op.

Scope. This version defines the context for the Voice Channel's 1:1 sessions. A Conference Channel (Section 12.10) MUST NOT pass a context to its TTS. A Realtime Voice Channel (Section 12.4) is out of scope: a speech-to-speech provider keeps its own conversation.

12.2.3 Speaker Attribution from STT

Streaming recognisers now label speakers themselves, so who spoke can come from the STT as well as from the pipeline's diarization stage (Section 12.3.9). The STT's label is aligned to the words it transcribed; this section defines what a provider reports and how a channel carries it to the room.

Status. Implemented: the provider contract, the refusals, a Voice Channel carrying labels in continuous mode, and the opt-in naming from the pipeline stage. A Voice Channel that does not implement Stream lifetime, Into the room and Speaker change MUST refuse a diarizing STT at construction, in every mode, with an error that says so.

Labels. A speaker label is assigned by the provider, is opaque, and is stable within one STT stream and only there: two streams may give one voice different labels, and one label to different voices. A provider MUST report labels as strings (a vendor's integer 0 becomes "0") and MUST report a speaker it could not attribute as null, whatever the vendor spells it (UU, PENDING, unknown in any case, or empty). A provider MUST NOT invent a label the vendor did not send, nor carry one over from another turn.

Segments. A provider reporting supports_diarization = true MUST give every final result the segments its text is made of, in order, each attributed to one speaker: a vendor that labels turns yields one segment per turn, a vendor that labels words yields one segment per run of consecutive words sharing a label, and text the vendor attributed to nobody is a segment with a null speaker. Partial results MAY carry segments. A provider reporting false MUST return no segments. A label MAY be revised by the vendor after the fact; a Voice Channel does not apply a revision to a result it has already routed.

Channels that cannot carry labels. Labels compare only within one stream, so a channel that opens one stream per utterance (Voice Channel in VAD mode) or per flush (batch mode) MUST refuse a diarizing STT at construction. So MUST a Conference Channel (Section 12.10): it attributes speech by participant track and transcribes each utterance on its own. Several people sharing one track (a room microphone) is out of scope in this revision.

Stream lifetime. In continuous mode, a Voice Channel with a diarizing STT MUST keep one stream open across turns, and open a new one only when the provider ends it, on an error, or on a language change (Section 12.2). The channel SHOULD keep the stream's audio level with the clock, filling a pause with silence when the stream is behind (a microphone muted during playback, packets lost over a long call): a provider may end a stream whose audio falls behind real time, which would start a new epoch. A transcript it discards as echo of its own playback does not end the stream either. Each stream a session opens starts a new label epoch, numbered from 0 for the session. The epoch travels with every label (below), and a consumer MUST NOT equate two labels from different epochs.

Into the room. For a final result with segments, the Voice Channel fires ON_SPEECH_END once, then:

1. Routes one inbound message per segment, in the order the finals and their segments were produced: a room message has one speaker. Ordering covers the commit of each message, not the reply to it: a reply still being produced MUST NOT hold back the next segment. A change of speaker ends the pending turn: a turn detector (Section 12.3.12) MUST NOT join two speakers' segments into one message. ON_TRANSCRIPTION fires per segment, and its event carries the segment's speaker, the speaker_epoch and the sender_name below; a hook MAY replace sender_name (a label matched to a known voice).
2. Keeps sender_id as the session's participant, the owner of the audio stream. A label is not a participant (Section 12.3.9): mapping one to a known person is the integrator's concern.
3. Sets on the message metadata.speaker_label, metadata.speaker_epoch and metadata.sender_name. sender_name defaults to "Speaker <label>" in epoch 0 and "Speaker <label>#<epoch>" after, so the same letter from two epochs never reads as one person. A segment with a null speaker carries no label and sender_name = "Unknown speaker": without a name, the AI channel would fall back to the stream owner's for words nobody was attributed. The AI channel attributes a turn to sender_name when the room holds more than one speaker, so the model reads "Speaker A: …" rather than one anonymous stream.

Speaker change. Evaluated per routed segment, in order. A segment with a null speaker neither fires nor resets the last label. The framework MUST fire ON_SPEAKER_CHANGE with source = "stt" when a segment's label differs from the last label routed in the same epoch, and on the first labelled segment of each epoch. The event's speaker_id is the label, speaker_epoch its epoch, is_new_speaker is true when the label had not been routed before in that epoch, and confidence is null unless the provider reports one. Events from the pipeline stage carry source = "pipeline".

With the pipeline stage. When a session has both a diarizing STT and a DiarizationProvider, the STT's label is the one attached to the transcript, being aligned to its words; the stage keeps firing ON_SPEAKER_CHANGE with its own source.

From the pipeline stage. When the STT is not diarizing, a Voice Channel MAY name a transcript's speaker from its DiarizationProvider (Section 12.3.9), on the integrator's opt-in. A channel MUST refuse the option at construction without a diarization stage, and in batch mode. With it:

1. A transcript's speaker is the stage's result heard the longest over the audio it transcribes: in VAD mode the utterance, its SPEECH_END frame included; in continuous mode the audio since the previous final. A result stands for the audio since the stage's previous result on that stream, so a stage that judges a buffer weighs each verdict by what it heard. The pipeline runs its VAD before its diarization stage, and a stage may identify only on the SPEECH_END frame, so a channel that reads the stage when SPEECH_END fires MUST wait until that frame has been through it.
2. The stage's speaker_id is normalised as an STT label (Labels). A result that attributes the audio to nobody (unknown) counts as a null speaker; if heard the longest, the transcript's sender_name is "Unknown speaker". A transcript over which the stage gave no result carries no speaker, as without the option.
3. The stage keeps its labels for the whole session, so the epoch is 0.
4. The transcript reaches the room as one message, as Into the room sets it: ON_TRANSCRIPTION carries the speaker and MAY rename it, and the message carries speaker_label, speaker_epoch and sender_name. The channel fires no ON_SPEAKER_CHANGE with source = "stt" for it: the stage fires its own.

12.2.4 Custom Voices

A voice can be made rather than picked: designed from a description, or replicated from a recording of a person. Making one manages the vendor's voice store; it is not synthesis, so it has an interface of its own, and the voice it returns is spoken through the vendor's TTS like any other (Section 12.2).

VoiceLibrary (interface)
├── name: string
├── supports_design: bool (default false)
├── supports_replication: bool (default false)
├── design_voice(description: string, store: bool,
│                name: string | null) → CustomVoice
│       # A voice written from a natural-language description
├── replicate_voice(sample: AudioContent, consent: AudioContent,
│                   store: bool, name: string | null) → CustomVoice
│       # A person's voice, from a recording of it and their recorded consent
├── get_voice(voice_id: string) → CustomVoice | null
└── delete_voice(voice_id: string) → void

CustomVoice
├── voice: VoiceInfo                        # voice.id is what the TTS accepts
├── stored: bool                            # Held by the vendor, listed by its TTS
├── expires_at: datetime | null             # When the vendor stops honouring the id
└── sample: AudioContent | null             # The vendor's preview of the voice

Kinds. A library MUST raise for a kind of creation its supports_design or supports_replication does not announce.

Storage. store = true asks the vendor to keep the voice for the caller's account until it is deleted or expires; such a voice appears in the vendor TTS's list_voices() and is read back with get_voice(). store = false asks for a voice the vendor keeps nothing addressable for: the returned id is the only handle, and get_voice() cannot find it. A vendor that offers one mode only MUST make the library raise for the other, rather than return the mode it has. get_voice() returns null for an id the vendor does not hold; deleting such an id is not an error.

name is the display name a stored voice is listed under; a vendor that names nothing ignores it.

Consent. replicate_voice() takes the consent recording as a required argument: a recording, by the person whose voice sample holds, agreeing to its replication. A vendor MAY prescribe the words, read verbatim, and refuse any other; a library SHOULD then expose the statement per language, so an application can show the person what to read. A library MUST send the recording to a vendor that verifies it, MUST NOT return a voice the vendor refused, and MUST raise the refusal as an error of its own, distinct from any other failure, so the application can ask for a new recording. It MUST raise without calling a vendor that has no consent check of its own when the integrator has not declared, through configuration, that consent is established outside the call. A TTS provider MAY also take reference audio in its configuration, a local model cloning a voice per call; the same obligation then lies with the integrator. Section 17.6 governs how reference and consent audio are handled.

12.3 Audio Processing Pipeline

The audio processing pipeline is a configurable layer between the transport and the conversation engine. The transport delivers raw audio frames; the pipeline processes them before they reach STT or speech-to-speech providers. All stages are optional — configure what the use case requires. A symmetric outbound pipeline processes audio before it reaches the transport.

This design ensures that audio processing capabilities (VAD, denoising, diarization) are available regardless of the transport choice (FastRTC, WebSocket, SIP, or any custom transport) and regardless of the voice architecture (STT/TTS or speech-to-speech).

Inbound (preprocessing):

Transport → [Resampler] → [Recorder ◉] → [AEC] → [AGC] → [Denoiser] → [VAD] → [Diarization] → STT
             (normalize)   (capture raw)  (echo)  (volume)   (clean)    (detect)  (who speaks)
                                 │
                           [DTMF Detector]
                           (parallel from resampled stream)

◉ Recorder taps the stream (non-blocking copy), does not modify frames.

Note on ordering rationale:

Position	Stage	Why here
1	Resampler	All downstream stages expect a consistent format (e.g., 16kHz mono PCM). Must run first.
1b	Recorder (inbound tap)	Capture raw user audio in consistent format before any processing — compliance requires unmodified signal.
2	AEC	Needs clean format but must run before denoiser — echo is not "noise", it's a known reference signal.
3	AGC	Normalize volume before denoiser and VAD — both are amplitude-sensitive.
4	Denoiser	Remove environmental noise from volume-normalized, echo-cancelled audio.
5	VAD	Detect speech in clean audio.
6	Diarization	Identify speaker from clean speech segments.
—	DTMF	Runs in parallel on the resampled stream — DTMF tones would be destroyed by denoiser/AGC.

Outbound (postprocessing):

TTS / Speech-to-Speech Provider → [PostProcessors] → [Recorder ◉] → [AEC ref †] → [Resampler] → Transport
                                   (normalize, etc.)  (capture final)  (feed AEC)   (match format)

◉ Recorder taps outbound stream after postprocessors (what the user actually hears).

† AEC reference feeding: for network transports, the pipeline feeds AEC.feed_reference() here.
  For local hardware transports, the transport itself feeds reference from the speaker output
  callback for precise time alignment (see Section 12.3.4). The pipeline skips this step when
  the transport handles AEC reference feeding directly.

Pipeline symmetry with text hooks:

Text Pipeline	Audio Pipeline
BEFORE_BROADCAST (sync hooks)	Inbound audio preprocessing (resampler → AEC → AGC → denoiser → VAD)
AFTER_BROADCAST (async hooks)	Outbound audio postprocessing (postprocessors → recorder → resampler)
ON_TRANSCRIPTION hook	Post-STT hook (exists in Section 12.5)
BEFORE_TTS hook	Pre-synthesis hook (exists in Section 12.5)
—	Turn detection (post-STT, pre-room event)
—	Interruption handling (barge-in → backchannel detection)

Note on subsection ordering: The subsections below (12.3.1–12.3.15) are organized by concern (configuration, then inbound stages, then outbound stages, then frame format, then post-STT stages, then execution flow, then debug taps) — NOT by pipeline execution order. For execution order, see the inbound/outbound diagrams above and the Pipeline Execution Flow (Section 12.3.14).

12.3.1 Pipeline Configuration

AudioPipelineConfig
├── resampler: ResamplerConfig | null        # OPTIONAL format normalization (RECOMMENDED)
├── aec: AECProvider | null                  # OPTIONAL echo cancellation
├── agc: AGCProvider | null                   # OPTIONAL automatic gain control
├── denoiser: DenoiserProvider | null        # OPTIONAL noise reduction
├── vad: VADProvider | null                  # OPTIONAL voice activity detection
├── diarization: DiarizationProvider | null  # OPTIONAL speaker identification
├── dtmf: DTMFDetector | null               # OPTIONAL DTMF tone detection
├── turn_detector: TurnDetector | null       # OPTIONAL semantic turn detection (post-STT)
├── recorder: AudioRecorder | null           # OPTIONAL bidirectional recording
├── recording_config: RecordingConfig | null # OPTIONAL recording parameters
├── postprocessors: list<AudioPostProcessor> # OPTIONAL outbound processing chain
├── vad_config: VADConfig | null             # OPTIONAL VAD tuning parameters
├── interruption: InterruptionConfig | null  # OPTIONAL barge-in behavior; when null,
│                                            # VoiceChannel uses its legacy parameters (A.10)
└── debug_taps: PipelineDebugTaps | null     # OPTIONAL diagnostic stage capture (Section 12.3.15)

All stages are OPTIONAL. At least one provider SHOULD be configured for the pipeline to be useful. Typical configurations:

• Voice Channel (STT/TTS): VAD (required for speech detection) + optional denoiser, AEC, AGC, and diarization.
• Realtime Voice Channel (speech-to-speech): Denoiser, AEC, and/or diarization; a VAD is optional and takes one of the two roles of Section 12.4.
• PSTN/SIP integration: Resampler (RECOMMENDED) + DTMF + recorder + VAD.

VADConfig
├── silence_threshold_ms: int | null = null   # Silence duration to trigger speech_end
├── speech_pad_ms: int | null = null          # Audio kept from before speech detection
├── min_speech_duration_ms: int | null = null # Minimum speech duration to emit event
└── extra: map<string, any> = {}              # Provider-specific settings

vad_config tunes the configured vad provider when the pipeline is built. Each field that is set MUST replace the provider's own value for that setting; a field left null MUST leave the provider's value unchanged, so the provider's defaults hold for everything the application did not set. extra carries settings specific to the provider (for example a model threshold); a provider MUST reject a key it does not know rather than ignore it. A provider that cannot apply a VADConfig SHOULD make that visible (a logged warning) instead of ignoring it silently.

The audio pipeline is configured per Voice Channel or per Realtime Voice Channel. Different channels in the same room MAY have different pipeline configurations.

Pipeline format contract:

AudioPipelineContract
├── internal_format: AudioFormat             # All stages operate on this format
├── transport_inbound_format: AudioFormat    # Format received from transport
└── transport_outbound_format: AudioFormat   # Format expected by transport

AudioFormat
├── sample_rate: int                         # Sample rate in Hz
├── channels: int                            # Channel count
├── sample_width: int                        # Bytes per sample
└── codec: string                            # Encoding (e.g., "pcm_s16le", "opus", "mulaw")

When the pipeline starts, implementations MUST validate that all configured stages are compatible with the internal format. If a stage requires a different format (e.g., a specific VAD requires 16kHz), the implementation MUST either: - Configure the resampler to match, or - Raise a configuration error at startup (fail fast).

Stream identity contract:

One AudioPipeline serves many audio streams. A Voice Channel bridging participants carries one stream per session; a conference carries one per lane. These are different speakers sharing one set of stage instances.

Every frame stage — resampler, AEC, AGC, denoiser, VAD, DTMF, diarization and postprocessors — therefore receives a stream key and MUST keep its state under it:

process(audio_frame: AudioFrame, stream: string) → ...
reset(stream: string) → void
close() → void

The stream key is an opaque identifier chosen by the caller — a session id, a lane id — and is stable for the life of that stream. Implementations MUST NOT share adaptive state between streams: VAD hangover and speech buffers, denoiser history, AGC gain, AEC adaptive filters and delay estimates, and diarization accumulators are all per stream. Sharing them makes one speaker's silence close another speaker's utterance.

stream MUST be a required parameter with no default value. A default would let an implementation accept the argument, ignore it, satisfy the interface, and mix streams silently — which is the failure this contract exists to prevent. A conformance check SHOULD verify that a stream's output sequence is unchanged when a second stream is interleaved with it.

The resampler is the one exception: its reset(stream | null) reads null as every stream, which is how a pipeline-wide reset calls it. A forgotten key there resets every stream rather than mixing two of them, so the rule above does not need to bind it.

For providers backed by a native SDK, the model and the adaptive state typically live in the same object (SpeexDSP's SpeexEchoState, RNNoise's DenoiseState, WebRTC's AudioProcessing). Per-stream state then means one native instance per stream; these SDKs offer no way to share the model alone. Implementations SHOULD create it lazily on the stream's first frame.

State that is not per stream stays on the instance: immutable configuration, process-level resources, and shared registries such as a diarization provider's enrolled-speaker set — enrolment belongs to the room, not to a speaker's current stream.

Session lifecycle contract:

Implementations MUST call reset(stream) on every configured stage when that stream ends, releasing its state. Omitting this leaks one speaker's worth of buffers — and, for native providers, C memory — for every stream a long-running room ever had.

Implementations MUST also call reset(stream) before the first audio frame of a new stream, so a reused key never inherits adaptive state (denoiser filter coefficients, VAD speech buffers, AGC gain history, diarization accumulators) from a previous stream with the same id.

close() remains global and MUST release every stream's state.

The stages this contract covers are the frame stages: resampler, AEC, AGC, denoiser, VAD, DTMF, diarization and postprocessors. The recorder is addressed by its handle (Section 12.3.7). The turn detector (Section 12.3.12) and the backchannel detector (Section 12.3.13) take no audio frame and no stream key; their reset() takes none either, and the session lifecycle does not call it.

12.3.2 Denoiser Provider

The denoiser removes background noise from inbound audio before VAD and STT process it. Running the denoiser first improves accuracy of all downstream stages.

DenoiserProvider (interface)
├── name: string                             # Provider name (e.g., "sherpa_onnx", "rnnoise")
├── process(audio_frame: AudioFrame, stream: string) → AudioFrame
│       # Process one frame of `stream`, return cleaned frame
├── reset(stream: string) → void
│       # Drop this stream's state when the stream ends
└── close() → void
        # Release resources

Implementations SHOULD provide at least one built-in denoiser (e.g., sherpa-onnx).

12.3.3 Resampler

The resampler normalizes audio format at pipeline entry (inbound) and exit (outbound). All pipeline stages operate on a consistent internal format, eliminating format mismatches between transport, STT, TTS, and VAD.

ResamplerConfig
├── internal_sample_rate: int = 16000       # Pipeline-internal sample rate (Hz)
├── internal_channels: int = 1              # Pipeline-internal channel count (mono)
├── internal_sample_width: int = 2          # Pipeline-internal bytes per sample (16-bit)
└── codec: string | null                    # Codec hint for transport negotiation (e.g., "opus", "pcm")

Behavior:

• Inbound: After receiving a raw AudioFrame from transport, the resampler converts it to the internal format before passing to downstream stages.
• Outbound: After postprocessors, the resampler converts from internal format to the transport's expected format (sample rate, channels, codec).

The resampler MUST preserve AudioFrame.timestamp_ms accurately through conversion. The resampler MUST add metadata.original_sample_rate and metadata.original_channels to the frame for audit purposes.

Implementations SHOULD use high-quality resampling (e.g., libsamplerate, sox-quality) to avoid introducing artifacts that degrade VAD and STT accuracy.

When no resampler is configured, all pipeline stages MUST accept the transport's native format. Implementations SHOULD log a warning if the transport format differs from the expected format of downstream stages.

12.3.4 Echo Canceller (AEC)

The echo canceller removes acoustic echo — the bot's own TTS audio picked up by the user's microphone. Without AEC, the VAD triggers on the bot's speech, causing false speech detections and feedback loops.

AECProvider (interface)
├── name: string                             # Provider name (e.g., "speex_aec", "webrtc_aec3")
├── process(audio_frame: AudioFrame, stream: string) → AudioFrame
│       # Remove echo from `stream`'s inbound frame using its buffered reference
├── feed_reference(audio_frame: AudioFrame, stream: string) → void
│       # Feed `stream`'s outbound audio as its echo reference
├── reset(stream: string) → void
│       # Drop this stream's state when the stream ends
└── close() → void
        # Release resources

Integration with outbound path:

The AEC requires a reference signal — the audio being played back to the user. The AEC provider manages the reference internally via feed_reference(), so process() operates on the inbound frame alone — this avoids callers having to track and pass the reference signal. The bidirectional dependency is:

Reference: Speaker output → AEC.feed_reference(frame, stream)
Inbound:   Transport → [resampler] → AEC.process(frame, stream) → [AGC] → ...

Both calls MUST carry the same stream key. Each stream owns its echo canceller, so an unkeyed reference could not reach the right one; in a conference every lane hears a different mix, and feeding one lane's output into another lane's canceller models an echo that never occurred.

Reference feeding strategies:

Implementations MUST call AECProvider.feed_reference() with the outbound audio and the stream key that audio is being played to. The point at which this call is made has a critical impact on echo cancellation quality. Two strategies are defined:

1. Transport-level feeding (RECOMMENDED for local hardware). When the transport has direct access to the playback hardware (e.g., a local speaker via PortAudio), the transport SHOULD feed reference audio from within the speaker output callback — at the exact moment audio is written to the DAC. This provides the best time alignment between reference and echo. The VoiceBackend accepts an optional AECProvider and feeds reference internally; the pipeline's outbound path does not call feed_reference().

2. Pipeline-level feeding (for network transports). When the transport sends audio over a network (WebRTC, SIP, WebSocket), the pipeline's outbound path calls AECProvider.feed_reference() after postprocessors and recorder:

TTS → [postprocessors] → [recorder] → AEC.feed_reference(frame, stream) → [resampler] → Transport

In this scenario the remote client's playback latency is handled by the AEC provider's internal ring buffer (e.g., SpeexDSP's split API).

Implementations MUST NOT feed reference from both levels simultaneously for the same AEC instance — this would double-feed and corrupt the adaptive filter.

Important considerations:

• AEC effectiveness depends on accurate time alignment between the reference signal and the echo in the inbound stream. Transport-level feeding provides the best alignment for local hardware; pipeline-level feeding is sufficient for network transports where latency is inherently variable.
• A transport feeding the reference from its speaker callback SHOULD keep one output stream open for the session, playing silence between responses. An output stream opened per response starts at a new render-to-capture delay each time, which the audio server may then take seconds to settle; an adaptive filter cannot follow a delay that moves under it (measured on a desktop audio server: a quarter of the echo left above -50 dBFS, against one twentieth with the delay held still).
• A transport running its own AEC SHOULD run it without pause for as long as it captures: through a response, through a cut, and on the silent reference between responses, feeding every played block and processing every captured frame, including frames it then drops (mute, gating, half-duplex). Paused and resumed, the canceller's render and capture timelines come back out of step and its converged filter misses the next response's echo (measured: up to 89 % of a response's first-second echo left above -50 dBFS, against none when it never pauses). With a silent reference it leaves the user's voice untouched.
• The same holds for a pipeline canceller whose reference the transport hands over block by block as it plays it, at the timing of strategy 1 but without a canceller of its own. A transport that reports every played block, the silence between responses included, SHOULD say so on each played block (metadata.reference_continues = true), and the pipeline SHOULD then keep its canceller active from the first response until the stream ends: the end of a response's playback (metadata.playback_ended) SHOULD NOT bypass it. Bypassed there, it drops the silent reference while capture runs on, and resumes at the next response with the timelines out of step (measured on a recorded session replayed through WebRTC AEC3: the next response's echo 0 to 5 dB below the microphone signal, against 34 to 44 dB kept active). A transport whose played blocks stop between responses leaves the canceller no reference to run on; the pipeline bypasses it once that playback has ended, keeping its converged filter for the next response.
• The reference and capture (inbound) audio MUST have the same sample rate and frame size. When the transport uses different sample rates for input and output, the implementation MUST either resample the reference to match the inbound rate, or require matching rates when AEC is enabled.
• When the VoiceBackend declares NATIVE_AEC capability, the pipeline MUST skip the AEC stage automatically, even if an AECProvider is configured, to avoid double processing.
• AEC MUST run before the denoiser — echo is a known signal to subtract, not random noise to filter.

AEC SHOULD add metadata.echo_cancelled = true to processed frames for observability.

12.3.5 Automatic Gain Control (AGC)

The AGC normalizes input audio volume to a consistent level. Users on different devices, at different distances from their microphone, or in different environments produce widely varying audio levels. AGC ensures that the VAD and STT receive audio at a predictable amplitude.

AGCProvider (interface)
├── name: string                             # Provider name (e.g., "webrtc_agc", "simple_agc")
├── process(audio_frame: AudioFrame, stream: string) → AudioFrame
│       # Apply gain control to `stream`, return normalized frame
├── reset(stream: string) → void
│       # Drop this stream's state when the stream ends
└── close() → void
        # Release resources

AGCConfig
├── target_level_dbfs: float = -3.0         # Target audio level in dBFS
├── max_gain_db: float = 30.0               # Maximum gain to apply
├── attack_ms: float = 10.0                 # Attack time (how fast gain increases)
└── release_ms: float = 100.0               # Release time (how fast gain decreases)

The AGC algorithm is well-standardized. Implementations MUST provide at least one built-in AGCProvider (e.g., based on WebRTC's AGC). Custom AGCProvider implementations MAY be registered for specialized requirements.

AGC MUST run after AEC (to avoid amplifying echo) and before the denoiser (to give the denoiser consistent input levels).

When the VoiceBackend declares NATIVE_AGC capability, the pipeline MUST skip the AGC stage automatically, even if an AGCProvider is configured.

AGC SHOULD add metadata.gain_applied_db to processed frames for observability.

12.3.6 DTMF Detector

The DTMF detector identifies telephone keypad tones (0-9, *, #, A-D) in the audio stream. This is essential for IVR (Interactive Voice Response) scenarios, call transfers, and PSTN/SIP integrations where users interact via keypad.

DTMFDetector (interface)
├── name: string                             # Provider name (e.g., "goertzel", "webrtc_dtmf")
├── process(audio_frame: AudioFrame, stream: string) → DTMFEvent | null
│       # Detect DTMF tone, return event if detected, null otherwise
├── reset(stream: string) → void
│       # Drop this stream's state when the stream ends
└── close() → void
        # Release resources

DTMFEvent
├── digit: string                            # Detected digit ("0"-"9", "*", "#", "A"-"D")
├── duration_ms: float                       # Tone duration
└── confidence: float | null                 # Detection confidence [0.0, 1.0]

Critical: DTMF runs in parallel, not in series.

DTMF tones are sinusoidal signals that would be destroyed or distorted by the denoiser, AGC, or AEC. The DTMF detector MUST process the frame the inbound resampler produced (the transport frame when no resampler runs) before AEC/AGC/denoiser stages, in parallel with the main pipeline:

Transport → [Resampler] → ┬─→ [AEC] → [AGC] → [Denoiser] → [VAD] → ...
                           │
                           └─→ [DTMF Detector] (parallel)

When a DTMF tone is detected, the framework MUST fire the ON_DTMF hook (Section 9.2). Implementations MAY optionally suppress the DTMF audio from the main pipeline to prevent the tone from being transcribed as speech.

Outbound DTMF. VoiceBackend exposes send_dtmf(session, digit, duration_ms) for sending DTMF digits to the remote party via RFC 4733 telephone-events. This is essential for AI agents navigating IVR menus, entering PINs, or interacting with phone systems programmatically. Valid digits are '0'-'9', '*', '#', and 'A'-'D'. The default duration is 160 ms. Only backends with the DTMF_SIGNALING capability support this method; others MUST raise NotImplementedError. VoiceChannel provides the public entry point with input validation (digit set, duration range 1–10000 ms, session state).

12.3.7 Audio Recorder

The audio recorder captures bidirectional audio for compliance, audit, quality assurance, and training purposes. In regulated industries (financial services, healthcare), call recording is often mandatory.

Scope. This interface records a VoiceSession, and everything about it says so: the handle names a session, and both the mode and the channel mode are expressed in terms of two directions. That is the shape of a call, not of a meeting. Recording a room's many participants is Section 12.11, and a conference records through that (Section 12.10.8) rather than through this.

AudioRecorder (interface)
├── name: string                             # Recorder name
├── start(session: VoiceSession, config: RecordingConfig) → RecordingHandle
│       # Start recording for a voice session
├── record_inbound(handle: RecordingHandle, frame: AudioFrame) → void
│       # Record an inbound (user) audio frame
├── record_outbound(handle: RecordingHandle, frame: AudioFrame) → void
│       # Record an outbound (bot/TTS) audio frame
├── stop(handle: RecordingHandle) → RecordingResult
│       # Stop recording, finalize file(s)
└── close() → void
        # Release resources

RecordingHandle
├── id: string                               # Unique handle identifier
├── session_id: string                       # Associated VoiceSession
├── started_at: datetime                     # When recording started
└── state: string                            # "recording" or "stopped"

The RecordingHandle is an opaque token returned by start() and passed to all subsequent recorder calls. Implementations MAY extend it with provider-specific fields.

RecordingConfig
├── format: string = "wav"                   # Output format ("wav", "mp3", "ogg", "flac")
├── mode: RecordingMode                      # What to record
├── channels: RecordingChannelMode           # How to mix audio
├── storage: string                          # Storage backend identifier (integrator-defined)
├── retention_days: int | null               # Auto-delete after N days (null = indefinite)
├── encryption: RecordingEncryption | null   # Encrypts finished files (Section 17.6)
├── storage_encrypted_at_rest: bool = false  # Integrator's statement: storage encrypts
└── metadata: map<string, any>               # Recording metadata (room_id, participant_id, etc.)

RecordingEncryption (interface)
├── name: string                             # Cipher/provider name
└── encrypt_file(path: string) → string      # Encrypt a finished file, leave no
                                             # plaintext; returns the artifact's path

A recorder that stores files MUST refuse to start when its configuration carries neither encryption nor storage_encrypted_at_rest = true (Section 17.6): an omitted security decision fails closed instead of producing a plaintext recording. With encryption, the recorder hands each finished file to encrypt_file() and reports the path it returns; a file the cipher cannot encrypt MUST be deleted rather than kept in the clear. storage_encrypted_at_rest is a deployment assertion the implementation cannot verify. The same two fields, with the same rule, configure the room media recorder (Section 12.11) and conference recording (Section 12.10.8).

The storage field is an integrator-defined identifier resolved by the implementation at runtime — similar to how provider names reference registered providers. Implementations MUST document which storage backends they support (e.g., "local", "s3", "gcs") and MUST raise a configuration error if an unknown identifier is provided.

RecordingMode enumeration:

Value	Description
INBOUND_ONLY	Record user audio only
OUTBOUND_ONLY	Record bot/TTS audio only
BOTH	Record both directions

RecordingChannelMode enumeration:

Value	Description
MIXED	Single file, both directions mixed
STEREO	Single file, user on left channel, bot on right channel
SEPARATE	Two separate files, one per direction

RecordingResult
├── recording_id: string                     # Unique recording identifier
├── urls: list<string>                       # Storage URLs for recording file(s)
├── duration_seconds: float                  # Total recording duration
├── format: string                           # Output format used
├── mode: RecordingChannelMode               # Channel mode used
└── metadata: map<string, any>               # Recording metadata

Recording position in the pipeline:

The recorder captures audio at two points: - Inbound: After resampler (raw user audio, consistent format) — before any processing, to preserve the original signal for compliance. - Outbound: After postprocessors (final bot audio) — what the user actually heard.

Implementations MAY provide a configuration option to record at different pipeline positions (e.g., after denoiser for cleaner recordings).

Inbound:  Transport → [Resampler] → recorder.record_inbound() → [AEC] → [AGC] → ...
Outbound: [PostProcessors] → recorder.record_outbound() → [Resampler] → Transport

When recording starts, the framework SHOULD fire ON_RECORDING_STARTED hook. When recording stops, the framework SHOULD fire ON_RECORDING_STOPPED hook with the RecordingResult.

12.3.8 VAD Provider

The VAD provider detects speech activity in audio frames and emits events that drive the voice pipeline. VAD events are the source for the ON_SPEECH_START, ON_SPEECH_END, ON_VAD_SILENCE, and ON_VAD_AUDIO_LEVEL hooks defined in Section 9.2.

Note: VAD is OPTIONAL in the AudioPipelineConfig schema to support speech-to-speech architectures where the AI provider handles turn detection. However, for Voice Channels (STT/TTS pipeline), VAD is effectively required — without it, the pipeline has no way to detect speech boundaries for STT.

VADProvider (interface)
├── name: string                             # Provider name (e.g., "silero", "ten_vad", "webrtc")
├── process(audio_frame: AudioFrame, stream: string) → VADEvent | null
│       # Process a frame, return event if state changed, null otherwise
├── configure(config: VADConfig) → void
│       # Apply the pipeline's vad_config when the pipeline is built (Section 12.3.1)
├── reset(stream: string) → void
│       # Drop this stream's state when the stream ends
└── close() → void
        # Release resources

VADEvent
├── type: VADEventType                       # What was detected
├── audio_bytes: bytes | null                # Captured speech audio (on SPEECH_END)
├── confidence: float | null                 # Detection confidence [0.0, 1.0]
├── duration_ms: float | null                # Duration of speech or silence
└── level_db: float | null                   # Audio level in dB (on AUDIO_LEVEL)

VADEventType enumeration:

Value	Description
SPEECH_START	Speech activity began
SPEECH_END	Speech activity ended — audio_bytes contains captured segment
SILENCE	Silence duration exceeded silence_threshold_ms
AUDIO_LEVEL	Periodic audio level report

Relationship with speech-to-speech providers:

When using a speech-to-speech provider (Section 12.4), the audio pipeline acts as a preprocessor — denoising audio before it reaches the provider and running diarization to identify speakers. Without a VAD, the provider manages its own turn detection. A configured VAD takes one of the two roles of Section 12.4: on a provider that takes turns it drives turn-taking (endpointing); on a full-duplex provider it observes (activity logging, audio level monitoring for UI) and turn-taking stays with the provider.

12.3.9 Diarization Provider

The diarization provider identifies which speaker produced each audio segment. This is particularly relevant in multi-participant voice rooms or when multiple speakers share a single audio stream (e.g., speakerphone).

DiarizationProvider (interface)
├── name: string                             # Provider name
├── process(audio_frame: AudioFrame, stream: string) → DiarizationResult | null
│       # Identify speaker, return result if determined, null otherwise
├── reset(stream: string) → void
│       # Drop this stream's state when the stream ends
└── close() → void
        # Release resources

DiarizationResult
├── speaker_id: string                       # Identified speaker label
├── confidence: float | null                 # Identification confidence [0.0, 1.0]
└── is_new_speaker: bool                     # True if this speaker was not previously seen

When diarization detects a speaker change, the framework MUST fire the ON_SPEAKER_CHANGE hook (Section 9.2). A result that attributes the audio to nobody — a speaker_id the label rule of Section 12.2.3 reads as unattributed (UU, PENDING, unknown in any case, or empty) — is not a speaker, as for an STT label: it neither fires the hook nor resets the last speaker, so a stretch the stage could not match between two stretches of one voice is no change. The result still reaches the frame, where a channel naming speakers from the stage counts it (Section 12.2.3, From the pipeline stage).

Speaker-to-Participant mapping: DiarizationResult.speaker_id is a provider-assigned label (e.g., "speaker_0", "speaker_1"), not a RoomKit Participant.id. Mapping diarization labels to participants is an integrator concern — typically resolved via the ON_SPEAKER_CHANGE hook, where the integrator can match speaker labels to known participants using voice enrollment, channel metadata, or heuristics (e.g., the participant who owns the voice channel is always "speaker_0"). A label from a diarizing STT follows the same rule, and Section 12.2.3 defines how it reaches the room.

12.3.10 Audio Post-Processor

Audio post-processors form an ordered chain on the outbound path. Each processor receives an audio frame and returns a (possibly modified) frame. Common use cases include volume normalization, audio watermarking (for compliance), and recording.

AudioPostProcessor (interface)
├── name: string                             # Processor name
├── process(audio_frame: AudioFrame, stream: string) → AudioFrame
│       # Process `stream`'s outbound audio frame
├── reset(stream: string) → void
│       # Drop this stream's state when the stream ends
└── close() → void
        # Release resources

Multiple post-processors are executed in the order they appear in the postprocessors list. Each processor receives the output of the previous one.

12.3.11 Audio Frame

The common data structure passed through the pipeline:

AudioFrame
├── data: bytes                              # Raw audio samples
├── sample_rate: int                         # Sample rate in Hz (e.g., 16000, 48000)
├── channels: int                            # Number of audio channels (1 = mono, 2 = stereo)
├── sample_width: int                        # Bytes per sample (2 = 16-bit)
├── timestamp_ms: float | null               # Frame timestamp relative to session start
└── metadata: map<string, any>               # Pipeline metadata (accumulated by stages)

AudioChunk is a distinct type, not an alias:

AudioChunk
├── data: bytes                              # Audio samples
├── sample_rate: int                         # Sample rate in Hz
├── channels: int                            # Number of audio channels
├── format: string                           # Sample encoding (default "pcm_s16le")
├── timestamp_ms: int | null                 # Chunk timestamp
└── is_final: bool                           # Last chunk of the stream

The two exist because inbound and outbound audio are not the same problem. Inbound audio is a stream to be analysed: an AudioFrame is a standalone unit that accumulates stage annotations in metadata as it descends the pipeline — the VAD records that speech was present, diarization records which speaker — and it is validated on construction, because a misaligned frame breaks the stages that consume it.

Outbound audio is a stream to be finished: TTS emits a sequence of chunks and must be able to say which one is last, so the transport knows when an utterance has ended, can drain its buffers, and can hand control back to barge-in handling. is_final carries that, and AudioFrame has no equivalent because a live microphone has no last frame. Nothing annotates an outbound chunk, so it carries no metadata and no validation.

Implementations MUST NOT merge the two. Doing so means either dragging is_final and format into the pipeline type, where they are meaningless, or losing stream termination on the TTS path.

Conversion is one-directional and lossy by design: an AudioFrame becomes an AudioChunk by dropping its pipeline metadata; the reverse requires validating alignment and starting a fresh annotation record.

12.3.12 Turn Detector

The turn detector determines whether the user has finished their conversational turn. This is distinct from VAD, which detects acoustic silence. A user may pause mid-sentence (VAD triggers silence) but not be done speaking. Conversely, a user may finish a question without a long pause.

Note: Unlike the frame-level stages above (denoiser, AEC, AGC, VAD), the turn detector operates on transcripts (text from STT), not audio frames. It does not implement process(AudioFrame). It sits between STT output and the room event pipeline, making it a post-STT stage rather than an audio-frame processor. It is included in the audio pipeline configuration for convenience, as it is integral to the voice processing flow.

TurnDetector (interface)
├── name: string                             # Detector name (e.g., "llm_turn", "heuristic", "hybrid")
├── evaluate(context: TurnContext) → TurnDecision
│       # Evaluate whether the user's turn is complete
├── reset() → void
│       # Reset internal state
└── close() → void
        # Release resources

TurnContext
├── transcript: string                       # Current accumulated transcript (from STT)
├── is_final: bool                           # Whether STT transcript is final
├── silence_duration_ms: float               # Current silence duration (from VAD)
├── speech_duration_ms: float                # Duration of the speech segment
├── conversation_history: list<TurnEntry> | null  # Recent turns for context
└── metadata: map<string, any>               # Additional context

TurnEntry
├── role: string                             # "user" or "assistant"
└── text: string                             # Transcript text of the turn

TurnDecision
├── is_complete: bool                        # Whether the turn is considered complete
├── confidence: float                        # Decision confidence [0.0, 1.0]
├── reason: string | null                    # Why (e.g., "question_mark", "long_pause", "semantic_complete")
└── suggested_wait_ms: float | null          # If not complete, how long to wait before re-evaluating

Turn detection modes:

Mode	Description	Latency	Accuracy
VAD-only	Use VAD silence threshold (current behavior)	Low	Low — triggers on pauses
Heuristic	Punctuation, sentence structure, silence combo	Low	Medium
LLM-based	Send transcript to fast LLM for completion check	Medium	High
Hybrid	VAD for fast path, LLM for ambiguous cases	Adaptive	High

Integration point:

The turn detector sits between STT and the room event pipeline. It replaces the simple "VAD silence → send to STT → route event" flow with a more nuanced path:

VAD SPEECH_END → STT transcribes → TurnDetector.evaluate(context)
                                          │
                                    is_complete?
                                    ├── YES → Create RoomEvent, route normally
                                    └── NO  → Wait (suggested_wait_ms), accumulate
                                              next speech, re-evaluate

Waiting on an incomplete turn: when is_complete is false, the channel MUST NOT hold the turn indefinitely. It waits for suggested_wait_ms of silence, or a configured default when the decision gives none. The wait counts silence, not time: speech that starts during it — including speech that started while the turn was still being judged — keeps the turn open, and silence counts again from the end of that speech. A transcript from that speech joins the accumulated turn, which is then evaluated again; speech that yields no transcript does not hold the turn. That transcript may still be on its way when the silence has lasted the wait (a streaming STT finalizes after the speech, sometimes for longer than the wait): the wait MUST NOT end before the transcript of speech that has ended has joined the turn or turned out empty, so the turn is routed with it. An implementation MAY bound this extra wait, so that a transcription that never ends does not hold the turn indefinitely. Once the wait ends in silence, the accumulated turn MUST be routed as complete, and its completion event carries confidence = 0.0. A user who pauses on a sentence the detector judged unfinished is answered, never ignored.

Speech that resumes before the turn is routed: a turn the detector judges complete MUST NOT be routed while speech that started after the evaluated segment is in progress. The channel holds it as it would an incomplete turn: the new speech joins it and the accumulated turn is evaluated again, or routed once the wait ends in silence. A user who resumes a sentence after a short pause is answered once, on the whole sentence.

Speech that resumes after the turn is routed, before its response is heard: this applies with or without a TurnDetector. From the routing of a turn until the first audio of its response reaches the transport, the response is unheard. Speech that starts while a session's response is unheard, and while no other audio is audible on the session, MUST hold that response: no audio of it is sent while the speech lasts, and the speech is processed as a user turn, not evaluated as a barge-in (nothing plays, so there is no echo to guard against). Speech over audible audio keeps its usual echo guard. Each speech segment is measured from its own onset. When the speech ends:

• if it lasted at least min_speech_ms and yields a transcript that is routed, the unheard turn MUST be cancelled with reason superseded before the new transcript is routed. The committed user message stays stored; the new transcript is routed on its own, and the model sees both messages in turn;
• otherwise (shorter, no transcript, blocked by an ON_TRANSCRIPTION hook, or discarded as echo) the held response is released and plays as it would have, unless new speech has started meanwhile, which holds it in turn.

A response cancelled as superseded was never heard, so it is not part of the conversation, and its consumption on the voice channel is unconsumed (Section 8.5). Its text may already be stored as finished (generation often ends before playback starts): the channel marks every MESSAGE response event of the turn metadata.cancelled = true and metadata.cancellation_reason = "superseded", and an intelligence channel MUST NOT replay such an event as history. A user who resumes a sentence after the turn was sent is answered once, not twice.

When no TurnDetector is configured, the pipeline falls back to VAD-only behavior (current default).

The turn detector MUST NOT add more than 200ms latency in the fast path. For LLM-based detection, implementations SHOULD use a small, fast model or cache common patterns.

12.3.13 Interruption Strategy

When the user speaks while the bot is responding (barge-in), the framework needs a configurable strategy to handle the interruption. Not all speech during bot playback is a genuine interruption — backchannels ("mmhmm", "ok", "yes") are acknowledgments, not requests to stop.

InterruptionConfig
├── strategy: InterruptionStrategy = CONFIRMED  # How to handle barge-in
├── min_speech_ms: int = 300                 # Minimum user speech duration to trigger interruption
├── allow_during_first_ms: int = 0           # Response audio (ms) played before barge-in is allowed (0 = no gate)
├── backchannel_detector: BackchannelDetector | null  # OPTIONAL backchannel filter
├── transcript_wait_ms: int = 1000           # SEMANTIC: how long to wait for the first words of transcribed speech
├── echo_guard_ms: int = 6000                # Session response audio (ms) during which speech echoing the response is no interruption (0 = off)
├── flush_partial_tts: bool = true           # Whether to discard unplayed TTS audio on interrupt
└── keep_partial_transcript: bool = true     # Whether to store bot's partial response in timeline

InterruptionStrategy enumeration:

Value	Description
IMMEDIATE	Cancel TTS as soon as VAD detects speech (v1 behavior). Fast but aggressive.
CONFIRMED	Wait for min_speech_ms of sustained speech before cancelling. Tolerates brief sounds. Default.
SEMANTIC	Use backchannel detector to decide — only interrupt on non-backchannel speech. Most natural.
DISABLED	Never interrupt — user speech is queued until bot finishes. For non-interactive playback.

BackchannelDetector interface:

BackchannelDetector (interface)
├── name: string                             # Detector name
├── classify(context: BackchannelContext) → BackchannelDecision
│       # Classify whether speech is a backchannel or genuine interruption
├── reset() → void
│       # Reset internal state (the session lifecycle does not call it)
└── close() → void
        # Release resources

BackchannelContext
├── audio_bytes: bytes | null                # Short audio segment (if available)
├── transcript: string | null                # Partial transcript of the speech (if STT fast enough)
├── speech_duration_ms: float                # How long the user has been speaking
├── bot_speech_progress: float               # How far into the bot's response (0.0 to 1.0)
└── metadata: map<string, any>               # Additional context

BackchannelDecision
├── is_backchannel: bool                     # True = acknowledgment, False = real interruption
├── confidence: float                        # Decision confidence [0.0, 1.0]
└── label: string | null                     # Classification label (e.g., "agreement", "filler", "question")

Timing constraint: In the SEMANTIC strategy, the BackchannelContext.transcript field may be null or incomplete if STT has not yet produced a result for the speech segment. In practice, the classifier will often operate on audio_bytes and speech_duration_ms alone, falling back to transcript-based classification only when streaming STT provides partial results fast enough. Implementations SHOULD design backchannel detectors to work with audio features alone and treat transcript availability as a bonus signal.

An implementation MUST NOT consult the detector at speech onset with neither a transcript nor any speech duration: no detector can tell an acknowledgement from an interruption on an empty utterance, and a transcript-based one would turn SEMANTIC into IMMEDIATE. It waits instead, and the speech is held (not processed as a user turn) meanwhile:

• When a streaming STT is transcribing the speech, it waits for the first partial transcript, for at most max(min_speech_ms, transcript_wait_ms) of sustained speech. A streaming STT typically needs longer than min_speech_ms for its first words, and judging before them classifies an empty utterance.
• Otherwise it waits for min_speech_ms of sustained speech.

Once the wait ends without words, the classification relies on speech_duration_ms (and audio_bytes) alone. A detector MAY instead judge an utterance without words a backchannel, so that only words interrupt: sound the STT made nothing of (echo an AEC left, a cough, room noise) then never cuts the bot, and a wordless interruption does not either. It is a configuration of the detector, not its default. When a streaming STT is available, the held speech SHOULD be transcribed during playback so its partials can be classified; a backchannel then fires ON_BACKCHANNEL once and the speech is discarded, a genuine interruption cancels TTS and the speech is processed as the user's turn.

The held speech may end before its first word and before the wait ends: a streaming STT often releases a short word ("okay", "stop") only when the speech is over. The streaming STT SHOULD then be finalized and its final transcript classified, rather than the speech discarded unheard. A backchannel, or no words at all, is discarded (a backchannel fires ON_BACKCHANNEL once) and TTS continues; anything else cancels TTS if it is still playing, and the speech is processed as the user's turn.

Echo guard. An echo canceller learns the room from the responses it hears: at a session's start it has heard none, and a response's opening words come back through the microphone, where an STT transcribes them as if someone had spoken. While the session has played less than echo_guard_ms of response audio in all, whatever the strategy, speech whose words are only words of the response being played (the text handed to the TTS so far), consecutive and in their order, is that response's echo. Words are compared without case, punctuation or accents. In a partial transcript the last word heard may be the start of the response's, as a partial catches a word being said, but only after a whole word: a lone "No" is no echo of "not", nor "Stop" of "stopped". A final transcript is compared in whole words. An echo does not interrupt, is not queued (DISABLED), and is no turn: its partial transcripts are not published (ON_PARTIAL_TRANSCRIPTION), and its final transcript is discarded, including when it arrives once the response is over, as long as it adds no word the response did not say there. The verdict on an utterance holds while its response plays and up to 10 seconds after the session's playback ends; it ends with its utterance (a final, the end of the STT stream) and with any speech that starts once nothing plays, which is nobody's echo. While it holds, the utterance's words decide nothing else: no barge-in on the response that cut the echoed one short, or on the next, is decided on them. When it lapses with its utterance still open (an STT that never closed it), the words go with it. A word that is not the response's makes the speech an ordinary utterance, judged as any other: when only the final says it, during the playback, the final is judged then, as a whole utterance past any minimum duration, and when the strategy interrupts, the playback is cut and the final is the turn. While the guard is open, a stretch of loud audio with no words yet waits for the STT's first words, up to transcript_wait_ms, before it may interrupt on its duration: only words tell the echo from the person. Audio played from a file (VoiceChannel.play()) counts toward the session's played audio as a response's does. Past echo_guard_ms, the response's own words interrupt as any words do: someone answering "the sword" to "the sword or the shield?" is not an echo. echo_guard_ms = 0 turns the guard off. With an STT that tells voices apart (Section 12.2.3), whatever echo_guard_ms, a voice heard speaking while nothing played is a person's; a final from a voice never heard so, said over a response, whose words are that response's in their order, one misheard at most and three at least heard right, is that response come back past the canceller and taken for another speaker: it is discarded, no turn. The barge-in its first words decided stands: the voice is only known with the final. The rule has a cost, accepted: a person not yet heard while nothing played who answers over a response with three or more of its words in their order ("the kitchen lamp" to "the kitchen lamp or the living room lamp?") cuts the response and is not heard, so the agent stops and says nothing; what they say next while nothing plays is heard. A voice is known within one STT stream: when the provider closes a kept stream, the voices heard alone are forgotten, as the next stream numbers its voices afresh.

In continuous-STT mode, where the STT transcribes all audio and no local VAD segments it, every evaluation of an ongoing stretch of speech during playback, whichever signal triggered it (a partial transcript, or an energy or transport barge-in), classifies the latest words of that stretch. A stretch classified as a backchannel fires ON_BACKCHANNEL once and is not cut on duration alone afterwards; later words that are not a backchannel still interrupt.

Interruption flow:

User speaks during TTS playback
      │
      ▼
VoiceBackend fires on_barge_in
      │
      ▼
Check InterruptionStrategy:
      │
      ├── IMMEDIATE → Cancel TTS immediately
      │
      ├── CONFIRMED → Start timer
      │   ├── Speech continues > min_speech_ms → Cancel TTS
      │   └── Speech stops before threshold → Ignore, resume TTS
      │
      ├── SEMANTIC → Run BackchannelDetector
      │   ├── is_backchannel = true → Ignore, resume TTS
      │   │   └── Fire ON_BACKCHANNEL hook (async)
      │   └── is_backchannel = false → Cancel TTS
      │       └── Fire ON_BARGE_IN hook (async)
      │
      └── DISABLED → Ignore speech, queue for after TTS completes

When TTS is cancelled:

1. If flush_partial_tts = true: discard all unplayed audio in the TTS buffer.
2. If keep_partial_transcript = true: store the bot's partial response in the timeline with metadata.interrupted = true and metadata.played_ms (how far playback got). metadata.played_percentage is added only when the utterance's total duration is known; a streamed response (Section 12.2, step 12s) has none, so it carries played_ms alone. For a streamed response, the stored text is the sentences already handed to TTS at the moment of the interruption. The response is recorded partial on the voice channel, with the same played_ms (Section 8.5), whether or not the partial response is kept. The record is the agent's words, not the listener's: it carries no participant id, and names the answer it cut with metadata.answer_channel_id (the intelligence channel that produced it) and metadata.answer_responds_to (the event that answer responds to, Section 8.5), so the AI channel can mark that answer as cut (Section 6.4).
3. If the TTS consumes context: record the assistant turn with interrupted = true and the same played_ms (Section 12.2.2), so the provider knows where the user stopped hearing it.
4. Process the user's speech normally through the inbound pipeline.

12.3.14 Pipeline Execution Flow

Session start (once per VoiceSession):

1. VoiceSession transitions to ACTIVE
2. Call reset(stream) on every configured frame stage (Section 12.3.1), the
   session id being the stream key
3. Backend fires on_session_ready callback when audio path is live
   └── VoiceChannel fires ON_SESSION_STARTED hook (dual-signal: requires
       both bind_session() and backend ready, in either order)
4. IF recorder AND recording_config configured:
   ├── handle = recorder.start(session, recording_config)
   └── Fire ON_RECORDING_STARTED hook

Session end (once per VoiceSession):

1. VoiceSession transitions to ENDED
2. Call reset(stream) on every configured frame stage (Section 12.3.1), the
   session id being the stream key
3. IF recorder configured AND recording active:
   ├── result = recorder.stop(handle)
   └── Fire ON_RECORDING_STOPPED hook with RecordingResult
4. Call close() on pipeline stages only if the channel is being destroyed
   (not on session end — stages are reused across sessions)

Inbound flow (per audio frame):

1. Transport emits raw AudioFrame for stream `stream` (session id, or lane id
   in a conference)
2. IF resampler configured:
   └── frame = resample(frame, internal_format)
       └── frame.metadata.original_sample_rate = original_rate
3. IF recorder configured:
   └── recorder.record_inbound(handle, frame)
4. IF dtmf configured (PARALLEL with steps 5-9):
   ├── dtmf_event = dtmf.process(frame, stream)
   └── IF dtmf_event is not null:
       └── Fire ON_DTMF hook
5. IF aec configured AND NOT backend.NATIVE_AEC:
   └── frame = aec.process(frame, stream)
6. IF agc configured AND NOT backend.NATIVE_AGC:
   └── frame = agc.process(frame, stream)
7. IF denoiser configured:
   └── frame = denoiser.process(frame, stream)
8. IF vad configured:
   ├── vad_event = vad.process(frame, stream)
   └── IF vad_event is not null:
       ├── Fire corresponding hook (ON_SPEECH_START, ON_SPEECH_END, etc.)
       └── IF vad_event.type == SPEECH_END:
           ├── STT transcribes audio_bytes → TranscriptionResult
           ├── Fire ON_TRANSCRIPTION hook (can modify transcript)
           └── IF turn_detector configured:
               ├── decision = turn_detector.evaluate(context)
               ├── IF decision.is_complete:
               │   ├── Fire ON_TURN_COMPLETE hook
               │   └── Create RoomEvent, route to Room
               └── ELSE:
                   ├── Fire ON_TURN_INCOMPLETE hook
                   └── Accumulate, wait for next speech segment
           └── ELSE (no turn detector):
               └── Create RoomEvent, route to Room (v1 behavior)
9. IF diarization configured:
   ├── result = diarization.process(frame, stream)
   └── IF result names a speaker (not unattributed, Section 12.3.9)
          AND it differs from the stream's last one:
       └── Fire ON_SPEAKER_CHANGE hook

10. WHEN the stream ends:
   └── FOR EACH configured frame stage: stage.reset(stream)
       (Releases that speaker's state. Skipping it leaks one stream's buffers —
        and native memory for SDK-backed stages — per speaker the room ever had.)

Outbound flow (per audio frame / chunk):

1. TTS emits AudioChunk stream (variable-size); speech-to-speech emits AudioFrame
   (Stages that require fixed-size AudioFrames MUST buffer and re-chunk internally)
2. FOR EACH postprocessor in order:
   └── frame = postprocessor.process(frame, stream)
3. IF recorder configured:
   └── recorder.record_outbound(handle, frame)
4. IF aec configured AND transport does NOT handle AEC reference feeding:
   └── aec.feed_reference(frame, stream)
   (When the transport feeds reference from its speaker output callback — e.g.,
    local audio hardware — the pipeline MUST skip this step to avoid double-feeding.
    See Section 12.3.4 for reference feeding strategies.)
5. IF resampler configured:
   └── frame = resample(frame, transport_format)
6. Transport sends processed frame to client
   (For local hardware transports, the speaker output callback feeds
    aec.feed_reference() here with that stream's key, time-aligned with actual
    playback.)

12.3.15 Pipeline Debug Taps

Pipeline Debug Taps provide lightweight diagnostic audio capture at every stage boundary in the processing pipeline. Unlike the production AudioRecorder (Section 12.3.7) — which captures raw audio for compliance and audit — debug taps capture audio at every processing stage, allowing developers to compare the signal before and after each transformation.

This is invaluable for debugging audio quality issues: "Is the denoiser removing too much signal?", "What does the VAD actually hear?", "Is AEC effective?". Without debug taps, these questions require custom instrumentation.

Configuration:

PipelineDebugTaps
├── output_dir: string                      # Directory for debug WAV files (REQUIRED)
├── stages: list<string> | "all" = "all"    # Which stages to capture
├── session_scoped: bool = true             # Prefix files with session ID + timestamp
└── sample_rate: int | null                 # Override sample rate for output files (null = use internal format)

When stages is "all", taps are inserted at every stage boundary. When a list is provided, only the named stages are captured. Valid stage names:

Stage name	Capture point	What it reveals
transport_raw	In a transport that cancels echo itself, before its AEC	What the microphone captured
raw	After resampler, before AEC	What the pipeline receives from transport (after the transport's own AEC, when it has one)
post_aec	After AEC	Echo cancellation effectiveness
post_agc	After AGC	Volume normalization result
post_denoiser	After denoiser	Noise reduction quality — what VAD sees
post_vad_speech	On SPEECH_END event	Accumulated speech audio bytes sent to STT
outbound_raw	Before postprocessors	TTS output before processing
outbound_final	After postprocessors, before resampler	Final audio sent to transport
aec_reference	In a transport that cancels echo itself, beside transport_raw	The reference its AEC received while that frame was captured

Output files:

Files are named with a numeric prefix reflecting pipeline order, making it easy to compare stages side by side in any audio editor:

{output_dir}/
  {session_id}_00_transport_raw.wav
  {session_id}_01_raw.wav
  {session_id}_02_post_aec.wav
  {session_id}_03_post_agc.wav
  {session_id}_04_post_denoiser.wav
  {session_id}_05_post_vad_speech.wav
  {session_id}_06_outbound_raw.wav
  {session_id}_07_outbound_final.wav
  {session_id}_08_aec_reference.wav

When session_scoped is false, files omit the session prefix (useful for quick single-session debugging).

A transport that cancels echo itself (NATIVE_AEC, transport-level reference feeding, Section 12.3.4) runs its AEC before the pipeline: raw is then already echo-cancelled and post_aec equals it. Such a transport emits transport_raw and aec_reference for every captured frame, from the capture path: the frame as captured, and the reference its AEC received meanwhile (as played, at the captured frame's rate), silence where it received none, so the frame's duration is kept. Both are then aligned sample for sample with raw, which makes the transport's echo cancellation measurable, and replayable offline. Writing them MUST NOT happen on the audio I/O threads: a file write there delays playback and capture.

Files are opened lazily on first write and closed on session end. Each file uses the pipeline's internal AudioFormat (from the AudioPipelineContract), unless sample_rate is overridden.

Integration with pipeline config:

AudioPipelineConfig
├── ... (existing fields)
└── debug_taps: PipelineDebugTaps | null    # OPTIONAL diagnostic capture (default: null)

Behavior:

• Debug taps MUST NOT modify audio frames — they are read-only observers. Tap processing is a non-blocking copy of the frame data.
• Tap writes SHOULD be non-blocking. Implementations MAY buffer frames and flush to disk asynchronously to avoid adding latency to the audio pipeline.
• When debug_taps is null, the pipeline MUST NOT perform any tap-related processing (zero overhead when disabled).
• The post_vad_speech tap captures the vad_event.audio_bytes blob — the accumulated speech segment that would be sent to STT. This is written as a separate WAV file per speech segment (appending a segment counter: {session_id}_05_post_vad_speech_001.wav, etc.).
• Implementations SHOULD log a warning at startup when debug taps are enabled, as the disk I/O and storage cost is not intended for production.

Session lifecycle:

1. On session_active:
   ├── Create output_dir if it does not exist
   └── Initialize per-stage WAV writers (lazily, on first frame)

2. On each inbound/outbound frame:
   └── IF stage is in configured stages:
       └── writer.write(frame.data)  # non-blocking copy

3. On SPEECH_END event (for post_vad_speech stage):
   └── Write audio_bytes to a new segment file

4. On session_ended:
   ├── Flush and close all WAV writers
   └── Finalize WAV headers with correct data sizes

Relationship to AudioRecorder:

Pipeline Debug Taps and AudioRecorder serve different purposes and MAY be used simultaneously:

	AudioRecorder	PipelineDebugTaps
Purpose	Compliance, audit, QA	Development and debugging
Capture points	Raw inbound + final outbound	Every stage boundary
Production use	Yes	No (SHOULD warn)
Modifies frames	No	No
Output	Single/stereo recording file	Multiple per-stage WAV files
Configuration	recorder + recording_config	debug_taps

12.4 Realtime Voice Channel (Speech-to-Speech)

The Realtime Voice Channel wraps speech-to-speech APIs (e.g., OpenAI Realtime, Gemini Live) that handle audio processing natively, bypassing STT/TTS.

RealtimeVoiceProvider interface:

RealtimeVoiceProvider (interface)
├── name → string                           # Provider identifier
├── model_name → string                     # Model behind the session; defaults to name
├── full_duplex: bool (default false)       # Model listens and speaks at once; no boundaries on the wire (Section 12.4.1)
├── supports_tools: bool (default true)     # The model can call tools; false declares none to it
├── supports_mid_session_reconfigure: bool (default true)
│       # Whether reconfigure() can run mid-session; false when it would
│       # replace the session or lose its state
├── supports_context_preservation: bool (default false)
│       # Whether provider_config {"preserve_context": true} is supported:
│       # the provider keeps tool-delivered instructions verbatim for the
│       # session, or ends it with an error before going on without them
├── available_voices() → VoiceInfo[]        # Curated offline catalog (VoiceInfo: Section 12.2)
├── list_voices() → VoiceInfo[]             # Live catalog; default: available_voices()
├── available_models() → ModelInfo[]        # Curated offline catalog (MAY be empty)
├── connect(session, system_prompt, voice, tools, temperature,
│           input_sample_rate = 16000, output_sample_rate = 24000,
│           server_vad = true, provider_config) → void
│       # server_vad false: the channel signals turns (endpointing role)
├── disconnect(session) → void
├── reconfigure(session, system_prompt, voice, tools, temperature,
│               provider_config) → void
│       # New settings mid-session; default: disconnect, then connect
│       # again with them
├── send_audio(session, audio_chunk) → void
├── start_audio_stream(session) → void
│       # Open the provider's audio input path before audio flows (a
│       # greeting before the caller speaks); no-op by default
├── inject_text(session, text, role = "user", silent = false)
│           → VoiceInjectionResult | null
│       # Insert text into the conversation; role is an intent ("system" or
│       # "user", below); silent adds it without requesting a response. The
│       # result's status is sent, not_sent or unknown, with a reason and
│       # whether a retry is allowed (Section 22.1); null reads as unknown
├── inject_image(session, image_data, mime_type = "image/png", prompt = "",
│                silent = false) → void
│       # Add an image for multimodal analysis; default: fails as unsupported
├── submit_tool_result(session, call_id, result) → void  # Return tool result to provider
├── submit_tool_error(session, call_id, result) → void   # Return a failed call's result; default: submit_tool_result
├── submit_delegation_output(session, delegation_id, text, spoken) → void  # Return a reasoning backend's output (Section 12.4.1)
├── interrupt(session) → void               # Signal user interruption to provider
├── truncate_audio(session, audio_end_ms) → void  # OPTIONAL: drop the unheard tail from provider context after an interruption; no-op by default
├── send_activity_start(session) → void     # Endpointing role: the user started speaking; no-op by default
├── send_activity_end(session) → void       # Endpointing role: the user stopped speaking; no-op by default
├── send_event(session, event) → void
│       # Send a raw provider-specific event; default: fails as unsupported
├── is_responding(session_id) → bool        # Whether a response is being generated; default false
├── close() → void                          # Release all resources
│
│   # Callback registration:
├── on_audio(callback) → void               # AI-generated audio
├── on_transcription(callback) → void       # User/AI speech transcript
├── on_speech_start(callback) → void
├── on_speech_end(callback) → void
├── on_tool_call(callback) → void           # AI requests a tool call
├── on_tool_call_cancelled(callback) → void # (session, call_ids): calls whose results the model will not read; fired for every call the provider abandons (below)
├── on_delegation(callback) → void          # (session, delegation_id, target): model handed reasoning to a backend (Section 12.4.1)
├── on_response_start(callback) → void
├── on_response_end(callback) → void
├── on_usage(callback) → void              # (session, usage): what the provider just recorded (Section 12.4.2)
└── on_error(callback) → void

What a provider owes its tool calls (normative). A call whose arguments read as an object reaches on_tool_call as that mapping; a call whose arguments do not (invalid JSON, an array, a fragment) reaches it as the text the model wrote, never as a mapping that could pass for arguments. The channel refuses such a call before the gate below, as a tool loop refuses a partial call (Section 6.4): nothing runs, the model reads that its arguments could not be read, and the observers receive the refusal. A call the provider issued is answered once, by the channel, through submit_tool_result, or through submit_tool_error when the call failed (refused, failed, blocked, served by nothing): a provider whose protocol marks a result as an error (ElevenLabs' client tools, Gemini Live's error key) MUST mark it, so the model does not read a refusal as a success, and MUST mark nothing else, so a served result whose body carries an error field of its own is not read as a failure; a provider whose protocol does not ignores the distinction, the result's body saying it. A provider MUST report through on_tool_call_cancelled every call it abandons, whatever the cause: the model discarding it, a reconnect orphaning it (call ids are connection-scoped), its own wait on the channel timing out, the conversation ending. A call it abandons without that report runs on unseen, and the observers record a success the model never read. The tasks a session lives on (its receive loop, its keepalive) MUST run in a context of their own: a connection opened inside a tool handler (a handoff reconnecting the session) would otherwise carry that call's context into every event of the new connection. A provider whose model cannot call tools says so (supports_tools false); the channel then declares none to it and warns once, rather than leave every call unanswerable. A provider whose service ends a turn on a function call it could not parse (Gemini Live's MALFORMED_FUNCTION_CALL) MUST tell the model that the call did not run, since nothing else will, once until the user speaks again, so a model that keeps failing does not loop on the reminder.

RealtimeVoiceProvider callback → hook mapping:

Provider Callback	Hook Fired	Notes
on_audio	—	Audio frames routed to transport; no hook (too high frequency)
on_transcription	ON_TRANSCRIPTION	User and AI transcripts emitted as RoomEvents
on_speech_start	ON_SPEECH_START	Provider-detected speech start
on_speech_end	ON_SPEECH_END	Provider-detected speech end
on_tool_call	ON_TOOL_CALL	Tool execution request from AI, behind the pre-execution gate below (ON_REALTIME_TOOL_CALL is superseded, Section 9.2)
on_tool_call_cancelled	ON_TOOL_CALL	Model abandoned outstanding calls; the channel interrupts their handlers, sends nothing back, and reports them to the observers with the cancelled marker, except a call whose own handler caused the reconnect (Section 9.3)
on_delegation	ON_REALTIME_DELEGATION	Model handed reasoning to a backend, hosted or integrator; the integrator target is served by the ReasoningBackend (Section 12.4.1)
on_response_start	—	Internal lifecycle; no hook (use AFTER_BROADCAST for response tracking)
on_response_end	—	Internal lifecycle; no hook (use AFTER_BROADCAST for response tracking)
on_usage	—	The provider's own usage report, relayed unaltered; no hook (an integrator that bills a call reads it here, Section 12.4.2)
on_error	ON_ERROR	Mapped to the global ON_ERROR hook (Section 9.2)

Note: on_response_start and on_response_end are internal provider lifecycle callbacks used for audio routing and session bookkeeping. They do not map to hooks because they don't represent events the integrator needs to act on. Integrators who need response-level tracking SHOULD use AFTER_BROADCAST on the transcription events emitted by the provider. A full-duplex provider has no response boundaries on its wire and synthesizes them (Section 12.4.1).

Session configuration. A session starts with the configuration it was opened with (its own prompt, voice or tools), else the one set for its room (the active agent of a pipeline installed there, Section 19.5), else the channel's. Reconfiguring a session (a handoff, an application changing one call's instructions) changes that session only: the channel's configuration for future sessions, and the sessions of other rooms, MUST NOT change with it, since one channel serves every room it is attached to (Section 19.7). The channel's configuration changes through the channel's own configuration call, never as a side effect of a session's.

Tool policy: A realtime channel MAY carry a ToolPolicy (Section 21.1), with the same meaning as an AI channel's and the same exempt names. It filters the tools declared to the session, at connection and at every reconfiguration (a Tool Search reveal and a skill activation included), the catalogue a reasoning backend receives (Section 12.4.1), and what Tool Search names; and it guards every call at the gate below, whichever entry brought it: the provider's own call, a call recovered from spoken text, a reasoning backend's call. A role override applies to the session's participant, with the role the store holds for it when the session's tools are first declared and again at each call: a role changed during the session MUST hold at the gate from the next call on, even while the declared tools still reflect the former one. A participant the store does not hold has no role, and the base rules apply to it. A policy that held on the provider's calls and not on a backend's would be a door around it.

Tool call pre-execution gate:

Before a realtime tool call is routed to whatever serves it, the channel MUST apply, in this order: the declared-tool check (a name absent from a non-empty declared catalogue is refused), the channel's ToolPolicy when it has one (above), skill gating (Section 24.2), argument validation against the declared schema, and the BEFORE_TOOL_USE hook (Section 9.2). A tool the policy denies or a skill keeps closed is refused before its arguments are read, so its refusal never names what its schema requires; each refusal reads its cause's text (Section 21.1).

The gate's scope is the call, not its servant. It MUST apply identically whether the call is served by a tool handler, by an ON_TOOL_CALL hook, or by a channel infrastructure tool (Tool Search, skill activation) — otherwise a hook that denies a tool would deny it only under some configurations, and a host auditing tool use would not see every call.

A call the gate refuses MUST still be reported through ON_TOOL_CALL's observers (Section 9.3), and the report MUST NOT precede the refusal on the wire: the provider is holding a turn open on the result, and an observer must not stand in front of it. Infrastructure tools are declared by the channel rather than by the caller's catalogue, so the declared-tool check admits them when the session declares them: Tool Search's while it hides the session's catalogue, run_skill_script with an executor. One the session does not declare is refused as any undeclared tool is (Section 6.4). Those that only read or unlock (skill activation and reference reading, Tool Search: the exact names of Section 21.1) remain exempt from skill gating, which they exist to operate; run_skill_script acts, and is gated like any other tool.

One call, one outcome (normative). Every realtime tool call, whatever door it came through (the provider's function call, a call recovered from speech, a reasoning backend's call, a conference's call), is served by the same steps: the gate above; the serving, inside the tool call context of Section 21.4, which a conference installs too, with the chain depth of the answer that issued the call (Section 8.3); ON_TOOL_CALL (Section 9.3); the bound of Section 21.5, which covers the results of Tool Search and of reading a skill's references as it covers a handler's, a tool's complete schema read by name aside, since the model needs it whole to call the tool (a refusal, a BLOCK or a hook's replacement in its place is bounded as any result); the delivery; the report. A call's result is delivered once and its outcome reported once: a reconfiguration that fails once the result went out sends no second one, and a cancellation that arrives after the result was delivered is not reported. A call an ending interrupts is reported once, as cancelled. The endings are the session's end, the channel's close, and on a conference a detach or the unplug of its realtime provider. A call whose own handler caused the ending (a hang-up tool ending its session, detaching its room, unplugging the provider, closing its channel or the framework) is not interrupted, on every host and every door, a reasoning backend's included: it runs on and reports its own outcome, while the other calls the ending reaches are interrupted. The call that caused an ending is known by the call context the ending runs in, which every task its handler starts inherits, not by the task that runs the ending; the context lasts until the call's outcome is delivered and reported. A handler that ends its session through a task of its own, awaited or not, and a close that runs in a task of its own, still spare it, and the task that holds it (a backend's delegation) with it while the call runs: the backend's other calls the ending reaches are interrupted, each answered to the backend as cancelled, and once the sparing call returns on the session its ending closed, the delegation ends with it, no further round run. An AI channel's close that cuts the turns holding such a call (a backend's agent) spares it the same way. A spared call still running when its channel closes is waited for within the close's bound, then interrupted and reported once, as cancelled; nothing of it outlives the close. A handler that closes the framework itself finishes once the framework is gone: it runs on, but its outcome reaches no hook, and is logged. A start that fails is a session's end for the calls the provider issued while it was pending, up to its disconnection. A report under way when an ending arrives (a call's refusal, or the cancellation of a call the provider abandoned) is still made, with the outcome already decided: a refusal is reported refused, whether or not its body went out, and an abandoned call cancelled (Section 9.3). An id names its call until the call's result goes out or the provider abandons the call: a second call under it before then is refused and reported once, and sends nothing, since the id's one result is the first call's, which runs on; a call under it after then is a new call, to the channel and to the provider alike, and gets its own answer while the first finishes its report. A provider frees the id when it sends the result, at the step the channel counts the call delivered, before anything can yield in between: a call issued meanwhile is otherwise new to the one and a duplicate to the other, and is never answered. A call the provider abandons (the model discarded it, a reconnect orphaned it, the provider's wait on it timed out, the connection ended) frees its id the same way, at the step the provider reports it: the channel frees it there too, before the handler it interrupts has finished, sends nothing for that call afterwards, and answers a call issued under the id meanwhile as a new call. A provider holds the ids it issued in one book, freed by the result or the abandonment, and drops, with a log, a result submitted for an id it does not hold, abandoned, never issued, or issued by a connection that is gone: nothing goes out for it, whoever submits it (an application calling the provider directly included), and an abandoned call is reported once. A session connected again under the same id abandons the calls its previous connection issued. A call a reconnect its own handler caused orphans, which runs on, frees its id at that step as well: the new connection never issued it. A call that came without an id is refused and reported once the same way, with nothing sent, since no result can name it. Such a refusal takes the path of any call: the session's end is read first (a call on an ended session is reported cancelled), a call_tool transport is unwrapped so the report names the tool it carries, and it waits for the transcription that precedes it. A text turn holds its round's calls by id the same way: a second call of a round under an id the first holds is refused with the same words and reported as a call of its own, the first keeping its report and its end row its own arguments; one the provider already ran under such an id is reported as a call of its own with the outcome the provider gave it, and a call under an id an earlier round used is a new call. A call that named no tool is refused before the gate, as an unreadable one, and answered under its id. A provider hands every call the model makes to the channel, these, a call the output cap cut and a call to a tool the channel never declared included: the channel decides it, and no provider or vendor SDK answers it in the channel's place. The id a provider hands on is the service's, never a key of the call's arguments: a vendor SDK that folds the id into the arguments, where a key the model wrote can replace it, is patched so the service's id stands and the model's key stays an argument, judged as any (Section 9.3). A provider books a call as awaiting its result only when the channel can answer it: a call refused with nothing sent MUST NOT hold the response, or the provider's continuation after it, open on a result that never comes. A channel that mutes the input while one of the provider's tool calls runs keeps it muted until the last such call ends; a call recovered from speech or a backend's call mutes nothing, the user's input being no answer to it.

An implementation MAY additionally recover a tool call the model emitted as assistant text rather than through the provider's function calling API. Such a recovered call MUST pass the same gate — its arguments are reconstructed from free text and are therefore less trustworthy than a function call's, not more. Its outcome, result or refusal, MUST be returned as injected context rather than as a tool result, because the model issued no call and has no pending response to close. The result it returns is set apart as a tool's result (<tool_result>, Section 6.4): it is data the tool returned, never the runtime's own words.

A response's calls are answered together (normative). A provider that must be asked to continue once tool results are in MUST ask once per model response, after that response has ended and every call it emitted has its result, not once per result. Two calls issued in parallel would otherwise ask twice, the first time while the response is still active, which the service rejects, and the second call's output would then wait, unspoken, for the caller's next turn. A result for a call of a response the conversation has already left (the caller spoke again and a new response began) joins the response in progress: that response's single request covers it, once the response has ended and its own calls have their results, and with no response in progress the provider asks at once. A request the provider has made counts as a response in progress until the response it asked for begins, and a result submitted in between is not part of that response: the next request covers it. Such a result never holds the new response back, and never waits for the caller's next turn.

Every request the provider makes reads that same state, whoever makes it: the continuation, a non-silent text injection, the request that closes the caller's turn in the endpointing role (Section 12.4, audio pipeline). None sends a request while a response is in progress, active or requested. And a continuation never starts while the caller is speaking (between activity start and activity end in the endpointing role, between the provider's own speech start and speech end otherwise): it is held until the caller's turn ends, and the request that answers the caller's turn covers it, since the results already sit in the conversation ahead of what the caller said. A continuation that started anyway would speak over the caller, and the caller's own request would then find a response in progress and leave the turn unanswered.

A caller's turn whose request found a response in progress is still owed one: the provider asks once that response ends. Any request the provider sends after the caller's turn was committed covers it, a continuation included, so the turn is never answered twice. A request the service rejects is no longer in progress: the error that answers it releases the state, or the session would never ask again.

Text injection refers to programmatically inserting text into a realtime session's conversation context (e.g., instructions, tool results, or context updates) rather than sending audio. The ON_REALTIME_TEXT_INJECTED hook fires after such an injection, allowing integrators to log or react to context changes. It receives the event of the injection itself, the same on every host of a realtime model (a realtime voice channel, a conference with a realtime model plugged in): its source the host channel, its content the text, and its metadata injected_role (the intent) and session_id; an injection that carries a broadcast event names it in injected_from (its channel_id and event_id). A session the host no longer holds, or one its provider ended before the host let it go, is not injected into: the injection is not_sent with reason realtime_session_gone, and the hook does not fire.

The role is an intent, not a wire field (normative). The providers do not share a vocabulary: one takes system and user messages, one takes only user and model turns, one takes no turns at all. inject_text therefore names what the integrator means, and each provider maps that meaning onto what its wire offers:

• system — an instruction from the application: how to behave, or what to do now ("greet the user by first name, then listen"). Non-silent, the model acts on it at once.
• user — content for the conversation. On a turn-based provider it is a user turn the model answers; on a full-duplex provider that takes only appends it is content the model says aloud, in its own words (Section 12.4.1).
• assistant — a line the agent says to the user now (a configured greeting, a scripted announcement). Where the wire has a primitive that makes the agent speak a text, the provider uses it, verbatim or paraphrased as that primitive does; otherwise it asks the model to say the line, as an instruction that quotes the line as Section 6.4 quotes a text, so the line cannot end the instruction and add to it. It MUST NOT reach the model as something the user said: the model would answer its own line.
• silent — any intent, added as context without asking for a response.

A provider MAY accept further intents its wire carries and MUST document them.

Outside a realtime session the same intent is an INSTRUCTION event (§10.1.1): it reaches a room's intelligence channels through the inbound pipeline, as the input of one turn marked as the application's, and is never stored.

Text from another channel (normative). A text event another channel broadcast into the room reaches a realtime session (a realtime voice channel's, a conference's with a realtime model plugged in) as what it is: content someone else wrote. It is injected with the user intent, never system nor assistant, its author named as a transcript names them (a person by their name and the channel they wrote through, anything else by its channel id) and its text quoted as Section 6.4 quotes a text from outside: Marie · sms: “…”. Nothing the event carries chooses the intent, its metadata included, which a remote client can write. The application directs the model through its own doors: inject_text with the system intent, or a delivery marked as an instruction (Section 22.1). A blank text reaches no session.

Text that directs the model MUST be injected with the system role, never as user: on a full-duplex provider a user injection is voiced as the model's own words, so an instruction sent that way is said, or improvised upon, instead of followed. An instruction to greet ("greet the user by first name") is system; the greeting itself, when the application wrote it, is assistant.

Intent	OpenAI Realtime	Gemini Live	OpenAI GPT-Live (full duplex)
system	system message item, then a response request	no system role in turns: a user turn	instructions append
user	user message item, then a response request	user turn	commentary append (spoken, paraphrased)
assistant	system message asking to say the line, then a response request	user turn asking to say the line	instructions append asking to say the line
silent	the item, no response request	turn not completed; once audio flows, a text marked as context (best effort)	thinking append

A provider with a speech primitive of its own carries assistant on it — an agent message (Deepgram's InjectAgentMessage), a direct text-to-speech call (Anam's talk). A primitive that relays information in the model's own words is not one: given a written greeting as a commentary append, GPT-Live improvised another, down to a name the application never supplied. A provider whose only instruction primitive rewrites the prompt keeps that for a silent system, and delivers a non-silent one where the model answers at once (Deepgram: a user message), or the instruction waits for the user to speak.

A provider whose wire cannot express an intent MUST map it to the nearest primitive that keeps the intent's effect (an instruction still directs, content is still content) and MUST document the mapping in its inject_text. A mapping that changes the effect is not an implementation detail: a provider with no way to carry an intent MUST refuse it (not_sent, with a reason) rather than deliver it as something else.

A provider that joins an injected text to instructions of its own MUST keep the two apart, so that the text never reads as more of the instructions: an instruction a resumption left unapplied (below) is set apart in a block of its own before the injected text, and content a provider adds to the prompt it rewrites for a silent injection (Deepgram) is set apart in a <context> block after the instructions.

A reconfiguration's instruction takes effect. A provider that applies a reconfiguration by resuming the session, where the service resumes it under the instruction it started with (observed on gemini-3.8-live), MUST still make the new instruction govern the session: a session that holds no conversation yet reconnects without resuming, since it has nothing to keep, and a session that holds one keeps its context and receives the new instruction with its next non-silent injection, which the model acts on. An integrator that reconfigures such a session directly follows it with an injection when the model must act on the change at once.

RealtimeAudioTransport interface:

RealtimeAudioTransport (interface)
├── name: string                             # Transport name (e.g., "websocket", "webrtc")
├── accept(session, connection) → void
├── send_audio(session, audio_chunk) → void
├── send_message(session, message) → void   # Send control/data message to client
├── on_audio_received(callback) → void      # Push-based audio reception
├── on_client_disconnected(callback) → void
├── set_trace_emitter(emitter | null) → void  # Set callback for emitting ProtocolTraces
├── disconnect(session) → void              # Disconnect a client session
└── close() → void                          # Release resources

Audio reception uses a push-based model via on_audio_received(callback) rather than a pull-based async iterator. This aligns with the transport pattern used by VoiceBackend and avoids the complexity of managing iterator lifecycles across session boundaries.

Session lifecycle:

1. Client connects → RealtimeAudioTransport.accept()
2. start_session(room_id, participant_id, connection, metadata, organization_id)
   ├── Read the room scoped to organization_id (§17.2); another
   │   organization's room is not found, and no session is created
   ├── Create RealtimeSession
   ├── Retain transport audio received from here until the provider has
   │   connected (at most 1 MiB; past it, new input is dropped and logged)
   ├── Connect provider with system_prompt, voice, tools
   ├── Wire callbacks: transport audio → provider, provider audio → transport
   └── Flush the retained audio to the provider in arrival order, frames
       arriving during the flush included, before ON_SESSION_STARTED; a
       failed start discards it
3. Audio flows bidirectionally: Client ↔ Transport ↔ Provider
4. Transcriptions emitted as RoomEvents (if configured), after the model
   heard them: identity verification neither labels nor holds them, and
   the integrator gates the session instead (§11.7.3). A user
   transcription has `chain_depth` 0. An assistant transcription answers
   what the model last heard, and carries that depth plus one (§8.3): 1
   after the user spoke, the injected event's depth plus one after a text
   injection.
5. Tool calls pass the pre-execution gate, then are handled via:
   ├── Async tool handler function (if provided)
   └── ON_TOOL_CALL hook (fallback)
6. end_session() → disconnect provider and transport

Audio pipeline in speech-to-speech mode:

When using a speech-to-speech provider, the audio pipeline (Section 12.3) MAY be configured as an optional preprocessor. In this mode the pipeline sits between the transport and the provider:

Client → Transport → [Resampler] → [AEC] → [AGC] → [Denoiser] → [Diarization] → Provider

Typical use cases:

• Resampling — normalize format for the provider's expected input
• AEC — prevent the provider from hearing its own output
• AGC — consistent volume for the provider's speech detection
• Denoising — cleaner audio improves AI recognition accuracy
• Diarization — identify which speaker is talking in multi-party calls
• Audio level monitoring (via optional VAD) — for UI indicators
• Activity logging and metrics — for observability

VAD is OPTIONAL in this mode, and a configured VAD serves one of two roles, set by the provider: endpointing on a provider that takes turns, observation on a full-duplex provider.

Observation. The VAD feeds diarization, level metering and metrics; it does NOT control when the speech-to-speech provider responds. Turn-taking is fully managed by the provider's own endpointing.

Endpointing. The pipeline VAD drives turn-taking and the provider's server VAD is disabled. This exists because a provider's endpointing is not always the right one: an integrator may have a VAD tuned to their acoustics, or need the same turn behaviour across several providers whose endpointing differs. In this role the implementation MUST signal activity start and end to the provider, and MUST NOT rely on provider-side speech events for turn boundaries.

An implementation that supports only the observation role is conformant. A full-duplex provider admits only the observation role (Section 12.4.1).

The RealtimeVoiceChannel accepts an AudioPipelineConfig in the same way as VoiceChannel. When a pipeline is configured, inbound audio frames are processed through the pipeline before being forwarded to the provider.

12.4.1 Full-Duplex Providers and Reasoning Delegation

Some speech-to-speech providers do not take turns. A full-duplex provider (e.g., OpenAI GPT-Live) listens and speaks at the same time: it decides on its own when to answer, when to fall silent because the user spoke over it, and when to answer only with a backchannel and let the user keep talking. Its wire carries audio and transcript deltas in both directions and little else — no response start or end, no speech start or end, no request to respond, no interruption. The same providers hold no tools: the conversational model hands reasoning and tool use to a backend model, a reasoning delegation, and keeps talking while the backend works. A provider declares both traits with full_duplex = true, and this section binds only providers that declare it. Support is OPTIONAL within Conformance Level 3.

Reasoning delegation is not Task Delegation (Section 23). Section 23 dispatches a described task to an agent in a child room and delivers a result into the conversation; a reasoning delegation is the provider's own model deferring the next thing it will say, inside the session, to a model that can think longer and call tools. The two compose: the backend of a reasoning delegation MAY itself delegate a task.

Boundaries are synthesized (normative). The channel contracts of this section — idle tracking, end-of-response flushing, AEC activation, one utterance at a time on the bot track (Section 12.10.12) — hold on on_response_start and on_response_end, and a full-duplex provider has neither on its wire. The provider MUST synthesize them: on_response_start when assistant output (audio or transcript) resumes after quiet, on_response_end when assistant output has been quiet for a configurable gap. The gap MUST exceed the provider's transcript frame granularity, or an ordinary pause inside a sentence would close the response; for a provider that emits transcript fragments on 200 ms frames a default in the order of 800 ms is RECOMMENDED. A provider MAY synthesize on_speech_start and on_speech_end from user-role transcript activity the same way, so that ON_SPEECH_START and ON_SPEECH_END keep firing; the channel MUST treat them as observation, never as the entry to the interruption path below. Synthesized boundaries lag the audio they describe by the gap, and a latency metric read from them MUST account for it.

An answer awaited starts with the model's output (normative). A host that sent the model a tool result or a delegation's spoken output is not idle until the model's answer to it starts. The answer starts with a response start, and, on a full-duplex provider, which may say it inside a response already open and synthesize no new start, with audible assistant audio or a non-empty partial transcript of the assistant's words. The host waits from before the output is sent, so an answer that starts before the send returns is not missed. An output that could not be sent is not waited for. Every host of a realtime model (a realtime voice channel, a conference) reads the answer's start the same way.

Interruption belongs to the model (normative). When the user speaks over a full-duplex provider, the model hears it and decides — it stops, or finishes its clause, or says "mhm" and carries on. The framework MUST NOT decide for it: on a full-duplex session the channel MUST NOT call interrupt() or truncate_audio(), MUST NOT flush or interrupt the transport's playback, and MUST NOT gate provider audio on user speech, whatever the sensor — provider speech events, pipeline VAD, or transport-level barge-in. Gating would cut the model's backchannels and its own recovery from being talked over, the very behaviour the provider was chosen for. The provider's interrupt() and truncate_audio() MUST be no-ops that neither fail nor log above debug level, so a channel that ignores the flag does no harm calling them. ON_BARGE_IN does not fire on a full-duplex 1:1 session: the framework interrupted nothing. A conference keeps its own authority over the bot track and is treated in Section 12.10.12.

The audio pipeline (Section 12.3) keeps its preprocessing role. Of the two VAD roles above, only observation is admissible: endpointing needs an activity signal the provider does not take. An implementation MUST NOT drive endpointing or interruption from a pipeline VAD on such a session, and SHOULD say so where the VAD is configured — a notice at session start suffices — rather than demote it silently.

The output is a stream, not a burst (normative). A full-duplex provider MAY emit output audio continuously at real-time pace, silence included, since it does not know in advance when it will speak. The outbound path MUST NOT rely on audio arriving faster than real time — a pacer that waits to prebuffer a response would wait for a burst that never comes — and a recorder (Section 12.3.7) records the silence as such.

The session is fixed at start (normative). Such a provider typically fixes model, instructions, voice, audio format, delegation mode and seeded history when the session starts, and takes only appends afterwards. The provider MUST declare supports_mid_session_reconfigure false — a mid-session reconfiguration replaces the session — so that the orchestration that changes a running agent — a handoff (Section 19.6), a skill activation (Section 24.4) — chooses an append where one exists and accepts the loss where it does not. A changed system prompt SHOULD be delivered as an instructions append where the provider offers one; a reconnect reseeds a text-only history and loses the audio context. Tools, in the hosted mode below, are the one field that MAY change after start and MUST be forwarded to the backend without replacing the session. A provider that takes one audio format for both directions MUST resample so that the channel's declared input and output rates hold at the channel boundary.

Text injection is paraphrased. On these providers a spoken injection is context the model relays in its own words, not a script it reads; there is no verbatim primitive, and inject_text MUST document that. Nor is there a user turn: the session takes no user text once started, so a user injection is content the model voices as its own, and an instruction — an opening greeting included — MUST travel as an instructions append (the system intent, Section 12.4). Where the provider bounds the size of one append, an implementation MUST split a longer text on sentence boundaries into as many appends as it takes, rather than truncate or refuse. A text the runtime set apart in a frame (Section 6.4) keeps it in each append: a block left open at the end of an append is closed there and opened again at the start of the next, and a quote left open is closed there and opened again after what precedes it on its line (its author's name, the instruction that quotes it), so that no piece of the text reaches the model outside its frame, an instructions append least of all.

Reasoning delegation, hosted backend (normative). In the hosted mode the provider's service runs the backend model, and the tools passed to connect() are that backend's tools. The channel sees nothing new: every function call the backend emits MUST reach on_tool_call and pass the pre-execution gate of this section unchanged, ON_TOOL_CALL fires as always, and the result returns through submit_tool_result. The provider MUST NOT resume the backend while any call of the same delegation is unanswered — the hosted service rejects a partial continuation — and MUST resume it once the last result is in: the rule of Section 12.4, counted per delegation. A result submitted after the session ended is dropped, as for any provider.

Reasoning delegation, integrator backend (normative). In the integrator mode the model signals only that it is handing work over: the provider fires on_delegation(session, delegation_id, target) with target = "integrator", and the request carries no task text. The hosted mode announces its delegations through the same callback with target = "hosted", so that a channel can observe both while serving one. The channel MUST serve the integrator target through a configured ReasoningBackend:

ReasoningBackend (interface)
├── run(request: ReasoningRequest) → async_iterator<ReasoningOutput>
│       # Work out the request from the transcript and answer it, with the
│       # backend's own model, context and tools
├── session_ended(session_id) → void        # OPTIONAL: release state kept for a session; no-op by default
├── served_names() → set<string>            # OPTIONAL: tool names the backend answers itself, before the gate; none by default
└── close() → void

ReasoningRequest
├── session: VoiceSession
├── delegation_id: string                   # Opaque; returned unchanged with every output
├── transcript: list<TranscriptLine>        # Both roles, since the previous request
├── first: bool                             # First request of the session: the transcript is the whole conversation
├── tools: list<ToolDefinition>             # The channel's declared catalogue, for the backend's model
├── unavailable: map<string, string>        # Session tools the model is not offered, each with the refusal its call reads (Section 21.1)
├── execute_tool(name, arguments) → string  # One call through the channel's pre-execution gate and handler
├── execute_tool_call(name, arguments) → ToolCallResult  # The same call, with its outcome
├── report_refusal(name, arguments, body, cancelled = false, refused = true, detail = null) → void  # Reports a call the backend's own loop ended before the gate, with its outcome, under the id model_call_id() names
├── report_call(name, arguments, result, is_error, detail, tool_call_id) → void  # Reports a call the backend's own provider served outside the gate, with its outcome
└── participant_role: string?               # The session participant's role when the delegation was handed over

ToolCallResult
├── text: string                            # What the backend's model reads
├── is_error: bool                          # Refused, failed, blocked, served by nothing or cancelled
└── refused: bool                           # Among those, refused: the backend's loop reads it refused, any other error failed

TranscriptLine
├── role: "user" | "assistant"
└── text: string

ReasoningOutput
├── text: string
├── spoken: bool                            # true: the model relays it in its own words; false: silent context it may draw on
└── is_final: bool                          # The answer to the request, as opposed to progress toward it

The channel MUST hand the backend the transcript recorded since the previous request, user and assistant lines both — the backend has no other way to learn what was asked — and MUST return every output through submit_delegation_output(session, delegation_id, text, spoken) as it arrives, so that a backend working in steps can tell the user what it learned on the way. The transcript ledger is session state, not a RoomEvent: it MUST NOT be stored, and in a conference (Section 12.10.12) its user lines are unattributed, as that section requires. A backend that yields nothing, fails, or exceeds the bound the channel SHOULD place on its run MUST still be answered with one spoken output saying the work could not be completed — a full-duplex model otherwise keeps the conversation open for an answer that never comes. One that fails on an error also fires ON_ERROR once, as the channel, with the category reasoning, as a room turn's failure fires it (Section 6.4): the fallback tells the user, the hook tells the host. A turn its round cap, deadline or budget cut is an expected end and fires none, as on a room, and so is a run past the channel's bound, the delegation's own deadline. An output returned for a delegation the conversation has moved past is the model's to ignore, and the provider's instructions SHOULD tell it so.

The backend is the integrator's — an AIProvider (Section 6.7) driven through a tool loop, an AIChannel, an Agent (Section 19). This specification defines the contract, not the component. Whatever hosts it, the tool calls the backend makes are tool calls of the framework and MUST pass the pre-execution gate and the ToolPolicy (Section 21) as any other: execute_tool on the request is that gate, followed by the channel's handler, ON_TOOL_CALL and result truncation, and a backend MUST route its calls through it. A delegation is not a way around them. execute_tool_call runs the same call and also says whether it failed, so the backend's model reads a refused or failed call as one, as every tool loop does (Section 9.3); a backend SHOULD use it, and execute_tool returns the text alone.

A backend's turn ends by the rules of a tool loop (Section 6.4), whatever drives its model: a call the provider could not parse, or an empty answer, is asked again within the loop's bounds; a call whose arguments do not read is refused, and a backend whose loop ends a call before the gate reports it to the channel's ON_TOOL_CALL observers through report_refusal, with the outcome its loop gave it (refused, cancelled or failed) and what failed, as the gate reports the calls it refuses itself; a call the backend's own provider served (a provider-side tool, outside the gate) is reported once to the channel's ON_TOOL_CALL hooks through report_call, served or failed, as an AI channel reports one (Section 9.3); each of these reports names the call by the id the backend's model gave it, within its delegation's, whichever of them reports it (report_call takes that id as tool_call_id; execute_tool, execute_tool_call and report_refusal read it from model_call_id(), which the framework's tool loop sets around the call it runs or reports; a call that comes with no id is reported under one minted within its delegation's); and a turn that did not complete (it ended for any reason but completed: its round cap, deadline or budget cut it, or its answer was cut or never came) has no answer. Its narration is progress, never the answer: the run fails, and the channel answers the delegation as it answers a failed backend. A backend call the end of its delegation cut (the channel's bound on the run) is reported once, cancelled, as a call the session's end cut is. A backend that is an agent (Section 19) runs on the AI channel's tool loop with the agent's settings and turn budget, the voice session's catalogue as its tools and the channel's gate as their handler, which bounds each call as it bounds every call of the session (Section 21.6): the agent's own bounds, its default and per tool, do not apply on top of it, and a tool that waits by design has no bound but the run's. The loop's own re-read of a result it stored (Section 21.5) stays the loop's, as on any turn, and answers from memory, with no wait to bound. Tools of its own (host tools, a skill registry even empty, a human-input handler, a sandbox, planning, an external handler) would run outside that gate, and a kit that registered it would judge each call a second time, so such an agent is refused. The names the agent serves itself (its re-read of a stored result, its tool search unless turned off) stay its own: a channel MUST refuse a session tool given under one of them, at construction and at configure, a human-input tool's included, which the agent would otherwise answer outside the gate; one that arrives later (a session's or a room's tools, a reconfiguration) is not declared, with a warning, as Section 21.1 drops a name the channel serves. The agent's own tool_policy composes with the channel's, as an agent's policy does on a realtime pipeline (Section 19.5): it is resolved for the session participant's role, which the channel reads when it hands the delegation over and passes in the request (participant_role), so its role overrides apply; a call it denies is refused in the agent's loop, before the channel's gate, and the channel's ON_TOOL_CALL observers see it as any refusal. Its conversation is the session's: a call a cut delegation left unanswered is answered as a room turn answers one before the next generation, and a session's delegations run one at a time, each reading what the one before it worked out. The backend's turn is traced under the voice session's span.

Observability. ON_REALTIME_DELEGATION (Section 9.2) fires when the model hands work over, in either mode, with the delegation id and its target (hosted or integrator); the time from it to the first spoken output is the latency the user hears. A provider billed by session duration MUST report duration as its usage and MUST NOT restate it as tokens; backend token usage, where the provider surfaces it, is the backend model's and is attributed to it.

12.4.2 Usage Reporting

What a spoken turn consumed is known to the service that billed it, not to the framework, so a realtime provider reports its usage rather than having it inferred. Two surfaces carry that report, and both are public.

The snapshot. VoiceSession.last_usage is what the provider recorded last for that session: a read-only map, empty until the first report. The framework fixes two of its keys — input_tokens and output_tokens, the totals a token-billed provider reports — and a provider MAY add its own breakdown beside them, per modality, cached share, reasoning or tool use, under the names its API uses. A reader MUST treat an absent key as unreported rather than as zero.

The callback. on_usage(callback) fires on every report, with the session and the map just recorded. An integrator that bills a call MUST use it rather than the snapshot: the next report replaces the snapshot and the channel clears it at the end of each turn, so a reader that polls can miss a turn, while a callback sees every one. Callbacks MUST NOT block the provider's event loop, and a failing one MUST NOT break the session.

A provider billed by session duration reports duration (Section 12.4.1) through these same two surfaces, under its own key and never restated as tokens; a hosted backend's token usage is likewise the backend model's and is reported apart, attributed to it.

12.5 Voice Hooks

Voice-specific hooks allow integrators to customize the voice pipeline:

Hook	Type	Use Case	Source
ON_SPEECH_START	ASYNC	Show "listening" indicator	Audio Pipeline (VAD)
ON_SPEECH_END	ASYNC	Log speech duration	Audio Pipeline (VAD)
ON_TRANSCRIPTION	SYNC	Fix STT errors, redact content	STT Provider
BEFORE_TTS	SYNC	Redact or rewrite text, hold speech back	Voice Channel
AFTER_TTS	ASYNC	Log synthesis metrics, cache	Voice Channel
ON_BARGE_IN	ASYNC	Track interruptions	VoiceBackend (transport)
ON_TTS_CANCELLED	ASYNC	Log cancellation reason	Voice Channel
ON_PARTIAL_TRANSCRIPTION	ASYNC	Show real-time captions	STT Provider
ON_VAD_SILENCE	ASYNC	Trigger silence timeout	Audio Pipeline (VAD)
ON_VAD_AUDIO_LEVEL	ASYNC	Audio level visualization	Audio Pipeline (VAD)
ON_INPUT_AUDIO_LEVEL	ASYNC	VU meter for mic input	Audio Pipeline
ON_OUTPUT_AUDIO_LEVEL	ASYNC	VU meter for speaker output	VoiceBackend
ON_SPEAKER_CHANGE	ASYNC	Identify speaker switch	Audio Pipeline (Diarization) or STT (Section 12.2.3)
ON_DTMF	ASYNC	IVR navigation, call transfer	Audio Pipeline (DTMF Detector)
ON_TURN_COMPLETE	ASYNC	Log turn-taking metrics	Audio Pipeline (Turn Detector)
ON_TURN_INCOMPLETE	ASYNC	Debug turn detection	Audio Pipeline (Turn Detector)
ON_BACKCHANNEL	ASYNC	Track user engagement	Audio Pipeline (Backchannel Detector)
ON_SESSION_STARTED	ASYNC	Send greeting, start telemetry	VoiceBackend / Inbound pipeline
ON_RECORDING_STARTED	ASYNC	Notify participants of recording	Audio Pipeline (Recorder) / Conference Channel / Room media recorder (Section 12.11)
ON_RECORDING_STOPPED	ASYNC	Store recording reference in timeline	Audio Pipeline (Recorder) / Conference Channel / Room media recorder (Section 12.11)
ON_TOOL_CALL	SYNC	Execute tool and return result	Realtime Provider (ON_REALTIME_TOOL_CALL is superseded, Section 9.2)
ON_TOOL_CALL	ASYNC	Audit tool use, including calls refused, failed, or abandoned by the model (Section 9.3)	Realtime Voice Channel
ON_REALTIME_TEXT_INJECTED	ASYNC	Log text injections	Realtime Voice Channel / Conference Channel (realtime model)
ON_REALTIME_DELEGATION	ASYNC	Measure delegation latency, log hand-offs to the backend	Realtime Provider (Section 12.4.1)
ON_PROTOCOL_TRACE	ASYNC	Log/inspect transport protocol traces (SIP, RTP)	Channel (via emit_trace)
BEFORE_BRIDGE_AUDIO	SYNC	Filter/modify audio before bridging (mute, gain)	AudioBridge (Section 12.7)

Audio level hooks (ON_INPUT_AUDIO_LEVEL, ON_OUTPUT_AUDIO_LEVEL):

These hooks provide real-time audio level (RMS in dB) for building VU meters and audio visualizations. Unlike ON_VAD_AUDIO_LEVEL (which requires VAD and includes speech classification), these fire independently of VAD for all processed audio.

Implementations SHOULD throttle ON_INPUT_AUDIO_LEVEL and ON_OUTPUT_AUDIO_LEVEL to at most 10 events per second per session (default interval: 100ms). Without throttling, per-frame firing at typical 20ms frame sizes would produce 50 events/sec per direction, each requiring a context build and store query.

The event payload is an AudioLevelEvent containing session, level_db (typically -60 to 0 dBFS), and timestamp.

12.6 Barge-In and Interruption Handling

Barge-in occurs when a user speaks while TTS is playing. The framework supports configurable interruption strategies (Section 12.3.13 — InterruptionConfig) ranging from immediate cancellation to semantic backchannel detection.

Default behavior (CONFIRMED strategy):

1. The VoiceBackend fires on_barge_in callback.
2. The pipeline checks InterruptionConfig.strategy:
3. IMMEDIATE: Cancel TTS immediately.
4. CONFIRMED: Wait for min_speech_ms of sustained speech.
5. SEMANTIC: Run BackchannelDetector to classify.
6. DISABLED: Ignore, queue speech for after TTS completes.
7. If interruption is confirmed: a. Cancel current TTS playback (if backend supports INTERRUPTION). b. If flush_partial_tts = true: discard unplayed audio buffer. c. If keep_partial_transcript = true: store partial bot response in timeline with metadata.interrupted = true. d. If the TTS consumes context: record the assistant turn truncated to played_ms (Section 12.2.2). e. Fire ON_BARGE_IN hook. f. Process the user's speech through the audio pipeline as the user's turn.
8. If classified as backchannel: a. Fire ON_BACKCHANNEL hook. b. TTS continues uninterrupted, and the speech is discarded (Section 12.3.13).

Implementations SHOULD support a configurable minimum playback duration before barge-in is allowed (InterruptionConfig.allow_during_first_ms, Section 12.3.13). This prevents interruption at the very start of a response.

Relationship between VoiceBackend barge-in and VAD:

VoiceBackend.on_barge_in and VAD are two separate detection systems:

• on_barge_in is a transport-level signal — the backend detects that the client is sending audio while outbound audio is playing. It fires immediately and is the entry point for the interruption strategy. Not all backends support this (requires BARGE_IN capability).
• VAD is a pipeline-level signal — it detects speech activity in the audio stream regardless of whether TTS is playing. VAD continues to process inbound frames during TTS playback.

When both are active during TTS playback:

1. on_barge_in fires first (transport-level, lowest latency).
2. The interruption strategy determines whether to act on the barge-in.
3. Meanwhile, VAD processes the same audio through the pipeline normally.
4. If the interruption is confirmed, the voice channel cancels TTS and the VAD's speech segment is routed to STT as usual.
5. If the interruption is rejected (backchannel or too short), the VAD speech segment is still captured but the pipeline SHOULD discard it (or queue it if InterruptionStrategy = DISABLED). A SEMANTIC segment that ends before its first word is not rejected as too short: its final transcript is classified first (Section 12.3.13).

When the backend does NOT support BARGE_IN, the pipeline MAY use VAD's SPEECH_START event during TTS playback as a fallback barge-in trigger. In this mode, the VAD effectively replaces the transport-level detection, but with higher latency (audio must traverse the full inbound pipeline before VAD fires).

12.7 Audio Bridging

Audio bridging enables human-to-human voice communication within a room by forwarding audio frames directly between voice sessions, bypassing the STT → text → TTS roundtrip. This is essential for use cases like SIP/RTP phone calls between participants, call center agent bridging, and multi-party conferencing.

12.7.1 Design Principles

1. Audio bridging is a parallel path, not a replacement. Bridge mode adds a direct audio forwarding path alongside the existing STT/TTS path. STT MAY still run concurrently for transcription, recording, or AI monitoring. The two paths are independent.

2. Audio bypasses the RoomEvent broadcast system. Real-time audio at 50 frames/second cannot traverse the async hook pipeline, store, and broadcast loop without unacceptable latency. Bridged audio flows directly from inbound pipeline output to outbound pipeline input, never becoming a RoomEvent.

3. Pipeline stages still apply. Inbound audio passes through the full pipeline (resampler, AEC, AGC, denoiser) before being forwarded. Outbound bridged audio passes through the outbound pipeline (resampler, recorder tap, AEC reference feeding) before reaching the transport.

4. The bridge is a concrete component, not an ABC. Unlike STT/TTS/VAD providers, there are no genuinely different "bridge implementations" to swap. Variations (direct forwarding vs N-party mixing) are configuration options on a single AudioBridge class.

12.7.2 AudioBridge

AudioBridge
├── config: AudioBridgeConfig
├── add_session(session, room_id, backend) → void
│       # Register a session for bridging
├── remove_session(session_id) → void
│       # Unregister a session
├── forward(session, audio_frame) → void
│       # Forward audio from this session to all other sessions in the room
│       # "forward": the frame goes unmixed to each other session
│       # "mix": each other session receives a mix of every session's
│       #   latest frame but its own
└── close() → void
        # Clean up all sessions

AudioBridgeConfig
├── enabled: bool = true
├── max_participants: int = 10         # Maximum sessions per room
└── mixing_strategy: "forward" | "mix" = "forward"
        # "forward": direct forwarding; each frame is sent unmixed to every
        #   other session (N>2: one send per source per target)
        # "mix": N-party additive mixing with clipping protection

Session management: Sessions are added and removed as participants join and leave the room. The bridge tracks sessions grouped by room_id. When a session is added and the room already has other bridged sessions, audio forwarding begins immediately.

Audio forwarding: When forward() is called with an audio frame from session A, the bridge sends that frame to all other sessions in the same room via VoiceBackend.send_audio_sync(). For the "forward" strategy, this is a direct send of the frame to each other session. For the "mix" strategy (N-party), the bridge MUST mix audio from all other active sessions and send each participant a mix of everyone else's audio (excluding their own, to prevent echo).

Thread safety: forward() is called from audio callback threads (same context as on_audio_received). All bridge operations MUST be thread-safe.

12.7.3 VoiceChannel Integration

VoiceChannel gains an optional bridge parameter:

VoiceChannel(
    "voice",
    backend=sip_backend,
    pipeline=pipeline,
    bridge=True,              # Enable bridging (AudioBridgeConfig defaults)
    # bridge=AudioBridgeConfig(mixing_strategy="mix"),  # Explicit config
    stt=deepgram,             # Optional: transcription alongside bridge
)

Inbound audio flow with bridge enabled:

Backend → on_audio_received → Pipeline inbound chain
    ├──→ AudioBridge.forward()       # Parallel: forward to other sessions
    └──→ VAD → STT (if configured)  # Parallel: transcription path (optional)

When bridge mode is enabled and audio exits the inbound pipeline, the VoiceChannel MUST feed the processed frame to AudioBridge.forward() in addition to the normal VAD/STT path. The bridge path and the STT path operate in parallel — neither blocks the other.

Outbound bridged audio flow:

AudioBridge.forward() → Pipeline outbound chain → Backend.send_audio_sync()
    [Recorder tap] → [AEC reference] → [Resampler] → Transport

Bridged audio sent to a session MUST pass through the outbound pipeline before reaching the transport. This ensures recording captures both sides and AEC reference is fed correctly.

Session lifecycle:

• When bind_session() is called with bridge enabled, the VoiceChannel MUST register the session with the AudioBridge via add_session().
• When unbind_session() is called, the VoiceChannel MUST remove the session via remove_session().
• The bridge does NOT manage session state transitions — that remains the responsibility of VoiceBackend and VoiceChannel.

Interaction with STT/TTS:

Configuration	Behavior
bridge=True, stt=None, tts=None	Pure audio bridge — human-to-human only
bridge=True, stt=provider	Bridge + live transcription (for recording, compliance, AI monitoring)
bridge=True, stt=provider, tts=provider	Bridge + AI can speak into the call via say() (TTS audio mixed with bridge audio)
bridge=False (default)	Current behavior — STT/TTS pipeline only

When both bridge and TTS are active, TTS audio from AI responses and bridged audio from other participants are both sent to the session. The bridge does NOT suppress TTS — both audio streams coexist. If audio mixing is needed (e.g., AI speaking while another participant is speaking), the backend handles concurrent send_audio() (TTS) and send_audio_sync() (bridge) calls.

Several transports on one channel. A bridged room often mixes transports: phone callers on SIP and browser participants on WebRTC. One VoiceChannel MAY serve sessions from more than one VoiceBackend:

VoiceChannel.add_backend(backend) → void
        # Register an additional transport. Its inbound audio enters the
        # channel's inbound pipeline, its session-ready and
        # client-disconnected signals drive the channel's session
        # lifecycle, and its own barge-in detection and out-of-band DTMF
        # reach the channel, exactly as for the backend given at
        # construction.

• The channel MUST have been constructed with a backend: the pipeline is built once, for that primary backend's capabilities, and every added transport's audio goes through it.
• A session belongs to exactly one backend. bind_session(…, backend=) names it; without it, the channel asks each added backend whether it holds the session (get_session) and falls back to the primary backend. A session joined on a pull-model backend is started listening on that backend.
• Everything addressed to one session — bridged audio, TTS audio, assistant transcriptions, the playback cancel of an interruption, DTMF, and the disconnect when the session leaves — MUST go through that session's own backend.
• A response delivered to the room reaches the sessions of every backend. What a backend's capabilities decide for one session — whether its playback can be cancelled (INTERRUPTION), whether its echo is cancelled at the transport (NATIVE_AEC) — is read on that session's own backend.
• Adding a backend already served is a no-op. Closing the channel closes the added backends with the primary one.

12.7.4 Sample Rate Handling

Voice sessions MAY have different sample rates (e.g., SIP at 8kHz, WebRTC at 48kHz). When forwarding audio between sessions with different rates, the outbound pipeline's resampler MUST convert the audio to the target session's native rate. Implementations SHOULD track each session's sample rate at registration time and apply resampling per-target in the outbound path.

12.7.5 N-Party Mixing

With the "mix" strategy, the bridge MUST mix audio so each participant hears all others but not themselves.

Mixing algorithm (additive with clipping protection):

1. For each target session T, collect audio frames from all other sessions.
2. Sum PCM samples (as signed 16-bit integers promoted to 32-bit to prevent overflow).
3. Apply soft clipping or normalization to prevent distortion:
4. Simple: clamp to [-32768, 32767]
5. Recommended: apply headroom scaling (e.g., divide by sqrt(N-1))
6. Convert back to 16-bit PCM and forward to session T.

Silence handling: If a session has no audio frame available in the current mixing window, it contributes silence (zero samples). The bridge SHOULD NOT forward silence-only mixes to save bandwidth.

12.7.6 Hooks

Audio bridging introduces one new hook and reuses existing hooks:

Hook	Type	Use Case	Source
BEFORE_BRIDGE_AUDIO	SYNC	Filter or modify audio before forwarding (e.g., gain, muting)	AudioBridge

BEFORE_BRIDGE_AUDIO fires for each frame before it is forwarded to other sessions. The hook receives the audio frame and context (source session, target room). Returning HookResult.block() drops the frame (effectively muting the speaker for that frame). Returning a modified frame applies the modification before forwarding.

Existing hooks that fire normally during bridge mode: - ON_SPEECH_START, ON_SPEECH_END — VAD still runs on bridged audio - ON_TRANSCRIPTION — if STT is configured - ON_RECORDING_STARTED, ON_RECORDING_STOPPED — recorder captures bridge audio - ON_SESSION_STARTED — session lifecycle unchanged

Note: BEFORE_BRIDGE_AUDIO is a SYNC hook that runs in the audio callback path. Implementations MUST ensure hook handlers complete quickly (< 1ms) to avoid audio glitches. Long-running operations MUST NOT be performed in this hook.

12.7.7 Capability Flag

VoiceCapability.NATIVE_BRIDGE

When a VoiceBackend declares NATIVE_BRIDGE, it can bridge audio at the transport level (e.g., RTP relay, SIP re-INVITE for direct media). The VoiceChannel MAY delegate bridging to the backend instead of using the AudioBridge when this capability is present. Native bridging provides lower latency and reduced CPU usage but bypasses pipeline stages.

Implementations that use native bridging MUST still fire session lifecycle hooks and MUST support fallback to AudioBridge when pipeline processing (recording, AEC, transcription) is required.

12.7.8 Conformance

Audio bridging is part of Conformance Level 3 (Real-Time Media) and is OPTIONAL.

Implementations that support audio bridging MUST: - Forward post-pipeline audio frames between sessions in the same room. - Support at least 2-party direct forwarding. - Apply outbound pipeline processing to bridged audio. - Register/unregister sessions on bind/unbind. - Ensure thread-safe operation of forward().

Implementations SHOULD: - Support N-party mixing for rooms with more than 2 voice sessions. - Handle cross-rate bridging via the outbound resampler. - Support concurrent bridge + STT operation.

12.8 Video Pipeline

The video subsystem mirrors the audio pipeline in structure. Video processing uses the same pluggable stage architecture, session model, and hook pattern as audio.

12.8.1 Architecture Overview

┌────────┐     ┌──────────────────────────────────────┐     ┌────────────┐
│ Video  │────→│          Video Pipeline               │────→│   Vision   │
│Backend │     │ Decode → Resize → Transform → Filter  │     │  Provider  │
└────────┘     └──────────────────────────────────────┘     └────────────┘
                              │                                    │
                         ┌────┴────┐                      ┌───────┴───────┐
                         │Recorder │                      │ AI Integration│
                         └─────────┘                      │ (inject into  │
                              │                           │  AIChannel)   │
                         ┌────┴─────┐                     └───────────────┘
                         │  Bridge  │
                         │(forward) │
                         └──────────┘

Three video channel variants:

Channel	Base Class	Purpose
VideoChannel	(standalone)	Video-only with vision analysis
AudioVideoChannel	VoiceChannel	Combined STT/TTS + video pipeline + avatar
RealtimeAudioVideoChannel	RealtimeVoiceChannel	Speech-to-speech + video

12.8.2 Video Data Models

VideoFrame — inbound video frame:

VideoFrame
├── data: bytes                             # Encoded NAL units OR raw pixels
├── codec: string                           # h264, vp8, vp9, av1 (encoded) or raw_rgb24, raw_bgr24, raw_yuv420p, raw_nv12 (raw)
├── width: int                              # Frame width in pixels
├── height: int                             # Frame height in pixels
├── timestamp_ms: float                     # Relative to session start
├── keyframe: bool                          # Whether this is a keyframe
├── sequence: int                           # Monotonically increasing per session
├── metadata: map<string, any>              # Accumulated by pipeline stages
│
├── is_encoded: bool (derived)              # True when codec in {h264, vp8, vp9, av1}
└── is_raw: bool (derived)                  # True when codec in {raw_rgb24, raw_bgr24, ...}

VideoChunk — outbound encoded video:

VideoChunk
├── data: bytes                             # Encoded frame data
├── codec: string                           # h264, vp8, vp9, av1 only
├── width: int
├── height: int
├── timestamp_ms: float | null
├── keyframe: bool
└── is_final: bool                          # Signals end of stream

VideoSession — active video connection:

VideoSession
├── id: string
├── room_id: string
├── participant_id: string
├── channel_id: string
├── state: VideoSessionState                # CONNECTING → ACTIVE → PAUSED → ENDED
├── provider_session_id: string | null      # Backend-specific identifier
├── created_at: datetime
└── metadata: map<string, any>

VideoCapability flags:

Flag	Description
SIMULCAST	Multiple resolution streams
SVC	Scalable Video Coding layers
SCREEN_SHARE	Separate screen-share track
RECORDING	Server-side recording support
BANDWIDTH_ESTIMATION	Adaptive bitrate

12.8.3 VideoBackend Interface

VideoBackend (interface)
├── name: string (property)                 # Backend identifier
│
│   # Session lifecycle:
├── connect(room_id, participant_id, channel_id, metadata) → VideoSession
├── accept(request) → VideoSession          # Server-side session acceptance
├── disconnect(session) → void
├── get_session(session_id) → VideoSession | null
├── list_sessions(room_id) → list<VideoSession>
│
│   # Video transport:
├── send_video(session, video) → void       # Send VideoChunk or async stream
├── send_video_sync(session, frame) → void  # Synchronous send (for callbacks)
├── request_keyframe(session) → void        # Request PLI/FIR for keyframe recovery
├── set_video_passthrough(session_id, enabled) → void  # Bridge mode toggle
│
│   # Callbacks:
├── on_video_received(callback) → void      # Raw inbound video frames
├── on_session_ready(callback) → void       # Video path is live
├── on_client_disconnected(callback) → void # Transport disconnect
│
├── capabilities: VideoCapability           # What the backend supports
└── close() → void

Required implementations:

Backend	Transport	Use Case
LocalVideoBackend	OpenCV webcam	Development, testing
ScreenCaptureBackend	mss screen capture	Screen sharing, monitoring

Optional backends:

Implementations MAY provide additional backends for WebRTC (FastRTC), RTP, SIP video, or WebSocket transport.

Combined audio/video backends. A transport that carries a call's audio and video together (a SIP INVITE with an m=video line, RTP with a parallel video stream, a WebRTC connection with both tracks) is a voice backend (Section 12.2) whose calls may carry video. One object MUST NOT implement both VoiceBackend and VideoBackend: their lifecycle methods share names and differ in meaning (on_session_ready is the audio path live for one, the video path for the other), so a channel registering its voice callbacks would be served the video ones, or none.

AudioVideoBackend (interface, a VoiceBackend)
└── video: VideoBackend (property)          # The video of its calls

The call is the voice session, and the backend's own lifecycle is the voice backend's: on_session_ready when a call's audio path is live and on_client_disconnected when the call ends, for every call, with or without video. The video of a call is served by the video facet, whose sessions carry the call's session id: its on_session_ready fires when the call's video path is live, for a call that negotiated video only, and its on_client_disconnected when that video ends, which is when its call ends. Video never starts or stops on its own: a call without video stays an audio-only call. The facet creates no session (connect is not supported; calls come from the transport), and its disconnect ends the call.

An audio/video channel (AudioVideoChannel) reports ON_SESSION_STARTED for the call, and ON_VIDEO_SESSION_STARTED and ON_VIDEO_SESSION_ENDED for its video, each once: the video ones only for a call with video, ON_VIDEO_SESSION_STARTED once the session is both bound and its video path live, whichever comes last, and ON_VIDEO_SESSION_ENDED only after it.

12.8.4 Video Processing Pipeline

The video pipeline processes inbound frames through ordered stages:

VideoPipelineConfig
├── decoder: VideoDecoderProvider | null     # Decode h264 → raw_rgb24
├── resizer: VideoResizerProvider | null     # Scale to target dimensions
├── transforms: list<VideoTransformProvider> # Pixel effects (chained)
├── filters: list<VideoFilterProvider>       # Inspect/replace frames (chained)
├── vision: VisionProvider | null            # Periodic frame analysis
├── recorder: VideoRecorder | null           # Save to file
└── recording_config: VideoRecordingConfig   # Recording parameters

VideoPipeline orchestrator:

VideoPipeline
├── process_inbound(session_id, frame) → VideoFrame | null
│       # Synchronous frame processing:
│       # 1. Decode (if frame.is_encoded)
│       # 2. Resize (if frame.is_raw and resizer configured)
│       # 3. Transforms (chained, pixel modifications)
│       # 4. Filters (chained, with FilterContext)
│       # Returns processed frame or null (dropped)
│
├── process_vision(session_id, frame) → VisionResult | null
│       # Async vision analysis (throttled by vision_interval_ms)
│
├── drain_events(session_id) → list<FilterEvent>
│       # Collect filter events for hook dispatch
│
├── update_filter_context(session_id, result) → void
│       # Sync vision results to active filter contexts
│
├── reset(session_id) → void
└── close() → void

Inbound processing flow:

Backend emits VideoFrame
  │
  ├─ [1] Decoder (if frame.is_encoded)
  │     VideoDecoderProvider.decode(frame) → raw_rgb24 frame
  │
  ├─ [2] Resizer (if frame.is_raw)
  │     VideoResizerProvider.resize(frame) → scaled frame
  │
  ├─ [3] Transforms (chained)
  │     For each VideoTransformProvider:
  │       transform(frame) → modified frame
  │     (grayscale, blur, color adjustment, etc.)
  │
  ├─ [4] Filters (chained, with FilterContext)
  │     For each VideoFilterProvider:
  │       filter(frame, context) → frame or replacement
  │     (censor, watermark, YOLO detection, face detection)
  │
  ├─ Vision Analysis (async, periodic)
  │     VisionProvider.analyze_frame(frame) → VisionResult
  │     Updates FilterContext for subsequent frames
  │     Fires ON_VISION_RESULT hook
  │     Injects description into AI channel context
  │
  └─ Recorder tap (if recording active)

12.8.5 Pipeline Stage Providers

VideoDecoderProvider:

VideoDecoderProvider (interface)
├── name: string
├── decode(frame: VideoFrame) → VideoFrame | null  # Decode encoded → raw
├── reset() → void
└── close() → void

Implementations: PyAV (FFmpeg-based, H.264/VP8/VP9/AV1), Mock.

VideoResizerProvider:

VideoResizerProvider (interface)
├── name: string
├── resize(frame: VideoFrame) → VideoFrame
└── close() → void

Implementations: PyAV (FFmpeg scaling), Mock.

VideoEncoderProvider:

VideoEncoderProvider (interface)
├── name: string
├── encode(frame: VideoFrame) → list<bytes>  # Raw → encoded NAL units
├── flush() → list<bytes>                    # Drain buffered frames
├── reset() → void
└── close() → void

Implementations: PyAV (H.264 Constrained Baseline, default encoder libx264).

VideoTransformProvider:

VideoTransformProvider (interface)
├── name: string
├── transform(frame: VideoFrame) → VideoFrame
├── reset() → void
└── close() → void

Transforms modify pixels without changing frame dimensions. Multiple transforms are chained in order. Example: grayscale, blur, color effects.

VideoFilterProvider:

VideoFilterProvider (interface)
├── name: string
├── filter(frame: VideoFrame, context: FilterContext) → VideoFrame
├── reset() → void
└── close() → void

Filters inspect frames (using vision context) and MAY replace them. They receive a FilterContext with the latest vision analysis results.

FilterContext:

FilterContext
├── session_id: string
├── last_vision_result: VisionResult | null  # Updated by vision analysis
├── labels_detected: set<string>             # Flattened from vision result
├── censoring: bool                          # Whether active censoring
├── metadata: map<string, any>               # Arbitrary state
└── events: list<FilterEvent>                # Detection events for hook dispatch

Built-in filter implementations:

Filter	Purpose
CensorVideoFilter	Replaces frames when blocked labels detected (black/solid frame)
YOLODetectorFilter	YOLO object detection, updates labels, optional bounding boxes
WatermarkFilter	Adds watermark overlay to frames
FaceDetectionFilter	Face detection with bounding boxes

12.8.6 Overlay System

Overlays render text, images, and rich content on top of video frames:

Overlay
├── id: string                              # Unique overlay identifier
├── content: string                         # Text, image URL, or markup
├── overlay_type: string                    # "text", "image", "rich", "subtitle"
├── position: OverlayPosition              # Predefined or CUSTOM with x/y
├── x: int (default 0)                     # X offset (for CUSTOM position)
├── y: int (default 0)                     # Y offset (for CUSTOM position)
├── z_order: int (default 0)               # Stacking order (higher = on top)
├── opacity: float (default 1.0)           # 0.0 to 1.0
├── style: map<string, any>                # Renderer-specific styling
└── version: int (default 0)               # Cache invalidation

OverlayPosition enumeration: TOP_LEFT, TOP_CENTER, TOP_RIGHT, CENTER_LEFT, CENTER, CENTER_RIGHT, BOTTOM_LEFT, BOTTOM_CENTER, BOTTOM_RIGHT, CUSTOM.

OverlayRenderer (interface):

OverlayRenderer (interface)
├── overlay_type: string (property)         # Which overlay type this renders
├── render(canvas, overlay, frame_width, frame_height) → canvas
├── invalidate_cache(overlay_id) → void
└── clear_cache() → void

Implementations: TextOverlayRenderer, ImageOverlayRenderer, RichOverlayRenderer, SubtitleOverlayRenderer. The OverlayFilterProvider applies overlays to frames as a pipeline filter stage.

12.8.7 VisionProvider

Vision providers analyze video frames and produce structured results:

VisionProvider (interface)
├── name: string (property)
├── analyze_frame(frame, prompt, response_schema) → VisionResult
│       # Analyze a single frame (async); response_schema is optional
│
├── analyze_stream(frames, interval_ms, assumed_fps) → async_iterator<VisionResult>
│       # Streaming analysis at configurable intervals
│
├── supports_streaming: bool (property)
├── supports_response_schema: bool (property, default false)
├── warmup() → void                         # Pre-load models
└── close() → void

Response schema. analyze_frame MAY take a response_schema, under the rules of Section 6.7: the same portable subset, and the same ResponseSchemaError reasons. When the provider's supports_response_schema is true, the description it returns is one JSON document satisfying the schema, checked before it is returned; when it is false, a call carrying a schema is refused before any request. A consumer that locates or extracts something in a frame passes a schema where the provider supports one, instead of repairing free text.

VisionResult:

VisionResult
├── description: string                     # Natural language description
├── labels: list<string>                    # Detected objects/scenes
├── confidence: float                       # 0.0 to 1.0
├── faces: list<FaceDetection>             # Detected faces with bounding boxes
├── text: string | null                     # OCR text
└── metadata: map<string, any>              # Provider-specific data

FaceDetection:

FaceDetection
├── x: int                                  # Bounding box x
├── y: int                                  # Bounding box y
├── width: int
├── height: int
├── confidence: float
└── label: string | null                    # Optional identity label

Implementations:

Provider	Backend	Description
OpenAIVisionProvider	GPT-4o, Ollama, vLLM	OpenAI-compatible vision API
GeminiVisionProvider	Google Gemini	Google Gemini vision
MockVisionProvider	—	Testing

AI integration: what a vision provider saw reaches a room's AI channels with no wiring. It changes from one result to the next, so it rides the turn's notes, not the system prompt, and is not written into the room's binding: it is state of the turn, not configuration of the room (Section 6.4). An AI channel's turn carries the latest result a video channel attached to its room produced, whichever video channel, as a <vision> block (Section 6.4): the description, the objects detected and the text read in the frame are what a model or the camera saw, data that cannot close its block. A standalone turn (Section 10.1.1) carries none. The result belongs to the video session that produced it: once that session ends, no turn reads it. A realtime session that is given what the camera sees takes the same block as a silent injection. setup_video_vision, which wrote the result into one AI channel's system prompt, is deprecated and writes nothing.

12.8.8 AvatarProvider (Lip-Sync Video Generation)

Avatar providers generate lip-synced video from TTS audio:

AvatarProvider (interface)
├── name: string (property)
├── fps: float (property)                   # Output frame rate
├── is_async: bool (property)               # Whether delivery is callback-based
│
├── start(reference_image, width, height) → void
│       # Initialize with a reference face image
│
├── feed_audio(pcm_data, sample_rate) → list<VideoFrame>
│       # Feed TTS audio, return generated frames
│       # Sync mode: returns frames immediately
│       # Async mode: returns [], frames arrive via on_video() callback
│
├── on_video(callback) → void               # Register frame callback (async mode)
├── end_turn() → void                       # Signal end of TTS response
├── get_idle_frame() → VideoFrame | null    # Frame to display when TTS not playing
├── flush() → list<VideoFrame>              # Drain buffered frames
├── stop() → void
└── close() → void

Two delivery modes: - Synchronous (local models): feed_audio() returns frames immediately. - Asynchronous (cloud APIs): feed_audio() returns empty list; frames arrive via the on_video() callback.

In AudioVideoChannel, the avatar is wired as an outbound audio tap. TTS audio chunks are fed to the avatar, and the resulting video frames are encoded and sent via the video backend.

12.8.9 VideoBridge

The VideoBridge forwards video frames between sessions in the same room, mirroring the AudioBridge pattern:

VideoBridgeConfig
├── enabled: bool (default true)
├── max_participants: int (default 10)
├── forwarding_strategy: "forward"          # Direct frame forwarding
└── keyframe_interval_s: float (default 5.0)  # PLI request interval

VideoBridge
├── add_session(session, room_id, backend) → void
├── remove_session(session_id) → void
├── forward(session, frame) → void          # Forward to all other sessions in room
├── set_frame_filter(fn) → void             # Optional pre-forward filter (can drop)
├── set_frame_processor(fn) → void          # Optional per-target processor
├── get_participant_count(room_id) → int
├── get_bridged_sessions(room_id) → list<(VideoSession, VideoBackend)>
└── close() → void

Key behaviors: - Keyframe buffering for late joiners. - Periodic PLI (Picture Loss Indication) requests to ensure keyframe delivery. - Delta frame gating — waits for a keyframe before forwarding delta frames to a new session. - Thread-safe session registration and removal. - BEFORE_BRIDGE_VIDEO sync hook fires before forwarding (can block/modify).

12.8.10 Video Recording

VideoRecordingConfig
├── storage: string                         # Output directory
├── format: string (default "mp4")          # Container format
├── codec: string (default "auto")          # Video codec
├── fps: float (default 15.0)              # Output frame rate
└── metadata: map<string, any>

VideoRecorder (interface)
├── name: string (property)
├── start(session, config) → VideoRecordingHandle
├── stop(handle) → VideoRecordingResult
├── tap_frame(handle, frame) → void         # Feed frame to recorder
├── reset() → void                          # Reset internal state
└── close() → void                          # Release resources

VideoRecordingResult:

VideoRecordingResult
├── id: string
├── url: string                             # File path or URL
├── duration_seconds: float
├── frame_count: int
├── format: string
├── size_bytes: int
└── metadata: map<string, any>

Implementations: PyAVVideoRecorder (H.264/H.265, NVIDIA GPU), OpenCVVideoRecorder (MJPEG/XVID), MockVideoRecorder.

Scope. Like the audio recorder, this one records a session's video and is handed one — a conference publishes N attributed video tracks and no session, and records them through Section 12.11 instead (Section 12.10.8).

12.8.11 Video Hooks

Hook	Execution	When
ON_VIDEO_SESSION_STARTED	ASYNC	Video session became active
ON_VIDEO_SESSION_ENDED	ASYNC	Video session ended
ON_VIDEO_TRACK_ADDED	ASYNC	Video track added to session (Planned, Section 9.2)
ON_VIDEO_TRACK_REMOVED	ASYNC	Video track removed from session (Planned, Section 9.2)
ON_VISION_RESULT	SYNC	VisionProvider returned an analysis result — can block or modify it
ON_VIDEO_DETECTION	ASYNC	Filter emitted a detection event (YOLO, face, etc.)
ON_SCREEN_SHARE_STARTED	ASYNC	Screen sharing started; fired by the conference channel only, not by a video channel (Section 12.10.4)
ON_SCREEN_SHARE_STOPPED	ASYNC	Screen sharing stopped; fired by the conference channel only, not by a video channel (Section 12.10.4)
BEFORE_BRIDGE_VIDEO	SYNC	Before frame forwarded via bridge — can block

12.9 (Removed)

This section held a draft AI Generation Pipeline of pluggable stages. It was removed in v18: each step of an AI turn already has its own contract, and a stage that rewrites the whole response before it is sent cannot coexist with a streamed response. Section 6.4 (Anatomy of an AI turn) maps each step to the contract that carries it. The number is kept so that references to Sections 12.10–12.12 stay valid.

12.10 Conference (SFU Orchestration)

Status: STABLE. The interfaces in this section were validated on paper against the published server APIs of multiple SFU implementations, then revised against the first conforming production backend (LiveKit) as its implementation landed — the revision window the PROVISIONAL status existed for, now closed. The section follows normal stability rules, and every other section referencing conference concepts follows them for those references too.

A conference extends a Room with a multi-party real-time media session — audio, video, and screen share among N participants. The defining architectural decision is that RoomKit does not own the conference media plane. An external SFU (Selective Forwarding Unit) routes media between human participants; RoomKit orchestrates the conference and joins it as a participant to provide intelligence: transcription, vision, AI voice, recording, and cross-channel integration.

12.10.1 Design Principles

1. RoomKit orchestrates; the SFU transports. Per-packet media routing between human participants — codec negotiation, simulcast layer selection, bandwidth estimation, NACK/PLI retransmission — is the SFU's responsibility. RoomKit MUST NOT sit in the media path between human participants. This is the scale boundary that the in-process AudioBridge (Section 12.7) cannot cross.

2. The conference is the room. Each conference maps 1:1 to a Room. Conference participants are Room participants. Transcriptions are RoomEvents. Hooks, permissions, and cross-channel broadcast apply with no conference-specific exceptions.

3. Human clients connect directly to the SFU. RoomKit mints access credentials (Section 12.10.3) and observes participant lifecycle through backend events; it never proxies client signaling or media. Client-side SDKs are provider-specific and out of scope for this specification — switching SFU providers implies switching the integrator's client SDK.

4. RoomKit joins as a bot participant. The framework's media access goes through one server-side connection per conference: subscribed tracks feed the audio/video pipelines (STT, vision, recording), and the AI's TTS output is published as a single bot track that the SFU distributes to everyone.

5. Tracks are the unit of media identity. Every media stream carries the identity of its publishing participant. Speaker attribution comes from track identity, so diarization is unnecessary. Server-side AEC is likewise unnecessary: clients perform their own echo cancellation and the SFU never mixes audio.

6. The backend boundary is decoded frames and opaque credentials. The ConferenceBackend delivers decoded PCM audio and raw (or encoded) video frames, and mints credentials the framework never inspects. SDP, ICE, codecs, simulcast layers, and bitrates do not appear in any framework interface.

12.10.2 Data Models

TrackKind enumeration:

Value	Description
AUDIO	Microphone audio, or the sound of a screen share
VIDEO	Camera video
SCREEN_SHARE	Screen share video

ConferenceTrack — a single published media stream:

ConferenceTrack
├── id: string                          # Backend-scoped stable identifier
├── room_id: string                     # Owning conference room
├── participant_id: string              # Publishing participant
├── kind: TrackKind
├── muted: bool = false
└── metadata: map<string, any>          # Provider-specific (sid, source, ...)

room_id is what makes the frame callbacks routable: a single ConferenceBackend instance serves many rooms, and on_track_audio / on_track_video carry only a ConferenceTrack. Every ConferenceTrack a backend emits MUST carry the room it belongs to.

ConferenceParticipant — a participant's media presence:

ConferenceParticipant
├── participant_id: string
├── display_name: string | null                 # Presentation, never identity
├── connected_at: datetime
├── tracks: list<ConferenceTrack>
├── metadata: map<string, any>                  # Provider-supplied participant attributes
└── asserted_metadata: map<string, any> | null  # The subset the SFU itself asserts

A backend MUST surface the provider's own participant attributes in metadata. This is not decoration: for a participant the framework did not name, those attributes are where the resolvable address lives — a PSTN dial-in carries its caller number there, and that number is precisely what identity resolution consumes (Section 5.6, channel_addresses). Dropping them leaves the framework with an opaque identifier and no way to connect the caller to a known Identity.

Where those attributes came from (normative). metadata says what the provider surfaced; it does not say who put it there. On most SFUs one attribute map carries two very different things: facts the server established — the caller number a SIP trunk reported, a claim in a token it authenticated, an attribute the integrator set through a server-side API — and values a participant's own client supplied when it joined. Nothing in the map's shape tells them apart, and the difference decides what may be believed.

asserted_metadata is where a backend states the difference. It is the subset of metadata the SFU itself asserts. Therefore:

1. A backend MUST NOT place an attribute in asserted_metadata unless its value was established by the SFU or by a server-side call: the SIP trunk, the admission it authenticated, its own API. Values a client supplied at join — directly, or through a token it was free to populate — MUST NOT appear there, whatever their key.
2. A backend that cannot tell the two apart MUST leave asserted_metadata null rather than fill it with everything it has. Null is a statement — "this backend does not distinguish" — and a consumer can act on it. A guess is indistinguishable from an assertion, and it is the assertion that decisions get made on.
3. An empty map is the other statement: this backend distinguishes, and the SFU asserts nothing about this participant.

The specification names the fact rather than the policy. What an implementation may do with each of the two sets is stated where it consumes them — for identity resolution, under Resolving the participant the framework did not name below.

ConferenceGrants — least-privilege permissions encoded into access:

ConferenceGrants
├── publish_audio: bool = true
├── publish_video: bool = true
├── publish_screen_share: bool = true
├── subscribe: bool = true
├── moderate: bool = false              # Can mute/remove other participants
├── hidden: bool = false                # Invisible observer (bots, monitors)
└── publish_screen_share_audio: bool = false  # The sound of a screen share

publish_screen_share_audio covers the audio track a client publishes beside a screen share — a browser tab or a whole screen shared with its sound — which SFUs carry as a source of its own. It is the one publish permission that defaults to false, and the reason is compatibility rather than caution: a caller that never names the field mints no credential carrying the right, so grants written without it in mind stay exactly as narrow as they read. A backend MUST grant that source only through this field — never as part of publish_audio or publish_screen_share — and the two screen-share fields are independent: a share without sound needs only publish_screen_share, and granting one never implies the other. Once published, the sound is a track of kind AUDIO: a separate right, not a separate kind, so the framework transcribes and records it like any other audio track, attributed to the participant sharing.

ConferenceAccess — credentials for a human client to join:

ConferenceAccess
├── url: string                         # Endpoint the client connects to
├── token: string                       # Provider-specific credential
├── expires_at: datetime | null
└── provider_data: map<string, any>     # Additional provider-specific fields

The framework treats ConferenceAccess as opaque: the backend mints it, the integrator returns it to the client application (e.g., via the REST surface), and the provider's client SDK consumes it. Framework code MUST NOT depend on its internal structure beyond the fields above.

Participant identity correlation (normative). Every attribution guarantee in this section — transcription RoomEvents, Participant records, interruption allowlists — depends on participant_id meaning the same thing on both sides of the backend boundary. Therefore:

1. The framework MUST pass the Room Participant.id (Section 5.5) as the participant_id argument to mint_access().
2. For a participant that joined with minted access, the backend MUST surface that same value as ConferenceParticipant.participant_id and as ConferenceTrack.participant_id. A backend whose SFU cannot carry a caller-supplied identity MUST maintain the mapping internally and translate at the boundary.
3. For a participant the framework did not mint — SIP/PSTN dial-in (Section 12.10.9), or admission arranged out of band — the backend MUST surface its own stable identity for the lifetime of that participant's connection, and MUST populate metadata with the provider's participant attributes. The channel MUST then create a Participant with that identity as external_id and identification: UNKNOWN, and MUST pass any resolvable address the SFU asserts — a caller number above all — to identity resolution (Section 11) rather than resolving on the opaque identity alone. A phone participant joining a conference SHOULD reach the same Identity it would have reached over the SMS or Voice channel.
4. Identities MUST NOT be reused across participants within a room's lifetime.

Resolving the participant the framework did not name. Rule 3 says the address must reach identity resolution; when it does so, and what becomes of the answer, decide whether the rest of the section still holds.

Resolution MUST run when the participant arrives, not when it first speaks. A conference participant may listen for an hour without publishing a word, and one that stays unidentified until it does is unidentified to every hook, roster read and disclosure obligation in the meantime — which is the opposite of what rule 3 exists for. It also means resolution runs for a participant that never speaks at all.

The answer MUST be recorded on the Participant rule 3 created, whose id remains the backend's identity: the resolved Identity is carried by identity_id (and display_name), and the participant MUST NOT be re-keyed to the Identity's own id. Re-keying would break rule 2 in the one place it matters most — transcription RoomEvents would be attributed to the Identity while the recording's RecordingTrack.participant_id (Section 12.10.8), the interruption allowlist (Section 12.10.5) and the roster stayed on the backend's identity, leaving one human under two identifiers in one room. One participant record, one identifier, an Identity linked to it.

What counts as a resolvable address is provider-specific, so an implementation MUST document the participant-attribute keys it reads and SHOULD let an integrator override them — a provider whose key is not on the list is a configuration matter, not a reason to fork the framework. The number a caller dialled is not the caller: keys carrying the trunk or destination number (sip.trunkPhoneNumber and equivalents) MUST NOT be taken for the caller's address. Where no resolvable address is found, the channel MUST leave the participant UNKNOWN rather than resolving on the opaque identity, which is the case rule 3 rules out.

An address is only as good as its provenance. Which key carries the address is the second question; the first is who put the value there. An attribute a participant's own client supplied is a claim about itself: a caller that writes its own phone_number and is resolved on it reaches whatever Identity that number belongs to — someone else's — and the Participant then carries the victim's identity_id, on the record every later attribution reads. Therefore an implementation MUST resolve identity only on addresses found in asserted_metadata, and MUST NOT take an attribute the backend did not assert for an address. A null asserted_metadata asserts nothing: a backend that does not distinguish (preceding subsection, point 2) yields no resolvable address at all, and its participants stay UNKNOWN.

Provenance outranks specificity. An asserted attribute on a generic key is believed over an unasserted one on the provider's own key, because the attacker chooses the key and never the provenance.

This is a default, not a prohibition. An integrator whose deployment has its own reason to trust unasserted attributes — a closed client fleet, a backend whose provenance the integrator establishes elsewhere — MAY widen it, and an implementation SHOULD offer that as configuration rather than leave forking as the only route. What an implementation MUST NOT do is widen it silently: the safe reading is the one that holds unconfigured. Where an implementation carries the provider's attributes into resolution as context, the ones it did not vouch for MUST remain distinguishable from the ones it did, so that a resolver reading them does so knowingly.

What the Participant record keeps of them. Provider attributes are worth persisting — they are how an integrator finds out that a participant dialled in, on what trunk, under which SIP call id. But a Participant's metadata (Section 5.5) is the integrator's own map, and a conference is a place where strangers write into it. An implementation that records provider attributes on a Participant MUST therefore:

• keep them under a single dedicated key rather than merged flat, so that no provider attribute can overwrite a field the integrator put there;
• keep the provenance split of the preceding subsection intact in the record, so that what an identity was founded on remains auditable after the fact;
• bound what is persisted — the number of attributes and the size of each — since on a conference open to dial-in, their content is chosen by whoever connects.

An arrival is not an inbound message, and the parts of Section 11 that act on one do not apply to it: there is nothing to hold, nothing to reject and nowhere to inject a challenge, so an arrival MUST NOT fire the ON_IDENTITY_* hooks or run the challenge and rejection flows — those belong to the inbound pipeline, and the participant's first utterance reaches it normally. An AMBIGUOUS or PENDING result SHOULD be recorded as a pending identification carrying its candidates; UNKNOWN, REJECTED and CHALLENGE_SENT leave the participant UNKNOWN. Section 11.5 applies to the call: on timeout the implementation MUST treat the result as UNKNOWN, emit identity_timeout, and let the participant join regardless. A resolver that raises MUST NOT keep a participant out of the conference.

BotSession — the framework's own connection to a conference:

BotSession
├── id: string
├── room_id: string
├── identity: string                    # Display identity in the conference
├── joined_at: datetime                 # When the bot connected; MUST be timezone-aware
└── metadata: map<string, any>

joined_at is what conference_ended's duration_ms is measured from (Section 8.2). It defaults to the moment the session is constructed, which is when a backend builds the one it returns from join_as_bot(); a backend holding a more accurate figure — one the SFU reports — SHOULD set it instead. It MUST be timezone-aware: a naive value cannot be subtracted from an aware clock, and the failure surfaces during teardown where it costs the conference its end announcement rather than the duration alone. Implementations SHOULD settle the timezone when the session is recorded, so that a backend's omission is reported at the boundary it came from rather than raised from the teardown.

ConferenceCapability flags:

Flag	Description
SCREEN_SHARE	Separate screen-share tracks
EGRESS_RECORDING	Server-side (SFU) recording/export
SIP_GATEWAY	PSTN/SIP participants can dial into the conference
ACTIVE_SPEAKER	Dominant-speaker change events
CONNECTION_QUALITY	Per-participant quality reports
VIDEO_PUBLISH	Bot can publish video tracks (avatar)
REMOTE_UNMUTE	A moderator can unmute another participant's track
BOT_GRANT_UPDATE	A connected bot session's grants can be changed in place
E2EE	End-to-end encryption between clients

REMOTE_UNMUTE is a capability rather than an assumed operation because unmuting someone else's microphone is a privacy decision, not a technical one: SFUs commonly refuse it by default and require an explicit server-side opt-in. Muting is always available; unmuting is not.

BOT_GRANT_UPDATE says the SFU can change what a connected session may do without reconnecting it — a server-side participant update (LiveKit's UpdateParticipant is the informative example). It is a capability because many SFUs can only set permissions at admission: against those, the one way to change a live bot's grants is to replace the session, and Section 12.10.4 (Hot-plugging intelligence) specifies exactly that fallback. What the capability buys is continuity — a re-permission with the session, its subscriptions and the event bridge intact.

E2EE and framework media access (normative). E2EE is the one capability that constrains rather than extends what the framework can do: when it is active, the bot receives ciphertext it cannot decode, so STT lanes, vision analysis, framework recording, and SFU egress recording all become unavailable.

The flag above states only that a backend supports E2EE. Whether a given conference uses it is per-conference state, requested through the channel's e2ee field (Section 12.10.4) and passed to ensure_room(); setting it on a backend without the E2EE capability MUST raise a configuration error. An implementation offering E2EE MUST then do one of:

• admit the bot as a key holder in the conference's key exchange, so subscribed tracks decode normally; or
• run the conference without framework media intelligence, and raise a configuration error when STT, vision, or recording is configured on a channel with e2ee set.

Silently delivering undecodable frames to a lane is NOT conforming.

12.10.3 ConferenceBackend Interface

ConferenceBackend (interface)
├── name: string (property)             # Backend identifier
├── capabilities: ConferenceCapability
│
│   # Control plane:
├── ensure_room(room_id, metadata?, e2ee?) → void
│       # Idempotent: create the conference room if absent; e2ee
│       # (default false) requests E2EE (Section 12.10.2)
├── close_room(room_id) → void
├── mint_access(room_id, participant_id, grants, display_name?, attributes?) → ConferenceAccess
├── list_participants(room_id) → list<ConferenceParticipant>
├── remove_participant(room_id, participant_id) → void
├── mute_track(room_id, track_id) → void     # Moderation mute
├── unmute_track(room_id, track_id) → void   # Requires REMOTE_UNMUTE
│
│   # Bot participant (framework media access):
├── join_as_bot(room_id, identity, grants) → BotSession
├── leave(bot) → void
├── update_bot_grants(bot, grants) → void         # Requires BOT_GRANT_UPDATE
├── subscribe_track(bot, track_id) → void
├── unsubscribe_track(bot, track_id) → void
├── publish_audio(bot, chunk: AudioChunk) → void
├── stop_playback(bot) → void                      # Barge-in: discard queued bot audio
├── publish_video(bot, frame: VideoFrame) → void   # Requires VIDEO_PUBLISH
│
│   # Callbacks:
├── on_participant_joined(callback)     # (room_id, ConferenceParticipant)
├── on_participant_left(callback)       # (room_id, ConferenceParticipant)
├── on_track_published(callback)        # (room_id, ConferenceTrack)
├── on_track_unpublished(callback)      # (room_id, ConferenceTrack)
├── on_track_muted(callback)            # (room_id, ConferenceTrack)
├── on_track_unmuted(callback)          # (room_id, ConferenceTrack)
├── on_track_audio(callback)            # (ConferenceTrack, AudioFrame)
├── on_track_video(callback)            # (ConferenceTrack, VideoFrame)
├── on_active_speaker_changed(callback) # (room_id, participant_id)
├── on_connection_quality(callback)     # (room_id, participant_id, quality)
├── on_bot_session_ended(callback)      # (BotSession, reason)
└── close() → void

Presence is observable only through a connection (normative): the participant, track, audio and speaker callbacks describe a conference that a bot session of this backend is connected to. Before the first join_as_bot() — and between a reported end and a re-join — a backend is not required to deliver any of them, and against most SFUs it cannot: the server reports a room's events to the connections in it, and the framework does not hold one. A backend with a server-side view of its own — webhooks, a polling loop — MAY report presence for a room no session is connected to, but a channel MUST NOT depend on such reports for anything correctness rests on, the first join above all (Section 12.10.4): a channel whose only bootstrap is a presence callback waits on an event that only the join it is waiting to make would let the backend observe.

What a newly joined session is owed is the present, not the past. A lazily joining bot enters a meeting already underway, and the participants and tracks it finds there may never have been reported — there was no connection to report them through. On join_as_bot(), a backend MUST therefore catch the channel up: every participant currently in the room and every track currently published that it has not already reported is delivered through the ordinary joined and published callbacks — the catch-up the reported-discontinuity rules below lean on. A backend that observed the room without a connection may have nothing left to say; one that could not observe it reports everything it finds. What ended before the join — an arrival that left again, a track unpublished — is not recoverable, exactly as across a reported discontinuity.

The bot's own connection (normative): an SFU can end the bot's session without a leave() — the connection drops, a moderator evicts the bot, the room is deleted under it. A backend that observes such an end MUST report it through on_bot_session_ended, naming the session and a human-readable reason, MUST forget the session (a later leave() for it is a no-op), and MUST NOT accept further media calls for it. The channel MUST treat the report as the session's end in fact: the session comes off the books — bot_present stops answering yes for a connection that is gone — conference_ended is announced, the session's lanes are closed and its recordings finalized, and the next need re-joins lazily, exactly as the first one did. Without this report a dropped bot is a session the channel reports present forever and a conference that has silently lost its transcription; a backend that genuinely cannot observe the loss has nothing to report, and inherits that failure mode knowingly.

The report is also the honest exit for a session whose view of the conference can no longer be trusted — an event bridge that overflowed, a state divergence the backend detects. A backend MUST NOT drop a participant or track lifecycle event silently: past whatever bound it keeps, it ends the session with a reason naming the loss instead. When the session being ended still has a live connection — an overflow is not a dropped link — the end MUST NOT be reported until the backend has actually disconnected it: the report empties the registry and seats a re-joined replacement, and issued early it puts two bots in one meeting. A session whose disconnect cannot be completed is kept — registered, reported present, refusing a replacement — and a later leave() retries.

Such an end is a reported discontinuity, and this specification is explicit about what it does and does not recover. The events queued and undelivered at the end are discarded, counted, and named in the reason; the re-join's catch-up announces the conference's current state. What happened entirely inside the outage window — a participant who joined and left while the consumer was stalled — is not recoverable, by the catch-up or by anything else. The per-participant lifecycle obligations of this section apply to the events the backend delivered; across a reported discontinuity, an implementation with disclosure or admission obligations (Section 17.7) MUST treat the window as unaccounted rather than as observed-and-empty — the reason string is the signal to do so.

A channel SHOULD attempt a bounded, backed-off re-join after any reported end while the room remains attached and collecting — the ended session was what received the frames and events, so no backend event can produce the lazy join's "next need"; a later mint or delivery could, but mid-meeting there may never be another of either.

Frame delivery requirements:

• on_track_audio MUST deliver decoded PCM as AudioFrames with a declared sample rate. The backend owns decoding (Opus, etc.).
• on_track_video SHOULD deliver raw frames. Backends MAY deliver encoded frames; the channel's VideoPipeline decoder stage (Section 12.8.4) then applies, reusing the existing is_encoded mechanic.
• Frames MUST be attributable: the ConferenceTrack passed with each frame carries the publishing participant_id and room_id.

Frame publishing requirements:

• publish_audio takes an AudioChunk (Section 12.3.11) — the outbound stream type, carrying decoded PCM rather than an encoded payload. AudioChunk names its own encoding in format, and a conference backend MUST reject a chunk that is not PCM rather than forwarding it to the SFU: encoding belongs to the backend, and a caller choosing the wire format would defeat the abstraction boundary below.
• publish_video takes a raw VideoFrame (is_raw true). The backend owns encoding, symmetrically with the decoding it owns on the inbound side. A backend MUST NOT require the framework to supply an encoded VideoChunk: not because the framework cannot encode — VideoEncoderProvider (Section 12.8.5) exists for VideoChannel's own transport path — but because deciding which codec to hand an SFU would put codec selection back into a framework interface, which the abstraction boundary below forbids. Avatar and other bot-video producers therefore emit raw frames.

Inbound tolerates encoded frames while outbound does not, and the asymmetry is deliberate: an encoded inbound frame is self-describing and the VideoPipeline decoder stage handles whatever arrives, whereas an encoded outbound frame requires choosing a codec the SFU will accept before any frame exists.

Stopping playback (normative): stop_playback() is the barge-in gesture — the one call that says the room asked for silence now, rather than at the end of whatever the transport has buffered. A participant who cut the bot off is not asking for the queue to finish playing.

• A backend MUST immediately discard the bot-track audio it has accepted through publish_audio() but not yet delivered for playout. Audio that has already left the backend — buffered in the SFU or at clients — is beyond recall; a backend that queues locally SHOULD keep that queue small enough that the residue still reads as responsive.
• The call does NOT end the utterance. is_final remains the only boundary (Section 12.10.4): the closing chunk still follows, and a backend MUST accept it after the stop rather than treating the stop as having closed the utterance itself.
• The bot's track MUST remain usable: the next utterance publishes normally.
• A stop for a session the backend no longer holds — left, evicted, dropped — is a no-op, not an error: the silence the call asks for is already true of a session that is gone. This is deliberately unlike publish_audio, which refuses media for a session that is out; a refusal there protects a track from writes, and there is no track left to protect.
• There is deliberately no capability flag for this call. Every backend can discard at least what it still holds, and one that holds nothing — publishing synchronously into the SFU — simply returns. What varies between backends is the residue, not the meaning of the call.

Mute transitions (normative): ConferenceTrack.muted is the publisher's own state, and a backend able to observe it change MUST keep the record true to it and SHOULD report each transition through on_track_muted / on_track_unmuted, with the record already updated when the callback runs. The transitions are presence, not media — most clients express "camera off" as a muted VIDEO track rather than an unpublish, so a mute is often the only signal a camera toggle produces — and they are subject to the same rule as every other callback here: observable only through a connection.

Track subscription: the framework's subscription set is authoritative. subscribe_track() / unsubscribe_track() are the only mechanism by which the bot begins or stops receiving a track's frames, and a backend MUST NOT deliver on_track_audio / on_track_video for a track the bot has not subscribed to. Backends whose SDK auto-subscribes by default MUST disable that behaviour for the bot session. The channel's subscription policy is specified in Section 12.10.4.

Bot grants: join_as_bot() takes ConferenceGrants like mint_access() does, because the bot's own permissions vary by AI participation pattern (Section 12.10.6): a speaking bot needs publish_audio, an Observer bot is subscribe-only with hidden set. A backend MUST apply them to the bot session rather than assuming full permissions.

Where the permissive ConferenceGrants defaults are right for humans — whose needs the framework cannot know — they are wrong for the bot, whose behaviour the framework configured and therefore knows. An implementation that derives the bot's grants SHOULD derive every one of them, subscribe included: a channel with nothing to consume the tracks it would receive subscribes to none (Selective subscription), and the grant is then permission to receive every participant's media for nobody to read.

With BOT_GRANT_UPDATE, update_bot_grants() replaces a connected session's grants in place: same session, same connection, subscriptions and callbacks undisturbed — the SFU changes what the session may do, which is what lets a hot-plugged voice speak without a re-join (Section 12.10.4, Hot-plugging intelligence). Calling it on a backend without the capability MUST raise a configuration error rather than fail silently or appear to succeed — the unmute_track() rule, for the same reason. A backend MUST treat the grants it is given as the session's whole grant set, not a delta.

hidden rides the grant set like every other field, and its two directions are not equals. A backend SHOULD deliver a session's hidden-to-visible transition to the clients already connected — LiveKit announces the participant to them as newly joined, which is the behaviour the channel's reveal path leans on (Section 12.10.4). No backend is expected to retract a participant its SFU already announced: there is no interface for un-telling a client, so the channel never asks — a visible-to-hidden transition replaces the session instead (Section 12.10.4).

Display names: mint_access() MAY be given the participant's display_name, and a channel with a roster SHOULD pass the one on record. It is presentation, never identity: attribution rides participant_id alone, and a backend MUST NOT let a name stand in for it. A backend SHOULD carry the name into the credential so the SFU's own clients render the participant as the room named them, and SHOULD report it back on ConferenceParticipant.display_name. A channel SHOULD use a reported name to fill a roster record that has none — never to overwrite one the integrator set — which is what returns names to a roster rebuilt from the join's catch-up after a restart (Section 12.10.4 step 1): the credential outlives the process that minted it, and the name rides the credential.

Participant attributes: mint_access() MAY be given attributes, a map of string keys to string values the caller wants carried on the participant it is admitting. It exists because the identity is otherwise the only field that travels, and an integrator whose clients must be told who is behind a channel identity has nowhere else to put it — which turns participant_id into a format to parse, and loses the separation Section 5.5 draws between a channel identity and the Identity linked to it. A backend whose SFU carries per-participant attributes SHOULD put them in the credential and MUST surface whatever the SFU then reports on ConferenceParticipant.metadata, by the same reading as any other attribute. One whose SFU carries none ignores them; nothing downstream depends on their arrival.

They are surfaced, not vouched for. A backend MUST NOT place an attribute in asserted_metadata merely because it rode a credential that backend minted: the deployment may issue other credentials, and an SFU that lets a client rewrite its own attributes after joining leaves nothing to tell the two apart at read time. Point 1 of the asserted_metadata rules of Section 12.10.2 governs unchanged, and an implementation that cannot establish the value independently MUST leave it unasserted — where it can be read and rendered, but cannot found an identity.

What is minted is the caller's decision, per mint. A channel MUST NOT derive attributes of its own from what it knows about the participant — an identity_id a channel added unasked would publish the platform identity of everyone in the room to every peer of a conference that may be pseudonymous. And a channel MUST bound what it accepts to emit by what it would accept to persist (Section 12.10.2, provider attributes): emitting what the same implementation would refuse to store is a credential whose contents cannot survive the round trip. Refusing is RECOMMENDED over truncating here — the caller is the integrator, which is the one party in this exchange that can be told.

Moderation: mute_track() is always available. unmute_track() requires the REMOTE_UNMUTE capability; calling it on a backend without that capability MUST raise a configuration error rather than failing silently or appearing to succeed. Implementations MUST NOT assume the two operations are symmetric, and SHOULD surface the asymmetry to the integrator — a moderation UI that offers unmute against a backend that refuses it is a worse outcome than one that never offers it.

Abstraction boundary (normative): the interface deliberately omits SDP negotiation, ICE, codec selection, simulcast/SVC layer control, and bitrate management. A backend MUST NOT require the integrator or the framework to handle any of these; they are internal to the SFU and its client SDKs.

Required implementations:

Backend	Purpose
MockConferenceBackend	Unit testing with scripted participant/track event sequences

Implementations MAY provide backends for any SFU whose server API can satisfy this interface (e.g., LiveKit, Janus, Daily — informative).

12.10.4 ConferenceChannel

The ConferenceChannel orchestrates a conference for a room. It owns the ConferenceBackend, demultiplexes tracks into the existing audio and video pipelines, and represents the AI's voice in the conference.

ConferenceChannel
├── channel_type: CONFERENCE
├── category: TRANSPORT
├── direction: BIDIRECTIONAL
├── backend: ConferenceBackend
├── stt: STTProvider | null             # Per-track transcription
├── tts: TTSProvider | null             # AI voice via bot track
├── realtime: ConferenceRealtimeConfig | null   # Speech-to-speech (Section 12.10.12); excludes tts
├── pipeline: AudioPipelineConfig | null   # Lane stages with stt or realtime; null = default VAD and contract
├── vision: VisionProvider | null       # Video/screen-share analysis (Planned)
├── interruption: ConferenceInterruptionConfig
├── recording: ConferenceRecordingConfig | null   # Section 12.10.8
├── recorder: MediaRecorder | null      # Section 12.11; required with recording, refused without it
├── bot_identity: string                # Bot's display identity
├── bot_grants: ConferenceGrants | null # Explicit grants for join_as_bot(); null = derived (Section 12.10.3)
├── default_grants: ConferenceGrants    # Grants minted for participants
├── e2ee: bool = false                  # Request E2EE (requires E2EE capability)
├── close_room_on_detach: bool = false  # Whether detach calls close_room()
└── speak_text_events: bool = false     # Speak inbound text from other channels

Lifecycle:

1. When the channel is attached to a room, the channel MUST call ensure_room(). It SHOULD call join_as_bot(room_id, bot_identity, bot_grants) lazily — on the first successful mint_access(), the first delivery, or the first participant or track event a backend able to observe one reports — and MUST fire ON_SESSION_STARTED and emit conference_started once the bot connection is live. Whatever the set of triggers, it MUST include at least one signal that does not depend on the backend's callbacks, and the first successful mint_access() is that signal: presence is observable only through a connection (Section 12.10.3), so a channel whose first join waits on an arrival is waiting on a callback that only the join itself would unlock — the bot never enters, and a meeting where humans speak and the AI is meant to listen is never transcribed. A mint is the framework's own advance notice that a human is about to connect, and it reaches the channel without the backend's help. The join it starts MUST NOT delay the mint's answer and MUST NOT fail the mint when the join itself fails: the credential belongs to the participant regardless of whether the framework got its own session into the room.

The mint bootstraps a conference nobody has yet been admitted to; it cannot resume one already underway. An attach can land over a live conference — a channel restarted mid-meeting re-attaches above participants an earlier life admitted — and every trigger above is then out of reach: any re-join supervisor died with the process, no callback can arrive without a connection, and the humans already in the room may never mint again nor be delivered to. The trigger set MUST therefore also cover the conference's present occupancy at attach: after ensure_room(), the channel asks whether the conference already holds participants other than its own bot — list_participants() is the control-plane answer that requires no connection (Section 12.10.3); a persistent roster whose conference participants are still active MAY stand in for it or add to it — and a non-empty answer is a first need like any other. The probe and the join it may start MUST NOT delay the attach's answer, and their failure MUST NOT fail an attach the backend accepted: the lazy join remains behind them. Laziness is preserved — an empty conference stays unjoined, and a room nobody confers in costs one control-plane call per attach and nothing more. Without this trigger, a restart over a running meeting leaves it unheard for as long as nobody new is admitted, and the unaccounted window of Section 17.7 stretches without bound.

A host MAY request the join itself (ensure_bot(room_id)), a first need like the others for a host that must know the bot is in before it lets a meeting begin. Unlike the triggers above, it waits for the join and returns the bot's session: a live session is returned as it is, a lost one is joined again, the request is serialised with every other trigger so the room is joined once, and a new join fires ON_SESSION_STARTED and emits conference_started before it returns. It fails with RoomNotAttachedError when the channel is not attached to the room or is detached while joining, and with the backend's error when join_as_bot() fails. A channel configured with nothing to consume or say refuses it (ConferenceCapabilityError) rather than join: its bot would be the silent observer this section declines to make a mode.

The join exists for the intelligence. The bot's session is the framework's media access (Section 12.10.1 principle 4): subscribed tracks feed the pipelines through it, and the AI's voice is published on it. First need is therefore only a need when something configured on the channel consumes media — stt, vision, recording — or can speak — tts. A channel configured with none of these MUST NOT join on a mint, an arrival, or the occupancy probe, and SHOULD NOT make the probe at all: the join is the only consequence a probe can have. A delivery needs no guard of its own — without tts there is nothing to speak it with. An explicit bot_grants creates no need either: grants say what the SFU would let the bot do, not what the channel was configured to do, and a session justified by its privileges alone is a participant with no function — one server connection per conference, an unexplained name on the meeting's roster, and a Section 17.7 disclosure surface reporting a bot that collects nothing. Everything that is not the join stands: ensure_room() is still called, mint_access() still admits, and the participant records of step 2 are still written — the channel remains the room's admission gate and roster, which is exactly what a pure-transport deployment asks of it. The price is stated rather than hidden: the bot's connection is the event bridge (Section 12.10.3), so with a backend that observes presence only through a connection, a conference this channel never joins produces no participant, track, speaker or quality callbacks, and the unaccounted window of Section 17.7 spans the whole meeting. An integrator whose obligations require attendance MUST configure a consumer or track attendance through its own client surface — joining a bot with nothing to consume just to keep the callbacks is the silent observer Section 17.7 exists to surface, and this specification declines to make it a mode. The standing-down is a state, not a sentence: the configuration it is read from can change while the channel runs, and Hot-plugging intelligence (below) is how a need arrives mid-meeting — bringing the join, the probe and the grants with it. 2. on_participant_joined MUST create or update the corresponding Room Participant record (Section 5.5, correlated per Section 12.10.2) and fire ON_CONFERENCE_PARTICIPANT_JOINED. 3. on_track_published MUST fire ON_CONFERENCE_TRACK_PUBLISHED, and MUST start a per-track processing lane (below) for tracks the channel subscribes to — an unsubscribed track never yields a frame, so it gets no lane (see Selective subscription). For SCREEN_SHARE tracks, ON_SCREEN_SHARE_STARTED fires additionally. 4. on_track_unpublished MUST tear down the lane, unsubscribe if subscribed, and fire the corresponding hooks. 5. When the channel is detached, the channel MUST call leave() and emit conference_ended if a bot session is active — the lazy join of step 1 may never have run — and MUST call close_room() when close_room_on_detach is set, whether or not a bot ever joined. The detach first closes the room to new work, then waits for the work already in flight for it — announcements and the hooks they run, publications on the bot track, mints — on one bounded budget for the whole wait. Past the budget the teardown goes ahead without what remains, and a mint still outstanding is taken back as Admitting participants requires. A detach issued from inside such work does not wait for it: admission closes at once, and the rest of the teardown runs once that work has ended or the same bound has passed. The speech-to-speech session, the lanes and the recordings are closed before leave(), and conference_ended is announced last.

The SFU's own signals about the conference are relayed on their hooks while the bot's session is connected: on_active_speaker_changed fires ON_ACTIVE_SPEAKER_CHANGED, on_connection_quality fires ON_CONNECTION_QUALITY_CHANGED, and on_track_muted / on_track_unmuted fire ON_CONFERENCE_TRACK_MUTED / ON_CONFERENCE_TRACK_UNMUTED — each naming the participant, and the mute pair the track and its kind, since a muted VIDEO track is how most clients say "camera off". None of these is collection — no media is read to relay them — so none is gated by the binding's collection state, and the bot's own tracks are excluded exactly as they are from every other track event.

One conference per room (normative): principle 2's mapping is 1:1 in both directions, and the attachment is where that is enforceable: implementations MUST refuse to attach a conference channel to a room that already has a channel of type CONFERENCE attached. A second conference channel is a second bot session, a second transcription of every utterance and a second AI voice speaking the same deliveries — duplicates the roster, the transcript and the meeting have no way to express. Re-attaching the same conference channel is not a second conference; it remains an ordinary attach over a live attachment.

The reservation outlives the binding. A detach removes the binding at its start and takes the bot out at its end, and a teardown can be deferred past the detach that asked for it — so a check that reads only the bindings admits a second conference while the first one's bot is still in the meeting. The refusal MUST therefore hold for as long as the previous conference channel still holds the room — a session active or on its books, a teardown still running — not merely while its binding exists. And it is a refusal, not a wait: the attach may be issued from inside the very announcement the teardown is deferred behind, where waiting is a deadlock. The caller retries once the teardown has ended.

Bot self-exclusion (normative): the bot is itself a conference participant, and some backends report it back through on_participant_joined or echo its published track. The channel MUST ignore participant and track events whose participant identity matches the active BotSession, MUST NOT create a Participant record for it, and MUST NOT subscribe to a track it published. Without this rule the bot's own TTS returns through a lane, the STT transcribes the AI's own speech, and the AIChannel answers itself — a feedback loop the chain depth limit (Section 8.3) would only bound, not prevent.

Closing and shared state (normative): a conference channel's bookkeeping lives in resources it does not own — the conversation store and the room lock manager belong to the framework, which releases them after the channels close. That order faces both ways, and each direction carries an obligation.

The channel's: close() MUST NOT wait without bound on anything the channel does not own. Its own media plane comes first — admission closed, playbacks stopped, the bot out of the conference, the backend closed — under bounded budgets throughout, and the media calls are no exception: leave(), the backend's own close and a lane's recogniser all end in code the channel does not own, so each is cancelled past its budget, given a bounded grace to unwind, and then abandoned and reported rather than waited for again. Nothing an abandoned call was using is freed on its account, and past the budgets the bookkeeping is not the channel's to wait for either. The framework closes channels in sequence, so a channel waiting without bound is not spending its own time: it is holding every channel behind it in its conference, which is the failure the budgets exist to prevent.

There is one logical shutdown per channel. A close() that overlaps another MUST join the shutdown already running rather than start a second: concurrent callers await one shared attempt, and a caller being cancelled cancels only its own wait — never the shutdown, whose steps the other callers and the channel's own invariants still depend on. Once the shutdown reaches its terminal result, that result is the channel's answer for good: after a success, later calls MUST return immediately; after a failure, later calls MUST reproduce the same terminal failure rather than run the steps again. close() is a report of what the one shutdown achieved, not a retry of it — retrying what failed (removing a session an SFU refuses to release) is the operator's task the failure names.

Departures are exact-once. A session has at most one leave() in flight, whatever path asked for it — a detach, an abandoned join, the channel's close: a path that finds a departure already running MUST join that attempt instead of issuing a second leave() for the same session. Two concurrent leave() calls for one session ask the backend to remove a participant twice, and whichever answer arrives second is about a session that no longer exists.

An operation the shutdown abandons keeps what it was using. Every resource an abandoned operation holds — the backend under a leave() that swallowed its cancellation, the pipeline and recogniser under a lane's recogniser call, the recorder under a finalisation — MUST stay open until the operation has genuinely ended, however long past the budget that is; the resource is closed then, off the shutdown's clock. The current close() does not wait for it: the resource is reported as retained and the close fails explicitly, because a close that neither freed a resource nor said so has misreported the channel's state.

A close has succeeded only when all of the following hold: no session remains on the channel's books; no operation the channel admitted is still running; every resource the channel owns is closed; and every failure met along the way was surfaced in the close's result rather than in a log alone. A close missing any of these MUST fail, whatever its steps individually reported.

Failing to remove a session is failing to close. A session still on the channel's books when every step has run — a leave() the SFU refused, or one a budget had to cancel — MUST be raised, by name, at the very end of close(), not summarised into a log: a close that returns cleanly while a bot may still be listening to a meeting misstates the one thing the roster exists to answer. The channel goes on reporting the session, and removing it is an operator's task. For the same reason a channel whose close() fails MUST NOT prevent the channels after it from closing — and MUST NOT be reported as a success either. The failure is surfaced to the caller once the shutdown has run to completion: a channel that failed to close may still be holding its media — a bot possibly still in a meeting, listening — and a shutdown that returns cleanly over that turns a logged error into an operational and disclosure risk.

The framework's: close() MUST NOT release the store or the lock manager while an operation issued through them is still in flight. An operation a shared resource has been given cannot be taken back, and releasing the resource underneath it does not stop the work — it arranges a failure for a write that was already accepted. So the wait for such operations belongs to the owner of the resources, and it comes after every channel has closed and every channel's media is released, where a slow store costs the shutdown its latency and nothing else. There it is bounded by nothing, because there is no third option: giving up on the wait is releasing the resource under the operation.

Two boundaries keep that promise finite. It covers the operations the framework and its channels start of their own accord — participant callbacks, roster bookkeeping, announcements — which never suspend in integrator code while holding the resources, so the wait terminates. An operation the integrator starts through the public API is the integrator's to order against close(), as with any resource-owning library: the framework does not hold its shutdown open for a call the caller left suspended across it. And the wait has an end: once it concludes, the resources are sealed. Work that was suspended somewhere the shutdown could not see — a callback parked in a backend past every closing budget — and that resumes afterwards MUST be refused with an error naming the shutdown, never run against a resource being released.

Admitting participants: the channel exposes mint_access() for a participant of the room, applying default_grants unless the caller overrides them. The integrator delivers the resulting ConferenceAccess to the client application through its own surface (Section 16); the framework does not serve it. ConferenceGrants defaults are permissive so the common case works unconfigured; narrowing them is the integrator's call and is RECOMMENDED wherever a role does not need to publish — a listener-only attendee SHOULD receive grants without publish permissions (Section 17.7). The caller MAY also pass attributes for the backend to carry on the participant (Section 12.10.3); the channel passes them through, bounds them as it bounds the provider attributes it persists, and adds none of its own. The caller MAY pass the organization_id it acts for: the room is read with it before anything is admitted, and another organization's room is not found (Section 17.2).

mint_access() MUST NOT issue a credential for a room the channel is no longer attached to, and the check MUST hold across every await the call makes. Minting is not a read: the token is honoured by the SFU, the ConferenceBackend contract offers no revocation, and a check that was true before a roster lookup and false after it has already handed out admission to a conference the framework has left. Implementations MUST therefore treat the mint as work a detach cannot land in the middle of — the in-flight work a detach waits for (step 5 of the lifecycle above) — rather than as a precondition tested once.

A bounded drain does not satisfy this on its own. Every other kind of in-flight work degrades gracefully past the deadline — a chunk arrives late, an event lands out of order — and a credential does not: it stays valid for as long as it says it does, against a conference the framework has left. So a mint still in flight when the deadline passes MUST be taken back rather than left behind: implementations MUST cancel the outstanding backend request, and MUST re-read the attachment when a request that survived cancellation returns, refusing to hand the credential on. Refusing is not revocation, and implementations SHOULD say so — the backend may have recorded a credential nobody received, and only the operator can decide whether it needs revoking there. close() carries the same obligation before it closes the backend.

Per-track audio lanes:

on_track_audio (one lane per AUDIO track)
  → Resampler → [AGC] → [Denoiser] → VAD → STT (streaming)
  → transcription RoomEvents attributed to the publishing participant

The lane preserves the inbound stage ordering of Section 12.3, minus the stages a conference makes unnecessary: AEC (no server-side echo path) and Diarization (track identity already attributes speech).

A transcription enters the inbound pipeline under the identity its track was published under, which is a room Participant.id and not an address. Identity resolution therefore MUST NOT run per utterance (Section 11.6, case 1): a conference resolves when a participant arrives and the address its provider attached is there to resolve (Section 12.10.2), and speaking again asks nothing new. Running it anyway resolves on the backend's opaque identity — what rule 3 rules out — and fails as rule 3 says it would: UNKNOWN every time, so ON_IDENTITY_UNKNOWN fires per utterance and an implementation refusing unknown senders there discards the transcripts of a participant it identified on arrival.

Each AUDIO track gets its own lane, holding its stage state under the stream identity contract of Section 12.3 with the track as the stream key. That independence is a property of the state, not of the instances: one pipeline serves every lane. Instantiating a pipeline per track buys no isolation the stream key does not already provide, and costs a denoiser model loaded once per participant.

Within a lane:

• Format normalisation MUST run before the other stages. Participants negotiate their own formats with the SFU, so tracks arrive in whatever each publisher sent, while every stage downstream assumes one format. This is the Resampler position of Section 12.3.
• VAD MUST be present. It is what divides the stream into utterances, and without it the lane has no utterance to speak of: it calls the recognizer once per frame, spending a round trip per 20 ms and producing a transcript cut at frame boundaries instead of at turns. An implementation given a recognizer and no VAD MUST refuse the configuration rather than degrade to per-frame recognition.
• AEC MUST NOT be required (no server-side echo path — Section 12.10.1).
• Diarization MUST NOT be required (track identity provides attribution).
• AGC and Denoiser MAY apply per lane.
• ON_TRANSCRIPTION fires per lane; the hook context identifies the track and participant.
• ON_SPEECH_START and ON_SPEECH_END fire per lane at the VAD's own utterance boundaries, naming the publishing participant and track. They are the real-time answer to "who is speaking right now": the dominant-speaker signal cannot say that nobody is (its interface carries one identity), and the transcription arrives only after the recognizer's round trip. An unsubscribed track has no lane and therefore no speech events — this is collection, and the binding's gate applies.

Lane isolation: processing one track MUST NOT delay frame delivery for any other. A backend hands frames to its subscribers in sequence, so a lane that does its work — speech recognition above all — inside the delivery callback makes one provider's latency into every participant's latency, and one slow track stalls the whole conference. A lane therefore accepts a frame and returns, doing its work on its own schedule.

This is what separates a lane from a callback, and unlike the stage list it is observable from outside: an implementation can be checked by delaying recognition on one track and measuring frame delivery on another.

Overload: a lane accepting frames faster than it processes them MUST bound what it holds. An unbounded backlog turns a slow recognizer into unbounded memory and a lag behind the live conversation that only grows.

Which audio to discard is a genuine trade-off, and this specification does not settle it. Dropping the oldest frames keeps the lane near live at the cost of a gap mid-utterance; dropping the utterance in progress and resynchronising loses more audio but never emits a transcript stitched across a hole. Implementations MUST document which they do, and MUST expose how much was discarded — an integrator who cannot see the loss will read a damaged transcript as a bad recognizer.

Per-track video lanes:

Planned. No implementation analyses conference video yet: the reference channel consumes AUDIO tracks only. When it does, VIDEO and SCREEN_SHARE tracks route through the VideoPipeline (Section 12.8.4) — decoder (if needed), transforms, filters, vision analysis — and VisionResults are attributed to the publishing participant and reach the AI turns as the <vision> block of Section 12.8.7.

Selective subscription: the channel MUST call subscribe_track() only for tracks it consumes, and MUST NOT rely on backend auto-subscription (Section 12.10.3). AUDIO tracks are subscribed when STT, recording, or speech-to-speech composition is configured. VIDEO and SCREEN_SHARE tracks MUST NOT be subscribed unless a VisionProvider or framework-side video recording is configured — otherwise no video frame ever reaches the framework process. Tracks published by the bot itself are never subscribed (Bot self-exclusion above). Backends MAY subscribe to a reduced simulcast layer for vision analysis; this is provider-internal.

Subscription is re-evaluated when configuration changes at runtime: adding a VisionProvider to a live conference MUST subscribe the already-published video tracks, and removing the last consumer of a track MUST unsubscribe_track() it.

Collection gate (normative): the room binding permits collection while the channel is attached to the room and the binding can write under Section 7.5 — access ∈ {READ_WRITE, WRITE_ONLY} and muted = false. While it does not, the channel MUST NOT subscribe a track, hand a frame to a lane or a recording, deliver a transcription, or start a re-join after a session loss. When the gate closes, frames stop being routed at once; the channel then unsubscribes every track, closes its lanes and finalizes its recordings. When it reopens with a bot in the conference, the published tracks the channel consumes are subscribed again.

Hot-plugging intelligence (normative): the configuration first need is read from — stt, tts, recording, and vision where an implementation carries it — is not fixed at construction. Implementations MUST let each be plugged into and unplugged from a running channel, as explicit per-kind operations rather than bare attribute assignment: a plug can put a bot into a live meeting and an unplug can take one out, and an operation with consequences of that order needs a surface whose refusals and effects have somewhere to land. This is principle 7 (Section 18) reaching the conference's intelligence, and it is the other half of the pure-transport contract of step 1: a meeting can begin purely human and gain its notetaker when the host asks for one — the explicit gesture a consent obligation wants (Section 17.7) — and lose it again the same way, without the channel being rebuilt around either.

Plugging a need is a first need. The lazy-join triggers of step 1 answer "has the conference started to matter"; a plug answers "has the channel started to care", and the two compose: if the channel is attached and holds no session, the occupancy probe of step 1 MUST be re-run at the plug — the join is once more a consequence a probe can have, so the probe's justification returns with the need — and a conference already holding participants is joined at once. An empty conference stays unjoined: laziness is preserved, and the mint, delivery and arrival triggers stand ready as on any configured channel. The join a plug starts follows the discipline of every trigger before it: its failure MUST NOT fail the plug — the configuration stands, and the lazy join remains behind it. What a plug refuses, it refuses for the reasons construction does, held identically at runtime: the E2EE exclusions (Section 12.10.2) gate a plugged stt or recording exactly as a constructed one, and a recording plugged without a recorder is refused wherever it is asked. A plug into a slot already holding a provider MUST be refused rather than replace it: a swap of streaming intelligence is a teardown and a rebuild whatever single verb offers it, and the observation gap belongs in the open — unplug, then plug. Unplugging an empty slot is a no-op: the state the caller asked for already holds. Configuration changes are serialised per channel: implementations MUST NOT let two interleave, since everything below — the grants derived, the subscriptions re-evaluated, the join or leave decided — is read from the configuration as a whole.

A plugged consumer reaches back. Selective subscription's runtime clause is the mechanism: an stt or recording plugged mid-meeting MUST subscribe the already-published tracks it consumes, and their lanes open as for any subscription — the meeting is transcribed from the plug forward, not from the next publication. An unplug re-evaluates the same way: lanes close and tracks are unsubscribed when nothing else consumes them, recordings are finalized and announced exactly as a detach finalizes them, and an utterance in flight when the voice is unplugged is ended as a live conference ends one — stop_playback() and a terminal chunk (One utterance at a time), since the bot is still in the meeting and the turn genuinely ended. Who closes an unplugged provider follows the channel's one provider-ownership rule, the same that governs its close().

Unplugging the last need takes the bot out. A session kept past the last consumer and the last voice is the participant with no function step 1 declines to admit — the silent observer of Section 17.7, now with a connection it was given back when it had a purpose — so the channel MUST leave, with conference_ended announced, and stand down exactly as a channel constructed pure-transport: triggers quiet, probe unmade, the admission gate and roster still on duty. The round trip is the point: transport-only to intelligent and back is a lifecycle the channel serves, not two deployments.

The session's grants MUST cover the configured needs, at the join and across every change. Derived grants (Section 12.10.3, Bot grants) are derived from the configuration in force at the join, not at construction. A plug that widens what a live session must be able to do — a voice plugged onto a listening bot needs publish_audio; a consumer plugged onto a speaking bot needs subscribe — MUST bring the session's grants in line before reporting the plug complete: through update_bot_grants() where the backend declares BOT_GRANT_UPDATE — the session, its subscriptions and the event bridge all survive — and otherwise by re-joining, a leave() and a join_as_bot(), each announced as the session event it is, because a session the SFU will not re-permission can only be replaced. An unplug that narrows the needs SHOULD narrow the grants in place where the backend can, and MAY leave them standing where it cannot: an unused privilege against a cut in the event bridge is a trade this specification settles for continuity. Explicit bot_grants are never rewritten by a plug or an unplug — the caller who set them took coverage on themselves (the coverage rule binds derived grants only, below), and that holds at runtime exactly as at construction.

Runtime ownership of explicit grants (normative): that the plugs never rewrite an explicit bot_grants does not make the grant set immutable — it makes it owned. The owner is the caller who set it, and implementations MUST give that caller a runtime operation that replaces the explicit grant set on a running channel, serialised with the plugs under the same configuration-change discipline. Replacing an explicit set with another explicit set carries the construction-time bargain forward unchanged: the caller took coverage of the configured needs on themselves, so a replacement that does not cover them — subscribe withdrawn under a plugged recognizer — is accepted exactly as construction would accept it, not refused. The coverage rule of this section binds derived grants only, at the setter as at the join. Replacing the explicit set with nothing returns the channel to derivation: from there the grants follow the configuration in force, exactly as on a channel that never had an explicit set — the round trip is available here as everywhere else in this section. Grants remain channel-wide in this revision, as the rest of the channel's configuration is; per-conference grants are not modelled. An explicit set still creates no need (step 1): a set on a channel with no consumer and no voice changes what the next session would be allowed, and joins nothing.

The setter is an instruction, not a hint. Where the alignment a plug makes may leave a narrowing standing — an unused privilege traded for continuity — the setter's change MUST be applied in full: through update_bot_grants() where the backend declares BOT_GRANT_UPDATE, and by the announced re-join otherwise or when the in-place update fails, whatever the direction of the change. A privilege the caller asked to withdraw is not an unused privilege; it is an ignored order. A caller who prefers continuity to the change expresses that by not calling. An implementation MUST let the caller learn before calling whether the change will be applied in place or cost the event bridge a re-join — the capability is consultable on the backend, and the channel's disclosure surface SHOULD answer it directly.

Visibility is the one grant the SFU's clients are told about, and it does not move symmetrically (verified against LiveKit): an SFU that can announce a newly visible participant to the clients already connected delivers a reveal in place, but no interface exists to un-tell them — a session re-hidden in place stays on the roster of every client that saw it. A hidden-to-visible transition is therefore applied like any other change: in place where the backend can (Section 12.10.3). A visible-to-hidden transition MUST be applied by replacing the session, whatever the backend's capabilities: the announced leave is the retraction, and it is the only one every backend can deliver.

When a connected session's effective grants change — in place, or carried by the replacement session of the re-join — the implementation MUST emit conference_bot_grants_changed (Section 12.10.7): with an in-place update nothing else says anything happened, and the code that asked for the change and the code that renders the room are rarely the same component. A set that changes nothing emits nothing.

The disclosure surface follows the configuration, not the constructor. Section 17.7 already requires that whether STT, vision or recording is active be observable at any time; with hot-plugging the configured half of the answer moves too, and an implementation's surface MUST report the configuration in force, not the construction — a meeting that gained a transcriber mid-way reads as transcribed from that moment, and one that lost it reads as clean.

Outbound (deliver):

When a broadcast event reaches the conference binding and TTS is configured, the channel MUST synthesize once (BEFORE_TTS / AFTER_TTS hooks apply) and publish the audio on the bot track via publish_audio(). The SFU distributes it to all participants. Targeted per-participant audio is NOT supported in this revision: one bot track, heard by all.

One utterance at a time (normative): the channel MUST NOT interleave the chunks of two utterances on the bot track. There is one track and the SFU mixes nothing for it, so two utterances published together do not arrive as two voices — they arrive as one stream that is intelligible as neither. Nothing upstream guarantees this on the channel's behalf: an implementation whose broadcast pipeline serialises per room (Section 10.1) may still consume a streaming response outside that serialisation, so the single track is the channel's own to protect. Whether a second utterance waits for the first or replaces it is left to the implementation — both are defensible, and they differ in whether an answer the AI produced may go unheard.

Every utterance the channel publishes MUST end with a chunk whose is_final is set, including one cut short by a barge-in (Section 12.10.5). This is what makes is_final a boundary a backend can rely on rather than a hint: without it, an interrupted utterance leaves the SFU believing the bot is still mid-sentence. The closing chunk MAY carry no audio (data empty) — there is nothing left to play, only an end to declare — and backends MUST accept one.

The closing chunk is a boundary, not a flush. An utterance cut short and one that ended by itself close identically, so a backend MUST NOT discard queued audio on is_final — a synthesizer whose real ending is already queued behind an empty final chunk would lose it. What silences the room is stop_playback() (Section 12.10.3): when an interruption lands, the channel MUST call it on the session the utterance is publishing on, so the audio the transport already holds is discarded rather than played out to the end of whatever buffer holds it. The call and the closing chunk are two different statements — the gesture that stops the sound, and the boundary that ends the turn — and the channel owes the backend both. No ordering is required between them: the closing chunk of an interrupted utterance carries nothing a flush could truncate.

There is exactly one exception, and it is the end of the session rather than the end of an utterance: an utterance the channel abandons because it has left the conference — a detach, or the channel closing — MUST NOT be closed with a terminal chunk. The bot session that chunk would name is on its way out of the conference and may already be gone, so publishing into it is either a write to a session the framework has left or a race with leave(). The framework cannot honour the guarantee here in any case: the process may also have crashed, the network may have dropped, and no in-band marker survives either. So a backend MUST treat leave(), and a session's disconnection by any other means, as terminating whatever utterance was in flight on it — the same way it already has to treat the connection dropping. An utterance interrupted by a participant is not this case and is still closed: the conference is live, the bot is in it, and the turn genuinely ended.

By default only AI/agent responses are spoken. When speak_text_events is true, inbound text events from other channels (SMS, WebSocket, ...) are also spoken into the conference.

This default deliberately diverges from VoiceChannel, which speaks every TextContent it receives (Section 12.3). A 1:1 voice session has one listener who is the conversation's subject; a conference is a multi-party meeting where unrelated channel traffic read aloud is disruptive to everyone at once. The restrictive default is therefore normative, not a convention: implementations MUST NOT speak non-AI text events unless speak_text_events is set.

12.10.5 Multi-Party Interruption Policy

In a 1:1 voice session, any user speech may interrupt TTS playback (Section 12.6). In a conference, who can interrupt the AI is policy:

ConferenceInterruptionConfig
├── strategy: InterruptionStrategy = IMMEDIATE   # Section 12.6 semantics
├── scope: "any" | "none" | "allowlist" = "any"
└── allowlist: list<participant_id> = []

• "any" — speech detected (VAD) on any audio track interrupts bot playback, subject to the configured InterruptionStrategy.
• "none" — the bot always finishes speaking (presentation/IVR style).
• "allowlist" — only listed participants can interrupt (moderator pattern).

SEMANTIC is NOT available here, and an implementation SHOULD refuse it at configuration time rather than degrade silently. The strategy classifies an utterance from its transcript, and a conference lane only has a transcript once the utterance has ended — by which point the interruption it would have authorised is moot. The remaining strategies apply with the InterruptionConfig defaults of Section 12.3.13, evaluated per track: CONFIRMED waits for min_speech_ms (300 ms) of one track's sustained speech, and ConferenceInterruptionConfig has no field to change it. scope is the conference's own axis and composes with all of them.

No backchannel detection runs on a conference lane: it belongs to the SEMANTIC strategy. An interruption that lands is more than the chunk stream stopping: the channel calls stop_playback() so the audio the transport already holds is discarded instead of playing on over the participant (Sections 12.10.3 and 12.10.4). The utterances still waiting for their turn in that room are dropped unpublished. ON_BARGE_IN fires with the interrupting participant identified.

12.10.6 AI Participation Patterns

STT/TTS (default). Per-track STT lanes produce attributed transcription RoomEvents; an AIChannel in the room responds through the normal broadcast pipeline; the response is spoken via the bot track. The AI requires no conference-specific mechanics and sees a fully attributed multi-speaker transcript.

Speech-to-speech (OPTIONAL). A RealtimeVoiceProvider (Section 12.4) may be composed with a conference: subscribe to all AUDIO tracks, mix N→1, feed the provider, and publish its audio output on the bot track. Section 12.10.12 specifies the composition — the mixing rule, where attribution ends, how the provider's voice meets the utterance contract, and how it crosses the interruption policy. Per-speaker attribution is lost at the provider boundary; implementations SHOULD run per-track STT lanes in parallel when transcripts are required.

Observer. A bot joined with bot_grants set to subscribe-only and hidden, with STT lanes but no TTS — transcription, compliance monitoring, or meeting summarization without a voice presence. Whether a silent transcribing bot may stay invisible to participants is a legal question, not a framework one; this specification does not restrict hidden, and Section 17.7 sets out what the framework exposes so the integrator can meet the disclosure rules that apply to it.

12.10.7 Conference State Events

Hooks (Section 9.2): ON_CONFERENCE_PARTICIPANT_JOINED, ON_CONFERENCE_PARTICIPANT_LEFT, ON_CONFERENCE_TRACK_PUBLISHED, ON_CONFERENCE_TRACK_UNPUBLISHED, ON_CONFERENCE_TRACK_MUTED, ON_CONFERENCE_TRACK_UNMUTED, ON_ACTIVE_SPEAKER_CHANGED, ON_CONNECTION_QUALITY_CHANGED. Screen-share track publication additionally fires the existing ON_SCREEN_SHARE_STARTED / ON_SCREEN_SHARE_STOPPED.

The track triggers carry the CONFERENCE_ prefix to keep them distinct from ON_VIDEO_TRACK_ADDED / ON_VIDEO_TRACK_REMOVED (Section 12.8), which describe tracks within a single VideoChannel session rather than publications by other participants in a conference.

Ephemeral events (Section 8.4): high-frequency media state MUST NOT be stored as RoomEvents; implementations SHOULD publish it via the RealtimeBackend instead:

• conference_track_muted / conference_track_unmuted
• conference_active_speaker
• conference_connection_quality

Framework events (Section 8.2), with their emission points:

Event	Emitted when
conference_started	join_as_bot() completes and the bot connection is live — the same point as ON_SESSION_STARTED
conference_ended	The bot session ends: leave() completes (channel detach, the unplug of the last need, or the leave of a re-join), or the backend reports the session's end (on_bot_session_ended)
conference_participant_joined	A Participant record is created or reactivated from on_participant_joined
conference_participant_left	on_participant_left is processed
conference_bot_grants_changed	A connected session's effective grants changed (Section 12.10.4): replaced in place through update_bot_grants(), or carried in by the replacement session of the announced re-join. Carries bot_session_id and the session's hidden status

conference_started tracks the framework's own media presence, not the existence of the SFU room: ensure_room() MAY have run much earlier, and human participants MAY be conferring before the bot joins.

conference_ended MUST carry the bot_session_id of the session that left. A detach whose destructive half is deferred (Section 12.10.4) can complete after the room has been re-attached and a second bot has been announced, so the end is not always the last conference event a room emits; without the session identifier an observer cannot tell which conference it is the end of. The same identifier appears on conference_started, and pairing them is the only ordering guarantee available in that case.

12.10.8 Recording

ConferenceRecordingConfig
├── mode: "framework" | "egress" = "framework"
├── storage: string                     # Integrator-defined identifier
├── format: string (default "wav" for audio, "mp4" for composed video)
├── metadata: map<string, any>
├── encryption: RecordingEncryption | null   # Section 12.3.7, handed to each track's recording
└── storage_encrypted_at_rest: bool = false

• framework (default) — bot-subscribed tracks are recorded through the room media recorder interface (Section 12.11), one recording per track. This is the path that always works: it needs no backend capability, it functions against MockConferenceBackend, and the recording is written wherever the implementation writes it, which matters where data residency is constrained. Since audio tracks are already subscribed for STT (Section 12.10.4), recording them adds a file write and no additional media subscription.
• egress (OPTIONAL) — recording is delegated to the SFU, and requires EGRESS_RECORDING. Configuring it on a backend without the capability MUST raise a configuration error. This mode exists for one reason: a composed video recording — grid or active-speaker layout — cannot be produced by the framework without subscribing every video track, decoding all of them, compositing and re-encoding, which is exactly the media-plane work Section 12.10.1 forbids. Delegating it is the only conforming way to obtain it.

Egress carries no unified result contract. The framework does not promise a RecordingResult or ON_RECORDING_STOPPED for a delegated recording: the SFU announces completion out of band — typically a webhook to an endpoint the framework does not host — and reaching back into that flow would require either provider-specific payloads inside the backend interface or polling the SFU's job API. An integrator that opts into egress collects its output through the provider's own mechanism. Switching between the two modes is therefore not a transparent configuration change, and implementations SHOULD document it as such.

Recorder mapping. AudioRecorder (Section 12.3.7) and VideoRecorder (Section 12.8.10) are specified around a session, with a handle carrying a session_id: AudioRecorder records its two directions (record_inbound / record_outbound, an INBOUND_ONLY | OUTBOUND_ONLY | BOTH mode), VideoRecorder the video the session receives (tap_frame). A conference has no session and no single inbound direction — it has N attributed tracks plus one bot track — so neither interface applies to it, and neither changes on its account. Framework-mode conference recording binds to MediaRecorder (Section 12.11) instead, which is addressed by track and already carries the attribution.

An implementation recording a conference in framework mode MUST:

• Open one recording per subscribed track: one on_recording_start() carrying exactly that track. A conference gains participants while it runs, and a recording that admits its tracks only at the start cannot record a late arrival — MP4, the usual container, refuses a new stream once muxing has begun. Per-track recordings also make "who said what" a property of the output rather than of a mixing decision.
• Attribute every track: RecordingTrack.participant_id carries the publishing participant as Section 12.10.2 resolved it, and channel_id the conference channel.
• Declare the format that track actually carries — codec, sample_rate and channels describing the frames the backend delivered, not the format the framework normalizes to elsewhere. Section 12.10.3 obliges a backend to deliver decoded PCM with a declared sample rate; it obliges no two participants to agree, so a conference of three may carry three formats, and the first frame of a track is where its own is known.
• Record the bot's published track as its own attributed track, never mixed into a participant's. The bot track is the only one that resembles an outbound direction, and treating it as one would make the framework mix media it was given separately. What the AI said is part of what was said.
• End a track's recording when that track ends — unpublished, unsubscribed, or the conference left — without ending the others.
• Stop recording when the room binding stops permitting collection, on the same gate as transcription (Section 12.10.4). Recording is collection.

A recording is opened on a format. That declaration is made once, on the frame that opens the track's recording, and a container fixes its streams at the first write — so a later frame in another format has no honest place in it. An implementation MUST NOT write one anyway: it is discarded, reported through the same loss the overload bound owes below, and said in the log. A recording missing the seconds where a publisher renegotiated has a defect anyone can see; one that wrote them is a file that opens and lies.

Writing is not the callback's work. Recording a track MUST NOT delay frame delivery for any other. This is the lane isolation rule of Section 12.10.4, and it applies to the recorder for the same reason it applies to the recogniser: the backend hands frames to its subscribers in sequence, so an encode and a file write performed where the frame arrives makes one track's storage latency into every participant's delivery latency. That the recorder interface is synchronous (Section 12.11) does not exempt it — it is what makes the obligation bite, since a caller cannot await its way out of a blocking write and MUST therefore get it off the delivery path altogether.

An implementation MUST bound what it holds for a track whose writes fall behind, MUST document which audio it discards when that bound is reached, and MUST expose how much it discarded — the same three obligations Section 12.10.4 places on a lane's backlog, and for the same reason: a recording with a hole in it that nothing reports reads as a defective recorder.

What is still queued when a track's recording ends MUST be written before the recording is finalized, within a bounded budget. A container closed over frames that were still in flight is not a truncated recording but a misleading one — it ends early and says nothing about it — and a budget is what keeps a recorder that has stopped draining from holding the teardown, and with it the bot, in the conference. What the budget could not write is loss, and is reported as such.

Reporting the result. A framework-mode recording ends with a MediaRecordingResult naming where it was written, and the framework SHOULD report it: ON_RECORDING_STARTED when a track's recording opens, ON_RECORDING_STOPPED when it ends, carrying that result — the same pair Section 12.3.7 fires for a session. Without it an integrator has files on a disk and no programmatic way to learn their paths: the recorder's return value stops inside the framework, and a log line is not an interface.

Per track, not per conference. The recording is a track, and the tracks of one meeting do not end together — a participant who leaves halfway through has a finished recording while the meeting continues. A single report at the end would have to hold every earlier result until then, and would arrive after the point where an observer could have acted on it. An integrator wanting the meeting's full list accumulates it by room, which is state it was going to keep anyway.

The report carries the attribution the result alone need not: the room, the track, and the participant publishing it as Section 12.10.2 resolved it. Both hooks are ASYNC — a recording already written is a fact, not a decision, and nothing about it is a hook's to block. A track subscribed but never published on reports nothing at all: the recording opens on the first frame, so silence produces no file and no pair of hooks describing one. Egress mode fires neither, for the reason given above — it promises no result contract.

How many files come out is the recorder implementation's decision: the framework guarantees attribution, not file layout. A recorder MAY write one file per track or mux several — it is handed the tracks separately and attributed, so either remains open to it.

Video tracks take the same shape where an implementation records them (RecordingTrack.kind distinguishes them), which is per-track and therefore never a composed recording. A grid or active-speaker layout stays egress territory, for the reason given above.

12.10.9 Cross-Channel Integration

Because the conference is a room, integration with other channels requires no new mechanism:

• Transcripts broadcast everywhere. Transcription RoomEvents flow through the broadcast pipeline (Section 10.2) to every bound channel — a WebSocket or messaging binding receives a live, attributed meeting transcript for free.
• Text can be voiced in. With speak_text_events, a message arriving from any channel is spoken into the conference by the bot.
• AI is just an AIChannel. Responding to transcripts, summarizing on close (ON_ROOM_CLOSED), or steering via hooks all use existing primitives.
• SIP/PSTN interop (OPTIONAL). Backends with SIP_GATEWAY MAY admit phone participants directly. Alternatively, an implementation MAY bridge an existing VoiceChannel SIP session into the conference by republishing its audio as an additional bot participant (audio-only gateway). A phone participant admitted by the SFU is the canonical case of a participant the framework did not name: its caller number arrives in ConferenceParticipant.metadata and MUST reach identity resolution (Section 12.10.2), so that dialling into a conference identifies the caller as reliably as messaging the room would. A gateway is also where the provenance rule earns its keep: the number the trunk reported is a fact the SFU established, and a backend admitting phone participants SHOULD say so through asserted_metadata — otherwise the one address the framework can act on is indistinguishable from one a client typed.

12.10.10 Relationship to Audio Bridging (Section 12.7)

Criterion	AudioBridge (12.7)	Conference (12.10)
Media plane	In-process (framework forwards/mixes)	External SFU
Media types	Audio only	Audio + video + screen share
Scale	Small rooms (default max 10)	SFU-bound (typically dozens+)
Infrastructure	None	SFU deployment required
Participants	Any VoiceBackend session (SIP, RTP, ...)	SFU-native clients (+ SIP gateway)
Use when	Phone bridging, call center, small audio conf	Video conferences, AI in meetings

Both MAY coexist in one implementation. They do not share interfaces: AudioBridge operates on VoiceSessions; conferences operate on ConferenceTracks.

12.10.11 Conformance

Conference support is part of Conformance Level 3 (Real-Time Media) and is OPTIONAL. It is STABLE (see the status note above).

Implementations that support conferencing MUST:

• Implement the ConferenceBackend interface with a Mock implementation.
• Map each conference 1:1 to a Room.
• Keep the framework out of the human-to-human media path.
• Deliver decoded, participant-attributed, room-attributed PCM audio per track.
• Correlate participant_id across the backend boundary per Section 12.10.2, including the external_id fallback for participants the framework did not mint, and surface provider participant attributes so their resolvable addresses reach identity resolution — on arrival rather than on first utterance, with the answer linked to the participant by identity_id rather than re-keying it.
• State which participant attributes the SFU itself asserts, leaving asserted_metadata null where the backend cannot tell, and resolve identity only on those — never on an attribute a client supplied, unless an integrator configured that explicitly (Section 12.10.2).
• Record provider attributes on the Participant under a dedicated key rather than merged into the integrator's own metadata, keeping their provenance and bounding what is persisted (Section 12.10.2).
• Gate unmute_track() on the REMOTE_UNMUTE capability, raising a configuration error rather than failing silently where it is unsupported.
• Subscribe explicitly: consume only tracks passed to subscribe_track(), and never auto-subscribe the bot session.
• Exclude the bot's own participant and tracks from Participant records, lanes, and subscriptions.
• Run per-track STT lanes emitting attributed transcription RoomEvents, one per utterance rather than one per frame (Section 12.10.4).
• Keep lanes independent: no track's processing delays frame delivery for another.
• Bound each lane's backlog, document what it discards under overload, and expose how much it discarded.
• Publish AI TTS as a single bot track (synthesize once, publish once).
• Publish bot media as decoded frames — PCM AudioChunk, raw VideoFrame — leaving encoding to the backend.
• Where framework-mode recording is offered, record one attributed recording per track — the bot's own included, unmixed — describing the format that track actually carries, and stop it on the same collection gate as transcription (Section 12.10.8).
• Keep those recordings off the delivery path: write outside the frame callback, bound each track's write backlog, document what it discards and expose how much, and flush what is queued before finalizing (Section 12.10.8).
• Fire conference lifecycle hooks and create/update Participant records from conference events.
• Treat ConferenceAccess as opaque.
• Either admit the bot to E2EE key exchange or refuse media-intelligence configuration on E2EE conferences (Section 12.10.2).
• Expose bot presence, hidden status, and whether STT/vision/recording is active, so integrators can satisfy their own disclosure obligations (Section 17.7). The specification mandates no disclosure policy.

Implementations SHOULD:

• Support vision analysis on VIDEO and SCREEN_SHARE tracks.
• Surface active speaker and connection quality as ephemeral events.
• Record conferences through the framework recorder path by default, treating SFU egress as an optional delegation with no result contract (Section 12.10.8).
• Report each framework-mode recording where it opens and where it ends — its result, its track and its participant — rather than leaving the integrator to find the files (Section 12.10.8).
• Enforce ConferenceInterruptionConfig scope.

Implementations MAY:

• Publish bot video (avatar embodiment, requires VIDEO_PUBLISH).
• Compose speech-to-speech providers via N→1 mixing (Section 12.10.12).
• Bridge SIP sessions into conferences.

Implementations that compose speech-to-speech providers (Section 12.10.12) MUST:

• Feed the provider a single stream mixed per Section 12.7.5, and forward no silence-only window.
• Discard the provider's user-role transcriptions rather than store or attribute them.
• Emit the provider's final assistant transcriptions as RoomEvents attributed to the channel, and never back into the conference's own voice path, with the chain depth Section 12.4 gives an assistant transcription: after a text injection, that event's depth plus one.
• Refuse a configuration holding both a synthesizer and a speech-to-speech provider.
• Publish provider audio under the utterance contract of Section 12.10.4, terminal chunk included.
• Keep the per-lane VAD as the interruption sensor and enforce ConferenceInterruptionConfig scope on it.
• Run per-track VAD whenever a speech-to-speech provider is composed.

12.10.12 Speech-to-Speech Composition

A conference MAY compose a RealtimeVoiceProvider (Section 12.4) as its intelligence: the participants' voices reach one realtime session, and the provider's voice is the bot's. This is the sub-second, natively turn-taking agent in a multi-party meeting — the speech-to-speech pattern of Section 12.10.6 — and it is OPTIONAL within Conformance Level 3. This section binds only implementations that offer it.

The composition is configured through the channel's realtime field (Section 12.10.4):

ConferenceRealtimeConfig
├── provider: RealtimeVoiceProvider           # One provider session per conference
├── system_prompt: string | null
├── voice: string | null
├── tools: list<ToolDefinition> | null        # Each needs tool_handler or human_input_handler
├── tool_handler: callable | null             # (room_id, name, arguments) → serialized result
├── temperature: float | null
├── input_sample_rate: int = 24000            # Rate the mix is resampled to
├── output_sample_rate: int = 24000           # Rate of the provider's audio, published as-is
├── server_vad: bool = true                   # Provider turn-taking only, never interruption
├── provider_config: map<string, any> | null  # Provider-specific session options
├── tool_policy: ToolPolicy | null            # Section 21.1
├── tool_timeout_seconds: float | null = 10   # Per-call bound (Section 21.6); null = unbounded
├── tool_timeouts: map<string, float | null> = {}  # Per-tool-name bounds
└── human_input_handler: callable | null     # Human-input tools (Section 9.3), served before tool_handler

The composition changes no boundary it crosses. The provider keeps the interface of Section 12.4 — one session, one audio input, no notion of a conference. The backend keeps the interface of Section 12.10.3 — tracks in, one bot track out. The channel between them keeps every contract of Sections 12.10.4 and 12.10.5. What this section specifies is the crossing itself.

N→1: the provider hears a mix (normative). Implementations MUST feed the provider one stream mixed from every subscribed AUDIO track, using the additive algorithm of Section 12.7.5: samples promoted and summed per mixing window, headroom-scaled by the square root of the number of tracks contributing to that window, clamped back to the sample width. A track with no audio in the window contributes silence; a window to which no track contributed MUST NOT be forwarded. The mixed stream is resampled to the provider's declared input rate before it is sent.

Routing the active speaker instead is not an admissible narrowing. It reads better on paper — the provider hears one clean voice — but ACTIVE_SPEAKER is an optional capability (Section 12.10.2), so the routing would not survive a backend change; the switch lands mid-syllable, so every hand-over chops a phoneme; and overlapping speech — the thing a meeting has that a 1:1 call does not — is audible in a mix and lost in a switch. The mix is the one complete signal every backend can produce. The mixer itself stays private: the algorithm of Section 12.7.5 is reusable, the AudioBridge interface is not (Section 12.10.10), and this specification defines no mixer provider interface — an implementation MAY factor one out privately.

Attribution ends at the provider boundary (normative). A mix carries no speaker identity, so nothing downstream of it can. The provider's user-role transcriptions MUST NOT be stored as RoomEvents and MUST NOT be attributed to a participant by inference — correlating them with active-speaker events or lane VAD timing is a guess wearing attribution's clothes, and principle 5 places identity on tracks precisely so that no component has to guess. Implementations MUST discard user-role transcriptions; they MAY surface them to observability, unattributed, and MAY hand them, unattributed, to a reasoning backend (Section 12.4.1) as state the session drops with itself. The attributed transcript is the one the conference already has: per-track STT lanes (Section 12.10.4), running in parallel with the mix, each event carrying its speaker. A deployment that needs the meeting transcribed configures stt beside the provider; the two consume the same lanes and do not contend.

The provider's assistant-role transcriptions are the other half, and they are kept: they are the only record of what the AI said, because no AIChannel generation stands behind this voice for the store to hold. Final assistant transcriptions MUST be emitted as RoomEvents attributed to the channel — no participant is their author — and MUST NOT be delivered back into the conference's own voice path: the words were already heard once, on the bot track.

Proactive delivery (normative). The provider's room session is the conference's realtime destination (Section 22): a delivery naming the conference, with no intelligence address, MUST be injected into that session with the intent of Section 12.4, as on a realtime voice channel, and is unavailable while no session is connected. A strategy waiting for idle waits until the model's answer has ended and reached the bot track, the provider hears nobody speak, no tool call of the session is in flight, and no result (or delegation fallback) sent to the model is still waiting for its answer (Section 12.4.1), as on a realtime voice channel. The injection fires ON_REALTIME_TEXT_INJECTED.

A delegation is always answered (normative). A full-duplex provider's delegation fires ON_REALTIME_DELEGATION; a conference serves no reasoning backend, so an integrator-side delegation is answered with the spoken fallback a realtime voice channel without a backend gives (Section 12.4.1), never left waiting. A delegation the provider issues while the room's session is still connecting is answered once the session is the room's, as a tool call issued then is served.

One voice per bot (normative). A channel MUST refuse a configuration holding both a synthesizer (tts) and a speech-to-speech provider, at construction and at the plugs alike. There is one bot track and one voice on it at a time; two components that each answer the room would answer over each other, and no turn-taking rule turns two intelligences into one voice. A deployment that wants to trade the STT/TTS pattern for the realtime one does it through the plugs — unplug one, plug the other — and never holds both.

The provider's voice is an utterance (normative). Provider audio reaches the conference through the outbound contract of Section 12.10.4 unchanged: a response takes its turn and opens an utterance, its audio deltas are the utterance's chunks, and its end closes the utterance with is_final — as does a barge-in, and a leave abandons it with no terminal chunk, exactly as written there. One utterance at a time holds: a second response waits for its turn. BEFORE_TTS and AFTER_TTS do not fire — they are text-synthesis hooks and no text precedes this synthesis. ON_BARGE_IN fires as always, though its interrupted_text MAY be empty when the interruption lands before the first assistant transcript delta has arrived.

Interruption has one sensor and two effects (normative). Section 12.10.5 stays the policy authority, and the per-lane VAD stays its sensor: ON_BARGE_IN MUST identify the interrupting participant, and only track identity can — the provider's own speech detection hears the mix and can name no one, so it MUST NOT trigger the interruption path. An interruption that lands does what Section 12.10.5 says — the response publishes nothing past its next chunk and closes with is_final, stop_playback() is called, ON_BARGE_IN fires, and what the provider still sends for that response is discarded — and additionally SHOULD signal the provider to cancel the response in flight, where the provider can: cancellation is best-effort, since Section 12.4 providers exist whose sessions cannot cancel a response, and it is stop_playback() that silences the room either way. A full-duplex provider (Section 12.4.1) is the standing case: it has no cancel signal, and the mix carries the interruption to it; stop_playback() and ON_BARGE_IN remain the framework's.

The provider brings its own turn-taking, and the two authorities can disagree: a provider whose native detection halts generation on any speech in the mix will fall silent for a speaker the configured scope would not admit — no barge-in fires, the bot simply trails off. Implementations SHOULD align the provider's own interruption behaviour with the configured scope where the provider exposes it (Section 12.4 provider configuration), and MUST NOT substitute the provider's detection for the policy: scope enforcement, stop_playback() and ON_BARGE_IN remain the framework's, whatever the provider does with its generation.

The session follows the bot (normative). One provider session serves one conference, scoped within the lifetime of the bot session whose track it speaks on. Speech-to-speech is a first need under Hot-plugging intelligence (Section 12.10.4): it consumes AUDIO tracks and it is a voice, so its derived grants carry both subscribe and publish_audio, and plugging and unplugging it follow every rule of that section — the occupancy probe re-runs at the plug, an occupied slot refuses a second provider, and unplugging the last need takes the bot out. Connecting the provider is lazy: the session is established when there is something for it to hear or say, and a connect failure MUST NOT fail the join or the plug, exactly as a lazy join's own failure does not (Section 12.10.4) — the configuration stands, and implementations SHOULD retry with a cooldown rather than on every mixing window. With no caller to raise to, each failed connect fires ON_ERROR (error_category = realtime_provider, error_type the error's type name), so the integrator learns the model is unreachable. When the bot session ends — detach, unplug, backend-side loss, close — the provider session is disconnected with it.

The lanes stay on. Composing a provider suspends none of the per-track machinery: VAD, speech edges, recording, and STT lanes where configured all run as before. The pipeline requirement of Section 12.10.4 extends accordingly: a conference composing speech-to-speech MUST run per-track VAD even when no STT is configured — without it the interruption policy of Section 12.10.5 has no sensor, and the bot cannot be interrupted at all.

A tool call is a realtime tool call (normative). A call the provider issues passes the pre-execution gate of Section 12.4 before it reaches the configured handler, in that section's order: the declared-tool check against the configuration's tools, the configuration's ToolPolicy when it carries one, argument validation against the declared schema, and BEFORE_TOOL_USE (Section 9.2), whose arguments, replaced or edited, are validated again. The gate has no skill-gating step because a conference declares no skills. The call fires ON_TOOL_CALL (Section 9.3) with its outcome, a refusal and a failure included, and as on any realtime channel the report of a refusal MUST NOT precede the refusal on the wire. Its result is bounded as any tool result is (Section 21.5), and a call the provider abandons interrupts its handler, sends no result, and is reported once, as cancelled; a call whose outcome ON_TOOL_CALL already received is not reported again. A handler's exception is logged; the model reads that the tool failed, not the exception's text. The ToolPolicy (Section 21.1) also filters the tools declared to the provider; the mix names no participant, so its base rules apply, never a role override. The conference is a door to the same tools as any other channel, and a gate that holds on one door and not another holds on neither.

12.11 Room Media Recording

Sections 12.3.7 and 12.8.10 each record a session: one participant, one medium — audio in the two directions a session has (12.3.7), the video the session receives (12.8.10). A room is not a session. It holds several participants, each of which may publish more than one kind of media, and they arrive and leave across the room's lifetime rather than all at its start. Room media recording is the interface for that shape, and it is what conference recording binds to (Section 12.10.8).

MediaRecorder (interface)
├── name: string                                          # Recorder name
├── on_recording_start(config: MediaRecordingConfig) → MediaRecordingHandle
│       # Open one recording
├── on_track_added(handle, track: RecordingTrack) → void
│       # Declare a track this recording carries
├── on_track_removed(handle, track: RecordingTrack) → void
│       # The track ended — flush what it holds
├── on_data(handle, track: RecordingTrack, data: bytes,
│           timestamp_ms: float | null) → void
│       # Media for one of the recording's tracks
├── on_recording_stop(handle) → MediaRecordingResult
│       # Close the recording, finalize output
└── close() → void
        # Release resources

RecordingTrack
├── id: string                        # Track identifier
├── kind: string                      # "audio" | "video" | "screen_share"
├── channel_id: string                # Channel the media came through
├── participant_id: string | null     # Who published it
├── codec: string
├── sample_rate: int | null           # Audio
├── channels: int | null              # Audio
├── width: int | null                 # Video
└── height: int | null                # Video

MediaRecordingConfig                  MediaRecordingResult
├── storage: string                   ├── id: string
├── format: string = "mp4"            ├── url: string
├── video_codec: string               ├── duration_seconds: float
├── video_fps: int                    ├── tracks: list<RecordingTrack>
├── audio_codec: string               ├── format: string
├── audio_sample_rate: int            └── size_bytes: int
├── metadata: map<string, any> = {}
├── encryption: RecordingEncryption | null
└── storage_encrypted_at_rest: bool = false

encryption and storage_encrypted_at_rest follow the rule of Section 12.3.7: a recorder that stores files MUST refuse on_recording_start() when neither is set, and with encryption the result's url names the encrypted artifact.

Recorders bound when a room is created start before the room is written, all or nothing. When one refuses on_recording_start(), the recordings already started for that room MUST be stopped and room creation MUST fail with the refusal, with no room written: a room that exists without the recording it was created with is a room the integrator believes is recorded. When writing the room fails, the recordings started for it MUST be stopped, and the recordings of a room already holding that id MUST NOT be touched. ON_RECORDING_STARTED fires for each recording once the room exists, which still precedes any media: a room recording captures nothing until a track is added.

Recorders started on an existing room obey the same rules: a room that is missing, another organization's (§17.2), or whose status refuses new events (§5.1) is refused before any recorder starts; the recorders start all or nothing, under the room lock; and ON_RECORDING_STARTED fires for each before any media, as at creation. Media may already flow in the room: a recording joins it only once announced, with each of the room's tracks declared to it first, so what reaches it comes after its consent point and in the format its track declares. What such a recording captures is the room's declared tracks: those declared from then on, and those already declared while an earlier recording ran. A channel the framework wires itself declares a session's tracks when the session joins a room that records, once the join has read the room with the caller's scope (§17.2); a session that joined while the room recorded nothing declares none, and is not recorded until it joins again. A recording resumed after a restart is one of these: its consent point is announced again. A caller feeding a room recording from a source the framework does not wire itself declares each track to the room's recordings and hands them its media through the framework, never through a recorder it holds.

A room recording's end is announced. When a room's recordings stop, on an explicit stop, when the room closes or is archived, or when the framework shuts down, ON_RECORDING_STOPPED fires for each recording that stopped, with its result, as for a session's or a conference track's. A room recording has no session: the event names the room instead. Stopping happens after the room is read with its scope, so a call that is refused stops nothing. A room whose record is gone while its recordings still run has them stopped by an explicit stop all the same, since a file nothing stops is never finalized; another organization's room is refused, and a gone room that records nothing is not found, as either would be told. A gone room has no record left to read its organization from, so the framework keeps, while a room records, the organization its recordings were started under: a scoped call on a gone room naming another organization is refused as well. With no room left to read a context from, such an end MAY be logged rather than announced. Listing a room's recordings and declaring a track to them read the room the same way, with its scope and the same rule for a gone room: through these verbs a caller learns of, or feeds, no other organization's recording.

MediaRecordingHandle                  RoomRecorderBinding
├── id: string                        ├── recorder: MediaRecorder
├── room_id: string                   ├── config: MediaRecordingConfig
├── state: string                     ├── enabled: bool = true
├── started_at: datetime              └── name: string
└── path: string

The unit is the handle, not the room: a handle is one recording, and how many a room opens is the caller's decision. A room that records its channels into a single file opens one and adds every track to it; a conference opens one per track (Section 12.10.8). Both are the same interface used at different granularities, which is why the interface names a track on every call rather than assuming one.

participant_id is what makes an output answerable to "who said what", and it is the recorder's only source for it: the interface carries no session and no direction. An implementation MUST populate it wherever the framework knows the publisher.

A track describes its own media. codec, sample_rate and channels are how a recorder learns to interpret the bytes it is then handed: on_data carries none of it, and PCM in particular is indistinguishable from any other PCM by inspection. A caller MUST therefore describe what it will actually deliver rather than what the framework normally carries, and MUST NOT deliver media in a format other than the one it declared. The obligation is a caller's because the failure is silent: a recorder told 16-bit mono and handed stereo writes a file that opens, plays at the wrong speed, and reports no error at all. A recorder MAY refuse a declaration it cannot honour, and MUST say so.

storage is an integrator-defined identifier resolved by the implementation at runtime, as in Section 12.3.7. Implementations MUST document which storage backends they support and MUST raise a configuration error for an unknown identifier. A storage path MUST NOT be allowed to escape its configured directory.

A recorder MAY refuse a track declared after it has begun writing — a container that fixes its streams at the first write cannot honour a late one — and MUST say so rather than silently dropping the media. A caller that cannot predict its tracks opens one recording per track instead, which is why a conference does.

Which thread calls it. Every call in this interface is synchronous, and what they do — encode a frame, write a container, close a file — blocks for as long as the storage takes. An implementation MAY therefore call a recorder outside the thread its event loop runs on, and one recording media as it arrives will have to: Section 12.10.8 forbids a write that delays another track's delivery, and a blocking call made on the loop delays everything on it. A recorder MUST NOT assume it is called on the loop thread, and MUST NOT assume the calling thread is the same one across calls.

What an implementation owes in return is serialization per handle: the calls belonging to one recording are ordered and never overlap, so a recorder's per-recording state needs no locking of its own and its media arrives in the order the framework received it. Different handles carry no such promise and may be written concurrently, which is what lets one track's slow storage stay one track's problem.

There is one call that may overlap, and only one: a write that has exhausted the budget of Section 12.10.8 cannot be taken back — a call made on another thread runs to completion whatever the framework does about it — and holding the close until it returns would let a wedged recorder hold a teardown, and with it a bot, in a conference indefinitely. An implementation MAY therefore finalize a recording whose write has passed that budget and is still running, and MUST report it when it does. Recorders are not asked to make that case safe; they are told it exists, because the alternative reading is that it cannot happen.

Every wait is bounded, the close included. The budget of Section 12.10.8 covers what a recording holds before it is finalized; on_track_removed() and on_recording_stop() are calls into the same recorder and block the same way, and a conference teardown waits for them before its bot leaves. An implementation MUST therefore bound the finalization too, and MUST report a recording it stopped waiting for — the result is unknown, so where the media was written goes unreported rather than guessed. Recorders MUST NOT read the absence of a further call as permission to discard: what they were told to close is still theirs to close.

A recorder is not released while a call is running in it. close() frees whatever the recorder holds, and freeing the context a call is executing inside is not an error a recorder can return — for one wrapping a native muxer it is a crash. So an implementation MUST NOT call close() while any call it stopped waiting for is still outstanding, and this wait is separate from the ones above: those belong to a teardown with a bot waiting behind them, this one belongs to releasing a provider and has nothing queued behind it. An implementation that still cannot settle them MUST leave the recorder unreleased and MUST say so. Leaking a recorder is a bounded cost; calling into a freed one is not.

---

12.12 Capture Sources

Every voice transport (VoiceBackend, Section 12.2) acquires its capture device when a session starts and releases it when that session ends. That is the right lifetime for a call, and the wrong one for anything that must be listening before there is a session to listen for — a wake word, a hotkey that arms on speech, an always-on level meter. Such a consumer holds the device, must hand it back so the session can take it, and the reacquisition lands precisely while the person is still talking. No application buffer repairs a closed device.

A capture source owns the device instead, and a session becomes one subscriber among several.

AudioCaptureSource (interface)
├── sample_rate: int                                      # Format it emits
├── channels: int
├── sample_width: int
├── start() → void                                        # Acquire the device
├── stop() → void                                         # Release it
├── close() → void                                        # Release all resources
├── mark() → CaptureMark
│       # Opaque token for the ring's current write position
└── subscribe(callback: (AudioFrame) → void,
              since: CaptureMark | null = null,
              name: string | null = null) → CaptureSubscription

CaptureSubscription                   CaptureMark
├── name: string | null               └── (opaque)
├── replayed_bytes: int
├── truncated: bool
└── unsubscribe() → void

A capture source is not a VoiceBackend and MUST NOT implement that interface. A backend is keyed by VoiceSession throughout; a source has no notion of a session, and conflating the two reintroduces the lifetime this section exists to break.

Lifetime is explicit and independent of subscribers. start() and stop() are the only things that acquire and release the device. A source MUST NOT stop capturing because its subscriber count reached zero, and MUST NOT start because it became non-zero. A consumer may therefore unsubscribe at any moment without endangering another subscriber's audio — which is what makes it safe for a wake-word detector to detach for the duration of a session and reattach after.

Frames are raw. A source emits what the device gave it. Echo cancellation is per-session — it needs the reference signal of what that session is playing — so it belongs downstream, applied by the backend after fan-out, not in the source. A subscriber that stays attached while a session is playing audio therefore hears the far end unattenuated, and MUST NOT assume otherwise. Sample-rate conversion is likewise the subscriber's concern: a source has one format, and a consumer that needs another resamples it.

Fan-out is synchronous on the capture thread. This is what keeps the existing AEC invariant intact — capture and reference must stay in step, which they cannot do across a queue hop. The consequence is a contract every subscriber MUST honour:

A subscriber MUST NOT perform unbounded work inside the fan-out callback. It enqueues the frame and returns. The source guarantees no isolation between subscribers: one slow subscriber degrades capture for all of them.

The bar is lower than it sounds. A block is typically 20 ms, and a wake-word subscriber running an ONNX encoder spends about that long per segment — enough on its own to starve the device. Implementations SHOULD make the breach observable rather than leaving it to be diagnosed from audio artifacts: timing each callback and logging a rate-limited warning past a fraction of the block duration turns "the audio crackles" into "this subscriber is slow". A subscriber that raises MUST be caught and logged; an exception MUST NOT propagate into the capture stream.

The backlog. A source retains recent frames in a bounded ring — bounded by duration and by bytes, evicting oldest-first. mark() names a position in it; subscribe(since=mark) replays from there before delivering anything live. Ordering is total: every replayed frame reaches the subscriber before every live one, with no interleaving, even for frames captured during the replay itself. Replay MUST NOT hold the fan-out lock for its duration — seconds of audio delivered under that lock would stall the capture thread — so an implementation queues live frames for a catching-up subscriber and switches to direct delivery under the lock once the queue drains.

A mark whose position has been evicted is stale. The source MUST replay what remains rather than fail: raising at the moment a session opens would discard the very utterance the backlog exists to preserve, and a partial phrase is worth more than none. It MUST report the degradation — truncated on the subscription, and a log line — so that silence is never mistaken for completeness. replayed_bytes is reported in every case.

Addressing by mark, not by duration. A "replay the last N seconds" request forces the caller to guess: too short truncates the utterance, too long replays the tail of the previous conversation into the provider. A mark placed where the speech actually began has neither failure.

Composition with the realtime path. The session-start sequence of Section 12.4 already retains inbound audio across the provider handshake and flushes it in order once the handshake completes. Replayed frames enter through the transport's ordinary inbound callback, so they land in that same retention buffer and are flushed with it. A wake word therefore needs no new control point on the inbound path: the caller passes its mark in the session metadata, the transport subscribes with it, and the utterance that preceded the session arrives ahead of the audio that followed. If the handshake fails, the retained audio is discarded with the session and nothing reaches the provider.

One consequence is worth stating plainly, because it is a design choice and not an oversight: the replayed backlog contains the wake word itself. Where the trigger phrase and the request share a single utterance — "hey jarvis what are you up to" — no boundary exists to split them on, and inventing one would require word-level timestamps the detector does not have. Providers handle the address form well. Implementations SHOULD treat the wake word's presence as intended rather than filter it.

---

13. Resilience and Error Handling

13.1 Circuit Breaker

Each channel SHOULD have an associated circuit breaker to prevent cascading failures.

States:

State	Behavior
CLOSED	Normal operation — deliveries proceed
OPEN	Fail-fast — deliveries immediately return error
HALF_OPEN	Probe — allow one delivery to test recovery

Transitions:

• CLOSED → OPEN: After N consecutive delivery failures (configurable). A delivery refused before any send because its binding names no recipient (§10.2 step 3d) is not one: it says nothing about the provider, and the breaker is shared by every room on the channel.
• OPEN → HALF_OPEN: After a configurable cooldown period.
• HALF_OPEN → CLOSED: Probe delivery succeeds.
• HALF_OPEN → OPEN: Probe delivery fails.

When a circuit breaker opens, implementations MUST emit a circuit_breaker_opened framework event.

13.2 Retry Policy

RetryPolicy
├── max_retries: int                        # Maximum retry attempts
├── base_delay_seconds: float               # Initial delay
├── exponential_base: float                 # Multiplier per attempt (default: 2.0)
├── max_delay_seconds: float                # Maximum delay cap
└── retryable_errors: list<string> | null   # Error codes eligible for retry

Retry delay formula: delay = min(base_delay * exponential_base ^ attempt, max_delay)

Retry policies are configured per-channel-binding. Only errors marked as retryable = true in the DeliveryError SHOULD trigger retries.

13.3 Rate Limiting

RateLimit
├── max_per_second: float | null            # Maximum events per second
├── max_per_minute: float | null            # Maximum events per minute
└── burst: int | null                       # Burst allowance

Implementations SHOULD use a token bucket algorithm. When rate-limited, deliveries MUST be queued (not dropped).

13.4 Idempotency

If an InboundMessage carries an idempotency_key, the framework MUST:

1. Check under a room-level lock whether the key has been seen.
2. If seen → return the original result without reprocessing.
3. If not seen → process normally and record the key.

The point of step 2 is that a sender retries because it never observed the first answer, not because it wants the message delivered twice. Reporting the redelivery as refused conflates "you already said this" with "this was not accepted", and leaves the sender unable to learn the event it committed.

The original result is the committed event. A ConversationStore resolves it from the key it recorded at commit; the key SHOULD therefore be stored so it can be read back, not merely tested for presence. A store that can only answer "seen" MAY return a blocked result carrying the reason instead — a degraded but conforming answer, since the MUST that matters, without reprocessing, holds either way.

13.5 Room-Level Locking

All event processing within a room MUST be serialized. Implementations MUST provide a locking mechanism that prevents concurrent processing of events in the same room.

Serialization scope. The room lock MUST cover the locked pre-commit section, the commit point, and broadcast planning (§10.1 steps 6–12): everything that reads room state to decide, assigns the index, writes the timeline, or resolves the delivery set. A streamed response's rows are the exception (§4, §9.5.1): their hooks and their commit run without the lock. External delivery execution (§10.2 step 3) is NOT required to run under the room lock — it MUST preserve per-room order (the delivery lane, §10.2), which holding the lock trivially satisfies. Implementations SHOULD NOT extend the room's critical section with external I/O (provider calls, AI generation): under a distributed lock manager, lock tenure is what serializes the whole deployment, and every await spent under the lock is paid by every other worker waiting on that room. A BEFORE_BROADCAST check that reads only the event is the hook-side case of that I/O: registered with needs_lock = false, it runs off the lock, ordered by the room's admission ticket instead (§9.5.1).

RoomLockManager (interface)
├── acquire(room_id) → lock
└── release(lock) → void

For single-process deployments, an in-memory lock manager (per-room async mutex) is sufficient. When a persistent store is shared across multiple processes an in-memory lock manager is NOT sufficient — per-process locks do not coordinate, so concurrent writers on the same room can interleave. Such deployments MUST either use a distributed lock manager (e.g. Redis or a PostgreSQL advisory lock) or rely on storage-layer atomicity for the operations that require it (index assignment, §8.1). Implementations SHOULD ship at least one distributed lock manager, and SHOULD detect the unsafe combination of a shared persistent store with an in-memory lock manager and warn (or refuse) at initialization.

13.6 Processing Timeout

Implementations SHOULD support a configurable process_timeout. Its semantics are defined relative to the commit point (§10.1 step 12) — the atomic transaction that stores the event as DELIVERED and updates the room counters:

• Pre-commit phase (§10.1 steps 3–11: context build, identity resolution, BEFORE_BROADCAST hooks, write-permission check, index assignment). This phase decides only and performs no durable write of the inbound event: the BLOCKED record that steps 10–11 owe is committed once the phase has ended, outside process_timeout, as the step 12 commit is. process_timeout MUST bound this phase. On expiry the implementation MUST abort before the commit point, leaving no partial durable state, and return InboundResult(blocked=true, reason=process_timeout) (direct injection, whose result is the committed event, raises instead, §10.5). The event MUST NOT appear in the timeline.

• Commit point (§10.1 step 12). Once the DELIVERED transaction has committed, the event is authoritative. process_timeout MUST NOT cancel, block, or re-mark it. An implementation MUST NOT wrap the commit and the subsequent broadcast in a single cancellable unit whose expiry could leave a committed event reported as blocked/failed.

• Post-commit phase (§10.1 steps 14–17: delivery-lane execution, reentry passes, side effects, async hooks). Slowness here MUST NOT invalidate the committed event. Each external delivery call is bounded by its transport channel's or provider's own timeout (the framework adds none around deliver() or on_event()), and failures are reported per channel via delivery_failed framework events — never by changing the source event's status or the returned result to blocked/failed. A degraded broadcast SHOULD surface as broadcast_partial_failure, with the source event remaining DELIVERED.

A regeneration re-broadcasts an event already committed, so it has no commit point of its own: process_timeout bounds its wait for the room lock and its choice of the event to replay, and MUST NOT bound the broadcast, which runs in the room's delivery lane off the lock like any post-commit phase (§13.5). On expiry it returns InboundResult(blocked=true, reason=process_timeout) and emits process_timeout with operation = regenerate. The room's status is read again when the lane takes the regeneration: a room closed while it waited refuses it as at its start (§5.1), before the agent runs, with InboundResult(blocked=true, reason=room_closed) and room_refused_event.

Implementations MAY additionally model outbound delivery with an explicit PENDING → COMMITTED | FAILED state and an outbox, plus recovery of events left PENDING, so that a crash between commit and delivery is recoverable. Such a state applies to delivery, not to the committed timeline event.

---

14. Storage Interface

14.1 Conversation Store

Implementations MUST provide a pluggable storage backend via the following interface:

ConversationStore (interface)
│
├── Rooms
│   ├── create_room(organization_id, metadata) → Room
│   ├── get_room(room_id) → Room | null
│   ├── update_room(room_id, updates) → Room
│   ├── delete_room(room_id) → void
│   ├── list_rooms(filters) → list<Room>
│   ├── find_room(filters) → Room | null
│   ├── find_latest_room(filters) → Room | null
│   ├── find_room_id_by_binding(channel_id, participant_id, status) → string | null
│   │                                         # newest room whose binding of
│   │                                         # channel_id names participant_id
│   └── find_room_id_by_participant(participant_id, status) → string | null
│                                             # newest room where participant_id
│                                             # is a participant (§10.4 step 1)
│
├── Events
│   ├── add_event(room_id, event) → RoomEvent
│   ├── get_event(event_id) → RoomEvent | null
│   ├── list_events(room_id, filters) → list<RoomEvent>
│   ├── get_event_count(room_id, filter | null) → int
│   ├── update_event(event_id, updates) → RoomEvent
│   └── delete_event(room_id, event_id, cascade_replies = true) → list<string>
│
├── Bindings
│   ├── create_binding(room_id, binding) → ChannelBinding
│   ├── get_binding(room_id, channel_id) → ChannelBinding | null
│   ├── update_binding(binding) → ChannelBinding
│   ├── delete_binding(room_id, channel_id) → void
│   └── list_bindings(room_id) → list<ChannelBinding>
│
├── Participants
│   ├── create_participant(room_id, participant) → Participant
│   ├── get_participant(participant_id) → Participant | null
│   ├── update_participant(participant_id, updates) → Participant
│   ├── delete_participant(participant_id) → void
│   └── list_participants(room_id) → list<Participant>
│
├── Identity
│   ├── create_identity(identity) → Identity
│   ├── get_identity(identity_id) → Identity | null
│   ├── resolve_identity(channel_type, address, organization_id | null) → Identity | null
│   └── link_address(identity_id, channel_type, address, organization_id | null) → void
│       # The same address may belong to a different identity in each organization
│
├── Verification (§11.7; Level 2)
│   ├── get_verification(identity_id, organization_id | null) → Verification | null
│   ├── put_verification(verification) → Verification
│   ├── list_verifications_in_force() → list<Verification>
│   │   # Re-arms the expiry timers when the implementation starts
│   ├── create_verification_request(request) → VerificationRequest
│   ├── get_verification_request(request_id) → VerificationRequest | null
│   └── update_verification_request(request) → VerificationRequest
│
├── Tasks
│   ├── create_task(room_id, task) → Task
│   ├── list_tasks(room_id, status | null) → list<Task>
│   └── update_task(task) → Task
│
└── Observations
    ├── create_observation(room_id, observation) → Observation
    └── list_observations(room_id) → list<Observation>

Reading a timeline (list_events). Two pagination modes exist: offset-based (offset + limit) and cursor-based (after_index or before_index, mutually exclusive; offset is then ignored). Which events a page holds and the order they come back in are two different answers. before_index, and in offset mode the newest_first flag (ignored under a cursor), select the window: the limit events nearest the cursor, or the room's tail instead of its head, with offset then counted back from the newest end. The page itself MUST always be rendered in ascending index order, whichever window was selected and whatever the order the rows were written or stamped in (ties between events stored without an index are broken deterministically by the store): its first element is the oldest event of the window and its last element the newest. A caller that wants the most recent event of a page reads its last element, never its first; an implementation that fetches descending to find the window reverses the page before returning it. The conversation and timeline reads built on list_events inherit both halves of this rule.

A timeline read serves what the room received. By default list_events MUST NOT return an event stored BLOCKED (§10.1 step 10, §7.5 rule 2, §8.3): the room refused that turn, and a reader listing what was said in the room is not asking for what was refused. The refused rows MUST stay readable on explicit request (a filter flag, include_blocked) and by id (get_event), because a host audits, copies or deletes them by those paths, and the filter MUST apply before the page is cut, so a page of limit received events is full whatever was refused around them. The conversation read that fills RoomContext.recent_events asks for the whole timeline: hooks read it whole (§7.5 rule 8), and the per-reader filter of that rule drops the refused rows for channels. A refused row still consumes an index (§10.1 step 8), so the room's counters keep counting it.

A count follows the rule of the page it stands for. get_event_count without a filter counts every committed row, the refused ones included: a refused row consumed an index (§8.3), and that count is the timeline's size, the exact twin of the room's maintained event_count. get_event_count with an EventFilter MUST count exactly the rows list_events would serve under that filter, with no page: the same criteria, and the same received-rows default lifted by include_blocked, so a host that needs the exact number of a subset (an agent's turns before an instant) never measures a page to get it and never inherits the page's cap.

14.2 Required Implementations

Implementation	Purpose	Conformance
In-Memory Store	Testing, prototyping, single-process	MUST provide
Persistent Store	Production (SQL, document DB)	SHOULD provide

14.3 Consistency Requirements

• Event index assignment MUST be atomic within a room. For a persistent store that may be accessed by more than one process, this atomicity MUST be enforced at the storage layer — a single transaction plus a UNIQUE(room_id, index) constraint (§8.1) — not by an in-process lock alone.
• Room state updates MUST be consistent with event storage. Specifically, storing an event — DELIVERED at §10.1 step 12, BLOCKED at steps 10–11 — and bumping the room's event_count / latest_index form a single atomic commit: an observer MUST never see a stored event that is not reflected in the room counters, nor counters that count an event absent from the timeline.
• Once committed (DELIVERED), an event MUST NOT be retroactively re-marked BLOCKED or FAILED. A processing timeout or a delivery failure after the commit point (§13.6) MUST NOT alter the event's committed status.
• Idempotency key checks MUST be performed under the room lock.

14.4 Returned-Object Ownership

• A committed event changes only through the write interface (§4.3). An event object returned by a read (get_event, list_events, conversation reads) is a snapshot of the committed record: the caller MUST treat it as frozen and MUST NOT mutate it. A store MAY return the same object to multiple readers (an in-memory store sharing its stored objects) or a fresh object per read (a SQL store deserialising rows) — both are conformant, and a portable caller can rely on neither aliasing nor isolation.
• A store MUST own its stored representation from the moment a write returns: a caller's later mutation of an object it passed to add_event / update_event MUST NOT alter stored state (copy-in and serialisation both satisfy this).
• Rooms, bindings and participants returned by reads are caller-owned: mutating a returned object MUST NOT affect stored state — changes are applied through the write interface.

---

15. Observability

15.1 Logging

Implementations MUST use structured logging with named loggers. The following logger hierarchy is RECOMMENDED:

roomkit
├── roomkit.core.framework
├── roomkit.core.router
├── roomkit.core.hooks
├── roomkit.core.locks
├── roomkit.channels.sms
├── roomkit.channels.email
├── roomkit.channels.websocket
├── roomkit.channels.ai
├── roomkit.channels.whatsapp
├── roomkit.channels.voice
├── roomkit.channels.voice.pipeline
├── roomkit.providers.sms.*
├── roomkit.providers.email.*
├── roomkit.providers.ai.*
├── roomkit.providers.voice.*
├── roomkit.identity
└── roomkit.store

15.2 Log Levels

Level	What to Log
DEBUG	Full pipeline trace, raw payloads, hook decisions
INFO	Room created, event stored, channel attached, participant resolved
WARNING	Delivery failed (retryable), hook timeout, chain depth approaching limit, an AI turn that ended before its answer (a delegated worker's, a reasoning backend's: an expected outcome, logged without a traceback)
ERROR	Provider error (non-retryable), circuit breaker opened, store failure

15.3 Structured Log Context

Each log entry SHOULD include structured context:

{
    "room_id": "room_8f3a",
    "event_id": "evt_abc",
    "channel_id": "sms_customer",
    "provider": "twilio",
    "chain_depth": 0,
    "status": "sent",
    "latency_ms": 245
}

15.4 Voice Pipeline Observability

Voice pipeline stages SHOULD emit structured metrics for monitoring and debugging. The following metrics are RECOMMENDED:

Metric	Source	Description
pipeline.stage_latency_ms	All stages	Processing time per stage per frame
pipeline.stt_latency_ms	STTProvider	Time from speech end to final transcript
pipeline.tts_latency_ms	TTSProvider	Time from text to first audio frame
pipeline.vad_speech_duration_ms	VAD	Duration of detected speech segments
pipeline.vad_false_positive_rate	VAD	Speech detections that produced no transcript
pipeline.aec_convergence	AEC	Echo cancellation effectiveness (0.0 to 1.0)
pipeline.agc_gain_db	AGC	Current applied gain
pipeline.dtmf_detected	DTMF	Count of DTMF digits detected per session
pipeline.recording_duration_s	Recorder	Active recording duration
pipeline.turn_detector_latency_ms	TurnDetector	Time to reach turn decision
pipeline.backchannel_rate	BackchannelDetector	Ratio of backchannels to interruptions
pipeline.barge_in_count	InterruptionConfig	Count of barge-in events per session
pipeline.debug_tap_bytes_written	PipelineDebugTaps	Total bytes written to debug tap files
pipeline.tts_context_turns	Voice Channel	Turns in the TTS context passed to a call, per session (Section 12.2.2)
pipeline.tts_context_audio_s	Voice Channel	Seconds of audio held in the TTS context, per session
transport.packets_lost	VoiceBackend	Count of packets confirmed lost per session (Section 12.2.1)
transport.concealed_frames	VoiceBackend	Count of frames synthesized by packet loss concealment per session

Voice pipeline logs SHOULD use the roomkit.channels.voice.pipeline logger with the following structured context fields: session_id, room_id, stage_name, latency_ms, and frame_timestamp_ms.

15.5 Framework Events for Monitoring

See Section 8.2 for the complete list of framework events; an implementation owes the events of the features it provides. These MUST be emittable and subscribable by integrators for monitoring dashboards, alerting, and integration purposes.

15.6 Protocol Trace Infrastructure

Channels that interact with transport-level protocols (SIP, RTP, SMTP, etc.) SHOULD emit ProtocolTrace records (Section 5.14) for significant protocol events. The framework provides a layered trace infrastructure that routes these traces to integrator-registered hooks at the room level.

Channel-level trace API:

Channel (trace extensions)
├── emit_trace(trace: ProtocolTrace) → void
│       # Emit a trace to all registered handlers
│       # Called by backend/transport implementations
│
├── on_trace(callback, protocols: list<string> | null) → void
│       # Register a user-level trace callback
│       # If protocols is set, only traces matching the filter are forwarded
│
├── trace_enabled: bool (read-only)
│       # True if any trace handler is registered (user or framework)
│
└── resolve_trace_room(session_id: string | null) → string | null
        # Resolve a room_id from a session_id via session bindings
        # Default: null (override in voice channels)

emit_trace() invokes all registered handlers: user-level callbacks (registered via on_trace()), and the framework handler (set during register_channel()). Handlers MAY be sync or async — async handlers MUST be scheduled as tasks and MUST NOT block the caller.

Trace bridging for voice channels:

Voice channels (VoiceChannel, RealtimeVoiceChannel) bridge traces from their underlying backend or transport to the channel's emit_trace(). This is done via set_trace_emitter() on the backend/transport:

VoiceChannel:
    # Calls backend.set_trace_emitter(self.emit_trace) when trace is enabled

RealtimeVoiceChannel:
    # Calls transport.set_trace_emitter(self.emit_trace) when trace is enabled
    # Transport MAY forward to underlying backend (e.g., SIPRealtimeTransport → SIPVoiceBackend)

The bridge is activated lazily — only when trace_enabled becomes true (via on_trace() registration or framework handler assignment).

Framework-level trace routing:

When a channel is registered via register_channel(), the framework sets a framework trace handler on the channel. This handler:

1. Resolves the room_id for the trace:
2. Uses trace.room_id if set directly.
3. Falls back to channel.resolve_trace_room(trace.session_id) to look up the room via session bindings.
4. Fires the ON_PROTOCOL_TRACE hook for the resolved room.

Pre-room trace buffering:

Transport-level traces (e.g., SIP INVITE) often fire before process_inbound() creates the room. Since hooks require a room context, these traces cannot be delivered immediately. The framework MUST buffer such traces and replay them when the room is created and the channel is attached:

1. Backend emits trace (e.g., SIP INVITE accepted)
2. Framework handler tries to fire ON_PROTOCOL_TRACE hook
3. Room does not exist yet → buffer trace in _pending_traces[room_id]
4. process_inbound() creates room, attaches channel
5. attach_channel() calls _flush_pending_traces(room_id)
6. Buffered traces are replayed as ON_PROTOCOL_TRACE hooks

This ensures that no protocol traces are lost, even for the initial signaling messages that precede room creation.

SIP trace examples:

A SIP voice backend SHOULD emit the following traces:

Event	Direction	Protocol	Summary	Raw
Call received	inbound	sip	INVITE from +1555... to +1666...	Serialized SIP INVITE request
Call accepted	outbound	sip	200 OK (codec=16000Hz, ...)	SDP answer body
Remote hangup	inbound	sip	BYE from +1555...	Serialized SIP BYE request
Local hangup	outbound	sip	BYE (local hangup)	null (library may not expose serialized form)

15.7 Telemetry Provider

Implementations SHOULD provide a pluggable telemetry provider abstraction for distributed tracing and metrics collection:

TelemetryProvider (interface)
├── start_span(name, attributes, parent) → Span
│       # Begin a new trace span for an operation
│
├── record_metric(name, value, attributes) → void
│       # Record a numeric metric value
│
├── record_event(name, attributes) → void
│       # Record a discrete event within the current trace
│
└── close() → void
        # Flush pending data and release resources

Required implementations:

Provider	Description
OpenTelemetryProvider	Industry-standard distributed tracing (OTLP export)
ConsoleTelemetryProvider	Human-readable console output for development
NoopTelemetryProvider	No-op implementation (telemetry disabled)

Implementations SHOULD instrument the following operations with spans: process_inbound, broadcast, hook_execution, deliver, ai_generate, stt_transcribe, tts_synthesize, identity_resolve, and store_event.

Each span SHOULD include structured attributes: room_id, channel_id, event_id, provider, and latency_ms.

15.8 Audit System

RoomKit provides a two-tier auditing architecture for compliance, debugging, and monitoring: tool auditing captures individual tool calls, and session auditing captures the full conversation timeline.

15.8.1 Tool Auditing

Tool auditing records every tool call with input, output, timing, and status.

ToolAuditEntry:

ToolAuditEntry
├── ts: string                              # ISO 8601 timestamp
├── agent_id: string                        # Which agent made the call
├── tool_name: string                       # Tool function name
├── arguments: map<string, any>             # Input arguments
├── result: string                          # Tool output (truncated to 500 chars)
├── status: string                          # "ok", "failed", "error" or "cancelled"
├── duration_ms: float                      # Execution time in milliseconds
└── metadata: map<string, any>              # Optional extra data

Status detection: - "error" — set when the tool handler raises an exception. - "cancelled" — set when the call was cancelled before its handler answered (its turn or session ended, the provider abandoned it). A cancelled call MUST NOT be recorded as "ok", and the cancellation still reaches the caller. A call the channel's per-call bound cut short (Section 21.6) reaches a wrapped handler as a cancellation and is recorded "cancelled" as well, its duration the bound; the channel reports it as failed. - "failed", structurally — set when the handler refused the call (ToolRefusedError), stated its failure in its own words (ToolFailedError, an MCP isError result: the structured form of the envelope below, which reads "failed" too) or declined it as not its own (Section 21.4). - "failed", from the body — read from the result body, which MUST be inspected for every failure envelope the implementation itself emits, not for one convention only: {"status": "failed"}, the {"error": ...} envelope a refused call and a wrapped MCP isError result return, and the {"success": false} convention hosts commonly use. Recognising one shape and not the others records a denied tool — the outcome an audit exists for — as "ok". - "ok" — the default, and a degradation rather than a contract: it is what a body outside every envelope reads as. A producer that can state its outcome structurally SHOULD do so (Section 9.3) instead of relying on this reading.

ToolAuditor (interface):

ToolAuditor (interface)
├── record(entry: ToolAuditEntry) → void    # Record a tool execution
├── entries: list<ToolAuditEntry> (property) # All recorded entries
├── summary() → string                     # Human-readable summary
└── print_summary() → void                 # Log summary via logger

Built-in implementations:

Implementation	Persistence	Description
JSONLToolAuditor	JSONL file	Appends one JSON line per call; maintains in-memory list
ConsoleToolAuditor	Logger	Real-time logging via Python logging module

Handler wrapper:

The audit_tool_handler(handler, auditor, agent_id) function wraps any tool handler to automatically record audit entries. It:

1. Measures execution time.
2. Records a raised exception as "error", a refusal, a stated failure (ToolFailedError) or a declined call as "failed" and a cancelled call as "cancelled", then re-raises each, so the channel reads the outcome it would have read without the wrapper.
3. Auto-detects "failed" from the response body otherwise.
4. Truncates the recorded result to 500 characters.
5. Returns the original handler result unchanged: the same object, never a printed copy of it, so content parts and structured values reach the channel as the handler gave them.

Integrators MAY implement custom ToolAuditor backends (e.g., Datadog, Elasticsearch) by implementing the interface.

15.8.2 Session Auditing

Session auditing captures the full conversation timeline: speech turns, tool calls, vision events, interruptions, and session lifecycle.

SessionAuditEntry:

SessionAuditEntry
├── ts: string                              # ISO 8601 timestamp
├── type: string                            # Event type (see below)
├── role: string | null                     # "user", "assistant", or "system"
├── content: string                         # Text, transcription, or description
├── duration_ms: float | null               # For timed events (tool calls)
└── metadata: map<string, any>              # Extra data (tool_name, args, status, etc.)

Session audit event types:

Type	Role	Trigger	Content
speech	user/assistant	ON_TRANSCRIPTION hook	Transcribed speech text
tool_call	assistant	Manual record_tool()	Tool result
vision	system	ON_VISION_RESULT hook	Frame description
barge_in	user	ON_BARGE_IN hook	"User interrupted"
session	system	ON_SESSION_STARTED hook	"Session started"
error	system	Application-specific	Error message

SessionAuditor (interface):

SessionAuditor (interface)
├── record(entry: SessionAuditEntry) → void
├── entries: list<SessionAuditEntry> (property)
├── summary() → string
├── print_summary() → void
│
│   # ToolAuditor compatibility bridge:
├── record_tool(entry: ToolAuditEntry) → void  # Convert tool entry to session entry
├── tool_auditor: ToolAuditor (property)        # Bridge adapter
│
│   # Hook auto-registration:
└── attach(kit: RoomKit) → void                # Register hooks for automatic capture

Hook auto-registration: When attach(kit) is called, the session auditor registers hooks for ON_TRANSCRIPTION, ON_BARGE_IN, ON_SESSION_STARTED, and ON_VISION_RESULT to capture conversation events automatically.

ToolAuditor bridge: The tool_auditor property returns a bridge adapter that converts ToolAuditEntry records into SessionAuditEntry records. This allows a single SessionAuditor to capture both tool calls and conversation events in a unified timeline.

Built-in implementation: JSONLSessionAuditor — writes entries to a JSONL file and maintains an in-memory chronological list.

15.8.3 Audit Data Format

Both auditors use JSONL (JSON Lines) for persistence — one JSON object per line, independently parseable:

{"ts":"...","agent_id":"exec","tool_name":"search","arguments":{"q":"roomkit"},"result":"...","status":"ok","duration_ms":1234.5,"metadata":{}}
{"ts":"...","type":"speech","role":"user","content":"Open Chrome","duration_ms":null,"metadata":{}}

JSONL files are append-only. Implementations MUST create parent directories automatically. Write errors MUST be logged but MUST NOT propagate exceptions to the caller.

---

16. Integration Surfaces

The RoomKit core MUST NOT depend on any specific web framework. Integration surfaces are thin wrappers that expose core functionality.

16.1 REST API (OPTIONAL)

The framework is a library, and the host owns its HTTP surface. An implementation that offers a REST API SHOULD provide the following endpoints:

Rooms:

POST   /rooms                                   # Create room
GET    /rooms/{id}                              # Get room
PATCH  /rooms/{id}                              # Update room
DELETE /rooms/{id}                              # Delete room
GET    /rooms?organization_id=X&status=active   # List rooms

Channels:

POST   /rooms/{id}/channels                     # Attach channel
DELETE /rooms/{id}/channels/{cid}               # Detach channel
PATCH  /rooms/{id}/channels/{cid}               # Update binding (access, visibility)
POST   /rooms/{id}/channels/{cid}/mute          # Mute channel
POST   /rooms/{id}/channels/{cid}/unmute        # Unmute channel
GET    /rooms/{id}/channels                     # List bindings
GET    /channels                                # List registered channels

Events & Timeline:

POST   /rooms/{id}/events                       # Inject event
GET    /rooms/{id}/timeline                     # Get timeline (supports pagination, visibility filter)

Participants:

POST   /rooms/{id}/participants                 # Add participant
DELETE /rooms/{id}/participants/{pid}            # Remove participant
GET    /rooms/{id}/participants                 # List participants
POST   /rooms/{id}/participants/{pid}/resolve   # Resolve pending identity

Tasks & Observations:

GET    /rooms/{id}/tasks                        # List tasks for room
PATCH  /tasks/{id}                              # Update task status
GET    /rooms/{id}/observations                 # List observations for room

Identity:

POST   /identities                              # Create identity
GET    /identities/resolve?channel=sms&address=+1... # Resolve identity
POST   /identities/{id}/addresses               # Link a channel address to an identity

Webhooks:

POST   /webhooks/{channel_type}/{provider}          # Inbound webhook
POST   /webhooks/{channel_type}/{provider}/status   # Delivery status webhook

WebSocket:

WS     /ws/{room_id}                            # Real-time room connection (WebSocket channel, Appendix A.8)

16.2 MCP Server (OPTIONAL)

For AI agents to interact with rooms natively via the Model Context Protocol:

MCP Tools (actions):

Tool	Description
send_message	Send a message to a room
create_task	Create a task in a room
add_observation	Record an observation
attach_channel	Attach a channel to a room
detach_channel	Detach a channel
mute_channel	Mute a channel
unmute_channel	Unmute a channel
set_channel_visibility	Change visibility
update_room_metadata	Update room metadata
resolve_identity	Resolve an identity
escalate_to_human	Escalate to a human agent

MCP Resources (data):

Resource	Description
room://{room_id}	Room state
room://{room_id}/timeline	Room timeline
room://{room_id}/participants	Room participants
room://{room_id}/channels	Room channel bindings
room://{room_id}/tasks	Room tasks
identity://{identity_id}	Identity details

16.3 Surface Independence

Both REST and MCP surfaces MUST call the same core methods. No business logic should live in the integration surface layer.

---

17. Security Considerations

17.1 Input Validation

• All inbound payloads MUST be validated before processing.
• Webhook signatures SHOULD be verified when the provider supports them.
• Event content SHOULD be sanitized to prevent injection attacks.

17.2 Multi-Tenant Isolation

• Rooms are scoped by organization_id.
• Implementations MUST ensure that room operations are isolated per organization.
• Identity resolution MUST be scoped to the organization.
• A room the framework creates for another room's work (a delegation's child room, Section 23.3) MUST belong to that room's organization.
• A door that admits a participant into a room's media (a framework join, a realtime channel's session start, a conference channel's mint_access()) MUST read the room with the organization the caller acts for, before any session exists or is bound and before any credential is minted: another organization's room is not found, no session joins it, no credential admits to it, and none of its recordings is told of the session's tracks. Called without an organization, a framework join reads the room unscoped, and a session start or a mint reads none for the scope.

17.3 Sensitive Data

• raw_payload MAY contain sensitive data. Implementations SHOULD support configurable redaction or encryption at rest.
• Hook-based PII scanning (e.g., SIN/SSN detection) SHOULD be used to prevent sensitive data from traversing channels.
• Blocked events are stored for audit but their content SHOULD be handled according to the organization's data retention policy.

17.4 Rate Limiting

• Per-channel rate limits prevent abuse of external provider APIs.
• Implementations SHOULD support per-organization rate limits.

17.5 Chain Depth

• The chain depth limit prevents resource exhaustion from unbounded AI ↔ AI loops.
• Implementations MUST enforce the limit and MUST NOT allow it to be disabled, a discussion's own max_depth included (Section 19.7.5).

17.6 Voice and Audio Security

Recording consent and compliance:

• Implementations that support audio recording MUST provide a mechanism for recording consent management. In many jurisdictions (TCPA, GDPR, PIPEDA), recording requires explicit consent from all parties.
• The framework SHOULD fire ON_RECORDING_STARTED before any audio is captured, giving integrators the opportunity to notify participants.
• Implementations SHOULD support configurable consent modes: SINGLE_PARTY (one party consents), ALL_PARTY (all parties must consent), or NONE (integrator manages consent externally).

Audio data handling:

• Audio recordings MUST be encrypted at rest. Implementations MUST support configurable encryption for stored recordings. Every recording configuration carries it (encryption, storage_encrypted_at_rest; Sections 12.3.7, 12.10.8, 12.11), and a recorder that stores files fails closed when neither is set.
• Audio streams in transit SHOULD use encrypted transport (TLS, SRTP, DTLS).
• STT and TTS provider calls transmit audio to external services. Implementations SHOULD document which providers receive audio data and SHOULD support configurable provider selection based on data residency requirements.
• Voice session metadata (transcripts, recordings, DTMF digits) MUST follow the same data retention and redaction policies as other room events.

TTS conversation context (Section 12.2.2):

• Context audio is a copy of the session's dialogue audio (what the user said and what was synthesized), held so the TTS can hear it. It MUST be kept in memory only: never written to the conversation store, the timeline, a recording, or a log.
• It MUST stay within max_turns and max_audio_seconds, and MUST be dropped when the session is unbound or ends, together with the call to release_context.
• Keeping audio MUST be opt-in (include_audio = false by default). The text of the context is the transcript already stored as room events, and is under the same policies.
• A user turn whose transcript a hook modified MUST NOT keep its audio. When DTMF redaction is enabled, a user turn during which DTMF was detected MUST NOT keep its audio either: in-band tones carry the digits.
• A provider at level AUDIO that runs as an external service receives the user's voice on every call, not only the current sentence. Implementations SHOULD say so where they document which providers receive audio data.

Custom voices (Section 12.2.4):

• A recording of a person's voice made to replicate it, and the recording of their consent, are personal data identifying that person. An implementation MUST hand them to the vendor for the call and keep nothing: never written to the conversation store, the timeline, a recording, or a log. What it keeps is the evidence the integrator needs, as metadata: which voice was created, when, by whom, and whether the vendor verified the consent.
• Replicating a voice without the consent of the person it belongs to MUST NOT be possible through the framework (Section 12.2.4). The integrator remains responsible for that consent, and for its withdrawal: a withdrawn consent means deleting the voice.
• A stored custom voice is held by the vendor under the integrator's account. Implementations SHOULD document where it lives and for how long, as they do for the providers that receive audio.

Provenance of synthesized audio:

• A vendor MAY mark the audio it synthesizes: a manifest in the file (C2PA) or a watermark in the signal. An implementation MUST NOT remove a mark it received: a file is passed on as the vendor produced it, not rebuilt from its samples. A stage that must decode the audio (resampling, mixing, encoding for a transport) loses a file-level manifest by construction; implementations SHOULD document where that happens.

DTMF sensitivity:

• DTMF digits MAY contain sensitive data (credit card numbers, PINs, account numbers). Implementations MUST support configurable DTMF redaction — the ability to mask DTMF digits in stored events and logs (e.g., replacing "4111111111111111" with "4111**1111").
• When DTMF redaction is enabled, raw digits MUST NOT appear in raw_payload, transcripts, or log output.

Voice model privacy:

• Audio recordings and transcripts MUST NOT be used for model training without explicit integrator and end-user consent.
• Implementations SHOULD provide a configuration flag to disable any data sharing with STT/TTS/speech-to-speech providers beyond what is required for the API call.

---

17.7 Conference Access Security

ConferenceAccess tokens are credentials:

• Tokens SHOULD be minted per participant with least-privilege grants (e.g., hidden observers receive subscribe-only grants). ConferenceGrants (Section 12.10.2) defaults to permissive values so that the common case works unconfigured; narrowing them for a given participant is the integrator's call, and is RECOMMENDED wherever a role does not need to publish.
• Tokens SHOULD be short-lived (expires_at) and MUST NOT be logged.
• Moderation operations (remove_participant, mute_track, unmute_track) MUST only be reachable through integrator-controlled code paths, never exposed unauthenticated.

Bot disclosure. Whether a recording or transcribing bot must be announced to human participants — and how — is set by the jurisdiction the deployment operates in, and those rules differ by country and by sector. This specification therefore mandates no disclosure behaviour and no restriction on hidden observers (Section 12.10.6). It requires instead that the framework give the integrator what it needs to comply with whatever applies to it:

• The bot's presence, its identity, and its hidden status MUST be observable to integrator code — the status in force on the session, not the one it was configured or admitted with: grants can change while a session runs (Section 12.10.4), and a surface reporting the construction answers a different question than the one a disclosure obligation asks.
• Whether STT, vision, or recording is active on a conference MUST be observable at any time, not only at configuration time.
• ON_CONFERENCE_PARTICIPANT_JOINED MUST fire for every human participant the backend delivered, giving the integrator the point at which to announce an active bot, gate entry on consent, or record that disclosure was made. Across a reported discontinuity (Section 12.10.3) that guarantee cannot reach the participants whose whole presence fell inside the outage window: the discontinuity is the signal that the window is unaccounted, and an integrator whose obligations require complete attendance MUST treat it so rather than as observed-and-empty.

Implementations SHOULD document this responsibility rather than assume a default. Deployments in regulated sectors typically do need visible bot identification; the specification's role is to make that implementable, not to decide it.

17.8 Skill-Bundled Files

• A skill's references and scripts are addressed by a name the model chooses, which makes every such read or execution an untrusted-input boundary.
• Resolution MUST be confined to the skill's own directory by resolving the path and checking containment, never by filtering the name alone (Section 24.6).
• The same containment MUST apply to script execution, including where execution policy is delegated to the integrator.
• A skill an MCP server serves is text from outside (Section 6.4) whatever its digests say: a digest the same server supplies proves its content consistent with its entry, never trustworthy.
• Such a skill's files are read only from the server that serves it, and only those its entry lists. Its scripts never run through run_skill_script, its allowed_tools grants nothing, and while it is active a tool that runs code is served only with the host's approval of that skill (Section 24.7).

---

18. Design Principles

These principles define the conceptual architecture of RoomKit:

1. The Room is the truth. All state lives in the Room. The timeline records everything.

2. Everything is a Channel. SMS, browser, AI, observer — same interface. No special cases.

3. Primitives, not opinions. The framework provides access, mute, and visibility. Business logic decides when to use them.

4. Two output paths. Room events are subject to permissions. Side effects always flow. Muting silences the voice, not the brain.

5. Providers are swappable. Channel type ≠ provider. Twilio and Sinch both provide SMS. Anthropic and OpenAI both provide AI.

6. Hooks intercept. Sync hooks block and modify. Async hooks observe and react. Pipeline architecture.

7. Channels are dynamic. Attach, detach, mute, unmute, reconfigure at any time during a conversation.

8. Channel awareness at generation. AI knows target constraints and media types before generating — not after.

9. Three layers of metadata. Channel.info (instance), ChannelBinding.metadata (per-room), RoomEvent.channel_data (per-event). Never lose data.

10. Direction declares capability. Channels declare inbound/outbound/bidirectional. Permissions restrict per room.

11. Two event levels. Room events (per-room, stored) and framework events (global, for subscribers).

12. Media types are first-class. Text, audio, video — route to compatible channels. Ready for the future.

13. Chain depth safety. Event chains are bounded by a configurable depth limit. Uses existing blocked mechanism — no new concepts.

14. Voice is a channel. Real-time voice follows the same Room/Channel/Event model. STT/TTS are providers. No special "voice API."

15. Sources complement webhooks. Persistent connections (WhatsApp Personal, SSE) push events. Webhooks deliver per-message callbacks. Both feed the same inbound pipeline.

16. Framework-agnostic core. No web framework dependency. Integration surfaces are thin wrappers. Any language, any framework.

17. Audio pipeline is transport-independent. Resampler, AEC, AGC, denoiser, VAD, diarization, DTMF, turn detection, and recording run as a pluggable pipeline between transport and conversation engine. The transport delivers raw frames. The pipeline processes them. Stages are optional and composable. Same pattern as text hooks: preprocessing inbound, postprocessing outbound.

18. Video pipeline mirrors audio. Decoder, resizer, transforms, filters, overlays, vision, and recording run as a pluggable pipeline between the video transport and the conversation engine. Same stage pattern as audio: pluggable providers, optional stages, session-scoped state. Vision results feed back into AI channels and filter contexts.

19. Agents are channels. Multi-agent orchestration uses the same Room/Channel/Event model. Agents are intelligence channels with identity metadata. Routing, handoff, and coordination happen through existing primitives — no special agent API.

20. Orchestration through primitives. Multi-agent routing, handoff, and state management are built on existing Room/Channel/Event/Hook primitives. The router is a BEFORE_BROADCAST hook. State lives in room metadata. Handoffs are tool calls. No new core abstractions required.

21. Memory is pluggable. AI context construction is a swappable strategy, not a hardcoded sliding window. Implementations choose how to build conversation history: recent events, summarized history, vector retrieval, or token-budget-aware truncation.

22. Audit is two-tiered. Tool auditing captures every tool call (input, output, timing, status). Session auditing captures the full conversation timeline (speech, tools, vision, interruptions). Both use pluggable backends — same extensibility pattern as providers and stores.

23. The conference media plane is external. RoomKit orchestrates conferences — rooms, access, participant lifecycle, AI participation — but never forwards media between human participants. An external SFU owns packet routing; RoomKit joins it as one participant among many. Same boundary as storage: the framework defines the interface, the deployment provides the infrastructure.

---

19. Multi-Agent Orchestration

Multi-agent orchestration enables multiple AI agents to collaborate within a single room. Agents are intelligence channels with identity metadata. Routing, handoff, and state management use existing Room/Channel/Event primitives.

19.1 Overview

An Agent is an AIChannel subclass with structured identity:

Agent (extends AIChannel)
├── role: string | null                     # "Senior Engineer", "Triage Agent"
├── description: string | null              # Agent capabilities description
├── scope: string | null                    # "Backend systems", "Billing"
├── voice: string | null                    # TTS voice identifier
├── greeting: string | null                 # Auto-greeting text on session start
├── language: string | null                 # Preferred language
├── auto_greet: bool (default true)         # Play greeting on voice session start
├── identity_in_prompt: bool (default true) # false: the host renders identity itself
│
├── is_config_only: bool (read-only)        # True when no AI provider (speech-to-speech mode)
└── build_identity_block(language) → string # Generate identity context for system prompt

The identity block is appended to the agent's system prompt wherever RoomKit writes one: its turns, a handoff's language change, a realtime pipeline's sessions. An agent built with identity_in_prompt false writes it nowhere (build_identity_block returns nothing), its fields kept for orchestration.

When is_config_only is true, the Agent has no AI provider and serves as a configuration container for speech-to-speech channels (voice, greeting, language metadata). A null provider MUST be accepted — no generation calls are made.

Multiple Agents MAY be attached to a single room. The orchestration system determines which Agent handles each event, unless the event names its recipients itself (§19.3).

19.2 ConversationState

Each room MAY have a ConversationState stored in room metadata:

ConversationState
├── phase: string | null                    # Current conversation phase ("triage", "specialist")
├── active_agent_id: string | null          # Agent currently handling the conversation
├── language: string | null                 # Detected participant language
└── metadata: map<string, any>              # Custom state (intent, escalation level, etc.)

Phase transitions MUST be explicit — either via handoff or programmatic update. The framework fires ON_PHASE_TRANSITION hooks on state changes.

active_agent_id MAY be written by the host, not only by a handoff: a console that switches which agent it is talking to is a legitimate writer of this field. Implementations SHOULD expose a supported way to set it — reaching into room metadata from application code is not one.

19.3 Addressing

Routing rules answer which agent handles this kind of event. Addressing answers a different question, asked per message by whoever sends it: which agents are being asked to act, right now. A room where a human types @codex review hello.py needs the second question, and no rule expresses it.

An event MAY carry an address:

RoomEvent
└── addressed_to: list<channel_id> | null    # null = unaddressed

Addressing is not visibility. Visibility (§7.3) is configured on a binding and answers who may see what a source produces. Addressing is set on an event by its sender and answers who is asked to act. The two are orthogonal: addressing one agent MUST NOT hide the event from any channel visibility would have shown it to.

Normative rules:

1. Addressing constrains intelligence channels only. Delivery to transport channels MUST NOT be affected — a message addressed to one agent still reaches the humans in the room.
2. When addressed_to is non-null, the channels it names are the only intelligence channels solicited for that event. Named ids not bound to the room are ignored; if none of them is bound, no intelligence channel is solicited.
3. An empty list addresses nobody: the event is stored and delivered, and no agent is asked to respond.
4. addressed_to is part of the stored event, so a transcript can show who was asked and a replay reproduces the same solicitation (in a discussion, the same queued turns, Section 19.7.5).
5. An interruption marker (Section 6.4) solicits no intelligence channel, whatever its address or a router's stamp: it says an agent's turn was cut, not something to act on. It is still stored and delivered like any event.

Unaddressed events (addressed_to = null) keep the behaviour of earlier versions: the router decides (§19.4), and with no router installed every eligible intelligence channel is solicited. A discussion decides instead (Section 19.7.5, rule 8).

A discussion (Section 19.7.5) defers the solicitation an address asks for to the addressed agent's turn: the channels the address names are still the only ones asked to act, later, one at a time. It MUST NOT solicit a channel the address does not name.

Proactive RoomKit.deliver() calls carry the same intelligence address through the text publication pipeline (§22.1). A realtime session target is a separate transport selection, not an intelligence address.

Conformance. Addressing itself is Level 0 — the field exists on every event and MUST be honoured when set, which an implementation without intelligence channels satisfies trivially. The router precedence (§19.4, step 0) and the named policies below belong to Level 2 with the rest of this section.

19.3.1 Agent-sourced events

An agent's own output is an event like any other, and by default it solicits the other agents in the room — the chaining Appendix B.4 describes, bounded by max_chain_depth. That is a feature for a pipeline (analyst → writer) and a hazard for a room of independent agents, where two agents answer each other until the depth limit stops them.

Implementations MUST provide both policies (AgentResponsePolicy), selectable per room through the room's agent_response_policy:

Policy	An agent's output solicits
AGENT_CHAIN	every eligible intelligence channel — the behaviour of earlier versions
ADDRESSED_ONLY	only the channels named in addressed_to, if any

AGENT_CHAIN MUST remain the default, so a room written against an earlier version behaves unchanged. Under either policy an explicit address is honoured: an agent MAY address another agent and be answered by it alone. Section 19.7.5 is how: under a discussion, the @ names an agent writes become its event's address.

Per room means for the life of the room, not at its creation only. A room rarely knows when it is created how many agents it will end up holding: one that opens with a single assistant and gains a second when a human asks for it becomes a room of independent agents at that moment, and the policy it needs changes with it. Implementations MUST therefore allow the policy to be set on an existing room. A change applies to events processed after it; an event already broadcast is not reconsidered, because solicitation was decided when it was routed and reopening it would ask an agent to act on a turn that has already closed. A discussion's deferred turn (Section 19.7.5) reopens nothing: it carries out, later, a solicitation decided when the event was routed.

19.3.2 Delivered versus solicited

A channel that is not solicited MUST NOT be asked to produce a response, and MUST NOT be delivered the event either — the implementation skips it. At §10.2 step 3c the two are one decision, not two.

Skipping alone would not settle the question, because the cost is not the same for every channel. An agent whose context is rebuilt from the room's timeline on every turn loses nothing by being skipped. An agent that owns conversational state outside RoomKit — an ACP coding agent holding a session in its own process — permanently lacks whatever it was not delivered. Left there, a room of such agents is not a shared conversation but a bundle of private threads, and nothing in the transcript says so: asked what was said earlier, an agent answers from the only history it has, in good faith, about something else.

The counterpart is therefore normative. An implementation MUST make the room's timeline available to a channel at the moment it is solicited, so a channel that keeps its own state can rebuild what it missed instead of diverging from the room. That rebuilt history is subject to §7.5 rule 8 without exception: it is filtered per reader, and a channel MUST NOT obtain through it an event visibility withheld from it at delivery. Catching up is not a second door into the room.

The rebuild MAY be bounded — by a count of events, by a token budget, by whatever the channel can carry. When it is, the channel SHOULD be told that what it received is partial, and how much is missing. A bound the reader cannot see reproduces the same silent divergence one layer down: an agent that knows its history was truncated can say so, or ask for the rest; an agent that believes it holds the whole room cannot.

Nothing here obliges a channel to read the timeline. An implementation whose channels all rebuild their context from storage each turn satisfies this section by construction — which is why the requirement is that the timeline be available, not that it be consumed.

Conformance. Level 2, with the rest of §19.3.

19.4 ConversationRouter

The ConversationRouter selects which Agent processes each event. It is installed as a BEFORE_BROADCAST sync hook.

ConversationRouter
├── rules: list<RoutingRule>                # Ordered by priority
├── default_agent_id: string | null         # Fallback when no rule matches
└── supervisor_id: string | null            # Always receives events (if set)

RoutingRule:

RoutingRule
├── agent_id: string                        # Target agent
├── conditions: RoutingConditions           # When this rule applies
└── priority: int (default 0)               # Lower = evaluated first

RoutingConditions:

RoutingConditions
├── phases: set<string> | null              # Match when ConversationState.phase is in set
├── channel_types: set<ChannelType> | null  # Match when source channel type is in set
├── intents: set<string> | null             # Match on detected intent
├── source_channel_ids: set<string> | null  # Match on specific source channels
└── custom: function | null                 # Arbitrary predicate (event, context, state) → bool

Evaluation order:

1. If the event is addressed (§19.3), the address decides. The router MUST NOT override it, and steps 1–3 are skipped.
2. If ConversationState.active_agent_id is set and no phase transition occurred, route to the active agent (sticky affinity).
3. Evaluate rules in priority order. First matching rule wins.
4. If no rule matches, use default_agent_id.
5. If supervisor_id is set, the supervisor ALWAYS receives the event (in addition to the selected agent, or to the addressed channels).

This holds under every AgentResponsePolicy, ADDRESSED_ONLY included, because the two rules answer different questions. A policy decides who is solicited to act; the supervisor is not being asked to act, it is watching. Reading ADDRESSED_ONLY as excluding it would blind a supervisor to precisely the traffic it exists to oversee — every unaddressed agent-to-agent exchange — and the role would be worth nothing in the deployments that need it most. The policy table in Section 19.3.1 describes solicitation, and it is not narrowed by this step.

Step 0 exists because sticky affinity would otherwise swallow every explicit request. With the active agent consulted first, a rule written to honour an address is unreachable for as long as any agent holds the conversation — the two mechanisms could not coexist. An address is a direct instruction from the sender and outranks both the affinity and the rules.

The router stamps routing metadata on the event. The Event Router skips non-targeted intelligence channels during broadcast.

19.5 ConversationPipeline

A ConversationPipeline generates RoutingRule entries from a linear stage definition:

ConversationPipeline
├── stages: list<PipelineStage>
├── default_phase: string | null            # Initial phase (first stage if null)
└── supervisor_id: string | null            # Supervisor for all stages

PipelineStage:

PipelineStage
├── phase: string                           # Phase name ("triage", "specialist", "closing")
├── agent_id: string                        # Agent handling this phase
├── next: string | null                     # Next phase on handoff (null = terminal)
├── can_return_to: set<string>              # Phases this stage can return to
└── description: string | null              # Human-readable stage description

The pipeline generates one RoutingRule per stage with conditions.phases = {stage.phase}.

On a realtime channel. A pipeline MAY drive a speech-to-speech session (Section 12.4) instead of routing events: the active agent is then the session's configuration (its prompt, its voice, its tools), and a handoff reconfigures the session to the next agent's. The active agent is the room's: a session that starts in a room starts with that room's active agent, a handoff in one room reconfigures that room's sessions, and neither changes another room's sessions nor the channel's configuration for future sessions (Section 19.7). The tools an agent's configuration declares are the channel's own tools, the agent's tools and the handoff tool; a pipeline MUST NOT drop the channel's tools, nor declare an agent without its own. The channel's tools are read as they are when a session starts or a handoff lands: a channel configured after the install keeps them under every agent. A call to one of the active agent's tools is served by the handler the agent was given; a tool the agent declares without one is served by the channel's handler. A name the channel's tools carry is the channel's (Section 21.1): declared with the channel's schema and served by its handler, and an agent's tool of that name is neither declared nor served, a warning naming it at the install, nor declared once the channel drops its own: nothing of the agent's serves it. The channel's tools here are its host definitions, every name its human-input tools serve, declared or not, and the tools it serves itself (Tool Search's and the skills', whether or not a session declares them now); for a pipeline agent's own tool, this skip takes the place of the refusal Section 21.1 gives a tool orchestration sets up over the channel's, so the install goes through with the agent's other tools. The active agent's tool policy holds on its session beside the channel's: a tool that either policy denies, each resolved for the session's participant, is neither declared, nor listed by Tool Search, nor among the names a handler reads, and the gate refuses it with the policy's words. A handoff applies the next agent's policy, read for each session's participant, to every session of the room before any declares the next agent's tools. The gate MUST judge a call by the agent the room talks to when the call reaches it, the one that serves the call, whatever a session still declares. A realtime session serves the channel's skills, human-input tools and MCP servers' instructions (Section 21.2), never an agent's, and serves no agent's planner, sandbox or external tool handler: an agent that carries one of these, a skill registry even empty (a skill added to it later would open its gated tools with nothing to gate them), is refused at the install, before anything is installed, each cause named, with the channel's own option where the channel has one (its skills, its human-input handler, its servers' instructions). Its gated tools would run without their skill, and its other ones would be called by a model that was never told of them or of how to use them. A reasoning backend's agent is refused for the same causes by the same rule, its servers' instructions aside, which reach its own model (Section 21.2), and for tools of its own besides (Section 12.4.1).

Allowed transitions: An agent in phase P MAY hand off to: - stage.next (forward progression) - Any phase in stage.can_return_to (backward/lateral) - Transitions to other phases MUST be rejected.

19.6 HandoffHandler

Agents request handoffs by calling the handoff_conversation tool:

HandoffRequest:

HandoffRequest
├── target_agent_id: string                 # Agent to hand off to
├── reason: string                          # Why the handoff is happening
├── summary: string                         # Context summary for the target agent
├── context: map<string, any>               # Additional handoff context
├── channel_escalation: string | null       # "same", "voice", "email", "sms" (channel switch)
└── urgent: bool (default false)            # Priority flag

HandoffResult:

HandoffResult
├── accepted: bool                          # Whether the handoff was accepted
├── new_agent_id: string | null             # Active agent after handoff
├── new_phase: string | null                # Phase after handoff
├── message: string                         # Human-readable result
└── reason: string                          # Rejection reason (if not accepted)

Handoff protocol:

1. Agent calls handoff_conversation tool with HandoffRequest.
2. HandoffHandler validates:
3. Target agent exists and is attached to the room.
4. Phase transition is allowed (per pipeline rules, if applicable).
5. If valid:
6. Update ConversationState.phase and active_agent_id.
7. Fire ON_HANDOFF hook.
8. Fire ON_PHASE_TRANSITION hook.
9. Return HandoffResult(accepted=true).
10. If invalid:
11. Fire ON_HANDOFF_REJECTED hook.
12. Return HandoffResult(accepted=false, reason=...).

The handoff summary is injected into the target agent's context so it has continuity, set apart as a summary of the conversation (<conversation_summary>, Section 6.4) under the previous agent's id kept to an identifier's characters. The reason the room's timeline records for the handoff is quoted.

setup_handoff(agent, handler, tool, room_id) wires the handoff tool into an AI channel and serves its calls with the HandoffHandler. tool (default null) replaces the default handoff_conversation definition; room_id (default null) declares it in that one room's turns, as a strategy installed in the room does (Section 19.7), and null declares it in every room the agent serves. Setting it up again where it is already declared is refused.

The room is the call's (normative). A handoff acts on the room of the call that requested it, read from the tool call context (Section 21.4). One agent serves every room it is attached to, so a room captured when the tool was wired, or left behind by a hook of another event, names whichever room came first, not the one asking. A call made outside a tool call has no room to act on, and the tool MUST refuse it rather than guess.

19.7 Orchestration Strategies

The following strategies are common patterns built on router and pipeline primitives. Implementations SHOULD provide helpers for each.

A strategy in several rooms (normative). A strategy is installed per room, but its agents are shared: one agent object serves every room it is attached to, and so does a realtime channel. What a strategy adds for a room is set up for that room on the shared agent or channel, and MUST reach that room only:

• a tool it adds for a room (a supervisor's delegation tools, a delegation's result tool, a loop's tool, a handoff tool) is declared in that room's turns and in that room's realtime sessions, not in every room the agent or the channel serves, and a call to it from another room is served as a call to an undeclared tool;
• the configuration that tool runs with (the workers, the reviewers, the strategy, the limits, the handoff's routing) is the one installed for the room of the call: a second room's install, with another configuration, runs its own, and changes nothing the first room runs;
• a turn the strategy runs in place of an agent's (a loop's produce and review, a supervisor's delegation passes) runs in the rooms it was installed in only, and an agent attached to a room with no such install answers there as itself;
• on a realtime channel, the active agent's configuration (Section 19.5) is the room's.

What a strategy hands a model (normative). A strategy that composes a model's input from other models' outputs (the workers' results a supervisor reviews or presents, the team's work a next worker builds on, the content a reviewer judges, a reviewer's feedback a producer revises with) sets each output apart in a <worker_output> block of its own under its author's label: one output cannot close its block, nor pass itself off as another worker's or as a verdict. The user's goal or task it copies into such an input is a <task> block. The input a model acts on as its own (the task framed for a worker, a worker's own task) stays that model's input and is not set apart. When several people speak in the room (the conversation's threshold, Section 6.4; a one-to-one conversation left as it is), the user's task such an input holds as a <task> block is headed by who asked for it, by the label the conversation gives the turn that led to it: Alice asked: before a participant's own words, Requested by Alice (2), in the delegating agent's words: before a task the delegating agent wrote. The heading names the asker of the task the block holds and nothing else: the runtime's own prompts and the input a worker acts on as its own carry no asker.

No install and no turn writes into the shared agent or channel on behalf of one room: not its tools, its handler, its handling of events, its prompt, its voice, nor a binding it shares. A write made for one room is read by every other room the object serves, which would answer, or refuse, another room's user with this room's configuration. Installing a strategy in a second room MUST NOT declare a tool twice in a room. A flag it keeps while work runs for a room is that room's, and its tools act on the room of the call (Sections 19.6 and 23.4). A tool set up on an agent outside any strategy (setup_handoff without a room_id, setup_delegation, setup_realtime_delegation) belongs to the agent whatever the room: it is declared wherever the agent serves, and still acts on the room of the call.

One strategy per room, installed while the room lives (normative). A room holds at most one strategy. An implementation MUST provide install_strategy(room_id, strategy) and uninstall_strategy(room_id) on the framework, and create_room(orchestration=...) installs through the same path:

• Installing on a room that already lives does what creating the room with the strategy does: the strategy's agents are registered and attached as intelligence channels, and its hooks, tools, turn runners and state are set up. The room's timeline stays, and the agents' turns read it (Section 20.1).
• While a strategy is installed, installing another one is refused, whatever the strategy, the host's own included: one rule decides who speaks. uninstall_strategy comes first.
• Uninstalling removes what the install added for that room, and only that: its room hooks, the tools and turn runners it set up on the room's agents, the agents it attached (an agent bound before the install stays bound), and the room state it wrote (ConversationState, a discussion's speak queue). A strategy MAY have an uninstall step of its own, run first (a discussion stops giving turns). The timeline stays. The room then answers by its agent_response_policy, and another strategy may be installed.

19.7.1 Pipeline

Agents are chained linearly. Each agent handles one phase and hands off to the next:

Triage → Specialist → Resolution → Closing

The first agent is the entry point. Only forward transitions (and explicitly allowed backward transitions) are permitted.

19.7.2 Swarm

Every agent can hand off to every other agent. No linear ordering. Useful for collaborative agent pools where any agent may be most appropriate:

Agent A ⇄ Agent B ⇄ Agent C

Sticky affinity keeps the conversation with the current agent until an explicit handoff occurs.

19.7.3 Supervisor

A supervisor agent talks to the user and delegates work to specialist agents. Specialists run in isolated child rooms — their output is collected and returned to the supervisor.

User ↔ Supervisor
              ├── delegates to → Worker A (child room)
              └── delegates to → Worker B (child room)

The supervisor MAY delegate sequentially or in parallel. Results are delivered back via the delivery strategy system (Section 22), at the chain depth of the turn that delegated (Section 23.3); results presented within the turn leave the supervisor's answer one deeper than the event it answers (§8.3). Results of workers dispatched in the background are handed back to the supervisor as a background delegation's are (§23.3 step 8): an instruction addressed to it, each worker's output bounded, never published as a participant's message. A background pipeline that fails before its results (a delegation that raised) hands its failure back the same way: an instruction that the work could not be completed, carrying the reasoning backend's fallback text (Section 12.4.1) and never the error's message, which goes to the logs (Section 9.3) and the status bus (Section 19.8). The supervisor told the user results would follow; it is told they will not, as the dispatching caller of a pipeline run within the turn reads its error. A strategy's background tool serves one run per room at a time, whichever voice channel's session called it, and the outcome goes to the session that made the call (Section 23.3 step 8). The run posts one terminal status entry once its outcome was handed back: completed, or failed when the work failed, none of its workers' tasks completed, or its outcome reached no one. A per-worker tool dispatched in the background is followed by such a run, for that worker in that room: its delegation stays a task of the framework's task runner (its id in the dispatch answer, which cancels it), the run waits for it within the per-task timeout, past which it is cancelled, the worker is free again before its outcome is handed back, and the run alone hands it back. A background run ends with the framework: closing the framework cancels it, which frees its room and posts its terminal entry failed, with nothing handed back.

Each worker's delegation, within the turn or in the background, is bounded by the strategy's per-task timeout: a worker past it is cut, its task ended cancelled, and it reads as failed. The worker is posted pending on the status bus, then one terminal entry however the delegation ends: completed, failed, timed out, raised, or cancelled by its caller. A sequential team is supervised (the supervisor frames and validates each step) whether it runs within the turn or in the background; a supervisor without a model (a configuration-only agent, as a voice supervisor often is) cannot frame nor judge a step, and its team runs unsupervised, each worker given the task and the work done before it. A run whose supervised chain stopped on a step the supervisor left unvalidated did not complete its work, as the supervisor reads it within the turn. A run asked for once the framework began closing does not start.

An instruction a strategy needs for one turn (a supervisor's task-formulation pass) is passed for that turn only, over the prompt the turn would have had. The supervisor, like any channel, serves every room it is attached to, so its configuration carries no turn's instruction: one written there and restored afterwards leaks into the turns of other rooms running meanwhile.

The task-formulation pass is read as any streamed turn is (Section 6.4), and fails as one does: its failure fires ON_ERROR once, as a streamed turn's (streaming, with the correlation of the tool rows it wrote), is logged once, and reaches the room turn's caller without a second report. The task it hands the workers is its final answer, never the narration of its tool rounds joined to it, and a pass its round cap, deadline or budget cut short has none, so no worker runs. A pass that stopped short of its answer (it ended for any reason but completed, save a stop someone chose, cancelled) still answers the message it answered: the supervisor's fallback message, the text a reasoning backend speaks when its delegated work could not be completed (Section 12.4.1), stored and delivered as the supervisor's response, carrying the turn's record (its loop_end_reason, its ai_usage, what the turn wrote) as the turn's last message would. A pass that failed with an error has no fallback: the error is the turn's, as on any turn, logged once at its own level. Nothing of the pass's own stream reaches the room's transports. Its tool calls are stored in the room as any turn's TOOL_CALL rows, through the room's commit gate (Section 10.1) and in the answered event's scope and thread. Its text is not stored: it is the workers' task, an answer to no one in the room.

19.7.4 Loop

A single agent handles the conversation indefinitely, looping back for refinement. Useful for iterative workflows (editing, code review, tutoring). The loop's result is the agent's response to the event that started it, one deeper than that event (§8.3). A producer's task that failed (§23.3 step 6) stops the loop: the last output it produced, if any, is the result, not approved, and says the loop stopped on that failure; with no output at all the turn has no answer, never an empty response passed off as one. Either way the loop's caller reads how the producer's last turn ended under turns, keyed by the producer, with its usage, as a room turn's caller does (§6.4): a turn its round cap, deadline, budget or a stop cut short is read there, is no error and fires no ON_ERROR, as a room turn's does; a turn that failed reaches the caller as the error it raised, its type unchanged (a provider's failure as that provider's error), reported to ON_ERROR once, where the producer's turn failed (its task's room), and logged once. An asynchronous loop (a voice channel's delegate_loop) hands its outcome back to the voice channel that started it as a background delegation's result is (§23.3 step 8): an instruction to that channel, its output bounded and set apart as a worker's, never published as a participant's message. The text says how the loop ended; for a producer's task that failed, that it failed and the cut that ended its turn, never with the task's error. A loop that raised hands back that the work could not be completed, carrying the reasoning backend's fallback text (Section 12.4.1) and never the error's message, which goes to the logs (Section 9.3) and the status bus (Section 19.8), as a supervisor's background pipeline does (Section 19.7.3), one loop per room and in the session that made the call. The room is free for a new loop before the outcome is handed back: the model's turn on it may start one. A loop its producer's task stopped posts its terminal status entry as failed. The producer and the reviewers are delegated as a supervisor's workers are (Section 19.7.3), without a per-task bound: each is posted pending, then one terminal entry however its delegation ends. An asynchronous loop ends with the framework as a supervisor's background run does.

19.7.5 Discussion

Several agents and one or more people hold one conversation in a room. Any agent may address any other, a person may address any agent, and no flow is fixed in advance: who speaks next follows from who was addressed. One agent speaks at a time.

Person ⇄ { Agent A ⇄ Agent B ⇄ Agent C }      one at a time, addressed by @name

Swarm (Section 19.7.2) passes the conversation from one agent to the next: the active agent talks with the user and the others wait for a handoff. In a discussion every agent stays in the conversation and speaks when it is addressed, the way people in a group chat do. Neither replaces the other.

A room of several agents left to the default policy dogpiles: every agent answers the same event at once, each reading the room as it was before the others answered, and every answer solicits the others again (Section 19.3.1). ADDRESSED_ONLY stops the dogpile by stopping the agents from answering each other at all. A discussion keeps them talking and takes the turns itself.

This version covers text: AI channels (Section 6.4), Agents among them, and external agents (ACP agent channels, Section 6.4), in rooms whose people write. Its terms are those of Section 6.4, which speaks for text and voice alike, so that voice can join it without renaming anything (see "Left out").

Discussion
├── agents: list<AIChannel | ACPChannel>      # The agents of the discussion: AI channels (Agents among them) and external agents
├── people: set<string> | null                # Names agents address people by (null: the room's people, §6.4)
├── addressed_only: bool (default false)      # A person's message asks only the agents it names (rule 8)
├── everyone: list<channel_id> | null         # Asked, in this order, by a person's message that answers no one (null: all agents)
├── max_turns: int | null                     # Turns given in the room's lifetime (null: no bound but max_depth)
├── max_depth: int | null                     # The depth limit of its turns (null: the kit's max_chain_depth)
├── done: function(room_id) → bool | null     # The discussion is over once it holds
├── silent_token: string (default "(silent)") # The answer of an agent that stays silent
├── dispatch: DispatchPolicy | null           # Picks who takes a person's message that names no agent (rule 18)
└── dispatch_timeout: float (default 5 s)     # How long a decision may take (rule 18)

SpeakQueue (one per room, readable by the host)
├── speaking: channel_id | null               # The agent whose turn runs now
├── queue: list<channel_id>                   # Agents owed a turn, in the order they get it
├── listening: set<channel_id>                # Agents that only listen (rule 12)
├── asked: list<(channel_id, person)>         # Agents that asked a person, and whom, waiting for the answer
├── waiting: bool                             # Whether the discussion waits for a person
└── over: bool                                # Whether the discussion is over (rule 15)

Normative rules:

1. The room is the discussion's. The intelligence channels the room binds are the discussion's agents, and no other: the install refuses a room that binds another intelligence channel, a voice or realtime channel (Section 12), an agent that thinks while it listens (Section 6.4), or that has a router or another strategy installed; while the discussion is installed, binding such a channel to the room and installing a router or another strategy in it are refused. One rule decides who speaks, never two: the supervisor a router always solicits (Section 19.4 step 4) has no place in a discussion. The room's agent_response_policy plays no part while the discussion is installed; uninstalling it (Section 19.7, one strategy per room) gives the room back to its policy at once and forgets the discussion (its queue, who only listens, what agents asked people, the turns given, whether it was over), so a discussion installed later starts fresh.
2. People. A person is a participant of the room that is neither an agent nor a bot, as Section 6.4 names the people of a turn; a person's message is an event such a participant sent through a transport channel, or one a transport channel brought in with no participant record behind it (a room whose people the host never recorded still has people). A delivery (Section 22), which a transport brings in from the framework's own sender, is no person's message: it is an event from another sender (rule 8).
3. The strategy takes the turns. No event solicits an agent at broadcast (Section 10.2, solicitation): the strategy queues the agents an event asks for and gives each its turn later (rule 7). The address keeps its meaning (Section 19.3): only the channels it names are asked to act, it is stored with the event, and what changes is when they act. Transports receive every event as before (Section 19.3 rule 1). An agent is not delivered an event at broadcast, as no channel left unsolicited is (Section 19.3.2); a turn reads the room when it starts (rule 9), filtered by what visibility lets the agent see (Section 7.5 rule 8). Its memory provider (Section 20) is still handed every message the agent may see (visibility and access, rule 4), the agent's own excepted, as the message is committed, as a room with no discussion hands it: a provider that learns as messages arrive (an index, a summary) learns the whole conversation, not only the messages that asked for the agent. The turn that answers a message does not hand it again.
4. Names. An agent's answer and a person's message address agents by naming them: @ followed by an agent's channel id; @all names every agent. A name starts the text or follows a character that is neither a letter, a digit, _ nor @; it is the longest run of the characters an identifier holds (letters of any script, digits, _, -, .) that follows, a final . excepted, and it matches a channel id ignoring case. Names are read in the text the event commits with, once its BEFORE_BROADCAST hooks have run (Section 10.1): never in a tool's arguments or result, an agent's reasoning, a fenced code block or a quoted line. They set the event's addressed_to, the speaker left out, in the order named, alongside an address a transport set; a name that is no agent of the room addresses nobody. A name queues an agent only when the event's visibility includes it and its access lets it read the event (Section 7.5): a name is no way around visibility; the visibility is the event's own, else its source binding's (Section 7.5). A person's name addresses no intelligence channel: it records that the agent asked that person (rule 10). The names people are addressed by are those of the room's people as the transcript labels them (Section 6.4: a participant's name, a transport's stamp, else the channel of a sender with no name), only those people lists when it is set, kept to an identifier's characters and at most 32 of them: a longer name addresses nobody, never cut short, since a cut name could be another person's whole one. What is recorded is the person as the transcript labels them, and only when one person answers to the name: a name two people answer to (Alice Martin, AliceMartin) records no one, and a look-alike's ranked label (Alice (2), Section 6.4) is addressed by no name. Where a person's name and an agent's channel id read alike, the name is the agent's. This is the mechanism Section 19.3.1 leaves open: an agent MAY address another agent.
5. A turn queues each agent once. Each event a turn commits carries its own names when it is committed (Section 19.3 rule 4). The queue takes them when the turn has ended, from the turn's delivered events only: an event a hook blocked, the depth limit stopped or the source's access withheld queues nobody, and an agent named in three segments of one turn is queued once.
6. A turn's end. A turn has ended once every event it produced is committed. A turn that fails, or that a round cap, a deadline, a budget or a Cancel cuts short, has ended too: the events it committed before the cut stay in the room and queue whom they name, the agent is not queued again for that turn, and a failure is reported as any turn's is (Section 9.3).
7. One turn at a time. The strategy gives the next turn once the current one has ended. It goes to the first agent of the queue that can take it: one that does not only listen, or whose turn rule 12 lets it take while it listens, and whose turn stays within the depth limit (rule 11). The agent whose turn just ended does not take the next one while another agent of the queue can, a person's message that put it first included: an agent that asks a person at every turn would otherwise keep the turns from the agents it waits on. Every turn an agent of the room takes goes through the queue. An instruction addressed to agents (Section 10.1.1) queues each of them at the front for a turn of its own, which takes the instruction as its input (rule 16). A regenerated answer queues at the front, for a turn of its own, each agent that answered the event (each agent the event asked for, when no answer to it is left), once per agent and event. Such a turn is given even while the discussion waits for a person (rule 10), to an agent that only listens, and to the agent whose turn just ended. Requests that put agents at the front are served in the order they came.
8. The queue. An agent an agent names is queued at the back, or keeps its place when already queued: the request merges into its pending turn, one turn for every event that asked for it. A person's message puts the agents it addresses at the front of the queue, in the order addressed. A person's message with no address (addressed_to null, and no name read in it) answers the agents that asked that person (asked), and puts them at the front in the order they asked; when none asked, it puts everyone there, in their order; each only when the message's visibility includes it and its access lets it read the message (rule 4). The message is routed so before rule 10 clears what was asked. With a dispatch policy, such a message is left to the policy instead, whether agents asked that person or not (rule 18). A message whose address is empty, or that names only people, asks no agent (Section 19.3 rule 3). An event from any other sender (a bot, a webhook, an event a hook injects, a delivery, Section 22) queues the agents its address names, at the back; with no address it asks no agent. A merged turn answers the latest person's message among the events that asked for it, else the latest of them. The order of everyone is the host's, so the agent that should open (the one that looks at the facts before the others act) comes first; the discussion does not rotate it. With addressed_only, a person's message asks only the agents it addresses, as ADDRESSED_ONLY asks only the channels an event names (Section 19.3.1): one that names nobody asks no agent, the agents that asked a person included, which wait for a message that names them. People who talk among themselves in the room are then never cut into by an agent; a host that wants agents to judge for themselves whether an unaddressed message is for them leaves addressed_only off and gives them a speak policy (Section 6.4), or gives the discussion a dispatch policy, which judges once for the room (rule 18).
9. A turn reads the room as it is. A turn a message asked for is a rerun of the event it answers, planned for that one agent under the room lock and run in the room's delivery lane, as a regeneration re-broadcasts an event already committed (Section 13.6): its context is built when the turn starts, so what was said while the agent waited is in it. The event it answers is passed as the turn's current_event and keeps its place in the history, before what came after it (Section 20.1), and a speak policy reads the conversation up to the turn's start in SpeakTurn.recent (Section 6.4). The turn's notes say which event the turn answers (its author's label and a quote of it, Section 6.4) and every speaker that asked for it. An external agent, whose session keeps its own history, is sent what it has not read yet and the event it answers as its request (Section 6.4). A turn is never given by publishing a message or an instruction in the room for the agent to answer, which a model reads as a third party's words and may rightly refuse to obey.
10. Asking a person, waiting for one. When an agent names a person, the discussion owes that person an answer (asked records the agent and the person as the transcript labels them, rule 4), and a person's message, once routed (rule 8), clears what was asked of them: of that label alone, so a sender who takes another's name (ranked apart, Section 6.4) answers for nobody else. With a dispatch policy, the policy judges whether the message answers them (rule 18); what was asked is cleared all the same. With addressed_only, only a message that names the agent clears what it asked. When no agent of the queue can take a turn, the discussion waits for a person's message, its waiting reading true, if the room has a person to wait for (one of its participants is, or a person's message reached it) and it owes a person an answer, an agent of the queue only listens, or the depth limit stopped the next turn. Until a person writes, whatever the message asks, or the cause of the wait is gone (the agent it waited on talks again), it gives no turn but those of rule 7 (an instruction, a regenerated answer). With no person in the room, or nothing to wait for, a discussion that can give no turn is idle, not over: the next event that asks for a turn opens it again.
11. Depth. A turn the strategy gives answers an event (rules 8 and 9) and carries its chain_depth + 1 (Section 8.3). A person's message is at depth 0, so each one opens a chain. max_depth takes the place of max_chain_depth for the turns the discussion gives and the events they produce, and cannot be disabled (Section 17.5). It bounds how far a chain of agent turns reaches from a person's message (four hops with the default of 5), not how many turns follow it, since several agents a person's message asks for each answer at depth 1. A turn the limit stops is not given and is recorded once as Section 8.3 says; its agent stays queued, and its turn answers the next event that asks for it within the limit: a person's message, which opens a new chain, or a request from a shallower one (rule 8). Each turn has a reentry budget of its own, 10 × max_depth passes (Section 8.3), as a regeneration does. A turn of its own (rule 7) past the limit does not wait: an instruction's is dropped and reported as dropped (rule 16), there being no stored event to record it against; a regenerated answer's is recorded once and dropped. An autonomous discussion, with no person to open a new chain, needs a max_depth that lets its work through.
12. Listening only. An implementation that provides Discussion MUST provide listen_only(room_id, channel_ids) and talk_again(room_id, channel_ids) for the host, which binds them to what a person does (a console's "@sre hold on", "@sre you can carry on"). The discussion keeps the state, and it means what the listening state of Section 6.4 means: an agent that only listens keeps its place in the queue and takes no turn an agent's message asks for; a person's message that names it gives it one turn, and it goes on listening (one that names nobody, routed to everyone or to the agents that asked that person, gives it none); talk_again ends the state. An agent whose speak policy also listens (Section 6.4) is silenced by either. Setting the state on the agent whose turn runs cuts that turn, as a Cancel steering directive does (Section 21.3), or, for an external agent, as its channel's cancellation does (Section 6.4): what the turn committed stays in the room; a turn given but not running yet, which no Cancel reaches, is abandoned before it runs. Talking again, its next turn reads the whole room (rule 9).
13. Staying silent. A turn a message asked for is submitted to the agent's speak policy, when it has one, as any event the agent would answer (Section 6.4); an instruction's turn is not, as Section 6.4 says. A silent decision runs no generation and ends the turn. Without a policy, or when the policy lets it speak, the model may stay silent itself: a text row of the turn whose only text is silent_token, ignoring case, surrounding whitespace, a final period and the brackets around the token if it has some, says nothing; a turn whose text rows all say nothing stayed silent. Its row is stored with status = BLOCKED and blocked_by = "discussion_silent", read for no name and delivered nowhere: the start of a streamed answer is held back until its text can no longer be the token, since a streamed segment once delivered is not taken back (Section 6.4). A turn that ends with no text produces no row and says nothing too. A silent turn counts as a turn (rules 7 and 15).
14. What a turn knows. Each turn's notes (Section 6.4) carry, as the runtime's own block and not a participant's words: the other agents of the room by their channel ids, with the role and description of those that carry an identity (Agents, Section 19.1, and external agents given one, Section 6.4); the people the agent may address, by the names rule 4 gives them; which event the turn answers and who asked for it, a person's label and their words quoted (Section 6.4); and how the room works (name an agent with @ to address it, a message that names nobody wakes nobody, the silence token when it has nothing to add, one agent speaks at a time).
15. The end. The discussion is over once done holds, checked before each turn is given, or once max_turns turns were given in the room, from the moment the last of them is given. It gives no further turn and drops its queue; an instruction addressed to an agent afterwards, and a regenerated answer asked for afterwards, are refused, their result blocked with the reason discussion_over; a turn running then ends as it would; the room's later events are stored and delivered with no agent asked. done is the host's: a mission's state, a person's word, a budget. A host that wants the agents to answer again uninstalls the discussion (rule 1).

16. Its state, across processes. Installing a discussion stores it with the room, as ConversationState is (Section 19.2): its configuration (the agents' channel ids, people, addressed_only, everyone, max_turns, max_depth, silent_token, whether it has a dispatch policy) and its speak queue. Every process that serves the room follows the stored discussion, whether or not its host installed it there: it asks no agent at broadcast (rule 3), and reads names and queues (rules 4, 5, 8, 10) by the stored configuration. Every change of the queue is read and written under the room lock (Section 13.5), so the processes that serve one room keep one queue, and the queue and the count of turns given outlive a restart. Uninstalling, in any process, ends the discussion in all of them.

    A process where the host installed the discussion may give its turns, one process at a time: the one that holds the room's lease, stored with the queue with the time it expires (15 s by default). It gives the turns, renews the lease while it holds it, and another process that installed the discussion takes it once it has expired: a process that stopped gives up its turns that way, and its running turn has ended (rule 6). A process that queues a turn without holding the lease makes it known through the stored queue, which the holder reads again at least once a second; a change that concerns the running turn (its agent set to listen only, rule 12) reaches the holder the same way. speaking is stored with the lease and names the holder's running turn. done is the host's and is checked by the process holding the lease, which asks its dispatch policy too (rule 18). A process that only follows the discussion never gives a turn: with no process that installed it alive, messages are queued and wait.

    A queued turn's instruction is kept in memory only, by the process that received it, as an instruction is stored nowhere (Section 10.1.1). Its entry names that process. When the turn comes and the holder is another process, the holder hands the lease over; an instruction whose process does not take the lease within one lease period is dropped. An instruction the discussion drops (it is over, uninstalled, or its process stopped) is reported through ON_SPEAK_QUEUE as dropped. 17. Following the queue. The host reads a room's SpeakQueue at any time, and each change of it fires ON_SPEAK_QUEUE (Section 9) with the room, the queue as it now is and what changed: an agent queued, a turn given or ended, a queued instruction dropped, an agent listening only or talking again, the discussion waiting for a person or no longer waiting, the discussion over. The host's calls take the organization a room belongs to (Section 17.2): a room of another organization holds no discussion for it. A console that shows who speaks, who is next, who only listens and whether the discussion waits for a person follows it, and draws the room's agents from the room's intelligence bindings and the agents' identity (Section 19.1). A person's message returns once committed and broadcast, with no agent's answer in its result (Section 10.1 step 18): the answers come in the turns that follow, as a regenerated answer the discussion queued does. 18. Who takes an unaddressed message. A discussion MAY carry a dispatch policy, which decides, for a person's message with no address and no name (rule 8), which agents take it and in which order, or that none does. Without one, rule 8 gives that message to the agents that asked that person, else to everyone: a room where every agent is asked every remark spends turns on agents with nothing to add (each says nothing, rule 13, but its turn runs), the agent that should answer may come third, and an agent that asked the person a question takes the person's next message even when it answers nothing of it (a request for another agent, a thanks). The policy is asked for that message only: a message with an address, or that names an agent or only people, is routed as rule 8 says, and a discussion with addressed_only, where a message that names nobody asks no agent, takes no dispatch policy. A name stays the person's word: no policy overrides it.

    DispatchPolicy (interface)
    └── decide(turn: DispatchTurn) → DispatchDecision
    
    DispatchTurn
    ├── room_id: string
    ├── event: RoomEvent                    # The person's message
    ├── recent: list<RoomEvent>             # The conversation before it, oldest first: the room's messages a candidate may read
    ├── speakers: map<string, string>       # Who said `event` and each of `recent`, by event id: a person by their transcript label, an agent as `@channel_id`
    ├── candidates: list<DispatchCandidate> # The agents it may pick: those that asked the author first, then `everyone`'s order
    └── asked: list<channel_id>             # The candidates that asked the message's author and wait for the answer (rule 10)
    
    DispatchCandidate
    ├── channel_id: string
    ├── name: string | null                 # An Agent's identity (Section 19.1), as a turn's notes give it (rule 14)
    ├── role: string | null
    └── description: string | null
    
    DispatchDecision
    ├── agents: list<channel_id>            # Who takes the message, in the order they answer; empty: no agent
    ├── reason: string = ""                 # Why, in a few words, for logs and hooks
    └── judgments: map<string, float> = {}  # What the policy weighed, by name
    

    The candidates are the agents that asked that person, in the order they asked, then everyone, in its order, less the agents the message does not reach (its visibility and their access, rule 4) and those that only listen, which an unaddressed message gives no turn (rule 12). With no candidate, the policy is not asked and no agent is. The agents a decision names are put at the front of the queue in its order, as rule 8 puts everyone; one that is not a candidate, or named twice, is left out, so a policy is no way around visibility or the listening state. An empty decision asks no agent: the message is stored and delivered, and the discussion goes idle (rule 10). The recent events a policy reads are those a candidate may read: a policy may hand them to a classifier outside, and an event no candidate may read MUST NOT reach it that way.

    The decision is the room's, taken once per message, off the room lock, by the process holding the lease (rule 16): routing the message leaves it waiting for a decision in the stored queue, with the candidates it reaches, and the holder decides it before it gives another turn, so the agents a person's message asks for still come before the agents queued after it. Messages waiting are decided in the order they were committed; once the discussion is over, they are dropped with its queue (rule 15), and a process that stops while it decides leaves the message waiting for the next holder. At most 16 messages wait: past them, the oldest asks what it asks with no policy (below), undecided, so a room with no holder alive keeps a bounded queue. The discussion bounds the wait for a decision (dispatch_timeout, 5 s by default). A policy that fails, does not decide in time or returns no readable decision does not silence the room: the message asks what it asks with no policy, the candidates that asked its author, else every candidate, and the decision reported names them with the reason fallback; so does a process holding the lease whose host gave it no policy, and any failure in deciding.

    Every decision applied fires ON_DISPATCH_DECISION with the room, the message, the candidates, the decision and how long the policy took (the bound, when it did not decide in time), so who took what, why and at what cost can be followed and measured without wrapping the policy. A decision on a message another process decided first (it took the lease over meanwhile) is neither applied nor reported: each message is reported once. The decision is not stored with the message: its addressed_to stays null, and a regenerated answer to it with no answer left asks everyone (rule 7), not a new decision. An agent picked still answers through its speak policy, when it has one (rule 13).

    RoomKit provides a policy built on a classifier (Section 6.8). It asks one yes/no question per candidate, all in one call: whether that agent should take the message, given what each agent's identity says it does, over the message, its speaker and the recent conversation named by speaker; an agent that waits for the speaker's answer is marked as such, so that it is picked when the message answers it and not otherwise. The candidates whose probability reaches a threshold (0.5 by default) take it, likeliest first, at most a number of them (2 by default); with none reaching it, no agent takes it (thanks, small talk). Each candidate's probability is reported with the decision, under its channel id. An agent with no identity is judged by its channel id alone.

17. Talking and working. A turn talks; long work runs as a task (Section 23), off the floor. An agent given the background tool (Section 23.4) starts one and says once, in a sentence, that the work is under way, and its turn ends there; the host starts one for an agent with start_task, and a person through what the host gives them. Tasks run in parallel with the discussion's turns and with one another, and the floor stays one turn at a time: a question asked while a task runs is answered in its turn, not after the task. A task's result comes back as its hand-back, an instruction addressed to the agent the task is for, which the discussion queues at the front for a turn of its own (rule 7), results served in the order they come; a task cancelled or failed says so the same way (Section 23.3). A posted result (hand_back = "post") takes the same place, and its turn posts it as the agent's message without running the agent's model: it never cuts a turn under way, and the names it carries ask no one, its worker having written it outside the room. Each turn's notes list the room's tasks (Section 23.4), so an agent asked how a task goes answers from them, and cancel_task or the host's cancel_task(task_id) stops one. A worker runs in its own child room, outside the discussion, so an agent of the discussion may work through any intelligence channel, an external agent among them. A task outlives an uninstalled or ended discussion: a relayed hand-back is then an instruction like any other, refused once the discussion is over (rule 15), its result kept on ON_TASK_COMPLETED and the StatusBus; a posted result, which asks no turn, is posted once no turn runs, one queued when the discussion ends or is uninstalled included.

The rules of Section 19.7 for a strategy in several rooms hold: the speak queue and who only listens are the room's, and the same agents may hold a discussion in another room with a speak queue of its own.

Left out of this version. Voice: a turn would end when its playback ends, which no event reports yet; a person speaking names no one with @; an agent's written names would be spoken. Realtime channels, whose turns run outside the room's queue, and agents that think while they listen, which need every event without a turn. The room's tools for an external agent: its tools are its own (its workspace, what its agent accepts), and this version wires none of the room's into it. One turn at a time: work that does not need the floor runs as tasks (rule 19), but turns themselves never run in parallel, each reading a room the others' answers would have changed. The text an agent writes between tool rounds still reaches the room as messages; only the names they carry wait for the turn's end. An approval a person gives for one production change is a tool's concern (Section 21), and a mission's shared state is the host's (done). An agent that asked a person may still be named by another agent before the person answers. A dispatch policy decides for a person's message only: an agent's message that names nobody still asks no agent. Threads (parent_event_id): an agent's answer already stays in the thread of the event it answers, but this version keeps one queue for the room and gives every turn the whole room; a later version makes each thread a conversation of its own (its own queue and chains, a turn reading its thread with the room before it as background, an agent taking one turn at a time across the room).

Conformance. Level 2, with the rest of Section 19.

19.8 StatusBus

The StatusBus enables inter-agent coordination through status messages:

StatusBus (interface)
├── post(agent_id, action, status, detail = "", metadata = null) → StatusEntry
│       # Publishes without waiting for the subscribers
├── post_async(agent_id, action, status, detail = "", metadata = null) → StatusEntry
│       # Waits for the subscribers
├── subscribe(callback) → void              # callback(StatusEntry) on every post
├── unsubscribe(callback) → void
├── recent(n = 10, agent_id = null, status = null) → list<StatusEntry>
│       # Most recent entries, optionally filtered
├── recent_text(n = 10) → string            # Recent entries as text, for an agent's context
├── has_completed(agent_id = null) → bool
│       # Whether a COMPLETED entry is among the last 50 completed ones
└── close() → void

StatusEntry
├── ts: string                              # ISO 8601 time of the post
├── agent_id: string
├── action: string                          # What the agent is doing or did
├── status: StatusLevel                     # OK, FAILED, PENDING, INFO, COMPLETED
├── detail: string = ""
└── metadata: map<string, any>              # room_id here names the entry's room

StatusLevel values are written ok, failed, pending, info and completed. One bus serves the framework; an entry names its room, if any, in metadata.room_id. The framework's own posts bound detail, a summary for whoever follows the bus: cut at a word, the cut ending in …, never in the middle of a word or a number. A completed entry whose detail was cut keeps the whole outcome in metadata.result, bounded at 4,000 characters and its cut marked the same way: an agent reads an outcome back from it (Section 23.4), never from the summary, which once turned a 14.9 °C forecast into "1". Status posts fire ON_STATUS_POSTED hooks in that room; an entry that names no room reaches no room's hooks. Agents MAY use the StatusBus to signal completion, progress, or request attention without sending room events. The framework posts each delegated task on the bus (Section 23.3), and an agent reads its room's tasks through the task_status tool (Section 23.4); an agent is never pointed at a tool it was not given.

---

20. Memory System

The Memory System provides pluggable AI context construction. Instead of always using a fixed sliding window of recent events, implementations MAY choose different strategies for building conversation history.

20.1 MemoryProvider Interface

MemoryProvider (interface)
├── retrieve(room_id, current_event, context, channel_id) → MemoryResult
│       # Build conversation context for AI generation
│       # Called by AIChannel before each generation
│
└── close() → void
        # Release resources

A provider is handed history, it does not fetch it: the context it receives MUST already be the requesting channel's view of the room (§7.5 rule 8). Making each provider apply the visibility filter itself would put the rule in five places and leave every third-party provider outside it; making the caller apply it once puts it in one, and a provider that summarizes what it is given can then never summarize something the channel was not allowed to read.

The current_event is the event the turn answers, and what the turn reads MUST keep it whatever the provider's window or budget: when events follow it in the context (a discussion's deferred turn, Section 19.7.5), it reads at its place in the history, before them, and is never folded into a summary. A provider returns it there, or leaves it out as it does a turn's input and the channel places it there; either way a window cannot drop it, which would leave the model answering an event it cannot read.

20.2 MemoryResult

MemoryResult
├── messages: list<AIMessage>               # Pre-built AI messages (summaries, instructions)
├── events: list<RoomEvent>                 # Raw events for AIChannel to convert
└── notes: list<string>                     # What was retrieved for this turn alone

When messages is non-empty, the AIChannel uses them directly as conversation history. When events is provided, the AIChannel converts them to AI messages using its standard conversion logic. Both MAY be combined — messages are prepended before converted events.

notes carries what a provider retrieved for the current turn only, knowledge passages for one: it changes from one turn to the next, so it rides the turn's notes (Section 6.4), after the input, and never the history, which a provider that caches a prefix would otherwise bill again at every turn. A provider that wraps another and rebuilds its result MUST carry the inner provider's notes, and one that keeps the turn within a token budget MUST count them: nothing trims the turn's notes, so the history is what makes room for them.

20.3 Built-in Implementations

Implementation	Strategy	Use Case
SlidingWindowMemory	Last N events	Default, simple conversations
CompactingMemory	Merges older events into compact summaries	Long conversations, reduce tokens
SummarizingMemory	AI-powered summarization of history	Complex multi-topic conversations
RetrievalMemory	Knowledge sources queried with the turn's text	Knowledge-backed answers (RAG)
BudgetAwareMemory	Token-based truncation with priorities	Strict token budget enforcement

SlidingWindowMemory is the default when no MemoryProvider is configured. It returns the most recent max_context_events from the room timeline.

SummarizingMemory calls an AI provider to summarize older history, keeping recent events intact. The summary is cached and refreshed periodically or when the event count exceeds a threshold.

RetrievalMemory wraps another provider and queries its knowledge sources (vector stores, search engines, document indexes) with the text of the current turn; the passages it retrieves are the turn's notes, each set apart as data, while the inner provider supplies the history. It indexes each ingested event in its sources.

BudgetAwareMemory counts tokens and evicts oldest events when the total exceeds evict_threshold_tokens. It prioritizes keeping system messages, tool results, and recent events.

The turn's footprint. What the history may take is what the window leaves once the rest of the turn is in it: the system prompt, the tools declared, the notes the channel itself adds to the turn's input (the room's plan, the digest of the tools already used, the speaker attribution) and the reply budget. A host cannot always name that rest, since the prompt is composed per turn (skills, Tool Search, the agent's identity, a binding's tools). So an AI channel MUST measure it, as it will send it, before it reads its memory, and make the measure readable to the memory for that turn (current_turn_footprint()): the input it measured, and the reply budget the turn requests, zero when the channel sets none and the provider applies its own.

A budget-aware memory reserves the larger of the reserve it was given and the measured input: the given reserve is a floor on that same input, not an addition to it, so a host that measures its own system prompt is not counted twice. For the reply it reserves the larger of its safety margin and the reply budget: the margin is the reply's headroom when the reply budget is unknown or smaller, and the reply budget when it is larger. What a hook adds to the context after the memory read is not measured.

---

21. Tool Access Control

21.1 ToolPolicy

A ToolPolicy controls which tools an AI agent can invoke:

ToolPolicy
├── allow: list<string>                     # Glob patterns for allowed tools
├── deny: list<string>                      # Glob patterns for denied tools
└── role_overrides: map<string, RoleOverride>  # Per-role policy adjustments

Resolution rules:

1. If deny matches the tool name, the tool is DENIED (deny always wins).
2. If allow is non-empty and the tool name does NOT match any allow pattern, the tool is DENIED (whitelist mode).
3. Otherwise, the tool is ALLOWED.

Glob patterns support * (any characters) and ? (single character). Example: allow: ["search_*", "get_*"] permits all search and get tools.

Tools the channel provides. A channel MUST apply its policy to every tool it declares to the model or executes, the tools it injects itself included, with one exception: the tools that only read or unlock and never act — activate_skill and read_skill_reference (Section 24), read_stored_result (Section 21.5), find_tools and list_tools (Tool Search). Without them a skill could not be activated nor a deferred tool found, so they stay declared and callable whatever the policy says. The exemption covers the tool the channel serves itself under one of these exact names, and nothing else: it MUST NOT extend to a name that merely shares a prefix, nor to a tool of the host, of an MCP server, of orchestration or of a hook that carries one of these names, which the policy governs like any other. A channel that serves none of them (a conference) exempts nothing, and neither does a door through which the channel serves none of its own tools (a reasoning backend's calls, a call recovered from speech): the exemption is read door by door, the same for the policy and for skill gating. Every other tool the channel injects — sandbox commands (sandbox_*), run_skill_script, plan_tasks — is subject to the policy like a host tool: a host that wants them allows them. The same names, and only them, are exempt from skill gating (Section 24.2). A transport that only carries a call to another tool (the call_tool of Tool Search on a provider whose declarations are fixed for the session) is exempt the same way: it stays declared and is among the names a handler reads (Section 21.4), and the policy applies to the tool it names, at the gate, once the call is unwrapped. A provider's native tool, which has no name ({"google_search": {}}), stays declared: a policy names the tools it governs, and Tool Search neither hides nor lists a tool it cannot name. A call under an alias a handler serves as another tool (an MCP tool called mcp__<server>__<tool>, which the MCP handler runs as <tool>) is judged under both names, by the policy and by skill gating: refused when either is, so no tool runs under an alias that its own name is refused. The declaration filter and the execution guard MUST apply one rule, so a tool the model is offered is a tool it may call.

One refusal per cause. A call is refused in the same order and with the same text for the same cause on every door: an AI channel's turn, a reasoning backend's (Section 12.4.1), a realtime session, a conference. The order is: a call cut short, a tool withdrawn for the turn, the declared check, the policy, skill gating, the arguments, then BEFORE_TOOL_USE, so a tool refused for access never has its arguments read. Arguments that do not fit the declared schema read Invalid arguments for '<name>': <why>; arguments a BEFORE_TOOL_USE hook left that do not fit it, whether it returned them or edited them in place, read Invalid rewritten arguments for '<name>': <why>. A tool the policy denies reads Tool '<name>' is not permitted by the agent's tool policy.; a tool a skill keeps closed reads Tool '<name>' is gated by a skill. Activate the skill first using activate_skill. (a reasoning backend, which cannot activate one, reads that the conversation has not activated it); a name the turn or session does not carry reads Tool '<name>' is not declared., and under Tool Search, on a text turn as on a realtime session, its call_tool transport and its list_tools included, that no tool by that name exists, with the hint to search. A tool the policy denies and a skill gates reads the policy's text: activating the skill would not let it through. A tool of the turn's catalogue that a round did not declare because the policy denies it or a skill keeps it closed is refused with that cause's text, not as undeclared: the model learns to activate a skill rather than to give up. A reasoning backend's model is offered only the session tools it may call; a call it makes to one it was not offered reads the session's cause, which the channel hands the backend with the catalogue (unavailable). Both are read when the channel hands the delegation to the backend, with the participant's role and the room's agent as they stand then, as the gate reads them at each call, and inside the delegation's bound: a read that fails is answered by the spoken fallback. A role changed during the session holds for the catalogue of the delegations handed over after it; one a backend queues behind another keeps what was read when it was handed over, its calls judged at the gate as the role stands.

One tool per name. A name is served by one tool in a room, and declared with that tool's definition: a tool declared under a name with one schema and served by another would have the model call the one's schema on the other's server. A name the channel serves itself is declared with the channel's definition only. The channel MUST refuse, when it is given, a tool under a name it already has for the same room, naming the tool: a host tool at construction under a name the channel serves or under a name another host tool already carries (two tool servers exposing the same name), and a tool orchestration sets up over the host's, over the channel's, or over another setup's for the same room (a setup for one room does not collide with another room's; the same strategy installed again for a room replaces its own). A tool that arrives with the turn (a binding's or a config provider's tools, a generation hook's) comes too late to be refused: under a name the channel or a setup already serves it MUST NOT be declared, and a warning names it, unless it is that setup's own declaration (a realtime pipeline gives each agent's session its own handoff and tools this way, Section 19.5); a name it gives twice is declared once, with the later definition. No name is declared twice in one round: a provider rejects a duplicate name. find_tools and list_tools MUST NOT name a tool the policy denies or a skill gates: a name the model can never call is a false promise and discloses what the policy hides. list_tools is the inventory: it lists every other tool the turn or the session can call, declared or hidden, its discovery tools aside, the re-read of a stored result included once a round declares it, on every path. find_tools names what Tool Search hides only, and its related names (a match's family) exclude what it never returns.

Tools orchestration injects stay declared. When Tool Search hides a large catalogue behind its discovery tools, a tool that orchestration injects on an agent or a channel (the handoff tool, a delegation tool, a delegation's result tool, a strategy's tools; Sections 19 and 23) MUST stay declared as a pinned tool does, and MUST NOT count toward the catalogue whose size decides whether Tool Search hides it. The tool policy and skill gating still apply to it; Tool Search does not: find_tools does not name it, being declared already, and the declaration reports it as always (Section 6.4, declared_tools). The agent is told to call these tools, often that it must: a tool it would first have to find is a tool it does not call.

RoleOverride:

RoleOverride
├── allow: list<string> | null              # Allow patterns (see mode)
├── deny: list<string> | null               # Additional deny patterns
└── mode: "restrict" | "replace"            # Narrow the base or replace it entirely

When mode is "restrict", the override narrows the base policy: its deny patterns are added to the base's, and its allow patterns constrain alongside the base's, so a tool must match both allow lists (each when non-empty). A restricting override can take access away, never grant it. When "replace", the override completely replaces the base.

21.2 MCP Tool Provider

Implementations SHOULD support loading tools from MCP (Model Context Protocol) servers via an MCPToolProvider:

MCPToolProvider
├── server: one of
│   ├── url: string                         # a remote server (streamable HTTP or SSE)
│   └── command: string, args: list<string> # a local server started as a subprocess (stdio)
├── discover: bool = true                   # List the server's tools when connecting
├── tools() → list<ToolDefinition>          # Discover available tools
├── call(name, arguments) → string          # Execute a tool
├── server_extensions → map<string, object> # Extensions the server declared
├── instructions → string | null           # What the server published for the model
├── server_info → object                   # What the server said of itself
├── read_resource(uri) → contents           # Read one resource
└── request(method, params) → object        # A request the provider does not model

A server named by command is started when the provider connects and MUST be stopped when it disconnects. Its arguments MUST be passed as a list, never through a shell. A connection that fails part-way MUST release what it had opened, the server process included, before the error is reported.

A provider lists the server's tools once, when it connects, following the listing's cursor across every page, and keeps the output schema each tool was listed with. A listing that pages on without bringing anything new (an empty page that still carries a cursor, a cursor handed out twice) or past a bound of pages ends there, with a warning naming the server: a server that pages forever MUST NOT hold the connection open. With discover false it connects without listing: tools() is empty and no call is validated, for a host that only reads the server's resources. A call MUST NOT list the server's tools again. A tool listed with an output schema has a successful result's structured content validated against that schema, and a result that breaks it (or carries none) fails the call: the tool ran, so the call is failed, not refused (Section 9.3). A tool the listing did not name is called as it is, without validation. Listing again on each call of an unnamed tool costs the whole catalogue every time, and a gateway that serves tools per caller never names them to the connection that calls them.

MCP tools are subject to the same ToolPolicy as local tools.

Server instructions. A server MAY publish, when it connects, how a model should use its tools (the instructions of its handshake result). A provider keeps them (instructions, null when the server published none) with what the server said of itself (server_info: its name, its version), from the handshake the connection made: a client that negotiates the newer protocol's discovery reads them from that result instead. A channel that composes a model's system prompt (an AI channel, the agent a reasoning backend drives (Section 12.4.1), a realtime voice channel whose provider takes tools) SHOULD take, from its host, the instructions of the servers whose tools it serves, each under the host's label for the server: an identifier, never the name the server gives itself, and a label that is not one fails the channel's construction. The channel MUST add them to every system prompt it composes, after its own prompt (or the one a binding or a turn's configuration sets for the room) and before the skills' preambles (Section 24). The same on every turn, they stay in the prefix a provider caches (Section 6.4); they are not tools, so Tool Search does not hide them; and no eviction or compaction reaches them. They are text from outside (Section 6.4): each server's sits in a <server_instructions> block under a line naming its label, the whole under a heading saying that they are their servers' content, which does not outrank the system's or the user's instructions and grants no permission. A server's text past 8192 characters is cut, with a warning naming the server. A server the host does not name, or names with no text, adds nothing: that is how a host keeps a server's instructions out of the prompt. A realtime pipeline serves the channel's, never an agent's (Section 19.5). A conference's realtime bot sends the prompt its host wrote as it is (Section 12.10): the channel adds nothing to it, the servers' instructions included.

A provider also carries what the skills a server serves need (Section 24.7) and no more: the extensions the server declared when it connected (server_extensions, empty for a server that declared none), read_resource, and request, which sends a request the provider does not model and returns its result as the server sent it. The skills rely on nothing else the connection holds: a skill's identity, the entry it was loaded from and the content verified are the registry's, so a transport without a session serves them alike.

call returns a string whatever the server sent. The tool handler a provider hands to an AI channel SHOULD instead hand an image the server returns to the model as an image, when it is in a format vision vendors accept (PNG, JPEG, GIF, WebP) and its payload decodes. Binary content the model cannot take (an image in another format or with a corrupt payload, audio, a blob resource) SHOULD reach it as a short note naming what was left out, never as its encoding: base64 read as text is noise the model pays for, and a bad image fails the whole request at the vendor.

21.3 AI Steering Directives

Steering directives allow dynamic mid-conversation control of AI behavior. They are injected into the AI generation context:

Directive	Effect
Cancel	Abort the current generation immediately
UpdateSystemPrompt	Append additional instructions to the system prompt
InjectMessage	Add a synthetic user or assistant message to the conversation history, a copy of the turn's notes' header or of another runtime mark it holds replaced (Section 6.4)

A Cancel ends the turn at the first point the loop reaches, and a round's tools are one such point: a Cancel that arrives after the model's last event of a round, while its calls are announced but before they run, MUST stop them. None of them runs, and each announced call ends with status = failed, so no start row stays pending. The loop honours the same points for every provider.

Which loop a directive reaches. One AI channel object serves every room it is bound to, so its running tool loops can belong to several rooms' turns at once. A host names the loop a directive is for, or its room, never both: a Cancel addressed to a room MUST reach every loop of that room, any other directive addressed to a room MUST reach the room's most recent loop, and a directive addressed to a room MUST NOT reach a loop of another room. A directive addressed to neither reaches the channel's most recent loop, whatever its room. A loop is reachable once its turn has started; a directive that comes before reaches nothing, and the host is told how many loops it reached.

Steering directives are typically issued by orchestration logic (e.g., supervisor injecting context for a worker agent) or by hooks reacting to events.

21.4 Tool Call Context

What a handler returns (normative). A tool handler answers with text or with a list of content parts (text and images, Section 21.5), which it MAY give as mappings naming their type. A mapping counts as a part only when it has exactly a part's shape (a text part's type and text, an image part's type and image fields): one that carries a type key among other data is a value, and reading it as a part would drop the rest. Any other value it returns (a mapping, a list of values, a number, a null) MUST reach the model as its JSON serialization, the same on every channel, and a result a SYNC ON_TOOL_CALL hook supplies in its place (Section 9.3) is read the same way. A value outside the contract MUST NOT fail the turn, nor reach the model as a language's own printing of it. A handler that declines a call SHOULD say so as a refusal, which carries the failure marker (Section 9.3); a body it returns instead is read by its own convention (Section 15.8.1). A handler says a tool is not its to serve by raising the implementation's unserved signal (UnservedToolCallError), so that a composition of handlers passes the call on and the channel reads the call as served by nothing; the text envelope an earlier convention returned ({"error": "Unknown tool: …"}) is still read as that signal, from a handler's answer only, by one adapter. A call whose handlers all decline it was served by nothing (Section 9.3), on every channel and every entry of one: a conference's calls and the calls a speech-to-speech channel recovers from speech read it as the function-calling path does.

A tool handler is called with the tool's name and arguments and nothing else. One AI channel object serves every room and every speaker it is bound to, so whatever a handler captured when it was built describes whoever attached the channel, not the turn now running. An implementation MUST therefore expose the turn to the handler through a per-call context that follows the async call chain (a contextvar, in Python), readable at any depth without a signature change, and MUST answer with the empty value outside a tool call so a host keeps its own fallback there (a direct call). A realtime channel serving a tool call (§12) MUST install the same context around the handler, with the session's room and participant as the turn's room and actor, so one handler answers the same questions on both paths. The context carries at least:

Accessor	Answers
current_tool_room_id()	The id of the room the turn belongs to
current_tool_room()	The Room of the turn itself: the object the store loaded when the turn began, the same one the turn's RoomContext.room holds for its hooks, memory provider and config provider; on a realtime tool call, which runs no turn, the room as loaded for that call
current_tool_actor_id()	The participant id of the event that woke the channel this round; empty when the turn has no author (a system injection, a webhook, a scheduled run)
current_tool_allowed_names()	Every tool name the turn resolved, and every tool a round declared beyond it (the re-read of a stored result), nothing withdrawn, that its tool policy admits for its actor, so a call is validated against the live toolset rather than an attach-time snapshot: a tool the policy denies is left out on every door, since the gate refuses it before any handler, while a tool a skill keeps closed stays in; on a realtime door, every tool the session declares (its catalogue, what orchestration set up, the channel's own) that its policy admits, and none when the session declares no catalogue: it names no list, and its gate, policy included, still judges each call
current_tool_call()	The per-call record: the call's id, its channel, and the structured-result reverse channel
current_response_metadata()	The turn's one response-metadata record (§6.7); empty where no turn will merge it (a realtime tool call)

The context also carries the turn's chain depth: the depth of the response the turn produces (§8.3), on a realtime tool call the depth of the assistant's answer (§12.4). A result delivered later on the turn's behalf reads it there (Section 23.3); outside a turn it is 0. The implementation reads it; it need not expose an accessor for it.

Two rules bind the values that name the turn (the room and its id, the actor, the toolset); the response-metadata record and the per-call record are the turn's write channels, described in §6.7 and above. They name the turn; they do not authenticate it. The actor is a room Participant.id that reads back the same whether the sender is identified, pending or unknown, and the room's organization is a fact of the room, not a grant: a handler that reaches a person's or a tenant's data resolves the participant (§11) and applies the host's own rule before acting. The Room is the room as the store loaded it when the turn began, shared by reference with RoomContext.room and so with every hook, provider and channel of the turn. A patch written to the store during the turn is not in it: a handler whose decision depends on the turn's own writes re-reads the room. A handler MUST NOT mutate the object: a room changes through the store, and a write on the shared object would be read by the rest of the turn (the agent-response policy, the delivery plan) as if the room had changed. On a path that runs no turn, a realtime tool call, the Room is the one the store loaded for that call: shared with the pre-execution gate's context when a hook made the channel build one, read once for the call otherwise, and shared with nothing else, since no turn's RoomContext exists there; current_response_metadata() MUST answer the empty value on that path, because no MESSAGE event will carry the record. A tool loop the implementation starts under a turn MUST inherit the turn's context, so a handler called on any round of the loop reads the same room and actor as on the first.

21.5 Large Tool Results

An implementation that bounds large tool results (stores the full text and gives the model a bounded preview with a way to read the rest back) MUST apply the bound to every outcome the model reads: a handler's result, a result override from ON_TOOL_CALL, a refusal, and an error. A bound that covers the success path only leaves the others unbounded, and an error page or a rewritten result costs the same as the result it stands for, on every round of the tool loop that carries it.

• The preview MUST be bounded in characters, not only in lines: one long line would otherwise carry the whole result.
• A result made of content parts has its text bounded and its images kept in place.
• A model that cannot take images MUST be given the text of a content-part result, its images marked, the way it is given a message's.
• The bound applies to the model's copy, and it runs after ON_TOOL_CALL: the hook receives the handler's whole result, and what it returns (the result, or its override) is what is bounded and stored. Everything the model can read back has then passed through the hook, so a hook that redacts covers the full text; bounding first would let the model page the raw text back from the store. A refusal observer receives the raw message.
• An active skill's instructions are exempt (Section 24.4).
• The read-back MAY search a stored result: given a text on one line, it returns the lines of the result as stored that contain it, case aside, with the lines around them and their numbers, bounded as a page is, a line longer than its share of the page cut around the match. A search reads every line whole, so one that finds nothing says the text is absent from the result, which a page read never can; its answer says when it holds only part of the matches. The text is matched as it is, never as a pattern the implementation would run: a model-supplied pattern could stall the process.
• What the model can read back SHOULD stay readable while its room works. The store's capacity is counted per room, and a bound the whole store needs for memory MUST take from the room holding the most first, so a room holding few results keeps them while other rooms evict; a result read back counts as in use. A stored result's id MUST NOT be given to another result while a context can still name it, so a call id reused in a later turn, or an id whose result left the store, reads nothing it did not store. A summary rebuilt from persisted history (after a restart) MUST NOT offer an id the store no longer holds.

The event that records a tool call's outcome (TOOL_CALL_END) is persisted, broadcast and handed to the event pipeline's hooks. It SHOULD NOT carry a result's images without bound, whatever shape the result takes (content parts, or JSON such as an ACP agent's output) and in its structured copy as much as in the result: an implementation keeps them up to a size and states each one it leaves out. The bound is on the event only; the model's copy keeps every image.

21.6 Tool Call Timeout

A handler that never answers holds its turn for good: the model waits on the call, and nothing else in the room's turn can proceed. An implementation MUST therefore bound every wait on a tool handler's answer, on every path a call takes (the tool loop of an AI channel, every entry of a speech-to-speech channel, a conference's calls), through one rule:

• A channel has a default bound per call, and a bound per tool name that overrides it; either MAY be absent, meaning no bound. The default SHOULD be short where a person waits in silence (speech-to-speech, conference) and MAY be longer for text.
• A tool that keeps a bound of its own is not subject to the channel's default: one that waits on another agent or on a person by design (a delegation, an orchestration strategy's tool, a human-input tool), or a command run under its own timeout (a sandbox command). A bound set for its name still applies. A tool an orchestration strategy installs for an agent to serve (a pipeline agent's own tool) is the host's and is bounded like any other.
• When the bound expires, the handler MUST be cancelled, and the call fails as a call whose handler raised (Section 9.3): the model reads the failure and the error's class, the observers the detail. A timeout the handler raises itself is that handler's failure, not an expired bound.

The bound is per call, not per turn: a turn's own deadline still applies between rounds.

---

22. Delivery Strategies

Delivery strategies control how proactive content (background task results, delegated agent output, scheduled messages) is delivered into a conversation.

22.1 DeliveryStrategy Interface

DeliveryStrategy (interface)
└── deliver(context: DeliveryContext) → DeliveryOutcome | void

DeliveryContext:

DeliveryContext
├── kit: RoomKit                            # Framework instance
├── room_id: string                         # Target room
├── content: EventContent                   # Content to deliver
├── channel_id: string | null               # Transport/source channel
├── addressed_to: list<string> | null       # Intelligence solicitation (§19.3)
├── idempotency_key: string | null          # Publication/injection identity
├── session_id: string | null               # Exact realtime model session
├── instruction: bool                       # The application's direction (§10.1.1)
└── metadata: map<string, any>              # Delivery metadata

RoomKit.deliver() MUST return an observable outcome. Built-in strategies MUST distinguish queued (accepted by the delivery backend), sent (published into the room or accepted by the voice provider), blocked (explicit refusal), unavailable (destination cannot be reached), and failed (execution error). Each unsuccessful outcome MUST carry a reason. Custom strategies returning no outcome MAY remain usable, but their outcome MUST be reported as unknown, never inferred as successful.

The outcome SHOULD expose the queue item id, committed event id, duplicate indicator, unavailable target ids, injected session ids and structured error where applicable. sent MUST NOT imply that the agent completed its turn. An implementation MAY expose the existing inbound result and delivery handle to observe turn completion; replaying a committed event does not establish that its original turn completed.

For synthetic text inbound, addressed_to MUST pass unchanged into the existing publication pipeline and stored event, with all of §19.3's rules. null retains routing, [] stores and broadcasts without soliciting an intelligence channel, and unknown targets MUST NOT select a substitute. An unavailable explicit target MUST be observable even if publication itself succeeded. Addressing MUST NOT change visibility or bypass permissions.

channel_id selects a transport; naming an intelligence channel there MAY remain an alias for selecting the room's transport, but MUST NOT silently replace explicit addressed_to. Applications select intelligence channels with addressed_to, and realtime provider sessions with channel_id and session_id; the latter are not intelligence channel ids (§19.3).

Instruction delivery. instruction = true delivers the content as the application's direction to an agent rather than as something a participant said. Through the text pipeline it is an INSTRUCTION event (§10.1.1), with that section's rules: it needs a non-empty addressed_to and no idempotency_key, and is otherwise refused with blocked before anything is written. Injected into a realtime session it takes the system intent (§12.4). The strategy, the delivery hooks and the delivery backend apply to it unchanged. Content an application produces for an agent (a background task's result, a scheduled nudge) SHOULD be delivered this way, so it is never stored, shown or read as a participant's words.

An explicit realtime destination MUST NOT expand to all sessions. Without session_id, an explicitly selected realtime channel requires one active session; multiple sessions are ambiguous. Strategies waiting for idle MUST pin the selected session before waiting and verify it is still active before injecting. A pinned delivery stays an injection: a session or a model gone by the time it goes out refuses it, and it MUST NOT be published to the room instead. Missing or replaced sessions and injection errors MUST NOT report success. Unaddressed calls without explicit destinations MAY retain room-wide realtime injection. A non-null intelligence address uses the text pipeline, not a direct realtime injection; combining it with session_id is invalid.

Idempotency boundary. Text keys are scoped to the room and use the ConversationStore's existing locked publication and unique-key contract (§13.4). Replays, including concurrent ones, MUST NOT create a second trigger event while the key is retained. The same key identifies the same publication regardless of subsequent body or address changes. Retention follows the store: memory does not survive restart; durable stores retain committed keys until the corresponding event/key is removed. Distributed use requires a shared store and a lock manager appropriate to that store.

Realtime injection identity. A keyed proactive voice injection MUST reserve its attempt atomically in the ConversationStore before calling the provider. Its identity is the tuple (room_id, channel_id, session_id, idempotency_key); different sessions are independent destinations. A repeated identity MUST reuse its recorded outcome without reinjection, unless the previous attempt explicitly established that nothing was submitted and that retry is allowed. Reusing a voice identity for different effective text MUST be refused as an idempotency conflict. This does not change the text publication replay semantics above.

The store MUST persist the reservation, a unique attempt token and its outcome. Claiming a retry and completing an attempt MUST use atomic conditional writes; an old owner MUST NOT overwrite another attempt. The reservation MUST outlive queue acknowledgement. A process-local store provides process-local retention; durable stores MUST preserve reservations across restart until the room is deleted. A store without this capability MUST refuse keyed voice injection before sending. No expiry or abandoned-claim recovery may authorize an uncertain injection again.

A reservation without a final outcome is an unresolved attempt. A concurrent caller MAY wait for its owner through the existing lock manager; if completion cannot be established, its result MUST be unknown, with no automatic reinjection. Cancellation during submission, an exception without proof of non-submission, or failure to persist the final outcome MUST preserve this conservative boundary.

Realtime providers SHOULD report sent (their send operation completed), not_sent (no submission or pending submission exists), or unknown (acceptance cannot be established). Only an explicit retryable not_sent result permits automatic retry after invoking a provider. Returning no result MUST NOT be interpreted as proof of acceptance. Unsupported injection and missing connections MUST NOT return a successful result. Locally queued input whose remote submission is not known remains uncertain. An injection result does not prove that the peer heard audio or that the agent completed its turn.

The delivery outcome MUST expose each selected session's result for a multi-session request, retaining successes and duplicates alongside failures or uncertainty. A retry MUST NOT inject again into a session with a successful or unresolved reservation. When session_id is omitted, selection still happens at execution; applications targeting one original call MUST supply that session's id. A known outcome for an explicit session MAY be replayed after that session has ended.

Direct realtime injection has no atomic transaction with the remote service. Suppressing retries after uncertainty can leave an announcement undelivered; eliminating that tradeoff requires verified idempotency at the remote service. Unkeyed calls retain no deduplication guarantee. Queue delivery remains at-least-once, and neither a committed text event nor a queue acknowledgement proves exactly-once agent execution or resumes a turn interrupted after commit. A durable delivery-lane outbox and crash recovery of turns are separate capabilities.

22.2 Built-in Strategies

Strategy	Behavior
Immediate	Deliver synchronously. May interrupt active voice playback.
WaitForIdle(buffer = 1.0, playback_timeout = 15.0)	On a voice channel, wait until the room's TTS playback has finished; on a channel hosting a realtime model, wait until the model's sessions are idle (no answer in flight, nobody heard speaking, no tool call in flight, its audio forwarded); then wait buffer seconds and deliver. On any other channel, deliver at once, with no buffer. A wait longer than playback_timeout seconds delivers anyway.
Queued(buffer = 1.0, playback_timeout = 15.0, separator)	Batch multiple delivery items, then deliver together after buffer seconds of inactivity, their contents joined with separator (default a blank line). It waits for idle as WaitForIdle does, and a wait longer than playback_timeout seconds delivers the batch anyway.

WaitForIdle is RECOMMENDED for voice channels where interruptions are disruptive. On a channel hosting a realtime model it watches both the model's answer and the user's speech; on a voice channel, the TTS playback.

Queued MUST keep incompatible rooms, transports, intelligence addresses, sessions, intents (message or instruction) and metadata separate. Keyed requests MUST preserve their individual publication keys and outcome attribution rather than merging into an event with a different key. Every waiting caller MUST receive its own outcome, including requests arriving while a batch is being delivered.

Delivery backends MUST preserve addresses, keys, session targets and the instruction flag in their serialized items and across retries. Enqueue acceptance is not transmission. Workers MUST distinguish a refusal from a retryable unavailable/failed target; an absent recipient MUST NOT be acknowledged as successful transmission.

22.3 Delivery Hooks

Hook	Execution	When
BEFORE_DELIVER	SYNC	Before strategy executes — can block or modify content
AFTER_DELIVER	ASYNC	After delivery completes

AFTER_DELIVER MUST report the actual outcome, including explicit refusal, missing destination and duplicate publication. Its payload MUST include the effective content and targeting information; a rewrite in BEFORE_DELIVER must be reflected. Hook observation remains best-effort and MUST NOT change the delivery outcome.

Both hooks see an instruction delivery as an event of type INSTRUCTION, so a hook that gates proactive content gates it too.

---

23. Task Delegation

Task delegation enables agents to offload background work to specialized agents running in isolated child rooms. Results are collected and delivered back to the parent conversation.

23.1 TaskRunner Interface

TaskRunner (interface)
├── run(task: DelegatedTask) → void
│       # Execute the task asynchronously
│
├── cancel(task_id: string) → bool
│       # Cancel one task; it ends cancelled
│
└── close() → void
        # Cancel in-flight tasks and release resources

Implementations MUST provide an InMemoryTaskRunner for single-process deployments. A runner ends each task exactly once, through the completion it was handed: completed or failed once its work ran, cancelled when cancel or close cut it (even before it ran). That completion MUST run to its end even if the caller of cancel is cancelled meanwhile, and a task whose work already ran when cancel arrives ends as it stands.

23.2 DelegatedTask

DelegatedTask
├── id: string                              # Unique task identifier
├── parent_room_id: string                  # Room that delegated
├── child_room_id: string                   # Room the agent works in
├── agent_id: string                        # Agent to execute the task
├── task: string                            # Task description / instructions
├── status: TaskStatus = PENDING            # PENDING, IN_PROGRESS, COMPLETED, FAILED, CANCELLED
└── result: DelegatedTaskResult | null      # Set when the task ends

DelegatedTaskResult
├── task_id: string
├── parent_room_id: string
├── child_room_id: string
├── agent_id: string
├── status: TaskStatus = COMPLETED
├── output: string | null                   # The worker's answer; a worker cut short keeps its narration
├── error: string | null                    # Error message (on failure); "cancelled" for a task cancelled from outside
├── exception: error | null                 # The failure itself, its type kept; held in memory, never serialized
├── duration_ms: float = 0
└── metadata: map<string, any>              # loop_end_reason when a cut or a failed turn ended the worker; turns: the worker's turn record (§6.4)

23.3 Delegation Protocol

When delegate(room_id, agent_id, task, notify, post_status = true, brief = null, conversation = 0, hand_back = "relay") is called:

1. Create a child room linked to the parent room, in the parent room's organization (Section 17.2). A tool, a hook or a provider acting in the child room reads the organization it acts for from the room, as it does in the parent; a child room without one would have the task's work, and the parent's messages the description carries (step 4), run outside any tenant. A read scoped to the parent's organization finds the child room, and one scoped to another does not.
2. Attach the specified agent as an INTELLIGENCE channel in the child room.
3. Share the channels the caller names from the parent, each with its parent binding's permissions (§7.5 rule 6).
4. Inject the task description as a system event in the child room, visible to the child room's intelligence channels only. The task description is the delegating side's instruction to the worker, not a message for the people behind a transport: a shared transport MUST NOT receive it, nor the re-prompt a result tool sends after a turn that ended without its call. The description is the brief when one is given, else the task: the task is what was asked, as the delegating side wrote it, the label the hand-back (step 8), the StatusBus and the turn's notes name the task by; the brief is what the worker is told, which may say more (how to work, what to report) and never reaches the parent room. With conversation above 0, the description also carries the parent room's last conversation messages that the notified channel may read (Section 7.5), oldest first, each under the label the transcript gives its author and quoted as Section 6.4 quotes a text from outside, set apart as data: a worker asked to look into what the room is talking about reads what was said, which its child room, new and empty, does not hold.
5. The agent processes the task and generates a response. When no transport is shared into the child room, nobody is delivered the response, and the child room keeps the agent's trace (tool calls and text), written as any streamed turn's rows are (Section 12.2 step 13s) but committed without crossing BEFORE_BROADCAST or a delivery lane: a TOOL_CALL_END keeps the call's structured copy, and a delegation cancelled or failed while a call runs closes that call with status = failed, so no start row stays pending. When a transport is shared, the response is committed as a room's response is (§10.1 steps 9 to 14, §7.5 rule 2): each row, buffered or streamed, crosses BEFORE_BROADCAST and the agent's right to write and rides the child room's delivery lane, so the shared transport receives the agent's answers as its binding's access and the agent binding's visibility allow, and the rows the room stored are the trace.
6. Collect the agent's response as the task result: the answer the child room kept. With a shared transport, that is what the hooks decided — a rewrite holds for the result as for the delivery, and an answer the gate refused is no answer. A result the agent submits through a result tool is read from the child room's trace, wherever the call was served (the agent's tool loop or an MCP server), once ON_TOOL_CALL has judged it: it is the agent's own call that ended served. A refused, failed or blocked call is no result, and neither is a call of another channel shared into the child room. A turn that did not complete (it ended for any reason but completed: its round cap, deadline or budget cut it, a stop cancelled it, its answer was cut or never came; an ACP agent's prompt stopped on any reason but end_turn, or never returned) has no answer (Section 6.4): the task fails, its error naming how the turn ended, its output the agent's last narration and its metadata the loop_end_reason (an ACP agent's stop reason, or interrupted), which ON_TASK_COMPLETED carries too, whether the turn streamed or not, wrote text or none, and whether a transport is shared. The answer and how its turn ended are read off the same agent: its turn's record, else its last message, never another agent's that answered in the child room. A turn that failed after it began (the provider's error once a round ran, an ACP agent's prompt that raised) fails the task with that error, the task's error being the error's own text, and the task still carries what the child room's record holds of the turn: its loop_end_reason (error, an ACP agent's interrupted) in its metadata and on ON_TASK_COMPLETED, its narration as its output; the failure is logged as its error is. The end is the failing agent's own: with several agents in the child room, one agent's failure never carries another's end. Whatever ended the turn, the task's metadata carries the turn's record under turns, keyed by the agent, as a room turn's caller reads it (Section 6.4): how it ended and its ai_usage, the last turn's when a result tool re-prompted the agent. A failed task keeps the failure itself (exception), its type unchanged, so a caller that hands it on raises what the turn raised. A result the agent submitted through a result tool before the cut or the failure still counts; without one, the task fails without a re-prompt. What reads a task's work (an orchestration strategy, the notified agent) reads none from a failed task, its output whatever it keeps.
7. Fire ON_TASK_COMPLETED hook in the parent room.
8. If notify is set, hand the result back through deliver() (§22) with the framework's delivery strategy, so BEFORE_DELIVER, AFTER_DELIVER and the delivery backend apply to it as to any proactive delivery, and with instruction = true when notify names an intelligence channel or a channel that hosts a realtime model. notify names who is told: an intelligence channel receives it addressed to it, through the room's transport (§10.1.1); a channel that hosts a realtime model (a realtime voice or audio-video channel, a conference with a realtime model plugged in), injected with the system intent (§12.4) into the session whose tool call delegated, read from the tool call context (Section 21.4), or its one session for a delegation made outside a call; nothing of it is published to the room's other channels. Another transport has no model to direct and receives it as an ordinary delivery through it (channel_id = notify, instruction = false). The delivered content carries the result, bounded and delimited, presented as the worker's output rather than as an instruction; a task that did not complete says it failed or was cancelled, without its error (§9.3), whatever output or error text it left. It names the task as the delegating agent wrote it (task, quoted as Section 6.4 quotes a text from outside), and the worker by its id kept to an identifier's characters, so the notified agent says the result for what was asked, not for what the conversation moved on to meanwhile. Delivered to an intelligence channel, whose hand-back opens a turn of its own, it also says, after the result, that the turn gives that result only, another of the agent's replies taking care of whatever else was said, in the language the conversation is in rather than the result's: the turn reads the room's last messages, and one that answered them too repeated what another turn was saying, and a worker's result written in English was said in English to a conversation in French. A channel that hosts a realtime model gets no such line: its injection may open no turn of its own (a silent one, an intent its provider keeps as a standing instruction), where the line would tell the reply to the person's next question not to answer it. A transport's delivery carries none either. Only a completed task with nothing to say is not handed back. The delivery carries the task in its metadata, task_id, agent_id (the worker), task_status and task (what was asked, bounded), whichever channel it reaches, so a hook that sees it (BEFORE_BROADCAST, BEFORE_DELIVER) tells a task's result from any other instruction and names the task without reading its text. An instruction is not stored, so the result lives in the turn it opens and in the agent's answer, not in the history of later turns. The result is never written into the room's stored configuration (the system prompt of a binding), which would replace the notified agent's own and hand a worker's output the system role for every turn after. A hand-back that is not delivered (refused by a hook, a room without transport) is logged; ON_TASK_COMPLETED still carries the result. The delivered event carries the chain depth of the response whose turn delegated (the depth of that turn's trigger plus one, read from the tool call context, Section 21.4), so a cycle of delegation, result and delegation again ends at max_chain_depth (§8.3) like any chain. A delegation made outside a tool call delivers at 0. hand_back says how the result reaches the room. relay, the default, is the hand-back above: the notified agent is told, and says the result in a turn of its own, in the conversation's words and with what the room said meanwhile. post publishes the result to the parent room as it is, as a message of the notified agent, without a turn of its model: the result reaches the room at once and as the worker wrote it, where a relayed one waits the length of a turn and comes back rewritten. Which one fits is the host's call, by context: a result to put in the conversation's words is relayed; a long or exact one (an analysis, a list, a diff), or one a slow model would hold, is posted. post requires notify to name an intelligence channel attached to the parent room; a delegation that names another channel with post is refused when called. The message names the task (task, quoted as Section 6.4 quotes a text from outside) before the result, so the room tells what it answers; a task that did not complete posts that it failed or was cancelled, without its error (§9.3); a completed task with nothing to say posts nothing, and a task the agent cancelled itself posts nothing either. The message is committed as the notified agent's response is (§10.1 steps 9 to 14): it crosses BEFORE_BROADCAST and the agent's right to write, is stored, and reaches the room's channels as the agent binding's visibility allows, so later turns, of every agent, read it in the history, where a relayed result lives only in the answer its turn gave. It carries the task in its metadata as the relayed delivery does, and the chain depth that delivery would carry.

A task cancelled from outside (its caller's timeout, the framework's cancel_task(task_id), the task runner's cancel or close, the framework's own close(), even before it ran or while its delegation was still being set up) ends once, as any task does, whether it ran inline or in the background: cancelled, with no output. A task whose worker's turn had begun carries it under turns (step 6), its end cancelled, as a room turn's caller reads a read cancelled from outside (Section 6.4); one cut before its turn began carries none. The framework holds an inline task as it holds the background ones, so its close() cuts it and waits for its end, its completion included (step 7, and step 8 for a background task), before it releases the store; a task its delegation announced as the close began ends the same way, before it runs. Steps 7 and 8 still run, to their end though the task is being cancelled, the completion callback with them, and a notified agent is told the task was cancelled; then the cancellation goes on. The one exception is an agent that cancelled the task itself (its cancel_task call, Section 23.4): the tool's answer has told it, and step 8 hands it nothing, or it would say the cancellation twice. A step 8 whose notified agent's turn fails leaves the log line to that turn's failure, with its cause and at its level, and notes the result not delivered below it. A framework that is closing starts no turn: step 8 is skipped and logged, step 7 still runs. A step 8 already under way when the framework closes is cut with it, as a strategy's background run's hand-back is: the turn it started is cancelled, what it had produced kept as a cancelled response (Section 12.2 step 13s) and nothing more stored, and the task's completion callback and its waiters still run. A task whose work already ran when the cancellation arrives ends as it stands. A delegation's span ends with its task's status: ok completed, error failed, cancelled cancelled.

A delegation is followed on the framework's StatusBus (Section 19.8), so an agent or a host sees a task while it runs. Once its child room is ready, when ON_TASK_DELEGATED fires, it posts pending under the worker's agent_id, action task, its detail the task, bounded. Once it ends, when ON_TASK_COMPLETED fires, it posts completed, its detail the result, bounded, with the whole result in metadata.result when the detail was cut (Section 19.8), or failed for a task that failed or was cancelled, its detail saying which and never the error's text (Section 9.3). Both entries carry in their metadata room_id, the parent room, task_id and child_room_id; the terminal one adds task_status and duration_ms. A caller that follows its tasks on the bus itself (an orchestration strategy's worker run, Section 19.7.3) delegates with post_status = false, and the delegation posts nothing, so a task never shows twice.

Progress. A task MAY say how far it got while it runs ("12/30 s", "3 sources of 5"). The child room's metadata names its task (task_id, beside parent_room_id and task_agent_id), so a tool the worker calls in that room posts its task's progress with post_task_progress(kit, detail), read from the tool call context (Section 21.4): an info entry under the worker's agent_id, action task, its detail the progress, bounded, with the same metadata as the task's other entries. It returns whether it posted: a call outside a task's child room posts nothing. A progress entry does not end the task: a reader of the bus keeps the latest progress of a task apart from its status, and a task that ended stays ended whatever progress follows.

23.4 Delegation Tools

Implementations SHOULD provide helpers for AI-driven delegation:

• build_delegate_tool(agents) — Creates an AI tool definition that lets an agent delegate work to other agents. The agents parameter lists available delegation targets with descriptions.

• setup_delegation(agent, handler, tool) — Wires the delegation tool into an AI channel, connecting it to the TaskRunner. It takes no room: the tool is declared wherever the agent serves. tool (default null) replaces the default definition; setting it up twice on a channel is refused.

• setup_realtime_delegation(channel, handler, tool) — The same for a realtime voice channel, whose sessions declare the tool.

• task_status — A tool an agent can be given on its own, independently of the delegation helpers. It answers with the tasks posted on the StatusBus for the room of the call, read from the tool call context (Section 21.4): those running and those ended, the latest entry of each, with its worker, the task, its status, its latest progress and when it was posted (Section 23.3), its whole result (Section 19.8) or that it failed, and when it was posted; a task_id argument narrows the answer to one task. It reads the bus for that room only: one room's tasks never show in another. A tool reply that tells its agent to follow the work (a strategy's background dispatch, Section 19.7.3) names task_status only when the agent was given it.

• work_in_background — A tool an agent of a room is given (setup_background(agent, worker = null, hand_back = "relay")), to start long work as a task and free its turn: going through many records, files or traces, an analysis of several tool calls. Its argument is task, what to do and report. The task is the calling agent's own, and runs on that agent's worker: the agent itself, unless its background was set up with another (any intelligence channel the kit holds, another agent or an external agent among them, Section 6.4). A model never starts a task for another agent, whose worker may hold more than it does (a workspace, its tools) and in whose name the result would come back: it asks that agent by name, and only the host's start_task reaches any agent. It is handed the room's last messages (conversation, RoomKit: 12), and its result goes back to the calling agent (notify), as any delegation's (Section 23.3), relayed or posted as the setup's hand_back says: the host's choice for that agent's tasks, never the model's. The tool answers at once, with the task's id, and tells the agent to say once, in a sentence, that the work is under way (before the call or after it, never both) and to end its turn. A call from a task's child room is refused: a worker does its work and starts no task of its own. The host starts the same task with start_task(room_id, agent_id, task, hand_back = null), a delegation made outside a tool call whose hand_back is the agent's setup's when null; a person starts one through whatever the host gives them (a console's /task). Whether work is long is the agent's judgment through its tool or the host's: a model may keep long work in its own turn whatever its tool says, and a host whose agents do so starts their tasks itself.

• cancel_task — A tool an agent can be given on its own, as task_status, so that a request the person changed or dropped stops its work. It cancels one task of the room of the call, named by its task_id (from the delegation's answer or task_status): a task still running ends cancelled, as any task cancelled from outside does (Section 23.3). It reaches only the tasks the bus lists for that room: a call never cancels another room's task, nor one a strategy runs without posting it (post_status = false), which that strategy follows itself. It answers that the task was cancelled, that it had already ended (with its status), or that the room has no such task.

The room's tasks in the turn's notes. An AI channel's turn reads the room's tasks without a tool call: when the StatusBus lists tasks for the room of the turn, the channel's notes (Section 6.4) carry the latest of them (RoomKit: 6), each with its worker and what was asked, then either how long it has run, its latest progress with how long ago it was posted, and that no result has come back yet, so none of its data may be given, or how it ended. The notes say that each task's progress is the latest its worker gave, to answer from without a tool call when asked how far a task got, to speak of a running task only when asked, and to give nothing of its result before it comes back; a result itself comes back by its hand-back, never through the notes. What was asked and the progress are a worker's text: each is quoted as Section 6.4 quotes a text from outside (on one line, bounded, between marks it cannot close), set apart as data. What the notes give unquoted carries no text of its own: a worker's name keeps to an identifier's characters, and a task's ending is completed, failed, cancelled, or else ended. The notes are read from the bus as the turn is built, and nothing is stored: a task the bus no longer lists is no longer in them. A standalone turn carries none (Section 10.1.1), and a realtime session, which builds no turn, carries none either.

A delegation tool, whichever helper wired it (a supervisor's per-worker and strategy tools included), delegates from the room of the call, read from the tool call context (Section 21.4), as a handoff does (Section 19.6): the child room's parent is the room that issued the call, never the room the tool was wired in. A worker delegating in turn runs in its own child room, so its delegation hangs off that room. A call made outside a tool call MUST be refused.

23.5 Delegation Hooks

Hook	Execution	When
ON_TASK_DELEGATED	ASYNC	Task created and child room initialized
ON_TASK_COMPLETED	ASYNC	Task ended: completed with its result, failed with its error, or cancelled

---

24. Skills Framework

The Skills Framework enables agent capabilities to be defined as discoverable, self-contained packages.

24.1 Skill Definition

A Skill is defined by a SKILL.md file with YAML frontmatter:

---
name: "web-search"
description: "Search the web for current information"
license: "MIT"
compatibility: "Needs network access"
allowed_tools:
  - "search_*"
  - "fetch_*"
requires:
  - search_web
---

## Instructions

You are a web search specialist. When asked to find information...

## References

- [Search API docs](https://...)

The frontmatter is read as YAML, within bounds that keep a few hundred bytes from exhausting the host: a YAML alias is refused, and so are collections nested past 64 levels, more than 64 brackets or braces open at once (quoted ones counted too) and a frontmatter past 65536 characters, each before anything is built. Past a bound, the skill cannot be parsed (Section 24.5), and a served skill's content fails its check (Section 24.7).

A host's own skill whose frontmatter is not YAML, or is read without a YAML reader, MAY be read line by line, each key: value line at the top level a field; a line indented under a key is not one. That reading MUST keep both YAML list forms: a key alone on its line followed by - item lines, as in the example, and [a, b] whole on the key's line. Of a frontmatter that is also YAML, it MUST NOT gate or require less than the YAML reading does. Whichever reader, an allowed_tools or requires that is neither a text nor a list of texts, or that a line-by-line reading cannot read whole (a list spread over several lines, a block scalar, a line under the key that is not a - item), means the skill cannot be parsed: a skill read without what it gates hands the model, before any activation, the tools it was meant to keep closed.

24.2 SkillMetadata

SkillMetadata
├── name: string                            # Unique skill identifier
├── description: string                     # What the skill does
├── license: string | null                  # License identifier
├── compatibility: string | null            # Free text, not interpreted; shown with
│       # the license in to_prompt_xml()
├── allowed_tools: list<string>             # Tool access patterns (ToolPolicy globs);
│       # in frontmatter, a list or comma-separated (a text `[a, b]` and
│       # quoted entries read as that list; a text in brackets also gates
│       # itself, the glob it is: `[ab]` covers `a` and `b`)
├── requires: list<string>                  # Exact names of the tools the skill needs
│       # (Section 24.3), read from extra_metadata `requires` as
│       # allowed_tools is, its glob reading aside; empty by default
└── extra_metadata: map<string, string>     # Every other frontmatter key (`version`,
        # `requires`, ...), its value as a string, a list's items joined
        # by commas

24.3 SkillRegistry

SkillRegistry
├── discover(*directories) → int            # Scan directories for SKILL.md files, return count
├── register(skill_dir) → SkillMetadata     # Register a single skill
├── get_metadata(name) → SkillMetadata | null  # Look up skill metadata
├── get_skill(name) → Skill | null          # Load full skill (metadata, instructions and
│       # the path of its directory)
├── add(skill: Skill) → void                # Register a skill built in memory
├── add_mcp_server(provider, label, ...) → int  # Register the skills an MCP server
│       # serves (Section 24.7)
├── copy(names?, marks = true) → SkillRegistry  # A subset, its paths and marks kept
├── all_metadata() → list<SkillMetadata>    # List all registered skills
├── mark_unavailable(name, reason) → void   # The skill leaves the available set, its reason kept; what it gated stays closed
├── get_unavailable_reason(name) → string | null
├── unavailable_skills → map<string, string>  # Unavailable skill name → reason
├── mark_unlisted(name) → void              # Still activatable, left out of to_prompt_xml()
├── gated_tool_names(activated) → set<string>  # Patterns kept closed: unactivated skills', closed ones
├── closed_tool_names(activated) → set<string> # Patterns only unavailable skills gate, none activated opens
├── requires_match: (name, offered) → bool  # How a requires name is served (default: an exact tool name)
└── to_prompt_xml() → string               # Generate <available_skills> XML for AI context

A host that builds its skills elsewhere (a store, a marketplace) registers them with add, and one that narrows a registry for an agent copies it with copy: a copy finds every skill the source finds, a skill discovered but not yet loaded included, and keeps the source's marks unless asked not to.

Skills are injected into the AI agent's system prompt via to_prompt_xml(). The agent can then select and apply skills based on the user's request. A skill a host marks unavailable is listed there with its reason, and a registry whose every skill is unavailable still has something to tell: the prompt carries the reasons and the activation tools are declared, on every path, so the model says why instead of guessing. A skill's requires is checked by every activation, on a text turn as on a realtime session, with one rule: against every tool the conversation declares, the channel's own and what orchestration set up included, once its tool policy is applied and skill gating aside (the skill opens what it gates). A requires name is an exact tool name unless the registry's host says how its names are served (requires_match: a hub served by any tool named <hub>_*, say), and an activation hands over the schemas of the tools that serve it. A tool the policy denies is no tool the skill can use: the activation is refused (Required tools not available: <names>) and never hands over that tool's schema, to the model or to a hook. A skill marked unavailable can never be activated, so the tools it gates stay closed (marked while the registry holds it): marking it does not open them, a required tool only such a gate holds is missing, and a call to it reads that a skill not available here gates it. A pattern an activated skill gates is that skill's to open: an unavailable skill gating the same pattern does not hold it closed (closed_tool_names). gated_tool_names is the one reading of skill gating both doors use.

24.4 Activation Lifecycle

Loading a skill's body is an activation, and an activation is scoped to the conversation, not to the turn that made it. A model working across several turns on the same task is following the same rules throughout; a skill that silently stops applying between two turns is a skill the model cannot rely on.

An implementation MUST keep an active skill's instructions reachable by the model for as long as the skill is active. It MUST NOT report a skill as active while its instructions are absent from the model's context.

The instructions MAY travel through the system prompt rather than through the activation tool's result. This is RECOMMENDED where the context is rebuilt per turn from stored events: tool results that do not survive that rebuild force the model to re-activate — and re-pay the full body — on every turn, whereas the system prompt is composed fresh each turn and can carry the body once. Where the body is carried this way, the activation tool SHOULD answer subsequent calls with a short acknowledgement instead of the body, and that acknowledgement MUST state that the instructions are already in force.

An active skill's instructions MUST NOT be truncated, evicted, or summarized by a large-result or context-compression mechanism. Such mechanisms exist for data, which can be paginated or condensed without changing what it means; binding rules reduced to a preview are no longer binding rules. Skill references are data and remain subject to those mechanisms.

Activation state is per (implementation-defined) conversation scope — a room, a session — and is never global to a channel serving several conversations. An implementation MAY bound the number of simultaneously active skills; when it does, retiring a skill MUST be equivalent to it never having been activated, so that a later activation delivers the body again.

An implementation whose activation record is lost (process restart, a channel object replaced) MUST fall back to delivering the body: losing the record and losing the prompt block are the same event, and the model is never left holding an acknowledgement without rules.

Tool gating (Section 21) follows the same scope: tools revealed by activating a skill stay revealed while the skill is active.

A skill an MCP server serves follows this lifecycle with what Section 24.7 adds: its instructions are data set apart as that server's, never among the binding rules of the host's own skills, and a lost record of its activation is not restored from the stored history.

An activation that finds no skill under its name, when that name matches tools the turn may call (a model that took the spotify_play tool for a "spotify" skill), answers with the skill's absence and names those tools, and reveals them as find_tools reveals its matches (the window swapped, what is declared anyway left out), for the rest of the session, once the call is served: an ON_TOOL_CALL block reveals nothing. The same answer and the same reveal on every door, text or realtime; where the tools are reached through call_tool, the answer says so. Only the turn's own catalogue is matched, never the tools the channel serves itself. A skill that exists but is unavailable answers with its reason and no such hint.

24.5 Loading Failures

A skill that cannot be parsed, or whose metadata does not validate, is a deployment error rather than a runtime condition. The two have opposite remedies. A skill that exists but cannot be used in the present context is a fact the model can be told and can work around; a malformed SKILL.md is a mistake in what someone deployed, and nothing the model does at runtime will repair it.

An implementation MUST NOT silently omit a skill it failed to load. Omission is the worst outcome available: the catalogue simply does not list the capability, the model is never told it was meant to exist, and the agent answers as though nothing were missing. Nobody reading the conversation sees a gap either. The error surfaces much later, as an agent that inexplicably cannot do the thing it was configured to do, by which point the frontmatter typo that caused it is far out of view.

Discovery MUST therefore fail by default on the first skill it cannot load, and MUST report which skill failed and why. A directory that cannot be scanned is the same class of failure and MUST be reported the same way: returning an empty count for a path that does not exist is, to the caller, indistinguishable from a directory that is genuinely empty.

An implementation MAY offer a mode that logs each failure and continues. It is the right behaviour where skills come from a source the deployment does not control and a partial catalogue beats no service at all. That mode MUST be requested explicitly by the caller and MUST NOT be the default, and every skipped skill MUST be logged with the reason it was skipped.

Discovery MUST NOT leave the registry partially populated. Whatever the outcome, the registry a caller observes afterwards MUST be either the state that preceded the call or the state with every discovered skill present. A half-filled catalogue reproduces exactly the silent gap this section exists to prevent, and does so at the moment the caller has most reason to believe the opposite.

24.6 Access to Bundled Files

A skill's references and scripts are addressed by name, and the name is chosen by the model. Resolution MUST be confined to the skill's own directory.

Filtering the name does not achieve that, and an implementation MUST NOT rely on name filtering alone. A name containing no traversal sequence, no separator and no drive letter still leaves the directory when a symbolic link bearing that name sits inside it. An implementation MUST resolve the candidate path, with links followed, and MUST verify that the result is still inside the skill directory, rejecting it otherwise. Containment is a property of where a path ends up, not of how it is spelled.

Names MUST be normalized before they are inspected, so that a character which is not the ASCII separator but normalizes onto it cannot pass a check performed on the raw string. Absolute paths MUST be rejected, in every spelling the host platform recognises.

The requirement does not lapse where execution policy — sandboxing, timeouts, permitted interpreters — is delegated to the integrator. What runs is part of the skill boundary; how it runs is not. An implementation that delegates execution MUST resolve the script name under these same rules and MUST reject a name that fails them before any integrator-supplied executor is invoked, and SHOULD make the resolved path available to that executor. A check every integrator is expected to rediscover is a check some integrator will omit.

24.7 Skills Served over MCP

An MCP server MAY serve skills beside its tools under the MCP Skills extension (io.modelcontextprotocol/skills, SEP-2640): it lists them with skills/list, answers skills/get for one by the URI of its SKILL.md, and serves each file of a skill as a resource read with resources/read. The skill format is the one Section 24.1 reads. This section is what a registry, and every channel serving its skills, adds for such a skill; a skill the host deploys itself is unchanged.

SkillRegistry
└── add_mcp_server(provider, label, allow_dynamic = false,
                   approve_execution = null, execution_tools = []) → int

Registration. add_mcp_server lists the skills of the server provider is connected to (Section 21.2) and registers them under label, returning how many it registered. The label is the host's name for that server: an identifier, never the name the server gives itself, which any server can copy. A registry MUST refuse a label it already holds for another provider, and MUST raise for a server that does not declare the extension rather than register nothing in silence. The listing follows the cursor across every page within the bounds of a tool listing (Section 21.2) and reads no file: nothing of a skill is fetched before it is activated. A listing answered over a connection that negotiated an earlier protocol revision may lack ttlMs, cacheScope and resultType, and is read as complete. Registering a server again replaces its entries: a skill it no longer offers, or offers in a form the host does not take, retires and is served nowhere any more; one it still offers keeps, while active, the entry it was loaded from. A copy of a registry narrowed to some names (Section 24.3) reaches those names and no other: only a registry that took the server whole asks it for a skill its listing did not hold. A registry and its copies share the entries the servers gave: a skill one of them retires, when it registers the server again or when a file fails its check, is retired in each of them, an active one included, and a copy made without the names its source marks unavailable never asks a server for them either.

Entries. Each entry is read on its own. An entry whose resources is missing or neither a list nor "dynamic", whose URI's last segment before SKILL.md is not its frontmatter name, whose list omits its own SKILL.md, names a file outside the skill's directory or one file twice; an entry past 512 files or 16 MiB in all; and an entry whose resources is "dynamic" while the host did not allow them (allow_dynamic) is registered as unavailable with its reason (Section 24.5: a skill the server meant to offer is never dropped in silence). The server's other skills stay usable. A "dynamic" skill offers nothing to verify: a host that allows one takes its content unverified, and an implementation MUST stop reading such a skill once what it read reaches 16 MiB.

Names. A skill an MCP server serves is registered under <label>:<path>, which is its name in the registry (Section 24.2): its path is its URI without the trailing /SKILL.md, and without the scheme when the scheme is skill:// (hf:hf-cli, hf:acme/billing/refunds, docs:github://owner/repo/skills/refunds). A host's own skill is named as Section 24.1 names it, which never holds :, and a registry MUST refuse to give a name it holds to a skill of another origin, so that a served skill never shadows a host's skill nor another server's, and the reverse. Two entries of one listing sharing a frontmatter name keep distinct paths. The skill's identity is the pair of its label and its URI everywhere it is kept: the activation record, the entry held, what was verified. The manifest (to_prompt_xml()) names the label as the skill's origin and gives its description as Section 6.4 quotes a text from outside. requires is read from the frontmatter as for any skill (Section 24.3).

Activation. Activating such a skill reads its SKILL.md from its own server then, and not before. The content MUST match the entry before anything of it reaches the model: its length the entry's size, its SHA-256 digest the entry's digest, and its frontmatter, parsed, the entry's frontmatter field by field. A mismatch loads nothing: the registry reads the entry again with skills/get, and the activation answers that the skill changed on its server or could not be verified, without its body. An entry read again is held to the rules a listed one is (Entries): one the host does not take leaves the skill unavailable with its reason. The instructions then follow Section 24.4 (whole, carried by the system prompt, acknowledged afterwards, retired as if never activated) as data: in a <skill> block (Section 6.4) naming the label and the URI, after the host's own skills and never among their binding rules. Where the body travels in the activation's result (a realtime provider whose instructions are fixed), it is set apart the same way. A skill's content outranks nothing else in the context. An activation records what it loaded, never what a second lookup finds: a skill marked unavailable is never activated, one loaded before included. A session keeps an activated skill's body as it rendered it, set apart, so nothing read later delivers it otherwise.

A listing may be partial or empty, which proves no skill missing. An activation naming <label>:<path> that the registry does not hold, under a label it holds, asks that server for the entry with skills/get: an entry it returns is registered as a listed one would be, kept out of the manifest (mark_unlisted), and activated; an error answers that the server serves no such skill. A skill the server's own instructions point to is reached so.

Files. While such a skill is active, the registry holds the entry it was loaded from. read_skill_reference resolves the name it is given against the skill's directory (templates/invoice.md), reads the file from the same server, and reads only a file that entry lists: a name the entry does not list is refused as content that failed verification, whatever the server would serve. Each file read is verified against the entry as SKILL.md is, and reaches the model naming the same origin. A file that fails retires the skill (Section 24.4): the registry reads the entry again, and the model is told to activate the skill again, which delivers its current body. A read never reaches another server: the content of one server's skill causes no read on another.

Lost record. An activation of such a skill is not restored from the stored history: its body is no longer held, so its next activation delivers it again (Section 24.4). A delivery that hands every skill's body over when a session starts (a realtime session in inline_full mode) would fetch them all ahead of need: there, a skill an MCP server serves is unavailable, with that reason.

Permissions. What such a skill's frontmatter would grant is not granted: its allowed_tools gates nothing and opens nothing (Section 24.2), and a SKILL.md within its directory is one of its files, never a skill activated with it. Its files are never written to disk to be run: run_skill_script refuses it. While such a skill is active in a conversation, a call to a tool that runs code (a sandbox command, sandbox_*, and every tool the host names in execution_tools) is served only when approve_execution approves each such skill active there; an implementation MUST refuse the call otherwise (a refused call, Section 9.3), and always when the host gave no callback. The callback receives the label, the URI, the conversation and the entry's fingerprint (the SHA-256 of its files' URIs and digests, sorted), and an approval holds for that fingerprint only: an entry read again with another set of files is asked again. A host that keeps approvals keeps them by fingerprint. A "dynamic" skill has no fingerprint an approval could bind, and the callback is told so.

Trust. A digest the server supplies proves the content consistent with an entry from the same server, never trustworthy: the content stays text from outside (Section 6.4). Content kept on disk between reads MUST meet the extension's cache requirements (out of reach of what the model runs, verified again on each read, outside every directory a registry discovers skills in); content kept in memory for the entry held needs none of them.

---

25. Image Generation

An agent that can describe a picture cannot draw one. Image synthesis is a capability a small number of models hold, and the model holding a conversation is rarely one of them — so it is specified here as its own provider surface, the way speech-to-text and text-to-speech already are, and not as an extension of the conversational response.

25.1 A Surface of Its Own

An implementation MUST expose image generation through a provider interface distinct from the AI provider interface, and MUST NOT require that the two be served by the same vendor, credential, or object. An agent conversing through one provider MUST be able to draw through another, exactly as it transcribes through a third.

This is a normative requirement rather than an implementation preference. Image generation reached as a mode of the conversational response inherits that response's provider: the capability then exists only for the subset of conversations already held by a model that draws, and acquiring it means migrating the conversation. The decoupled form is what makes the capability available to every conversation, and it is the property this section protects.

An implementation MUST NOT widen the conversational response type to carry generated images. AIResponse.content is read by every consumer of the framework; turning it into a union imposes a migration on all of them for a capability that two providers offer.

25.2 ImageProvider

ImageProvider
├── name → string                             # Provider identifier
├── model_name → string                       # Active model id
├── supports_editing → bool                   # Whether reference_images is honoured
├── available_models() → list<ModelInfo>      # Offline catalogue (Section 25.6)
├── generate(prompt, size, n, reference_images) → list<ImageResult>
└── close()                                   # Release resources

generate MUST return exactly n results or raise. Returning fewer without error leaves the caller to discover the shortfall by indexing, which it will do in production and not in testing.

size is a "WIDTHxHEIGHT" string, or null for the provider's default. A provider whose API expresses geometry differently — an aspect ratio and a resolution tier, say — MUST translate, and MUST NOT require the caller to know which form its vendor speaks. A provider that cannot honour a requested size MUST raise rather than silently substitute another: an image of the wrong geometry is a failure the caller can neither see nor correct.

n greater than one MUST produce n distinct results. A provider whose API has no batch parameter satisfies this with concurrent calls: the alternative — honouring n on some providers and refusing it on others — leaves a parameter in the interface that a caller cannot use without first knowing which provider is behind it, which is the coupling this section exists to remove.

Implementations MAY accept an optional typed options argument for controls such as quality, output format, background, compression, aspect ratio and resolution tier. An explicit aspect ratio and tier are native geometry, not a promise of exact pixels. An implementation MUST reject incompatible or unsupported options before making a billable request. Omitted options inherit provider defaults; explicit values override them. Existing calls without options MUST remain valid. A mask MAY accompany references when supported.

An error after a request has started MUST preserve every available result and usage report. A partial generation still raises, carrying successful results and the outcomes of individual vendor calls. An implementation MUST NOT mark an ambiguous network failure as unbilled or automatically retry a paid generation. A failed status is marked retryable only where the vendor did not run the generation (408, 409, 429, 503); a 500, 502 or 504 may follow one it ran and billed, and is final: the host decides. A transport failure is retryable only when the request never left (the connection could not be made); a timeout or a connection lost once the request went out is ambiguous, and final. An image the vendor delivered in a form the provider cannot return was generated and billed: that failure is final too. Independent concurrent calls SHOULD settle so their outcomes can be collected. Cancellation MUST remain cancellation, with already received outcomes accessible through an optional asynchronous progress callback.

25.3 ImageResult

ImageResult
├── data: string                              # data:<mime_type>;base64,<payload>
├── mime_type: string                         # Media type of the payload
├── revised_prompt: string | null             # Prompt as the model rewrote it, if reported
└── usage: map<string, any>                   # Token counters (Section 25.5)

data MUST be a data URI, always, and its media type MUST equal mime_type. The alternative — a field documented as "a data URI or bare base64" — forces every consumer to sniff the value before it can use it, and one consumer will sniff it wrong. The invariant also makes the result directly usable: a data URI is what image message content and image message parts already accept, so a generated image enters a room, or returns as a reference for a subsequent edit, without conversion.

A provider whose API returns a fetchable URL rather than bytes MUST retrieve the bytes and encode them. Such URLs expire, and a result that decays into a dead link some time after it was returned is not a result.

revised_prompt is null where the vendor reports none. An implementation MUST NOT substitute the original prompt for it: the field exists to show the caller what the model actually drew from, and echoing the input back conceals exactly the divergence it exists to reveal.

Results MAY carry a vendor request or interaction identifier, the vendor's original usage report, effective parameters, actual dimensions where reported, and grounding attribution. Usage shared by a batch MUST be represented once, with its scope explicit; it MUST NOT be divided into invented per-image measurements. Original metadata MUST omit inline image payloads and secrets.

A progress callback MAY receive individual call outcomes or preview images. Previews MUST be distinguished from final results. Callback delivery does not imply vendor cancellation or a refund, and failure to deliver an outcome MUST NOT trigger another generation.

25.4 Editing

Editing — an image and a prompt in, an image out — is expressed through reference_images on generate, not through a second method.

The distinction is the vendor's, not the caller's: one API takes reference images as ordinary content on the same endpoint, another routes them to a separate one. A provider MUST absorb that difference. A provider that cannot edit MUST report supports_editing as false and MUST raise when reference_images is non-empty, rather than generating from the prompt alone — a caller that asked for an edit and silently received a fresh image has no signal that the reference was dropped.

A provider MAY support continuity through a previous interaction identifier. The caller remains responsible for authorizing access to that interaction. A stateful provider MUST expose an explicit storage choice so applications can disable remote history retention. Omitting this choice MAY preserve the provider default; applications MUST NOT assume that omitted means stateless. Grounding with external search MAY be requested only on compatible models; available citations and search-suggestion metadata MUST be retained for the caller to display according to the provider's requirements.

25.5 Usage Accounting

A generation reports what it consumed. An image and the text around it are metered separately by the vendors that meter them at all, so the counters stay apart rather than collapsing into one number:

usage
├── input_tokens: int                         # Text input tokens
├── input_image_tokens: int                   # Reference-image input tokens
├── output_tokens: int                        # Text output tokens, where the model emits any
└── output_image_tokens: int                  # Generated-image tokens

The counters MUST be disjoint: a token appears under exactly one of them. Where a vendor reports image tokens as a subset of a total, an implementation MUST subtract them before reporting the text counter, so a consumer summing the counters counts each token once. Without that guarantee the counters cannot be combined at all, and the breakdown is worth less than the single total it replaced.

Counters a vendor does not report are absent, not zero: a reported zero and an unreported quantity are different claims, and only one of them is safe to add up.

Usage is a report, not a price. What a generation costs depends on how its vendor meters it — per token in some lineups, a flat amount per image in others, varying by resolution or quality — and on commercial terms no specification can observe. An implementation MAY carry a price list and MAY derive a cost from these counters; neither the shape of such a list nor the arithmetic over it is specified here. What is specified is that the counters mean what they say, so that whatever does price them has something unambiguous to price.

25.6 Model Catalogue

An image catalogue is a set disjoint from the conversational one: a model that draws is not a model that converses, and neither list contains an id belonging to the other.

An implementation MUST NOT merge the two into the response of a single enumeration. Merging saves no maintenance — the entries are written once either way — while obliging every consumer of the conversational catalogue, including those populating a model field for a conversational agent, to filter a class of models it can never use. Entries SHOULD nevertheless carry a capability tag identifying them as image-generating, so a consumer that deliberately combines the lists can still tell them apart.

Catalogue entries MAY carry structured image capabilities, including accepted controls and values, geometry constraints, reference limits, continuity, masking, search, and retirement dates. Unknown capabilities MUST remain unknown rather than being inferred from another model in the same vendor's lineup.

25.7 Non-Goals

Image generation is not a pipeline stage and defines no hook triggers. It is invoked by a caller — a tool handler, an application — and its result enters a room as ordinary message content, which the inbound and broadcast pipelines (Section 10) already govern. Introducing a parallel path for images would duplicate ordering, permission, and visibility rules that Section 7 and Section 10 state once.

Video generation, image understanding (already covered by multimodal message parts) and avatar synthesis (Section 12) are outside this section.

---

26. Conformance Levels

26.1 Level 0: Core (REQUIRED)

A conforming implementation MUST support:

• Room model with lifecycle (ACTIVE, PAUSED, CLOSED, ARCHIVED), including refusing new events at every entry point once the status says so (Section 5.1)
• Room timers (auto-pause, auto-close)
• RoomEvent with all EventType values and all EventContent types
• Sequential event indexing
• Channel interface (handle_inbound, deliver, on_event, capabilities)
• ChannelBinding with access, mute, and visibility
• Permission enforcement (Section 7.5)
• Hook engine with SYNC and ASYNC execution
• BEFORE_BROADCAST and AFTER_BROADCAST hooks with HookResult
• InjectedEvent delivery on block
• Event chain depth tracking and limiting
• Inbound processing pipeline (Section 10.1)
• Broadcast pipeline (Section 10.2)
• Reentry passes (response events re-enter the pipeline as commit passes)
• Inbound room routing (pluggable)
• Room-level locking
• ConversationStore interface with in-memory implementation
• Content transcoding (at least text fallback)
• Framework events (Section 8.2)
• Structured logging
• Idempotency checking
• Event addressing: RoomEvent.addressed_to, honoured when set (Section 19.3)

26.2 Level 1: Transport (RECOMMENDED)

A Level 1 implementation SHOULD additionally support:

• At least one SMS provider
• At least one Email provider
• WebSocket channel
• HTTP/Webhook channel
• AI channel with at least one AI provider
• Agent class (AIChannel subclass with identity metadata)
• Provider abstraction (swappable per channel type)
• Identity resolution pipeline
• Identity hooks (ON_IDENTITY_AMBIGUOUS, ON_IDENTITY_UNKNOWN)
• Participant model with identification status
• Memory system with at least SlidingWindowMemory (Section 20)
• Circuit breaker
• Retry policy
• Rate limiting
• Telemetry provider abstraction (Section 15.7)

26.3 Level 2: Rich (OPTIONAL)

A Level 2 implementation MAY additionally support:

• WhatsApp channel (Business and/or Personal)
• Messenger channel
• Teams channel
• Telegram channel
• RCS channel with SMS fallback
• Template content support
• Source providers (persistent connections)
• REST API (Section 16.1)
• MCP Server (Section 16.2)
• ACP agent channel (Section 6.4)
• Realtime/ephemeral events backend
• Per-room hooks
• Multi-agent orchestration (Section 19):
• ConversationRouter with routing rules, addresses taking precedence (Section 19.4, step 0)
• Both agent-response policies, selectable per room (Section 19.3.1)
• ConversationPipeline with stages
• HandoffHandler with tool-based handoff protocol
• At least Pipeline and Swarm strategies; Discussion MAY be provided (Section 19.7.5)
• Tool access control with ToolPolicy (Section 21)
• Task delegation with child rooms (Section 23)
• Delivery strategies: Immediate, WaitForIdle, Queued (Section 22)
• Skills framework with SkillRegistry (Section 24)
• Image generation through an ImageProvider decoupled from the AI provider (Section 25)
• Response schemas: an AI provider constraining its answer to a portable JSON Schema (Section 6.7)
• Identity verification: requests, labels, holding and release, the three verification triggers (Section 11.7)
• AI steering directives (Section 21.3)
• Advanced memory providers: Summarizing, Retrieval (Section 20)

26.4 Level 3: Real-Time Media (OPTIONAL)

A Level 3 implementation MAY additionally support audio and/or video real-time media:

Audio (Voice)

• Audio Processing Pipeline with:
• AudioFrame and AudioFormat data models
• AudioPipelineConfig with pipeline format contract
• ResamplerConfig (RECOMMENDED)
• VADProvider interface with at least one implementation (e.g., Silero) — REQUIRED for VoiceChannel, OPTIONAL for RealtimeVoiceChannel
• DenoiserProvider interface (OPTIONAL, e.g., sherpa-onnx)
• AECProvider interface (OPTIONAL, RECOMMENDED for non-WebRTC transports)
• AGCProvider interface (OPTIONAL, built-in implementation REQUIRED)
• DTMFDetector interface (OPTIONAL, RECOMMENDED for SIP/PSTN)
• DiarizationProvider interface (OPTIONAL)
• TurnDetector interface (OPTIONAL)
• BackchannelDetector interface (OPTIONAL)
• AudioRecorder interface (OPTIONAL, RECOMMENDED for regulated industries)
• AudioPostProcessor interface (OPTIONAL)
• InterruptionConfig
• Voice channel (STT/TTS pipeline)
• VoiceBackend interface with at least one implementation
• AudioCaptureSource interface (OPTIONAL, REQUIRED for capture that outlives a session — Section 12.12)
• Packet loss concealment for lossy transports (OPTIONAL, RECOMMENDED for RTP backends — Section 12.2.1)
• TTS conversation context — TTSContextLevel, TTSContext, release_context (OPTIONAL — Section 12.2.2)
• STTProvider interface with at least one implementation
• TTSProvider interface with at least one implementation
• Voice catalog — VoiceInfo, available_voices, list_voices (OPTIONAL — Section 12.2)
• Dialogue synthesis — synthesize_dialogue, DialogueTurn (OPTIONAL — Section 12.2)
• VoiceLibrary interface — custom voices, designed or replicated with consent (OPTIONAL — Section 12.2.4)
• Voice hooks (ON_SPEECH_START through ON_RECORDING_STOPPED)
• Barge-in and interruption handling (InterruptionStrategy)
• Audio bridging — AudioBridge (OPTIONAL — Section 12.7)
• Realtime Voice channel (speech-to-speech)
• RealtimeVoiceProvider interface
• Full-duplex providers and reasoning delegation (OPTIONAL — Section 12.4.1)
• RealtimeAudioTransport interface
• Realtime voice hooks (ON_TOOL_CALL, ON_REALTIME_TEXT_INJECTED, ON_PROTOCOL_TRACE)
• Protocol trace infrastructure (ProtocolTrace, emit_trace, on_trace, pre-room buffering)

Video

• Video data models:
• VideoFrame — inbound frame container (encoded NAL units or raw pixels)
• VideoChunk — outbound encoded frame container
• VideoSession — active video connection state
• VideoCapability flags (SIMULCAST, SVC, SCREEN_SHARE, RECORDING, BANDWIDTH_ESTIMATION)
• VideoBackend interface — transport abstraction (connect, disconnect, send_video, callbacks)
• VideoChannel — session-based channel orchestrator:
• Session lifecycle with dual-signal ready mechanism (matching VoiceChannel pattern)
• Hook triggers: ON_VIDEO_SESSION_STARTED, ON_VIDEO_SESSION_ENDED, ON_VISION_RESULT, ON_VIDEO_DETECTION, BEFORE_BRIDGE_VIDEO (ON_VIDEO_TRACK_ADDED and ON_VIDEO_TRACK_REMOVED are Planned, Section 9.2)
• Optional VisionProvider integration with configurable analysis interval
• Vision results emitted as framework events (video_vision_result)
• VisionProvider interface — frame analysis abstraction:
• analyze_frame(frame, prompt, response_schema) → VisionResult (description, labels, confidence, faces, OCR text)
• analyze_stream(frames, interval_ms, assumed_fps) for streaming analysis (Section 12.8.7)
• Implementations: OpenAI-compatible (GPT-4o, Ollama, vLLM), Gemini, Mock
• VideoRecorder interface (OPTIONAL — Section 12.8.10)
• AI integration: vision results reach the AI channels' turns as a <vision> block (Section 12.8.7); setup_video_vision() is deprecated and wires nothing
• Video and voice channels operate independently in the same room, enabling combined audio+video sessions where the AI can both hear (via STT) and see (via VisionProvider)

Room media recording

• MediaRecorder interface (OPTIONAL — Section 12.11); REQUIRED for an implementation that records conferences in framework mode (Section 12.10.8)

Conference (SFU)

• Conference data models (ConferenceTrack, TrackKind, ConferenceParticipant, ConferenceGrants, ConferenceAccess, BotSession, ConferenceCapability)
• ConferenceBackend interface with MockConferenceBackend (Section 12.10.3)
• ConferenceChannel with per-track STT lanes and bot TTS publication (Section 12.10.4)
• Conference hooks (ON_CONFERENCE_PARTICIPANT_JOINED/LEFT, ON_CONFERENCE_TRACK_PUBLISHED/UNPUBLISHED, ON_CONFERENCE_TRACK_MUTED/UNMUTED, ON_ACTIVE_SPEAKER_CHANGED, ON_CONNECTION_QUALITY_CHANGED, and ON_SCREEN_SHARE_STARTED/STOPPED for screen-share tracks)
• Explicit track subscription, participant identity correlation, and bot self-exclusion (Sections 12.10.2–12.10.4)
• Multi-party interruption policy (ConferenceInterruptionConfig)
• Participant lifecycle integration (conference events create/update Participant records)
• Optional: vision on tracks (Planned), egress recording delegation, SIP gateway interop, bot video publication (avatar), speech-to-speech composition

---

Appendix A: Channel Reference

A.1 SMS Channel

SMSChannel
├── type: SMS
├── category: TRANSPORT
├── direction: BIDIRECTIONAL
├── media_types: [TEXT, MEDIA]
├── capabilities:
│   ├── max_length: 1600
│   ├── supports_read_receipts: true
│   ├── supports_media: true
│   ├── supported_media_types: ["image/jpeg", "image/png", "image/gif"]
│   ├── supports_edit: false
│   └── supports_delete: false
├── provider_interface: SMSProvider
│   ├── send(event, to, from) → ProviderResult
│   ├── parse_webhook(payload) → InboundMessage
│   └── verify_signature(payload, signature, timestamp) → bool
└── binding_metadata:
    └── phone_number: string (recipient)

A.2 Email Channel

EmailChannel
├── type: EMAIL
├── category: TRANSPORT
├── direction: BIDIRECTIONAL
├── media_types: [TEXT, RICH, MEDIA]
├── capabilities:
│   ├── max_length: null (unlimited)
│   ├── supports_threading: true
│   ├── supports_rich_text: true
│   ├── supports_media: true
│   ├── supports_edit: false
│   └── supports_delete: false
├── provider_interface: EmailProvider
│   ├── send(event, to, from, subject) → ProviderResult
│   └── parse_inbound(payload) → InboundMessage
└── binding_metadata:
    ├── email_address: string (recipient)
    └── subject: string | null

A.3 WhatsApp Channel

WhatsAppChannel
├── type: WHATSAPP
├── category: TRANSPORT
├── direction: BIDIRECTIONAL
├── media_types: [TEXT, RICH, MEDIA, LOCATION, TEMPLATE]
├── capabilities:
│   ├── max_length: 4096
│   ├── supports_reactions: true
│   ├── supports_templates: true
│   ├── supports_buttons: true (max 3)
│   ├── supports_quick_replies: true
│   ├── supports_read_receipts: true
│   ├── supports_media: true
│   ├── supports_edit: true
│   └── supports_delete: true
├── provider_interface: WhatsAppProvider
│   ├── send(event, to) → ProviderResult
│   ├── send_template(template, to) → ProviderResult
│   └── send_reaction(chat, message_id, emoji) → ProviderResult
└── binding_metadata:
    └── phone_number: string (recipient)

A.4 WhatsApp Personal Channel

WhatsAppPersonalChannel
├── type: WHATSAPP_PERSONAL
├── category: TRANSPORT
├── direction: BIDIRECTIONAL
├── media_types: [TEXT, RICH, MEDIA, AUDIO, VIDEO, LOCATION]
├── capabilities:
│   ├── max_length: 4096
│   ├── supports_reactions: true
│   ├── supports_typing: true
│   ├── supports_read_receipts: true
│   ├── supports_audio: true
│   ├── supports_video: true
│   ├── supports_media: true
│   ├── supports_edit: true
│   └── supports_delete: true
├── source: WhatsAppPersonalSourceProvider
│   └── Persistent multidevice connection via neonize or equivalent
├── provider_interface: WhatsAppPersonalProvider
│   ├── send_message(jid, text) → ProviderResult
│   ├── send_image(jid, data, caption) → ProviderResult
│   ├── send_audio(jid, data, ptt) → ProviderResult
│   ├── send_video(jid, data, caption) → ProviderResult
│   ├── send_document(jid, data, filename) → ProviderResult
│   ├── send_location(jid, lat, lon, name, address) → ProviderResult
│   └── send_reaction(chat, sender, message_id, emoji) → ProviderResult
└── binding_metadata:
    └── phone_number: string (recipient)

A.5 Messenger Channel

MessengerChannel
├── type: MESSENGER
├── category: TRANSPORT
├── direction: BIDIRECTIONAL
├── media_types: [TEXT, RICH, MEDIA, TEMPLATE]
├── capabilities:
│   ├── max_length: 2000
│   ├── supports_buttons: true (max 3)
│   ├── supports_quick_replies: true
│   ├── supports_read_receipts: true
│   ├── supports_media: true
│   ├── supports_edit: false
│   └── supports_delete: true
├── provider_interface: MessengerProvider
│   ├── send(event, recipient_id) → ProviderResult
│   └── parse_webhook(payload) → InboundMessage
└── binding_metadata:
    └── facebook_user_id: string (recipient)

A.6 Teams Channel

TeamsChannel
├── type: TEAMS
├── category: TRANSPORT
├── direction: BIDIRECTIONAL
├── media_types: [TEXT, RICH]
├── capabilities:
│   ├── max_length: 28000
│   ├── supports_threading: true
│   ├── supports_reactions: true
│   ├── supports_read_receipts: true
│   ├── supports_rich_text: true
│   ├── supports_edit: true
│   └── supports_delete: true
├── provider_interface: TeamsProvider
│   ├── send(event, conversation_reference) → ProviderResult
│   ├── parse_webhook(activity) → InboundMessage
│   └── save_conversation_reference(activity) → void
├── requires: ConversationReferenceStore
└── binding_metadata:
    └── teams_conversation_id: string

A.7 RCS Channel

RCSChannel
├── type: RCS
├── category: TRANSPORT
├── direction: BIDIRECTIONAL
├── media_types: [TEXT, RICH, MEDIA]
├── capabilities:
│   ├── max_length: 8000
│   ├── supports_buttons: true
│   ├── supports_cards: true
│   ├── supports_quick_replies: true
│   ├── supports_read_receipts: true
│   ├── supports_typing: true
│   ├── supports_media: true
│   ├── supports_edit: false
│   └── supports_delete: false
├── provider_interface: RCSProvider
│   ├── send(event, to) → RCSDeliveryResult
│   ├── check_capability(phone) → bool
│   └── RCSDeliveryResult includes: channel_used, fallback flag
└── configuration:
    └── fallback: bool (auto-fallback to SMS when RCS unavailable)

A.8 WebSocket Channel

WebSocketChannel
├── type: WEBSOCKET
├── category: TRANSPORT
├── direction: BIDIRECTIONAL
├── media_types: [TEXT, RICH, MEDIA, AUDIO, VIDEO, LOCATION]
├── capabilities:
│   ├── max_length: null (unlimited)
│   ├── supports_typing: true
│   ├── supports_read_receipts: true
│   ├── supports_reactions: true
│   ├── supports_buttons: true
│   ├── supports_cards: true
│   ├── supports_quick_replies: true
│   ├── supports_audio: true
│   ├── supports_video: true
│   ├── supports_media: true
│   ├── supports_edit: true
│   └── supports_delete: true
├── connection_registry: map<connection_id, send_function>
│   ├── register_connection(id, send_fn, room_id,
│   │       stream_send_fn: function | null) → void
│   │       # Subscribes the connection to room_id; with stream_send_fn
│   │       # it also receives streamed responses as they are produced
│   ├── subscribe(id, room_id) → void          # Also deliver room_id's events
│   ├── unsubscribe(id, room_id) → void        # Stop delivering room_id's events
│   └── unregister_connection(id) → void       # Also drops its subscriptions
└── delivery: sends to the connections subscribed to the event's room

A.9 AI Channel

AIChannel
├── type: AI
├── category: INTELLIGENCE
├── direction: BIDIRECTIONAL
├── media_types: [TEXT]
├── capabilities:
│   ├── supports_rich_text: true (provider-dependent)
│   ├── supports_edit: false
│   └── supports_delete: false
├── configuration:
│   ├── provider: AIProvider
│   ├── system_prompt: string | null
│   ├── temperature: float | null
│   ├── max_tokens: int | null
│   ├── max_context_events: int | null
│   ├── thinking_budget: int | null         # Token budget for extended thinking/reasoning
│   ├── enable_thinking: bool | null        # Reasoning switch, after thinking_budget (Section 6.7)
│   ├── reasoning_effort: string | null     # How much the model reasons (Section 6.7)
│   ├── response_schema: map | null         # Each turn answers in this JSON Schema (Section 6.7)
│   ├── max_tool_rounds: int (default 50)   # Maximum tool call iterations per generation
│   ├── max_empty_retries: int (default 1)  # Tries an empty round and the continuation share
│   ├── continuation: policy | null         # Goes on an answer that did not act (Section 6.4)
│   ├── turn_budget_tokens: int | null      # Billed tokens a turn may spend (Section 6.4)
│   ├── turn_budget_usd: float | null       # What a turn may cost at the catalogue price
│   ├── tool_loop_timeout_seconds: float | null (default 300)  # Timeout for entire tool loop
│   ├── fallback_provider: AIProvider | null # Fallback if primary provider fails
│   ├── evict_threshold_tokens: int (default 5000)  # Token threshold for context eviction
│   ├── memory: MemoryProvider | null       # Pluggable context construction (Section 20)
│   ├── describe_empty_event: callable | null # What an event with no content reads as
│   ├── tool_policy: ToolPolicy | null      # Tool access control (Section 21)
│   └── skills: SkillRegistry | null        # Available skills (Section 24)
├── per_room_overrides (via binding metadata):
│   ├── system_prompt
│   ├── temperature
│   ├── max_tokens
│   ├── thinking_budget
│   ├── enable_thinking
│   ├── reasoning_effort
│   ├── response_schema
│   ├── turn_budget_tokens
│   ├── turn_budget_usd
│   └── tools
├── ai_message_parts:
│   │   # AIMessage content parts (AIContentPart, Section 6.7), in the
│   │   # conversation a turn is built from and its tool loop extends:
│   ├── AITextPart                          # Text content
│   ├── AIImagePart                         # Image content
│   ├── AIThinkingPart                      # Chain-of-thought reasoning (preserved in history)
│   ├── AIToolCallPart                      # Function call with name and arguments
│   └── AIToolResultPart                    # Tool execution result
└── behavior:
    ├── on_event() builds conversation history + target capabilities
    ├── Reads provider.generate_structured_stream(context) (generate() wrapped when it does not stream)
    ├── Runs one tool loop: generate → call tools → feed results → re-generate (up to max_tool_rounds)
    ├── Skips events from self (loop prevention)
    └── Returns ChannelOutput with a response stream, which the framework
        delivers with deliver_stream() and stores segment by segment

Per-turn settings. Each per-turn setting (system_prompt, temperature, max_tokens, thinking_budget, enable_thinking, reasoning_effort, response_schema, turn_budget_tokens, turn_budget_usd) resolves from the binding metadata, then the channel's per-turn configuration, then the channel default. A level that does not set it, its key absent or null, defers to the next: null never clears a setting, at either level, so an integrator that serializes an empty form field as null keeps the next level's value. A setting is lifted for one room only by an explicit value that states off (thinking_budget: 0, enable_thinking: false, reasoning_effort: "none", system_prompt: ""). The reasoning switch is read across two keys (Section 6.7): an enable_thinking: false set at a level turns reasoning off whatever thinking_budget a less specific level set, while a budget set at the same or a more specific level still states the switch. A setting with no value that states off (temperature, max_tokens, response_schema, and the turn budgets of Section 6.4) cannot be lifted for a room once a less specific level sets it, only replaced. The binding's tools is a toolset of its own, and a null there declares none, like an absent key.

Constrained turn. A turn's response_schema resolves like the other per-turn settings. When it is set, the channel MUST NOT deliver or store any of the answer's text before the provider's check has passed (Section 6.7): a streamed answer is held until the done event, and a failed check fails the turn with its ResponseSchemaError, leaving nothing in the room. A turn whose tool loop stops before a final answer, on its round cap, its deadline or its spending budget, because it was stopped, or because the provider interrupted it after a round, has no document to deliver and fails with truncated. The turn's tools include those the channel adds itself (skills, sandbox, planning, orchestration), so a schema on a channel that adds any needs a provider whose supports_response_schema_with_tools is true.

A.9.1 ACP Agent Channel

ACPChannel
├── type: AI
├── category: INTELLIGENCE
├── direction: BIDIRECTIONAL
├── media_types: [TEXT, RICH]
├── protocol side: ACP client (external coding agent is the ACP server)
├── transport:
│   ├── stable default: stdio
│   ├── command: argument vector (never shell-expanded)
│   └── protocol_version: stable ACP version
├── sessions:
│   ├── one ACP connection per channel instance
│   ├── one ACP session per Room
│   ├── a standalone turn: its own session, declared as the turn's (§10.1.1)
│   ├── setup on every open: mcp_servers for the Room, session_config, then the
│   │   values the host set on the Room; a refused value fails the open
│   └── prompts serialized per session
├── configuration:
│   ├── command: list<string>
│   ├── cwd: absolute path
│   ├── additional_directories: list<absolute path>
│   ├── environment: map<string, string> | null
│   ├── authentication_method: string | null
│   ├── mcp_servers: list<ACP MCPServer> | (room_id) → list<ACP MCPServer>
│   ├── session_config: map<string, string | bool> | null  # set on every session opened
│   ├── external_tool_handler: ExternalToolHandler | null
│   ├── name, role, description: string | null  # its identity, as an Agent's (§6.4)
│   └── instructions: string | null             # sent once per session (§6.4)
├── update_mapping:
│   ├── agent_message_chunk → text stream delta
│   ├── agent_thought_chunk → thinking stream marker + ephemeral event
│   ├── tool_call_start/progress → tool-call markers + ephemeral events
│   └── agent_plan_update → ephemeral custom event
├── end_of_turn:
│   ├── completed turn → ON_AI_RESPONSE (text, tool count, duration, usage)
│   ├── errored or abandoned turn → no trigger
│   └── usage: agent's token counters relayed unaltered, plus context and cost
└── safety:
    ├── filesystem/terminal client capabilities disabled by default
    ├── permission requests denied by default
    ├── cancellation forwarded to ACP session
    └── close releases sessions and process transport

ACP response observations MAY include optional usage_metadata alongside the existing numeric usage map. This metadata identifies the native session, measurement source and scope, separately from the host room/channel/request. A session cost MUST remain a cumulative observation, not a prompt cost; context occupancy/capacity MUST NOT be treated as consumed-token counters. Missing measurements MUST remain absent and reported zero MUST remain zero.

Transport-provided report/session/result identities and timestamps MUST be preserved across recovery and replay when available. A recovered terminal snapshot MUST NOT be combined with live observations from another session. An observation without a source prompt identity MUST NOT be assigned one from the currently active prompt. Locally assigned receipt identities MUST be identified as local and MUST NOT imply durable continuity across resets. A model observed at prompt start or report receipt MUST NOT be claimed as the model responsible for a whole cumulative session cost.

The end-of-turn trigger remains unchanged: a consumed prompt response can report a non-success stop reason; an exception or abandoned stream is not a response. Intermediate usage observations MAY remain observable as ephemeral events in those cases, but do not establish successful completion or durable accounting. No tariff, credit policy or inferred cost delta is specified here.

A.10 Voice Channel

VoiceChannel
├── type: VOICE
├── category: TRANSPORT
├── direction: BIDIRECTIONAL
├── media_types: [AUDIO]
├── requires:
│   ├── stt: STTProvider
│   ├── tts: TTSProvider
│   ├── backend: VoiceBackend
│   └── audio_pipeline: AudioPipelineConfig
├── configuration:
│   ├── streaming: bool (default true)
│   ├── interruption: InterruptionConfig | null   # Section 12.6 — the current surface
│   ├── enable_barge_in: bool (default true)      # legacy; superseded by interruption
│   └── barge_in_threshold_ms: int (default 200)  # legacy; maps to allow_during_first_ms
│                                                 # (IMMEDIATE strategy; used only when
│                                                 # no InterruptionConfig is given)
├── streaming_delivery:
│   ├── supports_streaming_delivery: bool
│   │   # True when TTS supports streaming input AND backend is configured
│   └── deliver_stream(text_stream, event, binding, context) → ChannelOutput
│       # Pipes text stream → sentence_splitter → synthesize_stream_input → outbound → transport
│       # Fires AFTER_TTS hook after stream completes
├── session_management:
│   ├── bind_session(session, room_id, binding)
│   └── unbind_session(session)
└── delivery:
    ├── Standard path:
    │   ├── AudioContent → [postprocessors] → [recorder] → [resampler] → transport → [AEC ref †]
    │   ├── TextContent → TTS → AudioChunk stream → outbound pipeline → transport → [AEC ref †]
    │   └── Other → error
    ├── Streaming AI → TTS path (framework-native):
    │   └── Framework pipes AIChannel.response_stream → deliver_stream()
    │       → sentence_splitter → synthesize_stream_input() → outbound pipeline → transport
    └── † AEC ref fed at transport level (local hw) or pipeline level (network) — see 12.3.4

A.11 Realtime Voice Channel

RealtimeVoiceChannel
├── type: REALTIME_VOICE
├── category: TRANSPORT
├── direction: BIDIRECTIONAL
├── media_types: [AUDIO]
├── requires:
│   ├── provider: RealtimeVoiceProvider
│   ├── transport: RealtimeAudioTransport
│   ├── reasoning_backend: ReasoningBackend | null  # OPTIONAL: serves integrator-side reasoning delegation (Section 12.4.1)
│   └── audio_pipeline: AudioPipelineConfig | null  # OPTIONAL in speech-to-speech mode
├── configuration:
│   ├── system_prompt: string | null
│   ├── voice: string | null
│   ├── tools: list<ToolDefinition>
│   ├── temperature: float | null
│   ├── input_sample_rate: int
│   ├── output_sample_rate: int
│   ├── reasoning_timeout_s: float              # Bound on one delegation's run (Section 12.4.1)
│   └── emit_transcription_events: bool
├── session_management:
│   ├── start_session(room_id, participant_id, connection, metadata, organization_id)
│   └── end_session(session)
├── tool_handling:
│   ├── async handler function (priority)
│   └── ON_TOOL_CALL hook (fallback)
└── delegation_handling:                            # Section 12.4.1
    ├── hosted backend → its function calls go through tool_handling
    └── integrator backend → reasoning_backend, outputs returned via submit_delegation_output

A.12 Video Channel

VideoChannel
├── channel_type: VIDEO
├── category: TRANSPORT
├── direction: BIDIRECTIONAL
├── requires:
│   ├── backend: VideoBackend
│   ├── vision: VisionProvider | null     # OPTIONAL
│   └── vision_interval_ms: int           # default 2000
├── session_lifecycle:
│   ├── connect_video(room_id, participant_id, channel_id)
│   ├── bind_session(session, room_id, binding)
│   ├── unbind_session(session)
│   └── disconnect_video(session)
├── hooks:
│   ├── ON_VIDEO_SESSION_STARTED — SessionStartedEvent
│   ├── ON_VIDEO_SESSION_ENDED — SessionStartedEvent (same type as STARTED)
│   ├── ON_VISION_RESULT, ON_VIDEO_DETECTION
│   └── BEFORE_BRIDGE_VIDEO (Section 12.8.11)
├── framework_events:
│   ├── video_session_started
│   ├── video_session_ended
│   └── video_vision_result (description, labels, confidence, text, faces)
├── ai_integration:
│   └── vision results reach the AI turns as a `<vision>` block
│       (Section 12.8.7); setup_video_vision() is deprecated, wires nothing
└── backends:
    ├── LocalVideoBackend — OpenCV webcam capture (dev/testing)
    ├── ScreenCaptureBackend — mss screen capture (screen sharing, monitoring)
    ├── MockVideoBackend — Unit testing with call tracking
    └── optional (MAY): WebRTC (FastRTC), RTP, SIP video, WebSocket backends

A.13 Telegram Channel

TelegramChannel
├── type: TELEGRAM
├── category: TRANSPORT
├── direction: BIDIRECTIONAL
├── media_types: [TEXT, RICH, MEDIA, LOCATION]
├── capabilities:
│   ├── max_length: 4096
│   ├── supports_rich_text: true (HTML subset)
│   ├── supports_buttons: true (inline keyboards)
│   ├── supports_edit: true
│   ├── supports_delete: true
│   └── supports_typing: true
├── provider: TelegramBotProvider
│   ├── bot_token: string (secret)
│   ├── webhook_url: string | null
│   ├── parse_webhook(request) → InboundMessage
│   ├── verify_signature(request) → bool
│   └── send(chat_id, content, options) → DeliveryResult
└── delivery: sends via Telegram Bot API (sendMessage, sendPhoto, etc.)

A.14 CLI Channel

CLIChannel
├── type: CLI
├── category: TRANSPORT
├── direction: BIDIRECTIONAL
├── media_types: [TEXT]
├── capabilities:
│   ├── max_length: null (unlimited)
│   └── supports_rich_text: false
└── purpose: Development and testing via terminal stdin/stdout

---

A.15 Conference Channel

ConferenceChannel
├── channel_type: CONFERENCE
├── category: TRANSPORT
├── direction: BIDIRECTIONAL
├── requires:
│   ├── backend: ConferenceBackend
│   ├── stt: STTProvider | null           # Per-track transcription
│   ├── tts: TTSProvider | null           # AI voice via bot track
│   ├── realtime: ConferenceRealtimeConfig | null  # OPTIONAL — speech-to-speech; excludes tts
│   ├── pipeline: AudioPipelineConfig | null       # Lane stages; null = default VAD and format contract
│   ├── vision: VisionProvider | null     # Planned — video/screen tracks
│   ├── interruption: ConferenceInterruptionConfig
│   ├── recording: ConferenceRecordingConfig | null
│   ├── recorder: MediaRecorder | null    # Required with recording, refused without it
│   ├── bot_identity: string              # Bot's display identity
│   ├── bot_grants: ConferenceGrants | null  # Explicit grants for join_as_bot(); null = derived
│   ├── default_grants: ConferenceGrants  # Grants minted for participants
│   ├── e2ee: bool                        # default false — requires E2EE cap
│   ├── close_room_on_detach: bool        # default false
│   └── speak_text_events: bool           # default false
├── lifecycle:
│   ├── attach → ensure_room() [+ lazy join_as_bot(…, bot_grants)]
│   ├── on_participant_joined → Participant record + hook (bot excluded)
│   ├── on_track_published → subscribe_track() if consumed → pipeline lane
│   ├── on_track_unpublished → lane teardown + unsubscribe_track()
│   └── detach → leave() [+ close_room() if close_room_on_detach]
├── hooks:
│   ├── ON_CONFERENCE_PARTICIPANT_JOINED / LEFT
│   ├── ON_CONFERENCE_TRACK_PUBLISHED / UNPUBLISHED
│   ├── ON_CONFERENCE_TRACK_MUTED / UNMUTED
│   ├── ON_ACTIVE_SPEAKER_CHANGED
│   ├── ON_CONNECTION_QUALITY_CHANGED
│   ├── ON_SCREEN_SHARE_STARTED / STOPPED
│   └── ON_TRANSCRIPTION — per track, participant-attributed
├── framework_events:
│   ├── conference_started / conference_ended
│   ├── conference_participant_joined / conference_participant_left
│   └── conference_bot_grants_changed
└── backends:
    ├── MockConferenceBackend — unit testing with scripted event sequences
    └── SFU providers (e.g., LiveKit)

Appendix B: Complete Event Flow Examples

B.1 Customer (SMS) + AI — Basic Conversation

1. Customer sends SMS "Bonjour" to +15559876543
   │
   ▼
2. Twilio webhook → parse_webhook() → InboundMessage
   {channel_id: "sms_main", sender_id: "+15551234567",
    content: TextContent{body: "Bonjour"}}
   │
   ▼
3. process_inbound(message)
   │
   ├── Route: no active room for +15551234567 → create new room
   │   └── Fire ON_ROOM_CREATED hook → attach AI channel
   │
   ├── Identity: resolve(message from "+15551234567" on SMS, context) → IDENTIFIED (Jean Tremblay)
   │
   ├── Create RoomEvent {
   │     type: MESSAGE, content: TextContent{body: "Bonjour"},
   │     source: {channel_id: "sms_main", participant_id: "p_jean"},
   │     index: 0, chain_depth: 0
   │   }
   │
   ├── BEFORE_BROADCAST hooks → allow
   │
   ├── Store event (index=0, status=DELIVERED)
   │
   ├── Broadcast to channels:
   │   └── AI channel.on_event()
   │       ├── Build history: [{role: "user", text: "Bonjour"}]
   │       ├── Target: SMS capabilities (max 1600 chars, text only)
   │       ├── Run the tool loop over the provider's structured stream (§6.4)
   │       └── Return ChannelOutput{response_stream}
   │             SMS does not stream: the framework reads the stream, and
   │             its text becomes RoomEvent{content: TextContent{body: "Bonjour Jean! ..."}}
   │
   ├── Reentry: AI response event
   │   ├── chain_depth = 1 (< max 5)
   │   ├── Store event (index=1)
   │   ├── Broadcast:
   │   │   └── SMS channel.deliver() → Twilio sends SMS to +15551234567
   │   └── No further responses
   │
   └── AFTER_BROADCAST hooks → audit log

B.2 Sensitivity Scanning — Block and Inject

1. Customer on SMS sends: "Mon NAS est 123-456-789"
   │
   ▼
2. process_inbound → RoomEvent {
     content: TextContent{body: "Mon NAS est 123-456-789"},
     source: {channel_id: "sms_customer"}, index: 5
   }
   │
   ▼
3. BEFORE_BROADCAST sync hooks:
   │
   ├── [priority=0] sensitivity_scanner:
   │   ├── Detects SIN pattern: 123-456-789
   │   └── Returns HookResult.block(
   │         reason: "SIN detected",
   │         injected_events: [
   │           InjectedEvent{
   │             target_channel_ids: ["sms_customer"],
   │             event: RoomEvent{content: TextContent{body: "Message blocked. Do not send SIN by SMS."}}
   │           },
   │           InjectedEvent{
   │             target_channel_ids: ["ws_advisor"],
   │             event: RoomEvent{content: TextContent{body: "Client attempted to send SIN. Blocked."}}
   │           }
   │         ],
   │         observations: [
   │           Observation{category: "compliance_violation", content: "SIN sent by SMS", metadata: {pattern: "SIN"}}
   │         ]
   │       )
   │
   ▼
4. Event stored: status=BLOCKED, blocked_by="sensitivity_scanner"
   │
   ▼
5. Injected events delivered:
   ├── SMS to customer: "Message blocked..."
   └── WebSocket to advisor: "Client attempted..."
   │
   ▼
6. Observation persisted: compliance_violation

B.3 Ambiguous Identity — Shared Family Phone

1. SMS from +15551234567: "I need to check my account"
   │
   ▼
2. Identity resolution:
   resolver.resolve(message from "+15551234567" on SMS, context) → AMBIGUOUS
   candidates: [
     Identity{name: "Jean Tremblay", id: "id_jean"},
     Identity{name: "Marie Tremblay", id: "id_marie"},
     Identity{name: "Pierre Tremblay", id: "id_pierre"}
   ]
   │
   ▼
3. Fire ON_IDENTITY_AMBIGUOUS hook:
   Hook returns: pending(candidates=[id_jean, id_marie, id_pierre])
   │
   ▼
4. Create Participant {
     identification: PENDING,
     candidates: ["id_jean", "id_marie", "id_pierre"],
     display_name: "+15551234567"  (fallback)
   }
   │
   ▼
5. Event processed normally with pending participant.
   AI responds: "Welcome! Could you please tell me your name?"
   │
   ▼
6. Later: Advisor resolves via REST API:
   POST /rooms/{id}/participants/{pid}/resolve
   {identity_id: "id_marie"}
   │
   ▼
7. Participant updated: identification=IDENTIFIED, identity_id="id_marie"
   │
   ▼
8. PARTICIPANT_IDENTIFIED event added to timeline

B.4 AI ↔ AI Multi-Agent with Chain Depth

This is the AGENT_CHAIN policy (§19.3.1), which is the default: an agent's output solicits the other agents, and only the chain-depth limit ends the exchange. A room of independent agents — each answering the human, none meant to answer the others — selects ADDRESSED_ONLY instead.

1. Human sends message → Room with analyst_ai + writer_ai
   chain_depth = 0
   │
   ▼
2. Broadcast → analyst_ai.on_event()
   Returns: "Based on analysis, key findings are: ..."
   chain_depth = 1
   │
   ▼
3. Reentry → broadcast analyst response
   writer_ai.on_event()
   Returns: "Here is the report based on the analysis: ..."
   chain_depth = 2
   │
   ▼
4. Reentry → broadcast writer response
   analyst_ai.on_event()
   Returns: "Good report. One correction: ..."
   chain_depth = 3
   │
   ... continues until chain_depth reaches max_chain_depth ...
   │
   ▼
N. The next response would have chain_depth = 5 (== max_chain_depth)
   The other agent is NOT asked: no model call, no tool
   Record: status=BLOCKED, blocked_by="event_chain_depth_limit",
           source=that agent, empty text, chain_depth=5
   Framework event: chain_depth_exceeded

A streamed response chains the same way: each of its text segments is an event the other agent answers as it is committed, and the chain stops at the same depth.

B.5 Dynamic Channel Management — Advisor Joins

Timeline of a room with SMS customer + AI:

[0] customer→    "Bonjour"                          (chain_depth=0)
[1] ai→          "Bonjour! How can I help?"          (chain_depth=1)
[2] customer→    "I need help with my mortgage"      (chain_depth=0)
[3] ai→          "I can help with mortgage info..."   (chain_depth=1)

── Advisor joins ──
[4] CHANNEL_ATTACHED  ws_advisor (access=READ_WRITE, visibility=all)

── Integrator mutes AI, changes to assistant mode ──
[5] CHANNEL_MUTED     ai_support
[6] CHANNEL_UPDATED   ai_support (visibility → "ws_advisor" only)
[7] CHANNEL_UNMUTED   ai_support

── Now AI whispers only to advisor ──
[8] customer→    "What rate can I get?"               (chain_depth=0)
[9] ai→          "Suggest offering 4.5% based on..."  (visibility: ws_advisor only)
                  ↑ customer does NOT see this
[10] advisor→    "We can offer you 4.5% fixed."       (visibility: all)
                  ↑ customer sees this

── Advisor lets AI respond directly again ──
[11] CHANNEL_UPDATED  ai_support (visibility → "all")

[12] customer→   "What documents do I need?"          (chain_depth=0)
[13] ai→         "You'll need: 1. ID 2. Income..."    (visibility: all)
                  ↑ customer sees this directly

B.6 Message Edit — Cross-Channel with Fallback

1. WhatsApp user edits message [index=3] from "I need 5000$" to "I need 50000$"
   │
   ▼
2. WhatsApp webhook → parse_webhook() → InboundMessage
   {channel_id: "wa_customer", sender_id: "+15551234567",
    content: EditContent{
      target_event_id: "evt_003",
      new_content: TextContent{body: "I need 50000$"},
      edit_source: "sender"
    },
    event_type: EDIT}
   │
   ▼
3. process_inbound(message)
   │
   ├── Validate: evt_003 exists in room → ✓
   ├── Validate: sender is original author → ✓
   │
   ├── BEFORE_BROADCAST hooks → allow
   │   (a block leaves evt_003 unmutated and fires no ON_EVENT_UPDATED)
   │
   ├── Update original event via update_event():
   │   evt_003.content = TextContent{body: "I need 50000$"}
   │   evt_003.metadata.edited = true
   │
   ├── Store EDIT event {
   │     type: EDIT, content: EditContent{...},
   │     source: {channel_id: "wa_customer"},
   │     index: 7
   │   }
   │
   ├── Broadcast to channels:
   │   │
   │   ├── WebSocket (supports_edit: true)
   │   │   └── deliver() native edit → client updates message in-place
   │   │
   │   ├── SMS (supports_edit: false)
   │   │   └── Transcode → TextContent{body: "Correction: I need 50000$"}
   │   │       └── deliver() → SMS sent as new message
   │   │
   │   └── AI channel.on_event()
   │       └── Updates conversation history with corrected text
   │
   ├── ON_EVENT_UPDATED hooks → evt_003 as edited
   │   (async, once the room lock is released)
   │
   └── AFTER_BROADCAST hooks → audit log

B.7 Message Delete — Cross-Channel with Fallback

1. WhatsApp user deletes message [index=5]
   │
   ▼
2. WhatsApp webhook → parse_webhook() → InboundMessage
   {channel_id: "wa_customer", sender_id: "+15551234567",
    content: DeleteContent{
      target_event_id: "evt_005",
      delete_type: SENDER,
      reason: null
    },
    event_type: DELETE}
   │
   ▼
3. process_inbound(message)
   │
   ├── Validate: evt_005 exists in room → ✓
   ├── Validate: sender is original author (SENDER type) → ✓
   │
   ├── BEFORE_BROADCAST hooks → allow
   │   (a block leaves evt_005 unmutated and fires no ON_EVENT_DELETED)
   │
   ├── Mark original event as deleted via update_event():
   │   evt_005.metadata.deleted = true
   │
   ├── Store DELETE event {
   │     type: DELETE, content: DeleteContent{...},
   │     source: {channel_id: "wa_customer"},
   │     index: 8
   │   }
   │
   ├── Broadcast to channels:
   │   │
   │   ├── WebSocket (supports_delete: true)
   │   │   └── deliver() native delete → client removes message
   │   │
   │   ├── SMS (supports_delete: false)
   │   │   └── Transcode → TextContent{body: "[Message deleted]"}
   │   │       └── deliver() → SMS sent as new message
   │   │
   │   └── AI channel.on_event()
   │       └── Updates conversation history (removes or marks deleted)
   │
   ├── ON_EVENT_DELETED hooks → evt_005 as deleted
   │   (async, once the room lock is released)
   │
   └── AFTER_BROADCAST hooks → audit log

B.8 SIP Call → Unified process_inbound with Protocol Traces

1. Incoming SIP INVITE from +15551234567
   │
   ├── SIPVoiceBackend auto-accepts, negotiates G.722 (16 kHz)
   ├── Creates VoiceSession {id: "sess-42", room_id: "room-abc",
   │     participant_id: "+15551234567", metadata: {caller, callee, ...}}
   │
   ├── Backend emits ProtocolTrace:
   │     {direction: "inbound", protocol: "sip",
   │      summary: "INVITE from +1555... to +1666...",
   │      raw: <serialized SIP INVITE>, session_id: "sess-42",
   │      room_id: "room-abc"}
   │
   ├── Backend emits ProtocolTrace:
   │     {direction: "outbound", protocol: "sip",
   │      summary: "200 OK (codec=16000Hz)",
   │      raw: <SDP answer>, session_id: "sess-42",
   │      room_id: "room-abc"}
   │
   │   Note: Both traces are BUFFERED — room does not exist yet
   │
   ▼
2. on_call callback fires
   │
   ├── parse_voice_session(session, channel_id="realtime-voice")
   │   → InboundMessage {
   │       channel_id: "realtime-voice",
   │       sender_id: "+15551234567",
   │       event_type: SYSTEM,
   │       content: SystemContent{body: "Voice session started",
   │                 code: "session_started",
   │                 data: {session_id: "...", channel_id: "realtime-voice",
   │                        caller: "+1555..."}},
   │       session: <VoiceSession>
   │     }   # process_inbound(message, room_id="room-abc")
   │
   ▼
3. kit.process_inbound(message, room_id="room-abc")
   │
   ├── Resolve channel → RealtimeVoiceChannel "realtime-voice"
   ├── Route to room → create room "room-abc"
   ├── Attach channel → triggers _flush_pending_traces("room-abc")
   │   └── BUFFERED TRACES replayed as ON_PROTOCOL_TRACE hooks:
   │       ├── Hook receives: INVITE trace (with room context)
   │       └── Hook receives: 200 OK trace (with room context)
   │
   ├── channel.handle_inbound() → RoomEvent
   │     {type: MESSAGE, source: {channel_id: "realtime-voice",
   │      provider: "SIPRealtimeTransport"}, content: SystemContent{...}}
   │
   ├── BEFORE_BROADCAST hooks:
   │   └── gate_incoming() checks caller against blocklist → ALLOW
   │
   ├── Store event (status=DELIVERED)
   ├── channel.connect_session(session, room_id, binding)
   │   └── Starts realtime AI session (Gemini Live, OpenAI Realtime, etc.)
   │
   ├── Broadcast (no other channels need delivery)
   └── AFTER_BROADCAST hooks → log with provider="SIPRealtimeTransport"
   │
   ▼
4. Audio flows bidirectionally
   │
   ├── Caller speaks → Transport → Provider → AI responds → Transport → Caller
   ├── ON_TRANSCRIPTION hooks fire for user and AI speech
   └── Events emitted with source.provider = "SIPRealtimeTransport"
   │
   ▼
5. Remote hangup (SIP BYE)
   │
   ├── Backend emits ProtocolTrace:
   │     {direction: "inbound", protocol: "sip",
   │      summary: "BYE from +1555...",
   │      raw: <serialized SIP BYE>, session_id: "sess-42"}
   │
   ├── ON_PROTOCOL_TRACE hook fires immediately (room exists)
   │
   ├── on_call_disconnected callback:
   │   ├── realtime.end_session(session)
   │   └── kit.close_room("room-abc")
   │
   └── Room closed, resources released

B.9 Multi-Agent Pipeline — Triage → Specialist → Resolution

1. Customer sends SMS "My internet is down since yesterday"
   │
   ▼
2. process_inbound(message)
   │
   ├── Route: create new room
   ├── Attach channels:
   │   ├── "sms_main" (TRANSPORT, customer)
   │   ├── "triage" (INTELLIGENCE, Agent: role="Triage")
   │   ├── "network-specialist" (INTELLIGENCE, Agent: role="Network Engineer")
   │   └── "closer" (INTELLIGENCE, Agent: role="Resolution")
   │
   ├── ConversationState initialized:
   │     {phase: "triage", active_agent_id: "triage"}
   │
   ├── ConversationRouter installed as BEFORE_BROADCAST hook:
   │     rules: [
   │       {agent_id: "triage", conditions: {phases: {"triage"}}},
   │       {agent_id: "network-specialist", conditions: {phases: {"specialist"}}},
   │       {agent_id: "closer", conditions: {phases: {"closing"}}},
   │     ]
   │
   ├── BEFORE_BROADCAST: router stamps event metadata → target: "triage"
   ├── Broadcast: only "triage" agent processes event
   │
   └── Triage agent responds:
       ├── Generates: "I'll connect you with our network specialist."
       ├── Calls tool: handoff_conversation(target_agent_id="network-specialist",
       │     reason="network outage", summary="Customer reports internet down since yesterday")
       │
       ├── HandoffHandler processes:
       │   ├── Validates: "triage" → "specialist" transition is allowed
       │   ├── Updates ConversationState:
       │   │     {phase: "specialist", active_agent_id: "network-specialist"}
       │   ├── Fires ON_HANDOFF hook
       │   └── Fires ON_PHASE_TRANSITION hook
       │
       └── Response events broadcast to SMS customer
   │
   ▼
3. Customer sends "Yes, the router lights are blinking red"
   │
   ├── BEFORE_BROADCAST: router evaluates
   │   ├── ConversationState.phase = "specialist"
   │   ├── Route → "network-specialist"
   │
   ├── Network specialist agent processes:
   │   ├── Has handoff context: original issue + triage summary
   │   ├── Generates diagnostic steps
   │   ├── Delegates background task:
   │   │     kit.delegate(room_id, "diagnostics-bot",
   │   │       task="Check network status for customer area",
   │   │       notify="network-specialist")
   │   │     (the kit's delivery strategy: WaitForIdle(buffer=5.0))
   │   │
   │   ├── ON_TASK_DELEGATED hook fires
   │   └── Responds: "Let me check your area's network status..."
   │
   ▼
4. Delegated task completes
   │
   ├── ON_TASK_COMPLETED hook fires
   ├── BEFORE_DELIVER hook fires (can block or modify)
   ├── WaitForIdle strategy waits for conversation pause
   ├── Result injected: "Network status: area outage confirmed, ETA 2 hours"
   │
   ▼
5. Specialist resolves and hands off to closer
   │
   ├── Calls: handoff_conversation(target_agent_id="closer",
   │     reason="resolved", summary="Area outage confirmed, ETA communicated")
   │
   ├── ConversationState:
   │     {phase: "closing", active_agent_id: "closer"}
   │
   └── Closer agent summarizes resolution, asks for satisfaction

---

End of RFC