feat(voice): Add OpenAI Realtime SIP channel (POC) - #102
Draft
xinghaohuang91 wants to merge 2 commits into
Draft
Conversation
Adds OpenAIRealtimeSipChannel, bridging Twilio calls to OpenAI's Realtime API over SIP trunking. Twilio forwards accepted calls directly to OpenAI at the SIP/SRTP level, so audio never passes through TAC — the channel only handles the realtime.call.incoming webhook (accept/reject) and, optionally, a sideband control WebSocket for transcript capture and tool calling. Verified against a real Twilio SIP Trunk + OpenAI project: calls connect and complete successfully, transcript capture and tool calling both work end-to-end. This is a POC — no automated test suite yet, only manual and unit-level smoke testing of the event-dispatch logic. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds OpenAIRealtimeMediaStreamsChannel as a second S2S option alongside OpenAIRealtimeSipChannel: Twilio streams call audio to our own WebSocket via <Connect><Stream>, and this channel bridges it to/from OpenAI's Realtime WebSocket directly, so it owns the audio path for the life of the call (unlike the SIP channel, which never touches audio). - OpenAIRealtimeMediaStreamsChannel: audio bridge with precise barge-in (truncates the model's reply at the exact ms played, clears Twilio's buffer), inline transcript capture and tool calling (no sideband connection needed — already holding one audio connection open) - Call state lives on ConversationSession.metadata; a small RealtimeWebSocketManager tracks just the Twilio/OpenAI socket pair - Session config (model/voice/instructions/tools) is supplied by reusing TAC.on_message_ready (called with an empty user_message and no memory_response, returning a JSON-encoded session dict) rather than a parallel callback — see S2S_POC.md for the reasoning - TACTool.to_realtime_format() promoted to tools/base.py, shared by both Realtime channels instead of duplicated - OpenAIRealtimeMediaStreamsServer: minimal standalone FastAPI host (TwiML + Media Stream WebSocket) - S2S_POC.md Section 2: architecture, setup, trade-offs vs. Section 1 Verified end-to-end against a real Twilio number + OpenAI project: calls connect, barge-in truncates correctly mid-reply, tool calls execute and the model continues with the result, and transcript capture matches what was said on both sides. Known limitation: on_message_ready is shared with text channels on the same TAC instance — an app combining this channel with e.g. an SMS channel needs its own branching logic (on ConversationSession.channel) in that one callback, since there's no built-in way to distinguish "build a session config" from "build a reply" calls. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
POC for speech-to-speech (S2S) voice support in TAC:
OpenAIRealtimeSipChannelbridges Twilio phone calls to OpenAI's Realtime API over SIP trunking, as an
alternative to the existing ConversationRelay-based
VoiceChannel.Twilio's Elastic SIP Trunking forwards accepted calls directly to OpenAI at
the SIP/SRTP level — audio never passes through TAC. The channel only
handles:
realtime.call.incomingwebhook (decide whether/how to accept a call,via a new
on_call_incomingcallback)transcript capture and tool calling, opened only when a tool is registered
or transcription is requested
New pieces:
src/tac/channels/openai_realtime_sip/— channel, client (RESTaccept/reject/hangup/refer + control WebSocket), config, models
src/tac/server/openai_realtime_sip_server.py— minimal standaloneFastAPI server exposing just the one webhook route (deliberately separate
from
TACFastAPIServer, since this channel shares no routes/signaturescheme with existing voice/messaging channels)
getting_started/examples/features/openai_realtime_sip_voice.py— runnableexample with a registered tool and transcript printing
S2S_POC.md— architecture + setup + trade-offs write-up (first of aplanned 3-section doc; this PR covers section 1 only)
tac[openai-realtime-sip]How to test
uv sync --extra server --extra openai-realtime-sipS2S_POC.md/ the example's docstring (OpenAIwebhook + project ID, Twilio Elastic SIP Trunk with an Origination URI
pointing at
sip:$PROJECT_ID@sip.api.openai.com;transport=tls)uv run python getting_started/examples/features/openai_realtime_sip_voice.pybehind a public tunnel (e.g. ngrok), then call the Twilio number
Verified end-to-end against a real Twilio SIP Trunk + OpenAI project: calls
connect and complete, transcript capture (both caller and model speech) and
tool calling both confirmed working.
Type of Change
Checklist
unit-level smoke tests of the event-dispatch logic (
_handle_control_event,_handle_function_call) so far; notests/test_openai_realtime_*.pyyet. Flagging for follow-up before this leaves POC status.
S2S_POC.md, example docstring)SDK Parity
This is the Python SDK.
exploring a new architecture, not yet a stable API to port.