fix: bound the output transport drain so EndFrame can't be stranded - #5217
Open
a6kme wants to merge 2 commits into
Open
fix: bound the output transport drain so EndFrame can't be stranded#5217a6kme wants to merge 2 commits into
a6kme wants to merge 2 commits into
Conversation
Transport writes have no timeout of their own. When the remote peer stops reading — a half-open socket, or a telephony call already torn down on the provider's side — the media sender's audio task parks inside its write and never returns. `MediaSender.stop()` awaited that task unconditionally, and `BaseOutputTransport.process_frame()` pushes the EndFrame downstream only after `stop()` returns. So the EndFrame was stranded inside the transport, never reached the sink, and `PipelineWorker._wait_for_pipeline_end()` — which applies a timeout on the CancelFrame path but not the EndFrame one — waited on it forever. A subsequent cancel() cannot rescue it either, because the CancelFrame is queued onto the same `_push_queue` that is already blocked. Bound the wait with the new `TransportParams.audio_out_drain_timeout_secs` (default 5s). It bounds how long the audio task may make *no progress at all*, not the total drain: queued audio is played out in real time and is legitimately slow, so a flat cap would truncate speech. The audio task stamps a monotonic timestamp each iteration and shutdown polls it, cancelling only once the task has genuinely stalled. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Rename the changelog fragment to 5217, drop the leading underscore from DRAIN_POLL_SECS to match the neighbouring module constants, keep it next to BOT_VAD_STOP_FALLBACK_SECS instead of splitting that pair, and trim the comments and docstrings down to what the code doesn't already say. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Codecov Report❌ Patch coverage is
🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #5216
Problem
BaseOutputTransport.process_frame()pushes anEndFramedownstream only afterawait self.stop(frame)returns, andMediaSender.stop()awaits the audio task unconditionally. Transport writes have no timeout of their own, so a peer that has stopped reading parks that task inside its write forever. TheEndFrameis then stranded inside the transport, never reaches the sink, andPipelineWorker._wait_for_pipeline_end()— which bounds theCancelFramepath but not theEndFrameone — waits on it indefinitely. A follow-upcancel()can't help either: itsCancelFramegoes onto the same_push_queuethat is already blocked.Full analysis, a minimal repro, and parked stack traces from a real socket are in #ISSUE_NUMBER.
Fix
Bound the drain in
MediaSender.stop()with a newTransportParams.audio_out_drain_timeout_secs(default 5s).The bound is on stall, not duration. A first attempt using a flat wall-clock timeout was rejected because it truncates legitimate speech: queued audio is played out in real time, so a healthy drain of a few seconds of audio takes a few seconds. In a websocket repro a flat 5s cap cut a healthy client's output from 687,576 to 484,348 bytes — a clipped goodbye.
Instead the audio task stamps
_audio_progress_timeon each loop iteration, and shutdown polls it, cancelling only once the task has made no progress at all for the configured window. Long playout is unaffected; a wedged write is cut loose.The clock task keeps a flat bound — it returns as soon as it pops the
EndFrame, so there is nothing to stall on.Tests
Two regression tests in
tests/test_base_output_transport.py:test_end_frame_proceeds_when_audio_write_never_returns— fails onmainwithTimeoutError, passes here.test_slow_but_progressing_drain_is_not_cancelled— passes both ways by design; it's the guard against the flat-timeout mistake above.Also verified end-to-end against a real uvicorn server and a real RFC6455 client that completes the handshake then stops reading:
ruff check/ruff format --checkclean.tests/test_base_output_transport.py,tests/test_pipeline.py,tests/test_websocket_transport.pypass.Notes
EndFrametimeout in_wait_for_pipeline_end()or letCancelFramebypass the blocked queue — both still worth doing as defence in depth, and both are out of scope here.