Skip to content

fix: bound the output transport drain so EndFrame can't be stranded - #5217

Open
a6kme wants to merge 2 commits into
pipecat-ai:mainfrom
a6kme:fix/bound-output-transport-drain
Open

fix: bound the output transport drain so EndFrame can't be stranded#5217
a6kme wants to merge 2 commits into
pipecat-ai:mainfrom
a6kme:fix/bound-output-transport-drain

Conversation

@a6kme

@a6kme a6kme commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Fixes #5216

Problem

BaseOutputTransport.process_frame() pushes an EndFrame downstream only after await self.stop(frame) returns, and MediaSender.stop() awaits the audio task unconditionally. Transport writes have no timeout of their own, so a peer that has stopped reading parks that task inside its write forever. The EndFrame is then stranded inside the transport, never reaches the sink, and PipelineWorker._wait_for_pipeline_end() — which bounds the CancelFrame path but not the EndFrame one — waits on it indefinitely. A follow-up cancel() can't help either: its CancelFrame goes onto the same _push_queue that is already blocked.

Full analysis, a minimal repro, and parked stack traces from a real socket are in #ISSUE_NUMBER.

Fix

Bound the drain in MediaSender.stop() with a new TransportParams.audio_out_drain_timeout_secs (default 5s).

The bound is on stall, not duration. A first attempt using a flat wall-clock timeout was rejected because it truncates legitimate speech: queued audio is played out in real time, so a healthy drain of a few seconds of audio takes a few seconds. In a websocket repro a flat 5s cap cut a healthy client's output from 687,576 to 484,348 bytes — a clipped goodbye.

Instead the audio task stamps _audio_progress_time on each loop iteration, and shutdown polls it, cancelling only once the task has made no progress at all for the configured window. Long playout is unaffected; a wedged write is cut loose.

The clock task keeps a flat bound — it returns as soon as it pops the EndFrame, so there is nothing to stall on.

Tests

Two regression tests in tests/test_base_output_transport.py:

  • test_end_frame_proceeds_when_audio_write_never_returns — fails on main with TimeoutError, passes here.
  • test_slow_but_progressing_drain_is_not_cancelled — passes both ways by design; it's the guard against the flat-timeout mistake above.

Also verified end-to-end against a real uvicorn server and a real RFC6455 client that completes the handshake then stops reading:

before after
stalled peer hangs indefinitely clean shutdown in 6.8s
healthy peer clean, 687,576 bytes clean, 687,832 bytes — unchanged

ruff check / ruff format --check clean. tests/test_base_output_transport.py, tests/test_pipeline.py, tests/test_websocket_transport.py pass.

Notes

a6kme and others added 2 commits August 4, 2026 09:16
Transport writes have no timeout of their own. When the remote peer stops
reading — a half-open socket, or a telephony call already torn down on the
provider's side — the media sender's audio task parks inside its write and
never returns.

`MediaSender.stop()` awaited that task unconditionally, and
`BaseOutputTransport.process_frame()` pushes the EndFrame downstream only
after `stop()` returns. So the EndFrame was stranded inside the transport,
never reached the sink, and `PipelineWorker._wait_for_pipeline_end()` — which
applies a timeout on the CancelFrame path but not the EndFrame one — waited
on it forever. A subsequent cancel() cannot rescue it either, because the
CancelFrame is queued onto the same `_push_queue` that is already blocked.

Bound the wait with the new `TransportParams.audio_out_drain_timeout_secs`
(default 5s). It bounds how long the audio task may make *no progress at
all*, not the total drain: queued audio is played out in real time and is
legitimately slow, so a flat cap would truncate speech. The audio task stamps
a monotonic timestamp each iteration and shutdown polls it, cancelling only
once the task has genuinely stalled.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Rename the changelog fragment to 5217, drop the leading underscore from
DRAIN_POLL_SECS to match the neighbouring module constants, keep it next to
BOT_VAD_STOP_FALLBACK_SECS instead of splitting that pair, and trim the
comments and docstrings down to what the code doesn't already say.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@codecov

codecov Bot commented Aug 4, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 89.47368% with 2 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
src/pipecat/transports/base_output.py 88.88% 2 Missing ⚠️
Files with missing lines Coverage Δ
src/pipecat/transports/base_transport.py 95.65% <100.00%> (+0.09%) ⬆️
src/pipecat/transports/base_output.py 72.01% <88.88%> (+0.55%) ⬆️
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

EndFrame is stranded in BaseOutputTransport when the peer stops reading, hanging pipeline shutdown forever

1 participant