fix(whip): stop an unstamped buffer at the session bridge before qtmux sees it - #759
Draft
wagenet wants to merge 1 commit into
Draft
fix(whip): stop an unstamped buffer at the session bridge before qtmux sees it#759wagenet wants to merge 1 commit into
wagenet wants to merge 1 commit into
Conversation
…x sees it
A WHIP session pipeline is bridged into the flow's main pipeline through an
appsink/appsrc pair. The bridge rebased the PTS of buffers that had one and
forwarded the rest untouched. Downstream of that appsrc sits a recorder, and
qtmux cannot place an untimestamped buffer in a track:
ERROR Could not multiplex stream.
gst_qt_mux_add_buffer (): .../GstMP4Mux:rec_p2:mux: Buffer has no PTS.
The GST_FLOW_ERROR propagates out of the recorder and up through the mixer, so
one seat's stray buffer takes the whole flow down. Observed on a live rig: a
second participant joining killed the first.
Drop the buffer at the bridge instead. There is no correct stamp to invent:
"main pipeline running time now" is a different time base from pts + offset and
the two drift, so a synthesised stamp can land before its own predecessor, and
a muxer rejects backwards timestamps as hard as missing ones. A dropped frame
on a live source is the recoverable failure; keyframe_request already covers a
decoder that lost part of its GOP.
Drops are counted per session and warned on the first, then every hundredth, so
a publisher sending nothing usable stays visible without a line per frame.
Moving the bridge into gst::whip_bridge also fixes three smaller things in the
same code: the shared A/V offset is now taken from the first buffer that
carries a PTS rather than the first buffer of the session, DTS is shifted with
PTS instead of being left in the session's time base, and a sample with no caps
no longer panics.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bug
A WHIP session runs in its own pipeline and is bridged into the flow's main pipeline through an
appsink/appsrcpair. The bridge rebased the PTS of buffers that carried one and forwarded the rest untouched:Downstream of that appsrc sits a recorder, and
qtmuxcannot place an untimestamped buffer in a track:The resulting
GST_FLOW_ERRORpropagates out of the recorder and up through the mixer, so one seat's stray buffer takes the whole flow down. The same incident logged errors fromwhip_p2:appsrc_audio_0,rec_p2:audio_0_queue,vmix:queue_1,vmix:appsrc_overlayand others. Observed on a live rig: a second participant joining killed the first.The fix
Drop the buffer at the bridge. The
new_sampleclosure moves intogst::whip_bridge::SessionBridge, which owns the per-session timestamp offset and the drop count.Why drop rather than stamp from the pipeline clock. There is no correct value to invent. The main pipeline's running time now is a different time base from
pts + offset, and the two drift apart over a session, so a synthesised stamp can land before its own predecessor — and a muxer rejects backwards timestamps exactly as hard as missing ones. That is the same cascade by another route. A stamp that is merely a little wrong is worse still: it desynchronises audio from video for the rest of the session, silently. A dropped frame on a live source is the ordinary recoverable failure — the receiver already tolerates loss, and thekeyframe_requestmachinery already in this file recovers a decoder that lost part of its GOP.Not silent. Drops are counted per session and warned on the first, then every hundredth, so a publisher sending nothing usable stays visible without writing a line per frame.
Three smaller things in the same code, fixed by the move:
sample.caps().unwrap()no longer panics on a sample without caps.Blast radius — deliberately not in this PR
A recorder failing should degrade that seat, not kill the mixer, the program and every other participant. That is a larger piece of work than a bridge fix: the recorder needs its own bin with a bus handler that fails the seat and leaves the rest of the flow running, plus a way to surface a failed seat through the API and UI. Bundling it here would mix a two-line behavioural fix with a pipeline-topology change. Filing separately.
Tests
backend/src/gst/whip_bridge.rs— 7 unit tests driving a realappsrc ! appsinkpair, so the assertion is on what actually comes out of the sink, not on a returned enum:an_unstamped_buffer_never_crosses_the_bridge— three samples in, stamped/unstamped/stamped; only the two stamped ones arrive, rebased.an_unstamped_first_buffer_is_dropped_and_does_not_fix_the_offset— the other route in: no offset yet, so the restamping path is never reached; the buffer must still not cross, and must not poison the offset for the one that follows.a_missing_running_time_forwards_unadjusted_and_retries,dts_is_shifted_with_pts,both_streams_share_one_offset,a_negative_shift_clamps_to_zero,drops_are_logged_first_then_sparsely.The guard bites. Re-adding
push_sampleon the PTS-less path fails 2 of the 7 withleft: Some(None)— the unstamped buffer arriving at the sink. It is a real guard, not a demonstration.Uses only
appsrc/appsinkfromgstreamer1.0-plugins-base, already in.github/workflows/ci.yml. No element skip, nothing to add.What I ran
cargo test --package strom --features efp,nvidiawithSTROM_REQUIRE_GST_PLUGINS=1— 551 lib + 57 integration tests, all pass.cargo clippy --package strom --all-targets --features efp,nvidia -- -D warnings— clean.cargo fmt --all --check— clean.whipclientsinkpublishers: both sessions' A/V pads bridged, exactly one shared ts-offset per session, zero unstamped drops on healthy traffic, zero mux errors, neither publisher killed the other.What I could not run. I could not drive the Meeting Rig to PLAYING on
upstream/main— it needs #744 (builtin.audioenc), #749 (idle seats holding the pipeline out of PLAYING) and #750 (recorders with no data), all still open. Soqtmuxnever actually ran in my live sessions and I did not reproduce the cascade end to end. The unit test is the guard; the live run is a smoke test that the bridge still behaves on healthy traffic. The original occurrence is intermittent anyway, which is why the guard is a unit test rather than an integration test.Conflicts to expect
new_sampleclosure (watchdog store toactivity.touch()). Small mechanical conflict, whichever lands second.whip.rshunk is elsewhere.🤖 Generated with Claude Code