Skip to content

fix(whip): stop an unstamped buffer at the session bridge before qtmux sees it - #759

Draft
wagenet wants to merge 1 commit into
Eyevinn:mainfrom
wagenet:wagenet/whip-ptsless-buffer
Draft

fix(whip): stop an unstamped buffer at the session bridge before qtmux sees it#759
wagenet wants to merge 1 commit into
Eyevinn:mainfrom
wagenet:wagenet/whip-ptsless-buffer

Conversation

@wagenet

@wagenet wagenet commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

The bug

A WHIP session runs in its own pipeline and is bridged into the flow's main pipeline through an appsink/appsrc pair. The bridge rebased the PTS of buffers that carried one and forwarded the rest untouched:

if offset_ns != 0 {
    if let Some(pts_val) = pts { /* restamp, push */ }
    else { let _ = appsrc.push_sample(&sample); }   // forwarded UNSTAMPED
} else {
    let _ = appsrc.push_sample(&sample);            // also unstamped
}

Downstream of that appsrc sits a recorder, and qtmux cannot place an untimestamped buffer in a track:

ERROR Could not multiplex stream.
  gstqtmux.c(5927): gst_qt_mux_add_buffer ():
  /GstSplitMuxSink:rec_p2:splitmuxsink/GstMP4Mux:rec_p2:mux:
  Buffer has no PTS.

The resulting GST_FLOW_ERROR propagates out of the recorder and up through the mixer, so one seat's stray buffer takes the whole flow down. The same incident logged errors from whip_p2:appsrc_audio_0, rec_p2:audio_0_queue, vmix:queue_1, vmix:appsrc_overlay and others. Observed on a live rig: a second participant joining killed the first.

The fix

Drop the buffer at the bridge. The new_sample closure moves into gst::whip_bridge::SessionBridge, which owns the per-session timestamp offset and the drop count.

Why drop rather than stamp from the pipeline clock. There is no correct value to invent. The main pipeline's running time now is a different time base from pts + offset, and the two drift apart over a session, so a synthesised stamp can land before its own predecessor — and a muxer rejects backwards timestamps exactly as hard as missing ones. That is the same cascade by another route. A stamp that is merely a little wrong is worse still: it desynchronises audio from video for the rest of the session, silently. A dropped frame on a live source is the ordinary recoverable failure — the receiver already tolerates loss, and the keyframe_request machinery already in this file recovers a decoder that lost part of its GOP.

Not silent. Drops are counted per session and warned on the first, then every hundredth, so a publisher sending nothing usable stays visible without writing a line per frame.

Three smaller things in the same code, fixed by the move:

  • The shared A/V offset is now taken from the first buffer that carries a PTS, not the first buffer of the session.
  • DTS is shifted with PTS. It is normally unset here (WebRTC H.264 has no B-frames), but leaving a session-base DTS beside a main-base PTS is the muxer's other way to reject the buffer.
  • sample.caps().unwrap() no longer panics on a sample without caps.

Blast radius — deliberately not in this PR

A recorder failing should degrade that seat, not kill the mixer, the program and every other participant. That is a larger piece of work than a bridge fix: the recorder needs its own bin with a bus handler that fails the seat and leaves the rest of the flow running, plus a way to surface a failed seat through the API and UI. Bundling it here would mix a two-line behavioural fix with a pipeline-topology change. Filing separately.

Tests

backend/src/gst/whip_bridge.rs — 7 unit tests driving a real appsrc ! appsink pair, so the assertion is on what actually comes out of the sink, not on a returned enum:

  • an_unstamped_buffer_never_crosses_the_bridge — three samples in, stamped/unstamped/stamped; only the two stamped ones arrive, rebased.
  • an_unstamped_first_buffer_is_dropped_and_does_not_fix_the_offset — the other route in: no offset yet, so the restamping path is never reached; the buffer must still not cross, and must not poison the offset for the one that follows.
  • a_missing_running_time_forwards_unadjusted_and_retries, dts_is_shifted_with_pts, both_streams_share_one_offset, a_negative_shift_clamps_to_zero, drops_are_logged_first_then_sparsely.

The guard bites. Re-adding push_sample on the PTS-less path fails 2 of the 7 with left: Some(None) — the unstamped buffer arriving at the sink. It is a real guard, not a demonstration.

Uses only appsrc/appsink from gstreamer1.0-plugins-base, already in .github/workflows/ci.yml. No element skip, nothing to add.

What I ran

  • cargo test --package strom --features efp,nvidia with STROM_REQUIRE_GST_PLUGINS=1 — 551 lib + 57 integration tests, all pass.
  • cargo clippy --package strom --all-targets --features efp,nvidia -- -D warnings — clean. cargo fmt --all --check — clean.
  • Live rig, headless on macOS, two whipclientsink publishers: both sessions' A/V pads bridged, exactly one shared ts-offset per session, zero unstamped drops on healthy traffic, zero mux errors, neither publisher killed the other.

What I could not run. I could not drive the Meeting Rig to PLAYING on upstream/main — it needs #744 (builtin.audioenc), #749 (idle seats holding the pipeline out of PLAYING) and #750 (recorders with no data), all still open. So qtmux never actually ran in my live sessions and I did not reproduce the cascade end to end. The unit test is the guard; the live run is a smoke test that the bridge still behaves on healthy traffic. The original occurrence is intermittent anyway, which is why the guard is a unit test rather than an integration test.

Conflicts to expect

🤖 Generated with Claude Code

…x sees it

A WHIP session pipeline is bridged into the flow's main pipeline through an
appsink/appsrc pair. The bridge rebased the PTS of buffers that had one and
forwarded the rest untouched. Downstream of that appsrc sits a recorder, and
qtmux cannot place an untimestamped buffer in a track:

    ERROR Could not multiplex stream.
      gst_qt_mux_add_buffer (): .../GstMP4Mux:rec_p2:mux: Buffer has no PTS.

The GST_FLOW_ERROR propagates out of the recorder and up through the mixer, so
one seat's stray buffer takes the whole flow down. Observed on a live rig: a
second participant joining killed the first.

Drop the buffer at the bridge instead. There is no correct stamp to invent:
"main pipeline running time now" is a different time base from pts + offset and
the two drift, so a synthesised stamp can land before its own predecessor, and
a muxer rejects backwards timestamps as hard as missing ones. A dropped frame
on a live source is the recoverable failure; keyframe_request already covers a
decoder that lost part of its GOP.

Drops are counted per session and warned on the first, then every hundredth, so
a publisher sending nothing usable stays visible without a line per frame.

Moving the bridge into gst::whip_bridge also fixes three smaller things in the
same code: the shared A/V offset is now taken from the first buffer that
carries a PTS rather than the first buffer of the session, DTS is shifted with
PTS instead of being left in the session's time base, and a sample with no caps
no longer panics.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant