fix(whip): judge a WHIP session by the media it produces, not the bytes it receives - #756
Draft
wagenet wants to merge 2 commits into
Draft
fix(whip): judge a WHIP session by the media it produces, not the bytes it receives#756wagenet wants to merge 2 commits into
wagenet wants to merge 2 commits into
Conversation
A WHIP publisher that dies without sending a DELETE — network loss, the common case for a real participant — leaves its slot occupied. Every reconnect until the inactivity watchdog notices is refused with 503, measured at ~20 s on a 5-seat rig. The ICE callback does not help: an abruptly killed peer leaves webrtcbin at `completed` until consent freshness expires. When all slots are taken, the POST now watches the sitting session for up to 3 s instead of refusing outright. The session carries a `SessionActivity` counter, stamped by the appsink callback for every buffer that crosses the bridge. A counter that moves between two polls is a publisher that is still sending: that client gets its 503 after one poll interval, and the live session is never touched. A counter frozen past 2 s is a dead transport, and the session is handed to the ordinary cleanup path (`SessionCleanupRequest`, guarded by `cleanup_sent`) so the new client can take the slot it releases. Both cases start out identical — a reconnect landing 300 ms after the drop sees the same near-zero idle time as a healthy stream — which is why the decision rests on the counter moving rather than on a single reading of it. A session that has never delivered a buffer is never displaced: it may just be slow to negotiate through a TURN relay. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…es it receives A seat can receive RTP for a whole session while producing nothing the flow can use: its decoder never gets a keyframe it can use, or a consumer below the slot's tee blocks and backs pressure up the chain until nothing comes out. Liveness was stamped in the session pipeline's appsink, upstream of all of that, so such a seat looked perfectly healthy. Slot takeover never displaced it and the inactivity watchdog never reaped it, and every rejoining client got 503 for as long as it sat there — 45 minutes on the live rig. The slot's chain now carries its own `ActivityStamp`, written by a pad probe on the slot's output tee, past `decodebin` and the converter. `SessionActivity` holds that alongside the appsink's and reports the staler of the two, so a session counts as usable only while media both arrives and comes out. The stamp belongs to the slot, which outlives the sessions passing through it, so claiming a slot resets it — otherwise a newcomer is judged on its predecessor's frames. Media arrives before it can be decoded, because H.264 carries its parameter sets with a keyframe, so a session is not judged at all until DECODE_GRACE after its first buffer. Measured 0.9-1.0 s from the video pad appearing to `decodebin` exposing its own, against a 5 s grace. The watchdog reads the same signal through `SessionActivity::idle()` rather than recomputing it from loose parameters, so it now also reaps a seat that receives without producing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
Author
|
Note for whoever merges this alongside #752. #752 replaces the watchdog's sleep-then-check with Done that way on my fork's main; its two tests carry over with the signature. Happy to |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacks on #753. Eyevinn has no branch for that PR, so this targets
mainand itsdiff carries 753's commit as well; review only
be15584, or diff againstwagenet:wagenet/whip-slot-takeover.The bug
A browser seat on the live rig stopped producing usable video but kept receiving
RTP. It held its slot for 45 minutes: every rejoining client got
503 All session slots are occupied, and the takeover added in #753 never fired.SessionActivity::touch()was called from the appsink'snew_samplecallback.That appsink lives in the isolated session pipeline, upstream of the flow
pipeline's
decodebin, so it stamped liveness when bytes arrived, not whenframes decoded or reached a consumer. A session whose media arrives but never
becomes usable frames looked perfectly healthy to
idlest_sessionand toTAKEOVER_IDLE_THRESHOLD, so it was neither displaced nor reaped — which isexactly the state takeover exists for.
Two ways a seat gets there: its decoder never gets a keyframe it can use, or a
consumer below the slot's tee blocks and backs pressure up the chain. A recorder
whose splitmuxsink stalls on one track is one cause and is being fixed
separately; network paths can produce the same state, so this makes takeover able
to recover from it however it arises.
What changed
ActivityStampis one stream of buffers reduced to when its first and mostrecent buffer went past. There are now two per session:
decodebinand the converter. Says frames are reaching the flow's consumers.SessionActivity::idle()returns the staler of the two: a session is usable onlywhile media both arrives and comes out. A publisher going away freezes ingress
first; a stall below the decoder freezes output first.
The output stamp belongs to the slot, whose chain is built once at flow build
time and outlives the sessions passing through it, so
SessionActivity::newresets it. Without that a newcomer is judged on its predecessor's frames and the
next client evicts it the moment its own media starts.
The grace period. Media arrives before it can be decoded — H.264 carries its
parameter sets with a keyframe, and
decodebinhas a decoder to autoplug first —so
idle()refuses to judge a session untilDECODE_GRACE(5 s) after itsfirst buffer. Measured 0.9-1.0 s from the video pad appearing to
decodebinexposing its own, on two runs. Measuring from the first buffer, not from session
start, means a session that spent a minute negotiating through TURN still gets
the full grace once media starts.
The watchdog reads
SessionActivity::idle()instead of recomputing idlenessfrom loose
(epoch, last_buffer_ms)parameters, so session idleness has onedefinition. It consequently also reaps a seat that receives without producing,
which previously sat forever.
Verification
Headless, fresh backend,
max_sessions: 1, one publisher viawhipclientsinkinto a flow that muxes both tracks tomp4mux ! filesink—so a stopped audio track stalls the seat the way a stalled recorder does.
num-buffers=300on the audiotestsrc ends audio ~7 s in while video keepsarriving.
The publisher process was still sending video throughout, including at the
moment of displacement. 2027 ms is
TAKEOVER_IDLE_THRESHOLD; the watchdog couldnot have fired before ~15:15:51, so this is takeover, not the reaper. Before this
change the same seat was never displaced at all.
The healthy half is the five consecutive 503s: while the seat was producing, a
second client was refused every time, in ~100 ms (one poll interval), and the
sitting session survived. A separate earlier run refused a second client against
a fully healthy publisher with zero
Displacinglines logged.Tests
All new, and all confirmed to fail with the fix reverted (checked by temporarily
restoring ingress-only liveness, and separately by deleting the tee probe).
whip_session_manager.rs:a_session_receiving_media_it_never_decodes_is_displaced— the bug. Ingressticks the whole time; nothing ever comes out of the slot.
a_session_whose_output_has_stalled_is_displaced— decoded, then froze, whileRTP keeps arriving.
a_session_still_waiting_for_its_first_decoded_frame_is_not_displaced— thegrace period. Passes both before and after; it guards the new code against
being too aggressive.
a_new_session_does_not_inherit_the_slots_previous_liveness— the reset.whip.rs:the_slot_stamp_follows_media_out_of_the_decode_chain_not_into_it— builds theblock, assembles the chain from its own
internal_links, and walks buffersthrough
appsrc -> decodebin -> videoconvert -> tee. Asserts the stamp followsmedia that gets through, then blocks the tee's consumer and asserts the stamp
does not move while buffers keep arriving at the appsrc. Uses only appsrc,
decodebin, videoconvert, tee and fakesink, all in
gstreamer1.0-plugins-base,which CI already installs.
The two negative guards from #753 —
a_live_session_is_never_displacedanda_session_that_has_not_delivered_media_yet_is_not_displaced— still passunchanged.
Ran locally:
cargo test, 607 passed, 0 failed, 1 ignored (a pre-existingignore in
jitterbuffer_mute_test).cargo clippy --all-targetsclean.Trade-offs worth a reviewer's attention
flow pipeline stalls, every seat's output freezes and a new POST can displace
one. That seat was not delivering anything either, and the displaced session is
torn down through the ordinary cleanup path, so the cost is a reconnect rather
than lost media — but it is a behaviour change from "only a dead transport is
displaced".
DECODE_GRACEis the safety margin against evicting a healthy publishermid-preroll, and 5 s against a measured ~1 s is the only number here without
a hard bound behind it. It has to stay under
INACTIVITY_TIMEOUT(10 s) forthe watchdog to reap a session that never decodes at all.
pipeline. The callback is one
Instant::elapsed(), one relaxed store, and onerelaxed load — no lock, no allocation, no formatting.
🤖 Generated with Claude Code