Skip to content

fix(whip): judge a WHIP session by the media it produces, not the bytes it receives - #756

Draft
wagenet wants to merge 2 commits into
Eyevinn:mainfrom
wagenet:wagenet/whip-liveness-decoded
Draft

fix(whip): judge a WHIP session by the media it produces, not the bytes it receives#756
wagenet wants to merge 2 commits into
Eyevinn:mainfrom
wagenet:wagenet/whip-liveness-decoded

Conversation

@wagenet

@wagenet wagenet commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

Stacks on #753. Eyevinn has no branch for that PR, so this targets main and its
diff carries 753's commit as well; review only be15584, or diff against
wagenet:wagenet/whip-slot-takeover.

The bug

A browser seat on the live rig stopped producing usable video but kept receiving
RTP. It held its slot for 45 minutes: every rejoining client got
503 All session slots are occupied, and the takeover added in #753 never fired.

SessionActivity::touch() was called from the appsink's new_sample callback.
That appsink lives in the isolated session pipeline, upstream of the flow
pipeline's decodebin, so it stamped liveness when bytes arrived, not when
frames decoded or reached a consumer. A session whose media arrives but never
becomes usable frames looked perfectly healthy to idlest_session and to
TAKEOVER_IDLE_THRESHOLD, so it was neither displaced nor reaped — which is
exactly the state takeover exists for.

Two ways a seat gets there: its decoder never gets a keyframe it can use, or a
consumer below the slot's tee blocks and backs pressure up the chain. A recorder
whose splitmuxsink stalls on one track is one cause and is being fixed
separately; network paths can produce the same state, so this makes takeover able
to recover from it however it arises.

What changed

ActivityStamp is one stream of buffers reduced to when its first and most
recent buffer went past. There are now two per session:

  • ingress — the session appsink, as before. Says the publisher is sending.
  • output — a pad probe on the slot's output tee in the main pipeline, past
    decodebin and the converter. Says frames are reaching the flow's consumers.

SessionActivity::idle() returns the staler of the two: a session is usable only
while media both arrives and comes out. A publisher going away freezes ingress
first; a stall below the decoder freezes output first.

The output stamp belongs to the slot, whose chain is built once at flow build
time and outlives the sessions passing through it, so SessionActivity::new
resets it. Without that a newcomer is judged on its predecessor's frames and the
next client evicts it the moment its own media starts.

The grace period. Media arrives before it can be decoded — H.264 carries its
parameter sets with a keyframe, and decodebin has a decoder to autoplug first —
so idle() refuses to judge a session until DECODE_GRACE (5 s) after its
first buffer. Measured 0.9-1.0 s from the video pad appearing to decodebin
exposing its own, on two runs. Measuring from the first buffer, not from session
start, means a session that spent a minute negotiating through TURN still gets
the full grace once media starts.

The watchdog reads SessionActivity::idle() instead of recomputing idleness
from loose (epoch, last_buffer_ms) parameters, so session idleness has one
definition. It consequently also reaps a seat that receives without producing,
which previously sat forever.

Verification

Headless, fresh backend, max_sessions: 1, one publisher via
whipclientsink into a flow that muxes both tracks to mp4mux ! filesink
so a stopped audio track stalls the seat the way a stalled recorder does.
num-buffers=300 on the audiotestsrc ends audio ~7 s in while video keeps
arriving.

15:15:33.98  slot 0 allocated
15:15:34.55  Pad audio_0 / Pad video_0            (media arriving)
15:15:35.57  decodebin video pad linked           (producing)
15:15:35.65  rejoin -> 503                        publisher=sending
15:15:36.80  rejoin -> 503                        publisher=sending
15:15:37.97  rejoin -> 503                        publisher=sending
15:15:39.14  rejoin -> 503                        publisher=sending
15:15:40.31  rejoin -> 503                        publisher=sending
             ... audio ends, mp4mux stalls, seat produces nothing ...
15:15:42.81  Displacing session 'ff49a5e2' on port 60987
             (2027 ms without usable media)       publisher=sending
15:15:42.91  slot 0 allocated to the rejoining client

The publisher process was still sending video throughout, including at the
moment of displacement. 2027 ms is TAKEOVER_IDLE_THRESHOLD; the watchdog could
not have fired before ~15:15:51, so this is takeover, not the reaper. Before this
change the same seat was never displaced at all.

The healthy half is the five consecutive 503s: while the seat was producing, a
second client was refused every time, in ~100 ms (one poll interval), and the
sitting session survived. A separate earlier run refused a second client against
a fully healthy publisher with zero Displacing lines logged.

Tests

All new, and all confirmed to fail with the fix reverted (checked by temporarily
restoring ingress-only liveness, and separately by deleting the tee probe).

whip_session_manager.rs:

  • a_session_receiving_media_it_never_decodes_is_displaced — the bug. Ingress
    ticks the whole time; nothing ever comes out of the slot.
  • a_session_whose_output_has_stalled_is_displaced — decoded, then froze, while
    RTP keeps arriving.
  • a_session_still_waiting_for_its_first_decoded_frame_is_not_displaced — the
    grace period. Passes both before and after; it guards the new code against
    being too aggressive.
  • a_new_session_does_not_inherit_the_slots_previous_liveness — the reset.

whip.rs:

  • the_slot_stamp_follows_media_out_of_the_decode_chain_not_into_it — builds the
    block, assembles the chain from its own internal_links, and walks buffers
    through appsrc -> decodebin -> videoconvert -> tee. Asserts the stamp follows
    media that gets through, then blocks the tee's consumer and asserts the stamp
    does not move while buffers keep arriving at the appsrc. Uses only appsrc,
    decodebin, videoconvert, tee and fakesink, all in gstreamer1.0-plugins-base,
    which CI already installs.

The two negative guards from #753a_live_session_is_never_displaced and
a_session_that_has_not_delivered_media_yet_is_not_displaced — still pass
unchanged.

Ran locally: cargo test, 607 passed, 0 failed, 1 ignored (a pre-existing
ignore in jitterbuffer_mute_test). cargo clippy --all-targets clean.

Trade-offs worth a reviewer's attention

  • A stall that is not the seat's fault still costs it its slot. If the whole
    flow pipeline stalls, every seat's output freezes and a new POST can displace
    one. That seat was not delivering anything either, and the displaced session is
    torn down through the ordinary cleanup path, so the cost is a reconnect rather
    than lost media — but it is a behaviour change from "only a dead transport is
    displaced".
  • DECODE_GRACE is the safety margin against evicting a healthy publisher
    mid-preroll
    , and 5 s against a measured ~1 s is the only number here without
    a hard bound behind it. It has to stay under INACTIVITY_TIMEOUT (10 s) for
    the watchdog to reap a session that never decodes at all.
  • The output probe is a BUFFER probe, so it is on the hottest path in the
    pipeline. The callback is one Instant::elapsed(), one relaxed store, and one
    relaxed load — no lock, no allocation, no formatting.

🤖 Generated with Claude Code

wagenet and others added 2 commits September 3, 2026 10:51
A WHIP publisher that dies without sending a DELETE — network loss, the
common case for a real participant — leaves its slot occupied. Every
reconnect until the inactivity watchdog notices is refused with 503,
measured at ~20 s on a 5-seat rig. The ICE callback does not help: an
abruptly killed peer leaves webrtcbin at `completed` until consent
freshness expires.

When all slots are taken, the POST now watches the sitting session for
up to 3 s instead of refusing outright. The session carries a
`SessionActivity` counter, stamped by the appsink callback for every
buffer that crosses the bridge. A counter that moves between two polls
is a publisher that is still sending: that client gets its 503 after one
poll interval, and the live session is never touched. A counter frozen
past 2 s is a dead transport, and the session is handed to the ordinary
cleanup path (`SessionCleanupRequest`, guarded by `cleanup_sent`) so the
new client can take the slot it releases.

Both cases start out identical — a reconnect landing 300 ms after the
drop sees the same near-zero idle time as a healthy stream — which is
why the decision rests on the counter moving rather than on a single
reading of it. A session that has never delivered a buffer is never
displaced: it may just be slow to negotiate through a TURN relay.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…es it receives

A seat can receive RTP for a whole session while producing nothing the flow
can use: its decoder never gets a keyframe it can use, or a consumer below
the slot's tee blocks and backs pressure up the chain until nothing comes
out. Liveness was stamped in the session pipeline's appsink, upstream of all
of that, so such a seat looked perfectly healthy. Slot takeover never
displaced it and the inactivity watchdog never reaped it, and every
rejoining client got 503 for as long as it sat there — 45 minutes on the
live rig.

The slot's chain now carries its own `ActivityStamp`, written by a pad probe
on the slot's output tee, past `decodebin` and the converter.
`SessionActivity` holds that alongside the appsink's and reports the staler
of the two, so a session counts as usable only while media both arrives and
comes out. The stamp belongs to the slot, which outlives the sessions
passing through it, so claiming a slot resets it — otherwise a newcomer is
judged on its predecessor's frames.

Media arrives before it can be decoded, because H.264 carries its parameter
sets with a keyframe, so a session is not judged at all until DECODE_GRACE
after its first buffer. Measured 0.9-1.0 s from the video pad appearing to
`decodebin` exposing its own, against a 5 s grace.

The watchdog reads the same signal through `SessionActivity::idle()` rather
than recomputing it from loose parameters, so it now also reaps a seat that
receives without producing.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@wagenet

wagenet commented Sep 4, 2026

Copy link
Copy Markdown
Contributor Author

Note for whoever merges this alongside #752.

#752 replaces the watchdog's sleep-then-check with wait_for_inactivity, which polls
idle every WATCHDOG_POLL tick. This PR rewrites the same loop to read
SessionActivity::idle(). Both are wanted, and the resolution is not "keep one":
keep #752's per-tick polling and change wait_for_inactivity to take
&SessionActivity instead of (epoch, last_buffer_ms, ...), so session idleness has a
single definition shared with slot takeover rather than two that can drift.

Done that way on my fork's main; its two tests carry over with the signature. Happy to
fold it into this PR once #752 lands.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant