fix(whip): detect a dropped ingest sooner and retry with backoff - #754
fix(whip): detect a dropped ingest sooner and retry with backoff#754wagenet wants to merge 1 commit into
Conversation
Recovery from a dead transport takes ~26 s: Chrome needs ~6 s to report ICE `disconnected`, the page then allows 10 s for it to recover, and the first retry waits another flat 10 s. The server frees the seat at 10 s, so the page asks long after it could have. Give `disconnected` 4 s to recover, and back the retries off — 1, 2, 4, 8, 16, then 30 s — so fifteen attempts cover 331 s. The first attempt is early on purpose: the server decides a dropped publisher is gone a couple of seconds after its media stops, and holds an arriving POST while it decides, so an early retry is answered rather than refused. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
cccc205 to
a06661c
Compare
srperens
left a comment
There was a problem hiding this comment.
Verdict: Comment — correct diagnosis, sound fix, and every checkable claim confirms, but no check run in this repo's CI exercises the changed logic, so approval is blocked on that ground alone.
Claims
| Claim | Verdict | Evidence |
|---|---|---|
| Server's inactivity watchdog fires at 10s, matching the "server frees the seat" claim | CONFIRMED | backend/src/blocks/builtin/whip.rs:506 — `const INACTIVITY_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(10);` |
RECONNECT_DELAYS indexing never hits undefined, including past attempt 6 |
CONFIRMED | backend/static/whip/ingest.html:293 — `const delay = RECONNECT_DELAYS[Math.min(reconnectAttempt, RECONNECT_DELAYS.length) - 1];` — reconnectAttempt starts at 0 (ingest.html:113) and is incremented before use, so the index is [0, 5] for attempts [1, 15] |
#753 is not yet merged, so this PR's measured numbers reflect current main, not a #753-assisted handoff |
CONFIRMED | gh pr view 753: state=OPEN, mergedAt=null |
| CI never exercises this JS | CONFIRMED | .github/workflows/ci.yml contains no reference to static, whip, node --check, or a JS runner |
Diagnosis — Root cause and fix are both right: two stacked flat 10s waits are slower than the server's own 10s reclaim, so the page's retry consistently loses the race. Backoff starting at 1s instead of a flat 10s, and a shorter (still nonzero) ICE grace period, directly close that gap. The PR is explicit about what it does not fix (Chrome's own ~6s ICE-detection floor) rather than overselling the win — that's the right scope statement.
Not covered — backend/static/whep/player.html has the identical pattern: its own flat RECONNECT_DELAY = 10000 (player.html:120) and a hardcoded 10000ms wait on its own disconnect path (player.html:810), untouched by this diff and not sharing whip.js. BOUNDED, not ABSOLUTE — WHEP playback has no server-side "seat" to lose the way WHIP ingest does, so the urgency argument in this PR's body doesn't automatically transfer, but the same slow-reconnect UX symptom is still there.
Radius — LOCAL: whip.js has exactly one consumer (ingest.html; grepped backend/static/ for other includes), and the diff changes no server code.
Tests & CI — Check (Linux), both Build (Linux ...), Check & Build (WASM), API Contract Check green at this head — expected, since none of them touch JS; the repo genuinely has no JS runner (package.json doesn't exist under backend/static/), matching the PR's own statement. The Playwright numbers in the body are not something I ran or can verify here; the source-level logic they describe (bounds, delay values) I did verify directly, above.
Confidence: HIGH
Problem
The WHIP ingest page is slow to come back after its transport dies. Two timers stack up: ICE
disconnectedgets 10 s to recover (whip.js), then every retry waits a flat 10 s (ingest.html). On top of that Chrome takes ~6 s to notice at all, so the first reconnect attempt lands ~26 s after the drop.The server frees the seat long before that. The inactivity watchdog reclaims it at 10 s, and #753 will hand it over as soon as ~2 s. Neither helps while the page is not asking.
What this does
ICE_DISCONNECT_GRACE_MS = 4000—disconnectedis genuinely transient and often recovers, so it still gets a grace period, just a shorter one. Tearing down early costs a needless renegotiation; waiting costs the publisher, who keeps encoding into a dead transport meanwhile.RECONNECT_DELAYS = [1000, 2000, 4000, 8000, 16000, 30000]replaces the flat 10 s, last value repeating. Better on both axes: first attempt at 1 s instead of 10 s, and 15 attempts now cover 331 s instead of 150 s.The first retry is early on purpose. With #753 the server holds an arriving POST for up to 3 s while it decides whether the sitting session is dead, so a retry that lands before the seat is free is answered rather than refused. Without #753 it collects a 503 and the next attempt is 2 s later.
Verification
Playwright drives the real page in Chromium with a fake camera and mic, waits for media to reach the server (
Pad video_0), then issues a WHIP DELETE for the session from outside the browser. The page is not told — from its point of view the server has vanished, which is what network loss looks like. Same rig, same drop, before and after:disconnectedBackoff sequence and grace value read out of the page source and stepped through all 15 attempts:
1, 2, 4, 8, 16, 30 × 10, noundefinedat the boundary. Both files passnode --check(the inline script extracted from the HTML).Server was built from
main— no #753 — so this measures the client timing on its own.What this does not fix
Chrome's ~6 s detection is now the floor, and a page cannot shorten it. That keeps the first re-POST at ~11.6 s, still past the 10 s watchdog, so #753's takeover window remains out of reach for this page even after this change. Getting under it means detecting the drop ourselves — polling
getStats()for outbound RTP with no returning RTCP — rather than waiting for the browser's ICE state. Worth doing, not here.Risk
An ICE blip that would have recovered between 4 s and 10 s now causes a full renegotiation instead: new POST, new session, new keyframe wait, a second or two of black. The measurements above put Chrome's own detection at ~6 s, so a blip has to be quite specifically timed to land in that band.
Tests
None. This is a timing change in static browser JS, and the repo has no JS test harness — no
package.json, no test runner, and CI never touchesbackend/static/. A meaningful guard would have to drive a real browser against a real server, which is the Playwright run above; standing that up in CI is a bigger change than this one.cargo buildonly, to re-embed the static files. No Rust changed.🤖 Generated with Claude Code