Skip to content

fix(whip): detect a dropped ingest sooner and retry with backoff - #754

Open
wagenet wants to merge 1 commit into
Eyevinn:mainfrom
wagenet:wagenet/whip-ingest-fast-reconnect
Open

fix(whip): detect a dropped ingest sooner and retry with backoff#754
wagenet wants to merge 1 commit into
Eyevinn:mainfrom
wagenet:wagenet/whip-ingest-fast-reconnect

Conversation

@wagenet

@wagenet wagenet commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

Problem

The WHIP ingest page is slow to come back after its transport dies. Two timers stack up: ICE disconnected gets 10 s to recover (whip.js), then every retry waits a flat 10 s (ingest.html). On top of that Chrome takes ~6 s to notice at all, so the first reconnect attempt lands ~26 s after the drop.

The server frees the seat long before that. The inactivity watchdog reclaims it at 10 s, and #753 will hand it over as soon as ~2 s. Neither helps while the page is not asking.

What this does

ICE_DISCONNECT_GRACE_MS = 4000disconnected is genuinely transient and often recovers, so it still gets a grace period, just a shorter one. Tearing down early costs a needless renegotiation; waiting costs the publisher, who keeps encoding into a dead transport meanwhile.

RECONNECT_DELAYS = [1000, 2000, 4000, 8000, 16000, 30000] replaces the flat 10 s, last value repeating. Better on both axes: first attempt at 1 s instead of 10 s, and 15 attempts now cover 331 s instead of 150 s.

The first retry is early on purpose. With #753 the server holds an arriving POST for up to 3 s while it decides whether the sitting session is dead, so a retry that lands before the seat is free is answered rather than refused. Without #753 it collects a 503 and the next attempt is 2 s later.

Verification

Playwright drives the real page in Chromium with a fake camera and mic, waits for media to reach the server (Pad video_0), then issues a WHIP DELETE for the session from outside the browser. The page is not told — from its point of view the server has vanished, which is what network loss looks like. Same rig, same drop, before and after:

v10 (before) v11 (after)
Chrome reports ICE disconnected +6.2 s +6.3 s
first re-POST reaches the server +26.4 s +11.6 s
media flowing again +27.4 s +12.1 s

Backoff sequence and grace value read out of the page source and stepped through all 15 attempts: 1, 2, 4, 8, 16, 30 × 10, no undefined at the boundary. Both files pass node --check (the inline script extracted from the HTML).

Server was built from main — no #753 — so this measures the client timing on its own.

What this does not fix

Chrome's ~6 s detection is now the floor, and a page cannot shorten it. That keeps the first re-POST at ~11.6 s, still past the 10 s watchdog, so #753's takeover window remains out of reach for this page even after this change. Getting under it means detecting the drop ourselves — polling getStats() for outbound RTP with no returning RTCP — rather than waiting for the browser's ICE state. Worth doing, not here.

Risk

An ICE blip that would have recovered between 4 s and 10 s now causes a full renegotiation instead: new POST, new session, new keyframe wait, a second or two of black. The measurements above put Chrome's own detection at ~6 s, so a blip has to be quite specifically timed to land in that band.

Tests

None. This is a timing change in static browser JS, and the repo has no JS test harness — no package.json, no test runner, and CI never touches backend/static/. A meaningful guard would have to drive a real browser against a real server, which is the Playwright run above; standing that up in CI is a bigger change than this one.

cargo build only, to re-embed the static files. No Rust changed.

🤖 Generated with Claude Code

@wagenet
wagenet marked this pull request as ready for review September 3, 2026 18:45
Recovery from a dead transport takes ~26 s: Chrome needs ~6 s to report
ICE `disconnected`, the page then allows 10 s for it to recover, and the
first retry waits another flat 10 s. The server frees the seat at 10 s,
so the page asks long after it could have.

Give `disconnected` 4 s to recover, and back the retries off — 1, 2, 4,
8, 16, then 30 s — so fifteen attempts cover 331 s. The first attempt is
early on purpose: the server decides a dropped publisher is gone a couple
of seconds after its media stops, and holds an arriving POST while it
decides, so an early retry is answered rather than refused.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@wagenet
wagenet force-pushed the wagenet/whip-ingest-fast-reconnect branch from cccc205 to a06661c Compare September 3, 2026 18:50
@srperens srperens mentioned this pull request Sep 4, 2026

@srperens srperens left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: Comment — correct diagnosis, sound fix, and every checkable claim confirms, but no check run in this repo's CI exercises the changed logic, so approval is blocked on that ground alone.

Claims

Claim Verdict Evidence
Server's inactivity watchdog fires at 10s, matching the "server frees the seat" claim CONFIRMED backend/src/blocks/builtin/whip.rs:506`const INACTIVITY_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(10);`
RECONNECT_DELAYS indexing never hits undefined, including past attempt 6 CONFIRMED backend/static/whip/ingest.html:293`const delay = RECONNECT_DELAYS[Math.min(reconnectAttempt, RECONNECT_DELAYS.length) - 1];`reconnectAttempt starts at 0 (ingest.html:113) and is incremented before use, so the index is [0, 5] for attempts [1, 15]
#753 is not yet merged, so this PR's measured numbers reflect current main, not a #753-assisted handoff CONFIRMED gh pr view 753: state=OPEN, mergedAt=null
CI never exercises this JS CONFIRMED .github/workflows/ci.yml contains no reference to static, whip, node --check, or a JS runner

Diagnosis — Root cause and fix are both right: two stacked flat 10s waits are slower than the server's own 10s reclaim, so the page's retry consistently loses the race. Backoff starting at 1s instead of a flat 10s, and a shorter (still nonzero) ICE grace period, directly close that gap. The PR is explicit about what it does not fix (Chrome's own ~6s ICE-detection floor) rather than overselling the win — that's the right scope statement.

Not coveredbackend/static/whep/player.html has the identical pattern: its own flat RECONNECT_DELAY = 10000 (player.html:120) and a hardcoded 10000ms wait on its own disconnect path (player.html:810), untouched by this diff and not sharing whip.js. BOUNDED, not ABSOLUTE — WHEP playback has no server-side "seat" to lose the way WHIP ingest does, so the urgency argument in this PR's body doesn't automatically transfer, but the same slow-reconnect UX symptom is still there.

RadiusLOCAL: whip.js has exactly one consumer (ingest.html; grepped backend/static/ for other includes), and the diff changes no server code.

Tests & CICheck (Linux), both Build (Linux ...), Check & Build (WASM), API Contract Check green at this head — expected, since none of them touch JS; the repo genuinely has no JS runner (package.json doesn't exist under backend/static/), matching the PR's own statement. The Playwright numbers in the body are not something I ran or can verify here; the source-level logic they describe (bounds, delay values) I did verify directly, above.

Confidence: HIGH

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants