fix(recorder): end a track that stops so the rest of the recording continues

[wagenet]

Stacks on #750, whose branch is not on this remote, so its commit shows up in this diff too — review the second commit, fix(recorder): end a track that stops..., only. #750 fixed an input that never carries data; this is the different case of an input that carried data and then stopped.

The bug

splitmuxsink releases a GOP only once every one of its sink pads has advanced past it. A track that stops delivering never advances, so the recording freezes — and because the recorder shares a tee with the seat's other consumers, the backpressure reaches them too. A participant whose microphone dies at the source takes down their own video and recording, and the program output as well if they were the only live source.

Reproduced in-process through the real RecorderBuilder, no WHIP needed: one recorder with a continuous H.264 track and an AAC track that stops after two seconds with its EOS dropped (EOS does not cross a WebRTC hop, so a dead microphone just stops sending). Video reaching the muxer over a 5 s window: 0 buffers, 0 bytes. Control with both tracks continuous over the same window: 180 buffers — exactly 30 fps — and 1.5 MB.

The fix

A watchdog thread ends the stalled track with EOS. Measurements behind the two decisions that are not obvious:

EOS, not GAP. A GAP event is the natural way to say "nothing here for a while", but splitmuxsink ignores it on a non-reference stream. Pushed at the block boundary, 20 GAPs reached the audio_0 sink pad and video stayed at 0 buffers; delivered straight to that pad, still 0. EOS in the same position recovered +270 buffers and +2.26 MB, and the resulting file is a valid 10 s MP4 with both tracks (gst-discoverer-1.0). So the track has to end rather than idle, and the audio track ends where the microphone died.

The trigger is the whole recording being frozen, not one quiet input. The first version watched arrival at each recorder input and ended whichever had been quiet longest. That is not sound, and the end-to-end rig showed why: once the muxer blocks, every other input backs up behind it within a second, so all inputs look equally dead. Across three runs it ended audio, then video, then video — and in one run it ended a track on the control seat, whose publisher never muted, because the box was CPU-bound and delivery genuinely paused for 5 s. A single quiet input cannot distinguish a dead track from a loaded machine.

So detection moved to the splitmuxsink sink pads: a probe records when the muxer last took a buffer from each track and how far that buffer carried the recording. Nothing is ended unless every track has been frozen for the timeout, and the one ended is whichever last muxed buffer is furthest behind — the same track splitmuxsink is waiting on internally. Ending one track pauses the watchdog for a further timeout so the others can recover before anything else is considered.

Buffers arriving after a track ends are dropped at the block boundary, so upstream never sees the flow error a dead branch returns: the seat's encoder feeds the vision mixer through the same tee and must not stop with the recording.

The two BUFFER probes are on the hot path and stay within the rules in CLAUDE.md — one relaxed atomic load on the input probe, two relaxed stores plus Instant::elapsed on the intake probe, no locks or allocation.

Tests

backend/tests/recorder_stalled_track_test.rs, two tests, both run locally against the real builder:

• a_track_that_stops_does_not_freeze_the_recording — fails when the fix is reverted: 0 video buffers reached the muxer in 5 s (0 bytes written). It also pins down which track is ended: had the recorder ended the video track, no video would reach the muxer either.
• tracks_that_keep_running_are_left_alone — the counterpart. Ending tracks on a timer regardless of liveness would satisfy the first test and destroy every recording.

They need avenc_aac and aacparse beyond what recorder_idle_input_test already required; recorder_unfed_track_test uses both today, and gstreamer1.0-libav is installed in the Linux and macOS CI jobs that set STROM_REQUIRE_GST_PLUGINS=1.

Ran locally: the full backend suite, cargo test --package strom --features efp with STROM_REQUIRE_GST_PLUGINS=1 — 605 tests across 22 binaries, all passing, including pipeline_lifecycle_test and #750's recorder_idle_input_test. cargo fmt --check and cargo clippy --all-targets --features efp -D warnings clean. Nothing was skipped.

Not covered, found while testing this

A track that is connected but never carries a buffer stalls the recording in exactly the same way, and neither #750 nor this change covers it. #750 keeps the sink locked until the first caps arrive on any track, so a recorder whose video track has data and whose audio track never does unlocks the sink and then waits forever on the audio pad. Measured in the same harness: 0 video buffers over 3 s, and EOS delivered straight to the audio_0 sink pad releases it (+31 buffers). This is a participant who joins video-only, and it is what the end-to-end WHIP rig kept landing in.

It needs a different arming rule — the timeout has to run from when the recording starts, not from flow start, or every recorder whose publisher connects late would lose its audio track — and a different delivery point, since the input identity is still unlinked at that stage. Worth its own change rather than bundling it here. The watchdog stays silent in that case by design: it needs at least two tracks that have both been muxed, so it will not end the one healthy track.

Verified end to end

One WHIP seat, decode: true, into builtin.videoenc and an audioconvert ! audioresample ! avenc_aac chain, both into a two-track recorder. Same rig, same publisher, same machine, one variable — this branch against its base commit, run concurrently so load is equal.

The publisher's microphone is closed at runtime with a valve, not with num-buffers: num-buffers sends EOS into whipclientsink and gst-launch then tears the whole publisher down, video included (Got EOS from element "pipeline0"). A valve stops the branch with no EOS, which is also what the server sees, since EOS does not cross a WebRTC hop.

	base commit	this branch
recording after the microphone dies	frozen, +0 bytes over 80 s	+84 KB / 8 s, steady for 90 s
track ended	—	audio 0, the one that died
WHIP session	torn down, Inactivity timeout (10s idle)	alive, publisher still connected
pipeline errors	0	0

The finished file is a valid 2:41 MP4 with both streams (gst-discoverer-1.0): video to the end, audio up to where the microphone died. A control run of the same rig, untouched for two minutes, ended nothing and reported no errors.

Two defects came out of that end-to-end work and are fixed here; neither was visible in the isolated reproduction:

• It ended the wrong track. Ranking by raw PTS is meaningless across these two tracks — a WHIP seat's video arrives at running time 3600053010 ms against its audio's 43541 ms, so audio is always "furthest behind" and video could never be chosen. Positions are now converted to running time with the segment from each sink pad, which is what splitmuxsink compares.
• It killed the seat five seconds later. Ending a track makes aacparse drain a partial frame with no PTS; mp4mux answers Buffer has no PTS, errors, and takes the seat's video path down with it (appsrc_video_0: streaming stopped, reason error (-5)) — strictly worse than the stall. Buffers with no PTS are now dropped at the muxer's sink pad, which loses nothing, since the muxer cannot accept them.

An earlier design that watched arrival at each recorder input was discarded for the reason described above; across three runs it ended audio, then video, then video, and once ended a track on a control seat whose publisher never muted.

🤖 Generated with Claude Code