feat(liveaudiorouter): live audio router on synchronising buses, with per-sample crosspoint fades

[srperens]

Closes #661. An alternative to #737, which targets the same issue with a different design.

#737 expresses the routing as coefficients in a single audiomixmatrix, fed by interleave and split back out with deinterleave. I built and ran it locally before writing this, and found three problems that are properties of that foundation rather than bugs in the implementation. All numbers below are measured on this machine (GStreamer 1.28.6, macOS), not inferred.

The router stalls when an input is not connected

interleave is a GstCollectPads element: it waits for one buffer on every sink pad before producing output. audiomixer/audiointerleave are GstAudioAggregators, which align on timestamps, fill gaps with silence and aggregate on a timeout in live mode.

Same graph, 3-second windows, counting level messages:

	interleave	audiointerleave
both inputs connected	61	61
one input pad requested but unconnected	0	59
one source stops mid-stream	17 before, 0 after	17 before, 43 after

A configured-but-unconnected input is the default state of any router the operator has not finished wiring, and a source dropping out is the thing a router exists to survive. #737 lists "interleave on an unconnected input" among its own unverified items.

A crosspoint change cannot fade

Only the standalone volume element samples its volume property per sample (volume_transform_ip in gstvolume.c) — which is what crate::gst::volume_ramp already relies on, and which its module header already documents. Driving a 40 ms linear ramp and counting how often the gain actually changed:

coefficient	flags	gain changed over 40 ms
audiomixmatrix.matrix	readable, writable	not controllable
audiomixer sink pad converter-config	readable, writable	not controllable
audiomixer sink pad volume, 10 ms blocks	controllable	4
volume element	controllable	880 (of 1680 samples)

The mixer pad produces a staircase in step with output-buffer-duration; the envelope went 0 → 25 → 50 → 75 → 100 %FS in 10 ms steps. A matrix property cannot interpolate at all. Both step, and a step is an audible click on every take.

It was unreachable from the UI

routing_matrix is declared live: true, but the routing editor's Save writes the whole flow through POST /api/flows/{id}, which applies pad properties live and never block properties. Verified directly: a live write moved audio in 64 ms, while the same change through the flow-save path left the pipeline on the old routing. The only way to see a change was to restart the flow — which renegotiates every WHEP session and resets the meters.

This design

The pattern the mixer block has used for its aux sends all along (tee → volume → audiomixer), which answers all three at once:

audio_in_I → identity_in_I → deinterleave_in_I → tee_iIcC
tee_iIcC → xp_iIcC_oOcD (volume) → mixer_O sink pad (places channel D)
mixer_O → caps_out_O → capssetter_out_O → queue_out_O → audio_out_O

audiomixer does the synchronisation. The coefficient is a volume element, so it fades per sample. And because the coefficient is a gdouble on a real element, a crosspoint is a level rather than a checkbox — {"i0c0": {"o0c0": 1.0, "o1c0": 0.35}} alongside the existing {"i0c0": ["o0c0"]}, so routing written for builtin.audiorouter keeps working. builtin.audiorouter itself is untouched.

The whole crossbar is built up front and stays linked, so a routing change only writes gains — it never relinks and nothing restarts. interleave, audiomixmatrix, capssetter_in and the dynamic pad-added queue construction are all gone, as is the global layout registry keyed by element id that #737 flagged as worth a look: crosspoints are enumerated from the running pipeline's element map instead.

No queue per crosspoint

A tee branch normally needs a queue so it cannot block its siblings, but every branch here ends on a GstAggregatorPad, which queues the buffer and returns rather than waiting for the mix — and the bus is built with force-live, a latency timeout and ignore-inactive-pads, so it never waits indefinitely on a pad and always drains. Verified over 20 s at two scales:

--- 7 in-ch x 10 out-ch = 70 crosspoints ---
  with queues:  401 level msgs, 70 queues
  NO queues  :  401 level msgs,  0 queues
--- 32 in-ch x 16 out-ch = 512 crosspoints ---
  with queues:  401 level msgs, 512 queues
  NO queues  :  401 level msgs,  0 queues

Each queue is a streaming thread. On the test flow below this took one router block from 161 elements and 70 queue threads to 91 elements and 2 — one per output bus, which is where downstream decoupling belongs.

Verified in a running flow

A test flow with three inputs of different channel counts (1 / 2 / 4), each channel a distinct tone at a distinct level ~2.5 dB apart, into an 8-channel monitor bus on a meter and a stereo bus on a WHEP output. Measured peaks against the routing:

meter out	source	measured	expected
o1c0	i0c0 @ 0.75	−2.50	−2.50
o1c1	i1c0 @ 0.50	−6.02	−6.02
o1c2	i1c1 @ 0.375	−8.52	−8.52
o1c3..c6	i2c0..c3	−11.06 … −18.42	matches each source
o1c5	i2c2 @ crosspoint gain 0.5	−21.94	−21.94
o1c7	i1c0 + i1c1 (fan-in)	−1.16	−1.16 (0.875)

Fan-out, fan-in, per-crosspoint gain, all-closed (silent, with no silence source), and extreme cases all behave. Live changes applied without a single pipeline restart, and the fade is visible on the meter — with crosspoint_fade_ms at 800 ms: −64.2 → −56.2 → −52.6 → … → −2.5 dB, evenly spaced in dB, which is volume_ramp.rs' log curve for ramps over 50 ms.

Properties

crosspoint_fade_ms defaults to the mixer's DEFAULT_VOLUME_RAMP_MS. force_live, latency, min_upstream_latency and output_buffer_duration use the mixer block's names, types and defaults — both blocks sum onto an audiomixer, so an operator who has tuned one should not have to learn a second vocabulary. output_buffer_duration is documented as a latency-versus-CPU knob only: because the coefficient moved to volume, it no longer affects how smooth a fade is.

UI

Save now applies live, and a "Live" toggle (on by default) sends every change as it is made — block properties persist by default, so in live mode the endpoint stores the value too and Save has nothing left to do. Changes are debounced through the same filter the property inspector's live controls use. Each open crosspoint gets a compact dB control next to its checkbox.

Both routing forms are parsed, which matters beyond convenience: the old editor parsed only the list form and fell back to an empty matrix, so opening it on a matrix with gains and pressing Save wiped the routing. This actually happened once during development.

Live mode and the gain control are gated on capability rather than a block id — the live flag of the block's own routing_matrix property, and the presence of crosspoint_fade_ms. builtin.audiorouter therefore keeps its Save button and its plain checkboxes, and the_original_audiorouter_still_routes_audio runs real audio through it on this code.

One routing-matrix format

The backend and the editor each had their own parser for the same JSON, and the editor's copy is the one that decides whether a saved routing survives a round trip. The format now lives in strom_types::routing and both sides use it. The editor's internals are keyed by Crosspoint rather than formatted strings, which makes cleanup_routing one bounds check instead of two key sets.

The tests moved there for a second reason: the workspace has default-members = ["backend"] and CI runs cargo test per package for strom, strom-mcp-server and strom-types — never strom-frontend. Tests next to the editor would never have run.

Two things found while checking that the old block still works: the live write was sent unconditionally, so saving on builtin.audiorouter made a request the backend answers with rejected and the frontend discards (harmless, but a wasted round-trip and a swallowed rejection) — now gated on the property being declared live. And serialisation was unordered, so saving the same routing twice produced different JSON and churned the flow file; a_plain_routing_round_trips_as_the_list_form now pins that the list form comes back byte for byte.

Tests

16 integration tests, 4 backend unit tests and 9 format tests in strom-types. The audio-path tests run real audio through the block's own builder rather than re-implementing the routing model — #737's two matrix tests were pure unit tests with no audio, and its one end-to-end test failed on its own baseline. New guards for each measured failure above: an_unconnected_input_does_not_stall_the_router, a_source_that_stops_does_not_stall_the_other_inputs, every_crosspoint_is_a_volume_element_with_a_controllable_gain, the_crossbar_costs_no_thread_per_crosspoint. Every element they need is in the CI package list, so none of them skip.

Ran locally: the 16 integration tests, the 8 unit tests, cargo test --lib (545), cargo test --package strom-types (67), openapi_test, volume_ramp_test, pipeline_lifecycle_test, block_properties_persist_test, api_tests. cargo fmt --check and cargo clippy --all-targets clean, including wasm32.

Not verified

The frontend has no test for the new cell or for live mode, and CI does not test the strom-frontend package at all — worth fixing separately. Rates other than 48 kHz, more than 8 channels per stream, Windows and Linux (developed and measured on macOS). MAX_CROSSPOINTS is 1024 — no longer 1024 threads, but still 1024 elements in one block, and worth a second opinion. The capability-parity test now allows five documented additions instead of requiring an exact property match; it still fails on any other addition, but it is a relaxation of the guard requested on #661. audiomixer sums without headroom, so enough crosspoints on one output channel can clip — the same is true of builtin.audiorouter, but the matrix form makes it easier to hit.

🤖 Generated with Claude Code

[srperens]

Verdict: Comment — a well-argued alternative to #737 for the same issue, with genuine end-to-end audio coverage. Not approvable only because Check (Linux) (the job that runs these tests) was still pending at review time — nothing found in the diff blocks it.

Claims

Claim	Verdict	Evidence
Check (Linux) (runs cargo test) has completed and is green	UNVERIFIED	gh pr checks 740 at review time: Check (Linux) pending; only API Contract Check and Check & Build (WASM) had finished
The element guard fails hard rather than skipping silently	CONFIRMED	backend/tests/liveaudiorouter_test.rs:221 — `assert!(` on missing.is_empty(), not a skip — satisfies CLAUDE.md
builtin.audiorouter's own routing_matrix is unaffected by the new live-write intercept	CONFIRMED	backend/src/blocks/builtin/audiorouter.rs:842-847 — `element_id: "_block".to_string(),` with `live: false,`, never reaches instance_from_anchor's ":mixer_0" match
No crosspoint captures a strong pipeline/element ref in the deinterleave pad-added closure	CONFIRMED	backend/src/blocks/builtin/liveaudiorouter.rs:400 — `let instance_for_cb = instance_id.to_string();` — only a String moves in; the bin comes from the callback's own arg
Omitting the per-crosspoint queue matches this repo's own tee→volume→mixer aux-send pattern	CONTRADICTED	backend/src/blocks/builtin/mixer/builder.rs:971 — `` `// Link: (pre

Diagnosis — Directly rebuts #737's foundation (interleave stalling on an unconnected/stopped input, a matrix coefficient that cannot fade, live writes the flow-save path silently drops) with measurements; #737's own body already lists the first and third as its own unverified gaps, so the critique lines up with what #737 admits. tee→volume→audiomixer is not new to this tree — aux sends already do it — a real point in this design's favour.

On the missing queue — the PR argues every crosspoint lands on a GstAggregatorPad, which queues and returns rather than blocking the tee, so a per-crosspoint queue buys no decoupling the bus doesn't already have. A claim about upstream behaviour (EXTERNAL), plausibly correct, but it's the one place this diverges from a pattern the codebase already uses for the same shape, and the_crossbar_costs_no_thread_per_crosspoint only counts elements — no backpressure stress. Worth a sanity-check given the aux-send precedent, not a blocker.

Blast radius — SHARED: new branches in update_element_property (properties.rs) and update_block_properties (state.rs) sit ahead of the existing per-property paths, but both are guarded — by instance_from_anchor's ":mixer_0" suffix match, and by definition_id == liveaudiorouter::BLOCK_ID — and leave builtin.audiorouter untouched (row above). make_audiomixer widens pub(super) → pub(crate), one new caller, no signature change.

Tests & CI — API Contract Check, Check & Build (WASM) green at b920c5f. Check (Linux), Build (Linux x86_64/ARM64) still running; macOS/Windows skipping as expected. The 4 tests named as new guards for the PR's own measured failures, plus the_original_audiorouter_still_routes_audio, all exist in liveaudiorouter_test.rs as named. The frontend Live-toggle/debounce path has no CI coverage — not unique to this PR (strom-frontend never appears in a cargo test --package line in .github/workflows/ci.yml), and the PR body already flags it.

Design record — States the rejected alternative (#737) and why.

Confidence: HIGH

[srperens]

Verdict: Approve — supersedes my 2026-09-01 v3 review (id 5079073686, COMMENTED) at head b920c5f2, which withheld approval only on Check (Linux) being pending. That run is now green, and the commits added since (4a240d4..784a58e) are backend-tested and don't reopen any row from that review.

Claims

Claim	Verdict	Evidence
Check (Linux)/Build (Linux)/WASM/API-contract all completed green at the new head	CONFIRMED	gh pr checks 740 at 784a58e: all pass except macOS/Windows skipping (expected — no push/PR trigger)
The new straight-through default is confined to liveaudiorouter and doesn't change builtin.audiorouter	CONFIRMED	backend/tests/liveaudiorouter_test.rs:976 — `fn the_original_audiorouter_keeps_its_silent_default()` builds the old block and asserts zero :xp_ elements
An explicitly-empty matrix (an operator closing every crosspoint) survives and isn't overwritten by the new default	CONFIRMED	backend/tests/liveaudiorouter_test.rs:957 — `fn an_explicitly_empty_routing_is_honoured_rather_than_defaulted()`
Tests covering the new code path actually execute in CI rather than skip	CONFIRMED	.github/workflows/ci.yml: Check (Linux) runs cargo test --package strom --features efp,nvidia with STROM_REQUIRE_GST_PLUGINS=1 (skip-to-failure) and a separate cargo test --package strom-types step, covering both new test files

Diagnosis — backend/src/blocks/builtin/liveaudiorouter.rs:305 now calls routing::default_routing() when routing_matrix is absent (a block just dropped in), while an explicit "{}" still parses to an empty matrix — the two "no routing" states are distinguished, which is the right invariant for the reason the tests name (an upgrade must not silently reopen a deliberately-closed router). types/src/routing.rs:182 builds the diagonal by zipping flattened (stream, channel) pairs and stops at the shorter side, which is the same behaviour the maintainer would get by hand-wiring 1:1 and cheap to reason about.

Minor, non-blocking — frontend/src/audiorouter.rs:588 comments that a folded input row "still shows whether it is routed at all, so nothing is hidden without a trace," but the row underneath emits out_ch_count empty ui.label("") calls with no such indicator, in both the normal and flipped grid renderers. The claim isn't in the PR body and doesn't affect functionality — flagging so the comment gets fixed or the indicator gets added, whichever was intended.

Blast radius — unchanged from my last review: SHARED (new branches in update_element_property/update_block_properties, both gated on the liveaudiorouter block/definition id), not reopened by this delta — default_routing and the frontend fold state are additive and touch no existing call site outside this block.

Tests & CI — green at 784a58e0 per the claim row above. WASM frontend changes (knob rendering, fold state) have no UI-level test, consistent with strom-frontend never appearing in a cargo test --package line in CI — not new to this PR.

Confidence: HIGH