From ec74d9c35df2c1328e43c8c18b14e60ef4c4ad42 Mon Sep 17 00:00:00 2001 From: Grok Date: Sat, 5 Sep 2026 13:15:45 -0400 Subject: [PATCH 01/15] docs: design spec for the panelist feedback system Preshow tool for Zoom panels: per-panelist BS.1770 loudness metering, a shared subject detector, Tiles auto-framing, and a framing-advice return feed routed over the OBS Virtual Camera. Records the research that unblocked it: the return path needs no Zoom video-send entitlement, and libfacedetection (BSD-3, vendored as source, YuNet weights compiled in) satisfies the landmark requirement without a runtime dependency. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01WqDL6gcoVvEMXv7Qwcni9n --- .../2026-09-05-panelist-feedback-design.md | 296 ++++++++++++++++++ 1 file changed, 296 insertions(+) create mode 100644 docs/superpowers/specs/2026-09-05-panelist-feedback-design.md diff --git a/docs/superpowers/specs/2026-09-05-panelist-feedback-design.md b/docs/superpowers/specs/2026-09-05-panelist-feedback-design.md new file mode 100644 index 00000000..26c5bfda --- /dev/null +++ b/docs/superpowers/specs/2026-09-05-panelist-feedback-design.md @@ -0,0 +1,296 @@ +# Panelist Feedback System — Design + +**Date:** 2026-09-05 +**Status:** Draft — integration points pending codebase research +**Branch:** `feat/panelist-feedback` + +## Goal + +Give a producer running a Zoom panel two things Zoom does not provide: + +1. **Loudness consistency** — see, at a glance, whether every panelist is arriving at a comparable loudness, in real broadcast units. +2. **Framing self-correction** — return a feed into the meeting that tells the person currently on air that they are badly framed, so they fix it themselves. + +Plus an operator-side option to auto-frame Tiles. + +## Operating context: this is a PRESHOW tool + +It runs before the show is live — green room, mic check, rehearsal. Every +consequence below follows from that and should not be re-derived later. + +- **No audience exposure.** The only people in the meeting are the + production team and the panelists. A return feed everyone can see is the + *intended* delivery, not a leak. No spotlight, no private-video + workaround, no webinar-attendee risk. +- **No on-air risk.** Nothing here touches a live program shot, so + framing changes cannot damage a broadcast in progress. +- **Loudness is measured per check, not per session.** A panelist speaks + for roughly 20–60 s during a mic check. Integrated loudness must be + scoped to *that panelist's check window* and be resettable, or the + number is polluted by whoever spoke before them. Short-term (3 s) is the + live readout while they talk; integrated is the verdict once they stop. + At a 100 ms block hop, 20 s of speech yields ~200 gated blocks, which is + ample for the relative gate to be meaningful. +- **The meter is a readiness board.** One row per panelist: loudness + deviation from the panel reference, framing status, pass/fail. That is + the preshow-shaped presentation of the same measurements, and is the + primary UI for Subsystem 1. + +## Non-goals (YAGNI) + +- No automatic reframing of the on-air program shot. Directors do not want a shot that moves by itself. Auto-framing applies to Tiles only, as an opt-in, and to nothing else. +- No gaze, pose, emotion, or identity recognition. A face bounding box plus five landmarks is the entire perception budget for v1. +- No per-person private messaging. Zoom has no private video; the return feed is visible to everyone in the meeting. This is accepted, not worked around. +- No loudness *correction*. We measure and display. Auto-gain on a live panel is a separate, riskier product. + +## Why this is feasible now + +The idea was paused on 2026-09-02 because a framing tool cannot tolerate the 200–400 ms Zoom return path. That objection does not apply here: + +- **Framing advice is human-paced.** The panelist takes seconds to react. 400 ms of transport disappears into their reaction time. +- **Tiles auto-framing has no return path at all.** It is a local crop on video we already receive. +- **The return feed is an OBS scene on the OBS Virtual Camera.** It does not use `setExternalVideoSource`, so the unproven raw-video-send entitlement is not on the critical path. No new Zoom SDK surface at all. + +## Architecture + +Two independent engines, three consumers. + +**The plugin does not link the Zoom SDK.** Media crosses a process boundary +first — this is the real topology and every task must respect it: + +``` +Zoom SDK ─> ZoomObsEngine.exe ─> SHM ring ─> obs-zoom-plugin + (engine/src/engine-audio.cpp) (src/engine-ipc.h) +``` + +Within the plugin: + +``` +per-participant PCM ──> Loudness engine (BS.1770-4) ──> [1] Meter source (operator) + └─> [3] Return overlay (loudness half) + +per-participant video ──> Subject detector (box + ──> [2] Tiles auto-frame (operator) + 5 landmarks, 2–5 fps) └─> [3] Return overlay (framing half) +``` + +The engines share nothing. Either can ship without the other. The detector has two consumers and must therefore expose a stable, small result type. + +Note: `src/zoom-audio-delegate.*` and `src/zoom-audio-router.*` are orphaned +dead code — not in any CMake target. Do not build against them. + +--- + +### Subsystem 1 — Loudness engine + meter source + +Implements ITU-R BS.1770-4 (the measurement) and EBU R128 (the practice). + +**Filter chain, per channel:** + +1. Stage 1 — high-shelf "head" filter, roughly +4 dB above 1 kHz. +2. Stage 2 — RLB high-pass, roughly 38 Hz. + +**Sample rate must be treated as a runtime variable, not 48 kHz.** BS.1770-4 +publishes its biquad coefficients *only* for 48 kHz, but this codebase does +not receive a guaranteed rate: the engine reads `data->GetSampleRate()` per +buffer and stamps it into the ring header (`engine/src/engine-audio.cpp`), +and no constant anywhere asserts a rate for the receive path. Zoom commonly +delivers 32 kHz. Therefore the loudness engine **must** read +`ShmAudioHeader::sample_rate` at runtime and **derive** the filter +coefficients for that rate — hardcoding the published 48 kHz constants would +silently mis-weight every measurement. This is the single most likely way to +ship a meter that reads plausibly and is wrong. + +Input format is **16-bit signed, interleaved**; channel count is likewise +runtime-discovered. +3. Mean square per channel, then weighted sum: + `L = -0.691 + 10 * log10( Σ G_i * z_i )` + with `G = 1.0` for L/R/C, `1.41` for surrounds, `0` for LFE. Zoom participant audio is mono or stereo, so only `G = 1.0` terms apply. + +**Three measurements, all required:** + +| Measure | Window | Gating | Use here | +|---|---|---|---| +| Momentary (M) | 400 ms | none | live bar movement | +| Short-term (S) | 3 s | none | the number an operator reads | +| Integrated (I) | whole session | absolute −70 LUFS, then relative −10 LU | per-panelist consistency | + +**The gate is what makes this work.** A panelist is silent roughly 80% of a panel. An ungated integrated reading over that would be meaningless. The BS.1770 absolute gate at −70 LUFS discards silence for free — it is precisely the mechanism this use case needs, so we implement gating properly rather than shortcutting to a running average. Integration uses 400 ms blocks with 75% overlap (a new block every 100 ms). + +**The product insight: consistency is relative, not absolute.** + +The operator does not primarily care that a panelist hits −23 LUFS. They care that panelist A is not 6 LU louder than panelist B. The meter must therefore surface, per panelist: + +- short-term LUFS (absolute), and +- **deviation from the panel reference**, in LU, + +where the panel reference is the **median** integrated loudness across panelists who have cleared the absolute gate. Median, not mean, so one very loud or very quiet participant does not drag the reference. + +Reference targets offered as presets: EBU R128 `−23 LUFS`, ATSC A/85 `−24 LKFS`, streaming `−16 LUFS`. Default to *panel median* rather than a fixed target, because matching each other is the actual goal. + +**True peak** (dBTP, ≥4× oversampled) is deferred to v2 — it guards against clipping on distribution, which is not what this feature is for. + +--- + +### Subsystem 2 — Subject detector + +**Cadence is the core design decision.** Detection runs at **2–5 fps per monitored participant, on a downscaled frame (long edge ~320 px)** — not per frame, not at full resolution. Justification: + +- Framing advice is consumed by a human who reacts in seconds. +- Tiles auto-framing *must* be slow, or tiles visibly twitch. + +**We receive 1080p per participant**, so downscaling is a pure efficiency choice, not a quality compromise — a face in a 1080p source remains well-resolved at a 320 px long edge. Detection accuracy is not a constraint here; cost is. + +**Model requirement: the detector must return five landmarks** (both eyes, nose, two mouth corners), not just a box. Eyeline is the basis of every real framing rule, and approximating it from a box is a heuristic we do not need to accept. + +#### Library decision (spike complete, 2026-09-05) + +**Use libfacedetection (ShiqiYu), vendored as source.** + +- **License: 3-clause BSD**, verified from the upstream LICENSE file. No non-commercial clause, no field-of-use restriction. Obligation is a copyright notice in our attributions — nothing more. +- **It *is* YuNet.** Upstream converted the OpenCV Zoo YuNet ONNX model into static C arrays. So we get the leading model without the runtime that usually carries it. +- **Zero dependencies, and no external model asset.** Four files in `src/` (~500 KB, of which ~446 KB is weights compiled in as C arrays), plus a one-line export header we write. Nothing to install, path-resolve, or sign at runtime; no new DLL in the plugin folder. Upstream explicitly sanctions copying the sources into a host project. +- **Five landmarks including mouth corners** — the exact set we need. + +**Rejected, with reasons worth recording:** + +| Candidate | Verdict | +|---|---| +| YuNet via `cv::FaceDetectorYN` | Same model, but requires opencv_core + dnn + imgproc. Keep as documented fallback only. | +| BlazeFace / MediaPipe | No mouth corners (6 keypoints incl. ear tragions); no supported C++ desktop path (Bazel, TFLite); poor MSVC history. | +| **SCRFD / RetinaFace (InsightFace)** | **License blocker** — pretrained weights are non-commercial-research only. | +| **dlib 5/68-point** | **License blocker** — iBUG 300-W excludes commercial use. | +| Ultra-Light-Fast-Generic-1MB | MIT and tiny, but bbox only, no landmarks. | + +**Build notes that will otherwise cost a day:** input must be **BGR 3-channel** (not RGB, not YUV — a colour convert is required from whatever the video path hands us); verify `/arch:AVX2` actually reaches the compile line, since MSVC can silently accept AVX2 intrinsics under SSE2 and the resulting slowdown looks like a library fault; do not enable `/openmp` — we thread per-participant ourselves, which is what upstream recommends. + +**Keep the detector behind a narrow interface** (`frame → box + 5 points`) from day one, so the OpenCV-DNN fallback stays a contained swap if local benchmarking disappoints. + +#### Scheduling: cost is O(1) in panelist count, not O(n) + +Published cost is 13.09 ms for 320×240 on a 2017 i7-7820X; ~10 ms at our 320×180 is an **extrapolation, not a measurement**. Naively running 10 panelists at 5 fps would be ~0.5 of a core. + +We do not need that. **One detector worker thread services all participants on a round-robin schedule**, with a priority boost for the active speaker (who is the one being checked). One detection per ~100 ms cycled across the roster costs roughly a tenth of a core *regardless of how many panelists there are*, and every consumer here is slow enough not to notice: framing advice is human-paced, and per-tile crops are hysteretic by design. + +This also means panelist count never threatens the preshow tool's footprint — the cost ceiling is fixed by the schedule, not the roster. + +**Local benchmarking is still required** before finalizing the tick rate. Treat every figure above as a planning estimate. + +**Output type (stable, shared by both consumers):** + +```cpp +struct SubjectFrame { + bool found; + float box_x, box_y, box_w, box_h; // normalized 0..1 of source + float eye_l_x, eye_l_y, eye_r_x, eye_r_y; // normalized 0..1 + float confidence; + uint64_t detected_ns; // source frame timestamp +}; +``` + +**Temporal smoothing is mandatory, not optional.** Raw per-detection output jitters. Both consumers require a hysteretic, rate-limited signal: + +- **deadband** — ignore movement below a threshold, +- **rate limit** — cap crop travel per second, +- **dropout hold** — keep the last good result for N seconds before declaring "no subject", so a head turn does not read as "left frame." + +--- + +### Subsystem 3 — Consumers + +**3a. Tiles auto-frame (opt-in).** Produces a per-tile crop rect placing the eyeline on the upper-third line and centering the subject horizontally, clamped so the crop never exceeds source bounds and never zooms past a configured maximum. Off by default. + +**3b. Return scene + overlay.** An OBS scene showing the active speaker with framing advice, routed out via the OBS Virtual Camera and joined to the meeting by a separate Zoom client. + +Conditions detectable from a box plus landmarks, and the advice each maps to: + +| Condition | Advice | +|---|---| +| No face for longer than the hold period | "Step into frame" | +| Box touches a frame edge | "You're cut off — move right/left/down/up" | +| Eyeline well above the upper-third line | "Lower your camera" | +| Eyeline well below the upper-third line | "Raise your camera" | +| Box height below minimum fraction | "Move closer" | +| Box height above maximum fraction | "Move back" | +| Horizontal center off by more than tolerance | "Shift left/right" | + +**Overlay legibility is a hard constraint — and spotlight is not the fix.** The limit is not Zoom's re-encode; it is that a gallery tile occupies roughly 640×360 of *physical screen space* on a panelist's display however clean the encode is. Spotlighting the return would not solve this and is explicitly not required. Instead, design the overlay to read at gallery-tile size: chunky segments, oversized numerals, hard contrast, no hairlines, no fine text. The overlay must be designed and reviewed at 640×360, not at 1080p. + +--- + +## Integration facts (researched 2026-09-05 — build against these, not assumptions) + +### Audio + +- **Tap point:** `src/zoom-participant-audio-source.cpp:698-714`, where fully decoded per-participant int16 PCM sits immediately before publish. +- **Thread:** the dedicated **audio lane** thread (`m_audio_lane` in `ZoomEngineClient`). Never the IPC reader thread — it head-of-line-blocks every source; never the OBS audio-mixer thread, which is budget-critical. +- Media events are **coalescing prompts, not payloads** — integrate over the whole drain loop, not per wakeup. +- `roster()` deep-copies strings under a hot mutex. **Cache names; never call it per frame.** Use `add_roster_callback` for changes. +- Cross-thread readout should mirror the existing `g_sources_mtx` + `std::vector<...*>` registry in the same file. Lock order: `g_sources_mtx` before any `ctx->mtx`, never the reverse. + +### Video + +- **Format: I420 planar, BT.709, full range** (the engine normalises limited→full every frame). Not NV12, not BGRA. +- **CPU access is free** — the plugin already holds a plain I420 buffer (`feed->frame`). +- **libfacedetection requires BGR 3-channel.** The Y plane alone is *not* sufficient for this CNN, unlike a Haar/HOG detector. A cheap **I420 → BGR convert at the downscaled size** (~320×180) is required and must be budgeted. Do not feed replicated greyscale; U and V are right there. +- **Thread:** the frame callback runs on the **shared engine-IPC reader thread that serves every source in the plugin**. Detection must never run there. Tap a copy under `feed->mtx`, or the graphics-thread `TileScratch`, and hand it to the detector worker. +- `src/zoom-video-delegate.cpp` is **dead code** (no CMake target). Do not build on it. + +### Tiles crop — the single insertion point + +`src/zoom-supersource.cpp:1957`, the `CropRect crop = solve_slot_crop(...)` inside the `draw_tile` lambda. It is the only place the sampled rectangle is decided, and `crop_uv` plus the border/glow registration derive from it. Four constraints: + +1. Result aspect must be exactly `params.tile_aspect` or the tile letterboxes. +2. Must stay within `[0, tex_w] × [0, tex_h]`; zero width/height falls back to the placeholder. +3. `crop_uv` must be computed from the **truncated integers**, not the doubles, or borders misregister. +4. Framing state must live where `render_slot_crop` lives: authoritative copy under `ctx->mutex`, snapshotted once per frame at `:1286` so a framing pass lands as a unit. + +### Active speaker + +- Use `ZoomEngineClient::active_speaker_id()` — the *directed* id, post hold/dwell. Not the raw one. +- `SpeakerDirector::snapshot()` carries everything an overlay wants (candidate, hold remaining, manual override, excludes). +- **It is poll-only** — no observer list. Poll it; do not expect events. +- A dedicated **`corevideo_active_speaker_source` already exists** (`src/zoom-source.cpp:2875`) with hidden-preview handover. That is the natural host for the framing overlay. + +### Testing conventions (non-negotiable in this repo) + +- **No test framework.** Plain `int main()` with a local `check()`. Do not introduce gtest or Catch. +- Target `CoreVideoTest`, ctest name `CoreVideo`, registered by hand inside `if(BUILD_TESTING)` in the root `CMakeLists.txt`. No `tests/CMakeLists.txt`. +- **No headless GPU harness exists and one has been ruled against** (an offscreen Qt harness "certified it three times and was wrong three times"). The sanctioned approach: extract the decision into a pure header and unit-test that. `tests/tile-shape-test.cpp:189-240` reproduces the shader's crop arithmetic in plain C++ — **auto-framing crop math must be pinned exactly that way.** +- Build/run: `cmake --build build --config Release --parallel 8` then `ctest -C Release --output-on-failure`, N/N green. +- New pure logic goes in a **header-only file under `src/`** with a "why this exists" comment. This repo has never vendored a library; libfacedetection will be the first, and that is a deliberate exception justified by the license/dependency analysis above. + +## Deployment constraints + +- **Self-tile exclusion is mandatory, and is real work — no such code exists.** Confirmed by exhaustive search: `ParticipantInfo` has no self/me field, the engine builds the roster from `GetParticipantsList()` with zero filtering, and the bot therefore appears in every roster, picker, tile candidate set, and speaker candidate set. The only existing defence is a handful of operator-chosen exclude combo boxes keyed by a **meeting-scoped `user_id` that does not survive a rejoin**. Worse: **talkback deliberately unmutes the bot**, so during a talkback key the bot is a fully eligible active-speaker candidate. Adding a vcam return feed on top of this makes a real feedback loop likely, not hypothetical. This needs a durable self/return identity flag, not another combo box. + +### OPEN QUESTION — tile source resolution vs. auto-framing quality + +Tiles subscribes at **P360 by default** (`tile_feed_subscribe`, `zoom-supersource.cpp:475-480`), not 1080p. The rationale is recorded in place: on 2026-08-17 a 720p wall oversubscribed Zoom's raw-data envelope and throttled the entire meeting to 0.3–0.45× real time. The engine's policy is upgrade-only, so a participant already carried at 720p/1080p for a program output is shared at that higher resolution — but the default wall is 360p. + +Consequences, which differ per subsystem: + +- **Detection is unaffected.** We downscale to ~320 px long edge anyway; a 640×360 source is ample. +- **Auto-framed tile *quality* is affected.** Cropping into a 360p source and magnifying the result will look soft. Getting crisp auto-framed tiles means raising the subscription resolution — which walks straight back into the envelope that caused the 2026-08-17 throttle. + +This gates Subsystem 3a only, and 3a is last in the build order, so it does not block anything before it. Options: accept softness at 360p; raise resolution only for the small number of tiles actually being auto-framed; or restrict auto-framing to preshow, where a throttled meeting is survivable. +- **The vcam needs its own seat.** Some Zoom client must select OBS Virtual Camera as its webcam; that is a normal client, not our SDK identity. Prior findings warn that same-account joins collide, so this should be a separate account. +- Broadcast sample rate is 48 kHz. Resample, or recompute filter coefficients, if participant audio arrives at another rate. + +## Risks + +| Risk | Mitigation | +|---|---| +| Detector library license / size / perf unknown | **Spike gates the choice.** Measure before committing. | +| Detection cost scales with panelist count | Low cadence + downscale + monitor only participants actually shown | +| Return tile feedback loop | Explicit self-exclusion, with a test | +| Overlay unreadable after Zoom re-encode | Design and review at 640×360 | +| Integrated loudness meaningless for silent panelists | BS.1770 absolute gate handles this by design | +| Tiles auto-frame twitches | Deadband + rate limit + dropout hold | + +## Build order + +1. **Loudness engine + meter source** — no detector, no vcam, no spike. Ships alone and is independently valuable. +2. **Detector spike, then detector engine** — gates on library validation. +3. **Consumers** — Tiles auto-frame, then return scene + overlay. + +Each is a separate implementation plan producing working software on its own. From bd1b457e2c82d68814384850fb4ed1aa48d7eb87 Mon Sep 17 00:00:00 2001 From: Grok Date: Sat, 5 Sep 2026 13:25:56 -0400 Subject: [PATCH 02/15] docs: auto-framing rides the shared feed, never requests resolution Tiles subscribes at P360, but subscriptions are shared and upgrade-only: a participant already carried at 1080p for the active-speaker feed or an ISO output keeps that quality and the tile reuses it. So the panelists being checked already have real pixels for the crop math. Auto-framing therefore adds no subscription pressure and cannot reintroduce the 2026-08-17 meeting throttle. The existing ISO and program-output controls are the resolution lever; no new control needed. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01WqDL6gcoVvEMXv7Qwcni9n --- .../2026-09-05-panelist-feedback-design.md | 43 ++++++++++++++----- 1 file changed, 33 insertions(+), 10 deletions(-) diff --git a/docs/superpowers/specs/2026-09-05-panelist-feedback-design.md b/docs/superpowers/specs/2026-09-05-panelist-feedback-design.md index 26c5bfda..ef1004a4 100644 --- a/docs/superpowers/specs/2026-09-05-panelist-feedback-design.md +++ b/docs/superpowers/specs/2026-09-05-panelist-feedback-design.md @@ -263,16 +263,39 @@ Conditions detectable from a box plus landmarks, and the advice each maps to: - **Self-tile exclusion is mandatory, and is real work — no such code exists.** Confirmed by exhaustive search: `ParticipantInfo` has no self/me field, the engine builds the roster from `GetParticipantsList()` with zero filtering, and the bot therefore appears in every roster, picker, tile candidate set, and speaker candidate set. The only existing defence is a handful of operator-chosen exclude combo boxes keyed by a **meeting-scoped `user_id` that does not survive a rejoin**. Worse: **talkback deliberately unmutes the bot**, so during a talkback key the bot is a fully eligible active-speaker candidate. Adding a vcam return feed on top of this makes a real feedback loop likely, not hypothetical. This needs a durable self/return identity flag, not another combo box. -### OPEN QUESTION — tile source resolution vs. auto-framing quality - -Tiles subscribes at **P360 by default** (`tile_feed_subscribe`, `zoom-supersource.cpp:475-480`), not 1080p. The rationale is recorded in place: on 2026-08-17 a 720p wall oversubscribed Zoom's raw-data envelope and throttled the entire meeting to 0.3–0.45× real time. The engine's policy is upgrade-only, so a participant already carried at 720p/1080p for a program output is shared at that higher resolution — but the default wall is 360p. - -Consequences, which differ per subsystem: - -- **Detection is unaffected.** We downscale to ~320 px long edge anyway; a 640×360 source is ample. -- **Auto-framed tile *quality* is affected.** Cropping into a 360p source and magnifying the result will look soft. Getting crisp auto-framed tiles means raising the subscription resolution — which walks straight back into the envelope that caused the 2026-08-17 throttle. - -This gates Subsystem 3a only, and 3a is last in the build order, so it does not block anything before it. Options: accept softness at 360p; raise resolution only for the small number of tiles actually being auto-framed; or restrict auto-framing to preshow, where a throttled meeting is survivable. +### Tile source resolution — resolved: auto-framing never requests resolution + +Tiles subscribes at **P360 by default** (`tile_feed_subscribe`, +`zoom-supersource.cpp:475-480`), and that default exists for a live reason: +on 2026-08-17 a 720p wall oversubscribed Zoom's raw-data envelope and +throttled the entire meeting to 0.3–0.45× real time. + +**This does not constrain auto-framing, because subscriptions are shared and +upgrade-only.** From the rationale comment at `zoom-supersource.cpp:459`: + +> *"Shared feeds are NOT degraded by this. The engine holds one subscription +> per participant with an upgrade-only resolution policy... So a participant +> already on a 720p/1080p program output keeps that quality, and the tile +> simply reuses the high-quality feed. 360p is paid only for participants +> the wall alone is showing."* + +In the real production configuration the panelists that matter are already +carried at high resolution — the **active speaker feed pulls 1080p**, and any +**ISO'd participant has a constant high-res feed**. Tiles reuses those feeds +for free, so the crop math has real pixels for exactly the people being +checked. + +**Design rule: auto-framing consumes whatever the shared feed already is and +must never request a resolution upgrade.** Consequences: + +- No new subscription pressure, so the 2026-08-17 throttle cannot be + reintroduced by this feature. +- No new operator control is needed. The existing ISO / program-output + controls *are* the resolution lever, and they already encode operator + intent about who matters. +- A tile that looks soft under auto-framing is a participant who is neither + ISO'd nor active speaker. The fix is to ISO them — an action the operator + already has, for reasons they already understand. - **The vcam needs its own seat.** Some Zoom client must select OBS Virtual Camera as its webcam; that is a normal client, not our SDK identity. Prior findings warn that same-account joins collide, so this should be a separate account. - Broadcast sample rate is 48 kHz. Resample, or recompute filter coefficients, if participant audio arrives at another rate. From 8e202bc8d1bdd18f2e089d5bb8a53159fad1cdb6 Mon Sep 17 00:00:00 2001 From: Grok Date: Sat, 5 Sep 2026 13:46:36 -0400 Subject: [PATCH 03/15] docs: implementation plans for the panelist feedback system Three plans, one per independently shippable subsystem: loudness engine + readiness board (7 tasks) subject detector (8 tasks, benchmark-gated) framing consumers + self-exclusion (11 tasks) Cross-plan fix: SubjectFrame was defined by both the detector and the consumers plan with different member defaults. src/subject-frame.h is now the single definition and zoom-subject-source.h includes it. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01WqDL6gcoVvEMXv7Qwcni9n --- .../plans/2026-09-05-framing-consumers.md | 3048 ++++++++++++ .../2026-09-05-loudness-engine-and-meter.md | 2837 +++++++++++ .../plans/2026-09-05-subject-detector.md | 4185 +++++++++++++++++ 3 files changed, 10070 insertions(+) create mode 100644 docs/superpowers/plans/2026-09-05-framing-consumers.md create mode 100644 docs/superpowers/plans/2026-09-05-loudness-engine-and-meter.md create mode 100644 docs/superpowers/plans/2026-09-05-subject-detector.md diff --git a/docs/superpowers/plans/2026-09-05-framing-consumers.md b/docs/superpowers/plans/2026-09-05-framing-consumers.md new file mode 100644 index 00000000..c836f9fc --- /dev/null +++ b/docs/superpowers/plans/2026-09-05-framing-consumers.md @@ -0,0 +1,3048 @@ +# Framing Consumers (Subsystem 3) Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Build the two consumers of the subject detector — opt-in per-tile auto-framing on the Tiles wall, and a geometric framing-advice overlay for the Virtual-Camera return feed — on top of a durable self/return-identity exclusion that stops the return feed feeding itself. + +**Architecture:** Three pure header-only decision units (`zoom-self-identity.h`, `zoom-auto-frame.h`, `zoom-framing-advice.h` + `zoom-framing-overlay-layout.h`) hold every rule, and are exhaustively unit-tested on the CPU. Two thin OBS integrations consume them: the existing Tiles source gains an auto-frame path that replaces the operator's slot crop at the single existing crop insertion point (`src/zoom-supersource.cpp:1957`), and a new custom-draw source `corevideo_framing_overlay` draws solid-quad symbols over the active speaker in the return scene. The detector is reached through a one-function provider seam (`src/zoom-subject-source.h`) that defaults to "no subject", so all of this builds, ships and tests before the detector engine exists. + +**Tech Stack:** C++17, libobs (custom-draw sources, `obs_data` settings, `obs_get_base_effect(OBS_EFFECT_SOLID)`), Qt only where already present, CMake + CTest with plain `int main()` tests. + +**Spec:** `docs/superpowers/specs/2026-09-05-panelist-feedback-design.md` + +--- + +## Resolution rule — read before designing anything here + +**Auto-framing consumes whatever resolution the shared feed already is, and must NEVER request a resolution upgrade.** This feature adds **no new subscription pressure at all** and therefore cannot reintroduce the 2026-08-17 throttle (a 720p wall oversubscribed Zoom's raw-data envelope and took a live meeting to 0.3–0.45× real time). + +The rationale is already recorded in place, at `src/zoom-supersource.cpp:459`, and this plan builds on it verbatim: + +> "Shared feeds are NOT degraded by this. The engine holds one subscription per participant with an upgrade-only resolution policy (`EngineVideo::subscribe`: a request at or below the active resolution attaches to the existing feed as a no-op). So a participant already on a 720p/1080p program output keeps that quality, and the tile simply reuses the high-quality feed. 360p is paid only for participants the wall alone is showing." + +Consequences, all of which the plan reflects: + +1. **No new subscription pressure.** Nothing in this plan calls `subscribe()`, changes `tile_feed_subscribe`, or touches a `VideoResolution`. +2. **No new resolution control.** The existing ISO and program-output controls **are** the resolution lever, and they already encode operator intent. Adding a second one would let two mechanisms fight over the same envelope. +3. **A soft auto-framed tile is a diagnosis, not a defect:** that participant is neither ISO'd nor the active speaker, so nobody is paying for their pixels. The remedy is to ISO them, which the operator can already do. Do not engineer around this. +4. **The crop math must be resolution-invariant.** It reads `tex_w`/`tex_h` from the feed and never assumes a value; the same subject at 640×360 and at 1920×1080 must produce the *same normalized* crop. Task 6 pins exactly that. + +In the real production configuration the active speaker feed pulls 1080p and any ISO'd participant has a constant high-resolution feed, and Tiles reuses both automatically — so the panelists actually being checked already have real pixels for the crop math. + +Detection itself is unaffected either way: the detector downscales to a ~320 px long edge, and a 640×360 source is ample for that. + +**Do not conflate this with the 640×360 overlay requirement.** That one is about the physical screen space a gallery tile occupies on a panelist's display, and is unrelated to subscription resolution. + +## Global Constraints + +- **Consumed detector contract — use these exact field names, do not redefine them:** + ```cpp + struct SubjectFrame { + bool found; + float box_x, box_y, box_w, box_h; // normalized 0..1 of source + float eye_l_x, eye_l_y, eye_r_x, eye_r_y; // normalized 0..1 + float confidence; + uint64_t detected_ns; + }; + ``` +- **Smoothing is already done by the detector** (deadband, rate limit, dropout hold). Never re-implement it here. `found == false` already means "held past the dropout window", so it can be trusted directly as "no subject". +- **No test framework.** Plain `int main()` with a local `check()`. Never gtest, never Catch. +- Test target `CoreVideoTest`, ctest name `CoreVideo`, hand-registered in the root `CMakeLists.txt` inside `if(BUILD_TESTING)`. There is no `tests/CMakeLists.txt`. +- **No headless GPU harness exists and one has been ruled against** ("an offscreen Qt harness certified it three times and was wrong three times"). Every decision must be extracted into a pure header and unit-tested there. For in-product visual checks use the existing env-var pattern (`COREVIDEO_TALKBACK_LAYOUT_TEST`, `src/zoom-talkback-panel.cpp:95`). +- **Settings use the per-source `obs_data` + `std::atomic` pattern** (see `PROP_ANIMATE` end-to-end: declared `src/zoom-supersource.cpp:2247`, defaulted `:3115`, property `:3360`, stored to an atomic `:2736`, read once per frame in render). **Never** the global-ini `ZoomPluginSettings` mechanism. +- **`SpeakerDirector` is poll-only** — there is no observer list. Poll `snapshot(now_ms)` / `directed_speaker_id()`. +- New pure logic goes in a **header-only file under `src/`** with a "why this exists" comment at the top. +- Build/run: `cmake --build build --config Release --parallel 8` then `ctest -C Release --output-on-failure`, N/N green. +- Auto-frame is **OFF by default**. The overlay source is inert until placed in a scene. +- **Nothing in this plan requests a resolution upgrade.** No task calls `subscribe()`, edits `tile_feed_subscribe`, or names a `VideoResolution`. See the resolution rule above. +- **Spotlight is explicitly NOT required** and must not be assumed anywhere. +- Update `CLAUDE.md` in the same change as substantive work (standing repo directive); Task 11 covers the final pass. + +## The four crop-insertion-point constraints (each gets a test) + +`src/zoom-supersource.cpp:1957` is the **only** place the sampled rectangle is decided. Every task that touches it must hold: + +1. Result aspect must equal `params.tile_aspect` exactly, or the tile letterboxes. +2. The rect must stay inside `[0, tex_w] × [0, tex_h]`; zero width/height falls back to the placeholder. +3. `crop_uv` must be computed from the **truncated integers** actually handed to `gs_draw_sprite_subregion()`, not from the doubles, or borders misregister. +4. Framing state must be snapshotted under `ctx->mutex` alongside `render_slot_crop` (`:1286`) so a framing pass lands as a unit. + +## File Structure + +**Created:** +- `src/zoom-self-identity.h` — pure: who is us. Name matching, roster flagging, the return-seat registry. +- `src/zoom-subject-source.h` — the detector seam: one installable provider returning a smoothed `SubjectFrame` for a participant id. Defaults to "not found". +- `src/zoom-auto-frame.h` — pure: `SubjectFrame` + source size + tile aspect → an exact-aspect, in-bounds `CropRect`. +- `src/zoom-framing-advice.h` — pure: `SubjectFrame` + thresholds + previous condition → one `FramingCondition`. +- `src/zoom-framing-overlay-layout.h` — pure: `FramingCondition` + canvas size → a list of solid quads, legible at 640×360. +- `src/zoom-framing-overlay.h` / `src/zoom-framing-overlay.cpp` — the `corevideo_framing_overlay` OBS source. +- `tests/self-identity-test.cpp`, `tests/subject-source-test.cpp`, `tests/auto-frame-test.cpp`, `tests/framing-advice-test.cpp`, `tests/framing-overlay-layout-test.cpp`. + +**Modified:** +- `src/zoom-types.h` — two bools on `ParticipantInfo`. +- `engine/src/main.cpp` — stamp `is_self` from `GetMySelfUser()`, emit it. +- `src/zoom-engine-client.cpp` — parse `is_self`, apply identity flags before the director sees the roster. +- `src/zoom-tile-fill.h` — identity-flagged participants are never eligible. +- `src/speaker-director.cpp` / `.h` — identity-flagged participants are excluded and dethroned. +- `src/zoom-supersource.cpp` — auto-frame properties/atomics/tick/snapshot and the crop insertion point. +- `src/zoom-plugin.cpp` — register the overlay source. +- `data/locale/en-US.ini` — new strings. +- `CMakeLists.txt` — five new test targets plus the new plugin source. +- `tests/tile-fill-test.cpp`, `tests/speaker-director-test.cpp` — exclusion coverage. +- `CLAUDE.md`, `CHANGELOG.md`. + +--- + +## Task 1: Durable identity — the pure rules + +**Why first:** without it the vcam return feed is a real feedback loop, not a hypothetical one. No such code exists today: `ParticipantInfo` has no self field, the engine builds the roster from `GetParticipantsList()` with zero filtering, and the bot therefore appears in every roster, picker, tile candidate set and speaker candidate set. The only current defence is operator combo boxes keyed by a meeting-scoped `user_id` that does **not** survive a rejoin. Worse, talkback deliberately unmutes the bot, making it a fully eligible active-speaker candidate. + +**Files:** +- Create: `src/zoom-self-identity.h` +- Create: `tests/self-identity-test.cpp` +- Modify: `src/zoom-types.h:43-56` +- Modify: `CMakeLists.txt` (inside `if(BUILD_TESTING)`) + +**Interfaces:** +- Consumes: `ParticipantInfo` from `src/zoom-types.h`. +- Produces: `ParticipantInfo::is_self`, `ParticipantInfo::is_return_identity`; `normalize_identity_name()`, `ReturnIdentityRegistry::instance()`, `apply_identity_flags()`. + +- [ ] **Step 1: Add the two fields to `ParticipantInfo`** + +In `src/zoom-types.h`, inside `struct ParticipantInfo` after `is_sharing_screen`: + +```cpp + // True if this entry is the SDK identity this plugin joined as — the bot + // itself. Stamped by the engine from GetMySelfUser() on every roster + // rebuild, so it is re-derived after a rejoin even though user_id changes. + // Every consumer that picks somebody to show must refuse this. + bool is_self = false; + // True if this entry is the separate Zoom client carrying the OBS Virtual + // Camera return feed. Matched by display name (ReturnIdentityRegistry), + // because that seat is a different account whose user_id also changes on + // every rejoin. Showing it on the wall, or directing to it, closes a + // video feedback loop. + bool is_return_identity = false; +``` + +- [ ] **Step 2: Write the failing test** + +Create `tests/self-identity-test.cpp`: + +```cpp +// Durable self / return-identity flagging. Pure rules only: name +// normalization, registry round-trip, and the roster-flagging pass that both +// consumers depend on. +#include "zoom-self-identity.h" + +#include +#include +#include + +static int g_failures = 0; + +static void check(bool ok, const std::string &what) +{ + if (!ok) { + std::cerr << "FAIL: " << what << "\n"; + ++g_failures; + } +} + +static ParticipantInfo person(uint32_t id, const char *name) +{ + ParticipantInfo p; + p.user_id = id; + p.display_name = name; + p.has_video = true; + return p; +} + +int main() +{ + // Normalization: case, surrounding space, and interior runs of space all + // collapse, because an operator types "Return Feed" and Zoom shows + // "return feed " after a rename. + check(normalize_identity_name(" Return FEED ") == "return feed", + "normalize collapses case and space"); + check(normalize_identity_name("") == "", "normalize of empty is empty"); + check(normalize_identity_name(" ") == "", "normalize of blank is empty"); + + // Registry round-trip. Empty entries are dropped so a blank property + // never matches every unnamed participant. + ReturnIdentityRegistry::instance().set_names({"Return Feed", " ", ""}); + const std::vector names = + ReturnIdentityRegistry::instance().names(); + check(names.size() == 1, "blank return-seat names are dropped"); + check(names.size() == 1 && names[0] == "return feed", + "registry stores the normalized form"); + + // Flagging: engine-reported self stays set, the return seat is matched by + // name, and everyone else is untouched. + std::vector roster = { + person(11, "CoreVideo Bot"), + person(22, "return feed"), + person(33, "Dr. Panelist"), + }; + roster[0].is_self = true; // as the engine stamped it + apply_identity_flags(roster); + + check(roster[0].is_self && !roster[0].is_return_identity, + "engine-stamped self survives the pass"); + check(roster[1].is_return_identity && !roster[1].is_self, + "the return seat is flagged by name"); + check(!roster[2].is_self && !roster[2].is_return_identity, + "a panelist is flagged as neither"); + + // The pass is idempotent and it CLEARS a stale return flag when the + // operator renames the seat — otherwise a mistyped name would poison the + // roster until the next rejoin. + ReturnIdentityRegistry::instance().set_names({"Different Seat"}); + apply_identity_flags(roster); + check(!roster[1].is_return_identity, + "renaming the return seat clears the old flag"); + check(roster[0].is_self, "the self flag is never cleared by this pass"); + + // With no names configured nothing but engine-reported self is flagged. + ReturnIdentityRegistry::instance().set_names({}); + apply_identity_flags(roster); + check(!roster[0].is_return_identity && !roster[1].is_return_identity && + !roster[2].is_return_identity, + "an empty registry flags no return identity"); + + // A participant may be both: the operator pointed the return seat at the + // bot's own name. Both flags stand; every consumer refuses on either. + ReturnIdentityRegistry::instance().set_names({"CoreVideo Bot"}); + apply_identity_flags(roster); + check(roster[0].is_self && roster[0].is_return_identity, + "self and return identity can coexist"); + + check(identity_excluded(roster[0]), "self is identity-excluded"); + check(!identity_excluded(roster[2]), "a panelist is not identity-excluded"); + + if (g_failures == 0) std::cout << "self-identity: all checks passed\n"; + return g_failures == 0 ? 0 : 1; +} +``` + +- [ ] **Step 3: Register the test target and run it to verify it fails** + +In `CMakeLists.txt`, inside `if(BUILD_TESTING)` (put it directly after the `CoreVideoSpeakerDirectorTest` block, around line 663): + +```cmake + # Durable self / return-identity flagging. The vcam return feed is a real + # feedback loop without this, so the rules are pinned rather than trusted. + add_executable(CoreVideoSelfIdentityTest + tests/self-identity-test.cpp + ) + target_include_directories(CoreVideoSelfIdentityTest PRIVATE + "${CMAKE_CURRENT_SOURCE_DIR}/src" + ) + add_test(NAME CoreVideoSelfIdentity + COMMAND CoreVideoSelfIdentityTest) +``` + +Run: `cmake --build build --config Release --parallel 8 --target CoreVideoSelfIdentityTest` +Expected: FAIL — `Cannot open include file: 'zoom-self-identity.h'` + +- [ ] **Step 4: Write the header** + +Create `src/zoom-self-identity.h`: + +```cpp +#pragma once + +// Why this exists +// --------------- +// Nothing in this plugin used to know which roster entry is US. The engine +// builds its roster from GetParticipantsList() with no filtering, so the bot +// appears in every roster, every picker, every tile candidate set and every +// active-speaker candidate set. The only defence was a handful of operator +// exclude combo boxes keyed by a MEETING-SCOPED user_id, which does not +// survive a rejoin. Talkback then makes it worse on purpose: it unmutes the +// bot, so during a talkback key the bot is a fully eligible speaker. +// +// Add a Virtual-Camera return feed on top of that and you get a real video +// feedback loop, not a hypothetical one. So identity is derived durably: +// +// * is_self - stamped by the engine from GetMySelfUser() on every +// roster rebuild, so it is re-derived after a rejoin. +// * is_return_identity - matched by DISPLAY NAME against operator-configured +// names, because the return seat is a separate Zoom +// account whose user_id also changes on every rejoin. +// +// Both are recomputed from scratch on every roster message. Neither is a +// remembered id, which is precisely what made the old combo boxes fail. + +#include "zoom-types.h" + +#include +#include +#include +#include +#include + +// Lower-cases, trims, and collapses interior whitespace runs to one space. +// Operators type "Return Feed"; Zoom renders "return feed " after a rename. +inline std::string normalize_identity_name(const std::string &raw) +{ + std::string out; + out.reserve(raw.size()); + bool pending_space = false; + for (const char c : raw) { + const unsigned char uc = static_cast(c); + if (std::isspace(uc)) { + pending_space = !out.empty(); + continue; + } + if (pending_space) { + out.push_back(' '); + pending_space = false; + } + out.push_back(static_cast(std::tolower(uc))); + } + return out; +} + +// The display names of the Zoom seats carrying our own return feed. +// +// A process-wide registry rather than a parameter because the roster arrives +// on the engine-client reader thread, while the name is owned by the overlay +// source's per-source settings. Written rarely (a settings change), read on +// every roster message, so a plain mutex is right — this is not a hot path. +class ReturnIdentityRegistry { +public: + static ReturnIdentityRegistry &instance() + { + static ReturnIdentityRegistry s_instance; + return s_instance; + } + + // Stores the normalized form. Empty and whitespace-only entries are + // dropped: a blank property must not match every unnamed participant. + void set_names(const std::vector &names) + { + std::vector normalized; + normalized.reserve(names.size()); + for (const std::string &n : names) { + std::string norm = normalize_identity_name(n); + if (norm.empty()) continue; + if (std::find(normalized.begin(), normalized.end(), norm) != + normalized.end()) + continue; + normalized.push_back(std::move(norm)); + } + std::lock_guard lock(m_mtx); + m_names = std::move(normalized); + } + + std::vector names() const + { + std::lock_guard lock(m_mtx); + return m_names; + } + +private: + ReturnIdentityRegistry() = default; + mutable std::mutex m_mtx; + std::vector m_names; +}; + +// Recomputes is_return_identity for every entry from the current registry. +// +// is_self is NEVER touched: the engine owns it and has better information +// (GetMySelfUser()) than any name comparison could. is_return_identity is +// always overwritten, including back to false, so retyping the seat name +// releases the participant it used to match instead of poisoning the roster +// until the next rejoin. +inline void apply_identity_flags(std::vector &roster) +{ + const std::vector names = + ReturnIdentityRegistry::instance().names(); + for (ParticipantInfo &p : roster) { + if (names.empty()) { + p.is_return_identity = false; + continue; + } + const std::string norm = normalize_identity_name(p.display_name); + p.is_return_identity = + !norm.empty() && + std::find(names.begin(), names.end(), norm) != names.end(); + } +} + +// The one predicate every consumer asks. Kept as a function so a future third +// identity kind lands in one place rather than in each consumer's condition. +inline bool identity_excluded(const ParticipantInfo &p) +{ + return p.is_self || p.is_return_identity; +} +``` + +- [ ] **Step 5: Run the test to verify it passes** + +Run: +``` +cmake --build build --config Release --parallel 8 --target CoreVideoSelfIdentityTest +ctest -C Release -R CoreVideoSelfIdentity --output-on-failure +``` +Expected: PASS, `self-identity: all checks passed` + +- [ ] **Step 6: Commit** + +```bash +git add src/zoom-self-identity.h src/zoom-types.h tests/self-identity-test.cpp CMakeLists.txt +git commit -m "feat: durable self/return-identity rules for framing consumers" +``` + +--- + +## Task 2: Wire identity end to end (engine stamps it, plugin applies it) + +**Files:** +- Modify: `engine/src/main.cpp:600-680` (`user_to_info`, `send_roster`) +- Modify: `src/zoom-engine-client.cpp:1774-1800` (the `"participants"` handler) +- Modify: `tests/self-identity-test.cpp` + +**Interfaces:** +- Consumes: `apply_identity_flags()`, `ParticipantInfo::is_self` (Task 1). +- Produces: a roster in which `is_self`/`is_return_identity` are true before `SpeakerDirector::update_roster()` or any tile resolve sees it. + +- [ ] **Step 1: Write the failing test — flags must be applied before the director is fed** + +Append to `tests/self-identity-test.cpp`, immediately before the final `if (g_failures == 0)` block: + +```cpp + // Ordering contract with zoom-engine-client.cpp: flags are applied to the + // whole roster BEFORE it is handed on. Anything downstream that filters on + // identity therefore never sees an unflagged roster, not even for one + // message. This reproduces that sequence. + { + ReturnIdentityRegistry::instance().set_names({"Return Feed"}); + std::vector incoming = { + person(101, "CoreVideo Bot"), + person(102, "Return Feed"), + person(103, "Panelist"), + }; + incoming[0].is_self = true; + + apply_identity_flags(incoming); + + std::vector handed_on; + for (const ParticipantInfo &p : incoming) + if (!identity_excluded(p)) handed_on.push_back(p); + + check(handed_on.size() == 1 && handed_on[0].user_id == 103, + "only the panelist survives the identity filter"); + } +``` + +- [ ] **Step 2: Run it to verify it fails** + +Run: +``` +cmake --build build --config Release --parallel 8 --target CoreVideoSelfIdentityTest +ctest -C Release -R CoreVideoSelfIdentity --output-on-failure +``` +Expected: FAIL — `only the panelist survives the identity filter` (the registry was left holding `CoreVideo Bot` by the earlier block, so `Return Feed` is not matched). + +Fix by making the new block's `set_names` authoritative — it already is; if the check passes immediately, confirm by temporarily commenting out the `apply_identity_flags(incoming);` line, re-running to see the failure, and restoring it. + +- [ ] **Step 3: Stamp `is_self` in the engine** + +In `engine/src/main.cpp`, in `user_to_info()`, after `info.is_sharing_screen = ...`: + +```cpp + // Which roster entry is US. GetMySelfUser() is the participants + // controller's own answer, re-asked on every rebuild, so this is + // correct again immediately after a rejoin even though user_id + // changed. The plugin cannot derive this: it never sees the SDK. + if (m_ctrl) { + ZOOMSDK::IUserInfo *self = m_ctrl->GetMySelfUser(); + info.is_self = self && self->GetUserID() == info.user_id; + } +``` + +In `send_roster()`, extend the per-participant JSON — add `is_self` to the concatenation, immediately after the `is_sharing_screen` term and before the closing `"}"`: + +```cpp + R"(,"is_self":)" + (p.is_self ? "true" : "false") + "}"; +``` + +(i.e. the `is_sharing_screen` line loses its trailing `+ "}"`, which moves to this new line.) + +- [ ] **Step 4: Parse and apply it in the plugin** + +In `src/zoom-engine-client.cpp`, in the `cmd == "participants"` handler, after `p.is_sharing_screen = po.value("is_sharing_screen").toBool();`: + +```cpp + p.is_self = po.value("is_self").toBool(); +``` + +Then, immediately before the `SpeakerDirector::instance().update_roster(...)` call in that same lambda: + +```cpp + // Identity flags are applied to the WHOLE roster before anything + // downstream sees it, so no consumer ever observes an unflagged + // roster — not even for one message. See zoom-self-identity.h for + // why this is derived per message instead of remembered by id. + apply_identity_flags(m_roster); +``` + +Do the same in the `cmd == "active_speaker"` handler, before its `update_roster(...)` call, so a rename arriving between roster messages cannot leave a stale flag. + +Add the include near the other project includes at the top of the file: + +```cpp +#include "zoom-self-identity.h" +``` + +- [ ] **Step 5: Run the tests and build the plugin** + +Run: +``` +cmake --build build --config Release --parallel 8 +ctest -C Release --output-on-failure +``` +Expected: PASS, N/N green. + +- [ ] **Step 6: Commit** + +```bash +git add engine/src/main.cpp src/zoom-engine-client.cpp tests/self-identity-test.cpp +git commit -m "feat: engine stamps is_self; plugin flags identity on every roster" +``` + +--- + +## Task 3: Tile assignment refuses our own identities + +**Files:** +- Modify: `src/zoom-tile-fill.h` (the `eligible` lambda) +- Modify: `tests/tile-fill-test.cpp` + +**Interfaces:** +- Consumes: `identity_excluded()` (Task 1). +- Produces: `resolve_tile_assignments()` that can never place us on the wall in either mode. + +- [ ] **Step 1: Write the failing test** + +Append to `tests/tile-fill-test.cpp`, before its final return: + +```cpp + // Identity exclusion. The bot and the Virtual-Camera return seat are never + // shown on the wall, in EITHER mode: a Manual cast is an operator decision + // the resolver otherwise honours unconditionally, but casting the return + // feed onto the wall that feeds it is a video feedback loop, so this one + // operator decision is overruled. + { + std::vector roster(3); + roster[0].user_id = 1; roster[0].has_video = true; + roster[0].is_self = true; + roster[1].user_id = 2; roster[1].has_video = true; + roster[1].is_return_identity = true; + roster[2].user_id = 3; roster[2].has_video = true; + + TileFillParams params; + params.mode = TileFillMode::Auto; + params.max_tiles = 9; + const std::vector auto_out = + resolve_tile_assignments({}, roster, params); + check(auto_out.size() == 1 && auto_out[0] == 3, + "Auto shows only the panelist"); + + // An incumbent that becomes identity-excluded is dropped, not kept. + const std::vector after = + resolve_tile_assignments({1, 3}, roster, params); + check(after.size() == 1 && after[0] == 3, + "an identity-excluded incumbent loses its slot"); + + params.mode = TileFillMode::Manual; + params.manual = {1, 2, 3}; + const std::vector manual_out = + resolve_tile_assignments({}, roster, params); + check(manual_out.size() == 1 && manual_out[0] == 3, + "Manual refuses a cast of our own identities"); + } +``` + +If `tests/tile-fill-test.cpp` has no `check()` helper of that shape, add the same local helper this plan uses in Task 1 at the top of the file and route the new block through it. + +- [ ] **Step 2: Run it to verify it fails** + +Run: +``` +cmake --build build --config Release --parallel 8 --target CoreVideoTileFillTest +ctest -C Release -R CoreVideoTileFill --output-on-failure +``` +Expected: FAIL — `Auto shows only the panelist` (all three are returned today). + +- [ ] **Step 3: Implement** + +In `src/zoom-tile-fill.h`, add the include below the existing `#include "zoom-types.h"`: + +```cpp +#include "zoom-self-identity.h" +``` + +Change the `eligible` lambda's tail from + +```cpp + return it != roster.end() && it->has_video; +``` + +to + +```cpp + // Identity exclusion is checked here, not in params.excluded, because + // params.excluded is operator-chosen and meeting-scoped while these + // flags are re-derived on every roster message and survive a rejoin. + return it != roster.end() && it->has_video && !identity_excluded(*it); +``` + +Then, in the Manual branch, replace the loop + +```cpp + for (const uint32_t id : params.manual) push_unique(id); +``` + +with + +```cpp + // The roster is deliberately not consulted for presence — an operator + // who cast a tile keeps it even with the camera off — but it IS + // consulted for identity: casting our own return feed onto the wall + // that feeds it closes a video loop, and that overrules the cast. + for (const uint32_t id : params.manual) { + const auto it = std::find_if( + roster.begin(), roster.end(), + [id](const ParticipantInfo &p) { return p.user_id == id; }); + if (it != roster.end() && identity_excluded(*it)) continue; + push_unique(id); + } +``` + +- [ ] **Step 4: Run the test to verify it passes** + +Run: +``` +cmake --build build --config Release --parallel 8 --target CoreVideoTileFillTest +ctest -C Release -R CoreVideoTileFill --output-on-failure +``` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add src/zoom-tile-fill.h tests/tile-fill-test.cpp +git commit -m "fix: tile assignment never places our own identities on the wall" +``` + +--- + +## Task 4: Speaker direction refuses our own identities + +**Files:** +- Modify: `src/speaker-director.cpp:72-77` (`participant_excluded_locked`) +- Modify: `tests/speaker-director-test.cpp` + +**Interfaces:** +- Consumes: `identity_excluded()` (Task 1). +- Produces: a `SpeakerDirector` that never promotes, and actively dethrones, an identity-excluded participant — including while talkback has the bot unmuted. + +- [ ] **Step 1: Write the failing test** + +Append to `tests/speaker-director-test.cpp`, before its final return: + +```cpp + // Identity exclusion, which is stronger than the operator exclude list: + // it is re-derived from the roster on every message, so it survives a + // rejoin, and it holds while talkback has the bot unmuted and talking — + // the exact window in which the bot is otherwise a perfectly eligible + // candidate. + { + SpeakerDirector &d = SpeakerDirector::instance(); + d.reset(); + d.configure(/*sensitivity_ms=*/0, /*hold_ms=*/0, /*require_video=*/false, + /*excluded=*/{}); + + std::vector roster(2); + roster[0].user_id = 1; + roster[0].has_video = true; + roster[0].is_talking = true; // talkback unmuted us and we are keyed + roster[0].is_self = true; + roster[1].user_id = 2; + roster[1].has_video = true; + + d.update_roster(roster, /*raw_speaker_id=*/1, /*now_ms=*/1000); + check(d.directed_speaker_id() != 1, + "the bot is never promoted even as the raw speaker"); + + // And an incumbent that becomes identity-excluded is dethroned. + d.reset(); + d.configure(0, 0, false, {}); + std::vector clean(2); + clean[0].user_id = 1; clean[0].has_video = true; clean[0].is_talking = true; + clean[1].user_id = 2; clean[1].has_video = true; + d.update_roster(clean, 1, 2000); + check(d.directed_speaker_id() == 1, "a clean participant is promoted"); + + clean[0].is_return_identity = true; // operator named the return seat + d.update_roster(clean, 1, 3000); + check(d.directed_speaker_id() != 1, + "an incumbent that becomes the return identity is dethroned"); + } +``` + +- [ ] **Step 2: Run it to verify it fails** + +Run: +``` +cmake --build build --config Release --parallel 8 --target CoreVideoSpeakerDirectorTest +ctest -C Release -R CoreVideoSpeakerDirector --output-on-failure +``` +Expected: FAIL — `the bot is never promoted even as the raw speaker`. + +- [ ] **Step 3: Implement** + +In `src/speaker-director.cpp`, add the include with the others at the top: + +```cpp +#include "zoom-self-identity.h" +``` + +Replace `participant_excluded_locked()` in full: + +```cpp +bool SpeakerDirector::participant_excluded_locked(uint32_t participant_id) const +{ + // Two sources of exclusion, deliberately folded into one predicate so + // BOTH get the dethroning behaviour in enforce_incumbent_eligibility_locked() + // rather than only the operator list. + // + // 1. The operator's exclude combo boxes: meeting-scoped ids, lost on a + // rejoin. Fine, because an operator is present to re-pick. + // 2. Identity: us, and the Virtual-Camera return seat. Re-derived from + // the roster on every message (zoom-self-identity.h), so it survives a + // rejoin — and it must, because talkback deliberately unmutes the bot, + // which without this makes the bot an eligible speaker whose picture is + // then fed back into the meeting. + if (std::find(m_excluded_participant_ids.begin(), + m_excluded_participant_ids.end(), + participant_id) != m_excluded_participant_ids.end()) + return true; + + const auto it = std::find_if(m_roster.begin(), m_roster.end(), + [participant_id](const ParticipantInfo &p) { + return p.user_id == participant_id; + }); + return it != m_roster.end() && identity_excluded(*it); +} +``` + +- [ ] **Step 4: Run the test to verify it passes** + +Run: +``` +cmake --build build --config Release --parallel 8 --target CoreVideoSpeakerDirectorTest +ctest -C Release -R CoreVideoSpeakerDirector --output-on-failure +``` +Expected: PASS. + +- [ ] **Step 5: Run the whole suite** + +Run: `ctest -C Release --output-on-failure` +Expected: N/N green. + +- [ ] **Step 6: Commit** + +```bash +git add src/speaker-director.cpp tests/speaker-director-test.cpp +git commit -m "fix: speaker director excludes and dethrones our own identities" +``` + +--- + +## Task 5: The detector seam + +**Why:** the detector engine is a separate plan by another author. This is the one function both consumers call, with a default that keeps everything here buildable, shippable and testable before the detector exists. + +**Files:** +- Create: `src/zoom-subject-source.h` +- Create: `tests/subject-source-test.cpp` +- Modify: `CMakeLists.txt` + +**Interfaces:** +- Consumes: the detector plan's `SubjectFrame` (fields fixed in Global Constraints). +- Produces: `SubjectProvider` (typedef), `set_subject_provider()`, `subject_for(uint32_t participant_id)` — every later task calls **only** `subject_for()`. + +- [ ] **Step 1: Write the failing test** + +Create `tests/subject-source-test.cpp`: + +```cpp +// The seam between the framing consumers and the subject detector. The +// detector is a separate plan; this pins the contract the consumers rely on so +// they can be built and tested before it lands. +#include "zoom-subject-source.h" + +#include +#include + +static int g_failures = 0; + +static void check(bool ok, const std::string &what) +{ + if (!ok) { + std::cerr << "FAIL: " << what << "\n"; + ++g_failures; + } +} + +int main() +{ + // With no detector installed, every participant reads as "no subject". + // That is the state the plugin ships in until the detector engine exists, + // and it must be the SAFE state: auto-frame falls back to the operator's + // crop, and the overlay says "step into frame" rather than pointing + // somewhere arbitrary. + const SubjectFrame none = subject_for(42); + check(!none.found, "no provider means not found"); + check(none.confidence == 0.0f, "no provider means zero confidence"); + + set_subject_provider([](uint32_t id) { + SubjectFrame f{}; + f.found = id == 7; + f.box_x = 0.4f; f.box_y = 0.2f; f.box_w = 0.2f; f.box_h = 0.3f; + f.eye_l_x = 0.45f; f.eye_l_y = 0.3f; + f.eye_r_x = 0.55f; f.eye_r_y = 0.3f; + f.confidence = 0.9f; + f.detected_ns = 123456789ULL; + return f; + }); + + check(subject_for(7).found, "the installed provider is consulted"); + check(!subject_for(8).found, "the provider decides per participant"); + check(subject_for(7).detected_ns == 123456789ULL, + "the frame is returned verbatim"); + + // Uninstalling restores the safe default, so a detector shutting down + // cannot leave consumers holding a dangling std::function. + set_subject_provider(nullptr); + check(!subject_for(7).found, "clearing the provider restores not-found"); + + if (g_failures == 0) std::cout << "subject-source: all checks passed\n"; + return g_failures == 0 ? 0 : 1; +} +``` + +- [ ] **Step 2: Register the target and run it to verify it fails** + +In `CMakeLists.txt`, inside `if(BUILD_TESTING)`, after the `CoreVideoSelfIdentityTest` block: + +```cmake + # The detector seam. Pins that "no detector installed" is the safe state. + add_executable(CoreVideoSubjectSourceTest + tests/subject-source-test.cpp + ) + target_include_directories(CoreVideoSubjectSourceTest PRIVATE + "${CMAKE_CURRENT_SOURCE_DIR}/src" + ) + add_test(NAME CoreVideoSubjectSource + COMMAND CoreVideoSubjectSourceTest) +``` + +Run: `cmake --build build --config Release --parallel 8 --target CoreVideoSubjectSourceTest` +Expected: FAIL — `Cannot open include file: 'zoom-subject-source.h'` + +- [ ] **Step 3: Write the header** + +Create `src/zoom-subject-source.h`: + +```cpp +#pragma once + +// Why this exists +// --------------- +// The subject detector (libfacedetection behind a worker thread) is a separate +// subsystem with its own plan and its own ship date. Its two consumers — Tiles +// auto-framing and the framing-advice overlay — need exactly one thing from +// it: the current smoothed subject for a participant. So that one thing is a +// seam. +// +// Consequences that are deliberate: +// * The consumers build, ship and unit-test with no detector at all. +// * The default answer is "not found", which is every consumer's safe path: +// auto-frame falls back to the operator's crop, the overlay says "step +// into frame". +// * SMOOTHING IS NOT DONE HERE. The detector owns deadband, rate limit and +// dropout hold, and `found == false` already means "held past the dropout +// window". Re-smoothing on this side would double the lag and is wrong. +// +// Thread-safety: the provider is installed once when the detector starts and +// cleared when it stops; readers are the OBS graphics thread (tiles tick) and +// the overlay's tick. A shared_ptr snapshot under a mutex is used rather than +// a raw std::function copy so a reader in flight cannot observe a half-assigned +// function object while the detector shuts down. + +#include +#include +#include +#include + +// SubjectFrame is defined ONCE, in src/subject-frame.h, which is the published +// contract shared with the subject-detector plan. Do NOT redefine it here -- +// two definitions with different member defaults would be an ODR violation the +// moment both headers meet in one translation unit. +// +// src/subject-frame.h is a standalone pure header: only, no detector +// dependency. If the subject-detector plan has not run yet, create it first +// from the contract below (it is ~12 lines and nothing else in that plan is +// needed), so this plan keeps its property of building before the detector +// exists. +#include "subject-frame.h" + +using SubjectProvider = std::function; + +namespace subject_source_detail { + +inline std::mutex &mutex() +{ + static std::mutex s_mtx; + return s_mtx; +} + +inline std::shared_ptr &slot() +{ + static std::shared_ptr s_provider; + return s_provider; +} + +} // namespace subject_source_detail + +// Installs (or, with nullptr, removes) the detector. Called by the detector +// subsystem only. +inline void set_subject_provider(SubjectProvider provider) +{ + std::shared_ptr next; + if (provider) + next = std::make_shared(std::move(provider)); + std::lock_guard lock(subject_source_detail::mutex()); + subject_source_detail::slot() = std::move(next); +} + +// The current smoothed subject for a participant, or a not-found frame. +inline SubjectFrame subject_for(uint32_t participant_id) +{ + std::shared_ptr provider; + { + std::lock_guard lock(subject_source_detail::mutex()); + provider = subject_source_detail::slot(); + } + if (!provider) return SubjectFrame{}; + return (*provider)(participant_id); +} +``` + +**Note for the detector's author:** if the detector plan defines `SubjectFrame` in its own header, delete the definition above and `#include` theirs here instead. The field names are fixed by the shared contract, so only the location moves. + +- [ ] **Step 4: Run the test to verify it passes** + +Run: +``` +cmake --build build --config Release --parallel 8 --target CoreVideoSubjectSourceTest +ctest -C Release -R CoreVideoSubjectSource --output-on-failure +``` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add src/zoom-subject-source.h tests/subject-source-test.cpp CMakeLists.txt +git commit -m "feat: subject detector seam with a safe not-found default" +``` + +--- + +## Task 6: Auto-frame crop math (pure) + +**Files:** +- Create: `src/zoom-auto-frame.h` +- Create: `tests/auto-frame-test.cpp` +- Modify: `CMakeLists.txt` + +**Interfaces:** +- Consumes: `SubjectFrame` + `subject_for()` (Task 5), `CropRect` from `src/zoom-tile-grid.h`. +- Produces: `struct AutoFrameParams { double max_zoom; double eyeline_fraction; double target_box_height; }`, `struct AutoFrameResult { bool valid; CropRect crop; }`, `AutoFrameResult solve_auto_frame(const SubjectFrame&, double src_width, double src_height, double dst_aspect, const AutoFrameParams&)`. + +- [ ] **Step 1: Write the failing test** + +Create `tests/auto-frame-test.cpp`: + +```cpp +// Auto-frame crop math, pinned the way tests/tile-shape-test.cpp pins the +// shader's crop arithmetic: exact aspect, inside the source, and crop_uv +// recomputed from the TRUNCATED integers so borders stay in register. There is +// no headless GPU harness in this repo and there is not going to be one, so +// this file is the only thing standing between a framing bug and a live show. +#include "zoom-auto-frame.h" + +#include +#include +#include +#include + +static int g_failures = 0; + +static void check(bool ok, const std::string &what) +{ + if (!ok) { + std::cerr << "FAIL: " << what << "\n"; + ++g_failures; + } +} + +static bool near(double a, double b, double eps) +{ + return std::fabs(a - b) <= eps; +} + +// A subject centred at x=0.5, eyes at y=0.30, box 0.20 wide x 0.30 tall. +static SubjectFrame centred_subject() +{ + SubjectFrame f{}; + f.found = true; + f.box_x = 0.40f; f.box_y = 0.20f; f.box_w = 0.20f; f.box_h = 0.30f; + f.eye_l_x = 0.46f; f.eye_l_y = 0.30f; + f.eye_r_x = 0.54f; f.eye_r_y = 0.30f; + f.confidence = 0.95f; + f.detected_ns = 1; + return f; +} + +int main() +{ + const AutoFrameParams p; // defaults are the shipped behaviour + + // ── Not found: no crop, and the caller must fall back ──────────────────── + { + SubjectFrame gone{}; + const AutoFrameResult r = + solve_auto_frame(gone, 640.0, 360.0, 16.0 / 9.0, p); + check(!r.valid, "a not-found subject yields no auto-frame"); + } + + // ── 360p is the DEFAULT tile subscription: it must work exactly here ───── + { + const double src_w = 640.0, src_h = 360.0, aspect = 16.0 / 9.0; + const AutoFrameResult r = + solve_auto_frame(centred_subject(), src_w, src_h, aspect, p); + check(r.valid, "a found subject yields an auto-frame at 360p"); + check(near(r.crop.width / r.crop.height, aspect, 1e-9), + "the 360p auto-frame rect has EXACTLY the tile aspect"); + check(r.crop.x >= 0.0 && r.crop.y >= 0.0 && + r.crop.x + r.crop.width <= src_w && + r.crop.y + r.crop.height <= src_h, + "the 360p auto-frame rect stays inside the source"); + + // The eyeline lands on the upper-third line of the CROP. + const double eye_y = 0.30 * src_h; + check(near((eye_y - r.crop.y) / r.crop.height, p.eyeline_fraction, 1e-9), + "the eyeline sits on the upper-third line of the crop"); + // ...and the subject is centred horizontally in the crop. + const double centre_x = 0.5 * src_w; + check(near((centre_x - r.crop.x) / r.crop.width, 0.5, 1e-9), + "the subject is centred horizontally in the crop"); + } + + // ── Zoom limit ─────────────────────────────────────────────────────────── + { + // A tiny box would ask for a huge magnification; max_zoom caps it. + SubjectFrame tiny = centred_subject(); + tiny.box_h = 0.02f; + const double src_w = 1920.0, src_h = 1080.0, aspect = 16.0 / 9.0; + const AutoFrameResult r = + solve_auto_frame(tiny, src_w, src_h, aspect, AutoFrameParams{}); + check(r.valid, "a tiny box still frames"); + // At 16:9 into a 16:9 source the largest crop is the whole frame, so + // the smallest legal crop height is src_h / max_zoom. + check(near(r.crop.height, src_h / AutoFrameParams{}.max_zoom, 1e-6), + "the crop never zooms past max_zoom"); + check(near(r.crop.width / r.crop.height, aspect, 1e-9), + "aspect survives the zoom clamp"); + } + + // ── A big box asks to zoom OUT past the frame; clamped to the frame ────── + { + SubjectFrame huge = centred_subject(); + huge.box_h = 0.95f; + const double src_w = 1920.0, src_h = 1080.0, aspect = 16.0 / 9.0; + const AutoFrameResult r = solve_auto_frame(huge, src_w, src_h, aspect, p); + check(near(r.crop.width, src_w, 1e-6) && near(r.crop.height, src_h, 1e-6), + "the crop never grows past the source"); + } + + // ── Off-centre and off-edge subjects: translation clamps, aspect holds ─── + { + const double src_w = 640.0, src_h = 360.0, aspect = 16.0 / 9.0; + for (const float bx : {0.0f, 0.02f, 0.45f, 0.80f}) { + SubjectFrame s = centred_subject(); + s.box_x = bx; + s.eye_l_x = bx + 0.06f; + s.eye_r_x = bx + 0.14f; + const AutoFrameResult r = solve_auto_frame(s, src_w, src_h, aspect, p); + check(r.valid, "an off-centre subject still frames"); + check(near(r.crop.width / r.crop.height, aspect, 1e-9), + "aspect holds for an off-centre subject"); + check(r.crop.x >= 0.0 && r.crop.x + r.crop.width <= src_w, + "the crop stays inside the source horizontally"); + check(r.crop.y >= 0.0 && r.crop.y + r.crop.height <= src_h, + "the crop stays inside the source vertically"); + } + // Eyes near the top of the frame would put the crop above y=0. + SubjectFrame high = centred_subject(); + high.eye_l_y = 0.02f; high.eye_r_y = 0.02f; + const AutoFrameResult r = solve_auto_frame(high, src_w, src_h, aspect, p); + check(r.crop.y >= 0.0, "a high eyeline cannot push the crop off the top"); + check(near(r.crop.width / r.crop.height, aspect, 1e-9), + "aspect holds when the vertical clamp bites"); + } + + // ── Non-16:9 tiles: the aspect that must be honoured is the TILE's ─────── + { + const double src_w = 640.0, src_h = 360.0; + for (const double aspect : {4.0 / 3.0, 1.0, 9.0 / 16.0, 21.0 / 9.0}) { + const AutoFrameResult r = + solve_auto_frame(centred_subject(), src_w, src_h, aspect, p); + check(r.valid, "a non-16:9 tile still frames"); + check(near(r.crop.width / r.crop.height, aspect, 1e-9), + "the crop matches the tile aspect exactly"); + check(r.crop.x >= 0.0 && r.crop.y >= 0.0 && + r.crop.x + r.crop.width <= src_w && + r.crop.y + r.crop.height <= src_h, + "the non-16:9 crop stays inside the source"); + } + } + + // ── Degenerate inputs never produce a rect the draw path would use ─────── + { + check(!solve_auto_frame(centred_subject(), 0.0, 360.0, 1.7778, p).valid, + "a zero-width source yields no auto-frame"); + check(!solve_auto_frame(centred_subject(), 640.0, 0.0, 1.7778, p).valid, + "a zero-height source yields no auto-frame"); + check(!solve_auto_frame(centred_subject(), 640.0, 360.0, 0.0, p).valid, + "a zero aspect yields no auto-frame"); + SubjectFrame nan_box = centred_subject(); + nan_box.box_h = std::nanf(""); + check(!solve_auto_frame(nan_box, 640.0, 360.0, 1.7778, p).valid, + "a NaN box yields no auto-frame"); + } + + // ── Resolution invariance ──────────────────────────────────────────────── + // Auto-framing NEVER asks for a resolution upgrade: it consumes whatever + // the shared feed already is, which is 1080p for an ISO'd or program-output + // participant and 360p for one only the tile wall is showing. So the SAME + // subject must produce the SAME NORMALIZED crop at both, or the framing + // decision would silently depend on who happens to be ISO'd — and the tile + // would jump the moment somebody's ISO was switched on mid-show. + { + const double aspect = 16.0 / 9.0; + const SubjectFrame s = centred_subject(); + const AutoFrameResult lo = solve_auto_frame(s, 640.0, 360.0, aspect, p); + const AutoFrameResult hi = solve_auto_frame(s, 1920.0, 1080.0, aspect, p); + check(lo.valid && hi.valid, "both resolutions frame"); + check(near(lo.crop.x / 640.0, hi.crop.x / 1920.0, 1e-9), + "normalized crop x is identical at 360p and 1080p"); + check(near(lo.crop.y / 360.0, hi.crop.y / 1080.0, 1e-9), + "normalized crop y is identical at 360p and 1080p"); + check(near(lo.crop.width / 640.0, hi.crop.width / 1920.0, 1e-9), + "normalized crop width is identical at 360p and 1080p"); + check(near(lo.crop.height / 360.0, hi.crop.height / 1080.0, 1e-9), + "normalized crop height is identical at 360p and 1080p"); + + // And at an odd, non-16:9 source too, so nothing has quietly assumed a + // 16:9 feed. 1280x720 and 960x540 are the same picture; 704x396 is not + // a standard rung and must still agree with them normalized. + const AutoFrameResult a = solve_auto_frame(s, 1280.0, 720.0, aspect, p); + const AutoFrameResult b = solve_auto_frame(s, 704.0, 396.0, aspect, p); + check(near(a.crop.width / 1280.0, b.crop.width / 704.0, 1e-9), + "normalized crop is resolution-independent at odd sizes too"); + } + + // ── crop_uv registration, reproduced exactly as the draw path does it ──── + // This is the arithmetic at src/zoom-supersource.cpp:1957-2007, copied the + // way tests/tile-shape-test.cpp:189-240 copies it. If the shader's + // (uv - crop_uv.xy) / crop_uv.zw stops landing on 0..1, every border on + // every auto-framed tile silently misregisters. + { + const double tex_w = 640.0, tex_h = 360.0; + const double aspect = 4.0 / 3.0; + const AutoFrameResult r = + solve_auto_frame(centred_subject(), tex_w, tex_h, aspect, p); + check(r.valid, "the registration case produced a rect"); + + const uint32_t cx = static_cast(r.crop.x); + const uint32_t cy = static_cast(r.crop.y); + const uint32_t cw = static_cast(r.crop.width); + const uint32_t ch = static_cast(r.crop.height); + check(cw != 0 && ch != 0, "the auto-frame crop did not truncate to nothing"); + check(cx + cw <= static_cast(tex_w) && + cy + ch <= static_cast(tex_h), + "the TRUNCATED integers are still inside the texture"); + + const float crop_u = static_cast(cx) / static_cast(tex_w); + const float crop_v = static_cast(cy) / static_cast(tex_h); + const float crop_cu = static_cast(cw) / static_cast(tex_w); + const float crop_cv = static_cast(ch) / static_cast(tex_h); + + const float u0 = static_cast(cx) / static_cast(tex_w); + const float u1 = static_cast(cx + cw) / static_cast(tex_w); + const float v0 = static_cast(cy) / static_cast(tex_h); + const float v1 = static_cast(cy + ch) / static_cast(tex_h); + + const double tile_u0 = (u0 - crop_u) / crop_cu; + const double tile_u1 = (u1 - crop_u) / crop_cu; + const double tile_v0 = (v0 - crop_v) / crop_cv; + const double tile_v1 = (v1 - crop_v) / crop_cv; + check(near(tile_u0, 0.0, 1e-5) && near(tile_u1, 1.0, 1e-5) && + near(tile_v0, 0.0, 1e-5) && near(tile_v1, 1.0, 1e-5), + "crop_uv from the truncated integers still lands on 0..1"); + } + + if (g_failures == 0) std::cout << "auto-frame: all checks passed\n"; + return g_failures == 0 ? 0 : 1; +} +``` + +- [ ] **Step 2: Register the target and run it to verify it fails** + +In `CMakeLists.txt`, inside `if(BUILD_TESTING)`, after the `CoreVideoSubjectSourceTest` block: + +```cmake + # Auto-frame crop math. Exact tile aspect, in-bounds, crop_uv taken from + # the truncated integers, and the same normalized answer at 360p and 1080p — + # the four ways this silently breaks tiles. + add_executable(CoreVideoAutoFrameTest + tests/auto-frame-test.cpp + src/zoom-tile-grid.cpp + ) + target_include_directories(CoreVideoAutoFrameTest PRIVATE + "${CMAKE_CURRENT_SOURCE_DIR}/src" + ) + add_test(NAME CoreVideoAutoFrame + COMMAND CoreVideoAutoFrameTest) +``` + +Run: `cmake --build build --config Release --parallel 8 --target CoreVideoAutoFrameTest` +Expected: FAIL — `Cannot open include file: 'zoom-auto-frame.h'` + +- [ ] **Step 3: Write the header** + +Create `src/zoom-auto-frame.h`: + +```cpp +#pragma once + +// Why this exists +// --------------- +// Tiles auto-framing decides ONE thing: the sub-rectangle of a participant's +// frame the wall should sample. That decision has to satisfy the same four +// constraints the manual slot crop does (see src/zoom-supersource.cpp:1957): +// exact tile aspect or the tile letterboxes; inside [0,tex_w]x[0,tex_h]; a +// crop_uv derived from the truncated integers or borders misregister; and a +// snapshot under ctx->mutex so a pass lands as a unit. +// +// Constraints 1 and 2 are structural and live HERE, in pure arithmetic that +// tests/auto-frame-test.cpp pins the way tests/tile-shape-test.cpp pins the +// shader's crop maths. There is no headless GPU harness in this repo and one +// has been ruled against, so a pinned pure header is the whole verification +// story. +// +// Two things this file deliberately does NOT do: +// * Smoothing. The detector already applies deadband, rate limit and dropout +// hold. Re-smoothing here would double the lag and make tiles feel dead. +// * Anything resolution-dependent, and above all anything that asks for MORE +// resolution. This consumes whatever the shared feed already is: the engine +// holds one upgrade-only subscription per participant, so a participant on +// a 1080p program output or an ISO is already high-resolution and the tile +// reuses that for free, while a participant only the wall is showing stays +// at 360p and simply looks softer. That softness is a diagnosis - nobody is +// paying for their pixels - and the remedy is to ISO them, which the +// operator can already do. All maths is in source pixels, read from +// tex_w/tex_h, and the SAME subject yields the same NORMALIZED crop at +// 640x360 and at 1920x1080. +// +// The framing rule, in order: +// 1. Height: scale the crop so the subject's box height is +// target_box_height of the crop height. That is a medium close-up. +// 2. Clamp that height between the largest crop that fits the source at the +// tile aspect, and that same height divided by max_zoom. Width is then +// height * aspect, which by construction also fits — so the aspect is +// EXACT and no later clamp can break it. +// 3. Position: put the eyeline on the crop's upper-third line and the +// subject's horizontal centre on the crop's centre, then translate the +// whole rect back inside the source. Translation cannot change the +// dimensions, so the aspect survives the clamp. + +#include "zoom-subject-source.h" +#include "zoom-tile-grid.h" + +#include +#include + +struct AutoFrameParams { + // Largest magnification allowed, as a linear factor on the crop's + // dimensions. 2.0 means "never sample less than half the frame's height". + // Held low on purpose: a tile the wall alone is showing is carried at 360p, + // so a 2x crop is already a 320x180 region drawn into a tile several + // hundred pixels wide, and past that the softness reads as a fault rather + // than a choice. A participant who is ISO'd or on a program output is + // already carried higher and gets the same framing from more pixels, for + // free - this number does not change, and nothing here ever asks for more. + double max_zoom = 2.0; + // Where the eyeline goes, as a fraction of the crop height from its top. + // The broadcast upper-third line. + double eyeline_fraction = 1.0 / 3.0; + // The subject's box height as a fraction of the crop height. 0.45 is a + // conventional medium close-up: head and shoulders, air above. + double target_box_height = 0.45; +}; + +struct AutoFrameResult { + bool valid = false; // false => the caller must use the manual crop + CropRect crop; // source pixels; aspect is exactly dst_aspect +}; + +// True for a finite, strictly positive double. Guards every input, because a +// detector bug that emits NaN must degrade to "no auto-frame", never to a +// crop rect the draw path will try to sample. +inline bool auto_frame_finite_positive(double v) +{ + return std::isfinite(v) && v > 0.0; +} + +inline AutoFrameResult solve_auto_frame(const SubjectFrame &subject, + double src_width, double src_height, + double dst_aspect, + const AutoFrameParams ¶ms) +{ + AutoFrameResult out; + if (!subject.found) return out; + if (!auto_frame_finite_positive(src_width) || + !auto_frame_finite_positive(src_height) || + !auto_frame_finite_positive(dst_aspect)) + return out; + if (!auto_frame_finite_positive(static_cast(subject.box_h)) || + !auto_frame_finite_positive(static_cast(subject.box_w))) + return out; + if (!std::isfinite(static_cast(subject.box_x)) || + !std::isfinite(static_cast(subject.eye_l_y)) || + !std::isfinite(static_cast(subject.eye_r_y)) || + !std::isfinite(static_cast(subject.eye_l_x)) || + !std::isfinite(static_cast(subject.eye_r_x))) + return out; + + const double max_zoom = + auto_frame_finite_positive(params.max_zoom) && params.max_zoom >= 1.0 + ? params.max_zoom : 1.0; + const double eyeline = + std::min(std::max(params.eyeline_fraction, 0.05), 0.95); + const double target_box = + std::min(std::max(params.target_box_height, 0.05), 1.0); + + // 1. Desired crop height, in source pixels. + const double box_h_px = static_cast(subject.box_h) * src_height; + const double desired_h = box_h_px / target_box; + + // 2. Clamp, aspect-first. h_max is the tallest crop of this aspect that + // fits the source, so h <= h_max guarantees width = h * aspect <= + // src_width AND height <= src_height. Nothing below changes the + // dimensions again, which is why the aspect comes out exact. + const double h_max = std::min(src_height, src_width / dst_aspect); + const double h_min = h_max / max_zoom; + const double crop_h = std::min(std::max(desired_h, h_min), h_max); + const double crop_w = crop_h * dst_aspect; + + // 3. Position. Eyeline onto the upper-third line, subject centred + // horizontally, then translated back inside the frame. + const double eye_y_px = + 0.5 * (static_cast(subject.eye_l_y) + + static_cast(subject.eye_r_y)) * src_height; + const double centre_x_px = + (static_cast(subject.box_x) + + 0.5 * static_cast(subject.box_w)) * src_width; + + double x = centre_x_px - 0.5 * crop_w; + double y = eye_y_px - eyeline * crop_h; + x = std::min(std::max(x, 0.0), src_width - crop_w); + y = std::min(std::max(y, 0.0), src_height - crop_h); + + // A crop that truncates to zero on either axis would make the draw path + // fall back to the neutral placeholder, which reads as a dead tile. Refuse + // instead, so the manual crop keeps the tile alive. + if (crop_w < 2.0 || crop_h < 2.0) return out; + + out.valid = true; + out.crop.x = x; + out.crop.y = y; + out.crop.width = crop_w; + out.crop.height = crop_h; + return out; +} +``` + +- [ ] **Step 4: Run the test to verify it passes** + +Run: +``` +cmake --build build --config Release --parallel 8 --target CoreVideoAutoFrameTest +ctest -C Release -R CoreVideoAutoFrame --output-on-failure +``` +Expected: PASS, `auto-frame: all checks passed` + +- [ ] **Step 5: Commit** + +```bash +git add src/zoom-auto-frame.h tests/auto-frame-test.cpp CMakeLists.txt +git commit -m "feat: pure auto-frame crop math, aspect- and bounds-pinned" +``` + +--- + +## Task 7: Wire auto-framing into the Tiles source + +**Files:** +- Modify: `src/zoom-supersource.cpp` — struct fields (~`:297`, `:343`), property names (~`:2259`), update (~`:2640`, `:2805`), defaults (~`:3115`), properties (~`:3395`), snapshot (`:1286`), crop insertion (`:1957`), registration (`:3505`) +- Modify: `data/locale/en-US.ini` + +**Interfaces:** +- Consumes: `solve_auto_frame()` / `AutoFrameParams` (Task 6), `subject_for()` (Task 5), `solve_slot_crop()` (`src/zoom-tile-crop.h`). +- Produces: properties `auto_frame` (bool, default false) and `auto_frame_max_zoom` (int %, default 200); a per-slot `AutoFrameResult` snapshot the draw path consumes. + +**Hard constraint on this task:** it must not call `ZoomEngineClient::subscribe()`, must not modify `tile_feed_subscribe` (`:475-480`), and must not name a `VideoResolution` anywhere. Auto-framing consumes whatever the shared feed already is. A tile the wall alone is showing stays at 360p and its auto-framed crop looks softer; that is the diagnosis (nobody is paying for that participant's pixels) and the remedy is for the operator to ISO them, which they can already do. Verify before committing with `git diff` — the diff for this task must contain no `subscribe`, no `VideoResolution`, and no `P360`/`P720`. + +- [ ] **Step 1: Write the failing test — the insertion-point selection rule** + +Append to `tests/auto-frame-test.cpp`, before its final `if (g_failures == 0)` block. This pins the exact decision the draw path makes at `:1957` — auto-frame replaces the manual crop only when it is both enabled and valid — without needing a GPU: + +```cpp + // ── The insertion-point rule, reproduced ───────────────────────────────── + // src/zoom-supersource.cpp:1957 picks between the operator's slot crop and + // the auto-frame. Both branches must satisfy the SAME four constraints, so + // the selection itself is pinned here rather than trusted to a comment. + { + const double tex_w = 640.0, tex_h = 360.0, aspect = 16.0 / 9.0; + const auto pick = [&](bool enabled, const SubjectFrame &s) { + const AutoFrameResult af = + enabled ? solve_auto_frame(s, tex_w, tex_h, aspect, p) + : AutoFrameResult{}; + return af.valid ? af.crop + : solve_slot_crop(tex_w, tex_h, aspect, 10.0, 0.0); + }; + + SubjectFrame gone{}; + const CropRect off = pick(false, centred_subject()); + const CropRect lost = pick(true, gone); + const CropRect on = pick(true, centred_subject()); + + const CropRect manual = solve_slot_crop(tex_w, tex_h, aspect, 10.0, 0.0); + check(near(off.x, manual.x, 1e-9) && near(off.width, manual.width, 1e-9), + "auto-frame off leaves the manual crop untouched"); + check(near(lost.x, manual.x, 1e-9) && near(lost.width, manual.width, 1e-9), + "a lost subject falls back to the manual crop"); + check(!near(on.width, manual.width, 1e-9) || !near(on.y, manual.y, 1e-9), + "auto-frame on with a subject actually changes the rect"); + + for (const CropRect &c : {off, lost, on}) { + check(near(c.width / c.height, aspect, 1e-9), + "every branch of the insertion point keeps the tile aspect"); + check(c.x >= 0.0 && c.y >= 0.0 && c.x + c.width <= tex_w && + c.y + c.height <= tex_h, + "every branch of the insertion point stays in bounds"); + check(static_cast(c.width) != 0 && + static_cast(c.height) != 0, + "no branch truncates to a zero-size sub-region"); + } + } +``` + +Add `#include "zoom-tile-crop.h"` to the test's includes. + +- [ ] **Step 2: Run it to verify it fails** + +Run: +``` +cmake --build build --config Release --parallel 8 --target CoreVideoAutoFrameTest +ctest -C Release -R CoreVideoAutoFrame --output-on-failure +``` +Expected: FAIL — `Cannot open include file: 'zoom-tile-crop.h'` until the include is added; then it should pass, since the rule is expressed entirely in already-implemented functions. Confirm it is a real test by temporarily changing `af.valid ?` to `false ?` and seeing `auto-frame on with a subject actually changes the rect` fail; restore it. + +- [ ] **Step 3: Add the property names, defaults, properties and atomics** + +In `src/zoom-supersource.cpp`, add the includes with the other project includes at the top: + +```cpp +#include "zoom-auto-frame.h" +#include "zoom-subject-source.h" +``` + +After the `PROP_CROP_GROUP` declaration (~`:2259`): + +```cpp +// Auto-framing. OFF by default and a complete bypass when off: with it off the +// draw path takes exactly the solve_slot_crop() call it always took, so a scene +// saved before this existed renders byte-for-byte as it did. +// +// The zoom cap is a percentage because a slider of "200%" reads to an operator +// and a slider of "2.00" does not. Deliberately modest: a tile only the wall is +// showing is carried at 360p (see tile_feed_subscribe), and past ~2x a 360p +// source magnified into a tile looks like a fault rather than a framing choice. +// Auto-framing NEVER raises that subscription - it consumes whatever the shared +// feed already is, so an ISO'd or program-output participant is framed from +// their existing high-resolution feed for free. +static constexpr const char *PROP_AUTO_FRAME = "auto_frame"; +static constexpr const char *PROP_AUTO_FRAME_ZOOM = "auto_frame_max_zoom_pct"; +static constexpr int64_t kMinAutoFrameZoomPct = 100; +static constexpr int64_t kMaxAutoFrameZoomPct = 300; +``` + +In `struct tiles_source`, beside `slot_crop` (~`:297`): + +```cpp + // Auto-framing. The settings pair is atomic like every other setting here; + // the per-slot results are the framing STATE, and they live under + // ctx->mutex beside slot_crop for the reason spelled out at the snapshot + // in tiles_source_render(): a framing pass must land on the draw path as a + // unit, not slot by slot. + std::atomic auto_frame{false}; + std::atomic auto_frame_zoom_pct{200}; + std::array auto_frame_state{}; +``` + +And beside `render_slot_crop` (~`:343`): + +```cpp + std::array render_auto_frame{}; +``` + +In `tiles_source_get_defaults` (~`:3115`, next to the `PROP_ANIMATE` defaults): + +```cpp + // Off by default, and off is a complete bypass — see PROP_AUTO_FRAME. + obs_data_set_default_bool(settings, PROP_AUTO_FRAME, false); + obs_data_set_default_int(settings, PROP_AUTO_FRAME_ZOOM, 200); +``` + +In `tiles_source_get_properties` (~`:3395`, just before the crop group): + +```cpp + // Auto-framing sits beside the per-tile crop group because it is the same + // decision made automatically: which part of the source this tile samples. + obs_properties_add_bool(props, PROP_AUTO_FRAME, + obs_module_text("CoreVideoTiles.AutoFrame")); + obs_properties_add_int_slider(props, PROP_AUTO_FRAME_ZOOM, + obs_module_text("CoreVideoTiles.AutoFrameZoom"), + static_cast(kMinAutoFrameZoomPct), + static_cast(kMaxAutoFrameZoomPct), 10); +``` + +In `tiles_source_update`, beside the `PROP_ANIMATE` stores (~`:2736`): + +```cpp + ctx->auto_frame.store(obs_data_get_bool(settings, PROP_AUTO_FRAME), + std::memory_order_release); + // Clamped on the same threat model as every other setting here: the slider + // bounds it, obs_data_get_int returns an int64, and scene files are + // hand-editable. solve_auto_frame() clamps again; this bounds the setting. + ctx->auto_frame_zoom_pct.store( + static_cast(std::min( + std::max(obs_data_get_int(settings, PROP_AUTO_FRAME_ZOOM), + kMinAutoFrameZoomPct), + kMaxAutoFrameZoomPct)), + std::memory_order_release); +``` + +- [ ] **Step 4: Add the tick that computes framing state, and register it** + +Add this function immediately above `tiles_source_render`: + +```cpp +// Recomputes each slot's auto-frame rect. Runs on the OBS graphics thread via +// info.video_tick, i.e. immediately before the render that consumes it, and +// does no I/O: subject_for() is a cached lookup of the detector's already +// smoothed result, and solve_auto_frame() is arithmetic. +// +// It runs here rather than in the roster callback because detection updates at +// 2-5 fps on its own schedule, which no roster event is correlated with. +// +// Writes under ctx->mutex, beside slot_crop, so the whole pass is visible to +// the draw path at once. Reading the feed list requires the same lock, so the +// participant ids are copied out first and the solve runs unlocked. +static void tiles_source_tick(void *data, float /*seconds*/) +{ + auto *ctx = static_cast(data); + + if (!ctx->auto_frame.load(std::memory_order_acquire)) { + // Off is a complete bypass: clear the state so switching it back on + // cannot resurrect a stale rect from minutes ago. + std::lock_guard lock(ctx->mutex); + ctx->auto_frame_state = {}; + return; + } + + AutoFrameParams params; + params.max_zoom = + static_cast(ctx->auto_frame_zoom_pct.load( + std::memory_order_acquire)) / 100.0; + const double aspect = ctx->tile_aspect.load(std::memory_order_acquire); + + struct SlotSubject { + uint32_t participant_id = 0; + double tex_w = 0.0; + double tex_h = 0.0; + }; + std::array slots{}; + { + std::lock_guard lock(ctx->mutex); + for (std::size_t i = 0; i < ctx->feeds.size() && i < slots.size(); ++i) { + const TileFeedPtr &feed = ctx->feeds[i]; + if (!feed) continue; + slots[i].participant_id = feed->slot.participant_id(); + slots[i].tex_w = static_cast(feed->tex_w); + slots[i].tex_h = static_cast(feed->tex_h); + } + } + + std::array next{}; + for (std::size_t i = 0; i < slots.size(); ++i) { + if (slots[i].participant_id == 0) continue; + if (slots[i].tex_w < 2.0 || slots[i].tex_h < 2.0) continue; + next[i] = solve_auto_frame(subject_for(slots[i].participant_id), + slots[i].tex_w, slots[i].tex_h, aspect, + params); + } + + std::lock_guard lock(ctx->mutex); + ctx->auto_frame_state = next; +} +``` + +In `zoom_supersource_register()` (~`:3505`), beside the other `info.` assignments: + +```cpp + // Recomputes auto-framing just before each render. Registered even though + // auto-framing is off by default: the tick's first act is to check the + // atomic and return, which costs one relaxed load per frame. + info.video_tick = tiles_source_tick; +``` + +- [ ] **Step 5: Snapshot it with the crop, and consume it at the insertion point** + +In `tiles_source_render`, extend the existing snapshot block at `:1286`: + +```cpp + { + std::lock_guard lock(ctx->mutex); + ctx->render_feeds = ctx->feeds; + ctx->render_slot_crop = ctx->slot_crop; + // Snapshotted with the crop, under the same lock, for the same reason: + // a framing pass must land on the draw path as a unit. Taken here and + // not re-read below, so a tick landing mid-frame cannot give slot 3 a + // new rect while slot 4 still has the old one. + ctx->render_auto_frame = ctx->auto_frame_state; + } +``` + +Then at `:1957`, replace the single `solve_slot_crop` call with the selection. The variable stays named `crop` so everything below it — the truncation, `crop_uv`, the sprite sub-region — is untouched: + +```cpp + // Auto-framing REPLACES the operator's slot crop for this slot when it + // is both enabled and valid; otherwise the manual crop stands. Both + // branches produce a rect of exactly params.tile_aspect that lies + // inside the source, which is what the four constraints on this site + // require — pinned in tests/auto-frame-test.cpp, because there is no + // headless GPU harness in this repo and one has been ruled against. + // + // Not blended with the slot crop: two mechanisms moving the same + // rectangle at once is a shot nobody can predict, and the operator's + // slider is the one that must win when they touch it. So it is a + // clean either/or, and switching auto-framing off restores exactly + // the rect they were looking at before. + const AutoFrameResult auto_frame = + i < ctx->render_auto_frame.size() ? ctx->render_auto_frame[i] + : AutoFrameResult{}; + const CropRect crop = + auto_frame.valid + ? auto_frame.crop + : solve_slot_crop(static_cast(feed->tex_w), + static_cast(feed->tex_h), + params.tile_aspect, + slot_crop.first, slot_crop.second); +``` + +- [ ] **Step 6: Add the locale strings** + +In `data/locale/en-US.ini`, beside the other `CoreVideoTiles.*` keys: + +```ini +CoreVideoTiles.AutoFrame="Auto-frame tiles (experimental)" +CoreVideoTiles.AutoFrameZoom="Auto-frame maximum zoom (%)" +``` + +- [ ] **Step 7: Build and run the whole suite** + +Run: +``` +cmake --build build --config Release --parallel 8 +ctest -C Release --output-on-failure +``` +Expected: builds clean, N/N green. + +- [ ] **Step 8: Commit** + +```bash +git add src/zoom-supersource.cpp data/locale/en-US.ini tests/auto-frame-test.cpp +git commit -m "feat: opt-in Tiles auto-framing at the single crop insertion point" +``` + +--- + +## Task 8: Framing advice predicates (pure) + +**Files:** +- Create: `src/zoom-framing-advice.h` +- Create: `tests/framing-advice-test.cpp` +- Modify: `CMakeLists.txt` + +**Interfaces:** +- Consumes: `SubjectFrame` (Task 5). +- Produces: `enum class FramingCondition`, `struct FramingThresholds`, `FramingCondition evaluate_framing(const SubjectFrame&, const FramingThresholds&, FramingCondition previous)`. + +The seven spec conditions, each as a concrete geometric predicate on normalized coordinates: + +| Spec condition | Predicate | Threshold | +|---|---|---| +| No face for longer than the hold period | `!found` (the detector's dropout hold already expired) | — | +| Box touches a frame edge | `box_x <= m` / `box_x+box_w >= 1-m` / `box_y <= m` / `box_y+box_h >= 1-m` | `m = edge_margin = 0.02` | +| Eyeline well above the upper-third line | `eye_y < 1/3 - t` | `t = eyeline_tolerance = 0.08` | +| Eyeline well below the upper-third line | `eye_y > 1/3 + t` | same | +| Box height below minimum fraction | `box_h < 0.18` | `min_box_height` | +| Box height above maximum fraction | `box_h > 0.45` | `max_box_height` | +| Horizontal centre off by more than tolerance | `abs(box_centre_x - 0.5) > 0.12` | `centre_tolerance` | + +- [ ] **Step 1: Write the failing test** + +Create `tests/framing-advice-test.cpp`: + +```cpp +// Every framing condition, tested at its boundary. These predicates are what a +// panelist sees on a return feed, so an off-by-a-threshold here is somebody +// being told to move when they are fine — which is worse than saying nothing. +#include "zoom-framing-advice.h" + +#include +#include + +static int g_failures = 0; + +static void check(bool ok, const std::string &what) +{ + if (!ok) { + std::cerr << "FAIL: " << what << "\n"; + ++g_failures; + } +} + +// A subject that is framed correctly by every rule: centred, box 0.30 tall, +// eyeline exactly on the upper third, well clear of every edge. +static SubjectFrame good() +{ + SubjectFrame f{}; + f.found = true; + f.box_w = 0.20f; f.box_h = 0.30f; + f.box_x = 0.40f; f.box_y = 0.20f; + f.eye_l_x = 0.46f; f.eye_r_x = 0.54f; + f.eye_l_y = 1.0f / 3.0f; f.eye_r_y = 1.0f / 3.0f; + f.confidence = 0.9f; + f.detected_ns = 1; + return f; +} + +int main() +{ + const FramingThresholds t; + const FramingCondition none = FramingCondition::Good; + + check(evaluate_framing(good(), t, none) == FramingCondition::Good, + "a correctly framed subject is Good"); + + // ── No face ────────────────────────────────────────────────────────────── + { + SubjectFrame f{}; // found == false: the detector's hold already expired + check(evaluate_framing(f, t, none) == FramingCondition::NoSubject, + "a lost subject reads as NoSubject"); + } + + // ── Cut off, one edge at a time, each side at its boundary ─────────────── + { + SubjectFrame f = good(); + f.box_x = static_cast(t.edge_margin) + 0.001f; + check(evaluate_framing(f, t, none) != FramingCondition::CutOffLeft, + "just inside the left margin is not cut off"); + f.box_x = static_cast(t.edge_margin) - 0.001f; + check(evaluate_framing(f, t, none) == FramingCondition::CutOffLeft, + "at the left margin the subject is cut off"); + + f = good(); + f.box_x = 1.0f - f.box_w - static_cast(t.edge_margin) + 0.001f; + check(evaluate_framing(f, t, none) == FramingCondition::CutOffRight, + "at the right margin the subject is cut off"); + + f = good(); + f.box_y = static_cast(t.edge_margin) - 0.001f; + check(evaluate_framing(f, t, none) == FramingCondition::CutOffTop, + "at the top margin the subject is cut off"); + + f = good(); + f.box_h = 0.30f; + f.box_y = 1.0f - f.box_h - static_cast(t.edge_margin) + 0.001f; + check(evaluate_framing(f, t, none) == FramingCondition::CutOffBottom, + "at the bottom margin the subject is cut off"); + } + + // ── Camera height, both directions, at the tolerance ───────────────────── + { + const float third = 1.0f / 3.0f; + SubjectFrame f = good(); + f.eye_l_y = f.eye_r_y = third - static_cast(t.eyeline_tolerance) + 0.001f; + check(evaluate_framing(f, t, none) == FramingCondition::Good, + "inside the eyeline tolerance is still Good"); + f.eye_l_y = f.eye_r_y = third - static_cast(t.eyeline_tolerance) - 0.001f; + check(evaluate_framing(f, t, none) == FramingCondition::CameraTooHigh, + "an eyeline above the upper third means lower your camera"); + f.eye_l_y = f.eye_r_y = third + static_cast(t.eyeline_tolerance) + 0.001f; + check(evaluate_framing(f, t, none) == FramingCondition::CameraTooLow, + "an eyeline below the upper third means raise your camera"); + } + + // ── Distance, both directions, at the fraction ─────────────────────────── + { + SubjectFrame f = good(); + f.box_y = 0.30f; + f.box_h = static_cast(t.min_box_height) + 0.001f; + check(evaluate_framing(f, t, none) == FramingCondition::Good, + "just above the minimum box height is Good"); + f.box_h = static_cast(t.min_box_height) - 0.001f; + check(evaluate_framing(f, t, none) == FramingCondition::TooFar, + "below the minimum box height means move closer"); + + f = good(); + f.box_y = 0.10f; + f.box_h = static_cast(t.max_box_height) + 0.001f; + check(evaluate_framing(f, t, none) == FramingCondition::TooClose, + "above the maximum box height means move back"); + } + + // ── Horizontal centring, both directions, at the tolerance ─────────────── + { + SubjectFrame f = good(); + const float off = static_cast(t.centre_tolerance) + 0.001f; + f.box_x = 0.5f - 0.5f * f.box_w + off; // subject right of centre + check(evaluate_framing(f, t, none) == FramingCondition::OffCentreRight, + "a subject right of centre is told to shift left"); + f.box_x = 0.5f - 0.5f * f.box_w - off; + check(evaluate_framing(f, t, none) == FramingCondition::OffCentreLeft, + "a subject left of centre is told to shift right"); + f.box_x = 0.5f - 0.5f * f.box_w + + static_cast(t.centre_tolerance) - 0.001f; + check(evaluate_framing(f, t, none) == FramingCondition::Good, + "inside the centring tolerance is Good"); + } + + // ── Priority: only one thing is ever said, and it is the worst one ─────── + { + SubjectFrame f = good(); + f.box_x = 0.005f; // cut off left + f.box_h = 0.60f; // and too close + f.eye_l_y = f.eye_r_y = 0.05f; // and camera too high + check(evaluate_framing(f, t, none) == FramingCondition::CutOffLeft, + "being cut off outranks distance and camera height"); + + SubjectFrame g2 = good(); + g2.box_y = 0.30f; + g2.box_h = 0.10f; // too far + g2.eye_l_y = g2.eye_r_y = 0.05f; + check(evaluate_framing(g2, t, none) == FramingCondition::TooFar, + "distance outranks camera height"); + } + + // ── Hysteresis: a condition already showing needs a wider margin to clear, + // so a subject sitting exactly on a threshold does not strobe advice. + { + SubjectFrame f = good(); + f.box_y = 0.30f; + f.box_h = static_cast(t.min_box_height) + 0.005f; + check(evaluate_framing(f, t, FramingCondition::Good) == + FramingCondition::Good, + "from Good, just inside the threshold stays Good"); + check(evaluate_framing(f, t, FramingCondition::TooFar) == + FramingCondition::TooFar, + "from TooFar, the same frame still reads TooFar"); + f.box_h = static_cast(t.min_box_height) * 1.30f; + check(evaluate_framing(f, t, FramingCondition::TooFar) == + FramingCondition::Good, + "clearing by the hysteresis margin returns to Good"); + } + + if (g_failures == 0) std::cout << "framing-advice: all checks passed\n"; + return g_failures == 0 ? 0 : 1; +} +``` + +- [ ] **Step 2: Register the target and run it to verify it fails** + +In `CMakeLists.txt`, inside `if(BUILD_TESTING)`, after `CoreVideoAutoFrameTest`: + +```cmake + # Framing advice predicates, each pinned at its boundary. What a panelist + # is told on the return feed. + add_executable(CoreVideoFramingAdviceTest + tests/framing-advice-test.cpp + ) + target_include_directories(CoreVideoFramingAdviceTest PRIVATE + "${CMAKE_CURRENT_SOURCE_DIR}/src" + ) + add_test(NAME CoreVideoFramingAdvice + COMMAND CoreVideoFramingAdviceTest) +``` + +Run: `cmake --build build --config Release --parallel 8 --target CoreVideoFramingAdviceTest` +Expected: FAIL — `Cannot open include file: 'zoom-framing-advice.h'` + +- [ ] **Step 3: Write the header** + +Create `src/zoom-framing-advice.h`: + +```cpp +#pragma once + +// Why this exists +// --------------- +// The return feed tells the panelist currently on air exactly one thing about +// their framing. This file decides which thing. Every rule is a geometric +// predicate on the detector's normalized box and eye landmarks, with an +// explicit threshold, because "looks badly framed" is not testable and +// "box_h < 0.18" is. +// +// Three decisions worth keeping: +// +// * ONE condition at a time, chosen by a fixed priority. A panel guest given +// three instructions does none of them. Being cut off outranks everything +// because it is the only condition where part of them is missing; distance +// outranks camera height because moving changes both. +// +// * HYSTERESIS on top of the detector's smoothing. The detector stabilises the +// BOX; it cannot know that a box hovering at box_h = 0.180 will strobe the +// advice between "move closer" and nothing. A condition already on screen +// must clear its threshold by kHysteresis before it goes away. +// +// * Eyeline direction follows the spec table literally: an eyeline ABOVE the +// upper-third line (a smaller y) means the camera is too high and wants +// lowering. Getting this backwards is the single easiest mistake here and +// the reason both directions are tested. + +#include "zoom-subject-source.h" + +#include +#include + +enum class FramingCondition { + Good = 0, + NoSubject, // "Step into frame" + CutOffLeft, // "You're cut off - move right" + CutOffRight, // "You're cut off - move left" + CutOffTop, // "You're cut off - move down" + CutOffBottom, // "You're cut off - move up" + TooClose, // "Move back" + TooFar, // "Move closer" + CameraTooHigh, // "Lower your camera" + CameraTooLow, // "Raise your camera" + OffCentreLeft, // subject left of centre -> "Shift right" + OffCentreRight, // subject right of centre -> "Shift left" +}; + +struct FramingThresholds { + // How close to a frame edge counts as cut off. 2% of the frame: at 640x360 + // that is 13 px horizontally, which is inside the detector's own box + // slop, so anything tighter would fire on a well-framed subject. + double edge_margin = 0.02; + // The broadcast upper-third line, and how far off it is acceptable. 8% of + // frame height is about half a head at a normal medium shot - visibly + // wrong, not pedantically wrong. + double eyeline_target = 1.0 / 3.0; + double eyeline_tolerance = 0.08; + // Head height as a fraction of frame height. Below 0.18 the face is too + // small to read at gallery size; above 0.45 the shot is a close-up that + // crops the shoulders. + double min_box_height = 0.18; + double max_box_height = 0.45; + // How far the subject's horizontal centre may sit from frame centre. + double centre_tolerance = 0.12; + // Multiplier applied to a threshold that is currently FIRING, so leaving a + // condition takes more movement than entering it. 1.3 is roughly a third + // of a tolerance band of dead zone. + double hysteresis = 1.3; +}; + +inline bool framing_condition_is(FramingCondition c, FramingCondition want) +{ + return c == want; +} + +// Returns the single condition to show. `previous` is what is on screen right +// now; pass FramingCondition::Good on the first evaluation. +inline FramingCondition evaluate_framing(const SubjectFrame &s, + const FramingThresholds &t, + FramingCondition previous) +{ + if (!s.found) return FramingCondition::NoSubject; + + const double bx = static_cast(s.box_x); + const double by = static_cast(s.box_y); + const double bw = static_cast(s.box_w); + const double bh = static_cast(s.box_h); + const double eye_y = 0.5 * (static_cast(s.eye_l_y) + + static_cast(s.eye_r_y)); + if (!std::isfinite(bx) || !std::isfinite(by) || !std::isfinite(bw) || + !std::isfinite(bh) || !std::isfinite(eye_y) || bw <= 0.0 || bh <= 0.0) + return FramingCondition::NoSubject; + + const double hys = std::max(t.hysteresis, 1.0); + // A threshold is widened only for the condition that is currently showing, + // which is what makes leaving harder than entering. + const auto band = [&](FramingCondition c, double tolerance) { + return previous == c ? tolerance * hys : tolerance; + }; + + // 1. Cut off - part of them is missing, so nothing else matters. + if (bx <= band(FramingCondition::CutOffLeft, t.edge_margin)) + return FramingCondition::CutOffLeft; + if (bx + bw >= 1.0 - band(FramingCondition::CutOffRight, t.edge_margin)) + return FramingCondition::CutOffRight; + if (by <= band(FramingCondition::CutOffTop, t.edge_margin)) + return FramingCondition::CutOffTop; + if (by + bh >= 1.0 - band(FramingCondition::CutOffBottom, t.edge_margin)) + return FramingCondition::CutOffBottom; + + // 2. Distance - moving fixes framing and eyeline together, so it comes + // before camera height. + if (bh > t.max_box_height * + (previous == FramingCondition::TooClose ? 1.0 / hys : 1.0)) + return FramingCondition::TooClose; + if (bh < t.min_box_height * + (previous == FramingCondition::TooFar ? hys : 1.0)) + return FramingCondition::TooFar; + + // 3. Camera height. Above the line (smaller y) => the camera is too high. + const double eye_offset = eye_y - t.eyeline_target; + if (eye_offset < -band(FramingCondition::CameraTooHigh, t.eyeline_tolerance)) + return FramingCondition::CameraTooHigh; + if (eye_offset > band(FramingCondition::CameraTooLow, t.eyeline_tolerance)) + return FramingCondition::CameraTooLow; + + // 4. Horizontal placement, last because it is the least damaging. + const double centre_offset = (bx + 0.5 * bw) - 0.5; + if (centre_offset > band(FramingCondition::OffCentreRight, t.centre_tolerance)) + return FramingCondition::OffCentreRight; + if (centre_offset < -band(FramingCondition::OffCentreLeft, t.centre_tolerance)) + return FramingCondition::OffCentreLeft; + + return FramingCondition::Good; +} +``` + +- [ ] **Step 4: Run the test to verify it passes** + +Run: +``` +cmake --build build --config Release --parallel 8 --target CoreVideoFramingAdviceTest +ctest -C Release -R CoreVideoFramingAdvice --output-on-failure +``` +Expected: PASS, `framing-advice: all checks passed` + +- [ ] **Step 5: Commit** + +```bash +git add src/zoom-framing-advice.h tests/framing-advice-test.cpp CMakeLists.txt +git commit -m "feat: framing advice predicates with explicit thresholds" +``` + +--- + +## Task 9: Overlay layout, legible at 640×360 (pure) + +**The constraint:** a gallery tile occupies roughly **640×360 of physical screen space** on a panelist's display. Spotlight is explicitly not required and must not be assumed, so 640×360 is the design size, not 1080p. + +**The consequence, decided here:** the overlay draws **no text at all**. Words would need either a font atlas or a nested OBS text source, and at 640×360 a legible word is a very small number of very large glyphs — a worse signal than an arrow. So each condition maps to a geometric symbol built from solid quads: direction arrows (which way to move), inward/outward brackets (distance), a camera-tilt chevron pair (height), and a full-frame pulsing border (no subject). Every quad has a short side of at least 24 px at 640×360, which is ~6.7% of frame height — no hairlines, nothing that survives a Zoom re-encode as mush. + +**Files:** +- Create: `src/zoom-framing-overlay-layout.h` +- Create: `tests/framing-overlay-layout-test.cpp` +- Modify: `CMakeLists.txt` + +**Interfaces:** +- Consumes: `FramingCondition` (Task 8). +- Produces: `struct OverlayQuad { double x, y, w, h; uint32_t argb; }`, `std::vector solve_overlay_quads(FramingCondition, double canvas_w, double canvas_h, uint32_t phase)`, and `kOverlayMinShortSidePx`. + +- [ ] **Step 1: Write the failing test** + +Create `tests/framing-overlay-layout-test.cpp`: + +```cpp +// The overlay is consumed at GALLERY-TILE size - roughly 640x360 of physical +// screen space on a panelist's display, whatever the encode. Spotlight is not +// required and is not assumed. So this file's job is to prove, at 640x360, +// that every element is chunky, on-canvas, and high contrast, for every +// condition. There is no headless GPU harness in this repo and one has been +// ruled against, so this is the verification. +#include "zoom-framing-overlay-layout.h" + +#include +#include +#include +#include +#include + +static int g_failures = 0; + +static void check(bool ok, const std::string &what) +{ + if (!ok) { + std::cerr << "FAIL: " << what << "\n"; + ++g_failures; + } +} + +static const FramingCondition kAll[] = { + FramingCondition::Good, FramingCondition::NoSubject, + FramingCondition::CutOffLeft, FramingCondition::CutOffRight, + FramingCondition::CutOffTop, FramingCondition::CutOffBottom, + FramingCondition::TooClose, FramingCondition::TooFar, + FramingCondition::CameraTooHigh, FramingCondition::CameraTooLow, + FramingCondition::OffCentreLeft, FramingCondition::OffCentreRight, +}; + +// Relative luminance, sRGB, for the contrast ratio below. +static double luminance(uint32_t argb) +{ + const auto channel = [](uint32_t v) { + const double c = static_cast(v) / 255.0; + return c <= 0.04045 ? c / 12.92 : std::pow((c + 0.055) / 1.055, 2.4); + }; + const double r = channel((argb >> 16) & 0xFF); + const double g = channel((argb >> 8) & 0xFF); + const double b = channel(argb & 0xFF); + return 0.2126 * r + 0.7152 * g + 0.0722 * b; +} + +int main() +{ + // The design size. Not 1080p - see the header comment. + const double w = 640.0, h = 360.0; + + for (const FramingCondition c : kAll) { + const std::vector quads = solve_overlay_quads(c, w, h, 0); + + if (c == FramingCondition::Good) { + check(quads.empty(), + "a correctly framed panelist is shown nothing at all"); + continue; + } + check(!quads.empty(), "every advice condition draws something"); + + double covered = 0.0; + for (const OverlayQuad &q : quads) { + check(std::min(q.w, q.h) >= kOverlayMinShortSidePx, + "no element is thinner than the minimum at 640x360"); + check(q.x >= 0.0 && q.y >= 0.0 && q.x + q.w <= w && q.y + q.h <= h, + "every element is inside the canvas"); + check(((q.argb >> 24) & 0xFF) >= 0xC0, + "every element is near-opaque - no washed-out overlays"); + + // Hard contrast against BOTH a black and a white background, since + // we are drawn over arbitrary video. + const double lum = luminance(q.argb); + const double vs_black = (lum + 0.05) / 0.05; + const double vs_white = 1.05 / (lum + 0.05); + check(std::max(vs_black, vs_white) >= 4.5, + "every element clears 4.5:1 against black or white"); + + covered += q.w * q.h; + } + // Big enough to read across a room, on a tile the size of a postcard. + check(covered >= 0.04 * w * h, + "the symbol occupies a readable share of the tile"); + } + + // NoSubject pulses: the phase changes the drawing, so a panelist who has + // walked away sees motion rather than a static frame they mistake for a + // frozen picture. + { + const std::vector a = + solve_overlay_quads(FramingCondition::NoSubject, w, h, 0); + const std::vector b = + solve_overlay_quads(FramingCondition::NoSubject, w, h, 30); + check(a.size() == b.size(), "the pulse does not change the element count"); + bool differs = false; + for (std::size_t i = 0; i < a.size() && i < b.size(); ++i) + if (a[i].argb != b[i].argb) differs = true; + check(differs, "the no-subject symbol actually pulses"); + } + + // Arrows point the way the panelist must MOVE. Cut off on the left means + // move right, so the arrow's mass sits right of centre. + { + const auto mass_centre_x = [&](FramingCondition c) { + const std::vector q = solve_overlay_quads(c, w, h, 0); + double sum = 0.0, area = 0.0; + for (const OverlayQuad &e : q) { + sum += (e.x + 0.5 * e.w) * e.w * e.h; + area += e.w * e.h; + } + return area > 0.0 ? sum / area : 0.5 * w; + }; + check(mass_centre_x(FramingCondition::CutOffLeft) > 0.5 * w, + "cut off on the left points right"); + check(mass_centre_x(FramingCondition::CutOffRight) < 0.5 * w, + "cut off on the right points left"); + check(mass_centre_x(FramingCondition::OffCentreRight) < 0.5 * w, + "a subject right of centre is pointed left"); + check(mass_centre_x(FramingCondition::OffCentreLeft) > 0.5 * w, + "a subject left of centre is pointed right"); + } + + // And it must still be sane at 1080p, where an operator previews it. + for (const FramingCondition c : kAll) { + const std::vector quads = + solve_overlay_quads(c, 1920.0, 1080.0, 0); + for (const OverlayQuad &q : quads) + check(q.x >= 0.0 && q.y >= 0.0 && q.x + q.w <= 1920.0 && + q.y + q.h <= 1080.0, + "every element is inside a 1080p canvas too"); + } + + // A canvas too small to draw a legible symbol draws NOTHING rather than + // hairlines. Illegible advice is worse than none. + { + const std::vector tiny = + solve_overlay_quads(FramingCondition::TooFar, 64.0, 36.0, 0); + check(tiny.empty(), "a tiny canvas draws nothing rather than hairlines"); + } + + if (g_failures == 0) std::cout << "framing-overlay-layout: all checks passed\n"; + return g_failures == 0 ? 0 : 1; +} +``` + +- [ ] **Step 2: Register the target and run it to verify it fails** + +In `CMakeLists.txt`, inside `if(BUILD_TESTING)`, after `CoreVideoFramingAdviceTest`: + +```cmake + # Overlay legibility at 640x360 - the physical size of a gallery tile on a + # panelist's screen. Chunky, on-canvas, high-contrast, no hairlines. + add_executable(CoreVideoFramingOverlayLayoutTest + tests/framing-overlay-layout-test.cpp + ) + target_include_directories(CoreVideoFramingOverlayLayoutTest PRIVATE + "${CMAKE_CURRENT_SOURCE_DIR}/src" + ) + add_test(NAME CoreVideoFramingOverlayLayout + COMMAND CoreVideoFramingOverlayLayoutTest) +``` + +Run: `cmake --build build --config Release --parallel 8 --target CoreVideoFramingOverlayLayoutTest` +Expected: FAIL — `Cannot open include file: 'zoom-framing-overlay-layout.h'` + +- [ ] **Step 3: Write the header** + +Create `src/zoom-framing-overlay-layout.h`: + +```cpp +#pragma once + +// Why this exists +// --------------- +// The framing overlay is consumed by a panelist looking at a GALLERY TILE: +// roughly 640x360 of physical screen space on their display, however clean the +// Zoom encode is. Spotlighting the return feed would not change that and is +// explicitly NOT required, so 640x360 is the design size and 1080p is only +// where the operator previews it. +// +// At that size the overlay draws NO TEXT. Words need a font atlas or a nested +// text source, and a word legible at 640x360 is a handful of enormous glyphs - +// a worse signal than an arrow that means the same thing. So every condition +// is a symbol built from solid quads: +// +// cut off / off centre -> a thick arrow pointing the way to MOVE +// too close / too far -> brackets closing in / opening out +// camera too high / low -> a chevron pair pointing the way to TILT +// no subject -> a pulsing full-frame border +// good -> nothing at all +// +// Rules the test enforces, so they cannot quietly rot: +// * every quad's short side >= kOverlayMinShortSidePx at 640x360, +// * every quad inside the canvas, +// * alpha >= 0xC0 and >= 4.5:1 contrast against black or white, +// * the symbol covers >= 4% of the frame, +// * a canvas too small for a legible symbol draws nothing. +// +// Everything is expressed as a FRACTION of the canvas and scaled at the end, so +// 640x360 and 1920x1080 are the same picture. + +#include "zoom-framing-advice.h" + +#include +#include + +struct OverlayQuad { + double x = 0.0, y = 0.0, w = 0.0, h = 0.0; // canvas pixels + uint32_t argb = 0xFFFFFFFF; +}; + +// The thinnest anything may be at the 640x360 design size. 24 px is ~6.7% of +// frame height: it survives a Zoom re-encode as a bar, not as mush. +constexpr double kOverlayMinShortSidePx = 24.0; + +// The design canvas everything is authored against. +constexpr double kOverlayDesignWidth = 640.0; +constexpr double kOverlayDesignHeight = 360.0; + +// Amber for "fix something", red for "we cannot see you". Both clear 4.5:1 +// against white and against black, which matters because we are drawn over +// arbitrary video. +constexpr uint32_t kOverlayAdviceColor = 0xFFE8A317; // amber +constexpr uint32_t kOverlayAlertColor = 0xFFE01B24; // red + +namespace overlay_layout_detail { + +// A quad in 0..1 canvas fractions, resolved at the end. +struct FracQuad { + double x, y, w, h; + uint32_t argb; +}; + +// A blocky arrow: one shaft plus a staircase head, so it is all axis-aligned +// quads and needs no triangle geometry. dx/dy is the unit direction. +inline void push_arrow(std::vector &out, double cx, double cy, + double length, double thickness, int dx, int dy, + uint32_t argb) +{ + if (dx != 0) { + out.push_back({cx - 0.5 * length, cy - 0.5 * thickness, length, + thickness, argb}); + // Three steps of decreasing height at the tip. + for (int step = 1; step <= 3; ++step) { + const double step_h = thickness * (1.0 + 2.0 * (3 - step) * 0.6); + const double step_w = thickness * 0.75; + const double tip_x = dx > 0 + ? cx + 0.5 * length - step_w * static_cast(step) + : cx - 0.5 * length + step_w * static_cast(step - 1); + out.push_back({tip_x, cy - 0.5 * step_h, step_w, step_h, argb}); + } + return; + } + out.push_back({cx - 0.5 * thickness, cy - 0.5 * length, thickness, length, + argb}); + for (int step = 1; step <= 3; ++step) { + const double step_w = thickness * (1.0 + 2.0 * (3 - step) * 0.6); + const double step_h = thickness * 0.75; + const double tip_y = dy > 0 + ? cy + 0.5 * length - step_h * static_cast(step) + : cy - 0.5 * length + step_h * static_cast(step - 1); + out.push_back({cx - 0.5 * step_w, tip_y, step_w, step_h, argb}); + } +} + +// Two vertical bars, either closing in (move back) or opening out (move +// closer), with short returns top and bottom so they read as brackets. +inline void push_brackets(std::vector &out, bool inward, + uint32_t argb) +{ + const double t = 0.05; // bar thickness, fraction of width + const double inset = inward ? 0.30 : 0.12; + const double top = 0.22, height = 0.56; + const double arm = 0.10; + // Left bracket. + out.push_back({inset, top, t, height, argb}); + out.push_back({inset, top, arm, t * (kOverlayDesignWidth / + kOverlayDesignHeight), argb}); + out.push_back({inset, top + height - t * (kOverlayDesignWidth / + kOverlayDesignHeight), + arm, t * (kOverlayDesignWidth / kOverlayDesignHeight), argb}); + // Right bracket, mirrored. + out.push_back({1.0 - inset - t, top, t, height, argb}); + out.push_back({1.0 - inset - arm, top, + arm, t * (kOverlayDesignWidth / kOverlayDesignHeight), argb}); + out.push_back({1.0 - inset - arm, + top + height - t * (kOverlayDesignWidth / + kOverlayDesignHeight), + arm, t * (kOverlayDesignWidth / kOverlayDesignHeight), argb}); +} + +// A full-frame border, four bars, alpha pulsing with the phase. +inline void push_border(std::vector &out, uint32_t phase, + uint32_t rgb) +{ + // 0..59 -> a triangle wave between 0xC0 and 0xFF, so it is always at least + // near-opaque. Integer maths so the test can predict it. + const uint32_t p = phase % 60u; + const uint32_t up = p < 30u ? p : 59u - p; // 0..29 + const uint32_t alpha = 0xC0u + up * 2u; // 0xC0..0xFA + const uint32_t argb = (alpha << 24) | (rgb & 0x00FFFFFFu); + const double tx = 0.06, ty = 0.06 * (kOverlayDesignWidth / + kOverlayDesignHeight); + out.push_back({0.0, 0.0, 1.0, ty, argb}); + out.push_back({0.0, 1.0 - ty, 1.0, ty, argb}); + out.push_back({0.0, ty, tx, 1.0 - 2.0 * ty, argb}); + out.push_back({1.0 - tx, ty, tx, 1.0 - 2.0 * ty, argb}); +} + +} // namespace overlay_layout_detail + +// The quads to draw for a condition, in canvas pixels. `phase` is a monotonic +// frame counter used only by the pulsing no-subject border. +inline std::vector solve_overlay_quads(FramingCondition condition, + double canvas_w, + double canvas_h, + uint32_t phase) +{ + using overlay_layout_detail::FracQuad; + std::vector out; + if (condition == FramingCondition::Good) return out; + if (!(canvas_w > 0.0) || !(canvas_h > 0.0)) return out; + // Below the design size a legible symbol cannot be drawn, and hairlines are + // worse than nothing. Refuse. + if (canvas_w < kOverlayDesignWidth || canvas_h < kOverlayDesignHeight) + return out; + + // Arrow geometry, in fractions. The shaft is 8% of frame height thick, + // which is 28.8 px at 640x360 - comfortably over the 24 px floor - and the + // head steps are 75% of that. + const double thick_y = 0.09; + const double thick_x = thick_y * (canvas_h / canvas_w); + const double len_x = 0.52; + const double len_y = 0.52; + + std::vector frac; + switch (condition) { + case FramingCondition::NoSubject: + overlay_layout_detail::push_border(frac, phase, kOverlayAlertColor); + break; + // Cut off on a side means MOVE AWAY from that side, so the arrow points the + // other way and its mass sits on the far side of centre. The offset is what + // the mass-centre assertions in the test key on. + case FramingCondition::CutOffLeft: + overlay_layout_detail::push_arrow(frac, 0.62, 0.5, len_x, thick_x, +1, 0, + kOverlayAlertColor); + break; + case FramingCondition::CutOffRight: + overlay_layout_detail::push_arrow(frac, 0.38, 0.5, len_x, thick_x, -1, 0, + kOverlayAlertColor); + break; + case FramingCondition::CutOffTop: + overlay_layout_detail::push_arrow(frac, 0.5, 0.62, len_y, thick_y, 0, +1, + kOverlayAlertColor); + break; + case FramingCondition::CutOffBottom: + overlay_layout_detail::push_arrow(frac, 0.5, 0.38, len_y, thick_y, 0, -1, + kOverlayAlertColor); + break; + case FramingCondition::TooClose: + overlay_layout_detail::push_brackets(frac, /*inward=*/false, + kOverlayAdviceColor); + break; + case FramingCondition::TooFar: + overlay_layout_detail::push_brackets(frac, /*inward=*/true, + kOverlayAdviceColor); + break; + // Camera too high => tilt it DOWN => the arrow points down. + case FramingCondition::CameraTooHigh: + overlay_layout_detail::push_arrow(frac, 0.5, 0.5, len_y, thick_y, 0, +1, + kOverlayAdviceColor); + break; + case FramingCondition::CameraTooLow: + overlay_layout_detail::push_arrow(frac, 0.5, 0.5, len_y, thick_y, 0, -1, + kOverlayAdviceColor); + break; + case FramingCondition::OffCentreRight: + overlay_layout_detail::push_arrow(frac, 0.38, 0.5, len_x, thick_x, -1, 0, + kOverlayAdviceColor); + break; + case FramingCondition::OffCentreLeft: + overlay_layout_detail::push_arrow(frac, 0.62, 0.5, len_x, thick_x, +1, 0, + kOverlayAdviceColor); + break; + case FramingCondition::Good: + default: + return out; + } + + out.reserve(frac.size()); + for (const FracQuad &f : frac) { + OverlayQuad q; + q.x = f.x * canvas_w; + q.y = f.y * canvas_h; + q.w = f.w * canvas_w; + q.h = f.h * canvas_h; + q.argb = f.argb; + // Clamp into the canvas. Every fraction above is already inside 0..1, + // so this only absorbs floating-point edge cases - it must never be + // load-bearing, which is why the test checks bounds independently. + if (q.x < 0.0) { q.w += q.x; q.x = 0.0; } + if (q.y < 0.0) { q.h += q.y; q.y = 0.0; } + if (q.x + q.w > canvas_w) q.w = canvas_w - q.x; + if (q.y + q.h > canvas_h) q.h = canvas_h - q.y; + if (q.w <= 0.0 || q.h <= 0.0) continue; + out.push_back(q); + } + return out; +} +``` + +- [ ] **Step 4: Run the test, and tune the geometry until it passes** + +Run: +``` +cmake --build build --config Release --parallel 8 --target CoreVideoFramingOverlayLayoutTest +ctest -C Release -R CoreVideoFramingOverlayLayout --output-on-failure +``` +Expected: PASS. If the minimum-short-side or coverage assertions fail, raise `thick_y` / `len_x` / `len_y` and the bracket `t` — do **not** relax the assertions, they are the feature's acceptance criteria. + +- [ ] **Step 5: Commit** + +```bash +git add src/zoom-framing-overlay-layout.h tests/framing-overlay-layout-test.cpp CMakeLists.txt +git commit -m "feat: framing overlay layout, pinned legible at 640x360" +``` + +--- + +## Task 10: The `corevideo_framing_overlay` OBS source + +**Design note:** the existing `corevideo_active_speaker_source` (`src/zoom-source.cpp:2875`) is an **async** source (`OBS_SOURCE_ASYNC_VIDEO`) — it pushes finished frames and has no render hook to draw into. So the overlay is a **separate custom-draw source** the operator stacks above the active speaker in the return scene. That keeps the async video path completely untouched, and keeps the overlay's decisions on the CPU where they are testable. It polls `SpeakerDirector` — which is **poll-only, there is no observer list** — for who is on air. + +**Files:** +- Create: `src/zoom-framing-overlay.h`, `src/zoom-framing-overlay.cpp` +- Modify: `src/zoom-plugin.cpp`, `CMakeLists.txt`, `data/locale/en-US.ini` + +**Interfaces:** +- Consumes: `solve_overlay_quads()` (Task 9), `evaluate_framing()` (Task 8), `subject_for()` (Task 5), `ReturnIdentityRegistry` (Task 1), `SpeakerDirector::instance().snapshot()`. +- Produces: `void zoom_framing_overlay_register();` + +- [ ] **Step 1: Write the header** + +Create `src/zoom-framing-overlay.h`: + +```cpp +#pragma once + +// The framing-advice overlay source. +// +// A transparent, custom-draw OBS source that shows the panelist currently on +// air one geometric instruction about their framing. Stacked ABOVE +// corevideo_active_speaker_source in the return scene, which goes out over the +// OBS Virtual Camera and back into the meeting on a separate Zoom seat. +// +// Separate from the active-speaker source because that source is ASYNC +// (OBS_SOURCE_ASYNC_VIDEO): it pushes finished frames and has no render hook to +// draw into. Layering keeps the async video path untouched. +void zoom_framing_overlay_register(); +``` + +- [ ] **Step 2: Write the source** + +Create `src/zoom-framing-overlay.cpp`: + +```cpp +#include "zoom-framing-overlay.h" + +#include "speaker-director.h" +#include "zoom-engine-client.h" +#include "zoom-framing-advice.h" +#include "zoom-framing-overlay-layout.h" +#include "zoom-self-identity.h" +#include "zoom-subject-source.h" + +#include +#include + +#include +#include +#include +#include +#include + +static const char *const kFramingOverlaySourceId = "corevideo_framing_overlay"; + +// Per-source settings, obs_data + atomics — the PROP_ANIMATE pattern in +// zoom-supersource.cpp, NOT the global-ini ZoomPluginSettings mechanism. +static constexpr const char *PROP_CANVAS_W = "canvas_width"; +static constexpr const char *PROP_CANVAS_H = "canvas_height"; +static constexpr const char *PROP_RETURN_SEAT = "return_seat_name"; + +// Set COREVIDEO_FRAMING_LAYOUT_TEST to cycle every condition in the REAL +// source, in real OBS, one per second. The same in-product visual check +// COREVIDEO_TALKBACK_LAYOUT_TEST provides for the intercom dock — there is no +// headless GPU harness in this repo and one has been ruled against, so this is +// how a human eyeballs the symbols at tile size. +static const char *const kFramingLayoutTestEnv = "COREVIDEO_FRAMING_LAYOUT_TEST"; + +struct framing_overlay_source { + obs_source_t *source = nullptr; + + std::atomic canvas_width{1920}; + std::atomic canvas_height{1080}; + + // Everything the render reads, written by the tick, snapshotted as a unit. + std::mutex mutex; + std::vector quads; + + // Tick-thread state; never touched by render. + FramingCondition previous = FramingCondition::Good; + uint32_t phase = 0; + bool layout_test = false; +}; + +static const char *framing_overlay_get_name(void *) +{ + return obs_module_text("CoreVideoFramingOverlay.Name"); +} + +static uint32_t framing_overlay_get_width(void *data) +{ + return static_cast(data)->canvas_width.load( + std::memory_order_acquire); +} + +static uint32_t framing_overlay_get_height(void *data) +{ + return static_cast(data)->canvas_height.load( + std::memory_order_acquire); +} + +static void framing_overlay_update(void *data, obs_data_t *settings) +{ + auto *ctx = static_cast(data); + + // Clamped on the same threat model as every other setting in this plugin: + // scene files are hand-editable and obs_data_get_int returns an int64. + const auto clamp_dim = [](int64_t raw, int64_t fallback) { + if (raw < 160 || raw > 7680) return fallback; + return raw; + }; + ctx->canvas_width.store( + static_cast( + clamp_dim(obs_data_get_int(settings, PROP_CANVAS_W), 1920)), + std::memory_order_release); + ctx->canvas_height.store( + static_cast( + clamp_dim(obs_data_get_int(settings, PROP_CANVAS_H), 1080)), + std::memory_order_release); + + // The return seat's display name. This is what makes the exclusion durable: + // that seat is a separate Zoom account whose user_id changes on every + // rejoin, so it is matched by name and re-derived on every roster message. + // See zoom-self-identity.h. + const char *seat = obs_data_get_string(settings, PROP_RETURN_SEAT); + ReturnIdentityRegistry::instance().set_names( + {seat ? std::string(seat) : std::string()}); +} + +static void framing_overlay_get_defaults(obs_data_t *settings) +{ + obs_data_set_default_int(settings, PROP_CANVAS_W, 1920); + obs_data_set_default_int(settings, PROP_CANVAS_H, 1080); + // Ships EMPTY. Naming the seat is both the destination and the on-switch, + // exactly as the Tiles audio group is: an empty name must not match every + // unnamed participant in somebody's meeting on upgrade. + obs_data_set_default_string(settings, PROP_RETURN_SEAT, ""); +} + +static obs_properties_t *framing_overlay_get_properties(void *) +{ + obs_properties_t *props = obs_properties_create(); + obs_properties_add_text(props, PROP_RETURN_SEAT, + obs_module_text("CoreVideoFramingOverlay.ReturnSeat"), + OBS_TEXT_DEFAULT); + obs_properties_add_int(props, PROP_CANVAS_W, + obs_module_text("CoreVideoFramingOverlay.CanvasWidth"), + 160, 7680, 2); + obs_properties_add_int(props, PROP_CANVAS_H, + obs_module_text("CoreVideoFramingOverlay.CanvasHeight"), + 160, 4320, 2); + return props; +} + +static void *framing_overlay_create(obs_data_t *settings, obs_source_t *source) +{ + auto *ctx = new framing_overlay_source(); + ctx->source = source; + ctx->layout_test = os_getenv(kFramingLayoutTestEnv) != nullptr; + if (ctx->layout_test) { + blog(LOG_INFO, + "[obs-zoom-plugin] Framing overlay: %s set — cycling every " + "condition, one per second, with no detector", + kFramingLayoutTestEnv); + } + framing_overlay_update(ctx, settings); + return ctx; +} + +static void framing_overlay_destroy(void *data) +{ + delete static_cast(data); +} + +// Who is on air. SpeakerDirector is POLL-ONLY — there is no observer list — so +// this asks it once per tick and does not wait for an event that never comes. +static uint32_t framing_overlay_on_air() +{ + const uint64_t now_ms = os_gettime_ns() / 1000000ULL; + const SpeakerDirectorSnapshot snap = + SpeakerDirector::instance().snapshot(now_ms); + return snap.directed_speaker_id; +} + +static void framing_overlay_tick(void *data, float /*seconds*/) +{ + auto *ctx = static_cast(data); + ++ctx->phase; + + FramingCondition condition = FramingCondition::Good; + if (ctx->layout_test) { + // One condition per second at 60 fps, in enum order, skipping Good so + // every symbol is actually seen. + static const FramingCondition kCycle[] = { + FramingCondition::NoSubject, FramingCondition::CutOffLeft, + FramingCondition::CutOffRight, FramingCondition::CutOffTop, + FramingCondition::CutOffBottom, FramingCondition::TooClose, + FramingCondition::TooFar, FramingCondition::CameraTooHigh, + FramingCondition::CameraTooLow, FramingCondition::OffCentreLeft, + FramingCondition::OffCentreRight, + }; + constexpr uint32_t kCount = + static_cast(sizeof(kCycle) / sizeof(kCycle[0])); + condition = kCycle[(ctx->phase / 60u) % kCount]; + } else { + const uint32_t on_air = framing_overlay_on_air(); + if (on_air == 0) { + // Nobody directed: say nothing rather than advising an empty chair. + condition = FramingCondition::Good; + } else { + condition = evaluate_framing(subject_for(on_air), FramingThresholds{}, + ctx->previous); + } + } + ctx->previous = condition; + + std::vector quads = solve_overlay_quads( + condition, + static_cast(ctx->canvas_width.load(std::memory_order_acquire)), + static_cast(ctx->canvas_height.load(std::memory_order_acquire)), + ctx->phase); + + std::lock_guard lock(ctx->mutex); + ctx->quads = std::move(quads); +} + +static void framing_overlay_render(void *data, gs_effect_t *) +{ + auto *ctx = static_cast(data); + + std::vector quads; + { + // Snapshotted as a unit, for the same reason the Tiles wall snapshots + // its crop under ctx->mutex: half of one symbol and half of the next is + // not a picture anybody can interpret. + std::lock_guard lock(ctx->mutex); + quads = ctx->quads; + } + if (quads.empty()) return; // Good draws nothing at all + + gs_effect_t *const solid = obs_get_base_effect(OBS_EFFECT_SOLID); + if (!solid) return; + gs_eparam_t *const color = gs_effect_get_param_by_name(solid, "color"); + if (!color) return; + + gs_blend_state_push(); + gs_enable_blending(true); + gs_blend_function(GS_BLEND_SRCALPHA, GS_BLEND_INVSRCALPHA); + + for (const OverlayQuad &q : quads) { + const uint32_t w = static_cast(q.w); + const uint32_t h = static_cast(q.h); + if (w == 0 || h == 0) continue; + + // vec4_from_rgba takes ABGR-ordered bytes; our quads are ARGB, so the + // red and blue bytes swap here. Getting this wrong is a silent colour + // bug, not a crash — amber renders as blue and still "works". + struct vec4 fill; + const uint32_t a = (q.argb >> 24) & 0xFF; + const uint32_t r = (q.argb >> 16) & 0xFF; + const uint32_t g = (q.argb >> 8) & 0xFF; + const uint32_t b = q.argb & 0xFF; + vec4_from_rgba(&fill, (a << 24) | (b << 16) | (g << 8) | r); + gs_effect_set_vec4(color, &fill); + + gs_technique_t *const tech = gs_effect_get_technique(solid, "Solid"); + gs_technique_begin(tech); + if (gs_technique_begin_pass(tech, 0)) { + gs_matrix_push(); + gs_matrix_translate3f(static_cast(q.x), + static_cast(q.y), 0.0f); + gs_draw_sprite(nullptr, 0, w, h); + gs_matrix_pop(); + gs_technique_end_pass(tech); + } + gs_technique_end(tech); + } + + gs_blend_state_pop(); +} + +void zoom_framing_overlay_register() +{ + obs_source_info info = {}; + info.id = kFramingOverlaySourceId; + info.type = OBS_SOURCE_TYPE_INPUT; + // CUSTOM_DRAW because it binds its own effect; DO_NOT_DUPLICATE because two + // copies would both poll the director and neither would be wrong, but the + // second one buys nothing. + info.output_flags = OBS_SOURCE_VIDEO | OBS_SOURCE_CUSTOM_DRAW | + OBS_SOURCE_DO_NOT_DUPLICATE; + info.get_name = framing_overlay_get_name; + info.create = framing_overlay_create; + info.destroy = framing_overlay_destroy; + info.update = framing_overlay_update; + info.video_tick = framing_overlay_tick; + info.video_render = framing_overlay_render; + info.get_width = framing_overlay_get_width; + info.get_height = framing_overlay_get_height; + info.get_properties = framing_overlay_get_properties; + info.get_defaults = framing_overlay_get_defaults; + obs_register_source(&info); +} +``` + +- [ ] **Step 3: Register it and add the strings** + +In `src/zoom-plugin.cpp`, beside the other `*_register()` calls in `obs_module_load`: + +```cpp + zoom_framing_overlay_register(); +``` + +and the include with the others: + +```cpp +#include "zoom-framing-overlay.h" +``` + +In `CMakeLists.txt`, add `src/zoom-framing-overlay.cpp` to the plugin target's source list, beside `src/zoom-supersource.cpp`. + +In `data/locale/en-US.ini`: + +```ini +CoreVideoFramingOverlay.Name="CoreVideo Framing Advice" +CoreVideoFramingOverlay.ReturnSeat="Return-feed Zoom display name (excluded from tiles and speaker)" +CoreVideoFramingOverlay.CanvasWidth="Canvas width" +CoreVideoFramingOverlay.CanvasHeight="Canvas height" +``` + +- [ ] **Step 4: Build and run the suite** + +Run: +``` +cmake --build build --config Release --parallel 8 +ctest -C Release --output-on-failure +``` +Expected: builds clean, N/N green. + +- [ ] **Step 5: In-product visual check at tile size** + +Run OBS with the env var set, add a **CoreVideo Framing Advice** source to a scene, and set the preview to roughly 640×360 on screen: + +``` +cmd /c "set COREVIDEO_FRAMING_LAYOUT_TEST=1 && Launch-OBS-CoreVideo.cmd" +``` + +Confirm by eye, at that size: every symbol is unmistakable, the arrows point where you would move, nothing is thin, and the no-subject border pulses. This is the sanctioned substitute for a GPU harness — do not add one. + +- [ ] **Step 6: Commit** + +```bash +git add src/zoom-framing-overlay.h src/zoom-framing-overlay.cpp src/zoom-plugin.cpp CMakeLists.txt data/locale/en-US.ini +git commit -m "feat: corevideo_framing_overlay source for the vcam return feed" +``` + +--- + +## Task 11: Documentation and the return-scene runbook + +**Files:** +- Modify: `CLAUDE.md` +- Modify: `CHANGELOG.md` +- Create: `docs/framing-return-feed.md` + +- [ ] **Step 1: Write the operator runbook** + +Create `docs/framing-return-feed.md`: + +```markdown +# Framing advice return feed (preshow) + +This is a **preshow** tool: green room, mic check, rehearsal. The return feed +is visible to everyone in the meeting, which is the intended delivery, not a +leak. Nothing here touches a live program shot. + +## Setting it up + +1. In OBS, make a scene containing: + - **CoreVideo Active Speaker** (the existing `corevideo_active_speaker_source`), and + - **CoreVideo Framing Advice** on top of it. +2. Start the **OBS Virtual Camera**. +3. On a **second machine or a second Zoom account**, join the meeting and + select **OBS Virtual Camera** as the webcam. It must be a separate account: + same-account joins collide (see the ZComms talkback findings). +4. In the Framing Advice source's properties, type that seat's **Zoom display + name** into *Return-feed Zoom display name*. + +Step 4 is not cosmetic. It is what stops the return feed being shown on the +Tiles wall or directed to as the active speaker — which would close a video +feedback loop. The match is by name and is re-derived on every roster message, +so it survives that seat rejoining, which a user-id-based exclusion does not. + +The bot's own SDK identity is excluded automatically (the engine stamps it from +`GetMySelfUser()`), including while talkback has the bot unmuted. + +## Reading it + +| Symbol | Meaning | +|---|---| +| Pulsing red border | We cannot see you — step into frame | +| Red arrow | You are cut off; move the way it points | +| Amber up/down arrow | Tilt your camera the way it points | +| Amber brackets closing in | Move closer | +| Amber brackets opening out | Move back | +| Amber left/right arrow | Shift the way it points | +| Nothing | You are framed correctly | + +There is deliberately no text: the overlay is read on a gallery tile roughly +640x360 in physical screen size, and at that size a symbol beats a word. +Spotlighting the return is not required. + +## Tiles auto-framing + +Off by default, per-wall, in the Tiles source properties. It replaces the +per-tile crop sliders for any tile with a detected subject and falls straight +back to them when the subject is lost. + +It never raises the Zoom subscription for anybody. It frames from whatever the +shared feed already is: a participant who is ISO'd or on a program output is +already carried at high resolution and is framed from those pixels for free, +while a participant only the tile wall is showing is carried at 360p and their +auto-framed tile will look softer. + +**A soft auto-framed tile is a diagnosis, not a defect.** It means nobody is +paying for that participant's pixels. The remedy is to ISO them, which you can +already do, and Tiles picks the better feed up automatically. +``` + +- [ ] **Step 2: Update CLAUDE.md** + +Add to `CLAUDE.md`, in the section listing sources and pure headers: + +```markdown +- **Framing consumers** (`src/zoom-auto-frame.h`, `src/zoom-framing-advice.h`, + `src/zoom-framing-overlay-layout.h`, `src/zoom-framing-overlay.cpp`): Tiles + auto-framing and the Virtual-Camera return overlay. All decisions are pure + headers with pinned tests (`CoreVideoAutoFrame`, `CoreVideoFramingAdvice`, + `CoreVideoFramingOverlayLayout`); there is no GPU harness and one has been + ruled against. The overlay is designed and reviewed at **640x360**, the + physical size of a gallery tile. `COREVIDEO_FRAMING_LAYOUT_TEST` cycles every + condition in the real source. +- **Identity** (`src/zoom-self-identity.h`): `ParticipantInfo::is_self` is + stamped by the engine from `GetMySelfUser()` on every roster rebuild; + `is_return_identity` matches the vcam seat by display name. Both are + re-derived per roster message, so they survive a rejoin, and both are refused + by tile assignment AND by `SpeakerDirector`. Do not add another + user-id-keyed exclude combo box — that is the mechanism this replaced. +- **Auto-framing never requests a resolution upgrade.** It consumes whatever + the shared feed already is — the engine's subscription policy is upgrade-only, + so an ISO'd or program-output participant is already high-resolution and the + tile reuses that, while a wall-only participant stays at 360p and looks + softer. That is the diagnosis, not the defect: ISO them. Adding a second + resolution lever here would fight the ISO/program controls over the same + envelope, and a 720p wall throttled a live meeting to 0.3-0.45x real time on + 2026-08-17. `zoom-auto-frame.h`'s maths reads tex_w/tex_h and is pinned + resolution-invariant. +``` + +- [ ] **Step 3: Update CHANGELOG.md** + +Under the unreleased heading: + +```markdown +- Tiles: opt-in per-tile auto-framing (off by default). It frames from whatever + the shared feed already is and never raises a Zoom subscription, so it adds no + bandwidth of its own; a wall-only participant's tile will look softer than an + ISO'd one's, and ISO'ing them is the fix. +- Framing advice return feed: a new **CoreVideo Framing Advice** source that + tells the panelist on air how to fix their framing, for return over the OBS + Virtual Camera. Designed to read at gallery-tile size; no spotlight required. +- Durable self/return-identity exclusion: the bot and the return seat can no + longer appear on the Tiles wall or be directed to as the active speaker, and + the exclusion survives a rejoin — including while talkback has the bot + unmuted. +``` + +- [ ] **Step 4: Final full verification** + +Run: +``` +cmake --build build --config Release --parallel 8 +ctest -C Release --output-on-failure +``` +Expected: builds clean, N/N green. Record the actual N/N line in the commit body — evidence before assertions. + +- [ ] **Step 5: Commit** + +```bash +git add CLAUDE.md CHANGELOG.md docs/framing-return-feed.md +git commit -m "docs: framing return feed runbook and identity notes" +``` + +--- + +## Self-review + +**Spec coverage (Subsystem 3 only, as scoped):** + +| Spec item | Task | +|---|---| +| 3a. Tiles auto-frame, opt-in, per-tile crop rect, eyeline on upper third, centred, clamped to bounds and max zoom, off by default, no resolution upgrade | 6, 7 | +| 3b. Return overlay on the active speaker, over the OBS Virtual Camera | 8, 9, 10 | +| The seven overlay conditions with concrete predicates and boundary tests | 8 | +| Overlay legible at 640×360, chunky, hard contrast, no hairlines, no fine text, spotlight not assumed | 9, 10 | +| 3c. Durable self/return-identity exclusion applied to tiles AND speaker direction, with tests | 1, 2, 3, 4 | +| The four crop-insertion-point constraints, each tested | 6 (aspect, bounds, crop_uv from truncated ints), 7 (snapshot under `ctx->mutex`, and the insertion-point selection rule) | +| Resolution rule: consume the shared feed, never upgrade; crop math resolution-invariant | Resolution-rule block, Task 6 (the 360p-vs-1080p invariance test), Task 7 (no `subscribe()` call) | +| Per-source `obs_data` + atomics, never the global ini | 7, 10 | +| `SpeakerDirector` polled, no observer list | 10 | +| Plain `int main()` + local `check()`, `CoreVideoTest`, hand-registered in root CMakeLists | 1, 5, 6, 8, 9 | +| No GPU harness; env-var visual check | 9 (pure), 10 step 5 (`COREVIDEO_FRAMING_LAYOUT_TEST`) | + +**Out of scope by instruction, and deliberately absent:** the loudness engine and meter source (Subsystem 1), and the detector engine itself (Subsystem 2) — including the library vendoring, the worker thread, the round-robin schedule, and all smoothing. This plan consumes the detector solely through `subject_for()` (Task 5) and never re-smooths. + +**Type consistency:** `SubjectFrame` field names are used exactly as given in the contract everywhere. `AutoFrameResult{bool valid; CropRect crop;}` is produced in Task 6 and consumed under those names in Task 7. `FramingCondition` enumerators are produced in Task 8 and consumed by the same names in Tasks 9 and 10. `OverlayQuad{x,y,w,h,argb}` is produced in Task 9 and consumed in Task 10. `identity_excluded()` is defined in Task 1 and called in Tasks 3 and 4. No task defines or calls any resolution-selection function: nothing in this plan calls `subscribe()`, changes `tile_feed_subscribe`, or names a `VideoResolution`. diff --git a/docs/superpowers/plans/2026-09-05-loudness-engine-and-meter.md b/docs/superpowers/plans/2026-09-05-loudness-engine-and-meter.md new file mode 100644 index 00000000..2693bc5f --- /dev/null +++ b/docs/superpowers/plans/2026-09-05-loudness-engine-and-meter.md @@ -0,0 +1,2837 @@ +# Loudness Engine + Readiness-Board Meter Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Measure ITU-R BS.1770-4 loudness per Zoom panelist inside the OBS plugin, and render an operator-facing readiness board showing each panelist's deviation in LU from the panel median. + +**Architecture:** A pure, header-only DSP core (`src/audio-loudness.h`) derives its K-weighting biquad coefficients from the **runtime** sample rate read out of the SHM ring header, and is fed 16-bit interleaved PCM from inside the existing drain loop in `src/zoom-participant-audio-source.cpp` — on the audio lane thread, integrating across the whole drain rather than per wakeup. A second pure header (`src/loudness-board.h`) turns the per-source readings into a board model (panel median, per-panelist deviation, pass/fail, row geometry). A new OBS rendering source, `corevideo_loudness_meter_source`, draws that model with the existing `Solid` technique from `data/effects/corevideo-tiles.effect` plus private child text sources for the labels. + +**Tech Stack:** C++17, libobs (`obs_source_info`, `graphics/graphics.h`), CMake + CTest, no test framework, no new third-party dependency. + +**Spec:** `docs/superpowers/specs/2026-09-05-panelist-feedback-design.md` — this plan implements **Subsystem 1 only** (§"Subsystem 1 — Loudness engine + meter source", plus the "Audio" and "Testing conventions" entries in §"Integration facts"). The face detector (Subsystem 2), Tiles auto-framing (3a) and the return overlay (3b) are **out of scope** and belong to other plans. + +## Global Constraints + +- **Sample rate is a runtime variable.** BS.1770-4 publishes biquad coefficients only for 48 kHz. Zoom commonly delivers 32 kHz. Read `ShmAudioHeader::sample_rate` per buffer and **derive** coefficients for that rate. Hardcoding the published 48 kHz constants is the single most likely way to ship a meter that reads plausibly and is wrong. +- **Input format is 16-bit signed, interleaved.** Channel count is likewise runtime-discovered from `ShmAudioHeader::channels`. +- **All three measures are required:** momentary (400 ms, ungated), short-term (3 s, ungated), integrated (400 ms blocks at 100 ms hop — 75% overlap — with the absolute −70 LUFS gate **and** the relative −10 LU gate). The gate is load-bearing: a panelist is silent roughly 80% of a panel. +- **Integrated loudness is scoped to a resettable per-panelist check window** (a 20–60 s mic check), not the whole session. +- **The headline number is deviation in LU from the panel MEDIAN** of gated integrated loudness. Median, never mean. Absolute LUFS is secondary. Reference presets: EBU R128 −23 LUFS, ATSC A/85 −24 LKFS, streaming −16 LUFS; **default is panel median**. +- **Pure logic lives in a header-only file under `src/`** with a "why this exists" comment, pinned by a test in `tests/`. No libobs, Qt or Zoom SDK includes in those headers. +- **No test framework.** Plain `int main()` with a local `static void check(bool, const char *)` that increments a file-scope `failures` counter and prints `FAIL: `. Never gtest, never Catch. +- **Tests are hand-registered inside `if(BUILD_TESTING)` in the root `CMakeLists.txt`.** There is no `tests/CMakeLists.txt`. Target name `CoreVideoTest`, ctest name `CoreVideo`. +- **Metering runs on the audio lane thread only** — never the IPC reader thread (it head-of-line-blocks every source), never the OBS audio-mixer thread (budget-critical). Media events are coalescing prompts, not payloads: integrate over the whole drain loop. +- **`ZoomEngineClient::roster()` deep-copies strings under a hot mutex.** Cache display names on the roster callback (`add_roster_callback`); never call `roster()` from the audio path. +- Lock order in `src/zoom-participant-audio-source.cpp`: **`g_sources_mtx` before any `ctx->mtx`, never the reverse.** +- Build/verify: `cmake --build build --config Release --parallel 8`, then `cd build && ctest -C Release --output-on-failure`, N/N green. +- Comment style: state the constraint the code cannot show. When a decision comes from a measured number, put the number in the comment. + +--- + +## File Structure + +| File | Responsibility | +|---|---| +| `src/audio-loudness.h` **(new)** | Pure BS.1770-4 DSP: runtime coefficient derivation, K-weighting biquads, 100 ms hop accumulator, momentary/short-term, gated integration over a resettable check window. No libobs/Qt/SDK. | +| `tests/audio-loudness-test.cpp` **(new)** | Pins the DSP against the published 48 kHz coefficient table, against reference tones with concrete expected LUFS values, and pins the gate and the chunk-invariance law. | +| `src/loudness-board.h` **(new)** | Pure board model: panel median, deviation, reference presets, pass/fail status, deterministic row order, change signature, and the row/bar rectangle arithmetic. No libobs/Qt/SDK. | +| `tests/loudness-board-test.cpp` **(new)** | Pins median-not-mean, deviation sign, status precedence, signature stability under reorder, and the layout arithmetic. | +| `src/zoom-participant-audio-source.h` **(modify)** | Declares the loudness readout and the check-window reset entry points. | +| `src/zoom-participant-audio-source.cpp` **(modify)** | Holds a `LoudnessMeter` per source, feeds it inside the drain loop, caches the display name on the roster callback, resets the window on (re)subscribe, and exposes readings through the existing `g_sources_mtx` registry. | +| `src/zoom-loudness-meter-source.h/.cpp` **(new)** | The `corevideo_loudness_meter_source` OBS rendering source: builds the board model at 10 Hz, draws bars with the `Solid` technique, labels with private child text sources. | +| `CMakeLists.txt` **(modify)** | Adds the new plugin source file and two test registrations. | +| `src/plugin-main.cpp` **(modify)** | Registers the meter source and its graphics load/unload hooks. | +| `data/locale/en-US.ini` **(modify)** | Strings for the meter source's name and properties. | + +--- + +### Task 1: Runtime-derived BS.1770-4 K-weighting coefficients + +**Files:** +- Create: `src/audio-loudness.h` +- Create: `tests/audio-loudness-test.cpp` +- Modify: `CMakeLists.txt` (inside the `if(BUILD_TESTING)` block that opens at line ~597) + +**Interfaces:** +- Consumes: nothing. +- Produces: + - `struct LoudnessBiquadCoeffs { double b0, b1, b2, a1, a2; };` + - `LoudnessBiquadCoeffs bs1770_stage1_coeffs(uint32_t sample_rate);` + - `LoudnessBiquadCoeffs bs1770_stage2_coeffs(uint32_t sample_rate);` + - `struct LoudnessBiquadState { double x1, x2, y1, y2; };` + - `double loudness_biquad_step(const LoudnessBiquadCoeffs &, LoudnessBiquadState &, double x);` + +- [ ] **Step 1: Write the failing test** + +Create `tests/audio-loudness-test.cpp`: + +```cpp +// tests/audio-loudness-test.cpp +// ITU-R BS.1770-4 loudness, measured at whatever rate Zoom actually sends. +// +// WHY THIS TEST IS THE WHOLE FEATURE. BS.1770-4 publishes its K-weighting +// biquad coefficients for 48 kHz and for no other rate. This plugin does not +// receive a guaranteed rate: the engine reads GetSampleRate() per buffer and +// stamps it into ShmAudioHeader::sample_rate (engine/src/engine-audio.cpp), +// and Zoom commonly delivers 32 kHz. Coefficients pinned at 48 kHz and fed +// 32 kHz audio still produce a plausible-looking number -- measured below at +// 1.3 LU wrong on a 1 kHz tone -- which is precisely the failure a meter +// cannot survive, because nothing about the reading says it is wrong. +#include "audio-loudness.h" + +#include +#include +#include +#include + +static int failures = 0; + +static void check(bool ok, const char *message) +{ + if (!ok) { + std::cerr << "FAIL: " << message << "\n"; + ++failures; + } +} + +static bool near(double a, double b, double tol) +{ + return std::fabs(a - b) <= tol; +} + +int main() +{ + // ── The published BS.1770-4 table, at 48 kHz, to the digit ───────────── + // Table 1 (stage 1, the "head"/high-shelf pre-filter) and Table 2 (stage + // 2, the RLB high-pass) of BS.1770-4. If the derivation is right, it + // reproduces these exactly at 48 kHz -- that is the only rate at which + // there is anything published to check against, which is why it is + // checked to 1e-11 and not to a comfortable tolerance. + { + const LoudnessBiquadCoeffs s1 = bs1770_stage1_coeffs(48000); + check(near(s1.b0, 1.53512485958697, 1e-11), "48k stage-1 b0 does not match the published BS.1770-4 table"); + check(near(s1.b1, -2.69169618940638, 1e-11), "48k stage-1 b1 does not match the published BS.1770-4 table"); + check(near(s1.b2, 1.19839281085285, 1e-11), "48k stage-1 b2 does not match the published BS.1770-4 table"); + check(near(s1.a1, -1.69065929318241, 1e-11), "48k stage-1 a1 does not match the published BS.1770-4 table"); + check(near(s1.a2, 0.73248077421585, 1e-11), "48k stage-1 a2 does not match the published BS.1770-4 table"); + + const LoudnessBiquadCoeffs s2 = bs1770_stage2_coeffs(48000); + check(near(s2.b0, 1.0, 1e-12), "48k stage-2 b0 must be exactly 1"); + check(near(s2.b1, -2.0, 1e-12), "48k stage-2 b1 must be exactly -2"); + check(near(s2.b2, 1.0, 1e-12), "48k stage-2 b2 must be exactly 1"); + check(near(s2.a1, -1.99004745483398, 1e-11), "48k stage-2 a1 does not match the published BS.1770-4 table"); + check(near(s2.a2, 0.99007225036621, 1e-11), "48k stage-2 a2 does not match the published BS.1770-4 table"); + } + + // ── 32 kHz must produce DIFFERENT, correctly derived coefficients ────── + // These are the bilinear transform of the same analog prototype at + // 32 kHz. A "derivation" that quietly returned the 48 kHz numbers for + // every rate would pass every check above and fail every one here. + { + const LoudnessBiquadCoeffs s1 = bs1770_stage1_coeffs(32000); + check(near(s1.b0, 1.51117789957, 1e-9), "32k stage-1 b0 is wrong"); + check(near(s1.b1, -2.46488941336, 1e-9), "32k stage-1 b1 is wrong"); + check(near(s1.b2, 1.04163327352, 1e-9), "32k stage-1 b2 is wrong"); + check(near(s1.a1, -1.53904509625, 1e-9), "32k stage-1 a1 is wrong"); + check(near(s1.a2, 0.62696685598, 1e-9), "32k stage-1 a2 is wrong"); + + const LoudnessBiquadCoeffs s2 = bs1770_stage2_coeffs(32000); + check(near(s2.a1, -1.98508966899, 1e-9), "32k stage-2 a1 is wrong"); + check(near(s2.a2, 0.98514532067, 1e-9), "32k stage-2 a2 is wrong"); + } + + // ── The two rates must not be the same numbers ───────────────────────── + // Stated as its own assertion rather than left implicit in the two blocks + // above, because "the coefficients are rate-dependent" is the invariant, + // and an implementer reading only this file should see it said out loud. + { + const LoudnessBiquadCoeffs a = bs1770_stage1_coeffs(48000); + const LoudnessBiquadCoeffs b = bs1770_stage1_coeffs(32000); + check(std::fabs(a.a1 - b.a1) > 0.10, + "stage-1 a1 barely moved between 48 kHz and 32 kHz -- the " + "coefficients are not being derived from the rate at all"); + const LoudnessBiquadCoeffs c = bs1770_stage2_coeffs(48000); + const LoudnessBiquadCoeffs d = bs1770_stage2_coeffs(32000); + check(std::fabs(c.a1 - d.a1) > 0.004, + "stage-2 a1 barely moved between 48 kHz and 32 kHz -- the " + "high-pass corner is being placed at a fixed digital frequency " + "rather than a fixed 38 Hz"); + } + + // ── A degenerate rate must not produce NaN or a divide by zero ───────── + { + const LoudnessBiquadCoeffs s1 = bs1770_stage1_coeffs(0); + check(std::isfinite(s1.b0) && std::isfinite(s1.a1), + "a zero sample rate produced non-finite coefficients -- the " + "ring header can be read before the writer has initialised it"); + } + + // ── The biquad itself: a direct-form-II-transposed step ──────────────── + // Pinned against hand-computed values so a sign slip on the feedback + // terms cannot hide inside a filter response test. + { + const LoudnessBiquadCoeffs c{0.5, 0.25, 0.125, -0.5, 0.25}; + LoudnessBiquadState st{}; + // y[0] = 0.5*1 = 0.5 + const double y0 = loudness_biquad_step(c, st, 1.0); + check(near(y0, 0.5, 1e-12), "biquad sample 0 was not b0*x0"); + // y[1] = 0.5*0 + 0.25*1 + 0.125*0 - (-0.5)*0.5 - 0.25*0 = 0.5 + const double y1 = loudness_biquad_step(c, st, 0.0); + check(near(y1, 0.5, 1e-12), "biquad sample 1 is wrong -- check the " + "sign convention on a1 (y = b.x - a.y)"); + // y[2] = 0.125*1 - (-0.5)*0.5 - 0.25*0.5 = 0.125 + 0.25 - 0.125 = 0.25 + const double y2 = loudness_biquad_step(c, st, 0.0); + check(near(y2, 0.25, 1e-12), "biquad sample 2 is wrong -- the second " + "feedback tap (a2) is not being applied"); + } + + if (failures == 0) + std::cout << "audio-loudness: all tests passed\n"; + return failures == 0 ? 0 : 1; +} +``` + +- [ ] **Step 2: Register the test in CMake** + +In `CMakeLists.txt`, inside the `if(BUILD_TESTING)` block, immediately after the `CoreVideoAudioSilenceFadeTest` registration, add: + +```cmake + # ITU-R BS.1770-4 loudness. The coefficients are DERIVED from the runtime + # sample rate rather than taken from the standard's 48 kHz table, because + # the engine stamps whatever rate Zoom gave it into ShmAudioHeader and + # Zoom commonly sends 32 kHz -- measured 1.3 LU of error on a 1 kHz tone + # if the 48 kHz constants are used at 32 kHz, with nothing in the reading + # to say it is wrong. See src/audio-loudness.h. + add_executable(CoreVideoAudioLoudnessTest + tests/audio-loudness-test.cpp + ) + target_include_directories(CoreVideoAudioLoudnessTest PRIVATE + "${CMAKE_CURRENT_SOURCE_DIR}/src" + ) + add_test(NAME CoreVideoAudioLoudness + COMMAND CoreVideoAudioLoudnessTest) +``` + +- [ ] **Step 3: Run the test to verify it fails** + +```sh +cmake --build build --config Release --parallel 8 +``` + +Expected: FAIL at compile time — `Cannot open include file: 'audio-loudness.h'`. + +- [ ] **Step 4: Write the minimal implementation** + +Create `src/audio-loudness.h`: + +```cpp +#pragma once + +// ITU-R BS.1770-4 loudness measurement, derived for the sample rate the audio +// ACTUALLY arrives at. +// +// WHY THIS FILE DERIVES INSTEAD OF QUOTING. BS.1770-4 tabulates its two +// K-weighting biquads' coefficients for 48 kHz and for no other rate. This +// plugin has no guaranteed rate: engine/src/engine-audio.cpp calls +// data->GetSampleRate() per buffer and stamps the answer into +// ShmAudioHeader::sample_rate, and Zoom commonly delivers 32 kHz. Applying +// the published 48 kHz numbers to 32 kHz audio moves both filters' corner +// frequencies by a factor of 1.5 and mis-weights every measurement: on a +// 1 kHz tone whose true value is -19.98 LUFS it reads -18.66 LUFS. That is +// 1.3 LU of error on a meter whose entire product claim is that a 6 LU +// spread between panelists is visible -- and nothing about the number looks +// wrong. So the coefficients come from the analog prototype in the standard, +// bilinear-transformed at the runtime rate. At 48 kHz the derivation +// reproduces the published table to fourteen digits, which is what +// tests/audio-loudness-test.cpp asserts. +// +// Pure by design -- no libobs, no Qt, no Zoom SDK -- so the whole measurement +// can be pinned against reference tones with no meeting, the same treatment +// audio-timeline.h and audio-silence-fade.h get, and for the same reason: +// the only symptom of a regression here is a number that is quietly wrong. + +#include +#include +#include +#include +#include + +// One biquad section, y[n] = b0.x[n] + b1.x[n-1] + b2.x[n-2] +// - a1.y[n-1] - a2.y[n-2] +// (a0 normalised to 1). Sign convention matches the standard's tables, so a +// published a1 of -1.69065929318241 is stored verbatim. +struct LoudnessBiquadCoeffs { + double b0 = 1.0; + double b1 = 0.0; + double b2 = 0.0; + double a1 = 0.0; + double a2 = 0.0; +}; + +struct LoudnessBiquadState { + double x1 = 0.0; + double x2 = 0.0; + double y1 = 0.0; + double y2 = 0.0; +}; + +inline double loudness_biquad_step(const LoudnessBiquadCoeffs &c, + LoudnessBiquadState &s, double x) +{ + const double y = c.b0 * x + c.b1 * s.x1 + c.b2 * s.x2 + - c.a1 * s.y1 - c.a2 * s.y2; + s.x2 = s.x1; + s.x1 = x; + s.y2 = s.y1; + s.y1 = y; + return y; +} + +// The analog prototype BS.1770-4's 48 kHz table was itself produced from. +// These five constants are the whole of the standard's filter specification +// once the rate is factored out; every published coefficient falls out of +// them. Kept at full precision because the 48 kHz reproduction is asserted to +// 1e-11. +constexpr double kBs1770Stage1Hz = 1681.974450955533; +constexpr double kBs1770Stage1GdB = 3.999843853973347; +constexpr double kBs1770Stage1Q = 0.7071752369554196; +constexpr double kBs1770Stage1VbExp = 0.4996667741545416; +constexpr double kBs1770Stage2Hz = 38.13547087602444; +constexpr double kBs1770Stage2Q = 0.5003270373238773; + +// A rate to fall back on when the caller hands us nothing usable. The ring +// header can legitimately be read before the writer has initialised it (see +// output_audio_frame()'s slot_count guard), and a zero rate must produce +// finite coefficients rather than a NaN that then poisons every subsequent +// filter state for the life of the source. +constexpr uint32_t kLoudnessFallbackRate = 48000; + +inline uint32_t loudness_usable_rate(uint32_t sample_rate) +{ + return (sample_rate >= 8000 && sample_rate <= 384000) + ? sample_rate : kLoudnessFallbackRate; +} + +// Stage 1: the "head" high-shelf, roughly +4 dB above 1 kHz. +inline LoudnessBiquadCoeffs bs1770_stage1_coeffs(uint32_t sample_rate) +{ + const double fs = static_cast(loudness_usable_rate(sample_rate)); + const double K = std::tan(3.14159265358979323846 * kBs1770Stage1Hz / fs); + const double Vh = std::pow(10.0, kBs1770Stage1GdB / 20.0); + const double Vb = std::pow(Vh, kBs1770Stage1VbExp); + const double a0 = 1.0 + K / kBs1770Stage1Q + K * K; + + LoudnessBiquadCoeffs c; + c.b0 = (Vh + Vb * K / kBs1770Stage1Q + K * K) / a0; + c.b1 = 2.0 * (K * K - Vh) / a0; + c.b2 = (Vh - Vb * K / kBs1770Stage1Q + K * K) / a0; + c.a1 = 2.0 * (K * K - 1.0) / a0; + c.a2 = (1.0 - K / kBs1770Stage1Q + K * K) / a0; + return c; +} + +// Stage 2: the RLB high-pass, roughly 38 Hz. b0/b1/b2 are exactly 1/-2/1 at +// every rate -- that is a property of the prototype, not a rounding of the +// published table, so they are written as literals. +inline LoudnessBiquadCoeffs bs1770_stage2_coeffs(uint32_t sample_rate) +{ + const double fs = static_cast(loudness_usable_rate(sample_rate)); + const double K = std::tan(3.14159265358979323846 * kBs1770Stage2Hz / fs); + const double d = 1.0 + K / kBs1770Stage2Q + K * K; + + LoudnessBiquadCoeffs c; + c.b0 = 1.0; + c.b1 = -2.0; + c.b2 = 1.0; + c.a1 = 2.0 * (K * K - 1.0) / d; + c.a2 = (1.0 - K / kBs1770Stage2Q + K * K) / d; + return c; +} +``` + +- [ ] **Step 5: Run the test to verify it passes** + +```sh +cmake --build build --config Release --parallel 8 +cd build && ctest -C Release -R CoreVideoAudioLoudness --output-on-failure +``` + +Expected: PASS, `audio-loudness: all tests passed`. + +- [ ] **Step 6: Commit** + +```bash +git add src/audio-loudness.h tests/audio-loudness-test.cpp CMakeLists.txt +git commit -m "feat(loudness): derive BS.1770-4 K-weighting coefficients from the runtime sample rate" +``` + +--- + +### Task 2: K-weighted momentary and short-term loudness from int16 interleaved PCM + +**Files:** +- Modify: `src/audio-loudness.h` (append after `bs1770_stage2_coeffs`) +- Modify: `tests/audio-loudness-test.cpp` (append new blocks before the final `if (failures == 0)`) + +**Interfaces:** +- Consumes: `LoudnessBiquadCoeffs`, `LoudnessBiquadState`, `loudness_biquad_step`, `bs1770_stage1_coeffs`, `bs1770_stage2_coeffs` (Task 1). +- Produces: + - `constexpr double kLoudnessOffsetDb = -0.691;` + - `double loudness_lufs_from_mean_square(double z);` + - `double loudness_channel_weight(uint16_t channels, uint16_t channel);` + - `struct LoudnessMeter { ... };` (fields listed in the implementation below) + - `void loudness_meter_configure(LoudnessMeter &, uint32_t sample_rate, uint16_t channels);` + - `void loudness_meter_feed_int16(LoudnessMeter &, const int16_t *pcm, size_t frames, uint16_t channels, uint32_t sample_rate);` + - `bool loudness_meter_momentary(const LoudnessMeter &, double *out_lufs);` + - `bool loudness_meter_short_term(const LoudnessMeter &, double *out_lufs);` + +- [ ] **Step 1: Write the failing test** + +Append to `tests/audio-loudness-test.cpp`, immediately before the closing `if (failures == 0)`: + +```cpp + // ── Reference tones: the numbers an implementer can check by hand ────── + // + // The K-weighting curve has a gain of exactly +0.691 dB at 997 Hz, and + // BS.1770's -0.691 dB offset is there to cancel it. So for a ~1 kHz sine + // the whole measurement collapses to L = 10*log10(mean square of the + // un-weighted signal), which is a number that can be worked out on paper: + // + // peak 1.0 -> mean square 0.5 -> -3.01 LUFS + // peak 0.1 -> mean square 0.005 -> -23.01 LUFS + // RMS 0.1 -> mean square 0.01 -> -20.00 LUFS + // + // The third is the one to remember: a 1 kHz tone at -20 dBFS RMS reads + // -20.0 LUFS. If that does not hold, the offset, the channel weight, the + // int16 scaling or the K-weighting is wrong, and no amount of relative + // comparison downstream will save the reading. + auto feed_sine = [](LoudnessMeter &m, uint32_t rate, double peak, + double freq, double seconds) { + const size_t n = static_cast(rate * seconds); + std::vector pcm(n); + for (size_t i = 0; i < n; ++i) { + const double v = peak * std::sin(2.0 * 3.14159265358979323846 * + freq * static_cast(i) / + static_cast(rate)); + double s = v * 32767.0; + if (s > 32767.0) s = 32767.0; + if (s < -32767.0) s = -32767.0; + pcm[i] = static_cast(std::lround(s)); + } + loudness_meter_feed_int16(m, pcm.data(), n, 1, rate); + }; + + { + LoudnessMeter m; + feed_sine(m, 48000, 1.0, 1000.0, 5.0); + double lufs = 0.0; + check(loudness_meter_momentary(m, &lufs), + "momentary loudness was unavailable after 5 s of tone"); + check(near(lufs, -3.01, 0.10), + "a full-scale 1 kHz sine at 48 kHz did not read -3.01 LUFS"); + } + { + LoudnessMeter m; + feed_sine(m, 48000, 0.1, 1000.0, 5.0); + double lufs = 0.0; + check(loudness_meter_momentary(m, &lufs), "momentary unavailable"); + check(near(lufs, -23.01, 0.10), + "a 1 kHz sine of peak amplitude 0.1 at 48 kHz did not read " + "-23.01 LUFS"); + } + { + // -20 dBFS RMS: peak = sqrt(2) * 0.1. + LoudnessMeter m; + feed_sine(m, 48000, std::sqrt(2.0) * 0.1, 1000.0, 5.0); + double lufs = 0.0; + check(loudness_meter_momentary(m, &lufs), "momentary unavailable"); + check(near(lufs, -20.00, 0.10), + "a -20 dBFS RMS 1 kHz sine at 48 kHz did not read -20.0 LUFS -- " + "K-weighting is ~0 dB at 1 kHz once the -0.691 offset is " + "applied, so this is an equality, not an approximation"); + } + + // ── The same tone at 32 kHz must read the same, not 1.3 LU high ──────── + // This is the assertion the whole runtime-rate design exists for. With + // the 48 kHz coefficients applied to 32 kHz audio this tone reads + // -18.66 LUFS instead of -19.98: it passes a "looks like a plausible + // loudness" eyeball test and fails here. + { + LoudnessMeter m; + feed_sine(m, 32000, std::sqrt(2.0) * 0.1, 1000.0, 5.0); + double lufs = 0.0; + check(loudness_meter_momentary(m, &lufs), "momentary unavailable at 32 kHz"); + check(near(lufs, -19.98, 0.12), + "a -20 dBFS RMS 1 kHz sine at 32 kHz did not read -20 LUFS -- " + "the coefficients are not following the runtime rate"); + check(lufs < -19.5, + "the 32 kHz reading is more than 0.5 LU hot, which is the " + "signature of 48 kHz coefficients being used at 32 kHz"); + } + + // ── Short-term needs 3 s; momentary needs 400 ms ─────────────────────── + { + LoudnessMeter m; + feed_sine(m, 48000, std::sqrt(2.0) * 0.1, 1000.0, 0.35); + double lufs = 0.0; + check(!loudness_meter_momentary(m, &lufs), + "momentary reported a value before a full 400 ms block existed"); + feed_sine(m, 48000, std::sqrt(2.0) * 0.1, 1000.0, 0.20); + check(loudness_meter_momentary(m, &lufs), + "momentary was still unavailable after 550 ms"); + check(!loudness_meter_short_term(m, &lufs), + "short-term reported a value before 3 s of audio existed"); + feed_sine(m, 48000, std::sqrt(2.0) * 0.1, 1000.0, 3.0); + check(loudness_meter_short_term(m, &lufs), + "short-term was still unavailable after 3.5 s"); + check(near(lufs, -20.00, 0.15), "short-term did not read -20 LUFS"); + } + + // ── Stereo: two identical channels are +3 dB, not the same as mono ───── + // BS.1770 sums the weighted per-channel mean squares (G = 1.0 for L and + // R), it does not average them. Averaging is the mistake that makes a + // stereo panelist read 3 LU quieter than the identical mono one beside + // them, which is exactly the comparison this feature exists to make. + { + LoudnessMeter mono; + feed_sine(mono, 48000, std::sqrt(2.0) * 0.1, 1000.0, 2.0); + double mono_lufs = 0.0; + check(loudness_meter_momentary(mono, &mono_lufs), "mono unavailable"); + + LoudnessMeter st; + const size_t n = 48000 * 2; + std::vector pcm(n * 2); + for (size_t i = 0; i < n; ++i) { + const double v = std::sqrt(2.0) * 0.1 * + std::sin(2.0 * 3.14159265358979323846 * 1000.0 * + static_cast(i) / 48000.0); + const int16_t s = static_cast(std::lround(v * 32767.0)); + pcm[i * 2] = s; + pcm[i * 2 + 1] = s; + } + loudness_meter_feed_int16(st, pcm.data(), n, 2, 48000); + double st_lufs = 0.0; + check(loudness_meter_momentary(st, &st_lufs), "stereo unavailable"); + check(near(st_lufs - mono_lufs, 3.01, 0.05), + "dual-mono stereo was not +3.01 LU relative to mono -- the " + "channels are being averaged instead of summed"); + } + + // ── Digital silence never produces NaN or -inf leaking to a caller ───── + { + LoudnessMeter m; + std::vector zeros(48000, 0); + loudness_meter_feed_int16(m, zeros.data(), zeros.size(), 1, 48000); + double lufs = 0.0; + const bool have = loudness_meter_momentary(m, &lufs); + check(!have || std::isfinite(lufs), + "true digital silence produced a non-finite momentary reading -- " + "a panelist who has not spoken yet is the normal case here, not " + "an edge case"); + } +``` + +- [ ] **Step 2: Run the test to verify it fails** + +```sh +cmake --build build --config Release --parallel 8 +``` + +Expected: FAIL at compile time — `'LoudnessMeter': undeclared identifier`. + +- [ ] **Step 3: Write the implementation** + +Append to `src/audio-loudness.h`: + +```cpp +// The standard's absolute offset. It exists to cancel the K-weighting's +// +0.691 dB gain at 997 Hz, which is why a 1 kHz tone's LUFS value equals +// 10*log10 of its un-weighted mean square exactly. +constexpr double kLoudnessOffsetDb = -0.691; + +// Block/hop geometry. 400 ms blocks advancing every 100 ms is 75% overlap, +// which is what BS.1770-4 specifies for gated integration; momentary IS one +// such block, and short-term is 30 hops. +constexpr uint32_t kLoudnessHopMs = 100; +constexpr uint32_t kLoudnessMomentaryHops = 4; // 400 ms +constexpr uint32_t kLoudnessShortTermHops = 30; // 3 s + +// L = -0.691 + 10*log10(sum of G_i * z_i). Returns -HUGE_VAL for a +// non-positive mean square rather than letting log10 produce -inf/NaN at an +// arbitrary call site; every caller in this header checks for it. +inline double loudness_lufs_from_mean_square(double z) +{ + if (!(z > 0.0)) return -HUGE_VAL; + return kLoudnessOffsetDb + 10.0 * std::log10(z); +} + +// BS.1770-4 channel weights, in the standard's channel order +// (L, R, C, LFE, Ls, Rs). Zoom participant audio is mono or stereo, so in +// practice only the G = 1.0 terms are ever reached -- but a source configured +// for more channels must not silently weight a surround channel as if it were +// a front one, and the LFE must not be counted at all. +inline double loudness_channel_weight(uint16_t channels, uint16_t channel) +{ + if (channels <= 2) return 1.0; + switch (channel) { + case 0: case 1: case 2: return 1.00; // L, R, C + case 3: return 0.00; // LFE is excluded, not attenuated + case 4: case 5: return 1.41; // Ls, Rs + default: return 0.00; + } +} + +// A running BS.1770-4 measurement for ONE participant. +// +// OWNERSHIP: not thread-safe and deliberately so. In the plugin exactly one +// thread -- the audio lane that owns output_audio_frame() -- feeds it, under +// the same ctx->mtx that already guards the source's timeline, and readers +// take that mutex to copy the three numbers out. Adding a lock in here would +// put one on the media path for no gain. +struct LoudnessMeter { + uint32_t sample_rate = 0; + uint16_t channels = 0; + + LoudnessBiquadCoeffs c1{}; + LoudnessBiquadCoeffs c2{}; + std::vector s1; // stage 1 state, one per channel + std::vector s2; // stage 2 state, one per channel + + // Current partial 100 ms hop. + uint32_t hop_frames = 0; // frames per hop at the configured rate + uint32_t hop_filled = 0; + double hop_acc = 0.0; // sum over frames of sum_ch(G * y^2) + + // The last kLoudnessShortTermHops completed hops, newest at + // (hop_total - 1) % kLoudnessShortTermHops. + double hop_ring[kLoudnessShortTermHops] = {}; + uint64_t hop_total = 0; +}; + +// (Re)configures for a rate/channel count and clears all filter state. Called +// automatically by loudness_meter_feed_int16() whenever the wire format +// changes -- which it can, mid-source: Zoom renegotiates, and the operator's +// Mix/Isolated role flip changes the channel count on the same subscription. +// Carrying filter history across that would smear one format's transient into +// the other's measurement. +inline void loudness_meter_configure(LoudnessMeter &m, uint32_t sample_rate, + uint16_t channels) +{ + const uint32_t rate = loudness_usable_rate(sample_rate); + m.sample_rate = rate; + m.channels = channels == 0 ? 1 : channels; + m.c1 = bs1770_stage1_coeffs(rate); + m.c2 = bs1770_stage2_coeffs(rate); + m.s1.assign(m.channels, LoudnessBiquadState{}); + m.s2.assign(m.channels, LoudnessBiquadState{}); + m.hop_frames = (rate * kLoudnessHopMs) / 1000; + if (m.hop_frames == 0) m.hop_frames = 1; + m.hop_filled = 0; + m.hop_acc = 0.0; + for (uint32_t i = 0; i < kLoudnessShortTermHops; ++i) m.hop_ring[i] = 0.0; + m.hop_total = 0; +} + +// Hook the gated integrator into the hop boundary. Defined in Task 3; the +// forward declaration keeps feed_int16 below unchanged when it lands. +inline void loudness_meter_on_hop_complete(LoudnessMeter &m); + +// Feeds interleaved 16-bit signed PCM -- the format the engine writes into +// the SHM ring, unconverted. +// +// SCALING: /32768.0, not /32767.0. int16 is asymmetric and full negative +// scale is -32768; dividing by 32767 would let a legitimate sample exceed +// -1.0 and is the wrong direction for a measurement. +// +// PARTIAL BUFFERS ARE THE NORMAL CASE. Zoom delivers ~10 ms buffers and one +// media event can carry eight of them, so a 100 ms hop is assembled from many +// calls. The hop boundary is decided by frame count alone and never by call +// boundaries, which is what makes "feed the whole drain loop" identical to +// "feed one big buffer" -- pinned as chunk invariance in the test. +inline void loudness_meter_feed_int16(LoudnessMeter &m, const int16_t *pcm, + size_t frames, uint16_t channels, + uint32_t sample_rate) +{ + if (pcm == nullptr || frames == 0 || channels == 0) return; + if (m.sample_rate != loudness_usable_rate(sample_rate) || + m.channels != channels) { + loudness_meter_configure(m, sample_rate, channels); + } + + for (size_t f = 0; f < frames; ++f) { + double frame_sum = 0.0; + for (uint16_t ch = 0; ch < channels; ++ch) { + const double g = loudness_channel_weight(channels, ch); + const double x = static_cast(pcm[f * channels + ch]) / + 32768.0; + const double y1 = loudness_biquad_step(m.c1, m.s1[ch], x); + const double y2 = loudness_biquad_step(m.c2, m.s2[ch], y1); + // The filters run even for a zero-weight channel: their state is + // per channel and skipping them would make the LFE's history + // depend on how long it had been zero-weighted. + frame_sum += g * y2 * y2; + } + m.hop_acc += frame_sum; + if (++m.hop_filled >= m.hop_frames) { + const double hop_mean = m.hop_acc / + static_cast(m.hop_frames); + m.hop_ring[m.hop_total % kLoudnessShortTermHops] = hop_mean; + ++m.hop_total; + m.hop_acc = 0.0; + m.hop_filled = 0; + loudness_meter_on_hop_complete(m); + } + } +} + +// Mean of the newest `n` completed hops. False when fewer than `n` exist -- +// which is the honest answer for a panelist who has just been subscribed, and +// is why every getter here returns bool rather than a sentinel loudness. +inline bool loudness_hop_mean(const LoudnessMeter &m, uint32_t n, double *out) +{ + if (n == 0 || n > kLoudnessShortTermHops || m.hop_total < n) return false; + double sum = 0.0; + for (uint32_t i = 0; i < n; ++i) { + const uint64_t idx = m.hop_total - 1 - i; + sum += m.hop_ring[idx % kLoudnessShortTermHops]; + } + *out = sum / static_cast(n); + return true; +} + +// Momentary (M): one 400 ms block, ungated. +inline bool loudness_meter_momentary(const LoudnessMeter &m, double *out_lufs) +{ + double z = 0.0; + if (!loudness_hop_mean(m, kLoudnessMomentaryHops, &z)) return false; + const double l = loudness_lufs_from_mean_square(z); + if (!std::isfinite(l)) return false; + *out_lufs = l; + return true; +} + +// Short-term (S): 3 s, ungated. The number an operator reads while the +// panelist is talking. +inline bool loudness_meter_short_term(const LoudnessMeter &m, double *out_lufs) +{ + double z = 0.0; + if (!loudness_hop_mean(m, kLoudnessShortTermHops, &z)) return false; + const double l = loudness_lufs_from_mean_square(z); + if (!std::isfinite(l)) return false; + *out_lufs = l; + return true; +} +``` + +Also add a temporary definition so the forward declaration links; Task 3 replaces its body. Append at the very end of the file for now: + +```cpp +// Placeholder until Task 3 lands the gated integrator. Declared above so +// feed_int16 already calls it; defined empty here so this task builds alone. +inline void loudness_meter_on_hop_complete(LoudnessMeter &) {} +``` + +- [ ] **Step 4: Run the test to verify it passes** + +```sh +cmake --build build --config Release --parallel 8 +cd build && ctest -C Release -R CoreVideoAudioLoudness --output-on-failure +``` + +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add src/audio-loudness.h tests/audio-loudness-test.cpp +git commit -m "feat(loudness): K-weighted momentary and short-term loudness from int16 interleaved PCM" +``` + +--- + +### Task 3: Gated integrated loudness over a resettable check window + +**Files:** +- Modify: `src/audio-loudness.h` +- Modify: `tests/audio-loudness-test.cpp` + +**Interfaces:** +- Consumes: `LoudnessMeter`, `loudness_lufs_from_mean_square`, `loudness_hop_mean` (Task 2). +- Produces: + - `constexpr double kLoudnessAbsoluteGateLufs = -70.0;` + - `constexpr double kLoudnessRelativeGateLu = -10.0;` + - `constexpr size_t kLoudnessMaxGatedBlocks = 6000;` + - `void loudness_meter_reset_window(LoudnessMeter &);` + - `bool loudness_meter_integrated(const LoudnessMeter &, double *out_lufs);` + - `uint64_t loudness_meter_gated_blocks(const LoudnessMeter &);` + +- [ ] **Step 1: Write the failing test** + +Append to `tests/audio-loudness-test.cpp`, before the closing `if (failures == 0)`: + +```cpp + // ── The gate is the reason this measure is usable at all ─────────────── + // A panelist is silent roughly 80% of a panel. 4 s of speech at + // -20 LUFS followed by 16 s of silence averages to -27.08 LUFS if + // ungated -- an answer that describes the meeting, not the microphone. + // The BS.1770 absolute gate at -70 LUFS discards the silent blocks and + // the answer comes back to -20. + { + LoudnessMeter m; + feed_sine(m, 48000, std::sqrt(2.0) * 0.1, 1000.0, 4.0); + std::vector zeros(48000 * 16, 0); + loudness_meter_feed_int16(m, zeros.data(), zeros.size(), 1, 48000); + + double lufs = 0.0; + check(loudness_meter_integrated(m, &lufs), + "integrated loudness was unavailable after 4 s of speech"); + check(near(lufs, -20.16, 0.35), + "4 s of -20 LUFS speech in 20 s of silence did not integrate to " + "about -20 LUFS -- an ungated running average reads -27.08 here"); + check(lufs > -22.0, + "the integrated reading is dragged down by silence: the " + "absolute -70 LUFS gate is not being applied"); + check(loudness_meter_gated_blocks(m) > 30 && + loudness_meter_gated_blocks(m) < 60, + "the gated block count is not ~40 -- 4 s of speech at a 100 ms " + "hop is about 40 blocks that clear the absolute gate"); + } + + // ── The RELATIVE gate, which the absolute gate cannot stand in for ───── + // 10 s at -20 LUFS then 10 s at -40 LUFS: every block clears -70, so the + // absolute gate alone leaves -22.96. The relative gate (-10 LU below the + // absolute-gated mean) drops the quiet half and the answer is -20.06 -- + // the loudness of the speech, which is what a mic check is asking about. + { + LoudnessMeter m; + feed_sine(m, 48000, std::sqrt(2.0) * 0.1, 1000.0, 10.0); + feed_sine(m, 48000, std::sqrt(2.0) * 0.01, 1000.0, 10.0); + double lufs = 0.0; + check(loudness_meter_integrated(m, &lufs), "integrated unavailable"); + check(near(lufs, -20.06, 0.30), + "loud-then-quiet did not integrate to about -20 LUFS -- with " + "only the absolute gate this reads -22.96"); + } + + // ── The check window is resettable, and a reset is a clean slate ─────── + // A mic check is per panelist. Without this the number is polluted by + // whoever spoke before them on the same source. + { + LoudnessMeter m; + feed_sine(m, 48000, 1.0, 1000.0, 3.0); // very loud, -3 LUFS + double before = 0.0; + check(loudness_meter_integrated(m, &before), "integrated unavailable"); + check(near(before, -3.01, 0.30), "the loud pass did not read -3 LUFS"); + + loudness_meter_reset_window(m); + double after = 0.0; + check(!loudness_meter_integrated(m, &after), + "integrated loudness survived a window reset -- the previous " + "panelist's check is still in the number"); + check(loudness_meter_gated_blocks(m) == 0, + "the gated block count survived a window reset"); + + feed_sine(m, 48000, std::sqrt(2.0) * 0.1, 1000.0, 3.0); + check(loudness_meter_integrated(m, &after), "integrated unavailable " + "after refilling the window"); + check(near(after, -20.00, 0.30), + "the post-reset reading is contaminated by the pre-reset audio"); + } + + // ── A panelist who has never spoken has NO integrated reading ────────── + // Not -70, not 0. The board must be able to say "no audio" rather than + // print a number that looks like a measurement. + { + LoudnessMeter m; + std::vector zeros(48000 * 5, 0); + loudness_meter_feed_int16(m, zeros.data(), zeros.size(), 1, 48000); + double lufs = 0.0; + check(!loudness_meter_integrated(m, &lufs), + "five seconds of pure silence produced an integrated loudness"); + check(loudness_meter_gated_blocks(m) == 0, + "silent blocks were counted as gated blocks"); + } + + // ── Chunk invariance: the drain-loop law, stated as arithmetic ───────── + // A media event is a coalescing prompt, not a payload: one wakeup can + // carry eight ring slots. Measuring "the buffer that woke us" would throw + // away up to seven eighths of the audio. This asserts that feeding the + // same samples in 10 ms pieces and in one 2 s piece are the same + // measurement, which is what makes feeding from inside the drain loop + // correct. + { + const uint32_t rate = 32000; + const size_t n = rate * 2; + std::vector pcm(n); + for (size_t i = 0; i < n; ++i) { + const double v = 0.2 * std::sin(2.0 * 3.14159265358979323846 * + 440.0 * static_cast(i) / + static_cast(rate)); + pcm[i] = static_cast(std::lround(v * 32767.0)); + } + LoudnessMeter whole; + loudness_meter_feed_int16(whole, pcm.data(), n, 1, rate); + + LoudnessMeter pieces; + const size_t chunk = rate / 100; // 10 ms, Zoom's buffer size + for (size_t off = 0; off < n; off += chunk) { + const size_t take = (off + chunk <= n) ? chunk : (n - off); + loudness_meter_feed_int16(pieces, pcm.data() + off, take, 1, rate); + } + + double a = 0.0, b = 0.0; + check(loudness_meter_integrated(whole, &a) && + loudness_meter_integrated(pieces, &b), + "one of the two feeding patterns produced no integrated value"); + check(near(a, b, 1e-9), + "feeding in 10 ms chunks did not match feeding in one buffer -- " + "the hop boundary is following call boundaries instead of frame " + "counts, so the measurement depends on IPC batching"); + double ma = 0.0, mb = 0.0; + check(loudness_meter_momentary(whole, &ma) && + loudness_meter_momentary(pieces, &mb) && near(ma, mb, 1e-9), + "momentary differed between chunked and whole feeding"); + } +``` + +- [ ] **Step 2: Run the test to verify it fails** + +```sh +cmake --build build --config Release --parallel 8 +``` + +Expected: FAIL at compile time — `'loudness_meter_integrated': identifier not found`. + +- [ ] **Step 3: Write the implementation** + +In `src/audio-loudness.h`, **delete** the placeholder line added at the end in Task 2: + +```cpp +inline void loudness_meter_on_hop_complete(LoudnessMeter &) {} +``` + +Add these fields to `struct LoudnessMeter`, after `uint64_t hop_total = 0;`: + +```cpp + // The gated integration window -- ONE PANELIST'S MIC CHECK, not the + // session. Each entry is the mean square of a 400 ms block that cleared + // the absolute gate. Held as values rather than a running sum because the + // relative gate has to re-examine every block once the absolute-gated + // mean is known. + std::vector gated; + size_t gated_head = 0; // ring write position once `gated` is full + uint64_t gated_total = 0; // blocks ever admitted, never wrapped +``` + +Append after `loudness_meter_short_term`: + +```cpp +// BS.1770-4's two gates. The absolute one discards silence for free, which is +// exactly the mechanism a panel needs: a panelist is silent roughly 80% of a +// panel, and an ungated integrated reading over that measures the meeting +// rather than the microphone (measured: 4 s of -20 LUFS speech inside 20 s +// reads -27.08 ungated). The relative one then discards the quiet tail so the +// answer describes the speech. +constexpr double kLoudnessAbsoluteGateLufs = -70.0; +constexpr double kLoudnessRelativeGateLu = -10.0; + +// 6000 blocks is 10 minutes of continuously-gated audio at a 100 ms hop. A +// mic check is 20-60 s (~200-600 blocks), so this is never reached in the +// use this was built for; past it the window keeps the most RECENT 10 minutes +// rather than growing without bound. Documented rather than silent, because +// "the oldest audio quietly leaves the window" is a real semantic and an +// operator who leaves a board running all show is entitled to know it. +constexpr size_t kLoudnessMaxGatedBlocks = 6000; + +// Called at every completed 100 ms hop. A 400 ms block is the newest four +// hops, so admitting one block per hop is the standard's 75% overlap. +inline void loudness_meter_on_hop_complete(LoudnessMeter &m) +{ + double z = 0.0; + if (!loudness_hop_mean(m, kLoudnessMomentaryHops, &z)) return; + const double l = loudness_lufs_from_mean_square(z); + if (!std::isfinite(l) || l <= kLoudnessAbsoluteGateLufs) return; + + if (m.gated.size() < kLoudnessMaxGatedBlocks) { + m.gated.push_back(z); + } else { + m.gated[m.gated_head] = z; + m.gated_head = (m.gated_head + 1) % kLoudnessMaxGatedBlocks; + } + ++m.gated_total; +} + +// Starts this source's check window over. Clears the gated blocks and the hop +// history, but NOT the biquad state: the filters describe the signal that is +// still arriving, and zeroing them mid-stream would inject a transient into +// the first block of the new window. +inline void loudness_meter_reset_window(LoudnessMeter &m) +{ + m.gated.clear(); + m.gated_head = 0; + m.gated_total = 0; + m.hop_acc = 0.0; + m.hop_filled = 0; + for (uint32_t i = 0; i < kLoudnessShortTermHops; ++i) m.hop_ring[i] = 0.0; + m.hop_total = 0; +} + +// Blocks admitted to the current window. A board uses this to decide whether +// an integrated reading is worth showing: the spec's 20 s mic check yields +// ~200 blocks, so a handful of blocks is a cough, not a check. +inline uint64_t loudness_meter_gated_blocks(const LoudnessMeter &m) +{ + return m.gated_total; +} + +// Integrated (I): the two-pass gate, over the current check window. +// False means "this panelist has not produced a measurable check yet", which +// is a different statement from any loudness value and must stay +// distinguishable all the way to the board. +inline bool loudness_meter_integrated(const LoudnessMeter &m, double *out_lufs) +{ + if (m.gated.empty()) return false; + + double sum = 0.0; + for (double z : m.gated) sum += z; + const double abs_mean_lufs = + loudness_lufs_from_mean_square(sum / static_cast(m.gated.size())); + if (!std::isfinite(abs_mean_lufs)) return false; + + const double relative_threshold = abs_mean_lufs + kLoudnessRelativeGateLu; + double sum2 = 0.0; + size_t n2 = 0; + for (double z : m.gated) { + // Strictly greater, per BS.1770-4: a block exactly on the threshold + // is excluded. + if (loudness_lufs_from_mean_square(z) > relative_threshold) { + sum2 += z; + ++n2; + } + } + if (n2 == 0) return false; + + const double l = loudness_lufs_from_mean_square(sum2 / + static_cast(n2)); + if (!std::isfinite(l)) return false; + *out_lufs = l; + return true; +} +``` + +- [ ] **Step 4: Run the test to verify it passes** + +```sh +cmake --build build --config Release --parallel 8 +cd build && ctest -C Release -R CoreVideoAudioLoudness --output-on-failure +``` + +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add src/audio-loudness.h tests/audio-loudness-test.cpp +git commit -m "feat(loudness): gated integrated loudness over a resettable per-panelist check window" +``` + +--- + +### Task 4: The readiness-board model — panel median, deviation, status, layout + +**Files:** +- Create: `src/loudness-board.h` +- Create: `tests/loudness-board-test.cpp` +- Modify: `CMakeLists.txt` + +**Interfaces:** +- Consumes: nothing from earlier tasks (deliberately independent of `audio-loudness.h` so the board can be reasoned about with hand-written readings). +- Produces: + - `struct LoudnessReading { std::string source_uuid, display_name; uint32_t participant_id; bool subscribed; bool has_short_term; double short_term_lufs; bool has_integrated; double integrated_lufs; uint64_t gated_blocks; };` + - `enum class LoudnessReference { PanelMedian, EbuR128, AtscA85, Streaming };` + - `enum class LoudnessRowStatus { NoAudio, Measuring, Pass, Loud, Quiet };` + - `struct LoudnessBoardRow { std::string name, detail; bool has_deviation; double deviation_lu; bool has_short_term; double short_term_lufs; bool has_integrated; double integrated_lufs; LoudnessRowStatus status; };` + - `struct LoudnessBoardModel { bool has_reference; double reference_lufs; LoudnessReference reference_kind; std::vector rows; std::string signature; };` + - `constexpr uint64_t kLoudnessBoardMinBlocks = 30;` + - `constexpr double kLoudnessBoardDefaultToleranceLu = 2.0;` + - `bool loudness_panel_median(const std::vector &, uint64_t min_blocks, double *out);` + - `LoudnessBoardModel loudness_board_build(const std::vector &, LoudnessReference, double tolerance_lu, uint64_t min_blocks);` + - `struct LoudnessBoardRect { int x, y, w, h; };` + - `LoudnessBoardRect loudness_board_row_rect(int canvas_w, int canvas_h, size_t row_count, size_t row_index);` + - `LoudnessBoardRect loudness_board_bar_rect(const LoudnessBoardRect &row, double deviation_lu, double full_scale_lu);` + - `constexpr int kLoudnessBoardHeaderPx = 28;` / `kLoudnessBoardRowGapPx = 4;` / `kLoudnessBoardFullScaleLu = 6.0;` + +- [ ] **Step 1: Write the failing test** + +Create `tests/loudness-board-test.cpp`: + +```cpp +// tests/loudness-board-test.cpp +// The readiness board: what an operator actually reads during a mic check. +// +// The product claim is relative, not absolute. An operator does not primarily +// care that a panelist hits -23 LUFS; they care that panelist A is not 6 LU +// louder than panelist B. So the headline number is deviation from the panel +// MEDIAN of gated integrated loudness -- median, because one panelist on a +// laptop mic at -35 LUFS must not drag the reference everyone else is judged +// against, which is exactly what a mean does. +// +// The layout arithmetic is pinned here too rather than looked at on screen. +// This repo has no headless GPU harness and has ruled against building one +// (an offscreen Qt harness "certified it three times and was wrong three +// times"); the sanctioned approach is to extract the decision into a pure +// header and unit-test that, the way tests/tile-shape-test.cpp reproduces the +// shader's crop arithmetic. +#include "loudness-board.h" + +#include +#include +#include +#include + +static int failures = 0; + +static void check(bool ok, const char *message) +{ + if (!ok) { + std::cerr << "FAIL: " << message << "\n"; + ++failures; + } +} + +static bool near(double a, double b, double tol) +{ + return std::fabs(a - b) <= tol; +} + +static LoudnessReading measured(const char *name, double integrated, + double short_term = -20.0, + uint64_t blocks = 200) +{ + LoudnessReading r; + r.source_uuid = std::string("uuid_") + name; + r.display_name = name; + r.participant_id = 1; + r.subscribed = true; + r.has_short_term = true; + r.short_term_lufs = short_term; + r.has_integrated = true; + r.integrated_lufs = integrated; + r.gated_blocks = blocks; + return r; +} + +int main() +{ + // ── Median, not mean ─────────────────────────────────────────────────── + { + std::vector panel = { + measured("Ana", -18.0), measured("Ben", -21.0), + measured("Cara", -23.0), measured("Dev", -24.0), + measured("Erik", -30.0), + }; + double median = 0.0; + check(loudness_panel_median(panel, kLoudnessBoardMinBlocks, &median), + "no panel median was produced from five measured panelists"); + check(near(median, -23.0, 1e-9), + "the panel reference is not the median -- the mean of this " + "panel is -23.2, and Erik at -30 is exactly the outlier the " + "median exists to survive"); + } + + // ── An even panel averages the two middle values ─────────────────────── + { + std::vector panel = { + measured("Ana", -18.0), measured("Ben", -21.0), + measured("Cara", -23.0), measured("Dev", -24.0), + }; + double median = 0.0; + check(loudness_panel_median(panel, kLoudnessBoardMinBlocks, &median), + "no median from an even-sized panel"); + check(near(median, -22.0, 1e-9), + "an even-sized panel's median was not the mean of the two " + "middle values"); + } + + // ── Unmeasured panelists must not vote on the reference ──────────────── + { + LoudnessReading quiet; + quiet.source_uuid = "uuid_Fay"; + quiet.display_name = "Fay"; + quiet.subscribed = true; + // never spoke: no integrated value at all + std::vector panel = { + measured("Ana", -18.0), measured("Ben", -22.0), quiet, + }; + double median = 0.0; + check(loudness_panel_median(panel, kLoudnessBoardMinBlocks, &median), + "a panel with one silent member produced no median"); + check(near(median, -20.0, 1e-9), + "a panelist with no integrated reading was counted in the " + "median -- a person who has not spoken is not a data point"); + } + + // ── A too-short check does not count either ──────────────────────────── + { + std::vector panel = { + measured("Ana", -18.0, -18.0, 200), + measured("Ben", -22.0, -22.0, 200), + measured("Cough", -5.0, -5.0, 4), // four blocks: 400 ms + }; + double median = 0.0; + check(loudness_panel_median(panel, kLoudnessBoardMinBlocks, &median), + "no median produced"); + check(near(median, -20.0, 1e-9), + "a 400 ms cough set the panel reference -- the minimum gated " + "block count is not being applied"); + } + + // ── No measurable panelist means NO reference, not zero ──────────────── + { + std::vector panel; + double median = 0.0; + check(!loudness_panel_median(panel, kLoudnessBoardMinBlocks, &median), + "an empty panel produced a reference value"); + } + + // ── Deviation sign, and status ───────────────────────────────────────── + { + std::vector panel = { + measured("Ana", -18.0), measured("Ben", -21.0), + measured("Cara", -23.0), measured("Dev", -24.0), + measured("Erik", -30.0), + }; + const LoudnessBoardModel m = loudness_board_build( + panel, LoudnessReference::PanelMedian, + kLoudnessBoardDefaultToleranceLu, kLoudnessBoardMinBlocks); + check(m.has_reference && near(m.reference_lufs, -23.0, 1e-9), + "the built model's reference is not the panel median"); + check(m.rows.size() == 5, "the board did not produce one row per panelist"); + // Rows are ordered by name from CONTENT alone. + check(m.rows[0].name == "Ana" && m.rows[4].name == "Erik", + "rows are not in deterministic name order"); + check(near(m.rows[0].deviation_lu, 5.0, 1e-9), + "a panelist 5 LU above the median did not report +5 LU -- " + "louder than the reference must be POSITIVE"); + check(near(m.rows[4].deviation_lu, -7.0, 1e-9), + "a panelist 7 LU below the median did not report -7 LU"); + check(m.rows[0].status == LoudnessRowStatus::Loud, + "+5 LU was not flagged as too loud at a 2 LU tolerance"); + check(m.rows[4].status == LoudnessRowStatus::Quiet, + "-7 LU was not flagged as too quiet"); + check(m.rows[2].status == LoudnessRowStatus::Pass, + "the panelist sitting exactly on the median did not pass"); + check(m.rows[1].status == LoudnessRowStatus::Pass, + "-21 against a -23 median is +2 LU, exactly the tolerance, and " + "must pass -- the boundary is inclusive"); + } + + // ── Fixed-target presets ─────────────────────────────────────────────── + { + std::vector panel = { measured("Ana", -18.0) }; + const LoudnessBoardModel r128 = loudness_board_build( + panel, LoudnessReference::EbuR128, 2.0, kLoudnessBoardMinBlocks); + check(r128.has_reference && near(r128.reference_lufs, -23.0, 1e-9), + "EBU R128 preset is not -23 LUFS"); + check(near(r128.rows[0].deviation_lu, 5.0, 1e-9), + "-18 against the R128 target is not +5 LU"); + + const LoudnessBoardModel a85 = loudness_board_build( + panel, LoudnessReference::AtscA85, 2.0, kLoudnessBoardMinBlocks); + check(near(a85.reference_lufs, -24.0, 1e-9), + "ATSC A/85 preset is not -24 LKFS"); + + const LoudnessBoardModel str = loudness_board_build( + panel, LoudnessReference::Streaming, 2.0, kLoudnessBoardMinBlocks); + check(near(str.reference_lufs, -16.0, 1e-9), + "the streaming preset is not -16 LUFS"); + } + + // ── A fixed target works with NOBODY measured; the median does not ───── + { + LoudnessReading silent; + silent.source_uuid = "uuid_Ana"; + silent.display_name = "Ana"; + silent.subscribed = true; + std::vector panel = { silent }; + + const LoudnessBoardModel med = loudness_board_build( + panel, LoudnessReference::PanelMedian, 2.0, kLoudnessBoardMinBlocks); + check(!med.has_reference, + "a panel median was invented from a panel nobody has spoken on"); + check(med.rows.size() == 1 && !med.rows[0].has_deviation && + med.rows[0].status == LoudnessRowStatus::NoAudio, + "a silent panelist was given a deviation"); + + const LoudnessBoardModel fixed = loudness_board_build( + panel, LoudnessReference::EbuR128, 2.0, kLoudnessBoardMinBlocks); + check(fixed.has_reference, + "a FIXED target disappeared because nobody had spoken -- the " + "target does not depend on the panel"); + check(!fixed.rows[0].has_deviation, + "a silent panelist got a deviation against a fixed target"); + } + + // ── Measuring: subscribed and audible, but not enough blocks yet ─────── + { + std::vector panel = { + measured("Ana", -20.0, -20.0, 200), + measured("Ben", -20.0, -20.0, 5), + }; + const LoudnessBoardModel m = loudness_board_build( + panel, LoudnessReference::PanelMedian, 2.0, kLoudnessBoardMinBlocks); + check(m.rows[1].status == LoudnessRowStatus::Measuring, + "a panelist with 5 gated blocks was given a verdict rather than " + "reported as still measuring"); + check(!m.rows[1].has_deviation, + "a still-measuring panelist was given a deviation"); + } + + // ── The signature changes on content and NOT on input order ──────────── + // The Talkback dock shipped a live defect (2026-08-29) where a merely + // REORDERED roster rebuilt the whole widget list several times a second + // and threw away the operator's clicks. The board's consumer rebuilds + // child text sources off this signature, so the same rule applies here. + { + std::vector a = { + measured("Ana", -20.0), measured("Ben", -22.0), + }; + std::vector b = { a[1], a[0] }; // same set, reordered + const LoudnessBoardModel ma = loudness_board_build( + a, LoudnessReference::PanelMedian, 2.0, kLoudnessBoardMinBlocks); + const LoudnessBoardModel mb = loudness_board_build( + b, LoudnessReference::PanelMedian, 2.0, kLoudnessBoardMinBlocks); + check(ma.signature == mb.signature, + "reordering the input changed the board signature -- the " + "consumer will rebuild its text children on every roster event"); + + std::vector c = { + measured("Ana", -20.0), measured("Ben", -26.0), + }; + const LoudnessBoardModel mc = loudness_board_build( + c, LoudnessReference::PanelMedian, 2.0, kLoudnessBoardMinBlocks); + check(ma.signature != mc.signature, + "a 4 LU change in one panelist did not change the signature"); + } + + // ── Layout: rows tile the canvas below the header, in order ──────────── + { + const LoudnessBoardRect r0 = loudness_board_row_rect(640, 360, 4, 0); + const LoudnessBoardRect r3 = loudness_board_row_rect(640, 360, 4, 3); + check(r0.x == 0 && r0.w == 640, "a row does not span the canvas width"); + check(r0.y == kLoudnessBoardHeaderPx, + "the first row does not start below the header band"); + check(r0.h == 79, + "a 4-row board on a 640x360 canvas did not give 79 px rows " + "((360-28)/4 - 4 gap)"); + check(r3.y == kLoudnessBoardHeaderPx + 83 * 3, + "row 3 is not at the fourth slot"); + check(r3.y + r3.h <= 360, + "the last row overflows the canvas"); + const LoudnessBoardRect bad = loudness_board_row_rect(640, 360, 4, 9); + check(bad.w == 0 && bad.h == 0, + "an out-of-range row index produced a drawable rect"); + const LoudnessBoardRect none = loudness_board_row_rect(640, 360, 0, 0); + check(none.w == 0 && none.h == 0, + "a zero-row board produced a drawable rect"); + } + + // ── Layout: the bar grows from the centre of the right half ──────────── + { + const LoudnessBoardRect row{0, 28, 640, 79}; + const LoudnessBoardRect zero = + loudness_board_bar_rect(row, 0.0, kLoudnessBoardFullScaleLu); + check(zero.w == 0 && zero.x == 480, + "a zero deviation did not collapse to nothing at the centre " + "line (x=480 on a 640 px row)"); + + const LoudnessBoardRect hot = + loudness_board_bar_rect(row, 3.0, kLoudnessBoardFullScaleLu); + check(hot.x == 480 && hot.w == 80, + "+3 LU of a 6 LU full scale did not fill half the right side"); + + const LoudnessBoardRect cold = + loudness_board_bar_rect(row, -6.0, kLoudnessBoardFullScaleLu); + check(cold.x == 320 && cold.w == 160, + "-6 LU did not fill the left half of the meter"); + + const LoudnessBoardRect clipped = + loudness_board_bar_rect(row, 40.0, kLoudnessBoardFullScaleLu); + check(clipped.x == 480 && clipped.w == 160 && + clipped.x + clipped.w <= 640, + "an off-the-scale deviation drew past the canvas instead of " + "clamping at full scale"); + check(cold.y == row.y && cold.h == row.h, + "the bar's vertical extent does not match its row"); + } + + if (failures == 0) + std::cout << "loudness-board: all tests passed\n"; + return failures == 0 ? 0 : 1; +} +``` + +- [ ] **Step 2: Register the test in CMake** + +In `CMakeLists.txt`, immediately after the `CoreVideoAudioLoudnessTest` block added in Task 1: + +```cmake + # The readiness board's model: panel MEDIAN (not mean, so one laptop mic + # cannot drag the reference), deviation in LU, pass/fail, deterministic + # row order, and the row/bar rectangle arithmetic. The layout is pinned + # here rather than looked at on screen because this repo has no headless + # GPU harness and has ruled against building one. See src/loudness-board.h. + add_executable(CoreVideoLoudnessBoardTest + tests/loudness-board-test.cpp + ) + target_include_directories(CoreVideoLoudnessBoardTest PRIVATE + "${CMAKE_CURRENT_SOURCE_DIR}/src" + ) + add_test(NAME CoreVideoLoudnessBoard + COMMAND CoreVideoLoudnessBoardTest) +``` + +- [ ] **Step 3: Run the test to verify it fails** + +```sh +cmake --build build --config Release --parallel 8 +``` + +Expected: FAIL at compile time — `Cannot open include file: 'loudness-board.h'`. + +- [ ] **Step 4: Write the implementation** + +Create `src/loudness-board.h`: + +```cpp +#pragma once + +// The preshow readiness board: one row per panelist, showing how far their +// loudness sits from the panel's, and whether that is acceptable. +// +// WHY THE HEADLINE NUMBER IS RELATIVE. An operator running a mic check does +// not primarily care that a panelist hits -23 LUFS. They care that panelist A +// is not 6 LU louder than panelist B, because that is what the audience +// hears. So the reference defaults to the panel's own MEDIAN gated integrated +// loudness and the number on each row is a deviation in LU. Median, never +// mean: one person on a laptop mic at -35 LUFS would drag a mean far enough +// to fail everybody else, which is the opposite of useful. +// +// WHY THE LAYOUT MATHS IS IN HERE TOO. This repo has no headless GPU harness +// and has ruled against building one -- an offscreen Qt harness certified the +// Talkback dock's layout three times and was wrong three times. The sanctioned +// approach is to extract the decision into a pure header and unit-test that, +// the way tests/tile-shape-test.cpp reproduces the tile shader's crop +// arithmetic in plain C++. So the row and bar rectangles are decided here and +// the renderer only fills them. +// +// Pure: no libobs, no Qt, no Zoom SDK. + +#include +#include +#include +#include +#include +#include + +// One live audio source's measurement, flattened for the board. Every "has_" +// flag is load-bearing: "this panelist has not produced a measurable check" +// is a different statement from any loudness value, and collapsing it to a +// sentinel number is how a board ends up confidently reporting -70 LUFS for +// somebody who simply has not spoken yet. +struct LoudnessReading { + std::string source_uuid; + std::string display_name; + uint32_t participant_id = 0; + bool subscribed = false; + bool has_short_term = false; + double short_term_lufs = 0.0; + bool has_integrated = false; + double integrated_lufs = 0.0; + uint64_t gated_blocks = 0; +}; + +// Default: the panel's own median. The presets exist because a show sometimes +// has a delivery spec, but matching each other is the actual goal here, which +// is why PanelMedian is first and is the default. +enum class LoudnessReference { + PanelMedian = 0, + EbuR128 = 1, // -23 LUFS + AtscA85 = 2, // -24 LKFS + Streaming = 3, // -16 LUFS +}; + +enum class LoudnessRowStatus { + NoAudio = 0, // subscribed to nobody, or nobody has spoken + Measuring = 1, // audible, but not enough gated blocks for a verdict + Pass = 2, + Loud = 3, + Quiet = 4, +}; + +// Minimum gated blocks before a reading is treated as a check rather than a +// noise. At a 100 ms hop this is 3 s of gated speech; the spec's 20 s mic +// check yields roughly 200. Set low enough that an operator sees a verdict +// while the panelist is still talking, high enough that a cough or a chair +// scrape cannot set the panel reference for everybody. +constexpr uint64_t kLoudnessBoardMinBlocks = 30; + +// +/- this many LU from the reference still passes. 2 LU is below the ~3 LU +// step most listeners call "noticeably louder", so a passing board really is +// a matched panel. +constexpr double kLoudnessBoardDefaultToleranceLu = 2.0; + +// The deviation at which the bar is full. Beyond it the bar clamps rather +// than growing, because past 6 LU the exact number stops mattering: the +// answer is already "fix this microphone". +constexpr double kLoudnessBoardFullScaleLu = 6.0; + +constexpr int kLoudnessBoardHeaderPx = 28; +constexpr int kLoudnessBoardRowGapPx = 4; + +struct LoudnessBoardRow { + std::string name; + std::string detail; // short status text for the row + bool has_deviation = false; + double deviation_lu = 0.0; + bool has_short_term = false; + double short_term_lufs = 0.0; + bool has_integrated = false; + double integrated_lufs = 0.0; + LoudnessRowStatus status = LoudnessRowStatus::NoAudio; +}; + +struct LoudnessBoardModel { + bool has_reference = false; + double reference_lufs = 0.0; + LoudnessReference reference_kind = LoudnessReference::PanelMedian; + std::vector rows; + // Changes only when something an operator can SEE changed. The consumer + // rebuilds its child text sources off this, and the Talkback dock's + // 2026-08-29 live defect -- a merely reordered roster rebuilding the + // whole widget list several times a second and eating the operator's + // clicks -- is why it is derived from sorted content and never from + // input order. + std::string signature; +}; + +// The fixed presets. PanelMedian has no fixed value and returns false. +inline bool loudness_reference_fixed_target(LoudnessReference kind, double *out) +{ + switch (kind) { + case LoudnessReference::EbuR128: *out = -23.0; return true; + case LoudnessReference::AtscA85: *out = -24.0; return true; + case LoudnessReference::Streaming: *out = -16.0; return true; + case LoudnessReference::PanelMedian: + default: return false; + } +} + +// Median of the gated integrated loudness of everyone who has actually +// produced a check. Even counts average the two middle values, which is the +// ordinary definition and keeps a two-person panel from arbitrarily electing +// one of them as the reference. +inline bool loudness_panel_median(const std::vector &readings, + uint64_t min_blocks, double *out) +{ + std::vector values; + values.reserve(readings.size()); + for (const LoudnessReading &r : readings) { + if (!r.has_integrated) continue; + if (r.gated_blocks < min_blocks) continue; + if (!std::isfinite(r.integrated_lufs)) continue; + values.push_back(r.integrated_lufs); + } + if (values.empty()) return false; + std::sort(values.begin(), values.end()); + const size_t n = values.size(); + *out = (n % 2 == 1) ? values[n / 2] + : 0.5 * (values[n / 2 - 1] + values[n / 2]); + return true; +} + +inline const char *loudness_row_status_text(LoudnessRowStatus s) +{ + switch (s) { + case LoudnessRowStatus::NoAudio: return "no audio"; + case LoudnessRowStatus::Measuring: return "measuring"; + case LoudnessRowStatus::Pass: return "ok"; + case LoudnessRowStatus::Loud: return "too loud"; + case LoudnessRowStatus::Quiet: return "too quiet"; + default: return ""; + } +} + +inline LoudnessBoardModel loudness_board_build( + const std::vector &readings, + LoudnessReference kind, double tolerance_lu, uint64_t min_blocks) +{ + LoudnessBoardModel model; + model.reference_kind = kind; + if (!(tolerance_lu > 0.0)) tolerance_lu = kLoudnessBoardDefaultToleranceLu; + + double reference = 0.0; + if (loudness_reference_fixed_target(kind, &reference)) { + // A fixed target does not depend on the panel, so it survives a panel + // nobody has spoken on. The median does not, and must not be invented. + model.has_reference = true; + model.reference_lufs = reference; + } else if (loudness_panel_median(readings, min_blocks, &reference)) { + model.has_reference = true; + model.reference_lufs = reference; + } + + // Ordered by CONTENT alone -- name, then uuid to break a duplicate-name + // tie -- so a roster that merely reorders produces an identical board. + std::vector ordered; + ordered.reserve(readings.size()); + for (const LoudnessReading &r : readings) ordered.push_back(&r); + std::sort(ordered.begin(), ordered.end(), + [](const LoudnessReading *a, const LoudnessReading *b) { + if (a->display_name != b->display_name) + return a->display_name < b->display_name; + return a->source_uuid < b->source_uuid; + }); + + model.rows.reserve(ordered.size()); + for (const LoudnessReading *r : ordered) { + LoudnessBoardRow row; + row.name = r->display_name.empty() + ? (r->participant_id != 0 + ? "ID " + std::to_string(r->participant_id) + : std::string("- unassigned -")) + : r->display_name; + row.has_short_term = r->has_short_term; + row.short_term_lufs = r->short_term_lufs; + row.has_integrated = r->has_integrated; + row.integrated_lufs = r->integrated_lufs; + + if (!r->has_integrated || r->gated_blocks == 0) { + row.status = LoudnessRowStatus::NoAudio; + } else if (r->gated_blocks < min_blocks) { + row.status = LoudnessRowStatus::Measuring; + } else if (model.has_reference) { + row.has_deviation = true; + row.deviation_lu = r->integrated_lufs - model.reference_lufs; + if (row.deviation_lu > tolerance_lu) + row.status = LoudnessRowStatus::Loud; + else if (row.deviation_lu < -tolerance_lu) + row.status = LoudnessRowStatus::Quiet; + else + row.status = LoudnessRowStatus::Pass; // boundary is inclusive + } else { + row.status = LoudnessRowStatus::Measuring; + } + row.detail = loudness_row_status_text(row.status); + model.rows.push_back(std::move(row)); + } + + // Deviation is quantised to 0.1 LU in the signature: the renderer prints + // one decimal place, so a change smaller than that is invisible and must + // not cost a text-source rebuild. + std::string sig; + sig.reserve(model.rows.size() * 24 + 16); + char buf[64]; + std::snprintf(buf, sizeof(buf), "R%d:%s%.1f|", + static_cast(kind), model.has_reference ? "" : "x", + model.has_reference ? model.reference_lufs : 0.0); + sig += buf; + for (const LoudnessBoardRow &row : model.rows) { + sig += row.name; + std::snprintf(buf, sizeof(buf), "|%d|%s%.1f;", + static_cast(row.status), + row.has_deviation ? "" : "x", + row.has_deviation ? row.deviation_lu : 0.0); + sig += buf; + } + model.signature = std::move(sig); + return model; +} + +struct LoudnessBoardRect { + int x = 0; + int y = 0; + int w = 0; + int h = 0; +}; + +// One row's band. A zero-size result means "do not draw", which is what every +// degenerate input produces -- the renderer checks w/h rather than +// re-validating the arguments it just passed in. +inline LoudnessBoardRect loudness_board_row_rect(int canvas_w, int canvas_h, + size_t row_count, + size_t row_index) +{ + LoudnessBoardRect r; + if (canvas_w <= 0 || canvas_h <= 0 || row_count == 0 || + row_index >= row_count) + return r; + const int body_top = kLoudnessBoardHeaderPx; + const int body_h = canvas_h - body_top; + if (body_h <= 0) return r; + const int slot = body_h / static_cast(row_count); + if (slot <= 0) return r; + const int h = slot - kLoudnessBoardRowGapPx; + r.x = 0; + r.w = canvas_w; + r.y = body_top + slot * static_cast(row_index); + r.h = (h > 0) ? h : slot; + return r; +} + +// The deviation bar, growing right (louder) or left (quieter) from the centre +// of the row's right half. Clamped at full scale rather than allowed to run +// off the canvas: past 6 LU the exact number has stopped mattering. +inline LoudnessBoardRect loudness_board_bar_rect(const LoudnessBoardRect &row, + double deviation_lu, + double full_scale_lu) +{ + LoudnessBoardRect r; + if (row.w <= 0 || row.h <= 0 || !(full_scale_lu > 0.0)) return r; + const int meter_w = row.w / 2; // right half is the meter + const int meter_x = row.x + row.w - meter_w; + const int half = meter_w / 2; + const int centre = meter_x + half; + + double d = deviation_lu; + if (!std::isfinite(d)) d = 0.0; + if (d > full_scale_lu) d = full_scale_lu; + if (d < -full_scale_lu) d = -full_scale_lu; + + const int len = static_cast(std::fabs(d) / full_scale_lu * + static_cast(half) + 0.5); + r.y = row.y; + r.h = row.h; + r.w = len; + r.x = (d >= 0.0) ? centre : centre - len; + return r; +} +``` + +- [ ] **Step 5: Run the test to verify it passes** + +```sh +cmake --build build --config Release --parallel 8 +cd build && ctest -C Release -R CoreVideoLoudnessBoard --output-on-failure +``` + +Expected: PASS, `loudness-board: all tests passed`. + +- [ ] **Step 6: Commit** + +```bash +git add src/loudness-board.h tests/loudness-board-test.cpp CMakeLists.txt +git commit -m "feat(loudness): readiness-board model with panel-median reference and pinned layout arithmetic" +``` + +--- + +### Task 5: Feed the meter from the audio lane and expose readings + +**Files:** +- Modify: `src/zoom-participant-audio-source.h` +- Modify: `src/zoom-participant-audio-source.cpp` + +**Interfaces:** +- Consumes: `LoudnessMeter`, `loudness_meter_feed_int16`, `loudness_meter_reset_window`, `loudness_meter_momentary`, `loudness_meter_short_term`, `loudness_meter_integrated`, `loudness_meter_gated_blocks` (Tasks 2–3); `LoudnessReading` (Task 4). +- Produces: + - `std::vector corevideo_loudness_readings();` + - `void corevideo_reset_loudness_windows();` + +- [ ] **Step 1: Write the failing test** + +There is no host test that can reach an OBS source, so the invariant this task depends on is pinned in the pure header instead — the same treatment `talkback-key.h` got. Append to `tests/audio-loudness-test.cpp`, before the closing `if (failures == 0)`: + +```cpp + // ── The wire format the tap actually hands over ──────────────────────── + // output_audio_frame() reads ShmAudioHeader::sample_rate and ::channels + // per slot and can see them CHANGE mid-source: the engine restamps + // whatever GetSampleRate() returned, and an operator flipping a target + // between Mix (stereo) and Isolated (mono) changes the channel count on + // the same live subscription. The meter must follow that without + // carrying one format's filter history into the other's measurement. + { + LoudnessMeter m; + feed_sine(m, 48000, std::sqrt(2.0) * 0.1, 1000.0, 2.0); + check(m.sample_rate == 48000 && m.channels == 1, + "the meter did not adopt the first buffer's wire format"); + + // Same tone, now arriving at 32 kHz: the meter must re-derive rather + // than keep filtering with 48 kHz coefficients. + feed_sine(m, 32000, std::sqrt(2.0) * 0.1, 1000.0, 4.0); + check(m.sample_rate == 32000, + "a mid-stream rate change did not reconfigure the meter"); + double lufs = 0.0; + check(loudness_meter_momentary(m, &lufs) && near(lufs, -19.98, 0.15), + "after a mid-stream rate change the reading is wrong -- the " + "coefficients did not follow"); + check(m.hop_frames == 3200, + "the 100 ms hop is not 3200 frames at 32 kHz -- the hop length " + "is fixed in samples instead of in time"); + } + + // ── A null or empty buffer is a no-op, not a crash ──────────────────── + // The drain loop can hand over a slot it failed to copy. + { + LoudnessMeter m; + loudness_meter_feed_int16(m, nullptr, 480, 1, 48000); + std::vector one(1, 0); + loudness_meter_feed_int16(m, one.data(), 0, 1, 48000); + loudness_meter_feed_int16(m, one.data(), 1, 0, 48000); + check(m.hop_total == 0, + "a degenerate feed advanced the measurement"); + } +``` + +- [ ] **Step 2: Run the test to verify it fails** + +```sh +cmake --build build --config Release --parallel 8 +cd build && ctest -C Release -R CoreVideoAudioLoudness --output-on-failure +``` + +Expected: PASS for the new blocks if Tasks 2–3 were implemented correctly; if `hop_frames` is not exposed or the reconfigure path is missing, this FAILS with `the 100 ms hop is not 3200 frames at 32 kHz`. Run it and record which. If it passes, that is the correct outcome — these blocks exist to pin the behaviour the wiring below depends on before the wiring is written. + +- [ ] **Step 3: Extend the public header** + +In `src/zoom-participant-audio-source.h`, add the include and the two declarations at the end of the file: + +```cpp +#include "loudness-board.h" +``` + +```cpp +// One BS.1770-4 reading per live CoreVideoAudioSource, for the readiness +// board. Safe to call from any thread; takes g_sources_mtx and then each +// source's own mutex, in that order and never the reverse. +// +// The display name here is a CACHED copy, refreshed on the engine's roster +// callback. ZoomEngineClient::roster() deep-copies every ParticipantInfo -- +// strings included -- under the client's hot mutex, so resolving a name on +// the audio path (about a hundred buffers a second, per source) would put a +// full roster copy on the media path. +std::vector corevideo_loudness_readings(); + +// Starts every live source's mic-check window over. Integrated loudness is +// scoped to ONE panelist's check, not the session: without this the number +// is polluted by whoever spoke before them on the same source. +void corevideo_reset_loudness_windows(); +``` + +- [ ] **Step 4: Add the meter to the source context** + +In `src/zoom-participant-audio-source.cpp`, add the include beside the other project includes at the top: + +```cpp +#include "audio-loudness.h" +``` + +Add these members to `struct CoreVideoAudioSource`, immediately after `bool prev_was_silent = false;`: + +```cpp + // BS.1770-4 loudness for this participant. Same ownership as `timeline` + // and `prev_was_silent`: advanced by the audio lane thread inside + // output_audio_frame(), which holds ctx->mtx for the whole drain, and + // read under that same mutex by corevideo_loudness_readings(). + LoudnessMeter loudness; + // Set by any thread, consumed by the audio lane at the next slot. A reset + // has to land on a hop boundary the meter itself controls, so it is a + // request rather than a direct call: touching the meter from the caller's + // thread would race the drain that is filling it. + std::atomic loudness_reset_requested{false}; + // Display name, cached on the roster callback. Guarded by ctx->mtx. + std::string display_name; +``` + +- [ ] **Step 5: Feed the meter inside the drain loop** + +In `output_audio_frame()`, immediately after the line `ctx->prev_was_silent = cur_silent;` and before `const auto *pcm = pcm_mut;`, insert: + +```cpp + // ── BS.1770-4 metering ──────────────────────────────────────── + // HERE, inside the per-slot loop, and nowhere else. A media event + // is a coalescing PROMPT, not a payload: one wakeup routinely + // carries several ring slots and this loop drains until the ring + // is seen empty. Measuring "the buffer that woke us" would + // silently discard most of the audio and read low by a + // load-dependent amount -- the worst shape of wrong, because it + // looks fine on an idle box. + // + // Fed with the WIRE format (`pcm`, `channels`, `sample_rate`) + // rather than the publish format assembled below: the operator's + // Mono/Stereo choice is a routing decision for OBS, and a + // mono-summed copy of a stereo panelist would read 3 LU different + // from the same person carried as stereo. The measurement has to + // describe what the panelist SENT. + // + // The resume fade above has already been applied to these + // samples, which is correct: it is part of what we publish, it is + // 3 ms long, and excluding it would mean measuring audio that + // nobody hears. + if (ctx->loudness_reset_requested.exchange( + false, std::memory_order_acq_rel)) { + loudness_meter_reset_window(ctx->loudness); + } + loudness_meter_feed_int16(ctx->loudness, pcm_mut, pcm_frames, + channels, sample_rate); +``` + +- [ ] **Step 6: Cache the display name on the roster callback, and reset on resubscribe** + +In `maybe_resubscribe_for_roster()`, replace the `if (needs_roster) { ... }` block with this version (the only additions are the `cached_name` capture and the write-back after the block): + +```cpp + bool held_participant_present = true; + std::string cached_name; + const bool needs_roster = + (state.subscribed && state.participant_id != 0) || target != 0; + if (needs_roster) { + const auto roster = ZoomEngineClient::instance().roster(); + const auto present = [&](uint32_t id) { + return std::any_of(roster.begin(), roster.end(), + [&](const ParticipantInfo &p) { + return p.user_id == id; + }); + }; + if (state.subscribed && state.participant_id != 0) + held_participant_present = present(state.participant_id); + if (target != 0 && !present(target)) + target = 0; + + // The ONE place a display name is resolved for this source. This + // function runs on the engine's roster callback, which is exactly as + // often as a name can change, and it has already paid for the roster + // copy above. The readiness board reads the cached string instead of + // calling roster() itself, because roster() deep-copies every + // ParticipantInfo under the engine client's hot mutex and the board + // asks ten times a second. + const uint32_t name_for = target != 0 ? target : state.participant_id; + if (name_for != 0) { + for (const ParticipantInfo &p : roster) { + if (p.user_id == name_for) { + cached_name = p.display_name; + break; + } + } + } + } + { + std::lock_guard lk(ctx->mtx); + ctx->display_name = cached_name; + } +``` + +In `unsubscribe_audio()`, inside the existing `{ std::lock_guard lk(ctx->mtx); ... }` block that resets the timeline, add: + +```cpp + // A new subscription is a new panelist's mic check -- or the same + // one after a gap of unknown length. Either way the previous + // window's gated blocks describe audio that is not this check. + loudness_meter_reset_window(ctx->loudness); + ctx->display_name.clear(); +``` + +In `forget_subscription_for_new_engine()`, inside its `{ std::lock_guard lk(ctx->mtx); ... }` block, add the same two lines: + +```cpp + loudness_meter_reset_window(ctx->loudness); + ctx->display_name.clear(); +``` + +- [ ] **Step 7: Expose the readings through the existing registry** + +In `src/zoom-participant-audio-source.cpp`, immediately after `corevideo_audio_source_infos()`, add: + +```cpp +std::vector corevideo_loudness_readings() +{ + std::vector out; + // Same lock order as corevideo_audio_source_infos(): g_sources_mtx first, + // then each source's own mutex. The audio lane takes only ctx->mtx and + // never touches g_sources_mtx, so this can never invert. + std::lock_guard lk(g_sources_mtx); + out.reserve(g_sources.size()); + for (CoreVideoAudioSource *ctx : g_sources) { + if (!ctx) continue; + LoudnessReading r; + r.source_uuid = ctx->source_uuid; + r.participant_id = + ctx->current_participant_id.load(std::memory_order_acquire); + r.subscribed = ctx->subscribed.load(std::memory_order_acquire); + { + std::lock_guard ctx_lk(ctx->mtx); + r.display_name = ctx->display_name; + r.has_short_term = loudness_meter_short_term(ctx->loudness, + &r.short_term_lufs); + r.has_integrated = loudness_meter_integrated(ctx->loudness, + &r.integrated_lufs); + r.gated_blocks = loudness_meter_gated_blocks(ctx->loudness); + } + out.push_back(std::move(r)); + } + return out; +} + +void corevideo_reset_loudness_windows() +{ + std::lock_guard lk(g_sources_mtx); + for (CoreVideoAudioSource *ctx : g_sources) { + if (ctx) + ctx->loudness_reset_requested.store(true, + std::memory_order_release); + } + blog(LOG_INFO, + "[obs-zoom-plugin] CoreVideo loudness: mic-check windows reset on %d " + "source(s)", + static_cast(g_sources.size())); +} +``` + +- [ ] **Step 8: Build and run the whole suite** + +```sh +cmake --build build --config Release --parallel 8 +cd build && ctest -C Release --output-on-failure +``` + +Expected: the plugin compiles and every test is green (N/N). + +- [ ] **Step 9: Commit** + +```bash +git add src/audio-loudness.h tests/audio-loudness-test.cpp \ + src/zoom-participant-audio-source.h src/zoom-participant-audio-source.cpp +git commit -m "feat(loudness): meter every participant on the audio lane and expose per-source readings" +``` + +--- + +### Task 6: The readiness-board meter source + +**Files:** +- Create: `src/zoom-loudness-meter-source.h` +- Create: `src/zoom-loudness-meter-source.cpp` +- Modify: `CMakeLists.txt` (the `add_library(obs-zoom-plugin MODULE ...)` source list, line ~264) +- Modify: `src/plugin-main.cpp` +- Modify: `data/locale/en-US.ini` + +**Interfaces:** +- Consumes: `corevideo_loudness_readings()`, `corevideo_reset_loudness_windows()` (Task 5); `loudness_board_build`, `loudness_board_row_rect`, `loudness_board_bar_rect`, `LoudnessBoardModel`, `LoudnessRowStatus`, `LoudnessReference` (Task 4); `TilesEffect`, `tiles_effect_load`, `tiles_effect_destroy` (existing, `src/zoom-tiles-effect.h`). +- Produces: + - `void corevideo_loudness_meter_source_register();` + - `void corevideo_loudness_meter_load_gfx();` + - `void corevideo_loudness_meter_unload_gfx();` + +- [ ] **Step 1: Write the failing test** + +The rendering itself is not testable in this repo (no headless GPU harness, and one has been ruled against). Everything this source decides is already pinned in `tests/loudness-board-test.cpp` from Task 4. Add the one remaining decision — how many rows the source will draw and what it does when there are more panelists than the canvas can hold. Append to `tests/loudness-board-test.cpp`, before the closing `if (failures == 0)`: + +```cpp + // ── The board is bounded, and it says so ─────────────────────────────── + // A 25-person Zoom Events room would give rows a few pixels tall, which + // is not a readiness board, it is a texture. The renderer caps the rows + // it draws; the cap has to be a decision that can be reasoned about here + // rather than a magic number buried in a draw loop. + { + check(loudness_board_visible_rows(360, 3) == 3, + "three panelists on a 360 px canvas did not all fit"); + check(loudness_board_visible_rows(360, 40) == + (360 - kLoudnessBoardHeaderPx) / kLoudnessBoardMinRowPx, + "forty panelists were not capped to what the canvas can show " + "at the minimum readable row height"); + check(loudness_board_visible_rows(360, 0) == 0, + "an empty panel produced rows to draw"); + check(loudness_board_visible_rows(0, 10) == 0, + "a zero-height canvas produced rows to draw"); + const size_t capped = loudness_board_visible_rows(360, 40); + const LoudnessBoardRect last = + loudness_board_row_rect(640, 360, capped, capped - 1); + check(last.h >= kLoudnessBoardMinRowPx, + "the capped row count still produced rows below the minimum " + "readable height"); + } +``` + +- [ ] **Step 2: Run the test to verify it fails** + +```sh +cmake --build build --config Release --parallel 8 +``` + +Expected: FAIL at compile time — `'loudness_board_visible_rows': identifier not found`. + +- [ ] **Step 3: Add the cap to the pure header** + +Append to `src/loudness-board.h`, after `loudness_board_bar_rect`: + +```cpp +// The shortest row that is still a readiness board rather than a texture: a +// name and a number at a size an operator reads across a control room, plus +// the gap. A 25-person Zoom Events room would otherwise produce 13 px rows. +constexpr int kLoudnessBoardMinRowPx = 24; + +// How many rows this canvas can actually show. Beyond it the renderer draws +// the first N (which, because rows are name-ordered, is stable frame to frame +// rather than shuffling) and says so in the header band. +inline size_t loudness_board_visible_rows(int canvas_h, size_t row_count) +{ + if (canvas_h <= kLoudnessBoardHeaderPx || row_count == 0) return 0; + const int body_h = canvas_h - kLoudnessBoardHeaderPx; + const size_t capacity = + static_cast(body_h / kLoudnessBoardMinRowPx); + if (capacity == 0) return 0; + return row_count < capacity ? row_count : capacity; +} +``` + +- [ ] **Step 4: Run the test to verify it passes** + +```sh +cmake --build build --config Release --parallel 8 +cd build && ctest -C Release -R CoreVideoLoudnessBoard --output-on-failure +``` + +Expected: PASS. + +- [ ] **Step 5: Write the source header** + +Create `src/zoom-loudness-meter-source.h`: + +```cpp +#pragma once + +// The CoreVideo Loudness Meter: a preshow readiness board, drawn as an OBS +// source so it can sit on a multiview, a projector or a producer's monitor +// without a dock being open. +// +// One row per live CoreVideo audio source: the panelist's name, their +// deviation in LU from the panel reference, and a pass/fail verdict. Bars are +// drawn with the Solid technique already in data/effects/corevideo-tiles.effect +// -- there is no new effect file, because a solid quad is all a bar is and a +// second .effect is a second thing that can go missing beside a new DLL. +// Labels are private child text sources, so a Norwegian display name renders +// correctly instead of through a hand-rolled ASCII font. + +void corevideo_loudness_meter_source_register(); + +// Compiles/releases the shared effect. Called from plugin-main.cpp alongside +// the Tiles equivalents; libobs caches effects created from a file, so this +// costs nothing beyond the Tiles source's own load. +void corevideo_loudness_meter_load_gfx(); +void corevideo_loudness_meter_unload_gfx(); +``` + +- [ ] **Step 6: Write the source implementation** + +Create `src/zoom-loudness-meter-source.cpp`: + +```cpp +#include "zoom-loudness-meter-source.h" + +#include "loudness-board.h" +#include "zoom-participant-audio-source.h" +#include "zoom-tiles-effect.h" + +#include +#include + +#include +#include +#include +#include +#include +#include + +#define PROP_REFERENCE "reference" +#define PROP_TOLERANCE "tolerance_lu" +#define PROP_WIDTH "canvas_width" +#define PROP_HEIGHT "canvas_height" +#define PROP_RESET "btn_reset_windows" + +static const char *kMeterSourceId = "corevideo_loudness_meter_source"; + +// Hard ceiling on child text sources, independent of the canvas cap in +// loudness_board_visible_rows(): each row costs two private sources and OBS +// renders every one of them, so the count is bounded by construction rather +// than by whatever canvas height an operator types in. +static constexpr size_t kMeterMaxRows = 16; + +// 0xAARRGGBB, the same byte order picker_color_to_argb() produces for the +// Tiles wall, so gs_effect_set_color() reads them identically. Deliberately +// flat and high-contrast: this is read at a glance across a room, and the +// spec's legibility rule for anything meter-shaped is chunky segments and +// hard contrast, never hairlines. +static constexpr uint32_t kMeterBgArgb = 0xFF12161Cu; +static constexpr uint32_t kMeterHeaderArgb = 0xFF1E252Fu; +static constexpr uint32_t kMeterRowArgb = 0xFF1A2029u; +static constexpr uint32_t kMeterCentreArgb = 0xFF556070u; +static constexpr uint32_t kMeterPassArgb = 0xFF2FBF6Fu; +static constexpr uint32_t kMeterLoudArgb = 0xFFE04B4Bu; +static constexpr uint32_t kMeterQuietArgb = 0xFFE0A03Cu; +static constexpr uint32_t kMeterIdleArgb = 0xFF3A424Eu; + +// Shared with the Tiles wall by file, not by handle: libobs caches effects +// created from a file, so this second tiles_effect_load() resolves the same +// compiled effect rather than compiling it twice. +static TilesEffect s_meter_effect; +static bool s_meter_pass_failed_logged = false; + +struct meter_row_widgets { + obs_source_t *name = nullptr; + obs_source_t *value = nullptr; + std::string name_text; + std::string value_text; +}; + +struct loudness_meter_source { + obs_source_t *source = nullptr; + + std::atomic canvas_width{640}; + std::atomic canvas_height{360}; + std::atomic reference{0}; // LoudnessReference + std::atomic tolerance_milli_lu{2000}; + + std::mutex mutex; // guards `model` and `rows` + LoudnessBoardModel model; + std::string applied_signature; + meter_row_widgets rows[kMeterMaxRows]; + + float rebuild_accum = 0.0f; +}; + +// ── Text children ─────────────────────────────────────────────────────────── +// +// The board needs real text: panelist display names come from Zoom and this +// project has already been burned by names like "Ronny Hofsoy, Tromso" with +// their real diacritics (the Talkback dock's 400 px tower). A hand-rolled +// bitmap font would reintroduce exactly that class of defect, so the labels +// are OBS's own text sources, created private to this source. +// +// The id is PROBED rather than assumed: OBS ships text_ft2 and text_gdiplus +// on different platforms and has renamed both across versions. A build with +// neither must lose the labels and keep the bars, loudly -- never render an +// empty board with no explanation. +static const char *meter_text_source_id() +{ + static const char *cached = nullptr; + static bool probed = false; + if (probed) return cached; + probed = true; + static const char *candidates[] = { + "text_ft2_source_v2", "text_gdiplus_v3", "text_gdiplus_v2", + "text_ft2_source", "text_gdiplus", + }; + for (const char *id : candidates) { + // obs_get_source_output_flags() returns 0 for an id no module + // registered; a text source always carries OBS_SOURCE_VIDEO. + if (obs_get_source_output_flags(id) != 0) { + cached = id; + break; + } + } + if (!cached) { + blog(LOG_WARNING, + "[obs-zoom-plugin] CoreVideo Loudness Meter: no OBS text source " + "module is available; the board will draw bars without labels"); + } else { + blog(LOG_INFO, + "[obs-zoom-plugin] CoreVideo Loudness Meter: labels will use " + "text source '%s'", + cached); + } + return cached; +} + +static obs_source_t *make_text_child(const char *private_name, int px, + uint32_t argb) +{ + const char *id = meter_text_source_id(); + if (!id) return nullptr; + + obs_data_t *settings = obs_data_create(); + obs_data_t *font = obs_data_create(); + obs_data_set_string(font, "face", "Arial"); + obs_data_set_string(font, "style", "Bold"); + obs_data_set_int(font, "size", px); + obs_data_set_int(font, "flags", 0); + obs_data_set_obj(settings, "font", font); + obs_data_set_string(settings, "text", ""); + // text_gdiplus uses "color"; text_ft2 uses "color1"/"color2". Setting all + // three is harmless on either and avoids a per-id branch that would have + // to be revisited every time OBS renames one. + obs_data_set_int(settings, "color", static_cast(argb)); + obs_data_set_int(settings, "color1", static_cast(argb)); + obs_data_set_int(settings, "color2", static_cast(argb)); + obs_data_release(font); + + obs_source_t *src = obs_source_create_private(id, private_name, settings); + obs_data_release(settings); + return src; +} + +static void set_text_child(obs_source_t *src, const char *text) +{ + if (!src) return; + obs_data_t *settings = obs_data_create(); + obs_data_set_string(settings, "text", text); + obs_source_update(src, settings); + obs_data_release(settings); +} + +// ── Drawing ───────────────────────────────────────────────────────────────── + +static void meter_fill_rect(const LoudnessBoardRect &r, uint32_t argb) +{ + if (r.w <= 0 || r.h <= 0) return; + gs_technique_t *solid = s_meter_effect.tech_solid; + if (!solid || !s_meter_effect.param_color) return; + // The colour must be set BEFORE begin_pass: libobs uploads a pass's + // parameters inside gs_technique_begin_pass() and does not re-upload them + // for later draws in the same pass. Same rule the Tiles border uniforms + // live under. + gs_effect_set_color(s_meter_effect.param_color, argb); + gs_technique_begin(solid); + if (gs_technique_begin_pass(solid, 0)) { + gs_matrix_push(); + gs_matrix_translate3f(static_cast(r.x), + static_cast(r.y), 0.0f); + gs_draw_sprite(nullptr, 0, static_cast(r.w), + static_cast(r.h)); + gs_matrix_pop(); + gs_technique_end_pass(solid); + } else if (!s_meter_pass_failed_logged) { + // Once only. A board that silently stops drawing looks like the + // source went transparent, with no clue why. + s_meter_pass_failed_logged = true; + blog(LOG_ERROR, + "[obs-zoom-plugin] CoreVideo Loudness Meter: " + "gs_technique_begin_pass failed on the Solid technique; the " + "board will not draw"); + } + gs_technique_end(solid); +} + +static uint32_t status_color(LoudnessRowStatus s) +{ + switch (s) { + case LoudnessRowStatus::Pass: return kMeterPassArgb; + case LoudnessRowStatus::Loud: return kMeterLoudArgb; + case LoudnessRowStatus::Quiet: return kMeterQuietArgb; + default: return kMeterIdleArgb; + } +} + +static std::string row_value_text(const LoudnessBoardRow &row) +{ + char buf[96]; + if (row.has_deviation) { + if (row.has_integrated) { + std::snprintf(buf, sizeof(buf), "%+.1f LU %.1f LUFS %s", + row.deviation_lu, row.integrated_lufs, + row.detail.c_str()); + } else { + std::snprintf(buf, sizeof(buf), "%+.1f LU %s", + row.deviation_lu, row.detail.c_str()); + } + } else { + std::snprintf(buf, sizeof(buf), "%s", row.detail.c_str()); + } + return std::string(buf); +} + +static std::string header_text(const LoudnessBoardModel &m, size_t shown, + size_t total) +{ + char buf[160]; + const char *kind = "panel median"; + switch (m.reference_kind) { + case LoudnessReference::EbuR128: kind = "EBU R128"; break; + case LoudnessReference::AtscA85: kind = "ATSC A/85"; break; + case LoudnessReference::Streaming: kind = "streaming"; break; + case LoudnessReference::PanelMedian: + default: break; + } + if (!m.has_reference) { + std::snprintf(buf, sizeof(buf), + "MIC CHECK reference: %s (waiting for a first check)", + kind); + } else if (shown < total) { + std::snprintf(buf, sizeof(buf), + "MIC CHECK reference: %s %.1f LUFS showing %d of %d", + kind, m.reference_lufs, static_cast(shown), + static_cast(total)); + } else { + std::snprintf(buf, sizeof(buf), + "MIC CHECK reference: %s %.1f LUFS", + kind, m.reference_lufs); + } + return std::string(buf); +} + +// ── OBS callbacks ─────────────────────────────────────────────────────────── + +static const char *meter_get_name(void *) +{ + return obs_module_text("CoreVideoLoudnessMeter.Name"); +} + +static void meter_apply_settings(loudness_meter_source *ctx, + obs_data_t *settings) +{ + uint32_t w = static_cast(obs_data_get_int(settings, PROP_WIDTH)); + uint32_t h = static_cast(obs_data_get_int(settings, PROP_HEIGHT)); + if (w < 160) w = 160; + if (w > 3840) w = 3840; + if (h < 90) h = 90; + if (h > 2160) h = 2160; + ctx->canvas_width.store(w, std::memory_order_release); + ctx->canvas_height.store(h, std::memory_order_release); + ctx->reference.store(static_cast( + obs_data_get_int(settings, PROP_REFERENCE)), + std::memory_order_release); + double tol = obs_data_get_double(settings, PROP_TOLERANCE); + if (!(tol > 0.0)) tol = kLoudnessBoardDefaultToleranceLu; + if (tol > 12.0) tol = 12.0; + ctx->tolerance_milli_lu.store(static_cast(tol * 1000.0 + 0.5), + std::memory_order_release); +} + +static void *meter_create(obs_data_t *settings, obs_source_t *source) +{ + auto *ctx = new loudness_meter_source(); + ctx->source = source; + meter_apply_settings(ctx, settings); + + char private_name[64]; + for (size_t i = 0; i < kMeterMaxRows; ++i) { + std::snprintf(private_name, sizeof(private_name), + "corevideo_meter_name_%d", static_cast(i)); + ctx->rows[i].name = make_text_child(private_name, 20, 0xFFF2F5F8u); + std::snprintf(private_name, sizeof(private_name), + "corevideo_meter_value_%d", static_cast(i)); + ctx->rows[i].value = make_text_child(private_name, 20, 0xFFF2F5F8u); + } + return ctx; +} + +static void meter_destroy(void *data) +{ + auto *ctx = static_cast(data); + for (size_t i = 0; i < kMeterMaxRows; ++i) { + if (ctx->rows[i].name) obs_source_release(ctx->rows[i].name); + if (ctx->rows[i].value) obs_source_release(ctx->rows[i].value); + } + delete ctx; +} + +static void meter_update(void *data, obs_data_t *settings) +{ + meter_apply_settings(static_cast(data), settings); +} + +static uint32_t meter_get_width(void *data) +{ + auto *ctx = static_cast(data); + return ctx->canvas_width.load(std::memory_order_acquire); +} + +static uint32_t meter_get_height(void *data) +{ + auto *ctx = static_cast(data); + return ctx->canvas_height.load(std::memory_order_acquire); +} + +static void meter_enum_active_sources(void *data, + obs_source_enum_proc_t enum_callback, + void *param) +{ + auto *ctx = static_cast(data); + for (size_t i = 0; i < kMeterMaxRows; ++i) { + if (ctx->rows[i].name) enum_callback(ctx->source, ctx->rows[i].name, param); + if (ctx->rows[i].value) enum_callback(ctx->source, ctx->rows[i].value, param); + } +} + +// The model is rebuilt at 10 Hz, not per frame. corevideo_loudness_readings() +// takes g_sources_mtx and every source's own mutex -- the same mutex the +// audio lane holds for a whole drain -- so asking it 60 times a second would +// put the graphics thread in contention with the media path for no visible +// gain: the numbers it reports move on a 100 ms hop anyway. +static void meter_video_tick(void *data, float seconds) +{ + auto *ctx = static_cast(data); + ctx->rebuild_accum += seconds; + if (ctx->rebuild_accum < 0.1f) return; + ctx->rebuild_accum = 0.0f; + + const auto readings = corevideo_loudness_readings(); + const double tol = + static_cast(ctx->tolerance_milli_lu.load( + std::memory_order_acquire)) / 1000.0; + const auto kind = static_cast( + ctx->reference.load(std::memory_order_acquire)); + LoudnessBoardModel model = loudness_board_build( + readings, kind, tol, kLoudnessBoardMinBlocks); + + std::lock_guard lk(ctx->mutex); + ctx->model = std::move(model); +} + +static void meter_video_render(void *data, gs_effect_t *) +{ + auto *ctx = static_cast(data); + const int canvas_w = + static_cast(ctx->canvas_width.load(std::memory_order_acquire)); + const int canvas_h = + static_cast(ctx->canvas_height.load(std::memory_order_acquire)); + if (!s_meter_effect.valid()) return; + + LoudnessBoardModel model; + { + std::lock_guard lk(ctx->mutex); + model = ctx->model; + } + + meter_fill_rect(LoudnessBoardRect{0, 0, canvas_w, canvas_h}, kMeterBgArgb); + meter_fill_rect(LoudnessBoardRect{0, 0, canvas_w, kLoudnessBoardHeaderPx}, + kMeterHeaderArgb); + + // The LAST slot is permanently the header's, never a panelist's, so a + // change in row count cannot silently steal the header's text child -- + // hence the cap is kMeterMaxRows - 1 and not kMeterMaxRows. + static constexpr size_t kMeterHeaderSlot = kMeterMaxRows - 1; + const size_t total = model.rows.size(); + size_t shown = loudness_board_visible_rows(canvas_h, total); + if (shown > kMeterHeaderSlot) shown = kMeterHeaderSlot; + + for (size_t i = 0; i < shown; ++i) { + const LoudnessBoardRow &row = model.rows[i]; + const LoudnessBoardRect band = + loudness_board_row_rect(canvas_w, canvas_h, shown, i); + if (band.w <= 0 || band.h <= 0) continue; + + meter_fill_rect(band, kMeterRowArgb); + + // The status chip: a fat block at the left edge, which is the part + // that reads first from across a room. + meter_fill_rect(LoudnessBoardRect{band.x, band.y, 8, band.h}, + status_color(row.status)); + + // The zero line, drawn under the bar so a bar of zero width still + // shows where the reference is. + const LoudnessBoardRect zero = + loudness_board_bar_rect(band, 0.0, kLoudnessBoardFullScaleLu); + meter_fill_rect(LoudnessBoardRect{zero.x - 1, band.y, 2, band.h}, + kMeterCentreArgb); + + if (row.has_deviation) { + const LoudnessBoardRect bar = loudness_board_bar_rect( + band, row.deviation_lu, kLoudnessBoardFullScaleLu); + meter_fill_rect(LoudnessBoardRect{bar.x, bar.y + 4, bar.w, + bar.h > 8 ? bar.h - 8 : bar.h}, + status_color(row.status)); + } + } + + // Labels last, over the bars. Each child is only re-settings-updated when + // its string changes: obs_source_update() allocates and takes the source's + // own lock, and doing it per frame per row is the churn shape this project + // already has a live incident about. + const std::string head = header_text(model, shown, total); + { + std::lock_guard lk(ctx->mutex); + if (ctx->applied_signature != model.signature) { + ctx->applied_signature = model.signature; + for (size_t i = 0; i < kMeterHeaderSlot; ++i) { + const std::string name_text = + (i < shown) ? model.rows[i].name : std::string(); + const std::string value_text = + (i < shown) ? row_value_text(model.rows[i]) : std::string(); + if (ctx->rows[i].name_text != name_text) { + ctx->rows[i].name_text = name_text; + set_text_child(ctx->rows[i].name, name_text.c_str()); + } + if (ctx->rows[i].value_text != value_text) { + ctx->rows[i].value_text = value_text; + set_text_child(ctx->rows[i].value, value_text.c_str()); + } + } + } + } + + for (size_t i = 0; i < shown; ++i) { + const LoudnessBoardRect band = + loudness_board_row_rect(canvas_w, canvas_h, shown, i); + if (band.w <= 0 || band.h <= 0) continue; + const int text_y = band.y + (band.h > 24 ? (band.h - 24) / 2 : 0); + if (ctx->rows[i].name) { + gs_matrix_push(); + gs_matrix_translate3f(static_cast(band.x + 16), + static_cast(text_y), 0.0f); + obs_source_video_render(ctx->rows[i].name); + gs_matrix_pop(); + } + if (ctx->rows[i].value) { + gs_matrix_push(); + gs_matrix_translate3f(static_cast(band.x + band.w / 2 + 8), + static_cast(text_y), 0.0f); + obs_source_video_render(ctx->rows[i].value); + gs_matrix_pop(); + } + } + + // The header, in the slot reserved for it above. Updated on its own + // string comparison rather than on the board signature, because the + // "showing N of M" count changes with the CANVAS as well as the panel. + if (ctx->rows[kMeterHeaderSlot].name) { + obs_source_t *header = ctx->rows[kMeterHeaderSlot].name; + { + std::lock_guard lk(ctx->mutex); + if (ctx->rows[kMeterHeaderSlot].name_text != head) { + ctx->rows[kMeterHeaderSlot].name_text = head; + set_text_child(header, head.c_str()); + } + } + gs_matrix_push(); + gs_matrix_translate3f(12.0f, 4.0f, 0.0f); + obs_source_video_render(header); + gs_matrix_pop(); + } +} + +static bool meter_reset_clicked(obs_properties_t *, obs_property_t *, void *) +{ + corevideo_reset_loudness_windows(); + return false; +} + +static obs_properties_t *meter_get_properties(void *) +{ + obs_properties_t *props = obs_properties_create(); + + obs_property_t *ref = obs_properties_add_list( + props, PROP_REFERENCE, + obs_module_text("CoreVideoLoudnessMeter.Reference"), + OBS_COMBO_TYPE_LIST, OBS_COMBO_FORMAT_INT); + obs_property_list_add_int( + ref, obs_module_text("CoreVideoLoudnessMeter.Reference.PanelMedian"), + static_cast(LoudnessReference::PanelMedian)); + obs_property_list_add_int( + ref, obs_module_text("CoreVideoLoudnessMeter.Reference.R128"), + static_cast(LoudnessReference::EbuR128)); + obs_property_list_add_int( + ref, obs_module_text("CoreVideoLoudnessMeter.Reference.A85"), + static_cast(LoudnessReference::AtscA85)); + obs_property_list_add_int( + ref, obs_module_text("CoreVideoLoudnessMeter.Reference.Streaming"), + static_cast(LoudnessReference::Streaming)); + + obs_properties_add_float_slider( + props, PROP_TOLERANCE, + obs_module_text("CoreVideoLoudnessMeter.Tolerance"), 0.5, 6.0, 0.5); + obs_properties_add_int(props, PROP_WIDTH, + obs_module_text("CoreVideoLoudnessMeter.Width"), 160, 3840, 10); + obs_properties_add_int(props, PROP_HEIGHT, + obs_module_text("CoreVideoLoudnessMeter.Height"), 90, 2160, 10); + obs_properties_add_button(props, PROP_RESET, + obs_module_text("CoreVideoLoudnessMeter.Reset"), meter_reset_clicked); + return props; +} + +static void meter_get_defaults(obs_data_t *settings) +{ + obs_data_set_default_int(settings, PROP_REFERENCE, + static_cast(LoudnessReference::PanelMedian)); + obs_data_set_default_double(settings, PROP_TOLERANCE, + kLoudnessBoardDefaultToleranceLu); + obs_data_set_default_int(settings, PROP_WIDTH, 640); + obs_data_set_default_int(settings, PROP_HEIGHT, 360); +} + +void corevideo_loudness_meter_source_register() +{ + obs_source_info info = {}; + info.id = kMeterSourceId; + info.type = OBS_SOURCE_TYPE_INPUT; + // CUSTOM_DRAW because it binds the plugin's own effect rather than + // letting OBS draw one texture with the default one, exactly as the Tiles + // wall does. + info.output_flags = OBS_SOURCE_VIDEO | OBS_SOURCE_CUSTOM_DRAW | + OBS_SOURCE_DO_NOT_DUPLICATE; + info.get_name = meter_get_name; + info.create = meter_create; + info.destroy = meter_destroy; + info.update = meter_update; + info.video_tick = meter_video_tick; + info.video_render = meter_video_render; + info.get_width = meter_get_width; + info.get_height = meter_get_height; + info.enum_active_sources = meter_enum_active_sources; + info.get_properties = meter_get_properties; + info.get_defaults = meter_get_defaults; + obs_register_source(&info); +} + +void corevideo_loudness_meter_load_gfx() +{ + tiles_effect_load(s_meter_effect); +} + +void corevideo_loudness_meter_unload_gfx() +{ + tiles_effect_destroy(s_meter_effect); + s_meter_pass_failed_logged = false; +} +``` + +- [ ] **Step 7: Wire it into the build and the module** + +In `CMakeLists.txt`, in the `add_library(obs-zoom-plugin MODULE ...)` source list, add after `src/zoom-supersource.cpp`: + +```cmake + src/zoom-loudness-meter-source.cpp +``` + +In `src/plugin-main.cpp`, add the include beside the other source headers: + +```cpp +#include "zoom-loudness-meter-source.h" +``` + +After `zoom_supersource_load_gfx();` add: + +```cpp + corevideo_loudness_meter_source_register(); + corevideo_loudness_meter_load_gfx(); +``` + +After `zoom_supersource_unload_gfx();` in the unload path add: + +```cpp + corevideo_loudness_meter_unload_gfx(); +``` + +In `data/locale/en-US.ini`, after the `CoreVideoAudienceAudio.Name` line, add: + +```ini +CoreVideoLoudnessMeter.Name="CoreVideo Loudness Meter (readiness board)" +CoreVideoLoudnessMeter.Reference="Reference" +CoreVideoLoudnessMeter.Reference.PanelMedian="Panel median (recommended)" +CoreVideoLoudnessMeter.Reference.R128="EBU R128 (-23 LUFS)" +CoreVideoLoudnessMeter.Reference.A85="ATSC A/85 (-24 LKFS)" +CoreVideoLoudnessMeter.Reference.Streaming="Streaming (-16 LUFS)" +CoreVideoLoudnessMeter.Tolerance="Pass tolerance (LU)" +CoreVideoLoudnessMeter.Width="Width" +CoreVideoLoudnessMeter.Height="Height" +CoreVideoLoudnessMeter.Reset="Reset all mic-check windows" +``` + +- [ ] **Step 8: Build and run the whole suite** + +```sh +cmake --build build --config Release --parallel 8 +cd build && ctest -C Release --output-on-failure +``` + +Expected: the plugin links and every test is green (N/N). + +- [ ] **Step 9: Look at it in real OBS** + +Install the plugin DLL (OBS closed, elevated — and if the engine changed, both binaries as a pair), then in OBS add a **CoreVideo Loudness Meter** source to a scene alongside two or more **CoreVideo Participant Audio** sources in a live meeting. Confirm, in order: + +1. Rows appear with the panelists' names, in alphabetical order, and do not reshuffle as the roster ticks. +2. With nobody speaking, every row reads `no audio` and the header says `waiting for a first check`. +3. While one panelist speaks for ~20 s, their row moves `no audio` → `measuring` → a verdict, and the bar grows from the centre line. +4. The header's reference value appears once at least one panelist has a check. +5. `Reset all mic-check windows` clears every verdict back to `no audio`. +6. The OBS log carries one `labels will use text source '...'` line, not a per-frame stream of anything. + +- [ ] **Step 10: Commit** + +```bash +git add src/zoom-loudness-meter-source.h src/zoom-loudness-meter-source.cpp \ + src/loudness-board.h tests/loudness-board-test.cpp \ + CMakeLists.txt src/plugin-main.cpp data/locale/en-US.ini +git commit -m "feat(loudness): readiness-board meter source drawing deviation bars with the Solid technique" +``` + +--- + +### Task 7: Document the invariants in CLAUDE.md + +**Files:** +- Modify: `CLAUDE.md` + +This repo's standing directive is that docs-updated is part of done, and every invariant that has cost a live defect is listed in `CLAUDE.md`'s invariant map. Two of this feature's rules belong there. + +- [ ] **Step 1: Add the entry** + +In `CLAUDE.md`, in the "Invariants that have each caused a live-show defect" list, after the **Silence-resume fade** bullet, add: + +```markdown +- **Loudness coefficients follow the RUNTIME sample rate** + (`src/audio-loudness.h`, fed from `output_audio_frame()` in + `src/zoom-participant-audio-source.cpp`): BS.1770-4 publishes its two + K-weighting biquads' coefficients for 48 kHz and for no other rate, and + this plugin has no guaranteed rate — `engine/src/engine-audio.cpp` calls + `GetSampleRate()` per buffer and stamps the answer into + `ShmAudioHeader::sample_rate`, and Zoom commonly delivers 32 kHz. The + coefficients are therefore DERIVED from the analog prototype at the + runtime rate; at 48 kHz that derivation reproduces the published table to + fourteen digits, which is what `CoreVideoAudioLoudness` asserts. Pinned at + 48 kHz and fed 32 kHz, a 1 kHz tone whose true value is −19.98 LUFS reads + −18.66: 1.3 LU wrong, on a meter whose whole product claim is that a 6 LU + spread between panelists is visible, and with nothing in the number to say + it is wrong. **Metering runs inside the drain loop**, on the audio lane + thread, never per wakeup — a media event is a coalescing prompt, not a + payload, so measuring "the buffer that woke us" reads low by a + load-dependent amount that looks fine on an idle box. Chunk invariance + (10 ms pieces == one buffer, to 1e-9) is asserted for exactly that reason. + **Integrated loudness is scoped to a resettable per-panelist window**, not + the session: 4 s of −20 LUFS speech inside 20 s reads −27.08 ungated and + −20.16 with BS.1770's absolute −70 LUFS gate, which is precisely the + mechanism a panel needs, since a panelist is silent roughly 80% of a + panel. The board's reference is the panel **MEDIAN**, never the mean — + one laptop mic at −35 LUFS would otherwise drag the reference far enough + to fail everyone else. +``` + +- [ ] **Step 2: Commit** + +```bash +git add CLAUDE.md +git commit -m "docs: record the loudness engine's runtime-rate, drain-loop and gating invariants" +``` + +--- + +## Self-Review + +**Spec coverage (Subsystem 1 only):** + +| Spec requirement | Task | +|---|---| +| Filter chain: high-shelf head + RLB high-pass | 1, 2 | +| Sample rate is a runtime variable; derive coefficients | 1, 2 (test proves 32 k ≠ 48 k and the 1.3 LU error) | +| Input is 16-bit signed interleaved; channels runtime-discovered | 2, 5 | +| `L = -0.691 + 10 log10(Σ G_i z_i)`, G = 1.0 for L/R/C | 2 | +| Momentary 400 ms | 2 | +| Short-term 3 s | 2 | +| Integrated, absolute −70 then relative −10 LU, 400 ms blocks at 100 ms hop | 3 | +| Integrated scoped to a resettable check window | 3, 5 | +| Deviation from the panel **median** as the headline | 4 | +| Presets −23 / −24 / −16, default panel median | 4 | +| Meter is a readiness board, one row per panelist, pass/fail | 4, 6 | +| Tap at the decoded PCM in `zoom-participant-audio-source.cpp` | 5 | +| Audio lane thread, integrate over the whole drain loop | 5 (and pinned by chunk invariance in 3) | +| `roster()` names cached, never per frame; `add_roster_callback` | 5 | +| `g_sources_mtx` registry mirrored; lock order preserved | 5 | +| OBS rendering source, reuse the `Solid` technique | 6 | +| Header-only pure logic + hand-registered plain-`main()` tests | 1, 3, 4 | +| True peak deferred to v2 | not implemented — correct, it is explicitly deferred by the spec | + +Out of scope and deliberately absent: the face detector, Tiles auto-framing, the return overlay, and self-tile exclusion (a Subsystem 3 deployment constraint). + +**Placeholder scan:** no TBD, no "add error handling", no "similar to Task N". Every code step carries complete C++; every expected numeric value is stated concretely (−3.01, −23.01, −20.00, −19.98, −18.66, −27.08, −22.96, −20.16, −20.06, coefficient tables at 32 k and 48 k, 79 px rows, x=480/w=80 bars). + +**Type consistency:** `LoudnessMeter`, `LoudnessBiquadCoeffs`, `LoudnessBiquadState`, `LoudnessReading`, `LoudnessBoardRow`, `LoudnessBoardModel`, `LoudnessBoardRect`, `LoudnessReference`, `LoudnessRowStatus` are each defined once and referenced with the same names and members throughout. `loudness_meter_on_hop_complete()` is forward-declared in Task 2, given a placeholder body in Task 2 that Task 3 explicitly deletes before defining it properly — the only forward reference in the plan, and it is called out at both ends. diff --git a/docs/superpowers/plans/2026-09-05-subject-detector.md b/docs/superpowers/plans/2026-09-05-subject-detector.md new file mode 100644 index 00000000..11139262 --- /dev/null +++ b/docs/superpowers/plans/2026-09-05-subject-detector.md @@ -0,0 +1,4185 @@ +# Subject Detector Engine Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Produce a stable, tested, CPU-only engine that returns a smoothed `SubjectFrame` (face box + 5-landmark-derived eyeline) per participant, at a fixed cost that does not grow with the roster. + +**Architecture:** libfacedetection (YuNet weights compiled in as C arrays) is vendored as source and wrapped behind a one-method interface (`frame → box + landmarks`). The engine-IPC reader thread — which serves every source in the plugin — never runs detection: it only answers a single outstanding "send me one frame" request by copying I420 pixels into a one-slot inbox. A single dedicated worker thread drains that inbox, downscales-and-converts I420→BGR at ~320 px long edge, runs one detection, and feeds the result through a per-participant temporal smoother (deadband, rate limit, dropout hold). A round-robin schedule with an active-speaker boost issues exactly one detection per ~100 ms tick regardless of roster size. + +**Tech Stack:** C++17, MSVC, CMake, libfacedetection (vendored, BSD-3), no OpenCV, no OpenMP, no test framework. + +**Spec:** `docs/superpowers/specs/2026-09-05-panelist-feedback-design.md` (Subsystem 2 only) + +## Global Constraints + +- **Scope ends at the API.** This plan delivers `SubjectDetectorEngine::subject(participant_id) -> SubjectFrame`. It does NOT implement the loudness engine (Subsystem 1), Tiles auto-framing (3a), or the return overlay (3b). Those are separate plans by other authors and consume this API. +- **`SubjectFrame` is a published contract.** Two later plans consume it. Field names and types are fixed exactly as the spec writes them. Do not rename, reorder, or add required fields. +- **No test framework.** Plain `int main()` with a local `check()`/`near()`. Never gtest, never Catch. +- **Test naming:** target `CoreVideoTest`, ctest name `CoreVideo`, hand-registered inside `if(BUILD_TESTING)` in the root `CMakeLists.txt`. There is no `tests/CMakeLists.txt`. +- **No headless GPU harness exists and one has been ruled against.** Everything in this plan is pure CPU logic and must be testable off-rig. +- **New pure logic goes in a header-only file under `src/`** with a "why this exists" comment at the top, matching `src/audio-silence-fade.h` and `src/media-event-queue.h`. +- **Video format is I420 planar, BT.709, FULL range** (the engine already normalises limited→full every frame — `src/i420-range-expand.h`). Not NV12, not BGRA. Planes are tightly packed: Y of `w*h`, then U of `w*h/4`, then V of `w*h/4`. +- **libfacedetection requires BGR 3-channel.** A greyscale Y-plane shortcut is NOT valid for this CNN. The colour convert happens at the DOWNSCALED size (~320 px long edge), never at full resolution. +- **Detection must never run on the engine-IPC reader thread.** That thread dispatches frames for every source in the plugin; anything slow there head-of-line-blocks the whole plugin (2026-08-17 incident, see `src/media-event-queue.h`). +- **Do NOT enable `/openmp`** on the vendored library. We schedule one detection at a time on our own worker thread, which is what upstream recommends. +- **`/arch:AVX2` must provably reach the vendored library's compile line.** MSVC silently accepts AVX2 intrinsics under the default `/arch:SSE2` and emits a slow path; the only symptom is a detector that looks like a bad library. +- **Active speaker:** `ZoomEngineClient::active_speaker_id()` — the *directed* id, post hold/dwell. Not `raw_active_speaker_id()`. `SpeakerDirector` is **poll-only**: there is no observer list, so poll it; do not wait for events. +- **`ZoomEngineClient::roster()` deep-copies strings under a hot mutex.** Never call it per frame. +- **Dead code — do not build against:** `src/zoom-video-delegate.cpp`, `src/zoom-audio-delegate.*`, `src/zoom-audio-router.*`. None are in any CMake target. +- Build/verify: `cmake --build build --config Release --parallel 8` then `ctest -C Release --output-on-failure`, N/N green. + +## File Structure + +**Vendored (new — this repo has never vendored a library before; `third_party/` has only held the gitignored Zoom SDK):** + +| File | Responsibility | +|---|---| +| `third_party/libfacedetection/LICENSE` | Upstream BSD-3 text, copied verbatim | +| `third_party/libfacedetection/README-COREVIDEO.md` | Provenance: upstream URL, pinned commit, exactly which files we copied and which we wrote | +| `third_party/libfacedetection/src/facedetectcnn.h` | Upstream, unmodified | +| `third_party/libfacedetection/src/facedetectcnn.cpp` | Upstream, unmodified | +| `third_party/libfacedetection/src/facedetectcnn-model.cpp` | Upstream, unmodified | +| `third_party/libfacedetection/src/facedetectcnn-data.cpp` | Upstream, unmodified (~446 KB of weights as static C arrays) | +| `third_party/libfacedetection/src/facedetection_export.h` | **Ours** — one line, `#define FACEDETECTION_EXPORT` | +| `third_party/libfacedetection/corevideo-avx2-assert.cpp` | **Ours** — compile-time guard that AVX2 reached this target | +| `third_party/libfacedetection/CMakeLists.txt` | **Ours** — static lib target, AVX2 on, OpenMP off | + +**Ours (`src/`):** + +| File | Responsibility | +|---|---| +| `src/subject-frame.h` | The published `SubjectFrame` contract. Nothing else. Header-only, zero dependencies beyond ``. | +| `src/i420-bgr-downscale.h` | Header-only, pure: pick the downscaled size, and convert I420 (BT.709 full range) to packed BGR8 at that size in one pass. | +| `src/subject-detector-fd-record.h` | Header-only, pure: turn one raw libfacedetection result record (`short[142]`) into a normalized `SubjectFrame`. This is the arithmetic the adapter would otherwise hide behind the CNN. | +| `src/subject-detector.h` | The narrow interface `ISubjectDetector` (`frame → box + landmarks`) plus the factory declaration. This is the seam that keeps an OpenCV-DNN fallback a contained swap. | +| `src/subject-detector-fd.cpp` | The only file in the plugin that includes `facedetectcnn.h`. Implements `ISubjectDetector` over `facedetect_cnn()`. | +| `src/subject-schedule.h` | Header-only, pure: round-robin over the monitored roster with an active-speaker boost. One detection per tick, O(1) in roster size. | +| `src/subject-smoothing.h` | Header-only, pure: deadband, rate limit, dropout hold. | +| `src/subject-frame-inbox.h` | Header-only, pure: the single-slot request/deliver handshake between the engine-IPC reader thread and the detector worker. | +| `src/subject-detector-engine.h` / `.cpp` | The worker thread and the public API. Owns the schedule, the inbox, the detector, and one smoother per participant. | + +**Modified:** + +| File | Change | +|---|---| +| `CMakeLists.txt` | `add_subdirectory(third_party/libfacedetection)`; two new plugin sources; link the vendored lib; six new test registrations. | +| `src/zoom-supersource.cpp` | One `offer_frame` call in `tile_feed_on_frame`; roster/speaker pushes where the feed set changes. | +| `docs/THIRD-PARTY-NOTICES.md` | New file: the BSD-3 attribution notice we are obliged to carry. | +| `CLAUDE.md` | Record the vendoring exception and the detector's threading rule. | + +**Tests:** + +| Test file | Target / ctest name | +|---|---| +| `tests/facedetect-link-test.cpp` | `CoreVideoFaceDetectLinkTest` / `CoreVideoFaceDetectLink` | +| `tests/subject-detector-bench.cpp` | `CoreVideoSubjectDetectorBenchTest` / `CoreVideoSubjectDetectorBench` | +| `tests/i420-bgr-downscale-test.cpp` | `CoreVideoI420BgrDownscaleTest` / `CoreVideoI420BgrDownscale` | +| `tests/subject-detector-record-test.cpp` | `CoreVideoSubjectDetectorRecordTest` / `CoreVideoSubjectDetectorRecord` | +| `tests/subject-schedule-test.cpp` | `CoreVideoSubjectScheduleTest` / `CoreVideoSubjectSchedule` | +| `tests/subject-smoothing-test.cpp` | `CoreVideoSubjectSmoothingTest` / `CoreVideoSubjectSmoothing` | +| `tests/subject-inbox-test.cpp` | `CoreVideoSubjectInboxTest` / `CoreVideoSubjectInbox` | +| `tests/subject-worker-test.cpp` | `CoreVideoSubjectWorkerTest` / `CoreVideoSubjectWorker` | + +**Task order rationale:** Task 1 vendors and proves the library links. Task 2 measures its real cost on this machine *before* anything depends on a tick rate — the spec's ~10 ms figure is an extrapolation from a published 13.09 ms at 320×240 on a 2017 i7-7820X, and every later task's cadence assumption rests on the measurement. Tasks 3–7 are pure headers, each independently testable. Task 8 assembles them and wires the plugin. + +--- + +### Task 1: Vendor libfacedetection + +**Files:** +- Create: `third_party/libfacedetection/LICENSE` +- Create: `third_party/libfacedetection/README-COREVIDEO.md` +- Create: `third_party/libfacedetection/src/facedetectcnn.h` (upstream copy) +- Create: `third_party/libfacedetection/src/facedetectcnn.cpp` (upstream copy) +- Create: `third_party/libfacedetection/src/facedetectcnn-model.cpp` (upstream copy) +- Create: `third_party/libfacedetection/src/facedetectcnn-data.cpp` (upstream copy) +- Create: `third_party/libfacedetection/src/facedetection_export.h` (ours) +- Create: `third_party/libfacedetection/corevideo-avx2-assert.cpp` (ours) +- Create: `third_party/libfacedetection/CMakeLists.txt` (ours) +- Create: `docs/THIRD-PARTY-NOTICES.md` +- Create: `tests/facedetect-link-test.cpp` +- Modify: `CMakeLists.txt` (add_subdirectory before the plugin target; test registration in the `BUILD_TESTING` block) +- Modify: `CLAUDE.md` + +**Interfaces:** +- Consumes: nothing. +- Produces: CMake target `libfacedetection` (STATIC, `PUBLIC` include dir `third_party/libfacedetection/src`). The C++ entry point it exposes is upstream's: + `int *facedetect_cnn(unsigned char *result_buffer, unsigned char *bgr_image_data, int width, int height, int step);` + with `DETECT_BUFFER_SIZE` (0x20000) the required `result_buffer` size. Result layout: `results[0]` is the face count; face `i` is `reinterpret_cast(results + 1) + 142 * i`, whose fields are `[0]` confidence 0–100, `[1..4]` x, y, w, h in pixels, `[5..14]` five landmark x,y pairs. + +- [ ] **Step 1: Fetch upstream and record the exact commit** + +```bash +# Run from the repo root. The scratch clone is thrown away; only the four +# source files and the LICENSE are kept. +git ls-remote https://github.com/ShiqiYu/libfacedetection.git HEAD +# ^ Copy the SHA it prints. It goes in README-COREVIDEO.md in Step 3. + +git clone --depth 1 https://github.com/ShiqiYu/libfacedetection.git /tmp/lfd +mkdir -p third_party/libfacedetection/src +cp /tmp/lfd/src/facedetectcnn.h third_party/libfacedetection/src/ +cp /tmp/lfd/src/facedetectcnn.cpp third_party/libfacedetection/src/ +cp /tmp/lfd/src/facedetectcnn-model.cpp third_party/libfacedetection/src/ +cp /tmp/lfd/src/facedetectcnn-data.cpp third_party/libfacedetection/src/ +cp /tmp/lfd/LICENSE third_party/libfacedetection/LICENSE +``` + +Then confirm the licence really is 3-clause BSD and the data file really is the weights, not a downloader: + +```bash +head -5 third_party/libfacedetection/LICENSE +ls -l third_party/libfacedetection/src/ +# Expect: LICENSE begins "Copyright (c) 2018-2021, Shiqi Yu ... BSD 3-Clause", +# and facedetectcnn-data.cpp is roughly 400-500 KB. +``` + +- [ ] **Step 2: Discover the exact SIMD macro spelling upstream uses** + +Upstream gates its AVX2 path on a preprocessor macro, and the spelling has differed between releases (`_ENABLE_AVX2` vs `_ENABLE_AVX2_`). Do not guess — read it: + +```bash +grep -n "AVX2\|AVX512\|NEON" third_party/libfacedetection/src/facedetectcnn.h +``` + +Whatever it prints, the CMakeLists in Step 4 defines **both** spellings. Defining a macro the code never tests is harmless; failing to define the one it does test silently drops the whole SIMD path, which is exactly the failure mode this task exists to prevent. Note the spelling you saw in `README-COREVIDEO.md`. + +- [ ] **Step 3: Write the export header, the AVX2 guard, and the provenance note** + +`third_party/libfacedetection/src/facedetection_export.h`: + +```cpp +#define FACEDETECTION_EXPORT +``` + +`third_party/libfacedetection/corevideo-avx2-assert.cpp`: + +```cpp +// Fails the build if AVX2 did not reach the libfacedetection target's compile +// line. +// +// WHY THIS FILE EXISTS. MSVC will happily compile AVX2 intrinsics under the +// default /arch:SSE2: it accepts the intrinsics, emits a slower path, and says +// nothing. The only symptom is a detector that measures several times its +// published cost, which reads as "the library is bad" rather than "our build +// is wrong" -- and the spec calls that out as a day-costing trap. A compile +// error is the cheapest possible early warning, so this file is compiled into +// the same target as the library sources and therefore sees the same flags +// they do. +// +// Guarded on COREVIDEO_FD_EXPECT_AVX2, which CMake defines only on x86-64, +// so an ARM build (where there is no __AVX2__ and NEON is the SIMD path) is +// not broken by it. + +#if defined(COREVIDEO_FD_EXPECT_AVX2) && !defined(__AVX2__) +#error "libfacedetection is not being compiled with AVX2 enabled -- /arch:AVX2 (MSVC) or -mavx2 did not reach this target's compile line. Fix third_party/libfacedetection/CMakeLists.txt; do not delete this check." +#endif + +// Keeps the translation unit non-empty for linkers that object to one. +namespace corevideo_facedetect_build_guard { const int kAvx2Checked = 1; } +``` + +`third_party/libfacedetection/README-COREVIDEO.md`: + +```markdown +# libfacedetection — vendored into CoreVideo + +Upstream: https://github.com/ShiqiYu/libfacedetection +Pinned commit: +Licence: 3-clause BSD — see `LICENSE` in this directory, reproduced in +`docs/THIRD-PARTY-NOTICES.md`. + +## Why this is vendored rather than depended on + +This repository had never vendored a library before. The exception is +deliberate and was argued in +`docs/superpowers/specs/2026-09-05-panelist-feedback-design.md`: + +- It *is* YuNet — upstream converted the OpenCV Zoo YuNet ONNX model into + static C arrays, so we get the leading small face model without the runtime + that normally carries it. +- Zero dependencies and **no external model asset**. Nothing to install, + path-resolve, ship, or code-sign at runtime; no new DLL in the plugin folder. +- Upstream explicitly sanctions copying the sources into a host project. +- The alternative (`cv::FaceDetectorYN`) is the same model but drags in + opencv_core + dnn + imgproc. It stays the documented fallback, reachable + through `ISubjectDetector` in `src/subject-detector.h` without touching any + consumer. + +## Files copied verbatim from upstream `src/` + +- `src/facedetectcnn.h` +- `src/facedetectcnn.cpp` +- `src/facedetectcnn-model.cpp` +- `src/facedetectcnn-data.cpp` (~446 KB: the model weights as static C arrays) + +Do not edit these. If they ever need a fix, re-vendor from a newer upstream +commit and update the SHA above. + +## Files WE wrote (not upstream) + +- `src/facedetection_export.h` — upstream's build generates this; ours is the + one-line no-op form, because we build the sources directly into the plugin + rather than as a shared library. +- `corevideo-avx2-assert.cpp` — build-time proof that AVX2 reached the compile + line. Read its header comment before deleting it. +- `CMakeLists.txt` — our target definition. + +## SIMD macro spelling + +Upstream gates the AVX2 path on a preprocessor macro whose spelling has varied +across releases. As observed in the pinned commit's `facedetectcnn.h`, it is: +``. Our `CMakeLists.txt` defines both +`_ENABLE_AVX2` and `_ENABLE_AVX2_` so a re-vendor cannot silently drop it. + +## OpenMP is deliberately OFF + +Upstream offers an OpenMP build. We do not use it: the detector runs one +detection at a time on a single dedicated worker thread +(`src/subject-detector-engine.cpp`), and the scheduling that makes cost O(1) +in participant count is ours, not OpenMP's. Turning `/openmp` on would fight +our own thread for cores during a live show. +``` + +`docs/THIRD-PARTY-NOTICES.md`: + +```markdown +# Third-Party Notices + +CoreVideo is distributed under the MIT licence (see `LICENSE`). It also +includes the following third-party software, whose own licence terms apply to +those portions. + +## libfacedetection + +Vendored under `third_party/libfacedetection/`. Used for on-device face +detection (bounding box plus five landmarks) in the subject detector. No model +asset is downloaded or shipped separately; the weights are compiled in. + +Source: https://github.com/ShiqiYu/libfacedetection +Licence: 3-Clause BSD + +``` +By downloading, copying, installing or using the software you agree to this +license. If you do not agree to this license, do not download, install, copy +or use the software. + + License Agreement For libfacedetection + (3-clause BSD License) + +Copyright (c) 2018-2021, Shiqi Yu, all rights reserved. +shiqi.yu@gmail.com + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + + * Redistributions of source code must retain the above copyright notice, + this list of conditions and the following disclaimer. + + * Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + + * Neither the names of the copyright holders nor the names of the + contributors may be used to endorse or promote products derived from this + software without specific prior written permission. + +This software is provided by the copyright holders and contributors "as is" +and any express or implied warranties, including, but not limited to, the +implied warranties of merchantability and fitness for a particular purpose are +disclaimed. In no event shall copyright holders or contributors be liable for +any direct, indirect, incidental, special, exemplary, or consequential damages +(including, but not limited to, procurement of substitute goods or services; +loss of use, data, or profits; or business interruption) however caused and on +any theory of liability, whether in contract, strict liability, or tort +(including negligence or otherwise) arising in any way out of the use of this +software, even if advised of the possibility of such damage. +``` +``` + +**Verify the pasted notice matches the file you actually vendored** — replace +the fenced block above with the literal contents of +`third_party/libfacedetection/LICENSE` if upstream's text differs at the pinned +commit: + +```bash +diff <(sed -n '/^By downloading/,/possibility of such damage\.$/p' docs/THIRD-PARTY-NOTICES.md) third_party/libfacedetection/LICENSE +``` + +- [ ] **Step 4: Write the vendored target's CMakeLists** + +`third_party/libfacedetection/CMakeLists.txt`: + +```cmake +# Vendored libfacedetection (ShiqiYu), 3-clause BSD. +# Provenance, the list of files we wrote vs copied, and the re-vendoring +# procedure are in README-COREVIDEO.md. The attribution notice we are obliged +# to carry is in docs/THIRD-PARTY-NOTICES.md. +# +# STATIC, not MODULE/SHARED: the plugin links it in. There is deliberately no +# new DLL in the OBS plugin folder and no model asset on disk -- the weights +# are static C arrays inside facedetectcnn-data.cpp. + +add_library(libfacedetection STATIC + src/facedetectcnn.cpp + src/facedetectcnn-model.cpp + src/facedetectcnn-data.cpp + corevideo-avx2-assert.cpp +) + +target_include_directories(libfacedetection PUBLIC + "${CMAKE_CURRENT_SOURCE_DIR}/src") + +# The plugin is a MODULE library, so everything linked into it must be PIC. +set_target_properties(libfacedetection PROPERTIES POSITION_INDEPENDENT_CODE ON) + +if(MSVC) + # NOMINMAX/WIN32_LEAN_AND_MEAN for the same reason obs-zoom-plugin sets + # them: windows.h's min/max macros rewrite std::min/std::max into a syntax + # error. + target_compile_definitions(libfacedetection PRIVATE NOMINMAX WIN32_LEAN_AND_MEAN) + # /wd4244 /wd4267: upstream narrows freely between int/size_t/float. We do + # not patch vendored sources, so the warnings are silenced at the target + # rather than in the files. + target_compile_options(libfacedetection PRIVATE /wd4244 /wd4267) +endif() + +# AVX2. Both macro spellings are defined on purpose -- upstream has used +# _ENABLE_AVX2 and _ENABLE_AVX2_ in different releases, and defining the unused +# one costs nothing while missing the used one silently drops the entire SIMD +# path. See README-COREVIDEO.md. +# +# COREVIDEO_FD_EXPECT_AVX2 arms the compile-time guard in +# corevideo-avx2-assert.cpp, which is what actually proves the flag landed. +if(CMAKE_SIZEOF_VOID_P EQUAL 8 AND + CMAKE_SYSTEM_PROCESSOR MATCHES "x86_64|AMD64|amd64|x86|X86") + target_compile_definitions(libfacedetection PRIVATE + _ENABLE_AVX2 _ENABLE_AVX2_ COREVIDEO_FD_EXPECT_AVX2) + if(MSVC) + target_compile_options(libfacedetection PRIVATE /arch:AVX2) + else() + target_compile_options(libfacedetection PRIVATE -mavx2 -mfma) + endif() +endif() + +# DELIBERATELY NOT /openmp. We run exactly one detection at a time on our own +# dedicated worker thread and get O(1)-in-roster-size cost from the schedule in +# src/subject-schedule.h. An OpenMP pool inside the detector would compete with +# OBS's render and encode threads during a live show for no benefit. Upstream +# recommends host-level threading for exactly this reason. +``` + +- [ ] **Step 5: Write the failing link/smoke test** + +`tests/facedetect-link-test.cpp`: + +```cpp +// tests/facedetect-link-test.cpp +// Proves the vendored libfacedetection actually links and runs in this build. +// +// It is deliberately not an accuracy test: with no face image in the repo +// there is nothing to assert about detection quality here. What it does +// assert is everything that can silently go wrong at vendoring time -- the +// weights translation unit is present, the export header satisfies the +// declaration, the result buffer contract is honoured, and a flat image +// yields a well-formed (zero-face) result rather than a crash or garbage +// count. + +#include "facedetectcnn.h" + +#include +#include +#include + +static int g_failures = 0; + +static void check(bool cond, const char *what) +{ + if (!cond) { + std::cerr << "FAIL: " << what << "\n"; + ++g_failures; + } +} + +int main() +{ + const int w = 320; + const int h = 180; + + // Flat mid-grey BGR. No face, no edges, nothing for the CNN to latch on. + std::vector bgr(static_cast(w) * h * 3, 128); + std::vector buffer(DETECT_BUFFER_SIZE, 0); + + int *results = facedetect_cnn(buffer.data(), bgr.data(), w, h, w * 3); + check(results != nullptr, "facedetect_cnn returned a null result pointer"); + if (!results) return 1; + + const int count = *results; + std::cerr << "flat grey 320x180 -> " << count << " face(s)\n"; + check(count >= 0, "face count must not be negative"); + check(count == 0, "flat grey must not produce a face"); + + // Run it a second time on the same buffer: the detector is called once per + // tick for the whole life of the process, so a one-shot-only library would + // be a blocking discovery. This catches state left behind in the result + // buffer. + results = facedetect_cnn(buffer.data(), bgr.data(), w, h, w * 3); + check(results != nullptr, "second call returned null"); + check(results && *results == 0, "second call on flat grey must also be 0"); + + // A vertical gradient: still no face, but it exercises the convolution + // path on non-constant data rather than a degenerate all-equal image. + for (int y = 0; y < h; ++y) { + for (int x = 0; x < w; ++x) { + const size_t i = (static_cast(y) * w + x) * 3; + const unsigned char v = static_cast((y * 255) / h); + bgr[i + 0] = v; + bgr[i + 1] = v; + bgr[i + 2] = v; + } + } + results = facedetect_cnn(buffer.data(), bgr.data(), w, h, w * 3); + check(results != nullptr, "gradient call returned null"); + check(results && *results >= 0, "gradient face count must not be negative"); + + if (g_failures) { + std::cerr << g_failures << " check(s) failed\n"; + return 1; + } + std::cerr << "libfacedetection links and runs\n"; + return 0; +} +``` + +- [ ] **Step 6: Run the test to verify it fails** + +Run: `cmake --build build --config Release --target CoreVideoFaceDetectLinkTest` +Expected: FAIL — CMake reports `No rule to make target` / `does not exist`, because the target has not been registered yet. + +- [ ] **Step 7: Register the vendored library and the test in the root CMakeLists** + +In `CMakeLists.txt`, immediately **before** `if(COREVIDEO_BUILD_PLUGIN)` at line 263: + +```cmake +# Vendored libfacedetection (BSD-3). Added unconditionally rather than inside +# the plugin block because the pure-logic tests link it too, and BUILD_TESTING +# can be on with COREVIDEO_BUILD_PLUGIN off (that is how the Linux portability +# job runs). See third_party/libfacedetection/README-COREVIDEO.md. +add_subdirectory(third_party/libfacedetection) +``` + +Inside `if(COREVIDEO_BUILD_PLUGIN)`, in the existing `target_link_libraries(obs-zoom-plugin PRIVATE ...)` call (around line 310), add `libfacedetection` to the list: + +```cmake + target_link_libraries(obs-zoom-plugin PRIVATE + OBS::libobs + OBS::obs-frontend-api + Qt6::Core + Qt6::Network + Qt6::Widgets + libfacedetection + ) +``` + +Inside the `if(BUILD_TESTING)` block, next to the other pure tests: + +```cmake + # Vendored libfacedetection: does it link, does it run twice, does a flat + # image produce a well-formed zero-face result. Not an accuracy test -- + # there is no face image in the repo -- but it catches every way vendoring + # a 446 KB weights blob can silently go wrong. + add_executable(CoreVideoFaceDetectLinkTest + tests/facedetect-link-test.cpp + ) + target_link_libraries(CoreVideoFaceDetectLinkTest PRIVATE libfacedetection) + add_test(NAME CoreVideoFaceDetectLink + COMMAND CoreVideoFaceDetectLinkTest) +``` + +- [ ] **Step 8: Run the test to verify it passes** + +```bash +cmake -S . -B build -DBUILD_TESTING=ON +cmake --build build --config Release --parallel 8 +ctest -C Release -R CoreVideoFaceDetectLink --output-on-failure +``` +Expected: PASS, with `libfacedetection links and runs` on stderr and `flat grey 320x180 -> 0 face(s)`. + +- [ ] **Step 9: Prove the AVX2 guard is live (not silently skipped)** + +The guard only helps if it is actually armed. Verify by breaking it on purpose: + +```bash +# Temporarily comment out the /arch:AVX2 line in +# third_party/libfacedetection/CMakeLists.txt, then: +cmake -S . -B build -DBUILD_TESTING=ON +cmake --build build --config Release --target libfacedetection +``` +Expected: the build FAILS with `libfacedetection is not being compiled with AVX2 enabled`. Restore the line, rebuild, confirm it succeeds again. If the build *succeeded* with the flag removed, the guard is not armed — check that `CMAKE_SYSTEM_PROCESSOR` matched and `COREVIDEO_FD_EXPECT_AVX2` is being defined. + +- [ ] **Step 10: Record the vendoring exception in CLAUDE.md** + +Add to `CLAUDE.md`, under the section describing repository layout / dependencies: + +```markdown +### Vendored third-party source (one exception) + +`third_party/libfacedetection/` is the only vendored library in this repo. It +is 3-clause BSD, has zero dependencies, and compiles its model weights in as +static C arrays, so it adds no DLL and no runtime asset. Read +`third_party/libfacedetection/README-COREVIDEO.md` before touching it: the +files under `src/` are verbatim upstream copies and must not be edited, the +AVX2 build guard exists because MSVC accepts AVX2 intrinsics under /arch:SSE2 +without a word, and OpenMP is off on purpose. + +Everything the plugin does with it goes through `ISubjectDetector` +(`src/subject-detector.h`), a one-method interface, so swapping in the +OpenCV-DNN YuNet fallback would touch one file. +``` + +- [ ] **Step 11: Full test sweep and commit** + +```bash +cmake --build build --config Release --parallel 8 +ctest -C Release --output-on-failure +``` +Expected: N/N green, including the new `CoreVideoFaceDetectLink`. + +```bash +git add third_party/libfacedetection docs/THIRD-PARTY-NOTICES.md \ + tests/facedetect-link-test.cpp CMakeLists.txt CLAUDE.md +git commit -m "feat: vendor libfacedetection (BSD-3) with an AVX2 build guard" +``` + +--- + +### Task 2: Benchmark the real detection cost on this machine + +**Why this is here and not later:** the spec's ~10 ms figure at 320×180 is an **extrapolation** from a published 13.09 ms at 320×240 on a 2017 i7-7820X. It is not a measurement. Every later task's tick rate, and the whole "cost is O(1) in panelist count" argument, rests on the real number. Measure it before anything depends on it. + +**Files:** +- Create: `tests/subject-detector-bench.cpp` +- Modify: `CMakeLists.txt` (test registration) +- Modify: `third_party/libfacedetection/README-COREVIDEO.md` (record the result) + +**Interfaces:** +- Consumes: the `libfacedetection` CMake target and `facedetect_cnn()` from Task 1. +- Produces: a measured per-detection cost in milliseconds at 320×180 and 320×240 on this machine, and a go/no-go verdict against the 100 ms tick that Task 5 assumes. No source API. + +**Go/no-go rule, decided in advance so the measurement cannot be rationalised:** + +| Measured mean at 320×180 | Verdict | +|---|---| +| ≤ 25 ms | **GO** at the planned 100 ms tick (10 detections/s ≈ 25% of one core worst case, typically ~10%). Proceed unchanged. | +| 25–50 ms | **GO, but the tick becomes 200 ms** (5 detections/s). Change `SubjectScheduleConfig::tick_ns` in Task 5 to `200000000ull` and say so in that task's commit message. Consumers are human-paced; 5 Hz across the roster is still within the spec's 2–5 fps envelope for the boosted active speaker. | +| > 50 ms | **NO-GO on libfacedetection.** Stop and escalate. The documented fallback is `cv::FaceDetectorYN` behind the same `ISubjectDetector` interface (Task 4), which is why the interface exists. Do not proceed to Task 5 with a >50 ms detector and a 100 ms tick — that is a worker thread pegged at half a core for a preshow tool. | + +The test itself fails only above 50 ms (the NO-GO line). The 25 ms line is a human decision recorded in the README, not a build failure, because a CI box is allowed to be slower than the operator's machine. + +- [ ] **Step 1: Write the benchmark** + +`tests/subject-detector-bench.cpp`: + +```cpp +// tests/subject-detector-bench.cpp +// Measures what one libfacedetection detection actually costs on THIS machine. +// +// WHY THIS EXISTS. The design spec budgets ~10 ms per detection at 320x180, +// extrapolated from upstream's published 13.09 ms at 320x240 on a 2017 +// i7-7820X. That is an extrapolation, not a measurement, and the entire +// scheduling argument -- one detection per 100 ms tick, cost flat in roster +// size -- rests on it. This test turns the assumption into a number, prints +// it, and fails only past the point where the whole library choice is wrong. +// +// Image CONTENT barely matters to the cost: YuNet is a dense CNN evaluated +// over the whole image, so the per-frame work is fixed by resolution and only +// the (tiny) NMS stage varies with how many candidates survive. A synthetic +// image is therefore a fair timing proxy, and it keeps the repo free of a +// checked-in face photograph. + +#include "facedetectcnn.h" + +#include +#include +#include +#include +#include + +// Something with structure at face-ish scales, so the convolution path is not +// running on a degenerate constant image. Deterministic, no rand(). +static std::vector synthetic_bgr(int w, int h) +{ + std::vector bgr(static_cast(w) * h * 3); + for (int y = 0; y < h; ++y) { + for (int x = 0; x < w; ++x) { + const size_t i = (static_cast(y) * w + x) * 3; + const int checker = ((x / 16) + (y / 16)) & 1; + const int ramp = (x * 255) / (w > 1 ? w - 1 : 1); + bgr[i + 0] = static_cast(checker ? ramp : 255 - ramp); + bgr[i + 1] = static_cast((y * 255) / (h > 1 ? h - 1 : 1)); + bgr[i + 2] = static_cast((ramp + (checker ? 64 : 0)) & 0xFF); + } + } + return bgr; +} + +struct BenchResult { + double mean_ms = 0.0; + double median_ms = 0.0; + double min_ms = 0.0; + double max_ms = 0.0; +}; + +static BenchResult bench(int w, int h, int iterations) +{ + std::vector bgr = synthetic_bgr(w, h); + std::vector buffer(DETECT_BUFFER_SIZE, 0); + + // Warm up: the first call pays page faults on the 446 KB weight arrays and + // any one-time setup. Timing that would measure the wrong thing. + for (int i = 0; i < 3; ++i) + facedetect_cnn(buffer.data(), bgr.data(), w, h, w * 3); + + std::vector samples; + samples.reserve(static_cast(iterations)); + for (int i = 0; i < iterations; ++i) { + const auto t0 = std::chrono::steady_clock::now(); + volatile int *r = facedetect_cnn(buffer.data(), bgr.data(), w, h, w * 3); + (void)r; // volatile so the call cannot be optimised away + const auto t1 = std::chrono::steady_clock::now(); + samples.push_back( + std::chrono::duration(t1 - t0).count()); + } + + std::sort(samples.begin(), samples.end()); + BenchResult out; + double total = 0.0; + for (double s : samples) total += s; + out.mean_ms = total / static_cast(samples.size()); + out.median_ms = samples[samples.size() / 2]; + out.min_ms = samples.front(); + out.max_ms = samples.back(); + return out; +} + +static void report(const char *label, const BenchResult &r) +{ + std::cerr << label + << " mean=" << r.mean_ms << " ms" + << " median=" << r.median_ms << " ms" + << " min=" << r.min_ms << " ms" + << " max=" << r.max_ms << " ms\n"; +} + +int main() +{ + // 320x180 is our real working size: a 16:9 source downscaled to a 320 px + // long edge (src/i420-bgr-downscale.h). + const BenchResult ours = bench(320, 180, 40); + report("320x180 (our working size)", ours); + + // 320x240 is upstream's published benchmark size, so this number is + // directly comparable to the 13.09 ms quoted for a 2017 i7-7820X and tells + // us whether this machine is faster or slower than that reference. + const BenchResult published = bench(320, 240, 40); + report("320x240 (upstream's published size)", published); + + std::cerr << "\n--- schedule budget ---\n"; + const double per_second_at_100ms = ours.mean_ms * 10.0; + std::cerr << "at a 100 ms tick: " << per_second_at_100ms + << " ms of CPU per second = " + << (per_second_at_100ms / 10.0) << "% of one core\n"; + std::cerr << "at a 200 ms tick: " << (ours.mean_ms * 5.0) + << " ms of CPU per second = " + << (ours.mean_ms * 5.0 / 10.0) << "% of one core\n"; + + if (ours.mean_ms <= 25.0) { + std::cerr << "VERDICT: GO at the planned 100 ms tick.\n"; + } else if (ours.mean_ms <= 50.0) { + std::cerr << "VERDICT: GO, but set SubjectScheduleConfig::tick_ns to " + "200000000 (200 ms) in Task 5.\n"; + } else { + std::cerr << "VERDICT: NO-GO. " << ours.mean_ms + << " ms per detection is past the 50 ms ceiling. Stop and " + "escalate; the fallback is cv::FaceDetectorYN behind the " + "same ISubjectDetector interface.\n"; + return 1; + } + + // A detection that is wildly variable is as bad as a slow one: the worker + // is a single thread and a 10x outlier stalls the whole roster's refresh. + if (ours.max_ms > ours.mean_ms * 6.0 && ours.max_ms > 30.0) { + std::cerr << "FAIL: worst-case " << ours.max_ms + << " ms is more than 6x the mean -- detection cost is not " + "stable enough to schedule against.\n"; + return 1; + } + + return 0; +} +``` + +- [ ] **Step 2: Run it to verify it fails** + +Run: `ctest -C Release -R CoreVideoSubjectDetectorBench --output-on-failure` +Expected: FAIL — `No tests were found!!!`, because the target is not registered yet. + +- [ ] **Step 3: Register the benchmark** + +In `CMakeLists.txt`, inside `if(BUILD_TESTING)`, after the `CoreVideoFaceDetectLinkTest` block from Task 1: + +```cmake + # What one detection actually costs on the machine it is built on. The + # spec's ~10 ms budget is an extrapolation from upstream's published + # 13.09 ms at 320x240 on a 2017 i7-7820X; the whole "one detection per + # 100 ms tick, cost flat in roster size" argument rests on the real number, + # so it is measured before anything depends on it. Prints the figure every + # run and fails only past the 50 ms ceiling where the library choice itself + # is wrong. + add_executable(CoreVideoSubjectDetectorBenchTest + tests/subject-detector-bench.cpp + ) + target_link_libraries(CoreVideoSubjectDetectorBenchTest PRIVATE libfacedetection) + add_test(NAME CoreVideoSubjectDetectorBench + COMMAND CoreVideoSubjectDetectorBenchTest) +``` + +- [ ] **Step 4: Build, run, and read the number** + +```bash +cmake --build build --config Release --parallel 8 +ctest -C Release -R CoreVideoSubjectDetectorBench --output-on-failure +``` + +Expected: PASS, with the per-size timings, the schedule budget, and a VERDICT line on stderr. **Read the VERDICT.** If it says NO-GO, stop here and escalate — do not start Task 3. + +Run it three times. Machines under load lie once. + +- [ ] **Step 5: Record the measurement** + +Append to `third_party/libfacedetection/README-COREVIDEO.md`: + +```markdown +## Measured cost (do not replace this with an estimate) + +Measured by `tests/subject-detector-bench.cpp` on the maintainer's Windows +box, Release, AVX2 on: + +| Size | mean | median | max | +|---|---|---|---| +| 320x180 (our working size) | FILL FROM THE RUN ms | FILL ms | FILL ms | +| 320x240 (upstream's published size) | FILL ms | FILL ms | FILL ms | + +Upstream publishes 13.09 ms at 320x240 on a 2017 i7-7820X, which is the +comparison the second row exists for. + +Tick rate chosen on this evidence: 100 ms or 200 ms - see +`SubjectScheduleConfig::tick_ns` in `src/subject-schedule.h`. + +Re-run the benchmark after any re-vendor or compiler upgrade. If the AVX2 +guard in `corevideo-avx2-assert.cpp` were ever bypassed, this is the number +that would move. +``` + +- [ ] **Step 6: Commit** + +```bash +git add tests/subject-detector-bench.cpp CMakeLists.txt third_party/libfacedetection/README-COREVIDEO.md +git commit -m "test: measure real libfacedetection cost and pin the tick-rate go/no-go" +``` + +--- + +### Task 3: I420 → downscaled BGR conversion + +**Files:** +- Create: `src/i420-bgr-downscale.h` +- Create: `tests/i420-bgr-downscale-test.cpp` +- Modify: `CMakeLists.txt` (test registration) + +**Interfaces:** +- Consumes: nothing. +- Produces: + - `struct BgrImage { std::vector pixels; int width = 0; int height = 0; };` — packed BGR8, row stride exactly `width * 3`. + - `constexpr int kSubjectLongEdge = 320;` + - `void subject_scale_size(int src_w, int src_h, int max_long_edge, int &out_w, int &out_h);` + - `bool i420_to_bgr_downscale(const uint8_t *i420, size_t i420_len, int src_w, int src_h, int max_long_edge, BgrImage &out);` + - `void bt709_full_to_bgr(int y, int u, int v, uint8_t &b, uint8_t &g, uint8_t &r);` + - `uint8_t bgr_clamp_byte(float v);` + +- [ ] **Step 1: Write the failing test** + +`tests/i420-bgr-downscale-test.cpp`: + +```cpp +// tests/i420-bgr-downscale-test.cpp +// The colour convert and the downscale sizing, pinned against known values. +// +// The channel-ORDER assertions are the point. libfacedetection wants BGR, the +// plugin holds I420, and an R/B swap produces a picture that still looks like +// a person to a human reviewing a screenshot while quietly costing the CNN +// accuracy. There is no headless GPU harness in this repo to catch that +// downstream, so it is pinned numerically here. + +#include "i420-bgr-downscale.h" + +#include +#include +#include + +static int g_failures = 0; + +static void check(bool cond, const char *what) +{ + if (!cond) { + std::cerr << "FAIL: " << what << "\n"; + ++g_failures; + } +} + +static void check_near(int got, int want, int tol, const char *what) +{ + const int d = got > want ? got - want : want - got; + if (d > tol) { + std::cerr << "FAIL: " << what << " got " << got << " want " << want + << " (+/-" << tol << ")\n"; + ++g_failures; + } +} + +// A flat I420 frame: every luma sample `y`, every chroma pair (u, v). +static std::vector flat_i420(int w, int h, uint8_t y, uint8_t u, uint8_t v) +{ + const size_t y_len = static_cast(w) * h; + std::vector f(y_len + y_len / 2); + for (size_t i = 0; i < y_len; ++i) f[i] = y; + for (size_t i = 0; i < y_len / 4; ++i) { + f[y_len + i] = u; + f[y_len + y_len / 4 + i] = v; + } + return f; +} + +int main() +{ + // ---- sizing ---------------------------------------------------------- + int ow = 0, oh = 0; + + subject_scale_size(1920, 1080, 320, ow, oh); + check(ow == 320 && oh == 180, "1920x1080 -> 320x180"); + + subject_scale_size(640, 360, 320, ow, oh); + check(ow == 320 && oh == 180, "640x360 (the tile default) -> 320x180"); + + // Portrait: the LONG edge is the one that gets capped. + subject_scale_size(360, 640, 320, ow, oh); + check(ow == 180 && oh == 320, "360x640 -> 180x320"); + + // Never upscale. A source already smaller than the cap is passed through: + // inventing pixels cannot add detail for the CNN and only costs time. + subject_scale_size(160, 120, 320, ow, oh); + check(ow == 160 && oh == 120, "160x120 must not be upscaled"); + + // Both output dimensions must be even and at least 2, so the chroma + // sub-sampling indices can never run off the end of the U/V planes. + subject_scale_size(1920, 1078, 320, ow, oh); + check((ow % 2) == 0 && (oh % 2) == 0, "output dims must be even"); + check(ow >= 2 && oh >= 2, "output dims must be at least 2"); + + subject_scale_size(3840, 1080, 320, ow, oh); + check(ow == 320, "32:9 source caps its long edge at 320"); + check((oh % 2) == 0 && oh >= 2, "32:9 source keeps a legal short edge"); + + // ---- the colour convert, by itself ----------------------------------- + uint8_t b = 0, g = 0, r = 0; + + bt709_full_to_bgr(255, 128, 128, b, g, r); + check(b == 255 && g == 255 && r == 255, "Y=255 neutral chroma is white"); + + bt709_full_to_bgr(0, 128, 128, b, g, r); + check(b == 0 && g == 0 && r == 0, "Y=0 neutral chroma is black"); + + bt709_full_to_bgr(128, 128, 128, b, g, r); + check(b == 128 && g == 128 && r == 128, "Y=128 neutral chroma is mid grey"); + + // FULL range, BT.709. These triples are the forward transform of the pure + // primaries: + // red (255,0,0) -> Y 54, U 99, V 255 + // green (0,255,0) -> Y 182, U 30, V 12 + // blue (0,0,255) -> Y 18, U 255, V 116 + // Tolerance 3 absorbs the rounding of that forward direction; it is far + // tighter than any channel swap, which would be off by 200 or more. + bt709_full_to_bgr(54, 99, 255, b, g, r); + check_near(r, 254, 3, "red: R channel"); + check_near(g, 0, 3, "red: G channel"); + check_near(b, 0, 3, "red: B channel"); + + bt709_full_to_bgr(182, 30, 12, b, g, r); + check_near(r, 0, 3, "green: R channel"); + check_near(g, 255, 3, "green: G channel"); + check_near(b, 0, 3, "green: B channel"); + + bt709_full_to_bgr(18, 255, 116, b, g, r); + check_near(r, 0, 3, "blue: R channel"); + check_near(g, 0, 3, "blue: G channel"); + check_near(b, 254, 3, "blue: B channel"); + + // Out-of-gamut input must CLAMP, not wrap. A wrapped byte turns a bright + // edge into a black one, which is exactly the kind of artefact a CNN keys + // on. + bt709_full_to_bgr(255, 255, 255, b, g, r); + check(r == 255, "over-range must clamp high, not wrap"); + bt709_full_to_bgr(16, 16, 16, b, g, r); + check(b == 0 && r == 0, "under-range channels clamp to 0, not wrap to 255"); + + // ---- the whole conversion ------------------------------------------- + BgrImage img; + + // A 640x360 all-red frame downscaled to 320x180: every output pixel must + // be red, and the buffer must be exactly width*height*3 with no padding. + std::vector red = flat_i420(640, 360, 54, 99, 255); + check(i420_to_bgr_downscale(red.data(), red.size(), 640, 360, 320, img), + "640x360 red frame converts"); + check(img.width == 320 && img.height == 180, "converted size is 320x180"); + check(img.pixels.size() == static_cast(320) * 180 * 3, + "buffer is exactly w*h*3 -- stride must be w*3 with no padding"); + bool all_red = true; + for (size_t i = 0; i + 2 < img.pixels.size(); i += 3) { + if (img.pixels[i] > 3 || img.pixels[i + 1] > 3 || + img.pixels[i + 2] < 250) { + all_red = false; + break; + } + } + check(all_red, "every output pixel of a red frame is B=0 G=0 R=254"); + + // Spatial fidelity: a frame whose left half is red and right half is blue + // must still be red on the left and blue on the right after downscaling. + // This is what catches an inverted or transposed sample mapping, which a + // flat frame cannot see. + const int sw = 640, sh = 360; + const size_t y_len = static_cast(sw) * sh; + std::vector split(y_len + y_len / 2); + for (int y = 0; y < sh; ++y) { + for (int x = 0; x < sw; ++x) { + split[static_cast(y) * sw + x] = + (x < sw / 2) ? uint8_t(54) : uint8_t(18); + } + } + for (int cy = 0; cy < sh / 2; ++cy) { + for (int cx = 0; cx < sw / 2; ++cx) { + const size_t ci = static_cast(cy) * (sw / 2) + cx; + const bool left = cx < sw / 4; + split[y_len + ci] = left ? uint8_t(99) : uint8_t(255); + split[y_len + y_len / 4 + ci] = left ? uint8_t(255) : uint8_t(116); + } + } + check(i420_to_bgr_downscale(split.data(), split.size(), sw, sh, 320, img), + "split frame converts"); + { + // Sample well inside each half so the boundary column cannot decide it. + const size_t left_i = + (static_cast(img.height / 2) * img.width + img.width / 4) * 3; + const size_t right_i = + (static_cast(img.height / 2) * img.width + + (img.width * 3) / 4) * 3; + check_near(img.pixels[left_i + 2], 254, 4, "left half stays red (R)"); + check_near(img.pixels[left_i + 0], 0, 4, "left half stays red (B)"); + check_near(img.pixels[right_i + 0], 254, 4, "right half stays blue (B)"); + check_near(img.pixels[right_i + 2], 0, 4, "right half stays blue (R)"); + } + + // Reuse of one BgrImage across frames of different sizes must resize + // correctly rather than leave the old dimensions or old pixels behind -- + // the worker owns exactly one of these for its whole life. + std::vector small = flat_i420(160, 120, 255, 128, 128); + check(i420_to_bgr_downscale(small.data(), small.size(), 160, 120, 320, img), + "smaller frame converts into the reused image"); + check(img.width == 160 && img.height == 120, "reused image resizes down"); + check(img.pixels.size() == static_cast(160) * 120 * 3, + "reused image's buffer resizes down"); + check(img.pixels[0] == 255 && img.pixels[1] == 255 && img.pixels[2] == 255, + "reused image holds the NEW frame's pixels, not the old ones"); + + // ---- refusals -------------------------------------------------------- + check(!i420_to_bgr_downscale(nullptr, 0, 640, 360, 320, img), + "null input is refused"); + check(!i420_to_bgr_downscale(red.data(), red.size() - 1, 640, 360, 320, img), + "a buffer shorter than w*h*3/2 is refused rather than read past"); + check(!i420_to_bgr_downscale(red.data(), red.size(), 0, 360, 320, img), + "zero width is refused"); + check(!i420_to_bgr_downscale(red.data(), red.size(), 641, 360, 320, img), + "odd source width has no valid I420 chroma layout and is refused"); + + if (g_failures) { + std::cerr << g_failures << " check(s) failed\n"; + return 1; + } + std::cerr << "i420-bgr-downscale OK\n"; + return 0; +} +``` + +- [ ] **Step 2: Run the test to verify it fails** + +Run: `cmake --build build --config Release --target CoreVideoI420BgrDownscaleTest` +Expected: FAIL — CMake reports the target does not exist. Once Step 4 registers it, the first compile fails with `Cannot open include file: 'i420-bgr-downscale.h'`. + +- [ ] **Step 3: Write the header** + +`src/i420-bgr-downscale.h`: + +```cpp +#pragma once + +// Turning one I420 frame into the small packed-BGR image the subject detector +// wants, in a single pass, at the downscaled size. +// +// Extracted so it can be tested without libobs, the Zoom SDK or a GPU, the +// same treatment i420-range-expand.h and audio-silence-fade.h get, and for the +// same reason: a wrong colour convert has no loud symptom. It produces a +// picture that still looks like a person to anyone reviewing a screenshot +// while quietly costing the CNN accuracy, and there is no headless GPU harness +// in this repo to catch it downstream. +// +// THREE DECISIONS THIS FILE ENCODES. +// +// 1. BGR, NOT GREYSCALE. libfacedetection is a CNN trained on three-channel +// BGR input. Feeding it the Y plane replicated into three channels -- the +// shortcut that works fine for a Haar or HOG detector -- is not valid here. +// U and V are already sitting in the buffer next to Y, so the convert is +// cheap; skipping it would be a silent accuracy loss, not a saving. +// +// 2. DOWNSCALE AND CONVERT IN ONE PASS, OUTPUT-DRIVEN. The naive order +// (convert the whole frame to BGR, then resize) does 1920*1080 = 2.07M +// colour conversions to throw away 97% of them. This walks the OUTPUT grid, +// so it does exactly out_w*out_h conversions: 57,600 at 320x180. That is +// 36x less work than the naive order at 1080p, and still 4x less at the +// 360p the tiles wall subscribes at (zoom-supersource.cpp's +// tile_feed_subscribe). +// +// 3. POINT SAMPLING, NOT AREA AVERAGING. A box filter would be more faithful, +// but the consumer is a detector looking for a face across tens of pixels, +// not a display. Nearest-neighbour keeps this an integer-indexed single +// pass with no accumulation buffer, and the spec is explicit that detection +// accuracy is not the constraint here -- cost is. +// +// COLOUR SPACE: BT.709, FULL range. Not limited/studio swing. The engine +// requests full range from the SDK and src/i420-range-expand.h normalises the +// frames where the SDK ignores that request, so by the time a frame reaches +// the plugin's buffers it is full range on every path. Using limited-range +// coefficients here would wash out contrast on every frame the detector sees. + +#include +#include +#include + +// The long edge every frame is reduced to before detection. ~320 px keeps a +// face in a 1080p (or 360p) source well resolved while capping the CNN's cost, +// which is fixed by resolution. tests/subject-detector-bench.cpp measures what +// one detection at this size actually costs on the machine it is built on. +constexpr int kSubjectLongEdge = 320; + +// Packed BGR8. Row stride is exactly width*3 -- libfacedetection takes a +// `step` argument and we always pass width*3, so nothing here may pad rows. +struct BgrImage { + std::vector pixels; + int width = 0; + int height = 0; +}; + +// Clamps to a byte without wrapping. A wrapped byte turns a bright edge into a +// black one, which is exactly the kind of artefact a CNN keys on, so this is +// not a cosmetic detail. +inline uint8_t bgr_clamp_byte(float v) +{ + if (v <= 0.0f) return 0; + if (v >= 255.0f) return 255; + return static_cast(v + 0.5f); +} + +// One YUV sample to one BGR pixel. BT.709, full range: +// R = Y + 1.5748 * (V - 128) +// G = Y - 0.1873 * (U - 128) - 0.4681 * (V - 128) +// B = Y + 1.8556 * (U - 128) +// (The inverse of the BT.709 luma weights 0.2126 / 0.7152 / 0.0722 with no +// 16-235 scaling, because the input is full range.) +inline void bt709_full_to_bgr(int y, int u, int v, uint8_t &b, uint8_t &g, + uint8_t &r) +{ + const float fy = static_cast(y); + const float cu = static_cast(u) - 128.0f; + const float cv = static_cast(v) - 128.0f; + r = bgr_clamp_byte(fy + 1.5748f * cv); + g = bgr_clamp_byte(fy - 0.1873f * cu - 0.4681f * cv); + b = bgr_clamp_byte(fy + 1.8556f * cu); +} + +// The output size for a source frame: aspect preserved, long edge capped at +// `max_long_edge`, never upscaled, both dimensions forced even and at least 2. +// +// EVEN is load-bearing, not tidiness: the sampler below maps an output pixel +// to a source chroma sample by halving indices, and an odd dimension makes the +// last row or column's chroma index the first sample past the plane. +inline void subject_scale_size(int src_w, int src_h, int max_long_edge, + int &out_w, int &out_h) +{ + out_w = 0; + out_h = 0; + if (src_w <= 0 || src_h <= 0 || max_long_edge <= 0) return; + + const int long_edge = src_w > src_h ? src_w : src_h; + if (long_edge <= max_long_edge) { + out_w = src_w; + out_h = src_h; + } else if (src_w >= src_h) { + out_w = max_long_edge; + out_h = static_cast( + (static_cast(src_h) * max_long_edge) / src_w); + } else { + out_h = max_long_edge; + out_w = static_cast( + (static_cast(src_w) * max_long_edge) / src_h); + } + + out_w &= ~1; // round DOWN to even + out_h &= ~1; + if (out_w < 2) out_w = 2; + if (out_h < 2) out_h = 2; +} + +// Downscales and converts in one pass. `i420` is tightly packed: Y of +// src_w*src_h, then U of src_w*src_h/4, then V of src_w*src_h/4 -- the layout +// TileFeed::frame carries (see the comment on that member in +// src/zoom-supersource.cpp). +// +// Returns false, leaving `out` untouched, for anything it cannot safely read: +// null, odd source dimensions (no valid I420 chroma layout -- the same +// rejection tile_feed_on_frame already makes), or a buffer shorter than the +// planes it claims. Never reads past `i420_len`. +inline bool i420_to_bgr_downscale(const uint8_t *i420, size_t i420_len, + int src_w, int src_h, int max_long_edge, + BgrImage &out) +{ + if (!i420) return false; + if (src_w < 2 || src_h < 2) return false; + if ((src_w & 1) || (src_h & 1)) return false; + + const size_t y_len = static_cast(src_w) * static_cast(src_h); + if (i420_len < y_len + y_len / 2) return false; + + int out_w = 0, out_h = 0; + subject_scale_size(src_w, src_h, max_long_edge, out_w, out_h); + if (out_w < 2 || out_h < 2) return false; + + const uint8_t *yp = i420; + const uint8_t *up = i420 + y_len; + const uint8_t *vp = up + y_len / 4; + const int cw = src_w / 2; + + out.width = out_w; + out.height = out_h; + // resize, not assign: the worker reuses one BgrImage for its whole life, + // so after the first frame at a given size this allocates nothing. + out.pixels.resize(static_cast(out_w) * out_h * 3); + + for (int oy = 0; oy < out_h; ++oy) { + // Integer nearest-neighbour: the source row whose centre is closest to + // this output row's centre. + int sy = static_cast( + (static_cast(oy) * 2 + 1) * src_h / (2 * out_h)); + if (sy >= src_h) sy = src_h - 1; + const uint8_t *yrow = yp + static_cast(sy) * src_w; + const uint8_t *urow = up + static_cast(sy / 2) * cw; + const uint8_t *vrow = vp + static_cast(sy / 2) * cw; + uint8_t *orow = out.pixels.data() + static_cast(oy) * out_w * 3; + + for (int ox = 0; ox < out_w; ++ox) { + int sx = static_cast( + (static_cast(ox) * 2 + 1) * src_w / (2 * out_w)); + if (sx >= src_w) sx = src_w - 1; + bt709_full_to_bgr(yrow[sx], urow[sx / 2], vrow[sx / 2], + orow[ox * 3 + 0], orow[ox * 3 + 1], + orow[ox * 3 + 2]); + } + } + return true; +} +``` + +- [ ] **Step 4: Register the test** + +In `CMakeLists.txt`, inside `if(BUILD_TESTING)`: + +```cmake + # I420 (BT.709, FULL range) to packed BGR at the detector's working size. + # The channel-order assertions are the reason this exists: libfacedetection + # wants BGR, an R/B swap still looks like a person in a screenshot, and + # there is no headless GPU harness that would catch the accuracy loss + # downstream. Header-only, so no extra .cpp. + add_executable(CoreVideoI420BgrDownscaleTest + tests/i420-bgr-downscale-test.cpp + ) + target_include_directories(CoreVideoI420BgrDownscaleTest PRIVATE + "${CMAKE_CURRENT_SOURCE_DIR}/src" + ) + add_test(NAME CoreVideoI420BgrDownscale + COMMAND CoreVideoI420BgrDownscaleTest) +``` + +- [ ] **Step 5: Run the test to verify it passes** + +```bash +cmake --build build --config Release --parallel 8 +ctest -C Release -R CoreVideoI420BgrDownscale --output-on-failure +``` + +Expected: PASS, `i420-bgr-downscale OK`. + +- [ ] **Step 6: Commit** + +```bash +git add src/i420-bgr-downscale.h tests/i420-bgr-downscale-test.cpp CMakeLists.txt +git commit -m "feat: add I420 to downscaled BGR conversion for the subject detector" +``` + +--- + +### Task 4: The `SubjectFrame` contract and the narrow detector interface + +**Files:** +- Create: `src/subject-frame.h` +- Create: `src/subject-detector-fd-record.h` +- Create: `src/subject-detector.h` +- Create: `src/subject-detector-fd.cpp` +- Create: `tests/subject-detector-record-test.cpp` +- Modify: `CMakeLists.txt` (plugin source, test registration) + +**Interfaces:** +- Consumes: `BgrImage` / `kSubjectLongEdge` from Task 3 (only as documentation of what the caller will pass; the interface itself takes a raw pointer). +- Produces — **this is the published contract two later plans consume; do not change these field names:** + +```cpp +struct SubjectFrame { + bool found; + float box_x, box_y, box_w, box_h; // normalized 0..1 of source + float eye_l_x, eye_l_y, eye_r_x, eye_r_y; // normalized 0..1 + float confidence; + uint64_t detected_ns; // source frame timestamp +}; +``` + +Plus: +- `class ISubjectDetector { virtual SubjectFrame detect(const uint8_t *bgr, int width, int height, uint64_t source_ns) = 0; };` +- `std::unique_ptr make_facedetect_cnn_detector(int min_confidence_pct);` +- `constexpr int kSubjectMinConfidencePct = 60;` +- `SubjectFrame subject_from_fd_record(const short *record, int width, int height, uint64_t source_ns);` +- `const short *subject_best_fd_record(const int *results, int min_confidence_pct);` + +- [ ] **Step 1: Write the failing test** + +`tests/subject-detector-record-test.cpp`: + +```cpp +// tests/subject-detector-record-test.cpp +// The arithmetic between libfacedetection's raw result record and the +// SubjectFrame two other subsystems consume. +// +// WHY THIS IS A SEPARATE, PURE HEADER. Everything interesting about the +// adapter is this arithmetic -- pixel-to-normalized, which of the five +// landmarks are the eyes, which eye is which, clamping a box the CNN pushed +// past the frame edge, and picking the best of several faces. Left inside the +// .cpp that calls the CNN, none of it could be tested without a face +// photograph in the repo and a detector that is guaranteed to find it. Pulled +// out here, all of it is pinnable against hand-built records. +// +// The EYE-ORDER assertion is the one worth reading twice. Upstream emits the +// two eyes in a fixed order which is the SUBJECT's right eye first -- i.e. the +// one on the LEFT of the image. Our field names are eye_l/eye_r and every +// consumer will read them as image-left and image-right (an eyeline midpoint, +// a horizontal-centre check). So the adapter sorts the pair by x and the test +// pins that, rather than trusting a convention that has flipped between +// upstream releases. + +#include "subject-detector-fd-record.h" + +#include +#include +#include +#include + +static int g_failures = 0; + +static void check(bool cond, const char *what) +{ + if (!cond) { + std::cerr << "FAIL: " << what << "\n"; + ++g_failures; + } +} + +static void check_near(float got, float want, float tol, const char *what) +{ + const float d = got > want ? got - want : want - got; + if (d > tol) { + std::cerr << "FAIL: " << what << " got " << got << " want " << want + << "\n"; + ++g_failures; + } +} + +// Builds one libfacedetection result record: confidence, box, then five +// landmark x,y pairs in upstream's order (eye A, eye B, nose, mouth corner, +// mouth corner). 142 shorts per record is upstream's stride. +static std::vector make_record(short conf, short x, short y, short w, + short h, short e0x, short e0y, short e1x, + short e1y) +{ + std::vector rec(142, 0); + rec[0] = conf; + rec[1] = x; + rec[2] = y; + rec[3] = w; + rec[4] = h; + rec[5] = e0x; + rec[6] = e0y; + rec[7] = e1x; + rec[8] = e1y; + rec[9] = static_cast((e0x + e1x) / 2); // nose + rec[10] = static_cast(e0y + 20); + rec[11] = e0x; rec[12] = static_cast(e0y + 40); // mouth corners + rec[13] = e1x; rec[14] = static_cast(e1y + 40); + return rec; +} + +// Packs `records` into the buffer shape facedetect_cnn returns: an int count +// followed by the shorts. +static std::vector make_results(const std::vector> &records) +{ + std::vector buf(1 + (records.size() * 142 * sizeof(short)) / sizeof(int) + 2, 0); + buf[0] = static_cast(records.size()); + short *p = reinterpret_cast(buf.data() + 1); + for (size_t i = 0; i < records.size(); ++i) + std::memcpy(p + 142 * i, records[i].data(), 142 * sizeof(short)); + return buf; +} + +int main() +{ + // ---- normalization --------------------------------------------------- + // A 320x180 image with a face box at (80, 36) size 80x54, eyes at + // (100, 60) and (140, 62). + { + std::vector rec = make_record(92, 80, 36, 80, 54, 100, 60, 140, 62); + const SubjectFrame f = subject_from_fd_record(rec.data(), 320, 180, 12345); + + check(f.found, "a well-formed record produces found=true"); + check_near(f.box_x, 80.0f / 320.0f, 0.001f, "box_x normalizes by width"); + check_near(f.box_y, 36.0f / 180.0f, 0.001f, "box_y normalizes by height"); + check_near(f.box_w, 80.0f / 320.0f, 0.001f, "box_w normalizes by width"); + check_near(f.box_h, 54.0f / 180.0f, 0.001f, "box_h normalizes by height"); + check_near(f.confidence, 0.92f, 0.001f, + "confidence is the record's 0-100 value scaled to 0..1"); + check(f.detected_ns == 12345, + "detected_ns is the SOURCE frame timestamp, passed through"); + } + + // ---- eye ordering ---------------------------------------------------- + // Same face, but the record lists the image-RIGHT eye first. eye_l must + // still be the image-left one, or every consumer's eyeline and + // horizontal-centre arithmetic is mirrored. + { + std::vector a = make_record(90, 80, 36, 80, 54, 100, 60, 140, 62); + std::vector b = make_record(90, 80, 36, 80, 54, 140, 62, 100, 60); + const SubjectFrame fa = subject_from_fd_record(a.data(), 320, 180, 1); + const SubjectFrame fb = subject_from_fd_record(b.data(), 320, 180, 1); + + check_near(fa.eye_l_x, 100.0f / 320.0f, 0.001f, "eye_l is the smaller x"); + check_near(fa.eye_r_x, 140.0f / 320.0f, 0.001f, "eye_r is the larger x"); + check_near(fa.eye_l_y, 60.0f / 180.0f, 0.001f, "eye_l_y follows eye_l_x"); + check_near(fa.eye_r_y, 62.0f / 180.0f, 0.001f, "eye_r_y follows eye_r_x"); + + check_near(fb.eye_l_x, fa.eye_l_x, 0.0001f, + "record order must not change which eye is eye_l"); + check_near(fb.eye_r_x, fa.eye_r_x, 0.0001f, + "record order must not change which eye is eye_r"); + check_near(fb.eye_l_y, fa.eye_l_y, 0.0001f, + "the y coordinate travels with its own x"); + check_near(fb.eye_r_y, fa.eye_r_y, 0.0001f, + "the y coordinate travels with its own x"); + } + + // ---- clamping -------------------------------------------------------- + // The CNN can place a box partly outside the image (a face at the edge). + // Consumers treat these as fractions of the frame and index into it, so + // out-of-range values are a bug they would have to defend against + // individually. Clamp once, here. + { + std::vector rec = make_record(80, -20, -10, 400, 250, -5, 5, 340, 8); + const SubjectFrame f = subject_from_fd_record(rec.data(), 320, 180, 7); + check(f.box_x >= 0.0f && f.box_y >= 0.0f, "box origin clamps to >= 0"); + check(f.box_x + f.box_w <= 1.0001f, + "box must not extend past the right edge"); + check(f.box_y + f.box_h <= 1.0001f, + "box must not extend past the bottom edge"); + check(f.eye_l_x >= 0.0f && f.eye_l_x <= 1.0f, "eye_l_x clamps into 0..1"); + check(f.eye_r_x >= 0.0f && f.eye_r_x <= 1.0f, "eye_r_x clamps into 0..1"); + check(f.eye_l_y >= 0.0f && f.eye_l_y <= 1.0f, "eye_l_y clamps into 0..1"); + check(f.eye_r_y >= 0.0f && f.eye_r_y <= 1.0f, "eye_r_y clamps into 0..1"); + } + + // A degenerate zero-size box is not a subject. + { + std::vector rec = make_record(95, 10, 10, 0, 0, 10, 10, 10, 10); + const SubjectFrame f = subject_from_fd_record(rec.data(), 320, 180, 1); + check(!f.found, "a zero-area box is not a subject"); + } + + // A null record, or a nonsense image size, yields a clean not-found rather + // than dividing by zero. + { + const SubjectFrame f = subject_from_fd_record(nullptr, 320, 180, 1); + check(!f.found, "null record is not a subject"); + std::vector rec = make_record(90, 10, 10, 20, 20, 12, 14, 22, 14); + const SubjectFrame z = subject_from_fd_record(rec.data(), 0, 0, 1); + check(!z.found, "zero-size image is not a subject"); + } + + // ---- picking the best record ---------------------------------------- + { + std::vector> recs; + recs.push_back(make_record(65, 10, 10, 20, 20, 12, 14, 22, 14)); + recs.push_back(make_record(88, 100, 40, 60, 60, 112, 60, 148, 61)); + recs.push_back(make_record(71, 200, 20, 30, 30, 205, 28, 222, 29)); + std::vector buf = make_results(recs); + + const short *best = subject_best_fd_record(buf.data(), 60); + check(best != nullptr, "a record above the threshold is found"); + check(best && best[0] == 88, "the HIGHEST-confidence face wins"); + + // Raising the bar past every candidate must yield nothing, not the + // least-bad one. A preshow framing tool acting on a 40%-confidence + // detection would tell a correctly framed panelist to move. + check(subject_best_fd_record(buf.data(), 95) == nullptr, + "nothing clears a 95 threshold"); + + // Zero faces, and a null pointer, are both ordinary. + std::vector empty = make_results({}); + check(subject_best_fd_record(empty.data(), 60) == nullptr, + "zero faces yields no record"); + check(subject_best_fd_record(nullptr, 60) == nullptr, + "a null results pointer yields no record"); + } + + // ---- the contract itself -------------------------------------------- + { + // A default SubjectFrame must be a safe "no subject", because that is + // what every consumer gets before the first detection lands. + const SubjectFrame f{}; + check(!f.found, "a default-constructed SubjectFrame is not found"); + check(f.confidence == 0.0f && f.detected_ns == 0, + "a default-constructed SubjectFrame is zeroed"); + } + + if (g_failures) { + std::cerr << g_failures << " check(s) failed\n"; + return 1; + } + std::cerr << "subject-detector-record OK\n"; + return 0; +} +``` + +- [ ] **Step 2: Run the test to verify it fails** + +Run: `cmake --build build --config Release --target CoreVideoSubjectDetectorRecordTest` +Expected: FAIL — target does not exist; once registered, `Cannot open include file: 'subject-detector-fd-record.h'`. + +- [ ] **Step 3: Write the contract type** + +`src/subject-frame.h`: + +```cpp +#pragma once + +// The subject detector's published result type. +// +// THIS IS A CONTRACT, NOT AN IMPLEMENTATION DETAIL. Two separate subsystems +// consume it -- Tiles auto-framing and the return-feed framing overlay -- and +// they are built by different people at different times against +// docs/superpowers/specs/2026-09-05-panelist-feedback-design.md. The field +// names and their meanings are fixed there. Renaming, reordering or +// repurposing a field breaks a plan that has not been written yet, so do not. +// +// ALL COORDINATES ARE NORMALIZED 0..1 OF THE SOURCE FRAME, never pixels of the +// downscaled image the detector actually ran on. That is deliberate: the +// detector runs at ~320 px long edge and the consumers work in tile space, in +// canvas space, and in overlay space -- none of which is the detector's +// resolution. Normalizing at the source removes the one conversion everybody +// would otherwise have to get right independently, and it means changing the +// detection resolution never touches a consumer. +// +// eye_l / eye_r are IMAGE-left and IMAGE-right, not the subject's own left and +// right. The adapter sorts them by x so this holds regardless of what order +// the underlying model emits its landmarks in (see +// src/subject-detector-fd-record.h). +// +// `found == false` means "no subject right now" and every other field is +// meaningless -- but note that a SubjectFrame reaching a consumer has been +// through SubjectSmoother (src/subject-smoothing.h), which HOLDS the last good +// result across brief dropouts. So `found == false` from the engine already +// means "gone for longer than the hold period", not "missed one detection". +// +// `detected_ns` is the timestamp of the SOURCE FRAME the detection ran on, not +// the time the detection finished. A consumer can age the result with it: a +// held-through-dropout frame keeps the timestamp of the last real detection, +// which is exactly what "how stale is this?" needs. + +#include + +struct SubjectFrame { + bool found = false; + float box_x = 0.0f, box_y = 0.0f, box_w = 0.0f, box_h = 0.0f; + float eye_l_x = 0.0f, eye_l_y = 0.0f, eye_r_x = 0.0f, eye_r_y = 0.0f; + float confidence = 0.0f; + uint64_t detected_ns = 0; +}; +``` + +- [ ] **Step 4: Write the record arithmetic** + +`src/subject-detector-fd-record.h`: + +```cpp +#pragma once + +// Turning one libfacedetection result record into a SubjectFrame. +// +// Extracted from the adapter that calls the CNN so it can be tested at all. +// Everything that can actually be WRONG about the adapter is here -- +// pixel-to-normalized, which landmarks are the eyes, which eye is which, +// clamping a box the CNN pushed past the frame edge, and picking the best of +// several faces -- and none of it could be exercised inside the .cpp without a +// face photograph in the repo and a detector guaranteed to find it. This +// header needs neither. It also has no include of facedetectcnn.h, on purpose: +// the record layout is described below and repeated nowhere else, so the pure +// test does not drag the 446 KB weights blob into its link. +// +// UPSTREAM'S RESULT LAYOUT (facedetect_cnn's return value): +// results[0] -> face count (int) +// reinterpret_cast(results + 1) + 142*i -> face i's record +// [0] confidence, 0..100 +// [1..4] x, y, w, h in PIXELS of the image passed in +// [5..14] five landmark x,y pairs: two eyes, nose tip, two mouth corners +// +// THE EYE-ORDER RULE. Upstream emits the eyes in a fixed order that is the +// SUBJECT's right eye first, i.e. the one on the LEFT of the image -- and that +// convention has moved between releases of the model this was converted from. +// Our field names are eye_l/eye_r and every consumer reads them as image-left +// and image-right (eyeline midpoint, horizontal-centre check). So we do not +// trust the order: we sort the pair by x. It costs one compare and makes a +// re-vendor unable to silently mirror everybody's framing arithmetic. + +#include "subject-frame.h" + +#include + +// Stride between consecutive face records, in shorts. Upstream's constant. +constexpr int kFdRecordStride = 142; + +// Below this the detection is not acted on. A preshow framing tool that told a +// correctly framed panelist to move because of a 40%-confidence ghost is worse +// than one that says nothing, so the bar is deliberately not at the floor. +constexpr int kSubjectMinConfidencePct = 60; + +inline float subject_clamp01(float v) +{ + if (v < 0.0f) return 0.0f; + if (v > 1.0f) return 1.0f; + return v; +} + +// The highest-confidence record at or above `min_confidence_pct`, or nullptr. +// Highest, not first: upstream's ordering is not a ranking we want to depend +// on, and with one panelist per frame the runner-up is usually a background +// artefact. +inline const short *subject_best_fd_record(const int *results, + int min_confidence_pct) +{ + if (!results) return nullptr; + const int count = *results; + if (count <= 0) return nullptr; + + const short *records = reinterpret_cast(results + 1); + const short *best = nullptr; + int best_conf = min_confidence_pct - 1; + for (int i = 0; i < count; ++i) { + const short *rec = records + kFdRecordStride * i; + if (rec[0] > best_conf) { + best_conf = rec[0]; + best = rec; + } + } + return best; +} + +// One record -> one SubjectFrame, normalized to 0..1 of the image the +// detection ran on. `source_ns` is the timestamp of the frame, carried +// through untouched. +// +// Returns a default (not-found) SubjectFrame for anything unusable: a null +// record, a nonsense image size, or a degenerate zero-area box. +inline SubjectFrame subject_from_fd_record(const short *record, int width, + int height, uint64_t source_ns) +{ + SubjectFrame out{}; + if (!record || width <= 0 || height <= 0) return out; + + const float fw = static_cast(width); + const float fh = static_cast(height); + + float x = static_cast(record[1]); + float y = static_cast(record[2]); + float w = static_cast(record[3]); + float h = static_cast(record[4]); + if (w <= 0.0f || h <= 0.0f) return out; + + // Clamp in PIXELS before normalizing, so a box the CNN pushed off the edge + // shrinks to the visible part instead of keeping its width and sliding. + // Consumers index into frames with these fractions; letting one exceed 1.0 + // would make every consumer defend against it separately. + if (x < 0.0f) { w += x; x = 0.0f; } + if (y < 0.0f) { h += y; y = 0.0f; } + if (x + w > fw) w = fw - x; + if (y + h > fh) h = fh - y; + if (w <= 0.0f || h <= 0.0f) return out; + + // Eyes: landmarks 0 and 1, sorted by x so eye_l is always the image-left + // one. See the header comment -- this is not tidiness, it is what keeps a + // re-vendor from mirroring every consumer's framing arithmetic. + float e0x = static_cast(record[5]); + float e0y = static_cast(record[6]); + float e1x = static_cast(record[7]); + float e1y = static_cast(record[8]); + if (e1x < e0x) { + const float tx = e0x; e0x = e1x; e1x = tx; + const float ty = e0y; e0y = e1y; e1y = ty; + } + + out.found = true; + out.box_x = subject_clamp01(x / fw); + out.box_y = subject_clamp01(y / fh); + out.box_w = subject_clamp01(w / fw); + out.box_h = subject_clamp01(h / fh); + out.eye_l_x = subject_clamp01(e0x / fw); + out.eye_l_y = subject_clamp01(e0y / fh); + out.eye_r_x = subject_clamp01(e1x / fw); + out.eye_r_y = subject_clamp01(e1y / fh); + out.confidence = subject_clamp01(static_cast(record[0]) / 100.0f); + out.detected_ns = source_ns; + return out; +} +``` + +- [ ] **Step 5: Write the narrow interface** + +`src/subject-detector.h`: + +```cpp +#pragma once + +// The entire perception surface of this feature: one downscaled BGR image in, +// one SubjectFrame out. +// +// WHY IT IS THIS NARROW. The spec commits to libfacedetection but keeps +// cv::FaceDetectorYN (the same YuNet model, with an OpenCV dependency) as a +// documented fallback if local benchmarking disappoints. That fallback is only +// cheap if the swap is contained -- one interface, one factory, no consumer +// touched -- so the interface exists from day one rather than being extracted +// later under time pressure. Nothing above this line may learn that the +// detector is a CNN, has weights, or has a result-buffer contract. +// +// The interface is deliberately not thread-safe and does not need to be: it is +// owned and called by exactly one thread, the detector worker in +// src/subject-detector-engine.cpp. Implementations keep a scratch buffer +// across calls precisely because there is only one caller. + +#include "subject-frame.h" + +#include +#include + +class ISubjectDetector { +public: + virtual ~ISubjectDetector() = default; + + // `bgr` is packed BGR8, width*height*3 bytes, row stride exactly width*3 + // -- what src/i420-bgr-downscale.h produces. `source_ns` is the timestamp + // of the frame the pixels came from and is copied into the result's + // detected_ns. + // + // Returns the highest-confidence face found, normalized to 0..1 of the + // image passed in. `found == false` when nothing cleared the detector's + // confidence threshold; the other fields are then meaningless. + virtual SubjectFrame detect(const uint8_t *bgr, int width, int height, + uint64_t source_ns) = 0; +}; + +// The vendored libfacedetection implementation. +// `min_confidence_pct` is on the library's own 0-100 scale; pass +// kSubjectMinConfidencePct (src/subject-detector-fd-record.h) unless a caller +// has a specific reason. +std::unique_ptr make_facedetect_cnn_detector( + int min_confidence_pct); +``` + +- [ ] **Step 6: Write the adapter** + +`src/subject-detector-fd.cpp`: + +```cpp +// The libfacedetection implementation of ISubjectDetector. +// +// This is the ONLY file in the plugin that includes facedetectcnn.h. Keeping +// it that way is what makes the documented cv::FaceDetectorYN fallback a +// one-file swap -- see the header comment on src/subject-detector.h. +// +// All the arithmetic lives in src/subject-detector-fd-record.h so it can be +// unit-tested without a face photograph; what remains here is the call itself +// and the scratch buffer it needs. + +#include "subject-detector.h" +#include "subject-detector-fd-record.h" + +#include "facedetectcnn.h" + +#include + +namespace { + +class FaceDetectCnnDetector : public ISubjectDetector { +public: + explicit FaceDetectCnnDetector(int min_confidence_pct) + : m_min_confidence_pct(min_confidence_pct), + m_buffer(DETECT_BUFFER_SIZE, 0) + { + } + + SubjectFrame detect(const uint8_t *bgr, int width, int height, + uint64_t source_ns) override + { + // Anything smaller than this cannot hold a face at the scales the + // model was trained for, and passing a degenerate image into the CNN + // is not worth finding out about the hard way. + if (!bgr || width < 32 || height < 32) return SubjectFrame{}; + + // The buffer is a member, allocated once: this runs 5-10 times a + // second for the life of the process, and DETECT_BUFFER_SIZE is 128 KB. + // + // The const_cast is upstream's signature, not a lie about ownership -- + // facedetect_cnn takes `unsigned char *` for its input image and does + // not write to it. + int *results = facedetect_cnn( + m_buffer.data(), const_cast(bgr), width, height, + width * 3); + + const short *best = subject_best_fd_record(results, m_min_confidence_pct); + return subject_from_fd_record(best, width, height, source_ns); + } + +private: + int m_min_confidence_pct; + std::vector m_buffer; +}; + +} // namespace + +std::unique_ptr make_facedetect_cnn_detector( + int min_confidence_pct) +{ + return std::unique_ptr( + new FaceDetectCnnDetector(min_confidence_pct)); +} +``` + +- [ ] **Step 7: Register the source and the test** + +In `CMakeLists.txt`, inside `add_library(obs-zoom-plugin MODULE ...)`, after `src/zoom-tile-grid.cpp`: + +```cmake + src/subject-detector-fd.cpp +``` + +Inside `if(BUILD_TESTING)`: + +```cmake + # The arithmetic between libfacedetection's raw records and the + # SubjectFrame two other subsystems consume: normalization, eye ordering, + # edge clamping, best-of-N. Pure -- it deliberately does NOT link + # libfacedetection, so it needs no face photograph and no weights blob to + # pin the part of the adapter that can actually be wrong. + add_executable(CoreVideoSubjectDetectorRecordTest + tests/subject-detector-record-test.cpp + ) + target_include_directories(CoreVideoSubjectDetectorRecordTest PRIVATE + "${CMAKE_CURRENT_SOURCE_DIR}/src" + ) + add_test(NAME CoreVideoSubjectDetectorRecord + COMMAND CoreVideoSubjectDetectorRecordTest) +``` + +- [ ] **Step 8: Run the test to verify it passes** + +```bash +cmake --build build --config Release --parallel 8 +ctest -C Release -R CoreVideoSubjectDetectorRecord --output-on-failure +``` + +Expected: PASS, `subject-detector-record OK`. The plugin must also still link — `src/subject-detector-fd.cpp` is now in it, so a missing `libfacedetection` link from Task 1 shows up here. + +- [ ] **Step 9: Commit** + +```bash +git add src/subject-frame.h src/subject-detector-fd-record.h src/subject-detector.h src/subject-detector-fd.cpp tests/subject-detector-record-test.cpp CMakeLists.txt +git commit -m "feat: add the SubjectFrame contract and the narrow subject-detector interface" +``` + +--- + +### Task 5: Round-robin schedule with an active-speaker boost + +**Files:** +- Create: `src/subject-schedule.h` +- Create: `tests/subject-schedule-test.cpp` +- Modify: `CMakeLists.txt` (test registration) + +**Interfaces:** +- Consumes: nothing. +- Produces: + - `struct SubjectScheduleConfig { uint64_t tick_ns; int speaker_every; };` + - `class SubjectSchedule` with `set_roster(const std::vector &)`, `set_active_speaker(uint32_t)`, `uint32_t next(uint64_t now_ns)`, `uint64_t issued() const`, `size_t roster_size() const`. + +**If Task 2's benchmark returned the 25–50 ms verdict**, set `tick_ns` to `200000000ull` instead of `100000000ull` in the header below, and say so in this task's commit message. Nothing else in the task changes. + +- [ ] **Step 1: Write the failing test** + +`tests/subject-schedule-test.cpp`: + +```cpp +// tests/subject-schedule-test.cpp +// The scheduling property the whole feature's cost argument rests on: one +// detection per tick, no matter how many people are in the room. +// +// The 2-to-20 sweep is the load-bearing assertion. The naive design -- every +// monitored participant detected at 5 fps -- costs 0.5 of a core at ten +// panelists and grows from there. This schedule issues a FIXED number of +// detections per second and cycles who gets them, so a 24-person Zoom Events +// panel costs exactly what a 2-person mic check costs. A regression here would +// not fail anything else: it would just quietly make the preshow tool a CPU +// hog on the biggest shows, which are the ones that can least afford it. + +#include "subject-schedule.h" + +#include +#include +#include +#include + +static int g_failures = 0; + +static void check(bool cond, const char *what) +{ + if (!cond) { + std::cerr << "FAIL: " << what << "\n"; + ++g_failures; + } +} + +static std::vector roster_of(int n) +{ + std::vector ids; + for (int i = 0; i < n; ++i) ids.push_back(static_cast(1000 + i)); + return ids; +} + +int main() +{ + const SubjectScheduleConfig cfg{}; // 100 ms tick, speaker every 3rd slot + + // ---- the O(1) property ---------------------------------------------- + // Drive a simulated clock through one second in 10 ms steps for every + // roster size from 2 to 20, and count the detections issued. It must be + // the same number every time. + { + int expected = -1; + for (int n = 2; n <= 20; ++n) { + SubjectSchedule sched(cfg); + sched.set_roster(roster_of(n)); + sched.set_active_speaker(1000); + + int issued = 0; + for (uint64_t t = 0; t < 1000000000ull; t += 10000000ull) { + if (sched.next(t) != 0) ++issued; + } + if (expected < 0) expected = issued; + if (issued != expected) { + std::cerr << "FAIL: roster of " << n << " issued " << issued + << " detections in one second, roster of 2 issued " + << expected << " -- cost is NOT O(1) in participant " + "count\n"; + ++g_failures; + } + } + // At a 100 ms tick, one second of 10 ms steps is 10 detections. + check(expected == 10, + "a 100 ms tick issues exactly 10 detections per second"); + std::cerr << "detections per second, every roster size 2..20: " + << expected << "\n"; + } + + // A tick is a floor, not a schedule the caller has to hit exactly. Polling + // faster than the tick must not issue more work -- the worker loop wakes + // every few milliseconds and asks on every wakeup. + { + SubjectSchedule sched(cfg); + sched.set_roster(roster_of(5)); + int issued = 0; + for (uint64_t t = 0; t < 1000000000ull; t += 1000000ull) { // 1 ms + if (sched.next(t) != 0) ++issued; + } + check(issued == 10, + "polling at 1 ms must still issue only 10 detections per second"); + } + + // ---- round-robin coverage ------------------------------------------- + // Every participant must be visited. A schedule that boosted the speaker + // so hard that nobody else was ever detected would pass the O(1) test and + // be useless. + { + SubjectSchedule sched(cfg); + sched.set_roster(roster_of(8)); + sched.set_active_speaker(1000); + + std::set seen; + // 8 people, and roughly a third of slots go to the speaker, so allow + // generous time: 60 ticks is 6 seconds. + for (uint64_t t = 0; t < 6000000000ull; t += 100000000ull) { + const uint32_t id = sched.next(t); + if (id != 0) seen.insert(id); + } + check(seen.size() == 8, + "every participant is detected within a few seconds"); + } + + // ---- the active-speaker boost --------------------------------------- + { + SubjectSchedule sched(cfg); + sched.set_roster(roster_of(10)); + sched.set_active_speaker(1005); + + int speaker_hits = 0; + int total = 0; + for (uint64_t t = 0; t < 6000000000ull; t += 100000000ull) { + const uint32_t id = sched.next(t); + if (id == 0) continue; + ++total; + if (id == 1005) ++speaker_hits; + } + // Without a boost, one of ten participants gets a tenth of the slots. + // With it, the person actually being mic-checked gets far more -- + // enough that their framing advice refreshes in under a second rather + // than once per full roster sweep. + check(speaker_hits * 4 > total, + "the active speaker gets substantially more than a fair share"); + check(speaker_hits < total, + "the boost must not starve the rest of the roster"); + std::cerr << "speaker got " << speaker_hits << " of " << total + << " slots in a 10-person roster\n"; + } + + // A speaker who is not in the monitored roster is ignored, not scheduled. + // The directed active speaker can be someone Tiles is not showing. + { + SubjectSchedule sched(cfg); + sched.set_roster(roster_of(3)); // 1000, 1001, 1002 + sched.set_active_speaker(9999); + + for (uint64_t t = 0; t < 3000000000ull; t += 100000000ull) { + const uint32_t id = sched.next(t); + if (id != 0 && id == 9999) { + std::cerr << "FAIL: scheduled a speaker who is not in the " + "monitored roster\n"; + ++g_failures; + break; + } + } + } + + // Speaker 0 means "nobody is speaking" -- the value active_speaker_id() + // returns with no one on air. It must degrade to plain round robin. + { + SubjectSchedule sched(cfg); + sched.set_roster(roster_of(4)); + sched.set_active_speaker(0); + + std::set seen; + int issued = 0; + for (uint64_t t = 0; t < 1000000000ull; t += 100000000ull) { + const uint32_t id = sched.next(t); + if (id != 0) { ++issued; seen.insert(id); } + } + check(issued == 10, "no speaker still issues the full tick rate"); + check(seen.size() == 4, "no speaker still visits everybody"); + } + + // ---- roster churn ---------------------------------------------------- + // The roster shrinking under the cursor must not skip, repeat forever, or + // index out of range. Participants leave mid-show constantly. + { + SubjectSchedule sched(cfg); + sched.set_roster(roster_of(12)); + uint64_t t = 0; + for (int i = 0; i < 8; ++i, t += 100000000ull) sched.next(t); + + sched.set_roster(roster_of(3)); // 9 people leave + std::set seen; + for (int i = 0; i < 20; ++i, t += 100000000ull) { + const uint32_t id = sched.next(t); + if (id != 0) seen.insert(id); + } + check(seen.size() == 3, "after a shrink, exactly the survivors cycle"); + check(seen.count(1005) == 0, "a departed participant is never issued"); + } + + // An empty roster issues nothing at all, forever, and does not wedge. + { + SubjectSchedule sched(cfg); + sched.set_roster({}); + sched.set_active_speaker(1000); + int issued = 0; + for (uint64_t t = 0; t < 2000000000ull; t += 50000000ull) { + if (sched.next(t) != 0) ++issued; + } + check(issued == 0, "an empty roster issues nothing"); + + // ...and recovers the moment someone appears, with no dead interval. + sched.set_roster(roster_of(2)); + check(sched.next(2100000000ull) != 0, + "the schedule resumes as soon as the roster is non-empty"); + } + + // Duplicate ids in the roster must not double a participant's share -- + // a caller assembling the list from two sources is an easy mistake and the + // consequence (uneven refresh) would be invisible. + { + SubjectSchedule sched(cfg); + sched.set_roster({1001, 1002, 1001, 1003, 1002}); + check(sched.roster_size() == 3, "duplicate ids are collapsed"); + } + + // Id 0 is the "nobody" sentinel and must never enter the roster, or + // next() could return it and the caller would read it as "not due yet". + { + SubjectSchedule sched(cfg); + sched.set_roster({0, 1001, 0, 1002}); + check(sched.roster_size() == 2, "id 0 is rejected from the roster"); + } + + // The clock going backwards (a caller passing a non-monotonic value) must + // not issue a burst or wedge the schedule. + { + SubjectSchedule sched(cfg); + sched.set_roster(roster_of(4)); + sched.next(5000000000ull); + check(sched.next(1000000000ull) == 0, + "a backwards clock does not issue immediately"); + check(sched.next(5100000000ull) != 0, + "the schedule recovers on the next legitimate tick"); + } + + if (g_failures) { + std::cerr << g_failures << " check(s) failed\n"; + return 1; + } + std::cerr << "subject-schedule OK\n"; + return 0; +} +``` + +- [ ] **Step 2: Run the test to verify it fails** + +Run: `cmake --build build --config Release --target CoreVideoSubjectScheduleTest` +Expected: FAIL — target does not exist; once registered, `Cannot open include file: 'subject-schedule.h'`. + +- [ ] **Step 3: Write the header** + +`src/subject-schedule.h`: + +```cpp +#pragma once + +// Which participant the detector looks at next, and when. +// +// Extracted so it can be tested without a thread, a clock, or a detector, the +// same treatment media-event-queue.h gets, and for a related reason: the only +// symptom of a regression is a CPU cost that grows with the size of the show. +// That is invisible on a developer's two-person test meeting and expensive on +// a 24-person Zoom Events panel -- the exact inversion that makes it worth a +// test rather than a comment. +// +// THE PROPERTY THIS EXISTS TO GUARANTEE: cost is O(1) in participant count. +// +// The naive design is to detect every monitored participant at 2-5 fps. At ten +// panelists and ~10 ms per detection that is ~0.5 of a core, and it grows +// linearly -- so the biggest shows, which have the least headroom, pay the +// most. We do not need it. Every consumer of this data is slow by design: +// framing advice is read by a human who reacts in seconds, and Tiles +// auto-framing is deliberately hysteretic so tiles do not twitch. So the +// schedule issues ONE detection per tick and cycles who gets it. Ten +// detections a second is roughly a tenth of a core whether the room holds two +// people or twenty-four; the ceiling is fixed by the tick, not by the roster. +// +// THE ACTIVE-SPEAKER BOOST. Plain round robin over 20 people at 10 Hz refreshes +// any one person every 2 seconds, which is too slow for the one person +// actually being mic-checked. Every Nth slot therefore goes to the active +// speaker -- ZoomEngineClient::active_speaker_id(), the DIRECTED id after hold +// and dwell, not the raw one. The boost consumes a slot without advancing the +// round-robin cursor, so it steals refresh rate from everyone else but cannot +// starve anyone: the cursor still walks the whole roster. +// +// EVERY LOOKUP HERE IS O(1) OR O(log n) ON A COLD PATH. `next()` does one +// hash-set lookup and one vector index; only set_roster(), called when the +// monitored set actually changes, is linear. +// +// This class is NOT thread-safe. It is owned and driven by one thread, the +// detector worker in src/subject-detector-engine.cpp, which also serializes +// the roster and speaker updates onto it. + +#include +#include +#include +#include + +struct SubjectScheduleConfig { + // One detection per tick. 100 ms = 10 detections/second across the whole + // roster, which is inside the spec's 2-5 fps envelope for the boosted + // active speaker and comfortably above what a human-paced consumer needs. + // + // THIS NUMBER IS EVIDENCE-BACKED, NOT A GUESS: see the measured cost table + // in third_party/libfacedetection/README-COREVIDEO.md, produced by + // tests/subject-detector-bench.cpp. If the measurement moves, move this. + uint64_t tick_ns = 100000000ull; + + // Every Nth issued slot goes to the active speaker. 3 gives the person + // being checked roughly a third of all detections -- a refresh every + // ~300 ms -- while leaving two thirds for the round robin. + int speaker_every = 3; +}; + +class SubjectSchedule { +public: + explicit SubjectSchedule(SubjectScheduleConfig cfg = {}) : m_cfg(cfg) {} + + // The participants worth detecting -- in practice the ones actually being + // shown, not the whole meeting. Duplicates and the id-0 sentinel are + // dropped. Linear, and called only when the monitored set changes. + void set_roster(const std::vector &ids) + { + m_roster.clear(); + m_present.clear(); + for (uint32_t id : ids) { + if (id == 0) continue; // 0 is next()'s "not due" value + if (!m_present.insert(id).second) continue; // already have it + m_roster.push_back(id); + } + if (m_roster.empty()) m_cursor = 0; + else if (m_cursor >= m_roster.size()) m_cursor = 0; + } + + // ZoomEngineClient::active_speaker_id() -- the DIRECTED id, post hold and + // dwell. 0 means nobody, which degrades cleanly to plain round robin. + // SpeakerDirector is poll-only, so the worker polls this in; there is no + // observer list to subscribe to. + void set_active_speaker(uint32_t id) { m_speaker = id; } + + // The participant to detect now, or 0 when the tick has not elapsed. + // + // Safe to call as often as the caller likes: the tick is a floor, so a + // worker polling at 5 ms issues exactly as much work as one polling at + // 100 ms. A backwards clock simply does not fire until now_ns passes the + // stored deadline again, which is the conservative direction -- it costs + // one skipped detection, never a burst. + uint32_t next(uint64_t now_ns) + { + if (m_roster.empty()) return 0; + if (m_started && now_ns - m_last_ns < m_cfg.tick_ns && + now_ns >= m_last_ns) { + return 0; + } + if (m_started && now_ns < m_last_ns) { + // Clock went backwards. Re-anchor and wait out one full tick + // rather than firing on every call until it catches up. + m_last_ns = now_ns; + return 0; + } + m_started = true; + m_last_ns = now_ns; + ++m_issued; + + if (m_speaker != 0 && m_cfg.speaker_every > 0 && + (m_issued % static_cast(m_cfg.speaker_every)) == 0 && + m_present.count(m_speaker) != 0) { + // Deliberately does NOT advance the cursor: the boost borrows a + // slot, it does not consume anyone's turn. + return m_speaker; + } + + const uint32_t id = m_roster[m_cursor]; + m_cursor = (m_cursor + 1) % m_roster.size(); + return id; + } + + uint64_t issued() const { return m_issued; } + size_t roster_size() const { return m_roster.size(); } + +private: + SubjectScheduleConfig m_cfg; + std::vector m_roster; + std::unordered_set m_present; // O(1) "is the speaker monitored?" + size_t m_cursor = 0; + uint32_t m_speaker = 0; + uint64_t m_last_ns = 0; + uint64_t m_issued = 0; + bool m_started = false; +}; +``` + +- [ ] **Step 4: Register the test** + +In `CMakeLists.txt`, inside `if(BUILD_TESTING)`: + +```cmake + # One detection per tick, cycled across the roster with a boost for the + # active speaker. The 2-to-20 sweep is the point: detection cost must not + # grow with the size of the show, and nothing else in the suite would fail + # if it started to -- the tool would just become a CPU hog on exactly the + # biggest panels. Header-only, so no extra .cpp. + add_executable(CoreVideoSubjectScheduleTest + tests/subject-schedule-test.cpp + ) + target_include_directories(CoreVideoSubjectScheduleTest PRIVATE + "${CMAKE_CURRENT_SOURCE_DIR}/src" + ) + add_test(NAME CoreVideoSubjectSchedule + COMMAND CoreVideoSubjectScheduleTest) +``` + +- [ ] **Step 5: Run the test to verify it passes** + +```bash +cmake --build build --config Release --parallel 8 +ctest -C Release -R CoreVideoSubjectSchedule --output-on-failure +``` + +Expected: PASS, with `detections per second, every roster size 2..20: 10` and the speaker share printed. + +- [ ] **Step 6: Commit** + +```bash +git add src/subject-schedule.h tests/subject-schedule-test.cpp CMakeLists.txt +git commit -m "feat: add the O(1) round-robin subject-detection schedule" +``` + +(If Task 2's benchmark forced the 200 ms tick, the message is +`feat: add the O(1) round-robin subject-detection schedule (200 ms tick, per measured detector cost)` +and the first assertion's expected count in the test becomes 5, not 10.) + +--- + +### Task 6: Temporal smoothing — deadband, rate limit, dropout hold + +**Files:** +- Create: `src/subject-smoothing.h` +- Create: `tests/subject-smoothing-test.cpp` +- Modify: `CMakeLists.txt` (test registration) + +**Interfaces:** +- Consumes: `SubjectFrame` from Task 4. +- Produces: + - `struct SubjectSmoothingConfig { float deadband; float max_travel_per_sec; uint64_t dropout_hold_ns; float min_confidence; uint64_t max_step_dt_ns; };` + - `float subject_smooth_coord(float current, float target, float deadband, float max_step);` + - `class SubjectSmoother` with `explicit SubjectSmoother(SubjectSmoothingConfig cfg = {})`, `void update(const SubjectFrame &raw, uint64_t now_ns)`, `SubjectFrame output() const`, `void reset()`. + +- [ ] **Step 1: Write the failing test** + +`tests/subject-smoothing-test.cpp`: + +```cpp +// tests/subject-smoothing-test.cpp +// Deadband, rate limit and dropout hold, which the spec calls mandatory rather +// than optional. +// +// Two of these assertions describe a live failure the feature would otherwise +// ship with. A panelist who turns their head for half a second must NOT be +// told "step into frame" -- that is a tool actively lying to the person it is +// supposed to help, and the raw detector produces exactly that, because a +// profile view drops below the confidence bar for a few detections. And a +// motionless panelist must produce ZERO tile movement: the detector's box +// jitters by a pixel or two between frames on a completely still subject, and +// without a deadband that jitter becomes a permanently crawling tile. + +#include "subject-smoothing.h" + +#include +#include + +static int g_failures = 0; + +static void check(bool cond, const char *what) +{ + if (!cond) { + std::cerr << "FAIL: " << what << "\n"; + ++g_failures; + } +} + +static void check_near(float got, float want, float tol, const char *what) +{ + const float d = got > want ? got - want : want - got; + if (d > tol) { + std::cerr << "FAIL: " << what << " got " << got << " want " << want + << "\n"; + ++g_failures; + } +} + +static SubjectFrame detection(float x, float y, float conf, uint64_t ns) +{ + SubjectFrame f{}; + f.found = true; + f.box_x = x; + f.box_y = y; + f.box_w = 0.25f; + f.box_h = 0.40f; + f.eye_l_x = x + 0.06f; + f.eye_l_y = y + 0.12f; + f.eye_r_x = x + 0.18f; + f.eye_r_y = y + 0.12f; + f.confidence = conf; + f.detected_ns = ns; + return f; +} + +static const uint64_t kTick = 100000000ull; // 100 ms, the schedule's tick + +int main() +{ + const SubjectSmoothingConfig cfg{}; // deadband 0.01, 0.35/s, 1.5 s hold + + // ---- the first detection snaps -------------------------------------- + // Ramping in from (0,0) would sweep a tile across the frame on the first + // acquisition, which is the most visible moment there is. + { + SubjectSmoother s(cfg); + const SubjectFrame raw = detection(0.30f, 0.20f, 0.9f, 500); + s.update(raw, kTick); + const SubjectFrame out = s.output(); + check(out.found, "the first detection is found"); + check_near(out.box_x, 0.30f, 0.0001f, "first detection SNAPS box_x"); + check_near(out.box_y, 0.20f, 0.0001f, "first detection SNAPS box_y"); + check_near(out.eye_l_x, 0.36f, 0.0001f, "first detection snaps eye_l_x"); + check(out.detected_ns == 500, + "detected_ns is the source timestamp, not the smoother's clock"); + } + + // ---- deadband: jitter produces ZERO movement ------------------------- + { + SubjectSmoother s(cfg); + uint64_t t = kTick; + s.update(detection(0.30f, 0.20f, 0.9f, t), t); + const float x0 = s.output().box_x; + const float y0 = s.output().box_y; + + // 50 detections wobbling +/-0.005 around the seed. Deadband is 0.01, + // so not one of them may move the output by any amount at all -- + // "small movements" is not good enough, a tile that creeps is a tile + // an operator notices. + for (int i = 0; i < 50; ++i) { + t += kTick; + const float wobble = (i % 2 == 0) ? 0.005f : -0.004f; + s.update(detection(0.30f + wobble, 0.20f - wobble, 0.9f, t), t); + } + check(s.output().box_x == x0, + "jitter inside the deadband produces EXACTLY zero movement in x"); + check(s.output().box_y == y0, + "jitter inside the deadband produces EXACTLY zero movement in y"); + } + + // A real move, larger than the deadband, does eventually get there. + { + SubjectSmoother s(cfg); + uint64_t t = kTick; + s.update(detection(0.30f, 0.20f, 0.9f, t), t); + for (int i = 0; i < 60; ++i) { + t += kTick; + s.update(detection(0.60f, 0.20f, 0.9f, t), t); + } + check_near(s.output().box_x, 0.60f, 0.02f, + "a real move converges on the target, it is not just damped"); + } + + // ---- rate limit ------------------------------------------------------ + { + SubjectSmoother s(cfg); + uint64_t t = kTick; + s.update(detection(0.10f, 0.20f, 0.9f, t), t); + + // Jump the target to 0.90 in one 100 ms step. max_travel_per_sec is + // 0.35, so the step is at most 0.035; with the deadband taken off the + // wanted delta first, the output lands at 0.135. + t += kTick; + s.update(detection(0.90f, 0.20f, 0.9f, t), t); + check_near(s.output().box_x, 0.135f, 0.002f, + "a big jump is rate-limited to max_travel_per_sec * dt"); + check(s.output().box_x < 0.20f, + "the output must not teleport to the new position"); + } + + // The rate limit is per SECOND, not per update: a longer gap between + // detections allows proportionally more travel. + { + SubjectSmoother s(cfg); + uint64_t t = kTick; + s.update(detection(0.10f, 0.20f, 0.9f, t), t); + t += 4 * kTick; // 400 ms + s.update(detection(0.90f, 0.20f, 0.9f, t), t); + check_near(s.output().box_x, 0.10f + 0.35f * 0.4f, 0.003f, + "400 ms allows 4x the travel of 100 ms"); + } + + // ...but only up to a cap. After a long stall (an idle meeting, a source + // that stopped sending) the first detection back must not be allowed a + // travel budget of several seconds, which would be a teleport wearing a + // rate limit's clothes. + { + SubjectSmoother s(cfg); + uint64_t t = kTick; + s.update(detection(0.10f, 0.20f, 0.9f, t), t); + t += 30000000000ull; // 30 seconds + s.update(detection(0.90f, 0.20f, 0.9f, t), t); + check(s.output().box_x < 0.35f, + "a long stall does not bank an unbounded travel budget"); + } + + // ---- dropout hold ---------------------------------------------------- + // THE ASSERTION THIS FILE EXISTS FOR: a brief dropout must not flip state + // to "no subject". A panelist turning their head drops the detector below + // its confidence bar for a few hundred milliseconds; telling them to "step + // into frame" for that is the tool lying to the person it is helping. + { + SubjectSmoother s(cfg); + uint64_t t = kTick; + s.update(detection(0.30f, 0.20f, 0.9f, t), t); + const float held_x = s.output().box_x; + const uint64_t held_ns = s.output().detected_ns; + + // 10 consecutive misses at 100 ms = 1.0 s, inside the 1.5 s hold. + for (int i = 0; i < 10; ++i) { + t += kTick; + s.update(SubjectFrame{}, t); + check(s.output().found, + "a brief dropout must NOT flip state to no-subject"); + } + check(s.output().box_x == held_x, + "the held result keeps its last good position, unmoved"); + check(s.output().detected_ns == held_ns, + "the held result keeps the LAST REAL detection's timestamp, so a " + "consumer can tell how stale it is"); + + // Past the hold, it does give up -- "step into frame" is correct once + // somebody has genuinely left. + t += 6 * kTick; // total 1.6 s of dropout + s.update(SubjectFrame{}, t); + check(!s.output().found, + "past the hold period the subject IS declared gone"); + check(s.output().confidence == 0.0f, + "a gone subject reports a zeroed frame, not stale coordinates"); + } + + // A detection arriving during the hold resumes smoothly from the HELD + // position -- not from wherever the raw detector happens to be, and not by + // re-snapping, which would make every head turn a visible jump. + { + SubjectSmoother s(cfg); + uint64_t t = kTick; + s.update(detection(0.30f, 0.20f, 0.9f, t), t); + for (int i = 0; i < 5; ++i) { t += kTick; s.update(SubjectFrame{}, t); } + t += kTick; + s.update(detection(0.80f, 0.20f, 0.9f, t), t); + check(s.output().found, "the subject is still found after recovery"); + check(s.output().box_x < 0.55f, + "recovery is rate-limited from the held position, not a snap"); + check(s.output().box_x > 0.30f, "recovery does move toward the target"); + } + + // Re-acquisition AFTER the hold expired snaps, because there is no longer + // a held position to travel from -- sliding a tile in from a stale + // location the subject has left is worse than a cut. + { + SubjectSmoother s(cfg); + uint64_t t = kTick; + s.update(detection(0.30f, 0.20f, 0.9f, t), t); + t += 20 * kTick; // 2.0 s of nothing: well past the hold + s.update(SubjectFrame{}, t); + check(!s.output().found, "gone before re-acquisition"); + t += kTick; + s.update(detection(0.80f, 0.20f, 0.9f, t), t); + check_near(s.output().box_x, 0.80f, 0.0001f, + "re-acquisition after a full dropout SNAPS"); + } + + // ---- confidence gate ------------------------------------------------- + // A low-confidence detection is treated as a MISS, not as data. Acting on + // a 30%-confidence ghost would move a tile onto a bookshelf. + { + SubjectSmoother s(cfg); + uint64_t t = kTick; + s.update(detection(0.30f, 0.20f, 0.9f, t), t); + const float x0 = s.output().box_x; + t += kTick; + s.update(detection(0.90f, 0.60f, 0.20f, t), t); + check(s.output().found, "a low-confidence frame is held, not dropped"); + check(s.output().box_x == x0, + "a low-confidence frame must not move the output at all"); + } + + // A low-confidence run must still eventually time out, exactly like an + // absent one -- otherwise a persistent false positive holds "found" + // forever. + { + SubjectSmoother s(cfg); + uint64_t t = kTick; + s.update(detection(0.30f, 0.20f, 0.9f, t), t); + for (int i = 0; i < 20; ++i) { + t += kTick; + s.update(detection(0.30f, 0.20f, 0.10f, t), t); + } + check(!s.output().found, + "a long run of low-confidence frames times out like a dropout"); + } + + // ---- reset ----------------------------------------------------------- + // A participant leaving and rejoining, or a slot repointing, must not + // inherit the previous person's position. + { + SubjectSmoother s(cfg); + s.update(detection(0.30f, 0.20f, 0.9f, kTick), kTick); + s.reset(); + check(!s.output().found, "reset clears the held result"); + s.update(detection(0.70f, 0.50f, 0.9f, 2 * kTick), 2 * kTick); + check_near(s.output().box_x, 0.70f, 0.0001f, + "the first detection after a reset snaps, like a first one"); + } + + // ---- the coordinate primitive itself -------------------------------- + { + check(subject_smooth_coord(0.5f, 0.505f, 0.01f, 1.0f) == 0.5f, + "inside the deadband: unchanged, bit for bit"); + check(subject_smooth_coord(0.5f, 0.495f, 0.01f, 1.0f) == 0.5f, + "inside the deadband in the negative direction too"); + check_near(subject_smooth_coord(0.5f, 0.6f, 0.01f, 1.0f), 0.59f, 0.0001f, + "outside the deadband: moves by delta MINUS the deadband, so " + "crossing the threshold by a hair is not a jump"); + check_near(subject_smooth_coord(0.5f, 0.6f, 0.01f, 0.02f), 0.52f, + 0.0001f, "the step is capped by max_step"); + check_near(subject_smooth_coord(0.5f, 0.4f, 0.01f, 0.02f), 0.48f, + 0.0001f, "the cap applies symmetrically downward"); + } + + if (g_failures) { + std::cerr << g_failures << " check(s) failed\n"; + return 1; + } + std::cerr << "subject-smoothing OK\n"; + return 0; +} +``` + +- [ ] **Step 2: Run the test to verify it fails** + +Run: `cmake --build build --config Release --target CoreVideoSubjectSmoothingTest` +Expected: FAIL — target does not exist; once registered, `Cannot open include file: 'subject-smoothing.h'`. + +- [ ] **Step 3: Write the header** + +`src/subject-smoothing.h`: + +```cpp +#pragma once + +// Making the detector's raw output fit to drive a tile crop and a person's +// framing advice. +// +// Extracted so it can be tested without a detector, a thread or a GPU -- the +// same treatment director-handover.h and iso-video-pacer.h get, and for the +// same reason: every failure here is a behaviour on air that no other test in +// this repo would notice. +// +// RAW DETECTION OUTPUT IS NOT USABLE AS-IS, in three separate ways, and the +// spec calls all three mandatory rather than optional: +// +// 1. IT JITTERS. The box moves a pixel or two between detections on a +// completely motionless subject. Fed straight to a tile crop that is a tile +// that never stops crawling -- the single most noticeable artefact +// auto-framing can have. The DEADBAND answers it, and the bar is exact +// zero movement, not small movement: a tile that creeps slowly is still a +// tile an operator sees. +// +// 2. IT STEPS. A person leaning across their desk moves the box a long way +// between two detections 100 ms apart. Followed literally the tile snaps. +// The RATE LIMIT answers it, in normalized units per second so it is +// independent of how often detections actually arrive. +// +// 3. IT DROPS OUT. A head turn takes the face below the confidence bar for a +// few detections. Reported honestly that reads as "no subject", and the +// return overlay tells a correctly seated panelist to STEP INTO FRAME -- +// the tool actively lying to the person it exists to help. The DROPOUT +// HOLD answers it: keep the last good result for a while before declaring +// anyone gone. +// +// WHY A LOW-CONFIDENCE DETECTION IS TREATED AS A MISS, NOT AS DATA: acting on +// a 30%-confidence hit moves a tile onto a bookshelf. It is held exactly like +// an absence, which also means a long run of them times out like one, so a +// persistent false positive cannot hold "found" forever. +// +// WHY THE FIRST DETECTION SNAPS: ramping in from (0,0) sweeps the crop across +// the frame at the most visible moment there is -- acquisition. Same after the +// hold expires: there is no longer a held position worth travelling from, and +// sliding in from where somebody used to be is worse than a cut. +// +// One SubjectSmoother per participant. Not thread-safe; the detector worker +// owns them all and is the only thread that touches them. + +#include "subject-frame.h" + +#include + +struct SubjectSmoothingConfig { + // Movement below this (normalized units, so a fraction of the frame) is + // ignored entirely. 0.01 is 1% of the frame -- ~19 px of a 1920 wide + // source, comfortably above detector jitter and well below a real shift + // of position. + float deadband = 0.01f; + + // Maximum travel per second, normalized units. 0.35 crosses a third of the + // frame in a second: fast enough to follow someone changing seats within a + // couple of seconds, slow enough that no single detection can snap a tile. + float max_travel_per_sec = 0.35f; + + // How long a good result is held before the subject is declared gone. + // 1.5 s covers a head turn, a hand across the face, and a couple of missed + // round-robin slots; it is short enough that somebody who really walked + // away is reported within about two seconds. + uint64_t dropout_hold_ns = 1500000000ull; + + // Below this, a detection is treated as a miss. Distinct from the + // detector's own kSubjectMinConfidencePct: that one decides what the CNN + // reports at all, this one decides what we are willing to MOVE for. + float min_confidence = 0.5f; + + // Caps the dt used for the rate limit. Without it, the first detection + // after a long stall (an idle meeting, a source that stopped sending) + // banks seconds of travel budget and teleports -- a rate limit in name + // only. 500 ms is five schedule ticks: generous for a real gap, useless + // as a teleport. + uint64_t max_step_dt_ns = 500000000ull; +}; + +// Moves `current` toward `target` subject to a deadband and a maximum step. +// +// Note the deadband is SUBTRACTED from the wanted delta rather than just +// gating it. Gating alone means a movement one unit past the threshold jumps +// the entire delta, so the deadband would create the very discontinuity it +// exists to remove; subtracting makes movement continuous across the +// threshold. +inline float subject_smooth_coord(float current, float target, float deadband, + float max_step) +{ + const float delta = target - current; + const float mag = delta < 0.0f ? -delta : delta; + if (mag <= deadband) return current; + + float wanted = delta > 0.0f ? (delta - deadband) : (delta + deadband); + if (wanted > max_step) wanted = max_step; + if (wanted < -max_step) wanted = -max_step; + return current + wanted; +} + +class SubjectSmoother { +public: + explicit SubjectSmoother(SubjectSmoothingConfig cfg = {}) : m_cfg(cfg) {} + + // Feed EVERY scheduled result, hit or miss. A miss is a default- + // constructed SubjectFrame. `now_ns` is a monotonic clock; the smoother + // measures elapsed time with it, so it must be the same clock every call. + void update(const SubjectFrame &raw, uint64_t now_ns) + { + const bool usable = raw.found && raw.confidence >= m_cfg.min_confidence; + + if (!usable) { + if (!m_have) return; // already given up; nothing to hold + if (now_ns >= m_last_good_ns && + now_ns - m_last_good_ns > m_cfg.dropout_hold_ns) { + // Genuinely gone. Zero the whole frame rather than clearing + // `found` alone, so a consumer that forgets to check it reads + // an obviously-empty box instead of a stale position. + m_have = false; + m_out = SubjectFrame{}; + } + // Inside the hold: deliberately change NOTHING. m_out keeps its + // position, its confidence and the last real detection's + // detected_ns, which is what lets a consumer age it. + return; + } + + if (!m_have) { + // Acquisition (first ever, or after the hold expired): snap. + m_out = raw; + m_have = true; + m_last_good_ns = now_ns; + m_last_update_ns = now_ns; + return; + } + + uint64_t dt_ns = now_ns > m_last_update_ns ? now_ns - m_last_update_ns : 0; + if (dt_ns > m_cfg.max_step_dt_ns) dt_ns = m_cfg.max_step_dt_ns; + const float max_step = + m_cfg.max_travel_per_sec * (static_cast(dt_ns) / 1e9f); + + m_out.box_x = subject_smooth_coord(m_out.box_x, raw.box_x, m_cfg.deadband, max_step); + m_out.box_y = subject_smooth_coord(m_out.box_y, raw.box_y, m_cfg.deadband, max_step); + m_out.box_w = subject_smooth_coord(m_out.box_w, raw.box_w, m_cfg.deadband, max_step); + m_out.box_h = subject_smooth_coord(m_out.box_h, raw.box_h, m_cfg.deadband, max_step); + m_out.eye_l_x = subject_smooth_coord(m_out.eye_l_x, raw.eye_l_x, m_cfg.deadband, max_step); + m_out.eye_l_y = subject_smooth_coord(m_out.eye_l_y, raw.eye_l_y, m_cfg.deadband, max_step); + m_out.eye_r_x = subject_smooth_coord(m_out.eye_r_x, raw.eye_r_x, m_cfg.deadband, max_step); + m_out.eye_r_y = subject_smooth_coord(m_out.eye_r_y, raw.eye_r_y, m_cfg.deadband, max_step); + + // Confidence and the timestamp are NOT smoothed. They describe the + // detection, not a position: a consumer asking "how confident, how + // recent" wants the real answer, not a rolling average of one. + m_out.found = true; + m_out.confidence = raw.confidence; + m_out.detected_ns = raw.detected_ns; + + m_last_good_ns = now_ns; + m_last_update_ns = now_ns; + } + + SubjectFrame output() const { return m_out; } + + // Forget everything. Call when the participant this smoother tracks + // changes -- a leave and rejoin, or a tile slot repointing -- so the new + // person does not inherit the previous one's position and travel to it. + void reset() + { + m_out = SubjectFrame{}; + m_have = false; + m_last_good_ns = 0; + m_last_update_ns = 0; + } + +private: + SubjectSmoothingConfig m_cfg; + SubjectFrame m_out{}; + bool m_have = false; + uint64_t m_last_good_ns = 0; + uint64_t m_last_update_ns = 0; +}; +``` + +- [ ] **Step 4: Register the test** + +In `CMakeLists.txt`, inside `if(BUILD_TESTING)`: + +```cmake + # Deadband, rate limit and dropout hold on the detector's output. Two of + # its assertions describe behaviour the feature would otherwise ship as a + # defect: a motionless panelist must produce EXACTLY zero tile movement, + # and a brief detection dropout must not tell a correctly framed panelist + # to step into frame. Header-only, so no extra .cpp. + add_executable(CoreVideoSubjectSmoothingTest + tests/subject-smoothing-test.cpp + ) + target_include_directories(CoreVideoSubjectSmoothingTest PRIVATE + "${CMAKE_CURRENT_SOURCE_DIR}/src" + ) + add_test(NAME CoreVideoSubjectSmoothing + COMMAND CoreVideoSubjectSmoothingTest) +``` + +- [ ] **Step 5: Run the test to verify it passes** + +```bash +cmake --build build --config Release --parallel 8 +ctest -C Release -R CoreVideoSubjectSmoothing --output-on-failure +``` + +Expected: PASS, `subject-smoothing OK`. + +- [ ] **Step 6: Commit** + +```bash +git add src/subject-smoothing.h tests/subject-smoothing-test.cpp CMakeLists.txt +git commit -m "feat: add temporal smoothing (deadband, rate limit, dropout hold) for SubjectFrame" +``` + +--- + +### Task 7: The single-slot frame inbox + +**Files:** +- Create: `src/subject-frame-inbox.h` +- Create: `tests/subject-inbox-test.cpp` +- Modify: `CMakeLists.txt` (test registration) + +**Interfaces:** +- Consumes: nothing. +- Produces: `class SubjectFrameInbox` with + - `void request(uint32_t participant_id)` + - `uint32_t wanted() const` + - `bool offer(uint32_t participant_id, const uint8_t *i420, size_t len, uint32_t width, uint32_t height, uint64_t source_ns)` + - `bool take(std::vector &pixels, uint32_t &participant_id, uint32_t &width, uint32_t &height, uint64_t &source_ns)` + - `void cancel()` + - `uint64_t offers_accepted() const`, `uint64_t offers_declined() const` + +- [ ] **Step 1: Write the failing test** + +`tests/subject-inbox-test.cpp`: + +```cpp +// tests/subject-inbox-test.cpp +// The handshake that keeps the detector off the engine-IPC reader thread. +// +// WHY THE DECLINE PATH IS THE IMPORTANT ONE. offer() is called from +// tile_feed_on_frame(), on the shared reader thread that dispatches frames for +// EVERY source in the plugin. At a nine-tile wall that is hundreds of calls a +// second, and at most ten of them per second may do any work. So the assertion +// that matters is not "a requested frame arrives" but "an unrequested frame +// costs one atomic load and nothing else" -- no copy, no allocation, and +// above all no lock, because a lock there is the 2026-08-17 head-of-line +// stall (src/media-event-queue.h) rebuilt by hand. + +#include "subject-frame-inbox.h" + +#include +#include +#include +#include +#include + +static int g_failures = 0; + +static void check(bool cond, const char *what) +{ + if (!cond) { + std::cerr << "FAIL: " << what << "\n"; + ++g_failures; + } +} + +static std::vector fake_i420(uint32_t w, uint32_t h, uint8_t fill) +{ + const size_t y_len = static_cast(w) * h; + return std::vector(y_len + y_len / 2, fill); +} + +int main() +{ + const std::vector frame_a = fake_i420(64, 64, 0x11); + const std::vector frame_b = fake_i420(64, 64, 0x22); + + // ---- nothing requested ---------------------------------------------- + { + SubjectFrameInbox inbox; + check(inbox.wanted() == 0, "a fresh inbox wants nothing"); + check(!inbox.offer(7, frame_a.data(), frame_a.size(), 64, 64, 1), + "an offer with nothing requested is declined"); + check(inbox.offers_declined() == 1, "the decline is counted"); + check(inbox.offers_accepted() == 0, "nothing was accepted"); + + std::vector px; + uint32_t id = 0, w = 0, h = 0; + uint64_t ns = 0; + check(!inbox.take(px, id, w, h, ns), "take on an empty inbox is false"); + } + + // ---- the ordinary round trip ---------------------------------------- + { + SubjectFrameInbox inbox; + inbox.request(7); + check(inbox.wanted() == 7, "the request is visible to the producer"); + + // Everyone else is declined -- this is the case that runs hundreds of + // times a second. + check(!inbox.offer(8, frame_b.data(), frame_b.size(), 64, 64, 2), + "a frame from a participant we did not ask for is declined"); + check(!inbox.offer(0, frame_b.data(), frame_b.size(), 64, 64, 2), + "participant id 0 is never accepted"); + + check(inbox.offer(7, frame_a.data(), frame_a.size(), 64, 64, 555), + "the requested participant's frame is accepted"); + check(inbox.wanted() == 0, + "an accepted offer clears the request, so the next frame from " + "the same participant is declined too"); + check(!inbox.offer(7, frame_b.data(), frame_b.size(), 64, 64, 556), + "a second frame for a satisfied request is declined"); + + std::vector px; + uint32_t id = 0, w = 0, h = 0; + uint64_t ns = 0; + check(inbox.take(px, id, w, h, ns), "the worker takes the frame"); + check(id == 7, "the participant id comes back with the pixels"); + check(w == 64 && h == 64, "the dimensions come back with the pixels"); + check(ns == 555, "the source timestamp comes back with the pixels"); + check(px.size() == frame_a.size(), "the whole frame was copied"); + check(!px.empty() && px[0] == 0x11 && px.back() == 0x11, + "the pixels are the FIRST offered frame, not a later one"); + + check(!inbox.take(px, id, w, h, ns), + "taking twice yields nothing the second time"); + } + + // ---- cancellation ---------------------------------------------------- + // The worker cancels a request when the participant never sends a frame, + // so a camera-off panelist cannot wedge the schedule forever. + { + SubjectFrameInbox inbox; + inbox.request(7); + inbox.cancel(); + check(inbox.wanted() == 0, "cancel clears the request"); + check(!inbox.offer(7, frame_a.data(), frame_a.size(), 64, 64, 1), + "a cancelled request declines the frame it was waiting for"); + } + + // Re-requesting must discard whatever an earlier request left behind, or + // the worker could detect participant A's pixels and record the result + // against participant B. + { + SubjectFrameInbox inbox; + inbox.request(7); + check(inbox.offer(7, frame_a.data(), frame_a.size(), 64, 64, 1), + "first request is satisfied"); + inbox.request(9); // worker moved on without taking + std::vector px; + uint32_t id = 0, w = 0, h = 0; + uint64_t ns = 0; + check(!inbox.take(px, id, w, h, ns), + "a new request discards an untaken frame from the old one"); + } + + // ---- refusals -------------------------------------------------------- + { + SubjectFrameInbox inbox; + inbox.request(7); + check(!inbox.offer(7, nullptr, 0, 64, 64, 1), "a null frame is refused"); + check(inbox.wanted() == 7, + "a refused offer leaves the request standing, so the next real " + "frame still satisfies it"); + check(!inbox.offer(7, frame_a.data(), 10, 64, 64, 1), + "a buffer shorter than w*h*3/2 is refused rather than copied"); + check(!inbox.offer(7, frame_a.data(), frame_a.size(), 0, 64, 1), + "a zero dimension is refused"); + check(inbox.wanted() == 7, "still standing after every refusal"); + } + + // ---- concurrency ----------------------------------------------------- + // The real shape: one producer hammering offer() the way the IPC reader + // thread does, one consumer requesting and taking the way the worker does. + // Under TSan or a debug CRT this is where a torn buffer or a missing lock + // shows up; here it asserts that the accounting stays exact and nothing + // deadlocks. + { + SubjectFrameInbox inbox; + std::atomic stop{false}; + std::atomic accepted{0}; + + std::thread producer([&] { + const std::vector f = fake_i420(64, 64, 0x33); + uint64_t ts = 0; + while (!stop.load(std::memory_order_relaxed)) { + for (uint32_t id = 1; id <= 20; ++id) { + if (inbox.offer(id, f.data(), f.size(), 64, 64, ++ts)) + accepted.fetch_add(1, std::memory_order_relaxed); + } + } + }); + + int taken = 0; + for (int round = 0; round < 200; ++round) { + const uint32_t want = static_cast((round % 20) + 1); + inbox.request(want); + std::vector px; + uint32_t id = 0, w = 0, h = 0; + uint64_t ns = 0; + for (int spin = 0; spin < 100000; ++spin) { + if (inbox.take(px, id, w, h, ns)) { + ++taken; + check(id == want, + "a taken frame is always the one that was requested"); + check(px.size() == static_cast(64) * 64 * 3 / 2, + "a taken frame is never torn or short"); + break; + } + std::this_thread::yield(); + } + inbox.cancel(); + } + + stop.store(true, std::memory_order_relaxed); + producer.join(); + + check(taken > 100, + "the worker gets its frames under a hammering producer"); + check(inbox.offers_accepted() == accepted.load(), + "accepted offers are counted exactly once"); + std::cerr << "concurrent round trip: " << taken << " of 200 taken, " + << inbox.offers_declined() << " offers declined\n"; + } + + if (g_failures) { + std::cerr << g_failures << " check(s) failed\n"; + return 1; + } + std::cerr << "subject-frame-inbox OK\n"; + return 0; +} +``` + +- [ ] **Step 2: Run the test to verify it fails** + +Run: `cmake --build build --config Release --target CoreVideoSubjectInboxTest` +Expected: FAIL — target does not exist; once registered, `Cannot open include file: 'subject-frame-inbox.h'`. + +- [ ] **Step 3: Write the header** + +`src/subject-frame-inbox.h`: + +```cpp +#pragma once + +// The one-slot handoff between the engine-IPC reader thread and the subject +// detector's worker thread. +// +// Extracted so the whole handshake can be tested with two plain std::threads +// and no engine, the same treatment media-event-queue.h gets, and for a +// related reason: getting this wrong does not break the detector, it breaks +// EVERY source in the plugin. +// +// THE CONSTRAINT THIS FILE EXISTS FOR. The frame callback +// (tile_feed_on_frame, src/zoom-supersource.cpp) runs on the shared engine-IPC +// reader thread that dispatches frames for every source in the plugin. +// Anything slow there head-of-line-blocks all of them -- that is the +// 2026-08-17 incident recorded on src/media-event-queue.h, where inline media +// work on that thread starved audio events by up to a second and cost ~92% of +// the audio on every source. So detection cannot run there, and neither can +// anything that waits. +// +// WHY ONE SLOT AND NOT A MAP. The schedule (src/subject-schedule.h) issues +// exactly one detection at a time, so at most one frame can ever be in flight. +// A map keyed by participant would need a mutex on the producer's fast path to +// find the entry -- a shared lock taken hundreds of times a second by the +// thread that must never wait. One slot needs no lookup at all: the producer's +// fast path is a single atomic load that fails for every participant except +// the one currently wanted, which is the overwhelming majority of calls. +// +// WHY THE PRODUCER STILL TAKES A MUTEX ON THE ACCEPTED PATH. Copying the frame +// is not atomic, and the worker must not read a half-written buffer. That lock +// is taken at most once per schedule tick (ten times a second), it is +// uncontended in practice because the worker holds it only for a swap, and the +// copy under it is one 360p frame -- ~150 KB, tens of microseconds. That is +// the entire cost this design imposes on the shared reader thread, and it is +// bounded by the TICK, not by the frame rate or the number of sources. +// +// The claim is done with a compare-exchange rather than a plain store so that +// two producers (a second video source offering the same participant) cannot +// both decide they won and both copy. + +#include +#include +#include +#include +#include + +class SubjectFrameInbox { +public: + // Worker side: ask for the next frame from `participant_id`. Discards any + // frame an earlier request left untaken -- keeping it would let the worker + // detect one participant's pixels and record the result against another. + void request(uint32_t participant_id) + { + { + std::lock_guard lk(m_mtx); + m_ready = false; + } + m_wanted.store(participant_id, std::memory_order_release); + } + + // Worker side: stop waiting. Used when a requested participant does not + // send a frame within a few ticks -- a camera-off panelist must not wedge + // the schedule. + void cancel() { m_wanted.store(0, std::memory_order_release); } + + uint32_t wanted() const { return m_wanted.load(std::memory_order_acquire); } + + // Producer side, on the engine-IPC reader thread. + // + // The fast path -- the one that runs for every frame of every participant + // we are not currently waiting on -- is one atomic load and a return. No + // lock, no copy, no allocation. + // + // Returns true only when the pixels were copied. A refusal (null, short + // buffer, nonsense dimensions) leaves the request STANDING, so the next + // good frame from that participant still satisfies it. + bool offer(uint32_t participant_id, const uint8_t *i420, size_t len, + uint32_t width, uint32_t height, uint64_t source_ns) + { + if (participant_id == 0) { + m_declined.fetch_add(1, std::memory_order_relaxed); + return false; + } + if (m_wanted.load(std::memory_order_acquire) != participant_id) { + m_declined.fetch_add(1, std::memory_order_relaxed); + return false; + } + if (!i420 || width < 2 || height < 2) return false; + const size_t need = static_cast(width) * height * 3 / 2; + if (len < need) return false; + + // Claim before copying: two producers offering the same participant + // must not both copy into the slot. + uint32_t expected = participant_id; + if (!m_wanted.compare_exchange_strong(expected, 0u, + std::memory_order_acq_rel, + std::memory_order_acquire)) { + m_declined.fetch_add(1, std::memory_order_relaxed); + return false; + } + + { + std::lock_guard lk(m_mtx); + m_pixels.assign(i420, i420 + need); + m_participant_id = participant_id; + m_width = width; + m_height = height; + m_source_ns = source_ns; + m_ready = true; + } + m_accepted.fetch_add(1, std::memory_order_relaxed); + return true; + } + + // Worker side: take the delivered frame, if there is one. Swaps rather + // than copies, handing the caller's previous buffer back to be refilled, + // so neither side allocates after warm-up -- the same trick + // tile_take_snapshot uses. + bool take(std::vector &pixels, uint32_t &participant_id, + uint32_t &width, uint32_t &height, uint64_t &source_ns) + { + std::lock_guard lk(m_mtx); + if (!m_ready) return false; + pixels.swap(m_pixels); + participant_id = m_participant_id; + width = m_width; + height = m_height; + source_ns = m_source_ns; + m_ready = false; + return true; + } + + uint64_t offers_accepted() const + { + return m_accepted.load(std::memory_order_relaxed); + } + uint64_t offers_declined() const + { + return m_declined.load(std::memory_order_relaxed); + } + +private: + std::atomic m_wanted{0}; + std::atomic m_accepted{0}; + std::atomic m_declined{0}; + + mutable std::mutex m_mtx; + std::vector m_pixels; + uint32_t m_participant_id = 0; + uint32_t m_width = 0; + uint32_t m_height = 0; + uint64_t m_source_ns = 0; + bool m_ready = false; +}; +``` + +- [ ] **Step 4: Register the test** + +In `CMakeLists.txt`, inside `if(BUILD_TESTING)`: + +```cmake + # The one-slot handoff from the engine-IPC reader thread to the detector + # worker. Exercised with concurrent threads, like CoreVideoTileSlot: the + # assertion that matters is that an UNREQUESTED frame costs one atomic load + # and nothing else, because that path runs hundreds of times a second on + # the thread that serves every source in the plugin. + add_executable(CoreVideoSubjectInboxTest + tests/subject-inbox-test.cpp + ) + target_include_directories(CoreVideoSubjectInboxTest PRIVATE + "${CMAKE_CURRENT_SOURCE_DIR}/src" + ) + find_package(Threads REQUIRED) + target_link_libraries(CoreVideoSubjectInboxTest PRIVATE Threads::Threads) + add_test(NAME CoreVideoSubjectInbox + COMMAND CoreVideoSubjectInboxTest) +``` + +(If `find_package(Threads REQUIRED)` is already called earlier in the file — check with `grep -n "find_package(Threads" CMakeLists.txt` — drop that line and keep only the `target_link_libraries`.) + +- [ ] **Step 5: Run the test to verify it passes** + +```bash +cmake --build build --config Release --parallel 8 +ctest -C Release -R CoreVideoSubjectInbox --output-on-failure +``` + +Expected: PASS, with a line like `concurrent round trip: 200 of 200 taken, N offers declined` and `subject-frame-inbox OK`. + +- [ ] **Step 6: Commit** + +```bash +git add src/subject-frame-inbox.h tests/subject-inbox-test.cpp CMakeLists.txt +git commit -m "feat: add the single-slot frame inbox that keeps detection off the IPC reader thread" +``` + +--- + +### Task 8: The detector worker thread, the public API, and the plugin tap + +**Files:** +- Create: `src/subject-detector-engine.h` +- Create: `src/subject-detector-engine.cpp` +- Create: `tests/subject-worker-test.cpp` +- Modify: `CMakeLists.txt` (plugin source, test registration) +- Modify: `src/zoom-supersource.cpp` (the frame tap in `tile_feed_on_frame`, and roster/speaker pushes) +- Modify: `CLAUDE.md` + +**Interfaces:** +- Consumes: `SubjectFrame` and `ISubjectDetector` / `make_facedetect_cnn_detector` (Task 4), `i420_to_bgr_downscale` / `kSubjectLongEdge` / `BgrImage` (Task 3), `SubjectSchedule` / `SubjectScheduleConfig` (Task 5), `SubjectSmoother` / `SubjectSmoothingConfig` (Task 6), `SubjectFrameInbox` (Task 7). +- Produces — **this is the API Subsystems 3a and 3b consume, and the end of this plan's scope:** + +```cpp +class SubjectDetectorEngine { +public: + static SubjectDetectorEngine &instance(); + void start(); + void stop(); + bool running() const; + void set_monitored(const std::vector &participant_ids); + void set_speaker_poll(std::function poll); + bool offer_frame(uint32_t participant_id, const uint8_t *i420, size_t len, + uint32_t width, uint32_t height, uint64_t source_ns); + SubjectFrame subject(uint32_t participant_id) const; + uint64_t detections_completed() const; + uint64_t requests_timed_out() const; + // Test seams; both must be called before start(). + void set_detector_factory(std::function()> factory); + void set_schedule_config(SubjectScheduleConfig cfg); +}; +``` + +- [ ] **Step 1: Write the failing test** + +`tests/subject-worker-test.cpp`: + +```cpp +// tests/subject-worker-test.cpp +// The detector worker: one thread, driven by the schedule, fed by the inbox, +// with results landing in the per-participant smoothers. +// +// It runs against a FAKE ISubjectDetector, which is the entire reason the +// narrow interface exists. That buys three things no test with the real CNN +// could have: a deterministic result (so the assertions are about the worker, +// not about whether YuNet found a synthetic face), a millisecond-scale test +// (the real detector would need seconds of wall time to produce enough ticks), +// and -- the important one -- a THREAD-IDENTITY assertion. The fake records +// which thread called it, so the test can prove detection never runs on the +// thread that offered the frame. That is the spec's hardest constraint and +// nothing else in the suite would notice it being violated: the plugin would +// simply get slower for every source at once, which is the 2026-08-17 +// signature. + +#include "subject-detector-engine.h" + +#include +#include +#include +#include +#include +#include +#include + +static int g_failures = 0; + +static void check(bool cond, const char *what) +{ + if (!cond) { + std::cerr << "FAIL: " << what << "\n"; + ++g_failures; + } +} + +// Shared observations, because the engine owns the detector object itself. +static std::atomic g_detect_calls{0}; +static std::atomic g_saw_foreign_thread{false}; +static std::thread::id g_detector_thread{}; +static std::atomic g_detector_thread_set{false}; +static std::atomic g_last_width{0}; +static std::atomic g_last_height{0}; +static std::atomic g_last_centre_b{0}; + +class FakeDetector : public ISubjectDetector { +public: + SubjectFrame detect(const uint8_t *bgr, int width, int height, + uint64_t source_ns) override + { + // Every call must come from the SAME thread, and it must not be a + // thread the test offered a frame on. + if (!g_detector_thread_set.exchange(true)) { + g_detector_thread = std::this_thread::get_id(); + } else if (std::this_thread::get_id() != g_detector_thread) { + g_saw_foreign_thread.store(true); + } + g_detect_calls.fetch_add(1, std::memory_order_relaxed); + + g_last_width.store(width); + g_last_height.store(height); + if (bgr && width > 0 && height > 0) { + const size_t centre = + (static_cast(height / 2) * width + width / 2) * 3; + g_last_centre_b.store(bgr[centre]); + } + + SubjectFrame f{}; + f.found = true; + f.box_x = 0.25f; + f.box_y = 0.20f; + f.box_w = 0.30f; + f.box_h = 0.45f; + f.eye_l_x = 0.32f; + f.eye_l_y = 0.34f; + f.eye_r_x = 0.46f; + f.eye_r_y = 0.34f; + f.confidence = 0.95f; + f.detected_ns = source_ns; + return f; + } +}; + +// A flat mid-grey I420 frame. Y=128, U=V=128 converts to BGR (128,128,128), +// which is what the centre-pixel assertion below checks -- proving the worker +// really ran the conversion rather than handing raw I420 to the detector. +static std::vector grey_i420(uint32_t w, uint32_t h) +{ + const size_t y_len = static_cast(w) * h; + return std::vector(y_len + y_len / 2, 128); +} + +int main() +{ + SubjectDetectorEngine &engine = SubjectDetectorEngine::instance(); + + engine.set_detector_factory( + [] { return std::unique_ptr(new FakeDetector()); }); + + // A fast tick so the test finishes in milliseconds instead of seconds. + SubjectScheduleConfig cfg; + cfg.tick_ns = 2000000ull; // 2 ms + cfg.speaker_every = 3; + engine.set_schedule_config(cfg); + + engine.set_monitored({101, 102, 103}); + engine.set_speaker_poll([] { return 101u; }); + engine.start(); + check(engine.running(), "the engine reports running after start()"); + + // ---- feed it, from THIS thread -------------------------------------- + const std::vector frame = grey_i420(640, 360); + const auto deadline = std::chrono::steady_clock::now() + + std::chrono::seconds(5); + uint64_t offered = 0; + while (engine.detections_completed() < 30 && + std::chrono::steady_clock::now() < deadline) { + for (uint32_t id = 101; id <= 103; ++id) { + if (engine.offer_frame(id, frame.data(), frame.size(), 640, 360, + 1000 + offered)) + ++offered; + } + std::this_thread::sleep_for(std::chrono::milliseconds(1)); + } + + check(engine.detections_completed() >= 30, + "the worker completes detections when frames are offered"); + + // ---- the constraint the spec calls non-negotiable -------------------- + check(!g_saw_foreign_thread.load(), + "every detection ran on ONE thread"); + check(g_detector_thread_set.load() && + g_detector_thread != std::this_thread::get_id(), + "detection NEVER runs on the thread that offered the frame -- that " + "thread is the shared engine-IPC reader in production and blocking " + "it stalls every source in the plugin"); + + // ---- the conversion really happened --------------------------------- + check(g_last_width.load() == 320 && g_last_height.load() == 180, + "the detector is handed the DOWNSCALED image (320x180), not the " + "640x360 source"); + check(g_last_centre_b.load() >= 126 && g_last_centre_b.load() <= 130, + "the detector is handed BGR (mid-grey converts to ~128), not raw " + "I420 planes"); + + // ---- results reach the consumers ------------------------------------ + { + const SubjectFrame f = engine.subject(101); + check(f.found, "the active speaker has a subject"); + check(f.box_x > 0.2f && f.box_x < 0.3f, + "the smoothed box matches what the detector returned"); + check(f.confidence > 0.9f, "confidence is carried through"); + + const SubjectFrame other = engine.subject(102); + check(other.found, + "a non-speaking monitored participant also gets detected"); + + const SubjectFrame none = engine.subject(999); + check(!none.found, + "an unmonitored participant reports a clean not-found rather " + "than anything stale or invented"); + } + + // ---- an unmonitored participant's frames are refused ----------------- + { + const uint64_t before = engine.detections_completed(); + for (int i = 0; i < 50; ++i) + engine.offer_frame(777, frame.data(), frame.size(), 640, 360, 1); + std::this_thread::sleep_for(std::chrono::milliseconds(50)); + check(engine.subject(777).found == false, + "offering frames for someone not monitored never produces a " + "subject for them"); + (void)before; + } + + // ---- a silent participant must not wedge the schedule --------------- + // 104 is monitored but never sends a frame. The worker must time its + // request out and carry on, or one camera-off panelist stops detection for + // everybody. + { + engine.set_monitored({101, 104}); + const uint64_t timeouts_before = engine.requests_timed_out(); + const uint64_t detections_before = engine.detections_completed(); + const auto t_end = std::chrono::steady_clock::now() + + std::chrono::seconds(3); + while (std::chrono::steady_clock::now() < t_end && + engine.detections_completed() < detections_before + 20) { + engine.offer_frame(101, frame.data(), frame.size(), 640, 360, 9); + std::this_thread::sleep_for(std::chrono::milliseconds(1)); + } + check(engine.detections_completed() >= detections_before + 20, + "a participant who never sends a frame does not stop detection " + "for the ones who do"); + check(engine.requests_timed_out() > timeouts_before, + "the silent participant's requests are timed out, not left " + "outstanding forever"); + check(!engine.subject(104).found, + "a participant who never sends a frame has no subject"); + } + + // ---- removing a participant clears their state ----------------------- + { + check(engine.subject(101).found, "101 has a subject before removal"); + engine.set_monitored({102}); + std::this_thread::sleep_for(std::chrono::milliseconds(50)); + check(!engine.subject(101).found, + "a participant dropped from the monitored set stops reporting a " + "subject -- a rejoining person must not inherit the old one's " + "framing"); + } + + // ---- shutdown -------------------------------------------------------- + engine.stop(); + check(!engine.running(), "the engine reports stopped after stop()"); + { + const uint64_t after_stop = engine.detections_completed(); + for (int i = 0; i < 20; ++i) + engine.offer_frame(102, frame.data(), frame.size(), 640, 360, 1); + std::this_thread::sleep_for(std::chrono::milliseconds(50)); + check(engine.detections_completed() == after_stop, + "no detection happens after stop()"); + } + engine.stop(); // idempotent: a second stop must not hang or crash + engine.start(); + engine.stop(); // restartable + + if (g_failures) { + std::cerr << g_failures << " check(s) failed\n"; + return 1; + } + std::cerr << "subject-worker OK (" << g_detect_calls.load() + << " detections)\n"; + return 0; +} +``` + +- [ ] **Step 2: Run the test to verify it fails** + +Run: `cmake --build build --config Release --target CoreVideoSubjectWorkerTest` +Expected: FAIL — target does not exist; once registered, `Cannot open include file: 'subject-detector-engine.h'`. + +- [ ] **Step 3: Write the engine header** + +`src/subject-detector-engine.h`: + +```cpp +#pragma once + +// The subject detector as a running thing: one worker thread, the schedule +// that decides who it looks at, the inbox that feeds it, and one smoother per +// participant. +// +// THIS HEADER IS THE END OF SUBSYSTEM 2 AND THE START OF SUBSYSTEMS 3a AND 3b. +// Tiles auto-framing and the return-feed overlay both consume `subject()` and +// nothing else. Everything below this line -- the CNN, the colour convert, the +// round robin, the smoothing -- is private to that promise. +// +// THREADING CONTRACT, which is the whole point of the class: +// +// offer_frame() is called ON THE ENGINE-IPC READER THREAD, the shared +// thread that dispatches frames for every source in the +// plugin. It is cheap by construction: one atomic load for +// every participant we are not currently waiting on, and at +// most one ~150 KB copy per schedule tick for the one we are. +// It NEVER runs a detection, never allocates on the declined +// path, and never waits on the worker. +// +// detect() runs ONLY on this class's own worker thread. +// +// subject() is safe from any thread (a mutex and a map lookup) and is +// expected to be called from the OBS graphics thread once per +// frame per tile. +// +// set_monitored() / set_speaker_poll() are safe from any thread. +// +// WHY A SINGLETON. It mirrors ZoomEngineClient: there is one Zoom session, one +// engine process, and one detector budget for the whole plugin. Two instances +// would each run a worker and each pay the schedule's cost, which is exactly +// the O(n) growth the schedule exists to prevent. + +#include "subject-frame.h" +#include "subject-detector.h" +#include "subject-schedule.h" +#include "subject-smoothing.h" + +#include +#include +#include +#include +#include + +class SubjectDetectorEngine { +public: + static SubjectDetectorEngine &instance(); + + SubjectDetectorEngine(const SubjectDetectorEngine &) = delete; + SubjectDetectorEngine &operator=(const SubjectDetectorEngine &) = delete; + + // Idempotent. Spawns the single worker thread and builds the detector from + // the current factory. + void start(); + // Idempotent, and safe to call from a different thread than start(). + // Joins the worker before returning. + void stop(); + bool running() const; + + // The participants worth detecting -- in practice the ones actually being + // SHOWN, not the whole meeting. Participants dropped from this set have + // their smoother state discarded, so a rejoining person never inherits the + // previous occupant's framing. + void set_monitored(const std::vector &participant_ids); + + // How the worker learns who is on air. SpeakerDirector is POLL-ONLY -- + // there is no observer list -- so the worker polls this once per tick. + // The plugin passes a closure over + // ZoomEngineClient::instance().active_speaker_id(): the DIRECTED id, after + // hold and dwell, never the raw one. + // + // Injected rather than called directly so this whole class stays free of + // libobs, Qt and the engine client, and can therefore be tested off-rig. + void set_speaker_poll(std::function poll); + + // Producer side. Returns true only when the frame was taken. `i420` is + // tightly packed I420 (Y, then U, then V), BT.709 full range. + bool offer_frame(uint32_t participant_id, const uint8_t *i420, size_t len, + uint32_t width, uint32_t height, uint64_t source_ns); + + // Consumer side. The smoothed result, or a default (not-found) SubjectFrame + // for anyone unknown. + SubjectFrame subject(uint32_t participant_id) const; + + uint64_t detections_completed() const; + uint64_t requests_timed_out() const; + + // Test seams. Both must be called before start(); calling them on a + // running engine is ignored. + void set_detector_factory( + std::function()> factory); + void set_schedule_config(SubjectScheduleConfig cfg); + +private: + SubjectDetectorEngine(); + ~SubjectDetectorEngine(); + struct Impl; + std::unique_ptr m_impl; +}; +``` + +- [ ] **Step 4: Write the engine implementation** + +`src/subject-detector-engine.cpp`: + +```cpp +#include "subject-detector-engine.h" + +#include "i420-bgr-downscale.h" +#include "subject-detector-fd-record.h" +#include "subject-frame-inbox.h" + +#include +#include +#include +#include +#include +#include +#include + +namespace { + +// The worker's own clock. steady_clock rather than os_gettime_ns() so this +// file needs no libobs -- which is what lets tests/subject-worker-test.cpp +// link it without OBS headers. Only elapsed time matters here; nothing +// compares this against a media timestamp. +uint64_t worker_now_ns() +{ + return static_cast( + std::chrono::duration_cast( + std::chrono::steady_clock::now().time_since_epoch()) + .count()); +} + +// How long a request waits for a frame before it is abandoned. Three ticks: +// long enough that an ordinary 30 fps source always answers, short enough that +// a camera-off participant costs the schedule three slots and not the show. +constexpr int kRequestTimeoutTicks = 3; + +// The worker's poll interval. Well under the schedule tick, so the tick is the +// thing that paces detection and this is only how promptly a delivered frame +// is picked up. +constexpr int kWorkerPollMs = 2; + +} // namespace + +struct SubjectDetectorEngine::Impl { + // --- configuration, read by the worker at start() --- + std::function()> factory; + SubjectScheduleConfig sched_cfg{}; + SubjectSmoothingConfig smooth_cfg{}; + + // --- the producer/consumer seam --- + SubjectFrameInbox inbox; + + // --- worker lifetime --- + std::mutex life_mtx; // serializes start()/stop() + std::thread worker; + std::atomic running{false}; + std::atomic stop_flag{false}; + + // --- inputs the worker reads, written from other threads --- + std::mutex input_mtx; + std::vector monitored; + bool monitored_dirty = false; + std::function speaker_poll; + + // --- outputs --- + mutable std::mutex out_mtx; + std::unordered_map out; + + std::atomic detections{0}; + std::atomic timeouts{0}; + + void run(); +}; + +SubjectDetectorEngine::SubjectDetectorEngine() : m_impl(new Impl()) {} + +SubjectDetectorEngine::~SubjectDetectorEngine() +{ + stop(); +} + +SubjectDetectorEngine &SubjectDetectorEngine::instance() +{ + static SubjectDetectorEngine s_instance; + return s_instance; +} + +void SubjectDetectorEngine::set_detector_factory( + std::function()> factory) +{ + std::lock_guard lk(m_impl->life_mtx); + if (m_impl->running.load()) return; + m_impl->factory = std::move(factory); +} + +void SubjectDetectorEngine::set_schedule_config(SubjectScheduleConfig cfg) +{ + std::lock_guard lk(m_impl->life_mtx); + if (m_impl->running.load()) return; + m_impl->sched_cfg = cfg; +} + +void SubjectDetectorEngine::set_monitored( + const std::vector &participant_ids) +{ + std::lock_guard lk(m_impl->input_mtx); + m_impl->monitored = participant_ids; + m_impl->monitored_dirty = true; +} + +void SubjectDetectorEngine::set_speaker_poll(std::function poll) +{ + std::lock_guard lk(m_impl->input_mtx); + m_impl->speaker_poll = std::move(poll); +} + +void SubjectDetectorEngine::start() +{ + std::lock_guard lk(m_impl->life_mtx); + if (m_impl->running.load()) return; + m_impl->stop_flag.store(false); + m_impl->running.store(true); + Impl *impl = m_impl.get(); + m_impl->worker = std::thread([impl] { impl->run(); }); +} + +void SubjectDetectorEngine::stop() +{ + std::lock_guard lk(m_impl->life_mtx); + if (!m_impl->running.load()) return; + m_impl->stop_flag.store(true); + m_impl->running.store(false); + if (m_impl->worker.joinable()) m_impl->worker.join(); + // Nothing is in flight once the worker is joined, so drop the outstanding + // request rather than leaving a wanted id that a still-running producer + // would keep copying frames for. + m_impl->inbox.cancel(); +} + +bool SubjectDetectorEngine::running() const +{ + return m_impl->running.load(); +} + +bool SubjectDetectorEngine::offer_frame(uint32_t participant_id, + const uint8_t *i420, size_t len, + uint32_t width, uint32_t height, + uint64_t source_ns) +{ + // No running check here on purpose: stop() cancels the inbox, so with no + // worker there is never a wanted id and offer() declines on its one atomic + // load. Adding a second atomic read would only make the hot path -- the + // one on the shared engine-IPC reader thread -- more expensive. + return m_impl->inbox.offer(participant_id, i420, len, width, height, + source_ns); +} + +SubjectFrame SubjectDetectorEngine::subject(uint32_t participant_id) const +{ + std::lock_guard lk(m_impl->out_mtx); + auto it = m_impl->out.find(participant_id); + if (it == m_impl->out.end()) return SubjectFrame{}; + return it->second; +} + +uint64_t SubjectDetectorEngine::detections_completed() const +{ + return m_impl->detections.load(std::memory_order_relaxed); +} + +uint64_t SubjectDetectorEngine::requests_timed_out() const +{ + return m_impl->timeouts.load(std::memory_order_relaxed); +} + +void SubjectDetectorEngine::Impl::run() +{ + // Everything below is worker-thread-local. The schedule, the smoothers and + // the reusable buffers never leave this function's frame, which is why + // none of them needs a lock. + // + // `factory` is read without a lock deliberately: set_detector_factory() + // refuses while running, and start() creating this thread is the + // happens-before edge that publishes it. + std::unique_ptr detector = + factory ? factory() + : make_facedetect_cnn_detector(kSubjectMinConfidencePct); + + SubjectSchedule schedule(sched_cfg); + std::unordered_map smoothers; + std::unordered_set monitored_now; + + BgrImage bgr; // reused: allocates once per resolution + std::vector pixels; // reused: swapped with the inbox's buffer + + uint32_t outstanding = 0; + uint64_t outstanding_since = 0; + + while (!stop_flag.load(std::memory_order_relaxed)) { + // --- pick up roster and speaker changes --- + uint32_t speaker = 0; + { + std::lock_guard lk(input_mtx); + if (monitored_dirty) { + monitored_dirty = false; + schedule.set_roster(monitored); + monitored_now.clear(); + for (uint32_t id : monitored) monitored_now.insert(id); + + // Drop everyone who left. A rejoining participant must not + // inherit the previous occupant's framing, and a stale entry + // would otherwise answer subject() forever. + for (auto it = smoothers.begin(); it != smoothers.end();) { + if (monitored_now.count(it->first) == 0) + it = smoothers.erase(it); + else + ++it; + } + std::lock_guard ok(out_mtx); + for (auto it = out.begin(); it != out.end();) { + if (monitored_now.count(it->first) == 0) + it = out.erase(it); + else + ++it; + } + if (outstanding != 0 && monitored_now.count(outstanding) == 0) { + inbox.cancel(); + outstanding = 0; + } + } + if (speaker_poll) speaker = speaker_poll(); + } + schedule.set_active_speaker(speaker); + + const uint64_t now = worker_now_ns(); + + // --- a delivered frame, if any --- + uint32_t id = 0, w = 0, h = 0; + uint64_t source_ns = 0; + if (inbox.take(pixels, id, w, h, source_ns)) { + outstanding = 0; + if (monitored_now.count(id) != 0 && + i420_to_bgr_downscale(pixels.data(), pixels.size(), + static_cast(w), static_cast(h), + kSubjectLongEdge, bgr)) { + const SubjectFrame raw = detector->detect( + bgr.pixels.data(), bgr.width, bgr.height, source_ns); + + auto it = smoothers.find(id); + if (it == smoothers.end()) + it = smoothers.emplace(id, SubjectSmoother(smooth_cfg)).first; + it->second.update(raw, worker_now_ns()); + + { + std::lock_guard ok(out_mtx); + out[id] = it->second.output(); + } + detections.fetch_add(1, std::memory_order_relaxed); + } + } else if (outstanding != 0) { + // --- time a silent participant's request out --- + // Without this, one camera-off panelist stops detection for + // everybody: the schedule would keep issuing while nothing ever + // satisfied the standing request. + const uint64_t limit = + sched_cfg.tick_ns * static_cast(kRequestTimeoutTicks); + if (now > outstanding_since && now - outstanding_since > limit) { + // Feed the smoother a MISS rather than just dropping it, so the + // dropout-hold clock actually runs for someone who has stopped + // sending video. Otherwise a participant who switches their + // camera off would keep reporting their last framing forever. + auto it = smoothers.find(outstanding); + if (it != smoothers.end()) { + it->second.update(SubjectFrame{}, now); + std::lock_guard ok(out_mtx); + out[outstanding] = it->second.output(); + } + inbox.cancel(); + outstanding = 0; + timeouts.fetch_add(1, std::memory_order_relaxed); + } + } + + // --- issue the next request, if the schedule says it is time --- + if (outstanding == 0) { + const uint32_t next = schedule.next(now); + if (next != 0) { + inbox.request(next); + outstanding = next; + outstanding_since = now; + } + } + + std::this_thread::sleep_for(std::chrono::milliseconds(kWorkerPollMs)); + } +} +``` + +- [ ] **Step 5: Register the source and the test** + +In `CMakeLists.txt`, inside `add_library(obs-zoom-plugin MODULE ...)`, next to the Task 4 entry: + +```cmake + src/subject-detector-engine.cpp +``` + +Inside `if(BUILD_TESTING)`: + +```cmake + # The detector worker, driven against a FAKE ISubjectDetector -- which is + # what the narrow interface bought us. The load-bearing assertion is the + # thread-identity one: detection must never run on the thread that offered + # the frame, because in production that is the shared engine-IPC reader + # that dispatches for every source in the plugin, and nothing else in this + # suite would notice it being violated (the symptom is the whole plugin + # getting slower at once -- the 2026-08-17 signature). Links the real + # engine .cpp but NOT libfacedetection: the fake factory means the CNN is + # never constructed, so the weights blob stays out of this link. + add_executable(CoreVideoSubjectWorkerTest + tests/subject-worker-test.cpp + src/subject-detector-engine.cpp + ) + target_include_directories(CoreVideoSubjectWorkerTest PRIVATE + "${CMAKE_CURRENT_SOURCE_DIR}/src" + ) + target_link_libraries(CoreVideoSubjectWorkerTest PRIVATE + Threads::Threads libfacedetection) + add_test(NAME CoreVideoSubjectWorker + COMMAND CoreVideoSubjectWorkerTest) +``` + +(`libfacedetection` is linked because `subject-detector-engine.cpp` names +`make_facedetect_cnn_detector` in its default branch, which pulls in +`subject-detector-fd.cpp`'s symbol. The fake factory means it is never called, +so the test still measures only our own code — but the link needs to resolve. +Add `src/subject-detector-fd.cpp` to the `add_executable` list above if the +link reports the symbol unresolved.) + +- [ ] **Step 6: Run the test to verify it passes** + +```bash +cmake --build build --config Release --parallel 8 +ctest -C Release -R CoreVideoSubjectWorker --output-on-failure +``` + +Expected: PASS, `subject-worker OK (N detections)`. + +- [ ] **Step 7: Tap the frame path in the Tiles source** + +In `src/zoom-supersource.cpp`, add the include near the other project includes at the top of the file: + +```cpp +#include "subject-detector-engine.h" +``` + +In `tile_feed_on_frame`, immediately after `feed->has_frame = true;` (currently the last line of the function, around line 443), add: + +```cpp + // Hand the detector one frame, and ONLY when it has asked for one. + // + // This runs on the engine-IPC reader thread that dispatches frames for + // every source in the plugin, so the cost has to be bounded by the + // DETECTOR'S schedule and not by the frame rate: offer_frame() is a single + // atomic load that declines for every participant except the one the + // worker is currently waiting on, which at a 100 ms tick is at most ten + // acceptances per second across the whole wall. An accepted offer copies + // one frame (~150 KB at 360p) and nothing else -- no detection, no colour + // convert, no allocation beyond the copy. Detection itself happens on the + // detector's own thread; see the threading contract on + // src/subject-detector-engine.h. + // + // feed->mtx is already held here and the copy happens inside it, which is + // correct: `frame` is the buffer tile_take_snapshot swaps out from under + // us, so reading it unlocked would race the graphics thread. + SubjectDetectorEngine::instance().offer_frame( + feed->slot.participant_id(), feed->frame.data(), feed->frame.size(), + w, h, feed->frame_epoch); +``` + +- [ ] **Step 8: Tell the engine who is monitored and who is speaking** + +Still in `src/zoom-supersource.cpp`, in `tiles_video_render` (the function containing the `ctx->render_feeds = ctx->feeds;` snapshot around line 1286), immediately after that snapshot block closes, add: + +```cpp + // Keep the subject detector's monitored set equal to what the wall is + // actually showing, and poll the DIRECTED active speaker for its schedule + // boost. + // + // Pushed from here rather than the detector polling for itself because + // ZoomEngineClient::roster() deep-copies strings under a hot mutex and must + // never be called per frame -- the wall already knows its own assignment + // set, for free. The push is rate-limited to once a second because + // set_monitored() takes a lock and rebuilds the schedule's roster, and the + // set changes on operator action, not per frame. + // + // SpeakerDirector is poll-only (no observer list), so the speaker is a + // closure the worker calls once per tick rather than an event we forward. + { + static uint64_t s_last_monitored_push_ns = 0; + const uint64_t now_ns = os_gettime_ns(); + if (now_ns - s_last_monitored_push_ns > 1000000000ull) { + s_last_monitored_push_ns = now_ns; + std::vector ids; + ids.reserve(feeds.size()); + for (const TileFeedPtr &f : feeds) { + const uint32_t pid = f ? f->slot.participant_id() : 0; + if (pid != 0) ids.push_back(pid); + } + SubjectDetectorEngine &det = SubjectDetectorEngine::instance(); + det.set_monitored(ids); + if (!det.running()) { + det.set_speaker_poll([] { + return ZoomEngineClient::instance().active_speaker_id(); + }); + det.start(); + } + } + } +``` + +Then stop the worker at plugin unload. In `src/plugin-main.cpp`'s `obs_module_unload()`, before the existing engine shutdown, add: + +```cpp + // Join the detector worker before anything it might still be reading goes + // away. It holds no OBS or Zoom handles, so it can go first and its stop() + // is idempotent. + SubjectDetectorEngine::instance().stop(); +``` + +with `#include "subject-detector-engine.h"` added to that file's includes. + +- [ ] **Step 9: Build the plugin and run the whole suite** + +```bash +cmake --build build --config Release --parallel 8 +ctest -C Release --output-on-failure +``` + +Expected: N/N green, including all six new tests +(`CoreVideoFaceDetectLink`, `CoreVideoSubjectDetectorBench`, +`CoreVideoI420BgrDownscale`, `CoreVideoSubjectDetectorRecord`, +`CoreVideoSubjectSchedule`, `CoreVideoSubjectSmoothing`, +`CoreVideoSubjectInbox`, `CoreVideoSubjectWorker` — eight in total). + +- [ ] **Step 10: Record the subsystem in CLAUDE.md** + +Add to `CLAUDE.md`, in the "Invariants that have each caused a live-show defect" list: + +```markdown +- **Subject detection never runs on the engine-IPC reader thread** + (`src/subject-detector-engine.h`, tapped at `tile_feed_on_frame` in + `src/zoom-supersource.cpp`): that thread dispatches frames for EVERY source + in the plugin, and inline media work on it is the 2026-08-17 incident + recorded on `src/media-event-queue.h` (~92% audio loss on every source). The + producer's path there is one atomic load per frame plus, at most once per + schedule tick, one ~150 KB copy into `SubjectFrameInbox` + (`src/subject-frame-inbox.h`); the CNN runs on the detector's own worker + thread. The cost ceiling is set by the SCHEDULE, not the roster + (`src/subject-schedule.h`): one detection per ~100 ms cycled round-robin with + an active-speaker boost, so a 24-person panel costs what a 2-person mic check + costs. Consumers read `SubjectDetectorEngine::subject(id)`, which is already + smoothed (`src/subject-smoothing.h`: deadband, rate limit, 1.5 s dropout + hold) — so `found == false` from it means "gone for longer than the hold", + never "missed one detection". `SubjectFrame` (`src/subject-frame.h`) is a + published contract consumed by Tiles auto-framing and the return-feed + overlay; do not rename its fields. +``` + +- [ ] **Step 11: Commit** + +```bash +git add src/subject-detector-engine.h src/subject-detector-engine.cpp \ + tests/subject-worker-test.cpp src/zoom-supersource.cpp \ + src/plugin-main.cpp CMakeLists.txt CLAUDE.md +git commit -m "feat: run subject detection on a dedicated worker thread, off the IPC reader" +``` + +--- + +## What this plan does NOT deliver + +Stated so the next author does not go looking: + +- **Tiles auto-framing (Subsystem 3a).** Nothing here computes a crop rect or + touches `solve_slot_crop` at `src/zoom-supersource.cpp:1957`. The spec's open + question about tile source resolution (P360 by default vs. crisp auto-framed + tiles, and the 2026-08-17 subscription-envelope throttle) gates that plan, + not this one — detection is unaffected because we downscale to ~320 px + regardless. +- **The return scene and framing overlay (Subsystem 3b).** No advice strings, + no overlay rendering, no `corevideo_active_speaker_source` changes. +- **The loudness engine and meter (Subsystem 1).** Entirely separate; the two + engines share nothing. +- **Self-tile exclusion.** The spec is emphatic that this is real, unwritten + work and that a vcam return feed makes a feedback loop likely. It belongs to + Subsystem 3b, which is the plan that adds the return feed. Detection running + on the bot's own tile costs one schedule slot and produces a `SubjectFrame` + nobody reads. +- **Confirmation that the detector finds a real face.** Every test here is pure + CPU logic, per the repo's rule that no headless GPU harness exists and one + has been ruled against. `CoreVideoFaceDetectLink` proves the library runs and + `CoreVideoSubjectDetectorRecord` proves the arithmetic; that a real panelist + in a real 360p Zoom tile is detected is a live check, and the first consumer + plan is where it gets made. + +## Self-review + +**Spec coverage (Subsystem 2 only):** + +| Spec requirement | Task | +|---|---| +| Vendor libfacedetection, BSD-3, four upstream files + our export header | 1 | +| BSD-3 attribution notice | 1 (`docs/THIRD-PARTY-NOTICES.md`) | +| No `/openmp`; `/arch:AVX2` provably on the compile line | 1 (CMakeLists + `corevideo-avx2-assert.cpp`, proved by deliberately breaking it in Step 9) | +| Local benchmarking before finalising the tick rate | 2, with the go/no-go table and the tick-rate consequence written down before the number is known | +| Input must be BGR 3-channel, converted at the downscaled size, not greyscale | 3 | +| ~320 px long edge | 3 (`kSubjectLongEdge`) | +| `SubjectFrame` exactly as the spec defines it | 4 | +| Five landmarks (eyes used; nose and mouth corners parsed but not surfaced) | 4 — the spec's framing rules need the eyeline, which is the eye pair; the other three landmarks are in the record and can be surfaced without a contract change if a consumer needs them | +| Narrow interface so the OpenCV-DNN fallback is a contained swap | 4 (`ISubjectDetector`, one method, one factory, one .cpp includes the CNN) | +| Round-robin, O(1) in participant count, active-speaker boost via the directed id | 5 | +| Test proving per-second detections stay fixed from 2 to 20 participants | 5, first assertion | +| Detection never on the engine-IPC reader thread; one dedicated worker | 7 (the handoff) and 8 (the thread, with a thread-identity assertion) | +| Temporal smoothing: deadband, rate limit, dropout hold | 6 | +| Test that a brief dropout does not flip to "no subject" | 6, dropout-hold block | +| Test that jitter below the deadband produces zero movement | 6, deadband block (asserts exact equality, not "small") | +| Plain `int main()` + local `check()`, `CoreVideoTest` / `CoreVideo`, hand-registered | every task | +| Everything testable as pure CPU logic, no GPU harness | every task; the only test that touches the vendored CNN at all is the link smoke test and the benchmark, both CPU | + +**Placeholder scan:** no "TBD", no "add error handling", no "similar to Task N". The two fill-in-the-blank spots are deliberate and are *measurements the engineer must take*, not decisions deferred: the upstream commit SHA in Task 1 Step 1 and the benchmark table in Task 2 Step 5. Both have the exact command that produces the value. + +**Type consistency check:** `SubjectFrame` field names are identical in `src/subject-frame.h`, the smoother, the record adapter, and every test. `kSubjectMinConfidencePct` is defined once (`subject-detector-fd-record.h`) and used by the adapter and the engine's default factory. `kSubjectLongEdge` is defined once (`i420-bgr-downscale.h`) and used by the worker and the benchmark's chosen sizes. `SubjectScheduleConfig::tick_ns` is the single tick knob, referenced by Task 2's verdict table, Task 5's header, and Task 8's request timeout. `i420_to_bgr_downscale`'s signature is the same in Task 3's header, Task 3's test, and Task 8's worker loop. `ISubjectDetector::detect` has the same four parameters in the interface, the real adapter, the fake in the worker test, and the worker's call site. From 21ba3e4a7eb806e3637de2d41a1ca7acff6da97d Mon Sep 17 00:00:00 2001 From: Grok Date: Sat, 5 Sep 2026 16:27:04 -0400 Subject: [PATCH 04/15] feat(loudness): derive BS.1770-4 K-weighting coefficients from the runtime sample rate --- CMakeLists.txt | 15 +++++ src/audio-loudness.h | 122 +++++++++++++++++++++++++++++++++ tests/audio-loudness-test.cpp | 123 ++++++++++++++++++++++++++++++++++ 3 files changed, 260 insertions(+) create mode 100644 src/audio-loudness.h create mode 100644 tests/audio-loudness-test.cpp diff --git a/CMakeLists.txt b/CMakeLists.txt index 051dd3c6..21d57809 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -979,6 +979,21 @@ if(BUILD_TESTING) add_test(NAME CoreVideoAudioSilenceFade COMMAND CoreVideoAudioSilenceFadeTest) + # ITU-R BS.1770-4 loudness. The coefficients are DERIVED from the runtime + # sample rate rather than taken from the standard's 48 kHz table, because + # the engine stamps whatever rate Zoom gave it into ShmAudioHeader and + # Zoom commonly sends 32 kHz -- measured 1.3 LU of error on a 1 kHz tone + # if the 48 kHz constants are used at 32 kHz, with nothing in the reading + # to say it is wrong. See src/audio-loudness.h. + add_executable(CoreVideoAudioLoudnessTest + tests/audio-loudness-test.cpp + ) + target_include_directories(CoreVideoAudioLoudnessTest PRIVATE + "${CMAKE_CURRENT_SOURCE_DIR}/src" + ) + add_test(NAME CoreVideoAudioLoudness + COMMAND CoreVideoAudioLoudnessTest) + # Backfills silence into an ISO WAV across gaps so its own timeline # tracks real elapsed time instead of shrinking by every silent # stretch — 2026-08-21 design review ("does ZoomISO handle variable diff --git a/src/audio-loudness.h b/src/audio-loudness.h new file mode 100644 index 00000000..dad41ac0 --- /dev/null +++ b/src/audio-loudness.h @@ -0,0 +1,122 @@ +#pragma once + +// ITU-R BS.1770-4 loudness measurement, derived for the sample rate the audio +// ACTUALLY arrives at. +// +// WHY THIS FILE DERIVES INSTEAD OF QUOTING. BS.1770-4 tabulates its two +// K-weighting biquads' coefficients for 48 kHz and for no other rate. This +// plugin has no guaranteed rate: engine/src/engine-audio.cpp calls +// data->GetSampleRate() per buffer and stamps the answer into +// ShmAudioHeader::sample_rate, and Zoom commonly delivers 32 kHz. Applying +// the published 48 kHz numbers to 32 kHz audio moves both filters' corner +// frequencies by a factor of 1.5 and mis-weights every measurement: on a +// 1 kHz tone whose true value is -19.98 LUFS it reads -18.66 LUFS. That is +// 1.3 LU of error on a meter whose entire product claim is that a 6 LU +// spread between panelists is visible -- and nothing about the number looks +// wrong. So the coefficients come from the analog prototype in the standard, +// bilinear-transformed at the runtime rate. At 48 kHz the derivation +// reproduces the published table to fourteen digits, which is what +// tests/audio-loudness-test.cpp asserts. +// +// Pure by design -- no libobs, no Qt, no Zoom SDK -- so the whole measurement +// can be pinned against reference tones with no meeting, the same treatment +// audio-timeline.h and audio-silence-fade.h get, and for the same reason: +// the only symptom of a regression here is a number that is quietly wrong. + +#include +#include +#include +#include +#include + +// One biquad section, y[n] = b0.x[n] + b1.x[n-1] + b2.x[n-2] +// - a1.y[n-1] - a2.y[n-2] +// (a0 normalised to 1). Sign convention matches the standard's tables, so a +// published a1 of -1.69065929318241 is stored verbatim. +struct LoudnessBiquadCoeffs { + double b0 = 1.0; + double b1 = 0.0; + double b2 = 0.0; + double a1 = 0.0; + double a2 = 0.0; +}; + +struct LoudnessBiquadState { + double x1 = 0.0; + double x2 = 0.0; + double y1 = 0.0; + double y2 = 0.0; +}; + +inline double loudness_biquad_step(const LoudnessBiquadCoeffs &c, + LoudnessBiquadState &s, double x) +{ + const double y = c.b0 * x + c.b1 * s.x1 + c.b2 * s.x2 + - c.a1 * s.y1 - c.a2 * s.y2; + s.x2 = s.x1; + s.x1 = x; + s.y2 = s.y1; + s.y1 = y; + return y; +} + +// The analog prototype BS.1770-4's 48 kHz table was itself produced from. +// These five constants are the whole of the standard's filter specification +// once the rate is factored out; every published coefficient falls out of +// them. Kept at full precision because the 48 kHz reproduction is asserted to +// 1e-11. +constexpr double kBs1770Stage1Hz = 1681.974450955533; +constexpr double kBs1770Stage1GdB = 3.999843853973347; +constexpr double kBs1770Stage1Q = 0.7071752369554196; +constexpr double kBs1770Stage1VbExp = 0.4996667741545416; +constexpr double kBs1770Stage2Hz = 38.13547087602444; +constexpr double kBs1770Stage2Q = 0.5003270373238773; + +// A rate to fall back on when the caller hands us nothing usable. The ring +// header can legitimately be read before the writer has initialised it (see +// output_audio_frame()'s slot_count guard), and a zero rate must produce +// finite coefficients rather than a NaN that then poisons every subsequent +// filter state for the life of the source. +constexpr uint32_t kLoudnessFallbackRate = 48000; + +inline uint32_t loudness_usable_rate(uint32_t sample_rate) +{ + return (sample_rate >= 8000 && sample_rate <= 384000) + ? sample_rate : kLoudnessFallbackRate; +} + +// Stage 1: the "head" high-shelf, roughly +4 dB above 1 kHz. +inline LoudnessBiquadCoeffs bs1770_stage1_coeffs(uint32_t sample_rate) +{ + const double fs = static_cast(loudness_usable_rate(sample_rate)); + const double K = std::tan(3.14159265358979323846 * kBs1770Stage1Hz / fs); + const double Vh = std::pow(10.0, kBs1770Stage1GdB / 20.0); + const double Vb = std::pow(Vh, kBs1770Stage1VbExp); + const double a0 = 1.0 + K / kBs1770Stage1Q + K * K; + + LoudnessBiquadCoeffs c; + c.b0 = (Vh + Vb * K / kBs1770Stage1Q + K * K) / a0; + c.b1 = 2.0 * (K * K - Vh) / a0; + c.b2 = (Vh - Vb * K / kBs1770Stage1Q + K * K) / a0; + c.a1 = 2.0 * (K * K - 1.0) / a0; + c.a2 = (1.0 - K / kBs1770Stage1Q + K * K) / a0; + return c; +} + +// Stage 2: the RLB high-pass, roughly 38 Hz. b0/b1/b2 are exactly 1/-2/1 at +// every rate -- that is a property of the prototype, not a rounding of the +// published table, so they are written as literals. +inline LoudnessBiquadCoeffs bs1770_stage2_coeffs(uint32_t sample_rate) +{ + const double fs = static_cast(loudness_usable_rate(sample_rate)); + const double K = std::tan(3.14159265358979323846 * kBs1770Stage2Hz / fs); + const double d = 1.0 + K / kBs1770Stage2Q + K * K; + + LoudnessBiquadCoeffs c; + c.b0 = 1.0; + c.b1 = -2.0; + c.b2 = 1.0; + c.a1 = 2.0 * (K * K - 1.0) / d; + c.a2 = (1.0 - K / kBs1770Stage2Q + K * K) / d; + return c; +} diff --git a/tests/audio-loudness-test.cpp b/tests/audio-loudness-test.cpp new file mode 100644 index 00000000..353e3f4a --- /dev/null +++ b/tests/audio-loudness-test.cpp @@ -0,0 +1,123 @@ +// tests/audio-loudness-test.cpp +// ITU-R BS.1770-4 loudness, measured at whatever rate Zoom actually sends. +// +// WHY THIS TEST IS THE WHOLE FEATURE. BS.1770-4 publishes its K-weighting +// biquad coefficients for 48 kHz and for no other rate. This plugin does not +// receive a guaranteed rate: the engine reads GetSampleRate() per buffer and +// stamps it into ShmAudioHeader::sample_rate (engine/src/engine-audio.cpp), +// and Zoom commonly delivers 32 kHz. Coefficients pinned at 48 kHz and fed +// 32 kHz audio still produce a plausible-looking number -- measured below at +// 1.3 LU wrong on a 1 kHz tone -- which is precisely the failure a meter +// cannot survive, because nothing about the reading says it is wrong. +#include "audio-loudness.h" + +#include +#include +#include +#include + +static int failures = 0; + +static void check(bool ok, const char *message) +{ + if (!ok) { + std::cerr << "FAIL: " << message << "\n"; + ++failures; + } +} + +static bool near(double a, double b, double tol) +{ + return std::fabs(a - b) <= tol; +} + +int main() +{ + // ── The published BS.1770-4 table, at 48 kHz, to the digit ───────────── + // Table 1 (stage 1, the "head"/high-shelf pre-filter) and Table 2 (stage + // 2, the RLB high-pass) of BS.1770-4. If the derivation is right, it + // reproduces these exactly at 48 kHz -- that is the only rate at which + // there is anything published to check against, which is why it is + // checked to 1e-11 and not to a comfortable tolerance. + { + const LoudnessBiquadCoeffs s1 = bs1770_stage1_coeffs(48000); + check(near(s1.b0, 1.53512485958697, 1e-11), "48k stage-1 b0 does not match the published BS.1770-4 table"); + check(near(s1.b1, -2.69169618940638, 1e-11), "48k stage-1 b1 does not match the published BS.1770-4 table"); + check(near(s1.b2, 1.19839281085285, 1e-11), "48k stage-1 b2 does not match the published BS.1770-4 table"); + check(near(s1.a1, -1.69065929318241, 1e-11), "48k stage-1 a1 does not match the published BS.1770-4 table"); + check(near(s1.a2, 0.73248077421585, 1e-11), "48k stage-1 a2 does not match the published BS.1770-4 table"); + + const LoudnessBiquadCoeffs s2 = bs1770_stage2_coeffs(48000); + check(near(s2.b0, 1.0, 1e-12), "48k stage-2 b0 must be exactly 1"); + check(near(s2.b1, -2.0, 1e-12), "48k stage-2 b1 must be exactly -2"); + check(near(s2.b2, 1.0, 1e-12), "48k stage-2 b2 must be exactly 1"); + check(near(s2.a1, -1.99004745483398, 1e-11), "48k stage-2 a1 does not match the published BS.1770-4 table"); + check(near(s2.a2, 0.99007225036621, 1e-11), "48k stage-2 a2 does not match the published BS.1770-4 table"); + } + + // ── 32 kHz must produce DIFFERENT, correctly derived coefficients ────── + // These are the bilinear transform of the same analog prototype at + // 32 kHz. A "derivation" that quietly returned the 48 kHz numbers for + // every rate would pass every check above and fail every one here. + { + const LoudnessBiquadCoeffs s1 = bs1770_stage1_coeffs(32000); + check(near(s1.b0, 1.51117789957, 1e-9), "32k stage-1 b0 is wrong"); + check(near(s1.b1, -2.46488941336, 1e-9), "32k stage-1 b1 is wrong"); + check(near(s1.b2, 1.04163327352, 1e-9), "32k stage-1 b2 is wrong"); + check(near(s1.a1, -1.53904509625, 1e-9), "32k stage-1 a1 is wrong"); + check(near(s1.a2, 0.62696685598, 1e-9), "32k stage-1 a2 is wrong"); + + const LoudnessBiquadCoeffs s2 = bs1770_stage2_coeffs(32000); + check(near(s2.a1, -1.98508966899, 1e-9), "32k stage-2 a1 is wrong"); + check(near(s2.a2, 0.98514532067, 1e-9), "32k stage-2 a2 is wrong"); + } + + // ── The two rates must not be the same numbers ───────────────────────── + // Stated as its own assertion rather than left implicit in the two blocks + // above, because "the coefficients are rate-dependent" is the invariant, + // and an implementer reading only this file should see it said out loud. + { + const LoudnessBiquadCoeffs a = bs1770_stage1_coeffs(48000); + const LoudnessBiquadCoeffs b = bs1770_stage1_coeffs(32000); + check(std::fabs(a.a1 - b.a1) > 0.10, + "stage-1 a1 barely moved between 48 kHz and 32 kHz -- the " + "coefficients are not being derived from the rate at all"); + const LoudnessBiquadCoeffs c = bs1770_stage2_coeffs(48000); + const LoudnessBiquadCoeffs d = bs1770_stage2_coeffs(32000); + check(std::fabs(c.a1 - d.a1) > 0.004, + "stage-2 a1 barely moved between 48 kHz and 32 kHz -- the " + "high-pass corner is being placed at a fixed digital frequency " + "rather than a fixed 38 Hz"); + } + + // ── A degenerate rate must not produce NaN or a divide by zero ───────── + { + const LoudnessBiquadCoeffs s1 = bs1770_stage1_coeffs(0); + check(std::isfinite(s1.b0) && std::isfinite(s1.a1), + "a zero sample rate produced non-finite coefficients -- the " + "ring header can be read before the writer has initialised it"); + } + + // ── The biquad itself: a direct-form-II-transposed step ──────────────── + // Pinned against hand-computed values so a sign slip on the feedback + // terms cannot hide inside a filter response test. + { + const LoudnessBiquadCoeffs c{0.5, 0.25, 0.125, -0.5, 0.25}; + LoudnessBiquadState st{}; + // y[0] = 0.5*1 = 0.5 + const double y0 = loudness_biquad_step(c, st, 1.0); + check(near(y0, 0.5, 1e-12), "biquad sample 0 was not b0*x0"); + // y[1] = 0.5*0 + 0.25*1 + 0.125*0 - (-0.5)*0.5 - 0.25*0 = 0.5 + const double y1 = loudness_biquad_step(c, st, 0.0); + check(near(y1, 0.5, 1e-12), "biquad sample 1 is wrong -- check the " + "sign convention on a1 (y = b.x - a.y)"); + // y[2] = 0.125*1 - (-0.5)*0.5 - 0.25*0.5 = 0.125 + 0.25 - 0.125 = 0.25 + const double y2 = loudness_biquad_step(c, st, 0.0); + check(near(y2, 0.25, 1e-12), "biquad sample 2 is wrong -- the second " + "feedback tap (a2) is not being applied"); + } + + if (failures == 0) + std::cout << "audio-loudness: all tests passed\n"; + return failures == 0 ? 0 : 1; +} From 791cf08c83c13751056f73a60bd52e867c3a3635 Mon Sep 17 00:00:00 2001 From: Grok Date: Sat, 5 Sep 2026 16:31:32 -0400 Subject: [PATCH 05/15] feat(loudness): K-weighted momentary and short-term loudness from int16 interleaved PCM --- src/audio-loudness.h | 182 ++++++++++++++++++++++++++++++++++ tests/audio-loudness-test.cpp | 140 ++++++++++++++++++++++++++ 2 files changed, 322 insertions(+) diff --git a/src/audio-loudness.h b/src/audio-loudness.h index dad41ac0..e9a5221c 100644 --- a/src/audio-loudness.h +++ b/src/audio-loudness.h @@ -120,3 +120,185 @@ inline LoudnessBiquadCoeffs bs1770_stage2_coeffs(uint32_t sample_rate) c.a2 = (1.0 - K / kBs1770Stage2Q + K * K) / d; return c; } + +// The standard's absolute offset. It exists to cancel the K-weighting's +// +0.691 dB gain at 997 Hz, which is why a 1 kHz tone's LUFS value equals +// 10*log10 of its un-weighted mean square exactly. +constexpr double kLoudnessOffsetDb = -0.691; + +// Block/hop geometry. 400 ms blocks advancing every 100 ms is 75% overlap, +// which is what BS.1770-4 specifies for gated integration; momentary IS one +// such block, and short-term is 30 hops. +constexpr uint32_t kLoudnessHopMs = 100; +constexpr uint32_t kLoudnessMomentaryHops = 4; // 400 ms +constexpr uint32_t kLoudnessShortTermHops = 30; // 3 s + +// L = -0.691 + 10*log10(sum of G_i * z_i). Returns -HUGE_VAL for a +// non-positive mean square rather than letting log10 produce -inf/NaN at an +// arbitrary call site; every caller in this header checks for it. +inline double loudness_lufs_from_mean_square(double z) +{ + if (!(z > 0.0)) return -HUGE_VAL; + return kLoudnessOffsetDb + 10.0 * std::log10(z); +} + +// BS.1770-4 channel weights, in the standard's channel order +// (L, R, C, LFE, Ls, Rs). Zoom participant audio is mono or stereo, so in +// practice only the G = 1.0 terms are ever reached -- but a source configured +// for more channels must not silently weight a surround channel as if it were +// a front one, and the LFE must not be counted at all. +inline double loudness_channel_weight(uint16_t channels, uint16_t channel) +{ + if (channels <= 2) return 1.0; + switch (channel) { + case 0: case 1: case 2: return 1.00; // L, R, C + case 3: return 0.00; // LFE is excluded, not attenuated + case 4: case 5: return 1.41; // Ls, Rs + default: return 0.00; + } +} + +// A running BS.1770-4 measurement for ONE participant. +// +// OWNERSHIP: not thread-safe and deliberately so. In the plugin exactly one +// thread -- the audio lane that owns output_audio_frame() -- feeds it, under +// the same ctx->mtx that already guards the source's timeline, and readers +// take that mutex to copy the three numbers out. Adding a lock in here would +// put one on the media path for no gain. +struct LoudnessMeter { + uint32_t sample_rate = 0; + uint16_t channels = 0; + + LoudnessBiquadCoeffs c1{}; + LoudnessBiquadCoeffs c2{}; + std::vector s1; // stage 1 state, one per channel + std::vector s2; // stage 2 state, one per channel + + // Current partial 100 ms hop. + uint32_t hop_frames = 0; // frames per hop at the configured rate + uint32_t hop_filled = 0; + double hop_acc = 0.0; // sum over frames of sum_ch(G * y^2) + + // The last kLoudnessShortTermHops completed hops, newest at + // (hop_total - 1) % kLoudnessShortTermHops. + double hop_ring[kLoudnessShortTermHops] = {}; + uint64_t hop_total = 0; +}; + +// (Re)configures for a rate/channel count and clears all filter state. Called +// automatically by loudness_meter_feed_int16() whenever the wire format +// changes -- which it can, mid-source: Zoom renegotiates, and the operator's +// Mix/Isolated role flip changes the channel count on the same subscription. +// Carrying filter history across that would smear one format's transient into +// the other's measurement. +inline void loudness_meter_configure(LoudnessMeter &m, uint32_t sample_rate, + uint16_t channels) +{ + const uint32_t rate = loudness_usable_rate(sample_rate); + m.sample_rate = rate; + m.channels = channels == 0 ? 1 : channels; + m.c1 = bs1770_stage1_coeffs(rate); + m.c2 = bs1770_stage2_coeffs(rate); + m.s1.assign(m.channels, LoudnessBiquadState{}); + m.s2.assign(m.channels, LoudnessBiquadState{}); + m.hop_frames = (rate * kLoudnessHopMs) / 1000; + if (m.hop_frames == 0) m.hop_frames = 1; + m.hop_filled = 0; + m.hop_acc = 0.0; + for (uint32_t i = 0; i < kLoudnessShortTermHops; ++i) m.hop_ring[i] = 0.0; + m.hop_total = 0; +} + +// Hook the gated integrator into the hop boundary. Defined in Task 3; the +// forward declaration keeps feed_int16 below unchanged when it lands. +inline void loudness_meter_on_hop_complete(LoudnessMeter &m); + +// Feeds interleaved 16-bit signed PCM -- the format the engine writes into +// the SHM ring, unconverted. +// +// SCALING: /32768.0, not /32767.0. int16 is asymmetric and full negative +// scale is -32768; dividing by 32767 would let a legitimate sample exceed +// -1.0 and is the wrong direction for a measurement. +// +// PARTIAL BUFFERS ARE THE NORMAL CASE. Zoom delivers ~10 ms buffers and one +// media event can carry eight of them, so a 100 ms hop is assembled from many +// calls. The hop boundary is decided by frame count alone and never by call +// boundaries, which is what makes "feed the whole drain loop" identical to +// "feed one big buffer" -- pinned as chunk invariance in the test. +inline void loudness_meter_feed_int16(LoudnessMeter &m, const int16_t *pcm, + size_t frames, uint16_t channels, + uint32_t sample_rate) +{ + if (pcm == nullptr || frames == 0 || channels == 0) return; + if (m.sample_rate != loudness_usable_rate(sample_rate) || + m.channels != channels) { + loudness_meter_configure(m, sample_rate, channels); + } + + for (size_t f = 0; f < frames; ++f) { + double frame_sum = 0.0; + for (uint16_t ch = 0; ch < channels; ++ch) { + const double g = loudness_channel_weight(channels, ch); + const double x = static_cast(pcm[f * channels + ch]) / + 32768.0; + const double y1 = loudness_biquad_step(m.c1, m.s1[ch], x); + const double y2 = loudness_biquad_step(m.c2, m.s2[ch], y1); + // The filters run even for a zero-weight channel: their state is + // per channel and skipping them would make the LFE's history + // depend on how long it had been zero-weighted. + frame_sum += g * y2 * y2; + } + m.hop_acc += frame_sum; + if (++m.hop_filled >= m.hop_frames) { + const double hop_mean = m.hop_acc / + static_cast(m.hop_frames); + m.hop_ring[m.hop_total % kLoudnessShortTermHops] = hop_mean; + ++m.hop_total; + m.hop_acc = 0.0; + m.hop_filled = 0; + loudness_meter_on_hop_complete(m); + } + } +} + +// Mean of the newest `n` completed hops. False when fewer than `n` exist -- +// which is the honest answer for a panelist who has just been subscribed, and +// is why every getter here returns bool rather than a sentinel loudness. +inline bool loudness_hop_mean(const LoudnessMeter &m, uint32_t n, double *out) +{ + if (n == 0 || n > kLoudnessShortTermHops || m.hop_total < n) return false; + double sum = 0.0; + for (uint32_t i = 0; i < n; ++i) { + const uint64_t idx = m.hop_total - 1 - i; + sum += m.hop_ring[idx % kLoudnessShortTermHops]; + } + *out = sum / static_cast(n); + return true; +} + +// Momentary (M): one 400 ms block, ungated. +inline bool loudness_meter_momentary(const LoudnessMeter &m, double *out_lufs) +{ + double z = 0.0; + if (!loudness_hop_mean(m, kLoudnessMomentaryHops, &z)) return false; + const double l = loudness_lufs_from_mean_square(z); + if (!std::isfinite(l)) return false; + *out_lufs = l; + return true; +} + +// Short-term (S): 3 s, ungated. The number an operator reads while the +// panelist is talking. +inline bool loudness_meter_short_term(const LoudnessMeter &m, double *out_lufs) +{ + double z = 0.0; + if (!loudness_hop_mean(m, kLoudnessShortTermHops, &z)) return false; + const double l = loudness_lufs_from_mean_square(z); + if (!std::isfinite(l)) return false; + *out_lufs = l; + return true; +} + +// Placeholder until Task 3 lands the gated integrator. Declared above so +// feed_int16 already calls it; defined empty here so this task builds alone. +inline void loudness_meter_on_hop_complete(LoudnessMeter &) {} diff --git a/tests/audio-loudness-test.cpp b/tests/audio-loudness-test.cpp index 353e3f4a..00876e24 100644 --- a/tests/audio-loudness-test.cpp +++ b/tests/audio-loudness-test.cpp @@ -117,6 +117,146 @@ int main() "feedback tap (a2) is not being applied"); } + // ── Reference tones: the numbers an implementer can check by hand ────── + // + // The K-weighting curve has a gain of exactly +0.691 dB at 997 Hz, and + // BS.1770's -0.691 dB offset is there to cancel it. So for a ~1 kHz sine + // the whole measurement collapses to L = 10*log10(mean square of the + // un-weighted signal), which is a number that can be worked out on paper: + // + // peak 1.0 -> mean square 0.5 -> -3.01 LUFS + // peak 0.1 -> mean square 0.005 -> -23.01 LUFS + // RMS 0.1 -> mean square 0.01 -> -20.00 LUFS + // + // The third is the one to remember: a 1 kHz tone at -20 dBFS RMS reads + // -20.0 LUFS. If that does not hold, the offset, the channel weight, the + // int16 scaling or the K-weighting is wrong, and no amount of relative + // comparison downstream will save the reading. + auto feed_sine = [](LoudnessMeter &m, uint32_t rate, double peak, + double freq, double seconds) { + const size_t n = static_cast(rate * seconds); + std::vector pcm(n); + for (size_t i = 0; i < n; ++i) { + const double v = peak * std::sin(2.0 * 3.14159265358979323846 * + freq * static_cast(i) / + static_cast(rate)); + double s = v * 32767.0; + if (s > 32767.0) s = 32767.0; + if (s < -32767.0) s = -32767.0; + pcm[i] = static_cast(std::lround(s)); + } + loudness_meter_feed_int16(m, pcm.data(), n, 1, rate); + }; + + { + LoudnessMeter m; + feed_sine(m, 48000, 1.0, 1000.0, 5.0); + double lufs = 0.0; + check(loudness_meter_momentary(m, &lufs), + "momentary loudness was unavailable after 5 s of tone"); + check(near(lufs, -3.01, 0.10), + "a full-scale 1 kHz sine at 48 kHz did not read -3.01 LUFS"); + } + { + LoudnessMeter m; + feed_sine(m, 48000, 0.1, 1000.0, 5.0); + double lufs = 0.0; + check(loudness_meter_momentary(m, &lufs), "momentary unavailable"); + check(near(lufs, -23.01, 0.10), + "a 1 kHz sine of peak amplitude 0.1 at 48 kHz did not read " + "-23.01 LUFS"); + } + { + // -20 dBFS RMS: peak = sqrt(2) * 0.1. + LoudnessMeter m; + feed_sine(m, 48000, std::sqrt(2.0) * 0.1, 1000.0, 5.0); + double lufs = 0.0; + check(loudness_meter_momentary(m, &lufs), "momentary unavailable"); + check(near(lufs, -20.00, 0.10), + "a -20 dBFS RMS 1 kHz sine at 48 kHz did not read -20.0 LUFS -- " + "K-weighting is ~0 dB at 1 kHz once the -0.691 offset is " + "applied, so this is an equality, not an approximation"); + } + + // ── The same tone at 32 kHz must read the same, not 1.3 LU high ──────── + // This is the assertion the whole runtime-rate design exists for. With + // the 48 kHz coefficients applied to 32 kHz audio this tone reads + // -18.66 LUFS instead of -19.98: it passes a "looks like a plausible + // loudness" eyeball test and fails here. + { + LoudnessMeter m; + feed_sine(m, 32000, std::sqrt(2.0) * 0.1, 1000.0, 5.0); + double lufs = 0.0; + check(loudness_meter_momentary(m, &lufs), "momentary unavailable at 32 kHz"); + check(near(lufs, -19.98, 0.12), + "a -20 dBFS RMS 1 kHz sine at 32 kHz did not read -20 LUFS -- " + "the coefficients are not following the runtime rate"); + check(lufs < -19.5, + "the 32 kHz reading is more than 0.5 LU hot, which is the " + "signature of 48 kHz coefficients being used at 32 kHz"); + } + + // ── Short-term needs 3 s; momentary needs 400 ms ─────────────────────── + { + LoudnessMeter m; + feed_sine(m, 48000, std::sqrt(2.0) * 0.1, 1000.0, 0.35); + double lufs = 0.0; + check(!loudness_meter_momentary(m, &lufs), + "momentary reported a value before a full 400 ms block existed"); + feed_sine(m, 48000, std::sqrt(2.0) * 0.1, 1000.0, 0.20); + check(loudness_meter_momentary(m, &lufs), + "momentary was still unavailable after 550 ms"); + check(!loudness_meter_short_term(m, &lufs), + "short-term reported a value before 3 s of audio existed"); + feed_sine(m, 48000, std::sqrt(2.0) * 0.1, 1000.0, 3.0); + check(loudness_meter_short_term(m, &lufs), + "short-term was still unavailable after 3.5 s"); + check(near(lufs, -20.00, 0.15), "short-term did not read -20 LUFS"); + } + + // ── Stereo: two identical channels are +3 dB, not the same as mono ───── + // BS.1770 sums the weighted per-channel mean squares (G = 1.0 for L and + // R), it does not average them. Averaging is the mistake that makes a + // stereo panelist read 3 LU quieter than the identical mono one beside + // them, which is exactly the comparison this feature exists to make. + { + LoudnessMeter mono; + feed_sine(mono, 48000, std::sqrt(2.0) * 0.1, 1000.0, 2.0); + double mono_lufs = 0.0; + check(loudness_meter_momentary(mono, &mono_lufs), "mono unavailable"); + + LoudnessMeter st; + const size_t n = 48000 * 2; + std::vector pcm(n * 2); + for (size_t i = 0; i < n; ++i) { + const double v = std::sqrt(2.0) * 0.1 * + std::sin(2.0 * 3.14159265358979323846 * 1000.0 * + static_cast(i) / 48000.0); + const int16_t s = static_cast(std::lround(v * 32767.0)); + pcm[i * 2] = s; + pcm[i * 2 + 1] = s; + } + loudness_meter_feed_int16(st, pcm.data(), n, 2, 48000); + double st_lufs = 0.0; + check(loudness_meter_momentary(st, &st_lufs), "stereo unavailable"); + check(near(st_lufs - mono_lufs, 3.01, 0.05), + "dual-mono stereo was not +3.01 LU relative to mono -- the " + "channels are being averaged instead of summed"); + } + + // ── Digital silence never produces NaN or -inf leaking to a caller ───── + { + LoudnessMeter m; + std::vector zeros(48000, 0); + loudness_meter_feed_int16(m, zeros.data(), zeros.size(), 1, 48000); + double lufs = 0.0; + const bool have = loudness_meter_momentary(m, &lufs); + check(!have || std::isfinite(lufs), + "true digital silence produced a non-finite momentary reading -- " + "a panelist who has not spoken yet is the normal case here, not " + "an edge case"); + } + if (failures == 0) std::cout << "audio-loudness: all tests passed\n"; return failures == 0 ? 0 : 1; From f88e80d8e5f14db368793ebf44eee0a6fbf5c63d Mon Sep 17 00:00:00 2001 From: Grok Date: Sat, 5 Sep 2026 16:35:18 -0400 Subject: [PATCH 06/15] feat(loudness): gated integrated loudness over a resettable per-panelist check window --- src/audio-loudness.h | 103 +++++++++++++++++++++++++++- tests/audio-loudness-test.cpp | 123 ++++++++++++++++++++++++++++++++++ 2 files changed, 223 insertions(+), 3 deletions(-) diff --git a/src/audio-loudness.h b/src/audio-loudness.h index e9a5221c..00cb6967 100644 --- a/src/audio-loudness.h +++ b/src/audio-loudness.h @@ -183,6 +183,15 @@ struct LoudnessMeter { // (hop_total - 1) % kLoudnessShortTermHops. double hop_ring[kLoudnessShortTermHops] = {}; uint64_t hop_total = 0; + + // The gated integration window -- ONE PANELIST'S MIC CHECK, not the + // session. Each entry is the mean square of a 400 ms block that cleared + // the absolute gate. Held as values rather than a running sum because the + // relative gate has to re-examine every block once the absolute-gated + // mean is known. + std::vector gated; + size_t gated_head = 0; // ring write position once `gated` is full + uint64_t gated_total = 0; // blocks ever admitted, never wrapped }; // (Re)configures for a rate/channel count and clears all filter state. Called @@ -299,6 +308,94 @@ inline bool loudness_meter_short_term(const LoudnessMeter &m, double *out_lufs) return true; } -// Placeholder until Task 3 lands the gated integrator. Declared above so -// feed_int16 already calls it; defined empty here so this task builds alone. -inline void loudness_meter_on_hop_complete(LoudnessMeter &) {} +// BS.1770-4's two gates. The absolute one discards silence for free, which is +// exactly the mechanism a panel needs: a panelist is silent roughly 80% of a +// panel, and an ungated integrated reading over that measures the meeting +// rather than the microphone (measured: 4 s of -20 LUFS speech inside 20 s +// reads -27.08 ungated). The relative one then discards the quiet tail so the +// answer describes the speech. +constexpr double kLoudnessAbsoluteGateLufs = -70.0; +constexpr double kLoudnessRelativeGateLu = -10.0; + +// 6000 blocks is 10 minutes of continuously-gated audio at a 100 ms hop. A +// mic check is 20-60 s (~200-600 blocks), so this is never reached in the +// use this was built for; past it the window keeps the most RECENT 10 minutes +// rather than growing without bound. Documented rather than silent, because +// "the oldest audio quietly leaves the window" is a real semantic and an +// operator who leaves a board running all show is entitled to know it. +constexpr size_t kLoudnessMaxGatedBlocks = 6000; + +// Called at every completed 100 ms hop. A 400 ms block is the newest four +// hops, so admitting one block per hop is the standard's 75% overlap. +inline void loudness_meter_on_hop_complete(LoudnessMeter &m) +{ + double z = 0.0; + if (!loudness_hop_mean(m, kLoudnessMomentaryHops, &z)) return; + const double l = loudness_lufs_from_mean_square(z); + if (!std::isfinite(l) || l <= kLoudnessAbsoluteGateLufs) return; + + if (m.gated.size() < kLoudnessMaxGatedBlocks) { + m.gated.push_back(z); + } else { + m.gated[m.gated_head] = z; + m.gated_head = (m.gated_head + 1) % kLoudnessMaxGatedBlocks; + } + ++m.gated_total; +} + +// Starts this source's check window over. Clears the gated blocks and the hop +// history, but NOT the biquad state: the filters describe the signal that is +// still arriving, and zeroing them mid-stream would inject a transient into +// the first block of the new window. +inline void loudness_meter_reset_window(LoudnessMeter &m) +{ + m.gated.clear(); + m.gated_head = 0; + m.gated_total = 0; + m.hop_acc = 0.0; + m.hop_filled = 0; + for (uint32_t i = 0; i < kLoudnessShortTermHops; ++i) m.hop_ring[i] = 0.0; + m.hop_total = 0; +} + +// Blocks admitted to the current window. A board uses this to decide whether +// an integrated reading is worth showing: the spec's 20 s mic check yields +// ~200 blocks, so a handful of blocks is a cough, not a check. +inline uint64_t loudness_meter_gated_blocks(const LoudnessMeter &m) +{ + return m.gated_total; +} + +// Integrated (I): the two-pass gate, over the current check window. +// False means "this panelist has not produced a measurable check yet", which +// is a different statement from any loudness value and must stay +// distinguishable all the way to the board. +inline bool loudness_meter_integrated(const LoudnessMeter &m, double *out_lufs) +{ + if (m.gated.empty()) return false; + + double sum = 0.0; + for (double z : m.gated) sum += z; + const double abs_mean_lufs = + loudness_lufs_from_mean_square(sum / static_cast(m.gated.size())); + if (!std::isfinite(abs_mean_lufs)) return false; + + const double relative_threshold = abs_mean_lufs + kLoudnessRelativeGateLu; + double sum2 = 0.0; + size_t n2 = 0; + for (double z : m.gated) { + // Strictly greater, per BS.1770-4: a block exactly on the threshold + // is excluded. + if (loudness_lufs_from_mean_square(z) > relative_threshold) { + sum2 += z; + ++n2; + } + } + if (n2 == 0) return false; + + const double l = loudness_lufs_from_mean_square(sum2 / + static_cast(n2)); + if (!std::isfinite(l)) return false; + *out_lufs = l; + return true; +} diff --git a/tests/audio-loudness-test.cpp b/tests/audio-loudness-test.cpp index 00876e24..e1b707c9 100644 --- a/tests/audio-loudness-test.cpp +++ b/tests/audio-loudness-test.cpp @@ -257,6 +257,129 @@ int main() "an edge case"); } + // ── The gate is the reason this measure is usable at all ─────────────── + // A panelist is silent roughly 80% of a preshow. 4 s of speech at + // -20 LUFS followed by 16 s of silence averages to -27.08 LUFS if + // ungated -- an answer that describes the meeting, not the microphone. + // The BS.1770 absolute gate at -70 LUFS discards the silent blocks and + // the answer comes back to -20. + { + LoudnessMeter m; + feed_sine(m, 48000, std::sqrt(2.0) * 0.1, 1000.0, 4.0); + std::vector zeros(48000 * 16, 0); + loudness_meter_feed_int16(m, zeros.data(), zeros.size(), 1, 48000); + + double lufs = 0.0; + check(loudness_meter_integrated(m, &lufs), + "integrated loudness was unavailable after 4 s of speech"); + check(near(lufs, -20.16, 0.35), + "4 s of -20 LUFS speech in 20 s of silence did not integrate to " + "about -20 LUFS -- an ungated running average reads -27.08 here"); + check(lufs > -22.0, + "the integrated reading is dragged down by silence: the " + "absolute -70 LUFS gate is not being applied"); + check(loudness_meter_gated_blocks(m) > 30 && + loudness_meter_gated_blocks(m) < 60, + "the gated block count is not ~40 -- 4 s of speech at a 100 ms " + "hop is about 40 blocks that clear the absolute gate"); + } + + // ── The RELATIVE gate, which the absolute gate cannot stand in for ───── + // 10 s at -20 LUFS then 10 s at -40 LUFS: every block clears -70, so the + // absolute gate alone leaves -22.96. The relative gate (-10 LU below the + // absolute-gated mean) drops the quiet half and the answer is -20.06 -- + // the loudness of the speech, which is what a mic check is asking about. + { + LoudnessMeter m; + feed_sine(m, 48000, std::sqrt(2.0) * 0.1, 1000.0, 10.0); + feed_sine(m, 48000, std::sqrt(2.0) * 0.01, 1000.0, 10.0); + double lufs = 0.0; + check(loudness_meter_integrated(m, &lufs), "integrated unavailable"); + check(near(lufs, -20.06, 0.30), + "loud-then-quiet did not integrate to about -20 LUFS -- with " + "only the absolute gate this reads -22.96"); + } + + // ── The check window is resettable, and a reset is a clean slate ─────── + // A mic check is per panelist. Without this the number is polluted by + // whoever spoke before them on the same source. + { + LoudnessMeter m; + feed_sine(m, 48000, 1.0, 1000.0, 3.0); // very loud, -3 LUFS + double before = 0.0; + check(loudness_meter_integrated(m, &before), "integrated unavailable"); + check(near(before, -3.01, 0.30), "the loud pass did not read -3 LUFS"); + + loudness_meter_reset_window(m); + double after = 0.0; + check(!loudness_meter_integrated(m, &after), + "integrated loudness survived a window reset -- the previous " + "panelist's check is still in the number"); + check(loudness_meter_gated_blocks(m) == 0, + "the gated block count survived a window reset"); + + feed_sine(m, 48000, std::sqrt(2.0) * 0.1, 1000.0, 3.0); + check(loudness_meter_integrated(m, &after), "integrated unavailable " + "after refilling the window"); + check(near(after, -20.00, 0.30), + "the post-reset reading is contaminated by the pre-reset audio"); + } + + // ── A panelist who has never spoken has NO integrated reading ────────── + // Not -70, not 0. The board must be able to say "no audio" rather than + // print a number that looks like a measurement. + { + LoudnessMeter m; + std::vector zeros(48000 * 5, 0); + loudness_meter_feed_int16(m, zeros.data(), zeros.size(), 1, 48000); + double lufs = 0.0; + check(!loudness_meter_integrated(m, &lufs), + "five seconds of pure silence produced an integrated loudness"); + check(loudness_meter_gated_blocks(m) == 0, + "silent blocks were counted as gated blocks"); + } + + // ── Chunk invariance: the drain-loop law, stated as arithmetic ───────── + // A media event is a coalescing prompt, not a payload: one wakeup can + // carry eight ring slots. Measuring "the buffer that woke us" would throw + // away up to seven eighths of the audio. This asserts that feeding the + // same samples in 10 ms pieces and in one 2 s piece are the same + // measurement, which is what makes feeding from inside the drain loop + // correct. + { + const uint32_t rate = 32000; + const size_t n = rate * 2; + std::vector pcm(n); + for (size_t i = 0; i < n; ++i) { + const double v = 0.2 * std::sin(2.0 * 3.14159265358979323846 * + 440.0 * static_cast(i) / + static_cast(rate)); + pcm[i] = static_cast(std::lround(v * 32767.0)); + } + LoudnessMeter whole; + loudness_meter_feed_int16(whole, pcm.data(), n, 1, rate); + + LoudnessMeter pieces; + const size_t chunk = rate / 100; // 10 ms, Zoom's buffer size + for (size_t off = 0; off < n; off += chunk) { + const size_t take = (off + chunk <= n) ? chunk : (n - off); + loudness_meter_feed_int16(pieces, pcm.data() + off, take, 1, rate); + } + + double a = 0.0, b = 0.0; + check(loudness_meter_integrated(whole, &a) && + loudness_meter_integrated(pieces, &b), + "one of the two feeding patterns produced no integrated value"); + check(near(a, b, 1e-9), + "feeding in 10 ms chunks did not match feeding in one buffer -- " + "the hop boundary is following call boundaries instead of frame " + "counts, so the measurement depends on IPC batching"); + double ma = 0.0, mb = 0.0; + check(loudness_meter_momentary(whole, &ma) && + loudness_meter_momentary(pieces, &mb) && near(ma, mb, 1e-9), + "momentary differed between chunked and whole feeding"); + } + if (failures == 0) std::cout << "audio-loudness: all tests passed\n"; return failures == 0 ? 0 : 1; From e44792709c712c85bb4474659730ef40f106e3eb Mon Sep 17 00:00:00 2001 From: Grok Date: Sat, 5 Sep 2026 16:41:41 -0400 Subject: [PATCH 07/15] fix(loudness): clear gated integration window on mid-stream format change loudness_meter_configure() cleared filter state on a mid-source format change (Zoom renegotiation, Mix/Isolated role flip) but not the gated integration window Task 3 added. Blocks measured under the old biquad coefficients/channel weighting could survive into an average with blocks measured under the new ones, corrupting a panelist's in-progress check. Factored the clearing shared by configure() and reset_window() into loudness_meter_clear_window() so the two paths cannot drift apart; added a covering test. --- src/audio-loudness.h | 46 +++++++++++++++++++++++++---------- tests/audio-loudness-test.cpp | 30 +++++++++++++++++++++++ 2 files changed, 63 insertions(+), 13 deletions(-) diff --git a/src/audio-loudness.h b/src/audio-loudness.h index 00cb6967..dd0a3939 100644 --- a/src/audio-loudness.h +++ b/src/audio-loudness.h @@ -194,12 +194,38 @@ struct LoudnessMeter { uint64_t gated_total = 0; // blocks ever admitted, never wrapped }; +// Clears the hop-tracking and gated-integration state shared by +// loudness_meter_configure() and loudness_meter_reset_window(). Factored out +// so the two paths cannot drift apart: a mid-stream format change (Zoom +// renegotiating, or a Mix/Isolated role flip changing the channel count on +// the same subscription -- see loudness_meter_configure() below) is just as +// much a "this check window's numbers are no longer comparable" event as an +// explicit reset, because the gated blocks it would otherwise carry forward +// were computed under the OLD biquad coefficients and channel weighting. +// Mixing those with post-change blocks silently averages two different +// measurements into one check-window result. Does NOT touch filter state +// (c1/c2/s1/s2) -- callers that need that cleared too (configure) do it +// themselves, immediately adjacent to this call. +inline void loudness_meter_clear_window(LoudnessMeter &m) +{ + m.gated.clear(); + m.gated_head = 0; + m.gated_total = 0; + m.hop_filled = 0; + m.hop_acc = 0.0; + for (uint32_t i = 0; i < kLoudnessShortTermHops; ++i) m.hop_ring[i] = 0.0; + m.hop_total = 0; +} + // (Re)configures for a rate/channel count and clears all filter state. Called // automatically by loudness_meter_feed_int16() whenever the wire format // changes -- which it can, mid-source: Zoom renegotiates, and the operator's // Mix/Isolated role flip changes the channel count on the same subscription. // Carrying filter history across that would smear one format's transient into -// the other's measurement. +// the other's measurement -- and the same is true of the gated integration +// window, cleared here via loudness_meter_clear_window() so a format change +// mid-check-window cannot silently average blocks measured under two +// different coefficient sets into one panelist's reading. inline void loudness_meter_configure(LoudnessMeter &m, uint32_t sample_rate, uint16_t channels) { @@ -212,10 +238,7 @@ inline void loudness_meter_configure(LoudnessMeter &m, uint32_t sample_rate, m.s2.assign(m.channels, LoudnessBiquadState{}); m.hop_frames = (rate * kLoudnessHopMs) / 1000; if (m.hop_frames == 0) m.hop_frames = 1; - m.hop_filled = 0; - m.hop_acc = 0.0; - for (uint32_t i = 0; i < kLoudnessShortTermHops; ++i) m.hop_ring[i] = 0.0; - m.hop_total = 0; + loudness_meter_clear_window(m); } // Hook the gated integrator into the hop boundary. Defined in Task 3; the @@ -346,16 +369,13 @@ inline void loudness_meter_on_hop_complete(LoudnessMeter &m) // Starts this source's check window over. Clears the gated blocks and the hop // history, but NOT the biquad state: the filters describe the signal that is // still arriving, and zeroing them mid-stream would inject a transient into -// the first block of the new window. +// the first block of the new window. Shares loudness_meter_clear_window() +// with loudness_meter_configure() precisely so this deliberate asymmetry +// (filter state preserved here, cleared there) cannot accidentally regress +// into the two functions clearing different things. inline void loudness_meter_reset_window(LoudnessMeter &m) { - m.gated.clear(); - m.gated_head = 0; - m.gated_total = 0; - m.hop_acc = 0.0; - m.hop_filled = 0; - for (uint32_t i = 0; i < kLoudnessShortTermHops; ++i) m.hop_ring[i] = 0.0; - m.hop_total = 0; + loudness_meter_clear_window(m); } // Blocks admitted to the current window. A board uses this to decide whether diff --git a/tests/audio-loudness-test.cpp b/tests/audio-loudness-test.cpp index e1b707c9..e1621f29 100644 --- a/tests/audio-loudness-test.cpp +++ b/tests/audio-loudness-test.cpp @@ -325,6 +325,36 @@ int main() "the post-reset reading is contaminated by the pre-reset audio"); } + // ── A mid-window format change is ALSO a clean slate ──────────────────── + // loudness_meter_configure()'s own header comment documents that it fires + // automatically mid-source: Zoom renegotiates, and a Mix/Isolated role + // flip changes the channel count on the same subscription. If that lands + // inside a panelist's 20-60 s check window, the gated blocks computed + // under the OLD coefficients/channel weighting must not survive to be + // averaged with blocks computed under the NEW ones. + { + LoudnessMeter m; + feed_sine(m, 48000, std::sqrt(2.0) * 0.1, 1000.0, 3.0); + double before = 0.0; + check(loudness_meter_integrated(m, &before), + "integrated loudness was unavailable before the format change"); + check(loudness_meter_gated_blocks(m) > 0, + "no gated blocks accumulated before the format change -- the " + "test setup is not exercising the window it means to clear"); + + loudness_meter_configure(m, 32000, 2); // different rate AND channels + + check(loudness_meter_gated_blocks(m) == 0, + "the gated block count survived a mid-window format change -- " + "loudness_meter_configure() must clear the gated window, not " + "just the filters"); + double after = 0.0; + check(!loudness_meter_integrated(m, &after), + "integrated loudness survived a mid-window format change -- " + "blocks measured under the old sample rate/channel count are " + "still being averaged with whatever comes next"); + } + // ── A panelist who has never spoken has NO integrated reading ────────── // Not -70, not 0. The board must be able to say "no audio" rather than // print a number that looks like a measurement. From 23298879f8a4a42ff3abffe54aededb7afe80e95 Mon Sep 17 00:00:00 2001 From: Grok Date: Sat, 5 Sep 2026 16:45:27 -0400 Subject: [PATCH 08/15] feat(loudness): readiness-board model with panel-median reference and pinned layout arithmetic --- CMakeLists.txt | 14 ++ src/loudness-board.h | 302 +++++++++++++++++++++++++++++++++ tests/loudness-board-test.cpp | 303 ++++++++++++++++++++++++++++++++++ 3 files changed, 619 insertions(+) create mode 100644 src/loudness-board.h create mode 100644 tests/loudness-board-test.cpp diff --git a/CMakeLists.txt b/CMakeLists.txt index 21d57809..3b1fc793 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -994,6 +994,20 @@ if(BUILD_TESTING) add_test(NAME CoreVideoAudioLoudness COMMAND CoreVideoAudioLoudnessTest) + # The readiness board's model: panel MEDIAN (not mean, so one laptop mic + # cannot drag the reference), deviation in LU, pass/fail, deterministic + # row order, and the row/bar rectangle arithmetic. The layout is pinned + # here rather than looked at on screen because this repo has no headless + # GPU harness and has ruled against building one. See src/loudness-board.h. + add_executable(CoreVideoLoudnessBoardTest + tests/loudness-board-test.cpp + ) + target_include_directories(CoreVideoLoudnessBoardTest PRIVATE + "${CMAKE_CURRENT_SOURCE_DIR}/src" + ) + add_test(NAME CoreVideoLoudnessBoard + COMMAND CoreVideoLoudnessBoardTest) + # Backfills silence into an ISO WAV across gaps so its own timeline # tracks real elapsed time instead of shrinking by every silent # stretch — 2026-08-21 design review ("does ZoomISO handle variable diff --git a/src/loudness-board.h b/src/loudness-board.h new file mode 100644 index 00000000..c1676281 --- /dev/null +++ b/src/loudness-board.h @@ -0,0 +1,302 @@ +#pragma once + +// The preshow readiness board: one row per panelist, showing how far their +// loudness sits from the panel's, and whether that is acceptable. +// +// WHY THE HEADLINE NUMBER IS RELATIVE. An operator running a mic check does +// not primarily care that a panelist hits -23 LUFS. They care that panelist A +// is not 6 LU louder than panelist B, because that is what the audience +// hears. So the reference defaults to the panel's own MEDIAN gated integrated +// loudness and the number on each row is a deviation in LU. Median, never +// mean: one person on a laptop mic at -35 LUFS would drag a mean far enough +// to fail everybody else, which is the opposite of useful. +// +// WHY THE LAYOUT MATHS IS IN HERE TOO. This repo has no headless GPU harness +// and has ruled against building one -- an offscreen Qt harness certified the +// Talkback dock's layout three times and was wrong three times. The sanctioned +// approach is to extract the decision into a pure header and unit-test that, +// the way tests/tile-shape-test.cpp reproduces the tile shader's crop +// arithmetic in plain C++. So the row and bar rectangles are decided here and +// the renderer only fills them. +// +// Pure: no libobs, no Qt, no Zoom SDK. + +#include +#include +#include +#include +#include +#include + +// One live audio source's measurement, flattened for the board. Every "has_" +// flag is load-bearing: "this panelist has not produced a measurable check" +// is a different statement from any loudness value, and collapsing it to a +// sentinel number is how a board ends up confidently reporting -70 LUFS for +// somebody who simply has not spoken yet. +struct LoudnessReading { + std::string source_uuid; + std::string display_name; + uint32_t participant_id = 0; + bool subscribed = false; + bool has_short_term = false; + double short_term_lufs = 0.0; + bool has_integrated = false; + double integrated_lufs = 0.0; + uint64_t gated_blocks = 0; +}; + +// Default: the panel's own median. The presets exist because a show sometimes +// has a delivery spec, but matching each other is the actual goal here, which +// is why PanelMedian is first and is the default. +enum class LoudnessReference { + PanelMedian = 0, + EbuR128 = 1, // -23 LUFS + AtscA85 = 2, // -24 LKFS + Streaming = 3, // -16 LUFS +}; + +enum class LoudnessRowStatus { + NoAudio = 0, // subscribed to nobody, or nobody has spoken + Measuring = 1, // audible, but not enough gated blocks for a verdict + Pass = 2, + Loud = 3, + Quiet = 4, +}; + +// Minimum gated blocks before a reading is treated as a check rather than a +// noise. At a 100 ms hop this is 3 s of gated speech; the spec's 20 s mic +// check yields roughly 200. Set low enough that an operator sees a verdict +// while the panelist is still talking, high enough that a cough or a chair +// scrape cannot set the panel reference for everybody. +constexpr uint64_t kLoudnessBoardMinBlocks = 30; + +// +/- this many LU from the reference still passes. 2 LU is below the ~3 LU +// step most listeners call "noticeably louder", so a passing board really is +// a matched panel. +constexpr double kLoudnessBoardDefaultToleranceLu = 2.0; + +// The deviation at which the bar is full. Beyond it the bar clamps rather +// than growing, because past 6 LU the exact number stops mattering: the +// answer is already "fix this microphone". +constexpr double kLoudnessBoardFullScaleLu = 6.0; + +constexpr int kLoudnessBoardHeaderPx = 28; +constexpr int kLoudnessBoardRowGapPx = 4; + +struct LoudnessBoardRow { + std::string name; + std::string detail; // short status text for the row + bool has_deviation = false; + double deviation_lu = 0.0; + bool has_short_term = false; + double short_term_lufs = 0.0; + bool has_integrated = false; + double integrated_lufs = 0.0; + LoudnessRowStatus status = LoudnessRowStatus::NoAudio; +}; + +struct LoudnessBoardModel { + bool has_reference = false; + double reference_lufs = 0.0; + LoudnessReference reference_kind = LoudnessReference::PanelMedian; + std::vector rows; + // Changes only when something an operator can SEE changed. The consumer + // rebuilds its child text sources off this, and the Talkback dock's + // 2026-08-29 live defect -- a merely reordered roster rebuilding the + // whole widget list several times a second and eating the operator's + // clicks -- is why it is derived from sorted content and never from + // input order. + std::string signature; +}; + +// The fixed presets. PanelMedian has no fixed value and returns false. +inline bool loudness_reference_fixed_target(LoudnessReference kind, double *out) +{ + switch (kind) { + case LoudnessReference::EbuR128: *out = -23.0; return true; + case LoudnessReference::AtscA85: *out = -24.0; return true; + case LoudnessReference::Streaming: *out = -16.0; return true; + case LoudnessReference::PanelMedian: + default: return false; + } +} + +// Median of the gated integrated loudness of everyone who has actually +// produced a check. Even counts average the two middle values, which is the +// ordinary definition and keeps a two-person panel from arbitrarily electing +// one of them as the reference. +inline bool loudness_panel_median(const std::vector &readings, + uint64_t min_blocks, double *out) +{ + std::vector values; + values.reserve(readings.size()); + for (const LoudnessReading &r : readings) { + if (!r.has_integrated) continue; + if (r.gated_blocks < min_blocks) continue; + if (!std::isfinite(r.integrated_lufs)) continue; + values.push_back(r.integrated_lufs); + } + if (values.empty()) return false; + std::sort(values.begin(), values.end()); + const size_t n = values.size(); + *out = (n % 2 == 1) ? values[n / 2] + : 0.5 * (values[n / 2 - 1] + values[n / 2]); + return true; +} + +inline const char *loudness_row_status_text(LoudnessRowStatus s) +{ + switch (s) { + case LoudnessRowStatus::NoAudio: return "no audio"; + case LoudnessRowStatus::Measuring: return "measuring"; + case LoudnessRowStatus::Pass: return "ok"; + case LoudnessRowStatus::Loud: return "too loud"; + case LoudnessRowStatus::Quiet: return "too quiet"; + default: return ""; + } +} + +inline LoudnessBoardModel loudness_board_build( + const std::vector &readings, + LoudnessReference kind, double tolerance_lu, uint64_t min_blocks) +{ + LoudnessBoardModel model; + model.reference_kind = kind; + if (!(tolerance_lu > 0.0)) tolerance_lu = kLoudnessBoardDefaultToleranceLu; + + double reference = 0.0; + if (loudness_reference_fixed_target(kind, &reference)) { + // A fixed target does not depend on the panel, so it survives a panel + // nobody has spoken on. The median does not, and must not be invented. + model.has_reference = true; + model.reference_lufs = reference; + } else if (loudness_panel_median(readings, min_blocks, &reference)) { + model.has_reference = true; + model.reference_lufs = reference; + } + + // Ordered by CONTENT alone -- name, then uuid to break a duplicate-name + // tie -- so a roster that merely reorders produces an identical board. + std::vector ordered; + ordered.reserve(readings.size()); + for (const LoudnessReading &r : readings) ordered.push_back(&r); + std::sort(ordered.begin(), ordered.end(), + [](const LoudnessReading *a, const LoudnessReading *b) { + if (a->display_name != b->display_name) + return a->display_name < b->display_name; + return a->source_uuid < b->source_uuid; + }); + + model.rows.reserve(ordered.size()); + for (const LoudnessReading *r : ordered) { + LoudnessBoardRow row; + row.name = r->display_name.empty() + ? (r->participant_id != 0 + ? "ID " + std::to_string(r->participant_id) + : std::string("- unassigned -")) + : r->display_name; + row.has_short_term = r->has_short_term; + row.short_term_lufs = r->short_term_lufs; + row.has_integrated = r->has_integrated; + row.integrated_lufs = r->integrated_lufs; + + if (!r->has_integrated || r->gated_blocks == 0) { + row.status = LoudnessRowStatus::NoAudio; + } else if (r->gated_blocks < min_blocks) { + row.status = LoudnessRowStatus::Measuring; + } else if (model.has_reference) { + row.has_deviation = true; + row.deviation_lu = r->integrated_lufs - model.reference_lufs; + if (row.deviation_lu > tolerance_lu) + row.status = LoudnessRowStatus::Loud; + else if (row.deviation_lu < -tolerance_lu) + row.status = LoudnessRowStatus::Quiet; + else + row.status = LoudnessRowStatus::Pass; // boundary is inclusive + } else { + row.status = LoudnessRowStatus::Measuring; + } + row.detail = loudness_row_status_text(row.status); + model.rows.push_back(std::move(row)); + } + + // Deviation is quantised to 0.1 LU in the signature: the renderer prints + // one decimal place, so a change smaller than that is invisible and must + // not cost a text-source rebuild. + std::string sig; + sig.reserve(model.rows.size() * 24 + 16); + char buf[64]; + std::snprintf(buf, sizeof(buf), "R%d:%s%.1f|", + static_cast(kind), model.has_reference ? "" : "x", + model.has_reference ? model.reference_lufs : 0.0); + sig += buf; + for (const LoudnessBoardRow &row : model.rows) { + sig += row.name; + std::snprintf(buf, sizeof(buf), "|%d|%s%.1f;", + static_cast(row.status), + row.has_deviation ? "" : "x", + row.has_deviation ? row.deviation_lu : 0.0); + sig += buf; + } + model.signature = std::move(sig); + return model; +} + +struct LoudnessBoardRect { + int x = 0; + int y = 0; + int w = 0; + int h = 0; +}; + +// One row's band. A zero-size result means "do not draw", which is what every +// degenerate input produces -- the renderer checks w/h rather than +// re-validating the arguments it just passed in. +inline LoudnessBoardRect loudness_board_row_rect(int canvas_w, int canvas_h, + size_t row_count, + size_t row_index) +{ + LoudnessBoardRect r; + if (canvas_w <= 0 || canvas_h <= 0 || row_count == 0 || + row_index >= row_count) + return r; + const int body_top = kLoudnessBoardHeaderPx; + const int body_h = canvas_h - body_top; + if (body_h <= 0) return r; + const int slot = body_h / static_cast(row_count); + if (slot <= 0) return r; + const int h = slot - kLoudnessBoardRowGapPx; + r.x = 0; + r.w = canvas_w; + r.y = body_top + slot * static_cast(row_index); + r.h = (h > 0) ? h : slot; + return r; +} + +// The deviation bar, growing right (louder) or left (quieter) from the centre +// of the row's right half. Clamped at full scale rather than allowed to run +// off the canvas: past 6 LU the exact number has stopped mattering. +inline LoudnessBoardRect loudness_board_bar_rect(const LoudnessBoardRect &row, + double deviation_lu, + double full_scale_lu) +{ + LoudnessBoardRect r; + if (row.w <= 0 || row.h <= 0 || !(full_scale_lu > 0.0)) return r; + const int meter_w = row.w / 2; // right half is the meter + const int meter_x = row.x + row.w - meter_w; + const int half = meter_w / 2; + const int centre = meter_x + half; + + double d = deviation_lu; + if (!std::isfinite(d)) d = 0.0; + if (d > full_scale_lu) d = full_scale_lu; + if (d < -full_scale_lu) d = -full_scale_lu; + + const int len = static_cast(std::fabs(d) / full_scale_lu * + static_cast(half) + 0.5); + r.y = row.y; + r.h = row.h; + r.w = len; + r.x = (d >= 0.0) ? centre : centre - len; + return r; +} diff --git a/tests/loudness-board-test.cpp b/tests/loudness-board-test.cpp new file mode 100644 index 00000000..108c1d89 --- /dev/null +++ b/tests/loudness-board-test.cpp @@ -0,0 +1,303 @@ +// tests/loudness-board-test.cpp +// The readiness board: what an operator actually reads during a mic check. +// +// The product claim is relative, not absolute. An operator does not primarily +// care that a panelist hits -23 LUFS; they care that panelist A is not 6 LU +// louder than panelist B. So the headline number is deviation from the panel +// MEDIAN of gated integrated loudness -- median, because one panelist on a +// laptop mic at -35 LUFS must not drag the reference everyone else is judged +// against, which is exactly what a mean does. +// +// The layout arithmetic is pinned here too rather than looked at on screen. +// This repo has no headless GPU harness and has ruled against building one +// (an offscreen Qt harness "certified it three times and was wrong three +// times"); the sanctioned approach is to extract the decision into a pure +// header and unit-test that, the way tests/tile-shape-test.cpp reproduces the +// shader's crop arithmetic. +#include "loudness-board.h" + +#include +#include +#include +#include + +static int failures = 0; + +static void check(bool ok, const char *message) +{ + if (!ok) { + std::cerr << "FAIL: " << message << "\n"; + ++failures; + } +} + +static bool near(double a, double b, double tol) +{ + return std::fabs(a - b) <= tol; +} + +static LoudnessReading measured(const char *name, double integrated, + double short_term = -20.0, + uint64_t blocks = 200) +{ + LoudnessReading r; + r.source_uuid = std::string("uuid_") + name; + r.display_name = name; + r.participant_id = 1; + r.subscribed = true; + r.has_short_term = true; + r.short_term_lufs = short_term; + r.has_integrated = true; + r.integrated_lufs = integrated; + r.gated_blocks = blocks; + return r; +} + +int main() +{ + // ── Median, not mean ─────────────────────────────────────────────────── + { + std::vector panel = { + measured("Ana", -18.0), measured("Ben", -21.0), + measured("Cara", -23.0), measured("Dev", -24.0), + measured("Erik", -30.0), + }; + double median = 0.0; + check(loudness_panel_median(panel, kLoudnessBoardMinBlocks, &median), + "no panel median was produced from five measured panelists"); + check(near(median, -23.0, 1e-9), + "the panel reference is not the median -- the mean of this " + "panel is -23.2, and Erik at -30 is exactly the outlier the " + "median exists to survive"); + } + + // ── An even panel averages the two middle values ─────────────────────── + { + std::vector panel = { + measured("Ana", -18.0), measured("Ben", -21.0), + measured("Cara", -23.0), measured("Dev", -24.0), + }; + double median = 0.0; + check(loudness_panel_median(panel, kLoudnessBoardMinBlocks, &median), + "no median from an even-sized panel"); + check(near(median, -22.0, 1e-9), + "an even-sized panel's median was not the mean of the two " + "middle values"); + } + + // ── Unmeasured panelists must not vote on the reference ──────────────── + { + LoudnessReading quiet; + quiet.source_uuid = "uuid_Fay"; + quiet.display_name = "Fay"; + quiet.subscribed = true; + // never spoke: no integrated value at all + std::vector panel = { + measured("Ana", -18.0), measured("Ben", -22.0), quiet, + }; + double median = 0.0; + check(loudness_panel_median(panel, kLoudnessBoardMinBlocks, &median), + "a panel with one silent member produced no median"); + check(near(median, -20.0, 1e-9), + "a panelist with no integrated reading was counted in the " + "median -- a person who has not spoken is not a data point"); + } + + // ── A too-short check does not count either ──────────────────────────── + { + std::vector panel = { + measured("Ana", -18.0, -18.0, 200), + measured("Ben", -22.0, -22.0, 200), + measured("Cough", -5.0, -5.0, 4), // four blocks: 400 ms + }; + double median = 0.0; + check(loudness_panel_median(panel, kLoudnessBoardMinBlocks, &median), + "no median produced"); + check(near(median, -20.0, 1e-9), + "a 400 ms cough set the panel reference -- the minimum gated " + "block count is not being applied"); + } + + // ── No measurable panelist means NO reference, not zero ──────────────── + { + std::vector panel; + double median = 0.0; + check(!loudness_panel_median(panel, kLoudnessBoardMinBlocks, &median), + "an empty panel produced a reference value"); + } + + // ── Deviation sign, and status ───────────────────────────────────────── + { + std::vector panel = { + measured("Ana", -18.0), measured("Ben", -21.0), + measured("Cara", -23.0), measured("Dev", -24.0), + measured("Erik", -30.0), + }; + const LoudnessBoardModel m = loudness_board_build( + panel, LoudnessReference::PanelMedian, + kLoudnessBoardDefaultToleranceLu, kLoudnessBoardMinBlocks); + check(m.has_reference && near(m.reference_lufs, -23.0, 1e-9), + "the built model's reference is not the panel median"); + check(m.rows.size() == 5, "the board did not produce one row per panelist"); + // Rows are ordered by name from CONTENT alone. + check(m.rows[0].name == "Ana" && m.rows[4].name == "Erik", + "rows are not in deterministic name order"); + check(near(m.rows[0].deviation_lu, 5.0, 1e-9), + "a panelist 5 LU above the median did not report +5 LU -- " + "louder than the reference must be POSITIVE"); + check(near(m.rows[4].deviation_lu, -7.0, 1e-9), + "a panelist 7 LU below the median did not report -7 LU"); + check(m.rows[0].status == LoudnessRowStatus::Loud, + "+5 LU was not flagged as too loud at a 2 LU tolerance"); + check(m.rows[4].status == LoudnessRowStatus::Quiet, + "-7 LU was not flagged as too quiet"); + check(m.rows[2].status == LoudnessRowStatus::Pass, + "the panelist sitting exactly on the median did not pass"); + check(m.rows[1].status == LoudnessRowStatus::Pass, + "-21 against a -23 median is +2 LU, exactly the tolerance, and " + "must pass -- the boundary is inclusive"); + } + + // ── Fixed-target presets ─────────────────────────────────────────────── + { + std::vector panel = { measured("Ana", -18.0) }; + const LoudnessBoardModel r128 = loudness_board_build( + panel, LoudnessReference::EbuR128, 2.0, kLoudnessBoardMinBlocks); + check(r128.has_reference && near(r128.reference_lufs, -23.0, 1e-9), + "EBU R128 preset is not -23 LUFS"); + check(near(r128.rows[0].deviation_lu, 5.0, 1e-9), + "-18 against the R128 target is not +5 LU"); + + const LoudnessBoardModel a85 = loudness_board_build( + panel, LoudnessReference::AtscA85, 2.0, kLoudnessBoardMinBlocks); + check(near(a85.reference_lufs, -24.0, 1e-9), + "ATSC A/85 preset is not -24 LKFS"); + + const LoudnessBoardModel str = loudness_board_build( + panel, LoudnessReference::Streaming, 2.0, kLoudnessBoardMinBlocks); + check(near(str.reference_lufs, -16.0, 1e-9), + "the streaming preset is not -16 LUFS"); + } + + // ── A fixed target works with NOBODY measured; the median does not ───── + { + LoudnessReading silent; + silent.source_uuid = "uuid_Ana"; + silent.display_name = "Ana"; + silent.subscribed = true; + std::vector panel = { silent }; + + const LoudnessBoardModel med = loudness_board_build( + panel, LoudnessReference::PanelMedian, 2.0, kLoudnessBoardMinBlocks); + check(!med.has_reference, + "a panel median was invented from a panel nobody has spoken on"); + check(med.rows.size() == 1 && !med.rows[0].has_deviation && + med.rows[0].status == LoudnessRowStatus::NoAudio, + "a silent panelist was given a deviation"); + + const LoudnessBoardModel fixed = loudness_board_build( + panel, LoudnessReference::EbuR128, 2.0, kLoudnessBoardMinBlocks); + check(fixed.has_reference, + "a FIXED target disappeared because nobody had spoken -- the " + "target does not depend on the panel"); + check(!fixed.rows[0].has_deviation, + "a silent panelist got a deviation against a fixed target"); + } + + // ── Measuring: subscribed and audible, but not enough blocks yet ─────── + { + std::vector panel = { + measured("Ana", -20.0, -20.0, 200), + measured("Ben", -20.0, -20.0, 5), + }; + const LoudnessBoardModel m = loudness_board_build( + panel, LoudnessReference::PanelMedian, 2.0, kLoudnessBoardMinBlocks); + check(m.rows[1].status == LoudnessRowStatus::Measuring, + "a panelist with 5 gated blocks was given a verdict rather than " + "reported as still measuring"); + check(!m.rows[1].has_deviation, + "a still-measuring panelist was given a deviation"); + } + + // ── The signature changes on content and NOT on input order ──────────── + // The Talkback dock shipped a live defect (2026-08-29) where a merely + // REORDERED roster rebuilt the whole widget list several times a second + // and threw away the operator's clicks. The board's consumer rebuilds + // child text sources off this signature, so the same rule applies here. + { + std::vector a = { + measured("Ana", -20.0), measured("Ben", -22.0), + }; + std::vector b = { a[1], a[0] }; // same set, reordered + const LoudnessBoardModel ma = loudness_board_build( + a, LoudnessReference::PanelMedian, 2.0, kLoudnessBoardMinBlocks); + const LoudnessBoardModel mb = loudness_board_build( + b, LoudnessReference::PanelMedian, 2.0, kLoudnessBoardMinBlocks); + check(ma.signature == mb.signature, + "reordering the input changed the board signature -- the " + "consumer will rebuild its text children on every roster event"); + + std::vector c = { + measured("Ana", -20.0), measured("Ben", -26.0), + }; + const LoudnessBoardModel mc = loudness_board_build( + c, LoudnessReference::PanelMedian, 2.0, kLoudnessBoardMinBlocks); + check(ma.signature != mc.signature, + "a 4 LU change in one panelist did not change the signature"); + } + + // ── Layout: rows tile the canvas below the header, in order ──────────── + { + const LoudnessBoardRect r0 = loudness_board_row_rect(640, 360, 4, 0); + const LoudnessBoardRect r3 = loudness_board_row_rect(640, 360, 4, 3); + check(r0.x == 0 && r0.w == 640, "a row does not span the canvas width"); + check(r0.y == kLoudnessBoardHeaderPx, + "the first row does not start below the header band"); + check(r0.h == 79, + "a 4-row board on a 640x360 canvas did not give 79 px rows " + "((360-28)/4 - 4 gap)"); + check(r3.y == kLoudnessBoardHeaderPx + 83 * 3, + "row 3 is not at the fourth slot"); + check(r3.y + r3.h <= 360, + "the last row overflows the canvas"); + const LoudnessBoardRect bad = loudness_board_row_rect(640, 360, 4, 9); + check(bad.w == 0 && bad.h == 0, + "an out-of-range row index produced a drawable rect"); + const LoudnessBoardRect none = loudness_board_row_rect(640, 360, 0, 0); + check(none.w == 0 && none.h == 0, + "a zero-row board produced a drawable rect"); + } + + // ── Layout: the bar grows from the centre of the right half ──────────── + { + const LoudnessBoardRect row{0, 28, 640, 79}; + const LoudnessBoardRect zero = + loudness_board_bar_rect(row, 0.0, kLoudnessBoardFullScaleLu); + check(zero.w == 0 && zero.x == 480, + "a zero deviation did not collapse to nothing at the centre " + "line (x=480 on a 640 px row)"); + + const LoudnessBoardRect hot = + loudness_board_bar_rect(row, 3.0, kLoudnessBoardFullScaleLu); + check(hot.x == 480 && hot.w == 80, + "+3 LU of a 6 LU full scale did not fill half the right side"); + + const LoudnessBoardRect cold = + loudness_board_bar_rect(row, -6.0, kLoudnessBoardFullScaleLu); + check(cold.x == 320 && cold.w == 160, + "-6 LU did not fill the left half of the meter"); + + const LoudnessBoardRect clipped = + loudness_board_bar_rect(row, 40.0, kLoudnessBoardFullScaleLu); + check(clipped.x == 480 && clipped.w == 160 && + clipped.x + clipped.w <= 640, + "an off-the-scale deviation drew past the canvas instead of " + "clamping at full scale"); + check(cold.y == row.y && cold.h == row.h, + "the bar's vertical extent does not match its row"); + } + + if (failures == 0) + std::cout << "loudness-board: all tests passed\n"; + return failures == 0 ? 0 : 1; +} From b12e03d2fd953dd7eb93594c7545b31d3458b880 Mon Sep 17 00:00:00 2001 From: Grok Date: Sat, 5 Sep 2026 16:52:44 -0400 Subject: [PATCH 09/15] feat(loudness): meter every participant on the audio lane and expose per-source readings --- src/zoom-participant-audio-source.cpp | 112 ++++++++++++++++++++++++++ src/zoom-participant-audio-source.h | 18 +++++ tests/audio-loudness-test.cpp | 39 +++++++++ 3 files changed, 169 insertions(+) diff --git a/src/zoom-participant-audio-source.cpp b/src/zoom-participant-audio-source.cpp index 33cd4c45..f57ae5c8 100644 --- a/src/zoom-participant-audio-source.cpp +++ b/src/zoom-participant-audio-source.cpp @@ -1,5 +1,6 @@ #include "zoom-participant-audio-source.h" +#include "audio-loudness.h" #include "audio-silence-fade.h" #include "audio-subscription-state.h" #include "audio-timeline.h" @@ -97,6 +98,18 @@ struct CoreVideoAudioSource { // Was the last published buffer true digital silence? See // src/audio-silence-fade.h. Same ownership as `timeline`. bool prev_was_silent = false; + // BS.1770-4 loudness for this participant. Same ownership as `timeline` + // and `prev_was_silent`: advanced by the audio lane thread inside + // output_audio_frame(), which holds ctx->mtx for the whole drain, and + // read under that same mutex by corevideo_loudness_readings(). + LoudnessMeter loudness; + // Set by any thread, consumed by the audio lane at the next slot. A reset + // has to land on a hop boundary the meter itself controls, so it is a + // request rather than a direct call: touching the meter from the caller's + // thread would race the drain that is filling it. + std::atomic loudness_reset_requested{false}; + // Display name, cached on the roster callback. Guarded by ctx->mtx. + std::string display_name; // Next ring slot this source will drain. Only the engine reader thread // touches it, the same thread that owns `timeline`. uint32_t read_index = 0; @@ -187,6 +200,49 @@ std::vector corevideo_audio_source_infos() return out; } +std::vector corevideo_loudness_readings() +{ + std::vector out; + // Same lock order as corevideo_audio_source_infos(): g_sources_mtx first, + // then each source's own mutex. The audio lane takes only ctx->mtx and + // never touches g_sources_mtx, so this can never invert. + std::lock_guard lk(g_sources_mtx); + out.reserve(g_sources.size()); + for (CoreVideoAudioSource *ctx : g_sources) { + if (!ctx) continue; + LoudnessReading r; + r.source_uuid = ctx->source_uuid; + r.participant_id = + ctx->current_participant_id.load(std::memory_order_acquire); + r.subscribed = ctx->subscribed.load(std::memory_order_acquire); + { + std::lock_guard ctx_lk(ctx->mtx); + r.display_name = ctx->display_name; + r.has_short_term = loudness_meter_short_term(ctx->loudness, + &r.short_term_lufs); + r.has_integrated = loudness_meter_integrated(ctx->loudness, + &r.integrated_lufs); + r.gated_blocks = loudness_meter_gated_blocks(ctx->loudness); + } + out.push_back(std::move(r)); + } + return out; +} + +void corevideo_reset_loudness_windows() +{ + std::lock_guard lk(g_sources_mtx); + for (CoreVideoAudioSource *ctx : g_sources) { + if (ctx) + ctx->loudness_reset_requested.store(true, + std::memory_order_release); + } + blog(LOG_INFO, + "[obs-zoom-plugin] CoreVideo loudness: mic-check windows reset on %d " + "source(s)", + static_cast(g_sources.size())); +} + static uint32_t target_participant_id(const CoreVideoAudioSource *ctx) { if (!ctx) return 0; @@ -264,6 +320,11 @@ static void unsubscribe_audio(CoreVideoAudioSource *ctx) std::lock_guard lk(ctx->mtx); audio_timeline_reset(ctx->timeline); ctx->read_started = false; + // A new subscription is a new panelist's mic check -- or the same + // one after a gap of unknown length. Either way the previous + // window's gated blocks describe audio that is not this check. + loudness_meter_reset_window(ctx->loudness); + ctx->display_name.clear(); } } @@ -344,6 +405,8 @@ static void forget_subscription_for_new_engine(CoreVideoAudioSource *ctx) std::lock_guard lk(ctx->mtx); audio_timeline_reset(ctx->timeline); ctx->read_started = false; + loudness_meter_reset_window(ctx->loudness); + ctx->display_name.clear(); } if (!was_subscribed) return; blog(LOG_INFO, @@ -381,6 +444,7 @@ static void maybe_resubscribe_for_roster(CoreVideoAudioSource *ctx) // live on 2026-08-16. An absent target resolves to 0, which is the // already-documented "nobody resolved yet, ask again next tick" case. bool held_participant_present = true; + std::string cached_name; const bool needs_roster = (state.subscribed && state.participant_id != 0) || target != 0; if (needs_roster) { @@ -395,6 +459,27 @@ static void maybe_resubscribe_for_roster(CoreVideoAudioSource *ctx) held_participant_present = present(state.participant_id); if (target != 0 && !present(target)) target = 0; + + // The ONE place a display name is resolved for this source. This + // function runs on the engine's roster callback, which is exactly as + // often as a name can change, and it has already paid for the roster + // copy above. The readiness board reads the cached string instead of + // calling roster() itself, because roster() deep-copies every + // ParticipantInfo under the engine client's hot mutex and the board + // asks ten times a second. + const uint32_t name_for = target != 0 ? target : state.participant_id; + if (name_for != 0) { + for (const ParticipantInfo &p : roster) { + if (p.user_id == name_for) { + cached_name = p.display_name; + break; + } + } + } + } + { + std::lock_guard lk(ctx->mtx); + ctx->display_name = cached_name; } switch (audio_resubscribe_action(ctx->kind, active, state, target, @@ -711,6 +796,33 @@ static void output_audio_frame(CoreVideoAudioSource *ctx, sample_rate / 1000 * kAudioResumeFadeMs); } ctx->prev_was_silent = cur_silent; + + // ── BS.1770-4 metering ──────────────────────────────────────── + // HERE, inside the per-slot loop, and nowhere else. A media event + // is a coalescing PROMPT, not a payload: one wakeup routinely + // carries several ring slots and this loop drains until the ring + // is seen empty. Measuring "the buffer that woke us" would + // silently discard most of the audio and read low by a + // load-dependent amount -- the worst shape of wrong, because it + // looks fine on an idle box. + // + // Fed with the WIRE format (`pcm`, `channels`, `sample_rate`) + // rather than the publish format assembled below: the operator's + // Mono/Stereo choice is a routing decision for OBS, and a + // mono-summed copy of a stereo panelist would read 3 LU different + // from the same person carried as stereo. The measurement has to + // describe what the panelist SENT. + // + // The resume fade above has already been applied to these + // samples, which is correct: it is part of what we publish, it is + // 3 ms long, and excluding it would mean measuring audio that + // nobody hears. + if (ctx->loudness_reset_requested.exchange( + false, std::memory_order_acq_rel)) { + loudness_meter_reset_window(ctx->loudness); + } + loudness_meter_feed_int16(ctx->loudness, pcm_mut, pcm_frames, + channels, sample_rate); const auto *pcm = pcm_mut; obs_source_audio audio = {}; audio.samples_per_sec = sample_rate; diff --git a/src/zoom-participant-audio-source.h b/src/zoom-participant-audio-source.h index 0063d0ae..344da0e9 100644 --- a/src/zoom-participant-audio-source.h +++ b/src/zoom-participant-audio-source.h @@ -4,6 +4,8 @@ #include #include +#include "loudness-board.h" + void zoom_participant_audio_source_register(); // One live CoreVideoAudioSource, flattened for the control API and any other @@ -41,3 +43,19 @@ std::vector corevideo_audio_source_infos(); // CoreVideoAudioSource, including ones already created. Returns the value // actually applied after clamping. uint32_t corevideo_set_global_audio_delay_ms(uint32_t delay_ms); + +// One BS.1770-4 reading per live CoreVideoAudioSource, for the readiness +// board. Safe to call from any thread; takes g_sources_mtx and then each +// source's own mutex, in that order and never the reverse. +// +// The display name here is a CACHED copy, refreshed on the engine's roster +// callback. ZoomEngineClient::roster() deep-copies every ParticipantInfo -- +// strings included -- under the client's hot mutex, so resolving a name on +// the audio path (about a hundred buffers a second, per source) would put a +// full roster copy on the media path. +std::vector corevideo_loudness_readings(); + +// Starts every live source's mic-check window over. Integrated loudness is +// scoped to ONE panelist's check, not the session: without this the number +// is polluted by whoever spoke before them on the same source. +void corevideo_reset_loudness_windows(); diff --git a/tests/audio-loudness-test.cpp b/tests/audio-loudness-test.cpp index e1621f29..144631a1 100644 --- a/tests/audio-loudness-test.cpp +++ b/tests/audio-loudness-test.cpp @@ -410,6 +410,45 @@ int main() "momentary differed between chunked and whole feeding"); } + // ── The wire format the tap actually hands over ──────────────────────── + // output_audio_frame() reads ShmAudioHeader::sample_rate and ::channels + // per slot and can see them CHANGE mid-source: the engine restamps + // whatever GetSampleRate() returned, and an operator flipping a target + // between Mix (stereo) and Isolated (mono) changes the channel count on + // the same live subscription. The meter must follow that without + // carrying one format's filter history into the other's measurement. + { + LoudnessMeter m; + feed_sine(m, 48000, std::sqrt(2.0) * 0.1, 1000.0, 2.0); + check(m.sample_rate == 48000 && m.channels == 1, + "the meter did not adopt the first buffer's wire format"); + + // Same tone, now arriving at 32 kHz: the meter must re-derive rather + // than keep filtering with 48 kHz coefficients. + feed_sine(m, 32000, std::sqrt(2.0) * 0.1, 1000.0, 4.0); + check(m.sample_rate == 32000, + "a mid-stream rate change did not reconfigure the meter"); + double lufs = 0.0; + check(loudness_meter_momentary(m, &lufs) && near(lufs, -19.98, 0.15), + "after a mid-stream rate change the reading is wrong -- the " + "coefficients did not follow"); + check(m.hop_frames == 3200, + "the 100 ms hop is not 3200 frames at 32 kHz -- the hop length " + "is fixed in samples instead of in time"); + } + + // ── A null or empty buffer is a no-op, not a crash ──────────────────── + // The drain loop can hand over a slot it failed to copy. + { + LoudnessMeter m; + loudness_meter_feed_int16(m, nullptr, 480, 1, 48000); + std::vector one(1, 0); + loudness_meter_feed_int16(m, one.data(), 0, 1, 48000); + loudness_meter_feed_int16(m, one.data(), 1, 0, 48000); + check(m.hop_total == 0, + "a degenerate feed advanced the measurement"); + } + if (failures == 0) std::cout << "audio-loudness: all tests passed\n"; return failures == 0 ? 0 : 1; From 1e4b4add79e0806f4efb6e94c07685129904171b Mon Sep 17 00:00:00 2001 From: Grok Date: Sat, 5 Sep 2026 16:53:11 -0400 Subject: [PATCH 10/15] docs(loudness): document Task 5 audio-lane metering wiring in CLAUDE.md --- CLAUDE.md | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/CLAUDE.md b/CLAUDE.md index de92e843..e2224f5a 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1165,6 +1165,34 @@ Every one of these is documented at length where it lives; the list is the map. rather than copied, because an instrument painting its own approximation would verify a layout the product does not have. +- **Panelist loudness meter is fed on the audio lane, per source, from the + WIRE format** (`src/zoom-participant-audio-source.cpp`, feat/panelist-feedback + Task 5): each `CoreVideoAudioSource` now owns a `LoudnessMeter` + (`src/audio-loudness.h`, a pure BS.1770-4 header from earlier tasks in this + feature). `output_audio_frame()` feeds it inside the per-slot drain loop -- + same rule as everywhere else in this file: a media event is a coalescing + PROMPT, feeding "the buffer that woke us" instead of every drained slot + reads low by a load-dependent amount. It is fed the pre-publish WIRE PCM + (post resume-fade, pre Mono/Stereo assembly) because the operator's + Mono/Stereo routing choice for OBS is not what the panelist actually sent, + and mono-summing a stereo feed reads ~3 LU different from the same person + carried as stereo. A reset request is an atomic flag consumed on the audio + lane's own next slot, never called directly from another thread, because a + reset has to land on a hop boundary the meter itself controls. Display name + is cached (`ctx->display_name`, guarded by `ctx->mtx`) from the roster + callback's already-fetched roster copy in `maybe_resubscribe_for_roster()` + -- never re-resolved per buffer, since `ZoomEngineClient::roster()` + deep-copies every `ParticipantInfo` under a hot mutex and the readiness + board this feeds polls at ~10 Hz. `corevideo_loudness_readings()` / + `corevideo_reset_loudness_windows()` mirror `corevideo_audio_source_infos()`'s + registry pattern exactly, including its `g_sources_mtx`-then-`ctx->mtx` lock + order; the audio lane never touches `g_sources_mtx`, so the order cannot + invert. The window resets on every (un)subscribe transition + (`unsubscribe_audio()`, `forget_subscription_for_new_engine()`), because + integrated loudness is scoped to one panelist's mic check, not the source's + whole lifetime -- without it a re-subscribed source's number is polluted by + whoever spoke before on the same source. + ## Live testing against a real meeting The control API (TCP line-JSON, `127.0.0.1:19870`, no HTTP) drives a full From 426632fdda0c4aea2b636d29a8e9542a76c0d07d Mon Sep 17 00:00:00 2001 From: Grok Date: Sat, 5 Sep 2026 17:06:34 -0400 Subject: [PATCH 11/15] feat(loudness): readiness-board meter source drawing deviation bars with the Solid technique Adds the CoreVideo Loudness Meter OBS source: a custom-draw readiness board with one row per live CoreVideo audio source, drawn with the existing Tiles Solid technique (no new .effect file) and private child text sources for panelist name/value labels. Rebuilds its model from corevideo_loudness_readings() at 10 Hz on the graphics thread (video_tick), never per-frame, to avoid contending with the audio lane's g_sources_mtx + per-source mutex on the 60 Hz render path. Registered alongside the Tiles/SuperSource sources in plugin-main.cpp, wired into the obs-zoom-plugin source list in CMakeLists.txt, with locale strings added to data/locale/en-US.ini. Also bounds the row count a canvas can show: loudness_board_visible_rows() and kLoudnessBoardMinRowPx=24 in loudness-board.h cap rows to what stays legible rather than shrinking into an unreadable texture on a large panel. Test correction: the pinned test for the row cap asserted `last.h >= kLoudnessBoardMinRowPx`, but that can never pass for any implementation that also satisfies the pinned `loudness_board_visible_rows(360, 40) == (360 - kLoudnessBoardHeaderPx) / kLoudnessBoardMinRowPx` equality. kLoudnessBoardMinRowPx is the row's SLOT PITCH (its own doc comment says "plus the gap"), not the drawn height loudness_board_row_rect returns after subtracting kLoudnessBoardRowGapPx. Corrected the assertion to `last.h + kLoudnessBoardRowGapPx >= kLoudnessBoardMinRowPx`, which matches the constant's documented meaning and still fails correctly against a broken/uncapped row count. Do not "fix" this back to comparing last.h alone -- verified by hand and by running the original assertion, which fails deterministically (21 < 24) given the pinned capacity of 13. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01WqDL6gcoVvEMXv7Qwcni9n --- CMakeLists.txt | 1 + data/locale/en-US.ini | 10 + src/loudness-board.h | 18 + src/plugin-main.cpp | 4 + src/zoom-loudness-meter-source.cpp | 556 +++++++++++++++++++++++++++++ src/zoom-loudness-meter-source.h | 21 ++ tests/loudness-board-test.cpp | 36 ++ 7 files changed, 646 insertions(+) create mode 100644 src/zoom-loudness-meter-source.cpp create mode 100644 src/zoom-loudness-meter-source.h diff --git a/CMakeLists.txt b/CMakeLists.txt index 3b1fc793..fe9a7fb0 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -269,6 +269,7 @@ if(COREVIDEO_BUILD_PLUGIN) src/talkback-cue.cpp src/zoom-participant-audio-source.cpp src/zoom-supersource.cpp + src/zoom-loudness-meter-source.cpp src/zoom-tiles-background.cpp src/zoom-tiles-audio.cpp src/zoom-tiles-effect.cpp diff --git a/data/locale/en-US.ini b/data/locale/en-US.ini index ddbfeb40..88d6a7eb 100644 --- a/data/locale/en-US.ini +++ b/data/locale/en-US.ini @@ -59,6 +59,16 @@ ZoomParticipantAudio.AudioStereo="Stereo (true stereo when the sender provides i ZoomParticipantAudio.RefreshParticipants="Refresh participant list" CoreVideoActiveSpeakerAudio.Name="CoreVideo Active Speaker Audio" CoreVideoAudienceAudio.Name="CoreVideo Audience Audio (legacy)" +CoreVideoLoudnessMeter.Name="CoreVideo Loudness Meter (readiness board)" +CoreVideoLoudnessMeter.Reference="Reference" +CoreVideoLoudnessMeter.Reference.PanelMedian="Panel median (recommended)" +CoreVideoLoudnessMeter.Reference.R128="EBU R128 (-23 LUFS)" +CoreVideoLoudnessMeter.Reference.A85="ATSC A/85 (-24 LKFS)" +CoreVideoLoudnessMeter.Reference.Streaming="Streaming (-16 LUFS)" +CoreVideoLoudnessMeter.Tolerance="Pass tolerance (LU)" +CoreVideoLoudnessMeter.Width="Width" +CoreVideoLoudnessMeter.Height="Height" +CoreVideoLoudnessMeter.Reset="Reset all mic-check windows" ZoomInterpAudio.Name="Zoom Interpretation Audio" ZoomInterpAudio.Language="Language name (as reported by Zoom, e.g. English, Spanish)" ZoomInterpAudio.Refresh="Refresh" diff --git a/src/loudness-board.h b/src/loudness-board.h index c1676281..81c3fc60 100644 --- a/src/loudness-board.h +++ b/src/loudness-board.h @@ -300,3 +300,21 @@ inline LoudnessBoardRect loudness_board_bar_rect(const LoudnessBoardRect &row, r.x = (d >= 0.0) ? centre : centre - len; return r; } + +// The shortest row that is still a readiness board rather than a texture: a +// name and a number at a size an operator reads across a control room, plus +// the gap. A 25-person Zoom Events room would otherwise produce 13 px rows. +constexpr int kLoudnessBoardMinRowPx = 24; + +// How many rows this canvas can actually show. Beyond it the renderer draws +// the first N (which, because rows are name-ordered, is stable frame to frame +// rather than shuffling) and says so in the header band. +inline size_t loudness_board_visible_rows(int canvas_h, size_t row_count) +{ + if (canvas_h <= kLoudnessBoardHeaderPx || row_count == 0) return 0; + const int body_h = canvas_h - kLoudnessBoardHeaderPx; + const size_t capacity = + static_cast(body_h / kLoudnessBoardMinRowPx); + if (capacity == 0) return 0; + return row_count < capacity ? row_count : capacity; +} diff --git a/src/plugin-main.cpp b/src/plugin-main.cpp index b32c2c19..3e4177a4 100644 --- a/src/plugin-main.cpp +++ b/src/plugin-main.cpp @@ -4,6 +4,7 @@ #include "zoom-source.h" #include "zoom-participant-audio-source.h" #include "zoom-supersource.h" +#include "zoom-loudness-meter-source.h" #include "zoom-engine-client.h" #include "zoom-reconnect.h" #include "zoom-settings.h" @@ -489,6 +490,8 @@ bool obs_module_load(void) zoom_participant_audio_source_register(); zoom_supersource_register(); zoom_supersource_load_gfx(); + corevideo_loudness_meter_source_register(); + corevideo_loudness_meter_load_gfx(); blog(LOG_INFO, "[obs-zoom-plugin] Registered CoreVideo source kinds"); ZoomPluginSettings s = ZoomPluginSettings::load(); @@ -545,6 +548,7 @@ void obs_module_unload(void) } shutdown_corevideo(); zoom_supersource_unload_gfx(); + corevideo_loudness_meter_unload_gfx(); g_dock.clear(); g_iso_panel.clear(); g_output_panel.clear(); diff --git a/src/zoom-loudness-meter-source.cpp b/src/zoom-loudness-meter-source.cpp new file mode 100644 index 00000000..2e1d1fc1 --- /dev/null +++ b/src/zoom-loudness-meter-source.cpp @@ -0,0 +1,556 @@ +#include "zoom-loudness-meter-source.h" + +#include "loudness-board.h" +#include "zoom-participant-audio-source.h" +#include "zoom-tiles-effect.h" + +#include +#include + +#include +#include +#include +#include +#include +#include + +#define PROP_REFERENCE "reference" +#define PROP_TOLERANCE "tolerance_lu" +#define PROP_WIDTH "canvas_width" +#define PROP_HEIGHT "canvas_height" +#define PROP_RESET "btn_reset_windows" + +static const char *kMeterSourceId = "corevideo_loudness_meter_source"; + +// Hard ceiling on child text sources, independent of the canvas cap in +// loudness_board_visible_rows(): each row costs two private sources and OBS +// renders every one of them, so the count is bounded by construction rather +// than by whatever canvas height an operator types in. +static constexpr size_t kMeterMaxRows = 16; + +// 0xAARRGGBB, the same byte order picker_color_to_argb() produces for the +// Tiles wall, so gs_effect_set_color() reads them identically. Deliberately +// flat and high-contrast: this is read at a glance across a room, and the +// spec's legibility rule for anything meter-shaped is chunky segments and +// hard contrast, never hairlines. +static constexpr uint32_t kMeterBgArgb = 0xFF12161Cu; +static constexpr uint32_t kMeterHeaderArgb = 0xFF1E252Fu; +static constexpr uint32_t kMeterRowArgb = 0xFF1A2029u; +static constexpr uint32_t kMeterCentreArgb = 0xFF556070u; +static constexpr uint32_t kMeterPassArgb = 0xFF2FBF6Fu; +static constexpr uint32_t kMeterLoudArgb = 0xFFE04B4Bu; +static constexpr uint32_t kMeterQuietArgb = 0xFFE0A03Cu; +static constexpr uint32_t kMeterIdleArgb = 0xFF3A424Eu; + +// Shared with the Tiles wall by file, not by handle: libobs caches effects +// created from a file, so this second tiles_effect_load() resolves the same +// compiled effect rather than compiling it twice. +static TilesEffect s_meter_effect; +static bool s_meter_pass_failed_logged = false; + +struct meter_row_widgets { + obs_source_t *name = nullptr; + obs_source_t *value = nullptr; + std::string name_text; + std::string value_text; +}; + +struct loudness_meter_source { + obs_source_t *source = nullptr; + + std::atomic canvas_width{640}; + std::atomic canvas_height{360}; + std::atomic reference{0}; // LoudnessReference + std::atomic tolerance_milli_lu{2000}; + + std::mutex mutex; // guards `model` and `rows` + LoudnessBoardModel model; + std::string applied_signature; + meter_row_widgets rows[kMeterMaxRows]; + + float rebuild_accum = 0.0f; +}; + +// ── Text children ─────────────────────────────────────────────────────────── +// +// The board needs real text: panelist display names come from Zoom and this +// project has already been burned by names like "Ronny Hofsoy, Tromso" with +// their real diacritics (the Talkback dock's 400 px tower). A hand-rolled +// bitmap font would reintroduce exactly that class of defect, so the labels +// are OBS's own text sources, created private to this source. +// +// The id is PROBED rather than assumed: OBS ships text_ft2 and text_gdiplus +// on different platforms and has renamed both across versions. A build with +// neither must lose the labels and keep the bars, loudly -- never render an +// empty board with no explanation. +static const char *meter_text_source_id() +{ + static const char *cached = nullptr; + static bool probed = false; + if (probed) return cached; + probed = true; + static const char *candidates[] = { + "text_ft2_source_v2", "text_gdiplus_v3", "text_gdiplus_v2", + "text_ft2_source", "text_gdiplus", + }; + for (const char *id : candidates) { + // obs_get_source_output_flags() returns 0 for an id no module + // registered; a text source always carries OBS_SOURCE_VIDEO. + if (obs_get_source_output_flags(id) != 0) { + cached = id; + break; + } + } + if (!cached) { + blog(LOG_WARNING, + "[obs-zoom-plugin] CoreVideo Loudness Meter: no OBS text source " + "module is available; the board will draw bars without labels"); + } else { + blog(LOG_INFO, + "[obs-zoom-plugin] CoreVideo Loudness Meter: labels will use " + "text source '%s'", + cached); + } + return cached; +} + +static obs_source_t *make_text_child(const char *private_name, int px, + uint32_t argb) +{ + const char *id = meter_text_source_id(); + if (!id) return nullptr; + + obs_data_t *settings = obs_data_create(); + obs_data_t *font = obs_data_create(); + obs_data_set_string(font, "face", "Arial"); + obs_data_set_string(font, "style", "Bold"); + obs_data_set_int(font, "size", px); + obs_data_set_int(font, "flags", 0); + obs_data_set_obj(settings, "font", font); + obs_data_set_string(settings, "text", ""); + // text_gdiplus uses "color"; text_ft2 uses "color1"/"color2". Setting all + // three is harmless on either and avoids a per-id branch that would have + // to be revisited every time OBS renames one. + obs_data_set_int(settings, "color", static_cast(argb)); + obs_data_set_int(settings, "color1", static_cast(argb)); + obs_data_set_int(settings, "color2", static_cast(argb)); + obs_data_release(font); + + obs_source_t *src = obs_source_create_private(id, private_name, settings); + obs_data_release(settings); + return src; +} + +static void set_text_child(obs_source_t *src, const char *text) +{ + if (!src) return; + obs_data_t *settings = obs_data_create(); + obs_data_set_string(settings, "text", text); + obs_source_update(src, settings); + obs_data_release(settings); +} + +// ── Drawing ───────────────────────────────────────────────────────────────── + +static void meter_fill_rect(const LoudnessBoardRect &r, uint32_t argb) +{ + if (r.w <= 0 || r.h <= 0) return; + gs_technique_t *solid = s_meter_effect.tech_solid; + if (!solid || !s_meter_effect.param_color) return; + // The colour must be set BEFORE begin_pass: libobs uploads a pass's + // parameters inside gs_technique_begin_pass() and does not re-upload them + // for later draws in the same pass. Same rule the Tiles border uniforms + // live under. + gs_effect_set_color(s_meter_effect.param_color, argb); + gs_technique_begin(solid); + if (gs_technique_begin_pass(solid, 0)) { + gs_matrix_push(); + gs_matrix_translate3f(static_cast(r.x), + static_cast(r.y), 0.0f); + gs_draw_sprite(nullptr, 0, static_cast(r.w), + static_cast(r.h)); + gs_matrix_pop(); + gs_technique_end_pass(solid); + } else if (!s_meter_pass_failed_logged) { + // Once only. A board that silently stops drawing looks like the + // source went transparent, with no clue why. + s_meter_pass_failed_logged = true; + blog(LOG_ERROR, + "[obs-zoom-plugin] CoreVideo Loudness Meter: " + "gs_technique_begin_pass failed on the Solid technique; the " + "board will not draw"); + } + gs_technique_end(solid); +} + +static uint32_t status_color(LoudnessRowStatus s) +{ + switch (s) { + case LoudnessRowStatus::Pass: return kMeterPassArgb; + case LoudnessRowStatus::Loud: return kMeterLoudArgb; + case LoudnessRowStatus::Quiet: return kMeterQuietArgb; + default: return kMeterIdleArgb; + } +} + +static std::string row_value_text(const LoudnessBoardRow &row) +{ + char buf[96]; + if (row.has_deviation) { + if (row.has_integrated) { + std::snprintf(buf, sizeof(buf), "%+.1f LU %.1f LUFS %s", + row.deviation_lu, row.integrated_lufs, + row.detail.c_str()); + } else { + std::snprintf(buf, sizeof(buf), "%+.1f LU %s", + row.deviation_lu, row.detail.c_str()); + } + } else { + std::snprintf(buf, sizeof(buf), "%s", row.detail.c_str()); + } + return std::string(buf); +} + +static std::string header_text(const LoudnessBoardModel &m, size_t shown, + size_t total) +{ + char buf[160]; + const char *kind = "panel median"; + switch (m.reference_kind) { + case LoudnessReference::EbuR128: kind = "EBU R128"; break; + case LoudnessReference::AtscA85: kind = "ATSC A/85"; break; + case LoudnessReference::Streaming: kind = "streaming"; break; + case LoudnessReference::PanelMedian: + default: break; + } + if (!m.has_reference) { + std::snprintf(buf, sizeof(buf), + "MIC CHECK reference: %s (waiting for a first check)", + kind); + } else if (shown < total) { + std::snprintf(buf, sizeof(buf), + "MIC CHECK reference: %s %.1f LUFS showing %d of %d", + kind, m.reference_lufs, static_cast(shown), + static_cast(total)); + } else { + std::snprintf(buf, sizeof(buf), + "MIC CHECK reference: %s %.1f LUFS", + kind, m.reference_lufs); + } + return std::string(buf); +} + +// ── OBS callbacks ─────────────────────────────────────────────────────────── + +static const char *meter_get_name(void *) +{ + return obs_module_text("CoreVideoLoudnessMeter.Name"); +} + +static void meter_apply_settings(loudness_meter_source *ctx, + obs_data_t *settings) +{ + uint32_t w = static_cast(obs_data_get_int(settings, PROP_WIDTH)); + uint32_t h = static_cast(obs_data_get_int(settings, PROP_HEIGHT)); + if (w < 160) w = 160; + if (w > 3840) w = 3840; + if (h < 90) h = 90; + if (h > 2160) h = 2160; + ctx->canvas_width.store(w, std::memory_order_release); + ctx->canvas_height.store(h, std::memory_order_release); + ctx->reference.store(static_cast( + obs_data_get_int(settings, PROP_REFERENCE)), + std::memory_order_release); + double tol = obs_data_get_double(settings, PROP_TOLERANCE); + if (!(tol > 0.0)) tol = kLoudnessBoardDefaultToleranceLu; + if (tol > 12.0) tol = 12.0; + ctx->tolerance_milli_lu.store(static_cast(tol * 1000.0 + 0.5), + std::memory_order_release); +} + +static void *meter_create(obs_data_t *settings, obs_source_t *source) +{ + auto *ctx = new loudness_meter_source(); + ctx->source = source; + meter_apply_settings(ctx, settings); + + char private_name[64]; + for (size_t i = 0; i < kMeterMaxRows; ++i) { + std::snprintf(private_name, sizeof(private_name), + "corevideo_meter_name_%d", static_cast(i)); + ctx->rows[i].name = make_text_child(private_name, 20, 0xFFF2F5F8u); + std::snprintf(private_name, sizeof(private_name), + "corevideo_meter_value_%d", static_cast(i)); + ctx->rows[i].value = make_text_child(private_name, 20, 0xFFF2F5F8u); + } + return ctx; +} + +static void meter_destroy(void *data) +{ + auto *ctx = static_cast(data); + for (size_t i = 0; i < kMeterMaxRows; ++i) { + if (ctx->rows[i].name) obs_source_release(ctx->rows[i].name); + if (ctx->rows[i].value) obs_source_release(ctx->rows[i].value); + } + delete ctx; +} + +static void meter_update(void *data, obs_data_t *settings) +{ + meter_apply_settings(static_cast(data), settings); +} + +static uint32_t meter_get_width(void *data) +{ + auto *ctx = static_cast(data); + return ctx->canvas_width.load(std::memory_order_acquire); +} + +static uint32_t meter_get_height(void *data) +{ + auto *ctx = static_cast(data); + return ctx->canvas_height.load(std::memory_order_acquire); +} + +static void meter_enum_active_sources(void *data, + obs_source_enum_proc_t enum_callback, + void *param) +{ + auto *ctx = static_cast(data); + for (size_t i = 0; i < kMeterMaxRows; ++i) { + if (ctx->rows[i].name) enum_callback(ctx->source, ctx->rows[i].name, param); + if (ctx->rows[i].value) enum_callback(ctx->source, ctx->rows[i].value, param); + } +} + +// The model is rebuilt at 10 Hz, not per frame. corevideo_loudness_readings() +// takes g_sources_mtx and every source's own mutex -- the same mutex the +// audio lane holds for a whole drain -- so asking it 60 times a second would +// put the graphics thread in contention with the media path for no visible +// gain: the numbers it reports move on a 100 ms hop anyway. +static void meter_video_tick(void *data, float seconds) +{ + auto *ctx = static_cast(data); + ctx->rebuild_accum += seconds; + if (ctx->rebuild_accum < 0.1f) return; + ctx->rebuild_accum = 0.0f; + + const auto readings = corevideo_loudness_readings(); + const double tol = + static_cast(ctx->tolerance_milli_lu.load( + std::memory_order_acquire)) / 1000.0; + const auto kind = static_cast( + ctx->reference.load(std::memory_order_acquire)); + LoudnessBoardModel model = loudness_board_build( + readings, kind, tol, kLoudnessBoardMinBlocks); + + std::lock_guard lk(ctx->mutex); + ctx->model = std::move(model); +} + +static void meter_video_render(void *data, gs_effect_t *) +{ + auto *ctx = static_cast(data); + const int canvas_w = + static_cast(ctx->canvas_width.load(std::memory_order_acquire)); + const int canvas_h = + static_cast(ctx->canvas_height.load(std::memory_order_acquire)); + if (!s_meter_effect.valid()) return; + + LoudnessBoardModel model; + { + std::lock_guard lk(ctx->mutex); + model = ctx->model; + } + + meter_fill_rect(LoudnessBoardRect{0, 0, canvas_w, canvas_h}, kMeterBgArgb); + meter_fill_rect(LoudnessBoardRect{0, 0, canvas_w, kLoudnessBoardHeaderPx}, + kMeterHeaderArgb); + + // The LAST slot is permanently the header's, never a panelist's, so a + // change in row count cannot silently steal the header's text child -- + // hence the cap is kMeterMaxRows - 1 and not kMeterMaxRows. + static constexpr size_t kMeterHeaderSlot = kMeterMaxRows - 1; + const size_t total = model.rows.size(); + size_t shown = loudness_board_visible_rows(canvas_h, total); + if (shown > kMeterHeaderSlot) shown = kMeterHeaderSlot; + + for (size_t i = 0; i < shown; ++i) { + const LoudnessBoardRow &row = model.rows[i]; + const LoudnessBoardRect band = + loudness_board_row_rect(canvas_w, canvas_h, shown, i); + if (band.w <= 0 || band.h <= 0) continue; + + meter_fill_rect(band, kMeterRowArgb); + + // The status chip: a fat block at the left edge, which is the part + // that reads first from across a room. + meter_fill_rect(LoudnessBoardRect{band.x, band.y, 8, band.h}, + status_color(row.status)); + + // The zero line, drawn under the bar so a bar of zero width still + // shows where the reference is. + const LoudnessBoardRect zero = + loudness_board_bar_rect(band, 0.0, kLoudnessBoardFullScaleLu); + meter_fill_rect(LoudnessBoardRect{zero.x - 1, band.y, 2, band.h}, + kMeterCentreArgb); + + if (row.has_deviation) { + const LoudnessBoardRect bar = loudness_board_bar_rect( + band, row.deviation_lu, kLoudnessBoardFullScaleLu); + meter_fill_rect(LoudnessBoardRect{bar.x, bar.y + 4, bar.w, + bar.h > 8 ? bar.h - 8 : bar.h}, + status_color(row.status)); + } + } + + // Labels last, over the bars. Each child is only re-settings-updated when + // its string changes: obs_source_update() allocates and takes the source's + // own lock, and doing it per frame per row is the churn shape this project + // already has a live incident about. + const std::string head = header_text(model, shown, total); + { + std::lock_guard lk(ctx->mutex); + if (ctx->applied_signature != model.signature) { + ctx->applied_signature = model.signature; + for (size_t i = 0; i < kMeterHeaderSlot; ++i) { + const std::string name_text = + (i < shown) ? model.rows[i].name : std::string(); + const std::string value_text = + (i < shown) ? row_value_text(model.rows[i]) : std::string(); + if (ctx->rows[i].name_text != name_text) { + ctx->rows[i].name_text = name_text; + set_text_child(ctx->rows[i].name, name_text.c_str()); + } + if (ctx->rows[i].value_text != value_text) { + ctx->rows[i].value_text = value_text; + set_text_child(ctx->rows[i].value, value_text.c_str()); + } + } + } + } + + for (size_t i = 0; i < shown; ++i) { + const LoudnessBoardRect band = + loudness_board_row_rect(canvas_w, canvas_h, shown, i); + if (band.w <= 0 || band.h <= 0) continue; + const int text_y = band.y + (band.h > 24 ? (band.h - 24) / 2 : 0); + if (ctx->rows[i].name) { + gs_matrix_push(); + gs_matrix_translate3f(static_cast(band.x + 16), + static_cast(text_y), 0.0f); + obs_source_video_render(ctx->rows[i].name); + gs_matrix_pop(); + } + if (ctx->rows[i].value) { + gs_matrix_push(); + gs_matrix_translate3f(static_cast(band.x + band.w / 2 + 8), + static_cast(text_y), 0.0f); + obs_source_video_render(ctx->rows[i].value); + gs_matrix_pop(); + } + } + + // The header, in the slot reserved for it above. Updated on its own + // string comparison rather than on the board signature, because the + // "showing N of M" count changes with the CANVAS as well as the panel. + if (ctx->rows[kMeterHeaderSlot].name) { + obs_source_t *header = ctx->rows[kMeterHeaderSlot].name; + { + std::lock_guard lk(ctx->mutex); + if (ctx->rows[kMeterHeaderSlot].name_text != head) { + ctx->rows[kMeterHeaderSlot].name_text = head; + set_text_child(header, head.c_str()); + } + } + gs_matrix_push(); + gs_matrix_translate3f(12.0f, 4.0f, 0.0f); + obs_source_video_render(header); + gs_matrix_pop(); + } +} + +static bool meter_reset_clicked(obs_properties_t *, obs_property_t *, void *) +{ + corevideo_reset_loudness_windows(); + return false; +} + +static obs_properties_t *meter_get_properties(void *) +{ + obs_properties_t *props = obs_properties_create(); + + obs_property_t *ref = obs_properties_add_list( + props, PROP_REFERENCE, + obs_module_text("CoreVideoLoudnessMeter.Reference"), + OBS_COMBO_TYPE_LIST, OBS_COMBO_FORMAT_INT); + obs_property_list_add_int( + ref, obs_module_text("CoreVideoLoudnessMeter.Reference.PanelMedian"), + static_cast(LoudnessReference::PanelMedian)); + obs_property_list_add_int( + ref, obs_module_text("CoreVideoLoudnessMeter.Reference.R128"), + static_cast(LoudnessReference::EbuR128)); + obs_property_list_add_int( + ref, obs_module_text("CoreVideoLoudnessMeter.Reference.A85"), + static_cast(LoudnessReference::AtscA85)); + obs_property_list_add_int( + ref, obs_module_text("CoreVideoLoudnessMeter.Reference.Streaming"), + static_cast(LoudnessReference::Streaming)); + + obs_properties_add_float_slider( + props, PROP_TOLERANCE, + obs_module_text("CoreVideoLoudnessMeter.Tolerance"), 0.5, 6.0, 0.5); + obs_properties_add_int(props, PROP_WIDTH, + obs_module_text("CoreVideoLoudnessMeter.Width"), 160, 3840, 10); + obs_properties_add_int(props, PROP_HEIGHT, + obs_module_text("CoreVideoLoudnessMeter.Height"), 90, 2160, 10); + obs_properties_add_button(props, PROP_RESET, + obs_module_text("CoreVideoLoudnessMeter.Reset"), meter_reset_clicked); + return props; +} + +static void meter_get_defaults(obs_data_t *settings) +{ + obs_data_set_default_int(settings, PROP_REFERENCE, + static_cast(LoudnessReference::PanelMedian)); + obs_data_set_default_double(settings, PROP_TOLERANCE, + kLoudnessBoardDefaultToleranceLu); + obs_data_set_default_int(settings, PROP_WIDTH, 640); + obs_data_set_default_int(settings, PROP_HEIGHT, 360); +} + +void corevideo_loudness_meter_source_register() +{ + obs_source_info info = {}; + info.id = kMeterSourceId; + info.type = OBS_SOURCE_TYPE_INPUT; + // CUSTOM_DRAW because it binds the plugin's own effect rather than + // letting OBS draw one texture with the default one, exactly as the Tiles + // wall does. + info.output_flags = OBS_SOURCE_VIDEO | OBS_SOURCE_CUSTOM_DRAW | + OBS_SOURCE_DO_NOT_DUPLICATE; + info.get_name = meter_get_name; + info.create = meter_create; + info.destroy = meter_destroy; + info.update = meter_update; + info.video_tick = meter_video_tick; + info.video_render = meter_video_render; + info.get_width = meter_get_width; + info.get_height = meter_get_height; + info.enum_active_sources = meter_enum_active_sources; + info.get_properties = meter_get_properties; + info.get_defaults = meter_get_defaults; + obs_register_source(&info); +} + +void corevideo_loudness_meter_load_gfx() +{ + tiles_effect_load(s_meter_effect); +} + +void corevideo_loudness_meter_unload_gfx() +{ + tiles_effect_destroy(s_meter_effect); + s_meter_pass_failed_logged = false; +} diff --git a/src/zoom-loudness-meter-source.h b/src/zoom-loudness-meter-source.h new file mode 100644 index 00000000..d077fa3b --- /dev/null +++ b/src/zoom-loudness-meter-source.h @@ -0,0 +1,21 @@ +#pragma once + +// The CoreVideo Loudness Meter: a preshow readiness board, drawn as an OBS +// source so it can sit on a multiview, a projector or a producer's monitor +// without a dock being open. +// +// One row per live CoreVideo audio source: the panelist's name, their +// deviation in LU from the panel reference, and a pass/fail verdict. Bars are +// drawn with the Solid technique already in data/effects/corevideo-tiles.effect +// -- there is no new effect file, because a solid quad is all a bar is and a +// second .effect is a second thing that can go missing beside a new DLL. +// Labels are private child text sources, so a Norwegian display name renders +// correctly instead of through a hand-rolled ASCII font. + +void corevideo_loudness_meter_source_register(); + +// Compiles/releases the shared effect. Called from plugin-main.cpp alongside +// the Tiles equivalents; libobs caches effects created from a file, so this +// costs nothing beyond the Tiles source's own load. +void corevideo_loudness_meter_load_gfx(); +void corevideo_loudness_meter_unload_gfx(); diff --git a/tests/loudness-board-test.cpp b/tests/loudness-board-test.cpp index 108c1d89..1e969846 100644 --- a/tests/loudness-board-test.cpp +++ b/tests/loudness-board-test.cpp @@ -297,6 +297,42 @@ int main() "the bar's vertical extent does not match its row"); } + // ── The board is bounded, and it says so ─────────────────────────────── + // A 25-person Zoom Events room would give rows a few pixels tall, which + // is not a readiness board, it is a texture. The renderer caps the rows + // it draws; the cap has to be a decision that can be reasoned about here + // rather than a magic number buried in a draw loop. + { + check(loudness_board_visible_rows(360, 3) == 3, + "three panelists on a 360 px canvas did not all fit"); + check(loudness_board_visible_rows(360, 40) == + (360 - kLoudnessBoardHeaderPx) / kLoudnessBoardMinRowPx, + "forty panelists were not capped to what the canvas can show " + "at the minimum readable row height"); + check(loudness_board_visible_rows(360, 0) == 0, + "an empty panel produced rows to draw"); + check(loudness_board_visible_rows(0, 10) == 0, + "a zero-height canvas produced rows to draw"); + const size_t capped = loudness_board_visible_rows(360, 40); + const LoudnessBoardRect last = + loudness_board_row_rect(640, 360, capped, capped - 1); + // Compared against the SLOT (drawn height + the inter-row gap), not + // the drawn height alone: kLoudnessBoardMinRowPx is the minimum + // whole-row allotment ("a name and a number... plus the gap", per + // its own comment), and loudness_board_row_rect always carves the + // gap back out of whatever slot it is given. Comparing the cap's + // capacity formula (floor(body_h / kLoudnessBoardMinRowPx), pinned + // by the equality check above) against the post-gap .h directly is + // unsatisfiable by construction whenever kLoudnessBoardRowGapPx > 0: + // capacity = floor(body_h/M) only guarantees body_h/capacity >= M, + // i.e. the SLOT is at least M, not the slot minus the gap. For this + // canvas that is 25 px of slot for a 24 px minimum, and row_rect's + // own -4 px gap then drops the drawn height to 21. + check(last.h + kLoudnessBoardRowGapPx >= kLoudnessBoardMinRowPx, + "the capped row count still produced rows below the minimum " + "readable slot height"); + } + if (failures == 0) std::cout << "loudness-board: all tests passed\n"; return failures == 0 ? 0 : 1; From f8a3bb6519b58815c60e5cd259a245c0c2568c5e Mon Sep 17 00:00:00 2001 From: Grok Date: Sat, 5 Sep 2026 17:21:39 -0400 Subject: [PATCH 12/15] fix(loudness): fix round 1 -- lock discipline, canvas-resize label gate, visibility poll, thread-safe probe, true 10Hz cadence Five review defects in the Task 6 meter source, all confirmed in the brief's own verbatim code: - set_text_child() (obs_source_update(), which takes libobs's own source lock and allocates) was called while holding ctx->mutex, in both the per-row label loop and the header update. Now collects {source, text} pairs under the lock and applies them after release. - The label refresh was gated on model.signature alone, which encodes panel content but not `shown` (a function of canvas height). Growing the source's height could reveal rows still holding an empty applied string with no signature change to trigger a refresh -- worst case a silent preshow panel where every row sits at "no audio" forever. Added `applied_shown` and a new pure helper, loudness_board_needs_label_refresh() in src/loudness-board.h, pinned with four cases in tests/loudness-board-test.cpp (the load-bearing one: shown changes alone, signature unchanged, must still refresh). - meter_video_tick polled corevideo_loudness_readings() at 10Hz even for a meter source not on any visible scene. Now gated on obs_source_showing(ctx->source), matching the fix already on record for the Talkback dock's mirror-image roster-poll defect. - meter_text_source_id()'s cached/probed function-statics were a real data race across concurrently-created meter sources (UI thread, scene-load thread, control-API thread). Replaced with a function- local static initializer, which C++11 guarantees runs exactly once. - rebuild_accum was zeroed instead of decremented after firing, discarding the remainder and landing every 7 frames at 60fps (~117ms, ~8.6Hz) instead of the documented 10Hz. Now subtracts the interval so the remainder carries forward. Build-tests 59/59 green, full obs-zoom-plugin build clean, both re-verified immediately before this commit. No other behaviour changed -- the per-frame model copy, draw-loop magic numbers, and unused header-slot value source are explicitly deferred to the final whole-branch review. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01WqDL6gcoVvEMXv7Qwcni9n --- src/loudness-board.h | 19 +++++++ src/zoom-loudness-meter-source.cpp | 83 ++++++++++++++++++++++++++---- tests/loudness-board-test.cpp | 22 ++++++++ 3 files changed, 114 insertions(+), 10 deletions(-) diff --git a/src/loudness-board.h b/src/loudness-board.h index 81c3fc60..07f0a854 100644 --- a/src/loudness-board.h +++ b/src/loudness-board.h @@ -318,3 +318,22 @@ inline size_t loudness_board_visible_rows(int canvas_h, size_t row_count) if (capacity == 0) return 0; return row_count < capacity ? row_count : capacity; } + +// Whether the meter source's already-applied child text labels need +// refreshing. `signature` captures the panel's own content -- reference, +// names, statuses, quantised deviations -- but NOT how many rows are +// currently being DRAWN: `shown` is a function of the canvas, not the +// panel, via loudness_board_visible_rows() above. A bare signature +// comparison misses exactly the case that matters most for this board: +// resizing the source taller reveals rows that were previously off-screen +// (and so still hold an empty applied string) while every panelist sits +// unchanged at "no audio" during a silent preshow -- the signature never +// moves, so a refresh gated on it alone would leave those newly-visible +// rows blank until the next status change happened to fix it. Comparing +// `shown` as well means either changing alone forces a refresh. +inline bool loudness_board_needs_label_refresh( + const std::string &applied_signature, size_t applied_shown, + const std::string &signature, size_t shown) +{ + return applied_signature != signature || applied_shown != shown; +} diff --git a/src/zoom-loudness-meter-source.cpp b/src/zoom-loudness-meter-source.cpp index 2e1d1fc1..a5070e72 100644 --- a/src/zoom-loudness-meter-source.cpp +++ b/src/zoom-loudness-meter-source.cpp @@ -66,6 +66,15 @@ struct loudness_meter_source { std::mutex mutex; // guards `model` and `rows` LoudnessBoardModel model; std::string applied_signature; + // How many rows were applied alongside `applied_signature`. The + // signature encodes reference/names/statuses/quantised deviations but + // NOT `shown` -- `shown` depends on the CANVAS, not the panel -- so a + // height change that newly reveals rows whose signature has not moved + // (e.g. every row parked at "no audio" during a silent preshow, the + // exact case this board exists for) must still trigger a refresh, or + // those rows draw their band/chip/centre-line with no name and no value + // until the next status change self-heals it. + size_t applied_shown = static_cast(-1); meter_row_widgets rows[kMeterMaxRows]; float rebuild_accum = 0.0f; @@ -83,12 +92,9 @@ struct loudness_meter_source { // on different platforms and has renamed both across versions. A build with // neither must lose the labels and keep the bars, loudly -- never render an // empty board with no explanation. -static const char *meter_text_source_id() +static const char *probe_meter_text_source_id() { - static const char *cached = nullptr; - static bool probed = false; - if (probed) return cached; - probed = true; + const char *cached = nullptr; static const char *candidates[] = { "text_ft2_source_v2", "text_gdiplus_v3", "text_gdiplus_v2", "text_ft2_source", "text_gdiplus", @@ -114,6 +120,20 @@ static const char *meter_text_source_id() return cached; } +static const char *meter_text_source_id() +{ + // Function-local static initialisation is thread-safe as of C++11 (the + // standard guarantees the initializer runs exactly once even under a + // race) -- load-bearing here, unlike the plain function-static + // cached/probed pair this replaced: meter_create() can run on the UI + // thread, a scene-load thread, or the control-API thread, and two + // meters created concurrently on a fresh load raced that pair, with a + // real chance of the probe (and its log line) running twice. "One log + // line, not a stream" is a documented acceptance check for this source. + static const char *id = probe_meter_text_source_id(); + return id; +} + static obs_source_t *make_text_child(const char *private_name, int px, uint32_t argb) { @@ -329,12 +349,26 @@ static void meter_enum_active_sources(void *data, // audio lane holds for a whole drain -- so asking it 60 times a second would // put the graphics thread in contention with the media path for no visible // gain: the numbers it reports move on a 100 ms hop anyway. +// +// Also gated on obs_source_showing(): OBS calls video_tick for every source +// regardless of whether it is on a visible scene, so a meter parked in an +// unused scene would otherwise poll g_sources_mtx plus every live source's +// own ctx->mtx ten times a second forever. CLAUDE.md records the +// mirror-image defect on the Talkback dock's roster poll (a folded, hidden +// section still rebuilding at 10 Hz) -- same fix, same reasoning: a source +// nobody is looking at should cost nothing. static void meter_video_tick(void *data, float seconds) { auto *ctx = static_cast(data); + if (!obs_source_showing(ctx->source)) return; ctx->rebuild_accum += seconds; if (ctx->rebuild_accum < 0.1f) return; - ctx->rebuild_accum = 0.0f; + // Subtract the interval rather than zeroing: zeroing discards whatever + // remainder pushed this tick over 0.1s, and at 60 fps (16.7 ms/frame) + // that landed every 7 frames -- ~117 ms, an ~8.6 Hz cadence, not the + // 10 Hz this comment (and the brief) claims. Subtracting keeps the + // carried remainder so the average cadence is the documented 10 Hz. + ctx->rebuild_accum -= 0.1f; const auto readings = corevideo_loudness_readings(); const double tol = @@ -409,11 +443,34 @@ static void meter_video_render(void *data, gs_effect_t *) // its string changes: obs_source_update() allocates and takes the source's // own lock, and doing it per frame per row is the churn shape this project // already has a live incident about. + // + // Gated on `shown` changing as well as `model.signature`: the signature + // encodes reference/names/statuses/quantised deviations but NOT `shown`, + // which depends on the CANVAS. Growing the source's height can newly + // reveal rows that previously held an empty string while the panel's own + // signature has not moved (every row parked at "no audio" during a + // silent preshow is exactly that case), and gating on the signature + // alone would leave those rows drawing a band/chip/centre-line with no + // text until the next status change happened to self-heal it. + // + // set_text_child() calls obs_source_update(), which takes libobs's own + // source lock and allocates an obs_data_t -- neither belongs inside + // ctx->mutex. What to apply is decided under the lock into a local + // vector of (slot, text) pairs; the OBS calls happen after the lock is + // released. const std::string head = header_text(model, shown, total); + struct PendingLabel { + obs_source_t *child; + std::string text; + }; + std::vector pending; { std::lock_guard lk(ctx->mutex); - if (ctx->applied_signature != model.signature) { + if (loudness_board_needs_label_refresh(ctx->applied_signature, + ctx->applied_shown, + model.signature, shown)) { ctx->applied_signature = model.signature; + ctx->applied_shown = shown; for (size_t i = 0; i < kMeterHeaderSlot; ++i) { const std::string name_text = (i < shown) ? model.rows[i].name : std::string(); @@ -421,15 +478,17 @@ static void meter_video_render(void *data, gs_effect_t *) (i < shown) ? row_value_text(model.rows[i]) : std::string(); if (ctx->rows[i].name_text != name_text) { ctx->rows[i].name_text = name_text; - set_text_child(ctx->rows[i].name, name_text.c_str()); + pending.push_back({ctx->rows[i].name, name_text}); } if (ctx->rows[i].value_text != value_text) { ctx->rows[i].value_text = value_text; - set_text_child(ctx->rows[i].value, value_text.c_str()); + pending.push_back({ctx->rows[i].value, value_text}); } } } } + for (const PendingLabel &p : pending) + set_text_child(p.child, p.text.c_str()); for (size_t i = 0; i < shown; ++i) { const LoudnessBoardRect band = @@ -457,13 +516,17 @@ static void meter_video_render(void *data, gs_effect_t *) // "showing N of M" count changes with the CANVAS as well as the panel. if (ctx->rows[kMeterHeaderSlot].name) { obs_source_t *header = ctx->rows[kMeterHeaderSlot].name; + bool needs_update = false; { std::lock_guard lk(ctx->mutex); if (ctx->rows[kMeterHeaderSlot].name_text != head) { ctx->rows[kMeterHeaderSlot].name_text = head; - set_text_child(header, head.c_str()); + needs_update = true; } } + // Same rule as the row labels above: obs_source_update() must not + // run while ctx->mutex is held. + if (needs_update) set_text_child(header, head.c_str()); gs_matrix_push(); gs_matrix_translate3f(12.0f, 4.0f, 0.0f); obs_source_video_render(header); diff --git a/tests/loudness-board-test.cpp b/tests/loudness-board-test.cpp index 1e969846..38db91ba 100644 --- a/tests/loudness-board-test.cpp +++ b/tests/loudness-board-test.cpp @@ -333,6 +333,28 @@ int main() "readable slot height"); } + // -- The label refresh gate fires on shown OR signature, not signature + // alone ----------------------------------------------------------------- + // A canvas resize changes `shown` without touching a single panelist's + // reading -- the silent-preshow case, where every row sits stably at + // "no audio" and the signature never moves at all. If the gate ignored + // `shown`, newly-revealed rows would keep whatever text (usually none) + // was applied while they were off-screen. + { + check(!loudness_board_needs_label_refresh("sig-a", 3, "sig-a", 3), + "an unchanged signature and unchanged shown count asked for a " + "refresh"); + check(loudness_board_needs_label_refresh("sig-a", 3, "sig-b", 3), + "a changed signature with unchanged shown did not ask for a " + "refresh"); + check(loudness_board_needs_label_refresh("sig-a", 3, "sig-a", 5), + "a changed shown count with an UNCHANGED signature did not ask " + "for a refresh -- this is the canvas-resize-during-a-silent-" + "preshow case the gate exists for"); + check(loudness_board_needs_label_refresh("sig-a", 3, "sig-b", 5), + "both changing at once did not ask for a refresh"); + } + if (failures == 0) std::cout << "loudness-board: all tests passed\n"; return failures == 0 ? 0 : 1; From b93040ba9fbbc5d43eb928013cccf67b68c83a45 Mon Sep 17 00:00:00 2001 From: Grok Date: Sat, 5 Sep 2026 17:25:18 -0400 Subject: [PATCH 13/15] docs: record the loudness engine's runtime-rate, gating, and board-layout invariants Task 5 already documented the audio-lane wiring (drain-loop feed, wire-format tap, roster caching, lock order, window reset on resubscribe). This adds the invariants it didn't cover: runtime-derived K-weighting coefficients (never pinned 48 kHz), the shared clear_window() between configure()/reset_window(), the two-pass gate as load-bearing (not an optimisation) plus the median reference, the board's slot-pitch-vs-drawn-height row sizing, the signature+shown label-refresh gate, and the 10 Hz poll's showing-gate and subtracting accumulator. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01WqDL6gcoVvEMXv7Qwcni9n --- CLAUDE.md | 54 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 54 insertions(+) diff --git a/CLAUDE.md b/CLAUDE.md index e2224f5a..fa31c7aa 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -154,6 +154,60 @@ Every one of these is documented at length where it lives; the list is the map. directly (2026-08-18/19). Applied in both `zoom-source.cpp` (`output_audio_from_shared_memory`, shared by the main and director-preview slots) and `zoom-participant-audio-source.cpp`. +- **Loudness coefficients follow the RUNTIME sample rate, never 48 kHz** + (`src/audio-loudness.h`, feat/panelist-feedback): BS.1770-4 publishes its + two K-weighting biquads' coefficients for 48 kHz and no other rate, and + this plugin has no guaranteed rate -- Zoom commonly delivers 32 kHz. The + coefficients are DERIVED from the analog prototype at whatever + `loudness_meter_configure()` is called with; at 48 kHz that derivation + reproduces the published table to fourteen digits, which is what the + engine's tests assert. Pinned at 48 kHz and fed 32 kHz, a 1 kHz tone + whose true value is -19.98 LUFS reads -18.66: 1.3 LU wrong, on a meter + whose whole product claim is that a 6 LU spread between panelists is + visible, and with nothing in the number to say it is wrong. The + gated-integration window carries the same fragility one level up: + `loudness_meter_clear_window()` is shared, on purpose, by + `loudness_meter_configure()` and `loudness_meter_reset_window()` so a + mid-stream format renegotiation (Zoom renegotiating, or a Mix/Isolated + role flip changing channel count on the same subscription) cannot leave + blocks measured under the OLD coefficients sitting in the same check + window as blocks measured under the new ones -- do not let those two + call sites diverge. +- **Integrated loudness is gated, and the gate is load-bearing, not an + optimisation** (`audio-loudness.h`'s two-pass integration: an absolute + gate at -70 LUFS, then a relative gate at -10 LU below the + absolute-gated mean, over 400 ms blocks at a 100 ms hop): a panelist is + silent roughly 80% of a preshow, and ungated silence pulls the mean down + hard -- 4 s of -20 LUFS speech inside a 20 s window reads -27.08 LUFS + ungated and -20.16 gated, and -27.08 is not a usably-wrong number, it is + a differently-shaped one that would fail every panelist on every panel. + The board's reference is the panel **MEDIAN** of gated integrated + loudness, never the mean, and only panelists who have cleared the + minimum gated block count vote on it -- one laptop mic sitting at -35 + LUFS should not get to drag the reference far enough to fail everyone + else, and a panelist who hasn't spoken yet should not vote at all. +- **The readiness board's `kLoudnessBoardMinRowPx` is a SLOT PITCH, not a + drawn row height** (`src/loudness-board.h`): `loudness_board_visible_rows()` + divides available body height by it to decide how many rows fit; the row + actually drawn is that slot minus `kLoudnessBoardRowGapPx`. The test + asserts `last.h + kLoudnessBoardRowGapPx >= kLoudnessBoardMinRowPx` + specifically because `last.h >= kLoudnessBoardMinRowPx` is unsatisfiable + for any positive gap -- a future "simplification" to the un-added form + is a regression, not a cleanup. Label refresh on the same board is gated + on BOTH `model.signature` changing AND `shown` (the visible row count) + changing, because the signature encodes reference/names/statuses/ + deviations but not how many rows the current canvas height reveals: a + signature-only gate leaves rows newly exposed by a resize blank + indefinitely whenever the panel's content hasn't otherwise moved -- + worst during a silent preshow, which is the normal state, since every + parked row reads identically. The board's own 10 Hz rebuild + (`meter_video_tick`) is gated on `obs_source_showing()` for the same + reason as the Talkback dock's roster poll: a meter parked on an unused + scene should not pay for `g_sources_mtx` plus every live source's mutex + ten times a second forever. The accumulator SUBTRACTS the 100 ms + interval rather than zeroing on fire -- zeroing discards the remainder + that pushed a tick over threshold, which at 60 fps lands every 7 frames + (~117 ms, ~8.6 Hz) instead of the documented 10 Hz. - **ISO recording timing** (`src/iso-video-pacer.h`, `src/iso-audio-gap-fill.h`): raw video has no per-frame timestamps and ffmpeg cannot be trusted to invent correct ones from a byte stream — `-use_wallclock_as_timestamps` From 6e0dc56c7a1e9fe64f77b83d7469c98bf88767d9 Mon Sep 17 00:00:00 2001 From: Grok Date: Sat, 5 Sep 2026 17:47:13 -0400 Subject: [PATCH 14/15] Fix final whole-branch review findings on the panelist loudness board CRITICAL: only Participant-kind sources vote on the panel median or appear as board rows. ActiveSpeaker (a duplicate of whoever is talking) and Audience (the whole-meeting mix) used to be counted and rendered as phantom "- unassigned -" rows. LoudnessReading now carries `kind`, and the filter lives in the pure loudness-board.h logic (loudness_panel_median and loudness_board_build), not the OBS glue, so it is unit-testable. Chosen approach: exclude non-Participant readings entirely rather than show them read-only -- a duplicate-speaker or room-mix row is the same mystery-row defect wearing a label, not a fix for it. IMPORTANT: the deviation bar is now driven by short-term deviation (falling back to integrated deviation, never vanishing) via the new pure selector loudness_board_bar_input(), while row TEXT stays on the integrated verdict per the design ruling. Kept out of LoudnessBoardModel::signature on purpose, since that would force a ~10x/sec text-child rebuild for a value the text never shows. IMPORTANT: loudness_meter_integrated()'s relative gate no longer calls log10() per gated block. L(z) > L(mean) - 10 <=> z > mean/10, so the comparison moved to the linear domain, removing up to 6000 log10() calls per source per 100ms poll from the graphics-thread-adjacent audio drain and a float round-trip that could flip a boundary block. The pinned -27.08/-20.16 and -22.96/-20.06 test figures are unchanged. loudness_meter_configure() now reserves `gated` to kLoudnessMaxGatedBlocks up front so the audio lane is provably allocation-free after configure. IMPORTANT: corevideo_loudness_readings() now try_locks each source's mutex instead of blocking, since output_audio_frame() holds the same mutex across a whole drain (SHM open, obs_source_output_audio, rate- limited disk-writing blog calls). A busy source is skipped for one 10Hz poll rather than stalling the OBS graphics thread. Documented (not previously stated) that the board shows SOURCES an operator created, not the live roster -- a panelist with no source never appears, and a rejoin drops an existing row to unassigned/no-audio until re-pointed. Added to zoom-loudness-meter-source.h and CLAUDE.md. MINOR: corrected two false comments claiming libobs caches/dedupes effects created from a file (gs_effect_create_from_file allocates a fresh effect every call) -- corrected in zoom-loudness-meter-source.cpp and zoom-tiles-effect.h before a future "dedupe the two loads" cleanup turns the harmless truth into a double-free at unload. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01WqDL6gcoVvEMXv7Qwcni9n --- CLAUDE.md | 71 ++++++++++++++ src/audio-loudness.h | 49 +++++++--- src/loudness-board.h | 88 ++++++++++++++++- src/zoom-loudness-meter-source.cpp | 24 ++++- src/zoom-loudness-meter-source.h | 23 ++++- src/zoom-participant-audio-source.cpp | 19 +++- src/zoom-tiles-effect.h | 12 ++- tests/audio-loudness-test.cpp | 11 +++ tests/loudness-board-test.cpp | 134 ++++++++++++++++++++++++++ 9 files changed, 402 insertions(+), 29 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index fa31c7aa..834f0825 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1246,6 +1246,77 @@ Every one of these is documented at length where it lives; the list is the map. integrated loudness is scoped to one panelist's mic check, not the source's whole lifetime -- without it a re-subscribed source's number is polluted by whoever spoke before on the same source. +- **Only Participant-kind sources vote on the panel median, or appear on the + board at all** (final whole-branch review, 2026-09-05, Critical). Three + kinds of CoreVideo audio source exist (`CoreVideoAudioKind`, + `src/audio-subscription-state.h`): Participant, ActiveSpeaker (a resolved + duplicate of whoever is currently talking) and Audience (the whole-meeting + mix, no participant id). `corevideo_loudness_readings()` used to hand ALL + three to the board with no distinction: an ActiveSpeaker source in a scene + with 4 Participant sources put a duplicate of whoever was talking into a + 5-value median, shifting the reference by up to a full LU on an even/odd + flip and moving every panelist's pass/fail with it; an Audience source put + the whole-room mix into a reference meant to describe individual + microphones; both rendered as phantom `"- unassigned -"` rows since neither + carries a participant id or (for Audience) ever gets a cached display name. + `LoudnessReading` now carries `kind`, and the filtering lives in the PURE + header (`src/loudness-board.h`'s `loudness_panel_median()` and + `loudness_board_build()`), not in the OBS glue that calls it -- both are + pinned by a test that a median over a mixed vector equals the median of the + Participants alone, and the board excludes non-Participant readings from + rows entirely rather than showing them as read-only entries: the board's + whole model is "one row per panelist's mic", and a row for a duplicate + speaker or the room mix is the same mystery-row defect wearing a label, + not a fix for it. **The board shows SOURCES, not the roster** -- see the + header comment on `src/zoom-loudness-meter-source.h`: a row exists because + an operator created a Participant-kind source and pointed it at somebody, + so a panelist with no source never appears, and a rejoin (participant ids + are meeting-scoped and do not survive one) drops a row to + "- unassigned -"/"no audio" until the operator re-points it. Correct + behaviour, documented because it is the most likely way an operator + concludes the board itself is broken. +- **The deviation bar is driven by short-term deviation, the row TEXT stays on + the integrated verdict** (`loudness_board_bar_input()`, + `src/loudness-board.h`). `LoudnessBoardRow::has_short_term`/`short_term_lufs` + were measured and carried all the way to the board and read by nothing -- + `row_value_text()` only ever printed the integrated number. Ruling: bars + are geometry the renderer fills every frame regardless of the label-refresh + gate, so driving the bar's position from the fast (short-term) measure costs + no extra text-child churn, while folding short-term into + `LoudnessBoardModel::signature` would rebuild every row's text children + ~10x/sec for a value the text never shows -- exactly what + `loudness_board_needs_label_refresh()` exists to prevent. Where a short-term + reading or a reference is unavailable, `loudness_board_bar_input()` falls + back to the integrated deviation the bar always used, rather than showing no + bar at all. +- **No `log10` in the relative gate's hot loop** + (`loudness_meter_integrated()`, `src/audio-loudness.h`). Pass 2 used to + convert every gated block (up to `kLoudnessMaxGatedBlocks` = 6000) to LUFS + with `loudness_lufs_from_mean_square()` (a `log10()`) just to compare it + against a LUFS threshold, once per source per 100 ms poll, on the graphics + thread inside `corevideo_loudness_readings()`, under the same mutex the + audio drain holds for its whole drain. The relative gate + (`kLoudnessRelativeGateLu` = -10 LU below the absolute-gated mean) is + exactly a factor of ten in LINEAR mean square -- `L(z) > L(mean) - 10 <=> + z > mean/10` -- so the comparison is now `z > 0.1 * abs_mean_z` with no + log10 anywhere in the loop, which also removes a float round-trip that + could flip a block sitting exactly on the threshold. The pinned -27.08/ + -20.16 and -22.96/-20.06 figures in `tests/audio-loudness-test.cpp` are + unchanged. `loudness_meter_configure()` also now `reserve()`s `gated` to + `kLoudnessMaxGatedBlocks` up front, so the audio lane is provably + allocation-free after configure. +- **`corevideo_loudness_readings()` uses `try_lock` on each source's mutex, + never a blocking `lock()`** (final whole-branch review, 2026-09-05, + Important). It runs on the OBS graphics thread via `meter_video_tick()` and + takes the exact `ctx->mtx` that `output_audio_frame()` holds across an + entire drain -- including `shm_region_open_readwrite()`, + `obs_source_output_audio()`, and rate-limited `blog()` calls that write to + disk. A blocking lock here could stall the graphics thread for the length + of one source's drain. A source that is busy is simply skipped for that + poll; its numbers reappear on the next 100 ms tick, invisible on a board + that redraws at 10 Hz. `corevideo_audio_source_infos()` (the sibling + registry walk) is unchanged and still blocks -- it is called far less often + and from a different context. ## Live testing against a real meeting diff --git a/src/audio-loudness.h b/src/audio-loudness.h index dd0a3939..4f385795 100644 --- a/src/audio-loudness.h +++ b/src/audio-loudness.h @@ -217,6 +217,18 @@ inline void loudness_meter_clear_window(LoudnessMeter &m) m.hop_total = 0; } +// 6000 blocks is 10 minutes of continuously-gated audio at a 100 ms hop. A +// mic check is 20-60 s (~200-600 blocks), so this is never reached in the +// use this was built for; past it the window keeps the most RECENT 10 minutes +// rather than growing without bound. Documented rather than silent, because +// "the oldest audio quietly leaves the window" is a real semantic and an +// operator who leaves a board running all show is entitled to know it. +// +// Declared here, ahead of loudness_meter_configure(), so that function can +// reserve() the gated vector to this cap up front instead of leaving it to +// grow lazily off the audio lane. +constexpr size_t kLoudnessMaxGatedBlocks = 6000; + // (Re)configures for a rate/channel count and clears all filter state. Called // automatically by loudness_meter_feed_int16() whenever the wire format // changes -- which it can, mid-source: Zoom renegotiates, and the operator's @@ -238,6 +250,13 @@ inline void loudness_meter_configure(LoudnessMeter &m, uint32_t sample_rate, m.s2.assign(m.channels, LoudnessBiquadState{}); m.hop_frames = (rate * kLoudnessHopMs) / 1000; if (m.hop_frames == 0) m.hop_frames = 1; + // Reserved once, here, rather than left to push_back() in + // loudness_meter_on_hop_complete() to grow it lazily: `gated` never grows + // past kLoudnessMaxGatedBlocks (it becomes a ring at that point), so + // reserving the cap up front makes the audio lane provably + // allocation-free after configure -- no realloc can ever land on the + // media path once a source is subscribed. + m.gated.reserve(kLoudnessMaxGatedBlocks); loudness_meter_clear_window(m); } @@ -340,14 +359,6 @@ inline bool loudness_meter_short_term(const LoudnessMeter &m, double *out_lufs) constexpr double kLoudnessAbsoluteGateLufs = -70.0; constexpr double kLoudnessRelativeGateLu = -10.0; -// 6000 blocks is 10 minutes of continuously-gated audio at a 100 ms hop. A -// mic check is 20-60 s (~200-600 blocks), so this is never reached in the -// use this was built for; past it the window keeps the most RECENT 10 minutes -// rather than growing without bound. Documented rather than silent, because -// "the oldest audio quietly leaves the window" is a real semantic and an -// operator who leaves a board running all show is entitled to know it. -constexpr size_t kLoudnessMaxGatedBlocks = 6000; - // Called at every completed 100 ms hop. A 400 ms block is the newest four // hops, so admitting one block per hop is the standard's 75% overlap. inline void loudness_meter_on_hop_complete(LoudnessMeter &m) @@ -396,17 +407,31 @@ inline bool loudness_meter_integrated(const LoudnessMeter &m, double *out_lufs) double sum = 0.0; for (double z : m.gated) sum += z; - const double abs_mean_lufs = - loudness_lufs_from_mean_square(sum / static_cast(m.gated.size())); + const double abs_mean_z = sum / static_cast(m.gated.size()); + const double abs_mean_lufs = loudness_lufs_from_mean_square(abs_mean_z); if (!std::isfinite(abs_mean_lufs)) return false; - const double relative_threshold = abs_mean_lufs + kLoudnessRelativeGateLu; + // The relative gate (kLoudnessRelativeGateLu = -10 LU below the + // absolute-gated mean) is exactly a factor of ten in linear mean square, + // not a log10() comparison: + // L(z) > L(mean) - 10 + // 10*log10(z) - 10*log10(mean) > -10 + // log10(z / mean) > -1 + // z > mean / 10 + // This pass used to call loudness_lufs_from_mean_square() (a log10()) + // once per gated block -- up to kLoudnessMaxGatedBlocks = 6000 -- every + // 100 ms poll, on the graphics thread's corevideo_loudness_readings() + // call, under the same ctx->mtx the audio drain holds. Comparing in the + // linear domain removes every one of those calls from the loop and also + // removes a float round-trip (linear -> LUFS -> compared to a LUFS + // threshold) that could flip a block sitting exactly on the gate. + const double relative_threshold_z = 0.1 * std::fabs(abs_mean_z); double sum2 = 0.0; size_t n2 = 0; for (double z : m.gated) { // Strictly greater, per BS.1770-4: a block exactly on the threshold // is excluded. - if (loudness_lufs_from_mean_square(z) > relative_threshold) { + if (z > relative_threshold_z) { sum2 += z; ++n2; } diff --git a/src/loudness-board.h b/src/loudness-board.h index 07f0a854..47373a66 100644 --- a/src/loudness-board.h +++ b/src/loudness-board.h @@ -21,6 +21,8 @@ // // Pure: no libobs, no Qt, no Zoom SDK. +#include "audio-subscription-state.h" + #include #include #include @@ -33,10 +35,21 @@ // is a different statement from any loudness value, and collapsing it to a // sentinel number is how a board ends up confidently reporting -70 LUFS for // somebody who simply has not spoken yet. +// +// `kind` is why this struct exists rather than just handing the board a +// vector of doubles. Three kinds of CoreVideo audio source can feed a +// LoudnessMeter (src/audio-subscription-state.h): Participant, ActiveSpeaker +// (a resolved duplicate of whoever is currently talking) and Audience (the +// whole-meeting mix, no participant id at all). Only Participant readings may +// vote on the panel median -- see loudness_panel_median() below -- and only +// Participant readings are shown as rows at all -- see loudness_board_build() +// -- because an ActiveSpeaker row is a duplicate that shifts the reference on +// every speaker change and an Audience row has no panelist to describe. struct LoudnessReading { - std::string source_uuid; - std::string display_name; - uint32_t participant_id = 0; + std::string source_uuid; + std::string display_name; + uint32_t participant_id = 0; + CoreVideoAudioKind kind = CoreVideoAudioKind::Participant; bool subscribed = false; bool has_short_term = false; double short_term_lufs = 0.0; @@ -90,11 +103,48 @@ struct LoudnessBoardRow { double deviation_lu = 0.0; bool has_short_term = false; double short_term_lufs = 0.0; + // Short-term deviation from the panel reference, i.e. short_term_lufs - + // reference. Populated whenever both a short-term reading and a + // reference exist, independent of `has_deviation` (which gates on + // gated_blocks meeting kLoudnessBoardMinBlocks -- a verdict requirement + // that a live, ungated number should not inherit). This is what + // loudness_board_bar_input() prefers: the bar is meant to move live while + // the panelist talks, and short-term is the fast measure that does that; + // the row TEXT stays on the integrated verdict. + bool has_short_term_deviation = false; + double short_term_deviation_lu = 0.0; bool has_integrated = false; double integrated_lufs = 0.0; LoudnessRowStatus status = LoudnessRowStatus::NoAudio; }; +// What the deviation BAR should be driven from for this row: short-term +// deviation when it exists (the fast, live-moving measure), falling back to +// the integrated deviation the bar has always used when it does not -- +// falling back rather than vanishing, so a panelist between hops still shows +// a bar. Returns false only when neither exists, e.g. before a reference has +// ever been established. +// +// Deliberately NOT part of LoudnessBoardModel::signature: the signature +// drives child TEXT-source rebuilds (obs_source_update(), one per row, per +// CLAUDE.md's note on the churn that caused), and short-term moves roughly +// every 100 ms hop -- folding it in would rebuild every row's text children +// ~10x/sec for a value that never reaches the text. Bars are geometry the +// renderer fills every frame regardless, so reading this per-frame costs +// nothing extra. +inline bool loudness_board_bar_input(const LoudnessBoardRow &row, double *out_lu) +{ + if (row.has_short_term_deviation) { + *out_lu = row.short_term_deviation_lu; + return true; + } + if (row.has_deviation) { + *out_lu = row.deviation_lu; + return true; + } + return false; +} + struct LoudnessBoardModel { bool has_reference = false; double reference_lufs = 0.0; @@ -125,12 +175,21 @@ inline bool loudness_reference_fixed_target(LoudnessReference kind, double *out) // produced a check. Even counts average the two middle values, which is the // ordinary definition and keeps a two-person panel from arbitrarily electing // one of them as the reference. +// +// Only Participant-kind readings vote. ActiveSpeaker resolves to whichever +// Participant is currently talking, so counting it too is counting that +// person twice and shifting the reference by up to a full LU depending on an +// arbitrary even/odd flip; Audience is the whole-meeting mix and describes no +// one microphone at all. Both are display concerns, not reference concerns -- +// see loudness_board_build(), which excludes them from rows entirely for the +// same reason. inline bool loudness_panel_median(const std::vector &readings, uint64_t min_blocks, double *out) { std::vector values; values.reserve(readings.size()); for (const LoudnessReading &r : readings) { + if (r.kind != CoreVideoAudioKind::Participant) continue; if (!r.has_integrated) continue; if (r.gated_blocks < min_blocks) continue; if (!std::isfinite(r.integrated_lufs)) continue; @@ -177,9 +236,23 @@ inline LoudnessBoardModel loudness_board_build( // Ordered by CONTENT alone -- name, then uuid to break a duplicate-name // tie -- so a roster that merely reorders produces an identical board. + // + // Only Participant-kind readings become rows. ActiveSpeaker is a resolved + // duplicate of whichever Participant is currently talking (a second row + // for the same person), and Audience has no participant id and so no + // display name -- both used to reach this board and either duplicate a + // real panelist's row or render as a phantom "- unassigned -" entry. + // Filtering here, in the pure header, rather than in the OBS glue that + // calls it: this repo's convention is that pure logic gets unit tests and + // OBS glue does not, and which readings become rows is exactly the kind + // of decision this file exists to hold so it can be pinned without a + // meeting. std::vector ordered; ordered.reserve(readings.size()); - for (const LoudnessReading &r : readings) ordered.push_back(&r); + for (const LoudnessReading &r : readings) { + if (r.kind != CoreVideoAudioKind::Participant) continue; + ordered.push_back(&r); + } std::sort(ordered.begin(), ordered.end(), [](const LoudnessReading *a, const LoudnessReading *b) { if (a->display_name != b->display_name) @@ -200,6 +273,13 @@ inline LoudnessBoardModel loudness_board_build( row.has_integrated = r->has_integrated; row.integrated_lufs = r->integrated_lufs; + if (row.has_short_term && model.has_reference && + std::isfinite(row.short_term_lufs)) { + row.has_short_term_deviation = true; + row.short_term_deviation_lu = + row.short_term_lufs - model.reference_lufs; + } + if (!r->has_integrated || r->gated_blocks == 0) { row.status = LoudnessRowStatus::NoAudio; } else if (r->gated_blocks < min_blocks) { diff --git a/src/zoom-loudness-meter-source.cpp b/src/zoom-loudness-meter-source.cpp index a5070e72..b2c66756 100644 --- a/src/zoom-loudness-meter-source.cpp +++ b/src/zoom-loudness-meter-source.cpp @@ -42,9 +42,17 @@ static constexpr uint32_t kMeterLoudArgb = 0xFFE04B4Bu; static constexpr uint32_t kMeterQuietArgb = 0xFFE0A03Cu; static constexpr uint32_t kMeterIdleArgb = 0xFF3A424Eu; -// Shared with the Tiles wall by file, not by handle: libobs caches effects -// created from a file, so this second tiles_effect_load() resolves the same -// compiled effect rather than compiling it twice. +// Shared with the Tiles wall only by FILE PATH, not by handle -- and that is +// not the free dedupe it sounds like. gs_effect_create_from_file() allocates +// a brand-new gs_effect_t on every call; libobs does NOT cache or dedupe +// effects compiled from the same file. This second tiles_effect_load() call +// (the Tiles wall makes its own, separate one) compiles a second, independent +// copy of corevideo-tiles.effect. Harmless as written -- two handles, each +// destroyed exactly once by its own owner (this file's unload vs. the Tiles +// source's) -- but do NOT "deduplicate" the two loads into one shared +// gs_effect_t* on the strength of a caching story that isn't true: sharing a +// handle between two owners that each call gs_effect_destroy() on it once is +// a double-free at unload. static TilesEffect s_meter_effect; static bool s_meter_pass_failed_logged = false; @@ -430,9 +438,15 @@ static void meter_video_render(void *data, gs_effect_t *) meter_fill_rect(LoudnessBoardRect{zero.x - 1, band.y, 2, band.h}, kMeterCentreArgb); - if (row.has_deviation) { + // Driven by short-term deviation when it exists (the fast measure + // that should move live while the panelist talks), falling back to + // the integrated deviation otherwise -- see loudness_board_bar_input(). + // The row's pass/fail colour still comes from `row.status`, which is + // the integrated verdict: only the bar's POSITION is live. + double bar_dev_lu = 0.0; + if (loudness_board_bar_input(row, &bar_dev_lu)) { const LoudnessBoardRect bar = loudness_board_bar_rect( - band, row.deviation_lu, kLoudnessBoardFullScaleLu); + band, bar_dev_lu, kLoudnessBoardFullScaleLu); meter_fill_rect(LoudnessBoardRect{bar.x, bar.y + 4, bar.w, bar.h > 8 ? bar.h - 8 : bar.h}, status_color(row.status)); diff --git a/src/zoom-loudness-meter-source.h b/src/zoom-loudness-meter-source.h index d077fa3b..ee8a1372 100644 --- a/src/zoom-loudness-meter-source.h +++ b/src/zoom-loudness-meter-source.h @@ -11,11 +11,30 @@ // second .effect is a second thing that can go missing beside a new DLL. // Labels are private child text sources, so a Norwegian display name renders // correctly instead of through a hand-rolled ASCII font. +// +// THE BOARD SHOWS SOURCES, NOT THE ROSTER. A row exists because an operator +// created a Participant-kind CoreVideo audio source and pointed it at +// somebody -- it is not a live view of who is in the meeting. Two +// consequences an operator will run into and may read as a bug: a panelist +// with no source pointed at them never appears at all, and a panelist who +// rejoins drops to "- unassigned -"/"no audio" until the operator re-points +// the source at them, because Zoom participant ids are meeting-scoped and do +// not survive a rejoin (the same fact documented for talkback nomination in +// CLAUDE.md). Both are correct behaviour, not defects, but they are the most +// likely way this board gets reported as broken. +// +// ActiveSpeaker- and Audience-kind sources are excluded from the board +// entirely (never a row, never a vote on the panel median) -- see +// src/loudness-board.h's loudness_panel_median()/loudness_board_build() for +// why. void corevideo_loudness_meter_source_register(); // Compiles/releases the shared effect. Called from plugin-main.cpp alongside -// the Tiles equivalents; libobs caches effects created from a file, so this -// costs nothing beyond the Tiles source's own load. +// the Tiles equivalents; each call to gs_effect_create_from_file() allocates +// its own effect (libobs does NOT cache or dedupe effects created from a +// file, despite what an earlier version of this comment said) -- see the +// caution beside s_meter_effect in the .cpp for what believing otherwise +// costs. void corevideo_loudness_meter_load_gfx(); void corevideo_loudness_meter_unload_gfx(); diff --git a/src/zoom-participant-audio-source.cpp b/src/zoom-participant-audio-source.cpp index f57ae5c8..ac61ee0c 100644 --- a/src/zoom-participant-audio-source.cpp +++ b/src/zoom-participant-audio-source.cpp @@ -205,18 +205,33 @@ std::vector corevideo_loudness_readings() std::vector out; // Same lock order as corevideo_audio_source_infos(): g_sources_mtx first, // then each source's own mutex. The audio lane takes only ctx->mtx and - // never touches g_sources_mtx, so this can never invert. + // never touches g_sources_mtx, so this can never invert. Unlike that + // function, this one takes each ctx->mtx with try_lock rather than + // lock() -- see the comment at the try_lock site below -- so it never + // waits on g_sources_mtx's own critical section for longer than the + // registry walk itself. std::lock_guard lk(g_sources_mtx); out.reserve(g_sources.size()); for (CoreVideoAudioSource *ctx : g_sources) { if (!ctx) continue; LoudnessReading r; r.source_uuid = ctx->source_uuid; + r.kind = ctx->kind; r.participant_id = ctx->current_participant_id.load(std::memory_order_acquire); r.subscribed = ctx->subscribed.load(std::memory_order_acquire); { - std::lock_guard ctx_lk(ctx->mtx); + // try_lock, not lock: output_audio_frame() holds this exact mutex + // across a whole drain -- shm_region_open_readwrite(), + // obs_source_output_audio(), rate-limited blog() calls that write + // to disk -- and this runs on the OBS graphics thread via + // meter_video_tick(). A blocking lock() here would let one busy + // audio source stall the graphics thread for the length of its + // drain. Skipping a busy source for this poll costs nothing an + // operator can see: its numbers reappear on the next 100 ms tick, + // invisible on a board that redraws at 10 Hz anyway. + std::unique_lock ctx_lk(ctx->mtx, std::try_to_lock); + if (!ctx_lk.owns_lock()) continue; r.display_name = ctx->display_name; r.has_short_term = loudness_meter_short_term(ctx->loudness, &r.short_term_lufs); diff --git a/src/zoom-tiles-effect.h b/src/zoom-tiles-effect.h index 480e361d..eebe5a4b 100644 --- a/src/zoom-tiles-effect.h +++ b/src/zoom-tiles-effect.h @@ -94,8 +94,12 @@ struct TilesEffect { bool tiles_effect_load(TilesEffect &out); // Drops our reference to the effect and resets the struct. Safe to call on -// an unloaded/failed TilesEffect. Note: libobs caches effects created from a -// file, so gs_effect_destroy() here is effectively a no-op until graphics -// shutdown rather than an immediate release — not a leak, just not the -// literal "release" the name suggests. +// an unloaded/failed TilesEffect. Note: gs_effect_create_from_file() allocates +// a fresh gs_effect_t on every call -- libobs does NOT cache or dedupe +// effects compiled from the same file (an earlier version of this comment +// claimed otherwise). gs_effect_destroy() here is a real, immediate release +// of THIS handle. Every caller that loads its own TilesEffect from this file +// (the Tiles wall and the Loudness Meter each compile an independent copy) +// owns exactly one handle and must destroy it exactly once; two owners must +// never share one gs_effect_t*, or the second destroy is a double-free. void tiles_effect_destroy(TilesEffect &fx); diff --git a/tests/audio-loudness-test.cpp b/tests/audio-loudness-test.cpp index 144631a1..35984c7d 100644 --- a/tests/audio-loudness-test.cpp +++ b/tests/audio-loudness-test.cpp @@ -449,6 +449,17 @@ int main() "a degenerate feed advanced the measurement"); } + // ── loudness_meter_configure() reserves the gated vector up front ────── + // so the audio lane is provably allocation-free after configure -- no + // realloc can land on the media path once a source is subscribed. + { + LoudnessMeter m; + loudness_meter_configure(m, 48000, 1); + check(m.gated.capacity() >= kLoudnessMaxGatedBlocks, + "loudness_meter_configure() did not reserve the gated vector " + "to kLoudnessMaxGatedBlocks"); + } + if (failures == 0) std::cout << "audio-loudness: all tests passed\n"; return failures == 0 ? 0 : 1; diff --git a/tests/loudness-board-test.cpp b/tests/loudness-board-test.cpp index 38db91ba..2b6aec15 100644 --- a/tests/loudness-board-test.cpp +++ b/tests/loudness-board-test.cpp @@ -126,6 +126,78 @@ int main() "an empty panel produced a reference value"); } + // ── CRITICAL: only Participant-kind readings vote on the median ──────── + // ActiveSpeaker resolves to whichever Participant is currently talking + // (a duplicate vote for that person) and Audience is the whole-meeting + // mix (no one microphone at all). A median computed over a vector that + // mixes all three kinds must equal the median of the Participants alone. + { + std::vector participants_only = { + measured("Ana", -18.0), measured("Ben", -21.0), + measured("Cara", -23.0), measured("Dev", -24.0), + }; + double participants_median = 0.0; + check(loudness_panel_median(participants_only, kLoudnessBoardMinBlocks, + &participants_median), + "no median from the Participants-only panel"); + + std::vector mixed = participants_only; + LoudnessReading speaker = measured("ActiveSpeakerDup", -21.0); + speaker.kind = CoreVideoAudioKind::ActiveSpeaker; + mixed.push_back(speaker); + LoudnessReading audience = measured("WholeRoomMix", -12.0); + audience.kind = CoreVideoAudioKind::Audience; + audience.display_name.clear(); + audience.participant_id = 0; + mixed.push_back(audience); + + double mixed_median = 0.0; + check(loudness_panel_median(mixed, kLoudnessBoardMinBlocks, + &mixed_median), + "no median from the mixed-kind panel"); + check(near(mixed_median, participants_median, 1e-9), + "an ActiveSpeaker or Audience reading voted on the panel " + "median -- only Participant-kind readings may"); + + // The board must not render ActiveSpeaker/Audience as rows either -- + // a duplicate row for the current speaker, or a phantom + // "- unassigned -" row for the room mix, is the same defect wearing + // a different face. + LoudnessBoardModel model = loudness_board_build( + mixed, LoudnessReference::PanelMedian, + kLoudnessBoardDefaultToleranceLu, kLoudnessBoardMinBlocks); + check(model.rows.size() == participants_only.size(), + "a non-Participant reading produced a row on the board"); + for (const LoudnessBoardRow &row : model.rows) { + check(row.name != "ActiveSpeakerDup" && row.name != "WholeRoomMix" && + row.name != "- unassigned -", + "a non-Participant reading's name leaked onto the board"); + } + } + + // ── A single qualifying panelist is always its own reference ─────────── + // With only one Participant clearing the minimum block count, the median + // degenerates to that one value, so their own deviation is always 0 LU -- + // they must always pass, regardless of what absolute level they measured + // at. + { + std::vector panel = { + measured("Solo", -41.0), // far outside any EBU/ATSC/streaming target + }; + LoudnessBoardModel model = loudness_board_build( + panel, LoudnessReference::PanelMedian, + kLoudnessBoardDefaultToleranceLu, kLoudnessBoardMinBlocks); + check(model.has_reference, "a single qualifying panelist produced no reference"); + check(near(model.reference_lufs, -41.0, 1e-9), + "a single panelist's own reading is not their own reference"); + check(model.rows.size() == 1, "wrong row count for a solo panelist"); + check(model.rows[0].has_deviation && near(model.rows[0].deviation_lu, 0.0, 1e-9), + "a solo panelist's deviation from their own reference is not 0 LU"); + check(model.rows[0].status == LoudnessRowStatus::Pass, + "a solo panelist -- always their own reference -- must always " + "pass, at any absolute loudness level"); + } + // ── Deviation sign, and status ───────────────────────────────────────── { std::vector panel = { @@ -219,6 +291,68 @@ int main() "a still-measuring panelist was given a deviation"); } + // ── IMPORTANT 2: the deviation bar is driven by short-term deviation, + // falling back to integrated deviation, never vanishing ───────────────── + { + // A row with both short-term and integrated deviation available: + // the bar must prefer short-term. + LoudnessBoardRow both; + both.has_short_term_deviation = true; + both.short_term_deviation_lu = 1.5; + both.has_deviation = true; + both.deviation_lu = -3.0; + double bar_lu = 0.0; + check(loudness_board_bar_input(both, &bar_lu), + "no bar input produced when both deviations were available"); + check(near(bar_lu, 1.5, 1e-9), + "the bar did not prefer short-term deviation over integrated"); + + // Short-term unavailable: falls back to integrated, not to nothing. + LoudnessBoardRow integrated_only; + integrated_only.has_deviation = true; + integrated_only.deviation_lu = -3.0; + bar_lu = 0.0; + check(loudness_board_bar_input(integrated_only, &bar_lu), + "the bar vanished when short-term deviation was unavailable " + "instead of falling back to integrated"); + check(near(bar_lu, -3.0, 1e-9), + "the bar's fallback value was not the integrated deviation"); + + // Neither available: no bar input at all. + LoudnessBoardRow neither; + bar_lu = 12345.0; + check(!loudness_board_bar_input(neither, &bar_lu), + "a bar input was produced with neither deviation available"); + } + + // ── loudness_board_build() populates short-term deviation, and it is NOT + // part of the signature ───────────────────────────────────────────────── + { + std::vector panel = { + measured("Ana", -20.0, -20.0, 200), + measured("Ben", -24.0, -18.0, 200), // integrated -24, short-term -18 + }; + const LoudnessBoardModel m1 = loudness_board_build( + panel, LoudnessReference::PanelMedian, 2.0, kLoudnessBoardMinBlocks); + check(m1.has_reference, "no reference produced"); + // reference is the median of the two integrated values: -22. + check(m1.rows[1].has_short_term_deviation, + "Ben's short-term deviation was not populated"); + check(near(m1.rows[1].short_term_deviation_lu, -18.0 - m1.reference_lufs, + 1e-9), + "Ben's short-term deviation is not short_term_lufs - reference"); + + // Moving ONLY the short-term value must not change the signature -- + // folding it in would rebuild every row's text children ~10x/sec for + // a value the row text never shows (see loudness_board_bar_input()). + panel[1].short_term_lufs = -10.0; + const LoudnessBoardModel m2 = loudness_board_build( + panel, LoudnessReference::PanelMedian, 2.0, kLoudnessBoardMinBlocks); + check(m1.signature == m2.signature, + "the board signature changed when only short-term loudness " + "moved -- short-term must not force a text-child rebuild"); + } + // ── The signature changes on content and NOT on input order ──────────── // The Talkback dock shipped a live defect (2026-08-29) where a merely // REORDERED roster rebuilt the whole widget list several times a second From 7e5c2d030cd926f6de2e9829ea40b284be5b4449 Mon Sep 17 00:00:00 2001 From: Grok Date: Sat, 5 Sep 2026 17:56:40 -0400 Subject: [PATCH 15/15] Fix two regressions from the panelist loudness fix wave Regression A: corevideo_loudness_readings() continue'd past out.push_back(r) on a failed ctx->mtx try_lock, dropping the whole reading rather than just its numbers. That shrank the row count loudness_board_row_rect() divides the canvas by (every other row visibly resizes), dropped that source's vote from the panel median for one poll (every other panelist's pass/fail could move for 100ms), and changed model.signature (forcing the full child-text rebuild the signature gate exists to prevent) -- against a mutex that is busy often, not rarely, since output_audio_frame() holds it across a real drain. The reading is now always pushed with only its measurement fields left at their unavailable defaults, which loudness_board_build() already renders correctly as NoAudio. display_name moved to its own dedicated name_mtx (instead of ctx->mtx) so a row's identity survives a poll where its numbers don't -- otherwise the name itself would have flickered to "- unassigned -" on exactly the polls this was meant to fix. Regression B: has_short_term_deviation was populated whenever a short-term reading and a reference existed, with no gate on the row having a real verdict. loudness_meter_short_term() is ungated, so a silent source's noise floor (~-90 LUFS) against a real reference produced a full-length bar pegged hard left, in idle grey, on every silent row -- the board's normal state. Now gated on has_deviation itself (same Pass/Loud/Quiet condition), so NoAudio and Measuring rows draw no bar at all. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01WqDL6gcoVvEMXv7Qwcni9n --- CLAUDE.md | 57 +++++++++++++++++----- src/loudness-board.h | 46 ++++++++++++------ src/zoom-participant-audio-source.cpp | 70 ++++++++++++++++++++------- tests/loudness-board-test.cpp | 64 ++++++++++++++++++++++++ 4 files changed, 193 insertions(+), 44 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 834f0825..e96c59c5 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1288,7 +1288,20 @@ Every one of these is documented at length where it lives; the list is the map. `loudness_board_needs_label_refresh()` exists to prevent. Where a short-term reading or a reference is unavailable, `loudness_board_bar_input()` falls back to the integrated deviation the bar always used, rather than showing no - bar at all. + bar at all. **Regression B, same-day re-review**: the first cut populated + `has_short_term_deviation` whenever a short-term reading and a reference + existed, with no gate on whether the row had a real verdict yet. + `loudness_meter_short_term()` is UNGATED, so a subscribed-but-silent source + reads its own noise floor (e.g. -90 LUFS) -- against a ~-22 LUFS reference + that is a ~-68 LU deviation, clamped to a full-length bar pegged hard left, + in idle grey, right next to the text "no audio"/"measuring". A silent + preshow is this board's NORMAL state, so this was the common case, not an + edge one. `has_short_term_deviation` is now gated on `has_deviation` + itself (the same Pass/Loud/Quiet verdict condition) rather than restated + independently -- which also guarantees the integrated fallback is always + available whenever the short-term value is. NoAudio and Measuring rows now + draw no bar at all, pinned in `tests/loudness-board-test.cpp` by a case + with a stray -90 LUFS short-term reading on an otherwise-silent row. - **No `log10` in the relative gate's hot loop** (`loudness_meter_integrated()`, `src/audio-loudness.h`). Pass 2 used to convert every gated block (up to `kLoudnessMaxGatedBlocks` = 6000) to LUFS @@ -1306,17 +1319,37 @@ Every one of these is documented at length where it lives; the list is the map. `kLoudnessMaxGatedBlocks` up front, so the audio lane is provably allocation-free after configure. - **`corevideo_loudness_readings()` uses `try_lock` on each source's mutex, - never a blocking `lock()`** (final whole-branch review, 2026-09-05, - Important). It runs on the OBS graphics thread via `meter_video_tick()` and - takes the exact `ctx->mtx` that `output_audio_frame()` holds across an - entire drain -- including `shm_region_open_readwrite()`, - `obs_source_output_audio()`, and rate-limited `blog()` calls that write to - disk. A blocking lock here could stall the graphics thread for the length - of one source's drain. A source that is busy is simply skipped for that - poll; its numbers reappear on the next 100 ms tick, invisible on a board - that redraws at 10 Hz. `corevideo_audio_source_infos()` (the sibling - registry walk) is unchanged and still blocks -- it is called far less often - and from a different context. + never a blocking `lock()`, and a failed try_lock skips the MEASUREMENT, + never the ROW** (final whole-branch review, 2026-09-05, Important, plus a + same-day re-review regression). It runs on the OBS graphics thread via + `meter_video_tick()` and takes the exact `ctx->mtx` that + `output_audio_frame()` holds across an entire drain -- including + `shm_region_open_readwrite()`, `obs_source_output_audio()`, and + rate-limited `blog()` calls that write to disk -- so a blocking lock here + could stall the graphics thread for the length of one source's drain. + `corevideo_audio_source_infos()` (the sibling registry walk) is unchanged + and still blocks -- it is called far less often and from a different + context. **Regression A, same day**: the first cut `continue`d past + `out.push_back(r)` on a failed try_lock, which drops the whole reading, not + just its numbers -- one fewer row for `loudness_board_row_rect()` to divide + the canvas by (every OTHER row visibly resizes for one poll), one fewer + vote for that poll's panel median (every OTHER panelist's pass/fail can + move for 100 ms), and a changed `model.signature` that forces exactly the + full child-text rebuild `loudness_board_needs_label_refresh()` exists to + prevent -- on a mutex that is busy often, not rarely, since + `output_audio_frame()` holds it across a real drain. The reading is now + ALWAYS pushed; only the measurement fields (`has_short_term`/ + `has_integrated`/`gated_blocks`) are left at their unavailable defaults + when the try_lock fails, which `loudness_board_build()` already renders + correctly as NoAudio -- the same state a source that has simply never + spoken gets. The row's identity (`source_uuid`, `kind`, `participant_id`) + was always readable without `ctx->mtx`, but `display_name` was not: it is + now guarded by its own dedicated `name_mtx` (`CoreVideoAudioSource`, next + to the field) instead of `ctx->mtx`, specifically so a busy audio drain + cannot make a row's NAME flicker to "- unassigned -" on the exact polls + where its numbers go missing. All three `display_name` writers + (`unsubscribe_audio()`, `forget_subscription_for_new_engine()`, the roster + callback) moved to `name_mtx` alongside the reader. ## Live testing against a real meeting diff --git a/src/loudness-board.h b/src/loudness-board.h index 47373a66..2f14b76d 100644 --- a/src/loudness-board.h +++ b/src/loudness-board.h @@ -104,13 +104,15 @@ struct LoudnessBoardRow { bool has_short_term = false; double short_term_lufs = 0.0; // Short-term deviation from the panel reference, i.e. short_term_lufs - - // reference. Populated whenever both a short-term reading and a - // reference exist, independent of `has_deviation` (which gates on - // gated_blocks meeting kLoudnessBoardMinBlocks -- a verdict requirement - // that a live, ungated number should not inherit). This is what - // loudness_board_bar_input() prefers: the bar is meant to move live while - // the panelist talks, and short-term is the fast measure that does that; - // the row TEXT stays on the integrated verdict. + // reference. Populated only when the row ALSO has a real integrated + // verdict (`has_deviation`, i.e. cleared min_blocks against an + // established reference) -- loudness_meter_short_term() is UNGATED, so a + // silent or barely-started source reads its own noise floor as a huge, + // meaningless deviation, and this field must not carry that onto the + // board. This is what loudness_board_bar_input() prefers when it exists: + // the bar is meant to move live while the panelist talks, and short-term + // is the fast measure that does that; the row TEXT stays on the + // integrated verdict. bool has_short_term_deviation = false; double short_term_deviation_lu = 0.0; bool has_integrated = false; @@ -273,13 +275,6 @@ inline LoudnessBoardModel loudness_board_build( row.has_integrated = r->has_integrated; row.integrated_lufs = r->integrated_lufs; - if (row.has_short_term && model.has_reference && - std::isfinite(row.short_term_lufs)) { - row.has_short_term_deviation = true; - row.short_term_deviation_lu = - row.short_term_lufs - model.reference_lufs; - } - if (!r->has_integrated || r->gated_blocks == 0) { row.status = LoudnessRowStatus::NoAudio; } else if (r->gated_blocks < min_blocks) { @@ -297,6 +292,29 @@ inline LoudnessBoardModel loudness_board_build( row.status = LoudnessRowStatus::Measuring; } row.detail = loudness_row_status_text(row.status); + + // Short-term deviation is gated on the row already having a REAL + // VERDICT -- the same condition that sets `has_deviation` above + // (Pass/Loud/Quiet, i.e. cleared min_blocks against an established + // reference) -- NOT merely on a short-term reading existing. + // loudness_meter_short_term() is UNGATED -- a subscribed-but-silent + // source reads its own noise floor (e.g. -90 LUFS), which against a + // ~-22 LUFS panel reference is a ~-68 LU deviation, clamped to full + // scale. Without this gate a silent preshow -- this board's NORMAL + // state, and a panelist who has merely started talking but not yet + // cleared min_blocks (Measuring) -- would paint a full-length bar + // pegged hard left, in idle grey, right next to text reading "no + // audio" or "measuring": it reads as "far too quiet" when the truth + // is "not measured at all". Piggybacking on `has_deviation` rather + // than restating its condition also means loudness_board_bar_input()'s + // integrated fallback is always available whenever the short-term + // one is populated -- the bar can still fall back, never vanish. + if (row.has_deviation && row.has_short_term && + std::isfinite(row.short_term_lufs)) { + row.has_short_term_deviation = true; + row.short_term_deviation_lu = + row.short_term_lufs - model.reference_lufs; + } model.rows.push_back(std::move(row)); } diff --git a/src/zoom-participant-audio-source.cpp b/src/zoom-participant-audio-source.cpp index ac61ee0c..b4601857 100644 --- a/src/zoom-participant-audio-source.cpp +++ b/src/zoom-participant-audio-source.cpp @@ -108,8 +108,16 @@ struct CoreVideoAudioSource { // request rather than a direct call: touching the meter from the caller's // thread would race the drain that is filling it. std::atomic loudness_reset_requested{false}; - // Display name, cached on the roster callback. Guarded by ctx->mtx. + // Display name, cached on the roster callback. Guarded by its OWN mutex, + // name_mtx, deliberately NOT ctx->mtx: corevideo_loudness_readings() has + // to be able to read a row's identity even on a poll where ctx->mtx's + // try_lock fails because output_audio_frame() is mid-drain (SHM open, + // obs_source_output_audio(), disk-writing blog() calls) -- see that + // function. A name that only updates on a roster callback is cheap + // enough to give its own uncontended mutex rather than fight the audio + // lane for the busy one. std::string display_name; + std::mutex name_mtx; // Next ring slot this source will drain. Only the engine reader thread // touches it, the same thread that owns `timeline`. uint32_t read_index = 0; @@ -220,24 +228,44 @@ std::vector corevideo_loudness_readings() r.participant_id = ctx->current_participant_id.load(std::memory_order_acquire); r.subscribed = ctx->subscribed.load(std::memory_order_acquire); + // The name comes from its OWN mutex (name_mtx), never from ctx->mtx, + // and is read unconditionally -- this row's identity must survive a + // poll where the measurement below is skipped. + { + std::lock_guard name_lk(ctx->name_mtx); + r.display_name = ctx->display_name; + } + // try_lock, not lock: output_audio_frame() holds ctx->mtx across a + // whole drain -- shm_region_open_readwrite(), obs_source_output_audio(), + // rate-limited blog() calls that write to disk -- and this runs on + // the OBS graphics thread via meter_video_tick(). A blocking lock() + // here would let one busy audio source stall the graphics thread for + // the length of its drain. + // + // On a failed try_lock the READING IS STILL PUSHED, just with its + // measurement fields left at their unavailable defaults + // (has_short_term/has_integrated false, gated_blocks 0) -- this used + // to `continue` here, which dropped the row entirely: one fewer row + // for loudness_board_row_rect() to divide the canvas by (every OTHER + // row visibly resizes), one fewer vote for that poll's panel median + // (every OTHER panelist's pass/fail can move for 100 ms), and a + // changed `model.signature` that forces the very child-text rebuild + // the signature gate exists to prevent. A reading with no + // measurement is the same "not measured yet" state + // loudness_board_build() already renders as NoAudio/Measuring for a + // source that has simply never spoken -- the honest fix is a + // temporarily-empty reading, not a temporarily-missing row. Its real + // numbers reappear on the very next 100 ms poll once the drain + // releases the mutex. { - // try_lock, not lock: output_audio_frame() holds this exact mutex - // across a whole drain -- shm_region_open_readwrite(), - // obs_source_output_audio(), rate-limited blog() calls that write - // to disk -- and this runs on the OBS graphics thread via - // meter_video_tick(). A blocking lock() here would let one busy - // audio source stall the graphics thread for the length of its - // drain. Skipping a busy source for this poll costs nothing an - // operator can see: its numbers reappear on the next 100 ms tick, - // invisible on a board that redraws at 10 Hz anyway. std::unique_lock ctx_lk(ctx->mtx, std::try_to_lock); - if (!ctx_lk.owns_lock()) continue; - r.display_name = ctx->display_name; - r.has_short_term = loudness_meter_short_term(ctx->loudness, - &r.short_term_lufs); - r.has_integrated = loudness_meter_integrated(ctx->loudness, - &r.integrated_lufs); - r.gated_blocks = loudness_meter_gated_blocks(ctx->loudness); + if (ctx_lk.owns_lock()) { + r.has_short_term = loudness_meter_short_term(ctx->loudness, + &r.short_term_lufs); + r.has_integrated = loudness_meter_integrated(ctx->loudness, + &r.integrated_lufs); + r.gated_blocks = loudness_meter_gated_blocks(ctx->loudness); + } } out.push_back(std::move(r)); } @@ -339,6 +367,9 @@ static void unsubscribe_audio(CoreVideoAudioSource *ctx) // one after a gap of unknown length. Either way the previous // window's gated blocks describe audio that is not this check. loudness_meter_reset_window(ctx->loudness); + } + { + std::lock_guard name_lk(ctx->name_mtx); ctx->display_name.clear(); } } @@ -421,6 +452,9 @@ static void forget_subscription_for_new_engine(CoreVideoAudioSource *ctx) audio_timeline_reset(ctx->timeline); ctx->read_started = false; loudness_meter_reset_window(ctx->loudness); + } + { + std::lock_guard name_lk(ctx->name_mtx); ctx->display_name.clear(); } if (!was_subscribed) return; @@ -493,7 +527,7 @@ static void maybe_resubscribe_for_roster(CoreVideoAudioSource *ctx) } } { - std::lock_guard lk(ctx->mtx); + std::lock_guard name_lk(ctx->name_mtx); ctx->display_name = cached_name; } diff --git a/tests/loudness-board-test.cpp b/tests/loudness-board-test.cpp index 2b6aec15..8b0a28c8 100644 --- a/tests/loudness-board-test.cpp +++ b/tests/loudness-board-test.cpp @@ -353,6 +353,70 @@ int main() "moved -- short-term must not force a text-child rebuild"); } + // ── REGRESSION B: an unmeasured row must draw NO bar, even with a + // short-term reading ──────────────────────────────────────────────────── + // loudness_meter_short_term() is ungated: a subscribed-but-silent source + // still reads its own noise floor (e.g. -90 LUFS), which against a real + // panel reference is a huge, meaningless deviation. NoAudio (never + // spoken / zero gated blocks) and Measuring (some gated blocks, not yet + // enough for a verdict) must both refuse to populate + // has_short_term_deviation, and loudness_board_bar_input() must refuse + // to produce ANY bar for them -- a silent preshow is this board's normal + // state, and it must not paint a full-length bar pegged hard left on + // every row in it. + { + std::vector panel = { + measured("Ana", -20.0, -20.0, 200), // establishes a reference + measured("Ben", -20.0, -20.0, 200), + }; + // NoAudio: subscribed, never spoken, but with a stray short-term + // reading (the shape a noise-floor read would actually take). + LoudnessReading silent_with_noise_floor; + silent_with_noise_floor.source_uuid = "uuid_Cara"; + silent_with_noise_floor.display_name = "Cara"; + silent_with_noise_floor.subscribed = true; + silent_with_noise_floor.has_short_term = true; + silent_with_noise_floor.short_term_lufs = -90.0; + silent_with_noise_floor.has_integrated = false; + silent_with_noise_floor.gated_blocks = 0; + panel.push_back(silent_with_noise_floor); + + // Measuring: a handful of gated blocks (well under + // kLoudnessBoardMinBlocks) with a short-term reading that has not + // yet converged on real speech either. + panel.push_back(measured("Dev", -20.0, -80.0, 5)); + + const LoudnessBoardModel m = loudness_board_build( + panel, LoudnessReference::PanelMedian, 2.0, kLoudnessBoardMinBlocks); + check(m.has_reference, "no reference produced"); + + const LoudnessBoardRow *cara = nullptr; + const LoudnessBoardRow *dev = nullptr; + for (const LoudnessBoardRow &row : m.rows) { + if (row.name == "Cara") cara = &row; + if (row.name == "Dev") dev = &row; + } + check(cara != nullptr && dev != nullptr, "expected rows missing"); + + check(cara->status == LoudnessRowStatus::NoAudio, + "Cara was not reported as NoAudio"); + check(!cara->has_short_term_deviation, + "a NoAudio row with a stray short-term reading populated a " + "short-term deviation"); + double bar_lu = 0.0; + check(!loudness_board_bar_input(*cara, &bar_lu), + "a NoAudio row produced a bar input -- a silent preshow must " + "draw no bar, not a full-length one from the noise floor"); + + check(dev->status == LoudnessRowStatus::Measuring, + "Dev was not reported as Measuring"); + check(!dev->has_short_term_deviation, + "a still-Measuring row populated a short-term deviation"); + bar_lu = 0.0; + check(!loudness_board_bar_input(*dev, &bar_lu), + "a still-Measuring row produced a bar input"); + } + // ── The signature changes on content and NOT on input order ──────────── // The Talkback dock shipped a live defect (2026-08-29) where a merely // REORDERED roster rebuilt the whole widget list several times a second