feat(monitor): sample per-thread CPU on macOS via mach thread_info

[wagenet]

Why

Per-thread CPU sampling read /proc/{pid}/task/{tid}/stat and /proc/stat behind
cfg(target_os = "linux"), and the other arm returned nothing. The thread and CPU
views were therefore empty on macOS, so there was no way to see how GStreamer
pipeline threads are actually scheduled.

What

A macOS backend using mach's thread_info(THREAD_BASIC_INFO), which reports
cumulative user+system time per thread. Same shape and output as the Linux path,
so the existing UI and API surface work unchanged.

mach has no /proc/stat equivalent, but the Linux denominator is by construction
elapsed wall time times the core count, so computing it directly yields the same
percentage with the same meaning: 100.0 is one saturated core, N * 100.0 the
maximum on N cores. Both platforms now share cpu_usage_percent() for that
conversion, which leaves the Linux result unchanged.

Port lifetime

The registry stores a mach thread port, not a pthread_t, because
thread_info needs a mach_port_t. Capturing that port correctly is the delicate
part of this PR.

pthread_mach_thread_np() — the obvious way to get it — returns the name without
taking a user reference. When the thread exits the name is freed and the kernel may
hand it to any other port; I saw port 77831 move from one element to another across
a flow restart. thread_info() on a name that has since been recycled to a
guarded port (libdispatch and XPC guard theirs) raises EXC_GUARD and the kernel
kills the process. It killed a running server:

EXC_GUARD / GUARD_TYPE_MACH_PORT / ILLEGAL_MOVE on mach port 161251
ThreadCpuSampler::sample <- AppState::get_thread_stats <- handle_socket_inner

hit while reconnecting WHIP publishers with the vision mixer page open: thread
churn plus an open stats WebSocket. There is no error return to check, so the
"non-success means the thread is gone" handling below cannot catch it.

So ThreadHandle captures the port with mach_thread_self() instead — the same
name, but with a send right — and releases it in Drop. Every way a registry entry
goes away releases the reference exactly once, because they are all HashMap
drops: unregister, unregister_flow, dropping the registry, and one register
overwriting another key. The reference lives in an Arc, so a snapshot from
get_all() keeps the port alive across the sampler's mach calls even if the entry
is unregistered concurrently.

read_thread_cpu_time() takes a &ThreadHandle rather than a u64, which makes
the invariant structural rather than a comment: a cached port name reaches
thread_info() only while something guarantees it still names the same thread.

A thread that has exited keeps its name — the handle holds it — and mach declines
to answer for it. That yields no sample and drops the stored baseline, and is not
logged: it recurs on every tick until the registry entry is cleaned up.

Threads registered from the session pad probe never post a Leave message, so
their handles are held until the flow's entries are dropped at stop. That is a
bounded number of port names per running flow.

No new dependency. libc already exposes thread_info, THREAD_BASIC_INFO,
thread_basic_info and time_value_t; it has deprecated the rest of its mach
surface in favour of the mach2 crate and never exposed mach_port_deallocate,
so the four symbols beyond that set are declared directly.

Measured on a running server

Two identical videotestsrc pattern=snow ! queue ! x264enc ! queue ! fakesink
chains on an M2 (4P + 4E), cross-checked against ps -M per-thread CPU-time
deltas over the same 10s window:

ps rank	ps delta	reported by this change
1-2	8.30 / 8.40%	src1/src2 7.7-8.8%
3-8	~5.1% x 6	x264enc internal workers - not registered, correctly absent
9-10	1.00 / 0.90%	q1a/q2a 1.08-1.11%

Identical chains report near-identical CPU. The six unregistered encoder workers
(~31%) account for the gap between the 20% registered sum and 51% process-wide.
Stopping the flow drops to zero threads with no error output; restarting
re-registers and reads 0.00% on the first sample rather than a bogus delta.

One caveat worth knowing

The percentage measures occupancy, not work completed. Neither platform's
clock adjusts for how fast the core is, so on heterogeneous CPUs two threads both
reading 100.0 can differ several-fold in throughput — I measured 4.5x on an M2
between QOS_CLASS_USER_INTERACTIVE and QOS_CLASS_BACKGROUND. A thread moved
onto a faster core reports a lower percentage for the same work. This matches
Linux semantics (including big.LITTLE) and standard tools, so it is not a
deviation, but it means this number alone cannot judge a core-placement change.
The second commit documents this where the function is defined.

Tests

Fourteen unit tests. Four cover the delta-to-percentage arithmetic without mach.
Three exercise real mach calls: an exited thread yields no sample, cumulative time
advances on a spinning thread, and the sampler reads a live thread end-to-end
through a ThreadRegistry. Seven cover port lifetime — the name stays allocated
after the thread exits, a clone keeps it alive after the original is dropped, a
get_all() snapshot survives a concurrent unregister, and every removal path
releases the reference.

thread_port_lifetime_test is the end-to-end guard: five start/stop cycles of a
real 32-thread pipeline through the real bus sync handler and the real sampler,
asserting that port names held by a registry snapshot are still allocated after
teardown. It shortens glib's thread-pool idle retention first — GStreamer returns
streaming threads to a pool that keeps idle ones for 15 seconds, so at the default
no thread exits within a test run and the test guards nothing.

Reverting the port capture to pthread_mach_thread_np() fails five tests including
that one, with mach port name 0x1603 for element 'queue5' was freed while a registry snapshot still held it. sampler_reports_nonzero_for_a_busy_thread
remains a guard for the sampling arm itself: reverting that falls back to the stub
and reads 0.0.

Ran on macOS (Apple Silicon, macOS 26.5, GStreamer via Homebrew):
cargo test --features efp — 554 lib tests and every integration test pass,
six full runs clean.
cargo clippy --all-targets --features efp -- -D warnings and cargo fmt --check
are clean.

Not run: anything on Linux. This machine has only aarch64-apple-darwin
installed and cross-building the GStreamer stack was not practical, so the shared
cpu_usage_percent() refactor on the Linux path has been reviewed by hand but
never compiled. That is the main thing worth a careful look in review — Linux CI
will be its first real check.

CI gap

The macOS CI job is workflow_dispatch-only, so nothing here is guarded on a pull
request: not the mach-backed sampling tests, and not the port-lifetime guards that
would catch a return of the crash above.

[srperens]

Verdict: Comment — sound design and the shared arithmetic is genuinely tested on Linux CI, but this is entirely new macOS-only FFI that this repo's CI has never compiled, which blocks approval per protocol regardless of the author's own local run.

Claims

Claim	Verdict	Evidence
cpu_usage_percent() extraction is a pure refactor, Linux math unchanged	CONFIRMED	system_monitor.rs diff: old inline (delta_thread as f32/delta_total as f32)*100.0*self.num_cpus as f32 moved verbatim into the new cpu_usage_percent(), Linux call site now just calls it with the same three values
macOS delta_total = elapsed_us * num_cpus fed through the same cpu_usage_percent() produces the same percentage semantics as Linux's /proc/stat-derived total	CONFIRMED (by algebra)	Linux's delta_total is already an N-core-aggregate quantity (from /proc/stat's summed cpu line), so (thread/(N·T))·100·N = (thread/T)·100; macOS constructs delta_total = T_us·N the same way, so the identical function cancels the same way — verified by working the substitution, and cross-checked against the new one_saturated_core_reads_100_percent test which asserts cpu_usage_percent(1_000, 8_000, 8) == 100.0
get_current_thread_native_id() captures the mach port on the streaming thread itself, before any exit race	CONFIRMED	backend/src/gst/thread_priority.rs:305,377,599 call get_current_thread_native_id() and pass the result straight into registry.register(...) (:373,657) from within the same thread closure — no pthread_t is ever stored or dereferenced cross-thread
ThreadRegistry::register signature matches the new test's call	CONFIRMED	backend/src/thread_registry.rs:44-50 fn register(&self, thread_id: u64, element_name: String, flow_id: FlowId, block_id: Option<String>, pinned_cpus: Option<Vec<usize>>) matches registry.register(port, "test-thread".to_string(), flow_id, None, None) in the new test
macOS FFI path (thread_info, THREAD_BASIC_INFO_COUNT, pthread_mach_thread_np) compiles and behaves as written	UNVERIFIED	gh pr checks 722 at head c7842e2c shows Build (macOS) as skipping; per CLAUDE.md/CI this job is workflow_dispatch-only and never runs on push/PR, so none of the three macOS-only tests (invalid_mach_port_yields_no_sample, live_mach_port_reports_advancing_cpu_time, sampler_reports_nonzero_for_a_busy_thread) have ever executed in this repo's CI. The PR body itself flags this as the main open risk

Diagnosis — root cause (macOS thread/CPU views empty, no sampling backend) is real and the fix is at the right layer: mirrors the existing Linux ThreadCpuSampler shape rather than adding a parallel code path, and the port-recycling handling (drop baseline on any thread_info failure, no distinction between "exited" and "invalid") is the correct conservative choice — a stale baseline against a recycled port name would produce a silent wrong reading rather than a visible error.

Radius — LOCAL to system_monitor.rs/thread_priority.rs, macOS-only cfg arms; Linux behavior is refactor-only and covered by Linux CI. No API/WebSocket/strom-types surface touched.

Tests & CI — Build (Linux x86_64/ARM64), Check (Linux), Check & Build (WASM), API Contract Check pass at c7842e2c, and the four platform-agnostic cpu_usage_percent unit tests run for real under Check (Linux). Dispatch before merge to actually compile and run the mach-backed tests: gh workflow run ci.yml --ref wagenet/macos-per-thread-cpu -f platforms=macos.

Confidence: HIGH

[srperens]

This supersedes my Comment review of c7842e2c. Head moved to 507f8401, adding one real commit ("hold a reference to a thread's mach port while it is registered") that fixes a genuine crash bug, so this is a full re-review.

Claims

Claim	Verdict	Evidence
ThreadHandle takes a real reference on the mach port (mach_thread_self()), not a bare name, released exactly once via Drop	CONFIRMED	backend/src/thread_handle.rs:68 — let port = unsafe { sys::mach_thread_self() }; and backend/src/thread_handle.rs:120 — sys::mach_port_deallocate(sys::mach_task_self_, self.0);
get_all() snapshots keep the port alive across a concurrent unregister	CONFIRMED	backend/src/thread_registry.rs:119 — threads.values().cloned().collect() clones the Arc<MachThreadPort>, exercised by backend/src/thread_registry.rs:244 — fn a_snapshot_outlives_a_concurrent_unregister()
The sampler reads the port through the handle, not a cached raw name	CONFIRMED	backend/src/system_monitor.rs:636 — let port = handle.mach_port();
A regression test drives real GStreamer threads through churn and asserts the name stays allocated while a snapshot holds it, freed only after drop	CONFIRMED	backend/tests/thread_port_lifetime_test.rs:161 — mach_port_name_is_allocated(name), before :171 — drop(snapshots);
macOS FFI path still isn't exercised by any CI job here	UNVERIFIED (unchanged)	gh pr checks 722 at 507f8401: Build (macOS) skipping — workflow_dispatch-only

Diagnosis — the right fix, not a workaround. The bug (a bare pthread_mach_thread_np() name cached past the capturing thread's exit, then handed to thread_info() after the kernel recycled it to a possibly-guarded port, raising unrecoverable EXC_GUARD) is closed by tying the name's lifetime to a send-right reference only Drop can release. All three former raw-id call sites (backend/src/gst/thread_priority.rs:308, :381, :562) now capture a ThreadHandle on the streaming thread itself, the only place mach_thread_self() is safe to call.

Radius — SHARED: ThreadRegistry::register's signature changed from u64 to ThreadHandle; confirmed all call sites updated — backend/src/gst/thread_priority.rs:377, :621, and the test-only caller at backend/src/system_monitor.rs:788 — ThreadHandle::current(),. No other module builds a ThreadInfo directly.

Nits (non-blocking) — backend/src/system_monitor.rs:552 still says "(see get_current_thread_native_id)", a function this PR deletes (grep finds no definition); should point at ThreadHandle::current. Also, backend/src/gst/thread_priority.rs:562 takes a handle before the :569 already-configured check, but the probe self-removes after its first firing per pad (:570, :630), so this is at most one wasted mach call per extra pad on a shared thread — not worth blocking on.

Tests & CI — Build (Linux x86_64/ARM64), Check (Linux), Check & Build (WASM), API Contract Check, sccache preflight pass at 507f8401. Build (macOS)/Build (Windows) skip, as before — this repo's CI never compiles the macOS code this PR is about. thread_port_lifetime_test.rs ran under Check (Linux) but its #[cfg(target_os = "macos")] assertions did not execute there. Static reading only — I ran nothing myself.

Verdict stands at Comment: the fix is sound and well-tested by design, but proof the macOS code compiles and behaves is still unverified by CI. Recommend gh workflow run ci.yml --ref <branch> -f platforms=macos before merge.

Confidence: HIGH

[wagenet]

(from Claude, on behalf of @wagenet)

This PR is unapproved on the Build (macOS) coverage gap alone — no code change is requested. That blocker is shared with #721, #722, #726 and #735, and the documented remedy (gh workflow run ci.yml --ref <branch> -f platforms=macos) appears not to work for fork branches. Written up once in #735 (comment) rather than repeated on each PR.