Skip to content

The test log capture is a process-global layer routing to a task-local sink (#542) - #565

Merged
emooreatx merged 1 commit into
mainfrom
fix/542-log-capture-rebuilds-callsite-interest
Sep 8, 2026
Merged

The test log capture is a process-global layer routing to a task-local sink (#542)#565
emooreatx merged 1 commit into
mainfrom
fix/542-log-capture-rebuilds-callsite-interest

Conversation

@emooreatx

Copy link
Copy Markdown
Contributor

Three CI runs failed retention_loop's steady-state test with an EMPTY
capture; once looked for, 1 in 4 local runs did too, with the diagnostic
naming it a capture failure rather than a missing line. The cause is in
tracing-core, not in run_pass: a callsite's Interest is cached process-wide,
and when exactly one scoped dispatcher is alive it is recomputed from the
REGISTERING thread's current default. A sibling test's tokio::spawned
retention loop hits run_pass's callsites first, on a worker thread whose
default is the global (or nothing), and caches them never; the scoped
with_subscriber capture on the test thread is then never asked, because
the tracing::info! macro returns before it consults the current dispatcher.
Forcing a rebuild from the test thread makes it worse for the same reason.

The capture is now ONE global default that is always interested and routes
each event to the capture active for the current task (a tokio task-local),
dropping events from tasks with no capture. Interest is always from every
thread; where an event goes is decided at dispatch time, per task. A test
binary that installs its own global subscriber before the first capture is
refused with a message rather than silently capturing nothing.

retention_loop 12/12 green (was 3/4); capacity_scorer, replication_reconcile
and trace_plane_release_gate 3/3 each. The steady-state assertion now
renders through render_or_explain, so a recurrence says "capture failure"
instead of "no INFO line".

Closes #542.

Reproduction, before: for i in 1..12; cargo test --test retention_loop failed 3 of 12 with (NO EVENTS CAPTURED AT ALL — this is a capture failure …); a probe event through the scoped dispatch was missing on the failing thread while a sibling thread's probe was present. After: 12/12.

🤖 Generated with Claude Code

…l sink, so a sibling test cannot cache its callsites as never-interested (#542)

Three CI runs failed retention_loop's steady-state test with an EMPTY
capture; once looked for, 1 in 4 local runs did too, with the diagnostic
naming it a capture failure rather than a missing line. The cause is in
tracing-core, not in run_pass: a callsite's Interest is cached process-wide,
and when exactly one scoped dispatcher is alive it is recomputed from the
REGISTERING thread's current default. A sibling test's tokio::spawned
retention loop hits run_pass's callsites first, on a worker thread whose
default is the global (or nothing), and caches them `never`; the scoped
`with_subscriber` capture on the test thread is then never asked, because
the tracing::info! macro returns before it consults the current dispatcher.
Forcing a rebuild from the test thread makes it worse for the same reason.

The capture is now ONE global default that is always interested and routes
each event to the capture active for the current task (a tokio task-local),
dropping events from tasks with no capture. Interest is `always` from every
thread; where an event goes is decided at dispatch time, per task. A test
binary that installs its own global subscriber before the first capture is
refused with a message rather than silently capturing nothing.

retention_loop 12/12 green (was 3/4); capacity_scorer, replication_reconcile
and trace_plane_release_gate 3/3 each. The steady-state assertion now
renders through `render_or_explain`, so a recurrence says "capture failure"
instead of "no INFO line".

Closes #542.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@emooreatx
emooreatx merged commit 7b06270 into main Sep 8, 2026
9 checks passed
@emooreatx
emooreatx deleted the fix/542-log-capture-rebuilds-callsite-interest branch September 8, 2026 13:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant