The test log capture is a process-global layer routing to a task-local sink (#542) - #565
Merged
Merged
Conversation
…l sink, so a sibling test cannot cache its callsites as never-interested (#542) Three CI runs failed retention_loop's steady-state test with an EMPTY capture; once looked for, 1 in 4 local runs did too, with the diagnostic naming it a capture failure rather than a missing line. The cause is in tracing-core, not in run_pass: a callsite's Interest is cached process-wide, and when exactly one scoped dispatcher is alive it is recomputed from the REGISTERING thread's current default. A sibling test's tokio::spawned retention loop hits run_pass's callsites first, on a worker thread whose default is the global (or nothing), and caches them `never`; the scoped `with_subscriber` capture on the test thread is then never asked, because the tracing::info! macro returns before it consults the current dispatcher. Forcing a rebuild from the test thread makes it worse for the same reason. The capture is now ONE global default that is always interested and routes each event to the capture active for the current task (a tokio task-local), dropping events from tasks with no capture. Interest is `always` from every thread; where an event goes is decided at dispatch time, per task. A test binary that installs its own global subscriber before the first capture is refused with a message rather than silently capturing nothing. retention_loop 12/12 green (was 3/4); capacity_scorer, replication_reconcile and trace_plane_release_gate 3/3 each. The steady-state assertion now renders through `render_or_explain`, so a recurrence says "capture failure" instead of "no INFO line". Closes #542. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three CI runs failed retention_loop's steady-state test with an EMPTY
capture; once looked for, 1 in 4 local runs did too, with the diagnostic
naming it a capture failure rather than a missing line. The cause is in
tracing-core, not in run_pass: a callsite's Interest is cached process-wide,
and when exactly one scoped dispatcher is alive it is recomputed from the
REGISTERING thread's current default. A sibling test's tokio::spawned
retention loop hits run_pass's callsites first, on a worker thread whose
default is the global (or nothing), and caches them
never; the scopedwith_subscribercapture on the test thread is then never asked, becausethe tracing::info! macro returns before it consults the current dispatcher.
Forcing a rebuild from the test thread makes it worse for the same reason.
The capture is now ONE global default that is always interested and routes
each event to the capture active for the current task (a tokio task-local),
dropping events from tasks with no capture. Interest is
alwaysfrom everythread; where an event goes is decided at dispatch time, per task. A test
binary that installs its own global subscriber before the first capture is
refused with a message rather than silently capturing nothing.
retention_loop 12/12 green (was 3/4); capacity_scorer, replication_reconcile
and trace_plane_release_gate 3/3 each. The steady-state assertion now
renders through
render_or_explain, so a recurrence says "capture failure"instead of "no INFO line".
Closes #542.
Reproduction, before:
for i in 1..12; cargo test --test retention_loopfailed 3 of 12 with(NO EVENTS CAPTURED AT ALL — this is a capture failure …); a probe event through the scoped dispatch was missing on the failing thread while a sibling thread's probe was present. After: 12/12.🤖 Generated with Claude Code