timed_allocation_audit flaps on ubuntu: a stray allocation enters the measured region

[zackees]

Summary

benchmark-suite/tests/timed_allocation_audit.rs fails nondeterministically on test (ubuntu-latest), and it is currently the only red left on main. Two separate problems are tangled together:

1. The allocation-count assertion genuinely flaps — a control-plane allocation sometimes lands inside the measured region.
2. A poisoned-mutex cascade turns one real failure into two, at a line that has nothing to do with the bug — so both the failure count and which test failed are unreliable.

The second is cheap to fix and worth fixing regardless of the first, because right now the test output actively misleads whoever investigates.

Observed

Same commit 154633e6, two runs, different results:

Run	Failures	Where
main 32658015900	2	latency_sampling_buffers_do_not_allocate_after_warmup at :232, ordinary_measured_region_performs_zero_harness_allocations at :142
PR #233 32658147470	1	ordinary_measured_region_performs_zero_harness_allocations at :185

(#233 changed two markdown files, so it cannot have influenced either.)

The cascade

static TEST_GATE: Mutex<()> = Mutex::new(());          // :15
...
let _test_guard = TEST_GATE.lock().unwrap();           // :142  ordinary_...
let _test_guard = TEST_GATE.lock().unwrap();           // :199  latency_...

Both tests serialize on one gate because they share global state (HARNESS_ALLOCATIONS, set_measured_region_hook). On the main run:

• latency_sampling_... failed its real assertion at :232 while holding TEST_GATE.
• A panic while holding a std::sync::Mutex poisons it.
• ordinary_... then hit TEST_GATE.lock().unwrap() at :142 and panicked on PoisonError.

So the second failure was pure collateral. :142 is the lock acquisition — it tells you nothing about allocations, and it points at the wrong test. That is why the two runs disagree on how many tests "failed": one genuine failure, plus a phantom whose presence depends on execution order.

Fix: don't unwrap() a shared gate.

let _test_guard = TEST_GATE.lock().unwrap_or_else(|e| e.into_inner());

The guard exists for mutual exclusion, not for data integrity — the () payload cannot be left inconsistent — so recovering from poisoning is correct here, not a papering-over. With that change, one real failure reports as exactly one failure, at the assertion that actually failed.

The underlying flake

With the cascade removed, the real signal is:

assertion `left == right` failed: Rust control-plane allocation entered the measured interval for TinyFixed16
assertion `left == right` failed: latency sampling allocated in the measured region for TinyFixed64, control=false

HARNESS_ALLOCATIONS is expected to be exactly 0 across the measured region, and occasionally is not. Worth noting the file already carries precedent that this assertion is environment-sensitive:

#[cfg_attr(
    target_os = "macos",
    ignore = "macOS realloc may route through the global allocator"
)]

Plausible directions, not investigated:

• A lazily-initialised allocation on a first-touch path (map growth, formatting, thread-local init) that only sometimes falls inside the hook window.
• Hook install/uninstall racing the measured region, so an allocation from outside is attributed inside.
• Genuine harness allocation that is usually amortised by a previous test having warmed the same path — which would explain order-dependence.

The order-dependence is the most suggestive clue: whichever test runs first under the gate seems likelier to fail.

Why it matters

benchmark-suite is not in cross.yml's scope (see #234/#235), so this does not block cross-target validation. But it does mean main is permanently red, which is corrosive in the ordinary way: it trains everyone to ignore a red test (ubuntu-latest), and the next real regression there will be waved through.

Acceptance

• A single genuine assertion failure reports as exactly one failed test, at the assertion line — never as a PoisonError at :142/:199.
• test (ubuntu-latest) is green on main across, say, five consecutive runs — or the allocation assertion is explicitly relaxed/ignored with a stated reason, as macOS already is.

[zackees]

The reporting half of this is fixed and merged — #237, 587b1594.

TEST_GATE is now acquired with unwrap_or_else(PoisonError::into_inner), so an assertion failure no longer poisons the mutex and takes the sibling test down with it. One real failure now reports as exactly one failure, at the assertion that actually failed, instead of two with the second pointing at a lock acquisition in the wrong test.

The underlying flake is still open, which is why this issue stays open:

assertion `left == right` failed: Rust control-plane allocation entered the
measured interval for TinyFixed16

HARNESS_ALLOCATIONS is expected to be exactly 0 across the measured region and occasionally is not. What is known:

• The measured region spans std::thread::scope workers in execute_cell / execute_latency_cell, so a stray allocation can come from a thread the thread-local INSIDE_ADAPTER guard does not cover.
• It is order-dependent — whichever test ran first under the gate seemed likelier to fail, which is consistent with a lazily-initialised path being warmed by the earlier test.
• The file already carries precedent that this assertion is environment-sensitive: ordinary_measured_region_performs_zero_harness_allocations is #[cfg_attr(target_os = "macos", ignore)] for "macOS realloc may route through the global allocator".
• It does not reproduce locally on Windows.

I did not guess at a fix. What changed is that when it fires, the output now names the right test and the right line — so the next person to look at it gets an accurate starting point rather than a phantom PoisonError at :142.