Skip to content

Flaky: recall_trigger_holds_under_no_drift asserts a hard threshold on an unseeded random graph #792

Description

@ruvnet

crates/ruvector-diskann/src/reuse.rs:412 fails intermittently in CI:

test reuse::tests::recall_trigger_holds_under_no_drift ... FAILED
thread panicked at crates/ruvector-diskann/src/reuse.rs:412:9:
assertion failed: t.probe_recall(&v) >= 0.9

Seen on https://github.com/ruvnet/RuVector/actions/runs/30867347732/job/91861986023 (994 of 995 passed). Passes 6/6 locally on darwin-arm64 with --all-features.

Mechanism

The test's fixture is deterministic — reuse.rs contains no RNG at all. The graph build is not:

  • graph.rs:101order.shuffle(&mut rand::thread_rng())
  • graph.rs:346, graph.rs:370rand::thread_rng()
  • pq.rs:59rand::thread_rng()

So RecallTrigger::build produces a different graph every run, probe_recall is a random variable, and assert!(recall >= 0.9) fails whenever the draw lands low. The comment above the assertion ("same vectors → recall ~1.0") describes the expectation, not a guarantee.

Why it matters beyond this test

A hard threshold on an unseeded random process fails at some rate forever, and a CI job that is red for reasons unrelated to the change under review trains reviewers to merge past red. That cost is paid by every future PR, not just this one.

Options

  1. Seed the graph build for tests — thread a seedable RNG (StdRng::seed_from_u64) through GraphBuilder/pq, so the test is deterministic and a failure means a real regression.
  2. Assert a distribution rather than a draw — run N builds and require the recall median (or a lower bound like p05) to clear 0.9, so the test measures the property it actually cares about.
  3. Lower the threshold — cheapest, and the worst: it hides the variance instead of describing it, and the next unlucky draw still fails.

Option 1 is preferable; it is the only one that makes a failure mean something.

Found while verifying that this failure was not caused by #791, which touches no diskann file.

🤖 Generated with claude-flow

https://claude.ai/code/session_01WfyBMvexdN4bhWVaqSXKvR

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions