Skip to content

fix(delta-index): connect_node self-deadlocks pruning a neighbour list - #787

Open
ohdearquant wants to merge 2 commits into
ruvnet:mainfrom
ohdearquant:fix/delta-index-connect-node-deadlock
Open

fix(delta-index): connect_node self-deadlocks pruning a neighbour list#787
ohdearquant wants to merge 2 commits into
ruvnet:mainfrom
ohdearquant:fix/delta-index-connect-node-deadlock

Conversation

@ohdearquant

@ohdearquant ohdearquant commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

What this fixes

ruvector-delta-index's tests::test_insert_and_search does not fail, it hangs. The
test thread parks at 0% CPU and never returns; the test runner has to kill it.

DeltaHnsw::connect_node's reverse-connection loop held a node's write guard and then
took a second lock on the same node:

let mut neighbor = self.nodes[neighbor_idx as usize].write();
if l < neighbor.neighbors.len() {
    neighbor.neighbors[l].push(node_idx);

    if neighbor.neighbors[l].len() > max_conn {
        let node_vec = self.nodes[neighbor_idx as usize].read().vector.clone();
        self.prune_neighbors(&mut neighbor.neighbors[l], &node_vec, max_conn);
    }
}

parking_lot::RwLock is not reentrant, so that read() blocks forever the first time a
neighbour's adjacency list grows past max_conn. Sampling the stalled process puts the
test thread in parking_lot_core::parking_lot::park beneath RwLock::read beneath
connect_node beneath insert.

prune_neighbors is a latent case of the same hazard rather than a demonstrated one:
it calls distance for every entry of the list it is pruning, and distance takes
nodes[n].read(). In a normally constructed graph those entries are other nodes, so it
does not deadlock today; it would the moment a list contained its own owner. Keeping it
out of the guard's scope costs nothing and removes the case.

The change

The loop takes the write guard only long enough to push the backlink and, when pruning
is required, copy out the adjacency list and the node's own vector. The guard is
dropped before prune_neighbors runs, so no lock is held while it takes read locks,
and the pruned list is written back under a fresh write guard. prune_neighbors's
signature and neighbour-selection logic are unchanged, and no HNSW parameter moves.

Test

test_connect_node_reverse_prune_no_deadlock drives the pruning path with m0 = 4, so
the sixth insert crosses the threshold and the branch is reached deterministically
despite the random vectors. It runs the insert loop on a background thread behind a
bounded receive, so a regression fails the test instead of hanging the runner, and it
reports a worker panic, an insert error and a timeout as three different failures
rather than collapsing them into one message.

What the test proves is the direct self-lock. It would still pass against a partial fix
that copied the vector out but left prune_neighbors inside the guard, because a
normally constructed adjacency list does not contain its own owner.

Verification

  • cargo nextest run -p ruvector-delta-index: 15 tests pass in about 0.25s, including
    test_insert_and_search at 0.23s.
  • Mutation check, re-run against the test as it stands: with the previous locking
    restored by hand (reverse-applied in place, not by checking out over the work),
    cargo nextest run -p ruvector-delta-index -E 'test(test_connect_node_reverse_prune_no_deadlock)'
    reports FAIL [60.015s] with connect_node reverse-connection pruning did not finish in 60s, and test_insert_and_search hangs until the runner kills it. Restoring the
    fix returns the crate to 15/15 passing and the working tree to a clean diff.
  • cargo clippy -p ruvector-delta-index --all-targets -- -D warnings clean;
    cargo fmt applied.

Related, not fixed here

crates/ruvector-postgres/src/index/hnsw.rs around HnswIndex::connect has a similar
shape: a DashMap reference and a layer write guard are both live while the same map is
looked up again inside the pruning branch. That crate's insert takes &self, so the
fix cannot be this one copied across — it needs a design that is safe against a
concurrent insert. Noting it rather than reaching into it; it is a static reading, not
something reproduced.

Why this has not shown up in CI

Tests (core-and-rest) never reaches this crate's tests: it is cancelled at its
240-minute limit while still compiling. Two other changes are needed before that job
can report a result at all (#784 and #786); this is the third. Once those land, this
deadlock would hang the job in the test phase instead.

`DeltaHnsw::connect_node`'s reverse-connection loop held a node's write guard
and then took a second lock on the same node:

    let mut neighbor = self.nodes[neighbor_idx as usize].write();
    ...
    let node_vec = self.nodes[neighbor_idx as usize].read().vector.clone();
    self.prune_neighbors(&mut neighbor.neighbors[l], &node_vec, max_conn);

`parking_lot::RwLock` is not reentrant, so the `read()` blocks forever the
first time a neighbour's adjacency list exceeds `max_conn`. `prune_neighbors`
carries the same hazard one level down: it calls `distance`, which takes
`nodes[n].read()` for every entry in the list being pruned.

`tests::test_insert_and_search` (100 inserts of 128-dim vectors) reaches that
branch and parks at 0% CPU indefinitely; sampling the process shows the test
thread in `parking_lot_core::parking_lot::park` under `RwLock::read` under
`connect_node` under `insert`.

The loop now takes the write guard only long enough to push the backlink and,
when pruning is required, copy out the adjacency list and the node's own
vector. The guard is dropped before `prune_neighbors` runs, so no lock is held
while it takes read locks, and the pruned list is stored back under a fresh
write guard. `prune_neighbors` and the neighbour-selection logic are unchanged.

Adds `test_connect_node_reverse_prune_no_deadlock`, which drives the pruning
path with a small `m`/`m0` and bounds itself with a channel timeout so a
regression fails the test rather than hanging the runner. With the previous
locking restored by hand, that test fails after its timeout and
`test_insert_and_search` hangs until the runner kills it; with the fix, all 15
tests in the crate pass in about 0.25s.
…es it

The regression test's comment said the old code took a second lock both
directly and through `prune_neighbors`. Only the direct `read()` is
demonstrated: `prune_neighbors` reads the entries of the list it prunes, which
in a normally constructed graph are other nodes, so it deadlocks only for a
list that contains its own owner. Both comments now say which is which, and
the test records that `m0 = 4` makes the sixth insert reach the branch.

The test also no longer reports a worker panic, a failed insert and a real
timeout with the same message, and the bound is raised to 60s so a starved
runner is not read as a deadlock.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant