fix(delta-index): connect_node self-deadlocks pruning a neighbour list - #787
Open
ohdearquant wants to merge 2 commits into
Open
fix(delta-index): connect_node self-deadlocks pruning a neighbour list#787ohdearquant wants to merge 2 commits into
ohdearquant wants to merge 2 commits into
Conversation
`DeltaHnsw::connect_node`'s reverse-connection loop held a node's write guard
and then took a second lock on the same node:
let mut neighbor = self.nodes[neighbor_idx as usize].write();
...
let node_vec = self.nodes[neighbor_idx as usize].read().vector.clone();
self.prune_neighbors(&mut neighbor.neighbors[l], &node_vec, max_conn);
`parking_lot::RwLock` is not reentrant, so the `read()` blocks forever the
first time a neighbour's adjacency list exceeds `max_conn`. `prune_neighbors`
carries the same hazard one level down: it calls `distance`, which takes
`nodes[n].read()` for every entry in the list being pruned.
`tests::test_insert_and_search` (100 inserts of 128-dim vectors) reaches that
branch and parks at 0% CPU indefinitely; sampling the process shows the test
thread in `parking_lot_core::parking_lot::park` under `RwLock::read` under
`connect_node` under `insert`.
The loop now takes the write guard only long enough to push the backlink and,
when pruning is required, copy out the adjacency list and the node's own
vector. The guard is dropped before `prune_neighbors` runs, so no lock is held
while it takes read locks, and the pruned list is stored back under a fresh
write guard. `prune_neighbors` and the neighbour-selection logic are unchanged.
Adds `test_connect_node_reverse_prune_no_deadlock`, which drives the pruning
path with a small `m`/`m0` and bounds itself with a channel timeout so a
regression fails the test rather than hanging the runner. With the previous
locking restored by hand, that test fails after its timeout and
`test_insert_and_search` hangs until the runner kills it; with the fix, all 15
tests in the crate pass in about 0.25s.
…es it The regression test's comment said the old code took a second lock both directly and through `prune_neighbors`. Only the direct `read()` is demonstrated: `prune_neighbors` reads the entries of the list it prunes, which in a normally constructed graph are other nodes, so it deadlocks only for a list that contains its own owner. Both comments now say which is which, and the test records that `m0 = 4` makes the sixth insert reach the branch. The test also no longer reports a worker panic, a failed insert and a real timeout with the same message, and the bound is raised to 60s so a starved runner is not read as a deadlock.
ohdearquant
marked this pull request as ready for review
August 3, 2026 17:55
This was referenced Aug 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this fixes
ruvector-delta-index'stests::test_insert_and_searchdoes not fail, it hangs. Thetest thread parks at 0% CPU and never returns; the test runner has to kill it.
DeltaHnsw::connect_node's reverse-connection loop held a node's write guard and thentook a second lock on the same node:
parking_lot::RwLockis not reentrant, so thatread()blocks forever the first time aneighbour's adjacency list grows past
max_conn. Sampling the stalled process puts thetest thread in
parking_lot_core::parking_lot::parkbeneathRwLock::readbeneathconnect_nodebeneathinsert.prune_neighborsis a latent case of the same hazard rather than a demonstrated one:it calls
distancefor every entry of the list it is pruning, anddistancetakesnodes[n].read(). In a normally constructed graph those entries are other nodes, so itdoes not deadlock today; it would the moment a list contained its own owner. Keeping it
out of the guard's scope costs nothing and removes the case.
The change
The loop takes the write guard only long enough to push the backlink and, when pruning
is required, copy out the adjacency list and the node's own vector. The guard is
dropped before
prune_neighborsruns, so no lock is held while it takes read locks,and the pruned list is written back under a fresh write guard.
prune_neighbors'ssignature and neighbour-selection logic are unchanged, and no HNSW parameter moves.
Test
test_connect_node_reverse_prune_no_deadlockdrives the pruning path withm0 = 4, sothe sixth insert crosses the threshold and the branch is reached deterministically
despite the random vectors. It runs the insert loop on a background thread behind a
bounded receive, so a regression fails the test instead of hanging the runner, and it
reports a worker panic, an insert error and a timeout as three different failures
rather than collapsing them into one message.
What the test proves is the direct self-lock. It would still pass against a partial fix
that copied the vector out but left
prune_neighborsinside the guard, because anormally constructed adjacency list does not contain its own owner.
Verification
cargo nextest run -p ruvector-delta-index: 15 tests pass in about 0.25s, includingtest_insert_and_searchat 0.23s.restored by hand (reverse-applied in place, not by checking out over the work),
cargo nextest run -p ruvector-delta-index -E 'test(test_connect_node_reverse_prune_no_deadlock)'reports
FAIL [60.015s]withconnect_node reverse-connection pruning did not finish in 60s, andtest_insert_and_searchhangs until the runner kills it. Restoring thefix returns the crate to 15/15 passing and the working tree to a clean diff.
cargo clippy -p ruvector-delta-index --all-targets -- -D warningsclean;cargo fmtapplied.Related, not fixed here
crates/ruvector-postgres/src/index/hnsw.rsaroundHnswIndex::connecthas a similarshape: a
DashMapreference and a layer write guard are both live while the same map islooked up again inside the pruning branch. That crate's
inserttakes&self, so thefix cannot be this one copied across — it needs a design that is safe against a
concurrent insert. Noting it rather than reaching into it; it is a static reading, not
something reproduced.
Why this has not shown up in CI
Tests (core-and-rest)never reaches this crate's tests: it is cancelled at its240-minute limit while still compiling. Two other changes are needed before that job
can report a result at all (#784 and #786); this is the third. Once those land, this
deadlock would hang the job in the test phase instead.