fix(rvf-runtime): clear the deletion bit when an id is re-ingested - #759
Open
ohdearquant wants to merge 2 commits into
Open
fix(rvf-runtime): clear the deletion bit when an id is re-ingested#759ohdearquant wants to merge 2 commits into
ohdearquant wants to merge 2 commits into
Conversation
delete(ids) sets bits in the deletion bitmap and ingest_batch never cleared them, so re-inserting a deleted id left it tombstoned. query() then skips the id, making the newly written vector unreachable, and compact() reads the stale bit as dead and physically removes the vector, destroying a payload that was written after the deletion. DeletionBitmap::clear_ids already existed for this; its only caller was its own unit test. The call goes immediately after the insert loop, which is where the id becomes live again. Adds two regression tests, one per consequence. Both were observed failing before the fix and passing after, and removing the added line again turns exactly those two red while the other 243 stay green.
ohdearquant
marked this pull request as draft
August 2, 2026 14:37
ohdearquant
marked this pull request as ready for review
August 3, 2026 18:08
Add a regression test that ingests id 42, deletes it, re-ingests it with a distinguishable replacement vector, closes the store, reopens it from disk, and asserts the query returns the replacement rather than a soft-deleted (or stale) result. The two existing re-ingest tests only covered the in-memory and compaction paths; neither closed and reopened the store, so the persisted deleted_ids manifest snapshot was unpinned. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #748.
delete(ids)sets bits in the deletion bitmap.ingest_batchnever cleared them, so re-inserting adeleted id left the bit set. Two consequences, and the second is data loss:
query()skips the id, so the newly ingested vector is unreachable even though it is stored.compact()treats the id as dead and physically removes it, destroying a payload that waswritten after the deletion.
DeletionBitmap::clear_idsalready existed for exactly this; its only caller was its own unit test.The fix
One line, placed immediately after the loop that inserts the vectors, which is the point where the
id becomes live again:
Verified at
main, not inherited from the reportThe issue confirms 0.2.0 and says the interaction is "present in 0.3.0". I reproduced it at
mainrather than taking that forward, because the same crate had a separate issue that was already fixed
by the time it was filed. Both halves reproduce:
reingest_after_delete_makes_the_id_visible_againcompact_keeps_a_vector_reingested_after_deleteBoth are in the PR. They were written before the fix and observed failing first, then observed
passing, and the failure was re-confirmed afterwards by removing the added line and watching exactly
those two go red while the other 243 stayed green. A regression test that passes with the fix
reverted would not be guarding anything.
Full suite at the current head: 328 passed, 0 failed across the rvf-runtime library (246) plus integration and doc targets.
cargo fmt --all --checkclean.One thing I checked and deliberately did not change
delete()also drops the id from a COW child'smembership_filter, so my first version of thispatch re-added it there for symmetry. That turned out to be unnecessary: re-ingesting makes the
vector child-owned, and the query paths find it through
self.vectorswithout consulting membership.I confirmed it by writing a COW branch/delete/re-ingest/query test and watching it pass both with and
without that block, which means the block was doing nothing. It is not in this PR, and neither is
that test — it passes on unmodified
main, so shipping it alongside the two real ones would havemade it look like a regression guard when it characterises behaviour that was already correct.
Mentioning it because "does this need the same treatment for COW children" is the first question a
reviewer should ask about this fix, and the answer is no, measured rather than assumed.
Note on lint
cargo clippyon this crate reports three pre-existing errors, all inrvf-types(impl can be derived×2, missingDefaultforSha256). They are untouched by this PR. Worth knowing why theysurvive:
rvf-runtimeis not a workspace member, socargo clippy --workspaceandcargo test --workspacenever reach it or its path deps. Happy to open that separately if useful.Note on this PR's CI.
Tests (core-and-rest)is red on every branch in this repository,including
main: the job is cancelled at its 240-minute cap while still compiling and neverreaches the test phase. #786 restores the exclusion list that the shard's
packages:value losesto a shell comment, #784 unblocks the
ruvector-filtertest target that the compiler cannotfinish, and #787 fixes a deadlock waiting behind both. That failure is not caused by this branch.