Add --drop_vector_index to update collection - #171
Open
jfrancoa wants to merge 5 commits into
Open
Conversation
Adds `weaviate-cli update collection --drop_vector_index <name>` to drop the
index of a named vector (Weaviate 1.39 experimental endpoint, exposed by
weaviate-python-client#1991 as collection.config.delete_vector_index()).
- defaults.py: UpdateCollectionDefaults.drop_vector_index
- update.py: --drop_vector_index option, mutually exclusive with --vector_index
- collection_manager.py:
- drop runs AFTER config.update() (which reads-and-writes the whole schema),
so a "none"-marked vector is never PUT back to the server
- get collection listing renders a dropped vector as `none` and no longer
crashes on a null vector_index_config; shows all distinct index types
(e.g. `hnsw, none`)
- re-dropping an already-"none" vector is allowed (server returns 200): it is
a no-op while cleanup runs and re-enqueues a fresh task if the previous one
FAILED -- the only operator recovery path for a stalled drop
- version warning when the server is older than v1.39.0
- unit tests: happy path, validation, listing, and re-trigger
- requirements-dev.txt / setup.cfg: temporary pin to the client PR #1991 commit
(revert to a released weaviate-client before release)
- operating skill docs
Closes #170
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
7 tasks
There was a problem hiding this comment.
Orca Security Scan Summary
| Status | Check | Issues by priority | |
|---|---|---|---|
| Secrets | View in Orca |
A dropped vector index CAN be re-created (confirmed with the developer and verified live via config.add_vector() once the drop finalizes). Correct the docs/help/messages that wrongly said "cannot be re-created" / "irreversible": - update.py help, collection_manager success + conflict messages - operating skill (SKILL.md, collections.md): re-creation is allowed after the drop finalizes (fresh, empty index); "irreversible" removed create data: automatically skip named vectors whose index was dropped (vectorIndexType "none", reported as vector_index_config == None). The server rejects any object carrying a dropped vector, so every batch was failing after a drop. Now those vectors are excluded from generated objects (with an informational note, suppressed under --json) and ingestion keeps working. - data_manager.__ingest_data: filter dropped vectors, derive vectorizer from a live one, note skipped vectors - data.md: document the auto-skip - unit tests: skip one, skip all, skip none, --json suppression Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
There was a problem hiding this comment.
Orca Security Scan Summary
| Status | Check | Issues by priority | |
|---|---|---|---|
| Infrastructure as Code | View in Orca | ||
| SAST | View in Orca | ||
| Secrets | View in Orca | ||
| Vulnerabilities | View in Orca |
Advances the temporary weaviate-python-client pin to the current head of PR #1991. Since our previous pin (2dbfaedf) the PR added two parser fixes we need: b92dbb71 replaced an assert in the dropped-vector schema parser with an explicit SchemaValidationError (-O safe), and 9d7b9163 fixes a KeyError 'vectorizer' crash when parsing a legacy (single-vector) collection whose vector index was dropped -- which otherwise breaks `get collection` and `delete collection --all`. No weaviate-cli code change is required; the fix is entirely in the client parser. Lint clean, 336 unit tests pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`create collection --named_vector_name` now accepts a comma-separated list, creating one named vector per name (all sharing the chosen --vectorizer and --vector_index). A single name keeps the previous behavior, so existing callers are unaffected. Duplicate or empty names are rejected. This makes it possible to create collections with more than one named vector, e.g. to test dropping one named vector's index while keeping others. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`update collection --add_vector NAME` adds a named vector via the client's config.add_vector(), so a vector whose index was dropped can be re-created from the CLI (and then re-ingested). --add_vector_vectorizer picks the vectorizer (none/contextionary/transformers/model2vec, default none = self-provided) and --add_vector_index_type picks the index, including quantized variants (hnsw_pq, hnsw_sq, hnsw_bq, hnsw_rq, hfresh, flat_bq, hnsw_acorn); pq/sq honor --training_limit. Mutually exclusive with --drop_vector_index and --vector_index. Unit tests cover the added config, quantizers, and the validation guards. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
weaviate-cli update collection --drop_vector_index <name>to drop the index of anamed vector — the Weaviate 1.39 experimental endpoint
(
DELETE /v1/schema/{Class}/vectors/{name}/index), exposed by the Python client inweaviate/weaviate-python-client#1991 as
collection.config.delete_vector_index(). The dropis destructive and applied asynchronously (a 200 means accepted, not done). The vector can
be re-created afterwards (a fresh, empty index via
config.add_vector()) once the dropfinalizes — verified live.
Closes #170
What changed
update.py/defaults.py—--drop_vector_index <name>, mutually exclusive with--vector_index; warns when the server is older than v1.39.0.collection_manager.pyconfig.update().config.update()reads the whole schema andPUTs it back, so dropping first would send a
vectorIndexType: "none"vector back to theserver. Ordering is asserted by a unit test.
get collectionlisting fix: a dropped vector'svector_index_configisNone; theold listing called
.vector_index_type()on it and crashed. It now rendersnoneandshows every distinct index type across named vectors (e.g.
hnsw, none).nonevector returns 200 on theserver — a no-op while cleanup runs, and a fresh cleanup task if the previous one FAILED
(the only operator recovery path for a stalled drop). The CLI reports the re-trigger
instead of blocking it.
create dataauto-skips dropped vectors — after a drop, the server rejects any objectcarrying the dropped vector, so
--randomizeingestion failed every batch. Generatedobjects now exclude any named vector whose index is dropped (
vectorIndexType: "none"),with an informational note (suppressed under
--json). Ingestion keeps working; re-ingestafter re-creating the index.
--vector_indexconflict, unknown/absent vector, listing(
none/hnsw, none), re-trigger, and thecreate dataskip (one / all / none /--json).336 pass.
create dataauto-skip).requirements-dev.txtandsetup.cfgpinweaviate-clientto the commit on thepython-client PR #1991 branch
(which adds
delete_vector_index()andVectorIndexType.NONE). That PR is not yet merged.Before a weaviate-cli release this must be reverted to a released
weaviate-client>=X. A gitURL cannot be published to PyPI, so this should not merge to a release tag as-is.
Test plan
make lintmake test(336 passed)stay 100% available during cleanup; dropped-vector search rejected immediately;
hnsw, noneshown mid-drop; re-drop of a
nonevector accepted;--vector_indexconflict rejected;vector re-created after finalize via
config.add_vector()and searchable again;create dataskips a dropped vector and ingests successfully (note shown;--jsonclean).🤖 Generated with Claude Code