Summary
Upstream semble #225 (204ae4e) added partial (incremental) reindexing: only files that changed since the cached index was built are re-chunked and re-embedded. csp rebuilds the whole index whenever any file changes.
Upstream
index/types.py: FileManifestEntry {mtime_ns, start, count}, PreviousIndex, CACHE_FORMAT_VERSION = 1, make_chunk_id(path, slot) -> "{path}:{slot}".index/bm25.py: replaced bm25s with an own BM25 class supporting add_document / remove_document / set_doc_order and JSON persistence (documents + doc_order).index/create.py: create_index_from_path(..., previous=) reuses chunks + vector slices for files whose mtime_ns matches, re-embeds only changed files, removes postings for deleted files, and returns a files manifest; reuses the previous vector matrix in place when the layout is unchanged.cache.py: load_previous_for_incremental() validates metadata (model_path, content_type, chunk_size, cache_version) and the chunk/vector/BM25 alignment before reuse. Metadata file_paths → files manifest.index/index.py: from_path calls load_previous_for_incremental; load_from_disk rejects a missing/other cache_version.
csp status
crates/csp/src/indexing/cache_orchestrator.rs::try_reuse validates one whole-tree content_hash; any mismatch → full CspIndex::from_path rebuild + save. Not reflected.IndexManifest has schema_version, content_hash, chunk_size, model_id, model_kind but no per-file manifest.Bm25Index (indexing/sparse.rs) is built once from all chunks; no incremental add/remove.
Proposed adaptation
csp already uses a content-hash oracle instead of mtime (see #74 / PR #78). Extend the manifest to a per-file entry {hash, start, count} and keep hash-based validation instead of mtime_ns; port the incremental BM25 + vector-slice reuse from create.py. Bump INDEX_SCHEMA_VERSION so old caches rebuild.
Ordering
Land after #80 / #82 (both touch index.rs) and after the MCP content parity issue (touches the cache key).
Refs
Summary
Upstream semble #225 (
204ae4e) added partial (incremental) reindexing: only files that changed since the cached index was built are re-chunked and re-embedded. csp rebuilds the whole index whenever any file changes.Upstream
index/types.py:FileManifestEntry {mtime_ns, start, count},PreviousIndex,CACHE_FORMAT_VERSION = 1,make_chunk_id(path, slot) -> "{path}:{slot}".index/bm25.py: replacedbm25swith an ownBM25class supportingadd_document/remove_document/set_doc_orderand JSON persistence (documents+doc_order).index/create.py:create_index_from_path(..., previous=)reuses chunks + vector slices for files whosemtime_nsmatches, re-embeds only changed files, removes postings for deleted files, and returns afilesmanifest; reuses the previous vector matrix in place when the layout is unchanged.cache.py:load_previous_for_incremental()validates metadata (model_path,content_type,chunk_size,cache_version) and the chunk/vector/BM25 alignment before reuse. Metadatafile_paths→filesmanifest.index/index.py:from_pathcallsload_previous_for_incremental;load_from_diskrejects a missing/othercache_version.csp status
crates/csp/src/indexing/cache_orchestrator.rs::try_reusevalidates one whole-treecontent_hash; any mismatch → fullCspIndex::from_pathrebuild +save. Not reflected.IndexManifesthasschema_version,content_hash,chunk_size,model_id,model_kindbut no per-file manifest.Bm25Index(indexing/sparse.rs) is built once from all chunks; no incremental add/remove.Proposed adaptation
csp already uses a content-hash oracle instead of mtime (see #74 / PR #78). Extend the manifest to a per-file entry
{hash, start, count}and keep hash-based validation instead ofmtime_ns; port the incremental BM25 + vector-slice reuse fromcreate.py. BumpINDEX_SCHEMA_VERSIONso old caches rebuild..please/docs/references/semble.md).Bm25Index(add/remove document, stable chunk ids, doc order).create_index_from_path(previous)reuse path +load_previous_for_incrementalequivalent incache_orchestrator.rs.files+ schema version bump;load_from_diskrejects other versions.tests/index/test_create.py/test_bm25.pycases.Ordering
Land after #80 / #82 (both touch
index.rs) and after the MCPcontentparity issue (touches the cache key).Refs
204ae4e)949559b..6bbbd32, 2026-09-03