feat(pack-code): L2 Rust scanner and extractor (ADR-085 Amendment 2 B2) - #1087
Draft
ohdearquant wants to merge 7 commits into
Draft
feat(pack-code): L2 Rust scanner and extractor (ADR-085 Amendment 2 B2)#1087ohdearquant wants to merge 7 commits into
ohdearquant wants to merge 7 commits into
Conversation
Adds the first slice of the L2 symbol tier: a syntax-only Rust Scanner (syn-based) and the shared, ontology-driven Extractor it feeds. - Scanner (scanner_rust.rs): parses top-level fn/struct/enum/type-alias/ trait declarations plus impl Trait for Type relationships. Doc comments are transcribed verbatim into descriptions; nothing is synthesized. Function bodies are walked for bare/path call expressions as a coverage-floor call-target extraction (method calls are not resolvable from syntax alone and are skipped). - Extractor (extractor.rs): maps Scanner output into the D2 subtype tokens (function/datatype/interface) via one adapter per Scanner shape, with a content hash per declaration for change detection. - Wired into code.ingest as an opt-in tier: a new tiers param (l1 | l1.5 | l2) defaults to l1+l1.5 only, so existing callers see no behavior change. l2 additionally creates declaration entities, module-contains- declaration edges, datatype-implements-interface edges, and same-project function-depends_on-function edges resolved from the call extraction. - Identity follows B4: uuid5(source_project, language, module_path, name, kind) under the pack's existing ingest namespace, with a content_hash property for change detection and last_seen_at stamped per sweep. Delivered in this slice: Rust Scanner, shared Extractor, D2 declaration entities, module containment edges, impl-Trait-for-Type edges, same- project function call edges, B4 identity, B5 staleness stamping. Deferred to follow-up slices (called out explicitly, not silent gaps): - Python/TypeScript/Lean scanners (B2's delivery order). - Declarations nested inside another item (mod blocks, impl/trait methods) — this slice scans top-level file items only. - Method-call resolution (self.foo()/x.foo()) — requires type inference a syntax-only scan does not have. - Cross-project symbol-level depends_on/implements resolution and a deferred-edge re-resolve queue for unresolved symbol references (B6's v2 alternative) — this slice resolves only within the same source_project, with no re-resolve pass across ingests. - datatype/interface D3 depends_on rules (field types, supertrait/bound references) beyond the implements edge. Tests: scanner unit tests, extractor adapter tests, symbol_uuid identity tests (language/kind disjointness per B8 property 3), and an integration suite (source_ingest_l2.rs) covering declaration creation with verbatim docs, containment + implements edges, function depends_on edges supporting blast-radius and dead-symbol queries (B8 property 1, scoped to this slice's call coverage), idempotent re-ingest, and the enable_l2=false no-op default. Gates: cargo test -p khive-pack-code (68 passed), cargo clippy --workspace --all-targets -- -D warnings (clean), cargo fmt --all -- --check (clean). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…incremental batched ingest PR #1087 round-1 fixes (all adversarially confirmed): - Route every L2 entity write through the same secret-gate checks KhiveRuntime::create_entity applies (ADR-085 D6). Raw EntityStore writes previously bypassed it for doc-comment-derived descriptions. - Make l1/l1.5/l2 independently selectable: tiers=["l1"] runs manifest parsing only, ["l1.5"] runs the import scan only, ["l2"] runs the symbol tier only, and combinations compose. - Key the L2 symbol index by (project, module_path, name, kind) instead of (project, name, kind), and resolve call targets using the syntactic context available (crate/self/super qualifiers plus the caller's own module) instead of a project-wide name lookup, so two same-named declarations in different modules no longer collide. - Compare each declaration's content_hash before writing: unchanged declarations get a batched last_seen_at-only touch instead of a full entity/edge rewrite. - Batch declaration entity and edge writes per file via upsert_entities/upsert_edges instead of one upsert per declaration. - Dedup resolved call targets per function before creating edges. - Doc comments are now transcribed verbatim (no trimming), the code.ingest verb description reflects that L2 is implemented, and the duplicated FNV-1a hash is a single shared helper. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
… Rust extraction Round-2 PR #1087 blockers (9 blocking + 2 minors): 1. Stale L2 edges: for every declaration re-extracted this scan, delete (soft) its L2-derived depends_on/implements edges the current scan no longer resolves. Scoped to l2_derived-provenance edges from scanned_decl_ids only; ADR-085 B5 absence-based no-deletion rule is unaffected (that governs declarations NOT re-observed this scan). 2. Manifest ancestry: find_governing_manifest walks parent directories above the ingest root to the filesystem root; a subtree ingest now attributes to the ancestor manifest package name. 3. Scanner extraction: recursive inline-module scanning (mod path segments threaded through), impl/trait-default methods named Type::method / Trait::method, contains edges to the correct nested parent. 4. resolve_call_target consumes every consecutive leading super segment, not just the first; underflow past crate root becomes None. 5. Tier isolation: run_import_scan short-circuits before any file I/O when a language has no L2 scanner and L1.5 is not requested; the reresolve pass is gated on enable_l1 OR enable_l1_5 so an L2-only call never replays a differently-tiered ingest pending specifiers. 6/9. get_entities_by_ids chunks at 900 inside the runtime method (every caller safe); touch_last_seen_at chunks its UPDATE at 900. 8. Batch edge-existence checks via get_edges (already chunked in khive-db) instead of N serial get_edge awaits. 10. CallRef DTO moved to extractor (language-neutral); scanner_rust emits raw segment vecs, the adapter wraps them. 11. L2 scanner dispatch is a language match table (dispatch_l2_scan), not inline control flow in run_import_scan. Gates: cargo fmt/clippy/test -p khive-pack-code all green (54 tests). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…omplete trait extraction
- Remove reconcile_stale_l2_edges soft-delete reconciliation entirely.
Per ADR-085 B5 ("ingest performs no automatic deletion"), every L2 edge
re-resolved this sweep gets properties.last_seen_at stamped, matching
entity staleness semantics. An edge whose source declaration re-scanned
but no longer resolves keeps its old stamp untouched — currency is a
view-layer filter, never a data-layer delete.
- Harden db_target::same_path against hard-link fence bypass: compare
(device, inode) identity across the main file and -wal/-shm companions
in addition to canonicalized-path comparison, and re-verify identity on
the just-opened handle before any write.
- Name trait-impl methods `<Type as Trait>::method` (Rust's canonical
qualified form) so two trait impls on one type no longer collide with
each other or with an inherent method of the same name.
- Resolve qualified impl target/trait paths (`impl a::T for b::S`) through
the same crate/self/super-aware resolver call targets use, instead of a
bare-last-segment lookup scoped to the impl's own module.
- Extract signature-only trait methods (no default body) as declarations,
not just default-bodied ones.
- Skip content-hash-unchanged modules before parsing; their declarations
still load into the resolution symbol index (batched, 900-chunked) so
a changed module elsewhere can resolve into them.
- Memoize governing-manifest lookups per directory for one ingest pass.
- Consolidate L2 language support/scanner/resolver behind one
L2Language dispatch table so a new language can't produce a silent
no-op path.
- Neutralize internal review-round vocabulary from comments across the
touched files — technical rationale only, ADR references kept.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Make the db-target fence URI-aware, bind ingest to a verified content identity rather than a pathname, fence configured multi-backend and cross-platform alias targets, re-scan L2 symbols on an L1.5->L2 tier upgrade, memoize shared ancestor manifests, and name the L2 symbol-identity key.
…ndows drive-letter target URIs The connection pool's bound-identity fence (device/inode on Unix, volume serial and file index on Windows) was checked by the pool's own open_standalone_reader/open_standalone_writer, but khive-db/sql_bridge.rs defined its own pair of free functions with the same names that opened a raw rusqlite::Connection directly against pool.config().path and never consulted the fence at all. SqlBridge::reader()/writer() call through those functions, so a post-bind file swap at the bound path was still reachable through the bridge even though the pool-level open paths already refused it. check_verified_identity is now pub(crate) so sql_bridge.rs can call it, and both bridge open functions check it before opening. Added a regression test that swaps the bound file after bind and asserts both bridge open paths error with the fence message. Also accepts Windows drive-letter absolute paths (e.g. C:\data\khive.db) as database target URIs in db_target.rs, which previously only recognized Unix-style absolute paths and file:// URIs.
… edges, module containment, sweep-clock coverage, chunked edge writes Several correctness gaps in the L2 Rust ingest pipeline: - An incremental pass that loaded a module unchanged (matching content_hash) never replayed that module's still-unresolved call references against newly-added targets from other modules in the same or a later sweep. A module's unresolved calls are now stamped onto an l2_unresolved_calls property on the module entity and replayed on every load until resolved. - A Rust parse failure in the L2 scanner previously fell through to the same code path as an empty-but-successful parse, overwriting a module's prior declaration_ids with nothing and leaving no distinction between a genuinely empty module and one that failed to parse. dispatch_l2_scan and scan_rust_l2 now return an explicit None on parse failure, and the caller skips the declaration_ids stamp entirely so the module keeps its last-known-good state and stays eligible for retry on the next sweep. - Qualified impl method names collided when two impl target types shared only their final path segment (e.g. a::T and b::T both impl'd against the same trait). The impl target and trait name are now taken as the full qualified path instead of the last segment. - The scanner extracted call-target edges (D3 rule 1) but nothing for D3 rules 2-7 (type references in signatures, fields, type aliases, and trait bounds/supertraits). Added a type-ref collector wired into every declaration site, an adapter-level TypeRef shape, and a second resolution pass in the ingest pipeline that emits depends_on edges between a declaration and every project-local type it references. - A file-backed module nested under another module (e.g. src/sub.rs owned by src/lib.rs's mod declaration) was always linked as directly contained by the project entity instead of by its owning parent module. The containment edge source is now the parent module when one exists. - A negative impl (impl !Trait for T) was recorded as a positive implements edge same as a normal impl. Negative impls are now skipped during extraction. - Union declarations were not extracted at all. They are now recorded as datatype entities, matching struct and enum. - A multi-language sweep over two languages that fall back to the same basename-derived project name (no governing manifest) only stamped the project's sweep_clock for whichever language's file walk reached the project first; project_ids being cache-hit for the second language suppressed its own upsert_project call, so its sweep_clock entry was never written. Project-upsert-once-per-sweep is now tracked as (project, language) pairs across both the manifest-driven pass and every language's import-scan pass, so a project shared by N languages gets exactly one upsert_project call per language, per sweep. - The final L2 edge resolution wrote every resolved edge in one unbounded transaction. Resolved edges are now written in fixed-size chunks. Added or extended scanner and ingest-pipeline regression tests for each of the above.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
First PR-sized slice of the L2 symbol tier (ADR-085 Amendment 2 B2-B8): a syntax-only Rust Scanner (
syn-based) and the shared, ontology-driven Extractor it feeds, wired intocode.ingestas an opt-in tier.Scope delivered
crates/khive-pack-code/src/scanner_rust.rs): parses top-levelfn/struct/enum/type/traitdeclarations andimpl Trait for Typerelationships from a Rust file viasyn. No type-checking, no compilation. Doc comments (////#[doc]) are transcribed verbatim into descriptions — nothing synthesized (ADR-069 D5). Function bodies are walked for bare/path call expressions as a coverage-floor call-target extraction (analogous in spirit to L1.5's regex import scan); method calls (self.foo(),x.foo()) are not resolvable from syntax alone and are skipped.crates/khive-pack-code/src/extractor.rs): shared, source-blind mapping from Scanner output into the D2 subtype tokens (function/datatype/interface), with one per-Scanner-output-shape adapter (from_rust_scanin this slice) feeding it, and an FNV-1a content hash per declaration for change detection.code.ingestwiring: newtiersparam (l1|l1.5|l2), defaulting to["l1", "l1.5"]so existing callers see byte-identical behavior.l2additionally runs the Rust Scanner/Extractor pass and creates:content_hash,language,module_path,source_projectproperties andlast_seen_atstaleness stamping (B5),module contains {function,datatype,interface}edges (D3 rules 17-19),datatype implements interfaceedges fromimpl Trait for Type(D3 rule 13),function depends_on functionedges resolved from the call extraction (D3 rule 1).uuid5(source_project, language, module_path, name, kind)under the pack's existingCODE_INGEST_NAMESPACEseed (same patternmodule_uuidalready used),kindalways one of the four canonical D2 tokens.Scope deferred (explicit follow-up slices, not silent gaps)
mod { .. }blocks and methods insideimpl/traitbodies. This slice scans top-level file items only; a stable qualified-name scheme for nested/method declarations is follow-up work.self.foo(),x.foo()) — requires type information a syntax-only scan does not have.depends_on/implementsresolution, and a deferred-edge re-resolve queue for unresolved symbol references (B6's documented v2 alternative). This slice resolves calls/impls only within the samesource_project, with no re-resolve pass across ingests — unresolved targets are silently dropped and counted in a newsymbol_dependencies_unresolvedreport field, not queued.depends_onrules fordatatype/interface(field types, supertrait/bound references) beyond theimplementsedge.ADR-085 citations
docs/adr/ADR-085-code-pack.mdlines 535-552.last_seen_atper sweep) — lines 614-627.Test plan
cargo test -p khive-pack-code— 68 tests pass (scanner unit tests, extractor adapter tests,symbol_uuididentity tests, and a newtests/source_ingest_l2.rsintegration suite: declaration creation with verbatim docs, containment + implements edges, functiondepends_onedges supporting blast-radius and dead-symbol queries, idempotent re-ingest, and theenable_l2=falseno-op default).cargo clippy --workspace --all-targets -- -D warnings— clean.cargo fmt --all -- --check— clean.🤖 Generated with Claude Code