Establish managed Git vault lifecycle foundation

[alemhnan]

Current packet: #86 validated Vault definitions

Implements #86 only on the existing draft integration branch. Tested commit:
3ed9ecba8f1efa3e7a9c8335ce1d6c50d73d9496.

Definition-management contract

• Persists the accepted common name, enabled state, per-Vault exclusions, and a
  tagged local, existing_git, or managed_git source under the immutable
  UUID map key established by Persist a recoverable zero-Vault registry #85.
• Provides shared-core add, edit, enable, disable, and disconnect
  operations with optimistic registry revisions. Case-insensitive duplicate
  names and equal/ancestor/descendant canonical paths are rejected; disabled
  definitions continue reserving their paths.
• Source/repository/branch/subdirectory/location changes require a disabled
  definition and explicit same-logical-Vault confirmation. Name, Git mode,
  exclusions, and credentials may change normally without changing the UUID.
• Existing-Git definitions require a readable working checkout. Readable but
  non-writable local directories are valid. Re-enabling revalidates the source.
• Private HTTPS username/token values persist only inside the 0600 registry.
  Public snapshots expose only credential_configured; debug output, errors,
  recovery guidance, and repository URLs remain credential-free. Changing
  repository identity never carries an old credential implicitly.
• Disconnect removes only the registry record and its in-record credential. It
  performs no deletion of notes, checkouts, Git history, or remotes.

No runtime reconciliation, migration, Git acquisition/sync, HTTP, MCP,
frontend, cache, search, or deployment work is included. #67, #102, and #103
are unchanged, and no #87 or later implementation was begun.

#86 interface-change checklist

Interface change: validated Vault definitions and lifecycle operations
Producer boundary: Vault collection registry (src/vault_registry.rs)
Kind: additive and behavior-changing persistent/Rust contract
Old contract: schema-1 registry records were an intentionally empty #85 identity slot, with only the private raw commit seam
New contract: validated tagged definition records, redacted projections, write-only credential inputs/updates, and add/edit/enable/disable/disconnect operations with the invariants above; raw record commit is private
Consumer boundaries: no production consumer is integrated in #86; repository references, module assignments, and the complete fixed-point diff were searched. #87 is the first planned consumer, followed by the routed runtime and management adapters.
Compatibility/migration: zero-Vault schema-1 registries remain valid; no production #85 consumer could create records; legacy environment conversion remains #87; downgrade is unsupported inside the draft integration route
Rollout/rollback: PR #31 remains draft and has no deployment wiring; revert the #86 commit before dependent packets to restore the #85 boundary
Evidence: focused registry tests, full backend suite, strict lint/format/type checks, module-map validation, and independent Standards/Spec review

#86 validation

Run from the repository root at tested commit
3ed9ecba8f1efa3e7a9c8335ce1d6c50d73d9496:

• cargo test vault_registry — passed (33)
• cargo check --all-targets — passed
• cargo fmt --all -- --check — passed
• cargo clippy --all-targets -- -D warnings — passed
• cargo test --all — passed (516 total: 506 library, 7 evaluation, 3 CLI; 0 doc tests)
• node scripts/check-module-map.mjs — passed (168 production files assigned exactly once)
• git diff --check — passed
• Standards review — initial documentation and duplicated structural-validation findings fixed; independent re-review clean
• Spec review — initial canonical-path, checkout-root, and malformed-HTTPS findings fixed with regressions; independent re-review clean

Frontend, Compose, and container gates were not run because their paths and
deployment behavior are forbidden and unchanged in #86.

#86 intermediate failures and routed gaps

• TDD deliberately observed red failures before each new public lifecycle,
  validation, redaction, recovery, and filesystem behavior was implemented.
• The first strict Clippy run found one collapsible nested if; it was corrected
  before the recorded clean run.
• Independent review found stale module documentation, duplicated structural
  source rules, a symlink-alias overlap bypass, acceptance of a repository's
  .git metadata directory as its checkout location, and under-validation of
  HTTPS authorities/percent escapes. The implementation was hardened and three
  regression tests were added before the tested commit.
• The first post-review full-suite run encountered four existing settings tests
  contaminated by stale PID-named /tmp/hatchdoor-settings-test-*.json files.
  Two remained reproducible alone, proving cross-process residue; deleting only
  those disposable test artifacts made the clean rerun pass all 516 tests.
• The non-interactive shell did not expose node; the installed Node 24.16.0
  binary ran the module-map checker successfully.
• Incremental compilation remained disabled and Cargo used one build job to
  protect VM disk headroom. Final free space remained 7.0 GB (89% used).

Those implementation incidents require no product ticket. Expected draft gaps
remain owned by the accepted route: legacy import #87; per-Vault runtime and
coordination #88-#90; cache/read/search #91-#93; actual Git acquisition,
synchronization, scheduling, and recovery #94-#97; HTTP discovery and management
#98-#101; frontend #102; MCP #103; and final deployment/docs acceptance #104.

---

Completed packet: #85 Vault registry

Implements #85 only on the existing draft integration branch. The tested commit is
162f83c9af126124c2dd0b8b53f9340413fccb7e. It adds the authoritative,
revisioned Vault collection registry without beginning #86 or integrating any
runtime, HTTP, MCP, frontend, migration, search, cache, or Git lifecycle consumer.
It does not modify #67, #102, or #103.

Registry contract

• One authoritative /data/state/vaults.json with schema_version: 1, a
  monotonic collection revision, and a map keyed by immutable canonical random
  UUID v4 Vault IDs.
• An absent file loads lazily as a complete revision-0, zero-Vault state and is
  not created by reads.
• Commits are serialized by normalized registry path across all store handles in
  the process, require the expected persisted revision, increment exactly once,
  and atomically replace the file with owner-only 0600 permissions.
• Corrupt, unsupported, and future-schema files expose no Vault records, give
  recovery guidance, preserve the original bytes, and refuse commits.
• VaultRecord is intentionally an empty persisted identity slot. Manage validated Vault definitions without exposing credentials #86 owns the
  accepted definition fields, validation, redaction, and lifecycle operations.

#85 interface-change checklist

Interface change: persistent Vault collection registry
Producer boundary: src/vault_registry.rs
Kind: additive persistent domain state
Old contract: no authoritative Vault collection, collection revision, or durable Vault ID store
New contract: the exact schema, revision, ID, lazy-zero, optimistic-concurrency, atomicity, permissions, and recovery behavior above
Consumer boundaries: no production consumer is integrated by #85; repository references, module assignments, and the complete branch diff were searched. #86 and #87 are the first planned consumers, followed by the routed runtime/API packets.
Compatibility/migration: existing startup behavior is unchanged; the absent registry remains a zero-Vault state; legacy environment import belongs to #87
Rollout/rollback: this remains a draft integration PR with no deployment change; revert the #85 commit to remove the boundary before consumers exist
Evidence: focused registry tests, full backend suite, strict lint/format checks, module-map validation, and both Standards and Spec reviews

Architectural invariants checked: this adds no service, framework, trait layer,
code generation, ambient/default Vault, runtime consumer, or compatibility shim.
Markdown remains authoritative and SQLite disposable. Existing HTTP and MCP
mutations remain routed through src/vault/write/. Search behavior is unchanged.

#85 validation

Run from the repository root at tested commit
162f83c9af126124c2dd0b8b53f9340413fccb7e:

• cargo test vault_registry — passed (13)
• cargo check --all-targets — passed
• cargo fmt --all -- --check — passed
• cargo clippy --all-targets -- -D warnings — passed
• cargo test --all — passed (496 total: 486 library, 7 evaluation, 3 CLI; 0 doc tests)
• node scripts/check-module-map.mjs — passed (168 production files assigned exactly once)
• git diff --check — passed
• Standards review — initially found per-instance rather than path-scoped write serialization; fixed with a process-wide normalized-path lock and an independently constructed-store regression test
• Spec review — reported the same optimistic-concurrency gap; fixed by the same tested change

Frontend gates were not run because frontend/** is forbidden and unchanged.
Compose/container checks were not run because #85 changes no deployment wiring.

#85 known intermediate failures and routed gaps

• The TDD sequence intentionally observed failures for each missing registry seam;
  the final review regression then proved that two independently constructed store
  handles could both save revision N+1. The final path-scoped serialization fix
  makes one succeed and one receive the expected revision conflict.
• A first temporary build target filled the available temporary filesystem while
  compiling. Generated target output was removed; validation reused the repository
  target with incremental compilation disabled and one Cargo build job.
• VM disk pressure reached 92%. Only generated/reinstallable artifacts were
  removed: approximately 3.1 GB of Rust incremental output and 509 MB of prototype
  frontend/node_modules; free space recovered to 7.0 GB (89% used) after final
  builds.
• The non-interactive shell did not expose Node. The pinned installed Node 24.16.0
  binary ran the module-map checker successfully.
• A module documentation block was briefly inserted at the wrong location and
  made cargo fmt --check fail; it was corrected before the recorded validation.

Those setup incidents do not create product follow-up tickets. Expected functional
gaps remain owned by the accepted route: definition operations by #86; legacy
import by #87; per-Vault runtime by #88-#90; cache/read/search by #91-#93; Git
source lifecycle by #94-#97; HTTP discovery and mutation contracts by #98-#101;
frontend by #102; MCP by #103; and final deployment/docs acceptance by #104.

---

Completed baseline packet: #84

Implements #84 only: rebase the existing managed-Git vault foundation onto the development head recorded at ticket start, preserve the accepted multi-Vault decisions, and leave the result as a draft integration baseline for the routed follow-up tickets.

• Recorded development base: f77855c4c9d2a099a232ae8e36c4758a405e7f91
• Tested integration commit: 7c9ebb3ceacbe2fb4c5516f7f19b8bf91355e1b3
• Replayed integration commits only: 77f89a3, c968ded, bf13373; 7c9ebb3 reconciles the module map after the rebase.
• This does not begin Persist a recoverable zero-Vault registry #85, use Wayfinder: multi-Vault web workspace #67 as a prerequisite, or modify the Integrate the reviewed #67 workspace with the frozen API v1 contract #102 frontend / Migrate MCP to explicit Vault scope #103 MCP work. No /api/v1 multi-Vault contract is introduced here.

Preserved decisions and transitional boundary

The conflict resolution preserves current development runtime settings, authentication, scanning, and server composition while retaining the foundation's explicit vault lifecycle/readiness behavior. VaultRuntime is deliberately documented as the transitional single-configured-Vault seam; it is not the accepted registry or per-Vault runtime topology.

The accepted route remains:

• Persist a recoverable zero-Vault registry #85-Import legacy single-Vault deployments once #87: registry, definitions, and legacy import
• Activate isolated per-Vault runtimes #88-Make Vault runtime lifecycle restart- and shutdown-safe #90: per-Vault runtimes, FIFO admission, restart/shutdown
• Vault-qualify shared SQLite snapshots #91-Preserve exact search semantics across Vault scope #93: cache identity, reads, and search
• Apply safe Git behavior to local and existing checkouts #94-Schedule per-Vault Git work and isolate recovery #97: source implementations, managed checkout/sync, scheduling/recovery
• Expose Vault discovery, management, status, and events under API v1 #98-Expose scoped mutations, retire the unscoped API, and freeze API v1 #101: discovery and exact HTTP contracts through mutation/API freeze
• Integrate the reviewed #67 workspace with the frozen API v1 contract #102: frontend integration after Expose scoped mutations, retire the unscoped API, and freeze API v1 #101
• Migrate MCP to explicit Vault scope #103: MCP integration after Expose scoped mutations, retire the unscoped API, and freeze API v1 #101
• Prove the cross-surface acceptance boundary and merge once #104: final documentation, deployment, and cross-surface acceptance

Interface-change checklist

Interface change: vault readiness/status baseline
Producer boundary: Runtime composition (AppState / VaultRuntime) and HTTP diagnostics
Kind: additive and behavior-changing
Old contract: startup and request paths assumed one immediately usable configured vault; no lifecycle capability snapshot was exposed
New contract: /api/vault-status exposes lifecycle/source/mode/capabilities; vault-dependent paths return a stable unavailable/indexing response until ready
Consumer boundaries: server/router, HTTP handlers, settings reindex, MCP routes/tool advertisement; searched through Code/Rust references and the complete branch diff
Compatibility/migration: existing local-vault happy paths remain supported; managed mode is opt-in; this is an intentionally intermediate single-Vault baseline replaced by #85-#101
Rollout/rollback: deploy only as a draft integration baseline; rollback is the pre-rebase PR head/backup branch; later tickets replace the transitional topology in dependency order
Evidence: focused vault_runtime, app_state, server, and config tests plus the full checks below

Interface change: managed Git source configuration
Producer boundary: application configuration and runtime composition
Kind: additive
Old contract: HATCHDOOR_VAULT_SOURCE selected the local configured vault
New contract: opt-in git / managed-git source accepts repository URL, checkout path, optional branch/subdirectory, and pull-only or bidirectional mode
Consumer boundaries: server startup, runtime capability derivation, Git sync composition, external deploy configuration
Compatibility/migration: default remains local; invalid or incomplete managed configuration fails closed; no implicit Vault or compatibility shim is added
Rollout/rollback: configure only when the owning source tickets are complete; remove the managed-source variables to return to local mode
Evidence: configuration/server tests and full checks below

Interface change: lifecycle-aware MCP write advertisement
Producer boundary: MCP tools/routes
Kind: behavior-changing
Old contract: configured MCP writes were advertised without vault lifecycle capability gating
New contract: write tools are advertised only when both MCP writes and the current lifecycle permit mutation
Consumer boundaries: MCP clients, MCP route dispatch, server MCP composition; unknown external clients may cache tool lists
Compatibility/migration: read/setup tools remain available; the exact vault-scoped MCP surface is intentionally deferred to #103 after the #101 API freeze
Rollout/rollback: ship with the lifecycle baseline or revert the integration branch; #103 owns the final multi-Vault contract
Evidence: MCP/server coverage in the full suite below

Architectural invariants checked: HTTP and MCP mutations remain routed through src/vault/write/; Markdown remains authoritative and SQLite disposable; authentication/origin requirements were preserved; search remains consistent with ADR-05; no new framework, service, trait layer, code generation, default-Vault inference, or compatibility shim was added. Rust/TypeScript wire coordination is N/A because frontend production work is explicitly deferred to #102.

Validation

Run from the repository root at tested commit 7c9ebb3ceacbe2fb4c5516f7f19b8bf91355e1b3:

• cargo fmt --all -- --check — passed
• cargo clippy --all-targets -- -D warnings — passed
• cargo check --all-targets — passed
• cargo test vault_runtime — passed (2)
• cargo test app_state — passed (12)
• cargo test server — passed (51)
• cargo test config with a fresh per-command TMPDIR — passed (31)
• cargo test --all — passed (483 total: 473 library, 7 evaluation, 3 CLI; 0 doc tests)
• node scripts/check-module-map.mjs — passed (167 production files assigned exactly once)
• git diff --check origin/development...HEAD — passed
• docker compose config --quiet with a temporary .env -> .env.example symlink — passed; the symlink was removed
• Standards review — no code finding after recording this interface checklist
• Spec review — no findings; confirmed exact base, limited replay, preserved decisions, and no Persist a recoverable zero-Vault registry #85/Wayfinder: multi-Vault web workspace #67/Integrate the reviewed #67 workspace with the frozen API v1 contract #102/Migrate MCP to explicit Vault scope #103 scope

Frontend gates were not run because frontend/** is excluded and unchanged from development. Container build/runtime checks were not run because #84 changes neither packaging nor persistence behavior.

Known intermediate failures

Expected product gaps in this draft

• No /data/state/vaults.json, zero-Vault registry, existing_git source, or multi-Vault collection yet — owned by Persist a recoverable zero-Vault registry #85-Import legacy single-Vault deployments once #87.
• Runtime state is still a single configured Vault and does not yet have the accepted per-Vault runtime/FIFO/restart topology. The legacy StartupTracker forwarding seam and publicly mutable readiness fixture state also remain transitional — owned by Activate isolated per-Vault runtimes #88-Make Vault runtime lifecycle restart- and shutdown-safe #90.
• SQLite identities, reads, and search remain single-Vault — owned by Vault-qualify shared SQLite snapshots #91-Preserve exact search semantics across Vault scope #93.
• Local/existing-Git/managed checkout, synchronization, admission, and recovery are not complete — owned by Apply safe Git behavior to local and existing checkouts #94-Schedule per-Vault Git work and isolate recovery #97.
• Discovery and exact vault-qualified /api/v1 reads/mutations are absent — owned by Expose Vault discovery, management, status, and events under API v1 #98-Expose scoped mutations, retire the unscoped API, and freeze API v1 #101.
• The reviewed Wayfinder: multi-Vault web workspace #67 workspace is not integrated into production UI in this ticket — owned by Integrate the reviewed #67 workspace with the frozen API v1 contract #102 after Expose scoped mutations, retire the unscoped API, and freeze API v1 #101.
• MCP remains the current interim scope-less surface — owned by Migrate MCP to explicit Vault scope #103 after Expose scoped mutations, retire the unscoped API, and freeze API v1 #101.
• Final deployment/docs/cross-surface acceptance remain — owned by Prove the cross-surface acceptance boundary and merge once #104.

Rebase and validation incidents

• A naive rebase exposed 77 unrelated commits from divergent fork history. It was aborted without retaining changes; the successful rebase used --onto to replay only the three PR commits.
• The first /tmp build exhausted that tmpfs while compiling vendored OpenSSL. Generated target output was removed and validation used the repository disk target.
• The sandbox initially could not resolve the registry for the new arc-swap dependency. The authorized network retry downloaded it successfully.
• FastEmbed's native-TLS build needed OpenSSL development artifacts unavailable on the VM. Validation reused the repository's already-built vendored OpenSSL 3.6.3 artifacts.
• Rustup attempted to sync the pinned 1.78 toolchain through a read-only home path. Validation used the installed Rust 1.97.1 rustc/rustdoc binaries directly.
• A repeated focused config test inherited a stale PID-based temporary settings file in this container. Re-running with a fresh per-command TMPDIR passed; this is test-environment contamination, not a product failure.
• Compose config initially stopped because .env was absent. Validation with the repository's .env.example as a temporary .env passed.
• GitHub initially refused to reopen the closed PR after its head branch moved, reporting that the branch had been force-pushed or recreated. The exact old head was temporarily restored under a lease, PR Establish managed Git vault lifecycle foundation #31 was reopened, and the tested head was reapplied under a second exact lease; the final PR head was verified through the API.
• gh pr edit hit GitHub's deprecated Projects-classic GraphQL error. The same evidence body was applied through the pull-request REST endpoint and verified live.
• Git emitted stale credential-helper-path warnings around the authenticated HTTPS pushes, but each leased remote ref update succeeded and the final fork/PR SHA was independently confirmed as 7c9ebb3ceacbe2fb4c5516f7f19b8bf91355e1b3.

These setup incidents do not create product follow-up tickets. The remaining functional failures are explicitly owned by #85-#104 as listed above.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a5f076ed22

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Distinguish vault failures from model setup

When a managed source enters Unavailable—currently the managed-mode outcome set by run_server—this ready-only dispatch falls back to setup_tools_list, while tools/call also permits the Gemma accept/decline tools for every non-ready phase. Headless MCP clients are therefore told that model setup is required and can trigger an unnecessary model download even though the actual error is managed_vault_not_acquired. Expose lifecycle status for non-model failures and reserve the setup tools for TermsRequired or model-setup phases.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9fe7f77988

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Avoid rescanning the vault after every MCP note write

For every successful MCP note mutation, refresh_after_write has already performed a full vault scan and rebuilt the published SQLite index, but this unconditional current_index immediately performs another O(vault) filesystem walk while the caller still holds vault_write_lock. On large vaults this doubles scan work for common writes and unnecessarily blocks browser writes and Git synchronization; retain the extra scan only for the slug-recovery case and obtain the resulting layer from the newly published SQLite cache as before.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Preserve read-only attachment listing in pull-only mode

For a ready pull-only managed vault with MCP writes configured, this mutation-capability guard rejects list_note_attachments, and tools_list also omits it because the entire write-module list is gated by lifecycle_write_enabled. The tool only reads the vault and is explicitly annotated with read_only_tool_annotations, so pull-only clients lose a browsing operation even though the lifecycle advertises browsing as available; gate this tool only by its existing MCP configuration policy and expose it separately from mutation tools.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: bad5d374be

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Route non-model failures away from model retry

When HATCHDOOR_VAULT_SOURCE=git, run_server sets the lifecycle to Unavailable with managed_vault_not_acquired, but this conversion exposes every unavailable state through /api/startup-status as the legacy failed model/index state. The existing frontend/src/startup/StartupGate.tsx stops polling on that state and displays a “Retry setup” button that calls /api/model/retry; clicking it needlessly reloads the embedding model before spawn_model_startup detects the managed source and returns to the same failure. Preserve a distinct non-model status (or update the frontend to consume /api/vault-status) so managed-vault failures do not offer a model retry that cannot recover them.

Useful? React with 👍 / 👎.

[BattermanZ]

Superseded by #62

[alemhnan]

Moved to the canonical same-repository branch and draft PR #106.

The replacement PR has the same development base, title, full implementation/validation record, commit history, and tested head commit 3ed9ecba8f1efa3e7a9c8335ce1d6c50d73d9496. This fork-backed PR is being closed as superseded, but remains available as the archive for its historical review threads.