feat(hub): gated federation — push/pull + receiving-side review gate#536
Merged
Conversation
the receiving side of vouchhub. inbound bundle knowledge no longer writes straight to the committed store: bundle.import_as_proposals files each inbound claim as a pending proposal via propose_claim, so nothing lands without passing this kb's own proposals.approve() -- the load-bearing invariant (roadmap step 10) upheld by design, not by surface removal. - import_as_proposals: sources/evidence registered as substrate (the inbox shape), claims filed as pending proposals; pages/entities/relations reported as deferred, never silently dropped. - provenance: the proposing actor is hub:<origin-kb> and each proposed claim carries an origin:<kb> tag that survives approval, so a reviewer and the durable claim both show which kb vouched. - kb instance id: a lazy per-kb identity in a .vouch/instance_id sidecar, stamped into the bundle manifest as kb_id; kept out of config and every exported subdir so identity never travels inside a bundle or collides with a receiver's config on import. - hub pull rewired to the gated path (no on-conflict -- a colliding claim is another proposal for the reviewer, not a destructive overwrite); new vouch import-proposals cli for file-based gated federation. tests: test_gated_import.py covers the gated lifecycle, provenance, conformance (only approve makes it durable), and instance id; the hub pull tests are rewired to assert proposals instead of committed writes. full suite green, mypy clean.
closes the second federation receive path that bypassed the review gate. sync_apply --as-proposals lands another kb's NEW claims as pending proposals (sources/evidence registered as substrate; pages/entities/relations/sessions/ decided reported as deferred), so nothing reaches the committed store without passing this kb's own proposals.approve() -- the roadmap step-10 invariant. the direct-write default is unchanged: like import-apply it stays the human tool for reconciling your own kbs. --as-proposals (roadmap 8.2) is the federation-safe path for accepting another kb's knowledge. the propose-inbound-claim + origin-tag provenance logic is extracted from bundle.import_as_proposals into shared bundle.propose_inbound_claim and inbound_claim_id, so both gated receive paths (bundle import and sync) stamp identical provenance. tests: gated sync files pending not committed, approves into a claim carrying the origin tag, and the cli --as-proposals flow. full suite green, mypy clean.
a claim imported through the gate carries an origin:<kb> tag; build_context_pack now reads it onto each returned item (ContextItem.origin) and lists every contributing kb in the pack's origins field, so a reader can see which knowledge came from another kb and which is local -- the read-time half of "every result names the kb that vouched for it" (roadmap step 10). locally-authored claims have origin=None and add no origins field. tests: a federated claim surfaces its origin on recall; a local claim does not. full suite green, mypy clean.
github-actions
Bot
added
cli
WalkthroughThis change adds filtered bundle manifests, KB instance identity, proposal-gated bundle and sync imports, provenance-aware context results, VouchHub push/pull/status commands, and corresponding CLI, JSONL, and test coverage. ChangesFederated bundle synchronization
Estimated code review effort: 4 (Complex) | ~60 minutes Sequence Diagram(s)sequenceDiagram
participant CLI
participant HubClient
participant Bundle
participant VouchHub
CLI->>HubClient: run hub push or pull
HubClient->>Bundle: export or import bundle
HubClient->>VouchHub: authenticated PUT or GET
VouchHub-->>HubClient: bundle, ETag, or conflict
HubClient-->>CLI: return synchronization result
Possibly related PRs
Suggested labels: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/vouch/bundle.py`:
- Around line 804-809: Update the source import logic in src/vouch/bundle.py
lines 804-809 and the corresponding sync logic in src/vouch/sync.py lines
332-336 to load each source’s sibling meta.yaml alongside sources/<sha>/content,
preserve its title, locator, url, tags, and metadata fields, and pass those
values to store.put_source(...); apply the same metadata-preserving behavior at
both sites.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 0e83bfbf-abad-48bb-8829-9a8b7c6b5f89
📒 Files selected for processing (12)
src/vouch/bundle.pysrc/vouch/cli.pysrc/vouch/context.pysrc/vouch/hub_client.pysrc/vouch/jsonl_server.pysrc/vouch/models.pysrc/vouch/storage.pysrc/vouch/sync.pytests/test_bundle.pytests/test_gated_import.pytests/test_hub_client.pytests/test_sync.py
Comment on lines
+804
to
+809
| if m.name.startswith("sources/") and m.name.endswith("/content"): | ||
| sha = m.name.split("/")[1] | ||
| contents[sha] = tar.extractfile(m).read() # type: ignore[union-attr] | ||
| for _sha, body in sorted(contents.items()): | ||
| store.put_source(body) | ||
| sources_registered += 1 |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# inspect the relevant file layouts and symbols
git ls-files src/vouch/bundle.py src/vouch/sync.py src/vouch/storage.py
echo
echo "== bundle.py outline =="
ast-grep outline src/vouch/bundle.py --view expanded || true
echo
echo "== sync.py outline =="
ast-grep outline src/vouch/sync.py --view expanded || true
echo
echo "== storage.py outline =="
ast-grep outline src/vouch/storage.py --view expanded || true
echo
echo "== relevant call sites =="
rg -n "put_source\(" src/vouch/bundle.py src/vouch/sync.py src/vouch/storage.py
rg -n "meta\.yaml|sources/" src/vouch/bundle.py src/vouch/sync.py src/vouch/storage.pyRepository: vouchdev/vouch
Length of output: 10332
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== storage.py put_source/get_source/list_sources =="
sed -n '338,410p' src/vouch/storage.py
echo
echo "== bundle.py import_apply and import_as_proposals =="
sed -n '611,845p' src/vouch/bundle.py
echo
echo "== sync.py bundle-source path =="
sed -n '150,360p' src/vouch/sync.py
echo
echo "== source model definition =="
rg -n "^class Source\\b|^class Source\\(" src/vouch -g '*.py'Repository: vouchdev/vouch
Length of output: 21037
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== artifact kind classification =="
sed -n '60,125p' src/vouch/sync.py
echo
echo "== source model =="
sed -n '163,230p' src/vouch/models.py
echo
echo "== bundle source export/import docs =="
sed -n '285,335p' src/vouch/bundle.py
sed -n '490,540p' src/vouch/bundle.pyRepository: vouchdev/vouch
Length of output: 8526
preserve source metadata on gated import
both src/vouch/bundle.py#L804-L809 and src/vouch/sync.py#L332-L336 only persist sources/<sha>/content, so sources/<sha>/meta.yaml fields (title, locator, url, tags, metadata) are dropped on first import. if those fields should survive federation, load the sibling meta file here too and pass its values into store.put_source(...).
📍 Affects 2 files
src/vouch/bundle.py#L804-L809(this comment)src/vouch/sync.py#L332-L336
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/vouch/bundle.py` around lines 804 - 809, Update the source import logic
in src/vouch/bundle.py lines 804-809 and the corresponding sync logic in
src/vouch/sync.py lines 332-336 to load each source’s sibling meta.yaml
alongside sources/<sha>/content, preserve its title, locator, url, tags, and
metadata fields, and pass those values to store.put_source(...); apply the same
metadata-preserving behavior at both sites.
resolve the conflicts from the federation gate meeting the test branch's identity, prompt-gate and personal-kb work: - bundle.py: union both feature sets — the exclude filter + instance-id kb_id provenance (this branch) and identity-stripped config export + fenced_bundle_path + kb identity block (test). export now fences the dest, strips config identity, forwards exclude, and stamps both kb_id and kb; _yaml_dump drops its dead-code marker since test now calls it. - cli.py: the two branches each added a `hub` command group; a second @cli.group() silently shadowed the first. fold hub link/push/pull/status onto the single hub group so test's register/list/init-personal/fallback subcommands survive alongside them. - context.py: keep both result blocks — federated `origins` and the retrieval-honesty `retrieval` block. - jsonl_server.py: export handler fences the path (test) and forwards the exclude filter (this branch). also clear pre-existing ruff debt in the hub-client base commits so the now-computable merge passes ci (conflicts had blocked pull_request ci from running at all): line length, dead noqa directives, mutable class defaults via ClassVar, and a redundant session_split import surfaced by the merge.
…ield the gated-import work added ContextItem.origin (and the pack-level origins) but the generated json schemas were never regenerated — the drift was latent because the conflict blocked schema-check from running on the pr. regenerate context-item and context-pack from models.py so the drift gate passes.
github-actions
Bot
added
schemas
There was a problem hiding this comment.
🧹 Nitpick comments (1)
src/vouch/bundle.py (1)
126-167: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick win
export()re-walks and re-reads every file already processed bybuild_manifest.
build_manifest(Line 131-132) iterates_iter_export_filesand computes_export_file_bytes/sha256_hexfor every file, thenexport()(Line 195, 198-201) walks the same file set again and recomputes_export_file_bytesfor the tar. Every exported file is read from disk twice per export call. For a KB with many claims/pages/sources this doubles I/O on a path that is otherwise carefully optimized (see the comment about hash-based bundle IDs).Consider having
build_manifestreturn (or accept) the already-computed(rel, data)pairs soexport()reuses them instead of re-reading from disk.♻️ Suggested direction
-def build_manifest( - kb_dir: Path, exclude: tuple[str, ...] = (), *, kb_id: str | None = None -) -> dict[str, Any]: +def build_manifest( + kb_dir: Path, exclude: tuple[str, ...] = (), *, kb_id: str | None = None, + file_data: dict[str, bytes] | None = None, +) -> dict[str, Any]: exclude = _check_exclude(exclude) files: list[dict[str, Any]] = [] for rel, abs_path in _iter_export_files(kb_dir, exclude): data = _export_file_bytes(rel, abs_path) + if file_data is not None: + file_data[rel.as_posix()] = data files.append(...)Then
export()passes a dict in, and reuses it when writing tar members instead of calling_export_file_bytesagain.Also applies to: 187-216
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/vouch/bundle.py` around lines 126 - 167, Refactor the build_manifest/export flow so each export file is read only once: have build_manifest expose or accept the computed (rel, data) pairs, then update export() to pass and reuse those bytes when writing tar members instead of calling _export_file_bytes again. Preserve the existing manifest hashes, paths, counts, and archive contents.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@src/vouch/bundle.py`:
- Around line 126-167: Refactor the build_manifest/export flow so each export
file is read only once: have build_manifest expose or accept the computed (rel,
data) pairs, then update export() to pass and reuse those bytes when writing tar
members instead of calling _export_file_bytes again. Preserve the existing
manifest hashes, paths, counts, and archive contents.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 79629c02-bcfa-44c8-878f-b761843b5b46
📒 Files selected for processing (11)
schemas/context-item.schema.jsonschemas/context-pack.schema.jsonsrc/vouch/bundle.pysrc/vouch/cli.pysrc/vouch/context.pysrc/vouch/hub_client.pysrc/vouch/jsonl_server.pysrc/vouch/models.pysrc/vouch/storage.pytests/test_bundle.pytests/test_hub_client.py
🚧 Files skipped from review as they are similar to previous changes (6)
- src/vouch/models.py
- src/vouch/context.py
- src/vouch/jsonl_server.py
- tests/test_bundle.py
- tests/test_hub_client.py
- src/vouch/storage.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
what
vouch federation, end to end and gated. this branch stacks the hub client
(push/pull/link/status, pre-existing on feat/knowledge-bundle-export) with a new
receiving gate so inbound knowledge can never bypass review.
the load-bearing change: before this, every federation receive path
(bundle.import_apply, sync.sync_apply, hub pull) wrote inbound claims straight
into the committed store -- the invariant was upheld only by removing those
methods from agent surfaces. now it is upheld by design.
the three new slices (on top of the hub-client base)
a pending proposal via propose_claim; sources/evidence are registered as
substrate; pages/entities/relations are reported as deferred, never dropped.
provenance: the proposing actor is hub: and each claim carries an
origin: tag that survives approval. a per-kb instance id
(.vouch/instance_id sidecar) is stamped into the bundle manifest as kb_id. hub
pull is rewired to this path; a new vouch import-proposals cli covers
file-based gated import.
as proposals; the direct-write default is unchanged (a human reconciling their
own kbs, same posture as import-apply). the propose-inbound-claim logic is
shared so both receive paths stamp identical provenance.
field, so recalled federated knowledge names which kb vouched for it.
the base commits (pre-existing, not authored here)
b74f8dd..4380920 are the hub-client work (export exclude filter, knowledge-only
preset, hub link/push/pull/status). they are unmerged and unpushed elsewhere, so
they ride along here; the receiving gate depends on them. if you'd rather land
them separately, this branch rebases cleanly once they're in.
tests
new: test_gated_import.py (gated lifecycle, provenance, conformance, instance id,
read-time origin) and gated-sync cases in test_sync.py; the hub pull tests are
rewired to assert proposals instead of committed writes. full suite green, mypy
clean, ruff clean on the authored files.
invariant
any data path that lands writes without a receiving-side proposal is wrong
(roadmap step 10). test_conformance_gate_holds_only_approve_makes_it_durable
locks it: after import the committed claim file does not exist; only approve()
creates it.
Summary by CodeRabbit
--exclude, plus richer export metadata.sync-apply --as-proposalsmode with origin provenance.origin/origins).