fix(bootstrap): promote external dependencies to first-class components - #138
Open
Rontu22 wants to merge 5 commits into
Open
fix(bootstrap): promote external dependencies to first-class components#138Rontu22 wants to merge 5 commits into
Rontu22 wants to merge 5 commits into
Conversation
Contributor
|
This will take some time to review. I'll run this a few times to catch any regressions, and then will merge. From the top of it, looks like great work!! |
Issue Autoloops#79: bootstrap buried technologies the repo depends on (a message broker, datastore, third-party API, build/deploy platform) inside claim text instead of giving them their own component, so `graph context "<name>"` returned vague claims instead of a dedicated node. Nothing was mechanically blocking this before: a bare component plus claims `about` it and flows that `touch` it already worked end-to-end. The bootstrap and layered-refresh prompts, and both bootstrap skills, simply never instructed the agent to do it. Adds generic, principle-based promotion guidance (load-bearing externally-owned dependency, referenced by multiple claims/flows, or likely to be queried by name) using only existing primitives - no schema changes, no fixed technology list.
Autoloops#79 Three layers of verification for the promotion fix: - check-dependency-promotion-guidance.js (fast, in npm test): lints the four prompt/skill files for the required criteria, connection mechanism, and contains anti-pattern, and asserts none of them hardcode a technology name - a direct regression guard on the "must stay generic" requirement. - check-dependency-promotion-model.js (fast, in npm test): proves the data model deterministically - a promoted component ends up with real about/ touches edges and never a contains edge, while the unpromoted case has no dedicated node for the dependency. - check-dependency-promotion-retrieval.js (slow, opt-in via `npm run test:dependency-promotion-retrieval`): the real proof, using actual local embeddings. Reproduces the exact issue Autoloops#79 symptom (`graph context "Broker"` returns no dedicated node) and shows it fixed once the recipe the guidance teaches is followed. Kept out of the default suite because local embedding calls take 10s+ each. This still does not prove a real bootstrap agent will follow the prompt guidance - only that the mechanism produces the right outcome when it is followed.
…loops#79 Real agent-driven eval case testing the actual fix, using a small bundled fixture repo (Order API + Redis, git-initialized fresh on every run instead of cloning an external URL, so results are deterministic and not dependent on a third-party repo staying reachable). Drives a real codex agent through the bootstrap skill and checks whether Redis - an unambiguous external dependency the fixture never hints should become a component - actually gets promoted. Pass/fail is a deterministic check on the raw proposal (does any component id/name mention "redis"), independent of the optional OpenAI judge, which adds a richer rubric-based quality read on top when GREPLICA judge flags are passed. Verified the harness plumbing (install/validate/apply/detection) with --proposal against both a hand-authored good and bad sample proposal before spending real codex calls; both directions report correctly (exit 0 / exit 1). Run with: npm run eval:dependency-promotion-79 -- --agent codex
Ran the eval against a real codex agent for both the old (main) and new
SKILL.md as an A/B test. The old skill scored a false pass: it folded
Redis into an internal component's free-text name ("Order API server and
Redis publisher") instead of giving it its own component - structurally
the same bug issue Autoloops#79 describes, just disguised. The original check
matched on `id + name`, so that compound name tripped a false positive.
Now only `id` is checked (split into tokens), since ids are canonical
identifiers an agent assigns specifically to a concept, unlike prose
`name` fields that can smuggle a mention of something into an unrelated
component. Re-verified against both saved proposals: old skill now
correctly reports not-promoted, new skill still correctly reports
promoted - a real, causal, agent-driven confirmation that the fix changes
outcomes, not just a false-positive artifact of the eval script.
Also drop the eval:dependency-promotion-79 package.json script per
feedback that one-off eval cases don't belong there - run directly with
`node dist/evals/cases/dependency-promotion-79/run.js -- --agent codex`.
… technologies Widens all three layers of verification from a single dependency (Redis) to the fuller spread the original issue named plus a few more: Postgres, Kafka, Redis, ClickHouse, Docker, ECS, and Prisma. Also drops test:dependency-promotion-retrieval from package.json - one-off eval/test scripts don't belong there; run it directly with `node scripts/check-dependency-promotion-retrieval.js`. - check-dependency-promotion-model.js and check-dependency-promotion- retrieval.js are now table-driven over 7 dependency shapes instead of one, so the structural and retrieval proofs aren't an artifact of one technology's name/word-shape. - Declaration-point file types are deliberately mixed rather than assuming docker-compose covers everything: docker-compose.yml, Kubernetes manifests (k8s/kafka-statefulset.yaml, k8s/clickhouse-deployment.yaml), an ORM schema (prisma/schema.prisma), a Dockerfile, and an ECS task definition (deploy/ecs-task-definition.json). - The dependency-promotion-79 eval fixture is now a small but realistic multi-dependency service (Order API backed by Postgres/Prisma, Kafka, Redis, and ClickHouse; containerized with Docker; deployed on ECS) instead of a single-dependency repo, mirroring the issue's own example much more closely. run.ts's detection logic generalizes from a single Redis check to checkDependenciesPromoted(), which requires all six dependencies to get their own component for the eval to pass. Re-verified the harness end to end with --proposal against both an updated good sample (all six promoted) and a bad sample (all six folded into one compound component name); both directions report correctly.
Rontu22
force-pushed
the
fix/promote-dependency-components-79
branch
from
July 11, 2026 15:21
d97bc09 to
02e0df5
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
While bootstrapping memory for a repo, Greplica correctly identified high-level, internally-owned components (API Server, Frontend, etc.), but infrastructure/technology dependencies the repo does not own — a message broker, datastore, third-party API, build/deploy platform — were only ever mentioned inside claim text, never represented as their own component. As a result,
greplica graph context "<name>"for one of these dependencies returned general claims instead of a dedicated graph node with its own relationships.Closes #79.
Root cause
Nothing in the schema or storage layer was blocking this. A bare component plus a claim
aboutit and a flow thattouchesit already worked end-to-end before this change —Componentonly ever requiredidandname. The gap was purely in the bootstrap guidance: the prompts and skills that drive the bootstrapping agent never instructed it to promote these dependencies to components, so it defaulted to describing them as an aside inside claims about the internal component that used them.What changed
This PR is prompt/skill guidance only — no schema, storage, or CLI changes:
scripts/memory-build/prompts/deep-bootstrap.mdandlayered-deep-bootstrap.md(the internal deep-bootstrap and refresh prompts) gain a new "Promote First-Class Dependencies" workflow step.skills/greplica-bootstrap/SKILL.mdandskills/greplica-fast-session-bootstrap/SKILL.md(the user-facing bootstrap skills) get the same guidance in condensed form.The guidance is deliberately generic — no hardcoded technology list (Kafka/Docker/ECS/Prisma/ClickHouse/... never appear in it). It gives principle-based promotion criteria instead:
code_anchor) at its declaration point — a compose/deploy file, schema file, client-init module, or dependency manifest.aboutit, flows thattouchit.contains—containsmeans ownership, and the repo does not own the dependency. (This is exactly the anti-pattern implied by the issue's own diagram of Kafka/Prisma/etc. nested under API Server.)Testing
Three layers of automated coverage, all in this PR:
check-dependency-promotion-guidance.js(fast, innpm test) — lints all four prompt/skill files for the required criteria and connection mechanism, and asserts none of them contain a hardcoded technology name (a direct regression guard on the "must stay generic" requirement).check-dependency-promotion-model.js(fast, innpm test) — proves the data model deterministically: a promoted dependency ends up with realabout/touchesedges and never acontainsedge; the unpromoted case has no dedicated node.check-dependency-promotion-retrieval.js(slow, opt-in — real local embeddings take 10s+ per call, so it's not in the default suite) — reproduces the exact issue Feature: Promote infrastructure subsystems to first-class graph components during bootstrap #79 symptom and shows it fixed: same query,graph context "Broker"returns no dedicated node before promotion, and a dedicated "Broker" component ranked alongside its claim after.Real agent-driven verification (
evals/cases/dependency-promotion-79/): rather than only proving the mechanism works when followed, this PR includes an eval case that drives an actual codex agent through the bootstrap skill against a bundled fixture repo (an "Order API" that publishes events to Redis — an unambiguous external dependency the fixture never hints should become a component). I ran this as a real A/B test, same fixture and model, only the skill content differed:SKILL.md: the agent createdcomponent.redis, anchored atdocker-compose.yml, connected viaabout(3 claims) andtouches(2 flows), and explicitly excluded from the internal component'scontainslist.main'sSKILL.md): the agent did not give Redis its own component — it folded it into an internal component's free-text name ("Order API server and Redis publisher"), reproducing the exact bug this issue reports, just in a different disguise.npm run build npm test node dist/evals/cases/dependency-promotion-79/run.js -- --agent codexScope
Out of scope for this PR (tracked separately): an optional
kindcategorization field on components (external_service/datastore/library/infrastructure/module) for richer visualization/retrieval. The core fix here — a dependency existing as a component with real edges — is sufficient on its own to resolve the retrieval gap the issue reports; categorization is a separate enhancement, not required for the fix.