Rename @launchstack/search to @launchstack/retrieval and consolidate all retrieval algorithms and tools into it - #371
Conversation
Mechanical rename only — directory, package name, the 13 TS import sites, five workspace dependents, jest moduleNameMapper, next.config transpilePackages, the CI publish filter, the e2e-ingest paths, and the pending changeset. ESLint: the brick list, the per-package block, and the tools allowlist now say retrieval, and @launchstack/search joins the legacy-name ban so stale branches fail loudly instead of silently resolving nothing. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… documented folders
Every algorithm and tool is now a folder with its own README, index.ts
surface, and colocated tests: algorithms/{bm25,vector,ensemble,rlm,graph,
reranking} and tools/{citation-builder,rag-search-tool}. The RLM, graph,
and ensemble retrievers move in from apps/web/src/lib/tools/rag with two
seams cut on the way: ~/server/db becomes the store client (same engine
connection), and the ensemble's env reads become configureEnsemble(),
called from the app's new composition seam ~/server/rag/ensemble.ts.
Notes retrieval stays app-side (its tables are product schema the engine
store cannot own) and joins the ensemble as an injected leg — resolving
design question Q2 the low-risk way. The Neo4j graph backend keeps using
indexing's graph client; retrieval's lint allowlist gains that one edge
(Q4). The rag-search tool becomes a factory taking an app-supplied
AccessValidator, since document access lives in product schema too.
Both shim layers (lib/tools/rag, server/rag re-exports) are deleted; old
package subpaths (./retrievers, ./reranking, ./citation-builder) survive
one release as aliases. Ensemble searches now log a per-leg breakdown so
a silently dead leg shows in logs, not just a smaller total.
Verified: workspace typecheck, lint at baseline (0 errors / 62 warnings),
retrieval+tools+evidence+conversion+worker+orchestration+indexing suites,
and the full apps/web jest run (2242 passed) including the vector-retriever
and document-creation integration tests against a migrated pgvector 16.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…o the package annOptimizer's five ANN variants become named, documented modules under algorithms/vector/strategies/ (exact scan, ivf, prefiltered, matryoshka, adaptive hybrid), with ANNOptimizer surviving as the config-driven dispatcher — documentQ&A's AIQuery/AIChat routes now import it from the package instead of reaching into another feature's folder. The feature's private hybridSearch dissolves along the design's lines: the ts_rank full-text leg into algorithms/bm25/fts.ts, generic Reciprocal Rank Fusion plus the composed page-level hybrid search into algorithms/fusion/, with the query embedder injected rather than chosen by the algorithm. Two of the three independent embedding clients are retired: predictive's utils/embeddings and all three notes-vertical OpenAIEmbeddings instantiations now generate through @launchstack/llm's embedding service (same endpoints, models, and dimensions — the files keep only env-pair resolution, caching, and error semantics). One embedding transport remains. Verified: workspace typecheck, lint at baseline, retrieval + tools vitest, full apps/web jest (2242 passed) against the migrated pgvector database. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
grounded-retrieval relocates from packages/tools — it is a retrieval
policy, not a vertical capability, and holding it in tools forced a
cross-brick import for every consumer. A re-export keeps the old path
alive for brand-voice/company-context/claim-evidence/persona and the
pipelines' architecture rule ("no feature calls the RAG port directly").
The cost-aware RLM search service moves out of documentQ&A wholesale:
its only app dependencies turned out to be compositions of llm-package
primitives (createEmbeddingModel + resolveEmbeddingIndex) and the store
client, so it lands in tools/rlm-search with zero injected seams. The
Q&A services barrel and AIQueryRLM route re-point to the package.
Every tool folder now carries the convention: README, index.ts surface,
implementation beside it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…rface Four golden tests against the migrated pgvector database pin the moved ensemble's behavior: top-hit ranking on a fixed corpus, the RRF property that cross-leg agreement beats a single leg's top rank, a liveness check on the vector leg exactly as the ensemble constructs it, and the silent-death canary (retrieval on a seeded corpus must never be empty — the failure mode the design named worst). If a change legitimately improves ranking, the goldens move in the same commit, with a reason. Also: the package README now documents the folder-per-algorithm layout and subpath map, the changeset describes the full consolidation, and the publish path is verified — all 154 workspace exports load from built output (check-package-exports). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…eside the retrieval rename Conflicts: the CI publish filter (kept retrieval, added document-conversion-engine), REPOSITORY.md (kept main's aligned table with the two new bricks, re-applied the retrieval row and the tools-allowlist rename), and the lockfile (regenerated from the merged manifests). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit d0efbb2. Configure here.
| ); | ||
| const { buildCitations } = await import( | ||
| "../../packages/search/src/citation-builder.ts" | ||
| "../../packages/retrieval/src/citation-builder.ts" |
There was a problem hiding this comment.
E2E ingest uses stale retrieval paths
Medium Severity
The ingest e2e script still targets packages/retrieval/src/retrievers/bm25-retriever.ts and packages/retrieval/src/citation-builder.ts. Those files now live under algorithms/bm25 and tools/citation-builder, so the script cannot load the moved modules.
Reviewed by Cursor Bugbot for commit d0efbb2. Configure here.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d0efbb2d79
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| @@ -0,0 +1,151 @@ | |||
| { | |||
| "name": "@launchstack/retrieval", | |||
There was a problem hiding this comment.
Renaming the workspace here deletes packages/search/package.json, but the dependency stages in apps/web/Dockerfile:23, apps/web/Dockerfile.prebuilt:28, and apps/worker/Dockerfile:27 still copy that exact path. Every build using any of these three inspected Dockerfiles therefore stops at COPY before dependencies can be installed; update those manifest paths and destinations to packages/retrieval as part of the rename.
Useful? React with 👍 / 👎.
| "neo4j-driver": { | ||
| "optional": true | ||
| } |
There was a problem hiding this comment.
When a consumer omits the peer explicitly marked optional here, importing @launchstack/retrieval/algorithms/ensemble, @launchstack/retrieval/algorithms, or the compatibility retrievers export still evaluates ensemble.ts, which statically imports ../graph, whose neo4j.ts statically imports neo4j-driver. This throws a module-resolution error before the default graphRetrieval: false configuration can disable that backend, so the Neo4j module needs to be loaded lazily or the driver must be a required dependency.
Useful? React with 👍 / 👎.
1 participant
Summary
packages/search→packages/retrievaland finishes the long-stalled boundary inversion: every retrieval algorithm and retrieval-facing tool now lives in the one brick, organized as one documented folder per algorithm —algorithms/{bm25,vector,fusion,ensemble,rlm,graph,reranking}andtools/{citation-builder,grounded-retrieval,rag-search-tool,rlm-search}, each folder carrying a README (what it is, how it works, when it wins), anindex.tssurface, and colocated tests.algorithms/vector/strategies/(documentQ&A stops importing from another feature's folder), its private hybrid search dissolves intobm25/fts.ts+fusion/(RRF), and three independent embedding clients (predictive's, plus three notes-verticalOpenAIEmbeddingsinstantiations) collapse into@launchstack/llm's embedding service.RagPortcontract and retrieval behavior are unchanged; the ensemble's env reads becomeconfigureEnsemble()injected from the app's new composition seam (apps/web/src/server/rag/ensemble.ts), and old subpaths (./retrievers,./reranking,./citation-builder) survive one release as aliases.@launchstack/searchjoins the lint ban on legacy names.Design doc (LaunchStack template, with resolutions recorded): https://claude.ai/code/artifact/d4082c18-aa54-48d7-ada7-fe4448c8f204
Related
Implements the Retrieval Brick consolidation design. Deliberately does not touch the predictive-document-analysis kill-or-rebuild decision — only the shared algorithms trapped in its folder.
Checklist
pnpm checkpasses (lint + typecheck) — lint 0 errors, no new warningspnpm --filter @launchstack/web testpasses — 2,273 passed (196 suites), incl. integration tests against a migrated pgvector 16.changeset/retrieval-rename.md)Testing
indexingfor exactly one edge — the Neo4j graph backend reusing indexing's graph client).apps/webjest suite plus evidence/conversion/retrieval/tools/worker/orchestration/indexing package suites, run against a migratedpgvector/pgvector:pg16database (CI's image).apps/web/__tests__/server/retrieval-golden.test.ts, gated onDATABASE_URL): pinned ensemble ranking on a fixed corpus, the RRF cross-leg-agreement property, a liveness check on the vector leg exactly as the ensemble constructs it, and a non-empty canary for the worst failure mode (silent empty-context retrieval).pnpm -r build+scripts/ci/check-package-exports.mjs— all 154 exports load from built output.Notes for reviewers
~/server/db→ the store client (same engine connection — the app'sdbis already a proxy over it) and env flags →configureEnsemble({ graphRetrieval, notesLegs }).documentNoteEmbeddingsintopackages/store: the engine/product migration-ledger boundary (check-schema-boundary.mjs) makes promotion a migration-history move. Recorded as the Q2 resolution in the design doc; reversible later.grounded-retrievalmoved frompackages/toolswith a re-export kept, so its four internal consumers and the pipelines' "no direct RAG port calls" architecture test hold unchanged.AccessValidator(document access is product schema). It had no live consumers; behavior is preserved for future wiring.origin/mainas of4dc8b0da(Drive-linked files + PDF conversion bricks); conflicts were the CI publish filter, REPOSITORY.md's table, and the lockfile.@launchstack/search0.1.0 on npm (design Q5) and the predictive-analysis product decision.🤖 Generated with Claude Code
Note
Medium Risk
Large cross-cutting refactor of document Q&A and predictive matching retrieval paths; ranking behavior is guarded by new golden tests but any missed import or ensemble misconfiguration could still yield silent empty context.
Overview
Renames
@launchstack/searchto@launchstack/retrievaland folds scattered retrieval code into one brick: algorithms undersrc/algorithms/(bm25, vector with ANN strategies, fusion, ensemble, rlm, graph, reranking) and agent-facing tools undersrc/tools/. The old package name is ESLint-banned; legacy subpaths (./retrievers,./reranking,./citation-builder) remain as one-release aliases.Moves implementation out of
apps/web: the former~/lib/tools/ragstack is removed; Q&A and related routes import algorithms from@launchstack/retrievaland ensemble search only through~/server/rag/ensemble.ts, which callsconfigureEnsemble()with deployment flags and an injected notes leg (notes schema stays app-side). Document access for RAG tools lives in~/server/rag/access.ts. Predictive-analysisannOptimizerandhybridSearchare deleted in favor of@launchstack/retrieval/algorithms/vectorandfusion(hybridSearchWithRRFnow takes an injected embedder). Graph/RLM retrievers use@launchstack/store/clientinstead of the app DB proxy.Behavioral / wiring tweaks: the ensemble no longer reads
process.envinside the package; graph enablement and Neo4j backend selection are split (config flag vsisNeo4jConfigured()). Ensemble runs log per-leg candidate counts. Notes and predictive embedding paths route through@launchstack/llminstead of direct LangChain OpenAI clients. New DB golden tests pin ensemble ranking on a fixed corpus whenDATABASE_URLis set.CI,
REPOSITORY.md, engine/tools/worker/web dependencies, and Jest module maps are updated to@launchstack/retrieval.Reviewed by Cursor Bugbot for commit d0efbb2. Bugbot is set up for automated code reviews on this repo. Configure here.