feat(context-lake): org-wide context lake for agent retrieval

[ZaltaClaw]

Concept

Context Lake is a data lake for company knowledge — a queryable, org-wide store of unstructured and structured context (ADRs, decision logs, customer notes, glossary, code conventions, sales playbooks, security policies) that any agent in Frontier can retrieve from at runtime.

Architecture

Sources (file-upload, SharePoint, Confluence, Notion, GitHub)
  ↓  SourceAdapter interface  (src/lib/context-lake.ts)
Context Lake Core  (list, query, add-source)
  ↓
API routes  (GET /api/context-lake/sources|documents, POST /api/context-lake/query)
  ↓
UI  (Sources tab → Documents tab → Bindings tab → Query Playground)
  ↓
Agent Bindings  (attach ranked chunks to system prompt — stub, follow-up)

What's real vs. stubbed

Feature	Status
File-upload connector	✅ Real (backed by seed data in data/context-lake/)
Query API (keyword/tag ranker)	✅ Real — works without AI Search credentials
Query Playground UI	✅ Real — calls POST /api/context-lake/query
ContextDocument metadata layer	✅ Real (domain, freshness, ACL, lineage, tags)
Sources / Documents / Bindings tabs	✅ Real UI
SharePoint / Confluence / Notion / GitHub connectors	🔲 Stub — wizard UI + config persist; sync not implemented
Agent Bindings (runtime injection)	🔲 Stub UI — schema + empty state; CRUD is follow-up
AI Search / vector ranking	🔲 Follow-up — ranker falls back to keyword scoring

New files

API

• src/app/api/context-lake/sources/route.ts — GET (list) + POST (add source)
• src/app/api/context-lake/documents/route.ts — GET with optional ?domain= filter
• src/app/api/context-lake/documents/[id]/route.ts — GET single document
• src/app/api/context-lake/query/route.ts — POST {query, domains?, limit?} → ranked chunks

Pages

• src/app/context-lake/page.tsx — main 3-tab page (Sources / Documents / Bindings)
• src/app/context-lake/sources/page.tsx — sources deep-dive
• src/app/context-lake/documents/[id]/page.tsx — document detail

Components (src/components/context-lake/)

• source-card.tsx — source type, status badge, doc count, last sync
• connect-source-wizard.tsx — 3-step modal: pick type → fill credentials → confirm
• document-row.tsx — table row with domain badge + freshness indicator
• document-detail.tsx — slide-out panel with full metadata + content
• domain-filter.tsx — toggle row for all 6 domains (sales / engineering / product / customer / legal / other)
• freshness-indicator.tsx — live (green) vs snapshot (orange) badge
• query-playground.tsx — text input + domain filter → ranked results with score bars

Lib / Types

• src/lib/context-lake.ts — adapter interface, FileUploadAdapter, 4 stub adapters, keyword ranker
• src/lib/seed-context-lake.ts — seed document loader (reads data/context-lake/*.json)
• src/types/context-lake.ts — ContextDocument, ContextSource, ContextChunk, ContextBinding, domain/freshness enums

Seed data (data/context-lake/)

Six example documents spanning all domains: ADR, security policy, PRD, sales playbook, on-call runbook, glossary.

Navigation

• Sidebar: "Context Lake" added to Build group (below Knowledge) with Layers icon

Follow-up plan

1. Wire SharePoint adapter to MS Graph API
2. Implement Confluence / Notion / GitHub adapters
3. Replace keyword ranker with Azure AI Search or pgvector embedding search
4. Implement binding runtime: inject ranked chunks into agent system prompt before each run
5. Add freshness refresh scheduler (cron per source)
6. ACL enforcement (orgRead / teamWrite / ownerOnly checks at query time)

Acceptance checklist

• npm run lint — no errors in context-lake files (pre-existing warnings elsewhere unchanged)
• npm run build — compiled successfully, /context-lake builds as static page
• Query Playground works against seed data without AI Search credentials
• Sidebar nav entry present below Knowledge

🤖 Generated with Claude Code