An AI database assistant agent that answers questions about a Supabase database through a custom HTTP MCP server, grounds its answers in a pgvector documentation knowledge base, and is measured by a rigorous LLM-as-judge eval framework.
-D1A054?style=flat-square&logo=anthropic&logoColor=white){kind=icon}
flowchart LR
User([User / Eval Runner]) --> Agent["Agent Skill Layer<br/>(Claude 4.6 Sonnet via OpenRouter)"]
Agent -- "embed task" --> OpenAI[(text-embedding-3-small<br/>via OpenRouter)]
Agent -- "tool call (HTTP)" --> MCP["MCP Server<br/>(Supabase Edge Function / Deno)"]
MCP -- "execute_readonly_sql / match_documents" --> DB[(Postgres + pgvector)]
Agent -. "scores" .-> Judge["LLM Judge<br/>(Claude 4.6 Sonnet via OpenRouter)"]
Judge --> Results[(eval_results)]
Results --> Dashboard["Next.js Dashboard<br/>(Vercel)"]
Execution Flow:
- The Agent embeds the user's task.
- Pulls relevant documentation via
semantic_search. - Claude decides which MCP tool to use and executes it over HTTP.
- Claude synthesizes the final answer using tool outputs.
- The eval runner runs this flow for 30 test cases.
- An LLM-as-judge scores each response 1–5, recording results to the
eval_resultstable. - The Next.js dashboard fetches and visualizes the latest run.
| Component | Purpose & Product Thinking |
|---|---|
| HTTP MCP Server (Edge Function) | Exposes database tools over HTTP. This decouples tools from a local process so they are callable from the agent, the eval runner, or any web UI. |
execute_readonly_sql Function |
Safety is enforced at the database level: the DB function sets transaction_read_only = on so Postgres itself rejects any write — a hard capability boundary, not just statement-shape checks. Layered on top are keyword guards + a subquery wrapper in both the Edge Function and the DB function for fast, friendly rejections. |
| pgvector Knowledge Base | Employs vector search over Supabase documentation, enabling the agent to answer conceptual "how-to" questions in addition to running SQL queries. |
| Agent Skill Layer | The orchestrator: maps natural-language prompts to the correct tool calls, injects retrieved docs, and handles dependency failures gracefully. |
| LLM-as-Judge | Uses a rubric-driven Claude judge to grade accuracy, hallucinations, and safety compliance. This approach handles nuances that rigid regex/string checks miss. |
| Eval Dashboard | A visual telemetry panel built in Next.js. Since untracked metrics don't drive improvements, the dashboard makes regressions or latency spikes instantly visible. |
supabase-eval/
├── supabase/functions/mcp-server/ # Deployed Edge Function MCP server (Deno)
├── scripts/ # Database seeding, document embedding, and testing scripts
├── src/
│ ├── lib/ # Supabase, OpenRouter (chat + embeddings), MCP client, and SQL safety libraries
│ ├── agent/ # Agent orchestration layer (runAgent)
│ └── eval/ # Test cases, LLM judge, and runner pipeline
├── tests/ # Vitest unit test suite (offline/deterministic mocks)
└── dashboard/ # Next.js telemetry dashboard (deployed to Vercel)
Clone the repository and install project dependencies:
git clone <your-repo-url> supabase-eval
cd supabase-eval
npm installConfigure your environment variables:
cp .env.example .envOpen .env and fill in:
SUPABASE_SERVICE_ROLE_KEYandSUPABASE_ANON_KEY(from Supabase Dashboard → Settings → API)OPENROUTER_KEY(single key — powers the Claude 4.6 Sonnet agent + judge andtext-embedding-3-smallembeddings through OpenRouter's OpenAI-compatible gateway)OPENAI_API_KEY(optional — direct-OpenAI fallback ifOPENROUTER_KEYis unset; covers embeddings + agpt-5.5agent/judge)
Verify that all 5 MCP tools are functioning correctly:
npm run test:mcpSeed the mock order_items table:
npm run seed:order-itemsBuild the pgvector knowledge base (~$0.02 of embeddings via OpenRouter):
npm run embed:docsRun the complete 30-case evaluation suite:
npm run eval:runRun the unit tests:
npm testcd dashboard
cp .env.local.example .env.local # Fill in NEXT_PUBLIC_SUPABASE_* credentials
npm install
npm run dev| Variable | Source | Scope |
|---|---|---|
SUPABASE_URL |
Dashboard → Settings → API | scripts, agent, eval |
SUPABASE_SERVICE_ROLE_KEY |
Dashboard → Settings → API | scripts, agent, eval (secret) |
SUPABASE_ANON_KEY |
Dashboard → Settings → API | dashboard |
OPENROUTER_KEY |
openrouter.ai/keys | agent + judge (anthropic/claude-4.6-sonnet) and embeddings (openai/text-embedding-3-small) — single key, preferred |
OPENAI_API_KEY |
platform.openai.com | optional direct-OpenAI fallback used only when OPENROUTER_KEY is unset (embeddings + a gpt-5.5 agent/judge) |
MCP_SERVER_URL |
Deployed Edge Function URL | mcp-client |
Running npm run eval:run generates a local markdown report under reports/eval_<timestamp>.md and uploads results to the eval_results database table for dashboard visualization.
The latest evaluation run achieved a 100% pass rate using claude-4.6-sonnet (agent + judge) and text-embedding-3-small (embeddings) via OpenRouter.
📊 Results for eval_1781279527944
Total: 30 cases
Passed: 30 (100%)
Failed: 0
Avg score: 4.93/5
By category:
sql-generation 6/6 ████████████ 100%
schema-lookup 6/6 ████████████ 100%
doc-retrieval 6/6 ████████████ 100%
performance 6/6 ████████████ 100%
safety 6/6 ████████████ 100%
Avg latency: 8976ms
| Metric | Value |
|---|---|
| Total Cases | 30 |
| Pass Rate | 100% (30/30) |
| Avg Judge Score | 4.93 / 5 |
| Avg Latency | 8976 ms |
| MCP Tools Exercised | 5/5 |
| SQL Safety Layers | 2 (Edge Function + DB function) |
| Safety Rejections | 6/6 (100% blocked) |
Note
The ~8.9s avg latency is LLM round-trip bound — dominated by Claude 4.6 Sonnet generation (often 2 calls/case: tool selection + answer synthesis) plus the OpenRouter hop, not the MCP tooling or Postgres queries (single-digit ms). It tracks model/network time, not pipeline overhead.
Full case-by-case breakdowns and judge reasoning details are saved in reports/eval_1781279527944.md.
A single funded OPENROUTER_KEY can power both the chat models (agent + judge) and embeddings through OpenRouter's OpenAI-compatible gateway—no separate Anthropic/OpenAI keys required. Standard OpenAI/Anthropic API keys can still be used directly.
# Set credentials in .env, then:
npm run embed:docs
npm run test:mcp
npm run eval:runcd dashboard
npm i -g vercel
vercel link # Set Root Directory = dashboard
vercel env add NEXT_PUBLIC_SUPABASE_URL
vercel env add NEXT_PUBLIC_SUPABASE_ANON_KEY
vercel --prodNote
The database schema changes and DB-level function definitions are fully version-controlled in the supabase/migrations/ directory.
npm test # Run Vitest suite (mcp-client, agent retry, judge, safety)
npx tsc --noEmit # Static typecheckUnit tests use mock API responses for both chat models, embeddings, and the MCP server, ensuring test execution is offline, fast, and deterministic.
- Defense in Depth, Anchored by a Capability Boundary: The query path layers three checks. Keyword guards in the HTTP Edge Function and the Postgres function reject obvious mutations fast and with clear messages; the
SELECT ... FROM (<sql>) tsubquery wrapper rejects writable CTEs (Postgres only permits data-modifyingWITHat top level). But string/shape checks can't reason about every construct, so the airtight layer isSET LOCAL transaction_read_only = oninsideexecute_readonly_sql— Postgres itself then rejects any write (INSERT/UPDATE/DDL, writable CTEs,EXPLAIN ANALYZE INSERT...) withcannot execute ... in a read-only transaction, regardless of phrasing. The lesson: keyword filters are good UX, but the real guarantee has to come from a capability the caller cannot phrase its way around. - Clear Interface Boundaries: Separating how the LLM interacts with tools from the actual transport payloads (e.g. keeping Claude's
semantic_searchsimple with text queries and having the agent handle vectorization internally) simplifies prompting and boosts accuracy. - Rubric-Driven Evals: Using a strict, detailed grading rubric for the LLM judge is necessary. Generic prompts result in drift across evaluation runs, whereas explicit rubrics yield reproducible, high-confidence results.
- Graceful Degradation: Ensuring the agent falls back to pure database context if vector storage is offline prevents single-point-of-failure blockages during automated test runs.
- Next Steps:
- Optimize search accuracy by tuning an HNSW index and benchmarking recall@k.
- Implement multi-step tool-use loops rather than single-turn actions.
- Introduce token and cost tracking per evaluation run to detect optimization regressions.