DocExtract AI

Ship-gate first: versioned eval corpus, offline replay, and a variance-calibrated CI gate, followed by two-pass extraction and agentic RAG.

Reviewer path (about two minutes, zero API keys): follow DEMO.md and run the local fixture-backed demo (DEMO_MODE=true streamlit run frontend/app.py). The hosted Streamlit URL is intentionally omitted until anonymous access is verified. Static preview and trace visualizer available in site/ and frontend/pages/agent_trace.py.

Static marketing front door: site/ (HTML + CSS, no build step).

Eval gate {#eval-gate}

Prompts are code. DocExtract treats extraction quality as a merge-blocking CI signal, not a post-hoc dashboard number.

Signal	What runs	When
Offline replay (badge driver)	scripts/eval_offline_replay.py on 28 committed fixtures	Every eval-gated PR; zero API cost
Variance-calibrated gate	scripts/eval_gate.py vs autoresearch/baseline.json	PRs touching prompts / extraction services
Paid live eval	Promptfoo, RAGAS, LLM-judge	Only when ANTHROPIC_API_KEY is present in CI; skipped otherwise
Drift cron	Golden set vs production prompt version	Daily 13:23 UTC

Eval gate proof (red blocked PR): #32 — intentional regression (keep open / expect red). See also docs/eval-methodology.md.

Metric	Value	Basis
Extraction accuracy (field-level, critical fields weighted 2×)	95.5%	Always-on CI offline replay of 28 deterministic fixtures (scripts/eval_offline_replay.py); not a paid live grade
Test suite	80% CI coverage gate	--cov-fail-under=80; the changing collected-test total is intentionally omitted (portfolio-metrics.yaml)
Authoring corpus	202 cases (151 golden + 51 adversarial)	evals/golden_set.jsonl + evals/adversarial_set.jsonl (line counts); separate from the 28-fixture offline replay
Cost / latency	See cost-model.md	Modeled only until a funded scripts/benchmark.py run is committed

CI-replayed eval breakdown by document type (from committed autoresearch/baseline.json)

Document type	Score	Cases
invoice	0.9731	13
receipt	0.9107	4
purchase_order	0.9762	3
bank_statement	0.9581	4
medical_record	0.9923	3
identity_document	0.8139	1

Overall: 0.955 across 28 cases, replayed on every eval-gated PR at zero API cost.

More: CASE_STUDY.md · docs/eval-methodology.md · evals/

What this does

FastAPI document intelligence: upload PDFs and images, classify with cost-aware routing, extract structured fields via a two-pass Claude pipeline, embed into pgvector, and query with agentic RAG (ReAct loop with streaming SSE reasoning).

Upload → ARQ worker → classify → extract → validate → embed → search / agentic RAG
         ↑
    Optional trace exporters        Offline eval replay (CI only, not on request path)

Why this is interesting (engineering)

• Eval-gated CI: eval-gate.yml offline job replays 28-case deterministic baseline at zero API cost; PRs touching prompts or extraction services must pass before merge
• FastAPI & Strict Type Safety: End-to-end Pydantic V2 validation contracts, typed error domains, and deterministic schema enforcement preventing malformed extraction persistence
• PostgreSQL (pgvector) & ARQ Queue: Document chunk embeddings indexed via pgvector HNSW vectors, decoupled background document processing via Redis and ARQ worker queue
• Agentic RAG: ReAct Think → Act → Observe over hybrid retrieval tools; primary search story in API and Streamlit (agentic_rag.py, agent_trace.py)
• Cost-aware model routing: Haiku for classification, Sonnet for extraction; prompt caching on system prompts; circuit breaker with Haiku fallback
• Independent judge: Gemini grades extractions to reduce self-grading bias (ADR-0018)
• Optional observability: Langfuse integration, LangSmith, and OpenTelemetry exporters are available when configured (app/observability.py)
• Prompt-injection defense: runtime fence + scan + output sanitization (ADR-0020)

Architecture

graph LR
  A[Client / Streamlit] -->|POST /documents| B[FastAPI]
  B -->|enqueue| C[ARQ Worker]
  C -->|classify + extract| D{Model Router}
  D -->|primary| E[Claude Sonnet]
  D -->|fallback| F[Claude Haiku]
  E --> G[(pgvector)]
  G -->|search| H[Agentic RAG]
  H --> A
  C -->|Langfuse| I[Traces]
  B -->|SSE /jobs/events| A

Loading

Demo

Run the fixture-backed demo locally with no API key:

DEMO_MODE=true streamlit run frontend/app.py

Progress streams over SSE: /jobs/{id}/events (extraction stages) and /agent-search/stream (agentic retrieval reasoning).

Install

git clone https://github.com/ChunkyTortoise/docextract.git
cd docextract
cp .env.example .env  # Add ANTHROPIC_API_KEY + GEMINI_API_KEY
docker compose up -d
open http://localhost:8501  # Streamlit UI

Services: API :8000 (/docs for Swagger) | Frontend :8501 | PostgreSQL :5432 | Redis :6379

Tests

pytest tests/ --collect-only -q       # Discover the current suite; count is not a portfolio claim
python scripts/eval_offline_replay.py --floor 0.85   # Always-on CI offline replay (badge driver)
python scripts/run_eval_ci.py --ci                    # Wrapper; same 28-case deterministic path
make eval                             # Optional paid live eval; requires configured credentials

Architecture Decisions

20 ADRs at docs/adr/. Key decisions:

ADR	Decision
ADR-0003	Two-pass Claude extraction with confidence gating
ADR-0006	Circuit breaker model fallback chain
ADR-0015	Anthropic prompt caching for eval cost reduction
ADR-0018	Gemini as independent judge
ADR-0019	TF-IDF reranker + agentic self-reflection loop

Scope notes (honest): GraphRAG hybrid retrieval is opt-in (GRAPH_RETRIEVAL_ENABLED=false by default): regex entity graph, file-backed. Semantic cache (ADR-0017) is implemented but feature-flagged off and not wired into the extraction hot path. Langfuse, LangSmith, and OpenTelemetry integrations require configuration and are not presented as verified live telemetry.

More: DEMO.md | docs/cost-model.md | site/

License

MIT