Multi-Agent Self-Correcting Code Review Pipeline

A Generator → Critic → Fixer loop, orchestrated with LangGraph, where agents don't just produce code — they check each other's work, execute it against real tests, and iterate until it's actually correct or a human needs to step in.

Most "multi-agent" demos are three LLMs chatting and writing a report. This project targets the parts of production AI engineering that get skipped: validation, observability, and governance — not just another chat interface on top of a model.

How it works

   ┌────────────┐      ┌──────────┐      ┌───────────────────────┐
   │ Generator  │ ───▶ │  Critic  │ ───▶ │ approve? ──▶ END        │
   │ (writes    │      │ (runs    │      │ revise? ──▶ Fixer ──┐   │
   │  code)     │      │  tests + │      │ out of iters? ──▶   │   │
   └────────────┘      │  reviews)│      │   escalate ──▶ END  │   │
                        └──────────┘ ◀────────────────────────┘   │
                              ▲                                    │
                              └────────────────────────────────────┘

1. Generator writes a first-pass Python solution from a plain-English task.
2. Critic is the actual quality gate:
   • Runs the code against real test cases in an isolated subprocess (with a hard timeout) — correctness is measured, not an LLM opinion.
   • Asks the LLM to review for security/style issues (hardcoded secrets, eval() on untrusted input, unbounded recursion, etc.).
   • Renders a verdict: approve or revise.
3. Fixer rewrites the code using the Critic's specific failing tests and issues — not a generic "try again."
4. The loop repeats until the code passes, or max_iterations is hit, at which point the run is escalated to a human queue instead of silently shipping broken code or looping forever.

Every iteration is logged as a structured JSONL event for basic observability (pass rate, verdict, issue count over time) — the kind of thing you'd point Grafana or a notebook at in a real system.

Why this project

Built to demonstrate the "Frontier Engineering" skills that plain chatbot projects don't touch:

JD requirement	Where it lives in this repo
Build and orchestrate multi-agent AI systems	app/graph.py — LangGraph state machine, 3 cooperating agents
Validate AI-generated code for quality and correctness	app/executor.py — sandboxed test execution, not just LLM opinion
Integrate AI validation/testing into CI/CD-style gates	Critic node blocks on failing tests + blocker-severity issues
Observability, drift monitoring, reliability controls	app/logger.py, GET /metrics
Governance, escalation, human-AI collaboration patterns	Max-iteration escalation path in app/graph.py

Project structure

multi-agent-code-review/
├── app/
│   ├── state.py       # Shared TypedDict state passed through the graph
│   ├── executor.py     # Sandboxed subprocess test runner
│   ├── llm_client.py    # Thin Anthropic API wrapper
│   ├── agents.py        # Generator, Critic, Fixer node functions
│   ├── graph.py          # LangGraph orchestration + escalation routing
│   ├── logger.py          # JSONL observability logging
│   └── main.py              # FastAPI app (/review, /metrics, /health)
├── tests/
│   └── test_executor.py       # Sandbox tests (no API key required)
├── examples/
│   └── example_request.json     # Sample payload for /review
├── requirements.txt
└── .env.example

Setup

git clone <your-repo-url>
cd multi-agent-code-review
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
cp .env.example .env   # then add your ANTHROPIC_API_KEY

Run it

uvicorn app.main:app --reload

Then, in another terminal:

curl -X POST http://localhost:8000/review \
  -H "Content-Type: application/json" \
  -d @examples/example_request.json

Response includes the final code, how many iterations it took, the full per-iteration history (test results + review issues), and whether it was approved or escalated.

Check aggregate stats:

curl http://localhost:8000/metrics

Run the tests

The executor tests don't need an API key — they just verify the sandbox correctly passes good code, fails buggy code, and times out infinite loops:

pytest tests/

Design decisions worth knowing for an interview

• Execution, not vibes. The Critic doesn't ask an LLM "is this code correct?" — it runs the code against real test cases in a subprocess and only asks the LLM to review dimensions tests can't measure (security, style). This is the difference between an "AI opinion" and a quality gate.
• Bounded loops. max_iterations guarantees the system terminates and explicitly hands off to a human instead of pretending to be fully autonomous — this is the human-in-the-loop / governance pattern.
• Sandboxing. Generated code runs in a subprocess with a timeout, never in-process — a naive exec() in the API process would let generated code (or an infinite loop) take down the server.
• Narrow prompts per agent. Generator, Critic, and Fixer each have a tightly scoped system prompt instead of one giant "write good code" prompt — this is what makes the loop actually converge instead of repeating the same mistake.

Possible extensions

• Swap the LLM provider by editing app/llm_client.py only.
• Add a SecurityScanner node that runs bandit as another automated gate alongside the LLM review.
• Persist run history to Postgres instead of JSONL for a real dashboard.
• Add a small React frontend that polls /metrics for a live drift chart.