multicooker

Run several LLM agents (claude, codex, agy, grok) on the same task in parallel — each in its own docker container with its own subscription auth — then have other LLM agents read the outputs blind (under A / B / C labels), score them against your rubric, and write reviews.

You get a leaderboard.md plus a corpus of N divergent solutions to one brief. No API bills: it goes through your Claude Pro / ChatGPT Plus / Gemini Advanced / SuperGrok subscriptions.

There's also a composite triad flavor — Claude as the lead engineer with Codex and Grok in the same container as in-cell reviewers it consults itself (build → review → integrate, multi-model review inside one build). Good as the chef/lead, not a blind competitor. See docs/add-flavor.md.

«multicooker»: one task, several dishes cook in parallel in their own pots; you compare what came out of each.

Why

When a task is underspecified — design, copy, refactoring with architectural choice, code review — there is no single "correct" answer. Any model will fill in the gaps from the brief itself, and what it fills in is the interesting part. A single run through a single model doesn't show this; you only see one interpretation and assume it's "the answer".

multicooker gives you a corpus of divergent interpretations of the same brief in one shot. Useful when:

• You're picking between models for a recurring task (refactoring, design, doc writing, code review) and tired of deciding by vibes.
• You want to see where a brief is underspecified — disagreement between models highlights exactly those spots.
• You're doing design or copy work and want three takes from three different "heads" instead of one.
• You're studying how much models agree with each other on open tasks (often: not much).

How it works (one cook end-to-end)

                     ┌─────────────────────────────┐
                     │      cooks/260516-task/     │
                     │  BRIEF.md  JUDGE_BRIEF.md   │
                     │  brief.yaml      raw/       │
                     └──────────────┬──────────────┘
                                    │ multicooker cook
        ┌───────────────────────────┼───────────────────────────┐
        ▼                           ▼                           ▼
 ┌─────────────┐             ┌─────────────┐             ┌─────────────┐
 │  claude     │             │  codex      │             │  agy        │
 │  container  │   (parallel)│  container  │  (parallel) │  container  │
 │  net-A      │             │  net-B      │             │  net-C      │
 │  /work/...  │             │  /work/...  │             │  /work/...  │
 └──────┬──────┘             └──────┬──────┘             └──────┬──────┘
        │ out/                      │ out/                      │ out/
        └───────────────────────────┼───────────────────────────┘
                                    ▼
                       ┌─────────────────────────┐
                       │   anonymize → A/B/C     │
                       │   mapping stays on host │
                       └──────────────┬──────────┘
                                      │ multicooker judge
                ┌─────────────────────┼─────────────────────┐
                ▼                                           ▼
       ┌─────────────────┐                         ┌─────────────────┐
       │  judge-1        │                         │  judge-2        │
       │  (claude/codex/ │  scores everyone except │  (different     │
       │   agy)          │  its own flavor         │   flavor)       │
       └────────┬────────┘                         └────────┬────────┘
                │ scores.json + review.md                   │
                └─────────────────────┬─────────────────────┘
                                      ▼ multicooker report
                            ┌──────────────────┐
                            │ leaderboard.md   │
                            └──────────────────┘

The key properties:

• Isolation. Each participant runs in its own container on its own bridge network — can't see the other participants, the judge brief, or the A↔flavor mapping.
• Parallelism. All participants start at the same time. One being rate-limited doesn't block the others.
• Anonymization. Judges only see A / B / C with no model names. The mapping lives only on the host.
• Anti-self-judge. A judge never scores submissions from its own flavor — claude doesn't judge claude's output.
• No API keys. Subscription credentials (Claude Pro / ChatGPT Plus / Gemini Advanced) are passed into containers via bind-mount or named volume, read-only. See docs/auth.md.

Install

git clone https://github.com/faeton/multicooker
cd multicooker
pip install -e .

Requirements:

• macOS or Linux host with a running docker daemon. On macOS, OrbStack is the recommended runtime — noticeably faster startup, lower idle CPU, and friendlier resource handling than Docker Desktop. Docker Desktop and colima also work.
• Python 3.10+.
• At least one of these CLIs installed and logged in: claude (claude /login), codex (codex to log in), agy (reuses your ~/.gemini/oauth_creds.json OAuth), grok (grok login). Only the flavors you actually want to run.

Want to try the pipeline without subscription creds? There's a dummy flavor — see examples/hello-task.

Quick start: let an agent scaffold the cook (10 seconds)

The fastest way to use multicooker is to fire up an LLM agent inside the repo and let it scaffold and run the cook for you. The repo ships with a CLAUDE.md (and an AGENTS.md symlink for codex / agy) that already explains the project, the shape of a cook, and the rule that the rubric stays in sync between brief.yaml and JUDGE_BRIEF.md. Any agent reading it can do the boring part for you.

git clone https://github.com/faeton/multicooker && cd multicooker
pip install -e .

claude        # or: codex, or: agy — they all read AGENTS.md

Then describe what you want in plain language:

"Set up a cook called landing-redesign. Compare claude / codex / agy on a single-file HTML hero for [product]. Judge on visual-hierarchy, typography, color-discipline, content-fit, polish. References are at ~/work/brand/notes.md and ~/work/brand/voice.md. Then run cook + judge + report."

The agent reads CLAUDE.md and examples/design-landing/ as templates, drafts your BRIEF.md / JUDGE_BRIEF.md / brief.yaml, copies the refs into raw/, kicks off multicooker cook, waits for it to finish, then runs judge and report. You read the leaderboard.

Iterating is the same conversation:

"Feedback for everyone: too much whitespace, push for denser layout. Specifically for claude: keep the color palette but tighten the type scale. Refine."

Or — start a new cook reusing the same reference material (different task, same brand assets):

"Same refs as the previous cook. New brief: a 3-frame onboarding sequence instead of a single landing. Judge the same dimensions plus story-clarity. Run it."

This is the canonical workflow. The manual flow below is useful for understanding the moving parts, but it's not how you'd typically use the tool day-to-day.

Manual flow (5 minutes, full control)

# 1. Preflight — docker, compose, creds for each flavor
multicooker doctor

# 2. Scaffold (name is auto-prefixed with today's date → 260509-my-task)
multicooker new my-task

# 3. Describe the task
cd cooks/260509-my-task
$EDITOR BRIEF.md          # what participants must do
$EDITOR JUDGE_BRIEF.md    # how judges will score
$EDITOR brief.yaml        # participants, judges, timeout, rubric
cp ~/some-reference.* raw/   # reference materials (mounted RO)

# 4. Cook — all participants in parallel, each in its own container
multicooker cook 260509-my-task

# 5. Judge — blind: judges only see A/B/C labels
multicooker judge 260509-my-task

# 6. Summary → leaderboard.md
multicooker report 260509-my-task
cat cooks/260509-my-task/leaderboard.md

Examples

The repo includes two ready-to-run examples plus reusable cook shapes for common task types:

• examples/hello-task — sanitized smoke test on the dummy flavor, no LLM creds required. ~10 seconds from start to leaderboard. Run it once to see the shape of a cook on the simplest possible task.

• examples/design-landing — a real design task: each model designs its own landing page for multicooker. Three HTML files you then compare side-by-side in a browser. More on this below.

• examples/technical-proposal — abstract RFC / architecture proposal. Use when the desired output is a clear build recommendation with alternatives, staged execution, and risk honesty.

• examples/code-review-audit — source-reading review, known-issue root cause, and downstream refine guidance. Use before a risky patch or rewrite cook.

• examples/implementation-spike — narrow working prototype with README.md, STATUS.md, source, and runnable evidence. Use after the target has been scoped.

• examples/multi-concept-ui — three divergent self-contained UI concepts for one workflow. Use when you want interaction-model exploration, not one polished landing page.

Use case: design and creative tasks

The most illustrative use case is tasks where there's no right answer but there are quality criteria. Design, copy, naming, architectural essays. Here models diverge not because one is buggy but because they hold different "aesthetic beliefs", and comparison becomes substantive.

examples/design-landing is a working template for this kind of cook. Brief: "design a landing page for multicooker, single-file HTML, no build step". When you open the three index.html files side by side, you typically see:

• Palette. One model commits to strict monochrome; another scatters six accent colors and doesn't quite know what to do with them; another defaults to dark mode.
• Typography. Someone reaches for the system stack; someone pulls Inter from Google Fonts; someone leaves the default serif — and the hero blocks read completely differently as a result.
• Density. One packs features into a three-column grid with small text; another goes for one big half-screen block.
• Content fit. Someone quotes raw/product.md verbatim; someone reimagines the product according to their own theories of what a "proper landing" should be (the content-fit dimension in the rubric exists to catch this).
• Polish. Hover states, spacing rhythm, code-block styling, footer treatment — small decisions that separate "draft" from "shipped".

The rubric in examples/design-landing/JUDGE_BRIEF.md scores on visual-hierarchy / typography / color-discipline / content-fit / polish. Two judges of different flavors score blindly — and they often disagree with each other. That's a useful signal: on design tasks, judge disagreement means there's no "winner on points", just three different directions, and you pick with your eyes.

# Run the design example (requires claude/codex/agy logins; grok optional)
multicooker new landing --participants claude,codex,agy,grok
TASK=$(basename "$(ls -d cooks/*-landing | tail -1)")
cp examples/design-landing/{BRIEF.md,JUDGE_BRIEF.md,brief.yaml} cooks/$TASK/
cp examples/design-landing/raw/* cooks/$TASK/raw/

multicooker cook   $TASK
multicooker judge  $TASK
multicooker report $TASK

# Open all three variants side by side, plus the leaderboard
open cooks/$TASK/out/*/index.html
cat  cooks/$TASK/leaderboard.md

This template adapts to any design task — SVG logo, README header, email template, dashboard mockup. You only need to rewrite BRIEF.md for your output and tweak the rubric dimensions (brand-fit, accessibility, density, motion-restraint — anything, as long as the names match between brief.yaml and JUDGE_BRIEF.md). See examples/design-landing/README.md for the full adaptation guide.

Iterating on a result

$EDITOR cooks/260509-my-task/FEEDBACK.md          # general feedback
$EDITOR cooks/260509-my-task/FEEDBACK_claude.md   # per-participant (optional)

multicooker refine 260509-my-task    # round N+1 on top of previous out/
multicooker judge  260509-my-task
multicooker report 260509-my-task

Previous rounds are preserved in rounds/<N>/ — nothing is lost. multicooker diff <task> shows what moved at file level between two rounds — useful for spotting which model actually took the feedback to heart vs which one just rephrased the previous answer.

Multiple participants of the same flavor / different models

multicooker new comparison \
  --participants claude-a=claude,claude-b=claude,codex,agy

Per-participant model selection lives in brief.yaml:

participants:
  - { name: claude-sonnet, flavor: claude, model: claude-sonnet-5 }
  - { name: claude-opus,   flavor: claude, model: claude-opus-5 }
  - { name: codex }

Useful for, e.g., pitting sonnet against opus on the same task — two horses of the same flavor under different names, with different models.

Model aliases: a model is a participant

You don't have to hand-edit brief.yaml for that. A model alias can stand wherever a flavor can, in new, add-participant, and chef:

multicooker new modelcmp --participants fable,opus,sonnet,codex,grok
multicooker add-participant modelcmp haiku
multicooker chef modelcmp --chef fable

spec	expands to
fable	{ name: fable, flavor: claude, model: claude-fable-5 }
opus	{ name: opus, flavor: claude, model: claude-opus-5 }
sonnet	{ name: sonnet, flavor: claude, model: claude-sonnet-5 }
haiku	{ name: haiku, flavor: claude, model: claude-haiku-4-5 }
arm-a=opus	same, renamed to arm-a
x=claude:claude-opus-4-8	any id an alias doesn't cover

Bare flavors are untouched — claude still means "claude flavor, CLI default". Aliases live in new_cook.MODEL_ALIASES; the FLAVOR:MODEL form is passed through verbatim, so check it against multicooker models <flavor> first.

Two things to know before reading a model bake-off's leaderboard:

• Judges must not be the participants' flavor. With three claude arms, a claude judge is a self-judge; under judging.policy: require_distinct_flavor its scores are dropped and those arms can end up scored by fewer judges than the rest. doctor / cook now warn when coverage comes out uneven. The three claude arms are always mutually symmetric (same flavor), so a head-to-head between them is sound even when a codex or grok reference arm sits at a different judge count.
• Same-flavor arms share one subscription. Three claude cells run in parallel against a single Claude account; heavy briefs can hit rate limits. Raise timeout_s, and on a limit failure use multicooker resume <cook> once the window resets rather than cancelling the cook.

Valid model: values are per-CLI, not provider API ids

The model: string is forwarded verbatim to the participant CLI (MULTICOOKER_MODEL → claude --model, codex -c model=, agy --model, grok -m). It must be a model id that CLI accepts, which is often not the provider's public API id — and the CLI that matters is the one inside the base image, not the one on your host. Ask the container, never the host:

multicooker models grok        # runs `grok models` inside mc-base-grok
multicooker models             # every flavor with a base template

Rules of thumb per flavor:

• claude — Anthropic ids or aliases (claude-opus-5, opus, sonnet). An older in-image CLI won't know newer ids.
• agy — agy models. Older builds print labels only ("Gemini 3.1 Pro (High)"); newer ones print id<TAB>label. Use what that image prints.
• grok — grok models. Ids are the grok CLI's own aliases and change across CLI releases (grok-build/grok-build-latest on 0.1.x; grok-4.5/grok-4.6 on 1.0.x), not provider API ids. A rejected id fails fast: Invalid params: "unknown model id", cell exits 1.
• codex — the host's ~/.codex/config.toml is not mounted (only auth.json is), so a cell ignores your host model = setting entirely.

If a cell dies in a few seconds with an exit 1, check its logs/<name>/<flavor>.stdout.log for an "unknown model id" line before assuming an auth problem. Fix the model: in brief.yaml and re-run just that cell with multicooker resume <cook> (re-runs only non-succeeded cells) — no need to cancel the whole cook.

Omitting model: is not "the latest model"

With no model:, each CLI picks its own default — resolved by the in-image CLI against your subscription, at run time. That default is invisible in brief.yaml, differs from your host, and drifts as the provider moves its default. Base images are built once and never auto-rebuilt (ensure_built skips anything already present), so an untended install keeps running last quarter's default indefinitely:

multicooker doctor --images        # in-image CLI vs host CLI, + image age
multicooker build-base --force     # rebuild them all

Pin model: explicitly on anything you intend to compare, publish, or reproduce. Whatever actually ran is recorded after the fact in cooks/<task>/summary.json (usage_totals.models) — check it before trusting a leaderboard that compared "the defaults".

Two gaps in that after-the-fact record, so don't treat it as complete:

• grok cells report nothing. There's no usage-ledger mount or parser for grok (_usage_volumes returns []), so a grok participant never contributes a model name. Absence there means "not collected", not "didn't run".
• triad reports only the lead. MULTICOOKER_MODEL reaches the claude lead; the in-cell codex and grok reviewers always run their own defaults, and model: on a triad participant does not change them.

Isolation and security (short version)

• One docker compose project per cook (mc-<task>).
• Each participant is in its own container on its own bridge network (net-participant-<name>); they don't see each other via DNS/IP.
• Subscription creds are snapshotted into cooks/<task>/.auth/<flavor>/ (mode 0600, .gitignore'd) and bind-mounted RO only into the corresponding container.
• After the cook, sealed out/ is anonymized into A/B/C/… before judging. The A↔flavor mapping lives on the host only, never goes into judge containers.
• Egress to the internet is open. Sandbox = container, not network. Threat model: docs/security.md.

The long version: HOWTO.md. Internals: docs/orchestration.md, docs/auth.md, docs/lifecycle.md. Driving multicooker from an external control plane: docs/control-plane-integration.md.

Commands

Command	What it does
multicooker new <task> [--participants ...]	Create a cook from templates.
multicooker doctor [<task>]	Preflight: docker, compose, creds, Dockerfiles, base images.
multicooker build-base [<flavor>...]	Build the shared base image (auto-built before the first cook).
multicooker cook <task>	Launch all participants in parallel.
multicooker refine <task>	Round N+1 with feedback on top of previous out.
multicooker chef <task>	Run one synthesis participant over sealed prior outputs.
multicooker judge <task>	Anonymized scoring by all judges.
multicooker rejudge <task>	Re-run judging (e.g. after editing JUDGE_BRIEF.md).
multicooker lint <task>	Check brief.yaml ↔ JUDGE_BRIEF.md consistency (rubric dimension coverage).
multicooker report <task>	Roll-up into leaderboard.md + summary.json + artifacts.json.
multicooker artifacts <task> [--json]	Build/show the visibility-tagged file manifest.
multicooker archive <task> [--include-operator] [--format tar]	Copy only publishable artifacts into a shareable dir/tarball.
multicooker status <task> [--json]	Current state from status.json (live; orchestrator-friendly).
multicooker cancel <task>	Stop a running cook, mark it cancelled, keep partial outputs.
multicooker resume <task> [--force]	Re-run only the retryable cells of the latest round.
multicooker tail <task> [actor]	Stream cell logs, prefixed by actor.
multicooker diff <task>	File-level diff between two refine rounds.
multicooker add-participant <task> NAME[=FLAVOR]	Add another participant to an existing cook.
multicooker clean [<task>] [--all]	compose down -v --rmi local + remove .auth/ (keeps results).
multicooker prune --older-than DAYS [--keep-results]	Delete cooks older than N days (docker teardown + remove dir). Destructive.

Machine-readable contract (for orchestrators)

Every cook writes, alongside the human leaderboard.md:

• status.json — live point-in-time snapshot (cook + per-cell state), updated atomically through the run. Read it via multicooker status.
• events.jsonl — append-only event log (cook.created, cell.started, cell.exited, seal.finished, judge.*, report.written, cook.cancel_requested/cook.cancelled, …).
• summary.json — canonical final result after report: ranking, per-judge breakdown, run metrics for the latest round, excluded self-flavor pairs.
• artifacts.json — a manifest of every cook file tagged with a visibility class: public (leaderboard, summary, participant out/, judge reviews), operator (logs, traces, results), secret (.auth/), host_only (judge mappings, sealed inbox). Unknown files default to operator, never public. multicooker archive uses these classes to emit a shareable copy that never contains credentials or judge mappings.

An external control plane should drive cooks off these files rather than parsing stdout or markdown. Full schemas, states, and the worker pattern: docs/control-plane-integration.md.

Python API

For embedding callers (e.g. a worker process), multicooker.api wraps the CLI:

from multicooker import CookRequest, run_cook, run_judge, run_report, get_status

req = CookRequest(name="260527-example", root="/abs/path/cooks", namespace="zuzoo")
status = run_cook(req)      # runs `cook` in a subprocess, returns CookStatus
status = run_judge(req)
result = run_report(req)    # returns CookResult parsed from summary.json
print(result.ranking)
# poll a running cook from elsewhere without launching anything:
live = get_status("260527-example", "/abs/path/cooks")

Each run_* launches the CLI as a subprocess (no shared threads/locks with the caller) and reads the result from the on-disk contract files. Prefer an absolute root.

Namespaces (multi-orchestrator)

Pass --namespace <ns> (or set MULTICOOKER_NAMESPACE) on cook/judge/ refine/resume and the compose project becomes mc-<ns>-<cook>, so two orchestrators can run cooks with the same name without colliding on containers, images, or networks. The resolved name is persisted in compose.yaml, so cancel and clean find the right project without needing the flag again.

Retention

clean only tears down docker artifacts and never deletes your results. multicooker prune --older-than DAYS is the destructive one: it tears down each stale cook's docker project and removes the directory (age from status.json.updated_at). --keep-results preserves summary.json + leaderboard.md; --dry-run lists without touching; --prune-images also reclaims dangling images + build cache.

Required outputs (optional)

Declare the deliverables a participant must produce, and a clean run that doesn't write them is recorded as artifact_missing (not ok) — honest status without aborting judging:

outputs:
  required:
    - { path: RESULT.md, kind: markdown }   # path is relative to out/

A required path is satisfied only by a real, non-empty file. multicooker lint (and doctor) check that every rubric dimension id in brief.yaml is mirrored in JUDGE_BRIEF.md; cook/refine refuse to run if it isn't.

Strict judge schema (optional)

By default report is tolerant — it repairs common judge-output variants (unwraps {"scores": …}, lifts flat dimensions). For automation that needs to trust the scores, opt into strict validation:

judging:
  strict_schema: true

A judge whose scores.json doesn't match the canonical {"<label>": {"dimensions": {"<dim>": int}}} shape is recorded as malformed_schema (in status.json, JUDGE_RESULT.json, summary.json, and the leaderboard's judge-run table) and its scores are not aggregated — no silent repair. Re-run just the judging with multicooker rejudge.

Status

v0.2. Tested on macOS with OrbStack and Docker Desktop. Linux should work; claude creds on darwin come from Keychain, on Linux from ~/.claude/.credentials.json.

Bugs → GitHub issues. Security: SECURITY.md.

License

MIT.