Reliable LLM Agents using Planner-Executor Pattern

A proof of concept for reliable agents using the planner-executor model. Like neurosymbolic AI, but without the neurosymbolic part.

Core principle: LLMs propose a structured pipeline definition; a deterministic engine executes it.

Natural Language → LLM Agent → Pipeline DAG → Deterministic Execution Engine → Reproducible Output

Running the same pipeline always returns the same result. This makes (usually) unreliable LLM agents reliable and consistent. Thanks to determinism, we can do cool things like provenance tracing for auditability, undo, and replay.

Determinism Guarantees

• Seal - Fingerprints engine version + primitive signatures
• Charter - Explicit rules for edge cases (null handling, float precision, string collation)

The screenshot below shows the CLI with GPT-4o. Surprisingly, benchmarks show the single-LLM strategy is the most performant and just as reliable as n-candidates or multi-agent specialization strategies.

Installation

pip install deterministic-agents

# With LLM support
pip install deterministic-agents[llm]

Don't forget to set your API keys in an .env file.

Quick Start

Interactive CLI

Talk to your data and watch it reshape in real-time:

dagents chat orders.csv

╭──────────────────────────────────────────────────────────╮
│  deterministic-agents v0.1.0                             │
│  Loaded: orders.csv (1,247 rows × 5 columns)             │
│  Columns: id, customer_id, amount, status, date          │
╰──────────────────────────────────────────────────────────╯

You: Show me only completed orders over $100

⏳ Generating pipeline...
✓ Pipeline compiled (2 nodes)
✓ Executed in 12ms

┌─────────────────────────────────────────────────────┐
│ Result (127 rows)                                   │
├──────────┬─────────────┬─────────┬──────────────────┤
│ id       │ customer_id │ amount  │ status           │
├──────────┼─────────────┼─────────┼──────────────────┤
│ ord_001  │ cust_42     │ 150.00  │ complete         │
│ ord_003  │ cust_42     │ 230.00  │ complete         │
└──────────┴─────────────┴─────────┴──────────────────┘

Pipeline: filter(status = 'complete') → filter(amount > 100)
Hash: 7a3f2c1e

CLI Commands

Command	Description
/history	Show transformation history
/pipeline	Show current pipeline JSON
/reset	Go back to original data
/undo	Undo last transformation
/export <file>	Export current result
/hash	Show current result hash
/seal	Show engine seal
/quit	Exit

Library API

from deterministic_agents import create_engine, pipeline, builtins

# Create engine with built-in primitives
engine = create_engine(primitives=builtins)

# Build pipeline fluently
p = (
    pipeline()
    .from_("orders")
    .filter("amount", ">", 100)
    .group_by("customer_id").sum("amount", as_="total")
    .sort("total", "desc")
    .build()
)

# Execute deterministically
result = engine.execute(engine.compile(p), {"orders": my_data})

# Same inputs = same hash (always)
print(engine.hash(result))  # e.g., "7a3f2c1e..."

Core Concepts

1. Primitives

Atomic, deterministic operations. Built-in primitives:

• filter - Filter rows by condition
• select - Select specific columns
• derive - Create new columns from expressions
• aggregate - Group by and aggregate (sum, avg, min, max, count)
• sort - Sort rows by column

2. Pipelines

DAGs of primitive nodes with typed inputs/outputs. Built with a fluent API:

p = (
    pipeline()
    .from_("data")
    .filter("status", "=", "active")
    .select("name", "email")
    .sort("name", "asc")
    .build()
)

3. Charter

Explicit rules for edge-case behavior (null handling, float precision, string collation):

from deterministic_agents import Charter

charter = Charter(
    null_handling="nulls_first",
    float_precision=10,
)
engine = create_engine(primitives=builtins, charter=charter)

4. Seal

Fingerprint of engine version + primitive signatures. Ensures reproducibility:

seal = engine.seal()
print(seal.engine_version)  # "0.1.0"
print(seal.primitives)  # {"filter": "abc123...", ...}

LLM Generation

One-Shot Strategy

Single LLM call → single pipeline:

from deterministic_agents.agents import create_generator

gen = create_generator(provider="openai", strategy="one_shot")

result = await gen.generate(
    intent="Filter orders over $100 and sort by date",
    context={"schema": ["id", "amount", "date"], "primitives": engine.available_primitives()}
)

compiled = engine.compile(result.pipeline)

N-Candidates Strategy

Generate multiple candidates, select the best:

gen = create_generator(
    provider="openai",
    strategy="n_candidates",
    n=3,
    selector="validation",  # or "llm_judge" or "hybrid"
)

Development

# Install uv (fast Python package manager)
curl -LsSf https://astral.sh/uv/install.sh | sh

# Install dependencies
uv sync --all-extras

# Run tests
uv run pytest

# Type check
uv run mypy src

# Lint
uv run ruff check src

# Run CLI
uv run dagents chat tests/fixtures/orders.csv

License

MIT