Navigates Every Enqueued Deliverable, Logs Effort
Deterministic bead processing with explicit outcome paths.
NEEDLE is a universal wrapper for headless coding CLI agents. It processes a shared bead queue in deterministic order, dispatching work to any headless CLI (Claude Code, OpenCode, Codex, Aider) and handling every outcome through an explicit, predefined path.
Prerequisites: git, tmux, and an agent CLI on your PATH — the flow below uses
Claude Code (claude). Prebuilt binaries are Linux x86_64;
everything else builds from source (see below).
# 1. Install needle, its transform helpers, and the bead-rs backend
curl -fsSL https://github.com/jedarden/NEEDLE/releases/latest/download/install.sh | bash
# 2. Initialize your repo with the bead-rs backend
cd <your-repo> && needle init --backend bead-rs
# 3. Create the bead store
bead init --prefix <name>
# 4. Create your first bead
bead create --title "Add a CONTRIBUTING.md" --priority 2
# 5. Verify system health
needle doctor
# 6. Run a worker
needle run --agent claude --identity alpha
# 7. Check status and attach to the session
needle status
tmux attach -t needle-claude-alpha
# 8. Verify completion
bead list --status closedHeads-up: the built-in claude adapter invokes claude -p … --dangerously-skip-permissions.
Unattended operation means no permission prompts; read needle config before pointing a
worker at a repository you care about.
Build from source (Rust 1.85+; NEEDLE pins its toolchain in rust-toolchain.toml):
cargo install --git https://github.com/jedarden/NEEDLE
cargo install --git https://github.com/jedarden/bead-rs --bin beadThe installer drops binaries in ~/.local/bin (override with NEEDLE_INSTALL_PATH). An existing bead at or above the release version is kept.
The installer automatically verifies SHA-256 checksums to ensure the downloaded binary has not been corrupted or tampered with. This verification is enabled by default for your protection.
Checksum verification safeguards against:
- Corrupted downloads that could crash or behave unpredictably
- Tampered binaries that could execute arbitrary malicious code
- Supply chain attacks where a malicious actor modifies releases
To see full security options and tradeoffs:
curl -fsSL https://github.com/jedarden/NEEDLE/releases/latest/download/install.sh | bash -s -- --help
⚠️ Warning: The installer supports an opt-out flag (--skip-checksum) for emergency recovery scenarios, but this is strongly discouraged except as a temporary workaround when checksums.txt is unavailable due to network/infrastructure issues. See the help output for full security details.
A worker starts, claims the next bead, dispatches to your chosen agent CLI, and loops. Multiple workers can run in parallel against the same workspace — coordination is handled by the shared bead queue (no central orchestrator).
See docs/examples/quickstart/ for a minimal workspace configuration and docs/examples/otel-collector/ for OpenTelemetry integration.
A bead is a work item — the unit of work NEEDLE processes. Think of it as a structured task ticket: a title, a body describing the deliverable and acceptance criteria, a status (open, in_progress, done), and optional metadata like priority and dependencies.
Beads live in a bead store — a pluggable backend managed by a bead CLI. The current primary backend is bead-rs, which uses a SQLite database (beads.db) plus a checkpoint directory (.beads/checkpoint/ with current.json, forensic.jsonl, and objects/). The legacy bead-forge backend uses issues.jsonl.
# Create a bead (bead-rs backend)
bead create --title "Add pagination to search results" --priority 2 --issue-type task
# List open beads
bead list --status open
# NEEDLE does the rest: claims, dispatches, and closes beads automaticallyNEEDLE workers read from this store, claim the next available bead atomically, dispatch it to your chosen agent CLI, and close it on success — then loop.
Existing agent orchestration tools are built for one of two shapes:
- Conversational frameworks (LangGraph, AutoGen, CrewAI) assume a chat loop with a human-in-the-loop or another LLM. They are bad at headless, long-running, cost-bounded work.
- Workflow engines (Temporal, Argo Workflows, Inngest) assume each step is deterministic code. They are bad at non-deterministic agent steps whose outcomes have to be classified and routed.
NEEDLE is the missing middle: a deterministic state machine that drives non-deterministic agents. Every outcome an agent can produce has an explicit handler. The agent's work is fuzzy; the orchestration around it is not.
NEEDLE is a state machine, not a script. Every bead transitions through a finite set of states, and every transition has a defined handler. There are no implicit fallbacks, no swallowed errors, no undefined paths.
If an outcome can happen, it has a handler.
If it doesn't have a handler, it cannot happen.
A single worker executes this loop indefinitely:
┌─────────────────────────────────────────────────────┐
│ │
│ ┌───────────┐ │
│ │ 🔍 SELECT │◄────────────────────────────────┐ │
│ └─────┬─────┘ │ │
│ │ │ │
│ ▼ │ │
│ ┌───────────┐ race lost ┌──────────┐ │ │
│ │ 🔒 CLAIM │──────────────►│ 🔁 RETRY │─────┘ │
│ └─────┬─────┘ └──────────┘ │
│ │ claimed │
│ ▼ │
│ ┌───────────┐ │
│ │ 📋 BUILD │ │
│ └─────┬─────┘ │
│ │ │
│ ▼ │
│ ┌───────────┐ │
│ │ 🚀 DISPATCH│ │
│ └─────┬─────┘ │
│ │ │
│ ▼ │
│ ┌───────────┐ │
│ │ ⏳ EXECUTE │ │
│ └─────┬─────┘ │
│ │ │
│ ▼ │
│ ┌───────────┐ │
│ │ 📊 OUTCOME │ │
│ └─────┬─────┘ │
│ │ │
│ ├── ✅ success ──► close bead ──────────────┘
│ ├── ❌ failure ──► log + release ────────────┘
│ ├── ⏰ timeout ──► release + defer ──────────┘
│ └── 💀 crash ────► release + alert ──────────┘
│ │
└─────────────────────────────────────────────────────┘
Query the bead queue for the next claimable bead in deterministic priority order. Selection is not random — given the same queue state, every worker computes the same ordering. Ties are broken by creation time (oldest first).
Attempt an atomic claim via the bead CLI (bead claim for bead-rs, bf claim for bead-forge). SQLite transaction isolation guarantees exactly one worker succeeds. If the claim fails (race lost), return to Step 1 with the losing candidate excluded.
Construct the prompt from the bead's context: title, body, workspace path, relevant files, and any dependency context. The prompt is a deterministic function of the bead state — same bead, same prompt.
Load the agent adapter configuration (YAML), render the invoke template with the built prompt, and execute via bash -c. The agent runs headless — it receives a prompt, does work, and exits.
The agent runs. NEEDLE waits. The only inputs are the exit code and stdout/stderr. There is no interactive communication during execution.
Evaluate the result and follow the explicit path for the observed outcome:
| Outcome | Exit Code | Handler |
|---|---|---|
| ✅ Success | 0 |
Validate output → close bead → log effort → loop |
| ❌ Failure | 1 |
Log failure reason → release bead → increment retry count → loop |
| ⏰ Timeout | 124 |
Release bead → mark deferred → loop |
| 💀 Crash | >128 |
Release bead → create alert bead → loop |
| 🏁 Race Lost | 4 |
(Handled at Step 2) → exclude candidate → retry select |
| 🫙 Queue Empty | — | Enter strand escalation → explore / mend / knot |
Every row is implemented. There are no unhandled cases.
When the primary workspace has no claimable beads, NEEDLE follows a strand sequence to find or create work. Each strand is evaluated in order — the first strand that yields a bead wins.
| # | Strand | Agent? | Purpose |
|---|---|---|---|
| 1 | 🪡 Pluck | Yes | Process beads from the assigned workspace |
| 2 | 🔧 Mend | No | Cleanup: orphaned claims, stale locks, health checks |
| 3 | 🔭 Explore | No | Search other workspaces for claimable beads |
| 4 | 🕸️ Weave | Yes | Create beads from documentation gaps (opt-in) |
| 5 | 🪢 Unravel | Yes | Propose alternatives for HUMAN-blocked beads (opt-in) |
| 6 | 💓 Pulse | Yes | Codebase health scans, auto-generate beads (opt-in) |
| 7 | 🪞 Reflect | Yes | Consolidate learnings from recent beads (opt-in) |
| 8 | 🪡 Splice | No | Document worker failures, create alert beads |
| 9 | 🪢 Knot | No | All strands exhausted — alert human, wait |
Multiple NEEDLE workers run independently with no central orchestrator. Coordination happens through the shared bead queue:
- Atomicity — the bead CLI's claim command uses SQLite transactions; exactly one worker wins each claim
- Determinism — all workers compute the same priority order; races are resolved by the database, not by timing
- Independence — each worker is a self-contained loop in its own tmux session
- Naming — workers use NATO alphabet identifiers:
alpha,bravo,charlie, ...
needle-claude-sonnet-alpha ──┐
needle-claude-sonnet-bravo ──┤
needle-codex-gpt4-charlie ───┼──► Shared .beads/ (SQLite + checkpoint)
needle-opencode-qwen-delta ──┤
needle-aider-sonnet-echo ────┘
NEEDLE is agent-agnostic. Any CLI that accepts a prompt and exits works.
| Agent | CLI | Input Method | Notes |
|---|---|---|---|
| Claude Code (interactive) | claude-interactive |
stdin | Recommended — uses subscription billing; see plugin |
| Claude Code (API) | claude --print |
stdin | Uses programmatic/API billing |
| OpenCode | opencode |
file | |
| Codex CLI | codex |
args | |
| Aider | aider --message |
args | |
| Custom | any | configurable via YAML adapter |
Adding a new agent requires only a YAML configuration file — no code changes.
The claude-interactive plugin ships as a separate release asset. It wraps the Claude Code CLI in a PTY so workers run under your Claude subscription instead of consuming programmatic API credits.
How it works: NEEDLE pipes subprocess stdio, which causes claude to detect a non-TTY and switch to API billing. claude-interactive creates an internal PTY so claude sees a real terminal, keeping it in interactive/subscription mode.
Prerequisites:
- Python 3.10 or later
pytePython library (installed automatically by the installer, or manually viapip install pyte)- The
claudeCLI on PATH
Install:
# Download the latest claude-interactive release
gh release download --repo jedarden/NEEDLE --pattern 'claude-interactive*'
chmod +x claude-interactive-install.sh
./claude-interactive-install.shNote: The installer will attempt to install pyte automatically. If your Python environment is externally managed (PEP 668, e.g., Debian 12, Ubuntu 23.04+, Homebrew Python), the installer will guide you through alternative installation methods.
Run:
cd /path/to/workspace
needle run --agent claude-interactive --count 4 # or: -i alpha to name a single workerSource lives in plugins/claude-interactive/.
NEEDLE/
├── Cargo.toml # Rust crate manifest
├── install.sh # One-line installer for prebuilt binaries
├── plugins/
│ └── claude-interactive/ # PTY wrapper — subscription billing adapter for Claude Code
├── src/
│ ├── main.rs # Worker entry point
│ ├── lib.rs # Library root
│ ├── agent_event.rs # Agent event telemetry utilities
│ ├── claude_md_placement.rs # CLAUDE.md placement logic
│ ├── commit_hook.rs # Bead-Id trailer injection for git commits
│ ├── routing.rs # Model-based adapter routing
│ ├── bead_store/ # Abstract bead backend interface
│ ├── bin/ # Auxiliary binaries (transform helpers)
│ ├── canary/ # Release channel promotion, canary tests
│ ├── claim/ # Atomic bead claiming via SQLite transactions
│ ├── cli/ # Command-line interface parsing
│ ├── config/ # `.needle.yaml` parsing and defaults
│ ├── cost/ # Token + USD spend tracking per bead and worker
│ ├── decision/ # Decision point detection, ADR management
│ ├── dispatch/ # Agent invocation + YAML adapter loading
│ ├── drift/ # Session similarity, clustering, divergence detection
│ ├── health/ # Liveness, stale-claim cleanup, watchdog
│ ├── learning/ # Retrospective extraction, learnings management
│ ├── mitosis/ # Child-aware bead splitting
│ ├── outcome/ # Explicit handler per outcome type
│ ├── peer/ # Multi-worker coordination, peer discovery
│ ├── prompt/ # Deterministic prompt construction from bead
│ ├── rate_limit/ # Provider/model concurrency and RPM rate limiting
│ ├── registry/ # Worker state registry
│ ├── sanitize/ # Output redaction (gitleaks integration)
│ ├── skill/ # Skill library, retrieval, promotion
│ ├── span/ # W3C trace context utilities
│ ├── stats/ # Aggregation engine, A/B comparison
│ ├── strand/ # Pluck / Mend / Explore / Weave / Unravel / Pulse / Reflect / Splice / Knot
│ ├── supervisor/ # Fleet supervisor daemon (auto-scale)
│ ├── telemetry/ # OTLP exporter, gen_ai semantic conventions
│ ├── trace/ # Trace capture, storage, retention
│ ├── transcript/ # Session JSONL parsing, action-outcome extraction
│ ├── types/ # Shared types, error definitions
│ ├── upgrade/ # Self-update, hot-reload, rollback
│ ├── validation/ # Pre-dispatch and post-execution checks
│ └── worker/ # Worker session and identity management
├── tests/ # Integration tests
├── ci/ # Docker images used by CI (runs on Argo Workflows)
├── config/ # Vendored gitleaks rules
└── docs/ # Plan, research, examples, post-mortems
NEEDLE emits structured telemetry for every state transition, claim attempt, dispatch, and outcome. A silent worker is a broken worker.
| Signal | Description |
|---|---|
| Traces | Spans for worker.session, bead.lifecycle, bead.claim, agent.dispatch, strand.evaluated, outcome.handled |
| Metrics | needle.beads.completed, needle.beads.duration, needle.agent.tokens.input, needle.cost.usd, and more |
| Logs | All events not represented as spans, with severity mapping (ERROR for failures, WARN for stale peers) |
NEEDLE can export telemetry to any OpenTelemetry-compatible backend (Jaeger, Tempo, Grafana, Honeycomb, Datadog, etc.) via OTLP.
Minimal configuration (.needle.yaml):
telemetry:
otlp_sink:
enabled: true
endpoint: "http://localhost:4317" # gRPC, or :4318 for HTTP
protocol: "grpc"Semantic conventions: NEEDLE follows OpenTelemetry's gen_ai.* semantic conventions for LLM telemetry, enabling out-of-the-box integration with GenAI dashboards (Grafana GenAI app, Langfuse, Honeycomb AI, etc.).
See docs/plan/plan.md for the complete semantic mapping table.
NEEDLE currently powers my own headless multi-agent workflow — workers run continuously against shared bead queues, dispatching to Claude Code and other CLIs, with full OTLP telemetry wired through. APIs are stable enough that I rebuild on top of them daily.
This is alpha software in the sense that I'm the primary user, not in the sense that "it doesn't work." Resource governance is delegated to claude-governor; session monitoring is handled by ccdash.
If you want to run NEEDLE in your own workflow, open an issue and I'll help.
- Documentation Index — Complete index of all 148 documentation files (ADRs, architecture, operations, investigations, reference)
- Binary Freshness Verification — Guide for verifying automatic worker rotation when new binaries are deployed
- Plan — Complete project architecture and implementation plan
- Integration Tests — Comprehensive test suite demonstrating all core functionality
- claude-governor — caps API spend and enforces weekly Anthropic quotas across NEEDLE worker fleets
- ccdash — TUI for monitoring Claude Code sessions, token usage, and worker activity
- CLASP — drop-in proxy letting Claude Code target OpenAI, Gemini, Anthropic, or any LLM backend
- agentists-quickstart — opinionated DevPod workspaces for running Claude Code + NEEDLE
MIT
Part of jedarden.com · Read the write-up: jedarden.com/projects/needle/
This GitHub repo is a read-only mirror of git.ardenone.com/jedarden/NEEDLE — issues and PRs are welcome here either way.