Outcome-verified evaluation for agent capability infrastructure.
Hold the agent fixed. Change the capability provider. Verify what actually happened.
Agent Capability Benchmark measures whether capability infrastructure lets a fixed AI agent complete real, externally verifiable work. It evaluates the full path from capability discovery and account connection to execution, recovery, policy compliance, and final state—not self-reported integration counts or success claims.
Important
This project is pre-alpha. The contracts and reference runtime are executable; production test-tenant backends and official provider results are not published yet.
If this is a problem you want solved, star the repository and join the design discussions.
Requirements: Python 3.11 or newer.
git clone https://github.com/MiltonHeYan/agent-capability-benchmark.git
cd agent-capability-benchmark
python -m venv .venv && source .venv/bin/activate
python -m pip install -e ".[dev]"
make checkValidate the public task suite:
capability-bench validate tasks/public| Fixed inside a comparison block | Varied | Independently verified | |
|---|---|---|---|
| What | Agent runner, model, prompt, task, fixture, limits, verifier | Capability provider and its thin adapter | External state, policy boundaries, duplicates, cost ceilings |
| Why | Isolate the capability layer's contribution | Compare like with like | Prevent false success from becoming a pass |
An agent saying “done” is never sufficient evidence. A run passes only when the trusted verifier observes the declared outcome.
flowchart LR
T[Versioned task] --> R[Fixed AgentRunner]
A[Provider adapter] -->|CapabilityBundle| R
R --> P[Capability provider]
P --> X[Dedicated test tenant]
S[Sandbox control plane] -->|execution lease| A
S -->|baseline + final state| V[Independent verifier]
X --> V
The execution plane receives only run-scoped capability handles. Verifier and cleanup credentials stay in the trusted control plane. Each external object is namespaced by run_id, and final-state collection still runs after timeouts or transport failures.
- Verified completion — did the expected external state actually exist?
- Discovery and schemas — could the agent find and correctly call the capability?
- Authentication — were the correct account, scope, and recovery path used?
- Safety and governance — were approval, data, and spending boundaries respected?
- Recovery and idempotency — did retries avoid duplicate or contradictory side effects?
- Operational quality — latency, calls, retries, tokens, and observable cost.
Optional tracks cover authentication, memory, payments, governance, identity, sandboxed computation, and full-stack workflows. Non-eligible products are reported as not_eligible, not failed.
- General model intelligence or prompt-writing quality
- Self-reported feature or integration counts
- A provider result produced with a provider-selected agent
- A single opaque score that hides safety, reliability, latency, or cost
- Outcomes that cannot be checked independently
| Component | Status |
|---|---|
| Benchmark principles and v0.1 protocol | Implemented |
| Machine-readable task and evidence schemas | Implemented |
| Public task contracts | 12 tasks across 6 tracks |
| Deterministic verifier and loopback fixtures | Implemented and tested |
| Provider adapter / agent runner separation | Implemented and tested, including JSONL subprocess driver |
| Sandbox and test-tenant control-plane contract | Implemented and tested |
| Production service backends | Planned |
| Repeated-run statistics and official results | Planned |
adapters/ Provider integration boundary and manifests
agent_capability_benchmark/ Harness, contracts, verifier, and CLI
docs/ Methodology and architecture
runners/ Fixed agent-engine integrations
spec/ Machine-readable schemas
tasks/public/ Public benchmark task definitions
tests/ Contract and runtime tests
verifiers/ Independent outcome-verification guidance
Start with the benchmark specification, then use the documentation map to go deeper.
This repository is looking for maintainers interested in agent infrastructure, evaluation, authentication, sandboxes, developer tooling, and applied security.
For a scoped first contribution, browse good first issue or help wanted.
Good ways to begin:
- Propose a vendor-neutral task
- Propose a runner, adapter, verifier, or sandbox integration
- Improve a deterministic verifier or add a regression test
- Review the methodology in Discussions
Read CONTRIBUTING.md for development setup and GOVERNANCE.md for the path from contributor to maintainer.
- GitHub Discussions — methodology, architecture, roadmap, and long-form questions
- GitHub Issues — reproducible bugs and scoped proposals
- Security reporting — follow SECURITY.md; never post credentials or fixture secrets
Research and evaluation work can cite this repository using CITATION.cff.
Released under the MIT License.