GeneralAgentHarnessLab

Experiments and infrastructure for evaluating and improving general-purpose agent harnesses.

GeneralAgentHarnessLab provides the measurement and execution layer needed to study and improve general-purpose agent harnesses without tying experiments to one framework. It records tool-using runs into a small portable event schema, replays those traces, and derives transparent execution metrics. Strands Agents is the first supported runtime; the core recorder and analysis tools do not depend on any model provider or agent framework.

Status: early alpha. The trace contract is intentionally small and may evolve before v1.0.

Why GeneralAgentHarnessLab?

Agent experiments are difficult to compare when every runtime emits different logs and mixes model output, tool calls, provider metadata, and local artifacts. Improving general-purpose agents requires a stable way to observe the same execution signals, reproduce failures, and evaluate changes. GeneralAgentHarnessLab separates the runtime from the experiment record:

flowchart LR
    A["Task"] --> B["Runtime adapter"]
    B --> C["Agent runtime"]
    C --> D["Tools and environment"]
    B --> E["Portable JSONL trace"]
    E --> F["Replay, metrics, and policies"]

Loading

The first release establishes four foundations for agent improvement work:

• a minimal runtime adapter interface;
• append-only, redacted JSONL traces;
• deterministic replay and execution summaries;
• transparent baseline policies for common failure patterns.

Quick start

GeneralAgentHarnessLab requires Python 3.10 or newer.

python -m venv .venv
source .venv/bin/activate
python -m pip install -e ".[dev]"

# No API key or model download required.
general-agent-harness-lab demo --prompt "Trace this small agent run"
general-agent-harness-lab summarize runs/demo.jsonl
general-agent-harness-lab audit runs/demo.jsonl

The demo produces a trace like this:

{"event_type":"run.started","runtime":"demo","sequence":0,"payload":{"prompt":"Trace this small agent run"}}
{"event_type":"tool.call","runtime":"demo","sequence":1,"payload":{"name":"word_count"}}
{"event_type":"tool.result","runtime":"demo","sequence":2,"payload":{"status":"ok"}}
{"event_type":"run.finished","runtime":"demo","sequence":4,"payload":{"status":"succeeded"}}

Timestamps, run IDs, schema versions, arguments, and results are omitted above for readability.

Run with Strands Agents

Install the optional integration and configure credentials for a Strands-supported model provider:

python -m pip install -e ".[strands]"
python examples/strands_quickstart.py
general-agent-harness-lab summarize runs/strands-quickstart.jsonl

The adapter attaches Strands lifecycle hooks to capture tool calls and results while leaving the runtime's model and tool configuration under application control.

Runtime adapter contract

A runtime only needs a name and one run method:

from agent_harness import EventType, RunResult


class MyRuntime:
    name = "my-runtime"

    def run(self, prompt, emit):
        emit(EventType.MODEL_MESSAGE, {"message": "hello"})
        return RunResult(output="hello")

Pass it to the harness:

from agent_harness import AgentHarness

run = AgentHarness(MyRuntime()).run("Say hello", trace_path="runs/hello.jsonl")
print(run.summary.to_dict())

Trace events

The v0.1 schema includes:

• run.started
• model.message
• tool.call
• tool.result
• policy.decision
• run.finished

Every event includes a schema version, run ID, monotonic sequence number, UTC timestamp, runtime name, and JSON payload. Trace files are append-only, easy to inspect, and intentionally independent of a particular model provider.

Baseline policies

The included repeated-call policy detects exact consecutive calls with identical canonicalized arguments. It is deliberately described as a baseline: structural repetition is useful to catch, but it is not equivalent to understanding task progress.

general-agent-harness-lab audit runs/example.jsonl --repeat-threshold 3

Security and privacy

Common credential fields and token shapes are redacted before events are written. Generated traces, local environments, and .env files are ignored by Git. Redaction cannot identify every secret, so always inspect traces before sharing them. See SECURITY.md.

Roadmap

• additional runtime adapters;
• richer trace inspection and export;
• benchmark integrations;
• pluggable execution-improvement policies;
• comparative cost, reliability, and task-quality evaluation.

Development

python -m pip install -e ".[dev]"
ruff check .
pytest

License

Apache-2.0