Trustra Agent Eval Harness

An open-core evaluation and tracing harness for AI agents: run test suites against an agent, score the results, capture production traces, and build a tamper-evident record of what an agent actually did.

This repository is the below-the-trust-line rails layer of Trustra's broader independent AI verification product. It does the automated work, running evals, scoring outputs, hash-chaining evidence. It does not issue attestations; that is a separate, accountable, human-signed layer that sits on top of this record and is out of scope for this repository.

What it does

• Defines a small data model for agents, test cases, datasets, eval runs, and scores.
• Runs a dataset of test cases against an agent target and records the results.
• Scores results with a pluggable scorer interface (exact match, contains, latency thresholds out of the box; wrap your own or third-party scorers).
• Ingests OpenTelemetry-compatible traces from a running agent for online evaluation.
• Hash-chains every eval run and trace batch so the record is tamper-evident: each entry embeds the hash of the entry before it, so any retroactive edit breaks the chain and is detectable.
• Exposes a small FastAPI service and a CLI for running evals and querying the record.

What it deliberately does not do

• It does not issue attestations, certifications, or compliance sign-off. That requires a named, accountable human and is out of scope for open-source software by design.
• It does not claim to be an independent third party. Anyone can run this harness on their own systems; independence is a property of who operates it and what they do with the output, not of the code itself.

Quickstart

pip install -e .
trustra-eval init
trustra-eval run --dataset examples/sample_dataset.json --target examples/echo_target.py
trustra-eval report --run-id <run-id>

Project layout

src/trustra/
  models.py        Core data model (AgentTarget, TestCase, Dataset, EvalRun, Score, TraceEvent)
  provenance.py     Hash-chain implementation for tamper-evident records
  storage.py        SQLite-backed storage layer
  scorers/          Pluggable scorer interface and built-in scorers
  ingestion/        OpenTelemetry-compatible trace ingestion
  api/              FastAPI service
  cli.py            Command-line interface
tests/              Unit tests
docs/               Architecture notes

Status

Early stage (M0). Core data model, provenance chain, storage, a handful of built-in scorers, trace ingestion, and a minimal API and CLI are implemented and tested. Not yet implemented: distributed storage backends, a web dashboard, and wrappers for third-party scorer libraries. See docs/ARCHITECTURE.md and open issues for the roadmap.

Contributing

Contributions are welcome. See CONTRIBUTING.md. This project is licensed under AGPL-3.0-or-later specifically so that improvements made by anyone running this as a service flow back to the community; see LICENSE.

License

GNU Affero General Public License v3.0 or later. See LICENSE.