A local, evidence-guided harness for benchmarking language models under strict output contracts — built so that AI-generated work is verifiable, not just fast.
Models generate under contract. A human curates and hardens. Deterministic gates verify. Nothing a model claims about itself is trusted; every run is checked by code and recorded as a hashed evidence pack.
Most "AI benchmarking" trusts the model to self-report. This does not. The core is a small, testable Go pipeline that treats model output as untrusted input: TaskSpec -> raw model response -> normalizer (strips reasoning traces, placeholder artifacts) -> deterministic validators (schema, secret-leak, trace-leak, contract checks) -> verifier decision -> evidence pack: manifest with SHA-256 hashes
Around this core, hardened Bash harnesses drive local models (Devstral, Ministral, Magistral, Qwen MoE) via llama.cpp/ROCm, classify runtime failures (ROCm OOM, segfaults), quarantine broken models, and aggregate CSV summaries.
make test # run the Go test suite make smoke # run the evidence-writer smoke test make fmt # gofmt
Requires Go (standard library only). The benchmark harnesses additionally
require a local llama.cpp/ROCm setup and are parameterized via environment
variables (see run-long-role-bench-safe.sh).
EVIDENCE.md records claims against a strict source-of-truth hierarchy: local
repo artifacts and tags rank above terminal output, which ranks above
reconstructed context. Reconstructed context is never treated as confirmed
unless linked to a local artifact.
Verdicts state exactly what the data supports and what it does not. Example: the Qwen MoE 100x checkpoint is evidence for runtime/recovery/schema stability — not for autonomous model quality or semantic correctness.
go.mod uses module workspace intentionally: the benchmark seed contract
requires that module path, and the prompt_echo validator checks for it.
It is a contract fixture, not an oversight.
Early and deliberately scoped. The Go core is gofmt/vet-clean with passing tests; the benchmark epistemics and shell hardening are the mature parts. This is a working harness and a reference slice, not a finished product.
MIT — see LICENSE.