Proposal: EvalPort adapter for autoevals' Scorer/Score primitives

[adhabnr-ux]

Summary

I maintain EvalPort, an Apache-2.0 open spec (TestCase/Grader/EvalSuite/ResultSet/GraderResult as JSON Schema-validated documents) for making LLM eval datasets and results portable across frameworks. I read through py/autoevals/score.py, llm.py, value.py, number.py, and string.py on main and think autoevals' Score/Scorer contract is a genuinely good fit for an EvalPort adapter — better than most frameworks I've looked at, for one specific reason I'll get to below. I'm not asking for anything to land in this repo; this is a proposal/discussion issue for feedback before I build it, per your AGENTS.md/README norms (no CONTRIBUTING.md, so posting as an issue since Discussions isn't enabled here).

What I actually read in the code

• Scorer (score.py) is an ABC with eval(output, expected=None, **kwargs) -> Score, an async twin eval_async, and __call__ as an alias. Every scorer — ExactMatch, Levenshtein, EmbeddingSimilarity, NumericDiff, the LLMClassifier family (Factuality, Battle, ClosedQA, Humor, Security, Sql, Summary, Translation, Possible), the RAG metrics in ragas.py, json.py's validators — implements the same _run_eval_sync(output, expected, **kwargs) shape.
• Score (also score.py) is a dataclass: name: str, score: float | None (validated in __post_init__ to raise ValueError if outside [0, 1]), metadata: dict, and a deprecated error field (exceptions now propagate to the caller instead). That range check is the detail I meant above — it's already exactly EvalPort's GraderResult.score contract ({"type": ["number","null"], "minimum": 0, "maximum": 1}), with no clamping/normalization step needed on the way in. Most frameworks I've adapted so far (Likert scales, raw cosine similarity that can go negative, 0–100 scores) need metadata.openeval.raw_score to preserve the native value after normalizing; autoevals doesn't, because it's normalized at the source.
• There's no batch/Eval() runner in this repo. Batch execution and dataset/experiment tracking live in the separate braintrust package's Eval() function (your own README's "Using Braintrust with Autoevals" section shows this — it's explicitly optional). autoevals itself is scorer-only and works standalone (pip install autoevals, no braintrust import required).

Why this isn't redundant with the existing braintrust-openeval-adapter

EvalPort already has an adapter for the braintrust package (tracked in evalport#3) that converts Eval() result objects — the run-level shape (input/expected/output/scores per case, from an already-executed experiment). It happens to use autoevals.Factuality in its own usage example, but it's consuming Braintrust's experiment output, not autoevals' Score/Scorer objects directly. Anyone using autoevals scorers standalone — no braintrust import, no Eval() — has no adapter path today. That's the gap this issue is about.

Proposed mapping (standalone package, adapters/autoevals-openeval-adapter, zero changes needed in this repo)

Grader definition — one EvalPort Grader per scorer class used:

• ExactMatch → type: "custom", params.handler: "autoevals:ExactMatch". I initially wanted to map this to EvalPort's well-known exact_match type, but ExactMatch does JSON-aware normalization (dict/list serialization, JSON-string parsing) that EvalPort's exact_match params (ignore_case, trim_whitespace) don't describe — so custom + handler is the honest choice per the spec's type-openness rule, not a cosmetic one.
• LLMClassifier-based scorers (Factuality, Battle, ClosedQA, Humor, Security, Sql, Translation, Possible, or a hand-built LLMClassifier) → type: "llm_judge", params.model = self.model, params.prompt derived from the constructor's prompt_template. This is the closest real fit in the whole library — but it's a partial one, worth stating plainly: EvalPort's llm_judge grader assumes the receiving runner executes the prompt and gets back a free-form {score, reason}, whereas LLMClassifier grades by picking one of a fixed set of choice_scores via a select_choice tool call. I'd carry choice_scores through in params (an EvalPort-permitted extra key) so a receiving runner that also understands autoevals' choice-based grading can reproduce it exactly, and fall back to documenting model+prompt for one that can't.
• Levenshtein, NumericDiff → type: "custom" (autoevals:Levenshtein, autoevals:NumericDiff) — no well-known EvalPort type covers edit-distance or magnitude-relative numeric closeness.
• EmbeddingSimilarity → type: "semantic_similarity", params.model = self.extra_args["model"], params.threshold = self.expected_min — genuinely clean fit, module names line up almost 1:1.
• ragas.py metrics (ContextRelevancy, Faithfulness, AnswerCorrectness, etc.) and json.py's validators → type: "custom" with an autoevals:ragas:<Name> / autoevals:json:<Name> handler; none of these match a well-known EvalPort type closely enough to claim it without misleading a runner about what params to expect.

Result conversion (to_openeval) — this is the direction that matters most, since it's lossless in a way I don't get with most frameworks:

def score_to_grader_result(grader_id: str, grader_type: str,
                            result: "autoevals.Score", *, pass_threshold: float = 0.5) -> dict:
    return {
        "grader_id": grader_id,
        "type": grader_type,
        "score": result.score,                      # already None or [0,1] — no clamping needed
        "passed": result.score is not None and result.score >= pass_threshold,
        "reason": result.metadata.get("rationale"),  # populated by LLMClassifier scorers
        "metadata": dict(result.metadata),           # preserved verbatim, lossless
    }

The one honest caveat here: Score itself carries no notion of "passed" — it's a bare continuous (or discrete-mapped) value, by design. EvalPort's GraderResult.passed is required, so the adapter has to supply a pass_threshold (default 0.5, overridable) the same way the braintrust-openeval-adapter already does for Braintrust's own bare FeedbackScoreDict values — this isn't a new problem I'm inventing, it's the same shape as an already-shipped adapter in the same repo.

from_openeval direction is necessarily partial, and I'd rather say that here than overclaim it in a README: an EvalPort TestCase carries input/expected_output/context, but never an actual_output — that only exists in a ResultSet.Result, produced by whatever's being evaluated. So from_openeval(suite) can only produce the (output=<caller-supplied>, expected=test_case.expected_output, input=test_case.input, **context_kwargs) call arguments for the right Scorer class (selected from the suite's Grader.params.handler) — it can't synthesize a call to .eval() on its own, since the "output" is the caller's model output, not something the adapter has.

What I'm asking

Not a code change — this needs nothing merged here, it's a standalone package against your existing public Scorer/Score surface, same as the other ~35 EvalPort adapters. I'd like feedback before building it:

1. Is the llm_judge mapping above (carrying choice_scores alongside model/prompt in params) a reasonable way to describe LLMClassifier, or is there a cleaner way to think about it that I'm missing from the inside?
2. Would a link to this adapter be welcome from your README's ecosystem/integrations section once it exists (opt-in, your call, no obligation either way)?

Happy to open the actual package as a PR to adhabnr-ux/evalport once there's a rough consensus here, and to link back to this issue from it either way.

— Sahi, independent contributor (not affiliated with this project)