feat(evaluation): independent OpenAI judge with abstention scoring (ADR-0018 increment 5)

[brunovicco]

Summary

• Implements the canonical independent judge defined in ADR-0018: a provider-neutral port (AnswerQualityJudge) behind which RagasJudge gains a real OpenAI adapter (gpt-5.4-mini-2026-03-17), alongside the existing Gemini adapter, which now becomes an explicit development fallback.
• Adds abstention evaluation - whether abstaining is appropriate, based on whether the question is answerable. This requires unanswerable questions, which were previously excluded entirely from the generation-and-judging flow, to be generated and judged as well.
• Fixes a real bug discovered along the way: the judge reused the RETRIEVAL embedding model (e.g. Qwen/Qwen3-Embedding-0.6B in the default local configuration) for RAGAS embedding calls through Gemini, which would fail silently. judge.embedding_model is now a separate configuration key.
• The scope covers the mechanism. The actual human calibration dataset (~30 samples) and the published kappa score are excluded because they require genuine human curation and cannot be fabricated in this session.

Details

• New evaluation/judge_ports.py: ModelIdentity, JudgeSample, JudgeResult, and AnswerQualityJudge (a Protocol exposing .identity and .evaluate(sample)).
• Rewritten ragas_judge.py: RagasJudge.evaluate(sample) -> JudgeResult; new build_openai_ragas_judge using instructor.from_provider("openai/...") and OpenAIEmbeddings. reasoning_effort is passed directly to InstructorLLM, verified against the library’s actual implementation, which already handles gpt-5.* models as reasoning models. build_gemini_ragas_judge was adapted accordingly. Abstention is evaluated through a dedicated structured call because RAGAS has no native metric for it. It uses a second, isolated InstructorLLM instance with its own PT-BR prompt, so it does not affect RAGAS’s internal prompts. JudgeResult contains only what RAGAS actually produces—a scalar score—rather than the structured unsupported_claims and rationale fields shown in the ADR’s illustrative JSON, which would require abandoning RAGAS’s built-in classes.
• answer_harness.py: every question, including unanswerable ones, now goes through retrieve → generate → judge; citation_accuracy remains absent from the metrics dictionary only for unanswerable questions; adds abstention_appropriate and citation_n, which now differs from answer_n because the two groups have different sizes.
• run.py: adds _build_judge_factory, fail-closed and following the same pattern as _build_embedder; adds judge_provider, judge_reasoning_effort, judge_prompt_version, and judge_label ("exploratory_same_provider_judge" when judge_provider == "gemini") to the manifest and --resume validation; adds an abstention column to the answer-quality table.
• benchmark-v01.yaml: the default is now judge.provider: openai.
• Promotes openai from a transitive dependency to a direct dependency in pyproject.toml. No package is actually added because it was already resolved through ragas/instructor.
• New evaluation/judge_calibration.py and scripts/judge_calibration_report.py: Cohen’s weighted kappa, Spearman correlation using statistics.correlation from the standard library with no new dependency, false-supported/unsupported rates, and abstention agreement. The mechanism is ready to consume real human-labeled data once it becomes available.

Out of scope (deferred and documented in the code)

• Actual human calibration dataset with 30+ samples and the published kappa score—requires human curation.
• factual_correctness—requires reference answers, which the golden set does not include, an inherited gap from ADR-0007.
• Local open-weight judge (gpt-oss-20b)—the ADR itself marks it as experimental.
• Provider Batch API—the ADR explicitly defers this.

Test plan

• uv run pytest — 351 passed, 20 integration tests deselected
• uv run mypy src tests — 0 issues across 157 files
• uv run python scripts/quality_gate.py — all checks pass (lint, formatting, architecture, MCP, typing, tests, and security/Bandit) except dependency auditing (pip-audit): gitpython 3.1.53 has three known advisories fixed in 3.1.54. Confirmed via git stash that this issue already exists on main and predates this increment; traced with uv tree --invert to streamlit, an unrelated direct dependency.
• grep AnswerJudge / grep judge.score\( across src/ and tests/ — no orphaned references found