fix(evaluation): harden judge calibration workflow

[brunovicco]

Problem

The existing judge-calibration path could not reliably validate the published benchmark.

The legacy worksheet reconstructed cited source excerpts instead of retaining the exact ordered contexts evaluated by RAGAS Faithfulness. The legacy report script also accepted an arbitrary flat JSON file without validating evidence provenance, human-label scale, sample completeness, minimum sample size, or per-dimension agreement.

As a result, an aggregate score could hide a Faithfulness or Answer Relevancy failure.

Solution

Introduce one fail-closed calibration workflow based on the exact evidence seen by the judge.

The answer harness now persists ordered judge contexts. The calibration builder creates a deterministic, stratified, blind-labeling workspace, and the finalizer validates the workspace before enforcing overall and per-dimension agreement gates.

Main changes

• Persist exact ordered judge_contexts in answer and question records.
• Preserve backward-compatible reading of legacy records.
• Add deterministic calibration sampling across required strata and conceptual cases.
• Separate exact judge evidence from citation diagnostics in the worksheet.
• Validate human labels against the 0.0, 0.5, and 1.0 ordinal scale.
• Version sealed judge scores and evidence schemas.
• Require at least 30 distinct answer samples.
• Require exact alignment between human labels and sealed judge scores.
• Report agreement overall and by dimension.
• Enforce the kappa floor independently for overall, Faithfulness, and Answer Relevancy.
• Preserve existing human scores only when rebuilding the exact same sample.
• Write calibration artifacts atomically.
• Remove the legacy judge_calibration_report.py entrypoint and flat-file loader.
• Archive the ineligible 20260726T185553Z calibration workspace with hashes and an explicit warning.
• Document evidence-retention and privacy implications.

Test evidence

• Project quality gate passed.
• 554 tests passed; 20 opt-in integration tests deselected.
• Total coverage: 83.06%.
• Ruff lint and formatting passed.
• Mypy strict typing passed.
• Architecture validation passed.
• Bandit reported no issues.
• Dependency audit passed under the repository's documented advisory policy.

New entrypoint tests cover:

• successful report publication;
• rejection of incomplete human labels;
• non-zero exit when a required dimension misses the kappa floor.

Security and data impact

No credentials, secrets, or personal data are introduced.

Exact judge contexts are retained locally as run evidence. The current benchmark corpus consists of public official legal texts. The privacy documentation now records this retention explicitly.

The archived legacy workspace contains only public benchmark material and unfilled human labels. It is retained for audit history and is explicitly ineligible for calibration.

Operational and rollout impact

New benchmark runs include judge_contexts in records.jsonl, increasing artifact size.

Runs created before evidence schema v1 cannot be retrospectively calibrated and fail closed. They must be rerun to produce an eligible human-calibration workspace.

The canonical workflow is:

uv run python scripts/build_calibration_worksheet.py <run-id>
uv run python scripts/finalize_calibration.py --show-disagreements

Risks and follow-ups

• Run a new live benchmark using the updated evidence schema.
• Complete blind human labeling for at least 30 samples, preferably 50.
• Publish the calibration report and disagreement examples.
• Keep judge metrics qualified until the agreement gate passes.
• Deterministic benchmark replay remains separate ADR-0020 work.