feat: add caselaw benchmark snapshot

CASELAW_METHODOLOGY.md

@@ -0,0 +1,23 @@
+# EvidenceBench caselaw module
+
+The v2 sealed holdout adds 16 evidence-caselaw questions to the 96 FRE-focused
+items. The caselaw module is balanced across eight intersections:
+
+- federal and state authority;
+- civil and criminal proceedings; and
+- widely cited and less familiar decisions.
+
+Each intersection contains two questions. Popularity is a sampling stratum,
+not a claim about precedential weight. The questions test holdings concerning
+expert gatekeeping, scientific evidence, eyewitness identification, hearsay,
+prior statements, and Rule 403.
+
+Case existence and citation form are validated against a frozen reporter index.
+Question rationales were checked against official court publications or official
+reporters where available. The benchmark requires a reporter citation; a case
+name without a reporter citation does not satisfy citation recall.
+
+The v2 prompt asks for legal authorities. FRE questions continue to require
+normalized FRE citations, while caselaw questions require official reporter
+citations. Answer accuracy, authority precision, authority recall, authority
+F1, hallucination rate, and the 70/30 overall formula are otherwise unchanged.

METHODOLOGY.md

@@ -1,16 +1,16 @@
 # EvidenceBench v1 methodology
 
-EvidenceBench evaluates closed-book responses to attorney-authored Federal
-Rules of Evidence questions. A model receives a multiple-choice question and
-must return JSON containing one option ID, a concise explanation, and Federal
-Rules of Evidence citations. No browsing, retrieval, tools, or external
-authority text is available to the model.
+EvidenceBench evaluates closed-book responses to attorney-authored evidence
+questions. A model receives a multiple-choice question and must return JSON
+containing one option ID, a concise explanation, and supporting FRE or official
+reporter citations. No browsing, retrieval, tools, or external authority text
+is available to the model.
 
 ## Dataset
 
 - Development set: 24 public questions licensed CC BY-NC 4.0.
-- Official holdout: 96 newly authored sealed questions; only aggregate results
-  are published.
+- Official v2 holdout: 112 sealed questions—96 FRE-focused and 16 caselaw;
+  only aggregate results are published.
 - Every item has a stable ID, category, difficulty, rationale, gold answer,
   required citation groups, accepted citations, corpus version, and attorney
   review record.
@@ -21,8 +21,9 @@ Answer accuracy is exact option match. Citation precision is accepted cited
 rules divided by all cited rules. Citation recall is the share of required
 citation groups satisfied. Citation F1 is their harmonic mean. A citation that
 does not exist in the frozen FRE corpus is hallucinated; a real but
-unannotated rule is unsupported. Overall score is 70% answer accuracy and 30%
-citation F1.
+unannotated authority is unsupported. Overall score is 70% answer accuracy and
+30% authority-citation F1. Caselaw results are additionally aggregated across
+federal/state, civil/criminal, and popular/obscure dimensions.
 
 ## Reproducibility
 

README.md

@@ -1,7 +1,7 @@
 # EvidenceBench
 
 EvidenceBench is Objection Academy's public benchmark for evidence-law answer
-accuracy and Federal Rules of Evidence citation reliability. It is an
+accuracy and legal-authority citation reliability. It is an
 educational research project, not legal advice and not a substitute for
 professional legal research.
 
@@ -10,11 +10,14 @@ professional legal research.
 - A 24-question development set and its scoring annotations.
 - The prompt contract, scoring code, model manifests, methodology, and website
   export format.
-- Aggregate official results after an attorney-approved sealed holdout run.
+- Aggregate official results after an approved sealed holdout run, including
+  separate FRE and caselaw dimensions.
 
 ## What is sealed
 
-The 96-question official holdout will be newly authored, attorney-reviewed,
+The v2 official holdout contains 96 FRE-focused questions plus a 16-question
+caselaw module balanced across federal/state, civil/criminal, and
+popular/obscure decisions. It is attorney-reviewed,
 and never distributed with the Objection Academy app. Its prompts,
 annotations, and item-level results are not public. Once approved, published
 manifests will commit to the holdout with a SHA-256 hash, counts, categories,
@@ -33,11 +36,11 @@ credentials named in that manifest. It never enables web search or tools.
 
 ## Metrics
 
-Each item receives answer accuracy, citation precision, citation recall,
+Each item receives answer accuracy, authority precision, authority recall,
 citation F1, citation-existence status, and unsupported-citation status.
 Official overall score is `0.70 * answer accuracy + 0.30 * citation F1`.
 Missing citations, refusals, and invalid structured outputs score zero for
-citation F1.
+authority F1. Caselaw questions require an official reporter citation.
 
 ## Authority corpus
 

data/case-authorities-v2.json

@@ -0,0 +1,9 @@
+{
+  "version": "CASELAW-2026-07-14",
+  "citations": [
+    "509 U.S. 579", "526 U.S. 137", "528 U.S. 440", "552 U.S. 379",
+    "541 U.S. 36", "519 U.S. 172", "513 U.S. 150", "505 U.S. 317",
+    "55 Cal. 4th 747", "7 N.Y.3d 434", "6 N.Y.3d 636", "234 N.J. 340",
+    "17 Cal. 3d 24", "208 N.J. 208", "352 Or. 724", "92 A.3d 766"
+  ]
+}