Expand EvidenceBench with Data.gov and 2026 caselaw

CASELAW_METHODOLOGY.md

@@ -1,7 +1,7 @@
 # EvidenceBench caselaw module
 
-The v2 sealed holdout adds 16 evidence-caselaw questions to the 96 FRE-focused
-items. The caselaw module is balanced across eight intersections:
+The v3 sealed holdout contains 55 evidence-caselaw questions alongside the 96
+FRE-focused items. Its original 16-question module is balanced across eight intersections:
 
 - federal and state authority;
 - civil and criminal proceedings; and
@@ -17,7 +17,33 @@ Question rationales were checked against official court publications or official
 reporters where available. The benchmark requires a reporter citation; a case
 name without a reporter citation does not satisfy citation recall.
 
-The v2 prompt asks for legal authorities. FRE questions continue to require
+The v3 prompt asks for legal authorities. FRE questions continue to require
 normalized FRE citations, while caselaw questions require official reporter
 citations. Answer accuracy, authority precision, authority recall, authority
 F1, hallucination rate, and the 70/30 overall formula are otherwise unchanged.
+
+## Data.gov/DOJ-NIJ expansion
+
+The 33-question v3 tranche uses every unique, citable decision dated 2020–2024
+that the public-domain DOJ/NIJ Post-PCAST decisions table labels published.
+This triples caselaw coverage from 16 to 49 questions and adds explicit year,
+recency, publication-status, forensic-discipline, and source dimensions. The
+government table is frozen as a normalized 124-row JSON snapshot with hashes.
+
+Each sealed question asks for the evidentiary decision effect recorded for one
+case and requires that case's published reporter, slip-opinion, or LEXIS
+citation. Dataset labels and summaries are preserved in the sealed rationale.
+The source is primarily forensic and criminal; published decisions in this
+slice include 21 state and 12 federal cases, but only three civil cases. Scores
+therefore report source-specific dimensions rather than implying balance.
+
+## Published 2026 decisions
+
+Six current decisions supplement the Data.gov tranche: four published federal
+appellate opinions and two decisions in New York's Official Reports. They cover
+Rule 702 expert gatekeeping, Rule 403 and stipulations, documentary evidence,
+expert disclosure, hearsay exceptions, and the Confrontation Clause. Because
+official reporter pagination may not yet exist for recently issued opinions,
+the frozen authority index accepts the court docket or New York slip citation.
+The public `data/recent-caselaw-2026.json` file records official source URLs and
+SHA-256 hashes; no secondary case summary is treated as authority.

DATA_LICENSE.md

@@ -7,3 +7,8 @@ result artifacts in this repository are licensed under
 The sealed official holdout is not included in this repository and is not
 licensed for redistribution. It is maintained by Objection Academy solely for
 official benchmark runs.
+
+`data/doj-nij-post-pcast-2024.json` is a normalized snapshot of the U.S.
+Department of Justice/National Institute of Justice Post-PCAST decisions table.
+Data.gov identifies that source dataset as Public Domain. Its provenance and
+source hashes are embedded in the snapshot.

METHODOLOGY.md

@@ -1,4 +1,4 @@
-# EvidenceBench v1 methodology
+# EvidenceBench v3 methodology
 
 EvidenceBench evaluates closed-book responses to attorney-authored evidence
 questions. A model receives a multiple-choice question and must return JSON
@@ -9,12 +9,27 @@ is available to the model.
 ## Dataset
 
 - Development set: 24 public questions licensed CC BY-NC 4.0.
-- Official v2 holdout: 112 sealed questions—96 FRE-focused and 16 caselaw;
+- Official v3 holdout: 151 sealed questions—96 FRE-focused and 55 caselaw;
   only aggregate results are published.
 - Every item has a stable ID, category, difficulty, rationale, gold answer,
   required citation groups, accepted citations, corpus version, and attorney
   review record.
 
+The v3 expansion adds 33 unique decisions from 2020–2024 that the DOJ/NIJ
+Post-PCAST decisions dataset marks as published and for which a deterministic
+case citation can be frozen. The source is public domain and cataloged by
+Data.gov. The complete normalized 124-row source snapshot, its retrieval URL,
+raw HTML hash, and normalized-record hash are checked in. Duplicate captions,
+unpublished rows, Lexis-only rows not marked published, uncitable rows, date
+mismatches, and decisions outside the frozen window are excluded.
+
+A separate current-law tranche adds six published decisions issued in 2026:
+four federal appellate opinions and two New York Official Reports decisions.
+Each source is court- or GovInfo-hosted, and the public metadata snapshot records
+the official URL, retrieval date, source hash, holding, court, docket or slip
+citation, and coverage dimensions. Unpublished decisions and secondary summaries
+are excluded.
+
 ## Scoring
 
 Answer accuracy is exact option match. Citation precision is accepted cited
@@ -23,7 +38,8 @@ citation groups satisfied. Citation F1 is their harmonic mean. A citation that
 does not exist in the frozen FRE corpus is hallucinated; a real but
 unannotated authority is unsupported. Overall score is 70% answer accuracy and
 30% authority-citation F1. Caselaw results are additionally aggregated across
-federal/state, civil/criminal, and popular/obscure dimensions.
+federal/state, civil/criminal, popular/obscure, decision-year, recency,
+publication-status, forensic-discipline, and source dimensions.
 
 ## Reproducibility
 
@@ -36,7 +52,8 @@ of prompt rendering and scoring without exposing the official holdout.
 
 EvidenceBench measures this specific closed-book task. It does not measure
 case-law research, jurisdiction-specific practice, current good-law status,
-or the safety of using a model in litigation.
+or the safety of using a model in litigation. The DOJ/NIJ source is concentrated
+in forensic-science admissibility and is not representative of all evidence law.
 
 ## Authority freeze
 

README.md

@@ -15,9 +15,15 @@ professional legal research.
 
 ## What is sealed
 
-The v2 official holdout contains 96 FRE-focused questions plus a 16-question
-caselaw module balanced across federal/state, civil/criminal, and
-popular/obscure decisions. It is attorney-reviewed,
+The v3 official holdout contains 96 FRE-focused questions plus a 55-question
+caselaw module. The original 16-case matrix remains balanced across
+federal/state, civil/criminal, and popular/obscure decisions. A new 33-question
+tranche covers every unique, citable decision from 2020–2024 classified as
+published in the public-domain DOJ/NIJ Post-PCAST dataset cataloged by
+[Data.gov](https://catalog.data.gov/dataset/post-pcast-court-decisions-assessing-the-admissibility-of-forensic-science-evidence).
+Six additional published 2026 decisions are frozen from official federal court,
+GovInfo, and New York Official Reports sources.
+It is attorney-reviewed,
 and never distributed with the Objection Academy app. Its prompts,
 annotations, and item-level results are not public. Once approved, published
 manifests will commit to the holdout with a SHA-256 hash, counts, categories,
@@ -40,16 +46,16 @@ Each item receives answer accuracy, authority precision, authority recall,
 citation F1, citation-existence status, and unsupported-citation status.
 Official overall score is `0.70 * answer accuracy + 0.30 * citation F1`.
 Missing citations, refusals, and invalid structured outputs score zero for
-authority F1. Caselaw questions require an official reporter citation.
+authority F1. Caselaw questions require a frozen reporter, slip-opinion, or
+published LEXIS citation.
 
 ## Authority corpus
 
-v1 uses the Federal Rules of Evidence in effect December 1, 2025. The pending
-Rule 801 amendment effective December 1, 2026 is excluded from v1.
+v3 uses the Federal Rules of Evidence in effect December 1, 2025. The pending
+Rule 801 amendment effective December 1, 2026 is excluded.
 
 ## Official-run gate
 
-The sealed 96-question holdout has an attorney-approved citation and content
-review record and is committed in the public manifest by SHA-256. The initial
-leaderboard remains empty until six credentialed, closed-book provider runs
-complete and their signed aggregate results are exported.
+The sealed holdout has an attorney-approved citation and content review record
+and is committed in each public manifest by SHA-256. The normalized government
+source snapshot is public; sealed questions and item-level model outputs are not.

data/case-authorities-v3.json

@@ -0,0 +1,60 @@
+{
+  "version": "CASELAW-DATAGOV-2026-07-14",
+  "citations": [
+    "17 Cal. 3d 24",
+    "195 N.E.3d 19",
+    "2022-Ohio-1739",
+    "2023 U.S. Dist. LEXIS 208806",
+    "2024 Cal. App. LEXIS 532",
+    "2024-Ohio-2572",
+    "2026 NY Slip Op 26043",
+    "2026 NY Slip Op 26081",
+    "208 N.J. 208",
+    "223 N.E.3d 160",
+    "234 N.J. 340",
+    "275 N.C. App. 843",
+    "306 A.3d 126",
+    "308 Neb. 708",
+    "337 Conn. 527",
+    "35 N.Y.3d 24",
+    "352 Or. 724",
+    "442 F. Supp. 3d 1122",
+    "444 F. Supp. 3d 1248",
+    "463 N.J. Super. 355",
+    "464 F. Supp. 3d 1252",
+    "466 N.J. Super. 270",
+    "467 F. Supp. 3d 598",
+    "483 Md. 637",
+    "502 F. Supp. 3d 28",
+    "505 U.S. 317",
+    "509 U.S. 579",
+    "512 P.3d 269",
+    "513 U.S. 150",
+    "519 U.S. 172",
+    "526 U.S. 137",
+    "528 U.S. 440",
+    "536 F. Supp. 3d 856",
+    "541 U.S. 36",
+    "55 Cal. 4th 747",
+    "552 U.S. 379",
+    "576 F. Supp. 3d 827",
+    "58 Cal. App. 5th 504",
+    "6 N.Y.3d 636",
+    "623 S.W.3d 717",
+    "629 F. Supp. 3d 841",
+    "63 F.4th 1229",
+    "662 Pa. 672",
+    "68 Misc. 3d 899",
+    "7 N.Y.3d 434",
+    "75 Cal. App. 5th 694",
+    "81 Misc. 3d 550",
+    "86 Cal. App. 5th 1094",
+    "92 A.3d 766",
+    "973 F.3d 667",
+    "990 F.3d 457",
+    "No. 23-2226",
+    "No. 23-30854",
+    "No. 24-1743",
+    "No. 24-1907"
+  ]
+}