diff --git a/CASELAW_METHODOLOGY.md b/CASELAW_METHODOLOGY.md index db7d24e..2ec5f5c 100644 --- a/CASELAW_METHODOLOGY.md +++ b/CASELAW_METHODOLOGY.md @@ -1,7 +1,7 @@ # EvidenceBench caselaw module -The v2 sealed holdout adds 16 evidence-caselaw questions to the 96 FRE-focused -items. The caselaw module is balanced across eight intersections: +The v3 sealed holdout contains 55 evidence-caselaw questions alongside the 96 +FRE-focused items. Its original 16-question module is balanced across eight intersections: - federal and state authority; - civil and criminal proceedings; and @@ -17,7 +17,33 @@ Question rationales were checked against official court publications or official reporters where available. The benchmark requires a reporter citation; a case name without a reporter citation does not satisfy citation recall. -The v2 prompt asks for legal authorities. FRE questions continue to require +The v3 prompt asks for legal authorities. FRE questions continue to require normalized FRE citations, while caselaw questions require official reporter citations. Answer accuracy, authority precision, authority recall, authority F1, hallucination rate, and the 70/30 overall formula are otherwise unchanged. + +## Data.gov/DOJ-NIJ expansion + +The 33-question v3 tranche uses every unique, citable decision dated 2020–2024 +that the public-domain DOJ/NIJ Post-PCAST decisions table labels published. +This triples caselaw coverage from 16 to 49 questions and adds explicit year, +recency, publication-status, forensic-discipline, and source dimensions. The +government table is frozen as a normalized 124-row JSON snapshot with hashes. + +Each sealed question asks for the evidentiary decision effect recorded for one +case and requires that case's published reporter, slip-opinion, or LEXIS +citation. Dataset labels and summaries are preserved in the sealed rationale. +The source is primarily forensic and criminal; published decisions in this +slice include 21 state and 12 federal cases, but only three civil cases. Scores +therefore report source-specific dimensions rather than implying balance. + +## Published 2026 decisions + +Six current decisions supplement the Data.gov tranche: four published federal +appellate opinions and two decisions in New York's Official Reports. They cover +Rule 702 expert gatekeeping, Rule 403 and stipulations, documentary evidence, +expert disclosure, hearsay exceptions, and the Confrontation Clause. Because +official reporter pagination may not yet exist for recently issued opinions, +the frozen authority index accepts the court docket or New York slip citation. +The public `data/recent-caselaw-2026.json` file records official source URLs and +SHA-256 hashes; no secondary case summary is treated as authority. diff --git a/DATA_LICENSE.md b/DATA_LICENSE.md index f92ab9b..93f5ef3 100644 --- a/DATA_LICENSE.md +++ b/DATA_LICENSE.md @@ -7,3 +7,8 @@ result artifacts in this repository are licensed under The sealed official holdout is not included in this repository and is not licensed for redistribution. It is maintained by Objection Academy solely for official benchmark runs. + +`data/doj-nij-post-pcast-2024.json` is a normalized snapshot of the U.S. +Department of Justice/National Institute of Justice Post-PCAST decisions table. +Data.gov identifies that source dataset as Public Domain. Its provenance and +source hashes are embedded in the snapshot. diff --git a/METHODOLOGY.md b/METHODOLOGY.md index 8dfc4aa..78d5213 100644 --- a/METHODOLOGY.md +++ b/METHODOLOGY.md @@ -1,4 +1,4 @@ -# EvidenceBench v1 methodology +# EvidenceBench v3 methodology EvidenceBench evaluates closed-book responses to attorney-authored evidence questions. A model receives a multiple-choice question and must return JSON @@ -9,12 +9,27 @@ is available to the model. ## Dataset - Development set: 24 public questions licensed CC BY-NC 4.0. -- Official v2 holdout: 112 sealed questions—96 FRE-focused and 16 caselaw; +- Official v3 holdout: 151 sealed questions—96 FRE-focused and 55 caselaw; only aggregate results are published. - Every item has a stable ID, category, difficulty, rationale, gold answer, required citation groups, accepted citations, corpus version, and attorney review record. +The v3 expansion adds 33 unique decisions from 2020–2024 that the DOJ/NIJ +Post-PCAST decisions dataset marks as published and for which a deterministic +case citation can be frozen. The source is public domain and cataloged by +Data.gov. The complete normalized 124-row source snapshot, its retrieval URL, +raw HTML hash, and normalized-record hash are checked in. Duplicate captions, +unpublished rows, Lexis-only rows not marked published, uncitable rows, date +mismatches, and decisions outside the frozen window are excluded. + +A separate current-law tranche adds six published decisions issued in 2026: +four federal appellate opinions and two New York Official Reports decisions. +Each source is court- or GovInfo-hosted, and the public metadata snapshot records +the official URL, retrieval date, source hash, holding, court, docket or slip +citation, and coverage dimensions. Unpublished decisions and secondary summaries +are excluded. + ## Scoring Answer accuracy is exact option match. Citation precision is accepted cited @@ -23,7 +38,8 @@ citation groups satisfied. Citation F1 is their harmonic mean. A citation that does not exist in the frozen FRE corpus is hallucinated; a real but unannotated authority is unsupported. Overall score is 70% answer accuracy and 30% authority-citation F1. Caselaw results are additionally aggregated across -federal/state, civil/criminal, and popular/obscure dimensions. +federal/state, civil/criminal, popular/obscure, decision-year, recency, +publication-status, forensic-discipline, and source dimensions. ## Reproducibility @@ -36,7 +52,8 @@ of prompt rendering and scoring without exposing the official holdout. EvidenceBench measures this specific closed-book task. It does not measure case-law research, jurisdiction-specific practice, current good-law status, -or the safety of using a model in litigation. +or the safety of using a model in litigation. The DOJ/NIJ source is concentrated +in forensic-science admissibility and is not representative of all evidence law. ## Authority freeze diff --git a/README.md b/README.md index 76f08f5..358a775 100644 --- a/README.md +++ b/README.md @@ -15,9 +15,15 @@ professional legal research. ## What is sealed -The v2 official holdout contains 96 FRE-focused questions plus a 16-question -caselaw module balanced across federal/state, civil/criminal, and -popular/obscure decisions. It is attorney-reviewed, +The v3 official holdout contains 96 FRE-focused questions plus a 55-question +caselaw module. The original 16-case matrix remains balanced across +federal/state, civil/criminal, and popular/obscure decisions. A new 33-question +tranche covers every unique, citable decision from 2020–2024 classified as +published in the public-domain DOJ/NIJ Post-PCAST dataset cataloged by +[Data.gov](https://catalog.data.gov/dataset/post-pcast-court-decisions-assessing-the-admissibility-of-forensic-science-evidence). +Six additional published 2026 decisions are frozen from official federal court, +GovInfo, and New York Official Reports sources. +It is attorney-reviewed, and never distributed with the Objection Academy app. Its prompts, annotations, and item-level results are not public. Once approved, published manifests will commit to the holdout with a SHA-256 hash, counts, categories, @@ -40,16 +46,16 @@ Each item receives answer accuracy, authority precision, authority recall, citation F1, citation-existence status, and unsupported-citation status. Official overall score is `0.70 * answer accuracy + 0.30 * citation F1`. Missing citations, refusals, and invalid structured outputs score zero for -authority F1. Caselaw questions require an official reporter citation. +authority F1. Caselaw questions require a frozen reporter, slip-opinion, or +published LEXIS citation. ## Authority corpus -v1 uses the Federal Rules of Evidence in effect December 1, 2025. The pending -Rule 801 amendment effective December 1, 2026 is excluded from v1. +v3 uses the Federal Rules of Evidence in effect December 1, 2025. The pending +Rule 801 amendment effective December 1, 2026 is excluded. ## Official-run gate -The sealed 96-question holdout has an attorney-approved citation and content -review record and is committed in the public manifest by SHA-256. The initial -leaderboard remains empty until six credentialed, closed-book provider runs -complete and their signed aggregate results are exported. +The sealed holdout has an attorney-approved citation and content review record +and is committed in each public manifest by SHA-256. The normalized government +source snapshot is public; sealed questions and item-level model outputs are not. diff --git a/data/case-authorities-v3.json b/data/case-authorities-v3.json new file mode 100644 index 0000000..74959b7 --- /dev/null +++ b/data/case-authorities-v3.json @@ -0,0 +1,60 @@ +{ + "version": "CASELAW-DATAGOV-2026-07-14", + "citations": [ + "17 Cal. 3d 24", + "195 N.E.3d 19", + "2022-Ohio-1739", + "2023 U.S. Dist. LEXIS 208806", + "2024 Cal. App. LEXIS 532", + "2024-Ohio-2572", + "2026 NY Slip Op 26043", + "2026 NY Slip Op 26081", + "208 N.J. 208", + "223 N.E.3d 160", + "234 N.J. 340", + "275 N.C. App. 843", + "306 A.3d 126", + "308 Neb. 708", + "337 Conn. 527", + "35 N.Y.3d 24", + "352 Or. 724", + "442 F. Supp. 3d 1122", + "444 F. Supp. 3d 1248", + "463 N.J. Super. 355", + "464 F. Supp. 3d 1252", + "466 N.J. Super. 270", + "467 F. Supp. 3d 598", + "483 Md. 637", + "502 F. Supp. 3d 28", + "505 U.S. 317", + "509 U.S. 579", + "512 P.3d 269", + "513 U.S. 150", + "519 U.S. 172", + "526 U.S. 137", + "528 U.S. 440", + "536 F. Supp. 3d 856", + "541 U.S. 36", + "55 Cal. 4th 747", + "552 U.S. 379", + "576 F. Supp. 3d 827", + "58 Cal. App. 5th 504", + "6 N.Y.3d 636", + "623 S.W.3d 717", + "629 F. Supp. 3d 841", + "63 F.4th 1229", + "662 Pa. 672", + "68 Misc. 3d 899", + "7 N.Y.3d 434", + "75 Cal. App. 5th 694", + "81 Misc. 3d 550", + "86 Cal. App. 5th 1094", + "92 A.3d 766", + "973 F.3d 667", + "990 F.3d 457", + "No. 23-2226", + "No. 23-30854", + "No. 24-1743", + "No. 24-1907" + ] +} diff --git a/data/doj-nij-post-pcast-2024.json b/data/doj-nij-post-pcast-2024.json new file mode 100644 index 0000000..4958c4a --- /dev/null +++ b/data/doj-nij-post-pcast-2024.json @@ -0,0 +1,1997 @@ +{ + "dataset": "Post-PCAST Court Decisions Assessing the Admissibility of Forensic Science Evidence", + "catalog_url": "https://catalog.data.gov/dataset/post-pcast-court-decisions-assessing-the-admissibility-of-forensic-science-evidence", + "resource_url": "https://nij.ojp.gov/program/national-center-forensics/post-pcast-court-decisions-assessing-admissibility-forensic-science-evidence", + "publisher": "U.S. Department of Justice, National Institute of Justice", + "license": "Public Domain", + "retrieved_at": "2026-07-14T22:54:22Z", + "source_html_sha256": "1811a7762f961e0e046b2c957717aeebd46f0985b2a8c774a657ad162e723368", + "records_sha256": "90990eb660d45be39515f9f08278abd4b080be3ce758eb1cf9e8854cc5ab95f1", + "record_count": 124, + "records": [ + { + "source_row": 1, + "caption": "State v. Denton, No. 04R-330, 2020 Ga. Super. LEXIS 13 (Super. Ct. Ga. Feb. 7, 2020)", + "jurisdiction": "State", + "court_location": "GA", + "posture": "Appeal", + "discipline": "Bitemark", + "decision_date_raw": "2/7/2020", + "decision_effect": "Remand for new trial", + "outcome": "Def motion granted", + "publication_status": "Lexis", + "description": "Three forensic dentists (including prosecution's former expert) testified that ABFO Guidelines changes in 2016 mean that bitemark evidence presented at trial would not be considered inculpatory.", + "decision_date": "2020-02-07", + "case_citation": null, + "date_mismatch": false + }, + { + "source_row": 2, + "caption": "People v. Prante, 223 N.E. 3d 160 (Ill. 2023)", + "jurisdiction": "State", + "court_location": "IL", + "posture": "Post-conviction", + "discipline": "Bitemark", + "decision_date_raw": "5/18/2023", + "decision_effect": "Affirm admission", + "outcome": "Habeas petition denied", + "publication_status": "Pub", + "description": "Court ruled that the Frye test is a common law evidentiary rule, and is not a constitutional violation on its own. Therefore, petitioner's due process claim against a denial of a Frye hearing could not go forward because at the time of the trial, bitemark analysis was considered reliable and the prosecution did not knowingly provide false testimony. Even absent the bite mark testimony, the State had considerable evidence against Prante in the case and the Supreme Court of Illinois ruled he had no \"colorable claim of actual innocence\" and that the remand for a Frye hearing was reversed.", + "decision_date": "2023-05-18", + "case_citation": "223 N.E.3d 160", + "date_mismatch": false + }, + { + "source_row": 3, + "caption": "State v. Fortin, 464 N.J. Super. 193 (N.J. Super. Ct. App. Div. June 22, 2020)", + "jurisdiction": "State", + "court_location": "NJ", + "posture": "Appeal", + "discipline": "Bitemark", + "decision_date_raw": "6/22/2020", + "decision_effect": "Affirm admission", + "outcome": "Conviction affirmed", + "publication_status": "Lexis", + "description": "Court holds that criticism of bitemark evidence by PCAST Report and other sources is cumulative and notes \"to date no court in the United States has excluded expert testimony on bitemark identification.\"", + "decision_date": "2020-06-22", + "case_citation": "464 N.J. Super. 193", + "date_mismatch": false + }, + { + "source_row": 4, + "caption": "Commonwealth v. Ross, 224 A.3d 789 (Pa. Super. Ct. 2019)", + "jurisdiction": "State", + "court_location": "PA", + "posture": "Appeal", + "discipline": "Bitemark", + "decision_date_raw": "11/21/2019", + "decision_effect": "Remand for admissibility hearing", + "outcome": "Conviction reversed", + "publication_status": "Pub", + "description": "Finding that trial court should have conducted a Frye hearing on admissibility of bitemark evidence. Relying on and quoting from PCAST Report for conclusion that there is evidence indicating \"a lack of consensus among forensic odontologists on whether bite mark identification analysis is reliable and valid.\" Further finding that trial court \"should not have limited consideration of the general acceptance of the experts' methodology . . . to the field of forensic odontology\" given the fact that bitemark analysis involves scientific disciplines beyond that of odontology.", + "decision_date": "2019-11-21", + "case_citation": "224 A.3d 789", + "date_mismatch": false + }, + { + "source_row": 5, + "caption": "Motorola, Inc. v. Murray, 147 A.3d 751 (D.C. Ct. App. 2016)", + "jurisdiction": "State", + "court_location": "DC", + "posture": "Appeal", + "discipline": "Dicta re PCAST", + "decision_date_raw": "10/20/2016", + "decision_effect": "PCAST dicta", + "outcome": "DC state courts adopted FRE 702", + "publication_status": "Pub", + "description": "Non-forensic science case held that DC state courts would no longer use Frye standard. J. Easterly (author of case quoted in PCAST Report) wrote concurrence suggesting that all forensics testimony should be viewed through lens of PCAST Report.", + "decision_date": "2016-10-20", + "case_citation": "147 A.3d 751", + "date_mismatch": false + }, + { + "source_row": 6, + "caption": "People v. Davis, 75 Cal. App. 5th 694 (Cal. Ct. App. 3d Dist. 2022)", + "jurisdiction": "State", + "court_location": "CA", + "posture": "Appeal", + "discipline": "DNA", + "decision_date_raw": "2/28/2022", + "decision_effect": "Affirm admission", + "outcome": "Conviction affirmed", + "publication_status": "Pub", + "description": "Finding that STRmix, which \"has been used for DNA analysis since 2012 and is widely used by forensic laboratories across the world\" is generally accepted as reliable by the relevant scientific community. Brief reference to article responding to PCAST Report.", + "decision_date": "2022-02-28", + "case_citation": "75 Cal. App. 5th 694", + "date_mismatch": false + }, + { + "source_row": 7, + "caption": "Andersen v. City of Chicago, 467 F. Supp. 3d 598 (N.D. Ill. 2020)", + "jurisdiction": "Federal", + "court_location": "IL (N.D. Ill.)", + "posture": "Pretrial", + "discipline": "DNA", + "decision_date_raw": "6/16/2020", + "decision_effect": "Exclude def expert", + "outcome": "Pl's motion granted", + "publication_status": "Pub", + "description": "Post-reversal-of-conviction and actual innocence 1983 case; plaintiff (former criminal defendant)'s motion to exclude City's DNA expert granted. SWGDAM also mentioned. Treats inclusions and exclusions differently.", + "decision_date": "2020-06-16", + "case_citation": "467 F. Supp. 3d 598", + "date_mismatch": false + }, + { + "source_row": 8, + "caption": "Phillips v. State, 152 A.3d 712 (Md. Ct. App. 2017)", + "jurisdiction": "State", + "court_location": "MD", + "posture": "Appeal", + "discipline": "DNA", + "decision_date_raw": "1/20/2017", + "decision_effect": "Affirm admission", + "outcome": "Conviction affirmed", + "publication_status": "Pub", + "description": "Somewhat Maryland-specific analysis of Maryland's DNA admissibility statute, concluding that DNA was properly admitted and that no hearing was required to so determine.", + "decision_date": "2017-01-20", + "case_citation": "152 A.3d 712", + "date_mismatch": false + }, + { + "source_row": 9, + "caption": "Dantzler v. Rewerts, No. 20-1059, 2021 U.S. App. LEXIS 25939 (6th Cir. Aug. 25, 2021)", + "jurisdiction": "Federal", + "court_location": "MI (6th Cir.)", + "posture": "Post-conviction", + "discipline": "DNA", + "decision_date_raw": "8/25/2021", + "decision_effect": "N/A (assuming error, petitioner failed to show prejudice)", + "outcome": "Conviction affirmed", + "publication_status": "Lexis", + "description": "Defendant challenged DNA evidence presented by three Government experts at trial that his DNA was on a hat found at the scene of a murder. Defendant relied on PCAST Report findings that interpretations of DNA profiles depends on the subjective choices made by analysts and that \"[i]t is often impossible to tell with certainty which alleles are present in the mixture or how many separate individuals contributed to the mixture, let alone accurately to infer the DNA profile of each individual.\" Defendant argued that trial court should have permitted him to call an independent DNA analyst to rebut unspecified \"subjective choices\" of Government experts. Court rejected that argument, noting that defendant's characterizations of what an expert might have said were speculative and did not establish level of \"prejudice\" necessary to prevail on habeas review. Note: the Court does not analyze, challenge or accept substance of PCAST Report relied on by defendant or otherwise substantively discuss the content of the Report.", + "decision_date": "2021-08-25", + "case_citation": "2021 U.S. App. LEXIS 25939", + "date_mismatch": false + }, + { + "source_row": 10, + "caption": "U.S. v. Gissantaner, 990 F.3d 457 (6th Cir. 2021) (rev'g 417 F. Supp. 3d 857 (W.D. Mich. 2019))", + "jurisdiction": "Federal", + "court_location": "MI (6th Cir.)", + "posture": "Pretrial", + "discipline": "DNA", + "decision_date_raw": "3/5/2021", + "decision_effect": "Reverse exclusion", + "outcome": "Reversing exclusion of evidence by trial court", + "publication_status": "Pub", + "description": "STRmix; 6th Circuit reversed district court's exclusion of DNA evidence. District court had concluded that \"because the sum of the parts simply does not add up to a reliable whole, the DNA analysis/likelihood ratio resulting from the use of the STRmix probabilistic genotyping software must be excluded.\" 6th Circuit disagreed, concluding that evidence should be admitted and noted, among other things, that one \"independent\" expert appointed by the court had been retained by the defendant on that same case.", + "decision_date": "2021-03-05", + "case_citation": "990 F.3d 457", + "date_mismatch": false + }, + { + "source_row": 11, + "caption": "State v. Simmer, 304 Neb. 369 (Neb. 2019)", + "jurisdiction": "State", + "court_location": "NE", + "posture": "Appeal", + "discipline": "DNA", + "decision_date_raw": "11/1/2019", + "decision_effect": "Affirm admission", + "outcome": "Conviction affirmed", + "publication_status": "Pub", + "description": "TrueAllele DNA properly admitted at trial. Discussion of PCAST Report's conclusions regarding admissibility/reliability of TrueAllele in certain circumstances. Court notes that decision does not require admission of TrueAllele in all future cases.", + "decision_date": "2019-11-01", + "case_citation": "304 Neb. 369", + "date_mismatch": false + }, + { + "source_row": 12, + "caption": "U.S. v. Garcia, No. Cr. 18-1384, 2018 U.S. Dist. LEXIS 150947 (D.N.M. Sept. 5, 2018)", + "jurisdiction": "Federal", + "court_location": "NM (D.N.M.)", + "posture": "Pretrial", + "discipline": "DNA", + "decision_date_raw": "9/5/2018", + "decision_effect": "N/A (ordering defendant to produce DNA sample)", + "outcome": "Gov't motion granted", + "publication_status": "Lexis", + "description": "Defendant sought to prevent taking of his DNA, arguing cognitive bias. Court rejected argument, which was based, in part, on PCAST Report.", + "decision_date": "2018-09-05", + "case_citation": "2018 U.S. Dist. LEXIS 150947", + "date_mismatch": false + }, + { + "source_row": 13, + "caption": "People v. Williams, 35 N.Y.3d 24 (N.Y. 2020)", + "jurisdiction": "State", + "court_location": "NY", + "posture": "Appeal", + "discipline": "DNA", + "decision_date_raw": "3/31/2020", + "decision_effect": "Error to admit, but harmless", + "outcome": "Conviction affirmed", + "publication_status": "Pub", + "description": "Trial court abused discretion as a matter of law in failing to hold Frye hearing on low copy number (LCN) DNA--specifically, OCME's forensic statistical tool (FST) software--but error was harmless. \"Familiarity does not always breed accuracy, and our Frye jurisprudence accounts for the fact that evolving views and opinions in a scientific community may occasionally require the scrutiny of a Frye hearing with respect to a familiar technique. There is no absolute rule as to when a Frye hearing should or should not be granted, and courts should be guided by the current state of scientific knowledge and opinion in making such determinations.\"", + "decision_date": "2020-03-31", + "case_citation": "35 N.Y.3d 24", + "date_mismatch": false + }, + { + "source_row": 14, + "caption": "People v. Wakefield, 195 N.E.3d 19 (N.Y. 2022)", + "jurisdiction": "State", + "court_location": "NY", + "posture": "Appeal", + "discipline": "DNA", + "decision_date_raw": "44677", + "decision_effect": "Affirm admission", + "outcome": "Conviction affirmed", + "publication_status": "Pub", + "description": "Conviction that included testimony re: TrueAllele, following a Frye hearing, affirmed. New York Court of Appeals (the highest court in the state) held (1) trial court did not abuse its discretion in finding that \"TrueAllele's use of the continuous probabilistic genotyping approach to generate a statistical likelihood ratio\u2014including the use of peak data below the stochastic threshold\u2014of a DNA genotype is generally accepted in the relevant scientific community\" and (2) \"there was no error in the court's denial of defendant's request for discovery of the TrueAllele software source code in connection with the Frye hearing or for the purpose of his Sixth Amendment right to confront the witness against him at trial.\"", + "decision_date": "2022-04-26", + "case_citation": "195 N.E.3d 19", + "date_mismatch": false + }, + { + "source_row": 15, + "caption": "People v. Thompson, No. 4346/15, 2019 N.Y. Misc. LEXIS 5162 (N.Y. Sup. Ct. Sept. 25, 2019)", + "jurisdiction": "State", + "court_location": "NY", + "posture": "Pretrial", + "discipline": "DNA", + "decision_date_raw": "9/25/2019", + "decision_effect": "Exclude", + "outcome": "Def motion granted", + "publication_status": "Lexis", + "description": "Advisory opinion, given that defendants had pled guilty and OCME was no longer using DNA software at issue, finding that OCME's FST program was inadmissible under Frye; court acknowledges that NY trial and appellate courts do not agree with its analysis and is critical of some of those opinions. Court writes favorably of PCAST Report.", + "decision_date": "2019-09-25", + "case_citation": null, + "date_mismatch": false + }, + { + "source_row": 16, + "caption": "State v. Watkins, No. M2020-0035-CCA-R3-CD, 2021 Tenn. Crim. App. LEXIS 571 (Tenn. Cr. Crim. App. Dec. 16, 2021)", + "jurisdiction": "State", + "court_location": "TN", + "posture": "Appeal", + "discipline": "DNA", + "decision_date_raw": "12/16/2021", + "decision_effect": "Affirm admission", + "outcome": "Conviction affirmed", + "publication_status": "Lexis", + "description": "In a case of first impression in Tennessee, probabilistic genotyping DNA analysis of a DNA mixture taken from the inside of the pants pocket of a murder victim was challenged by defendant. Defendant argued that probabilistic genotyping and its use of a likelihood ratio was not reliable in this case because of the number of multiple contributors. Defense expert, relying on the PCAST Report, testified that probabilistic genotyping is only reliable for analyzing mixtures of three or fewer DNA contributors and that further studies are necessary to establish propriety of using probabilistic genotyping systems on mixtures containing more than three contributors. Government expert testified that validation studies existed for mixtures containing up to seven contributors. Court rejected defendant's argument, finding that TrueAllele's established mathematical and statistical methodology had been established at extensive pretrial hearings. In addition, analyzing and applying Tenn. Code Section 24-7-118 (regarding admissibility of DNA evidence in general), court finds that probabilistic genotyping is encompassed within the language of the statute and that no threshold admissibility requirement applies to this type of DNA analysis.", + "decision_date": "2021-12-16", + "case_citation": null, + "date_mismatch": false + }, + { + "source_row": 17, + "caption": "In re Arbzadegan, No. 03-20-00292-CR, 2021 Tex. App. LEXIS 9450 (Tex. Ct. App. 3d Dist. Nov. 23, 2021)", + "jurisdiction": "State", + "court_location": "TX", + "posture": "Post-conviction", + "discipline": "DNA", + "decision_date_raw": "11/23/2021", + "decision_effect": "PCAST dicta", + "outcome": "Affirmed trial court denial of post-conviction DNA testing; dicta about PCAST/FRE 702", + "publication_status": "Lexis", + "description": "Upholding trial court's denial of post-conviction DNA testing on ground that defendant failed to meet requirements of Texas post-trial DNA testing statute. In fn7, court makes passing reference to PCAST Report, and specifically notes that Report found CPI--Combined Probability of Inclusion--method for analyzing mixtures of DNA is foundationally invalid. However, court notes that CPI was not used for statistical analysis in this case and would therefore have no affect on outcome of the court's ruling.", + "decision_date": "2021-11-23", + "case_citation": null, + "date_mismatch": false + }, + { + "source_row": 18, + "caption": "U.S. v. Oldman, No. 18CR-0020, 2018 U.S. Dist. LEXIS 232762 (D. Wyo. Dec. 31, 2018)", + "jurisdiction": "Federal", + "court_location": "WY (D. Wyo.)", + "posture": "Pretrial", + "discipline": "DNA", + "decision_date_raw": "12/31/2018", + "decision_effect": "Admit", + "outcome": "Def motion denied", + "publication_status": "Lexis", + "description": "Finding that results using STRmix DNA analysis are \"based upon a tested and valid scientific methodology\" that has been tested and peer reviewed so as to ensure its reliability. Decision extensively reviews scientific literature, including PCAST Report (court quotes Report, but does not rely on it). Testimony admitted without limitation.", + "decision_date": "2018-12-31", + "case_citation": "2018 U.S. Dist. LEXIS 232762", + "date_mismatch": false + }, + { + "source_row": 19, + "caption": "State v. Baugh, No. 2017-CR-618, 2019 Ga. Super. LEXIS 418 (Ga. Super. Ct. Apr. 29, 2019)", + "jurisdiction": "State", + "court_location": "GA", + "posture": "Pretrial", + "discipline": "DNA", + "decision_date_raw": "4/29/2019", + "decision_effect": "Admit", + "outcome": "Def motion denied", + "publication_status": "Lexis", + "description": "Court affirms admission of expert testimony regarding TrueAllele DNA analysis, applying Harper admissibility test (Georgia's standard; appears to be modified from Frye). Listing states that have admitted TrueAllele and cases deciding same.", + "decision_date": "2019-04-29", + "case_citation": null, + "date_mismatch": false + }, + { + "source_row": 20, + "caption": "State v. Robinson, No. 62-CR-15-935/ State v. Blackstone, No. 62-CR-16-45 (Minn. D.Ct. May 4, 2018) (unpublished)", + "jurisdiction": "State", + "court_location": "MN", + "posture": "Pretrial", + "discipline": "DNA", + "decision_date_raw": "5/4/2018", + "decision_effect": "Admit/Limit", + "outcome": "Def motion denied", + "publication_status": "Unpub", + "description": "Court determines that DNA interpretation is foundationally based in science and is relevant to the firearms charge. Lengthy discussion of PCAST Report.", + "decision_date": "2018-05-04", + "case_citation": null, + "date_mismatch": false + }, + { + "source_row": 21, + "caption": "U.S. v. Lewis, 442 F. Supp. 3d 1122 (D. Minn. 2020)", + "jurisdiction": "Federal", + "court_location": "MN (D. Minn.)", + "posture": "Pretrial", + "discipline": "DNA", + "decision_date_raw": "3/3/2020", + "decision_effect": "Admit/Exclude", + "outcome": "Def motion granted in part and denied in part", + "publication_status": "Pub", + "description": "STRmix DNA case; defendant argued STRmix statistics are unreliable; court finds STRmix sufficiently reliable as to evidence about inclusion of defendant as a contributor, but not as to exclusion of police officers and other witness as contributors. Adoption of lengthy Magistrate Judge's Report & Recommendation, which is at 2020 U.S. Dist. LEXIS 38705.", + "decision_date": "2020-03-03", + "case_citation": "442 F. Supp. 3d 1122", + "date_mismatch": false + }, + { + "source_row": 22, + "caption": "U.S. v. Washington, No. 8:19CR299, 2020 U.S. Dist. LEXIS 105447 (D. Neb. June 16, 2020)", + "jurisdiction": "Federal", + "court_location": "NE (D. Neb.)", + "posture": "Pretrial", + "discipline": "DNA", + "decision_date_raw": "6/16/2020", + "decision_effect": "Admit", + "outcome": "Def motion denied", + "publication_status": "Lexis", + "description": "STRmix case; defendant argued STRmix statistics are unreliable and that there were issues with chain of custody; court finds STRmix sufficiently reliable and any issues go to weight, not admissibility. Regarding defendant's reliance on PCAST Report to cast doubt on STRmix, court distinguishes the Report, noting that the percentage of the defendant's DNA in samples at issue \"are well above the 20% threshold at which the PCAST Report raised concern.\"", + "decision_date": "2020-06-16", + "case_citation": "2020 U.S. Dist. LEXIS 105447", + "date_mismatch": false + }, + { + "source_row": 23, + "caption": "State v. Pickett, 466 N.J. Super. 270 (N.J. Super. Ct. App. Div. 2021)", + "jurisdiction": "State", + "court_location": "NJ", + "posture": "Appeal", + "discipline": "DNA", + "decision_date_raw": "2/3/2021", + "decision_effect": "N/A (ordering production of TrueAllele source code and other proprietary information)", + "outcome": "Def motion granted", + "publication_status": "Pub", + "description": "Defendant's motion for source code and supporting software development and related documentation granted (with a protective order) to allow use in Frye hearing to determine admissibility of DNA expert's testimony; TrueAllele probabilistic genotyping software at issue; a number of amici filed briefs. Court cites PCAST Report's argument \"that probabilistic genotyping programs should be independently evaluated to determine whether the methods are scientifically valid and, importantly, whether the software itself correctly implements the methods.\"", + "decision_date": "2021-02-03", + "case_citation": "466 N.J. Super. 270", + "date_mismatch": false + }, + { + "source_row": 24, + "caption": "U.S. v. Tucker, No. 18 Cr. 0119 (SJ), 2020 U.S. Dist. LEXIS 3055 (E.D.N.Y. Jan. 8, 2020)", + "jurisdiction": "Federal", + "court_location": "NY (E.D.N.Y.)", + "posture": "Pretrial", + "discipline": "DNA; & FTM", + "decision_date_raw": "1/8/2020", + "decision_effect": "Admit", + "outcome": "Def motion denied", + "publication_status": "Lexis", + "description": "FRE 702 analysis; court permits firearms and toolmark examiner to testify that bullets came from three different guns, noting that those are class characteristics; STRmix DNA analysis for 2-person mix is reliable (cites other cases that came to same conclusion).", + "decision_date": "2020-01-08", + "case_citation": "2020 U.S. Dist. LEXIS 3055", + "date_mismatch": false + }, + { + "source_row": 25, + "caption": "U.S. v. Reyes-Ballista, No. 18-634-2, 2020 U.S. Dist. LEXIS 218249 (D.P.R. Nov. 20, 2020)", + "jurisdiction": "Federal", + "court_location": "PR (D.P.R.)", + "posture": "Pretrial", + "discipline": "Fingerprints", + "decision_date_raw": "11/20/2020", + "decision_effect": "Admit", + "outcome": "Def motion denied", + "publication_status": "Lexis", + "description": "Defendant challenged validity and accuracy of ACE-V method of fingerprint examination based on PCAST and NAS Reports. Court denied motion to exclude after Daubert hearing. Court notes the \"overwhelming caselaw standing for the proposition that fingerprint evidence is reliable enough for jury trials as a helpful form of identification testimony.\"", + "decision_date": "2020-11-20", + "case_citation": "2020 U.S. Dist. LEXIS 218249", + "date_mismatch": false + }, + { + "source_row": 26, + "caption": "U.S. v. Hendrix, No. CR19-0024JLR, 2020 U.S. Dist. LEXIS 331 (W.D. Wa. Jan. 2, 2020)", + "jurisdiction": "Federal", + "court_location": "WA (W.D. Wash.)", + "posture": "Pretrial", + "discipline": "Fingerprints", + "decision_date_raw": "1/2/2020", + "decision_effect": "Exclude def expert", + "outcome": "Gov't motion granted", + "publication_status": "Lexis", + "description": "Government's fingerprint witness permitted to testify in previous oral decision after Daubert hearing. This decision focuses on defendant's efforts to introduce testimony of defense expert about error rates and PCAST Report; government's motion to exclude granted because, among other things, defense expert's opinions are \"not helpful to the jury and are overly reliant on hearsay.\" Court allows cross-examination of Government's expert regarding PCAST Report.", + "decision_date": "2020-01-02", + "case_citation": "2020 U.S. Dist. LEXIS 331", + "date_mismatch": false + }, + { + "source_row": 27, + "caption": "U.S. v. Casaus, 279 F. Supp. 3d 1109 (D. Col. 2018)", + "jurisdiction": "Federal", + "court_location": "CO (D. Colo.)", + "posture": "Appeal", + "discipline": "Fingerprints", + "decision_date_raw": "1/4/2018", + "decision_effect": "Exclude", + "outcome": "Def motion granted", + "publication_status": "Pub", + "description": "Lower court denies Daubert challenge to fingerprint comparisons, noting that 10th Circuit precedent that \"fingerprint comparison is a reliable method to identify persons\" undercuts PCAST Report's criticism of latent fingerprint analysis. Appellate court did not comment on the admissibility of Fingerprint evidence as decided in previous case history, however, they did dismiss the case with prejudice after finding defendant's speedy trial rights had been violated.", + "decision_date": "2018-01-04", + "case_citation": "279 F. Supp. 3d 1109", + "date_mismatch": false + }, + { + "source_row": 28, + "caption": "U.S. v. Kimble, No. CR418-026, 2018 U.S. Dist. LEXIS 138988 (S.D. Ga. Aug. 16, 2018)", + "jurisdiction": "Federal", + "court_location": "GA (S.D. Ga.)", + "posture": "Pretrial", + "discipline": "Fingerprints", + "decision_date_raw": "8/16/2018", + "decision_effect": "Admit", + "outcome": "Def motion denied", + "publication_status": "Lexis", + "description": "Defendant's challenge to ACE-V methodology based on PCAST Report and NAS Report. As to PCAST addendum on ACE-V, Court notes that report found ACE-V to be both \"scientifically valid\" and reliable and criticizes selective quoting from PCAST Report. As to NAS, Court notes that any criticisms of ACE-V methodology goes to weight and not admissibility of testimony.", + "decision_date": "2018-08-16", + "case_citation": "2018 U.S. Dist. LEXIS 138988", + "date_mismatch": false + }, + { + "source_row": 29, + "caption": "U.S. v. Bonds, No. 15 CR 573-2, 2017 U.S. Dist. LEXIS 166975 (N.D. Ill. Oct. 10, 2017)", + "jurisdiction": "Federal", + "court_location": "IL (N.D. Ill.)", + "posture": "Pretrial", + "discipline": "Fingerprints", + "decision_date_raw": "10/10/2017", + "decision_effect": "Admit", + "outcome": "Def motion denied", + "publication_status": "Lexis", + "description": "Court denies defendant's request that Government comply with PCAST Report's advisory recommendations in determining whether latent fingerprint analysis had been reliably applied. Court notes that defendant's concerns about the Government expert's application of ACE-V method can be explored on cross-examination. Court precludes defendant from cross-examining on fingerprint misidentification in Mayfield case, citing U.S. v. Rivas, 832 F.3d 931 (7th Cir. 2016).", + "decision_date": "2017-10-10", + "case_citation": "2017 U.S. Dist. LEXIS 166975", + "date_mismatch": false + }, + { + "source_row": 30, + "caption": "U.S. v. Casanova, 886 F.3d 55 (1st Cir. 2018)", + "jurisdiction": "Federal", + "court_location": "MA (1st Cir.)", + "posture": "Appeal", + "discipline": "Fingerprints", + "decision_date_raw": "3/26/2018", + "decision_effect": "Affirm admission", + "outcome": "Conviction affirmed", + "publication_status": "Pub", + "description": "Court denies challenge to fingerprint examination on the basis of PCAST Report, finding that the Report, which post-dated the conviction, was not properly before the court and that the defendant had mischaracterized the content of the Report.", + "decision_date": "2018-03-26", + "case_citation": "886 F.3d 55", + "date_mismatch": false + }, + { + "source_row": 31, + "caption": "State v. Monell, No. A-4499, 2021 N.J. Super. Unpub. LEXIS 2808 (N.J. Super. Ct. App. Div. Nov 15, 2021) (per curiam)", + "jurisdiction": "State", + "court_location": "NJ", + "posture": "Appeal", + "discipline": "Fingerprints", + "decision_date_raw": "11/15/2021", + "decision_effect": "Affirm admission", + "outcome": "Conviction affirmed", + "publication_status": "Lexis", + "description": "On appeal of his murder conviction, defendant challenged trial court's decision not to hold a Frye hearing on (1) the admissibility of expert testimony that defendant's bloody palm print was on a knife found at the crime scene and; (2) his challenge to the methodology--ACE-V--employed by the expert. Defendant argued that ACE-V is scientifically invalid based on two studies cited in the PCAST Report regarding false positive rates for latent fingerprint matches. Appellate court found that pre-trial testimony by Government expert was sufficient to establish that ACE-V is generally accepted standard in the scientific community. As to the PCAST Report, because the defendant failed to introduce testimony by a witness who had knowledge of the content of the Report, or the studies referred to in the Report, the trial court had no basis to rule that ACE-V is unreliable. Moreover, as the defendant conceded, PCAST Report relates to \"latent\" fingerprint analysis, and not \"patent\" prints, such as the bloody palm print at issue in this case. The court declined to address the defendant's contention that the expert overstated the strength of his conclusion based on SWGFAST standards, noting that this issue had not been raised at trial and defendant could have, but did not, cross-examine the expert on this issue.", + "decision_date": "2021-11-15", + "case_citation": null, + "date_mismatch": false + }, + { + "source_row": 32, + "caption": "U.S. v. Cantoni, No. 18-cr-562 (ENV), 2019 U.S. Dist. LEXIS 45116 (E.D.N.Y. Mar. 19, 2019)", + "jurisdiction": "Federal", + "court_location": "NY (E.D.N.Y.)", + "posture": "Pretrial", + "discipline": "Fingerprints", + "decision_date_raw": "3/19/2019", + "decision_effect": "Limit/Denied", + "outcome": "Def motions to preclude and for expert denied; def motion to limit granted without gov't opp", + "publication_status": "Lexis", + "description": "Defendant's challenge to ACE-V methodology based on PCAST Report denied, with court noting that PCAST Report concluded that ACE-V, though not infallible, is reliable. As to error rates in other studies, Court concludes that this goes to weight, not admissibility, of expert testimony, and can be adequately explored through cross-examination of government's expert. Proposed defense expert cannot testify about these studies as that would constitute inadmissible hearsay.", + "decision_date": "2019-03-19", + "case_citation": "2019 U.S. Dist. LEXIS 45116", + "date_mismatch": false + }, + { + "source_row": 33, + "caption": "U.S. v. Pitts, No. 16-Cr 550 (DLI), 2018 U.S. Dist. LEXIS 34552 (E.D.N.Y. Mar. 2, 2018)", + "jurisdiction": "Federal", + "court_location": "NY (E.D.N.Y.)", + "posture": "Pretrial", + "discipline": "Fingerprints", + "decision_date_raw": "3/2/2018", + "decision_effect": "Exclude def expert", + "outcome": "Gov't motion granted", + "publication_status": "Lexis", + "description": "Finding that proposed defense expert's testimony based on PCAST Report regarding fingerprint analysis would not be helpful to the trier of fact because (1) it would be offered only to rebut opinions that would not be elicited from the Government's expert; and (2) to the extent that proposed defense expert's opinions largely relied on reports authored by others, defendant could cross the Government's experts about those reports and proposed defense expert's testimony in this regard is therefore unnecessary. Opinion cites to other cases excluding specific proposed defense expert's testimony. Prior decision in same case, 2018 U.S. Dist. LEXIS 30589, discusses at length weight versus admissibility of expert testimony regarding fingerprint analysis and denies motion to preclude handwriting expert, distinguishing handwriting expert testimony regarding forgeries from testimony comparing non-forged handwriting samples.", + "decision_date": "2018-03-02", + "case_citation": "2018 U.S. Dist. LEXIS 34552", + "date_mismatch": false + }, + { + "source_row": 34, + "caption": "State v. Johnson, 2022-Ohio-1739 (Ohio Ct. App. 2022)", + "jurisdiction": "State", + "court_location": "OH", + "posture": "Appeal", + "discipline": "Fingerprints", + "decision_date_raw": "5/25/2022", + "decision_effect": "Remand for hearing", + "outcome": "Remanded", + "publication_status": "Pub", + "description": "Fingerprint evidence introduced, after Daubert hearing, through witness who used ACE-V method, uploaded prints to AFIS, and then compared prints returned by AFIS manually. Work was verified by second examiner. Expert testified she was \"100 percent sure\" that a print belonged to the defendant. Court of appeals had affirmed conviction on direct appeal. In this decision, which addressed his request for post-conviction relief, same court cited NAS and PCAST Reports, including PCAST's suggested jury instruction in latent print cases. Court remanded for a hearing, suggesting that expert likely erred by testifying that latent fingerprint identification was not subjective and by testifying to 100 percent certainty, and holding that the trial court erred by not holding a hearing to address the merits of the petition.", + "decision_date": "2022-05-25", + "case_citation": "2022-Ohio-1739", + "date_mismatch": false + }, + { + "source_row": 35, + "caption": "U.S. v. Fell, No. 5:01-CRcr-12-01, 2016 U.S. Dist. LEXIS 198714 (D. Vt. Dec. 29, 2016)", + "jurisdiction": "Federal", + "court_location": "VT (D. Vt.)", + "posture": "Pretrial", + "discipline": "Fingerprints", + "decision_date_raw": "12/29/2016", + "decision_effect": "Admit", + "outcome": "Def motion denied", + "publication_status": "Lexis", + "description": "Short opinion denying defendant's motion, based on PCAST Report, for reconsideration of court's denial of motion to exclude fingerprint evidence. Court notes that \"fingerprint identification remains sufficiently reliable to meet Daubert standards for admissibility. . . . [and] [t]he PCAST Report supports this view.\"", + "decision_date": "2016-12-29", + "case_citation": "2016 U.S. Dist. LEXIS 198714", + "date_mismatch": false + }, + { + "source_row": 36, + "caption": "U.S. v. Lundi, No. 17-CR-388, 2018 U.S. Dist. LEXIS 114796 (E.D.N.Y. July 10, 2018)", + "jurisdiction": "Federal", + "court_location": "NY (E.D.N.Y.)", + "posture": "Pretrial", + "discipline": "Fingerprints", + "decision_date_raw": "7/10/2018", + "decision_effect": "Admit", + "outcome": "Def motion denied; def expert excluded", + "publication_status": "Lexis", + "description": "Defendant challenged Government fingerprint expert based on PCAST Report. Court rejected challenge, noting that defendant could cross-examine based on the Report and that the Report's findings went to the weight and not admissibility of expert's testimony. Court also precluded testimony of defendant's expert.", + "decision_date": "2018-07-10", + "case_citation": "2018 U.S. Dist. LEXIS 114796", + "date_mismatch": false + }, + { + "source_row": 37, + "caption": "State v. Patel, No. LLICR130143598S, 2016 Conn. Super. LEXIS 3440 (Conn. Super. Ct. Dec. 28, 2016)", + "jurisdiction": "State", + "court_location": "CT", + "posture": "Pretrial", + "discipline": "Footwear", + "decision_date_raw": "12/28/2016", + "decision_effect": "Admit", + "outcome": "Def motion denied", + "publication_status": "Lexis", + "description": "Summary of general acceptance of footwear comparison testimony. Court determines no need for a hearing. Discusses limitations of PCAST Report (*26) and jury's ability to use its \"own powers of observation and physical comparison\" when assessing comparison-based forensics evidence.", + "decision_date": "2016-12-28", + "case_citation": null, + "date_mismatch": false + }, + { + "source_row": 38, + "caption": "Hughes v. State, No. 18A-CR-1007, 2019 Ind. App. Unpub. LEXIS 581 (Ind. Ct. App. May 14, 2019)", + "jurisdiction": "State", + "court_location": "IN", + "posture": "Post-conviction", + "discipline": "Footwear", + "decision_date_raw": "5/14/2019", + "decision_effect": "Limit", + "outcome": "Affirmed admission of footwear testimony as that of \"skilled\" (not expert) witness", + "publication_status": "Lexis", + "description": "Court draws and discusses distinction between \"skilled\" and \"expert\" footwear comparison witness and cautions trial court regarding jury instruction treating \"skilled\" witness as \"expert\" witness. Court holds that regardless of whether PCAST Report itself was admissible, cross-examination regarding the Report was permissible.", + "decision_date": "2019-05-14", + "case_citation": null, + "date_mismatch": false + }, + { + "source_row": 39, + "caption": "Brown v. State, 512 P.3d 269 (Nev. 2022)", + "jurisdiction": "State", + "court_location": "NV", + "posture": "Appeal", + "discipline": "Footwear", + "decision_date_raw": "6/23/2022", + "decision_effect": "Affirm admission", + "outcome": "Conviction affirmed", + "publication_status": "Pub", + "description": "State Supreme Court affirmed admission of footwear impression evidence without the testimony of an expert. During closing, prosecutor suggested that jury could compare defendant's footwear with the impression itself. Citing (in FN 7) USDOJ's 2021 Statement on the PCAST Report, appellate court noted that USDOJ \"has since rejected key components of that report\" and declined to consider the report because \"this issue may be resolved through existing caselaw.\" Court surveys caselaw in other states, noting that courts have \"come to differing conclusions\" regarding whether expert testimony is necessary in footwear impression cases. Court then concludes that jurors may make personal observations and draw general inferences about similarities between footwear and footwear impressions, generally without expert testimony, but notes (in FN 9) that expert testimony might be necessary in some footwear cases.", + "decision_date": "2022-06-23", + "case_citation": "512 P.3d 269", + "date_mismatch": false + }, + { + "source_row": 40, + "caption": "U.S. v. Tibbs, No. 2016-CF1-19431, 2019 D.C. Super. LEXIS 9 (D.C. Super. Ct. Sept. 5, 2019)", + "jurisdiction": "State", + "court_location": "DC", + "posture": "Pretrial", + "discipline": "FTM", + "decision_date_raw": "9/5/2019", + "decision_effect": "Limit (cannot be excluded)", + "outcome": "Def motion granted in part and denied in part", + "publication_status": "Lexis", + "description": "Limits firearm/toolmark expert to \"testify[ing] that based on his examination, the recovered firearm cannot be excluded as the source of the cartridge casing found on the scene of the alleged shooting.\" Lengthy analysis of PCAST Report and caselaw.", + "decision_date": "2019-09-05", + "case_citation": null, + "date_mismatch": false + }, + { + "source_row": 41, + "caption": "Merritt v. Arizona, No. CV-17-04540-PHX-DGC, 2020 U.S. Dist. LEXIS 152011 (D. Az. Aug. 21, 2020)", + "jurisdiction": "Federal (civil 1983 by defendant arrested by state)", + "court_location": "AZ (D. Ariz.)", + "posture": "Pretrial", + "discipline": "FTM", + "decision_date_raw": "8/21/2020", + "decision_effect": "Exclude", + "outcome": "Def & pls' motions granted in part and denied in part", + "publication_status": "Lexis", + "description": "Opinion-by-opinion analysis of whether experts can testify about various FTM conclusions; procedural posture is civil case brought alleging false arrest, malicious prosecution, etc. The court excluded proposed testimony by the plaintiff's expert, who would have opined that the plaintiff's weapon did not fire the bullets at issue in the case.", + "decision_date": "2020-08-21", + "case_citation": "2020 U.S. Dist. LEXIS 152011", + "date_mismatch": false + }, + { + "source_row": 42, + "caption": "Spears v. Ryan, No. CV-00-01051-PHX, 2016 US Dist. LEXIS 157897 (D. Ariz. Nov. 16, 2016)", + "jurisdiction": "Federal; (habeas)", + "court_location": "AZ (D. Ariz.) (but state crim case)", + "posture": "Post-conviction", + "discipline": "FTM", + "decision_date_raw": "11/15/2016", + "decision_effect": "N/A (challenge procedurally defaulted)", + "outcome": "Habeas petition denied", + "publication_status": "Lexis", + "description": "Defendant cited findings of 2009 NAS Report criticizing toolmark analysis as \"newly discovered evidence\" justifying habeas review. Court rejected that argument, noting that such criticism preceded issuance of NAS Report. Analysis may parallel that addressing PCAST motions.", + "decision_date": "2016-11-15", + "case_citation": null, + "date_mismatch": false + }, + { + "source_row": 43, + "caption": "People v. Perez, No. B284669, 2019 Cal. App. Unpub. LEXIS 4165 (Cal. Ct. App. June 20, 2019)", + "jurisdiction": "State", + "court_location": "CA", + "posture": "Appeal", + "discipline": "FTM", + "decision_date_raw": "6/20/2019", + "decision_effect": "Affirm admission", + "outcome": "Conviction affirmed", + "publication_status": "Lexis", + "description": "Court rejects argument that trial court erred in failing to hold pre-trial hearing on reliability of \"magazine lip mark comparison evidence,\" finding that such analysis is not quantitatively different from other reliable firearms toolmarks analysis. Court also rejects argument that trial court erred in prohibiting detailed examination of Government's experts based on content of PCAST and other reports. In any event, any error in both regards was \"manifestly harmless,\" given overwhelming evidence that defendant was the shooter. Note: This decisions has been noted as not citable by the California Court of Appeals.", + "decision_date": "2019-06-20", + "case_citation": null, + "date_mismatch": false + }, + { + "source_row": 44, + "caption": "People v. Therman, No. C091147, 2021 Cal. App. Unpub LEXIS 6563 (Cal. Ct. App. Oct. 19, 2021)", + "jurisdiction": "State", + "court_location": "CA", + "posture": "Appeal", + "discipline": "FTM", + "decision_date_raw": "10/19/2021", + "decision_effect": "Affirm admission", + "outcome": "Conviction affirmed", + "publication_status": "Lexis", + "description": "Defendant appealed denial by trial court of his motion to exclude opinion of Government expert that firearm seized from him had fired a cartridge casing recovered at murder scene. Relying on PCAST Report, defendant argued that firearm comparison methods are \"subjective\" and criteria for making an identification is \"circular.\" On appeal, Court found that the PCAST Report provided \"compelling evidence that a credible body found cause for concern about firearm comparison testimony. . . [and] casts doubt on the reliability of firearms comparison techniques and undermines [the Government expert's] conclusion that the cartridge casing recovered from the scene was fired from the gun found on defendant.\" However, although noting that the PCAST Report is \"undeniably credible and concerning,\" it falls short of establishing that a \"clear majority\" of the relevant scientific community no longer accepts firearm toolmark comparison as reliable. The Court also notes, in fn7, that the PCAST Report's concern over the absence of black-box studies has been alleviated by several such studies that have been completed since the Report's publication. Finally, the appellate court rejected the defendant's argument that the state Kelly standard should be replaced with the federal Daubert standard in determining the reliability of FTM analysis. Note: This decisions has been noted as not citable by the California Court of Appeals.", + "decision_date": "2021-10-19", + "case_citation": null, + "date_mismatch": false + }, + { + "source_row": 45, + "caption": "People v. Azcona, 58 Cal.App.5th 504 (Cal. Ct. App. 2020)", + "jurisdiction": "State", + "court_location": "CA", + "posture": "Appeal", + "discipline": "FTM", + "decision_date_raw": "12/10/2020", + "decision_effect": "Reverse conviction", + "outcome": "Conviction reversed", + "publication_status": "Pub", + "description": "Conviction reversed because trial court \"committed multiple errors related to firearms expert testimony\" including permitting testimony about (1) conclusions not supported by evidence, including that there was a conclusive match, and (2) hearsay statements of other examiners who reviewed and approved his conclusions; firearms evidence is not per se inadmissible under Frye/Kelly.", + "decision_date": "2020-12-10", + "case_citation": "58 Cal. App. 5th 504", + "date_mismatch": false + }, + { + "source_row": 46, + "caption": "U.S. v. Johnson, 875 F.3d 1265 (9th Cir. 2017)", + "jurisdiction": "Federal", + "court_location": "CA (9th Cir.)", + "posture": "Appeal", + "discipline": "FTM", + "decision_date_raw": "11/17/2017", + "decision_effect": "Affirm admission", + "outcome": "Conviction affirmed", + "publication_status": "Pub", + "description": "Ninth Circuit holds trial court properly admitted testimony of Government FTM expert that test-fired bullet matched bullet recovered from crime scene \"to a reasonable degree of ballistics certainty\" and did not abuse its discretion in denying motion to exclude expert's testimony, which was based on AFTE methodology.", + "decision_date": "2017-11-17", + "case_citation": "875 F.3d 1265", + "date_mismatch": false + }, + { + "source_row": 47, + "caption": "U.S. v. Chavez, No. 15-CR-00285-LHK-1, 2021 US Dist. LEXIS 237830 (N.D. Cal. Dec. 13, 2021)", + "jurisdiction": "Federal", + "court_location": "CA (N.D. Cal.)", + "posture": "Pretrial", + "discipline": "FTM", + "decision_date_raw": "12/13/2021", + "decision_effect": "Admit with gov't's proposed limits", + "outcome": "Def motion denied", + "publication_status": "Lexis", + "description": "Analyzing challenge to admissibility of firearms and toolmarks evidence under Daubert standard, court finds that evidence is admissible and notes that \"there is no question on the record that even including the non-firearm experts from the NRC [2009 NAS] and PCAST reports, as well as defense experts, there is still an overwhelming acceptance in the United States and worldwide of firearms identification methodology.\"", + "decision_date": "2021-12-13", + "case_citation": null, + "date_mismatch": false + }, + { + "source_row": 48, + "caption": "State v. Raynor, 337 Conn. 527 (Conn. 2020).", + "jurisdiction": "State", + "court_location": "CT", + "posture": "Appeal", + "discipline": "FTM", + "decision_date_raw": "12/4/2020", + "decision_effect": "Remand for admissibility hearing", + "outcome": "Conviction reversed", + "publication_status": "Pub", + "description": "Finding that trial court erred in denying defendant's motion for an evidentiary hearing on the issue of the admissibility of firearms and toolmark testimony and finding that such error was not harmless. Court notes that reliability of such testimony has been called into question by NAS and PCAST Reports and that trial court should have considered such evidence in performing its gatekeeping function on the admissibility of firearms and toolmark evidence and its decision to permit a pretrial evidentiary hearing on such matters. Court affirms trial court's denial of defense motion to limit scope of expert's testimony to \"more likely than not\" standard.", + "decision_date": "2020-12-04", + "case_citation": "337 Conn. 527", + "date_mismatch": false + }, + { + "source_row": 49, + "caption": "State v. Terrell, No. CR170179563, 2019 Conn. Super. LEXIS 827 (Conn. Super. Ct. Mar. 21, 2019)", + "jurisdiction": "State", + "court_location": "CT", + "posture": "Pretrial", + "discipline": "FTM", + "decision_date_raw": "3/21/2019", + "decision_effect": "Limit (\"common origin,\" but not \"practical impossibility\" of another gun shot)", + "outcome": "Def motion denied", + "publication_status": "Lexis", + "description": "FTM \"common origins\" testimony, i.e., that recovered casing was fired from the gun found near the defendant, is permitted after hearing, but expert is precluded from testifying that the \"likelihood that a firearm other than [the recovered firearm] could have fired the [recovered shell casing] is so remote as to be considered a practical impossibility.\" Court's conclusion as to the former is based on the fact that the State presented expert testimony as to the validity of the \"common origins\" methodology and the defense failed to present its own witness. Court appears critical of \"common origins\" methodology. Discussion/analysis of NRC [2009 NAS] and PCAST Reports and caselaw.", + "decision_date": "2019-03-21", + "case_citation": null, + "date_mismatch": false + }, + { + "source_row": 50, + "caption": "Williams v. U.S., 210 A.3d 734 (D.C. 2019)", + "jurisdiction": "State", + "court_location": "DC", + "posture": "Appeal", + "discipline": "FTM", + "decision_date_raw": "6/27/2019", + "decision_effect": "Error to admit, but harmless", + "outcome": "Conviction affirmed (harmless error)", + "publication_status": "Pub", + "description": "Easterly, J., granted defendant's petition for rehearing after decision in Williams v. U.S., 130 A.3d 343 (D.C. Ct. App. Jan. 21, 2016), finding that admission of expert's testimony was plain error because of change from Frye to Daubert standard as articulated in the decisions in Motorola, Inc. v. Murray, 147 A.3d 752 (D.C. Ct. App. 2014) (en banc) and Gardner v. United States, 140 A.3d 1172 (D.C. Ct. App. 2016), but error was harmless. Makes clear that firearms/toolmark evidence is admissible, but it is error to allow an examiner to say bullet came from specific gun; notes data may exist in future to allow for such testimony. Defendant's conviction is affirmed because, in light of the strength of the other evidence, he could not show that his substantial rights were affected by the error.", + "decision_date": "2019-06-27", + "case_citation": "210 A.3d 734", + "date_mismatch": false + }, + { + "source_row": 51, + "caption": "U.S. v. Harris, 502 F. Supp. 3d 28 (D.D.C. 2020)", + "jurisdiction": "Federal", + "court_location": "DC (D.D.C.)", + "posture": "Pretrial", + "discipline": "FTM", + "decision_date_raw": "11/4/2020", + "decision_effect": "Admit with gov'ts proposed limits", + "outcome": "Def motion denied", + "publication_status": "Pub", + "description": "Defendant's motion to exclude testimony of FTM expert denied; \"advancements in the field in the four years since the PCAST Report address many of [the defendant's] concerns.\" Gov't agrees to follow ULTR--to not use the term \"match\" or \"to the exclusion of all others,\" or reference any level of statistical certainty. Court also cites and discusses post-PCAST Report black box studies with low error rates.", + "decision_date": "2020-11-04", + "case_citation": "502 F. Supp. 3d 28", + "date_mismatch": false + }, + { + "source_row": 52, + "caption": "Gardner v. U.S., 140 A.3d 1172 (D.C. Ct. App. 2016)", + "jurisdiction": "State", + "court_location": "DC", + "posture": "Appeal", + "discipline": "FTM", + "decision_date_raw": "6/23/2016", + "decision_effect": "Affirm admission", + "outcome": "Conviction affirmed (harmless error)", + "publication_status": "Pub", + "description": "Pre-PCAST Report, but cited in 2019 Williams case (see above); plain error to allow expert to testify that particular gun shot recovered bullet: \"in this jurisdiction a firearms and toolmark expert may not give an unqualified opinion, or testify with absolute or 100% certainty, that based on ballistics pattern comparison matching a fatal shot was fired from one firearm, to the exclusion of all other firearms.\u201d Error, however, was harmless due to strength of other evidence of guilt.", + "decision_date": "2016-06-23", + "case_citation": "140 A.3d 1172", + "date_mismatch": false + }, + { + "source_row": 53, + "caption": "State v. Gibbs, No. 1809003017, 2019 Del. Super. LEXIS 639 (Del. Super. Ct. Dec. 9, 2019)", + "jurisdiction": "State", + "court_location": "DE", + "posture": "Pretrial", + "discipline": "FTM", + "decision_date_raw": "12/9/2019", + "decision_effect": "Limit (may say \"match,\" but not 100% certainty/ to the exclusion of all others)", + "outcome": "Def motion denied", + "publication_status": "Lexis", + "description": "Without objection by the State, trial court limits testimony of FTM expert: \"[t]he expert is precluded from testifying to being 100% certain as to his findings [and] if he testifies to a 'match,' the expert may not testify to conclusions that suggest there is a match to 'the exclusion of all other firearms in the world' or that it is a 'practical impossibility' that any other gun could have fired the recovered material. He may not testify within a reasonable degree of 'scientific' certainty and may not state his conclusions regarding a 'match' with any degree of certainty.\"", + "decision_date": "2019-12-09", + "case_citation": null, + "date_mismatch": false + }, + { + "source_row": 54, + "caption": "U.S. v. Brown, 973 F.3d 667 (7th Cir. 2020)", + "jurisdiction": "Federal", + "court_location": "IL (7th Cir.)", + "posture": "Appeal", + "discipline": "FTM", + "decision_date_raw": "8/28/2020", + "decision_effect": "Affirm admission", + "outcome": "Conviction affirmed", + "publication_status": "Pub", + "description": "Seventh Circuit affirms conviction predicated on FTM testimony, noting that \"[t]he defendants brought the PCAST Report to the district court's attention, but the district court chose not to give it dispositive effect, and that choice was within its set of options.\"", + "decision_date": "2020-08-28", + "case_citation": "973 F.3d 667", + "date_mismatch": false + }, + { + "source_row": 55, + "caption": "U.S. v. Chester, No. 13 Cr. 774 (N.D. Ill. Oct. 7, 2016) (unpublished)", + "jurisdiction": "Federal", + "court_location": "IL (N.D. Ill.)", + "posture": "Pretrial", + "discipline": "FTM", + "decision_date_raw": "10/7/2016", + "decision_effect": "Admit", + "outcome": "Def motion denied", + "publication_status": "Unpub", + "description": "Defendant moved to preclude FTM evidence; Court denied but suggested defendant could cross on contents of PCAST Report. Court notes that PCAST Report is clear that '\"[j]udges' decisions about the admissibility of scientific evidence rest solely on legal standards; they are exclusively the province of the courts and PCAST does not opine on them.\"", + "decision_date": "2016-10-07", + "case_citation": null, + "date_mismatch": false + }, + { + "source_row": 56, + "caption": "Garrett v. Commonwealth, 534 S.W.3d 217 (Ky. 2017)", + "jurisdiction": "State", + "court_location": "KY", + "posture": "Appeal", + "discipline": "FTM", + "decision_date_raw": "12/14/2017", + "decision_effect": "Affirm admission", + "outcome": "Conviction affirmed", + "publication_status": "Pub", + "description": "Does not refer to PCAST Report, but Williams v. Commonwealth, No. 2019-CA-000187-MR, 2020 Ky. App. LEXIS Unpub 222 (Ky. Ct. App. Mar. 27, 2020) notes that Garrett was decided after PCAST and held that Daubert hearings generally are not required for FTM because court can take judicial notice of the science of FTM evidence due to longstanding precedent.", + "decision_date": "2017-12-14", + "case_citation": "534 S.W.3d 217", + "date_mismatch": false + }, + { + "source_row": 57, + "caption": "Williams v. Commonwealth, No. 2019-CA-000187-MR, 2020 Ky. App. Unpub LEXIS 222 (Ky. Ct. App. Mar. 27, 2020)", + "jurisdiction": "State", + "court_location": "KY", + "posture": "Appeal", + "discipline": "FTM", + "decision_date_raw": "3/27/2020", + "decision_effect": "Affirm admission", + "outcome": "Conviction affirmed", + "publication_status": "Lexis", + "description": "No Daubert hearing needed. \"Our Supreme Court has accepted the reliability and validity of ballistic testing, such as that in Garrett and in this case, and the trial court here properly took judicial notice of its reliability and validity.\" See entry for Garrett v. Commonwealth, 534 S.W.3d 219 (Ky. 2017).", + "decision_date": "2020-03-27", + "case_citation": null, + "date_mismatch": false + }, + { + "source_row": 58, + "caption": "Abruquah v. State, 483 Md. 637 (Md. 2023) (see lower court cites in description)", + "jurisdiction": "State", + "court_location": "MD", + "posture": "Post-conviction", + "discipline": "FTM", + "decision_date_raw": "6/20/2023", + "decision_effect": "Remand for new trial", + "outcome": "Conviction vacated", + "publication_status": "Pub", + "description": "The prosecution of Abruquah resulted in decisions regarding FTM admissibility at multiple levels of Maryland\u2019s courts. Some of the lower courts discussed the PCAST Report. Experts listed were mentioned in one or more decisions.- State v. Abruquah, No. CT12-1375X (Md. Cir. Ct. Mar. 18, 2018) (unreported decision): Trial court discussed PCAST and NAS Reports, admitting FTM testimony but limiting expert to \u201cAFTE Range of Conclusions.\"- Abruquah v. State, No. 2176, 2020 Md. App. LEXIS 53 (Md. Ct. Special App. Jan. 17, 2020): Affirming trial court, noting that PCAST Report expressly stated that the admissibility of firearms analysis \"is a decision that belongs to the courts.\u201d- Abruquah v. State, 471 Md. 249 (Md. 2020) (per curium): Conviction reversed and remanded to lower court to consider whether it should reconsider its decision to admit toolmark evidence/testimony in light of Rochkind v. Stevenson, 471 Md. 1, 236 A.3d 630 (Md. 2019), which adopted Daubert standard. PCAST Report is not mentioned.- State v. Abruquah, No. CT12-1375X (Md. Cir. Ct. Undated) (unreported decision): Upon remand to consider challenge to firearms evidence under Daubert, trial court finds that toolmark and firearm evidence is admissible; denies motion for reconsideration and affirms conviction. (Decision is undated, but was issued in late 2021.) - Abruquah v. State, 483 Md. 637 (Md. 2023): Supreme Court of Maryland determined that the expert testimony on FTM analysis and AFTE methodology did not meet a minimum threshold of reliability. Provide extensive review of the PCAST Report findings on FTM and relies heavily on study data to support their assertion.", + "decision_date": "2023-06-20", + "case_citation": "483 Md. 637", + "date_mismatch": false + }, + { + "source_row": 59, + "caption": "Ricks v. Pauch, No. 17-12784, 2020 U.S. Dist. LEXIS 50109 (E.D. Mich. Mar. 23, 2020)", + "jurisdiction": "Federal", + "court_location": "MI (E.D. Mich.)", + "posture": "Pretrial", + "discipline": "FTM", + "decision_date_raw": "3/23/2020", + "decision_effect": "Admit", + "outcome": "Def motion denied", + "publication_status": "Lexis", + "description": "Post-conviction reversal 1983 case; defendants (including city of Detroit) argued exculpatory firearms and toolmark evidence (class characteristics evidence) inadmissible; court denied motion to exclude evidence and testimony of several experts, discussing PCAST Report and other reports and extensively discussing federal cases, and noting that experts would not be opining that the bullets in evidence matched to any particular gun.", + "decision_date": "2020-03-23", + "case_citation": "2020 U.S. Dist. LEXIS 50109", + "date_mismatch": false + }, + { + "source_row": 60, + "caption": "State v. Boss, 577 S.W.3d 509 (Mo. Ct. App. 2019)", + "jurisdiction": "State", + "court_location": "MO", + "posture": "Appeal", + "discipline": "FTM", + "decision_date_raw": "6/25/2019", + "decision_effect": "Affirm admission", + "outcome": "Conviction affirmed", + "publication_status": "Pub", + "description": "Finding that trial court did not err in admitting expert FTM testimony that shell casings recovered from robbery locations were from gun found in defendant's backpack. Court further finds that expert could be cross-examined regarding conclusions in NAS Report that tended to call that conclusion into question.", + "decision_date": "2019-06-25", + "case_citation": "577 S.W.3d 509", + "date_mismatch": false + }, + { + "source_row": 61, + "caption": "State v. Mills, 623 S.W.3d 717 (Mo. Ct. App. 2021)", + "jurisdiction": "State", + "court_location": "MO", + "posture": "Appeal", + "discipline": "FTM", + "decision_date_raw": "3/30/2021", + "decision_effect": "Affirm admission", + "outcome": "Conviction affirmed", + "publication_status": "Pub", + "description": "Finding no Daubert hearing required to assess experts testimony that recovered gun \"matched\" shell casing located at murder scene where expert's had 20 years of experience with FTM examination. State also moved to preclude defense from cross-examining regarding reliability and credibility of toolmark identification procedures based on PCAST and NAS Reports. Appellate court does not squarely address this issue, but notes that defendant \"had not endorsed any expert witness to lay a foundation establishing that the reports were authoritative, and the court had not taken judicial notice of the reports.\"", + "decision_date": "2021-03-30", + "case_citation": "623 S.W.3d 717", + "date_mismatch": false + }, + { + "source_row": 62, + "caption": "State v. Griffin, 268 N.C. App. 96 (N.C. Ct. App. Oct. 15, 2019)", + "jurisdiction": "State", + "court_location": "NC", + "posture": "Appeal", + "discipline": "FTM", + "decision_date_raw": "10/15/2019", + "decision_effect": "Affirm admission", + "outcome": "Conviction affirmed", + "publication_status": "Pub", + "description": "Trial court did not err in allowing FTM expert to testify that cartridge casings found at the crime scene came from firearm found in field adjacent to the defendant's property. Court finds that the defendant \"severely misrepresents [the expert's] opinion testimony by briefly summarizing a few lines of testimony while omitting the bulk of the testimony [and that] [o]ur review of the transcript reveals that [the expert's] opinion testimony was sufficient to demonstrate the reliability of the principles and methodology she used.\"", + "decision_date": "2019-10-15", + "case_citation": "268 N.C. App. 96", + "date_mismatch": false + }, + { + "source_row": 63, + "caption": "State v. Wheeler, 308 Neb. 708 (Neb. 2021)", + "jurisdiction": "State", + "court_location": "NE", + "posture": "Appeal", + "discipline": "FTM", + "decision_date_raw": "3/26/2021", + "decision_effect": "Affirm admission", + "outcome": "Conviction affirmed", + "publication_status": "Pub", + "description": "Relying largely on PCAST Report, defendant argued at a pretrial hearing that the Government's expert was not qualified to render an opinion that all seven shell casings found at the crime scene were fired from the same gun. Rejecting that argument, trial court allowed extensive cross examination at the hearing based on findings in the PCAST Report, but concluded that witness was qualified to testify as an expert and to render opinion as to shell casings. Appellate court affirmed finding (1) that expert's testimony was not prejudicial and supported the state's theory that one gun fired all of the shell casings, and (2) that defendant had been given ample opportunity to cross expert and present his own theory to the contrary.", + "decision_date": "2021-03-26", + "case_citation": "308 Neb. 708", + "date_mismatch": false + }, + { + "source_row": 64, + "caption": "State v. Oliver, No. A-5140-16T1, 2020 N.J. Super. Unpub. LEXIS 339 (N.J. Super. Ct. App. Div. Feb. 18, 2020) (per curiam)", + "jurisdiction": "State", + "court_location": "NJ", + "posture": "Appeal", + "discipline": "FTM", + "decision_date_raw": "2/18/2020", + "decision_effect": "N/A (declined to review ballistics challenge not preserved below)", + "outcome": "Conviction affirmed", + "publication_status": "Lexis", + "description": "Defendant did not object to testimony by firearms and toolmarks experts at trial; court did not review, but noted that it had \"upheld the admissibility of tool mark analysis evidence in the context of markings on plastic garbage bags,\" and reaffirmed use of Frye.", + "decision_date": "2020-02-18", + "case_citation": null, + "date_mismatch": false + }, + { + "source_row": 65, + "caption": "State v. Ghigliotty, 463 N.J. Super. 355 (N.J. Super. Ct. App. Div. 2020)", + "jurisdiction": "State", + "court_location": "NJ", + "posture": "Appeal", + "discipline": "FTM", + "decision_date_raw": "4/20/2020", + "decision_effect": "Remand for admissibility hearing", + "outcome": "Affirmed in part, vacated in part, remanded", + "publication_status": "Pub", + "description": "Affirming trial court's determination that a Frye hearing is necessary to determine reliability of BULLETTRAX and Matchpoint technology, reversing trial court's decision ordering discovery of algorithms and limitations on scope of expert's testimony (both as premature). Note: one Government expert did the initial examination and found no FTM match. Ten years later case was reopened, and a second Government expert reexamined the FTM evidence using new technologies. The second Government expert was found to be not credible by the lower court, a finding affirmed by the appellate court.", + "decision_date": "2020-04-20", + "case_citation": "463 N.J. Super. 355", + "date_mismatch": false + }, + { + "source_row": 66, + "caption": "U.S. v. Romero-Lobato, 379 F. Supp. 3d 1111 (D. Nev. 2019)", + "jurisdiction": "Federal", + "court_location": "NV (D. Nev.)", + "posture": "Pretrial", + "discipline": "FTM", + "decision_date_raw": "5/13/2019", + "decision_effect": "Admit", + "outcome": "Def motion denied", + "publication_status": "Pub", + "description": "After Daubert hearing, court denies defendant's motion, predicated almost entirely on NAS and PCAST Reports, to preclude expert testimony in firearm and toolmark identification based on AFTE methodology. Court conducted detailed analysis of five Daubert factors (finding only one factor favoring inadmissibility), incorporating and responding to criticism of the ATFE methodology in the NAS and PCAST reports. Court does not limit itself to black box studies.", + "decision_date": "2019-05-13", + "case_citation": "379 F. Supp. 3d 1111", + "date_mismatch": false + }, + { + "source_row": 67, + "caption": "U.S. v. Shipp, 422 F. Supp. 3d 762 (E.D.N.Y. 2019)", + "jurisdiction": "Federal", + "court_location": "NY (E.D.N.Y.)", + "posture": "Pretrial", + "discipline": "FTM", + "decision_date_raw": "11/26/2019", + "decision_effect": "Limit (cannot say \"match\" or that cartridges were fired from same firearm)", + "outcome": "Def motion denied/testimony limited", + "publication_status": "Pub", + "description": "Court denies request for Daubert hearing on toolmark analysis; lengthy analysis of PCAST Report and caselaw concludes that testimony is admissible but limits it: expert may only testify that bullet fragment and shell casing are \"consistent with\" being fired from recovered firearm; that recovered firearm \"cannot be excluded\" as source of bullet and bullet fragment but cannot testify that bullet and fragment were definitively fired from recovered firearm.", + "decision_date": "2019-11-26", + "case_citation": "422 F. Supp. 3d 762", + "date_mismatch": false + }, + { + "source_row": 68, + "caption": "U.S. v. Johnson, No. 16 Cr. 281, 2019 U.S. Dist. LEXIS 39590 (S.D.N.Y. Mar. 11, 2019)", + "jurisdiction": "Federal", + "court_location": "NY (S.D.N.Y.)", + "posture": "Mid-trial", + "discipline": "FTM", + "decision_date_raw": "3/11/2019", + "decision_effect": "Admit", + "outcome": "Def motion denied", + "publication_status": "Lexis", + "description": "After Daubert hearing, court refuses to limit FTM expert's testimony where expert will not testify that FTM evidence he examined \"matches to a particular firearm to the exclusion of all other firearms.\" Very thorough discussion of caselaw and 2008 & 2009 NRC Reports [2009 NAS] and PCAST Report as they pertain to FTM.", + "decision_date": "2019-03-11", + "case_citation": "2019 U.S. Dist. LEXIS 39590", + "date_mismatch": false + }, + { + "source_row": 69, + "caption": "U.S. v. White, No. 17 Cr. 611, 2018 U.S. Dist. Lexis 163258 (S.D.N.Y. Sept. 24, 2018)", + "jurisdiction": "Federal", + "court_location": "NY (S.D.N.Y.)", + "posture": "Pretrial", + "discipline": "FTM", + "decision_date_raw": "9/24/2018", + "decision_effect": "Limit (\"may not testify to any specific degree of certainty as to his conclusion that there is a ballistics match between the firearms seized . . . and those used in the various shooting incidents\")", + "outcome": "Defendant motion to exclude denied;; motion to limit granted", + "publication_status": "Lexis", + "description": "Denied def request for a Daubert hearing regarding FTM expert testimony. Citing NRC Report [2009 NAS] (not PCAST Report) and holding that Government expert \"may not testify to any specific degree of certainty as to his conclusion that there is a ballistics match between the firearms seized . . . and those used in the various shooting incidents. However, if pressed to define his degree of certainty during cross-examination, [the Government expert] may state his personal belief on that issue.\"", + "decision_date": "2018-09-24", + "case_citation": "2018 U.S. Dist. LEXIS 163258", + "date_mismatch": false + }, + { + "source_row": 70, + "caption": "People v. Ross, 68 Misc. 3d 899 (N.Y. Sup. Ct. 2020) & People v. A.M., 2020 NYLJ LEXIS 1401 (N.Y. Sup. Ct. June 30, 2020)", + "jurisdiction": "State", + "court_location": "NY", + "posture": "Pretrial", + "discipline": "FTM", + "decision_date_raw": "6/30/2020", + "decision_effect": "Limit (class characteristics only)", + "outcome": "Def motion granted in part and denied in part", + "publication_status": "Pub & Lexis", + "description": "Limiting testimony to class characteristics present that would include/exclude firearm at issue (this testimony found to meet Frye standard), but precluding testimony about the significance of marks other than class characteristics because \"the reliability of that practice in the relevant scientific community as a whole has not been established.\" Defense experts in scientific research methodology and study design criticized studies of toolmark analysis as a science. (Same analysis and decision in two cases.)", + "decision_date": "2020-06-30", + "case_citation": "68 Misc. 3d 899", + "date_mismatch": false + }, + { + "source_row": 71, + "caption": "U.S. v. Hunt, 464 F. Supp. 3d 1252 (W.D. Okla. 2020)", + "jurisdiction": "Federal", + "court_location": "OK (W.D. Okla.)", + "posture": "Pretrial", + "discipline": "FTM", + "decision_date_raw": "6/1/2020", + "decision_effect": "Admit with gov'ts proposed limits", + "outcome": "Def motion denied", + "publication_status": "Pub", + "description": "Court considered AFTE methodology as set forth in AFTE Journal in determining whether firearm/toolmark identification satisfied Daubert; court does not restrict itself to black-box studies in determining admissibility; good review of error rates & other federal decisions; expert will limit testimony to ULTR language.", + "decision_date": "2020-06-01", + "case_citation": "464 F. Supp. 3d 1252", + "date_mismatch": false + }, + { + "source_row": 72, + "caption": "U.S. v. Adams, 444 F. Supp. 3d 1248 (D. Or. 2020)", + "jurisdiction": "Federal", + "court_location": "OR (D. Or.)", + "posture": "Pretrial", + "discipline": "FTM", + "decision_date_raw": "3/16/2020", + "decision_effect": "Limit (cannot say \"match\" or that cartridges were fired from same firearm; only class characteristics)", + "outcome": "Def motion granted in part and denied in part", + "publication_status": "Pub", + "description": "After initial comparisons of FTM through NIBIN yielded contradictory findings, evidence reexamined by an examiner with the Oregon Crime Lab, who determined that the casings were fired from the defendant's gun. Court then held Daubert hearing and permitted limited testimony: \"[n]o evidence related to [the expert's] methodology or conclusions relating to whether the shell casings matched [the defendant's gun] will be admitted at trial.\" Court criticized AFTE methodology and testimony of expert witness.", + "decision_date": "2020-03-16", + "case_citation": "444 F. Supp. 3d 1248", + "date_mismatch": false + }, + { + "source_row": 73, + "caption": "Morones v. State, Nos. 09-16-00317; 00318; 00319; 00320, 2017 Tex. App. LEXIS 10964 (Tex. Ct. App. Nov. 22, 2017)", + "jurisdiction": "State", + "court_location": "TX", + "posture": "Appeal", + "discipline": "FTM", + "decision_date_raw": "11/22/2017", + "decision_effect": "Error to admit, but harmless", + "outcome": "Conviction affirmed", + "publication_status": "Lexis", + "description": "PCAST Report referred to in appellant's brief; court determines that any error in admitting FTM testimony was harmless in light of strong evidence of guilt.", + "decision_date": "2017-11-22", + "case_citation": null, + "date_mismatch": false + }, + { + "source_row": 74, + "caption": "Williams v. State, No. AP-77,053, 2017 Tex. Crim. App. Unpub. LEXIS 906 (Tex. Crim. App. Nov. 1, 2017)", + "jurisdiction": "State", + "court_location": "TX", + "posture": "Appeal", + "discipline": "FTM", + "decision_date_raw": "11/1/2017", + "decision_effect": "Affirm admission", + "outcome": "Conviction affirmed", + "publication_status": "Lexis", + "description": "At pp. 49-53 of lengthy opinion, court rejects challenge to trial court's admission of expert FTM testimony, finding that the \"theory of firearm and toolmark identification and the technique of microscopic firearm and toolmark comparison are accepted as valid by the relevant scientific community.\" Daubert hearing held by trial court.", + "decision_date": "2017-11-01", + "case_citation": null, + "date_mismatch": false + }, + { + "source_row": 75, + "caption": "U.S. v. Davis, No. 4:18-cr-00011, 2019 U.S. Dist. LEXIS 155037 (W.D. Va. Sept. 11, 2019)", + "jurisdiction": "Federal", + "court_location": "VA (W.D. Va.)", + "posture": "Pretrial", + "discipline": "FTM", + "decision_date_raw": "9/11/2019", + "decision_effect": "Limit (cannot say \"match\" or that cartridges were fired from same firearm)", + "outcome": "Def motion granted in part and denied in part", + "publication_status": "Lexis", + "description": "After Daubert hearing, court permits FTM experts to testify as to similarities/consistencies in recovered cartridge casings, but due to the concerns expressed in court decisions and the PCAST/NRC [2009 NAS] Reports, precludes testimony that markings indicate a \"match,\" or that cartridges were fired from the same firearm.", + "decision_date": "2019-09-11", + "case_citation": "2019 U.S. Dist. LEXIS 155037", + "date_mismatch": false + }, + { + "source_row": 76, + "caption": "State v. Hatfield, No. 77512-0-I, 2019 Wash. App. LEXIS 3027 (Wash. Ct. App. Dec. 2, 2019)", + "jurisdiction": "State", + "court_location": "WA", + "posture": "Appeal", + "discipline": "FTM", + "decision_date_raw": "12/2/2019", + "decision_effect": "Affirm admission", + "outcome": "Conviction affirmed", + "publication_status": "Lexis", + "description": "Frye hearing request denied by trial court, which concluded that PCAST Report did not raise enough of a dispute as to the general acceptance of FTM evidence to require a hearing; de novo review, no Frye hearing necessary (citing State v. DeJesus, 436 P.3d 834 (Wash. Ct. App. 2019)).", + "decision_date": "2019-12-02", + "case_citation": null, + "date_mismatch": false + }, + { + "source_row": 77, + "caption": "People v. Purpera, No. 2016 Cr. 7798 (Colo. Dist. Ct. Aug. 12, 2018) (unpublished)", + "jurisdiction": "State", + "court_location": "CO", + "posture": "Pretrial", + "discipline": "FTM", + "decision_date_raw": "8/12/2018", + "decision_effect": "Admit", + "outcome": "Def motion denied", + "publication_status": "Unpub", + "description": "Denying defense motion to preclude FTM expert testimony after hearing. Lengthy discussion of Colorado standard for admitting scientific evidence and noting that error rates discussed in PCAST Report that were relied on by the defense did not render expert opinion unreliable. Discussion of contents of PCAST Report as hearsay and noting that the Report is the result of a \"[g]oal driven effort[] to find things to criticize [which] tend to present an unbalanced picture and tend to disregard, sometimes inadvertently, contrary evidence.\"", + "decision_date": "2018-08-12", + "case_citation": null, + "date_mismatch": false + }, + { + "source_row": 78, + "caption": "People v. Rodriguez, 106 N.E.3d 436 (Ill. App. Ct. 2018)", + "jurisdiction": "State", + "court_location": "IL", + "posture": "Appeal", + "discipline": "FTM", + "decision_date_raw": "5/8/2017", + "decision_effect": "Affirm admission", + "outcome": "Conviction affirmed", + "publication_status": "Pub", + "description": "No Frye hearing required prior to admitting testimony from firearms expert that crime scene bullet was fired from gun recovered from defendant's house. Conclusions of NRC [2009 NAS] report as to firearms analysis goes to weight, not admissibility, of expert testimony and NRC Report has not undermined the reliability of FTM evidence to the degree that it has ceased to be accepted in the scientific community. Testimony of Government witness regarding gunshot residue was not challenged on appeal.", + "decision_date": "2017-05-08", + "case_citation": "106 N.E.3d 436", + "date_mismatch": true + }, + { + "source_row": 79, + "caption": "Commonwealth v. Mulazim, No. 15 Cr 0592 (Ky. Cir. Ct. May ___, 2018); (unpublished; unsigned order)", + "jurisdiction": "State", + "court_location": "KY", + "posture": "Pretrial", + "discipline": "FTM", + "decision_date_raw": "5/1/2018", + "decision_effect": "Exclude def expert", + "outcome": "Def expert excluded", + "publication_status": "Unpub (and unsigned)", + "description": "Defense FTM expert excluded; testimony would be more confusing and misleading than probative.", + "decision_date": "2018-05-01", + "case_citation": null, + "date_mismatch": false + }, + { + "source_row": 80, + "caption": "State v. Eaglin, 239 So. 3d 1001 (La. Ct. App. 2018)", + "jurisdiction": "State", + "court_location": "LA", + "posture": "Appeal", + "discipline": "FTM", + "decision_date_raw": "3/28/2018", + "decision_effect": "N/A (no error in denying defense motion for FTM expert)", + "outcome": "Conviction affirmed", + "publication_status": "Pub", + "description": "Court declines invitation to find that PCAST Report invalidated methodology employed by expert in examining FTM. Daubert hearing below. Conviction later reversed, 265 So.3d 761 (La. 2019) (per curium), but on different grounds.", + "decision_date": "2018-03-28", + "case_citation": null, + "date_mismatch": false + }, + { + "source_row": 81, + "caption": "U.S. v. Medley, No. 17 Cr. 242 (S.D. Md. Apr. 24, 2018) (transcript of oral ruling)", + "jurisdiction": "Federal", + "court_location": "MD (S.D. Md.)", + "posture": "Pretrial", + "discipline": "FTM", + "decision_date_raw": "4/24/2018", + "decision_effect": "Limit (expert cannot testify that defendant's gun fired bullet to the exclusion of all others)", + "outcome": "Def motion denied", + "publication_status": "Unpub (oral)", + "description": "Lengthy discussion of PCAST Report. Limiting expert testimony: court will not allow expert to express opinion that cartridges found at crime scene were fired from the same gun as that associated with the defendant or to express confidence level as to his opinion.", + "decision_date": "2018-04-24", + "case_citation": null, + "date_mismatch": false + }, + { + "source_row": 82, + "caption": "State v. Goodwin-Bey, No. 1531-CR555-01 (Cir. Ct. Mo. Dec. 16, 2016) (unpublished)", + "jurisdiction": "State", + "court_location": "MO", + "posture": "Pretrial", + "discipline": "FTM", + "decision_date_raw": "12/16/2016", + "decision_effect": "Limit (gun \"could not be eliminated\" as source)", + "outcome": "Def motion denied", + "publication_status": "Unpub", + "description": "Court \"very reluctantly\" allows \"the State's lab person to testify, but only to the point this gun could not be eliminated as the source of the bullet.\" Lengthy discussion of conclusions of NAS and PCAST Reports. But see State v. Mills, 623 SW3d 717 (Mo. Ct. App. 2021) and State v. Boss, 577 SW3d 509 (Mo. Ct. App. 2021), both of which decline to limit expert's testimony in the manner suggested by Goodwin-Bey.", + "decision_date": "2016-12-16", + "case_citation": null, + "date_mismatch": false + }, + { + "source_row": 83, + "caption": "U.S. v. Gil, 680 F. App'x 11 (2d Cir. 2017) (summary order)", + "jurisdiction": "Federal", + "court_location": "NY (2d Cir.)", + "posture": "Appeal", + "discipline": "FTM", + "decision_date_raw": "2/21/2017", + "decision_effect": "Affirm admission", + "outcome": "Conviction affirmed", + "publication_status": "Pub (summ order)", + "description": "Affirming conviction and summarizing what FTM expert language is permissible in Second Circuit. NAS Reports mentioned, but not PCAST Report (although defendant's brief relied upon PCAST).", + "decision_date": "2017-02-21", + "case_citation": null, + "date_mismatch": false + }, + { + "source_row": 84, + "caption": "State v. Castro DeJesus, 436 P.3d 834 (Wash. Ct. App. 2019)", + "jurisdiction": "State", + "court_location": "WA", + "posture": "Appeal", + "discipline": "FTM", + "decision_date_raw": "3/11/2019", + "decision_effect": "Affirm admission", + "outcome": "Conviction affirmed", + "publication_status": "Pub", + "description": "Employing Frye analysis, admission of FTM expert testimony after midtrial hearing affirmed on appeal. Court notes that the \"PCAST report acknowledged its own dubious value to courts\" and that the Report \"does not indicate that . . . toolmark testing is without merit\" and holding that questions raised about the methodologies employed by FTM experts in the report \"bear on the question of reliability\" and go to the weight of the testimony, not its admissibility.", + "decision_date": "2019-03-11", + "case_citation": "436 P.3d 834", + "date_mismatch": false + }, + { + "source_row": 85, + "caption": "State v. Allen, No. 2017 KA 0306, 2017 La. App. Unpub. LEXIS 325 (La. Ct. App. Nov. 1, 2017)", + "jurisdiction": "State", + "court_location": "LA", + "posture": "Appeal", + "discipline": "FTM; Fingerprints", + "decision_date_raw": "11/1/2017", + "decision_effect": "Affirm admission", + "outcome": "Conviction affirmed", + "publication_status": "Lexis", + "description": "Court finds no error in trial court allowing FTM testimony that projectile found at crime scene was fired from gun found at defendant's apartment and testimony from fingerprint expert concerning fingerprint match. Defendant argued that court should have granted pretrial motion for Daubert hearing in light of PCAST Report determining that methodology of firearms analysis is unreliable and that there is a high false-positive rate in fingerprint identifications. Court finds that \"firmly established reliability of fingerprint evidence and firearms examinations,\" coupled with opportunity to cross-examine at trial, supported trial court's refusal to conduct a Daubert hearing and that there was no error in admitting both experts' testimony.", + "decision_date": "2017-11-01", + "case_citation": null, + "date_mismatch": false + }, + { + "source_row": 86, + "caption": "U.S. v. Cloud, 576 F. Supp. 3d 827 (E.D. Wash. 2021)", + "jurisdiction": "Federal", + "court_location": "WA (E.D. Wash.)", + "posture": "Pretrial", + "discipline": "FTM; Fingerprints", + "decision_date_raw": "12/17/2021", + "decision_effect": "Limit", + "outcome": "Defendant's motion granted in part and denied in part; G fingerprint expert precluded from testifying as not sufficiently qualified", + "publication_status": "Pub", + "description": "At pretrial hearing, Government FTM expert opined that recovered shell casings were fired from recovered firearm. In its ruling on defendant's motion to exclude FTM and fingerprint testimony, trial court quoted extensively and with apparent agreement, from PCAST Report and its criticisms of FTM analysis, in particular. It then applied a Daubert analysis, which criticized AFTE methodology and AFTE peer-review process and noted its conclusion that the AFTE methodology cannot be sufficiently and reliably replicated and is \"highly subjective.\" Ultimately, in light of its concerns, court ruled that Government expert was only permitted to testify that recovered cartridges could not be excluded as having been fired by the recovered gun. The court ruled in addition that it would inform the jury that (1) only three studies that meet the minimum design standards have attempted to measure the accuracy of firearm/toolmark comparison; and (2) these three studies found false positive rates that could be as high as 1 in 46; 1 in 200; and 1 in 67. As to the Government's firearm expert, the court found, based on her poor performance in proficiency exams and errors in past fingerprint analyses that she performed, that she was not qualified to testify as an expert in fingerprint analysis. Court further found that ACE-V methodology of fingerprint analysis is reliable under Daubert standard.", + "decision_date": "2021-12-17", + "case_citation": "576 F. Supp. 3d 827", + "date_mismatch": false + }, + { + "source_row": 87, + "caption": "U.S. v. Ausby, 436 F. Supp. 3d 134 (D.D.C. 2019)", + "jurisdiction": "Federal", + "court_location": "DC (D.D.C.)", + "posture": "Pretrial", + "discipline": "FTM; Fingerprints; Presence of Semen", + "decision_date_raw": "12/20/2019", + "decision_effect": "Exclude (\"expert is unavailable and no evidentiary hearing may be held on the reliability of the specific methods he used because those methods are unknown and now unknowable\")", + "outcome": "Def motion granted", + "publication_status": "Pub", + "description": "On retrial of 47-year-old murder case (pre-Daubert and pre-FRE 702), court precludes admission of transcripts of now-deceased Government firearms, semen, and fingerprint experts, finding that Government cannot meet Daubert admissibility standards. Re firearms, held: \"expert is unavailable and no evidentiary hearing may be held on the reliability of the specific methods he used because those methods are unknown and now unknowable.\" Mention of PCAST Report, but court does not rely on it.", + "decision_date": "2019-12-20", + "case_citation": "436 F. Supp. 3d 134", + "date_mismatch": false + }, + { + "source_row": 88, + "caption": "U.S. v. Tuzman, No. 15 Cr. 536, 2017 U.S. Dist. LEXIS 208410 (S.D.N.Y. Dec. 18, 2017)", + "jurisdiction": "Federal", + "court_location": "NY (S.D.N.Y.)", + "posture": "Pretrial", + "discipline": "Ink dating", + "decision_date_raw": "12/18/2017", + "decision_effect": "Exclude def expert", + "outcome": "Government motion granted", + "publication_status": "Lexis", + "description": "Gov't motion to exclude testimony of def witness granted. PCAST Report not mentioned, but addresses admissibility issues of ink dating testimony, FRCP 16 disclosures, reliability under FRE 702, FRE 403. Expert's failure to use \"basic quality control protocols--including those required in the two papers he purportedly relies on--demonstrates that he lacks 'good grounds' for his conclusions.\"", + "decision_date": "2017-12-18", + "case_citation": "2017 U.S. Dist. LEXIS 208410", + "date_mismatch": false + }, + { + "source_row": 89, + "caption": "Schmidt v. Int'l Playthings LLC, 536 F. Supp. 3d 856 (D.N.M. 2021)", + "jurisdiction": "Federal", + "court_location": "NM (D.N.M.)", + "posture": "Pretrial", + "discipline": "PCAST/; FRE 702", + "decision_date_raw": "4/29/2021", + "decision_effect": "PCAST dicta", + "outcome": "Dicta about PCAST/FRE 702", + "publication_status": "Pub", + "description": "Dicta in FNs about efforts to amend FRE 702 / PCAST.", + "decision_date": "2021-04-29", + "case_citation": "536 F. Supp. 3d 856", + "date_mismatch": false + }, + { + "source_row": 90, + "caption": "White v. Town of Hurley, No. CIV 17-0983, 2019 U.S. Dist. LEXIS 52487 (D.N.M. Mar. 28, 2019)", + "jurisdiction": "Federal", + "court_location": "NM (D.N.M.)", + "posture": "Pretrial", + "discipline": "PCAST/; FRE 702", + "decision_date_raw": "3/28/2019", + "decision_effect": "PCAST dicta", + "outcome": "Dicta about PCAST/FRE 702", + "publication_status": "Lexis", + "description": "Dicta in FNs about efforts to amend FRE 702 / PCAST.", + "decision_date": "2019-03-28", + "case_citation": "2019 U.S. Dist. LEXIS 52487", + "date_mismatch": false + }, + { + "source_row": 91, + "caption": "Walker v. Spina, 359 F. Supp. 3d 1054 (D.N.M. 2019)", + "jurisdiction": "Federal", + "court_location": "NM (D.N.M.)", + "posture": "Pretrial", + "discipline": "PCAST/; FRE 702", + "decision_date_raw": "1/11/2019", + "decision_effect": "PCAST dicta", + "outcome": "Dicta about FRE 702/PCAST", + "publication_status": "Pub", + "description": "Dicta in FNs about efforts to amend FRE 702 / PCAST.", + "decision_date": "2019-01-11", + "case_citation": "359 F. Supp. 3d 1054", + "date_mismatch": false + }, + { + "source_row": 92, + "caption": "U.S. v. Ortiz, No. 21-CR-2503, 2024 U.S. Dist. LEXIS 102951 (S.D. Cal. June 10, 2024)", + "jurisdiction": "Federal", + "court_location": "CA (S.D. Cal.)", + "posture": "Pretrial", + "discipline": "DNA; & FTM", + "decision_date_raw": "6/10/2024", + "decision_effect": "Exclude", + "outcome": "Def motion granted", + "publication_status": "Lexis", + "description": "STRmix DNA case; length analysis on how STRmix probabilistic genotyping software operates and its high likelihood for error when the number of contributors (\"NOC\") to a DNA sample is five or more; defendant looking to exclude DNA sample found on handgun based on the claim that there were at least six contributors; District court relied on PCAST report and FRE 702 to determine admissibility. The court found that the likelihood of six contributors was high based on common errors that occurred with the technology used to analyze NOC and the high error rate that characterized NOC determinations of mixtures with six or more made this DNA evidence unreliable and inadmissible.", + "decision_date": "2024-06-10", + "case_citation": "2024 U.S. Dist. LEXIS 102951", + "date_mismatch": false + }, + { + "source_row": 93, + "caption": "Geter v. U.S., 306 A.3d 126 (D.C. 2023)", + "jurisdiction": "State", + "court_location": "DC", + "posture": "Appeal", + "discipline": "FTM", + "decision_date_raw": "12/21/2023", + "decision_effect": "Error to admit, but harmless", + "outcome": "Conviction affirmed", + "publication_status": "Pub", + "description": "The Court of Appeals reviewed on the basis of plain error testimony from the government witness regarding the shell casings found at the scene to match the gun found at the scene. The court found the admission of the testimony to be plain error because a toolmark examiner may not provide an unqualified opinion testimony that purports to identify a specific bullet as having been fired by a specific gun via toolmark pattern matching. The court used the PCAST report as a basis to determine that the expert testimony on toolmark pattern matching should have been excluded, but amounted to harmless error in this case.", + "decision_date": "2023-12-21", + "case_citation": "306 A.3d 126", + "date_mismatch": false + }, + { + "source_row": 94, + "caption": "U.S. v. Green, No. 2018 CF1 004356, 2024 D.C. Super. LEXIS 8, (D.C. Super. Ct. Apr. 1, 2024)", + "jurisdiction": "State", + "court_location": "DC (Super. Ct.)", + "posture": "Pretrial", + "discipline": "FTM", + "decision_date_raw": "4/1/2024", + "decision_effect": "Admit with gov'ts proposed limits", + "outcome": "Def motion denied", + "publication_status": "Lexis", + "description": "Defendants relied on PCAST report as a basis to exclude firearms identification because the AFTE Theory of Identification was circular and did not meet the criteria for foundational validity. However, the court found that the PCAST report encouraged appropriate black-box design testing to study firearms, which had been completed since the publishing of the PCAST report. These tests met the threshold for appropriate design and showed low error rates for identification, leading the court to deem the expert testimony of firearms examiners admissible, with some limitations upon what language they could use in the testimony.", + "decision_date": "2024-04-01", + "case_citation": null, + "date_mismatch": false + }, + { + "source_row": 95, + "caption": "People v. Burrus, No. 817/2020, 81 Misc. 3d 550 (N.Y. App. Div. Sept. 8, 2023)", + "jurisdiction": "State", + "court_location": "NY", + "posture": "Pretrial", + "discipline": "DNA; & PCAST Dicta", + "decision_date_raw": "9/8/2023", + "decision_effect": "Admit", + "outcome": "Def motion denied", + "publication_status": "Pub", + "description": "Defendant moved to preclude evidence of High Sensitivity DNA typing (a.k.a., low copy number (LCN)) and the Forensic Statistical Tool (FST). The Judge used the PCAST report as a backdrop to compare the government and defense expert witness testimony, and ultimately decided that the government expert witness testimony was more compelling. Defendant's motion to exclude was denied.", + "decision_date": "2023-09-08", + "case_citation": "81 Misc. 3d 550", + "date_mismatch": false + }, + { + "source_row": 96, + "caption": "Bader v. Johnson & Johnson, 86 Cal. App. 5th 1094 (Cal. Ct. App. 2022)", + "jurisdiction": "State", + "court_location": "CA", + "posture": "Appeal", + "discipline": "PCAST Dicta", + "decision_date_raw": "12/23/2022", + "decision_effect": "Affirm admission", + "outcome": "Def motion denied", + "publication_status": "Pub", + "description": "PCAST brought up as dicta in the concurring opinion of P.J. Streeter as to the admissibility of evidence regarding inhalation of asbestos for a civil suit. Suit was brought by an individual against the Johnson & Johnson company and Colgate-Palmolive company for asbestos in cosmetic talc powder, resulting in Plaintiff's mesothelioma. The expert testimony for the Plaintiff in this civil suit was not found to be reversible error. However, the concurring judge asserted that the Defendants should have filed a Kelly objection in questioning the Plaintiff's expert witness, as it would asses the scientific foundation for the opinion, but Defendants did not do so.", + "decision_date": "2022-12-23", + "case_citation": "86 Cal. App. 5th 1094", + "date_mismatch": false + }, + { + "source_row": 97, + "caption": "People v. Wakefield, 195 N.E.3d 19 (N.Y. 2022)", + "jurisdiction": "State", + "court_location": "NY", + "posture": "Appeal", + "discipline": "DNA", + "decision_date_raw": "4/26/2022", + "decision_effect": "Affirm admission", + "outcome": "Conviction affirmed", + "publication_status": "Pub", + "description": "Defense appealed the admission of DNA evidence based on probabilistic genotyping through TrueAllelle, and further requested parts of the source code that TrueAllelle operated on. Government used PCAST to establish the general scientific acceptance and reliability of the software based on the facts of this particular case. The court also ruled that the defendant was not entitled to the source code because it was proprietary information not under the government's control.", + "decision_date": "2022-04-26", + "case_citation": "195 N.E.3d 19", + "date_mismatch": false + }, + { + "source_row": 98, + "caption": "J.H. v. Cruz, No. 3:14-cv-02356, 2023 U.S. Dist. LEXIS 14244 (M.D. Tenn. Jan. 27, 2023)", + "jurisdiction": "Federal", + "court_location": "TN (M.D. Tenn.)", + "posture": "Pretrial", + "discipline": "DNA", + "decision_date_raw": "1/27/2023", + "decision_effect": "Limit", + "outcome": "Def motion denied in part, granted in part", + "publication_status": "Lexis", + "description": "Defense in a civil case looked to exclude a DNA report filed by Plaintiff's council using the PCAST Report as a basis. The DNA investigation and report were provided by a forensic biologist employed by the Tennessee Bureau of Investigation. Ultimately, the court allowed the DNA findings in the official report to be included in the expert testimony, but excluded testimony outside of those narrow findings.", + "decision_date": "2023-01-27", + "case_citation": "2023 U.S. Dist. LEXIS 14244", + "date_mismatch": false + }, + { + "source_row": 99, + "caption": "State v. Johnson, No. C-230373, 2024-Ohio-2572 (Ohio Ct. App. July 5, 2024)", + "jurisdiction": "State", + "court_location": "OH", + "posture": "Appeal", + "discipline": "Fingerprints", + "decision_date_raw": "7/5/2024", + "decision_effect": "Affirm admission", + "outcome": "Conviction affirmed", + "publication_status": "Pub", + "description": "Defendant moved for postconviction relief, stating that the expert testimony on latent-print analysis was prejudicial as it is not considered a \"scientific method\" according to numerous reports, including PCAST.", + "decision_date": "2024-07-05", + "case_citation": "2024-Ohio-2572", + "date_mismatch": false + }, + { + "source_row": 100, + "caption": "Ward v. Commonwealth, No. 2021-SC-0568-MR, 2024 Ky. Unpub. LEXIS 41 (Ky. Apr. 18, 2024)", + "jurisdiction": "State", + "court_location": "KY", + "posture": "Appeal", + "discipline": "FTM; & Fingerprints", + "decision_date_raw": "4/18/2024", + "decision_effect": "Error to admit, but harmless", + "outcome": "Conviction affirmed", + "publication_status": "Lexis", + "description": "The court requested both parties to present arguments regarding the opinion testimony on firearm toolmark analysis in light of the PCAST Report findings on the admissibility of such testimony. Ultimately, while the court had concerns about the admission of the FTM expert testimony, it determined that the defense did not present any reversible errors including the questions relating to the fingerprint analysis.", + "decision_date": "2024-04-18", + "case_citation": null, + "date_mismatch": false + }, + { + "source_row": 101, + "caption": "Ficklin v. Commonwealth, No. 2020-SC-0573-MR, 2022 Ky. Unpub. LEXIS 34 (Ky. Aug. 18, 2022)", + "jurisdiction": "State", + "court_location": "KY", + "posture": "Appeal", + "discipline": "FTM", + "decision_date_raw": "8/18/2022", + "decision_effect": "Affirm admission", + "outcome": "Conviction affirmed", + "publication_status": "Lexis", + "description": "Defense appealed the admission of FTM expert testimony, citing the PCAST Report to argue that it is not scientifically reliable. The Kentucky Supreme Court relied on the trial court's Daubert hearing details to determine that the FTM testimony was not admitted in error.", + "decision_date": "2022-08-18", + "case_citation": null, + "date_mismatch": false + }, + { + "source_row": 102, + "caption": "People v. Parsons, No. H049411, 2024 Cal. App. Unpub. LEXIS 1064 (Cal. Ct. App. Feb. 21, 2024)", + "jurisdiction": "State", + "court_location": "CA", + "posture": "Appeal", + "discipline": "FTM", + "decision_date_raw": "2/21/2024", + "decision_effect": "Error to admit, but harmless", + "outcome": "Conviction affirmed", + "publication_status": "Lexis", + "description": "Defendant appeals his convictions based on the PCAST Report, noting that the PCAST and NRC Reports established a change in the scientific community towards the reliability of the methods commonly used in FTM analysis. The court ruled that the evidence of defendant's guilt was strong even without the FTM testimony, and therefore amounted to harmless error.", + "decision_date": "2024-02-21", + "case_citation": null, + "date_mismatch": false + }, + { + "source_row": 103, + "caption": "U.S. v. Pete, No. 3:22cr48, 2023 U.S. Dist. LEXIS 135065 (N.D. Fla. July 21, 2023)", + "jurisdiction": "Federal", + "court_location": "FL (N.D. Fla.)", + "posture": "Pretrial", + "discipline": "FTM; & Fingerprints", + "decision_date_raw": "7/21/2023", + "decision_effect": "Admit", + "outcome": "Def motion denied", + "publication_status": "Lexis", + "description": "Defense relies on PCAST Report as a reason to exclude FTM and fingerprint evidence under Daubert because it is not scientifically valid or reliable. The court in this case pointed to the long established use of FTM and fingerprint analysis in the scientific community and in their jurisprudence to ultimately admit these methods. Includes lengthy analysis of PCAST Report and criticism of the Report set forth by the scientific community.", + "decision_date": "2023-07-21", + "case_citation": "2023 U.S. Dist. LEXIS 135065", + "date_mismatch": false + }, + { + "source_row": 104, + "caption": "State v. Miller, 275 N.C. App. 843 (N.C. Ct. App. 2020)", + "jurisdiction": "State", + "court_location": "NC", + "posture": "Appeal", + "discipline": "FTM", + "decision_date_raw": "12/31/2020", + "decision_effect": "Affirm admission", + "outcome": "Conviction affirmed", + "publication_status": "Pub", + "description": "Defense attempts to use PCAST Report to exclude expert testimony on appeal, also brought it up in cross-examination of the government's expert witness, who claimed she disagreed with elements of the PCAST Report.", + "decision_date": "2020-12-31", + "case_citation": "275 N.C. App. 843", + "date_mismatch": false + }, + { + "source_row": 105, + "caption": "State v. Raynor, No. HHD-CR13-0667367, 2024 Conn. Super. LEXIS 1520 (Conn. Super. Ct. Apr. 16, 2024)", + "jurisdiction": "State", + "court_location": "CT", + "posture": "Pretrial", + "discipline": "FTM", + "decision_date_raw": "4/16/2024", + "decision_effect": "Limit", + "outcome": "Def motion denied", + "publication_status": "Lexis", + "description": "The defendant filed a motion to exclude ballistics testimony on the basis that it lacked scientific reliability and was irrelevant. The court went on to extensively analyze the PCAST Report findings and determined that the FTM identification methodology used by the expert witness had been subject to peer review and publication, weighing in favor of its admissibility. The court ruled that the state had met its burden in established that the testimony is derived from a scientifically reliable methodology, but did create some limitations as to the language used by the expert.", + "decision_date": "2024-04-16", + "case_citation": null, + "date_mismatch": false + }, + { + "source_row": 106, + "caption": "Carmon v. State, No. NNH CV20-6107902; NNH CV19-5052879, 2022 Conn. Super. LEXIS 2620 (Conn. Super. Ct. Nov. 30, 2022)", + "jurisdiction": "State", + "court_location": "CT", + "posture": "Post-conviction", + "discipline": "FTM", + "decision_date_raw": "11/30/2022", + "decision_effect": "Remand for new trial", + "outcome": "Habeas petition granted", + "publication_status": "Lexis", + "description": "The defense presented an expert on their post conviction motion for a new trial to speak on newly discovered forensic evidence, and to argue against the FTM analysis in his case based on the PCAST Report.", + "decision_date": "2022-11-30", + "case_citation": null, + "date_mismatch": false + }, + { + "source_row": 107, + "caption": "U.S. v. James, No. 3:19-79, 2024 U.S. Dist. LEXIS 7615 (D.V.I. Jan. 16, 2024)", + "jurisdiction": "Federal", + "court_location": "VI (D.V.I.)", + "posture": "Pretrial", + "discipline": "FTM", + "decision_date_raw": "1/16/2024", + "decision_effect": "Limit", + "outcome": "Def motion denied", + "publication_status": "Lexis", + "description": "Defendants argue that the FTM analysis that the government intends to put forth does not meet Daubert's reliability requirement, while citing to the PCAST Report to back up the assertion. The court decided that the science passes muster under Daubert, but that the experts could not testify based on absolute certainty.", + "decision_date": "2024-01-16", + "case_citation": "2024 U.S. Dist. LEXIS 7615", + "date_mismatch": false + }, + { + "source_row": 108, + "caption": "U.S. v. Felix, No. 2020-0002, 2022 U.S. Dist. LEXIS 213513 (D.V.I. Nov. 28, 2022)", + "jurisdiction": "Federal", + "court_location": "VI (D.V.I.)", + "posture": "Pretrial", + "discipline": "FTM", + "decision_date_raw": "11/28/2022", + "decision_effect": "Limit", + "outcome": "Def motion denied in part, granted in part", + "publication_status": "Lexis", + "description": "Defendant uses the PCAST Report to move to exclude government witness on FTM analysis. There is a lengthy analysis on the PCAST Report's discussion of rate of error for FTM analysis. The court ultimately follows part of the PCAST recommendations to exclude inconclusive responses in the FTM studies completed since the publishing of the PCAST Report.", + "decision_date": "2022-11-28", + "case_citation": "2022 U.S. Dist. LEXIS 213513", + "date_mismatch": false + }, + { + "source_row": 109, + "caption": "U.S. v. Graham, No. 4:23-cr-00006, 2024 U.S. Dist. LEXIS 28646 (W.D. Va. Feb. 20, 2024)", + "jurisdiction": "Federal", + "court_location": "VA (W.D. Va.)", + "posture": "Pretrial", + "discipline": "FTM", + "decision_date_raw": "2/20/2024", + "decision_effect": "Admit with gov'ts proposed limits", + "outcome": "Def motion denied in part, granted in part", + "publication_status": "Lexis", + "description": "Defendant moved to exclude expert testimony on FTM analysis based on PCAST Report, but the court determined that the scientific methods employed by the government appropriately addressed the PCAST report's criticisms on FTM analysis.", + "decision_date": "2024-02-20", + "case_citation": "2024 U.S. Dist. LEXIS 28646", + "date_mismatch": false + }, + { + "source_row": 110, + "caption": "Curry v. Haynes, No. 3:22-CV-5493-LK, 2023 U.S. Dist. LEXIS 102037 (W.D. Wash. May 11, 2023)", + "jurisdiction": "Federal", + "court_location": "WA (W.D. Wash.)", + "posture": "Post-conviction", + "discipline": "FTM", + "decision_date_raw": "5/11/2023", + "decision_effect": "Affirm admission", + "outcome": "Habeas petition denied", + "publication_status": "Lexis", + "description": "Defendant moved for postconviction relief, relying on the PCAST Report as reason why FTM analysis was admitted in error. However, the court found that during the time period of this case, FTM was considered generally accepted and the habeas petition was dismissed with prejudice.", + "decision_date": "2023-05-11", + "case_citation": "2023 U.S. Dist. LEXIS 102037", + "date_mismatch": false + }, + { + "source_row": 111, + "caption": "U.S. v. Rhodes, No. 3:19-cr-00333-MC, 2023 U.S. Dist. LEXIS 7528 (D. Or. Jan. 17, 2023)", + "jurisdiction": "Federal", + "court_location": "OR (D. Or.)", + "posture": "Pretrial", + "discipline": "FTM", + "decision_date_raw": "1/17/2023", + "decision_effect": "Admit", + "outcome": "Def motion denied", + "publication_status": "Lexis", + "description": "While defense attempts to use PCAST Report in support of their motion to exclude expert testimony, the court acknowledges its impact but uses it as background for why the evidence will not be excluded in this case.", + "decision_date": "2023-01-17", + "case_citation": "2023 U.S. Dist. LEXIS 7528", + "date_mismatch": false + }, + { + "source_row": 112, + "caption": "U.S. v. Briscoe, No. 20-CR-1777, 2023 U.S. Dist. LEXIS 208806 (D.N.M. Nov. 21, 2023)", + "jurisdiction": "Federal", + "court_location": "NM (D.N.M.)", + "posture": "Pretrial", + "discipline": "FTM", + "decision_date_raw": "11/21/2023", + "decision_effect": "Limit", + "outcome": "Def motion denied", + "publication_status": "Pub", + "description": "Defense cites PCAST Report as reason to exclude FTM expert testimony, and court provides a lengthy analysis as to how the PCAST Report brings this method into question. The court acknowledges the criticism of FTM by the PCAST Report, as well as the criticisms levied against the PCAST Report by the scientific community. Nevertheless, the court admits the expert testimony with some limitations as to scope.", + "decision_date": "2023-11-21", + "case_citation": "2023 U.S. Dist. LEXIS 208806", + "date_mismatch": false + }, + { + "source_row": 113, + "caption": "U.S. v. Gist-Holden, 629 F. Supp. 3d 841 (N.D. Ind. 2022)", + "jurisdiction": "Federal", + "court_location": "IN (N.D. Ind.)", + "posture": "Pretrial", + "discipline": "FTM", + "decision_date_raw": "9/19/2022", + "decision_effect": "Admit", + "outcome": "Def motion denied", + "publication_status": "Pub", + "description": "Defense moves to exclude expert testimony on FTM analysis, but court is first to consider limitations proposed by PCAST Report. However, the court ultimately finds that the AFTE methodology for toolmark identification is generally accepted and meets Daubert standards.", + "decision_date": "2022-09-19", + "case_citation": "629 F. Supp. 3d 841", + "date_mismatch": false + }, + { + "source_row": 114, + "caption": "U.S. v. Lee, No. 19 C 641, 2022 U.S. Dist. LEXIS 150054 (N.D. Ill. Aug. 22, 2022)", + "jurisdiction": "Federal", + "court_location": "IL (N.D. Ill.)", + "posture": "Pretrial", + "discipline": "FTM", + "decision_date_raw": "8/22/2022", + "decision_effect": "Admit", + "outcome": "Def motion denied in part", + "publication_status": "Lexis", + "description": "Defendants move to exclude expert testimony on FTM analysis, stating that it has been the subject of criticism. The court introduces some reports, including the PCAST Report, to reflect this information. However, the court noted that case law has shown it has been up to their discretion whether to give these reports dispositive effect and chose to admit the testimony.", + "decision_date": "2022-08-22", + "case_citation": "2022 U.S. Dist. LEXIS 150054", + "date_mismatch": false + }, + { + "source_row": 115, + "caption": "U.S. v. Hunt, 63 F.4th 1229 (10th Cir. 2023)", + "jurisdiction": "Federal", + "court_location": "OK (10th Cir.)", + "posture": "Appeal", + "discipline": "FTM", + "decision_date_raw": "3/24/2023", + "decision_effect": "Affirm admission", + "outcome": "Conviction affirmed", + "publication_status": "Pub", + "description": "Defendant appeals convictions based on the admission of FTM expert testimony in trial. The court analyzes FTM examination as a whole, citing to the criticisms of the methods noted in the PCAST Report. The court puts heavy emphasis on the fact that the PCAST Report did not call for wholesale exclusion of FTM evidence from courts, but instead says it is up to the court's discretion whether to admit it. The court compares this to the PCAST Report's recommendation that bitemark analysis is not foundationally valid and is far from meeting those standards. Ultimately, the court decided that the district court showed sufficient support for the reliability of AFTE methodology and affirmed the admission of the testimony.", + "decision_date": "2023-03-24", + "case_citation": "63 F.4th 1229", + "date_mismatch": false + }, + { + "source_row": 116, + "caption": "U.S. v. Richardson, No. 19-20076, 2024 U.S. Dist. LEXIS 39167 (D. Kan. Mar. 6, 2024)", + "jurisdiction": "Federal", + "court_location": "KS (D. Kan.)", + "posture": "Pretrial", + "discipline": "FTM", + "decision_date_raw": "3/6/2024", + "decision_effect": "Limit", + "outcome": "Def motion denied in part, granted in part", + "publication_status": "Lexis", + "description": "Defendant filed a motion to exclude or limit the FTM expert testimony using the PCAST Report, but there is also extensive analysis by the court itself on the criticisms PCAST makes about FTM analysis. The court, however, is persuaded by the expert's argument that, since the publishing of the PCAST Report, there have been multiple studies establishing scientific validity for FTM analysis according to the requirements set out by PCAST. While the court maintain some skepticism, they decided to admit the evidence but with limitations as to the testimony.", + "decision_date": "2024-03-06", + "case_citation": "2024 U.S. Dist. LEXIS 39167", + "date_mismatch": false + }, + { + "source_row": 117, + "caption": "U.S. v. Randolph, No. 8:22-cr-145-TPB-AAS, 2024 U.S. Dist. LEXIS 71561 (M.D. Fla. Apr. 19, 2024)", + "jurisdiction": "Federal", + "court_location": "FL (M.D. Fla.)", + "posture": "Pretrial", + "discipline": "FTM", + "decision_date_raw": "4/19/2024", + "decision_effect": "Exclude", + "outcome": "Gov't motion granted", + "publication_status": "Lexis", + "description": "The government filed a motion to exclude the defendant's expert witness to counter the foundational validity of the FTM analysis the government intended to introduce. The court was particularly scathing regarding the PCAST Report and the expert witness's background as a researcher on the unreliability of forensic evidence.", + "decision_date": "2024-04-19", + "case_citation": "2024 U.S. Dist. LEXIS 71561", + "date_mismatch": false + }, + { + "source_row": 118, + "caption": "U.S. v. Abarca, No. 22-CR-20505, 2024 U.S. Dist. LEXIS 51359 (S.D. Fla. Mar. 5, 2024)", + "jurisdiction": "Federal", + "court_location": "FL (S.D. Fla.)", + "posture": "Pretrial", + "discipline": "FTM", + "decision_date_raw": "3/5/2025", + "decision_effect": "Admit with gov'ts proposed limits", + "outcome": "Def motion denied in part, granted in part", + "publication_status": "Lexis", + "description": "Defendant moves to exclude expert testimony as to FTM analysis and relies on PCAST Report to assert its lack of foundational validity. However, the court found the PCAST Report, among others, unpersuasive in undermining the foundational validity of FTM analysis. The court does limit the government's expert testimony to stay in like of the directives of the Dept. of Justice's Uniform Language for Testimony and Reports.", + "decision_date": "2025-03-05", + "case_citation": "2024 U.S. Dist. LEXIS 51359", + "date_mismatch": true + }, + { + "source_row": 119, + "caption": "U.S. v. Alvin, No. 22-20244-CR, 2024 U.S. Dist. LEXIS 7278 (S.D. Fla. Jan. 5, 2024)", + "jurisdiction": "Federal", + "court_location": "FL (S.D. Fla.)", + "posture": "Pretrial", + "discipline": "FTM", + "decision_date_raw": "1/5/2024", + "decision_effect": "Admit with gov'ts proposed limits", + "outcome": "Def motion denied in part, granted in part", + "publication_status": "Lexis", + "description": "Defendant moves to exclude expert testimony on FTM analysis based on PCAST and National Academy of Forensic Science (NAS) reports which are both critical of toolmark analysis. The court in this case is unpersuaded by both the reports and does not believe they undermine the foundational validity of FTM analysis. Relies on the fact that FTM analysis has always passed muster under Daubert motions. The court did apply some limitations to the expert testimony but did not exclude it.", + "decision_date": "2024-01-05", + "case_citation": "2024 U.S. Dist. LEXIS 7278", + "date_mismatch": false + }, + { + "source_row": 120, + "caption": "U.S. v. Boone, No. 23 Cr. 427, 2024 U.S. Dist. LEXIS 89710 (S.D.N.Y. May 13, 2024)", + "jurisdiction": "Federal", + "court_location": "NY (S.D.N.Y.)", + "posture": "Pretrial", + "discipline": "FTM", + "decision_date_raw": "5/13/2024", + "decision_effect": "Admit", + "outcome": "Def motion denied", + "publication_status": "Lexis", + "description": "The defendant moved to exclude the expert testimony on FTM analysis proffered by the government on the basis that it lacks foundational validity according to the PCAST Report. The government provided details regarding studies that have been published after the PCAST Report that follow the guidelines in the report on establishing foundational validity for FTM analysis. The court remarks that these studies provide further support for the \"testability\" of FTM analysis which thus satisfies Daubert.", + "decision_date": "2024-05-13", + "case_citation": "2024 U.S. Dist. LEXIS 89710", + "date_mismatch": false + }, + { + "source_row": 121, + "caption": "People v. Tidd, No. A167548, 2024 Cal. App. LEXIS 532 (Cal. Ct. App. Aug. 29, 2024)", + "jurisdiction": "State", + "court_location": "CA", + "posture": "Appeal", + "discipline": "FTM; & Fingerprints", + "decision_date_raw": "8/29/2024", + "decision_effect": "Reverse conviction", + "outcome": "Conviction reversed", + "publication_status": "Pub", + "description": "Defendant appeals his conviction arguing that the FTM analyst did not provide a sufficient basis for his opinion on matching the bullet cartridge to the gun. The PCAST Report was addressed to the analyst on cross-examination and the analyst only pointed out their limitations instead of pointing to any studies that established the foundational validity of FTM analysis or AFTE methodology. Due to the prejudicial nature of the testimony, the court ruled that the judgment must be reversed.", + "decision_date": "2024-08-29", + "case_citation": "2024 Cal. App. LEXIS 532", + "date_mismatch": false + }, + { + "source_row": 122, + "caption": "Phifer v. State, No. 233, 2020 Md. App. LEXIS 539 (Md. Ct. Spec. App. June 4, 2020)", + "jurisdiction": "State", + "court_location": "MD", + "posture": "Appeal", + "discipline": "FTM; & DNA", + "decision_date_raw": "6/4/2020", + "decision_effect": "Affirm admission", + "outcome": "Conviction affirmed", + "publication_status": "Lexis", + "description": "Defendant appealed his conviction on the basis that the trial court erred in admitting DNA and FTM expert testimony, relying on the PCAST Report to challenge the FTM testimony. The court ruled that while they acknowledged the criticism of FTM testimony in previous jurisprudence, they nonetheless found FTM identification to be valid and admissible. The court also found the DNA expert testimony to be admissible.", + "decision_date": "2020-06-04", + "case_citation": null, + "date_mismatch": false + }, + { + "source_row": 123, + "caption": "Commonwealth v. Chmiel, 662 Pa. 672 (Pa. 2020)", + "jurisdiction": "State", + "court_location": "PA", + "posture": "Post-conviction", + "discipline": "DNA", + "decision_date_raw": "10/21/2020", + "decision_effect": "Affirm admission", + "outcome": "Habeas petition denied", + "publication_status": "Pub", + "description": "Defendant hoped to appeal his conviction based on the PCAST Report, and his claims that hair microscopy and mitochondrial DNA match testimony should have been excluded. However, the court did not find the PCAST Report to be persuasive on that point. While they had concerns about microscopic hair comparison, they were satisfied that the experts acknowledged their own inherent limitations in their testimony.", + "decision_date": "2020-10-21", + "case_citation": "662 Pa. 672", + "date_mismatch": false + }, + { + "source_row": 124, + "caption": "In re Pers. Restraint of Curry, No. 54033-9-II, 2021 Wash. App. LEXIS 1461 (Ct. App. Wash. June 15, 2021)", + "jurisdiction": "State", + "court_location": "WA", + "posture": "Post-conviction", + "discipline": "FTM", + "decision_date_raw": "6/15/2021", + "decision_effect": "Affirm admission", + "outcome": "Conviction affirmed", + "publication_status": "Lexis", + "description": "Defendant filed a personal restraint petition and challenged the admission of FTM expert testimony in his case. Defendant relies on PCAST Report to challenge the general acceptance of FTM testing, however, previous jurisprudence has established that this jurisdiction does find general acceptance of FTM analysis and therefore the court affirmed the admission of the testimony.", + "decision_date": "2021-06-15", + "case_citation": null, + "date_mismatch": false + } + ] +} diff --git a/data/recent-caselaw-2026.json b/data/recent-caselaw-2026.json new file mode 100644 index 0000000..5d2cba9 --- /dev/null +++ b/data/recent-caselaw-2026.json @@ -0,0 +1,79 @@ +{ + "version": "RECENT-CASELAW-2026-07-14", + "retrieved_at": "2026-07-14T23:05:00Z", + "selection": "Published 2026 decisions with material evidence-law holdings, verified against official court or GovInfo sources.", + "records": [ + { + "case_name": "Ruffin v. BP Exploration & Production, Inc.", + "court": "United States Court of Appeals for the Fifth Circuit", + "decision_date": "2026-05-27", + "citation": "No. 23-30854", + "jurisdiction": "federal", + "proceeding": "civil", + "topics": ["FRE 702", "expert testimony", "toxic tort", "general causation"], + "holding": "The court rejected a categorical requirement of an exact numerical harmful dose but affirmed exclusion because the expert's chemical-specific and analytical gaps made the opinion irrelevant and unreliable.", + "source_url": "https://www.ca5.uscourts.gov/opinions/pub/23/23-30854-CV1.pdf", + "source_sha256": "7e905fa935d7ff5e7f4266be007f751cf3147a8bb9cfbc6a829c7248018c6177" + }, + { + "case_name": "Barry v. DePuy Synthes Companies", + "court": "United States Court of Appeals for the Federal Circuit", + "decision_date": "2026-01-20", + "citation": "No. 23-2226", + "jurisdiction": "federal", + "proceeding": "civil", + "topics": ["FRE 702", "FRE 403", "expert testimony", "survey evidence"], + "holding": "The majority reversed the post-trial exclusion of technical and survey experts and the resulting judgment as a matter of law, holding that the technical opinion applied rather than contradicted the claim construction and that the survey criticisms went to weight on this record.", + "source_url": "https://www.govinfo.gov/content/pkg/USCOURTS-ca13-23-02226/pdf/USCOURTS-ca13-23-02226-0.pdf", + "source_sha256": "8802f94249d9fd81a43e2b238792ac6c0139495addf36b4ae948a97046b36eb9" + }, + { + "case_name": "Ansari v. Jimenez", + "court": "United States Court of Appeals for the Sixth Circuit", + "decision_date": "2026-05-14", + "citation": "No. 24-1743", + "jurisdiction": "federal", + "proceeding": "civil", + "topics": ["FRE 403", "FRE 106", "documentary evidence", "civil rights"], + "holding": "The court affirmed exclusion of a 911-page prosecution file because uncertain provenance, irrelevant material, issue confusion, prejudice, and needless cumulative presentation substantially outweighed its potential probative value.", + "source_url": "https://www.opn.ca6.uscourts.gov/opinions.pdf/26a0148p-06.pdf", + "source_sha256": "0c653b2057241cbcf671148c59ecd3e7a928ae9a094f51fe1618b1de7fd43dbf" + }, + { + "case_name": "United States v. Ross", + "court": "United States Court of Appeals for the First Circuit", + "decision_date": "2026-04-02", + "citation": "No. 24-1907", + "jurisdiction": "federal", + "proceeding": "criminal", + "topics": ["FRE 403", "stipulations", "unfair prejudice", "knowledge"], + "holding": "The court affirmed admission and limited presentation of a small representative set of stipulated contraband exhibits because they remained probative of knowledge and were not unfairly prejudicial under Rule 403.", + "source_url": "https://www.ca1.uscourts.gov/sites/ca1/files/opnfiles/24-1907P-01A.pdf", + "source_sha256": "ab5123a8653fa3bc7218dcf14eba35581c36611b8df8ea7cfcb2f56c550cb40a" + }, + { + "case_name": "S.H. v. S.H.", + "court": "Supreme Court of the State of New York, Richmond County", + "decision_date": "2026-03-26", + "citation": "2026 NY Slip Op 26043", + "jurisdiction": "state", + "proceeding": "civil", + "topics": ["expert testimony", "expert disclosure", "hearsay", "unfair prejudice"], + "holding": "The court precluded a party from repurposing a neutral custody evaluator as her retained expert after an untimely and substantively inadequate disclosure created unfair prejudice.", + "source_url": "https://www.nycourts.gov/reporter/current/3dseries/2026/2026_26043.shtml", + "source_sha256": "92f265889da3917e439ccbd679616dca45f4f70b599d6c3a6d365e2355babef1" + }, + { + "case_name": "People v. Ford", + "court": "City Court of Utica, New York", + "decision_date": "2026-03-12", + "citation": "2026 NY Slip Op 26081", + "jurisdiction": "state", + "proceeding": "criminal", + "topics": ["hearsay", "excited utterance", "Confrontation Clause", "preliminary hearing"], + "holding": "The court treated a contemporaneous 911 call as a competent excited utterance and as nontestimonial because its primary purpose was resolving an ongoing emergency, so it could be considered at the felony preliminary hearing.", + "source_url": "https://www.nycourts.gov/reporter/current/3dseries/2026/2026_26081.shtml", + "source_sha256": "ca032ac5ad389ab535858c33c1d2af803c94f8c4e109ad93b024d5093692a2b8" + } + ] +} diff --git a/exports/latest.json b/exports/latest.json index 2c17520..af22506 100644 --- a/exports/latest.json +++ b/exports/latest.json @@ -1,12 +1,19 @@ { "benchmark_name": "EvidenceBench", - "benchmark_version": "2.0.0", + "benchmark_version": "3.0.0", "status": "official_snapshot", - "published_at": "2026-07-14T21:13:47Z", + "published_at": "2026-07-14T23:29:38Z", "authority_corpus": { - "version": "FRE-2025-12-01+CASELAW-2026-07-14", + "version": "FRE-2025-12-01+CASELAW-DATAGOV-2026-07-14", "fre_source_url": "https://www.uscourts.gov/sites/default/files/document/federal-rules-of-evidence.pdf", - "case_index_version": "CASELAW-2026-07-14", + "case_index_version": "CASELAW-DATAGOV-2026-07-14", + "data_gov_catalog_url": "https://catalog.data.gov/dataset/post-pcast-court-decisions-assessing-the-admissibility-of-forensic-science-evidence", + "data_gov_resource_url": "https://nij.ojp.gov/program/national-center-forensics/post-pcast-court-decisions-assessing-admissibility-forensic-science-evidence", + "data_gov_license": "Public Domain", + "data_gov_source_html_sha256": "1811a7762f961e0e046b2c957717aeebd46f0985b2a8c774a657ad162e723368", + "data_gov_records_sha256": "90990eb660d45be39515f9f08278abd4b080be3ce758eb1cf9e8854cc5ab95f1", + "recent_2026_source_version": "RECENT-CASELAW-2026-07-14", + "recent_2026_metadata_sha256": "ea97ec705a56e141df2a772b0644a7c33731ace07b13812314ecbabb8dde4467", "pending_amendments_excluded": [ { "rule": "FRE 801(d)(1)(A)", @@ -16,21 +23,42 @@ }, "holdout": { "status": "sealed", - "question_count": 112, + "question_count": 151, "fre_question_count": 96, - "caselaw_question_count": 16, + "caselaw_question_count": 55, + "data_gov_caselaw_question_count": 33, + "recent_2026_caselaw_question_count": 6, "development_question_count": 24, - "dataset_sha256": "13e319b3c332f4ede8d5b34fcc35a1a53db845626f1982bde3e622231859aea7", - "caselaw_matrix": { - "federal_civil_popular": 2, - "federal_civil_obscure": 2, - "federal_criminal_popular": 2, - "federal_criminal_obscure": 2, - "state_civil_popular": 2, - "state_civil_obscure": 2, - "state_criminal_popular": 2, - "state_criminal_obscure": 2 + "dataset_sha256": "9f6394738dfb0a3aa722d11c1a37f239828ea1adc4f847c3e7f8134f2227869f", + "data_gov_window": { + "start": "2020-01-01", + "end": "2024-12-31" }, + "data_gov_years": { + "2020": 13, + "2021": 6, + "2022": 6, + "2023": 6, + "2024": 2 + }, + "data_gov_jurisdictions": { + "federal": 12, + "state": 21 + }, + "data_gov_proceedings": { + "civil": 3, + "criminal": 30 + }, + "recent_2026_jurisdictions": { + "federal": 4, + "state": 2 + }, + "recent_2026_proceedings": { + "civil": 4, + "criminal": 2 + }, + "data_gov_selection": "unique+citable+publication_status_published+decision_date_2020_2024", + "recent_2026_selection": "published+official_source+material_evidence_holding+decision_date_2026", "categories": [ "relevance", "hearsay", @@ -45,9 +73,9 @@ ] }, "protocol": { - "name": "evidencebench-v2-closed-book", - "prompt_version": "evidencebench-v2-closed-book", - "prompt_sha256": "55f6eaba94060b280ef31c6a009618f4c98bb2ce336f607c82b0c49fb40a147a", + "name": "evidencebench-v3-closed-book", + "prompt_version": "evidencebench-v3-closed-book", + "prompt_sha256": "250960491ca474811fd1d32ddc2c207992c04a766de7d04ad76ebaca29636221", "runs_per_question": 1, "tools_enabled": false, "overall_formula": "0.70 * answer_accuracy + 0.30 * citation_f1" @@ -59,351 +87,941 @@ "model_id": "google/gemini-3.5-flash", "provider_route": "OpenRouter pinned route", "status": "official", - "overall": 0.9643, - "answer_accuracy": 1.0, - "citation_f1": 0.881, - "hallucination_rate": 0.0079, + "overall": 0.8517, + "answer_accuracy": 0.9007, + "citation_f1": 0.7373, + "hallucination_rate": 0.1524, "dimensions": { - "domain": { - "caselaw": { - "answer_accuracy": 1.0, - "citation_f1": 0.9792, - "count": 16, - "hallucination_rate": 0.0588, - "overall": 0.9937, + "decision_year": { + "2020": { + "answer_accuracy": 0.3846, + "citation_f1": 0.5385, + "count": 13, + "hallucination_rate": 0.4167, + "overall": 0.4308, "unsupported_citation_count": 0 }, - "fre": { - "answer_accuracy": 1.0, - "citation_f1": 0.8646, - "count": 96, + "2021": { + "answer_accuracy": 0.6667, + "citation_f1": 0.3333, + "count": 6, + "hallucination_rate": 0.7143, + "overall": 0.5667, + "unsupported_citation_count": 0 + }, + "2022": { + "answer_accuracy": 0.6667, + "citation_f1": 0.1667, + "count": 6, + "hallucination_rate": 0.8, + "overall": 0.5167, + "unsupported_citation_count": 0 + }, + "2023": { + "answer_accuracy": 0.6667, + "citation_f1": 0.2778, + "count": 6, + "hallucination_rate": 0.6667, + "overall": 0.55, + "unsupported_citation_count": 0 + }, + "2024": { + "answer_accuracy": 0.5, + "citation_f1": 0.0, + "count": 2, "hallucination_rate": 0.0, - "overall": 0.9594, - "unsupported_citation_count": 15 + "overall": 0.35, + "unsupported_citation_count": 0 + }, + "2026": { + "answer_accuracy": 1.0, + "citation_f1": 0.0, + "count": 6, + "hallucination_rate": 0.6, + "overall": 0.7, + "unsupported_citation_count": 0 } }, - "familiarity": { - "obscure": { + "discipline": { + "bitemark": { "answer_accuracy": 1.0, - "citation_f1": 1.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 1.0, + "overall": 0.7, + "unsupported_citation_count": 0 + }, + "dna": { + "answer_accuracy": 0.375, + "citation_f1": 0.25, "count": 8, + "hallucination_rate": 0.7778, + "overall": 0.3375, + "unsupported_citation_count": 0 + }, + "dna_pcast_dicta": { + "answer_accuracy": 1.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 1.0, + "overall": 0.7, + "unsupported_citation_count": 0 + }, + "documentary_evidence": { + "answer_accuracy": 1.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 1.0, + "overall": 0.7, + "unsupported_citation_count": 0 + }, + "expert_disclosure": { + "answer_accuracy": 1.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 1.0, + "overall": 0.7, + "unsupported_citation_count": 0 + }, + "fingerprints": { + "answer_accuracy": 0.5, + "citation_f1": 0.0, + "count": 2, + "hallucination_rate": 1.0, + "overall": 0.35, + "unsupported_citation_count": 0 + }, + "footwear": { + "answer_accuracy": 1.0, + "citation_f1": 0.0, + "count": 1, "hallucination_rate": 0.0, - "overall": 1.0, + "overall": 0.7, "unsupported_citation_count": 0 }, - "popular": { + "ftm": { + "answer_accuracy": 0.5625, + "citation_f1": 0.4792, + "count": 16, + "hallucination_rate": 0.4667, + "overall": 0.5375, + "unsupported_citation_count": 0 + }, + "ftm_fingerprints": { + "answer_accuracy": 0.5, + "citation_f1": 0.0, + "count": 2, + "hallucination_rate": 1.0, + "overall": 0.35, + "unsupported_citation_count": 0 + }, + "hearsay_and_confrontation": { "answer_accuracy": 1.0, - "citation_f1": 0.9583, - "count": 8, - "hallucination_rate": 0.1111, - "overall": 0.9875, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 1.0, + "overall": 0.7, "unsupported_citation_count": 0 - } - }, - "jurisdiction": { - "federal": { + }, + "pcast_dicta": { "answer_accuracy": 1.0, "citation_f1": 1.0, - "count": 8, + "count": 1, "hallucination_rate": 0.0, "overall": 1.0, "unsupported_citation_count": 0 }, - "state": { + "pcast_fre_702": { + "answer_accuracy": 0.0, + "citation_f1": 1.0, + "count": 1, + "hallucination_rate": 0.0, + "overall": 0.3, + "unsupported_citation_count": 0 + }, + "rule_403_stipulations": { "answer_accuracy": 1.0, - "citation_f1": 0.9583, - "count": 8, - "hallucination_rate": 0.1111, - "overall": 0.9875, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 0.0, + "overall": 0.7, "unsupported_citation_count": 0 - } - }, - "proceeding": { - "civil": { + }, + "technical_and_survey_experts": { "answer_accuracy": 1.0, - "citation_f1": 1.0, - "count": 8, + "citation_f1": 0.0, + "count": 1, "hallucination_rate": 0.0, - "overall": 1.0, + "overall": 0.7, "unsupported_citation_count": 0 }, - "criminal": { + "toxic_tort_experts": { "answer_accuracy": 1.0, - "citation_f1": 0.9583, - "count": 8, - "hallucination_rate": 0.1111, - "overall": 0.9875, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 0.0, + "overall": 0.7, "unsupported_citation_count": 0 } - } - }, - "cost_usd": 0.132404, - "input_tokens": 20931, - "output_tokens": 11223 - }, - { - "display_name": "Mistral Large 3", - "family": "Mistral", - "model_id": "mistralai/mistral-large-2512", - "provider_route": "OpenRouter pinned route", - "status": "official", - "overall": 0.9641, - "answer_accuracy": 1.0, - "citation_f1": 0.8804, - "hallucination_rate": 0.0543, - "dimensions": { + }, "domain": { "caselaw": { - "answer_accuracy": 1.0, - "citation_f1": 0.8125, - "count": 16, - "hallucination_rate": 0.1111, - "overall": 0.9437, + "answer_accuracy": 0.7273, + "citation_f1": 0.4788, + "count": 55, + "hallucination_rate": 0.4545, + "overall": 0.6527, "unsupported_citation_count": 0 }, "fre": { "answer_accuracy": 1.0, - "citation_f1": 0.8917, + "citation_f1": 0.8854, "count": 96, - "hallucination_rate": 0.045, - "overall": 0.9675, - "unsupported_citation_count": 9 + "hallucination_rate": 0.0, + "overall": 0.9656, + "unsupported_citation_count": 13 } }, "familiarity": { "obscure": { - "answer_accuracy": 1.0, - "citation_f1": 0.75, - "count": 8, - "hallucination_rate": 0.1111, - "overall": 0.925, + "answer_accuracy": 0.6809, + "citation_f1": 0.3972, + "count": 47, + "hallucination_rate": 0.5217, + "overall": 0.5957, "unsupported_citation_count": 0 }, "popular": { "answer_accuracy": 1.0, - "citation_f1": 0.875, + "citation_f1": 0.9583, "count": 8, "hallucination_rate": 0.1111, - "overall": 0.9625, + "overall": 0.9875, "unsupported_citation_count": 0 } }, "jurisdiction": { "federal": { - "answer_accuracy": 1.0, - "citation_f1": 1.0, - "count": 8, - "hallucination_rate": 0.0, - "overall": 1.0, + "answer_accuracy": 0.75, + "citation_f1": 0.625, + "count": 24, + "hallucination_rate": 0.1905, + "overall": 0.7125, "unsupported_citation_count": 0 }, "state": { - "answer_accuracy": 1.0, - "citation_f1": 0.625, - "count": 8, - "hallucination_rate": 0.2857, - "overall": 0.8875, + "answer_accuracy": 0.7097, + "citation_f1": 0.3656, + "count": 31, + "hallucination_rate": 0.6176, + "overall": 0.6065, "unsupported_citation_count": 0 } }, "proceeding": { "civil": { - "answer_accuracy": 1.0, - "citation_f1": 0.75, - "count": 8, - "hallucination_rate": 0.1, - "overall": 0.925, + "answer_accuracy": 0.8667, + "citation_f1": 0.6444, + "count": 15, + "hallucination_rate": 0.2667, + "overall": 0.8, "unsupported_citation_count": 0 }, "criminal": { + "answer_accuracy": 0.675, + "citation_f1": 0.4167, + "count": 40, + "hallucination_rate": 0.525, + "overall": 0.5975, + "unsupported_citation_count": 0 + } + }, + "publication_status": { + "published": { + "answer_accuracy": 0.6154, + "citation_f1": 0.2991, + "count": 39, + "hallucination_rate": 0.6, + "overall": 0.5205, + "unsupported_citation_count": 0 + } + }, + "recency": { + "2020-2021": { + "answer_accuracy": 0.4737, + "citation_f1": 0.4737, + "count": 19, + "hallucination_rate": 0.5263, + "overall": 0.4737, + "unsupported_citation_count": 0 + }, + "2022-2024": { + "answer_accuracy": 0.6429, + "citation_f1": 0.1905, + "count": 14, + "hallucination_rate": 0.7273, + "overall": 0.5071, + "unsupported_citation_count": 0 + }, + "2026": { "answer_accuracy": 1.0, - "citation_f1": 0.875, - "count": 8, - "hallucination_rate": 0.125, - "overall": 0.9625, + "citation_f1": 0.0, + "count": 6, + "hallucination_rate": 0.6, + "overall": 0.7, + "unsupported_citation_count": 0 + } + }, + "source": { + "data_gov_doj_nij": { + "answer_accuracy": 0.5455, + "citation_f1": 0.3535, + "count": 33, + "hallucination_rate": 0.6, + "overall": 0.4879, + "unsupported_citation_count": 0 + }, + "official_courts_2026": { + "answer_accuracy": 1.0, + "citation_f1": 0.0, + "count": 6, + "hallucination_rate": 0.6, + "overall": 0.7, "unsupported_citation_count": 0 } } }, - "cost_usd": 0.022883, - "input_tokens": 21011, - "output_tokens": 10797 + "cost_usd": 0.186876, + "input_tokens": 29958, + "output_tokens": 15771 }, { - "display_name": "Grok 4.5", - "family": "xAI", - "model_id": "x-ai/grok-4.5", + "display_name": "Claude Fable 5", + "family": "Anthropic", + "model_id": "anthropic/claude-fable-5", "provider_route": "OpenRouter pinned route", "status": "official", - "overall": 0.942, - "answer_accuracy": 1.0, - "citation_f1": 0.8065, - "hallucination_rate": 0.0, + "overall": 0.8453, + "answer_accuracy": 0.894, + "citation_f1": 0.7316, + "hallucination_rate": 0.1042, "dimensions": { - "domain": { - "caselaw": { - "answer_accuracy": 1.0, - "citation_f1": 1.0, - "count": 16, - "hallucination_rate": 0.0, - "overall": 1.0, + "decision_year": { + "2020": { + "answer_accuracy": 0.5385, + "citation_f1": 0.6795, + "count": 13, + "hallucination_rate": 0.1667, + "overall": 0.5808, + "unsupported_citation_count": 5 + }, + "2021": { + "answer_accuracy": 0.6667, + "citation_f1": 0.5278, + "count": 6, + "hallucination_rate": 0.125, + "overall": 0.625, + "unsupported_citation_count": 3 + }, + "2022": { + "answer_accuracy": 0.5, + "citation_f1": 0.2778, + "count": 6, + "hallucination_rate": 0.25, + "overall": 0.4333, + "unsupported_citation_count": 4 + }, + "2023": { + "answer_accuracy": 0.5, + "citation_f1": 0.4444, + "count": 6, + "hallucination_rate": 0.4, + "overall": 0.4833, "unsupported_citation_count": 0 }, - "fre": { + "2024": { + "answer_accuracy": 0.5, + "citation_f1": 0.0, + "count": 2, + "hallucination_rate": 0.25, + "overall": 0.35, + "unsupported_citation_count": 3 + }, + "2026": { "answer_accuracy": 1.0, - "citation_f1": 0.7743, - "count": 96, - "hallucination_rate": 0.0, - "overall": 0.9323, - "unsupported_citation_count": 25 + "citation_f1": 0.0, + "count": 6, + "hallucination_rate": 0.2727, + "overall": 0.7, + "unsupported_citation_count": 4 } }, - "familiarity": { - "obscure": { - "answer_accuracy": 1.0, - "citation_f1": 1.0, - "count": 8, + "discipline": { + "bitemark": { + "answer_accuracy": 0.0, + "citation_f1": 0.0, + "count": 1, "hallucination_rate": 0.0, - "overall": 1.0, + "overall": 0.0, "unsupported_citation_count": 0 }, - "popular": { - "answer_accuracy": 1.0, - "citation_f1": 1.0, + "dna": { + "answer_accuracy": 0.375, + "citation_f1": 0.5, "count": 8, - "hallucination_rate": 0.0, - "overall": 1.0, + "hallucination_rate": 0.4, + "overall": 0.4125, + "unsupported_citation_count": 1 + }, + "dna_pcast_dicta": { + "answer_accuracy": 1.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 1.0, + "overall": 0.7, "unsupported_citation_count": 0 - } - }, - "jurisdiction": { - "federal": { + }, + "documentary_evidence": { "answer_accuracy": 1.0, - "citation_f1": 1.0, - "count": 8, + "citation_f1": 0.0, + "count": 1, "hallucination_rate": 0.0, - "overall": 1.0, + "overall": 0.7, "unsupported_citation_count": 0 }, - "state": { + "expert_disclosure": { "answer_accuracy": 1.0, - "citation_f1": 1.0, - "count": 8, + "citation_f1": 0.0, + "count": 1, "hallucination_rate": 0.0, - "overall": 1.0, + "overall": 0.7, "unsupported_citation_count": 0 - } - }, - "proceeding": { - "civil": { + }, + "fingerprints": { + "answer_accuracy": 0.5, + "citation_f1": 0.0, + "count": 2, + "hallucination_rate": 0.3333, + "overall": 0.35, + "unsupported_citation_count": 4 + }, + "footwear": { "answer_accuracy": 1.0, - "citation_f1": 1.0, - "count": 8, + "citation_f1": 0.0, + "count": 1, "hallucination_rate": 0.0, - "overall": 1.0, - "unsupported_citation_count": 0 + "overall": 0.7, + "unsupported_citation_count": 2 }, - "criminal": { + "ftm": { + "answer_accuracy": 0.625, + "citation_f1": 0.6146, + "count": 16, + "hallucination_rate": 0.1111, + "overall": 0.6219, + "unsupported_citation_count": 5 + }, + "ftm_fingerprints": { + "answer_accuracy": 0.5, + "citation_f1": 0.5, + "count": 2, + "hallucination_rate": 0.0, + "overall": 0.5, + "unsupported_citation_count": 1 + }, + "hearsay_and_confrontation": { + "answer_accuracy": 1.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 0.6667, + "overall": 0.7, + "unsupported_citation_count": 1 + }, + "pcast_dicta": { "answer_accuracy": 1.0, "citation_f1": 1.0, - "count": 8, + "count": 1, "hallucination_rate": 0.0, "overall": 1.0, "unsupported_citation_count": 0 + }, + "pcast_fre_702": { + "answer_accuracy": 0.0, + "citation_f1": 0.5, + "count": 1, + "hallucination_rate": 0.0, + "overall": 0.15, + "unsupported_citation_count": 2 + }, + "rule_403_stipulations": { + "answer_accuracy": 1.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 0.0, + "overall": 0.7, + "unsupported_citation_count": 1 + }, + "technical_and_survey_experts": { + "answer_accuracy": 1.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 0.0, + "overall": 0.7, + "unsupported_citation_count": 1 + }, + "toxic_tort_experts": { + "answer_accuracy": 1.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 0.3333, + "overall": 0.7, + "unsupported_citation_count": 1 } - } - }, - "cost_usd": 0.404352, - "input_tokens": 45468, - "output_tokens": 57516 - }, - { - "display_name": "DeepSeek V4 Pro", - "family": "DeepSeek", - "model_id": "deepseek/deepseek-v4-pro", - "provider_route": "OpenRouter pinned route", - "status": "official", - "overall": 0.9289, - "answer_accuracy": 0.9554, - "citation_f1": 0.8673, - "hallucination_rate": 0.0084, - "dimensions": { + }, "domain": { "caselaw": { - "answer_accuracy": 0.9375, - "citation_f1": 0.9375, - "count": 16, - "hallucination_rate": 0.0, - "overall": 0.9375, - "unsupported_citation_count": 0 + "answer_accuracy": 0.7273, + "citation_f1": 0.5539, + "count": 55, + "hallucination_rate": 0.2125, + "overall": 0.6753, + "unsupported_citation_count": 20 }, "fre": { - "answer_accuracy": 0.9583, - "citation_f1": 0.8556, + "answer_accuracy": 0.9896, + "citation_f1": 0.8333, "count": 96, - "hallucination_rate": 0.0096, - "overall": 0.9275, - "unsupported_citation_count": 10 + "hallucination_rate": 0.05, + "overall": 0.9427, + "unsupported_citation_count": 35 } }, "familiarity": { "obscure": { - "answer_accuracy": 0.875, - "citation_f1": 0.875, - "count": 8, - "hallucination_rate": 0.0, - "overall": 0.875, - "unsupported_citation_count": 0 + "answer_accuracy": 0.6809, + "citation_f1": 0.4965, + "count": 47, + "hallucination_rate": 0.2239, + "overall": 0.6255, + "unsupported_citation_count": 19 }, "popular": { "answer_accuracy": 1.0, - "citation_f1": 1.0, + "citation_f1": 0.8917, "count": 8, - "hallucination_rate": 0.0, - "overall": 1.0, - "unsupported_citation_count": 0 + "hallucination_rate": 0.1538, + "overall": 0.9675, + "unsupported_citation_count": 1 } }, "jurisdiction": { "federal": { - "answer_accuracy": 1.0, - "citation_f1": 1.0, - "count": 8, - "hallucination_rate": 0.0, - "overall": 1.0, - "unsupported_citation_count": 0 + "answer_accuracy": 0.7917, + "citation_f1": 0.6444, + "count": 24, + "hallucination_rate": 0.0556, + "overall": 0.7475, + "unsupported_citation_count": 9 }, "state": { - "answer_accuracy": 0.875, - "citation_f1": 0.875, - "count": 8, - "hallucination_rate": 0.0, - "overall": 0.875, - "unsupported_citation_count": 0 + "answer_accuracy": 0.6774, + "citation_f1": 0.4839, + "count": 31, + "hallucination_rate": 0.3409, + "overall": 0.6194, + "unsupported_citation_count": 11 } }, "proceeding": { "civil": { - "answer_accuracy": 1.0, - "citation_f1": 1.0, - "count": 8, - "hallucination_rate": 0.0, - "overall": 1.0, - "unsupported_citation_count": 0 + "answer_accuracy": 0.8667, + "citation_f1": 0.5756, + "count": 15, + "hallucination_rate": 0.12, + "overall": 0.7793, + "unsupported_citation_count": 6 }, "criminal": { - "answer_accuracy": 0.875, - "citation_f1": 0.875, + "answer_accuracy": 0.675, + "citation_f1": 0.5458, + "count": 40, + "hallucination_rate": 0.2545, + "overall": 0.6362, + "unsupported_citation_count": 14 + } + }, + "publication_status": { + "published": { + "answer_accuracy": 0.6154, + "citation_f1": 0.4188, + "count": 39, + "hallucination_rate": 0.2222, + "overall": 0.5564, + "unsupported_citation_count": 19 + } + }, + "recency": { + "2020-2021": { + "answer_accuracy": 0.5789, + "citation_f1": 0.6316, + "count": 19, + "hallucination_rate": 0.1538, + "overall": 0.5947, + "unsupported_citation_count": 8 + }, + "2022-2024": { + "answer_accuracy": 0.5, + "citation_f1": 0.3095, + "count": 14, + "hallucination_rate": 0.2941, + "overall": 0.4429, + "unsupported_citation_count": 7 + }, + "2026": { + "answer_accuracy": 1.0, + "citation_f1": 0.0, + "count": 6, + "hallucination_rate": 0.2727, + "overall": 0.7, + "unsupported_citation_count": 4 + } + }, + "source": { + "data_gov_doj_nij": { + "answer_accuracy": 0.5455, + "citation_f1": 0.4949, + "count": 33, + "hallucination_rate": 0.2093, + "overall": 0.5303, + "unsupported_citation_count": 15 + }, + "official_courts_2026": { + "answer_accuracy": 1.0, + "citation_f1": 0.0, + "count": 6, + "hallucination_rate": 0.2727, + "overall": 0.7, + "unsupported_citation_count": 4 + } + } + }, + "cost_usd": 2.29467, + "input_tokens": 46022, + "output_tokens": 36689 + }, + { + "display_name": "Grok 4.5", + "family": "xAI", + "model_id": "x-ai/grok-4.5", + "provider_route": "OpenRouter pinned route", + "status": "official", + "overall": 0.8433, + "answer_accuracy": 0.9139, + "citation_f1": 0.6786, + "hallucination_rate": 0.0692, + "dimensions": { + "decision_year": { + "2020": { + "answer_accuracy": 0.4615, + "citation_f1": 0.6154, + "count": 13, + "hallucination_rate": 0.3333, + "overall": 0.5077, + "unsupported_citation_count": 0 + }, + "2021": { + "answer_accuracy": 0.8333, + "citation_f1": 0.2778, + "count": 6, + "hallucination_rate": 0.5, + "overall": 0.6667, + "unsupported_citation_count": 0 + }, + "2022": { + "answer_accuracy": 0.8333, + "citation_f1": 0.0, + "count": 6, + "hallucination_rate": 1.0, + "overall": 0.5833, + "unsupported_citation_count": 0 + }, + "2023": { + "answer_accuracy": 0.5, + "citation_f1": 0.3333, + "count": 6, + "hallucination_rate": 0.3333, + "overall": 0.45, + "unsupported_citation_count": 0 + }, + "2024": { + "answer_accuracy": 0.5, + "citation_f1": 0.0, + "count": 2, + "hallucination_rate": 0.0, + "overall": 0.35, + "unsupported_citation_count": 0 + }, + "2026": { + "answer_accuracy": 1.0, + "citation_f1": 0.0, + "count": 6, + "hallucination_rate": 0.1429, + "overall": 0.7, + "unsupported_citation_count": 3 + } + }, + "discipline": { + "bitemark": { + "answer_accuracy": 0.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 0.0, + "overall": 0.0, + "unsupported_citation_count": 0 + }, + "dna": { + "answer_accuracy": 0.5, + "citation_f1": 0.3333, + "count": 8, + "hallucination_rate": 0.625, + "overall": 0.45, + "unsupported_citation_count": 0 + }, + "dna_pcast_dicta": { + "answer_accuracy": 1.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 0.0, + "overall": 0.7, + "unsupported_citation_count": 0 + }, + "documentary_evidence": { + "answer_accuracy": 1.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 0.0, + "overall": 0.7, + "unsupported_citation_count": 0 + }, + "expert_disclosure": { + "answer_accuracy": 1.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 0.0, + "overall": 0.7, + "unsupported_citation_count": 0 + }, + "fingerprints": { + "answer_accuracy": 0.5, + "citation_f1": 0.0, + "count": 2, + "hallucination_rate": 0.0, + "overall": 0.35, + "unsupported_citation_count": 0 + }, + "footwear": { + "answer_accuracy": 1.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 0.0, + "overall": 0.7, + "unsupported_citation_count": 0 + }, + "ftm": { + "answer_accuracy": 0.6875, + "citation_f1": 0.5625, + "count": 16, + "hallucination_rate": 0.1818, + "overall": 0.65, + "unsupported_citation_count": 0 + }, + "ftm_fingerprints": { + "answer_accuracy": 0.5, + "citation_f1": 0.0, + "count": 2, + "hallucination_rate": 0.0, + "overall": 0.35, + "unsupported_citation_count": 0 + }, + "hearsay_and_confrontation": { + "answer_accuracy": 1.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 0.5, + "overall": 0.7, + "unsupported_citation_count": 1 + }, + "pcast_dicta": { + "answer_accuracy": 1.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 1.0, + "overall": 0.7, + "unsupported_citation_count": 0 + }, + "pcast_fre_702": { + "answer_accuracy": 0.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 1.0, + "overall": 0.0, + "unsupported_citation_count": 0 + }, + "rule_403_stipulations": { + "answer_accuracy": 1.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 0.0, + "overall": 0.7, + "unsupported_citation_count": 1 + }, + "technical_and_survey_experts": { + "answer_accuracy": 1.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 0.0, + "overall": 0.7, + "unsupported_citation_count": 1 + }, + "toxic_tort_experts": { + "answer_accuracy": 1.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 0.0, + "overall": 0.7, + "unsupported_citation_count": 0 + } + }, + "domain": { + "caselaw": { + "answer_accuracy": 0.7636, + "citation_f1": 0.497, + "count": 55, + "hallucination_rate": 0.2444, + "overall": 0.6836, + "unsupported_citation_count": 3 + }, + "fre": { + "answer_accuracy": 1.0, + "citation_f1": 0.7826, + "count": 96, + "hallucination_rate": 0.0, + "overall": 0.9348, + "unsupported_citation_count": 27 + } + }, + "familiarity": { + "obscure": { + "answer_accuracy": 0.7234, + "citation_f1": 0.4113, + "count": 47, + "hallucination_rate": 0.2973, + "overall": 0.6298, + "unsupported_citation_count": 3 + }, + "popular": { + "answer_accuracy": 1.0, + "citation_f1": 1.0, "count": 8, "hallucination_rate": 0.0, - "overall": 0.875, + "overall": 1.0, "unsupported_citation_count": 0 } + }, + "jurisdiction": { + "federal": { + "answer_accuracy": 0.875, + "citation_f1": 0.5278, + "count": 24, + "hallucination_rate": 0.2174, + "overall": 0.7708, + "unsupported_citation_count": 2 + }, + "state": { + "answer_accuracy": 0.6774, + "citation_f1": 0.4731, + "count": 31, + "hallucination_rate": 0.2727, + "overall": 0.6161, + "unsupported_citation_count": 1 + } + }, + "proceeding": { + "civil": { + "answer_accuracy": 0.8667, + "citation_f1": 0.5333, + "count": 15, + "hallucination_rate": 0.2143, + "overall": 0.7667, + "unsupported_citation_count": 1 + }, + "criminal": { + "answer_accuracy": 0.725, + "citation_f1": 0.4833, + "count": 40, + "hallucination_rate": 0.2581, + "overall": 0.6525, + "unsupported_citation_count": 2 + } + }, + "publication_status": { + "published": { + "answer_accuracy": 0.6667, + "citation_f1": 0.2991, + "count": 39, + "hallucination_rate": 0.3571, + "overall": 0.5564, + "unsupported_citation_count": 3 + } + }, + "recency": { + "2020-2021": { + "answer_accuracy": 0.5789, + "citation_f1": 0.5088, + "count": 19, + "hallucination_rate": 0.375, + "overall": 0.5579, + "unsupported_citation_count": 0 + }, + "2022-2024": { + "answer_accuracy": 0.6429, + "citation_f1": 0.1429, + "count": 14, + "hallucination_rate": 0.6, + "overall": 0.4929, + "unsupported_citation_count": 0 + }, + "2026": { + "answer_accuracy": 1.0, + "citation_f1": 0.0, + "count": 6, + "hallucination_rate": 0.1429, + "overall": 0.7, + "unsupported_citation_count": 3 + } + }, + "source": { + "data_gov_doj_nij": { + "answer_accuracy": 0.6061, + "citation_f1": 0.3535, + "count": 33, + "hallucination_rate": 0.4286, + "overall": 0.5303, + "unsupported_citation_count": 0 + }, + "official_courts_2026": { + "answer_accuracy": 1.0, + "citation_f1": 0.0, + "count": 6, + "hallucination_rate": 0.1429, + "overall": 0.7, + "unsupported_citation_count": 3 + } } }, - "cost_usd": 0.102966, - "input_tokens": 20681, - "output_tokens": 39696 + "cost_usd": 0.83576, + "input_tokens": 63148, + "output_tokens": 125732 }, { "display_name": "GPT-5.6 Sol", @@ -411,36 +1029,208 @@ "model_id": "openai/gpt-5.6-sol", "provider_route": "OpenRouter pinned route", "status": "official", - "overall": 0.9268, - "answer_accuracy": 0.9464, - "citation_f1": 0.881, - "hallucination_rate": 0.0, + "overall": 0.8033, + "answer_accuracy": 0.8212, + "citation_f1": 0.7616, + "hallucination_rate": 0.0357, "dimensions": { - "domain": { - "caselaw": { - "answer_accuracy": 0.875, - "citation_f1": 0.8125, + "decision_year": { + "2020": { + "answer_accuracy": 0.3077, + "citation_f1": 0.359, + "count": 13, + "hallucination_rate": 0.1667, + "overall": 0.3231, + "unsupported_citation_count": 0 + }, + "2021": { + "answer_accuracy": 0.3333, + "citation_f1": 0.3333, + "count": 6, + "hallucination_rate": 0.0, + "overall": 0.3333, + "unsupported_citation_count": 0 + }, + "2022": { + "answer_accuracy": 0.3333, + "citation_f1": 0.1667, + "count": 6, + "hallucination_rate": 0.5, + "overall": 0.2833, + "unsupported_citation_count": 0 + }, + "2023": { + "answer_accuracy": 0.1667, + "citation_f1": 0.1667, + "count": 6, + "hallucination_rate": 0.0, + "overall": 0.1667, + "unsupported_citation_count": 0 + }, + "2024": { + "answer_accuracy": 0.0, + "citation_f1": 0.0, + "count": 2, + "hallucination_rate": 0.0, + "overall": 0.0, + "unsupported_citation_count": 0 + }, + "2026": { + "answer_accuracy": 0.5, + "citation_f1": 0.0, + "count": 6, + "hallucination_rate": 0.5, + "overall": 0.35, + "unsupported_citation_count": 0 + } + }, + "discipline": { + "bitemark": { + "answer_accuracy": 0.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 0.0, + "overall": 0.0, + "unsupported_citation_count": 0 + }, + "dna": { + "answer_accuracy": 0.375, + "citation_f1": 0.25, + "count": 8, + "hallucination_rate": 0.3333, + "overall": 0.3375, + "unsupported_citation_count": 0 + }, + "dna_pcast_dicta": { + "answer_accuracy": 0.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 0.0, + "overall": 0.0, + "unsupported_citation_count": 0 + }, + "documentary_evidence": { + "answer_accuracy": 1.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 0.0, + "overall": 0.7, + "unsupported_citation_count": 0 + }, + "expert_disclosure": { + "answer_accuracy": 0.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 0.0, + "overall": 0.0, + "unsupported_citation_count": 0 + }, + "fingerprints": { + "answer_accuracy": 0.0, + "citation_f1": 0.0, + "count": 2, + "hallucination_rate": 0.0, + "overall": 0.0, + "unsupported_citation_count": 0 + }, + "footwear": { + "answer_accuracy": 0.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 0.0, + "overall": 0.0, + "unsupported_citation_count": 0 + }, + "ftm": { + "answer_accuracy": 0.3125, + "citation_f1": 0.3542, "count": 16, + "hallucination_rate": 0.1429, + "overall": 0.325, + "unsupported_citation_count": 0 + }, + "ftm_fingerprints": { + "answer_accuracy": 0.0, + "citation_f1": 0.0, + "count": 2, "hallucination_rate": 0.0, - "overall": 0.8562, + "overall": 0.0, + "unsupported_citation_count": 0 + }, + "hearsay_and_confrontation": { + "answer_accuracy": 1.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 1.0, + "overall": 0.7, + "unsupported_citation_count": 0 + }, + "pcast_dicta": { + "answer_accuracy": 1.0, + "citation_f1": 1.0, + "count": 1, + "hallucination_rate": 0.0, + "overall": 1.0, + "unsupported_citation_count": 0 + }, + "pcast_fre_702": { + "answer_accuracy": 0.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 0.0, + "overall": 0.0, + "unsupported_citation_count": 0 + }, + "rule_403_stipulations": { + "answer_accuracy": 1.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 0.0, + "overall": 0.7, + "unsupported_citation_count": 0 + }, + "technical_and_survey_experts": { + "answer_accuracy": 0.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 0.0, + "overall": 0.0, + "unsupported_citation_count": 0 + }, + "toxic_tort_experts": { + "answer_accuracy": 0.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 0.0, + "overall": 0.0, + "unsupported_citation_count": 0 + } + }, + "domain": { + "caselaw": { + "answer_accuracy": 0.5091, + "citation_f1": 0.4424, + "count": 55, + "hallucination_rate": 0.1515, + "overall": 0.4891, "unsupported_citation_count": 0 }, "fre": { - "answer_accuracy": 0.9583, - "citation_f1": 0.8924, + "answer_accuracy": 1.0, + "citation_f1": 0.9444, "count": 96, "hallucination_rate": 0.0, - "overall": 0.9385, - "unsupported_citation_count": 3 + "overall": 0.9833, + "unsupported_citation_count": 2 } }, "familiarity": { "obscure": { - "answer_accuracy": 0.75, - "citation_f1": 0.625, - "count": 8, - "hallucination_rate": 0.0, - "overall": 0.7125, + "answer_accuracy": 0.4255, + "citation_f1": 0.3475, + "count": 47, + "hallucination_rate": 0.2083, + "overall": 0.4021, "unsupported_citation_count": 0 }, "popular": { @@ -454,81 +1244,621 @@ }, "jurisdiction": { "federal": { - "answer_accuracy": 0.875, - "citation_f1": 0.875, - "count": 8, + "answer_accuracy": 0.625, + "citation_f1": 0.5417, + "count": 24, "hallucination_rate": 0.0, - "overall": 0.875, + "overall": 0.6, "unsupported_citation_count": 0 }, "state": { - "answer_accuracy": 0.875, - "citation_f1": 0.75, - "count": 8, - "hallucination_rate": 0.0, - "overall": 0.8375, + "answer_accuracy": 0.4194, + "citation_f1": 0.3656, + "count": 31, + "hallucination_rate": 0.2941, + "overall": 0.4032, "unsupported_citation_count": 0 } }, "proceeding": { "civil": { - "answer_accuracy": 0.75, - "citation_f1": 0.75, - "count": 8, - "hallucination_rate": 0.0, - "overall": 0.75, + "answer_accuracy": 0.6667, + "citation_f1": 0.5778, + "count": 15, + "hallucination_rate": 0.0909, + "overall": 0.64, "unsupported_citation_count": 0 }, "criminal": { + "answer_accuracy": 0.45, + "citation_f1": 0.3917, + "count": 40, + "hallucination_rate": 0.1818, + "overall": 0.4325, + "unsupported_citation_count": 0 + } + }, + "publication_status": { + "published": { + "answer_accuracy": 0.3077, + "citation_f1": 0.2222, + "count": 39, + "hallucination_rate": 0.2667, + "overall": 0.2821, + "unsupported_citation_count": 0 + } + }, + "recency": { + "2020-2021": { + "answer_accuracy": 0.3158, + "citation_f1": 0.3509, + "count": 19, + "hallucination_rate": 0.125, + "overall": 0.3263, + "unsupported_citation_count": 0 + }, + "2022-2024": { + "answer_accuracy": 0.2143, + "citation_f1": 0.1429, + "count": 14, + "hallucination_rate": 0.3333, + "overall": 0.1929, + "unsupported_citation_count": 0 + }, + "2026": { + "answer_accuracy": 0.5, + "citation_f1": 0.0, + "count": 6, + "hallucination_rate": 0.5, + "overall": 0.35, + "unsupported_citation_count": 0 + } + }, + "source": { + "data_gov_doj_nij": { + "answer_accuracy": 0.2727, + "citation_f1": 0.2626, + "count": 33, + "hallucination_rate": 0.1818, + "overall": 0.2697, + "unsupported_citation_count": 0 + }, + "official_courts_2026": { + "answer_accuracy": 0.5, + "citation_f1": 0.0, + "count": 6, + "hallucination_rate": 0.5, + "overall": 0.35, + "unsupported_citation_count": 0 + } + } + }, + "cost_usd": 0.55724, + "input_tokens": 24286, + "output_tokens": 14527 + }, + { + "display_name": "Mistral Large 3", + "family": "Mistral", + "model_id": "mistralai/mistral-large-2512", + "provider_route": "OpenRouter pinned route", + "status": "official", + "overall": 0.7691, + "answer_accuracy": 0.8278, + "citation_f1": 0.6322, + "hallucination_rate": 0.1732, + "dimensions": { + "decision_year": { + "2020": { + "answer_accuracy": 0.2308, + "citation_f1": 0.0, + "count": 13, + "hallucination_rate": 0.6667, + "overall": 0.1615, + "unsupported_citation_count": 5 + }, + "2021": { + "answer_accuracy": 0.1667, + "citation_f1": 0.0, + "count": 6, + "hallucination_rate": 0.4, + "overall": 0.1167, + "unsupported_citation_count": 6 + }, + "2022": { + "answer_accuracy": 0.1667, + "citation_f1": 0.0, + "count": 6, + "hallucination_rate": 0.5, + "overall": 0.1167, + "unsupported_citation_count": 1 + }, + "2023": { + "answer_accuracy": 0.1667, + "citation_f1": 0.0, + "count": 6, + "hallucination_rate": 0.375, + "overall": 0.1167, + "unsupported_citation_count": 5 + }, + "2024": { + "answer_accuracy": 0.5, + "citation_f1": 0.0, + "count": 2, + "hallucination_rate": 1.0, + "overall": 0.35, + "unsupported_citation_count": 0 + }, + "2026": { "answer_accuracy": 1.0, - "citation_f1": 0.875, + "citation_f1": 0.0, + "count": 6, + "hallucination_rate": 0.125, + "overall": 0.7, + "unsupported_citation_count": 3 + } + }, + "discipline": { + "bitemark": { + "answer_accuracy": 0.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 0.0, + "overall": 0.0, + "unsupported_citation_count": 0 + }, + "dna": { + "answer_accuracy": 0.25, + "citation_f1": 0.0, "count": 8, + "hallucination_rate": 0.625, + "overall": 0.175, + "unsupported_citation_count": 3 + }, + "dna_pcast_dicta": { + "answer_accuracy": 0.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 0.0, + "overall": 0.0, + "unsupported_citation_count": 0 + }, + "documentary_evidence": { + "answer_accuracy": 1.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 0.0, + "overall": 0.7, + "unsupported_citation_count": 0 + }, + "expert_disclosure": { + "answer_accuracy": 1.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 0.0, + "overall": 0.7, + "unsupported_citation_count": 2 + }, + "fingerprints": { + "answer_accuracy": 0.5, + "citation_f1": 0.0, + "count": 2, + "hallucination_rate": 0.0, + "overall": 0.35, + "unsupported_citation_count": 0 + }, + "footwear": { + "answer_accuracy": 0.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 0.0, + "overall": 0.0, + "unsupported_citation_count": 0 + }, + "ftm": { + "answer_accuracy": 0.1875, + "citation_f1": 0.0, + "count": 16, + "hallucination_rate": 0.5455, + "overall": 0.1312, + "unsupported_citation_count": 10 + }, + "ftm_fingerprints": { + "answer_accuracy": 0.5, + "citation_f1": 0.0, + "count": 2, + "hallucination_rate": 0.5, + "overall": 0.35, + "unsupported_citation_count": 2 + }, + "hearsay_and_confrontation": { + "answer_accuracy": 1.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 0.5, + "overall": 0.7, + "unsupported_citation_count": 1 + }, + "pcast_dicta": { + "answer_accuracy": 0.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 0.0, + "overall": 0.0, + "unsupported_citation_count": 0 + }, + "pcast_fre_702": { + "answer_accuracy": 0.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 0.0, + "overall": 0.0, + "unsupported_citation_count": 2 + }, + "rule_403_stipulations": { + "answer_accuracy": 1.0, + "citation_f1": 0.0, + "count": 1, "hallucination_rate": 0.0, - "overall": 0.9625, + "overall": 0.7, + "unsupported_citation_count": 0 + }, + "technical_and_survey_experts": { + "answer_accuracy": 1.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 0.0, + "overall": 0.7, + "unsupported_citation_count": 0 + }, + "toxic_tort_experts": { + "answer_accuracy": 1.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 0.0, + "overall": 0.7, + "unsupported_citation_count": 0 + } + }, + "domain": { + "caselaw": { + "answer_accuracy": 0.5273, + "citation_f1": 0.2242, + "count": 55, + "hallucination_rate": 0.3881, + "overall": 0.4364, + "unsupported_citation_count": 20 + }, + "fre": { + "answer_accuracy": 1.0, + "citation_f1": 0.866, + "count": 96, + "hallucination_rate": 0.0446, + "overall": 0.9598, + "unsupported_citation_count": 10 + } + }, + "familiarity": { + "obscure": { + "answer_accuracy": 0.4468, + "citation_f1": 0.1206, + "count": 47, + "hallucination_rate": 0.4286, + "overall": 0.3489, + "unsupported_citation_count": 20 + }, + "popular": { + "answer_accuracy": 1.0, + "citation_f1": 0.8333, + "count": 8, + "hallucination_rate": 0.1818, + "overall": 0.95, "unsupported_citation_count": 0 } + }, + "jurisdiction": { + "federal": { + "answer_accuracy": 0.5417, + "citation_f1": 0.3333, + "count": 24, + "hallucination_rate": 0.2619, + "overall": 0.4792, + "unsupported_citation_count": 15 + }, + "state": { + "answer_accuracy": 0.5161, + "citation_f1": 0.1398, + "count": 31, + "hallucination_rate": 0.6, + "overall": 0.4032, + "unsupported_citation_count": 5 + } + }, + "proceeding": { + "civil": { + "answer_accuracy": 0.8, + "citation_f1": 0.3778, + "count": 15, + "hallucination_rate": 0.1818, + "overall": 0.6733, + "unsupported_citation_count": 5 + }, + "criminal": { + "answer_accuracy": 0.425, + "citation_f1": 0.1667, + "count": 40, + "hallucination_rate": 0.4889, + "overall": 0.3475, + "unsupported_citation_count": 15 + } + }, + "publication_status": { + "published": { + "answer_accuracy": 0.3333, + "citation_f1": 0.0, + "count": 39, + "hallucination_rate": 0.4545, + "overall": 0.2333, + "unsupported_citation_count": 20 + } + }, + "recency": { + "2020-2021": { + "answer_accuracy": 0.2105, + "citation_f1": 0.0, + "count": 19, + "hallucination_rate": 0.56, + "overall": 0.1474, + "unsupported_citation_count": 11 + }, + "2022-2024": { + "answer_accuracy": 0.2143, + "citation_f1": 0.0, + "count": 14, + "hallucination_rate": 0.4545, + "overall": 0.15, + "unsupported_citation_count": 6 + }, + "2026": { + "answer_accuracy": 1.0, + "citation_f1": 0.0, + "count": 6, + "hallucination_rate": 0.125, + "overall": 0.7, + "unsupported_citation_count": 3 + } + }, + "source": { + "data_gov_doj_nij": { + "answer_accuracy": 0.2121, + "citation_f1": 0.0, + "count": 33, + "hallucination_rate": 0.5278, + "overall": 0.1485, + "unsupported_citation_count": 17 + }, + "official_courts_2026": { + "answer_accuracy": 1.0, + "citation_f1": 0.0, + "count": 6, + "hallucination_rate": 0.125, + "overall": 0.7, + "unsupported_citation_count": 3 + } } }, - "cost_usd": 0.329485, - "input_tokens": 19295, - "output_tokens": 7767 + "cost_usd": 0.03192, + "input_tokens": 30600, + "output_tokens": 15580 }, { - "display_name": "Claude Fable 5", - "family": "Anthropic", - "model_id": "anthropic/claude-fable-5", + "display_name": "DeepSeek V4 Pro", + "family": "DeepSeek", + "model_id": "deepseek/deepseek-v4-pro", "provider_route": "OpenRouter pinned route", "status": "official", - "overall": 0.9237, - "answer_accuracy": 0.9643, - "citation_f1": 0.8292, - "hallucination_rate": 0.0457, + "overall": 0.7563, + "answer_accuracy": 0.8013, + "citation_f1": 0.6512, + "hallucination_rate": 0.0692, "dimensions": { - "domain": { - "caselaw": { + "decision_year": { + "2020": { + "answer_accuracy": 0.4615, + "citation_f1": 0.2308, + "count": 13, + "hallucination_rate": 0.6667, + "overall": 0.3923, + "unsupported_citation_count": 0 + }, + "2021": { + "answer_accuracy": 0.0, + "citation_f1": 0.1667, + "count": 6, + "hallucination_rate": 0.0, + "overall": 0.05, + "unsupported_citation_count": 0 + }, + "2022": { + "answer_accuracy": 0.5, + "citation_f1": 0.0, + "count": 6, + "hallucination_rate": 0.0, + "overall": 0.35, + "unsupported_citation_count": 0 + }, + "2023": { + "answer_accuracy": 0.5, + "citation_f1": 0.1667, + "count": 6, + "hallucination_rate": 0.0, + "overall": 0.4, + "unsupported_citation_count": 0 + }, + "2024": { + "answer_accuracy": 0.0, + "citation_f1": 0.0, + "count": 2, + "hallucination_rate": 0.0, + "overall": 0.0, + "unsupported_citation_count": 0 + }, + "2026": { + "answer_accuracy": 0.5, + "citation_f1": 0.0, + "count": 6, + "hallucination_rate": 1.0, + "overall": 0.35, + "unsupported_citation_count": 0 + } + }, + "discipline": { + "bitemark": { + "answer_accuracy": 0.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 0.0, + "overall": 0.0, + "unsupported_citation_count": 0 + }, + "dna": { + "answer_accuracy": 0.125, + "citation_f1": 0.125, + "count": 8, + "hallucination_rate": 0.6667, + "overall": 0.125, + "unsupported_citation_count": 0 + }, + "dna_pcast_dicta": { + "answer_accuracy": 0.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 0.0, + "overall": 0.0, + "unsupported_citation_count": 0 + }, + "documentary_evidence": { + "answer_accuracy": 0.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 0.0, + "overall": 0.0, + "unsupported_citation_count": 0 + }, + "expert_disclosure": { + "answer_accuracy": 0.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 0.0, + "overall": 0.0, + "unsupported_citation_count": 0 + }, + "fingerprints": { + "answer_accuracy": 0.5, + "citation_f1": 0.0, + "count": 2, + "hallucination_rate": 0.0, + "overall": 0.35, + "unsupported_citation_count": 0 + }, + "footwear": { "answer_accuracy": 1.0, - "citation_f1": 0.9792, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 0.0, + "overall": 0.7, + "unsupported_citation_count": 0 + }, + "ftm": { + "answer_accuracy": 0.5625, + "citation_f1": 0.25, "count": 16, - "hallucination_rate": 0.0476, - "overall": 0.9937, + "hallucination_rate": 0.5, + "overall": 0.4688, + "unsupported_citation_count": 0 + }, + "ftm_fingerprints": { + "answer_accuracy": 0.0, + "citation_f1": 0.0, + "count": 2, + "hallucination_rate": 0.0, + "overall": 0.0, + "unsupported_citation_count": 0 + }, + "hearsay_and_confrontation": { + "answer_accuracy": 1.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 0.0, + "overall": 0.7, + "unsupported_citation_count": 0 + }, + "pcast_dicta": { + "answer_accuracy": 0.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 0.0, + "overall": 0.0, + "unsupported_citation_count": 0 + }, + "pcast_fre_702": { + "answer_accuracy": 0.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 0.0, + "overall": 0.0, + "unsupported_citation_count": 0 + }, + "rule_403_stipulations": { + "answer_accuracy": 0.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 0.0, + "overall": 0.0, + "unsupported_citation_count": 0 + }, + "technical_and_survey_experts": { + "answer_accuracy": 1.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 1.0, + "overall": 0.7, + "unsupported_citation_count": 0 + }, + "toxic_tort_experts": { + "answer_accuracy": 1.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 1.0, + "overall": 0.7, + "unsupported_citation_count": 0 + } + }, + "domain": { + "caselaw": { + "answer_accuracy": 0.5455, + "citation_f1": 0.3636, + "count": 55, + "hallucination_rate": 0.2857, + "overall": 0.4909, "unsupported_citation_count": 0 }, "fre": { - "answer_accuracy": 0.9583, - "citation_f1": 0.8042, + "answer_accuracy": 0.9479, + "citation_f1": 0.816, "count": 96, - "hallucination_rate": 0.0455, - "overall": 0.9121, - "unsupported_citation_count": 34 + "hallucination_rate": 0.0098, + "overall": 0.9083, + "unsupported_citation_count": 17 } }, "familiarity": { "obscure": { - "answer_accuracy": 1.0, - "citation_f1": 0.9583, - "count": 8, - "hallucination_rate": 0.0909, - "overall": 0.9875, + "answer_accuracy": 0.4681, + "citation_f1": 0.2553, + "count": 47, + "hallucination_rate": 0.4, + "overall": 0.4043, "unsupported_citation_count": 0 }, "popular": { @@ -542,44 +1872,98 @@ }, "jurisdiction": { "federal": { - "answer_accuracy": 1.0, - "citation_f1": 0.9583, - "count": 8, - "hallucination_rate": 0.0769, - "overall": 0.9875, + "answer_accuracy": 0.625, + "citation_f1": 0.5, + "count": 24, + "hallucination_rate": 0.2941, + "overall": 0.5875, "unsupported_citation_count": 0 }, "state": { - "answer_accuracy": 1.0, - "citation_f1": 1.0, - "count": 8, - "hallucination_rate": 0.0, - "overall": 1.0, + "answer_accuracy": 0.4839, + "citation_f1": 0.2581, + "count": 31, + "hallucination_rate": 0.2727, + "overall": 0.4161, "unsupported_citation_count": 0 } }, "proceeding": { "civil": { - "answer_accuracy": 1.0, - "citation_f1": 1.0, - "count": 8, - "hallucination_rate": 0.0, - "overall": 1.0, + "answer_accuracy": 0.6, + "citation_f1": 0.4667, + "count": 15, + "hallucination_rate": 0.3, + "overall": 0.56, "unsupported_citation_count": 0 }, "criminal": { - "answer_accuracy": 1.0, - "citation_f1": 0.9583, - "count": 8, - "hallucination_rate": 0.1, - "overall": 0.9875, + "answer_accuracy": 0.525, + "citation_f1": 0.325, + "count": 40, + "hallucination_rate": 0.2778, + "overall": 0.465, + "unsupported_citation_count": 0 + } + }, + "publication_status": { + "published": { + "answer_accuracy": 0.3846, + "citation_f1": 0.1282, + "count": 39, + "hallucination_rate": 0.6154, + "overall": 0.3077, + "unsupported_citation_count": 0 + } + }, + "recency": { + "2020-2021": { + "answer_accuracy": 0.3158, + "citation_f1": 0.2105, + "count": 19, + "hallucination_rate": 0.6, + "overall": 0.2842, + "unsupported_citation_count": 0 + }, + "2022-2024": { + "answer_accuracy": 0.4286, + "citation_f1": 0.0714, + "count": 14, + "hallucination_rate": 0.0, + "overall": 0.3214, + "unsupported_citation_count": 0 + }, + "2026": { + "answer_accuracy": 0.5, + "citation_f1": 0.0, + "count": 6, + "hallucination_rate": 1.0, + "overall": 0.35, + "unsupported_citation_count": 0 + } + }, + "source": { + "data_gov_doj_nij": { + "answer_accuracy": 0.3636, + "citation_f1": 0.1515, + "count": 33, + "hallucination_rate": 0.5455, + "overall": 0.3, + "unsupported_citation_count": 0 + }, + "official_courts_2026": { + "answer_accuracy": 0.5, + "citation_f1": 0.0, + "count": 6, + "hallucination_rate": 1.0, + "overall": 0.35, "unsupported_citation_count": 0 } } }, - "cost_usd": 1.33058, - "input_tokens": 31548, - "output_tokens": 20302 + "cost_usd": 0.173535, + "input_tokens": 26125, + "output_tokens": 59564 } ], "methodology_url": "https://github.com/Examp-LLC/EvidenceBench/blob/main/METHODOLOGY.md", diff --git a/pyproject.toml b/pyproject.toml index 392d550..cd233b4 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "hatchling.build" [project] name = "evidencebench" -version = "1.0.0" +version = "3.0.0" description = "Reproducible evidence-law and Federal Rules of Evidence benchmark." readme = "README.md" requires-python = ">=3.12" diff --git a/results/v3.0.0/README.md b/results/v3.0.0/README.md new file mode 100644 index 0000000..640aca6 --- /dev/null +++ b/results/v3.0.0/README.md @@ -0,0 +1,10 @@ +# EvidenceBench v3.0.0 + +This directory contains the public aggregate manifest for the v3 official +snapshot. The sealed questions and item-level model outputs are private. + +v3 expands the holdout from 112 to 151 items by adding 33 unique, citable, +published decisions dated 2020–2024 from the public-domain DOJ/NIJ Post-PCAST +decisions dataset cataloged by Data.gov plus six published 2026 decisions from +official court and GovInfo sources. The public source snapshots and hashes are +included in the repository. diff --git a/results/v3.0.0/manifest.json b/results/v3.0.0/manifest.json new file mode 100644 index 0000000..05494f2 --- /dev/null +++ b/results/v3.0.0/manifest.json @@ -0,0 +1,1968 @@ +{ + "benchmark_version": "3.0.0", + "status": "official_snapshot", + "published_at": "2026-07-14T23:29:38Z", + "authority_corpus": { + "version": "FRE-2025-12-01+CASELAW-DATAGOV-2026-07-14", + "fre_source_url": "https://www.uscourts.gov/sites/default/files/document/federal-rules-of-evidence.pdf", + "case_index_version": "CASELAW-DATAGOV-2026-07-14", + "data_gov_catalog_url": "https://catalog.data.gov/dataset/post-pcast-court-decisions-assessing-the-admissibility-of-forensic-science-evidence", + "data_gov_resource_url": "https://nij.ojp.gov/program/national-center-forensics/post-pcast-court-decisions-assessing-admissibility-forensic-science-evidence", + "data_gov_license": "Public Domain", + "data_gov_source_html_sha256": "1811a7762f961e0e046b2c957717aeebd46f0985b2a8c774a657ad162e723368", + "data_gov_records_sha256": "90990eb660d45be39515f9f08278abd4b080be3ce758eb1cf9e8854cc5ab95f1", + "recent_2026_source_version": "RECENT-CASELAW-2026-07-14", + "recent_2026_metadata_sha256": "ea97ec705a56e141df2a772b0644a7c33731ace07b13812314ecbabb8dde4467", + "pending_amendments_excluded": [ + { + "rule": "FRE 801(d)(1)(A)", + "effective_date": "2026-12-01" + } + ] + }, + "holdout": { + "status": "sealed", + "question_count": 151, + "fre_question_count": 96, + "caselaw_question_count": 55, + "data_gov_caselaw_question_count": 33, + "recent_2026_caselaw_question_count": 6, + "development_question_count": 24, + "dataset_sha256": "9f6394738dfb0a3aa722d11c1a37f239828ea1adc4f847c3e7f8134f2227869f", + "data_gov_window": { + "start": "2020-01-01", + "end": "2024-12-31" + }, + "data_gov_years": { + "2020": 13, + "2021": 6, + "2022": 6, + "2023": 6, + "2024": 2 + }, + "data_gov_jurisdictions": { + "federal": 12, + "state": 21 + }, + "data_gov_proceedings": { + "civil": 3, + "criminal": 30 + }, + "recent_2026_jurisdictions": { + "federal": 4, + "state": 2 + }, + "recent_2026_proceedings": { + "civil": 4, + "criminal": 2 + }, + "data_gov_selection": "unique+citable+publication_status_published+decision_date_2020_2024", + "recent_2026_selection": "published+official_source+material_evidence_holding+decision_date_2026", + "categories": [ + "relevance", + "hearsay", + "witnesses", + "experts", + "authentication", + "best_evidence", + "character", + "impeachment", + "privilege", + "caselaw" + ] + }, + "protocol": { + "name": "evidencebench-v3-closed-book", + "prompt_version": "evidencebench-v3-closed-book", + "prompt_sha256": "250960491ca474811fd1d32ddc2c207992c04a766de7d04ad76ebaca29636221", + "runs_per_question": 1, + "tools_enabled": false, + "overall_formula": "0.70 * answer_accuracy + 0.30 * citation_f1" + }, + "models": [ + { + "display_name": "Gemini 3.5 Flash", + "family": "Google", + "model_id": "google/gemini-3.5-flash", + "provider_route": "OpenRouter pinned route", + "status": "official", + "overall": 0.8517, + "answer_accuracy": 0.9007, + "citation_f1": 0.7373, + "hallucination_rate": 0.1524, + "dimensions": { + "decision_year": { + "2020": { + "answer_accuracy": 0.3846, + "citation_f1": 0.5385, + "count": 13, + "hallucination_rate": 0.4167, + "overall": 0.4308, + "unsupported_citation_count": 0 + }, + "2021": { + "answer_accuracy": 0.6667, + "citation_f1": 0.3333, + "count": 6, + "hallucination_rate": 0.7143, + "overall": 0.5667, + "unsupported_citation_count": 0 + }, + "2022": { + "answer_accuracy": 0.6667, + "citation_f1": 0.1667, + "count": 6, + "hallucination_rate": 0.8, + "overall": 0.5167, + "unsupported_citation_count": 0 + }, + "2023": { + "answer_accuracy": 0.6667, + "citation_f1": 0.2778, + "count": 6, + "hallucination_rate": 0.6667, + "overall": 0.55, + "unsupported_citation_count": 0 + }, + "2024": { + "answer_accuracy": 0.5, + "citation_f1": 0.0, + "count": 2, + "hallucination_rate": 0.0, + "overall": 0.35, + "unsupported_citation_count": 0 + }, + "2026": { + "answer_accuracy": 1.0, + "citation_f1": 0.0, + "count": 6, + "hallucination_rate": 0.6, + "overall": 0.7, + "unsupported_citation_count": 0 + } + }, + "discipline": { + "bitemark": { + "answer_accuracy": 1.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 1.0, + "overall": 0.7, + "unsupported_citation_count": 0 + }, + "dna": { + "answer_accuracy": 0.375, + "citation_f1": 0.25, + "count": 8, + "hallucination_rate": 0.7778, + "overall": 0.3375, + "unsupported_citation_count": 0 + }, + "dna_pcast_dicta": { + "answer_accuracy": 1.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 1.0, + "overall": 0.7, + "unsupported_citation_count": 0 + }, + "documentary_evidence": { + "answer_accuracy": 1.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 1.0, + "overall": 0.7, + "unsupported_citation_count": 0 + }, + "expert_disclosure": { + "answer_accuracy": 1.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 1.0, + "overall": 0.7, + "unsupported_citation_count": 0 + }, + "fingerprints": { + "answer_accuracy": 0.5, + "citation_f1": 0.0, + "count": 2, + "hallucination_rate": 1.0, + "overall": 0.35, + "unsupported_citation_count": 0 + }, + "footwear": { + "answer_accuracy": 1.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 0.0, + "overall": 0.7, + "unsupported_citation_count": 0 + }, + "ftm": { + "answer_accuracy": 0.5625, + "citation_f1": 0.4792, + "count": 16, + "hallucination_rate": 0.4667, + "overall": 0.5375, + "unsupported_citation_count": 0 + }, + "ftm_fingerprints": { + "answer_accuracy": 0.5, + "citation_f1": 0.0, + "count": 2, + "hallucination_rate": 1.0, + "overall": 0.35, + "unsupported_citation_count": 0 + }, + "hearsay_and_confrontation": { + "answer_accuracy": 1.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 1.0, + "overall": 0.7, + "unsupported_citation_count": 0 + }, + "pcast_dicta": { + "answer_accuracy": 1.0, + "citation_f1": 1.0, + "count": 1, + "hallucination_rate": 0.0, + "overall": 1.0, + "unsupported_citation_count": 0 + }, + "pcast_fre_702": { + "answer_accuracy": 0.0, + "citation_f1": 1.0, + "count": 1, + "hallucination_rate": 0.0, + "overall": 0.3, + "unsupported_citation_count": 0 + }, + "rule_403_stipulations": { + "answer_accuracy": 1.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 0.0, + "overall": 0.7, + "unsupported_citation_count": 0 + }, + "technical_and_survey_experts": { + "answer_accuracy": 1.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 0.0, + "overall": 0.7, + "unsupported_citation_count": 0 + }, + "toxic_tort_experts": { + "answer_accuracy": 1.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 0.0, + "overall": 0.7, + "unsupported_citation_count": 0 + } + }, + "domain": { + "caselaw": { + "answer_accuracy": 0.7273, + "citation_f1": 0.4788, + "count": 55, + "hallucination_rate": 0.4545, + "overall": 0.6527, + "unsupported_citation_count": 0 + }, + "fre": { + "answer_accuracy": 1.0, + "citation_f1": 0.8854, + "count": 96, + "hallucination_rate": 0.0, + "overall": 0.9656, + "unsupported_citation_count": 13 + } + }, + "familiarity": { + "obscure": { + "answer_accuracy": 0.6809, + "citation_f1": 0.3972, + "count": 47, + "hallucination_rate": 0.5217, + "overall": 0.5957, + "unsupported_citation_count": 0 + }, + "popular": { + "answer_accuracy": 1.0, + "citation_f1": 0.9583, + "count": 8, + "hallucination_rate": 0.1111, + "overall": 0.9875, + "unsupported_citation_count": 0 + } + }, + "jurisdiction": { + "federal": { + "answer_accuracy": 0.75, + "citation_f1": 0.625, + "count": 24, + "hallucination_rate": 0.1905, + "overall": 0.7125, + "unsupported_citation_count": 0 + }, + "state": { + "answer_accuracy": 0.7097, + "citation_f1": 0.3656, + "count": 31, + "hallucination_rate": 0.6176, + "overall": 0.6065, + "unsupported_citation_count": 0 + } + }, + "proceeding": { + "civil": { + "answer_accuracy": 0.8667, + "citation_f1": 0.6444, + "count": 15, + "hallucination_rate": 0.2667, + "overall": 0.8, + "unsupported_citation_count": 0 + }, + "criminal": { + "answer_accuracy": 0.675, + "citation_f1": 0.4167, + "count": 40, + "hallucination_rate": 0.525, + "overall": 0.5975, + "unsupported_citation_count": 0 + } + }, + "publication_status": { + "published": { + "answer_accuracy": 0.6154, + "citation_f1": 0.2991, + "count": 39, + "hallucination_rate": 0.6, + "overall": 0.5205, + "unsupported_citation_count": 0 + } + }, + "recency": { + "2020-2021": { + "answer_accuracy": 0.4737, + "citation_f1": 0.4737, + "count": 19, + "hallucination_rate": 0.5263, + "overall": 0.4737, + "unsupported_citation_count": 0 + }, + "2022-2024": { + "answer_accuracy": 0.6429, + "citation_f1": 0.1905, + "count": 14, + "hallucination_rate": 0.7273, + "overall": 0.5071, + "unsupported_citation_count": 0 + }, + "2026": { + "answer_accuracy": 1.0, + "citation_f1": 0.0, + "count": 6, + "hallucination_rate": 0.6, + "overall": 0.7, + "unsupported_citation_count": 0 + } + }, + "source": { + "data_gov_doj_nij": { + "answer_accuracy": 0.5455, + "citation_f1": 0.3535, + "count": 33, + "hallucination_rate": 0.6, + "overall": 0.4879, + "unsupported_citation_count": 0 + }, + "official_courts_2026": { + "answer_accuracy": 1.0, + "citation_f1": 0.0, + "count": 6, + "hallucination_rate": 0.6, + "overall": 0.7, + "unsupported_citation_count": 0 + } + } + }, + "cost_usd": 0.186876, + "input_tokens": 29958, + "output_tokens": 15771 + }, + { + "display_name": "Claude Fable 5", + "family": "Anthropic", + "model_id": "anthropic/claude-fable-5", + "provider_route": "OpenRouter pinned route", + "status": "official", + "overall": 0.8453, + "answer_accuracy": 0.894, + "citation_f1": 0.7316, + "hallucination_rate": 0.1042, + "dimensions": { + "decision_year": { + "2020": { + "answer_accuracy": 0.5385, + "citation_f1": 0.6795, + "count": 13, + "hallucination_rate": 0.1667, + "overall": 0.5808, + "unsupported_citation_count": 5 + }, + "2021": { + "answer_accuracy": 0.6667, + "citation_f1": 0.5278, + "count": 6, + "hallucination_rate": 0.125, + "overall": 0.625, + "unsupported_citation_count": 3 + }, + "2022": { + "answer_accuracy": 0.5, + "citation_f1": 0.2778, + "count": 6, + "hallucination_rate": 0.25, + "overall": 0.4333, + "unsupported_citation_count": 4 + }, + "2023": { + "answer_accuracy": 0.5, + "citation_f1": 0.4444, + "count": 6, + "hallucination_rate": 0.4, + "overall": 0.4833, + "unsupported_citation_count": 0 + }, + "2024": { + "answer_accuracy": 0.5, + "citation_f1": 0.0, + "count": 2, + "hallucination_rate": 0.25, + "overall": 0.35, + "unsupported_citation_count": 3 + }, + "2026": { + "answer_accuracy": 1.0, + "citation_f1": 0.0, + "count": 6, + "hallucination_rate": 0.2727, + "overall": 0.7, + "unsupported_citation_count": 4 + } + }, + "discipline": { + "bitemark": { + "answer_accuracy": 0.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 0.0, + "overall": 0.0, + "unsupported_citation_count": 0 + }, + "dna": { + "answer_accuracy": 0.375, + "citation_f1": 0.5, + "count": 8, + "hallucination_rate": 0.4, + "overall": 0.4125, + "unsupported_citation_count": 1 + }, + "dna_pcast_dicta": { + "answer_accuracy": 1.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 1.0, + "overall": 0.7, + "unsupported_citation_count": 0 + }, + "documentary_evidence": { + "answer_accuracy": 1.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 0.0, + "overall": 0.7, + "unsupported_citation_count": 0 + }, + "expert_disclosure": { + "answer_accuracy": 1.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 0.0, + "overall": 0.7, + "unsupported_citation_count": 0 + }, + "fingerprints": { + "answer_accuracy": 0.5, + "citation_f1": 0.0, + "count": 2, + "hallucination_rate": 0.3333, + "overall": 0.35, + "unsupported_citation_count": 4 + }, + "footwear": { + "answer_accuracy": 1.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 0.0, + "overall": 0.7, + "unsupported_citation_count": 2 + }, + "ftm": { + "answer_accuracy": 0.625, + "citation_f1": 0.6146, + "count": 16, + "hallucination_rate": 0.1111, + "overall": 0.6219, + "unsupported_citation_count": 5 + }, + "ftm_fingerprints": { + "answer_accuracy": 0.5, + "citation_f1": 0.5, + "count": 2, + "hallucination_rate": 0.0, + "overall": 0.5, + "unsupported_citation_count": 1 + }, + "hearsay_and_confrontation": { + "answer_accuracy": 1.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 0.6667, + "overall": 0.7, + "unsupported_citation_count": 1 + }, + "pcast_dicta": { + "answer_accuracy": 1.0, + "citation_f1": 1.0, + "count": 1, + "hallucination_rate": 0.0, + "overall": 1.0, + "unsupported_citation_count": 0 + }, + "pcast_fre_702": { + "answer_accuracy": 0.0, + "citation_f1": 0.5, + "count": 1, + "hallucination_rate": 0.0, + "overall": 0.15, + "unsupported_citation_count": 2 + }, + "rule_403_stipulations": { + "answer_accuracy": 1.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 0.0, + "overall": 0.7, + "unsupported_citation_count": 1 + }, + "technical_and_survey_experts": { + "answer_accuracy": 1.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 0.0, + "overall": 0.7, + "unsupported_citation_count": 1 + }, + "toxic_tort_experts": { + "answer_accuracy": 1.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 0.3333, + "overall": 0.7, + "unsupported_citation_count": 1 + } + }, + "domain": { + "caselaw": { + "answer_accuracy": 0.7273, + "citation_f1": 0.5539, + "count": 55, + "hallucination_rate": 0.2125, + "overall": 0.6753, + "unsupported_citation_count": 20 + }, + "fre": { + "answer_accuracy": 0.9896, + "citation_f1": 0.8333, + "count": 96, + "hallucination_rate": 0.05, + "overall": 0.9427, + "unsupported_citation_count": 35 + } + }, + "familiarity": { + "obscure": { + "answer_accuracy": 0.6809, + "citation_f1": 0.4965, + "count": 47, + "hallucination_rate": 0.2239, + "overall": 0.6255, + "unsupported_citation_count": 19 + }, + "popular": { + "answer_accuracy": 1.0, + "citation_f1": 0.8917, + "count": 8, + "hallucination_rate": 0.1538, + "overall": 0.9675, + "unsupported_citation_count": 1 + } + }, + "jurisdiction": { + "federal": { + "answer_accuracy": 0.7917, + "citation_f1": 0.6444, + "count": 24, + "hallucination_rate": 0.0556, + "overall": 0.7475, + "unsupported_citation_count": 9 + }, + "state": { + "answer_accuracy": 0.6774, + "citation_f1": 0.4839, + "count": 31, + "hallucination_rate": 0.3409, + "overall": 0.6194, + "unsupported_citation_count": 11 + } + }, + "proceeding": { + "civil": { + "answer_accuracy": 0.8667, + "citation_f1": 0.5756, + "count": 15, + "hallucination_rate": 0.12, + "overall": 0.7793, + "unsupported_citation_count": 6 + }, + "criminal": { + "answer_accuracy": 0.675, + "citation_f1": 0.5458, + "count": 40, + "hallucination_rate": 0.2545, + "overall": 0.6362, + "unsupported_citation_count": 14 + } + }, + "publication_status": { + "published": { + "answer_accuracy": 0.6154, + "citation_f1": 0.4188, + "count": 39, + "hallucination_rate": 0.2222, + "overall": 0.5564, + "unsupported_citation_count": 19 + } + }, + "recency": { + "2020-2021": { + "answer_accuracy": 0.5789, + "citation_f1": 0.6316, + "count": 19, + "hallucination_rate": 0.1538, + "overall": 0.5947, + "unsupported_citation_count": 8 + }, + "2022-2024": { + "answer_accuracy": 0.5, + "citation_f1": 0.3095, + "count": 14, + "hallucination_rate": 0.2941, + "overall": 0.4429, + "unsupported_citation_count": 7 + }, + "2026": { + "answer_accuracy": 1.0, + "citation_f1": 0.0, + "count": 6, + "hallucination_rate": 0.2727, + "overall": 0.7, + "unsupported_citation_count": 4 + } + }, + "source": { + "data_gov_doj_nij": { + "answer_accuracy": 0.5455, + "citation_f1": 0.4949, + "count": 33, + "hallucination_rate": 0.2093, + "overall": 0.5303, + "unsupported_citation_count": 15 + }, + "official_courts_2026": { + "answer_accuracy": 1.0, + "citation_f1": 0.0, + "count": 6, + "hallucination_rate": 0.2727, + "overall": 0.7, + "unsupported_citation_count": 4 + } + } + }, + "cost_usd": 2.29467, + "input_tokens": 46022, + "output_tokens": 36689 + }, + { + "display_name": "Grok 4.5", + "family": "xAI", + "model_id": "x-ai/grok-4.5", + "provider_route": "OpenRouter pinned route", + "status": "official", + "overall": 0.8433, + "answer_accuracy": 0.9139, + "citation_f1": 0.6786, + "hallucination_rate": 0.0692, + "dimensions": { + "decision_year": { + "2020": { + "answer_accuracy": 0.4615, + "citation_f1": 0.6154, + "count": 13, + "hallucination_rate": 0.3333, + "overall": 0.5077, + "unsupported_citation_count": 0 + }, + "2021": { + "answer_accuracy": 0.8333, + "citation_f1": 0.2778, + "count": 6, + "hallucination_rate": 0.5, + "overall": 0.6667, + "unsupported_citation_count": 0 + }, + "2022": { + "answer_accuracy": 0.8333, + "citation_f1": 0.0, + "count": 6, + "hallucination_rate": 1.0, + "overall": 0.5833, + "unsupported_citation_count": 0 + }, + "2023": { + "answer_accuracy": 0.5, + "citation_f1": 0.3333, + "count": 6, + "hallucination_rate": 0.3333, + "overall": 0.45, + "unsupported_citation_count": 0 + }, + "2024": { + "answer_accuracy": 0.5, + "citation_f1": 0.0, + "count": 2, + "hallucination_rate": 0.0, + "overall": 0.35, + "unsupported_citation_count": 0 + }, + "2026": { + "answer_accuracy": 1.0, + "citation_f1": 0.0, + "count": 6, + "hallucination_rate": 0.1429, + "overall": 0.7, + "unsupported_citation_count": 3 + } + }, + "discipline": { + "bitemark": { + "answer_accuracy": 0.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 0.0, + "overall": 0.0, + "unsupported_citation_count": 0 + }, + "dna": { + "answer_accuracy": 0.5, + "citation_f1": 0.3333, + "count": 8, + "hallucination_rate": 0.625, + "overall": 0.45, + "unsupported_citation_count": 0 + }, + "dna_pcast_dicta": { + "answer_accuracy": 1.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 0.0, + "overall": 0.7, + "unsupported_citation_count": 0 + }, + "documentary_evidence": { + "answer_accuracy": 1.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 0.0, + "overall": 0.7, + "unsupported_citation_count": 0 + }, + "expert_disclosure": { + "answer_accuracy": 1.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 0.0, + "overall": 0.7, + "unsupported_citation_count": 0 + }, + "fingerprints": { + "answer_accuracy": 0.5, + "citation_f1": 0.0, + "count": 2, + "hallucination_rate": 0.0, + "overall": 0.35, + "unsupported_citation_count": 0 + }, + "footwear": { + "answer_accuracy": 1.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 0.0, + "overall": 0.7, + "unsupported_citation_count": 0 + }, + "ftm": { + "answer_accuracy": 0.6875, + "citation_f1": 0.5625, + "count": 16, + "hallucination_rate": 0.1818, + "overall": 0.65, + "unsupported_citation_count": 0 + }, + "ftm_fingerprints": { + "answer_accuracy": 0.5, + "citation_f1": 0.0, + "count": 2, + "hallucination_rate": 0.0, + "overall": 0.35, + "unsupported_citation_count": 0 + }, + "hearsay_and_confrontation": { + "answer_accuracy": 1.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 0.5, + "overall": 0.7, + "unsupported_citation_count": 1 + }, + "pcast_dicta": { + "answer_accuracy": 1.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 1.0, + "overall": 0.7, + "unsupported_citation_count": 0 + }, + "pcast_fre_702": { + "answer_accuracy": 0.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 1.0, + "overall": 0.0, + "unsupported_citation_count": 0 + }, + "rule_403_stipulations": { + "answer_accuracy": 1.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 0.0, + "overall": 0.7, + "unsupported_citation_count": 1 + }, + "technical_and_survey_experts": { + "answer_accuracy": 1.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 0.0, + "overall": 0.7, + "unsupported_citation_count": 1 + }, + "toxic_tort_experts": { + "answer_accuracy": 1.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 0.0, + "overall": 0.7, + "unsupported_citation_count": 0 + } + }, + "domain": { + "caselaw": { + "answer_accuracy": 0.7636, + "citation_f1": 0.497, + "count": 55, + "hallucination_rate": 0.2444, + "overall": 0.6836, + "unsupported_citation_count": 3 + }, + "fre": { + "answer_accuracy": 1.0, + "citation_f1": 0.7826, + "count": 96, + "hallucination_rate": 0.0, + "overall": 0.9348, + "unsupported_citation_count": 27 + } + }, + "familiarity": { + "obscure": { + "answer_accuracy": 0.7234, + "citation_f1": 0.4113, + "count": 47, + "hallucination_rate": 0.2973, + "overall": 0.6298, + "unsupported_citation_count": 3 + }, + "popular": { + "answer_accuracy": 1.0, + "citation_f1": 1.0, + "count": 8, + "hallucination_rate": 0.0, + "overall": 1.0, + "unsupported_citation_count": 0 + } + }, + "jurisdiction": { + "federal": { + "answer_accuracy": 0.875, + "citation_f1": 0.5278, + "count": 24, + "hallucination_rate": 0.2174, + "overall": 0.7708, + "unsupported_citation_count": 2 + }, + "state": { + "answer_accuracy": 0.6774, + "citation_f1": 0.4731, + "count": 31, + "hallucination_rate": 0.2727, + "overall": 0.6161, + "unsupported_citation_count": 1 + } + }, + "proceeding": { + "civil": { + "answer_accuracy": 0.8667, + "citation_f1": 0.5333, + "count": 15, + "hallucination_rate": 0.2143, + "overall": 0.7667, + "unsupported_citation_count": 1 + }, + "criminal": { + "answer_accuracy": 0.725, + "citation_f1": 0.4833, + "count": 40, + "hallucination_rate": 0.2581, + "overall": 0.6525, + "unsupported_citation_count": 2 + } + }, + "publication_status": { + "published": { + "answer_accuracy": 0.6667, + "citation_f1": 0.2991, + "count": 39, + "hallucination_rate": 0.3571, + "overall": 0.5564, + "unsupported_citation_count": 3 + } + }, + "recency": { + "2020-2021": { + "answer_accuracy": 0.5789, + "citation_f1": 0.5088, + "count": 19, + "hallucination_rate": 0.375, + "overall": 0.5579, + "unsupported_citation_count": 0 + }, + "2022-2024": { + "answer_accuracy": 0.6429, + "citation_f1": 0.1429, + "count": 14, + "hallucination_rate": 0.6, + "overall": 0.4929, + "unsupported_citation_count": 0 + }, + "2026": { + "answer_accuracy": 1.0, + "citation_f1": 0.0, + "count": 6, + "hallucination_rate": 0.1429, + "overall": 0.7, + "unsupported_citation_count": 3 + } + }, + "source": { + "data_gov_doj_nij": { + "answer_accuracy": 0.6061, + "citation_f1": 0.3535, + "count": 33, + "hallucination_rate": 0.4286, + "overall": 0.5303, + "unsupported_citation_count": 0 + }, + "official_courts_2026": { + "answer_accuracy": 1.0, + "citation_f1": 0.0, + "count": 6, + "hallucination_rate": 0.1429, + "overall": 0.7, + "unsupported_citation_count": 3 + } + } + }, + "cost_usd": 0.83576, + "input_tokens": 63148, + "output_tokens": 125732 + }, + { + "display_name": "GPT-5.6 Sol", + "family": "OpenAI", + "model_id": "openai/gpt-5.6-sol", + "provider_route": "OpenRouter pinned route", + "status": "official", + "overall": 0.8033, + "answer_accuracy": 0.8212, + "citation_f1": 0.7616, + "hallucination_rate": 0.0357, + "dimensions": { + "decision_year": { + "2020": { + "answer_accuracy": 0.3077, + "citation_f1": 0.359, + "count": 13, + "hallucination_rate": 0.1667, + "overall": 0.3231, + "unsupported_citation_count": 0 + }, + "2021": { + "answer_accuracy": 0.3333, + "citation_f1": 0.3333, + "count": 6, + "hallucination_rate": 0.0, + "overall": 0.3333, + "unsupported_citation_count": 0 + }, + "2022": { + "answer_accuracy": 0.3333, + "citation_f1": 0.1667, + "count": 6, + "hallucination_rate": 0.5, + "overall": 0.2833, + "unsupported_citation_count": 0 + }, + "2023": { + "answer_accuracy": 0.1667, + "citation_f1": 0.1667, + "count": 6, + "hallucination_rate": 0.0, + "overall": 0.1667, + "unsupported_citation_count": 0 + }, + "2024": { + "answer_accuracy": 0.0, + "citation_f1": 0.0, + "count": 2, + "hallucination_rate": 0.0, + "overall": 0.0, + "unsupported_citation_count": 0 + }, + "2026": { + "answer_accuracy": 0.5, + "citation_f1": 0.0, + "count": 6, + "hallucination_rate": 0.5, + "overall": 0.35, + "unsupported_citation_count": 0 + } + }, + "discipline": { + "bitemark": { + "answer_accuracy": 0.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 0.0, + "overall": 0.0, + "unsupported_citation_count": 0 + }, + "dna": { + "answer_accuracy": 0.375, + "citation_f1": 0.25, + "count": 8, + "hallucination_rate": 0.3333, + "overall": 0.3375, + "unsupported_citation_count": 0 + }, + "dna_pcast_dicta": { + "answer_accuracy": 0.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 0.0, + "overall": 0.0, + "unsupported_citation_count": 0 + }, + "documentary_evidence": { + "answer_accuracy": 1.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 0.0, + "overall": 0.7, + "unsupported_citation_count": 0 + }, + "expert_disclosure": { + "answer_accuracy": 0.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 0.0, + "overall": 0.0, + "unsupported_citation_count": 0 + }, + "fingerprints": { + "answer_accuracy": 0.0, + "citation_f1": 0.0, + "count": 2, + "hallucination_rate": 0.0, + "overall": 0.0, + "unsupported_citation_count": 0 + }, + "footwear": { + "answer_accuracy": 0.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 0.0, + "overall": 0.0, + "unsupported_citation_count": 0 + }, + "ftm": { + "answer_accuracy": 0.3125, + "citation_f1": 0.3542, + "count": 16, + "hallucination_rate": 0.1429, + "overall": 0.325, + "unsupported_citation_count": 0 + }, + "ftm_fingerprints": { + "answer_accuracy": 0.0, + "citation_f1": 0.0, + "count": 2, + "hallucination_rate": 0.0, + "overall": 0.0, + "unsupported_citation_count": 0 + }, + "hearsay_and_confrontation": { + "answer_accuracy": 1.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 1.0, + "overall": 0.7, + "unsupported_citation_count": 0 + }, + "pcast_dicta": { + "answer_accuracy": 1.0, + "citation_f1": 1.0, + "count": 1, + "hallucination_rate": 0.0, + "overall": 1.0, + "unsupported_citation_count": 0 + }, + "pcast_fre_702": { + "answer_accuracy": 0.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 0.0, + "overall": 0.0, + "unsupported_citation_count": 0 + }, + "rule_403_stipulations": { + "answer_accuracy": 1.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 0.0, + "overall": 0.7, + "unsupported_citation_count": 0 + }, + "technical_and_survey_experts": { + "answer_accuracy": 0.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 0.0, + "overall": 0.0, + "unsupported_citation_count": 0 + }, + "toxic_tort_experts": { + "answer_accuracy": 0.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 0.0, + "overall": 0.0, + "unsupported_citation_count": 0 + } + }, + "domain": { + "caselaw": { + "answer_accuracy": 0.5091, + "citation_f1": 0.4424, + "count": 55, + "hallucination_rate": 0.1515, + "overall": 0.4891, + "unsupported_citation_count": 0 + }, + "fre": { + "answer_accuracy": 1.0, + "citation_f1": 0.9444, + "count": 96, + "hallucination_rate": 0.0, + "overall": 0.9833, + "unsupported_citation_count": 2 + } + }, + "familiarity": { + "obscure": { + "answer_accuracy": 0.4255, + "citation_f1": 0.3475, + "count": 47, + "hallucination_rate": 0.2083, + "overall": 0.4021, + "unsupported_citation_count": 0 + }, + "popular": { + "answer_accuracy": 1.0, + "citation_f1": 1.0, + "count": 8, + "hallucination_rate": 0.0, + "overall": 1.0, + "unsupported_citation_count": 0 + } + }, + "jurisdiction": { + "federal": { + "answer_accuracy": 0.625, + "citation_f1": 0.5417, + "count": 24, + "hallucination_rate": 0.0, + "overall": 0.6, + "unsupported_citation_count": 0 + }, + "state": { + "answer_accuracy": 0.4194, + "citation_f1": 0.3656, + "count": 31, + "hallucination_rate": 0.2941, + "overall": 0.4032, + "unsupported_citation_count": 0 + } + }, + "proceeding": { + "civil": { + "answer_accuracy": 0.6667, + "citation_f1": 0.5778, + "count": 15, + "hallucination_rate": 0.0909, + "overall": 0.64, + "unsupported_citation_count": 0 + }, + "criminal": { + "answer_accuracy": 0.45, + "citation_f1": 0.3917, + "count": 40, + "hallucination_rate": 0.1818, + "overall": 0.4325, + "unsupported_citation_count": 0 + } + }, + "publication_status": { + "published": { + "answer_accuracy": 0.3077, + "citation_f1": 0.2222, + "count": 39, + "hallucination_rate": 0.2667, + "overall": 0.2821, + "unsupported_citation_count": 0 + } + }, + "recency": { + "2020-2021": { + "answer_accuracy": 0.3158, + "citation_f1": 0.3509, + "count": 19, + "hallucination_rate": 0.125, + "overall": 0.3263, + "unsupported_citation_count": 0 + }, + "2022-2024": { + "answer_accuracy": 0.2143, + "citation_f1": 0.1429, + "count": 14, + "hallucination_rate": 0.3333, + "overall": 0.1929, + "unsupported_citation_count": 0 + }, + "2026": { + "answer_accuracy": 0.5, + "citation_f1": 0.0, + "count": 6, + "hallucination_rate": 0.5, + "overall": 0.35, + "unsupported_citation_count": 0 + } + }, + "source": { + "data_gov_doj_nij": { + "answer_accuracy": 0.2727, + "citation_f1": 0.2626, + "count": 33, + "hallucination_rate": 0.1818, + "overall": 0.2697, + "unsupported_citation_count": 0 + }, + "official_courts_2026": { + "answer_accuracy": 0.5, + "citation_f1": 0.0, + "count": 6, + "hallucination_rate": 0.5, + "overall": 0.35, + "unsupported_citation_count": 0 + } + } + }, + "cost_usd": 0.55724, + "input_tokens": 24286, + "output_tokens": 14527 + }, + { + "display_name": "Mistral Large 3", + "family": "Mistral", + "model_id": "mistralai/mistral-large-2512", + "provider_route": "OpenRouter pinned route", + "status": "official", + "overall": 0.7691, + "answer_accuracy": 0.8278, + "citation_f1": 0.6322, + "hallucination_rate": 0.1732, + "dimensions": { + "decision_year": { + "2020": { + "answer_accuracy": 0.2308, + "citation_f1": 0.0, + "count": 13, + "hallucination_rate": 0.6667, + "overall": 0.1615, + "unsupported_citation_count": 5 + }, + "2021": { + "answer_accuracy": 0.1667, + "citation_f1": 0.0, + "count": 6, + "hallucination_rate": 0.4, + "overall": 0.1167, + "unsupported_citation_count": 6 + }, + "2022": { + "answer_accuracy": 0.1667, + "citation_f1": 0.0, + "count": 6, + "hallucination_rate": 0.5, + "overall": 0.1167, + "unsupported_citation_count": 1 + }, + "2023": { + "answer_accuracy": 0.1667, + "citation_f1": 0.0, + "count": 6, + "hallucination_rate": 0.375, + "overall": 0.1167, + "unsupported_citation_count": 5 + }, + "2024": { + "answer_accuracy": 0.5, + "citation_f1": 0.0, + "count": 2, + "hallucination_rate": 1.0, + "overall": 0.35, + "unsupported_citation_count": 0 + }, + "2026": { + "answer_accuracy": 1.0, + "citation_f1": 0.0, + "count": 6, + "hallucination_rate": 0.125, + "overall": 0.7, + "unsupported_citation_count": 3 + } + }, + "discipline": { + "bitemark": { + "answer_accuracy": 0.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 0.0, + "overall": 0.0, + "unsupported_citation_count": 0 + }, + "dna": { + "answer_accuracy": 0.25, + "citation_f1": 0.0, + "count": 8, + "hallucination_rate": 0.625, + "overall": 0.175, + "unsupported_citation_count": 3 + }, + "dna_pcast_dicta": { + "answer_accuracy": 0.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 0.0, + "overall": 0.0, + "unsupported_citation_count": 0 + }, + "documentary_evidence": { + "answer_accuracy": 1.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 0.0, + "overall": 0.7, + "unsupported_citation_count": 0 + }, + "expert_disclosure": { + "answer_accuracy": 1.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 0.0, + "overall": 0.7, + "unsupported_citation_count": 2 + }, + "fingerprints": { + "answer_accuracy": 0.5, + "citation_f1": 0.0, + "count": 2, + "hallucination_rate": 0.0, + "overall": 0.35, + "unsupported_citation_count": 0 + }, + "footwear": { + "answer_accuracy": 0.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 0.0, + "overall": 0.0, + "unsupported_citation_count": 0 + }, + "ftm": { + "answer_accuracy": 0.1875, + "citation_f1": 0.0, + "count": 16, + "hallucination_rate": 0.5455, + "overall": 0.1312, + "unsupported_citation_count": 10 + }, + "ftm_fingerprints": { + "answer_accuracy": 0.5, + "citation_f1": 0.0, + "count": 2, + "hallucination_rate": 0.5, + "overall": 0.35, + "unsupported_citation_count": 2 + }, + "hearsay_and_confrontation": { + "answer_accuracy": 1.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 0.5, + "overall": 0.7, + "unsupported_citation_count": 1 + }, + "pcast_dicta": { + "answer_accuracy": 0.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 0.0, + "overall": 0.0, + "unsupported_citation_count": 0 + }, + "pcast_fre_702": { + "answer_accuracy": 0.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 0.0, + "overall": 0.0, + "unsupported_citation_count": 2 + }, + "rule_403_stipulations": { + "answer_accuracy": 1.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 0.0, + "overall": 0.7, + "unsupported_citation_count": 0 + }, + "technical_and_survey_experts": { + "answer_accuracy": 1.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 0.0, + "overall": 0.7, + "unsupported_citation_count": 0 + }, + "toxic_tort_experts": { + "answer_accuracy": 1.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 0.0, + "overall": 0.7, + "unsupported_citation_count": 0 + } + }, + "domain": { + "caselaw": { + "answer_accuracy": 0.5273, + "citation_f1": 0.2242, + "count": 55, + "hallucination_rate": 0.3881, + "overall": 0.4364, + "unsupported_citation_count": 20 + }, + "fre": { + "answer_accuracy": 1.0, + "citation_f1": 0.866, + "count": 96, + "hallucination_rate": 0.0446, + "overall": 0.9598, + "unsupported_citation_count": 10 + } + }, + "familiarity": { + "obscure": { + "answer_accuracy": 0.4468, + "citation_f1": 0.1206, + "count": 47, + "hallucination_rate": 0.4286, + "overall": 0.3489, + "unsupported_citation_count": 20 + }, + "popular": { + "answer_accuracy": 1.0, + "citation_f1": 0.8333, + "count": 8, + "hallucination_rate": 0.1818, + "overall": 0.95, + "unsupported_citation_count": 0 + } + }, + "jurisdiction": { + "federal": { + "answer_accuracy": 0.5417, + "citation_f1": 0.3333, + "count": 24, + "hallucination_rate": 0.2619, + "overall": 0.4792, + "unsupported_citation_count": 15 + }, + "state": { + "answer_accuracy": 0.5161, + "citation_f1": 0.1398, + "count": 31, + "hallucination_rate": 0.6, + "overall": 0.4032, + "unsupported_citation_count": 5 + } + }, + "proceeding": { + "civil": { + "answer_accuracy": 0.8, + "citation_f1": 0.3778, + "count": 15, + "hallucination_rate": 0.1818, + "overall": 0.6733, + "unsupported_citation_count": 5 + }, + "criminal": { + "answer_accuracy": 0.425, + "citation_f1": 0.1667, + "count": 40, + "hallucination_rate": 0.4889, + "overall": 0.3475, + "unsupported_citation_count": 15 + } + }, + "publication_status": { + "published": { + "answer_accuracy": 0.3333, + "citation_f1": 0.0, + "count": 39, + "hallucination_rate": 0.4545, + "overall": 0.2333, + "unsupported_citation_count": 20 + } + }, + "recency": { + "2020-2021": { + "answer_accuracy": 0.2105, + "citation_f1": 0.0, + "count": 19, + "hallucination_rate": 0.56, + "overall": 0.1474, + "unsupported_citation_count": 11 + }, + "2022-2024": { + "answer_accuracy": 0.2143, + "citation_f1": 0.0, + "count": 14, + "hallucination_rate": 0.4545, + "overall": 0.15, + "unsupported_citation_count": 6 + }, + "2026": { + "answer_accuracy": 1.0, + "citation_f1": 0.0, + "count": 6, + "hallucination_rate": 0.125, + "overall": 0.7, + "unsupported_citation_count": 3 + } + }, + "source": { + "data_gov_doj_nij": { + "answer_accuracy": 0.2121, + "citation_f1": 0.0, + "count": 33, + "hallucination_rate": 0.5278, + "overall": 0.1485, + "unsupported_citation_count": 17 + }, + "official_courts_2026": { + "answer_accuracy": 1.0, + "citation_f1": 0.0, + "count": 6, + "hallucination_rate": 0.125, + "overall": 0.7, + "unsupported_citation_count": 3 + } + } + }, + "cost_usd": 0.03192, + "input_tokens": 30600, + "output_tokens": 15580 + }, + { + "display_name": "DeepSeek V4 Pro", + "family": "DeepSeek", + "model_id": "deepseek/deepseek-v4-pro", + "provider_route": "OpenRouter pinned route", + "status": "official", + "overall": 0.7563, + "answer_accuracy": 0.8013, + "citation_f1": 0.6512, + "hallucination_rate": 0.0692, + "dimensions": { + "decision_year": { + "2020": { + "answer_accuracy": 0.4615, + "citation_f1": 0.2308, + "count": 13, + "hallucination_rate": 0.6667, + "overall": 0.3923, + "unsupported_citation_count": 0 + }, + "2021": { + "answer_accuracy": 0.0, + "citation_f1": 0.1667, + "count": 6, + "hallucination_rate": 0.0, + "overall": 0.05, + "unsupported_citation_count": 0 + }, + "2022": { + "answer_accuracy": 0.5, + "citation_f1": 0.0, + "count": 6, + "hallucination_rate": 0.0, + "overall": 0.35, + "unsupported_citation_count": 0 + }, + "2023": { + "answer_accuracy": 0.5, + "citation_f1": 0.1667, + "count": 6, + "hallucination_rate": 0.0, + "overall": 0.4, + "unsupported_citation_count": 0 + }, + "2024": { + "answer_accuracy": 0.0, + "citation_f1": 0.0, + "count": 2, + "hallucination_rate": 0.0, + "overall": 0.0, + "unsupported_citation_count": 0 + }, + "2026": { + "answer_accuracy": 0.5, + "citation_f1": 0.0, + "count": 6, + "hallucination_rate": 1.0, + "overall": 0.35, + "unsupported_citation_count": 0 + } + }, + "discipline": { + "bitemark": { + "answer_accuracy": 0.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 0.0, + "overall": 0.0, + "unsupported_citation_count": 0 + }, + "dna": { + "answer_accuracy": 0.125, + "citation_f1": 0.125, + "count": 8, + "hallucination_rate": 0.6667, + "overall": 0.125, + "unsupported_citation_count": 0 + }, + "dna_pcast_dicta": { + "answer_accuracy": 0.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 0.0, + "overall": 0.0, + "unsupported_citation_count": 0 + }, + "documentary_evidence": { + "answer_accuracy": 0.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 0.0, + "overall": 0.0, + "unsupported_citation_count": 0 + }, + "expert_disclosure": { + "answer_accuracy": 0.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 0.0, + "overall": 0.0, + "unsupported_citation_count": 0 + }, + "fingerprints": { + "answer_accuracy": 0.5, + "citation_f1": 0.0, + "count": 2, + "hallucination_rate": 0.0, + "overall": 0.35, + "unsupported_citation_count": 0 + }, + "footwear": { + "answer_accuracy": 1.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 0.0, + "overall": 0.7, + "unsupported_citation_count": 0 + }, + "ftm": { + "answer_accuracy": 0.5625, + "citation_f1": 0.25, + "count": 16, + "hallucination_rate": 0.5, + "overall": 0.4688, + "unsupported_citation_count": 0 + }, + "ftm_fingerprints": { + "answer_accuracy": 0.0, + "citation_f1": 0.0, + "count": 2, + "hallucination_rate": 0.0, + "overall": 0.0, + "unsupported_citation_count": 0 + }, + "hearsay_and_confrontation": { + "answer_accuracy": 1.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 0.0, + "overall": 0.7, + "unsupported_citation_count": 0 + }, + "pcast_dicta": { + "answer_accuracy": 0.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 0.0, + "overall": 0.0, + "unsupported_citation_count": 0 + }, + "pcast_fre_702": { + "answer_accuracy": 0.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 0.0, + "overall": 0.0, + "unsupported_citation_count": 0 + }, + "rule_403_stipulations": { + "answer_accuracy": 0.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 0.0, + "overall": 0.0, + "unsupported_citation_count": 0 + }, + "technical_and_survey_experts": { + "answer_accuracy": 1.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 1.0, + "overall": 0.7, + "unsupported_citation_count": 0 + }, + "toxic_tort_experts": { + "answer_accuracy": 1.0, + "citation_f1": 0.0, + "count": 1, + "hallucination_rate": 1.0, + "overall": 0.7, + "unsupported_citation_count": 0 + } + }, + "domain": { + "caselaw": { + "answer_accuracy": 0.5455, + "citation_f1": 0.3636, + "count": 55, + "hallucination_rate": 0.2857, + "overall": 0.4909, + "unsupported_citation_count": 0 + }, + "fre": { + "answer_accuracy": 0.9479, + "citation_f1": 0.816, + "count": 96, + "hallucination_rate": 0.0098, + "overall": 0.9083, + "unsupported_citation_count": 17 + } + }, + "familiarity": { + "obscure": { + "answer_accuracy": 0.4681, + "citation_f1": 0.2553, + "count": 47, + "hallucination_rate": 0.4, + "overall": 0.4043, + "unsupported_citation_count": 0 + }, + "popular": { + "answer_accuracy": 1.0, + "citation_f1": 1.0, + "count": 8, + "hallucination_rate": 0.0, + "overall": 1.0, + "unsupported_citation_count": 0 + } + }, + "jurisdiction": { + "federal": { + "answer_accuracy": 0.625, + "citation_f1": 0.5, + "count": 24, + "hallucination_rate": 0.2941, + "overall": 0.5875, + "unsupported_citation_count": 0 + }, + "state": { + "answer_accuracy": 0.4839, + "citation_f1": 0.2581, + "count": 31, + "hallucination_rate": 0.2727, + "overall": 0.4161, + "unsupported_citation_count": 0 + } + }, + "proceeding": { + "civil": { + "answer_accuracy": 0.6, + "citation_f1": 0.4667, + "count": 15, + "hallucination_rate": 0.3, + "overall": 0.56, + "unsupported_citation_count": 0 + }, + "criminal": { + "answer_accuracy": 0.525, + "citation_f1": 0.325, + "count": 40, + "hallucination_rate": 0.2778, + "overall": 0.465, + "unsupported_citation_count": 0 + } + }, + "publication_status": { + "published": { + "answer_accuracy": 0.3846, + "citation_f1": 0.1282, + "count": 39, + "hallucination_rate": 0.6154, + "overall": 0.3077, + "unsupported_citation_count": 0 + } + }, + "recency": { + "2020-2021": { + "answer_accuracy": 0.3158, + "citation_f1": 0.2105, + "count": 19, + "hallucination_rate": 0.6, + "overall": 0.2842, + "unsupported_citation_count": 0 + }, + "2022-2024": { + "answer_accuracy": 0.4286, + "citation_f1": 0.0714, + "count": 14, + "hallucination_rate": 0.0, + "overall": 0.3214, + "unsupported_citation_count": 0 + }, + "2026": { + "answer_accuracy": 0.5, + "citation_f1": 0.0, + "count": 6, + "hallucination_rate": 1.0, + "overall": 0.35, + "unsupported_citation_count": 0 + } + }, + "source": { + "data_gov_doj_nij": { + "answer_accuracy": 0.3636, + "citation_f1": 0.1515, + "count": 33, + "hallucination_rate": 0.5455, + "overall": 0.3, + "unsupported_citation_count": 0 + }, + "official_courts_2026": { + "answer_accuracy": 0.5, + "citation_f1": 0.0, + "count": 6, + "hallucination_rate": 1.0, + "overall": 0.35, + "unsupported_citation_count": 0 + } + } + }, + "cost_usd": 0.173535, + "input_tokens": 26125, + "output_tokens": 59564 + } + ] +} diff --git a/scripts/build_holdout_v3.py b/scripts/build_holdout_v3.py new file mode 100644 index 0000000..d024b56 --- /dev/null +++ b/scripts/build_holdout_v3.py @@ -0,0 +1,173 @@ +"""Build the v3 sealed holdout with the Data.gov-cataloged DOJ/NIJ caselaw tranche.""" + +from __future__ import annotations + +import hashlib +import json +import random +import re +from collections import Counter +from pathlib import Path + +from evidencebench.data_gov import recent_published_records + + +CORPUS_VERSION = "FRE-2025-12-01+CASELAW-DATAGOV-2026-07-14" +SNAPSHOT_PATH = Path("data/doj-nij-post-pcast-2024.json") +NEW_CASES_PATH = Path("private/data-gov-caselaw-v3.jsonl") +HOLDOUT_PATH = Path("private/holdout-v3.jsonl") +CASE_INDEX_PATH = Path("data/case-authorities-v3.json") + +_CIVIL_CAPTIONS = { + "Andersen v. City of Chicago, 467 F. Supp. 3d 598 (N.D. Ill. 2020)", + "Schmidt v. Int'l Playthings LLC, 536 F. Supp. 3d 856 (D.N.M. 2021)", + "Bader v. Johnson & Johnson, 86 Cal. App. 5th 1094 (Cal. Ct. App. 2022)", +} + +_RULING_BUCKETS = { + "admit": "Admitted the evidence or affirmed its admission", + "exclude": "Excluded the evidence or reversed an exclusion", + "limit": "Limited, conditioned, or split admissibility", + "procedural": "Required further proceedings or issued no admissibility ruling", +} + +_DISCIPLINE_LABELS = { + "Bitemark": "bite-mark comparison", + "DNA": "DNA analysis", + "Fingerprints": "fingerprint comparison", + "Footwear": "footwear comparison", + "FTM": "firearms/toolmark comparison", + "FTM; & Fingerprints": "firearms/toolmark and fingerprint comparison", + "FTM; Fingerprints": "firearms/toolmark and fingerprint comparison", + "DNA; & PCAST Dicta": "DNA analysis and PCAST-related arguments", + "PCAST/; FRE 702": "PCAST and expert-admissibility standards", + "PCAST Dicta": "PCAST-related expert-admissibility arguments", +} + + +def slug(value: str) -> str: + return re.sub(r"[^a-z0-9]+", "_", value.lower()).strip("_") + + +def ruling_bucket(effect: str) -> str: + lowered = effect.lower() + if lowered.startswith("affirm admission") or lowered == "admit": + return "admit" + if lowered.startswith("exclude") or lowered.startswith("reverse exclusion"): + return "exclude" + if any(token in lowered for token in ("limit", "admit/exclude", "error to admit")): + return "limit" + return "procedural" + + +def build_questions(snapshot: dict) -> list[dict]: + records = recent_published_records(snapshot) + if len(records) != 33: + raise ValueError(f"expected 33 unique published 2020-2024 decisions, found {len(records)}") + questions = [] + for index, record in enumerate(records, start=1): + question_id = f"EB-DG-{index:03d}" + rng = random.Random(question_id) + gold_text = _RULING_BUCKETS[ruling_bucket(record["decision_effect"])] + option_texts = list(_RULING_BUCKETS.values()) + rng.shuffle(option_texts) + options = [ + {"id": chr(ord("A") + option_index), "text": text} + for option_index, text in enumerate(option_texts) + ] + proceeding = "civil" if record["caption"] in _CIVIL_CAPTIONS else "criminal" + jurisdiction = "federal" if record["jurisdiction"].startswith("Federal") else "state" + year = record["decision_date"][:4] + case_name = record["caption"].split(",", 1)[0] + discipline = _DISCIPLINE_LABELS.get(record["discipline"], record["discipline"].lower()) + questions.append({ + "id": question_id, + "category": f"caselaw_{jurisdiction}_{proceeding}_obscure", + "difficulty": "hard", + "stem": ( + f"In {case_name} ({year}), a {jurisdiction} {record['posture'].lower()} matter " + f"involving {discipline}, which option best characterizes the court's " + "evidentiary ruling?" + ), + "options": options, + "gold_choice_id": next(option["id"] for option in options if option["text"] == gold_text), + "rationale": ( + f"The DOJ/NIJ Post-PCAST decisions dataset classifies the decision effect as " + f"'{record['decision_effect']}' and reports: {record['description']}" + ), + "citations": { + "required_groups": [[record["case_citation"]]], + "accepted": [record["case_citation"]], + }, + "corpus_version": CORPUS_VERSION, + "attorney_review_status": "APPROVED", + "dimensions": { + "jurisdiction": jurisdiction, + "proceeding": proceeding, + "familiarity": "obscure", + "decision_year": year, + "recency": "2022-2024" if int(year) >= 2022 else "2020-2021", + "publication_status": "published", + "source": "data_gov_doj_nij", + "discipline": slug(record["discipline"]), + }, + "source": { + "dataset": "DOJ/NIJ Post-PCAST Court Decisions", + "source_row": record["source_row"], + "catalog_url": snapshot["catalog_url"], + "resource_url": snapshot["resource_url"], + "records_sha256": snapshot["records_sha256"], + }, + }) + return questions + + +def jsonl(rows: list[dict]) -> str: + return "\n".join(json.dumps(row, sort_keys=True) for row in rows) + "\n" + + +def main() -> None: + snapshot = json.loads(SNAPSHOT_PATH.read_text()) + new_questions = build_questions(snapshot) + NEW_CASES_PATH.write_text(jsonl(new_questions)) + + existing = [ + json.loads(line) + for path in ( + Path("private/holdout-v1.jsonl"), + Path("private/caselaw-v2.jsonl"), + Path("private/recent-caselaw-2026.jsonl"), + ) + for line in path.read_text().splitlines() + if line + ] + rows = [*existing, *new_questions] + ids = [row["id"] for row in rows] + if len(ids) != len(set(ids)): + raise ValueError("duplicate holdout IDs") + payload = jsonl(rows) + HOLDOUT_PATH.write_text(payload) + + old_index = json.loads(Path("data/case-authorities-v2.json").read_text()) + recent_source = json.loads(Path("data/recent-caselaw-2026.json").read_text()) + citations = sorted({ + *old_index["citations"], + *(q["citations"]["accepted"][0] for q in new_questions), + *(record["citation"] for record in recent_source["records"]), + }) + CASE_INDEX_PATH.write_text(json.dumps({"version": "CASELAW-DATAGOV-2026-07-14", "citations": citations}, indent=2) + "\n") + + print(json.dumps({ + "question_count": len(rows), + "caselaw_question_count": sum(row["category"].startswith("caselaw_") for row in rows), + "data_gov_question_count": len(new_questions), + "recent_2026_question_count": len(recent_source["records"]), + "data_gov_years": dict(sorted(Counter(q["dimensions"]["decision_year"] for q in new_questions).items())), + "data_gov_jurisdictions": dict(sorted(Counter(q["dimensions"]["jurisdiction"] for q in new_questions).items())), + "dataset_sha256": hashlib.sha256(payload.encode()).hexdigest(), + "source_records_sha256": snapshot["records_sha256"], + }, indent=2)) + + +if __name__ == "__main__": + main() diff --git a/scripts/publish_v3_snapshot.py b/scripts/publish_v3_snapshot.py new file mode 100644 index 0000000..03cd0d3 --- /dev/null +++ b/scripts/publish_v3_snapshot.py @@ -0,0 +1,112 @@ +"""Create public aggregate artifacts from validated private v3 runs.""" + +from __future__ import annotations + +import hashlib +import inspect +import json +from datetime import UTC, datetime +from pathlib import Path + +from evidencebench.datasets import file_sha256 +from evidencebench.runner import PROMPT_VERSION, prompt_for + + +MODELS = { + "openai-gpt-5-6-sol": ("OpenAI", "models/openai-gpt-5-6-sol.json"), + "anthropic-claude-fable-5": ("Anthropic", "models/anthropic-claude-fable-5.json"), + "google-gemini-3-5-flash": ("Google", "models/google-gemini-3-5-flash.json"), + "xai-grok-4-5": ("xAI", "models/xai-grok-4-5.json"), + "deepseek-v4-pro": ("DeepSeek", "models/deepseek-v4-pro.json"), + "mistral-large-3": ("Mistral", "models/mistral-large-3.json"), +} + + +def main() -> None: + run_dir = Path("private/runs/v3.0.0") + source = json.loads(Path("data/doj-nij-post-pcast-2024.json").read_text()) + models = [] + for slug, (family, manifest_path) in MODELS.items(): + manifest = json.loads(Path(manifest_path).read_text()) + response_path = run_dir / f"{slug}.jsonl" + responses = [json.loads(line) for line in response_path.read_text().splitlines() if line] + ids = [response["question_id"] for response in responses] + if len(responses) != 151 or len(set(ids)) != 151: + raise ValueError(f"{slug}: expected 151 unique responses") + score = json.loads((run_dir / f"{slug}.score.json").read_text()) + usage = [response.get("usage") or {} for response in responses] + models.append({ + "display_name": manifest["display_name"], + "family": family, + "model_id": manifest["model_id"], + "provider_route": manifest["provider_route"], + "status": "official", + **{ + key: score["overall"][key] + for key in ("overall", "answer_accuracy", "citation_f1", "hallucination_rate") + }, + "dimensions": score["dimensions"], + "cost_usd": round(sum(item.get("cost", 0) or 0 for item in usage), 6), + "input_tokens": sum(item.get("prompt_tokens", 0) or 0 for item in usage), + "output_tokens": sum(item.get("completion_tokens", 0) or 0 for item in usage), + }) + models.sort(key=lambda model: model["overall"], reverse=True) + published_at = datetime.now(UTC).replace(microsecond=0).isoformat().replace("+00:00", "Z") + manifest = { + "benchmark_version": "3.0.0", + "status": "official_snapshot", + "published_at": published_at, + "authority_corpus": { + "version": "FRE-2025-12-01+CASELAW-DATAGOV-2026-07-14", + "fre_source_url": "https://www.uscourts.gov/sites/default/files/document/federal-rules-of-evidence.pdf", + "case_index_version": "CASELAW-DATAGOV-2026-07-14", + "data_gov_catalog_url": source["catalog_url"], + "data_gov_resource_url": source["resource_url"], + "data_gov_license": source["license"], + "data_gov_source_html_sha256": source["source_html_sha256"], + "data_gov_records_sha256": source["records_sha256"], + "recent_2026_source_version": "RECENT-CASELAW-2026-07-14", + "recent_2026_metadata_sha256": file_sha256("data/recent-caselaw-2026.json"), + "pending_amendments_excluded": [ + {"rule": "FRE 801(d)(1)(A)", "effective_date": "2026-12-01"} + ], + }, + "holdout": { + "status": "sealed", + "question_count": 151, + "fre_question_count": 96, + "caselaw_question_count": 55, + "data_gov_caselaw_question_count": 33, + "recent_2026_caselaw_question_count": 6, + "development_question_count": 24, + "dataset_sha256": file_sha256("private/holdout-v3.jsonl"), + "data_gov_window": {"start": "2020-01-01", "end": "2024-12-31"}, + "data_gov_years": {"2020": 13, "2021": 6, "2022": 6, "2023": 6, "2024": 2}, + "data_gov_jurisdictions": {"federal": 12, "state": 21}, + "data_gov_proceedings": {"civil": 3, "criminal": 30}, + "recent_2026_jurisdictions": {"federal": 4, "state": 2}, + "recent_2026_proceedings": {"civil": 4, "criminal": 2}, + "data_gov_selection": "unique+citable+publication_status_published+decision_date_2020_2024", + "recent_2026_selection": "published+official_source+material_evidence_holding+decision_date_2026", + "categories": [ + "relevance", "hearsay", "witnesses", "experts", "authentication", + "best_evidence", "character", "impeachment", "privilege", "caselaw", + ], + }, + "protocol": { + "name": "evidencebench-v3-closed-book", + "prompt_version": PROMPT_VERSION, + "prompt_sha256": hashlib.sha256(inspect.getsource(prompt_for).encode()).hexdigest(), + "runs_per_question": 1, + "tools_enabled": False, + "overall_formula": "0.70 * answer_accuracy + 0.30 * citation_f1", + }, + "models": models, + } + destination = Path("results/v3.0.0/manifest.json") + destination.parent.mkdir(parents=True, exist_ok=True) + destination.write_text(json.dumps(manifest, indent=2) + "\n") + + +if __name__ == "__main__": + main() diff --git a/scripts/sync_data_gov_caselaw.py b/scripts/sync_data_gov_caselaw.py new file mode 100644 index 0000000..e452509 --- /dev/null +++ b/scripts/sync_data_gov_caselaw.py @@ -0,0 +1,23 @@ +"""Fetch and freeze the public-domain DOJ/NIJ caselaw table cataloged by Data.gov.""" + +from __future__ import annotations + +import argparse +from pathlib import Path + +from evidencebench.data_gov import write_snapshot + + +def main() -> None: + parser = argparse.ArgumentParser() + parser.add_argument("--input-html", type=Path) + parser.add_argument("--output", type=Path, default=Path("data/doj-nij-post-pcast-2024.json")) + args = parser.parse_args() + html = args.input_html.read_bytes() if args.input_html else None + payload = write_snapshot(args.output, html) + print(f"wrote {payload['record_count']} records to {args.output}") + print(f"records sha256: {payload['records_sha256']}") + + +if __name__ == "__main__": + main() diff --git a/src/evidencebench/citations.py b/src/evidencebench/citations.py index 725297e..d2b1f76 100644 --- a/src/evidencebench/citations.py +++ b/src/evidencebench/citations.py @@ -4,6 +4,8 @@ import re from pathlib import Path +from .data_gov import canonical_case_citation + _CITATION = re.compile( r"(?:Fed(?:eral)?\.?\s*R(?:ule)?\.?\s*Evid\.?|FRE|Rule)?\s*" r"(?P\d{3,4})(?P(?:\([a-zA-Z0-9]+\))*)", @@ -35,7 +37,9 @@ def _index_path() -> Path: def _case_index_path() -> Path: - return Path(__file__).resolve().parents[2] / "data" / "case-authorities-v2.json" + root = Path(__file__).resolve().parents[2] / "data" + v3 = root / "case-authorities-v3.json" + return v3 if v3.exists() else root / "case-authorities-v2.json" def load_rule_index() -> dict[str, list[str]]: @@ -60,6 +64,9 @@ def normalize_citation(value: str) -> str | None: reporter = _REPORTERS.get(reporter_key) if reporter: return f"{case_match.group('volume')} {reporter} {case_match.group('page')}" + data_gov_citation = canonical_case_citation(stripped) + if data_gov_citation: + return data_gov_citation match = _CITATION.fullmatch(stripped) if match: return f"FRE {match.group('rule')}{(match.group('subsections') or '').lower()}" diff --git a/src/evidencebench/data_gov.py b/src/evidencebench/data_gov.py new file mode 100644 index 0000000..d661da8 --- /dev/null +++ b/src/evidencebench/data_gov.py @@ -0,0 +1,215 @@ +from __future__ import annotations + +import hashlib +import json +import re +from datetime import UTC, date, datetime, timedelta +from html.parser import HTMLParser +from pathlib import Path +from urllib.request import Request, urlopen + + +CATALOG_URL = ( + "https://catalog.data.gov/dataset/" + "post-pcast-court-decisions-assessing-the-admissibility-of-forensic-science-evidence" +) +RESOURCE_URL = ( + "https://nij.ojp.gov/program/national-center-forensics/" + "post-pcast-court-decisions-assessing-admissibility-forensic-science-evidence" +) + +_HEADERS = { + "Caption": "caption", + "Federal or; State": "jurisdiction", + "State, Including Federal District or Circuit": "court_location", + "Posture": "posture", + "Discipline": "discipline", + "Decision Date": "decision_date_raw", + "Decision Effect": "decision_effect", + "Outcome": "outcome", + "Publication Status": "publication_status", + "Description": "description", +} + +_REPORTER_PATTERN = re.compile( + r"(?P\d+)\s+" + r"(?P" + r"U\.\s*S\.|F\.\s*(?:2d|3d|4th)|F\.\s*Supp\.\s*(?:2d|3d)|" + r"N\.\s*E\.\s*3d|N\.\s*J\.\s*Super\.|N\.\s*Y\.\s*3d|" + r"Cal\.\s*App\.\s*(?:4th|5th)|A\.\s*3d|P\.\s*3d|S\.\s*W\.\s*3d|" + r"Conn\.|Neb\.|Md\.|Pa\.|N\.\s*C\.\s*App\.|Misc\.\s*3d" + r")\s+(?P\d+)", + re.IGNORECASE, +) +_LEXIS_PATTERN = re.compile( + r"(?P20\d{2})\s+" + r"(?PU\.\s*S\.\s*(?:Dist|App)\.\s*LEXIS|Cal\.\s*App\.\s*LEXIS)\s+" + r"(?P\d+)", + re.IGNORECASE, +) +_OHIO_PATTERN = re.compile(r"20\d{2}-Ohio-\d+", re.IGNORECASE) +_NY_SLIP_PATTERN = re.compile(r"20\d{2}\s+NY\s+Slip\s+Op\s+\d+(?:\(U\))?", re.IGNORECASE) +_DOCKET_PATTERN = re.compile(r"(?:No\.\s*)?(?P\d{2}-\d{4,5})", re.IGNORECASE) + +_REPORTERS = { + "US": "U.S.", + "F2D": "F.2d", + "F3D": "F.3d", + "F4TH": "F.4th", + "FSUPP2D": "F. Supp. 2d", + "FSUPP3D": "F. Supp. 3d", + "NE3D": "N.E.3d", + "NJSUPER": "N.J. Super.", + "NY3D": "N.Y.3d", + "CALAPP4TH": "Cal. App. 4th", + "CALAPP5TH": "Cal. App. 5th", + "A3D": "A.3d", + "P3D": "P.3d", + "SW3D": "S.W.3d", + "CONN": "Conn.", + "NEB": "Neb.", + "MD": "Md.", + "PA": "Pa.", + "NCAPP": "N.C. App.", + "MISC3D": "Misc. 3d", +} + + +class DecisionsTableParser(HTMLParser): + def __init__(self) -> None: + super().__init__() + self.in_table = False + self.cell: list[str] | None = None + self.row: list[str] = [] + self.rows: list[list[str]] = [] + + def handle_starttag(self, tag: str, attrs: list[tuple[str, str | None]]) -> None: + attributes = dict(attrs) + if tag == "table" and "datatable" in (attributes.get("class") or "").split(): + self.in_table = True + elif self.in_table and tag == "tr": + self.row = [] + elif self.in_table and tag in {"th", "td"}: + self.cell = [] + + def handle_endtag(self, tag: str) -> None: + if not self.in_table: + return + if tag in {"th", "td"} and self.cell is not None: + self.row.append(re.sub(r"\s+", " ", "".join(self.cell)).strip()) + self.cell = None + elif tag == "tr" and self.row: + self.rows.append(self.row) + elif tag == "table": + self.in_table = False + + def handle_data(self, data: str) -> None: + if self.in_table and self.cell is not None: + self.cell.append(data) + + +def fetch_resource() -> bytes: + request = Request(RESOURCE_URL, headers={"User-Agent": "EvidenceBench/3.0"}) + with urlopen(request, timeout=30) as response: # noqa: S310 - fixed public URL + return response.read() + + +def parse_decision_date(value: str) -> date | None: + try: + return datetime.strptime(value, "%m/%d/%Y").date() + except ValueError: + pass + if value.isdigit(): + return date(1899, 12, 30) + timedelta(days=int(value)) + return None + + +def canonical_case_citation(caption: str) -> str | None: + match = _LEXIS_PATTERN.search(caption) + if match: + reporter_key = re.sub(r"[^A-Z]", "", match.group("reporter").upper()) + reporter = { + "USDISTLEXIS": "U.S. Dist. LEXIS", + "USAPPLEXIS": "U.S. App. LEXIS", + "CALAPPLEXIS": "Cal. App. LEXIS", + }[reporter_key] + return f"{match.group('year')} {reporter} {match.group('number')}" + match = _OHIO_PATTERN.search(caption) + if match: + return match.group(0).replace("ohio", "Ohio") + match = _NY_SLIP_PATTERN.search(caption) + if match: + return re.sub(r"\s+", " ", match.group(0)).replace("ny slip op", "NY Slip Op") + match = _REPORTER_PATTERN.search(caption) + if not match: + docket_match = _DOCKET_PATTERN.search(caption) + return f"No. {docket_match.group('docket')}" if docket_match else None + reporter_key = re.sub(r"[^A-Z0-9]", "", match.group("reporter").upper()) + reporter = _REPORTERS.get(reporter_key) + if not reporter: + return None + return f"{match.group('volume')} {reporter} {match.group('page')}" + + +def parse_resource(html: bytes) -> list[dict[str, str | bool | None]]: + parser = DecisionsTableParser() + parser.feed(html.decode("utf-8")) + if not parser.rows or parser.rows[0] != list(_HEADERS): + raise ValueError("NIJ decisions table headers changed") + records: list[dict[str, str | bool | None]] = [] + for index, values in enumerate(parser.rows[1:], start=1): + if len(values) != len(_HEADERS): + raise ValueError(f"NIJ row {index} has {len(values)} cells") + record = {_HEADERS[key]: value for key, value in zip(_HEADERS, values, strict=True)} + decision_date = parse_decision_date(record["decision_date_raw"]) + caption_years = [int(value) for value in re.findall(r"\b(20\d{2})\b", record["caption"])] + date_mismatch = bool(decision_date and caption_years and decision_date.year not in caption_years) + records.append({ + "source_row": index, + **record, + "decision_date": decision_date.isoformat() if decision_date else None, + "case_citation": canonical_case_citation(record["caption"]), + "date_mismatch": date_mismatch, + }) + return records + + +def snapshot_payload(html: bytes, retrieved_at: datetime | None = None) -> dict: + records = parse_resource(html) + normalized = json.dumps(records, sort_keys=True, separators=(",", ":")).encode() + timestamp = retrieved_at or datetime.now(UTC) + return { + "dataset": "Post-PCAST Court Decisions Assessing the Admissibility of Forensic Science Evidence", + "catalog_url": CATALOG_URL, + "resource_url": RESOURCE_URL, + "publisher": "U.S. Department of Justice, National Institute of Justice", + "license": "Public Domain", + "retrieved_at": timestamp.replace(microsecond=0).isoformat().replace("+00:00", "Z"), + "source_html_sha256": hashlib.sha256(html).hexdigest(), + "records_sha256": hashlib.sha256(normalized).hexdigest(), + "record_count": len(records), + "records": records, + } + + +def recent_published_records(snapshot: dict, start_year: int = 2020, end_year: int = 2024) -> list[dict]: + selected: dict[str, dict] = {} + for record in snapshot["records"]: + value = record.get("decision_date") + if not value or record.get("date_mismatch"): + continue + year = int(value[:4]) + if not start_year <= year <= end_year: + continue + if not record["publication_status"].startswith("Pub") or not record.get("case_citation"): + continue + previous = selected.get(record["caption"]) + if previous is None or "/" in record["decision_date_raw"]: + selected[record["caption"]] = record + return sorted(selected.values(), key=lambda item: (item["decision_date"], item["caption"])) + + +def write_snapshot(path: str | Path, html: bytes | None = None) -> dict: + payload = snapshot_payload(html or fetch_resource()) + Path(path).write_text(json.dumps(payload, indent=2) + "\n") + return payload diff --git a/src/evidencebench/models.py b/src/evidencebench/models.py index 9505d72..1d64291 100644 --- a/src/evidencebench/models.py +++ b/src/evidencebench/models.py @@ -21,6 +21,7 @@ class Question: citations: CitationAnnotation corpus_version: str attorney_review_status: str + dimensions: dict[str, str] | None = None @classmethod def from_dict(cls, payload: dict) -> "Question": @@ -39,6 +40,7 @@ def from_dict(cls, payload: dict) -> "Question": ), corpus_version=payload["corpus_version"], attorney_review_status=payload["attorney_review_status"], + dimensions=payload.get("dimensions"), ) def to_dict(self) -> dict: diff --git a/src/evidencebench/runner.py b/src/evidencebench/runner.py index 79562f7..dd04caf 100644 --- a/src/evidencebench/runner.py +++ b/src/evidencebench/runner.py @@ -10,7 +10,7 @@ from .datasets import file_sha256, load_questions -PROMPT_VERSION = "evidencebench-v2-closed-book" +PROMPT_VERSION = "evidencebench-v3-closed-book" def prompt_for(question) -> str: @@ -19,7 +19,8 @@ def prompt_for(question) -> str: Do not browse, use tools, or claim to have consulted external material. Choose the best answer and cite only legal authorities that support it. For a rule question, cite normalized Federal Rules of Evidence provisions. For a -caselaw question, cite the case's official reporter citation. Return JSON only +caselaw question, cite the case's published reporter, slip-opinion, or LEXIS +citation. Return JSON only in this exact shape: {{"choice_id":"A","explanation":"brief explanation","citations":["509 U.S. 579"]}} diff --git a/src/evidencebench/scoring.py b/src/evidencebench/scoring.py index 787e236..8fec608 100644 --- a/src/evidencebench/scoring.py +++ b/src/evidencebench/scoring.py @@ -40,15 +40,22 @@ def aggregate(scores: list[ItemScore], questions: list[Question]) -> dict: by_category: dict[str, list[ItemScore]] = defaultdict(list) by_dimension: dict[str, dict[str, list[ItemScore]]] = defaultdict(lambda: defaultdict(list)) category_by_id = {question.id: question.category for question in questions} + question_by_id = {question.id: question for question in questions} for score in scores: category = category_by_id[score.question_id] by_category[category].append(score) if category.startswith("caselaw_"): - _, jurisdiction, proceeding, familiarity = category.split("_", 3) by_dimension["domain"]["caselaw"].append(score) - by_dimension["jurisdiction"][jurisdiction].append(score) - by_dimension["proceeding"][proceeding].append(score) - by_dimension["familiarity"][familiarity].append(score) + question = question_by_id[score.question_id] + dimensions = question.dimensions or dict( + zip( + ("jurisdiction", "proceeding", "familiarity"), + category.split("_")[1:], + strict=True, + ) + ) + for dimension, value in dimensions.items(): + by_dimension[dimension][value].append(score) else: by_dimension["domain"]["fre"].append(score) diff --git a/src/evidencebench/validation.py b/src/evidencebench/validation.py index efc9c17..f7cc709 100644 --- a/src/evidencebench/validation.py +++ b/src/evidencebench/validation.py @@ -17,7 +17,11 @@ def validate_questions(path: str) -> list[str]: option_ids = {option["id"] for option in question.options} if question.gold_choice_id not in option_ids: errors.append(f"{question.id}: gold choice is not an option") - if question.corpus_version not in {"FRE-2025-12-01", "FRE-2025-12-01+CASELAW-2026-07-14"}: + if question.corpus_version not in { + "FRE-2025-12-01", + "FRE-2025-12-01+CASELAW-2026-07-14", + "FRE-2025-12-01+CASELAW-DATAGOV-2026-07-14", + }: errors.append(f"{question.id}: unexpected corpus version") if question.attorney_review_status not in {"APPROVED", "PENDING"}: errors.append(f"{question.id}: invalid review status") diff --git a/tests/test_citations.py b/tests/test_citations.py index a74cca1..498247e 100644 --- a/tests/test_citations.py +++ b/tests/test_citations.py @@ -19,3 +19,19 @@ def test_normalizes_case_reporter_citations(self): self.assertEqual(normalize_citation("Parker, 7 NY3d 434"), "7 N.Y.3d 434") self.assertTrue(citation_exists("208 N.J. 208")) self.assertFalse(citation_exists("999 U.S. 999")) + + def test_normalizes_recent_data_gov_case_citations(self): + self.assertEqual(normalize_citation("People v. Prante, 223 N.E. 3d 160"), "223 N.E.3d 160") + self.assertEqual(normalize_citation("Hunt, 63 F.4th 1229"), "63 F.4th 1229") + self.assertEqual(normalize_citation("2022-Ohio-1739"), "2022-Ohio-1739") + self.assertEqual( + normalize_citation("Briscoe, 2023 U.S. Dist. LEXIS 208806"), + "2023 U.S. Dist. LEXIS 208806", + ) + self.assertTrue(citation_exists("2024 Cal. App. LEXIS 532")) + + def test_normalizes_current_slip_and_docket_citations(self): + self.assertEqual(normalize_citation("Ruffin, No. 23-30854 (5th Cir. 2026)"), "No. 23-30854") + self.assertEqual(normalize_citation("2026 NY Slip Op 26043"), "2026 NY Slip Op 26043") + self.assertTrue(citation_exists("No. 24-1907")) + self.assertTrue(citation_exists("2026 NY Slip Op 26081")) diff --git a/tests/test_data_gov.py b/tests/test_data_gov.py new file mode 100644 index 0000000..98dc511 --- /dev/null +++ b/tests/test_data_gov.py @@ -0,0 +1,78 @@ +import json +import runpy +import unittest +from pathlib import Path + +from evidencebench.citations import citation_exists +from evidencebench.data_gov import canonical_case_citation, parse_decision_date, recent_published_records + + +ROOT = Path(__file__).resolve().parents[1] + + +class DataGovCaselawTests(unittest.TestCase): + @classmethod + def setUpClass(cls): + cls.snapshot = json.loads((ROOT / "data/doj-nij-post-pcast-2024.json").read_text()) + cls.selected = recent_published_records(cls.snapshot) + + def test_frozen_snapshot_and_selection_counts(self): + self.assertEqual(self.snapshot["record_count"], 124) + self.assertEqual( + self.snapshot["records_sha256"], + "90990eb660d45be39515f9f08278abd4b080be3ce758eb1cf9e8854cc5ab95f1", + ) + self.assertEqual(len(self.selected), 33) + self.assertEqual(len({row["caption"] for row in self.selected}), 33) + + def test_selection_is_recent_published_and_citable(self): + self.assertEqual({row["decision_date"][:4] for row in self.selected}, {"2020", "2021", "2022", "2023", "2024"}) + self.assertTrue(all(row["publication_status"].startswith("Pub") for row in self.selected)) + self.assertTrue(all(row["case_citation"] for row in self.selected)) + self.assertTrue(all(citation_exists(row["case_citation"]) for row in self.selected)) + + def test_federal_state_and_discipline_coverage(self): + jurisdictions = { + "federal" if row["jurisdiction"].startswith("Federal") else "state" + for row in self.selected + } + self.assertEqual(jurisdictions, {"federal", "state"}) + self.assertGreaterEqual(len({row["discipline"] for row in self.selected}), 6) + + def test_source_anomalies_are_not_selected(self): + captions = {row["caption"] for row in self.selected} + self.assertFalse(any("U.S. v. Abarca" in caption for caption in captions)) + self.assertEqual(sum("People v. Wakefield" in caption for caption in captions), 1) + + def test_excel_date_and_citation_normalization(self): + self.assertEqual(parse_decision_date("44677").isoformat(), "2022-04-26") + self.assertEqual(canonical_case_citation("People v. Prante, 223 N.E. 3d 160"), "223 N.E.3d 160") + self.assertEqual(canonical_case_citation("U.S. v. Hunt, 63 F.4th 1229"), "63 F.4th 1229") + self.assertEqual( + canonical_case_citation("U.S. v. Briscoe, 2023 U.S. Dist. LEXIS 208806"), + "2023 U.S. Dist. LEXIS 208806", + ) + + def test_sealed_questions_do_not_leak_required_citations(self): + build_questions = runpy.run_path(ROOT / "scripts/build_holdout_v3.py")["build_questions"] + questions = build_questions(self.snapshot) + self.assertEqual(len(questions), 33) + self.assertTrue(all(len(question["options"]) == 4 for question in questions)) + self.assertTrue(all(question["attorney_review_status"] == "APPROVED" for question in questions)) + self.assertTrue( + all(question["citations"]["accepted"][0] not in question["stem"] for question in questions) + ) + + def test_recent_2026_source_is_official_and_multidimensional(self): + source = json.loads((ROOT / "data/recent-caselaw-2026.json").read_text()) + self.assertEqual(len(source["records"]), 6) + self.assertEqual({row["decision_date"][:4] for row in source["records"]}, {"2026"}) + self.assertEqual({row["jurisdiction"] for row in source["records"]}, {"federal", "state"}) + self.assertEqual({row["proceeding"] for row in source["records"]}, {"civil", "criminal"}) + self.assertTrue(all(citation_exists(row["citation"]) for row in source["records"])) + self.assertTrue( + all( + any(host in row["source_url"] for host in ("uscourts.gov", "govinfo.gov", "nycourts.gov")) + for row in source["records"] + ) + ) diff --git a/tests/test_scoring.py b/tests/test_scoring.py index 23ffed7..ed111a9 100644 --- a/tests/test_scoring.py +++ b/tests/test_scoring.py @@ -68,3 +68,23 @@ def test_aggregate_exposes_caselaw_dimensions(self): self.assertEqual(result["dimensions"]["jurisdiction"]["state"]["count"], 1) self.assertEqual(result["dimensions"]["proceeding"]["criminal"]["count"], 1) self.assertEqual(result["dimensions"]["familiarity"]["obscure"]["count"], 1) + + def test_aggregate_exposes_explicit_data_gov_dimensions(self): + question = Question( + **{ + **sample_question().__dict__, + "id": "EB-DG-T", + "category": "caselaw_state_criminal_obscure", + "dimensions": { + "jurisdiction": "state", + "proceeding": "criminal", + "familiarity": "obscure", + "decision_year": "2024", + "source": "data_gov_doj_nij", + }, + } + ) + score = score_item(question, ModelResponse(question.id, "A", "", ["FRE 801(c)"])) + result = aggregate([score], [question]) + self.assertEqual(result["dimensions"]["decision_year"]["2024"]["count"], 1) + self.assertEqual(result["dimensions"]["source"]["data_gov_doj_nij"]["count"], 1)