EvidenceBench v4 research release

.github/ISSUE_TEMPLATE/evidencebench-v4-reviewer.yml

@@ -0,0 +1,69 @@
+name: EvidenceBench v4 professional reviewer
+description: Volunteer to independently review one or more sealed EvidenceBench v4 candidates.
+title: "[Reviewer application] "
+labels:
+  - reviewer-application
+body:
+  - type: markdown
+    attributes:
+      value: |
+        Thank you for volunteering. EvidenceBench v4 needs identity-verified legal professionals to review sealed evidence-law items before release.
+
+        **Do not post bar numbers, identification documents, client information, or other sensitive personal data in this public issue.** Objection Academy will arrange private verification and assignment after initial triage.
+  - type: input
+    id: professional_role
+    attributes:
+      label: Professional role
+      description: For example, litigation attorney, judge, law professor, appellate lawyer, or evidence instructor.
+      placeholder: Litigation attorney
+    validations:
+      required: true
+  - type: input
+    id: jurisdictions
+    attributes:
+      label: Jurisdictions or evidence-law experience
+      description: List relevant jurisdictions or describe your evidence-law teaching, research, or practice experience.
+      placeholder: California and federal courts; trial-practice instructor
+    validations:
+      required: true
+  - type: dropdown
+    id: capacity
+    attributes:
+      label: Review capacity
+      description: Each assignment is a private Doctrine item or Matter workspace.
+      options:
+        - 1–2 candidates
+        - 3–5 candidates
+        - 6–10 candidates
+        - More than 10 candidates
+    validations:
+      required: true
+  - type: checkboxes
+    id: tracks
+    attributes:
+      label: Preferred track
+      options:
+        - label: Doctrine — applied evidentiary rulings
+        - label: Matter — closed-universe files and work product
+  - type: textarea
+    id: background
+    attributes:
+      label: Relevant background
+      description: Briefly describe the experience that would help with this review.
+    validations:
+      required: true
+  - type: checkboxes
+    id: attestations
+    attributes:
+      label: Review commitments
+      options:
+        - label: I can keep assigned benchmark material confidential.
+          required: true
+        - label: I will disclose conflicts and will not use assigned material to train or prompt a model.
+          required: true
+        - label: I understand that identity verification will occur privately before access is granted.
+          required: true
+  - type: markdown
+    attributes:
+      value: |
+        If you prefer not to open a public issue, email **dylan@examp.com** with the subject “EvidenceBench v4 reviewer.”

.github/workflows/test.yml

@@ -15,3 +15,11 @@ jobs:
           python-version: '3.12'
       - run: uv run --python 3.12 python -m unittest discover -s tests -v
       - run: uv run --python 3.12 evidencebench validate data/dev-v1.jsonl
+      - run: uv run --python 3.12 evidencebench validate-v4 --track doctrine --input data/v4/dev/doctrine.jsonl
+      - run: uv run --python 3.12 evidencebench validate-v4 --track matter --input data/v4/dev/matter
+      - name: Confirm draft fixtures cannot be published as official
+        run: |
+          if uv run --python 3.12 evidencebench validate-v4 --track doctrine --input data/v4/dev/doctrine.jsonl --official; then
+            echo "Draft v4 data incorrectly passed the official gate"
+            exit 1
+          fi

BENCHMARK_CARD_V4.md

@@ -0,0 +1,78 @@
+# EvidenceBench v4 benchmark card
+
+## Intended use
+
+EvidenceBench v4 is a research evaluation for evidence-law reasoning and
+closed-universe legal-agent execution. Appropriate uses include model
+comparison, harness ablation, retrieval/tool experiments, error analysis, and
+targeted post-training research.
+
+It is not legal advice, a bar examination, a general legal-intelligence score,
+or evidence that a system can work without attorney review.
+
+## Unit of evaluation
+
+Doctrine units are reviewed applied-ruling fact patterns. Matter units are
+closed-universe workspaces containing instructions, documents, expected
+deliverables, and atomic criteria. `family_id` is the resampling and leakage
+boundary.
+
+## Public data
+
+`data/v4/dev` contains eight Doctrine items and two Matter tasks. All are
+synthetic and marked `DRAFT`. They exercise schemas, runners, validators, and
+scorers. They must not be described as the v4 holdout or used for an official
+leaderboard.
+
+## Private data
+
+The corpus remains under Objection Academy control in a private
+source-controlled repository. A public release commits to it by cryptographic
+hash and reports counts and coverage without releasing answers or item-level
+outputs. The July 2026 research corpus contains 240 Doctrine candidates and 60
+Matter candidates; none had completed professional review at run time.
+
+## Primary metric
+
+The single overall score is the equal-weighted mean of Doctrine and Matter.
+Both track scores and all submetrics remain mandatory, because equal headline
+scores can conceal materially different capabilities.
+
+Doctrine is family-first and then macro-averaged across 12 domains. Matter
+penalizes unsupported extra findings through precision-aware legal, authority,
+and factual F1. A separate all-critical-criteria task-resolution rate remains a
+strict secondary metric.
+
+## Quality controls
+
+- independent legal review is required for every official item;
+- authors cannot serve as their own sole reviewer;
+- exact IDs, document hashes, citation existence, and rubric consistency are
+  validated automatically;
+- DRAFT data fails the official validation command;
+- public/holdout family and exact-stem overlap is rejected;
+- headline scoring is deterministic;
+- uncertainty is clustered by task family; and
+- official results require repeated runs and disclosure of failures and costs.
+
+The July 2026 snapshot does not satisfy the independent-review or repeated-run
+controls. It is published only as an unreviewed research release so researchers
+can inspect the evaluator and initial measurement behavior while professional
+review remains open.
+
+## Foreseeable risks
+
+Contamination can inflate results. Synthetic matters may not capture the
+ambiguity of real litigation records. The pinned authority corpus can become
+stale. Rubric incompleteness can penalize a legally sound alternative.
+Provider routing can introduce unobserved model variation. Volunteer review can
+create selection bias and inconsistent depth.
+
+Mitigations include a sealed family-level holdout, annual authority freezes,
+adjudicated challenge logs, reviewer calibration, route disclosure, and
+prospective versioning rather than silent answer changes.
+
+## Maintainer
+
+Objection Academy. Security or embargo concerns should be reported privately to
+dylan@examp.com.

HARVEY_LAB_COMPARISON.md

@@ -0,0 +1,87 @@
+# Harvey LAB and EvidenceBench v4
+
+This design comparison was frozen on July 25, 2026. Harvey LAB is evolving, so
+counts are descriptive rather than permanent.
+
+| Dimension | Harvey LAB | EvidenceBench v4 |
+|---|---|---|
+| Primary target | Long-horizon legal work across many practice areas | Evidence-law doctrine and litigation-record analysis |
+| Unit | Matter files, instruction, and reviewable work product | Doctrine item or closed-universe evidence matter |
+| Breadth | Harvey's May 2026 announcement reported more than 1,200 tasks, 24 practice areas, and 75,000 expert-written criteria | Private research corpus has 240 Doctrine items and 60 Matter tasks; 8 Doctrine and 2 Matter fixtures are public |
+| Tools | File read/edit/write, glob, grep, bash, and document-format skills | List/read/search documents and write outputs; no model shell or network |
+| Core grading | Semantic LLM judge for each binary criterion; task score uses all-pass | Deterministic structured criteria; weighted track score plus all-critical Matter resolution |
+| Citation treatment | A rubric can require citations | Citations are normalized against a pinned authority corpus; invalid, nonexistent, and unsupported authorities are separated |
+| Uncertainty | Results emphasize all-pass and criterion pass rates | Family-clustered 95% bootstrap intervals, repeated-run requirement, and weight sensitivity |
+| Contamination strategy | Public tasks plus a held-out test set | Public DRAFT development set plus a private family-separated candidate corpus with a public cryptographic commitment |
+
+## Lessons adopted
+
+Matter-centric assignments are a better frontier-model test than more
+multiple-choice variants. v4 therefore requires agents to find facts across
+files and create artifacts. Criteria are atomic, attached to material legal,
+authority, factual, and delivery requirements, and critical omissions produce
+a strict task-resolution failure. The harness records transcripts and actual
+deliverables.
+
+LAB also makes a strong case for treating the harness as part of the evaluated
+system. EvidenceBench records prompt versions, tool policy, route, token and
+turn limits, failures, and costs rather than labeling a result with only a
+model name.
+
+## Deliberate differences
+
+EvidenceBench retains a Doctrine track because controlled diagnostics can
+localize whether a failure came from the legal rule, issue spotting, authority
+selection, factual grounding, or agent execution. A Matter-only score cannot
+separate these causes cleanly.
+
+The official v4 score does not depend on an LLM judge. LAB's semantic judge is
+flexible for diverse prose work product, but it adds judge-model sensitivity,
+cost, and a second failure surface. EvidenceBench narrows its outputs enough to
+score headline criteria deterministically. Human and LLM qualitative review
+may be published separately.
+
+EvidenceBench also measures citation reliability as a first-class construct.
+An unparseable citation remains in the precision denominator, and the evaluator
+distinguishes a fabricated authority from a real authority omitted from the
+annotation.
+
+## Improvements over EvidenceBench v3
+
+v3 was useful for closed-book answer and citation testing but was too narrow
+for frontier agent research. Its multiple-choice format encouraged saturation;
+its public development and mechanically varied holdout families created
+leakage risk; an explanation was requested but not scored; invalid citations
+could be normalized away; and aggregate tables lacked uncertainty.
+
+v4 addresses those problems with:
+
+- independent family IDs and a public/holdout overlap audit;
+- a private, source-controlled holdout;
+- structured issue and fact grounding;
+- a long-horizon Matter track;
+- explicit DRAFT/APPROVED review state enforced by the official validator;
+- retry-on-transport-only execution;
+- deterministic scoring and citation error accounting;
+- one preregistered overall score with mandatory track disclosure; and
+- clustered confidence intervals, run replication, and weighting sensitivity.
+
+## What remains before an official launch
+
+The candidate corpus has a frozen coverage matrix, original families, private
+provenance, review packets, and deterministic validation. It still needs
+independent professional review, adjudication, replicated model runs, external
+reproduction, and prospective challenge handling. Until those gates pass, the
+July 2026 results remain an unreviewed research release rather than an official
+leaderboard.
+
+## Sources
+
+- Harvey announcement:
+  <https://www.harvey.ai/blog/introducing-harveys-legal-agent-benchmark>
+- Harvey LAB repository:
+  <https://github.com/harveyai/harvey-labs>
+- Harvey evaluation methodology:
+  <https://github.com/harveyai/harvey-labs/blob/main/docs/eval-strategies.md>
+- Vals AI independent benchmark page:
+  <https://www.vals.ai/benchmarks/hlab>