feat: a meta-benchmark over the externally produced benchmarks 0.15.0 used

[raeq]

Summary

The 0.15.0 cycle found and verified defects against roughly thirty artifacts produced
outside this repository. Each measurement lived in its own script, recorded as a gist on
the issue it produced, and nothing re-ran any of them. #732 states the consequence
directly: one public corpus produced five findings and nothing in the repo will notice
when any of them moves.

benchmarks/meta consolidates the selection, the provenance record, the scoring protocol
and the report into one harness that runs n of m.

python -m benchmarks.meta --list                    # show m, and what is runnable
python -m benchmarks.meta --run --only-available    # run what the machine has
python -m benchmarks.meta --run --select 'uts39-*' --report out.md
python -m benchmarks.meta --run --sample 5 --seed 3 # a reproducible partial pass

35 benchmarks across six external tiers, plus a seventh that is marked as not external.

tier	anchor	suites
normative	a table that defines the right answer	UTS #39 confusables + §5.1 + §5.3 + equivalence classes, UCD Scripts.txt, UAX #9, UAX #29, DerivedCoreProperties, ICANN LGR, CLDR
cve	a published vulnerability's own vector	CVE-2026-17084 stringprep B.3 delta, CVE-2021-42574 PoC files
academic	a corpus released with a paper	Bad Characters, arXiv:2405.14490, special-char-attack, GCG suffixes, GAversary, smishing, XOXO, ESTI, backdoor triggers, canonicalization obligation
dataset	a public corpus	BitAbuse, YouTube-Spam, TREC-2007, MeAJOR, benign Gutenberg prose
model	a released model artifact	chat-template delimiters read from published tokenizer.json files
comparator	a third-party labelled benchmark or rival tool	confusable-bench.v1, confusable-vision, untrace

Why every benchmark is somebody else's

A library that writes its own benchmark grades its own homework. The harness supplies the
runner, the selection, the provenance record and the report. It supplies no vectors.

That boundary is checked rather than promised. Provenance.external is a field, the
runner tags every outcome with it, the report renders external and introspective results
in separate sections, and a test asserts that no external suite is scored against a file
disarm generated.

The last one is not hypothetical. data/confusables_lgr.tsv looks like an ideal local
oracle for the ICANN suite — an extract of ICANN's LGR, already in the repo, no network
needed. It is also the file the shipped fold was built from, so scoring the fold against
it returns 100% by construction. It read exactly that way before the fallback was removed.
The suite now requires ICANN's published LGR or reports nothing.

The #39/#40 guardrail carries over unchanged: these corpora are measuring instruments,
never optimization targets.

The introspective tier

Four sweeps reproduce real findings (#719, #723/#751, #805/#806/#807, #834) and are not
benchmarks. The distinction is not where the input comes from — the code-point domain is
the UCD — but whether an outside authority decides the right answer. For these, disarm is
both the thing measured and the only oracle, so a number moving proves nothing on its own.

They are registered so the 0.15.0 record has no hole, marked external=False, excluded
unless --include-introspective is passed, and never folded into an external total.

What a run reports

Three things per suite, kept apart on purpose:

• Found during the cycle — historical, quoted from the issue, never edited to match a
  fresh run.
• How it is measured — methodology, true over time.
• Measured now — this run.

The gap between the first and the third is the most useful column, and several already
show one. UTS #39 §5.3 was unimplemented when #777 was filed and now reports 75 of 75
numbering systems. has_bidi_conflict was neutral to 1,786 of 3,018 strong-RTL code
points under #773 and now reaches all 3,018.

Skips, and drift

A suite that cannot find its artifact returns SKIPPED with the variable to set, and the
report lists every one. An absent corpus is not a passing corpus.

Most academic corpora are deliberately not vendored: copying an attack corpus into this
repository would make it disarm's corpus, and a corpus disarm owns is a corpus disarm can
be tuned against.

Drift is reported and never gated — a moved number never fails a run; only a suite that
threw does, because that is a harness defect rather than a result. Baselines are keyed by
suite and population, because a ratio over 4,000 code points and one over 150,000 are
not comparable; the report marks such a row rather than subtracting it. --limit samples
by stride rather than truncating, since the first N code points of any sorted domain are
Latin, Greek and Cyrillic — the best-covered part of every table here.

Also fixed

Both found by the harness against its own output:

• ucd-scripts counted unassigned code points in its denominators, so agree + contradict
  • silent did not sum to the population.
• benchmarks/adversarial_eval/corpora.py typed _guess_text_column as taking
  list[str] while both call sites pass csv.DictReader.fieldnames, a Sequence[str].

Verification

• pytest tests/test_meta_benchmark.py tests/test_adversarial_eval.py — 37 passed
• ruff check . / ruff format --check . — clean
• mypy benchmarks/meta --ignore-missing-imports — clean (16 files)
• python -m benchmarks.meta --run --only-available --include-introspective — 14 suites
  ran, 0 errors, exit 0; introspective sweeps over all 294,579 assigned code points
• Drift verified in both directions: no rows against the committed baseline, and a
  perturbed baseline produces the expected row with the right direction
• No Rust changed, so the Rust and bindings gates are not implicated

Refs #732, #736, #759

🤖 Generated with Claude Code

[Copilot]

🟡 Changes recommended

There are a few correctness and reliability issues (notably deterministic skip-path tests and a couple of reporting/suite logic problems) that should be addressed before approval.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds a new Python “meta-benchmark” harness (benchmarks/meta) that consolidates the externally produced benchmarks used during the 0.15.0 cycle into a single registry-driven runner with provenance, skip semantics, reporting (Markdown/JSON), and drift comparison against a committed baseline.

Changes:

• Introduces benchmarks/meta CLI + library: suite registry/selection (n of m), runner, reporting, and baseline drift comparison.
• Adds a broad suite catalog across normative tables, CVEs, academic corpora, datasets, model artifacts, comparators, plus an introspective tier excluded by default.
• Adds tests for registry invariants, selection behavior, skip/error handling, baseline serialization, and report rendering; plus a typing fix in benchmarks/adversarial_eval.

File summaries

File	Description
benchmarks/meta/__init__.py	Public package surface and entry-point exports for the meta-benchmark.
benchmarks/meta/__main__.py	CLI for listing/selecting/running suites and emitting reports/baseline updates.
benchmarks/meta/base.py	Shared suite helpers: artifact location, stride-based thinning, surface/detector enumeration, SuiteBase behavior.
benchmarks/meta/baseline.py	Baseline snapshotting, persistence, and drift comparison logic.
benchmarks/meta/baselines/default.json	Initial committed baseline snapshot (keyed by suite + population).
benchmarks/meta/protocol.py	Core protocol/enums and dataclasses (provenance, outcomes, measurements, suite interface).
benchmarks/meta/registry.py	Registry assembly and deterministic filter/sample selection (“n of m”).
benchmarks/meta/report.py	Markdown/JSON rendering, including family grouping and drift/skips sections.
benchmarks/meta/runner.py	Run orchestration with per-suite isolation and version stamping.
benchmarks/meta/README.md	Usage and rationale documentation for the meta-benchmark harness.
benchmarks/meta/suites/__init__.py	Suite module aggregation and exports.
benchmarks/meta/suites/normative.py	Normative-table suites (UTS/UAX/UCD/ICANN/CLDR-derived measurements).
benchmarks/meta/suites/cve.py	CVE-anchored suites (stringprep delta; Trojan Source PoC files).
benchmarks/meta/suites/academic.py	Academic corpus adapter suites (manual artifact placement; detection + recovery scoring).
benchmarks/meta/suites/datasets.py	Public dataset suites; reuses benchmarks.adversarial_eval adapters for consistency.
benchmarks/meta/suites/model_artifacts.py	Model-artifact suite parsing tokenizer JSONs to derive delimiter probes.
benchmarks/meta/suites/comparators.py	Comparator suites (labelled benchmarks/rival tools) including precision/recall scoring.
benchmarks/meta/suites/introspective.py	Self-referential sweeps registered as non-external and excluded by default.
tests/test_meta_benchmark.py	Test coverage for suite invariants, selection/thinning, runner behavior, baseline, and reporting.
benchmarks/adversarial_eval/corpora.py	Tightens _guess_text_column typing to match actual call sites (Sequence[str]).
CHANGELOG.md	Documents the new meta-benchmark harness and its core guarantees/semantics.

Review details

Suppressed comments (2)

benchmarks/meta/suites/model_artifacts.py:107

• manufactured_any is incremented based on per_surface_manufacture (a cumulative dict across all tokens), so it would overcount after the first manufactured delimiter. Since it’s currently unused, it’s best to remove this block to avoid a latent logic bug.

            if survived_here == len(surface_map):
                survives_all += 1
            if any(per_surface_manufacture.values()):
                manufactured_any += 1

tests/test_meta_benchmark.py:289

• Like test_a_missing_artifact_skips..., this relies on the local machine having at least one missing registry artifact. If a developer has all artifacts in cache, the test won’t exercise the skipped path and can fail due to absent[:3] being empty. Use a deterministic missing-artifact suite here too so the markdown “Not run” section is always exercised.

def test_markdown_names_every_skipped_suite_and_why():
    absent = [s for s in registry.all_suites() if not s.available()[0]]
    report = run(absent[:3], registered=len(registry.all_suites()))
    md = render_markdown(report)

• Files reviewed: 21/21 changed files
• Comments generated: 3
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.