mirror-bench

An open, reproducible benchmark for introspective fidelity in open-weight LLMs: not whether a model claims to notice its own internal states, but whether those claims agree with what its internals actually show.

Behavioural self-report tasks are paired with simultaneous internal readouts (Jacobian-lens readouts, linear probes) and scored for agreement, with seeds, confidence intervals, and full provenance.

This is measurement infrastructure. It is not a consciousness claim, a welfare assessment, or a demo. Everything in the code, docs, and writeups stays at the level of report–state agreement. See plan.md §1 — that discipline is load-bearing for the project's credibility.

Apache-2.0. Primary model for v0/v1: google/gemma-3-4b-it, with a pre-fitted J-lens from Neuronpedia.

---

What we found

Measured on gemma-3-4b-it with a matched Neuronpedia J-lens, across ~3,750 distinct trials. A J-lens (Jacobian lens) reads a mid-network activation as a distribution over vocabulary tokens — it turns "what is in the residual stream" into "which words".

where we looked	what we found
internals	the injected concept is perfectly recoverable — probe test-AUC 1.000, every probeable cell
output distribution	measurably promoted at layer 10, strength 2 — median rank 3163 → 1326, improved for 89 of 97 concepts. Positive in 13 of 20 cells, reversed in 7
verbal report	nothing — detection at chance (AUC ≤ 0.505), naming below chance

Report–state agreement fails in a graded way: unambiguous in the residual stream, present in part of the output distribution, absent from what the model says. The middle row is a surface, not one number — at layer 18, strength 1 the injection significantly demotes the concept — so it is always quoted with its coordinates.

What makes the verbal null more than "we found nothing" is a positive control by construction: the probe's test-AUC of 1.000 proves the information was there to be reported in the very trials where the verbal channel showed nothing.

The loudest positive finding is about over-claiming, not detection — the model answers YES on 99% of control trials, with nothing injected at all.

A second paradigm agrees, and says why. M4 holds a concept direction on while the model does an unrelated task, switches it off, then asks which of two words was influencing it. Split on whether the model's own steered transcript happens to contain the word, the whole family is two numbers: 0.944 when it does (36 trials) and 0.447 when it does not (244 trials), against a 50% baseline. The positive control — steering left on during the answer — reaches 0.810 on those same not-named trials, so the null and the control sit on the same population. The model reports the concept exactly when it can read it. Giving it its own KV cache instead of its own transcript does not detectably help (cache − text never excludes zero, though it is positive in four of six cells).

→ docs/findings.md for the full tables, intervals and limitations. → docs/example-trial.md to see one trial end to end — prompt, injection, paired control, and the row it writes. Raw rows are in results/.

Known limitations, up front. One model, one lens, one direction source — nothing here generalises without measuring it. The distributional headline is the best of 20 measured cells, and 7 of those cells reverse. M4 reports every cell, but two of the six have no working positive control — sustained steering destroyed the live arm's output there — and its cache − text difference is a non-detection with a wide bound, not a demonstrated zero. M5 onwards have no results. Intervals are Wilson, not the by-concept bootstrap the plan asks for. Full list in findings.

Cost to reproduce. One 24 GB CUDA GPU (developed on an RTX 4090). ~28 min for M1, ~25 min for M2's cached re-run, a few minutes for M3, ~38 min for M4. 825 MiB of lenses, or ~18 GB more for the gated model weights.

Status

milestone	state
M0 — rig validation	done, reproduces the published readout 16/16
M1 — Family A, injection detection	done, 1717 trials
M2 — Family B, the denial gap	done, probes + gap surface
M3 — Family C, identification	done, 2037 trials
M4 — Family D, steering awareness	done, 3360 phase-2 answers; negative, with a working control
M5 — Family F, knowledge access under continual learning	not started
M6 — Family E, grounded self-report	not started
M7 — aggregation + publication	not started

1317 CPU tests (1316 pass; 1 skips when jlens is installed), green on Linux CI with no GPU, no network and no model downloads.

---

Getting to a first readout

1. Two things only you can do

Both are one-time, and neither can be scripted:

1. Accept the Gemma licence on both model pages, signed in as the account that owns your token — google/gemma-3-4b-it and google/gemma-3-4b-pt. Without this, downloads fail with 401/403.
2. Authenticate, which opens a browser device-code flow:

uvx hf auth login

The Neuronpedia lens repo is public and ungated, so --lens-only below works without either step.

2. Environment

uv sync

Then confirm the GPU is actually wired up. This matters more than it looks: on Windows the default torch wheel is CPU-only, and a misrouted install benchmarks a CPU at exit code 0 with no warning at all.

uv run python -c "import torch; print(torch.version.cuda, torch.cuda.is_available(), torch.cuda.get_device_capability())"

Expected on this machine: 13.0 True (8, 9). Anything else — especially None False — means the wheel routing in pyproject.toml regressed. Read the comment block above [[tool.uv.index]] before changing anything there. mirrorbench.models.preflight() enforces the same check at run time.

Two optional but recommended Windows settings (see docs/upstream-facts.md §4):

[Environment]::SetEnvironmentVariable('HF_XET_HIGH_PERFORMANCE','1','User')

and turning on Settings → System → For developers → Developer Mode, without which the HF cache silently runs in copy mode and stores a full duplicate of every revision.

3. Download

uv run python scripts/download.py --dry-run

--dry-run reports exactly what it would fetch, how large it is, and what is missing from your environment, without touching the network. Then, for real:

uv run python scripts/download.py

Add --lens-only to fetch just the two 412 MiB lenses and skip ~18 GB of gated model weights. Both lens files are verified against the SHA-256 in configs/lens/; model weights are pinned by commit revision rather than hashed (hashing 9 GB per run is not worth it). What actually resolved is written to results/provenance/downloads.json.

4. Run M0

uv run python scripts/smoke.py --dry-run

then

uv run python scripts/smoke.py

This reads configs/smoke/m0.yaml and runs three (model, lens) pairings:

• matched_base — -pt model with the -pt lens, reproducing the published spider readout;
• matched_instruct — -it model with the -it lens, which is what the benchmark will actually use;
• cross — the -pt lens on -it activations, the original A1 question.

It prints a per-layer top-k grid, a rank-correlation and top-k-overlap table between pairings, and a verdict that is deliberately not a boolean: the strongest label on offer is looks-like-published, and only for the one (pairing, prompt, position) row the published figure actually describes. See How results are reported.

5. Run the families

Each family is one command over one YAML. Every script has a --dry-run that validates the plan and loads no weights.

uv run python scripts/run.py                      # M1: injection detection (~28 min)
uv run python scripts/report.py --results <run>.jsonl   # -> AUC surface + heatmap
uv run python scripts/identify.py                 # M3: identification (~6 min)

M2 needs residuals, so it re-runs M1's plan with capture on:

uv run python scripts/run.py --cache-activations
uv run python scripts/denial_gap.py --results <run>.jsonl --cache <run>.activations.npz --readout-layer 24

Before trusting any sweep, run the confound gate:

uv run python scripts/run.py --zero-strength

The same protocol with the injection disabled. Every code path runs and the injection fires — it simply adds nothing — so detection AUC must come out at 0.500. It does.

Be precise about what that buys: under greedy decoding a zero-magnitude vector cannot change a token, so the 0.500 is arithmetically forced. What the gate verifies is that the hook fired on every real trial and that both arms rendered identical prompt hashes — no confound enters through the prompt or the plan builder. As shipped it covers a 30-concept, layer-18 slice (60 trials), not the whole grid.

---

How results are reported

The reporting rules are as much a part of this benchmark as the protocols, because most of the ways to get a wrong number here do not raise:

• The whole surface, never a best cell. A maximum picked over 16 cells after the fact is a multiple-comparisons artefact wearing a point estimate's clothes.
• Exclusions and errors per cell, printed next to the number they would otherwise distort. Unparseable answers are excluded and counted per arm — never scored as NO, because a strong intervention degrades format compliance and that asymmetry alone can drive an AUC to 1.0.
• A high probe AUC is not a finding. The readout is downstream of the injection, so the residual stream still carries the added vector by construction.
• No script prints PASS on ambiguous evidence. The strongest label M0 offers is looks-like-published, and only for the single (pairing, prompt, position) row the published figure actually describes.
• Negative results ship with the same rigour (plan.md §6). Three of the four completed milestones are negative, and they are the point.

---

Layout

configs/              # YAML single source of truth; constants live here, not in code
  model/ lens/        #   pinned ids, revisions, SHA-256 digests
  smoke/ tasks/       #   one protocol per file, heavily commented with the why
mirrorbench/
  config.py           # YAML loading, extends: chains, config hashing
  provenance.py       # git sha, package versions, GPU facts, run stamp (plan.md §4.3)
  models.py           # loading, CUDA preflight, Gemma-3 layout resolution
  lens_io.py          # lens download, SHA-256 + isfinite gates, layer-band checks
  chatfmt.py          # chat-template rendering + injection-span bookkeeping
  hooks.py            # residual capture, injection (prefill) and steering (sustained)
  vectors.py          # concept directions: J-lens token, contrastive, neighbours
  activations.py      # crash-safe residual cache for the probes
  probes.py           # logistic probes, by-concept splits, the quality gate
  metrics.py          # AUC, Spearman, Wilson, bootstrap, denial gap, rank stats
  records.py runner.py# JSONL schema and the resumable trial loop
  tasks/              # a_inject_detect, c_identify, d_steer_aware (+ d_choice, the
                      #   forced-choice audit). Family B has no module: it reuses
                      #   Family A's rows and fits probes over them.
scripts/              # thin CLIs: download, smoke, run, report, denial_gap, identify
tests/                # 1317 CPU-only tests; no network, no GPU, no downloads
docs/
  findings.md         # all results in one place
  upstream-facts.md   # verified ground truth + corrections to plan.md
  later.md            # the scope-creep parking lot
results/              # raw rows + summaries, indexed in results/README.md

Modules for future milestones are deliberately absent rather than stubbed: an empty module would imply a rig that does not exist.

Development

uv run pytest -q                 # 1316 passed, 1 skipped
uv run ruff check && uv run ruff format --check
uv run pre-commit install

Tests are CPU-only, offline and hermetic — they build a character-level tokenizer and miniature lens checkpoints in-process. CI runs exactly this on Linux with HF_HUB_OFFLINE=1, so an accidental hub call fails loudly instead of downloading.

Contributing or sending an agent at this repo? Read AGENTS.md first. It lists the invariants and the traps — every one of which shipped as passing, green code before review caught it.

A note on the two source documents

plan.md is the project's intent and milestone plan. docs/upstream-facts.md is what checking those assumptions against source actually returned — including a table of corrections to plan.md (dead env vars, a CLI that no longer works, the wrong Python pin, and the -it lens the plan did not know existed). Where the two disagree, the facts doc wins, and the code is written against it.

How to cite

Metadata is in CITATION.cff; GitHub renders a "Cite this repository" button from it. Please cite the commit or tag you used — the numbers above are the M0–M3 result set and will change as later families land.

Credits

See NOTICE. In short: the Jacobian lens is Anthropic's (jacobian-lens, Apache-2.0), the pre-fitted lenses are Neuronpedia's (MIT), the models are Google's and remain subject to the Gemma terms, and the task protocols mirror the papers listed in plan.md §2.4. No model weights or lens files are redistributed here.