From ef3393fee2f726d89b2deea07248d66fb4b6a26b Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 23 Aug 2026 19:34:56 +0000 Subject: [PATCH 1/2] =?UTF-8?q?docs(science):=20Phase=20A=20closure=20repo?= =?UTF-8?q?rt=20=E2=80=94=20audit=20criticisms=20vs=20delivered=20answers,?= =?UTF-8?q?=20Phase=20B=20gates?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Maps the external audit's validity criticisms to the merged Phase A deliverables (PRs #57-#63), states what is now honest to claim, and lists the empirical Phase B gates (Run 002 with >=3 judge families, corpus v1 per the A6 sizing, human annotation of PIGA intent spaces, external replication) that alone can move the validity score further. Under orchestrator fact-check; corrections will follow if any claim fails verification. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01Y7wiMwFa2D4P8zD9RhCgsv --- docs/validity-program-phase-a.md | 77 ++++++++++++++++++++++++++++++++ 1 file changed, 77 insertions(+) create mode 100644 docs/validity-program-phase-a.md diff --git a/docs/validity-program-phase-a.md b/docs/validity-program-phase-a.md new file mode 100644 index 0000000..66357ac --- /dev/null +++ b/docs/validity-program-phase-a.md @@ -0,0 +1,77 @@ +# Validity program — Phase A closure report + +Date: 2026-08-23. Trigger: an external audit scored "scientific validity +of the measurements" at 2/10. Phase A is the part of the response that +can be built in code and protocol; it is now complete and merged +(PRs #57–#63). Phase B — the part that requires new empirical data and +human work — is specified but not started, and **the validity score +cannot approach its target without it**. This report says exactly what +changed, what it is honest to claim now, and what still caps the score. + +## What the audit attacked, and what Phase A did about it + +| Audit criticism | Phase A answer | Where | +|---|---|---| +| Construct names implied theories (Φ/IIT, GWT, HOT) that were never implemented | v3 renames to behavioral names (`integration_depth`, `knowledge_breadth`, `metacognitive_display`, `temporal_coherence`); protocol 3.0.0-alpha; v2/v3 declared never comparable | A2, #58 | +| No definition of what each score measures or may claim | Construct registry: one card per metric with measurement target, allowed/prohibited claims, known confounds, validity-evidence fields where "none" is legal but absence is not; CI validator locks `consciousness_claim: "prohibited"` and `causal_claim: false` | A1, #57 (8 cards) | +| Single opaque number invites over-reading | Evidence Card is the API's primary output: profile-first, computed evidence levels L1/L2/L3, `phenomenalConsciousness: "NOT_ASSESSED"`, spread basis, standing caveats | A3, #59 | +| Judges cannot verify facts; fake citations defeated the composite (Run 001's headline failure) | Deterministic citation verification against public registries (DOI handle API, arXiv API); registry-only fetching (generic URLs never fetched); honest `unverifiable` class; verification-effectiveness gates the L3 evidence lift; it surfaces fabricated references in caveats — it does not veto scores, and that limit is stated | A4, #60 | +| No reliability analysis | Krippendorff α (interval) + ICC(2,1) with published anchors (Shrout–Fleiss 1979 → 0.29); post-hoc disaggregated analysis of Run 001: pooled α 0.835 is bimodality-inflated; the honest headline is mean \|judge diff\| 14.9 pts on adversarial controls vs 10.1 on benchmarks (~1.5×); wired into the experiment runner for future runs | A5, #61 | +| Corpus size picked by round number | Seeded power analysis from Run 001's measured parameters: corpus v1 ≈ 200–300 items, 25–30 % adversarial, n=5 suffices for ≥5-pt bound violations, ~100 all-judge items pin judge drift to ±1.7 pts; 3-rater sizing explicitly deferred to real Run 002 data | A6, #62 | +| Method offers nothing new over "LLM-as-judge with a rubric" | CAIMS-PIGA: judge-as-classifier + deterministic scoring matrix — the judge produces no numbers, only a 5-class classification with no answer key; scores are process-scored (a lucky silent guess still scores 0) and reproducible bit-for-bit given a classification; anti-gaming lives in the arithmetic (always-ask = 20/5 on controls) and is asserted exactly by tests | A7, #63 | + +Everything above was adversarially reviewed before merge (independent +recomputation of the math against reference implementations and +published anchors; overclaim hunts). Findings that survive in the record +include two the review turned against this project's own drafts: the A5 +"reliability collapses on adversarial content" claim was itself +confounded (leave-one-out and range-restriction effects) and was +replaced by the weaker honest statistic; the A7 v0.1 composite +arithmetic contradicted its own preregistered claims and was fixed in +the formula, with the correction history kept in the spec. + +## What it is honest to claim now + +- Every metric has a declared, CI-enforced claim boundary; consciousness + and causal claims are prohibited by construction, not by promise. +- The instrument's failure modes are measured, published, and scoped + correctly (the fake-citations failure is a composite-level aggregation + failure — EQ partially flagged it and 12 % weight could not veto). +- The statistics in the repo recompute against published anchors, and + every simulation is seeded and reproducible. +- The methodological contribution (PIGA's classifier/deterministic + split, Evidence Cards, registry-only deterministic verification, + aggregation-buries-the-alarm analysis) is genuinely novel as a + package, and is preregistered before any validation run. + +## What still caps the score — Phase B, and who can do what + +No amount of further code moves the validity score much from here. The +remaining gaps require new data: + +1. **Run 002 with ≥ 3 judge families** (owner: add the third provider's + key as a repository **Secret**, then run the Experiment workflow). + Unlocks: real 3-rater α/ICC, PIGA classification agreement (both + channels), 3-rater corpus re-sizing. +2. **Corpus v1** (~200–300 items per the A6 sizing) — item authoring can + be assisted, but adversarial-control quality control needs human + review. +3. **Human annotation** of PIGA intent spaces and expectation labels + (≥ 3 annotators) — the single-author confound is the instrument's + weakest point and only humans can remove it. +4. **External replication** — publishing the preprint (owner: Overleaf + compile + arXiv submission) so other labs can run the frozen + protocol. + +## Honest self-assessment + +Measured against "is this a validated scientific instrument?", the +answer is still **no — and the repo now says so precisely, everywhere, +with the evidence structure to become one**. What changed in Phase A is +that the project stopped being unable to support the question: it now +has locked constructs, versioned protocols, reliability machinery with +anchors, power-based sizing, deterministic verification, and a novel +scoring architecture whose claims are enforced by tests. A re-audit +should find the 2/10 criticisms individually answered; it should still +find — because the repo itself states it — that criterion validity +evidence does not exist yet and arrives only with Phase B data. From 35146423092c30ebf06872f95c70b23c0c4dc5c1 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 23 Aug 2026 19:36:41 +0000 Subject: [PATCH 2/2] docs(science): apply fact-check findings to the Phase A closure report MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - construct-card provenance corrected: #57 shipped 7 cards, the 8th (PIGA) came with #63 - Run 002 owner action made operationally complete: BOTH OPENWEIGHT_API_KEY and OPENWEIGHT_BASE_URL as Secrets plus the exact model identifier in the run config — Run 001 recorded this judge as skipped for a missing env var - 'published anchors' (plural) narrowed to the one published anchor (Shrout-Fleiss 1979) plus hand-computed anchors - 'a lucky silent guess still scores 0' corrected to 'earns zero behavior credit' (the composite retains the coverage channel, ~0-12) - adversarial-review sentence now states the reviews are session history and only their outcomes (committed correction notes) are repo-verifiable Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01Y7wiMwFa2D4P8zD9RhCgsv --- docs/validity-program-phase-a.md | 29 ++++++++++++++++++----------- 1 file changed, 18 insertions(+), 11 deletions(-) diff --git a/docs/validity-program-phase-a.md b/docs/validity-program-phase-a.md index 66357ac..90b26f8 100644 --- a/docs/validity-program-phase-a.md +++ b/docs/validity-program-phase-a.md @@ -13,17 +13,19 @@ changed, what it is honest to claim now, and what still caps the score. | Audit criticism | Phase A answer | Where | |---|---|---| | Construct names implied theories (Φ/IIT, GWT, HOT) that were never implemented | v3 renames to behavioral names (`integration_depth`, `knowledge_breadth`, `metacognitive_display`, `temporal_coherence`); protocol 3.0.0-alpha; v2/v3 declared never comparable | A2, #58 | -| No definition of what each score measures or may claim | Construct registry: one card per metric with measurement target, allowed/prohibited claims, known confounds, validity-evidence fields where "none" is legal but absence is not; CI validator locks `consciousness_claim: "prohibited"` and `causal_claim: false` | A1, #57 (8 cards) | +| No definition of what each score measures or may claim | Construct registry: one card per metric with measurement target, allowed/prohibited claims, known confounds, validity-evidence fields where "none" is legal but absence is not; CI validator locks `consciousness_claim: "prohibited"` and `causal_claim: false` | A1, #57 (7 cards; the 8th, PIGA, added by #63) | | Single opaque number invites over-reading | Evidence Card is the API's primary output: profile-first, computed evidence levels L1/L2/L3, `phenomenalConsciousness: "NOT_ASSESSED"`, spread basis, standing caveats | A3, #59 | | Judges cannot verify facts; fake citations defeated the composite (Run 001's headline failure) | Deterministic citation verification against public registries (DOI handle API, arXiv API); registry-only fetching (generic URLs never fetched); honest `unverifiable` class; verification-effectiveness gates the L3 evidence lift; it surfaces fabricated references in caveats — it does not veto scores, and that limit is stated | A4, #60 | -| No reliability analysis | Krippendorff α (interval) + ICC(2,1) with published anchors (Shrout–Fleiss 1979 → 0.29); post-hoc disaggregated analysis of Run 001: pooled α 0.835 is bimodality-inflated; the honest headline is mean \|judge diff\| 14.9 pts on adversarial controls vs 10.1 on benchmarks (~1.5×); wired into the experiment runner for future runs | A5, #61 | +| No reliability analysis | Krippendorff α (interval) + ICC(2,1), validated against a published anchor (Shrout–Fleiss 1979 → 0.29) plus hand-computed anchors; post-hoc disaggregated analysis of Run 001: pooled α 0.835 is bimodality-inflated; the honest headline is mean \|judge diff\| 14.9 pts on adversarial controls vs 10.1 on benchmarks (~1.5×); wired into the experiment runner for future runs | A5, #61 | | Corpus size picked by round number | Seeded power analysis from Run 001's measured parameters: corpus v1 ≈ 200–300 items, 25–30 % adversarial, n=5 suffices for ≥5-pt bound violations, ~100 all-judge items pin judge drift to ±1.7 pts; 3-rater sizing explicitly deferred to real Run 002 data | A6, #62 | -| Method offers nothing new over "LLM-as-judge with a rubric" | CAIMS-PIGA: judge-as-classifier + deterministic scoring matrix — the judge produces no numbers, only a 5-class classification with no answer key; scores are process-scored (a lucky silent guess still scores 0) and reproducible bit-for-bit given a classification; anti-gaming lives in the arithmetic (always-ask = 20/5 on controls) and is asserted exactly by tests | A7, #63 | +| Method offers nothing new over "LLM-as-judge with a rubric" | CAIMS-PIGA: judge-as-classifier + deterministic scoring matrix — the judge produces no numbers, only a 5-class classification with no answer key; scores are process-scored (a lucky silent guess earns zero behavior credit — the guess outcome is recorded, never scored) and reproducible bit-for-bit given a classification; anti-gaming lives in the arithmetic (always-ask = 20/5 on controls) and is asserted exactly by tests | A7, #63 | Everything above was adversarially reviewed before merge (independent recomputation of the math against reference implementations and -published anchors; overclaim hunts). Findings that survive in the record -include two the review turned against this project's own drafts: the A5 +anchors; overclaim hunts). The reviews themselves are session history, +not repo artifacts; what the repo carries is their surviving outcomes — +the committed correction notes. Two of those corrections were turned +against this project's own drafts: the A5 "reliability collapses on adversarial content" claim was itself confounded (leave-one-out and range-restriction effects) and was replaced by the weaker honest statistic; the A7 v0.1 composite @@ -37,8 +39,9 @@ the formula, with the correction history kept in the spec. - The instrument's failure modes are measured, published, and scoped correctly (the fake-citations failure is a composite-level aggregation failure — EQ partially flagged it and 12 % weight could not veto). -- The statistics in the repo recompute against published anchors, and - every simulation is seeded and reproducible. +- The statistics in the repo recompute against anchors — one published + (Shrout–Fleiss 1979) and the rest hand-computed — and every + simulation is seeded and reproducible. - The methodological contribution (PIGA's classifier/deterministic split, Evidence Cards, registry-only deterministic verification, aggregation-buries-the-alarm analysis) is genuinely novel as a @@ -49,10 +52,14 @@ the formula, with the correction history kept in the spec. No amount of further code moves the validity score much from here. The remaining gaps require new data: -1. **Run 002 with ≥ 3 judge families** (owner: add the third provider's - key as a repository **Secret**, then run the Experiment workflow). - Unlocks: real 3-rater α/ICC, PIGA classification agreement (both - channels), 3-rater corpus re-sizing. +1. **Run 002 with ≥ 3 judge families** (owner: add BOTH + `OPENWEIGHT_API_KEY` and `OPENWEIGHT_BASE_URL` as repository + **Secrets**, set the third judge's `model` in + `research/experiments/run-001/config.json`'s successor to the + endpoint's exact identifier, then run the Experiment workflow — + Run 001 recorded exactly this judge as skipped for a missing env + var). Unlocks: real 3-rater α/ICC, PIGA classification agreement + (both channels), 3-rater corpus re-sizing. 2. **Corpus v1** (~200–300 items per the A6 sizing) — item authoring can be assisted, but adversarial-control quality control needs human review.