Skip to content

feat(science): corpus sizing by seeded power analysis — Phase A6 - #62

Merged
pixelstrade-dev merged 2 commits into
mainfrom
claude/caims-consciousness-framework-2EOiI
Aug 23, 2026
Merged

feat(science): corpus sizing by seeded power analysis — Phase A6#62
pixelstrade-dev merged 2 commits into
mainfrom
claude/caims-consciousness-framework-2EOiI

Conversation

@pixelstrade-dev

Copy link
Copy Markdown
Owner

Phase A6 of the validity program — corpus sizing derived, not decreed

The growth plan inherited a "2 000–5 000 items" corpus target chosen by round number. This PR replaces it with numbers derived from committed Run 001 parameters, via seeded simulations that anyone can reproduce.

New module: lib/statistics/simulation.ts (mirrored to @caims/core)

  • mulberry32(seed) — deterministic PRNG; makeNormalSampler — Box–Muller with spare caching.
  • boundDetectionPower({sigma, delta, n, sims, seed}) — P(mean of n samples flags a true bound violation of Δ). Anchored against the analytic answer Φ(√5) ≈ 0.987.
  • bootstrapHalfWidth({observed, itemCount, stat, sims, seed}) — 95 % CI half-width of a statistic under item resampling; returns null (never a fake number) when the statistic is mostly undefined.
  • 11 tests including the analytic anchor, determinism, monotonicity, and the null-honesty cases.

New CLI + report: cli/power-analysis.ts, docs/power-analysis-a6.md

All parameters from run-001/results/summary.json (seed 20260823, 20 000 sims):

  • Samples per cell: n=5 already gives ≥ 0.93 power for ≥ 5-point bound violations even at the adversarial-worst σ=7.60. Chasing 2-point violations would need n=25 — the honest alternative is preregistering bounds so a meaningful violation is ≥ 5 points.
  • Judge-drift precision: ~100 all-judge items pin mean |judge diff| to ±1.7 pts.
  • α precision: adversarial strata dominate the cost (±0.15 at 100 items, and those half-widths are labeled optimistic lower bounds — the bootstrap resamples only 5–6 observed values).
  • Recommendation — corpus v1: ~200–300 items, 25–30 % adversarial, n=5, ≥ 3 judge families (≈ 3 750 calls/run). The 2 000–5 000 scale is re-sized only after Run 002 provides real 3-rater data — 3-rater sizing from 2-rater data would mean inventing a judge.

Orchestrator review — applied fixes

Adversarial review returned APPROVED WITH REQUIRED FIXES; all applied:

  • MAJOR: doc table labeled σ=2.05 while the CLI simulated at rounded σ=2.0, and the even-count median was the upper-middle element, not interpolated. Fixed: interpolated median (1.92), simulations at unrounded σ, labels via toFixed(2) — the table now states exactly what was simulated.
  • "Nothing is assumed that the run did not measure" was false (normality is assumed) → "no parameter is invented", with modeling assumptions listed.
  • Bit-for-bit reproducibility scoped to Node (Math.log/sin/cos are not bit-specified by IEEE 754).
  • All-1.000 saturated rows included; controls median α ≈ 0.159 cross-referenced to the A5 disaggregated analysis.

Validation

  • apps/web: 374/374 tests; packages/core: 187/187; cold tsc clean on core and on all touched files (the only web errors are the pre-existing Prisma-generated-client absence in this container — CI regenerates it).
  • sync-core drift check clean.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Y7wiMwFa2D4P8zD9RhCgsv


Generated by Claude Code

claude added 2 commits August 23, 2026 18:51
Replaces the roadmap's round-number corpus target with numbers DERIVED
from Run 001's committed parameters, reproducible bit-for-bit (fixed
seed 20260823, 20k sims):

- lib/statistics/simulation.ts: mulberry32 + Box-Muller + H1
  bound-detection power + bootstrap CI half-widths. Anchored against
  the analytic answer (sigma 5, delta 5, n 5 -> Phi(sqrt(5)) = 0.987,
  matched by simulation) with monotonicity and determinism tests.
- cli/power-analysis.ts reads summary.json (median cell SD 2.05, max
  7.60, the 11 observed |judge diffs|) and prints tables A/B/C.
- docs/power-analysis-a6.md records the real outputs and the derived
  recommendation: corpus v1 ~200-300 items (25-30% adversarial — the
  stratum that dominates precision cost: alpha still +/-0.15 at 100
  items there), n=5 samples/cell suffices for >=5-point violations
  even at adversarial-worst SD, ~100 all-judge items pin the
  inter-judge difference to +/-1.7 pts. The 2000-5000 full-benchmark
  scale is re-sized after Run 002 (3-rater sizing from 2-rater data
  would require inventing a judge — refused).
- Assumptions stated in code, CLI output AND doc: normality for table
  A; representativeness of the 11 items for B/C (the binding
  limitation — which is the argument for the corpus); 2-rater only.

app 374/374, core 187/187, cold tsc 0 both sides, sync no drift

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y7wiMwFa2D4P8zD9RhCgsv
…veats

- power-analysis CLI: even-count median is now interpolated (1.92, was
  upper-middle 2.05); simulations run at the unrounded observed sigmas
  and rows are labeled with sigma.toFixed(2) — the doc table now states
  exactly what was simulated
- bit-for-bit reproducibility claim scoped to Node (Box-Muller uses
  Math.log/sin/cos, not bit-specified by IEEE 754)
- doc: 'no parameter is invented' replaces the false 'nothing is assumed
  that the run did not measure' (normality IS an assumption, and is
  listed); table A includes all rows incl. saturated 1.000 ones; table C
  half-widths labeled optimistic lower bounds (bootstrap from 5-6
  observed values); controls median alpha ~0.159 cross-referenced to the
  A5 disaggregated analysis
- sync-core: statistics/simulation mirrored into packages/core

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y7wiMwFa2D4P8zD9RhCgsv
@pixelstrade-dev
pixelstrade-dev merged commit 52e01ae into main Aug 23, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants