feat(science): inter-rater reliability α + ICC(2,1) — Phase A5 (orchestrator-verified) - #61
Merged
pixelstrade-dev merged 3 commits intoAug 23, 2026
Conversation
…) (Phase A5) lib/statistics/agreement.ts: - Krippendorff's alpha, interval metric, missing-tolerant (units with a single rating ignored per Krippendorff); hand-computed anchor test (units (1,2),(3,4) -> alpha 0.7) plus systematic-offset case proving it measures ABSOLUTE agreement, unlike Pearson r - ICC(2,1) per Shrout & Fleiss (two-way random, absolute agreement, single rater) — validated against the PUBLISHED 1979 anchor dataset (6x4 matrix -> 0.29); ANOVA mean squares exposed for auditability - both return null when undefined, never a fake number Runner: ExperimentSummary.interRater emitted natively (additive) — from Run 002 the coefficients ship with every run. cli/agreement.ts computes them POST-HOC for completed runs with a per-dataset disaggregation. RUN 001 POST-HOC FINDING (the reason disaggregation is mandatory): pooled alpha 0.835 / ICC 0.846 look respectable — and are bimodality-inflated exactly as protocol-001 warned for Pearson r. Disaggregated: benchmarks 0.920/0.930 vs NEGATIVE CONTROLS 0.230/0.341. Inter-judge reliability collapses precisely on adversarial content — the strongest quantitative argument yet for multi-judge ensembles plus deterministic verification. Recorded in the COMPOSITE construct card with the caveat as a known confound; ROADMAP and CHANGELOG updated. app 362/362, core 176/176, cold tsc 0 both sides, sync no drift, registry 7/7 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y7wiMwFa2D4P8zD9RhCgsv
Orchestrator review verified the math to machine precision (independent coincidence-matrix reference, 200-case fuzz max diff 3.6e-16; Shrout & Fleiss ICC contrast-checked against (3,1) and (2,k) to prove the (2,1) form) — and caught the project applying asymmetric skepticism to its own headline claim. Fixed: - MAJOR: 'reliability collapses on adversarial content' overclaimed — the disaggregated contrast carries its OWN confounds, now stated symmetrically everywhere: the benchmark set is itself bimodal (leave-one-out: benchmarks alpha 0.920->0.425 without the designed- low item) and the controls' narrower range deflates their alpha. The honest raw statistic leads now: mean |inter-judge difference| 14.9 pts on controls vs 10.1 on benchmarks (direction real, ~1.5x, not the 4x the coefficients suggest). CLI prints mean |judge diff| per dataset and the false 'bimodality removed' label is gone. - MINOR: single-pairable-unit alpha is undefined (was identically 0 — a misleading artifact) + test - MINOR: runner wiring pinned by test (mock summary emits interRater) - NIT: iccItems reports 0 when expectedRaters < 2 app 365/365, core 179/179, cold tsc 0, sync no drift, registry 7/7 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y7wiMwFa2D4P8zD9RhCgsv
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y7wiMwFa2D4P8zD9RhCgsv
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Programme validité — Phase A5 : fiabilité inter-juges chance-corrigée
lib/statistics/agreement.tsinterRaternativement dans chaque summary (Run 002 les aura d'office) ;cli/agreement.tsles calcule post-hoc avec désagrégation par dataset + mean |judge diff| brut.Résultat Run 001 (post-hoc, 11 items × 2 juges) — cadrage à scepticisme symétrique
Revue orchestrateur : APPROVED WITH REQUIRED FIXES → tous appliqués
Maths vérifiées indépendamment : implémentation de référence à matrice de coïncidences (fuzz 200 cas, écart max 3.6e-16), ICC contrast-checké contre (3,1)=0.71 et (2,k)=0.62 publiés pour prouver la forme (2,1), chiffres Run 001 reproduits depuis
summary.jsonpar code séparé. Le MAJOR retenu était mon propre overclaim (« la fiabilité s'effondre ») — corrigé partout ; la carte COMPOSITE porte désormais le confound « structure de variance dans les deux sens » comme règle permanente.Test plan
interRater)🤖 Generated with Claude Code
https://claude.ai/code/session_01Y7wiMwFa2D4P8zD9RhCgsv
Generated by Claude Code