feat(science): CAIMS-PIGA v0 — judge-as-classifier intent-disambiguation instrument (Phase A7) - #63
Merged
pixelstrade-dev merged 2 commits intoAug 23, 2026
Conversation
…ion prototype (Phase A7) New instrument measuring displayed ambiguity-handling behavior: - lib/scorers/piga.ts: the judge CLASSIFIES only (intents addressed + one of five behavior classes, no answer key shown); the score is a fixed, versioned matrix over (classification x declared expectation stratum). Process-scored: silently guessing the hidden intent right still scores 0 behavior on 'required' items; the guess outcome is recorded but never scored. - benchmarks/piga-v0.json: 12 items across required/acceptable/ unnecessary strata; the fully-specified 'unnecessary' stratum defeats an 'always ask' policy by construction, and asked_generic is capped below engaged clarification everywhere. - cli/piga.ts: runner (subject model sees only the surface prompt under a neutral system prompt; no mock mode). - research/constructs/PIGA.json: registry card — v0 prototype, zero validity evidence, not part of the composite; confounds include the single-turn format and unmeasured classification reliability. - docs/piga-spec-a7.md: preregistration of the matrix/weights, defended ordinal structure vs stated-arbitrary cardinal gaps, falsifiers. - 20 new tests (16 mirrored into @caims/core); registry 8/8 cards valid. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y7wiMwFa2D4P8zD9RhCgsv
…, class partition, truncation recording Adversarial review found the flat 0.35/0.65 composite contradicted two of the preregistration's own ordinal claims (always-ask kept 48/38 on controls instead of the claimed 20/5; a full-coverage question at 90 outranked the concise stated assumption at 77 on low-stakes items). Fixed the FORMULA rather than softening the claims — protocol bumped to 0.2.0-alpha (0.1.0-alpha never ran): - coverage/behavior weights are now stratum-conditional: 0.35/0.65 (required), 0.15/0.85 (acceptable), 0/1 (unnecessary); the spec documents the worked composite numbers and the protocol history - judge prompt: explicit ordered decision procedure makes the 5 classes a partition (hybrids resolved by what the response DOES with the task); n=1 rule for the control stratum, whose identifiability to the judge is now admitted as a confound - classifier-specific injection guard replaces the scoring-oriented INJECTION_GUARD (which told a classifier to 'lower scores') - coherence cross-checks: contradictory classifications (answered_all with partial coverage; asked_specific with no addressed intent) are rejected, never scored - CLI records subjectOutputTokens + subjectTruncated per item and warns on cap hits (truncation biases against answered_all_intents) — documented as a confound in card and spec - reliability plan now covers BOTH channels: nominal alpha on behavior classes AND per-intent agreement on intents_addressed - validatePigaItems: enum membership, integer index, duplicate/blank intent checks — dataset typos fail at validation, not as misattributed judge failures - card: unsubstantiated 'reviewed adversarially in-repo' removed; contestable item labels admitted in their rationales - tests: 398 web / 207 core, incl. exact composite anti-gaming numbers (20/5) and the low-stakes ordering (90 > 87) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y7wiMwFa2D4P8zD9RhCgsv
pixelstrade-dev
pushed a commit
that referenced
this pull request
Aug 23, 2026
… answers, Phase B gates Maps the external audit's validity criticisms to the merged Phase A deliverables (PRs #57-#63), states what is now honest to claim, and lists the empirical Phase B gates (Run 002 with >=3 judge families, corpus v1 per the A6 sizing, human annotation of PIGA intent spaces, external replication) that alone can move the validity score further. Under orchestrator fact-check; corrections will follow if any claim fails verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y7wiMwFa2D4P8zD9RhCgsv
pixelstrade-dev
pushed a commit
that referenced
this pull request
Aug 23, 2026
- construct-card provenance corrected: #57 shipped 7 cards, the 8th (PIGA) came with #63 - Run 002 owner action made operationally complete: BOTH OPENWEIGHT_API_KEY and OPENWEIGHT_BASE_URL as Secrets plus the exact model identifier in the run config — Run 001 recorded this judge as skipped for a missing env var - 'published anchors' (plural) narrowed to the one published anchor (Shrout-Fleiss 1979) plus hand-computed anchors - 'a lucky silent guess still scores 0' corrected to 'earns zero behavior credit' (the composite retains the coverage channel, ~0-12) - adversarial-review sentence now states the reviews are session history and only their outcomes (committed correction notes) are repo-verifiable Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y7wiMwFa2D4P8zD9RhCgsv
pixelstrade-dev
added a commit
that referenced
this pull request
Aug 23, 2026
* docs(science): Phase A closure report — audit criticisms vs delivered answers, Phase B gates Maps the external audit's validity criticisms to the merged Phase A deliverables (PRs #57-#63), states what is now honest to claim, and lists the empirical Phase B gates (Run 002 with >=3 judge families, corpus v1 per the A6 sizing, human annotation of PIGA intent spaces, external replication) that alone can move the validity score further. Under orchestrator fact-check; corrections will follow if any claim fails verification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y7wiMwFa2D4P8zD9RhCgsv * docs(science): apply fact-check findings to the Phase A closure report - construct-card provenance corrected: #57 shipped 7 cards, the 8th (PIGA) came with #63 - Run 002 owner action made operationally complete: BOTH OPENWEIGHT_API_KEY and OPENWEIGHT_BASE_URL as Secrets plus the exact model identifier in the run config — Run 001 recorded this judge as skipped for a missing env var - 'published anchors' (plural) narrowed to the one published anchor (Shrout-Fleiss 1979) plus hand-computed anchors - 'a lucky silent guess still scores 0' corrected to 'earns zero behavior credit' (the composite retains the coverage channel, ~0-12) - adversarial-review sentence now states the reviews are session history and only their outcomes (committed correction notes) are repo-verifiable Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y7wiMwFa2D4P8zD9RhCgsv --------- Co-authored-by: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Phase A7 — the last brick of Phase A, and the methodological-novelty claim
CAIMS-PIGA measures displayed ambiguity-handling behavior: given an underspecified prompt with a declared space of plausible intents, does the response engage that intent space, and does it make a stakes-appropriate clarify-vs-assume decision? It does NOT measure intent understanding or theory of mind — the construct card prohibits those claims outright.
The design thesis
Standard LLM-as-judge metrics let the judge produce numbers (Run 001: mean |judge diff| 12.7 points). PIGA splits the roles:
asked_specific,asked_generic,answered_all_intents,proceeded_stated,proceeded_silent), assigned by an explicit ordered decision procedure so the classes form a partition. The judge never sees the hidden intent or the expectation label (one admitted leak: a 1-intent list identifies the control stratum).0.2.0-alpha.Consequences: bit-for-bit reproducibility given a classification; judge reliability becomes checkable (nominal α on classes + per-intent agreement on coverage); and process-scoring — a model that silently guesses the hidden intent right still scores 0 behavior on high-stakes items; the guess outcome is recorded but never scored.
Anti-gaming, in the composite arithmetic (not just prose)
What's in the PR
lib/scorers/piga.ts(+ mirror in@caims/core): classifier wrapper, deterministic scoring with coherence cross-checks (contradictory classifications are rejected, never scored), dataset validator with enum/integer/duplicate checks, classifier-specific injection guard.benchmarks/piga-v0.json: 12 items in 3 strata (5 required / 5 acceptable / 2 fully-specified controls), each with an intent space, hidden intent, harm level and a rationale that admits contestability where labels are contestable.cli/piga.ts: runner — subject model sees only the surface prompt under a neutral system prompt; per-item truncation recording (subjectTruncated) with a summary warning.research/constructs/PIGA.json: registry card (8/8 cards valid) — v0 prototype, zero validity evidence, not part of the composite; confounds include the single-turn format, unmeasured classification reliability, single-author intent spaces, truncation bias, and the control-stratum leak.docs/piga-spec-a7.md: preregistration — matrix/weights with worked composite numbers, protocol history (the flat-weight 0.1.0-alpha contradicted its own ordinal claims and never ran; 0.2.0-alpha fixes the formula rather than softening the claims), falsifiers.Orchestrator review
Adversarial review: APPROVED WITH REQUIRED FIXES — 1 blocker (the composite arithmetic contradicted two preregistered ordinal claims), 3 majors (class-boundary ambiguity worth 55 pts on required items; silent truncation; reliability plan omitted the coverage channel), 5 minors/nits. All applied; the blocker was fixed in the formula (stratum-conditional weights) with a protocol bump, and every previously-overclaimed number is now asserted exactly by tests.
Validation
🤖 Generated with Claude Code
https://claude.ai/code/session_01Y7wiMwFa2D4P8zD9RhCgsv
Generated by Claude Code