From 0741026fbea75aa9b2782f3ccf0d01e2590bb288 Mon Sep 17 00:00:00 2001 From: Griffin Long Date: Sun, 23 Aug 2026 22:17:41 -0400 Subject: [PATCH] feat(shadow-verify): add claim normalization, STALE verdict, scale cap, and gate verdict --- skills/shadow-verify/SKILL.md | 51 ++++++++++++++++++++++++++++++----- 1 file changed, 44 insertions(+), 7 deletions(-) diff --git a/skills/shadow-verify/SKILL.md b/skills/shadow-verify/SKILL.md index 72e25ae..c38c878 100644 --- a/skills/shadow-verify/SKILL.md +++ b/skills/shadow-verify/SKILL.md @@ -1,26 +1,45 @@ --- name: shadow-verify -description: "Dispatch a parallel adversarial verifier wave after any high-stakes sub-agent investigation (code reviews, audits, findings reports, large refactors) — or whenever a sub-agent asserts a claim with high-confidence language (\"confident\", \"certain\", \"clearly\", \u226580%), since confidence is a trigger, not a verdict. Shadow verifiers independently re-derive 2–3 key claims from scratch using tool calls only, returning CONFIRMED/REFUTED/UNVERIFIABLE, and flag disagreements before the user acts. Use when sub-agent output will drive decisions, file changes, commits, or external side-effects." +description: "Dispatch a parallel adversarial verifier wave after any high-stakes sub-agent investigation (code reviews, audits, findings reports, large refactors, gap analyses) — or whenever a sub-agent asserts a claim with high-confidence language (\"confident\", \"certain\", \"clearly\", ≥80%), since confidence is a trigger, not a verdict. Shadow verifiers independently re-derive 2–3 key claims from scratch using tool calls only, returning CONFIRMED/REFUTED/STALE/UNVERIFIABLE, and flag disagreements before the user acts. Use when sub-agent output will drive decisions, file changes, commits, or external side-effects." context: load --- ## Sub-agent contract /contract -When a sub-agent (or wave) returns investigation findings, code-review conclusions, audit claims, refactor plans, or counts that will drive user decisions or file changes, do NOT surface the report. Instead, run a shadow verification wave **before** merging. +When a sub-agent (or wave) returns investigation findings, code-review conclusions, audit claims, refactor plans, gap-analysis results, or counts that will drive user decisions or file changes, do NOT surface the report. Instead, run a shadow verification wave **before** merging. + +### Pre-flight: claim normalization and scale guard + +Before dispatching verifiers, the coordinating agent normalizes each claim to a canonical form: + +``` +[CLAIM_ID] :: :: +``` + +`evidence_ref` must be a concrete pointer — a file path, line number, git ref, config key, or API endpoint. Claims that cannot be anchored to a concrete `evidence_ref` are classified **`UNVERIFIABLE`** immediately and removed from the verification queue. They are preserved in the final output under a dedicated section with the reason they could not be anchored (no re-derivation sub-agent is dispatched for them). + +**Scale cap:** if the normalized claim list exceeds 50 items, the coordinating agent halts and asks the operator to scope the investigation before proceeding. Verification at scale degrades into noise; a 50-item cap prevents a finding flood from becoming an echo-chamber rubber-stamp. + +Select 2–3 of the highest-stakes normalized claims to send to the verifier wave. Prefer claims that are (a) decision-driving, (b) expressed with high-confidence language, or (c) hard to re-derive from a single artifact. + +--- **Wave 2 — Adversarial verifiers (parallel, independent):** -1. Extract 2–3 concrete, re-checkable claims from the returned report (e.g., "X function is unused", "file Y exceeds 300 lines", "PR targets main", "no tests cover Z"). -2. Dispatch one shadow sub-agent per claim, in parallel. Each receives the claim + the user's original goal + **the search surface** — the inventory of files, directories, or URLs the original investigation touched. It must NOT receive the original agent's reasoning, verdict, confidence language, or the specific line/region it concluded from. **Withhold the conclusion, not the map.** Withholding the map too does not buy extra independence — the verifier still has to reach the same evidence, it just spends its budget guessing paths to get there. Measured: one verifier denied the inventory spent 70 `grep` + 18 `read_file` calls re-locating files the parent already had paths for, guessed 5 nonexistent paths on the way, and hit its tool-loop ceiling before finishing. The independence that matters is epistemic (re-deriving the verdict), not navigational. +1. Extract 2–3 concrete, re-checkable claims from the returned report (e.g., "X function is unused", "file Y exceeds 300 lines", "PR targets main", "no tests cover Z") and normalize them to `[CLAIM_ID] subject :: predicate :: evidence_ref` form as described above. +2. Dispatch one shadow sub-agent per claim, in parallel. Each receives the normalized claim text + the user's original goal + **the search surface** — the inventory of files, directories, or URLs the original investigation touched. It must NOT receive the original agent's reasoning, verdict, confidence language, or the specific line/region it concluded from. **Withhold the conclusion, not the map.** Withholding the map too does not buy extra independence — the verifier still has to reach the same evidence, it just spends its budget guessing paths to get there. Measured: one verifier denied the inventory spent 70 `grep` + 18 `read_file` calls re-locating files the parent already had paths for, guessed 5 nonexistent paths on the way, and hit its tool-loop ceiling before finishing. The independence that matters is epistemic (re-deriving the verdict), not navigational. - The inventory is a **starting surface, not a boundary**: it does not satisfy the composition-axis guard below, and a verifier that reads only inside it still returns `evidence_base: artifact-internal`. At least one primary source outside that surface is still required for `independent-rederivation`. - **Default to `subagent_type: "research-agent"` (mechanically locked to Read/Grep/Glob/WebFetch/WebSearch — cannot Edit/commit/push).** If the claim requires Bash to verify (running a failing test, `gh pr view`, `git log origin/...`), fall back to a Bash-capable subagent type with `isolation: "worktree"` and prepend this prefix to the prompt: *"Verifier sub-agent — do not Edit, Write, commit, push, `gh pr create`, or `curl`. Return findings only."* - **Every verifier dispatch carries an explicit budget** — `max_tool_use_iterations` (a wave of 2–3 claim checks needs ~15–25 rounds each, not 50) plus the cheapest sufficient model. An unbudgeted verifier does not fail loudly: it exhausts the default tool-round ceiling, terminates `stopReason: "tool_use_loop_capped"`, and emits its verdict from a tools-stripped wind-down round built on partial evidence. A `CONFIRMED` produced that way is indistinguishable from a real one and silently defeats the entire point of the wave. Check each returned verifier's stop reason before merging its verdict; treat a capped or wind-down verifier as `UNVERIFIABLE`, not as a verdict. -3. Each verifier re-derives the verdict independently using tool calls only — never re-reading the original report's reasoning. Returns `{claim, verifier_verdict, evidence_pointer, evidence_base}`, where `verifier_verdict` is one of `CONFIRMED`, `REFUTED`, or `UNVERIFIABLE`, and `evidence_base` is `independent-rederivation` (read primary sources *outside* the cited artifact's boundary) or `artifact-internal` (re-read only the cited file/region). On `REFUTED`, the verifier also emits a corrected finding. +3. Each verifier re-derives the verdict independently using tool calls only — never re-reading the original report's reasoning. Returns `{claim_id, verifier_verdict, evidence_pointer, evidence_base}`, where `verifier_verdict` is one of `CONFIRMED`, `REFUTED`, `STALE`, or `UNVERIFIABLE`, and `evidence_base` is `independent-rederivation` (read primary sources *outside* the cited artifact's boundary) or `artifact-internal` (re-read only the cited file/region). On `REFUTED` or `STALE`, the verifier also emits a corrected or updated finding. + +--- **Merge:** - `CONFIRMED` → surface the claim as validated. - `REFUTED` → replace the claim with the verifier's corrected finding, annotated `[was: confident, now: refuted]`, and show it alongside the original with evidence. Do not act until the conflict is resolved. -- `UNVERIFIABLE` → surface with a `[needs-human-review]` tag rather than passing it through silently. +- `STALE` → the claim was true at some prior point but no longer holds. Surface annotated `[was true, now stale]` with the verifier's updated finding. Do not forward the original claim downstream; re-investigate if currency matters for the decision. +- `UNVERIFIABLE` → surface with a `[needs-human-review]` tag rather than passing it through silently. This includes both pre-flight unanchorable claims and claims a verifier could not resolve after exhausting its evidence surface. - **Budget-exhausted verifier** (`stopReason` of `tool_use_loop_capped` / `soft_deadline_wind_down`, or a `timeout`/`429` failure) → its verdict was produced without finishing the evidence gathering, so it is not a verdict. Downgrade to `UNVERIFIABLE [budget-exhausted]` and re-dispatch that one claim with a narrower scope and the search surface attached; this re-dispatch counts as the next verification round (and must respect the invoking workflow's round budget — e.g. `/review` permits one round with no repeats). If the loop cap (3 rounds total) is already reached, or the caller's budget forbids another round, escalate to the user instead of re-dispatching. Never merge a capped `CONFIRMED`. *The two verdicts below are **not** emitted by individual verifiers — they are produced by the Composition-axis guard (defined below) and handled here:* @@ -29,6 +48,22 @@ When a sub-agent (or wave) returns investigation findings, code-review conclusio Bound the loop: at most 3 verification rounds per session. Claims still unresolved after 3 rounds are escalated to the user, never silently dropped. +--- + +**Gate verdict:** + +After all verifier results are merged, the coordinating agent emits a single gate verdict before surfacing findings downstream: + +| Verdict | Condition | +|---|---| +| `PASS` | All verifiable claims confirmed; findings may proceed downstream | +| `WARN` | One or more claims are `STALE`, `UNVERIFIABLE`, or `UNVERIFIED-COMPOSITION`/`UNVERIFIED-ECHO-CHAMBER`; proceed only with explicit caveats attached | +| `FAIL` | One or more claims `REFUTED`; findings must not be forwarded until re-investigated | + +A `FAIL` verdict blocks downstream use. The coordinating agent surfaces the specific refuted claims and their contradicting evidence to the operator before any remediation work begins. A `WARN` may proceed but must carry the flagged claims and their tags — stripping the caveats before forwarding is a protocol violation. + +--- + **Composition-axis guard (echo-chamber check):** A verifier that re-derives a claim by re-reading the *same* file/region the original sub-agent cited has confirmed the citation, not the claim — it can be blind to composition-boundary failures (temporal interleaving, state threading, render/event-pipeline ordering, scrollback/call-graph adjacency) that only manifest outside the artifact's boundary. Before accepting a `CONFIRMED`: 1. Read each verifier's `evidence_base`. @@ -37,8 +72,10 @@ A verifier that re-derives a claim by re-reading the *same* file/region the orig **Scope guard:** skip the composition check when the claim cites an external referent (RFC, spec, threat model, upstream-API contract) that survives independently of the repo, or when the artifact is purely local with no composition surface. Runs once per artifact, not on every cite. +--- + **When to invoke:** -Any time sub-agent output will drive user decisions, file edits, commits, external side-effects, or is the basis of a user-facing summary. Treat **high-confidence language as a trigger in its own right**: when a review/audit sub-agent asserts a claim with markers like "confident", "certain", "clearly", "obviously", "must be", or a stated probability ≥ 80%, verify it as if it were decision-driving regardless of stakes. Confidence is a trigger, not a verdict. +Any time sub-agent output will drive user decisions, file edits, commits, external side-effects, or is the basis of a user-facing summary — including gap analyses, audit findings, and issue lists that will be forwarded to a downstream session. Treat **high-confidence language as a trigger in its own right**: when a review/audit sub-agent asserts a claim with markers like "confident", "certain", "clearly", "obviously", "must be", or a stated probability ≥ 80%, verify it as if it were decision-driving regardless of stakes. Confidence is a trigger, not a verdict. **Skip when:** Sub-agent ran inside an orchestrator skill that already verifies (`resolve`, `diagnose`, `appmap`); sub-agent returned explicit failure; work was purely exploratory and no decision follows; or the session is **text-terminal** — a pure explanation, architecture walkthrough, onboarding Q&A, or capability map that names no mutated artifact (file/PR/commit/test), where there are no re-checkable state claims for adversarial verifiers to re-derive (assess coverage, coherence, and citation density instead of dispatching re-derivation sub-agents).