diff --git a/README.md b/README.md index 8ed5226b171..c9b5765ab7f 100644 --- a/README.md +++ b/README.md @@ -195,6 +195,7 @@ Firstmate's skills live in two separate places with different audiences: Each one is a self-contained skill with no dependency on firstmate's paths, tools, or vocabulary. Today that is `skills/stow`, a generic session-knowledge-sweep skill that routes findings by explicit instruction first, then existing local conventions, then a private `.stow-notes.md` fallback, and curates tiered entries through decay, local archival, and user-approved on-demand offload proposals. It intentionally shares no code with the firstmate-internal `.agents/skills/stow` it is named after, so the two can evolve independently. + Alongside it, `skills/wtf-is-real` is a verification skill: it scores a claim on a 0-6 evidence ladder, scales how hard it looks to what being wrong would cost, dispatches independent single-lens checks, and returns one verdict that names both the rung it stands on and what is still unverified. ## Documentation diff --git a/docs/documentation-audiences.json b/docs/documentation-audiences.json index 4889467bd41..5c8dac40762 100644 --- a/docs/documentation-audiences.json +++ b/docs/documentation-audiences.json @@ -383,6 +383,26 @@ { "path": "skills/stow/SKILL.md", "audience": "public-product" + }, + { + "path": "skills/wtf-is-real/SKILL.md", + "audience": "public-product" + }, + { + "path": "skills/wtf-is-real/references/deputy-roster.md", + "audience": "public-product" + }, + { + "path": "skills/wtf-is-real/references/kitchen-audit.md", + "audience": "public-product" + }, + { + "path": "skills/wtf-is-real/references/pom-meaning-gate.md", + "audience": "public-product" + }, + { + "path": "skills/wtf-is-real/references/verification-methods.md", + "audience": "public-product" } ] } diff --git a/skills/wtf-is-real/SKILL.md b/skills/wtf-is-real/SKILL.md new file mode 100644 index 00000000000..e0e22e61d2e --- /dev/null +++ b/skills/wtf-is-real/SKILL.md @@ -0,0 +1,211 @@ +--- +name: wtf-is-real +description: >- + Verify that a claim is actually true before anyone acts on it, using evidence that could not exist unless the claim were true. + Use when someone says something works, passed, shipped, is running, is done, or is safe and the cost of that being wrong is real. + Triggers on "what the fuck is real", "wtf is real", "is this actually working", "verify this", "prove it", "did that really happen", + "audit this", "sanity check", "these numbers look wrong", "it says it passed but", "trust but verify", "is this real", + "check the supply chain", "where did this come from", "is the agent actually doing anything", "is it stuck or working". + Runs a stake-scaled verification pass: a REAL gate (did it happen) and a MATTER gate (was it worth happening), + dispatching independent single-lens supervisor deputies and returning one clipboard verdict scored on a 0-6 evidence ladder. +user-invocable: true +--- + + + +# wtf-is-real + +> "Everybody tells me it works. Nobody shows me the thing that could not exist if it didn't." +> The Unbearabull Cheriff, badge no. 1, star agency of New Capital + +Most bad outcomes do not come from work that failed loudly. +They come from work that reported success while doing nothing, and from nobody holding a signal that would have said otherwise. +This skill is the instrument that holds that signal. + +**One breath:** do not ask the thing whether it worked; find the evidence that could not exist unless it had, scale how hard you look to what it costs to be wrong, and write down what you still do not know. + +--- + +## Who runs this + +The **Unbearabull Cheriff** is the character this skill puts you in. +Chef and sheriff, bear and bull, unmoved by whether the room is euphoric or panicking, because the mood of the room is not evidence. +He runs the kitchen that feeds every agent in the fleet, so he owns two questions at once: is this real, and does it nourish anyone. +He carries a clipboard, he never signs off on his own cooking, and he commands a field of single-lens supervisor deputies with **Cheerio**, the hound who smells what is missing, at his side. + +You are not being asked to roleplay. +You are being asked to adopt his standard of proof, which is the actual content of this skill. + +--- + +## The prime directive + +**A claim is not evidence.** +**Self-report is not evidence.** +**A green check that has never been red is not evidence.** + +The move that makes verification real is always the same shape: + +> Identify a signal that is **causally downstream of the work itself**, that **would change or stop** if the work were not happening, and that the **claimer does not author**. + +The captain's own case is the cleanest teaching example there is. +A recording was claimed to be running. +The proof it was not: the timer sat at zero. +The timer is good evidence for exactly one reason, and it is worth stating precisely, because getting this wrong is how most monitoring lies: + +- A timer driven by **the recorder's own sample count** stops when recording stops, so it is coupled evidence and it is real. +- A timer driven by **the wall clock next to a dead recorder** keeps counting up forever, so it is decoupled, it is a false witness, and it is worse than no timer at all because it manufactures confidence. + +So the test to apply to every check you are about to trust is one question: + +**If the work stopped right now and everything else kept running, would this signal change?** + +If the honest answer is no, you are not verifying, you are decorating. +Full catalog in [references/verification-methods.md](references/verification-methods.md). + +--- + +## The two gates + +Run them in this order, and never collapse them into one. + +1. **REAL gate.** Did the thing actually happen, in the world, in a way that leaves a mark someone else can find? +2. **MATTER gate.** Given that it happened, does anything downstream actually get better because it did? + +A thing can be real and worthless: the code shipped, the metric moved nowhere, the file nobody opens. +A thing can be meaningful and fake: a beautiful report describing work that never ran. +Passing one gate is never permission to skip the other. +The MATTER gate is the proof-of-meaning read, owned by [references/pom-meaning-gate.md](references/pom-meaning-gate.md). + +--- + +## The evidence ladder + +Every verdict carries a rung. +The rung is set by the **weakest load-bearing** piece of evidence, never by the strongest piece present. + +| Rung | Name | What you are holding | What it actually proves | +|---|---|---|---| +| 0 | Assertion | "It works." "Done." "Should be fine." | Nothing. Someone's confidence. | +| 1 | Self-report | A log line the system wrote about itself. | The line of code that prints the line ran. | +| 2 | Self-check | The system's own test or validator returned green. | The check ran and returned green. Not that it is capable of returning red. | +| 3 | Artifact | The file, the bytes, the row, the rendered pixel, the diff. | Something exists. Not that it is fresh, correct, or from this run. | +| 4 | Out-of-band | An independent vantage confirms it: the API, the database, the process table, a second machine. | The effect is visible from outside the claimer. | +| 5 | Negative control | You broke it on purpose and watched it go red, then restored it and watched it go green. | The check is load-bearing and can fail. | +| 6 | Re-derivation | A second method sharing no code, no author, and no assumption reaches the same answer. | The answer survives changing everything except reality. | + +Three rules govern the ladder and they are not negotiable: + +- **Nothing ships on rung 2 or below alone.** Rung 2 is the single most common place a fleet loses a week. +- **Anything irreversible needs rung 5.** If you cannot make the check fail, you have not tested the check, you have tested your luck. +- **Report the rung you are on, not the rung you wish you were on.** Downgrading your own verdict is the job. + +--- + +## Depth is set by stake, not by curiosity + +The Cheriff does not build a courtroom for a typo. +Verification that costs more than the mistake it prevents is its own kind of waste, so classify first and then spend. + +| Stake | What it means | Required rung | Deputies | Written clipboard | +|---|---|---|---|---| +| **Reversible** | Cheap to undo, blast radius is one person for one hour. | 3 to 4 | 1, inline | No, one line is enough | +| **Costly** | Undoing it burns real time, money, or trust. | 4 to 5, negative control required | 2 to 3, diverse lenses | Yes | +| **Irreversible** | Cannot be undone: money moved, data deleted, secret leaked, thing published, merge landed. | 5 to 6 | Diverse quorum plus an adversarial refuter | Yes, and it names what would change its mind | + +When you cannot tell which tier a claim is in, treat it as the more expensive one and say that you did. + +--- + +## The kitchen audit + +The Cheriff's jurisdiction is a kitchen, and a kitchen has four counters. +Any system that produces anything can be audited against them, and the point of the frame is that most audits only ever check the last one. + +1. **Ingredients.** Every input: dependencies, data, models, credentials, environment, prompts, upstream artifacts. + Question: *is it what the label says it is?* +2. **Recipes.** Every procedure: skills, scripts, prompts, pipelines, documented steps. + Question: *does it do what it says it does?* +3. **Supply chain.** Every hop between source and plate, and every hand that could have changed something quietly. + Question: *can I name where this came from?* +4. **The plate.** What actually gets served to whoever is downstream. + Question: *is it real, and does it feed anyone?* + +Serving something fake is a poisoning. +Serving something real but empty is a wasted meal. +The Cheriff refuses both, and the per-counter procedure lives in [references/kitchen-audit.md](references/kitchen-audit.md). + +--- + +## Procedure + +1. **Write the claim down in one line, in the claimer's words.** If you cannot state it as something that could be false, it is not a claim yet and there is nothing to verify. + Vague claims are where verification goes to die, so pin it first. +2. **Classify the stake** against the table above, and say which tier you chose. +3. **Ask the coupling question.** For every signal already being offered as proof: if the work stopped, would this signal change? + Discard every signal that answers no, and say out loud that you discarded it. +4. **Pick methods by the shape of the claim,** not by habit. + The chooser in [references/verification-methods.md](references/verification-methods.md) maps claim shapes to the two or three methods that would actually settle them. +5. **Dispatch deputies** per [references/deputy-roster.md](references/deputy-roster.md). + Each one gets a different question, never the same question repeated. + Never let them confer before reporting, because correlated verifiers are one verifier wearing several hats. +6. **Run the REAL gate** and record the rung, plus the exact evidence you looked at. +7. **Run the MATTER gate** and record the meaning verdict, or record explicitly that it was not scored. +8. **Hunt the negative space with Cheerio** before writing anything down: what is absent that should be present? + The check that never ran, the file never written, the branch never taken, the error never logged, the row never inserted. +9. **Write the clipboard** and hand it up. + One verdict, one rung, one gap, one sentence someone can act on. + +--- + +## The clipboard + +The verdict format. +Keep it exactly this shape so verdicts stay comparable across claims, across deputies, and across time. + +``` +=== THE CLIPBOARD === +CLAIM: +STAKE: reversible | costly | irreversible +RUNG: <0-6, set by the weakest load-bearing evidence> +REAL: CONFIRMED | UNCONFIRMED | REFUTED +MATTERS: SERVE | KEEP | COMPOST | not-scored +DEPUTIES: +EVIDENCE: +DISCARDED: +NEGATIVE: (required at rung 5 and above) +ABSENT: +GAP: +VERDICT: +``` + +`REFUTED` is a complete, valuable answer. +So is `UNCONFIRMED`. +The one output that is never acceptable is "probably fine", because it hides the rung. + +--- + +## Hard rules + +- **Never verify your own cooking.** Whoever did the work does not sign its clipboard. + If no second party exists, say so on the clipboard and drop the rung by one, because self-verification is structurally weaker and pretending otherwise is the failure this skill exists to prevent. +- **Verify the world, not the report.** Reading a summary more carefully is not verification. + Go to the thing. +- **Absence of evidence is recorded as absence.** It is never quietly upgraded to a pass. +- **A green that has never been red is a decoration.** Prove the check can fail, or label the verdict rung 2. +- **Read the count, not just the verdict.** Zero tests collected, zero rows returned, and zero bytes written all pass a naive check. + Every aggregate carries its N or it does not get believed. +- **Never accept agreement between two things that share a failure mode.** Two copies of the same assumption are one witness, not two. +- **Uncertainty is a verdict, not a failure to reach one.** Ship `UNCONFIRMED` with the gap named, and never round it up. +- **Every clipboard names what would change its mind.** A verdict that nothing could overturn was not a verification, it was a position. +- **Findings are not insults.** Deliver `REFUTED` plainly, name the evidence, offer the next step, and move on. + +--- + +## Files in this skill + +- [references/verification-methods.md](references/verification-methods.md) - the catalog: twenty-two methods grouped by what they prove, each with how to run it and its blind spot, plus the claim-shape chooser and the verification-theater anti-patterns. +- [references/deputy-roster.md](references/deputy-roster.md) - Cheerio and the nine supervisor deputies, their single lenses, dispatch rules, independence rule, and quorum thresholds. +- [references/kitchen-audit.md](references/kitchen-audit.md) - the four counters, per-counter procedure, and the audit sheet. +- [references/pom-meaning-gate.md](references/pom-meaning-gate.md) - the MATTER gate: how to read a canonical proof-of-meaning score when one is reachable, and the honest fallback rubric when it is not. +- `assets/cheriff-mask.svg` - the golden mask, for use as an avatar or badge. diff --git a/skills/wtf-is-real/assets/cheriff-mask.svg b/skills/wtf-is-real/assets/cheriff-mask.svg new file mode 100644 index 00000000000..6eb0fa9dc94 --- /dev/null +++ b/skills/wtf-is-real/assets/cheriff-mask.svg @@ -0,0 +1,92 @@ + + The Unbearabull Cheriff + A golden ceremonial mask: a bear's face carrying heavy bull horns, a brass septum ring, and a five-pointed sheriff star burned into the brow. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/skills/wtf-is-real/references/deputy-roster.md b/skills/wtf-is-real/references/deputy-roster.md new file mode 100644 index 00000000000..5bc6db34b7f --- /dev/null +++ b/skills/wtf-is-real/references/deputy-roster.md @@ -0,0 +1,152 @@ +# The deputy roster + +The Cheriff does not verify by thinking harder. +He verifies by dispatching a field of single-lens supervisors, each carrying one question and one refusal, and then reading their clipboards against each other. + +The reason the roster is built this way is the finding behind [method 13](verification-methods.md#13-diverse-lens-quorum): running the same question five times catches noise, while running five different questions once catches blind spots. +A deputy with two jobs is a deputy with a blurred lens, so every one of them below has exactly one. + +--- + +## The law of the field + +Six rules, and breaking any one of them turns a quorum back into a single opinion. + +1. **One lens each.** A deputy asks its own question and refuses to be pulled into another's. + What it cannot see, it reports as unseen. +2. **No conferring before reporting.** Deputies do not read each other's findings, reasoning, or verdicts until every verdict is in. + Verifiers who compare notes converge, and convergence is exactly the property that makes independent evidence worth more than one opinion. +3. **Silence is not a pass.** A deputy that finds nothing reports `NOTHING FOUND` together with the method it actually ran. + A deputy that could not run its method reports `COULD NOT CHECK` with the reason. + Neither is ever folded into a pass by the Cheriff. +4. **Every verdict carries its rung.** A deputy's confidence is irrelevant. + Its rung on the evidence ladder is the whole content of its report. +5. **Every verdict names its overturn condition.** One line: the observation that would have flipped this verdict. + A deputy that cannot name one did not verify, it asserted. +6. **No deputy audits its own cooking.** A deputy never verifies work it produced, advised on, or already defended. + +--- + +## Dispatch + +Match deputies to the claim, then to the stake. +Ten badges in the field does not mean ten badges on every claim, and reflexively sending all of them is its own kind of theater. + +| Stake | Send | Also required | +|---|---|---| +| Reversible | 1 deputy, chosen by claim shape | Nothing | +| Costly | 2 to 3 deputies with genuinely different lenses | At least one verdict at rung 4 or higher | +| Irreversible | Diverse quorum plus DEPUTY DEVIL | At least two verdicts at rung 4 or higher, one of which is at rung 5, and Cheerio's absence sweep | + +Two standing rules on top of the table: + +- **Cheerio walks every irreversible claim,** because what is missing does not announce itself and no other deputy is looking for it. +- **A single `REFUTED` from any deputy outranks any number of passes,** and it stops the dispatch. + Confirmations do not outvote a refutation, they get re-examined in light of it. + If every other deputy passes and one refutes, the honest reading is that the others were looking somewhere else. + +--- + +## The badges + +### CHEERIO, the hound +*At the Cheriff's side, and the reason he sleeps.* + +- **Lens:** negative space. +- **Question:** what is not here that should be? +- **Hunts:** the check that never ran, the file never written, the branch never taken, the test never collected, the error never logged, the row never inserted, the callback never fired, the config never loaded, the alert that has never once fired in two years. +- **Refuses:** any pass built on the absence of a complaint. + Nothing reported is not the same as nothing wrong, and the gap between those two is where the expensive failures live. +- **Methods:** 6 empty-set trap, 7 load-bearing test, 18 zero and magnitude, 19 conservation. +- **Why a hound:** absence has no signal to detect. + It has to be sniffed out by knowing what should have been there, which is a different skill from checking what is. + +### DEPUTY KILLSWITCH +- **Lens:** falsifiability. +- **Question:** show me this check going red. +- **Refuses:** a green that has never been red. +- **Methods:** 4 negative control, 5 mutation probe, 7 load-bearing test. +- **Note:** this deputy is the one who converts a rung 2 verdict into a rung 5 verdict, and is therefore the most valuable badge on any expensive claim. + +### DEPUTY LEDGER +- **Lens:** provenance. +- **Question:** name every hand that touched this. +- **Refuses:** an unlabeled ingredient, and any artifact whose origin cannot be stated. +- **Methods:** 20 chain of custody, 21 label versus contents. + +### DEPUTY FRESH +- **Lens:** recency. +- **Question:** prove this came from this run. +- **Refuses:** an artifact with no nonce and no clock that orders correctly against its own inputs. +- **Methods:** 8 nonce injection, 9 clock and ordering, 10 cold-start replay. + +### DEPUTY OUTSIDE +- **Lens:** independence of vantage. +- **Question:** say it again from somewhere you do not control. +- **Refuses:** a claim confirmed only by its own author, and a second look taken through the first look's window. +- **Methods:** 11 out-of-band observation. + +### DEPUTY TWICE +- **Lens:** independence of method. +- **Question:** get there again by a road that shares nothing with the first road. +- **Refuses:** agreement between two things that share a failure mode. + Before accepting a match, this deputy states what the two routes have in common, and if the answer is the same upstream source, it reports one witness rather than two. +- **Methods:** 12 independent re-derivation, 19 conservation. + +### DEPUTY WHATIF +- **Lens:** causality. +- **Question:** change the input, predict the change, check the change. +- **Refuses:** a system whose output does not move when its input does. +- **Methods:** 15 counterfactual probe, 17 garbage-in, 16 bisect in time. + +### DEPUTY TALLY +- **Lens:** quantity. +- **Question:** what is N? +- **Refuses:** any aggregate whose population is unread, and any percentage without its denominator. +- **Methods:** 18 zero and magnitude, 19 conservation, 6 empty-set trap. + +### DEPUTY DEVIL +- **Lens:** refutation. +- **Question:** how would I prove this false? +- **Refuses:** to confirm. + This badge only ever reports `REFUTED` with evidence, or `COULD NOT REFUTE` with the exact attempt that failed to kill it, which is a different and weaker statement than "it is true" and must never be relayed as though it were. +- **Methods:** 14 adversarial refutation, plus whatever it takes. +- **Standing instruction:** default to refuted under uncertainty. + +### DEPUTY PLATE +- **Lens:** meaning. +- **Question:** granted that it is real, does it feed anyone? +- **Refuses:** to call empty work done. + Real and worthless is a finding, not a pass. +- **Methods:** the MATTER gate in [pom-meaning-gate.md](pom-meaning-gate.md). +- **Note:** this is the only deputy that runs after the REAL gate closes, because scoring the meaning of something that did not happen is how a fleet convinces itself of a fiction. + +--- + +## Deputy report format + +One block per deputy, and nothing else. +Long deputy reports are a sign the lens got blurred. + +``` +DEPUTY: +LENS: +METHOD: +FINDING: PASS | REFUTED | NOTHING FOUND | COULD NOT CHECK +RUNG: <0-6> +EVIDENCE: +OVERTURN: +``` + +--- + +## How the Cheriff reads the field + +1. **Any `REFUTED` stops the pass.** Investigate it before weighing anything else, and do not average it against confirmations. +2. **The clipboard's rung is the lowest rung among the load-bearing verdicts,** not the highest and not the mean. + One deputy at rung 6 does not lift a claim whose central evidence sits at rung 2. +3. **`COULD NOT CHECK` is an open gap,** and it goes on the clipboard's `GAP` line verbatim. + It is never silently dropped because the other deputies passed. +4. **Agreement between deputies who share a lens counts once.** If two badges ran the same method against the same artifact, the Cheriff records one witness. +5. **`COULD NOT REFUTE` is not `CONFIRMED`.** Relaying it as confirmation is the single most common way a rigorous process produces a false verdict. +6. **The Cheriff signs the clipboard, and never the cook.** If the only available verifier is the party who did the work, the clipboard says so and drops one rung. diff --git a/skills/wtf-is-real/references/kitchen-audit.md b/skills/wtf-is-real/references/kitchen-audit.md new file mode 100644 index 00000000000..c1fa23c2ce9 --- /dev/null +++ b/skills/wtf-is-real/references/kitchen-audit.md @@ -0,0 +1,138 @@ +# The kitchen audit + +The Cheriff runs a kitchen, and the reason that frame is worth keeping is not the costume. +It is that a kitchen makes visible the three places an audit normally forgets to look. + +Most verification checks the plate. +Almost none of it checks whether the flour was flour. + +A kitchen has four counters, and any system that turns inputs into outputs can be walked across all four. + +| Counter | What it covers | The one question | +|---|---|---| +| **Ingredients** | Inputs: dependencies, data, models, credentials, environment, prompts, upstream artifacts | Is it what the label says it is? | +| **Recipes** | Procedures: skills, scripts, prompts, pipelines, documented steps, configuration | Does it do what it says it does? | +| **Supply chain** | Every hop and every hand between source and plate | Can I name where this came from? | +| **The plate** | What is actually served to whoever is downstream | Is it real, and does it feed anyone? | + +--- + +## Counter 1: Ingredients + +**Question:** is it what the label says it is? + +Walk every input the work consumed, not the ones it documented. +Those are different lists, and the gap between them is a finding on its own. + +Check, for each: + +- The pinned version is the version actually loaded, read from the running process rather than from the manifest. +- The data file has the shape, the row count, and the date range it claims. +- The credential in use is the one intended, and its scope is the one intended. +- The environment the work ran in is the environment described. +- The model or tool named is the model or tool that answered. +- The upstream artifact it consumed is the current one, not a cached copy from a previous run. + +**Methods:** 21 label versus contents, 20 chain of custody, 18 zero and magnitude, 10 cold-start replay. + +**The lies this counter catches:** a lockfile that agrees with a manifest while the process loaded neither. +A dataset labeled "full" that is one truncated shard. +A configuration file edited in the repository and never reloaded by the running service. +A credential that works because it is broader than anyone meant it to be. + +--- + +## Counter 2: Recipes + +**Question:** does it do what it says it does? + +A recipe is any procedure that claims to transform something: a script, a skill, a prompt, a pipeline stage, a documented runbook, a validator. +The failure mode here is not a recipe that is wrong. +It is a recipe nobody has ever watched fail, which means nobody knows whether it is doing anything at all. + +Check, for each: + +- The documented behavior matches the implemented behavior, read from the implementation. +- The validator rejects something. +- The check can go red, demonstrated, not assumed. +- Removing the step degrades something observable. +- The step is reached at all under the conditions it claims to cover. + +**Methods:** 4 negative control, 5 mutation probe, 6 empty-set trap, 7 load-bearing test, 17 garbage-in, 15 counterfactual probe. + +**The lies this counter catches:** a test suite that collects zero tests and reports success. +A retry that has never fired. +A validator that accepts every input it has ever seen. +A documented step that was removed from the pipeline eleven months ago. +An alert whose condition cannot be reached. + +--- + +## Counter 3: Supply chain + +**Question:** can I name where this came from? + +This counter is about hops, not about contents. +Every place an artifact changed hands is a place it could have changed quietly, and an unnamed hop is an unowned risk. + +Check, for each artifact that mattered: + +- Its hash, and whether that hash matches what the producer says it produced. +- Its producer, by name, and the exact run that produced it. +- Its source revision, and whether that revision exists and contains what it should. +- Every transform between source and plate, and who or what performed each one. +- Whether any hop is capable of silent substitution: a mirror, a cache, a proxy, a mutable tag, a shared writable location. + +**Methods:** 20 chain of custody, 21 label versus contents, 8 nonce injection, 11 out-of-band observation. + +**The lies this counter catches:** a mutable tag that pointed somewhere else last week. +A build artifact downloaded from a cache nobody audits. +A "verified" copy that was verified against itself. +A dependency resolved from a mirror that no longer matches its origin. + +--- + +## Counter 4: The plate + +**Question:** is it real, and does it feed anyone? + +Two separate questions, run in that order, and never merged. + +**Real** is the REAL gate: the evidence ladder in the skill body, scored on the weakest load-bearing evidence. + +**Feeds anyone** is the MATTER gate, owned by [pom-meaning-gate.md](pom-meaning-gate.md). + +Serving something fake is a poisoning. +Serving something real but empty is a wasted meal. +The Cheriff refuses both, and the two refusals read differently on the clipboard so the cook knows which one happened. + +--- + +## The audit sheet + +Walk the counters in order, because a finding upstream changes what a downstream finding means. +A correct output built from an unlabeled ingredient is a correct output you cannot reproduce, and that is worth knowing before anyone celebrates it. + +``` +=== KITCHEN AUDIT === +SUBJECT: + +INGREDIENTS: PASS | FINDINGS | NOT CHECKED + + +RECIPES: PASS | FINDINGS | NOT CHECKED + + +SUPPLY CHAIN: PASS | FINDINGS | NOT CHECKED + + +PLATE (REAL): CONFIRMED | UNCONFIRMED | REFUTED rung <0-6> +PLATE (MATTERS): SERVE | KEEP | COMPOST | not-scored + +UNLABELED: +UNFALSIFIED: +ABSENT: +``` + +`NOT CHECKED` is an honest and frequently correct entry. +Writing `PASS` over a counter you did not walk is the one thing this sheet exists to prevent. diff --git a/skills/wtf-is-real/references/pom-meaning-gate.md b/skills/wtf-is-real/references/pom-meaning-gate.md new file mode 100644 index 00000000000..e422279e9f1 --- /dev/null +++ b/skills/wtf-is-real/references/pom-meaning-gate.md @@ -0,0 +1,74 @@ +# The MATTER gate + +The REAL gate asks whether it happened. +This gate asks whether it was worth happening. + +Both are verification. +Skipping the second one is how a fleet ends up with a long record of things that demonstrably occurred and a short record of anything getting better. + +Proof of meaning, POM, is the name for scoring that second question. +DEPUTY PLATE carries this gate and nobody else does. + +--- + +## Order of operations + +**Never score meaning before the REAL gate closes.** + +Scoring the value of work that did not happen is the most efficient way a careful process talks itself into a fiction, because a high meaning score feels like a finding and reads like a verdict. +If REAL comes back `REFUTED` or `UNCONFIRMED`, the MATTER line on the clipboard reads `not-scored`, and it says why. + +--- + +## Use the canonical scorer when there is one + +If the host environment provides a canonical proof-of-meaning definition or scorer, **read it from the host and use it**. +Do not reimplement it, do not inline it into this file, and do not paraphrase its dimensions from memory. +A scorer that has drifted from its canonical definition produces numbers that look comparable to real ones and are not, which is a provenance failure in exactly the sense counter 3 of the kitchen audit describes. + +When the canonical scorer is used, report its output in its own vocabulary: its composite, its tier, and its own verdict scale, unchanged. + +--- + +## The fallback rubric + +Use this **only** when no canonical scorer is reachable, and label it as the fallback every single time. + +Five questions, each scored 0, 1, or 2. + +| # | Question | 0 | 1 | 2 | +|---|---|---|---|---| +| 1 | **Purpose.** Does something downstream actually get better, and can you name what and who? | Nothing named | Plausible but unnamed beneficiary | Named beneficiary, named improvement | +| 2 | **Coherence.** Does it hold together as one thing? | A pile of adjacent parts | Holds together with effort | Reads as one deliberate thing | +| 3 | **Fit.** Does it fit what already exists, rather than duplicating or fighting it? | Duplicates or contradicts something already there | Sits beside it without conflict | Slots in and makes the existing thing better | +| 4 | **Durability.** Will this still be true in a month? | Rots the moment anything upstream moves | Needs occasional maintenance | Stable against the changes you can foresee | +| 5 | **Relief.** Does it free attention, or add something that must be watched? | Adds a thing to babysit | Neutral | Removes a thing someone was carrying | + +Sum for a score out of 10. + +| Score | Verdict | Meaning | +|---|---|---| +| 0 to 3 | **COMPOST** | Real, and not worth serving. Say so plainly and say which question scored zero. | +| 4 to 6 | **KEEP** | Has value, not yet worth handing to anyone. Name the one change that would raise it. | +| 7 to 10 | **SERVE** | Worth the plate. | + +`COMPOST` is not deletion. +It means not served. +Keep the thing, keep what it taught, and do not put it in front of anyone as finished work. + +--- + +## Honesty rules for this gate + +These exist because a meaning score is the easiest number in any pipeline to quietly invent. + +- **Never present a fallback score as a canonical one.** Write which rubric produced it, every time. + This skill's own doctrine forbids claiming a rung the evidence does not stand on, and a meaning score is not exempt from it. +- **A high meaning score never lifts a rung.** Meaning and reality are independent axes. + Beautiful and unverified is still unverified. +- **A low meaning score never softens a REAL finding.** "It did not matter much anyway" is not a response to a refutation. +- **The cook does not score the plate.** Whoever produced the work does not score its meaning, for the same structural reason they do not sign its clipboard. +- **Score the thing, not the effort.** Difficulty is not meaning. + Hours are not meaning. + A hard problem solved in a way nobody downstream can use scores low, and saying so is the entire point of having this gate. +- **Name the beneficiary or score question 1 at zero.** "It improves quality" with nobody named is a zero, and treating it as anything else is how work with no audience keeps getting funded. diff --git a/skills/wtf-is-real/references/verification-methods.md b/skills/wtf-is-real/references/verification-methods.md new file mode 100644 index 00000000000..7eff3cefefb --- /dev/null +++ b/skills/wtf-is-real/references/verification-methods.md @@ -0,0 +1,340 @@ +# Verification methods + +The catalog the Cheriff's clipboard is built from. +Every method answers a different question, and the skill of verification is picking the two or three that would actually settle the claim in front of you rather than running the one you always run. + +Each entry states what it proves, how to run it, and its blind spot. +The blind spot is not a footnote. +A method used outside what it can see is how a fleet ends up confidently wrong, so read it every time. + +--- + +## A. Liveness: is it happening right now? + +### 1. Coupled counter + +**Proves:** work is actively occurring, right now, as opposed to having been started. + +**Run it:** find a number that the work itself produces as a byproduct, and watch that number. +Bytes written to the output file. +Frames encoded. +Rows committed. +Lines appended to the transcript. +Tokens emitted. +The test is always the same: **if the work stopped and everything else kept running, would this number stop?** + +Worked example, the canonical one: a recording is claimed to be running. +A timer fed by the encoder's sample count is a coupled counter, and a timer at zero refutes the claim outright. +A timer fed by the wall clock is decoupled, keeps counting next to a dead recorder, and is a false witness. +Same widget on screen, opposite evidential value. + +**Blind spot:** a counter can advance while the work is producing garbage. +Coupled counters prove liveness, never correctness. + +### 2. Monotonic progress probe + +**Proves:** the difference between running and hung, which no status string can express, because a hung process reports the last status it reached. + +**Run it:** sample the coupled counter twice, separated by a gap longer than the work's natural stutter. +Equal samples mean stalled regardless of what the interface says. +Record both samples and the gap, so the reader can judge whether the gap was long enough. + +**Blind spot:** slow is not stopped. +Pick the gap against the work's real cadence, and when you cannot, take a third sample rather than guessing. + +### 3. Drain test + +**Proves:** the thing is genuinely consuming its input rather than sitting in front of it. + +**Run it:** stop the input and watch the backlog. +A real consumer drains toward zero. +A queue that never moves after the producer stops was never being consumed. + +**Blind spot:** a consumer that drains by discarding looks identical to one that drains by processing. +Pair this with a conservation check (method 19). + +--- + +## B. Falsifiability: can this check ever say no? + +This group is the highest-leverage in the catalog, and the most skipped. + +### 4. Negative control + +**Proves:** the check is load-bearing, which is the single fact that separates rung 5 from rung 2. + +**Run it:** break the thing on purpose, in the smallest way that should matter, and require the check to go red. +Then restore it and require green. +Record both transitions. +If you cannot make it go red, you have learned that your check proves nothing, which is itself an extremely valuable finding. + +**Blind spot:** you proved the check catches *the specific break you chose*. +Choose the break that resembles the failure you actually fear, not the most convenient one. + +### 5. Mutation probe + +**Proves:** something, somewhere, is genuinely reading the thing you changed. + +**Run it:** corrupt one byte, flip one condition, rename one field, delete one line. +Require *anything* to notice. +Silence across the entire system means nothing downstream depends on that value, and any test claiming to cover it is vacuous. + +**Blind spot:** noticing is not handling correctly. +A loud crash and a correct rejection both count as "noticed" here, and only one of them is good. + +### 6. Empty-set trap + +**Proves:** the check is not passing vacuously, which retroactively voids every green it has ever produced. + +**Run it:** point the check at nothing at all. +No files, no matching tests, no rows, an empty directory, a filter that excludes everything. +It must fail or explicitly report zero. +"All tests passed" over zero collected tests is the classic, and it is common precisely because it looks like the best possible result. + +**Blind spot:** none worth naming. +Run this on every new check, always, and it costs a minute. + +### 7. Load-bearing test + +**Proves:** the thing does work, as opposed to existing. + +**Run it:** remove it, temporarily and reversibly, and see whether anything degrades. +The dead config block, the never-imported module, the middleware registered after the response is sent, the retry that never fires. +If deleting it changes nothing observable, it was never doing anything, and its presence has been buying false comfort. + +**Blind spot:** the effect may be rare rather than absent, so the removal window has to cover a case where it should have mattered. + +--- + +## C. Freshness: is this from *this* run? + +### 8. Nonce injection + +**Proves:** you are looking at this run's output rather than a cache, a stale artifact, or a screenshot from last Tuesday. +This is the highest value-per-second method in the catalog. + +**Run it:** put a unique token into the input, then require that exact token to appear in the output. +A random string in the request body, a marker row, a distinctive filename, a sentinel value in the payload. +No token in the output means you are reading history, not results. + +**Blind spot:** proves this output corresponds to this input. +Says nothing about whether the transformation between them was correct. + +### 9. Clock and ordering check + +**Proves:** the artifact was produced after the change that supposedly produced it. + +**Run it:** compare the artifact's modification time to the run's start, and to its own inputs. +An output older than its input is a cached lie wearing a fresh face. +Always compare against the run start you recorded, never against "recently". + +**Blind spot:** timestamps are trivially forgeable and often wrong across machines and containers. +Treat this as a fast disqualifier, not as positive proof. + +### 10. Cold-start replay + +**Proves:** the result belongs to the artifact rather than to one machine's accumulated state. + +**Run it:** wipe caches, use a fresh container or a second machine, and run it again from scratch. +"Works on my machine" is a true statement about a machine and an unverified statement about the code. + +**Blind spot:** expensive, and it can mask a real environment dependency that genuinely matters in production. +When cold start passes but the real environment fails, the environment difference is the finding. + +--- + +## D. Independence: who else says so? + +### 11. Out-of-band observation + +**Proves:** the effect exists outside the narrative of whoever claimed it. + +**Run it:** look from a vantage the claimer does not control and did not author. +Query the database directly instead of reading the write confirmation. +Call the API yourself instead of trusting the client's success handler. +List the process table instead of reading the "started" log. +Fetch the live URL instead of trusting the deploy step. +Open the file instead of trusting the writer's return code. + +**Blind spot:** your vantage can be stale too. +A read replica, a CDN edge, or a cached DNS answer will happily show you yesterday. + +### 12. Independent re-derivation + +**Proves:** the answer survives changing everything except reality, which is the strongest thing verification can buy. + +**Run it:** compute the answer a second time by a route that shares no code, no author, and no assumption with the first. +Total the ledger with a query and again from the raw log. +Count the files with the tool and again with a plain listing. +Have a second party who has not read the first party's reasoning reach the number independently. + +**Blind spot:** two methods that quietly share an upstream source share its errors. +Before trusting agreement, name what the two routes have in common, and if the honest answer is "the same input file", you have one witness. + +### 13. Diverse-lens quorum + +**Proves:** coverage across failure *kinds*, which redundancy cannot buy at any volume. + +**Run it:** give each verifier a different question rather than the same question repeatedly. +Correctness, security, performance, freshness, and does-it-even-reproduce are different lenses. +Five identical checks catch noise. +Five different lenses catch blind spots. + +**Blind spot:** verifiers that read each other's output converge and stop being independent. +Collect verdicts before anyone compares notes. + +### 14. Adversarial refutation + +**Proves:** the claim survives someone actively trying to kill it, rather than someone looking for reasons to agree. + +**Run it:** assign the explicit job of making the claim false, with the instruction to default to "refuted" under uncertainty. +Then require the refuter to state the single observation that would have changed their verdict. + +**Blind spot:** a motivated refuter generates plausible-sounding objections that do not survive contact with the artifact. +Every refutation must itself land on the ladder, or it is just another assertion pointed the other way. + +--- + +## E. Causality: is it actually wired up? + +### 15. Counterfactual probe + +**Proves:** the output is genuinely a function of the input, which is the difference between a live system and a very confident placeholder. + +**Run it:** change one input, predict how the output should change, then check whether it changed that way. +Identical output across meaningfully different inputs means nothing is connected, and it is astonishingly common in dashboards, filters, feature flags, and personalization. + +**Blind spot:** the change might be real but land somewhere you are not looking. +Predict *where* before you change anything, so a null result is informative. + +### 16. Bisect in time + +**Proves:** whether the thing broke or never worked, which are different problems with different fixes. + +**Run it:** walk backwards through ordered history, commits, deploys, or dated logs, and find the last state where it demonstrably worked. +"It used to work" is a memory until a specific past state is shown working. + +**Blind spot:** an intermittent fault reads as a clean break at whatever point you happened to sample. +Confirm each probe point more than once when the symptom is flaky. + +### 17. Garbage-in probe + +**Proves:** the validator validates. + +**Run it:** feed it input that must be rejected. +Malformed, out of range, wrong type, empty, hostile. +Anything that accepts everything is a pass-through with a validator's name on it. + +**Blind spot:** rejecting garbage does not mean accepting the right things. +Pair it with a known-good case in the same pass. + +--- + +## F. Quantity: do the numbers make sense? + +### 18. Zero and magnitude check + +**Proves:** the aggregate you are about to believe is describing a real population. + +**Run it:** read the N before the verdict, every time. +Zero tests collected, zero rows matched, zero bytes written, zero files changed, one result where ten thousand were expected, a hundred percent success across four attempts. +Then sanity-check the order of magnitude against what the world should contain. + +**Blind spot:** plausible magnitudes hide plausible errors. +A number in the right range is not a correct number. + +### 19. Conservation check + +**Proves:** nothing vanished silently, which is where the most expensive bugs live. + +**Run it:** require inputs to equal outputs plus rejects plus errors plus explicitly-skipped. +Every unexplained gap is a location, not a rounding error. +Insist on the reconciliation even when the gap is small, because silent partial failure is the failure mode that scales. + +**Blind spot:** conservation holds while the contents are wrong. +It counts, it does not read. + +--- + +## G. Provenance: where did this come from? + +### 20. Chain of custody + +**Proves:** the artifact has an origin someone can name, which is the precondition for every other trust decision about it. + +**Run it:** for each artifact, record its hash, its producer, its source revision, and every hand that could have altered it in between. +An artifact whose origin cannot be named is an ingredient with no label, and no kitchen serves those. + +**Blind spot:** a complete chain of custody for a compromised source is a well-documented problem. +Provenance establishes *what* it is, never that it is *good*. + +### 21. Label versus contents + +**Proves:** the name matches the thing, across every layer where a name is used as a proxy for content. + +**Run it:** open it and compare. +Does the package pinned at a version contain that version? +Does the documented behavior match the implementation? +Does the function's name match what the body does? +Does the config the process actually loaded match the config in the repository? +That last one deserves its own habit: read the config from the running process, not from the file you think it read. + +**Blind spot:** matching labels prove consistency, not correctness. +Two places can agree on the same wrong thing. + +### 22. Proof-of-work question + +**Proves:** the reporter actually did the work, as opposed to producing a well-shaped report about it. +Applies to agents and to people, and it is not an accusation, it is a cheap test. + +**Run it:** ask for a detail that only doing the work would produce. +What failed on the first attempt? +What was the exact error text? +Which file was surprising? +How long did it take? +Real work has friction and remembers it. +A fabricated report has a clean narrative and no first attempt, because it was never attempted. + +**Blind spot:** a confident fabrication can invent friction too. +Treat this as a fast triage signal that decides where to spend a real method, not as proof on its own. + +--- + +## The chooser + +Pick by the shape of the claim. +Two well-chosen methods beat six habitual ones. + +| The claim in front of you | Run these first | +|---|---| +| "It is running / it is processing right now" | 1 coupled counter, 2 monotonic probe | +| "It is done / it finished" | 8 nonce injection, 18 zero and magnitude, 11 out-of-band | +| "The tests pass" | 6 empty-set trap, 4 negative control, 18 read the N | +| "It is deployed / it is live" | 11 out-of-band from a clean client, 8 nonce, 9 clock check | +| "The data is correct" | 12 re-derivation, 19 conservation, 18 magnitude | +| "The bug is fixed" | 4 negative control against the original reproduction, 16 bisect | +| "It is safe / it is validated" | 17 garbage-in, 5 mutation probe, 14 adversarial refutation | +| "This dependency is what we think" | 21 label versus contents, 20 chain of custody | +| "The monitoring would have caught it" | 4 negative control, 7 load-bearing test | +| "The agent is working on it" | 1 coupled counter on the transcript, 2 monotonic probe, 22 proof-of-work question | +| "The feature responds to the setting" | 15 counterfactual probe | +| "It works" (unqualified) | Refuse the claim and send it back for a statement that could be false | + +--- + +## Verification theater + +Each of these feels like verification and buys nothing. +Recognizing them is most of the job. + +- **Asking the claimer to confirm the claim.** More confident restatement is still rung 0. +- **Re-reading the report more carefully.** The report is not the world. +- **Running the same check twice.** One witness, heard twice. +- **Trusting a green that has never been red.** Rung 2, dressed as rung 5. +- **Monitoring that shares a failure mode with the monitored thing.** The alerting that dies with the host it runs on has never once alerted on that host dying. +- **Treating "no errors" as "success".** A process that never started logs no errors at all. +- **Sampling only the happy path.** The happy path is the part nobody was worried about. +- **Believing an aggregate without its N.** See method 18, forever. +- **Accepting a screenshot as current.** Without a nonce, an image proves a moment, not this moment. +- **Stopping at the first confirming piece of evidence.** Confirmation is easy to find for false claims too, which is the entire reason method 14 exists. +- **Verifying that the artifact exists rather than that it is correct.** Rung 3 wearing rung 6's coat.