From dd3c167e7c552792b9376cad4f18eb6bf6173e85 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 31 Aug 2026 04:19:34 +0000 Subject: [PATCH 1/3] ums: an untested negative capability claim is its own failure mode Records the shigella#46 learning: branch-protection and ruleset settings were reported "not readable from this session" across several turns, and in two deliverables, on the strength of no dedicated MCP tool existing. Plain curl against the REST API answered the whole question on the first attempt. - shared/workflow/growth-mindset.md gains "A limitation you never tested leaves no error to diagnose". Its three existing subsections each start from a call that was made and came back wrong, so none of their remedies reaches a limitation that was only inferred. - shared/workflow/metacognitive-monitoring.md gains a Capability claim type, whose remedy is to attempt the call rather than to re-query; four count-based back-references to that list are made count-free. - memories/gh-cli.md extends the existing ruleset bullet in place with the effective-rules endpoint and the bypass fields, measured live. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01YJnmjBt8DbqWZhAkuvj1fc --- memories/gh-cli.md | 17 ++++++ shared/workflow/challenge-the-assignment.md | 2 +- shared/workflow/growth-mindset.cases.md | 25 ++++++++ shared/workflow/growth-mindset.md | 61 +++++++++++++++++++ shared/workflow/metacognitive-monitoring.md | 13 +++- .../metacognitive-monitoring.rationale.md | 6 +- 6 files changed, 117 insertions(+), 7 deletions(-) diff --git a/memories/gh-cli.md b/memories/gh-cli.md index 955e48b8d..707fe270c 100644 --- a/memories/gh-cli.md +++ b/memories/gh-cli.md @@ -429,6 +429,23 @@ Org-level rulesets (`gh api "orgs//rulesets"`) can also gate branches in member repos and would still return "nothing required" with the repo queries alone; add that sweep when the repo belongs to an org. + **One endpoint answers both scopes, and `bypass_actors` is what turns a merge into evidence.** + `repos///rules/branches/` returns the rules actually **in effect** on that branch, org-level rulesets included, so it needs no separate org sweep; + each entry carries `ruleset_source_type` and `ruleset_id`, which is how a repo rule is told from an org one. + Then read the bypass fields off the single-ruleset object before treating a successful merge as evidence about the gates: + `mergeable_state: "clean"` says no required check is missing only when the actor could not have bypassed the rule instead. + ```bash + gh api "repos///rules/branches/" \ + --jq '.[] | "\(.type) \(.ruleset_source_type) \(.ruleset_id)"' + gh api "repos///rulesets/" --jq '{current_user_can_bypass, bypass_actors}' + ``` + `current_user_can_bypass` is one of `always`, `pull_requests_only`, `never`. + `bypass_actors` can be **absent from the response entirely** rather than an empty array, so test for the key rather than for a length. + (Measured 2026-08-31 against `Morrison-Lab/ai-config` ruleset 17712474, which is the only ruleset this observation comes from: `current_user_can_bypass: "never"` and no `bypass_actors` key at all.) + These are plain REST endpoints, so `curl` with `GH_TOKEN` reaches every one of them in a session with no `gh` on `PATH` and no ruleset MCP tool. + Do not read the absence of such a tool as the settings being unreadable, per + [`growth-mindset`](../shared/workflow/growth-mindset.md)'s "A limitation you never tested leaves no error to diagnose" (UCD-SERG/shigella#46, 2026-08-31). + **Required checks are not the only thing a ruleset carries -- Copilot code review is turned on there too.** A `copilot_code_review` rule schedules Copilot itself, so nothing in the PR requests the review and no per-PR reviewer entry explains where it came from. Read it off the same endpoint: diff --git a/shared/workflow/challenge-the-assignment.md b/shared/workflow/challenge-the-assignment.md index accd79e37..5b27f0fc2 100644 --- a/shared/workflow/challenge-the-assignment.md +++ b/shared/workflow/challenge-the-assignment.md @@ -654,7 +654,7 @@ the author had already read the same file twice without finding it.) - [`metacognitive-monitoring`](metacognitive-monitoring.md) governs a premise stated as background fact, and the claims you generate yourself. This governs the instruction, which asserts nothing and so trips none of its - five claim types. + claim types. Its stripping pass is the nearest authoring-side rule, and the section above says why it does not reach a load-bearing premise. - [`derive-dont-enumerate`](derive-dont-enumerate.md) also tells an author to diff --git a/shared/workflow/growth-mindset.cases.md b/shared/workflow/growth-mindset.cases.md index 93192c113..db3b63f6c 100644 --- a/shared/workflow/growth-mindset.cases.md +++ b/shared/workflow/growth-mindset.cases.md @@ -43,3 +43,28 @@ as a flat denial, and answered by searching the MCP registry and plugin catalog for a GitHub Discussions server to install. Neither could have helped -- a local server sits behind the same proxy. The REST route the refusal named worked on the first attempt.) + +## "A limitation you never tested" --- branch-protection settings reported unreadable across several turns, never once queried + +(`UCD-SERG/shigella#46`, 2026-08-31: across several turns a session told the +user that GitHub branch-protection and ruleset settings were "not readable +from this session" and that confirming a required-checks list needed a human +to open Settings. +The claim was inferred from the absence of a dedicated MCP tool for rulesets, +never tested, and then restated across several turns --- including in a merged +PR body and in a filed issue, where it became a "needs a human with +branch-protection access" note that outlived the conversation. + +`GH_TOKEN` was set in the environment throughout, and plain `curl` against the +REST API answered every part of the question on the first attempt: +`/repos/{owner}/{repo}/rulesets` for the list, +`/repos/{owner}/{repo}/rulesets/{id}` for a ruleset's rules and +`bypass_actors`, `/repos/{owner}/{repo}/rules/branches/{branch}` for the rules +in effect including org-level ones, and +`/repos/{owner}/{repo}/branches/{branch}` for the classic protection summary. +The endpoints are recorded in +[`memories/gh-cli.md`](../../memories/gh-cli.md). + +The false claim was load-bearing rather than incidental: it is what put the +"a human has to check this" note into two deliverables, so the cost was not a +mistaken sentence in chat but a premise shipped to later readers.) diff --git a/shared/workflow/growth-mindset.md b/shared/workflow/growth-mindset.md index 3af6b0c5f..e5a1ee357 100644 --- a/shared/workflow/growth-mindset.md +++ b/shared/workflow/growth-mindset.md @@ -175,6 +175,67 @@ still the right move, but it is the case where reading it does not help. - **Don't:** reach for installing something to get past a sandbox boundary; what you install inherits the boundary. +## A limitation you never tested leaves no error to diagnose + +The three sections above all begin with a call that was made and came back +wrong: a tool errored, a bounded probe returned nothing, a request was +refused. +Each remedy reads the artifact that call produced. +None of them reaches the case where no call was made at all. + +The shape is a negative claim about your own capability. +"That is not readable from this session." +"Confirming it needs a human with access to the settings page." +Such a claim is inferred from the shape of the tool surface --- no dedicated +tool covers the thing, so the data must be out of reach --- and a tool listing +is a menu rather than a boundary. + +Three things make it worse than an ordinary wrong guess. + +**It produces no failure to notice.** +Each of the three cases above announces itself with an error, a timeout, or a +status code, and each remedy is to read that output more carefully. +This one leaves nothing red, nothing logged, and no output to re-read. +The cheap test that would refute it is exactly the test the claim tells you +not to bother running, so the claim protects itself. + +**A positive claim gets tested by being acted on; this one never is.** +"I can read X" leads to reading X, which either works or does not. +"I cannot read X" ends the inquiry, so nothing downstream ever disagrees with +it, and repeating it across several turns feels like consistency rather than +like an unexamined premise hardening. + +**It ships.** +A capability claim is a natural thing to write into a PR body or an issue as +a "needs a human with access" note, where it becomes a premise for whoever +reads it next and outlives the conversation that could have corrected it. +That is [`challenge-the-assignment`](challenge-the-assignment.md)'s +"An issue body is an assignment you author" case with a capability claim in +place of a count, and it puts the sentence under the same bar as any other +factual claim in a deliverable, per +[`fact-check-prose`](../writing/fact-check-prose.md). + +The test is one call, and it costs less than the sentence asserting the +limitation. +Before writing that something cannot be read from this session, run the +plainest route: a raw HTTP request against the service's documented API with +whatever credential the environment already holds. +An MCP tool, a CLI subcommand, and a raw request are three routes to one API, +so the absence of the first two is no evidence about the third. + +- **Do:** attempt the plainest available route once, and report what came + back, before claiming a thing cannot be read from this session. +- **Do:** hold a capability claim in a PR body, an issue, or a handoff to the + same standard as any other factual claim in a deliverable. +- **Don't:** infer a limitation from the tool listing --- it enumerates what + is convenient, not what is reachable. +- **Don't:** count having repeated the limitation across turns as having + established it; a claim that ends the inquiry can never be contradicted by + it. + +See [`growth-mindset.cases.md`](growth-mindset.cases.md), "branch-protection +settings reported unreadable across several turns, never once queried". + ## Applies to our own metacognitive tooling, too The same bias governs the skills, memories, and self-improvement loops in these diff --git a/shared/workflow/metacognitive-monitoring.md b/shared/workflow/metacognitive-monitoring.md index 22f9a8e21..8163413db 100644 --- a/shared/workflow/metacognitive-monitoring.md +++ b/shared/workflow/metacognitive-monitoring.md @@ -22,6 +22,13 @@ check. - **State** --- is it green, is it pushed, does it exist, is it public. Re-query, never recall. +- **Capability** --- I cannot read that, no tool covers it, that needs a human + with access. + Attempt the plainest route once and report what came back. + There is nothing to re-query when no call was ever made, which is why this + is not the **State** case: + see [`growth-mindset`](growth-mindset.md)'s "A limitation you never tested + leaves no error to diagnose". - **Scope** --- all, every, none, only, the whole corpus. Check the population rather than the sample that came to mind. - **Cause** --- it failed because, this is flaky, that change broke it. @@ -34,8 +41,8 @@ check. ## A premise you were handed is still a claim -All five types above describe assertions **you** generate, so all five trigger -on the act of writing one. +Every type above describes an assertion **you** generate, so each one triggers +on the act of writing it. - **Do:** restate a load-bearing premise explicitly and name what would falsify it, before building on it. @@ -179,7 +186,7 @@ Each was fixed on its own terms and none of the fixes generalized.) ## An action you recommend is a claim about state -The five types above fire on an assertion, and the section above extends them +The claim types above fire on an assertion, and the section above extends them to a premise you were handed. - **Do:** re-query an artifact's state immediately before recommending an diff --git a/shared/workflow/metacognitive-monitoring.rationale.md b/shared/workflow/metacognitive-monitoring.rationale.md index 2b0185d1b..0338e6067 100644 --- a/shared/workflow/metacognitive-monitoring.rationale.md +++ b/shared/workflow/metacognitive-monitoring.rationale.md @@ -45,8 +45,8 @@ that is easy to skip. ## A premise you were handed is still a claim -All five types above describe assertions **you** generate, so all five trigger -on the act of writing one. +Every type above describes an assertion **you** generate, so each one triggers +on the act of writing it. A premise supplied by the user triggers on nothing. It arrives as context rather than as a claim, you did not reason your way to it, and adopting it feels like listening rather than like asserting. @@ -295,7 +295,7 @@ ends up protected while the person does not. ## An action you recommend is a claim about state -The five types above fire on an assertion, and the section above extends them +The claim types above fire on an assertion, and the section above extends them to a premise you were handed. Both are things somebody states. A recommendation states nothing about the world. From 5d1fbf59c34dcb8388542a5edd550797358bca3e Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 31 Aug 2026 04:22:52 +0000 Subject: [PATCH 2/3] memories(gh-cli): second ruleset for the bypass_actors absence The caveat was scoped to one ruleset by its author's own self-review. A second, UCD-SERG/shigella 6339629, reports the same shape: no bypass_actors key rather than an empty array, and current_user_can_bypass never. Records how the trap is actually met: reading the value through dict.get returns None for an absent key as well as a null one, so it gets written down as 'bypass_actors: null' when there is no such field in the response. That happened in the session that produced this entry. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01YJnmjBt8DbqWZhAkuvj1fc --- memories/gh-cli.md | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/memories/gh-cli.md b/memories/gh-cli.md index 707fe270c..75ab638fa 100644 --- a/memories/gh-cli.md +++ b/memories/gh-cli.md @@ -441,7 +441,13 @@ ``` `current_user_can_bypass` is one of `always`, `pull_requests_only`, `never`. `bypass_actors` can be **absent from the response entirely** rather than an empty array, so test for the key rather than for a length. - (Measured 2026-08-31 against `Morrison-Lab/ai-config` ruleset 17712474, which is the only ruleset this observation comes from: `current_user_can_bypass: "never"` and no `bypass_actors` key at all.) + (Measured 2026-08-31 against two rulesets, both reporting + `current_user_can_bypass: "never"` and no `bypass_actors` key at all: + `Morrison-Lab/ai-config` ruleset 17712474 and `UCD-SERG/shigella` ruleset 6339629. + The second is worth its own mention because it is how the trap was met: + a reader who tests the value rather than the key sees `None` from + `dict.get`, reports it as `bypass_actors: null`, and has recorded a field + that is not in the response.) These are plain REST endpoints, so `curl` with `GH_TOKEN` reaches every one of them in a session with no `gh` on `PATH` and no ruleset MCP tool. Do not read the absence of such a tool as the settings being unreadable, per [`growth-mindset`](../shared/workflow/growth-mindset.md)'s "A limitation you never tested leaves no error to diagnose" (UCD-SERG/shigella#46, 2026-08-31). From abd8fc802d352a77b00bfc01b0dff664d2a7d37b Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 31 Aug 2026 04:27:54 +0000 Subject: [PATCH 3/3] Sweep the two claim-type back-references the first pass missed Review found both. The Do-and-don't checklist still enumerated five types and omitted capability, so a reader following it literally would skip the type this change exists to add. The rationale companion still opened with "The five" in the section mirroring the taxonomy, while its two siblings a few lines below had been made count-free. Neither was caught by the first sweep because that sweep searched for the word "five". The checklist carries the staleness as an enumeration rather than a count, so a count-keyed query cannot see it. Derived the replacement population from the enumeration itself rather than from the count, which is address-every-comment.cases.md's own round-3 case in this same file. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01YJnmjBt8DbqWZhAkuvj1fc --- shared/workflow/metacognitive-monitoring.md | 5 +++-- shared/workflow/metacognitive-monitoring.rationale.md | 2 +- 2 files changed, 4 insertions(+), 3 deletions(-) diff --git a/shared/workflow/metacognitive-monitoring.md b/shared/workflow/metacognitive-monitoring.md index 8163413db..b285f945c 100644 --- a/shared/workflow/metacognitive-monitoring.md +++ b/shared/workflow/metacognitive-monitoring.md @@ -1079,8 +1079,9 @@ write the thing that can be wrong. ## Do and don't -- **Do:** classify each assertion as state, scope, cause, inference, or default - before it goes out, and re-measure any that is not from this turn. +- **Do:** classify each assertion as state, capability, scope, cause, + inference, or default before it goes out, and re-measure any that is not + from this turn. - **Do:** name the falsifying command beside a claim, and run it when it is cheap. - **Do:** treat a fluent, undeliberated answer as owing an alternative you can diff --git a/shared/workflow/metacognitive-monitoring.rationale.md b/shared/workflow/metacognitive-monitoring.rationale.md index 0338e6067..5d783fe6f 100644 --- a/shared/workflow/metacognitive-monitoring.rationale.md +++ b/shared/workflow/metacognitive-monitoring.rationale.md @@ -39,7 +39,7 @@ check. Claim **type** can, because it is observable in the sentence you are about to write: -The five are worth keeping as a list rather than collapsing into "check your +They are worth keeping as a list rather than collapsing into "check your claims", because each names a different *repair*, and the repair is the part that is easy to skip.