Skip to content

ums: hold a fallback self-review to the bot's prose-fact-check bar - #1094

Merged
d-morrison merged 2 commits into
mainfrom
ums/fallback-self-review-shallow
Aug 3, 2026
Merged

ums: hold a fallback self-review to the bot's prose-fact-check bar#1094
d-morrison merged 2 commits into
mainfrom
ums/fallback-self-review-shallow

Conversation

@dem-ucdh

@dem-ucdh dem-ucdh commented Aug 3, 2026

Copy link
Copy Markdown
Collaborator

Closes #1093.

What this records

Sharpens CLAUDE.md's "Do the review yourself when the @claude workflow doesn't produce a verdict" section with a Do/Don't-labelled note about a specific trap: a fallback self-review -- one posted because the automated reviewer was unavailable (quota-skipped, stub, or erroring on an infra failure) -- feels like a stopgap and so tends to get a shallower pass than the round deserves. The gap is predictable: a shallow self-review checks structure (dogfood back-reference, ASCII, semantic line breaks) and skips the prose fact-check, so a false mechanism claim or a misattributed citation sails straight through.

The fix: run the applicable prose-review skills against the diff's own factual claims -- fact-check-prose, the cause claim-type check in metacognitive-monitoring, and the read-the-cited-source rule in address-every-comment -- not just the structural checks. The standard does not relax because the reviewer it replaces was absent.

Case record

#1092, 2026-08-03: a fallback self-review posted while claude-review was erroring on infra failures reported "no findings / ready" and missed two content bugs the recovered bot then caught -- a false mechanism claim about a directional-word grep, and a citation misattributed to shared/writing/definition-crossrefs.md (which is about formal Quarto crossref-div ordering and says no such thing). Both are semantic errors escaping mechanical checks -- the very subject of the reviewed section.

Placement / near-duplicate check

  • CLAUDE.md "Do the review yourself..." is the closest home; it already says "Apply the same review standards the bot would" but does not warn about fallback shallowness or name the prose fact-check skills. This is the fallback-specific sharpening of that line.
  • shared/workflow/ardi.md's pre-push self-review section names fact-check-prose, but for the general pre-push convention check, not the fallback-substitute case or its stopgap-shallowness trap.
  • shared/workflow/fully-clean.md covers self-review-as-fallback but framed around "still need an external verdict when reachable", not the prose-fact-check bar for the fallback you post.
  • shared/workflow/metacognitive-monitoring.md supplies the cause claim-type check the fix points at (its claim-type list is State/Scope/Cause/unexamined-default); the citation-checking rule lives in shared/workflow/address-every-comment.md, so the fix links both.
  • Normalized searches for stopgap, shallow, same bar as the bot, same review standards found no existing coverage of this trap.

Collision note

Disjoint from the other open PRs. #1088 also touches CLAUDE.md but at a distant section (~L521, the review-verdict-query section); this edit is at ~L887. The other open PRs (#1047 hooks/, #1091/#1089/#1087 shared/, #1092 shared/writing/forward-references.md, #1042 hooks/) do not touch CLAUDE.md.

Self-checks

  • Added lines are pure ASCII (no em/en-dash, curly quotes, or U+00D7); --- used for dashes.
  • One sentence per line; the **Bold.** + next-sentence pattern matches the dominant corpus convention (the local reformatter's suggestion to merge them is the documented .**-boundary false-merge it makes on all 31 existing bold-lead entries too).
  • All referenced fragment paths verified to exist; the definition-crossrefs.md characterization in the case record was verified by reading the file.

Review round 1 (addressed)

claude-review correctly caught that the paragraph itself misattributed a "citation" claim-type check to metacognitive-monitoring.md (which has no such type) -- the exact bug the paragraph warns about. Fixed in 45fdb5c: the file's main sentence and first Do bullet now point the cited-source half at address-every-comment, and this PR body is synced to match.

Copilot AI review requested due to automatic review settings August 3, 2026 10:33
@github-actions
github-actions Bot removed the request for review from Copilot August 3, 2026 10:33
Comment thread CLAUDE.md Outdated
@claude

claude Bot commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Code review

Reviewed the diff (one new subsection added to CLAUDE.md, no code changes). Verified: both linked files exist (shared/writing/fact-check-prose.md, shared/workflow/metacognitive-monitoring.md), the cited case record (#1092, #1093) accurately reflects that PR's history, definition-crossrefs.md is correctly characterized, the Do/Don't block is properly labelled per CLAUDE.md's "Record both the pattern and the anti-pattern" convention, added lines are ASCII-only with no banned punctuation, and the insertion doesn't break any downstream count-based back-reference or duplicate the nearby "Do the review yourself..." / "Self-review is the immediate fallback..." material.

One finding, posted inline on CLAUDE.md:890: the new text credits shared/workflow/metacognitive-monitoring.md with a "citation" claim-type check, but that file defines exactly four claim types (State, Scope, Cause, An unexamined default) and contains zero occurrences of "citation." The citation-verification rule actually lives in shared/workflow/address-every-comment.md (and is touched on in shared/writing/fact-check-prose.md, already linked one line up). This is the same misattributed-citation defect the paragraph itself is warning readers to catch — worth a quick fix before merge.

Verdict

Needs minor changes — one non-blocking-but-ironic citation fix (inline comment above); otherwise ready for merge.

@github-actions

github-actions Bot commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Claude finished reviewView run


Review posted. Summary:

Finding (1, inline): CLAUDE.md:890 credits shared/workflow/metacognitive-monitoring.md with a "citation" claim-type check, but that file only defines four claim types — State, Scope, Cause, and "An unexamined default" — with zero mentions of "citation" (confirmed via grep and independently by three separate review agents). The citation-verification guidance actually lives in shared/workflow/address-every-comment.md. This is notably the exact class of bug — a misattributed citation — that the new paragraph itself is warning readers to watch for.

Everything else checked out clean: both linked files exist, the cited case record (#1092/#1093) accurately reflects history, the Do/Don't block is properly labelled, punctuation/line-break conventions are followed, and the insertion doesn't duplicate nearby content or break any back-references.

Verdict

Needs minor changes — one non-blocking citation fix flagged inline (with a suggested rewording); otherwise ready for merge.

@github-actions

github-actions Bot commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

💰 Cost: $13.9774 (review) — run

claude-review round 1 found the paragraph misattributed a 'citation'
claim-type check to metacognitive-monitoring.md, which has exactly four
claim types (State, Scope, Cause, unexamined default) and no citation
type. The read-the-cited-source rule lives in address-every-comment.md.
Fixed the main sentence and the parallel first Do bullet; the cause half
stays pointed at metacognitive-monitoring.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01C9YWAmy9vJ7A6pEfkofgMk
Copilot AI review requested due to automatic review settings August 3, 2026 10:45
@github-actions
github-actions Bot removed the request for review from Copilot August 3, 2026 10:45
@github-actions

github-actions Bot commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Claude finished reviewView run


Code review — #1094

Scope checked: --comment was not passed, so per the code-review skill's step 7 I stopped short of posting anything to GitHub and am reporting findings here only.

What changed since the last round: The PR added a new CLAUDE.md subsection ("A fallback self-review is prone to being shallow...") plus a Do/Don't block and a case record citing #1092/#1093. My prior review (posted 2026-08-03T10:42:31Z, comment) found one issue: the paragraph credited shared/workflow/metacognitive-monitoring.md with a "citation" claim-type check it doesn't have.

Verification of the fix: Commit 45fdb5c re-split the sentence so metacognitive-monitoring is now cited only for the cause claim-type check, and the "read the cited source" rule is now attributed to shared/workflow/address-every-comment.md. I independently re-verified:

  • metacognitive-monitoring.md defines exactly State/Scope/Cause/An-unexamined-default, 0 occurrences of "citation" (grep -ic citation → 0).
  • address-every-comment.md:863 does contain "When a finding cites a source, read the cited source before reproducing anything" — the citation now resolves correctly.
  • The parallel first Do bullet was fixed the same way.
  • The "above" back-reference to "Apply the same review standards the bot would" correctly points to line 859, which is above the insertion in the same section (no forward-reference / stale-back-reference issue, per the very rule class this PR is about).
  • The case record's claims about ums: inserting prose can stale a downstream count back-reference #1092 and A fallback self-review is prone to being shallow; hold it to the same prose-fact-check bar as the bot it replaces #1093 match those items' actual bodies.
  • Added lines are ASCII-only (no em/en-dash, curly quotes, ×) and one-sentence-per-line, consistent with the rest of the file.
  • No duplication with the adjacent "Self-review is the immediate fallback..." material — that block is about whether an external verdict is still required; this new block is about the depth of the self-review itself.

No new issues found in this round.

Verdict

Ready for merge.

@github-actions

github-actions Bot commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

💰 Cost: $5.2835 (review) — run

@d-morrison
d-morrison merged commit bafcc06 into main Aug 3, 2026
11 checks passed
@d-morrison
d-morrison deleted the ums/fallback-self-review-shallow branch August 3, 2026 16:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A fallback self-review is prone to being shallow; hold it to the same prose-fact-check bar as the bot it replaces

3 participants