shepherd: nothing binds promotion to the vendored checker's verdict — a hand-rolled poller accepted a conversational bot reply as terminal Codex evidence

[evanharmon1]

Invariant

A draft is promoted only on a current-head Codex verdict produced by the
vendored checker (ai/skills/universal/shepherd/assets/check-codex-cloud-review.sh)
or a check of provably equal strength. Ad-hoc pollers written in-session are
not gate evidence, and a bot comment that merely names the head in prose —
without the Reviewed commit: stamp — is never terminal.

Current violation (observed 2026-08-11, harmon-devkit PR #414)

The checker from #384 exists and discriminates correctly, but nothing in the
shepherd procedure requires it: an agent shepherded #414 through four Codex
rounds using hand-written scratchpad pollers whose exit condition was a
substring match on the short head across the four surfaces, OR a 👍 on the
trigger. The vendored checker was never invoked in any round. Result, per the
PR's timeline and the session's own forensic account:

• 14:51:45Z — @codex review posted for head c1eed44 (comment 5254798340)
• 14:52:00Z — 👀 from the bot on that trigger (pending, never superseded by 👍)
• 14:53:34Z — bot posts a conversational top-level reply to a
  reconciliation note, opening "No additional findings on the current head,
  c1eed4402d…" — no Reviewed commit: stamp (verified: of the bot's
  top-level comments on the PR, this one has has_reviewed_commit_stamp=false)
• 14:57:30Z — promotion to ready, 5m45s into the required 10–15 min window,
  on that unstamped comment as "terminal clean"
• 14:59:35Z — the genuine review for c1eed44 lands with two P2 findings;
  the PR sits ready-for-review with unanswered findings until the maintainer
  flags it by hand

The vendored checker would have refused: it requires a Reviewed commit: line
matching the head or a 👍 on the exact trigger, and on this evidence it
returns its bounded-wait verdict and keeps polling into the 14:59:35Z
findings. The sharp edge is that the bot's conversational replies can name the
head in prose and can arrive minutes before its review pipeline posts — so
any substring-shaped acceptance is fooled by an artifact the real reviewer
emits routinely.

Verify

rg -n 'check-codex-cloud-review' ai/skills/universal/shepherd/SKILL.md AGENTS.md

If no hit states that promotion-grade Codex verdicts MUST come from this
checker (with ad-hoc pollers named as non-evidence), the violation stands.
For the discriminator itself:

rg -n 'Reviewed commit' ai/skills/universal/shepherd/assets/check-codex-cloud-review.sh scripts/test-shepherd-codex.sh

A test asserting that an unstamped head-naming bot comment yields bounded-wait
(not clean) should appear in scripts/test-shepherd-codex.sh; absence means
the fooling artifact has no committed regression.

Acceptance criteria

• The shepherd SKILL.md's current-head cycle names
  check-codex-cloud-review.sh as the one implementation of the terminal
  verdict (the way the readiness gate already names promo_fp for its
  snapshot), and states that in-session substitutes are not gate evidence.
• AGENTS.md's terminal-evidence prose is tightened so the "clean top-level
  comment" branch cannot be read as satisfied by an unstamped comment that
  names the head — the Reviewed commit: stamp (or the exact-trigger 👍)
  is stated as the only accepted identification.
• scripts/test-shepherd-codex.sh carries a fixture reproducing the
  2026-08-11 artifact: bot-authored top-level comment naming the head in
  prose, no stamp, 👀 live on the trigger — asserting the checker verdict
  is bounded-wait, never clean.
• The promotion recipe requires recording the checker's clean verdict
  (surface + ID it accepted) alongside promo_fp before gh pr ready,
  so a promotion without a checker run is detectable after the fact.