fix(aw): require an explicit noop terminal state in agentic workflows - #1123
fix(aw): require an explicit noop terminal state in agentic workflows#1123groupthinking wants to merge 2 commits into
Conversation
The EventRelay CI Investigator has auto-filed 54 "[aw] ... produced no safe outputs" issues, 6 of which are still open (#1091, #1042, #1019, #978, #964, #950). They are not real findings — they are the workflow reporting on its own silence. Root cause is a contradiction between the prompt and the harness: - The harness treats a run that emits zero safe outputs as a failure and files a tracking issue. Its own warning says "The agent should have called `noop` to explicitly indicate no action was taken." - `noop` is available: the compiler auto-injects it (`max: 1`, required `message` field) and the runtime prompt advertises it in `<safe-output-tools>`. - But the workflow body never mentioned `noop` — zero occurrences — while actively instructing the agent to go silent: "Never create duplicate issues/comments for unchanged healthy state." "Exit before expensive analysis if preflight detects no state change." So on every healthy, unchanged CI run the agent did exactly what it was told, emitted nothing, and the harness filed a bogus issue. The workflow was guaranteed to generate noise precisely when the repository was healthy. Add an explicit terminal-state contract: a run must always emit at least one safe output, and the no-action paths must terminate with `noop` plus a one-line reason rather than returning silently. The two silent-exit constraints are amended to route through `noop`. The same contract is added to canonical-pr-remediator and focused-coverage-controller. Neither has manifested the failure yet, but both run on the identical harness with `noop` advertised and no guidance to call it, so both carry the same latent defect. Lock files recompiled with the CI-pinned compiler (gh aw v0.82.14). Only `body_hash` changes in each lock; `frontmatter_hash` and `compiler_version` are unchanged, confirming no configuration drift. Closes #1091 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
github-actions
Bot
added
ci/cd
documentation
|
Important Review skippedAuto reviews are limited based on label configuration. 🏷️ Required labels (at least one) (1)
Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Repository YAML (base), Repository UI (inherited), Organization UI (inherited) Review profile: ASSERTIVE Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Comment |
Dependency Review✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.Snapshot WarningsEnsure that dependencies are being submitted on PR branches and consider enabling retry-on-snapshot-warnings. See the documentation for more information and troubleshooting advice. Scanned FilesNone |
Agent Completion Truth Gate: BLOCKEDReasons: Machine-readable verdict{
"details": {
"collection_errors": [
"incomplete_linked_issue_contract",
"missing_intent_snapshot",
"missing_agent_run_id",
"missing_agent_login"
],
"invalid_fields": [
"policy.agent_login",
"policy.run_id"
]
},
"reasons": [
"invalid_payload"
],
"verdict": "blocked"
} |
There was a problem hiding this comment.
🟡 Not ready to approve
The contracts omit missing_tool and missing_data when determining whether another safe output already exists.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.
Pull request overview
Adds explicit noop terminal states to prevent false missing-output issues in agentic workflows.
Changes:
- Requires at least one safe output per run.
- Routes no-action and early-exit paths through
noop. - Regenerates workflow body hashes.
File summaries
| File | Description |
|---|---|
.github/workflows/focused-coverage-controller.md |
Adds terminal-state contract. |
.github/workflows/focused-coverage-controller.lock.yml |
Updates compiled body hash. |
.github/workflows/eventrelay-ci-investigator.md |
Adds noop guidance to silent exits. |
.github/workflows/eventrelay-ci-investigator.lock.yml |
Updates compiled body hash. |
.github/workflows/canonical-pr-remediator.md |
Adds terminal-state contract. |
.github/workflows/canonical-pr-remediator.lock.yml |
Updates compiled body hash. |
Review details
- Files reviewed: 6/6 changed files
- Comments generated: 3
- Review effort level: Balanced
We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.
|
Terminal state: HALTED — blocked by a fail-closed trust gate, not by this diff Both failing checks trace to one root cause that no change on this branch can satisfy:
This is the designed security posture, not a regression introduced by this PR. #1122, which touches no agent locks, passes the same gate as Human unblock (the only path): on protected Generated by Claude Code |
groupthinking
left a comment
groupthinking
left a comment
There was a problem hiding this comment.
Reviewed at head a1b00fd. The change is correct and well-scoped — prompt-only, frontmatter_hash identical in all three locks, no new capability granted, and noop is explicitly bounded to no-action outcomes. It addresses the root cause of the 54 produced no safe outputs auto-files. One substantive nit, which confirms the Copilot reviewer's finding, plus the current merge blocker.
1. The noop skip-list is missing two safe outputs (confirms Copilot)
In all three contracts the exclusion clause reads:
Skip
nooponly when you have already emitted another safe output (add_comment,create_issue,update_issue, orcreate_check_run).
Per this PR's own evidence, the harness advertises seven safe-output tools: add_comment, create_issue, update_issue, create_check_run, missing_tool, missing_data, noop. missing_tool and missing_data are themselves safe outputs that satisfy the "at least one safe output" requirement — but they're absent from the skip-list.
Consequence: a run whose only prior output was missing_tool or missing_data still matches "no listed output emitted," so the contract instructs the agent to also emit noop. That layers a "nothing to do" record on top of a genuine "a tool/data dependency was missing" signal — contradictory, and it dilutes exactly the diagnostic the harness wants preserved. It won't refile the bogus issue (the harness is satisfied by either output), so this is a correctness/clarity nit, not a regression — but it's worth fixing while the contract is being introduced.
Suggested one-line edit (identical in eventrelay-ci-investigator.md, canonical-pr-remediator.md, focused-coverage-controller.md):
Skip
nooponly when you have already emitted another safe output (add_comment,create_issue,update_issue,create_check_run,missing_tool, ormissing_data).
Then recompile so body_hash updates in the three locks — same pinned gh aw v0.82.14 no-op-verified flow you documented, otherwise the gh-aw Validation git diff --exit-code gate will reject the stale locks.
2. Merge blocker: agent-completion/truth-gate ❌ invalid_payload
The gate reports invalid_fields: [issue.number, policy.agent_login, policy.run_id] — it wants an agent-lock manifest in the PR body, not just the Closes #1091 line. This is a human-authored PR, so there is no honest agent-lock payload to supply; I'm flagging it rather than fabricating one. It needs either the manifest (if this PR should be treated as agent-completed) or a gate policy that exempts human-authored PRs. Owner decision.
Everything else is green: Vercel ✅, Vercel Deployments – garv_projects ✅, CodeRabbit skipped-by-label (expected). Not approving only because of the open nit + the truth-gate; the change itself is sound and I'd approve once (1) is applied and the gate is resolved.
Generated by Claude Code
Review caught that the skip clause enumerated safe outputs, which was wrong twice over: 1. It omitted `missing_tool` and `missing_data`. Both are safe outputs, so a run that emits one already satisfies the harness. The clause would have told the agent to emit `noop` as well -- a contradictory double signal on a run that had in fact reported something. 2. It named `create_issue`, `update_issue` and `create_check_run` in canonical-pr-remediator and focused-coverage-controller, which expose only `add_comment, missing_tool, missing_data, noop`. Those two prompts therefore referenced tools the agent does not have -- the same class of prompt/harness mismatch this PR exists to fix. Replace the enumeration with a property-based rule: skip `noop` when the run has already emitted any other safe output, and never pair `noop` with another safe output. This is correct for all three workflows and cannot drift as the per-workflow tool set changes. Recompiled with the CI-pinned compiler (gh aw v0.82.14): one line per lock, `body_hash` only; `frontmatter_hash` and `compiler_version` unchanged. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
|
|
Shepherd status (automated PR-remediation run): ready to merge — the single red check is a systemic gate bug, not a defect in this PR. Gate-by-gate on
The only failing check is Net: this PR is green on everything it can control and CodeRabbit-approved. It's blocked only by the systemic truth-gate Generated by Claude Code |
Closing — branch orphaned by the secret-purge force-pushNo common ancestor with current The work is unlanded — requiring an explicit noop terminal state in agentic workflows, scoped by property rather than a hardcoded tool list. Tracked in #1378. Branch retained for archive-tagging. Generated by Claude Code |
Canonical issue
Closes #1091
Outcome
[aw] EventRelay CI Investigator (report-first) produced no safe outputshas beenauto-filed 54 times (6 still open: #1091, #1042, #1019, #978, #964, #950). None
are real findings — they are the workflow reporting on its own silence.
The cause is a direct contradiction between the prompt and the harness it runs on:
noopto explicitly indicate no action was taken."noopis available — auto-injected with{"max":1}and a requiredmessagefield, and advertised to the agent at runtime in<safe-output-tools>:Tools: add_comment, create_issue, update_issue, create_check_run, missing_tool, missing_data, noopnoop(grep -c noop→ 0), while explicitly instructing silence:• "Never create duplicate issues/comments for unchanged healthy state."
• "Exit before expensive analysis if preflight detects no state change."
So on every healthy, unchanged CI run the agent did exactly what it was told, emitted
nothing, and the harness filed a bogus issue. The workflow was guaranteed to generate
noise precisely when the repository was healthy — which is why these issues recur
indefinitely and why closing them individually has never stopped them.
This PR adds an explicit terminal-state contract: a run must always emit at least one
safe output, and no-action paths must terminate with
noopplus a one-line reasoninstead of returning silently. The two silent-exit constraints are amended to route
through
noop.Risk
Low, and strictly noise-reducing.
frontmatter_hashis byte-identical before and after in allthree locks, so no trigger, permission, engine, model, network or
safe-outputsconfiguration is touched. Only
body_hashmoves.noopwas already compiled in and alreadyadvertised to the agent; the prompt simply never told it when to use it. The
allowlist is unchanged.
nooptono-action outcomes and instructs the agent to skip it whenever another safe output
(
add_comment,create_issue,update_issue,create_check_run) has been emitted.nooprecord on a run that should have reported something — whichis strictly more information than today's silent exit, and still visible.
canonical-pr-remediatorandfocused-coverage-controllerreceive the same contract.Neither has manifested the failure yet, but both run on the identical harness with
noopadvertised and no guidance to call it, so both carry the same latent defect.Verification
Toolchain match proven before editing anything. Local
gh awwas v0.50.7 from adifferent repo (
githubnext/gh-aw) than CI's pinnedgithub/gh-aw@v0.82.14.Installed the CI-pinned version and confirmed a no-op recompile is byte-identical:
Without that control a stale-compiler recompile would have silently rewritten all three
locks and broken the
actions-lock.jsonv0.82.14 SHA pin ingh-aw-validation.yml.Post-edit recompile is minimal and intentional — exactly one line per lock:
frontmatter_hashbody_hashcompiler_versionCI's own validation run locally:
(
actionlintneeds Docker and could not be invoked in this environment — a toolinglimitation, not a validation failure; it runs in CI.
zizmorandpoutineboth ranclean.)
The
gh-aw Validationworkflow gates this PR directly: it triggers on.github/workflows/*.mdand*.lock.yml, recompiles with the pinned compiler, and runsgit diff --exit-codeon the locks — so a stale lock cannot merge.Local
ghextension state was restored to the user's originalgithubnext/gh-aw@v0.50.7afterwards, and the
.poutine.ymlscratch file the scanner emitted was removed ratherthan committed.
Review round 1 (
ba5591799). The reviewer caught that the skip clause enumeratedsafe outputs rather than describing them. It was wrong twice over:
missing_toolandmissing_data, which are safe outputs — so a runemitting one already satisfies the harness, and the clause would have told the agent
to redundantly emit
noopas well.create_issue,update_issueandcreate_check_runincanonical-pr-remediatorandfocused-coverage-controller, which expose onlyadd_comment, missing_tool, missing_data, noop. Those two prompts referenced toolsthe agent does not have — the same prompt/harness mismatch this PR exists to fix,
reintroduced by the patch itself.
Replaced the enumeration with a property-based rule ("any other safe output", plus an
explicit never pair
noopwith another safe output), which is correct for all threeworkflows and cannot drift as tool sets change. Recompiled: again one line per lock,
body_hashonly.Production evidence
Measured directly against the live repository:
So this single defect accounts for 54 of 60 auto-filed agentic-workflow issues in
the repository's history — the dominant source of backlog noise, and the reason
priority:hightriage keeps resurfacing the same non-finding.The 5 open duplicates (#1042, #1019, #978, #964, #950) are closed separately as
duplicates of #1091 rather than through this PR, because
pr-governance.ymlrequiresexactly one
Closes #<issue>reference per PR.