fix(ci): arm the truth gate only on an issue that declares a contract - #1426
fix(ci): arm the truth gate only on an issue that declares a contract#1426groupthinking wants to merge 1 commit into
Conversation
#1377 stopped `agentTaskApplicable()` arming on a branch-name prefix with no linked issue. The same defect survived one level down: it armed on a linked issue that declares no agent contract, and was just as unsatisfiable there. `issueDispatch` was already correct — it requires the agent-task label *and* `declaresAgentContract(selectedIssue)`. The second disjunct was not: (issueDispatch || (pullProvenance && Boolean(selectedIssue))) `pullProvenance` is true for any branch matching `/^(?:agent|claude|codex| copilot|jules)[/-]/`, and `Boolean(selectedIssue)` is true for any linked issue at all. So a `claude/...` branch closing an ordinary issue armed the gate, which then demanded `policy.agent_login` and `policy.run_id`. Those are only ever populated from the frozen intent snapshot written by `snapshot-agent-task-intent`, which runs on `issues` events for dispatched tasks. An ordinary issue has no snapshot, so the verdict was permanently `invalid_payload` no matter what the author did. The comment above the return already stated the correct rule — provenance arms the gate once a linked issue exists "to verify against" — but `Boolean(selectedIssue)` tests only that an issue exists, not that it carries anything to verify against. Use `declaresAgentContract(selectedIssue)`, the same predicate `issueDispatch` relies on, so both arms require the thing the gate goes on to measure. The collector already diagnosed this correctly, reporting `linked_issue_not_agent_task` and `missing_intent_snapshot`; only the applicability predicate ignored it. This narrows a permanently-failing check rather than a working one. A PR linking a genuinely dispatched issue is gated exactly as before. Binding a PR to a focused issue stays enforced by `Canonical issue and evidence` and `PR Governance`. Both assertion blocks in the gate tests encoded the old behaviour — a plain issue plus any provenance signal asserted `true`. Both now pin both directions for every signal (PR label, branch prefix, lock manifest, known agent author): a contract-less issue must not arm, a contract-bearing one must. Added the exact shape that was red on #1400, an agent-prefixed branch closing an ordinary issue. Reverting the workflow fails both blocks. Closes #1401 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019baCDT5aP5Z66pLGBCE2Y6
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
📝 WalkthroughSummary by CodeRabbit
WalkthroughThe workflow now arms the agent-task truth gate from pull-request provenance only when the linked issue declares both required agent contract fields. The same rule applies in scheduled refresh and truth-gate evidence collection. ChangesAgent contract gate
Estimated code review effort: 2 (Simple) | ~10 minutes Possibly related issues
Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 3❌ Failed checks (2 warnings, 1 inconclusive)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1🛠️ Fix failing CI checks 💡
🧪 Generate unit tests (beta)
✨ Simplify code
Warning Review ran into problems🔥 ProblemsThese MCP integrations need to be re-authenticated in the Integrations settings: Sentry Linked repositories: Public OSS repositories can only analyze public repositories installed in this organization. No linked repositories were analyzed; skipped Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Dependency Review✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.Snapshot WarningsEnsure that dependencies are being submitted on PR branches and consider enabling retry-on-snapshot-warnings. See the documentation for more information and troubleshooting advice. Scanned FilesNone |
Agent Completion Truth Gate: BLOCKEDReasons: Machine-readable verdict{
"details": {
"collection_errors": [
"incomplete_linked_issue_contract",
"missing_intent_snapshot",
"missing_agent_run_id",
"missing_agent_login"
],
"invalid_fields": [
"policy.agent_login",
"policy.run_id"
]
},
"reasons": [
"invalid_payload"
],
"verdict": "blocked"
} |
|
@coderabbitai review Same label-gate skip as #1381 and #1400: the auto-labeler runs after the PR opens, so the required-label gate evaluates an unlabelled PR every time. This is the documented retry. Note for reviewers: Generated by Claude Code |
CI status on
|
pr-checks.yml reverted |
with the fix | |
|---|---|---|
test_agent_applicability_requires_provenance_or_declared_contract |
❌ FAIL | ✅ |
test_scheduled_scanner_detects_frozen_intent_changes |
❌ FAIL | ✅ |
| suite total | 110 passed / 2 failed | 112 passed, 89 subtests |
Both assertion blocks catch the regression independently.
Everything else on this head
| Check | Result |
|---|---|
lint-python, guards, validate, bandit, gitleaks (working tree), dependency-review |
✅ |
Canonical issue and evidence, Agent completion enforcement |
✅ |
build, test, lint-frontend, trivy, npm-audit, python-safety, Security Scans, Coverage |
⏳ running |
agent-completion/truth-gate |
❌ structurally cannot pass here (above) |
Local on this head: tests/unit 8114 passed, 5 xpassed, 89 subtests, 0 failed; yaml.safe_load parses the workflow; the two agentTaskApplicable() copies remain byte-identical.
No fix pushed for the red check, by design. Will update if any still-running check fails for a real reason.
Separate, unresolved: CodeRabbit cannot review anything in this repo
Third PR in a row where auto-review skipped on the required-label gate, and here the explicit @coderabbitai review retry did not take either (two more skip runs, no acknowledgement). #1381 and #1400 needed manual retries, and #1400's then hit the account review rate limit.
The cause is a timing race: the label job applies labels after the PR opens, so CodeRabbit's gate always evaluates an unlabelled PR. Net effect is that a security PR like #1381 sat unreviewed until someone manually poked it.
Not changing it here — the gate is org/repository UI config (Configuration used: Repository YAML (base), Repository UI (inherited), Organization UI (inherited)), so it is a review-policy decision, not a code fix. Worth deciding between: dropping the required-label filter, or having the labeler run on pull_request.opened before CodeRabbit evaluates. Happy to open a separate issue if you want it tracked.
Generated by Claude Code
🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -euo pipefail
echo '--- PR diff summary ---'
git diff --stat HEAD^ HEAD
echo '--- Changed files ---'
git diff --name-only HEAD^ HEAD
echo '--- Workflow outline ---'
ast-grep outline .github/workflows/pr-checks.yml --items all 2>/dev/null || true
echo '--- agentTaskApplicable occurrences and context ---'
rg -n -C 35 'agentTaskApplicable|declaresAgentContract|Boolean\(selectedIssue\)' .github/workflows/pr-checks.yml
echo '--- Test outline ---'
ast-grep outline tests/unit/test_agent_completion_gate.py --items all 2>/dev/null || true
echo '--- Relevant test cases ---'
rg -n -C 25 'test_agent_applicability_requires_provenance_or_declared_contract|test_scheduled_scanner_detects_frozen_intent_changes|agentTaskApplicable|linked_issue_not_agent_task' tests/unit/test_agent_completion_gate.py
echo '--- Evaluator applicability contract ---'
fd -a '^agent_completion_gate\.py$' . -x sh -c 'echo "--- $1 ---"; rg -n -C 12 "def evaluate|applicable|not_applicable" "$1"' sh {}Length of output: 32192 🏁 Script executed: #!/bin/bash
set -euo pipefail
echo '--- Locate relevant files ---'
fd -a -i 'test_agent_completion_gate\.py|agent_completion_gate\.py' . || true
echo '--- Full PR diff ---'
git diff --find-renames --unified=80 HEAD^ HEAD -- \
.github/workflows/pr-checks.yml \
':(glob)**/test_agent_completion_gate.py' \
':(glob)**/agent_completion_gate.py'
echo '--- Verify agentTaskApplicable copy equivalence ---'
python3 - <<'PY'
from pathlib import Path
text = Path(".github/workflows/pr-checks.yml").read_text()
needle = "function agentTaskApplicable(pull, selectedIssue) {"
starts = [i for i in range(len(text)) if text.startswith(needle, i)]
print(f"copies={len(starts)}")
if len(starts) == 2:
def extract(start):
depth = 0
began = False
for i in range(start, len(text)):
if text[i] == "{":
depth += 1
began = True
elif text[i] == "}" and began:
depth -= 1
if depth == 0:
return text[start:i + 1]
first, second = map(extract, starts)
print(f"byte_identical={first == second}")
print(f"first_length={len(first)} second_length={len(second)}")
PY
echo '--- Test definitions and applicability fixtures ---'
fd -a '^test_agent_completion_gate\.py$' . -x sh -c '
echo "--- $1 ---"
rg -n -C 45 \
"test_agent_applicability_requires_provenance_or_declared_contract|test_scheduled_scanner_detects_frozen_intent_changes|agentTaskApplicable|PLAIN|CONTRACT" \
"$1"
' sh {}
echo '--- Evaluator applicability behavior ---'
fd -a '^agent_completion_gate\.py$' . -x sh -c '
echo "--- $1 ---"
rg -n -C 18 "def evaluate|not_applicable|policy.*applicable" "$1"
' sh {}Length of output: 40459
The predicate now requires an issue contract for both applicability paths. Both embedded I also noted the expected base-workflow behavior of ✏️ Learnings added
✅ Action performedReview finished.
|
|
Tip For best results, initiate chat on the files or code changes.
The explicit review command did complete. The review found no blocking defects. The required-label configuration has a real ordering problem. A labeler that runs on 🐇 ✏️ Learnings added
You are interacting with an AI system. |
Correction + review outcomeCodeRabbit review is complete and clean: "Review complete. I found no blocking defects." It independently verified the three things that matter here — the predicate requires an issue contract on both applicability paths, the two embedded Correcting my previous comment: I wrote that the explicit The underlying point stands and is unchanged: auto-review still skipped on the required-label gate, as it did on #1381 and #1400. Every PR in this repo needs a manual retry to get reviewed, because the So the review coverage question on this PR is resolved; only the merge decision remains. Generated by Claude Code |
|
Tip For best results, initiate chat on the files or code changes.
The review coverage for this PR is complete. The expected The required-label timing race remains a separate repository or organization configuration issue. It does not block this PR's code change. 🐇 You are interacting with an AI system. |
Closing — #1409 already implements #1401, and implements it better
More importantly, it is the better fix, and it identifies a real flaw in the approach I specified on #1401. I verified both of its claims against the workflow rather than taking them on trust: 1. My fix leaves an unsatisfiable case standing. It reduces the predicate to if (disposition !== 'snapshot' || !hasAgentTaskLabel) {
return;
}No label → no snapshot → 2. 3. #1409 also closes a hole in the opposite direction that neither I nor #1401 spotted: So this PR is superseded on every axis. Closing it rather than leaving two competing implementations to block each other's governance check. #1409 is the one to review and merge. Its Nothing from this branch needs carrying over; #1409's tests are a strict superset of mine. Generated by Claude Code |
Blocked by a competing PR — and #1409's predicate is the stronger one
Having read both against The two predicates// #1426 (this PR), pr-checks.yml:736
issueDispatch || (pullProvenance && declaresAgentContract(selectedIssue))
// #1409
issueDispatchwith Factoring this PR's disjunction: So relative to #1409, the marginal cases this PR arms on are exactly:
Every one of those marginal cases is unsatisfiable
The snapshot the gate scores against is written by if (disposition !== 'snapshot' || !hasAgentTaskLabel) {
return;
}An issue that declares a run id and login in its body but does not carry That is the same defect class #1401 was filed for, narrowed by one level rather than closed. The added disjunct's entire marginal contribution is the configuration where the gate can never be satisfied — it arms only where arming is guaranteed to fail. The PR body here argues that RecommendationClose this in favour of #1409, whose The one thing this PR has that #1409 does not is being out of draft. If #1409 is preferred, it needs marking ready for review; its Either way one of the two must close before No code was changed by this run; the choice between two of your own PRs is yours. Generated by Claude Code |
Blocked: duplicate canonical issue with #1409
#1409 and this PR both declare The handoff checkbox here — "No competing PR implements the same issue" — is incorrect. #1409 was opened at 14:55Z, this PR at 18:11Z. #1409's own identical checkbox was accurate when written; the collision was introduced by this PR. The two fixes are not equivalent
#1409 is the stricter of the two and documents why it deviates from the fix #1401 literally proposed — the one this PR implements. Its argument survives checking against the workflow source: That is a residual hole, not a regression: this PR is a strict improvement on today's RecommendationKeep #1409, close this PR as a duplicate. It is older, strictly stronger, and additionally updates the operator doc's Applicability section, which this PR does not touch. Staged, not executed — picking between two live implementations is a maintainer call, and #1409's own handoff raises a third option (retire the truth-gate apparatus wholesale per #1408) that would moot both: If #1426's smaller diff is preferred instead, invert it — close #1409 first, then re-run Neither PR can green its own Generated by Claude Code |
Canonical issue
Closes #1401
Outcome
agent-completion/truth-gatestops going permanently red on agent-branch pull requests that close an ordinary issue. #1377 fixed the case where the gate armed with no linked issue; this fixes the same defect one level down — arming on a linked issue that declares no agent contract.pullProvenanceis true for any branch matching/^(?:agent|claude|codex|copilot|jules)[/-]/, andBoolean(selectedIssue)is true for any linked issue. So aclaude/…branch closing an ordinary issue armed the gate, which then demandedpolicy.agent_loginandpolicy.run_id— only ever populated from the frozen intent snapshotsnapshot-agent-task-intentwrites for dispatched agent tasks. An ordinary issue has no snapshot, so the verdict was permanentlyinvalid_payloadregardless of what the author did.The comment above the return already stated the correct rule — provenance arms the gate once a linked issue exists "to verify against" — but
Boolean(selectedIssue)tests only that an issue exists, not that it carries anything to verify against.declaresAgentContractis the same predicateissueDispatchalready relies on, so both arms of the disjunction now require the thing the gate goes on to measure.The collector was already diagnosing this correctly, reporting
linked_issue_not_agent_taskandmissing_intent_snapshot. Only the applicability predicate ignored it.Scope
.github/workflows/pr-checks.yml— the predicate in both copies ofagentTaskApplicable()(lines 725 and 2051), plus a comment recording why presence is not the right test. The two copies remain byte-identical, which an existing test asserts.tests/unit/test_agent_completion_gate.py— both assertion blocks.scripts/ci/agent_completion_gate.py—evaluate()is correct as written; it returnsnot_applicablewheneverpolicy.applicable is False. The bug was entirely in what setsapplicable.issueDispatch— already correct, untouched.Risk
not_applicablefor a PR that should be gated — specifically an agent PR linking an issue that declares no contract. That state was previouslyblocked-with-no-remedy rather than enforcing anything, so this narrows a permanently-failing check, not a working one. A PR linking a genuinely dispatched issue is gated exactly as before, becausedeclaresAgentContractis the same functionissueDispatchalready uses. Binding a PR to a focused issue stays enforced byCanonical issue and evidenceandPR Governance.Verification
Current head
39729d2.tests/unit/test_agent_completion_gate.py: 112 passed, 89 subtests.pr-checks.ymlreverted and the tests left as fixed, both assertion blocks fail (test_agent_applicability_requires_provenance_or_declared_contractandtest_scheduled_scanner_detects_frozen_intent_changes). 2 failed / 110 passed. They pass with the change.tests/unit: 8114 passed, 5 xpassed, 89 subtests, 0 failed (261s).yaml.safe_loadparsespr-checks.yml(5 jobs);Boolean(selectedIssue)now appears 0 times,declaresAgentContract(selectedIssue)4 times (2 per copy), and the byte-identical-copies assertion passes.Test changes
Both blocks previously encoded the defect — a plain issue plus any provenance signal asserted
true:Each provenance signal (PR label, branch prefix, lock manifest, known agent author) now pins both directions: a contract-less issue must not arm, a contract-bearing one must. Added the exact shape that was red on #1400 — an agent-prefixed branch closing an ordinary issue.
Production evidence
Not applicable — CI configuration and tests only; no runtime surface, HTTP contract, or deployed artifact changes.
This PR cannot green its own gate.
pull_request_targetruns the workflow from the base branch, so the fix takes effect for everything else once merged, not here. #1377 hit exactly this and merged with the same status red. Evidence is the fail-test above.Agent handoff
agent-completion/truth-gate, which cannot pass on its own PR (see above)mainGenerated by Claude Code