From aaca52ff6f9bd815f47db49a99fa6a249295ea34 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 7 Aug 2026 14:53:19 +0000 Subject: [PATCH 1/4] fix(ci): arm the truth gate only on a real dispatch contract `agent-completion/truth-gate` was red on roughly every pull request, including merged ones (#1368, #1408), because its arming rule and `PR Governance` were mutually unsatisfiable. The gate scores a pull request against the frozen intent snapshot on its linked issue. That snapshot is written only by `snapshot-agent-task-intent`, which runs on `issues` events alone and only for issues labelled `agent-task`/`mcp-agent` that already declare an agent run id and login. The rule armed on `issueDispatch || (pullProvenance && selectedIssue)` -- so any agent-authored branch closing *any* issue was armed, whether or not a dispatch contract existed. With no snapshot, `policy.agent_login` and `policy.run_id` are unsatisfiable and the verdict is permanently `invalid_payload`, with no action available to the author. Since `PR Governance` requires exactly one `Closes #`, satisfying it guaranteed failing this gate. The comment directly above that return already argued the correct rule -- "a branch named `claude/...` is a naming convention, not a dispatch" -- but the disjunct re-armed on exactly that. Drop it: only `issueDispatch` arms the gate now, which is already defined as label plus declared contract. This is not an escape hatch. A pull request linking a genuinely dispatched issue is still fully gated, and binding a pull request to a focused issue at all remains owned by `Canonical issue and evidence`, which states a requirement an author can meet. Applied to both copies of `agentTaskApplicable` (the `truth-gate` collector and the `refresh-open-pull-requests` scanner), which a test holds identical. Removes the now-dead `knownAgents`/`agentBranch`/`manifestPresent`/ `pullProvenance` definitions in those two blocks; the separate `knownAgents` in `dispatch-evidence-refresh` is untouched. Two tests encoded the old behaviour as intentional ("an agent producing work against a contract-less issue is still applicable, and therefore still blocked"). That expectation is the livelock, so both are updated to the corrected rule, with cases added proving a genuine dispatch still arms the gate. Operator doc updated to match. Verified: 112 passed, 89 subtests (baseline 112) in tests/unit/test_agent_completion_gate.py; all 8 inline github-script blocks pass `node --check`; `applicable: false` exits 0 as `not_applicable`. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_013Kc7prW6s237ERAMbnVDhH --- .github/workflows/pr-checks.yml | 148 +++++++++-------------- docs/agent-completion-truth-gate.md | 16 ++- tests/unit/test_agent_completion_gate.py | 87 ++++++++----- 3 files changed, 126 insertions(+), 125 deletions(-) diff --git a/.github/workflows/pr-checks.yml b/.github/workflows/pr-checks.yml index 8f5dec4fe..ca2ccd5c2 100644 --- a/.github/workflows/pr-checks.yml +++ b/.github/workflows/pr-checks.yml @@ -658,27 +658,6 @@ jobs: const login = String( pull && pull.user && pull.user.login || '' ); - const knownAgents = new Set([ - 'google-labs-jules[bot]', - 'github-copilot[bot]', - 'copilot-swe-agent[bot]', - 'openai-codex[bot]', - 'chatgpt-codex-connector[bot]' - ]); - const agentBranch = - /^(?:agent|claude|codex|copilot|jules)[/-]/i.test( - String(pull && pull.head && pull.head.ref || '') - ); - const manifestPresent = - //i.test( - String(pull && pull.body || '') - ); - // Provenance asserted by the pull request itself. Each of these - // is a claim by the producing side that this is agent work. - const pullProvenance = knownAgents.has(login) || agentBranch || - manifestPresent || - carriesLabel((pull && pull.labels) || [], - ['agent', 'agenttask', 'mcpagent']); // Issue-side dispatch. Label automation also applies agent task // labels as topic tags to issues that never declared a contract, // so the bare label is not evidence of a dispatch: it only @@ -698,31 +677,36 @@ jobs: const issueDispatch = carriesLabel(issueLabelSource, ['agenttask', 'mcpagent']) && declaresAgentContract(selectedIssue); - // Pull-side provenance says who produced the branch. It is not - // evidence that a dispatch contract exists to measure that - // branch against. The gate scores a pull request against the - // frozen intent snapshot on its linked issue, and that snapshot - // is only ever written by `snapshot-agent-task-intent`, which - // runs on `issues` events alone. With no linked issue there is - // no snapshot, no declared run id and no declared login, so - // `policy.agent_login`, `policy.run_id` and `issue.number` are - // all unsatisfiable and the verdict is permanently - // `invalid_payload` regardless of what the author does. A - // branch named `claude/...` is a naming convention, not a - // dispatch. Arming on it alone is what made this check red on - // pull requests that never had a contract to satisfy -- including - // #1368, which merged with this status failing. + // Only an issue-side dispatch arms the gate. Pull-side + // provenance says who produced the branch; it is not evidence + // that a dispatch contract exists to measure that branch + // against. The gate scores a pull request against the frozen + // intent snapshot on its linked issue, and that snapshot is only + // ever written by `snapshot-agent-task-intent`, which runs on + // `issues` events alone, and only for issues labelled + // `agent-task`/`mcp-agent` that already declare a run id and + // login. Without that snapshot `policy.agent_login` and + // `policy.run_id` are unsatisfiable and the verdict is + // permanently `invalid_payload` regardless of what the author + // does. A branch named `claude/...` is a naming convention, not + // a dispatch. + // + // Arming on `pullProvenance && selectedIssue` -- provenance plus + // *any* linked issue -- put this check in direct contradiction + // with `PR Governance`, which requires exactly one + // `Closes #` reference. Satisfying one guaranteed failing + // the other: every well-formed agent pull request was armed + // against a contract that had never been written, so the gate + // was red on ~100% of pull requests, including merged ones + // (#1368, #1408). Requiring a real dispatch instead restores the + // #1130 reasoning to the arming rule that overrode it. // - // So provenance arms the gate only once a linked issue exists to - // verify against; with none, there is nothing to measure and the - // verdict is `not_applicable`. This does not create an escape - // hatch: a pull request that links a dispatched issue is still - // fully gated, and requiring a pull request to bind to a focused - // issue at all is separately owned by `Canonical issue and - // evidence`, which states a requirement an author can actually - // meet. - return login !== 'dependabot[bot]' && - (issueDispatch || (pullProvenance && Boolean(selectedIssue))); + // This is not an escape hatch: a pull request that links a + // genuinely dispatched issue is still fully gated, and requiring + // a pull request to bind to a focused issue at all is separately + // owned by `Canonical issue and evidence`, which states a + // requirement an author can actually meet. + return login !== 'dependabot[bot]' && issueDispatch; } function intentContractErrors(issue, comments, pullCreatedAt) { const errors = []; @@ -1984,27 +1968,6 @@ jobs: const login = String( pull && pull.user && pull.user.login || '' ); - const knownAgents = new Set([ - 'google-labs-jules[bot]', - 'github-copilot[bot]', - 'copilot-swe-agent[bot]', - 'openai-codex[bot]', - 'chatgpt-codex-connector[bot]' - ]); - const agentBranch = - /^(?:agent|claude|codex|copilot|jules)[/-]/i.test( - String(pull && pull.head && pull.head.ref || '') - ); - const manifestPresent = - //i.test( - String(pull && pull.body || '') - ); - // Provenance asserted by the pull request itself. Each of these - // is a claim by the producing side that this is agent work. - const pullProvenance = knownAgents.has(login) || agentBranch || - manifestPresent || - carriesLabel((pull && pull.labels) || [], - ['agent', 'agenttask', 'mcpagent']); // Issue-side dispatch. Label automation also applies agent task // labels as topic tags to issues that never declared a contract, // so the bare label is not evidence of a dispatch: it only @@ -2024,31 +1987,36 @@ jobs: const issueDispatch = carriesLabel(issueLabelSource, ['agenttask', 'mcpagent']) && declaresAgentContract(selectedIssue); - // Pull-side provenance says who produced the branch. It is not - // evidence that a dispatch contract exists to measure that - // branch against. The gate scores a pull request against the - // frozen intent snapshot on its linked issue, and that snapshot - // is only ever written by `snapshot-agent-task-intent`, which - // runs on `issues` events alone. With no linked issue there is - // no snapshot, no declared run id and no declared login, so - // `policy.agent_login`, `policy.run_id` and `issue.number` are - // all unsatisfiable and the verdict is permanently - // `invalid_payload` regardless of what the author does. A - // branch named `claude/...` is a naming convention, not a - // dispatch. Arming on it alone is what made this check red on - // pull requests that never had a contract to satisfy -- including - // #1368, which merged with this status failing. + // Only an issue-side dispatch arms the gate. Pull-side + // provenance says who produced the branch; it is not evidence + // that a dispatch contract exists to measure that branch + // against. The gate scores a pull request against the frozen + // intent snapshot on its linked issue, and that snapshot is only + // ever written by `snapshot-agent-task-intent`, which runs on + // `issues` events alone, and only for issues labelled + // `agent-task`/`mcp-agent` that already declare a run id and + // login. Without that snapshot `policy.agent_login` and + // `policy.run_id` are unsatisfiable and the verdict is + // permanently `invalid_payload` regardless of what the author + // does. A branch named `claude/...` is a naming convention, not + // a dispatch. + // + // Arming on `pullProvenance && selectedIssue` -- provenance plus + // *any* linked issue -- put this check in direct contradiction + // with `PR Governance`, which requires exactly one + // `Closes #` reference. Satisfying one guaranteed failing + // the other: every well-formed agent pull request was armed + // against a contract that had never been written, so the gate + // was red on ~100% of pull requests, including merged ones + // (#1368, #1408). Requiring a real dispatch instead restores the + // #1130 reasoning to the arming rule that overrode it. // - // So provenance arms the gate only once a linked issue exists to - // verify against; with none, there is nothing to measure and the - // verdict is `not_applicable`. This does not create an escape - // hatch: a pull request that links a dispatched issue is still - // fully gated, and requiring a pull request to bind to a focused - // issue at all is separately owned by `Canonical issue and - // evidence`, which states a requirement an author can actually - // meet. - return login !== 'dependabot[bot]' && - (issueDispatch || (pullProvenance && Boolean(selectedIssue))); + // This is not an escape hatch: a pull request that links a + // genuinely dispatched issue is still fully gated, and requiring + // a pull request to bind to a focused issue at all is separately + // owned by `Canonical issue and evidence`, which states a + // requirement an author can actually meet. + return login !== 'dependabot[bot]' && issueDispatch; } const prNumber = Number(process.env.INPUT_PR_NUMBER || 0); diff --git a/docs/agent-completion-truth-gate.md b/docs/agent-completion-truth-gate.md index da14e3e4c..be93f9fbd 100644 --- a/docs/agent-completion-truth-gate.md +++ b/docs/agent-completion-truth-gate.md @@ -35,14 +35,18 @@ If an agent has repository-write credentials that can create Actions workflows o ## Applicability -The gate applies when any of these signals identify agent work: +The gate applies only when the PR's linked issue is a genuine dispatch — that is, when the issue both: -- a known agent bot authored the PR; -- the branch starts with agent/, claude/, codex/, copilot/, or jules/; -- the PR or linked issue has agent, agent-task, or mcp/agent; -- the PR contains an agent-lock-manifest comment. +- carries agent-task or mcp/agent (the generic agent label does not count, since neither the snapshot job nor the collector recognises it); and +- declares an agent run id and an agent login in its body. -Dependabot is exempt. Other human-authored PRs receive not_applicable. +Dependabot is exempt. Everything else receives not_applicable. + +Pull-side provenance — a known agent bot author, a branch starting with agent/, claude/, codex/, copilot/, or jules/, an agent-lock-manifest comment, or an agent label on the PR — does **not** arm the gate, on its own or against a linked issue. It identifies who produced the branch; it does not supply a contract to score that branch against. + +That distinction is load-bearing. The gate scores a PR against the frozen intent snapshot on its linked issue, and that snapshot is written only by `snapshot-agent-task-intent`, which runs on `issues` events alone and only for issues that already declare a run id and login. An ordinary issue has no snapshot, so `policy.agent_login` and `policy.run_id` cannot be populated and the verdict is permanently `invalid_payload` — with no action available to the author. Because **PR Governance** separately requires exactly one `Closes #` reference, arming on provenance-plus-any-linked-issue put the two checks in direct contradiction: satisfying one guaranteed failing the other, and this gate was red on roughly every pull request, including merged ones (#1368, #1408). + +Requiring a declared dispatch is not an escape hatch: a PR that links a genuinely dispatched issue is still fully gated, and the requirement that a PR bind to a focused issue at all is separately owned by **Canonical issue and evidence**, which states a requirement an author can actually meet. ## Input schema diff --git a/tests/unit/test_agent_completion_gate.py b/tests/unit/test_agent_completion_gate.py index c58715293..abff65739 100644 --- a/tests/unit/test_agent_completion_gate.py +++ b/tests/unit/test_agent_completion_gate.py @@ -2894,22 +2894,33 @@ def test_scheduled_scanner_detects_frozen_intent_changes(self): ['issue label without contract', base, issue(1, ['mcp/agent']), false], ['issue label with contract', base, issue(1, ['mcp/agent'], CONTRACT), true], // Pull-side provenance identifies the producer; it does not supply a - // contract to score against. With no linked issue there is no intent - // snapshot, so `policy.agent_login`, `policy.run_id` and `issue.number` - // can never be populated and the verdict is permanently `invalid_payload`. - // Each of these arms the gate only once an issue exists to verify against. + // contract to score against, with or without a linked issue. The intent + // snapshot the gate scores against is only ever written by + // `snapshot-agent-task-intent`, which runs on `issues` events alone and + // only for issues that already declare a run id and login. A plain linked + // issue has no snapshot, so `policy.agent_login` and `policy.run_id` stay + // unsatisfiable and the verdict is permanently `invalid_payload`. + // + // Arming on provenance plus *any* linked issue therefore contradicted + // `PR Governance`, which requires exactly one `Closes #`: satisfying + // it guaranteed failing this gate. None of these arm it now. ['PR label, no issue', {...base, labels: [{name: 'agent-task'}]}, null, false], - ['PR label + issue', {...base, labels: [{name: 'agent-task'}]}, PLAIN, true], + ['PR label + plain issue', + {...base, labels: [{name: 'agent-task'}]}, PLAIN, false], ['branch, no issue', {...base, head: {ref: 'codex/fix'}}, null, false], - ['branch + issue', {...base, head: {ref: 'codex/fix'}}, PLAIN, true], + ['branch + plain issue', {...base, head: {ref: 'codex/fix'}}, PLAIN, false], ['manifest, no issue', {...base, body: ''}, null, false], - ['manifest + issue', - {...base, body: ''}, PLAIN, true], + ['manifest + plain issue', + {...base, body: ''}, PLAIN, false], ['known agent, no issue', {...base, user: {login: 'google-labs-jules[bot]'}}, null, false], - ['known agent + issue', - {...base, user: {login: 'google-labs-jules[bot]'}}, PLAIN, true], + ['known agent + plain issue', + {...base, user: {login: 'google-labs-jules[bot]'}}, PLAIN, false], + // Provenance still arms the gate when the linked issue is a real dispatch. + ['known agent + dispatched issue', + {...base, user: {login: 'google-labs-jules[bot]'}}, + issue(1, ['agent-task'], CONTRACT), true], ['dependabot excluded', { ...base, user: {login: 'dependabot[bot]'}, @@ -3386,16 +3397,24 @@ def test_adapter_recognizes_agent_labels_scripts_and_blocking_threads(self): self.assertIn("!explicitlyNonBehavioral", workflow) self.assertIn("VADE-RECOMMENDATION", workflow) - def test_agent_applicability_requires_provenance_or_declared_contract(self): - """A bare `agent-task` label on a linked issue is not a dispatch. + def test_agent_applicability_requires_a_declared_dispatch_contract(self): + """Only a real issue-side dispatch arms the gate. Label automation applies `agent-task` and `mcp/agent` as topic tags to - issues that were never created from the agent task template. Before - this guard the gate unioned pull request labels with linked issue - labels, so any pull request closing such an issue was judged an agent - completion and then measured against a contract the issue had never - declared -- producing a permanent ``blocked``/``invalid_payload`` - verdict that no author could satisfy (#1130). + issues that were never created from the agent task template, so a bare + label is not a dispatch: it counts only once the issue declares the run + id and login the gate goes on to require (#1130). + + Pull-side provenance -- a known agent author, an `agent`/`claude`/ + `codex`/`jules` branch prefix, a lock manifest, or an agent label -- + does not arm the gate either, with or without a linked issue. It + identifies the producer; it does not supply a contract to score + against. Arming on provenance plus *any* linked issue contradicted + `PR Governance`, which requires exactly one ``Closes #``: + satisfying that check guaranteed failing this one, because an ordinary + issue has no intent snapshot and `policy.agent_login` / `policy.run_id` + are then unsatisfiable. That left the gate red on ~100% of pull + requests, merged ones included (#1368, #1408). """ workflow = self._workflow() @@ -3464,25 +3483,35 @@ def test_agent_applicability_requires_provenance_or_declared_contract(self): ['unlabelled issue with contract', human, { number: 7, labels: [{name: 'bug'}], body: CONTRACT }, false], - // Pull request provenance applies against a linked issue -- an agent - // producing work against a contract-less issue is still applicable, and - // therefore still blocked. - ['known agent author', { + // Pull request provenance does not arm the gate on its own, even against a + // linked issue. An agent producing work against a contract-less issue has + // no intent snapshot to be scored against, so arming here yielded a + // permanent `invalid_payload` that no author could clear: the snapshot is + // written only by `snapshot-agent-task-intent`, on `issues` events, for + // issues that already declare a run id and login. Because `PR Governance` + // separately requires exactly one `Closes #`, arming on + // provenance-plus-any-issue made the two checks mutually unsatisfiable and + // left this gate red on ~100% of pull requests, merged ones included + // (#1368, #1408). + ['known agent author, contract-less issue', { user: {login: 'google-labs-jules[bot]'}, head: {ref: 'feature/thing'}, labels: [], body: '' - }, {number: 7, labels: [{name: 'agent-task'}], body: '## Summary\n'}, true], - // ...but with no linked issue there is no intent snapshot and no declared - // run id or login, so the required payload fields are unsatisfiable and the - // gate would block permanently rather than ever reaching a verdict. These - // are `not_applicable`, not violations. + }, {number: 7, labels: [{name: 'agent-task'}], body: '## Summary\n'}, false], + // With no linked issue there is likewise no snapshot and no declared run id + // or login. These are `not_applicable`, not violations. ['agent branch prefix, no issue', { user: {login: 'groupthinking'}, head: {ref: 'jules/thing'}, labels: [], body: '' }, null, false], - ['agent branch prefix with issue', { + ['agent branch prefix, contract-less issue', { user: {login: 'groupthinking'}, head: {ref: 'jules/thing'}, labels: [], body: '' - }, {number: 7, labels: [], body: ''}, true], + }, {number: 7, labels: [], body: ''}, false], + // A genuine dispatch still arms the gate for an agent-authored branch. + ['agent branch prefix, dispatched issue', { + user: {login: 'groupthinking'}, + head: {ref: 'jules/thing'}, labels: [], body: '' + }, {number: 7, labels: [{name: 'agent-task'}], body: CONTRACT}, true], ['agent label on the pull request, no issue', { user: {login: 'groupthinking'}, head: {ref: 'feature/thing'}, From 384be14ae204456bc10ff259d27e4cf0d2238c24 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 7 Aug 2026 15:01:20 +0000 Subject: [PATCH 2/4] fix(ci): require the complete dispatch contract to arm the truth gate Addresses CodeRabbit's review finding on #1409: `issueDispatch` was weaker than the snapshot job's contract, so the previous commit still left one permanently-blocked case standing. `declaresAgentContract` checked only that the linked issue declared an agent run id and login. `snapshot-agent-task-intent` additionally requires an objective, acceptance criteria, either a declared file scope or an approved unrestricted-scope request, and a checked pre-dispatch confirmation -- and refuses to write a snapshot (`incomplete_agent_task_contract`) when any is missing. So an issue labelled `agent-task` carrying only a login and run id armed the gate while producing no snapshot, leaving the verdict permanently `missing_intent_snapshot` -> `invalid_payload`. That is the same unsatisfiable shape the previous commit removed, one level down, and it contradicted this PR's claim that the gate arms only on a genuine dispatch. `declaresAgentContract` now mirrors the snapshot job's `complete` predicate exactly, including the `scope-unrestricted-approved` label requirement when unrestricted scope is requested. The two predicates must stay in step; both now say so in a comment. Test fixtures updated: `CONTRACT` is now a complete contract, and `PARTIAL_CONTRACT` (login + run id only) is added as the negative case. New cases cover partial contract, missing acceptance criteria, unchecked pre-dispatch confirmation, and unrestricted scope both with and without the approval label. The scanner test's inline `second` issue likewise needed a complete contract to keep exercising the textual-fallback path. Verified: 112 passed, 89 subtests (unchanged baseline); YAML parses, 8/8 inline github-script blocks pass `node --check`; the extracted predicate returns the expected verdict across 12 replayed cases, including graphql-shaped labels, dependabot, and null inputs (no throws). Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_013Kc7prW6s237ERAMbnVDhH --- .github/workflows/pr-checks.yml | 82 +++++++++++++++++++--- docs/agent-completion-truth-gate.md | 4 +- tests/unit/test_agent_completion_gate.py | 88 +++++++++++++++++++++++- 3 files changed, 164 insertions(+), 10 deletions(-) diff --git a/.github/workflows/pr-checks.yml b/.github/workflows/pr-checks.yml index ca2ccd5c2..5401a7671 100644 --- a/.github/workflows/pr-checks.yml +++ b/.github/workflows/pr-checks.yml @@ -646,14 +646,47 @@ jobs: } return output.join('\n').trim(); } + const body = String((issue && issue.body) || ''); function declared(headings) { - const value = section( - String((issue && issue.body) || ''), headings - ).replace(/^\x60|\x60$/g, '').trim(); + const value = section(body, headings) + .replace(/^\x60|\x60$/g, '').trim(); return Boolean(value) && value !== '_No response_'; } + function checked(headings) { + const value = section(body, headings).trim(); + return value.split(/\r?\n/).some(line => + /^\s*[-*]\s*\[[xX]\]/.test(line) + ) || /^(?:yes|true)$/i.test(value); + } + // This must mirror the `complete` predicate in + // `snapshot-agent-task-intent`. That job refuses to write a + // snapshot (`incomplete_agent_task_contract`) unless every + // one of these holds, and a gate armed with no snapshot is + // permanently `invalid_payload`. Checking only run id and + // login armed the gate on issues the snapshot job rejects -- + // the same unsatisfiable shape this rule exists to prevent, + // one level down. + const unrestrictedRequested = + checked(['unrestricted scope', 'scope unrestricted']); + const labelSource = issue && issue.labels + ? Array.isArray(issue.labels) + ? issue.labels + : issue.labels.nodes || [] + : []; + if (unrestrictedRequested && + !carriesLabel(labelSource, + ['scopeunrestrictedapproved'])) { + return false; + } return declared(['agent run id', 'run id']) && - declared(['agent login']); + declared(['agent login']) && + declared(['objective', 'description']) && + declared(['acceptance criteria', 'acceptance tests']) && + (declared(['declared file scope', 'file scope', 'scope']) || + unrestrictedRequested) && + /-\s*\[[xX]\]/.test( + section(body, ['pre-dispatch confirmation']) + ); } const login = String( pull && pull.user && pull.user.login || '' @@ -1956,14 +1989,47 @@ jobs: } return output.join('\n').trim(); } + const body = String((issue && issue.body) || ''); function declared(headings) { - const value = section( - String((issue && issue.body) || ''), headings - ).replace(/^\x60|\x60$/g, '').trim(); + const value = section(body, headings) + .replace(/^\x60|\x60$/g, '').trim(); return Boolean(value) && value !== '_No response_'; } + function checked(headings) { + const value = section(body, headings).trim(); + return value.split(/\r?\n/).some(line => + /^\s*[-*]\s*\[[xX]\]/.test(line) + ) || /^(?:yes|true)$/i.test(value); + } + // This must mirror the `complete` predicate in + // `snapshot-agent-task-intent`. That job refuses to write a + // snapshot (`incomplete_agent_task_contract`) unless every + // one of these holds, and a gate armed with no snapshot is + // permanently `invalid_payload`. Checking only run id and + // login armed the gate on issues the snapshot job rejects -- + // the same unsatisfiable shape this rule exists to prevent, + // one level down. + const unrestrictedRequested = + checked(['unrestricted scope', 'scope unrestricted']); + const labelSource = issue && issue.labels + ? Array.isArray(issue.labels) + ? issue.labels + : issue.labels.nodes || [] + : []; + if (unrestrictedRequested && + !carriesLabel(labelSource, + ['scopeunrestrictedapproved'])) { + return false; + } return declared(['agent run id', 'run id']) && - declared(['agent login']); + declared(['agent login']) && + declared(['objective', 'description']) && + declared(['acceptance criteria', 'acceptance tests']) && + (declared(['declared file scope', 'file scope', 'scope']) || + unrestrictedRequested) && + /-\s*\[[xX]\]/.test( + section(body, ['pre-dispatch confirmation']) + ); } const login = String( pull && pull.user && pull.user.login || '' diff --git a/docs/agent-completion-truth-gate.md b/docs/agent-completion-truth-gate.md index be93f9fbd..211171d85 100644 --- a/docs/agent-completion-truth-gate.md +++ b/docs/agent-completion-truth-gate.md @@ -38,7 +38,9 @@ If an agent has repository-write credentials that can create Actions workflows o The gate applies only when the PR's linked issue is a genuine dispatch — that is, when the issue both: - carries agent-task or mcp/agent (the generic agent label does not count, since neither the snapshot job nor the collector recognises it); and -- declares an agent run id and an agent login in its body. +- declares the **complete** dispatch contract in its body: agent login, agent run id, objective/description, acceptance criteria, either a declared file scope or an unrestricted-scope request carrying the scope-unrestricted-approved label, and a checked pre-dispatch confirmation. + +That list is deliberately identical to the `complete` predicate in `snapshot-agent-task-intent`. The two must stay in step: the snapshot job refuses to write a snapshot (`incomplete_agent_task_contract`) unless every element is present, and a gate armed with no snapshot is permanently `invalid_payload`. Requiring only the login and run id armed the gate on issues the snapshot job rejects — the same unsatisfiable shape described below, one level down. Dependabot is exempt. Everything else receives not_applicable. diff --git a/tests/unit/test_agent_completion_gate.py b/tests/unit/test_agent_completion_gate.py index abff65739..ecc38fd15 100644 --- a/tests/unit/test_agent_completion_gate.py +++ b/tests/unit/test_agent_completion_gate.py @@ -2870,7 +2870,20 @@ def test_scheduled_scanner_detects_frozen_intent_changes(self): self.assertEqual(len(applicable_functions), 2) self.assertEqual(applicable_functions[0], applicable_functions[1]) applicable_assertions = r""" +// A complete contract, matching what `snapshot-agent-task-intent` demands +// before it will write a snapshot. Anything less produces no snapshot, so +// arming on it would be permanently unsatisfiable. const CONTRACT = [ + '### Agent Login', '', '`google-labs-jules[bot]`', '', + '### Agent Run ID', '', '`run-42`', '', + '### Objective', '', 'Fix the thing.', '', + '### Acceptance Criteria', '', '- It is fixed.', '', + '### Declared File Scope', '', '`src/thing.py`', '', + '### Pre-dispatch Confirmation', '', '- [x] Confirmed' +].join('\n'); +// Login + run id only -- the old `declaresAgentContract` accepted this, but +// the snapshot job rejects it as `incomplete_agent_task_contract`. +const PARTIAL_CONTRACT = [ '### Agent Login', '', '`google-labs-jules[bot]`', '', '### Agent Run ID', '', '`run-42`' ].join('\n'); @@ -2893,6 +2906,11 @@ def test_scheduled_scanner_detects_frozen_intent_changes(self): // actually declares the contract the gate goes on to require (#1130). ['issue label without contract', base, issue(1, ['mcp/agent']), false], ['issue label with contract', base, issue(1, ['mcp/agent'], CONTRACT), true], + // The contract must be the complete one the snapshot job requires; a + // login+run-id-only body yields no snapshot, so arming would be + // permanently unsatisfiable. + ['issue label with partial contract', base, + issue(1, ['mcp/agent'], PARTIAL_CONTRACT), false], // Pull-side provenance identifies the producer; it does not supply a // contract to score against, with or without a linked issue. The intent // snapshot the gate scores against is only ever written by @@ -3000,11 +3018,17 @@ def test_scheduled_scanner_detects_frozen_intent_changes(self): combined_assertions = r""" const first = {number: 1, labels: {nodes: []}, body: ''}; +// A complete dispatch contract -- login and run id alone are rejected by +// `snapshot-agent-task-intent`, so they would not arm the gate. const second = { number: 2, labels: {nodes: [{name: 'agent-task'}]}, body: '### Agent Login\n\n`google-labs-jules[bot]`\n\n' + - '### Agent Run ID\n\n`run-42`' + '### Agent Run ID\n\n`run-42`\n\n' + + '### Objective\n\nFix the thing.\n\n' + + '### Acceptance Criteria\n\n- It is fixed.\n\n' + + '### Declared File Scope\n\n`src/thing.py`\n\n' + + '### Pre-dispatch Confirmation\n\n- [x] Confirmed' }; const multi = { user: {login: 'maintainer'}, @@ -3425,7 +3449,37 @@ def test_agent_applicability_requires_a_declared_dispatch_contract(self): self.assertEqual(len(applicable), 2) assertions = r""" +// A complete contract, matching what `snapshot-agent-task-intent` demands +// before it will write a snapshot. Anything less produces no snapshot, so +// arming on it would be permanently unsatisfiable. const CONTRACT = [ + '### Agent Login', + '', + '`google-labs-jules[bot]`', + '', + '### Agent Run ID', + '', + '`run-42`', + '', + '### Objective', + '', + 'Fix the thing.', + '', + '### Acceptance Criteria', + '', + '- It is fixed.', + '', + '### Declared File Scope', + '', + '`src/thing.py`', + '', + '### Pre-dispatch Confirmation', + '', + '- [x] Confirmed' +].join('\n'); +// Login + run id only -- the old `declaresAgentContract` accepted this, but +// the snapshot job rejects it as `incomplete_agent_task_contract`. +const PARTIAL_CONTRACT = [ '### Agent Login', '', '`google-labs-jules[bot]`', @@ -3434,6 +3488,10 @@ def test_agent_applicability_requires_a_declared_dispatch_contract(self): '', '`run-42`' ].join('\n'); +const UNRESTRICTED = CONTRACT.replace( + '### Declared File Scope\n\n`src/thing.py`', + '### Unrestricted Scope\n\n- [x] Requested' +); const human = { user: {login: 'groupthinking'}, head: {ref: 'feature/thing'}, @@ -3457,6 +3515,34 @@ def test_agent_applicability_requires_a_declared_dispatch_contract(self): ['declared contract', human, { number: 7, labels: [{name: 'agent-task'}], body: CONTRACT }, true], + // ...but the contract must be the *complete* one the snapshot job demands. + // A login+run-id-only body is rejected there as + // `incomplete_agent_task_contract`, so no snapshot is written and arming on + // it would be unsatisfiable in exactly the way this rule exists to prevent. + ['partial contract, login and run id only', human, { + number: 7, labels: [{name: 'agent-task'}], body: PARTIAL_CONTRACT + }, false], + ['missing acceptance criteria', human, { + number: 7, + labels: [{name: 'agent-task'}], + body: CONTRACT.replace('### Acceptance Criteria\n\n- It is fixed.', '') + }, false], + ['unconfirmed pre-dispatch checkbox', human, { + number: 7, + labels: [{name: 'agent-task'}], + body: CONTRACT.replace('- [x] Confirmed', '- [ ] Confirmed') + }, false], + // Unrestricted scope substitutes for a declared file scope, but only once a + // maintainer has applied scope-unrestricted-approved -- the same condition + // the snapshot job enforces before writing. + ['unrestricted scope, unapproved', human, { + number: 7, labels: [{name: 'agent-task'}], body: UNRESTRICTED + }, false], + ['unrestricted scope, approved', human, { + number: 7, + labels: [{name: 'agent-task'}, {name: 'scope-unrestricted-approved'}], + body: UNRESTRICTED + }, true], ['contract via graphql label nodes', human, { number: 7, labels: {nodes: [{name: 'agent-task'}]}, body: CONTRACT }, true], From f14ea74d24edff42a7a4f15b2248e2ce974a379a Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 7 Aug 2026 15:02:21 +0000 Subject: [PATCH 3/4] docs: name the dispatch labels exactly in the truth-gate applicability rule From CodeRabbit's review of #1409. It asked for precise label names, which was fair -- unbackticked prose could not tell an operator which string to apply. It also asked to replace `mcp/agent` with `mcp-agent`. That part is wrong for this repository: `mcp-agent` does not exist (the API returns 404) and `mcp/agent` does. Label matching lowercases and strips non-alphanumerics, so both spellings normalise to the same key and either would work in code, but documenting a label nobody can apply would be a regression. The doc now names the real labels in backticks and explains the normalisation so the next reader does not have to re-derive it. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_013Kc7prW6s237ERAMbnVDhH --- docs/agent-completion-truth-gate.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/agent-completion-truth-gate.md b/docs/agent-completion-truth-gate.md index 211171d85..84cc0080d 100644 --- a/docs/agent-completion-truth-gate.md +++ b/docs/agent-completion-truth-gate.md @@ -37,7 +37,7 @@ If an agent has repository-write credentials that can create Actions workflows o The gate applies only when the PR's linked issue is a genuine dispatch — that is, when the issue both: -- carries agent-task or mcp/agent (the generic agent label does not count, since neither the snapshot job nor the collector recognises it); and +- carries `agent-task` or `mcp/agent` — those exact label names, as they exist in this repository. Label matching lowercases and strips non-alphanumerics, so `mcp/agent` and `mcp-agent` both normalise to the same key, but only `mcp/agent` is a real label here. The generic `agent` label does **not** count: neither the snapshot job nor the collector recognises it, so arming on it would be unsatisfiable; and - declares the **complete** dispatch contract in its body: agent login, agent run id, objective/description, acceptance criteria, either a declared file scope or an unrestricted-scope request carrying the scope-unrestricted-approved label, and a checked pre-dispatch confirmation. That list is deliberately identical to the `complete` predicate in `snapshot-agent-task-intent`. The two must stay in step: the snapshot job refuses to write a snapshot (`incomplete_agent_task_contract`) unless every element is present, and a gate armed with no snapshot is permanently `invalid_payload`. Requiring only the login and run id armed the gate on issues the snapshot job rejects — the same unsatisfiable shape described below, one level down. From 0be4d0aacfe24e67d0214f5df1e6ca4e70dabeb8 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 7 Aug 2026 15:06:08 +0000 Subject: [PATCH 4/4] fix(ci): align the snapshot placeholder test with the gate's Second CodeRabbit finding on #1409, and a real one in the opposite direction from the first: a gate bypass rather than a permanent block. Issue forms render an unfilled field as `_No response_`, and authors routinely wrap values in backticks, so a field left unfilled inside a code span arrives as '`_No response_`'. The two predicates disagreed on it: value snapshot gate '`_No response_`' true false '``' true false `declaresAgentContract`'s `declared` strips outer backticks before the placeholder test; `snapshot-agent-task-intent`'s `hasResponse` did not, so it read a backticked placeholder as a real answer. The snapshot was written while the gate stayed `not_applicable` -- a malformed dispatch skipping the check entirely. `hasResponse` now applies the same normalisation, so both reject it. That is the correct side to move: a backticked `_No response_` is an unfilled field whichever predicate reads it. No legitimate dispatch is affected -- a value only becomes empty under the strip if it was nothing but backticks. Adds `test_snapshot_and_arming_predicates_agree_on_placeholders`, which pins `hasResponse` against the gate's normalisation across seven inputs and asserts end to end, against both copies of `agentTaskApplicable`, that a backticked placeholder login or an empty-backtick run id does not arm. Verified: 113 passed, 89 subtests; YAML parses; 8/8 inline github-script blocks pass `node --check`; the two predicates now agree on every probed input, and the 12-case arming matrix is unchanged. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_013Kc7prW6s237ERAMbnVDhH --- .github/workflows/pr-checks.yml | 12 ++- tests/unit/test_agent_completion_gate.py | 112 +++++++++++++++++++++++ 2 files changed, 123 insertions(+), 1 deletion(-) diff --git a/.github/workflows/pr-checks.yml b/.github/workflows/pr-checks.yml index 5401a7671..f7611a77e 100644 --- a/.github/workflows/pr-checks.yml +++ b/.github/workflows/pr-checks.yml @@ -196,8 +196,18 @@ jobs: } return output.join('\n').trim(); } + // Strips outer backticks before the placeholder test, matching + // `declared` in `agentTaskApplicable`. Issue forms render an + // unfilled field as `_No response_`, and authors routinely wrap + // values in backticks, so a field left unfilled inside a code span + // arrives as '`_No response_`'. Without the strip that reads as a + // real answer here while the gate's predicate rejects it -- the + // snapshot would be written but the gate would never arm, letting + // a malformed dispatch skip the check entirely. Both predicates + // must agree on every input. function hasResponse(value) { - const response = String(value || '').trim(); + const response = String(value || '') + .replace(/^\x60|\x60$/g, '').trim(); return Boolean(response && response !== '_No response_'); } function checkboxChecked(value) { diff --git a/tests/unit/test_agent_completion_gate.py b/tests/unit/test_agent_completion_gate.py index ecc38fd15..8e825b4fe 100644 --- a/tests/unit/test_agent_completion_gate.py +++ b/tests/unit/test_agent_completion_gate.py @@ -3635,6 +3635,118 @@ def test_agent_applicability_requires_a_declared_dispatch_contract(self): ) self.assertEqual(completed.returncode, 0, completed.stderr) + def test_snapshot_and_arming_predicates_agree_on_placeholders(self): + """The snapshot job and the gate must accept exactly the same fields. + + `snapshot-agent-task-intent` decides whether to write an intent + snapshot; `agentTaskApplicable` decides whether to demand one. A field + either predicate reads differently opens a hole in one direction or the + other. + + Issue forms render an unfilled field as ``_No response_``, and authors + routinely wrap values in backticks, so an unfilled field inside a code + span arrives as ``` `_No response_` ```. The gate's ``declared`` strips + outer backticks before the placeholder test; the snapshot job's + ``hasResponse`` originally did not, so it read that as a real answer. + The snapshot was written while the gate stayed ``not_applicable`` -- + a malformed dispatch skipping the check entirely, the mirror image of + the permanent block fixed alongside it. + """ + + workflow = self._workflow() + has_response = _javascript_functions(workflow, "function hasResponse(") + self.assertEqual(len(has_response), 1) + applicable = _javascript_functions( + workflow, + "function agentTaskApplicable(", + ) + self.assertEqual(len(applicable), 2) + + assertions = r""" +function declared(value) { + const v = String(value || '').replace(/^\x60|\x60$/g, '').trim(); + return Boolean(v) && v !== '_No response_'; +} +const rows = [ + ['`_No response_`', false], + ['``', false], + ['_No response_', false], + ['', false], + [' ', false], + ['`run-42`', true], + ['run-42', true] +]; +for (const [value, expected] of rows) { + if (hasResponse(value) !== expected) { + throw new Error( + `snapshot hasResponse(${JSON.stringify(value)}) === ` + + `${hasResponse(value)}, expected ${expected}` + ); + } + if (declared(value) !== hasResponse(value)) { + throw new Error( + `predicates disagree on ${JSON.stringify(value)}: ` + + `snapshot=${hasResponse(value)} gate=${declared(value)}` + ); + } +} +""" + completed = subprocess.run( + ["node", "-e", has_response[0] + assertions], + check=False, + capture_output=True, + text=True, + ) + self.assertEqual(completed.returncode, 0, completed.stderr) + + # ...and end to end: a backticked placeholder must not arm the gate, + # matching the snapshot job's refusal to write for the same issue. + gate_assertions = r""" +const FIELDS = [ + ['### Agent Login', '`x[bot]`'], + ['### Agent Run ID', '`r-1`'], + ['### Objective', 'Do it.'], + ['### Acceptance Criteria', '- Done.'], + ['### Declared File Scope', '`src/a.py`'], + ['### Pre-dispatch Confirmation', '- [x] yes'] +]; +function body(overrides) { + return FIELDS.map(([heading, value]) => + heading + '\n\n' + (overrides[heading] || value) + ).join('\n\n'); +} +const pull = { + user: {login: 'someone'}, + head: {ref: 'claude/x'}, + labels: [], + body: '' +}; +const issue = text => ({ + number: 7, labels: [{name: 'agent-task'}], body: text +}); +const rows = [ + ['complete', body({}), true], + ['backticked placeholder login', + body({'### Agent Login': '`_No response_`'}), false], + ['empty backticks run id', + body({'### Agent Run ID': '``'}), false] +]; +for (const [name, text, expected] of rows) { + const actual = agentTaskApplicable(pull, issue(text)); + if (actual !== expected) { + throw new Error(`${name}: ${actual} !== ${expected}`); + } +} +""" + for copy in applicable: + completed = subprocess.run( + ["node", "-e", copy + gate_assertions], + check=False, + capture_output=True, + text=True, + ) + self.assertEqual(completed.returncode, 0, completed.stderr) + def test_agent_applicability_copies_stay_identical(self): """The scheduled sweep and the per-pull-request collector must agree.