fix(ci): route every invalid_payload verdict through a shared builder - #1357
fix(ci): route every invalid_payload verdict through a shared builder#1357groupthinking wants to merge 3 commits into
Conversation
…verdicts The agent-completion truth gate blocks ~47 of the 69 open PRs with a bare `invalid_payload` and no remediation path. Root cause: `agentTaskApplicable()` in pr-checks.yml classifies any branch matching /^(?:agent|claude|codex|copilot|jules)[\/-]/ as agent work, so human-authored Claude Code worktree branches are held to the full AgentTask provenance contract. With no linked AgentTask issue, the collector emits `policy.agent_login` and `policy.run_id` as null and records the real reasons in `collection_errors` (missing_linked_issue, missing_agent_login, missing_agent_run_id). `evaluate()` then returned at the schema check and discarded `collection_errors` entirely -- they are only read further down, after the early return. Authors saw `invalid_payload` and nothing else. This keeps the gate fail-closed and byte-identical in `verdict` and `reasons`, and only adds `details.collection_errors` so the gate says what to fix. Verified: 112 passed against the reproduced PR #1270 payload. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The collection_errors diagnostic was only attached to the late
field-validation return. The three early invalid_payload short-circuits
(payload not a dict, policy not a dict, missing/invalid policy.applicable)
returned bare verdicts, so a malformed payload that never reaches field
validation stayed just as opaque despite the collector having already
recorded why — exactly the case the review thread raised
(evaluate({"policy": {}, "collection_errors": [...]})).
Extract _invalid_payload(payload, invalid_fields) and route all four
invalid_payload returns through it so the diagnostic is applied
consistently. verdict and reasons stay byte-identical for every input;
only details is enriched, and only when the collector recorded errors —
the gate remains fail-closed.
Add regression coverage for both early paths and confirm a non-dict
payload still returns an empty details.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011L6jqdhrKTYLinTnJYKEg9
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Important Review skippedAuto reviews are limited based on label configuration. 🏷️ Required labels (at least one) (1)
Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Repository YAML (base), Repository UI (inherited), Organization UI (inherited) Review profile: ASSERTIVE Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Dependency Review✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.Snapshot WarningsEnsure that dependencies are being submitted on PR branches and consider enabling retry-on-snapshot-warnings. See the documentation for more information and troubleshooting advice. Scanned FilesNone |
Agent Completion Truth Gate: BLOCKEDReasons: Machine-readable verdict{
"details": {
"collection_errors": [
"incomplete_linked_issue_contract",
"missing_linked_issue",
"missing_closing_issue_reference",
"missing_agent_run_id",
"missing_agent_login"
],
"invalid_fields": [
"issue.number",
"policy.agent_login",
"policy.run_id"
]
},
"reasons": [
"invalid_payload"
],
"verdict": "blocked"
} |
|
Closing as a duplicate of the canonical open PR #1331 ( Why this PR is redundant and being withdrawn rather than driven to green:
Focused tests pass locally on these commits ( Generated by Claude Code |
|
Re-closing — this PR was reopened by an external actor (not a request I received), but nothing has changed that alters the earlier determination: it's a duplicate of the canonical open PR #1331 ( On the Deferring to #1331 for the merge. If you intentionally reopened this and want it kept open in preference to #1331, reopen and say so and I'll follow that instead. Generated by Claude Code |
Canonical issue
Closes #
Outcome
The agent completion gate (
scripts/ci/agent_completion_gate.py) now surfaces the collector's owncollection_errorson everyinvalid_payloadverdict, not only on the late field-validation path. When a payload is rejected early (non-dict payload, missing/invalidpolicy, non-boolpolicy.applicable), the author now sees the collector diagnostics that explain why, instead of an emptydetailsobject. The gate stays fail-closed —verdictandreasonsare byte-identical for every input; onlydetailsis enriched.Scope
_invalid_payload(payload, invalid_fields)helper that builds a single, consistentinvalid_payloadverdict and folds in surfacedcollection_errors._collection_errors(payload)helper (extracted from the inline comprehension) so both the happy path and everyinvalid_payloadpath share one implementation.invalid_payloadreturn sites route through the shared builder.verdict/reasonsand surfaces collection errors when present.not_applicable/evidence_collection_failedpaths.Risk
details;verdict/reasonsare covered by tests asserting byte-identical output, so a regression cannot flip the gate open or closed.detailson early rejects.Verification
Tied to head SHA
5a034b77182e04c227f09b18a481fe364233acec.PYTHONPATH=src pytest tests/unit/test_agent_completion_gate.py→ 109 passed, 89 subtests passedNote:
scripts/is outside the CI ruff scope (ci.ymllints onlysrc/youtube_extension/backend/andmain.py); the new helpers follow the file's existingDict/Listtyping convention.Production evidence
Not applicable — this is a CI-gate script change with no runtime or deployment surface.
Agent handoff
Human sign-off is requested at the merge gate: base branch
mainis protected, so this draft is left for a human to mark ready and merge.Generated by Claude Code