fix(ci): report Agent Lock gate as neutral when trust policy is unprovisioned - #1151
Conversation
…visioned
The "Agent completion enforcement" check currently fails on 100% of open
pull requests. It is not detecting policy violations - it is reporting that
its own trust infrastructure was never provisioned:
1. The workflow greps head-SHA check runs for a check named
"Agent Lock trusted publication". Nothing in this repository publishes
that check; the string occurs in exactly one place, the consumer itself.
Every PR therefore short-circuits to `missing_trusted_publication`.
2. Even with a publisher, .github/agent-lock/trusted-publishers.json has
all three allowlists empty with `custom_role_policy: "fail_closed"`, so
agent_completion_enforcement.py returns `trust_policy_unprovisioned`.
Both are repository-infrastructure states, not defects in any pull request,
but both were mapped to `failure`. The result is a permanently red check that
trains everyone to ignore CI, hiding genuine `build`, `test`, `Vercel` and
`trivy` failures behind constant noise.
This maps those two reasons to `neutral` and emits an explanatory warning.
This does not weaken enforcement:
* genuine violations (untrusted_publisher, head_sha_mismatch,
pull_request_mismatch, verifier_did_not_publish) still report `failure`
and still call core.setFailed;
* `main` has no required_status_checks, so a red result blocks no merge
today in any case - the gate protects nothing it is not already failing
to protect.
Once a trusted publisher App exists and trusted-publishers.json is populated,
these reasons stop occurring and the gate becomes live. It should be added to
branch protection required checks at that point.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Important Review skippedAuto reviews are limited based on label configuration. 🏷️ Required labels (at least one) (1)
Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Repository YAML (base), Repository UI (inherited), Organization UI (inherited) Review profile: ASSERTIVE Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Dependency Review✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.Snapshot WarningsEnsure that dependencies are being submitted on PR branches and consider enabling retry-on-snapshot-warnings. See the documentation for more information and troubleshooting advice. Scanned FilesNone |
Agent Completion Truth Gate: BLOCKEDReasons: Machine-readable verdict{
"details": {
"invalid_fields": [
"policy.agent_login",
"policy.run_id"
]
},
"reasons": [
"invalid_payload"
],
"verdict": "blocked"
} |
There was a problem hiding this comment.
Pull request overview
Changes Agent Lock’s unprovisioned states from failure to neutral.
Changes:
- Adds neutral handling and warnings for unprovisioned trust infrastructure.
- Preserves failures for other enforcement verdicts.
Suppressed comments (1)
.github/workflows/agent-completion-enforcement.yml:93
missing_trusted_publicationis not proof that the infrastructure is unprovisioned. After the App and allowlists are configured, an App outage, publication race, or absent report produces the same reason; this workflow has nocheck_runtrigger to reevaluate when a late publication arrives. Marking that reason neutral unconditionally therefore lets a required gate pass without any trusted evidence. Only downgrade it when an authoritative enabled/provisioned state says the gate is intentionally inactive; once enabled, missing publication must remain fail-closed.
const UNPROVISIONED_REASONS = new Set([
'missing_trusted_publication',
'trust_policy_unprovisioned'
]);
const unprovisioned = UNPROVISIONED_REASONS.has(verdict.reason);
|
Automated shepherd — CI triage (head Substantive signal is green. The 5 red/neutral checks are all non-required infra/governance noise, none caused by this single-file workflow diff. Since this PR exists precisely because phantom reds get mistaken for real ones, here is the breakdown so the merge decision isn't scared off by them: Green (the real signal): Red — none block merge (
Terminal state: HALTED(awaiting_merge_approval). Ready on the merits; holding merge for maintainer sign-off (unwatched automated run, default branch, no Unrelated follow-up worth a separate PR: the Generated by Claude Code |
…visioned; sync docs Addresses the two confirmed review findings on this PR. 1. Guard the neutral downgrade on trust-infrastructure signal, not the reason string alone. `trust_policy_unprovisioned` is derived from the policy file's own empty-allowlist state and stays advisory (neutral). But `missing_trusted_publication` merely means no trusted report was published: expected while unprovisioned, yet once the App is provisioned it is a real failure mode (App outage, publication race, late/absent check-run) and must stay fail-closed. The workflow now reads the protected default-branch trusted-publishers.json and only downgrades a missing report to neutral when all three allowlists are empty -- mirroring the emptiness gate in scripts/ci/agent_completion_enforcement.py. 2. Sync operator docs that still promised unconditional fail-closed: docs/agent-completion-truth-gate.md, .github/workflows/README.md, and .github/workflows/AUDIT.md now state the gate is neutral (advisory) while unprovisioned and returns to fail-closed once the App + allowlists are provisioned and the check is added to branch-protection required checks. Validated: workflow YAML parses; embedded github-script passes node --check; decision table verified across all verdict shapes in both policy states (provisioned + missing report -> failure; genuine violations -> failure). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HmUaMNhiJ1iQAMcQSuLu2G
Independent red-team + CI triage @
|
| Check | Result | Why it's not a blocker |
|---|---|---|
| Agent completion enforcement | ❌ failure | Phantom — the workflow runs base-branch code, so this PR's own fix can't self-apply on its own PR. Exactly the red this PR removes for future PRs; resolves once merged. |
| gitleaks (working tree) | ❌ failure | Pre-existing false positive: square-access-token rule matching a sha256: package hash at uv.lock:5129 — a file this PR never touches. Fires on every PR. |
| PR Governance / Canonical issue and evidence | ❌ failure | Agent-governance meta-checks; not required status checks on main. |
| CodeQL, bandit, trivy, npm-audit, python-safety, dependency-review, Security Scan (js/py), build, test, lint | ✅ | All real security/build gates pass. |
Terminal state: awaiting your merge decision. Not merging autonomously — there's no automerge label and merging to main is a human sign-off step. Staged command for whenever you're ready:
gh pr merge 1151 --repo groupthinking/EventRelay --squash
Reminder from the PR's own "Follow-up required": this restores signal but does not make the gate live — that still needs the publisher App + populated allowlists + adding Agent completion enforcement to branch-protection required checks.
Generated by Claude Code
|
Blocker-watch containment: this PR was returned to draft at exact |
- Correct the #1118 claim: it is CONFLICTING/DIRTY (needs rebase), under security review, possibly closeable as obsolete — not green, not fast-trackable. - Replace fast-track group A with an evaluate-after-rebase methodology: green/red is not a usable signal until PRs are rebased past the #1151/#1142 gate fixes; require >=1 green required check (build/test/Coverage/validate-gh-aw). - Strengthen the loop finding (46/50 open PRs are drafts; generation-rate problem) and adopt the owner's preferred remediation: write to an issue/workflow summary, pause until drained. Does not auto-close #1044/#1059 (owner's call). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016hdiTXBJUUCgw9QX23tp6G
Canonical issue
Closes #1160
Agent completion enforcementreportsfailureon 100% of open pull requests. Two independent repository-infrastructure states — neither of which is a defect in any PR — are mapped to a hard failure:Agent Lock trusted publication. Nothing in this repository ever publishes that check; the string occurs in exactly one place, the consumer itself. Every PR therefore short-circuits tomissing_trusted_publication..github/agent-lock/trusted-publishers.jsonshipstrusted_check_app_slugs,trusted_label_actors, andtrusted_human_exemption_actorsas[]withcustom_role_policy: "fail_closed", soagent_completion_enforcement.pyreturnstrust_policy_unprovisioned.The result is a permanently red check that carries no information. It is actively harmful: triage run #1128 read the uniform red as uniform risk, concluded PRs #999–#1008 were a "human fast-path… only need a maintainer to mark ready + merge," and would have merged broken builds. #999 fails
Vercel,build, andvalidate-gh-aw; #1000 failsCoverage,Vercel,build,test, andvalidate-gh-aw; #1003 failsbuildandVercel. A gate that is red for everything hides the PRs that are red for a reason.Outcome
The gate distinguishes "this repository has not provisioned a trust policy" from "this PR violated the trust policy."
missing_trusted_publicationandtrust_policy_unprovisioned→neutralconclusion pluscore.warning. The unprovisioned state stays visible without being fatal.untrusted_publisher,head_sha_mismatch,pull_request_mismatch, andverifier_did_not_publish→ unchangedfailurepluscore.setFailed.Once merged, a red
Agent completion enforcementmeans a real violation, and the pre-existing genuine CI failures across the backlog become visible instead of being camouflaged.Change
Adds a
trustPolicyProvisionedboolean derived from.github/agent-lock/trusted-publishers.json, requiring all three allowlists to be non-empty. Only when the policy is unprovisioned are the two infrastructure reasons downgraded toneutral.Files:
.github/workflows/agent-completion-enforcement.yml.github/workflows/AUDIT.md.github/workflows/README.mddocs/agent-completion-truth-gate.mdRisk
Low. This narrows the failure surface only for states that cannot currently be satisfied by any PR.
mainhas norequired_status_checks(verified viaGET /repos/groupthinking/EventRelay/branches/main/protection), so this check blocks no merge today in either state. Merging cannot loosen a control that is not currently enforcing.Verification
0c06052f069f0082006915a8dddd00b9b77e3801agent-completion/truth-gate: passed (not_applicable: all rules passed)build: passedCodeQL: passedbandit: passedSecurity Scan - python: passedSecurity Scan - javascript: passedGenerate and Upload Coverage: passeddependency-review: passedguards: passedVercel: passedAgent completion enforcement(self-referential; the bug being fixed) andgitleaks (working tree)(theuv.lockfalse positive addressed by fix(security): scope uv.lock checksum false-positive allowlist (#1141) #1142).Production evidence
Not applicable. The change is confined to CI workflow logic and documentation. No runtime code, dependency, schema, or deployed artifact is touched, so there is no production surface to observe.
Post-merge evidence is the check conclusion itself:
Agent completion enforcementreportsneutralwith a warning on PRs where the trust policy is unprovisioned, and continues to reportfailureon any PR that trips a genuine violation reason.