fix(ci): add concurrency groups so the Actions queue can drain - #1510
Conversation
The repository's Actions queue is saturated and not draining: 285 runs queued, the oldest sitting unstarted for ~24 minutes with only 2 runs in progress. Pushes to `main` are affected as well as PRs, so no PR can reach a green CI gate and nothing is mergeable. Cause: 17 of 30 workflows carried no `concurrency:` block, including the heaviest and most-triggered ones (ci, codeql-analysis, security, e2e-tests, pr-checks, auto-label, dependency-review). #1447 correctly removed their base-branch filters so every PR runs the gate, but without a concurrency group each push stacks a brand-new full run beside the runs it obsoletes. With ~30 workflows and dozens of open PRs the arrival rate exceeds the drain rate and the backlog grows without bound. Add a concurrency group to the 14 workflows that trigger on pull_request, pull_request_target, push, or issues: - Group is keyed on the PR number where one exists, falling back to github.ref. Keying on github.ref alone would be wrong for the pull_request_target workflows (pr-checks, dependabot-auto-merge), where github.ref resolves to the base branch and every PR would collide into a single group and cancel its siblings. - cancel-in-progress is true only for pull_request and pull_request_target events. Push and schedule runs queue rather than cancel, so each merged commit keeps its own verdict instead of being cancelled by the next merge. - auto-label triggers on both PRs and issues, so its key falls through pull_request.number -> issue.number -> github.ref. Left untouched: bulk-issue-processor, real-processing, and stale, which are workflow_dispatch- or schedule-only and are not queue pressure. Group names are prefixed by workflow slug and do not collide with the existing groups in coverage, secret-scan, pr-governance, verification, phase-goal-tracker, or the gh-aw lock files. This changes scheduling only. No job, step, trigger, or permission is modified; all 28 workflow files still parse. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FoRdXUHXgtt3U6HiHdcosV
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Important Review skippedAuto reviews are limited based on label configuration. 🏷️ Required labels (at least one) (1)
Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Repository YAML (base), Repository UI (inherited), Organization UI (inherited) Review profile: ASSERTIVE Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
groupthinking
added
ci/cd
github_actions
Claude
|
@coderabbitai full review Labels added ( Reviewer context: this is a scheduling-only change — 14 workflows each gain a Generated by Claude Code |
|
|
Kept as canonical over #1511 (gate 6), with one defect to fix first#1511 was the same change from a concurrent session, five seconds apart. Closed in favour of this one — 14 workflows vs 7, and this PR's key handles the This PR is the unblock for the whole repo right now, so it is worth getting right. Two things before it lands. 1. Defect —
|
…dicts Self-review correction to the previous commit, plus the same fix applied to two workflows that already had the defect. The previous commit keyed every group on `github.event.pull_request.number || github.ref` and set `cancel-in-progress` false for push events, on the reasoning that push runs would queue rather than cancel and so "each merged commit keeps its own verdict". That claim is wrong. `cancel-in-progress: false` prevents a new run from cancelling a *running* one. It does not prevent GitHub from cancelling a *pending* one: when a run is queued into a group that already has one in progress and one pending, the pending run is cancelled. Because `github.ref` is `refs/heads/main` for every merge, all main pushes shared one group, so a burst of merges -- exactly what this repository does, ten in twenty minutes -- would have silently dropped the middle commits' verdicts. That is a quieter version of the failure being fixed. Key non-PR runs on `github.sha` instead. Each push and scheduled run then lands in its own singleton group and can neither cancel nor be cancelled. This costs nothing: every `push:` trigger in these workflows is already filtered to `main` (verified across all 13), so pushes to PR branches fire no push runs at all and all queue pressure comes from `pull_request`/`pull_request_target` events, which still cancel their predecessors by PR number. The conditional `cancel-in-progress` is kept as a second, explicit safeguard. Also apply the same key to coverage.yml and secret-scan.yml. Both predate this change and both carried the bare `github.ref` group with `cancel-in-progress: true` -- the more aggressive form of the same bug, where a merge to main cancels the previous merge's coverage and secret scan outright. Leaving them as-is while documenting the hazard in fourteen other files would have been incoherent. Issue-triggered workflows (auto-assign, issue-triage, auto-label) keep their `github.event.issue.number` key; `github.sha` is meaningless for issues events and would collide every issue into one group. Scheduling only. No job, step, trigger, or permission changed; all 28 workflow files parse. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FoRdXUHXgtt3U6HiHdcosV
![vercel[bot]](https://avatars.githubusercontent.com/in/8329?s=60&v=4){kind=small}
Review finding from the Vercel review bot on #1510, confirmed and applied. dependabot-auto-merge is the one workflow here that triggers on check_suite, and on that event GITHUB_SHA is the tip of the default branch rather than the commit under test. It is therefore identical for every PR, so the `|| github.sha` fallback collapsed all concurrent check-suite runs into a single group. Since cancel-in-progress is false for that event, they would not cancel a running peer -- but GitHub still cancels the *pending* run in a group, so one PR's merge-gate verdict could be dropped by another PR's check suite arriving. Fall back to github.event.check_suite.head_sha first, which is the PR head and keeps the runs isolated, before github.sha. Audited the other events reached by these groups for the same class of degeneracy: - issues (auto-assign, auto-label, issue-triage) already key on github.event.issue.number, which is unique. - schedule (codeql-analysis, security) resolves github.sha to the default branch tip, which can coincide with that same workflow's push run at the same commit. That collision is between two runs of the same scan over the same tree, so collapsing them is harmless dedup rather than a lost verdict. check_suite was the only real case. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FoRdXUHXgtt3U6HiHdcosV
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
|
Unplanned live validation of this change, worth recording — and it comes with a caveat reviewers should know before merging. Two events landed on head
The That is the mechanism this PR is built on, observed rather than argued. The caveat: it also shows the fix is partially self-testing and not self-deploying. Queue depth meanwhile: 285 → 256 → 236 → 207 since 21:13. Still draining on recovered capacity; this PR's own required checks remain queued behind it, which is why the CI box above is still unchecked. Generated by Claude Code |
Dependency Review✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.Snapshot WarningsEnsure that dependencies are being submitted on PR branches and consider enabling retry-on-snapshot-warnings. See the documentation for more information and troubleshooting advice. Scanned FilesNone |
|
I opened #1508 a minute before this and independently added a concurrency group to Two things from that work worth handing over. 1. The
|
The open risk recorded when this PR was written -- that the merge gate hard-codes gate 2's check list and will drift from the policy it claims to enforce -- materialised in three weeks. #1449/#1480 made `test-frontend` a required check and updated MERGE_POLICY.md, but nothing pointed at this gate. Its list still ended at `test`, which runs Python pytest only, so it would have merged a Dependabot PR with the apps/web vitest suite unrun -- including the CWE-209 and billing-disclosure regressions that suite carries. That is the same fail-open class this PR exists to close, arriving through drift rather than through a coding error. Adds `test-frontend` to REQUIRED_CHECKS, and adds the test I previously declined to write. I argued then that parsing the policy prose was "its own small parser with its own failure mode"; having now seen the drift happen, that trade is clearly wrong -- the parse is one regex over a stable sentence, and the alternative is a gate that silently under-enforces. The test fails with the exact missing names, so the next addition to gate 2 lands here instead of in production. Also merges origin/main (56 commits), which brings the concurrency group added to this workflow by #1510. Verified present after the merge. 17 tests pass. Removing `test-frontend` from the gate fails the new test with `missing from gate: ['test-frontend']`.
2 participants
Canonical issue
Closes #1519
Outcome
Removes the structural amplifier that lets this repository's Actions queue reach hundreds of runs deep and stay there.
Measured via the Actions API:
queuedin_progress20:49:28Z(~24 min, never started)Be precise about what this PR does and does not claim. Between the first two samples, runner capacity recovered on its own and the queue began draining. So the acute stall was a capacity condition, not something missing
concurrency:blocks caused. I am not claiming this PR would have prevented it.What the missing blocks do cause is the standing run volume: with ~26 open PRs, ~30 workflows, and no deduplication, every push stacks a brand-new full run beside the ones it obsoletes. That is why a capacity dip produced a 285-deep backlog rather than a shallow one, why it took ~24 minutes to start moving, and why it will recur on the next dip. This PR shrinks the amplifier. It does not add runners.
The consequence while it lasted was concrete: no PR could reach a green CI gate. Ready PRs all reported
mergeable_state: unstablepurely because their checks were queued — nothing failed, nothing ran.Why the volume is unbounded
17 of 30 workflows carried no
concurrency:block, including the heaviest and most-triggered:ci,codeql-analysis,security,e2e-tests,pr-checks,auto-label,dependency-review.#1447 correctly removed those workflows' base-branch filters so every PR runs the gate. That was right and this PR does not touch it — but it raised per-PR run volume without adding any deduplication to absorb it.
Overlap with #1508 — read this before merging either
#1508 was opened one minute before this PR and independently adds a
concurrency:block toci.yml. An earlier revision of this body claimed no competing PR existed; that was written before #1508 appeared and is corrected here rather than left as a false checkbox.They are complementary, and neither needs to be closed:
.coderabbit.yamlhalf is right and this PR doesn't have it. Itsinheritance: falsediagnosis explains a symptom hit directly here: this PR carriesci/cdandgithub_actions— both on the inherited required list — and CodeRabbit still posts "Review skipped: excluded by label configuration". That is an inherited gate whichlabels: []alone never bound.ci.ymlhalf has the bug described below. It keys non-PR runs ongithub.refand carries a comment promising that "a rapid series of merges must not cancel each other" — which that key cannot deliver. Details in the comment on #1508.Suggested split: #1508 keeps
.coderabbit.yamland drops (or re-keys) itsci.ymlhunk; this PR keeps the workflow scheduling change across all 16 files. The reverse split is equally fine — the only thing that matters is that whichever version lands does not key main's post-merge runs ongithub.ref.Scope
concurrency:block on the 14 workflows triggering onpull_request,pull_request_target,push, orissues—ci,codeql-analysis,security,e2e-tests,pr-checks,dependency-review,api-cost-postgres,gh-aw-validation,anthropic-wif-test,branch-cleanup,dependabot-auto-merge,auto-label,auto-assign,issue-triage. Plus a re-key ofcoverageandsecret-scan, which already had groups carrying the defect below..coderabbit.yaml— that's fix(review): make the CodeRabbit label-gate override actually bind #1508's, and it should stay there.bulk-issue-processor,real-processing,stale—workflow_dispatch/scheduleonly, not queue pressure.The design, and two corrections found in review
The first commit (
3ef52a9) keyed groups ongithub.event.pull_request.number || github.refwithcancel-in-progress: falsefor push events, claiming push runs would "queue, so each merged commit keeps its own verdict."That claim was wrong, and
20cc78efixes it.cancel-in-progress: falsestops a new run cancelling a running one. It does not stop GitHub cancelling a pending one — when a run is queued into a group that already has one in progress and one pending, the pending run is cancelled. Sincegithub.refisrefs/heads/mainfor every merge, all main pushes shared one group, so a burst of merges — ten in twenty minutes here — would have silently dropped the middle commits' verdicts. (This is the same bug #1508'sci.ymlhunk currently carries.)A second defect, caught by the Vercel review bot, is fixed in
c00a664. Oncheck_suiteeventsGITHUB_SHAis the default-branch tip rather than the commit under test, so thegithub.shafallback collapsed every PR's check suite into one group.dependabot-auto-mergenow falls back togithub.event.check_suite.head_shafirst.The shipped design:
github.event.pull_request.numberand cancel their predecessors. This is where all queue pressure comes from — verified: everypush:trigger across these workflows is already filtered tomain(orclaude/branch-cleanup-*), so pushes to PR branches fire no push runs at all.github.sha, landing in singleton groups where they can neither cancel nor be cancelled. Every merged commit keeps its own verdict. This costs nothing given (1).github.refalone — besides the burst problem, on the twopull_request_targetworkflowsgithub.refresolves to the base branch, collapsing every open PR into one group where they would cancel each other.github.event.issue.number;github.shais meaningless for issues events.auto-labelfires on both, so it falls through PR → issue → ref.coverageandsecret-scanpredate this PR and both had the baregithub.refgroup withcancel-in-progress: true— the more aggressive form of the same bug. Re-keyed to match.Risk
git revert. No migration, config, schema, or permission change.Verification
Head
c00a664. Measured, not inferred.yaml.safe_loadover.github/workflows/*.yml, clean at every commit.on:keys of all 28 files are byte-identical before and after; the whole diff lives inside new or rewrittenconcurrency:blocks.push:triggers confirmed filtered tomain, exceptbranch-cleanup(claude/branch-cleanup-*). This is what makes point (2) free rather than a tradeoff.c00a664: a push at 21:23:43 andready_for_reviewat 21:24:29. Dependency Review (pull_request, uses the PR head's workflow file — has the block) cancelled the superseded run. PR Checks (pull_request_target, uses the base branch's file — no block yet) stacked both. Same PR, same commit, same window; the only variable is whether the file in play carries the block.A limitation this evidence exposes: the fix is partially self-testing but not self-deploying.
pull_requestworkflows take effect from the PR head and are already deduplicating here.pull_request_targetworkflows (pr-checks,dependabot-auto-merge) read the base branch, so they keep stacking until this lands onmain, and their behaviour cannot be verified from this PR. A green run here is not evidence for that half.Production evidence
Not applicable as a deployed artefact — CI configuration, no runtime or
apps/web/**surface. The Vercel preview built green on all three heads.The runtime evidence is the queue measurement and the same-SHA A/B above, both taken from the live Actions API rather than reasoned about — including the part that cuts against this PR's original framing. The post-merge check is direct and cheap: standing queue depth should sit materially lower at comparable PR volume.
Agent handoff
Recommended companion action (human)
Merging does not clear runs already queued — concurrency groups apply to new runs only. To drain immediately, cancel the existing backlog. I did not do this unilaterally: cancelling discards CI verdicts for commits already merged to
main, which is a call for a human, not an unattended routine.