You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
17 of 30 workflows carry no concurrency: block, including the heaviest and most-triggered ones: ci, codeql-analysis, security, e2e-tests, pr-checks, auto-label, dependency-review.
Without a concurrency group, each push stacks a brand-new full run beside the runs it obsoletes rather than superseding them. With ~26 open PRs and ~30 workflows, run volume is effectively unbounded: nothing is ever deduplicated, so obsolete runs compete for the same runner capacity as current ones.
Evidence
Measured via the Actions API on 2026-08-07:
Signal
21:13 UTC
21:19 UTC
21:26 UTC
Runs queued
285
256
207
Runs in_progress
2
23
—
Oldest queued run
20:49:28Z (~24 min, unstarted)
—
—
While it lasted, no PR in the repository could reach a green CI gate. Ready PRs reported mergeable_state: unstable purely because their checks were queued — nothing had failed, nothing had run.
Scoping the claim honestly: between the first two samples, runner capacity recovered on its own (in_progress 2 → 23) and the queue began draining. The acute stall was therefore a capacity condition, not caused by the missing concurrency blocks. This issue is not "concurrency blocks caused an outage."
What the missing blocks do cause is the standing run volume — which is why a capacity dip produced a 285-deep backlog rather than a shallow one, why it took ~24 minutes to start moving, and why the same dip will produce the same backlog next time. This is about removing an amplifier, not adding runners.
Contributing factor
#1447 correctly removed base-branch filters from ci, security, coverage, codeql-analysis, e2e-tests and dependency-review so that every PR runs the gate, including PRs stacked onto another PR's branch (#1440). That change was right and should not be reverted — but it raised per-PR run volume without adding any deduplication to absorb it.
Two traps for whoever implements this
Both were hit in practice while diagnosing this, so they are worth stating up front.
cancel-in-progress: false does not preserve every commit's verdict. It only stops a new run cancelling a run that is already executing. GitHub still cancels a pending run: when a run is queued into a group that already has one in progress and one pending, the previously pending run is cancelled. Since github.ref is refs/heads/main for every merge, a github.ref-keyed group collapses all post-merge runs together, and a burst of merges — ten in twenty minutes here — silently drops the middle commits' verdicts.
github.ref is wrong for pull_request_target, and github.sha is wrong for check_suite. On pull_request_target (pr-checks, dependabot-auto-merge) github.ref resolves to the base branch, collapsing every open PR into one group where they cancel each other. On check_suite (dependabot-auto-merge) GITHUB_SHA is the default-branch tip rather than the commit under test, so a github.sha fallback collapses every PR's check suite into one group.
Acceptance criteria
Every workflow triggered by pull_request, pull_request_target, push, or issues has a concurrency: block.
PR runs are keyed so that a new push to a PR supersedes that PR's own earlier runs, and only its own.
Non-PR runs (push to main, schedule, check_suite) are keyed so they can neither cancel nor be cancelled by one another — every merged commit keeps its own verdict.
No workflow gains or loses a trigger, job, step, or permission; no gate becomes skippable.
Group names do not collide with the pre-existing groups in pr-governance, verification, phase-goal-tracker, autonomous-video-processing, deploy-cloud-run, emergency-stop, repository-reconciliation, or the gh-aw lock files.
coverage and secret-scan, which already ship a bare github.ref group with cancel-in-progress: true (trap 1 in its more aggressive form), are brought in line.
All workflow files still parse.
Out of scope
Adding runner capacity or raising the concurrency ceiling. This reduces waste; it does not raise the ceiling. If the queue is saturated by genuinely distinct branches, runs still wait.
Cancelling any existing backlog. That discards CI verdicts for commits already merged to main and is an operational decision for a human.
Problem
17 of 30 workflows carry no
concurrency:block, including the heaviest and most-triggered ones:ci,codeql-analysis,security,e2e-tests,pr-checks,auto-label,dependency-review.Without a concurrency group, each push stacks a brand-new full run beside the runs it obsoletes rather than superseding them. With ~26 open PRs and ~30 workflows, run volume is effectively unbounded: nothing is ever deduplicated, so obsolete runs compete for the same runner capacity as current ones.
Evidence
Measured via the Actions API on 2026-08-07:
queuedin_progress20:49:28Z(~24 min, unstarted)While it lasted, no PR in the repository could reach a green CI gate. Ready PRs reported
mergeable_state: unstablepurely because their checks were queued — nothing had failed, nothing had run.Scoping the claim honestly: between the first two samples, runner capacity recovered on its own (
in_progress2 → 23) and the queue began draining. The acute stall was therefore a capacity condition, not caused by the missing concurrency blocks. This issue is not "concurrency blocks caused an outage."What the missing blocks do cause is the standing run volume — which is why a capacity dip produced a 285-deep backlog rather than a shallow one, why it took ~24 minutes to start moving, and why the same dip will produce the same backlog next time. This is about removing an amplifier, not adding runners.
Contributing factor
#1447 correctly removed base-branch filters from
ci,security,coverage,codeql-analysis,e2e-testsanddependency-reviewso that every PR runs the gate, including PRs stacked onto another PR's branch (#1440). That change was right and should not be reverted — but it raised per-PR run volume without adding any deduplication to absorb it.Two traps for whoever implements this
Both were hit in practice while diagnosing this, so they are worth stating up front.
cancel-in-progress: falsedoes not preserve every commit's verdict. It only stops a new run cancelling a run that is already executing. GitHub still cancels a pending run: when a run is queued into a group that already has one in progress and one pending, the previously pending run is cancelled. Sincegithub.refisrefs/heads/mainfor every merge, agithub.ref-keyed group collapses all post-merge runs together, and a burst of merges — ten in twenty minutes here — silently drops the middle commits' verdicts.github.refis wrong forpull_request_target, andgithub.shais wrong forcheck_suite. Onpull_request_target(pr-checks,dependabot-auto-merge)github.refresolves to the base branch, collapsing every open PR into one group where they cancel each other. Oncheck_suite(dependabot-auto-merge)GITHUB_SHAis the default-branch tip rather than the commit under test, so agithub.shafallback collapses every PR's check suite into one group.Acceptance criteria
pull_request,pull_request_target,push, orissueshas aconcurrency:block.main,schedule,check_suite) are keyed so they can neither cancel nor be cancelled by one another — every merged commit keeps its own verdict.pr-governance,verification,phase-goal-tracker,autonomous-video-processing,deploy-cloud-run,emergency-stop,repository-reconciliation, or thegh-awlock files.coverageandsecret-scan, which already ship a baregithub.refgroup withcancel-in-progress: true(trap 1 in its more aggressive form), are brought in line.Out of scope
mainand is an operational decision for a human.