Skip to content

ci: workflows have no concurrency groups, so obsolete runs stack instead of superseding #1519

Description

@groupthinking

Problem

17 of 30 workflows carry no concurrency: block, including the heaviest and most-triggered ones: ci, codeql-analysis, security, e2e-tests, pr-checks, auto-label, dependency-review.

Without a concurrency group, each push stacks a brand-new full run beside the runs it obsoletes rather than superseding them. With ~26 open PRs and ~30 workflows, run volume is effectively unbounded: nothing is ever deduplicated, so obsolete runs compete for the same runner capacity as current ones.

Evidence

Measured via the Actions API on 2026-08-07:

Signal 21:13 UTC 21:19 UTC 21:26 UTC
Runs queued 285 256 207
Runs in_progress 2 23
Oldest queued run 20:49:28Z (~24 min, unstarted)

While it lasted, no PR in the repository could reach a green CI gate. Ready PRs reported mergeable_state: unstable purely because their checks were queued — nothing had failed, nothing had run.

Scoping the claim honestly: between the first two samples, runner capacity recovered on its own (in_progress 2 → 23) and the queue began draining. The acute stall was therefore a capacity condition, not caused by the missing concurrency blocks. This issue is not "concurrency blocks caused an outage."

What the missing blocks do cause is the standing run volume — which is why a capacity dip produced a 285-deep backlog rather than a shallow one, why it took ~24 minutes to start moving, and why the same dip will produce the same backlog next time. This is about removing an amplifier, not adding runners.

Contributing factor

#1447 correctly removed base-branch filters from ci, security, coverage, codeql-analysis, e2e-tests and dependency-review so that every PR runs the gate, including PRs stacked onto another PR's branch (#1440). That change was right and should not be reverted — but it raised per-PR run volume without adding any deduplication to absorb it.

Two traps for whoever implements this

Both were hit in practice while diagnosing this, so they are worth stating up front.

  1. cancel-in-progress: false does not preserve every commit's verdict. It only stops a new run cancelling a run that is already executing. GitHub still cancels a pending run: when a run is queued into a group that already has one in progress and one pending, the previously pending run is cancelled. Since github.ref is refs/heads/main for every merge, a github.ref-keyed group collapses all post-merge runs together, and a burst of merges — ten in twenty minutes here — silently drops the middle commits' verdicts.

  2. github.ref is wrong for pull_request_target, and github.sha is wrong for check_suite. On pull_request_target (pr-checks, dependabot-auto-merge) github.ref resolves to the base branch, collapsing every open PR into one group where they cancel each other. On check_suite (dependabot-auto-merge) GITHUB_SHA is the default-branch tip rather than the commit under test, so a github.sha fallback collapses every PR's check suite into one group.

Acceptance criteria

  • Every workflow triggered by pull_request, pull_request_target, push, or issues has a concurrency: block.
  • PR runs are keyed so that a new push to a PR supersedes that PR's own earlier runs, and only its own.
  • Non-PR runs (push to main, schedule, check_suite) are keyed so they can neither cancel nor be cancelled by one another — every merged commit keeps its own verdict.
  • No workflow gains or loses a trigger, job, step, or permission; no gate becomes skippable.
  • Group names do not collide with the pre-existing groups in pr-governance, verification, phase-goal-tracker, autonomous-video-processing, deploy-cloud-run, emergency-stop, repository-reconciliation, or the gh-aw lock files.
  • coverage and secret-scan, which already ship a bare github.ref group with cancel-in-progress: true (trap 1 in its more aggressive form), are brought in line.
  • All workflow files still parse.

Out of scope

  • Adding runner capacity or raising the concurrency ceiling. This reduces waste; it does not raise the ceiling. If the queue is saturated by genuinely distinct branches, runs still wait.
  • Cancelling any existing backlog. That discards CI verdicts for commits already merged to main and is an operational decision for a human.
  • fix(ci): run PR quality gates on every base branch, not just main #1447's trigger design.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions