Skip to content

ci: supersede stale workflow runs with concurrency groups - #1511

Closed
groupthinking wants to merge 1 commit into
mainfrom
claude/clever-heisenberg-rndqcq
Closed

ci: supersede stale workflow runs with concurrency groups#1511
groupthinking wants to merge 1 commit into
mainfrom
claude/clever-heisenberg-rndqcq

Conversation

@groupthinking

@groupthinking groupthinking commented Aug 7, 2026

Copy link
Copy Markdown
Owner

Canonical issue

Closes # — no issue filed; this was found by the PR-remediation routine while trying to clear the CI gate on #1483 / #1485 / #1486 / #1494. Happy to file one if the repo wants the paper trail.

Outcome

Seven PR-triggered workflows had no concurrency block, so every push to a branch left its previous run queued rather than cancelling it.

Measured at 2026-08-07T21:12Z: 66 workflow runs live — 63 queued behind 3 executing. Nothing was failing; nothing could finish either. Every non-draft PR sat at mergeable_state: unstable with its checks queued, so no PR could reach a green gate.

21 of the 66 were superseded duplicates — a newer run for the same workflow on the same branch already existed, and nothing would ever read the older one's result:

Duplicates Workflow Branch
5x CI main
4x PR Checks claude/clever-heisenberg-kltzfb
3x CI claude/clever-heisenberg-j1yx9l
3x Phase Goal Tracker main
3x Auto Label main
2x each Dependency Review, PR Checks, CI ×5, Issue Triage various

That is ~⅓ of the queue reclaimable by configuration alone.

Scope

  • Included — a concurrency block on ci.yml, codeql-analysis.yml, security.yml, e2e-tests.yml, pr-checks.yml, dependency-review.yml, auto-label.yml. 49 added lines, ~20 of them comments. No job, step, trigger, or permission changed.
  • Explicitly excluded:
    • The other 45 live runs. Those are legitimate distinct runs; concurrency groups do not touch them. The queue is oversubscribed for a second reason this PR does not address — see below.
    • The workload itself. Whether ~8 workflows per PR event is the right gate is a policy question, not a config bug.
    • Cancelling the current backlog. A one-off operator action; this PR only stops it recurring.
    • The CodeRabbit label gate (see the reviewer note) — a dashboard/org-level setting, not fixable in-repo.
    • phase-goal-tracker.yml and issue-triage.yml, which also showed duplicates. Both are issues-triggered automation outside this PR's PR-gate scope.

Two traps in the group keys

Worth a reviewer's eye, because the obvious ${{ github.ref }} key is wrong in both cases:

  1. pr-checks.yml runs on pull_request_target, where github.ref is the base branch (refs/heads/main) — not the PR. A github.ref key would put every open PR in one group and have them cancel each other, converting a queue problem into a correctness problem. Keyed on github.event.pull_request.number, matching what pr-governance.yml already does.
  2. auto-label.yml also runs on issues, where github.ref is likewise the default branch — every issue would collapse into a single group. Keyed on pull_request.number || issue.number. PRs and issues share one numbering sequence on GitHub, so the two branches of that fallback cannot collide.

For the five pull_request-triggered workflows github.ref is refs/pull/<n>/merge, which is already unique per PR.

cancel-in-progress is deliberately not uniform

true only on the two PR-only workflows (dependency-review, pr-checks). The four that also run on push/schedule use ${{ github.event_name == 'pull_request' }}:

  • A push to main is the record of whether main is green. Cancelling it leaves that unanswered for the commit that landed.
  • A cancelled CodeQL or Security Scan run uploads no SARIF. That stales the security dashboard silently rather than failing loudly — the worst failure shape for a security gate.

This diverges from coverage.yml / secret-scan.yml, which use an unconditional true. Flagging rather than quietly matching them: those two may want the same treatment, but changing them is not this PR's business.

Risk

  • Risk level: low — configuration only, no job logic touched.
  • Failure mode: a mis-scoped group key cancels runs that should have proceeded. That is exactly the trap called out above, and the reason two of the seven are keyed on issue/PR number instead of github.ref. A cancelled required check reports as cancelled, not success, so the failure would be visible at the merge gate rather than silent.
  • Rollback: git revert. Behaviour returns to unbounded queuing immediately; no state to migrate.

Verification

Head a095dc6.

  • actionlint 1.7.7 — clean on all seven files.
  • YAML parses, and each concurrency block was read back to confirm the group key and cancel condition resolve as intended per triggering event.
  • Trigger sets unchangedon: blocks re-parsed and compared; no workflow gained or lost an event.
  • Required CI — cannot be demonstrated on this head. This PR's own checks enter the same saturated queue it exists to drain. The first real evidence will be the run count settling after it lands.
  • Review threads resolved — none open.

Production evidence

Not applicable — CI configuration, no runtime or UI surface.

The operational evidence is the queue measurement above, taken from the Actions API rather than inferred: 63 queued / 3 in progress, grouped by (workflow, branch) to identify the 21 superseded runs.

Agent handoff

  • One canonical issue is linked — none filed; see the top of this PR
  • No competing PR implements the same issue — no other open PR touches these concurrency blocks
  • Acceptance criteria are satisfied — superseded runs are cancelled; no gate loses coverage on main or on schedule
  • Required checks pass on the current head — blocked on the backlog this PR addresses
  • Human decision is requested only for: merge approval

Note for reviewers

Two things this PR does not fix, both found alongside it.

1. The queue is oversubscribed at the source

Nine PRs (#1475#1497) were opened in an 11-minute window by an automated remediation routine, each firing ~8 workflows. Concurrency groups reclaim the duplicate third; they do not address the burst. If that routine keeps opening PRs at that rate, the queue stays saturated.

2. #1425's CodeRabbit fix did not take — auto-review is still off

This supersedes a vaguer note in the first revision of this description; the mechanism is now pinned.

.coderabbit.yaml:45 sets labels: [], added by #1425 (c576b54) specifically to override an inherited org-level required-labels gate. Its comment describes the deadlock exactly: the gate demands one of ~26 labels, a PR opens unlabelled, so the review is skipped and no label is ever applied to un-skip it.

The override is not working. CodeRabbit's own run report on this PR (Run ID 60743c8d-31da-4804-8e23-e241981b2918) says:

Review skipped — Auto reviews are limited based on label configuration.
Required labels (at least one): ['architecture-gap', 'bug', 'ci-cd', 'ci/cd', 'copilot-rabbit', … 'v0']

Configuration used: Repository YAML (base), Repository UI (inherited), Organization UI (inherited)

That is the same ~26-label set #1425's comment describes, still armed, with the repo YAML loaded. Evidence it is not merely the draft rule (drafts: false):

PR Draft? Labels CodeRabbit
#1483 no none Review skipped: excluded by label configuration
#1494 no none Review skipped: excluded by label configuration
#1486 no javascript, tests Review rate limited — passed the gate, hit quota
#1511 yes none Review skipped: excluded by label configuration

Two non-draft PRs skipped for labels, and the one non-draft PR that had a matching label got through to a different failure. So the gate is live and labels: [] is not clearing it.

Likely cause is the hazard .coderabbit.yaml's own header warns about (lines 7–10): with inheritance: true, an empty list reads as unset and the inherited value layers back in. The two candidate remedies — inheritance: false, or removing the required-labels list in the Organization UI — both have blast radius beyond this PR (the first drops all org-level config), so I am flagging rather than guessing. Not fixable in-repo either way if it is the org setting.

Practical consequence today: any process step that assumes a CodeRabbit review gates these PRs is a no-op, and has been since before #1425 was believed to fix it. Reviews must be requested by hand with @coderabbitai review.

Agent provenance

Produced by a scheduled, unattended PR-remediation routine running under the repo owner's account. Left as a draft; it halts at the human merge-approval gate and requests no auto-merge to protected main.

Seven PR-triggered workflows had no `concurrency` block, so every push to a
branch left its previous run queued instead of cancelling it. A burst of PR
activity therefore saturated the Actions concurrency limit with runs whose
results nothing would ever read: measured at 21:12Z, 66 runs were live (63
queued behind 3 executing) and 21 of them were superseded duplicates of a
run on the same workflow and branch. No PR could reach a green gate, and
nothing was actually failing.

Group keys avoid two traps:

- `pr-checks.yml` runs on `pull_request_target`, where `github.ref` is the
  *base* branch. A `github.ref` key would put every open PR in one group and
  have them cancel each other. Keyed on the PR number, as pr-governance.yml
  already does.
- `auto-label.yml` also runs on `issues`, where `github.ref` is likewise the
  default branch. Keyed on `pull_request.number || issue.number`; PRs and
  issues share one numbering sequence, so the two cannot collide.

`cancel-in-progress` is limited to `pull_request` on the four workflows that
also run on push/schedule. A push to main is the record of whether main is
green, and a cancelled CodeQL or Security Scan run uploads no SARIF — that
would stale the security dashboard silently rather than fail loudly.

Verified with actionlint 1.7.7: clean on all seven files.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012S1Fh4RAWUcUbpy2TVtfnR
@vercel

vercel Bot commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
v0-uvai Ready Ready Preview, v0 Aug 7, 2026 9:17pm

@coderabbitai

coderabbitai Bot commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

Important

Review skipped

Auto reviews are limited based on label configuration.

🏷️ Required labels (at least one) (1)
  • [‘architecture-gap’, ‘bug’, ‘ci-cd’, ‘ci/cd’, ‘copilot-rabbit’, ‘documentation’, ‘duplicate’, ‘enhancement’, ‘frontend’, ‘github_actions’, ‘good first issue’, ‘help wanted’, ‘high-priority’, ‘invalid’, ‘javascript’, ‘ml-model’, ‘needs-triage’, ‘pipeline-critical’, ‘placeholder-code’, ‘priority:high’, ‘python’, ‘python:uv’, ‘question’, ‘styling’, ‘tests’, ‘v0’]

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository YAML (base), Repository UI (inherited), Organization UI (inherited)

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 60743c8d-31da-4804-8e23-e241981b2918

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Copy link
Copy Markdown
Owner Author

Closing as superseded by #1510MERGE_POLICY.md gate 6 reconciliation

This PR and #1510 were opened five seconds apart (21:16:42 and 21:16:47 UTC) by two concurrent agent sessions that independently diagnosed the same problem and independently wrote the same fix. Both add concurrency: groups so superseded Actions runs cancel instead of stacking.

Gate 6: reconciliation picks one, the other closes with a pointer. Picking on merits rather than the 72-hour freshness default, because the two are seconds apart and the coverage difference is decisive.

Why #1510 is the one to keep — it is a functional superset:

#1511 (this) #1510
Workflows covered 7 14 — adds anthropic-wif-test, api-cost-postgres, auto-assign, branch-cleanup, dependabot-auto-merge, gh-aw-validation, issue-triage
pull_request_target base-ref trap handled per-workflow, by hand, in pr-checks only handled structurally by the key itself — github.event.pull_request.number || github.ref prefers the PR number wherever one exists
No-cancel on main / schedule

Both of this PR's distinctive insights are already satisfied by #1510:

So nothing here is lost by closing. Since coverage is what actually drains the queue, the broader diff wins.

⚠️ #1510 is not defect-free — see the finding I left on it about dependabot-auto-merge.yml. That should be fixed on #1510 rather than by keeping this PR open.

Reopen if the reconciliation call looks wrong.


Generated by Claude Code

Copy link
Copy Markdown
Owner Author

Duplicate of #1510 — and I think #1510 should win

These two were opened ~20 seconds apart by parallel sessions, both fixing the same thing: the Actions queue can't drain. Measured just now via the API, so the premise is real and current:

  • 247 workflow runs queued
  • 22 in_progress — jobs created at 20:59 were still running at 21:19

So this is the root blocker for every open PR: nothing can reach a green CI gate, and the "green" statuses currently visible on #1486 / #1507 / #1485 are Vercel and CodeRabbit-skip, not tests.

Why #1510 rather than this one

1. The github.ref key has a failure mode this PR's comments argue it doesn't.

The reasoning here is that cancel-in-progress: ${{ github.event_name == 'pull_request' }} protects main:

A push to main is the record of whether main is green; cancelling it would leave that question unanswered for the commit that landed.

Right goal, but cancel-in-progress: false doesn't buy it. It only stops a running run from being cancelled. Group membership has a second, separate rule: when a run is queued into a group that already holds a pending run, GitHub cancels the older pending one. With group: ci-${{ github.ref }}, every push to main shares refs/heads/main, so a burst of merges — which is exactly what a remediation sweep produces — silently drops the middle commits' verdicts. The run that landed commit N gets cancelled while pending because commit N+1 arrived, and nothing reports it as a failure.

#1510 keys non-PR runs on github.sha, putting each in a singleton group, and its comment states this reasoning explicitly. That's the correct call.

2. Coverage: 16 workflows vs 7. This PR covers ci, codeql-analysis, dependency-review, e2e-tests, pr-checks, security, auto-label. #1510 adds anthropic-wif-test, api-cost-postgres, auto-assign, branch-cleanup, dependabot-auto-merge, gh-aw-validation, issue-triage, and re-keys the two that already had github.ref groups (coverage, secret-scan) — which have the same pending-cancellation bug on main today, and this PR leaves them alone.

Given the queue depth, the workflows this PR doesn't touch are a meaningful share of the pressure.

Where this PR is right and #1510 should absorb it

Your pr-checks.yml note is the sharper one:

on pull_request_target the ref is the base branch, so every open PR would share one group and cancel the others

That's correct and it's a trap #1510 avoids only incidentally (it keys on pull_request.number everywhere). Worth carrying the comment across — it explains why the number key is load-bearing there, which the generic block in #1510 doesn't say.

Suggested resolution

Close this in favour of #1510, moving the pull_request_target note onto pr-checks.yml there. Not closing it myself — it isn't my PR, and #1510 is still a draft with CI that can't run yet, so the safe order is to land #1510 first and close this after, rather than before.

Terminal state: HALTED(duplicate_pending_#1510).


Generated by Claude Code

@github-actions github-actions Bot added the ci/cd label Aug 7, 2026
@github-actions

github-actions Bot commented Aug 7, 2026

Copy link
Copy Markdown

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

Snapshot Warnings

⚠️: No snapshots were found for the head SHA a095dc6.
Ensure that dependencies are being submitted on PR branches and consider enabling retry-on-snapshot-warnings. See the documentation for more information and troubleshooting advice.

Scanned Files

None

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants