fix: route merge_group jobs to the dedicated arc-merge-queue pool (BLO-22428)

[allyblockcast]

Thinking Path

The presenting symptom was Blockcast/paperclip's master merge queue sitting 53 entries deep with a 10h47m gap between merges, while arc-default was pinned at 30/30 runner pods. The obvious reading — "we are out of CI capacity, add runners" — is wrong, and following it would have cost a cross-team capacity trade for no benefit.

The disconfirming measurement is GitHub's merge-queue max_entries_to_build, which is 1 on ruleset 20487141. The queue builds one entry at a time, so its runner demand is flat and small regardless of whether it is 5 deep or 53 deep. Queue depth is not check demand. What starved the queue was not insufficient total capacity but ordering: merge_group jobs competed for the same default label as an unbounded pull_request backlog, and lost.

That reframes the fix. The queue does not need capacity proportional to its depth; it needs a small pool that pull_request traffic cannot touch. So instead of raising arc-default past its documented 30-slot ceiling — which arc/arc-default-values.yaml states is derived imagefs arithmetic (2625Gi x 1.10 = 2887Gi max; 30 x 38Gi = 2876Gi, 11Gi slack) and whose raise "is a cross-team decision about whose capacity shrinks, not a knob turn" — the companion PR carves arc-default 30 -> 22 and adds arc-merge-queue at 8. Total stays 30, the ledger is untouched, and no team's capacity shrinks. This PR is the other half: routing the traffic onto that new pool.

Rejected alternatives: raising the ceiling by shrinking a named idle pool (spends a cross-team trade on the wrong axis, and a single 0-pod sample does not establish sustained idleness); and accepting the ceiling while throttling PR CI demand (treats the symptom, leaves the starvation mechanism intact).

What Changed

.github/workflows/pr.yml — the six heavy jobs shared between the pull_request and merge_group events (typecheck_release_registry, worktree_install, general_tests, build, verify_serialized_server, canary_dry_run) move from runs-on: default to a conditional:

runs-on: ${{ github.event_name == 'merge_group' && 'arc-merge-queue' || 'default' }}

merge_group runs land on the new dedicated pool; pull_request runs keep the shared default label. On pull_request the ternary evaluates to default, so PR CI behaviour is unchanged byte-for-byte.

pr.yml is the only workflow in the repo carrying a merge_group: trigger, so no straggler job is left starving on default. The runs-on values not touched here are 3x arc-light and 1x arc-e2e, both already partitioned.

scripts/check-github-runner-labels.mjs — adds arc-merge-queue to the allow-list, and teaches the guard to parse exactly the ${{ <cond> && 'X' || 'Y' }} shape so it validates both literal branches. Previously it treated any ${{ ... }} value as one opaque runner name, which would have flagged the new expression as an unknown label. Any other expression shape still falls through to the opaque-string path and fails closed.

scripts/check-github-runner-labels.test.js — three regression cases for the new parsing: a valid ternary with both branches allowed, one with a disallowed branch, and an unrelated opaque expression the parser cannot decompose (must still be rejected).

Risks

• Ordering risk, and it is the real one. This PR must not merge before Blockcast/onprem-k8s#2115, which creates the arc-merge-queue scale set. If this routing goes live first, every merge_group job targets a label with zero live listeners and queues indefinitely — the same starvation this change exists to fix, relocated to a new label. This PR must therefore not merge until feat: PAPERCLIP_APP_NAME — rebrand self-hosted deployment without forking paperclipai/paperclip#2115 is merged and arc-merge-queue pods are confirmed live in arc-runners. That gate is enforced by the sequencing hold on the Paperclip issue and by board approval 25fe072c — not by draft status, which has been lifted so the review gates can run.
• Unavoidable degraded window between the two merges. Once feat: PAPERCLIP_APP_NAME — rebrand self-hosted deployment without forking paperclipai/paperclip#2115 merges, arc-default drops to 22 while master's pr.yml still routes all six merge_group jobs to default — strictly worse than today until this PR lands. Running both pools at full size to avoid the window is not available: 30+8 slots is 3180Gi against a 2887Gi ceiling. The window is structural; the only lever is its duration, so this PR should land within minutes of feat: PAPERCLIP_APP_NAME — rebrand self-hosted deployment without forking paperclipai/paperclip#2115, not via the 53-deep queue.
• Mild oversubscription of the new pool. Peak merge_group concurrency is ~10 jobs, not 8: the 4 verify_serialized_server shards are gated behind general_tests, but typecheck + worktree_install + 6x general_tests + build + canary_dry_run overlap. Against maxRunners: 8 two jobs wait one slot-turn — materially better than starving behind 53 PRs, but the companion values-file comment's word "comfortably" is optimistic.
• Guard parsing is deliberately narrow. The regex matches one ternary shape only. A future author writing a differently-shaped expression gets a hard failure rather than a silent pass — intended (fail closed), but it will look like a false positive to whoever hits it first.
• Rollback is cheap. git revert this commit and all six jobs return to runs-on: default immediately; the arc-merge-queue pool then simply sits idle. No CI breakage. Reverting feat: PAPERCLIP_APP_NAME — rebrand self-hosted deployment without forking paperclipai/paperclip#2115 separately restores arc-default to 30.

Model Used

• Authored by PlatformSREEngineer (claude-sonnet-5[1m]), which also verified the ledger arithmetic and ran the guard suites locally.
• Reviewed, and this description restructured, by CTO (claude-opus-5[1m]). The CTO review independently confirmed the max_entries_to_build: 1 measurement that motivates partitioning over a capacity raise, checked that the companion ledger change nets to zero, and flagged the ~10-vs-8 peak-concurrency gap above.

Verification

• python3 -c "import yaml; yaml.safe_load(open('.github/workflows/pr.yml'))" — YAML still parses.
• node ./scripts/check-github-runner-labels.mjs — Validated 20 workflows: all runner labels use ARC.
• node --test scripts/check-github-runner-labels.test.js — 8/8 pass, including the 3 new ternary-parsing cases.
• node --test scripts/__tests__/pr-verify-lane-outcome.test.mjs — unaffected, 13/13 pass (does not assert on runs-on).
• Full build/typecheck is not runnable in the authoring environment (no local Postgres/build toolchain); CI is the gate for the rest.

Post-merge verifying signal: a merge_group run observed picking up a runner on the arc-merge-queue pool, and merge-queue drain rate measured by merged_at (not commit dates — under a REBASE queue a commit's committer date is its enqueue time).

Paperclip issue: https://paperclip.blockcast.net/BLO/issues/BLO-22428
Related: https://paperclip.blockcast.net/BLO/issues/BLO-21953

Related PRs

Searched the open and recent PR list for duplicates before filing this checklist. Nothing duplicates this change; two are adjacent and worth reading together:

• #1042 — ci: load-shed stale merge_group generations superseded by a queue re-stage (BLO-21953). Same incident, different mechanism (sheds superseded generations rather than partitioning capacity). Complementary, not overlapping.
• #1077 — fix(ci): run commitperclip-review against the merge queue's landing commit. Touches the review gate, not runs-on: routing.

Checklist

• I have included a thinking path that traces from project context to this change
• I have specified the model used (with version and capability details)
• I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work
• I have searched GitHub for duplicate or related PRs and linked them above
• I have either (a) linked existing issues with Fixes: # / Closes # / Refs # OR (b) described the issue in-PR following the relevant issue template — described in-PR above; tracking issues live in Paperclip (BLO-22428 / BLO-21953), not GitHub Issues
• I have run tests locally and they pass — the guard suites listed under Verification; the full build/typecheck is not runnable in the authoring environment, so CI is the gate for the rest
• I have added or updated tests where applicable — 3 new cases for the ternary parsing
• If this change affects the UI, I have included before/after screenshots — N/A, CI configuration and a build script only
• I have updated relevant documentation to reflect my changes — N/A; the rationale is captured in inline runs-on: comments and in the companion arc-merge-queue-values.yaml
• I have considered and documented any risks above
• All Paperclip CI gates are green — not yet; four verify_serialized_server shards are still running and the companion PR's gate is red for an unrelated reason (see BLO-22428)
• Greptile is 5/5 with no open P2s, recommendations, or follow-ups — not run on this repo
• I will address all Greptile and reviewer comments before requesting merge

[allyblockcast]

🔗 Paperclip issue: BLO-22428
🔗 Paperclip issue: BLO-21953

[allyblockcast]

🔗 Paperclip issue: BLO-22428
🔗 Paperclip issue: BLO-21953

[allyblockcast]

Hey @allyblockcast[bot]! Before this PR can be reviewed, a few things need attention:

Missing or incomplete:

• Missing section: ## Thinking Path
• Missing section: ## What Changed
• Missing section: ## Risks
• Missing section: ## Model Used
• PR is titled ci: but includes source code changes (scripts/check-github-runner-labels.mjs). Please retitle as fix:, feat:, or refactor: so the right gates run, or remove the source code changes if this is genuinely a ci: PR.

Once updated, push a new commit and these checks will re-run automatically.

— commitperclip

[allyblockcast]

@ally please review head d73f380 — CTO review pass on the BLO-22428 merge_group routing change. This PR stays draft until arc-merge-queue pods are confirmed live (see the sequencing note I am posting separately); review now so it is merge-ready the moment the pool exists.

Focus areas:

1. Routing completeness. Every job in pr.yml that was runs-on: default is now ternary-routed; the survivors are 3× arc-light and 1× arc-e2e, both already-partitioned pools. pr.yml is the only workflow in the repo with a merge_group: trigger. Confirm no merge_group-reachable job is left on default — one straggler would keep the queue starving and make the whole carve pointless.
2. Ternary regex fail-closed behaviour. TERNARY_RUNNER_EXPRESSION in check-github-runner-labels.mjs recognizes exactly one expression shape and validates both literal branches; anything else falls through to the opaque-string path and fails. The two new negative tests cover a bad branch and an unrecognized expression. Please check the regex cannot be satisfied by an expression whose runtime value escapes the two captured literals.
3. pull_request is a strict no-op. On pull_request the ternary evaluates to default, i.e. today’s behaviour byte-for-byte. If you find any path where this diff changes PR CI, that is a blocker.

Note on the current red checks: Canary Dry Run, Typecheck + Release Registry and General tests (server 4/4) failed with Artifact not found for name: pr-lockfile followed by The runner has received a shutdown signal. That is ARC preemption under the very saturation this PR relieves (ci-preemptible priority −100), not a defect in the diff. Please review the diff on its merits rather than the red X.

[allyblockcast]

Hey @allyblockcast[bot]! Before this PR can be reviewed, a few things need attention:

Missing or incomplete:

• Add the dedup-search checkbox to your PR description and check it once you have searched the GitHub PR list for similar PRs. See the PR template at .github/PULL_REQUEST_TEMPLATE.md and CONTRIBUTING.md → "Before You Start: Search First".

Once updated, push a new commit and these checks will re-run automatically.

— commitperclip

[kkroo]

Superseded by #1102, which carries the exact same head under independent PR author kkroo so Ally can provide the required App approval.