fix(ci): distinguish cancelled from failed lanes, add mass-cancellation detector metric (BLO-21078)

[allyblockcast]

What

Two of BLO-21078's acceptance criteria, addressed independently:

1. verify's summary now distinguishes cancelled from failed lanes (.github/workflows/pr.yml)

Previously verify was a bare test "$X" = "success" chain — the first non-success result aborted the step with a generic exit code, so gh pr checks rendered a GitHub-side runner kill identically to a real test failure. Every affected PR author spent a cycle disproving a break they didn't cause (this literally happened during the 2026-08-02 19:34-19:57Z mass-cancellation incident this issue tracks).

Now the step names each lane and its result, splits them into cancelled vs failed in both the step summary and the ::error:: annotations, and still fails the check either way (a cancelled lane still needs a re-run) -- but the reason is now visible without opening job logs.

2. paperclip_github_workflow_run_conclusion_total (server/src/services/metrics.ts, server/src/routes/github-webhook.ts)

A new bounded Counter, incremented once per completed workflow_run webhook delivery, labeled only by conclusion (success/failure/cancelled/skipped/timed_out/action_required/neutral/stale/startup_failure/other). This is the numerator for the mass-cancellation detector alert rule (companion PR in Blockcast/onprem-k8s, since that's where this cluster's actually-firing Prometheus rules live -- see that repo's monitoring/prometheus-configmap.yaml header).

Deliberately excludes repo/workflow-name/PR labels to keep cardinality fixed -- this counter's whole job is "how many terminal runs of each kind arrived recently," which increase(...{conclusion="cancelled"}[window]) answers without them.

Why

Investigation on BLO-21078 found the reported "19:13-19:31Z window" framing was wrong -- the actual kill instants cluster in three separate waves (19:34:01-23Z, 19:36:12-18Z, 19:56:36-19:57:02Z) regardless of when each job had started, and post-cleanup steps completing after the cancelled step ruled out an ARC/k8s runner death (GitHub-side graceful cancellation instead). Full writeup on the issue. The incident had zero metric surface -- this PR is the fix for that gap, plus the verify-summary fix so the next occurrence doesn't cost every affected author a debugging cycle.

Test plan

• npx tsc --noEmit on server/ -- 0 new errors (confirmed against baseline via git stash, 39 pre-existing unrelated errors before and after)
• npx vitest run server/src/__tests__/metrics-service.test.ts -- 50/50 pass, including 4 new tests for the counter (registration, bounded-label normalization, accumulation, unknown-conclusion collapse)
• node deploy/helm/paperclip/tests/prometheus-rule.test.mjs -- 5/5 pass (chart still renders valid YAML with the mirrored alert added)
• Manually unit-tested the new bash logic in verify against all-success / all-cancelled / mixed-failure-and-cancelled / skipped-dependency scenarios
• Full github-webhook.test.ts integration suite (embedded-postgres-backed) -- did not complete in the sandbox within a reasonable timeout; relying on CI for that pass

🤖 Generated with Claude Code

[allyblockcast]

🔗 Paperclip issue: BLO-21078

[allyblockcast]

🔗 Paperclip issue: BLO-21078

[allyblockcast]

Hey @allyblockcast[bot]! Before this PR can be reviewed, a few things need attention:

Missing or incomplete:

• Missing section: ## Thinking Path
• Missing section: ## What Changed
• Missing section: ## Verification
• Missing section: ## Risks
• Missing section: ## Model Used
• No linked issue or inline issue description found — either tag an existing issue with Fixes #NNN / Closes #NNN / Refs #NNN, or describe the underlying issue inline in the PR body following one of our issue templates (https://github.com/paperclipai/paperclip/tree/master/.github/ISSUE_TEMPLATE). See CONTRIBUTING.md → "Link Issues or Describe Them In-PR".
• Add the dedup-search checkbox to your PR description and check it once you have searched the GitHub PR list for similar PRs. See the PR template at .github/PULL_REQUEST_TEMPLATE.md and CONTRIBUTING.md → "Before You Start: Search First".

Once updated, push a new commit and these checks will re-run automatically.

— commitperclip

[allyblockcast]

Hey @allyblockcast[bot]! Before this PR can be reviewed, a few things need attention:

Missing or incomplete:

• Missing section: ## Thinking Path
• Missing section: ## What Changed
• Missing section: ## Verification
• Missing section: ## Risks
• Missing section: ## Model Used
• No linked issue or inline issue description found — either tag an existing issue with Fixes #NNN / Closes #NNN / Refs #NNN, or describe the underlying issue inline in the PR body following one of our issue templates (https://github.com/paperclipai/paperclip/tree/master/.github/ISSUE_TEMPLATE). See CONTRIBUTING.md → "Link Issues or Describe Them In-PR".
• Add the dedup-search checkbox to your PR description and check it once you have searched the GitHub PR list for similar PRs. See the PR template at .github/PULL_REQUEST_TEMPLATE.md and CONTRIBUTING.md → "Before You Start: Search First".

Once updated, push a new commit and these checks will re-run automatically.

— commitperclip

[allyblockcast]

Ally — Consolidated PR Review

Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex.
Reviewed head: 5a28116

Important Issues (1)

• [gstack/review + native-codex] .github/workflows/pr.yml:393 — The workflow equates every cancelled result with an infra/runner kill and tells authors it is not attributable to the PR, but cancelled carries no cause and this workflow itself has concurrency.cancel-in-progress: true. A newer push, a manual cancellation, or another intentional workflow cancellation can therefore produce this message without any runner failure. The same unsupported attribution also leaks into the mass-cancellation alert's “GitHub Actions is mass-cancelling” wording.
  • Report the state neutrally (for example, “cancelled; inspect the run/cancellation source”) and reserve infra/non-attributable language for a separate signal that actually identifies runner loss. Keep the alert framed as a cancellation burst unless it gains evidence that distinguishes expected concurrency cancellation from infrastructure failure.

Strengths

• The metric label is normalized to a fixed allow-list and pre-initialized, keeping cardinality bounded and making zero-valued series observable.
• The required verify check still fails closed for every non-success lane while presenting all lane outcomes instead of stopping at the first failure.

Recommended Action

1. Correct the cancellation attribution before merge.
2. Re-run the current failing CI lanes after the runner communication incident clears.

This PR is authored by app/allyblockcast, so the Ally GitHub App cannot review its own PR. The exact head 5a28116742815a30531e723ebfd2eb651a296dde must be reopened under an independent author before an App approval is possible; the shared User credential is not substitute gate evidence.

[allyblockcast]

Superseded by #989. Split into a clean cut against current master — this branch's pr.yml rewrite duplicated and would have regressed the skipped-lane handling already merged via #964 (BLO-20867 AC-3), and this branch is heavily stale (predates several dozen master commits, mergeStateStatus was CONFLICTING). Kept only the metrics counter addition, cleanly, in #989. See https://paperclip.blockcast.net/BLO/issues/BLO-20867 for the full context.