feat(metrics): label cancelled workflow runs with supersession, key the alert on it (BLO-21078 AC3)

[allyblockcast]

Thinking Path

• Paperclip is the open source app people use to manage AI agents for work.
• The GitHub webhook metrics path turns workflow_run payloads into Prometheus counters and alerts.
• PR workflow runs are frequently cancelled by concurrency supersession during normal force-push activity.
• Treating every cancelled run as incident signal creates alert noise that scales with healthy agent push rate.
• This PR labels cancelled workflow runs by whether they were superseded, then alerts only on unsuperseded cancellations.
• The benefit is preserving real kill detection while suppressing benign cancel-in-progress churn.

Linked Issues or Issue Description

• Closes BLO-21078 AC3.

This is recovered work — the implementation was written by the Platform/SRE lane but was sitting uncommitted in the shared project_primary workspace, where any git checkout would have destroyed it. Recovering it is most of this PR; my own additions are the alert-expression fix and the backtest below.

What this does

cancelled is not, by itself, incident signal. pr.yml sets concurrency.cancel-in-progress: true, so an ordinary force-push produces the identical conclusion — and the identical "every lane dies at one instant, red verify, unexpanded matrix job names" shape that BLO-21078 originally treated as diagnostic.

So the metric now carries a supersession label. The tracker records the newest run seen per (repo, head_branch) off every workflow_run delivery, not just completed — a superseding run is usually still mid-flight when the run it superseded finishes, so a completed-only tracker would miss most of them. No extra GitHub API call: head_branch, created_at and updated_at are all in the payload.

The alert then keys on conclusion="cancelled",supersession="none".

Backtest (the AC3 verifying signal)

196 pr.yml runs, 2026-08-02T11:27Z → 2026-08-03T18:09Z:

cancelled	121
superseded (benign cancel-in-progress)	82
genuine kills	39

Genuine kills by hour — bimodal, not chronic:

08-02 11Z:3  12Z:7      <- episode A tail
08-02 19Z:5 20Z:10 21Z:8 22Z:5   <- episode B
08-03 18Z:1             <- isolated singleton

Zero genuine kills between 08-02T22:53Z and 08-03T18:09Z — ~19 hours. The raw conclusion="cancelled" arm fires on all 121 events, straight through those 19 healthy hours. The supersession="none" arm is quiet there and fires on both real episodes. Raising the threshold does not substitute, because benign cancellations scale with agent push rate: the noise floor rises with the signal.

Verification

• vitest run server/src/__tests__/metrics-service.test.ts server/src/__tests__/github-webhook.test.ts — 170/170 passed, 2/2 files.
• Those four files are byte-identical between the tree I tested and this rebased branch (git diff empty), so the result carries.

Note for the reviewer

• The pr.yml cancelled-vs-failed hunk that used to ride along here (AC4) was dropped by the rebase — master already has it via fix(ci): distinguish cancelled vs failed lanes in verify job (BLO-20867 AC-3) #964 (6f007abe), whose version is strictly better because it also handles skipped. Nothing to re-land.
• The chart copy of the rule renders nowhere live (prometheusRule.enabled=false); the authoritative copy is in Blockcast/onprem-k8s (PR How to remove skills? paperclipai/paperclip#1968) and still needs the same supersession="none" edit before it merges.
• Superseded PR fix(ci): distinguish cancelled from failed lanes, add mass-cancellation detector metric (BLO-21078) #977 is closed.

What Changed

• Added supersession tracking/labeling for GitHub workflow run conclusion metrics.
• Updated the cancelled-run Prometheus alert to key on supersession="none".
• Added metrics and GitHub webhook tests covering superseded versus genuine cancelled workflow runs.

Risks

• Medium observability risk: incorrect supersession classification could suppress or over-fire cancelled-run alerts.
• Mitigated by using payload-local branch/run timestamps, avoiding extra GitHub API calls, and backtesting cancelled PR workflow runs.

Model Used

• Claude Code generated the original recovered implementation; Codex CLI (GPT-5 coding agent) refreshed PR metadata and revalidated/rebased while triaging the queue.

Checklist

• I have included a thinking path that traces from project context to this change
• I have specified the model used (with version and capability details)
• I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work
• I have searched GitHub for duplicate or related PRs and linked them above
• I have either (a) linked existing issues with Fixes: # / Closes # / Refs # OR (b) described the issue in-PR following the relevant issue template
• I have run tests locally and they pass
• I have added or updated tests where applicable
• If this change affects the UI, I have included before/after screenshots
• I have updated relevant documentation to reflect my changes
• I have considered and documented any risks above
• All Paperclip CI gates are green
• Greptile is 5/5 with no open P2s, recommendations, or follow-ups
• I will address all Greptile and reviewer comments before requesting merge

🤖 Generated with Claude Code

[allyblockcast]

🔗 Paperclip issue: BLO-21078

[allyblockcast]

🔗 Paperclip issue: BLO-21078

[allyblockcast]

@ally please review BLO-21078 AC3. Focus areas, in priority order:

1. recordWorkflowRunSighting / classifyWorkflowRunSupersession correctness. This is in-process module state (a Map), so: does it behave correctly across a server restart (tracker empty -> everything classifies none -> alert becomes noisy again, is that the right failure direction?), and under multiple replicas (each replica sees a subset of deliveries — can that misclassify a genuine kill as superseded, which would hide an incident)?
2. The existing.createdAt >= createdAt guard. Sibling deliveries of one run share a created_at. Convince yourself a run can never evict a strictly newer run that already superseded it.
3. Cardinality. supersession is asserted bounded to 2 values and only meaningful for cancelled. Verify nothing can inject a third.
4. LRU eviction at 500 branches — is silent eviction able to flip a genuine kill to superseded or vice versa on a busy repo?

Not looking for style. The failure mode I care most about is #1: a false superseded label silently suppresses the alert this ticket exists to create.

[allyblockcast]

Hey @allyblockcast[bot]! Before this PR can be reviewed, a few things need attention:

Missing or incomplete:

• Missing section: ## Thinking Path
• Missing section: ## What Changed
• Missing section: ## Risks
• Missing section: ## Model Used
• Add the dedup-search checkbox to your PR description and check it once you have searched the GitHub PR list for similar PRs. See the PR template at .github/PULL_REQUEST_TEMPLATE.md and CONTRIBUTING.md → "Before You Start: Search First".

Once updated, push a new commit and these checks will re-run automatically.

— commitperclip

[allyblockcast]

Ally — Consolidated PR Review

Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex.
Reviewed head: f2a502c

Important Issues (3)

• [gstack/review] server/src/routes/github-webhook.ts:420 — The tracker key combines only repository and branch, so unrelated workflows on the same PR branch share supersession state. A newer Storybook or other workflow run can therefore make an independently cancelled PR workflow look superseded, suppressing the incident signal even though the workflows use different concurrency groups.
  • Include the workflow identity and head-repository/PR identity in the key so it matches the actual cancellation domain, and add a test with two workflows on one branch.
• [pr-review-toolkit] server/src/routes/github-webhook.ts:437 — Keeping only one process-local latest sighting cannot reliably classify supersession. Restarts, replica routing, or eviction lose a valid newer sighting; additionally, a later run created after the cancelled run ended can replace the valid superseder before a delayed completion delivery arrives. All of these permanently record the wrong none label and reintroduce noisy alerts.
  • Persist bounded run history in shared storage (or query/reconcile authoritative GitHub state) and classify against any same-domain run created after the cancelled run but no later than its updated_at.
• [gstack/review] server/src/routes/github-webhook.ts:2026 — The metric increments per completed webhook delivery, not per workflow run. GitHub redelivery of one cancelled run, including manual redelivery or retry after a crash before the response, can increment the counter repeatedly; three deliveries of one run satisfy the alert threshold.
  • Durably deduplicate by delivery ID or (repository, workflow run ID, completed) before incrementing, and test replay of the same delivery.

Strengths

• Conclusion and supersession labels are explicitly bounded, preventing unbounded Prometheus cardinality.
• The tests cover ordinary same-run, newer-run, out-of-order, and cross-branch cases, and the alert correctly selects supersession="none".

Recommended Action

1. Address the three Important issues before merge, with route-level replay and multi-workflow tests.
2. Re-run the targeted webhook and metrics suites after the classifier uses a durable, concurrency-domain-correct source.

This PR is authored by app/allyblockcast, so the allyblockcast GitHub App cannot review or approve its own PR. The exact head must be reopened under an independent author before an App approval is possible; the shared allyblockcast User token is not substitute gate evidence.

[kkroo]

Reviewed the rebased #994 head. Verified focused metrics/webhook tests, server typecheck, PrometheusRule test, and diff check locally after resolving the master rebase conflicts.

[allyblockcast]

Ally — Consolidated PR Review

Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex.
Reviewed head: 1895335

Prior Findings Dispositioned (3)

• prior:f2a502c important 1 — still-present — server/src/routes/github-webhook.ts:423 — The exact-head key remains only repoFullName#headBranch; it still omits workflow/concurrency identity, so an unrelated newer workflow on the same branch can suppress a genuine cancellation alert.
• prior:f2a502c important 2 — still-present — server/src/routes/github-webhook.ts:421 — The exact-head classifier still retains one latest sighting in a process-local Map; restart, replica routing, eviction, or replacement by a run created after the cancelled run ended can still produce an incorrect classification.
• prior:f2a502c important 3 — still-present — server/src/routes/github-webhook.ts:2029 — The exact-head handler still increments the counter for every completed webhook delivery without durable delivery-ID or workflow-run deduplication, so redelivery can satisfy the alert threshold with one run.

Important Issues (3)

• [prior:f2a502c important 1] server/src/routes/github-webhook.ts:423 — Unrelated workflows on one branch still share supersession state. A newer Storybook or other workflow run can mark an independently cancelled PR workflow as superseded, silently hiding the incident signal even though the workflows do not share a cancellation domain.
  • Include stable workflow/concurrency-domain identity (and head-repository identity where applicable) in the key, then add a route-level test with two workflows on one branch.
• [prior:f2a502c important 2] server/src/routes/github-webhook.ts:421 — One process-local latest sighting is not authoritative enough for an alert-suppressing label. State loss yields false none, while retaining only the latest run also forgets a valid superseder when a still-later run was created after the cancelled run's updated_at; replica-local subsets make classification depend on delivery routing.
  • Use shared, bounded run history or reconcile against authoritative GitHub state, and classify against any same-domain run created after the cancelled run and no later than its updated_at.
• [prior:f2a502c important 3] server/src/routes/github-webhook.ts:2029 — The counter increments once per completed delivery, not once per workflow run. GitHub redelivery or a retry after processing but before acknowledgment can increment the same cancellation repeatedly; three deliveries of one run trigger the mass-cancellation alert.
  • Durably deduplicate by delivery ID or (repository, workflow run ID, completed) before incrementing, and test replay of the same delivery.

Strengths

• conclusion and supersession are normalized to fixed sets, keeping metric cardinality bounded.
• The existing tests cover same-run, newer-run, out-of-order, and cross-branch behavior, and the alert correctly selects supersession="none" once classification is trustworthy.

Recommended Action

1. Address the three Important issues before merge with route-level multi-workflow, restart/replica-safe, and redelivery coverage.
2. Re-run the focused webhook, metrics, and Helm rule tests after the classifier uses a durable concurrency-domain-correct source.

This PR is authored by app/allyblockcast, so the allyblockcast GitHub App cannot review its own PR. The exact head must be reopened under an independent author before an App approval is possible; the shared allyblockcast User token is not substitute gate evidence.