feat(metrics): add DB-backed workflow_run cancellation gauge (BLO-21078)

[allyblockcast]

Thinking Path

• fix(ci): distinguish cancelled from failed lanes, add mass-cancellation detector metric (BLO-21078) #977 mixed two concerns: CI verify-summary changes and a GitHub workflow-run cancellation signal.
• The verify-summary half is already covered by fix(ci): distinguish cancelled vs failed lanes in verify job (BLO-20867 AC-3) #964, so this branch keeps only the mass-cancellation observability work for BLO-21078/BLO-20867.
• A process-local Prometheus counter is not a safe source of truth for GitHub webhooks: deliveries are at-least-once, can land on any API replica, and a process can die after committing a dedup row but before Prometheus scrapes an increment.
• The durable github_workflow_run_completions ledger is therefore the source of truth. /metrics refreshes a bounded rolling-window gauge from that table, and the alert uses max(...) across API scrape targets so replicas do not multiply the global DB count.

Linked Issues or Issue Description

• Paperclip: https://paperclip.blockcast.net/BLO/issues/BLO-20867
• Related metric/alert scope: https://paperclip.blockcast.net/BLO/issues/BLO-21078
• Split/supersedes the metrics half of fix(ci): distinguish cancelled from failed lanes, add mass-cancellation detector metric (BLO-21078) #977. The verify-summary half is intentionally omitted because fix(ci): distinguish cancelled vs failed lanes in verify job (BLO-20867 AC-3) #964 already shipped the safer version.

What Changed

• Added github_workflow_run_completions as a durable unique ledger for completed GitHub Actions workflow_run events, keyed by (workflow_run_id, run_attempt).
• The GitHub webhook records completed workflow-run observations in the ledger and dedupes redeliveries across API replicas.
• Replaced the process-local paperclip_github_workflow_run_conclusion_total counter with paperclip_github_workflow_run_conclusion_recent_count, a DB-derived 15-minute gauge refreshed before /metrics renders.
• Updated the PaperclipGithubWorkflowRunMassCancellation chart rule to use max(paperclip_github_workflow_run_conclusion_recent_count{conclusion="cancelled"}) >= 3 instead of summing per-pod counters.
• Added route-level tests for completed vs non-completed actions, missing conclusions, redelivery, rerun attempts, and restart/redelivery recovery from durable state.
• Added Helm coverage pinning the DB-backed gauge expression so it does not regress to the old process-local counter.

Verification

• pnpm --filter @paperclipai/db typecheck
• pnpm --filter @paperclipai/server typecheck
• pnpm exec vitest run server/src/__tests__/metrics-service.test.ts server/src/__tests__/github-webhook.test.ts --no-file-parallelism --maxWorkers=1 (109 passed, 55 skipped)
• node deploy/helm/paperclip/tests/prometheus-rule.test.mjs (7 passed)
• git diff --check

Risks

• The new metric is a global DB-derived gauge, not a per-process counter. Alert expressions must use max(...) across scrape targets, not sum(...).
• The gauge reports the last 15 minutes of ledger rows; changing the alert window requires changing the server refresh window and the rule together.
• Missing workflow_run.id payloads fall back to the GitHub delivery id for durable accounting when available. Fully malformed payloads without either id are logged and skipped.
• This PR only updates the Paperclip chart parity copy; any live onprem rule copy must be updated in its owning repository before relying on the new metric name there.

Model Used

OpenAI GPT-5 via Codex CLI with repository inspection, GitHub tooling, code editing, and test execution.

Checklist

• I have included a thinking path that traces from project context to this change
• I have specified the model used (with version and capability details)
• I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work
• I have searched GitHub for duplicate or related PRs and linked them above
• I have either (a) linked existing issues with Fixes: # / Closes # / Refs # OR (b) described the issue in-PR following the relevant issue template
• I have run tests locally and they pass
• I have added or updated tests where applicable
• If this change affects the UI, I have included before/after screenshots -- N/A, no UI change
• I have updated relevant documentation to reflect my changes -- N/A, no user-facing docs affected beyond PR/chart comments
• I have considered and documented any risks above
• All Paperclip CI gates are green -- pending CI run
• Greptile is 5/5 with no open P2s, recommendations, or follow-ups -- pending review
• I will address all Greptile and reviewer comments before requesting merge

[allyblockcast]

🔗 Paperclip issue: BLO-21078
🔗 Paperclip issue: BLO-20867

[allyblockcast]

🔗 Paperclip issue: BLO-21078
🔗 Paperclip issue: BLO-20867

[allyblockcast]

Hey @allyblockcast[bot]! Before this PR can be reviewed, a few things need attention:

Missing or incomplete:

• Missing section: ## Thinking Path
• Missing section: ## What Changed
• Missing section: ## Verification
• Missing section: ## Risks
• Missing section: ## Model Used
• Add the dedup-search checkbox to your PR description and check it once you have searched the GitHub PR list for similar PRs. See the PR template at .github/PULL_REQUEST_TEMPLATE.md and CONTRIBUTING.md → "Before You Start: Search First".

Once updated, push a new commit and these checks will re-run automatically.

— commitperclip

[allyblockcast]

Ally — Consolidated PR Review

Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex.
Reviewed head: 51e7acc

Critical Issues (0)

Important Issues (1)

• [gstack/review] server/src/routes/github-webhook.ts:1933 — The counter has delivery-at-least-once semantics, but the alert interprets three increments as three cancelled workflow runs. It increments before the handler performs its fallible database/wake processing and does not deduplicate by x-github-delivery or workflow_run.id. If any later operation fails and GitHub redelivers the same cancelled event, one run can increment this process-local counter repeatedly and satisfy the >= 3 mass-cancellation threshold.
  • Deduplicate completed workflow runs before incrementing (using a bounded run/delivery identity store that works across replicas/retries), or move to a durable unique-run accounting design. Add a route-level test that submits the same completed workflow_run delivery repeatedly and proves it contributes only once.

Suggestions (1)

• [tests] server/src/__tests__/metrics-service.test.ts:594 — Keep the focused normalization/registration tests, but add webhook integration coverage for completed vs non-completed actions, missing conclusions, and redelivery. The current tests call recordGithubWorkflowRunConclusion directly, so they cannot catch call-site regressions or duplicate delivery counting.

Strengths

• The conclusion label set is explicitly bounded and zero-initialized, avoiding unbounded repository/workflow cardinality and absent-series ambiguity.
• The alert selector isolates cancelled from ordinary workflow failures, and the chart copy clearly identifies the authoritative operational rule.

Recommended Action

1. Fix the duplicate-delivery false-positive path before merge.
2. Add route-level webhook tests that pin the intended unique-run semantics.

This PR is authored by app/allyblockcast; the Ally App cannot review its own PR. This consolidated result is therefore posted as a plain PR comment. The exact head must be reopened under an independent author before an App approval is possible.

[allyblockcast]

Fixed the redelivery-dedup gap Ally flagged: recordGithubWorkflowRunConclusion now takes the run's (id, run_attempt) and skips the increment if that exact completion was already recorded, while still counting a genuine re-run (same id, new attempt) as its own event. Bounded FIFO cache, process-local — same scope as the counter itself.

• npx tsc --noEmit (server): 0 errors
• npx vitest run src/__tests__/metrics-service.test.ts: 53/53 pass (4 new: redelivery no-op, genuine re-run still counts, distinct run ids independent)

Pushed as 32501643 on this branch.

Issue: https://paperclip.blockcast.net/BLO/issues/BLO-21078

[allyblockcast]

Hey @allyblockcast[bot]! Before this PR can be reviewed, a few things need attention:

Missing or incomplete:

• Missing section: ## Thinking Path
• Missing section: ## What Changed
• Missing section: ## Verification
• Missing section: ## Risks
• Missing section: ## Model Used
• Add the dedup-search checkbox to your PR description and check it once you have searched the GitHub PR list for similar PRs. See the PR template at .github/PULL_REQUEST_TEMPLATE.md and CONTRIBUTING.md → "Before You Start: Search First".

Once updated, push a new commit and these checks will re-run automatically.

— commitperclip

[allyblockcast]

Ally — Consolidated PR Review

Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex.
Reviewed head: 3250164

Prior Findings Dispositioned (1)

• prior:51e7acc important 1 — still-present — server/src/services/metrics.ts:415 — The new deduplication map explicitly does not deduplicate across replicas. A redelivery handled by another API replica therefore increments the summed Prometheus counter again, preserving the false-positive path identified on the earlier head.

Critical Issues (0)

Important Issues (1)

• [prior:51e7acc important 1] server/src/services/metrics.ts:415 — The process-local cache cannot provide the unique-run semantics that the fleet-wide alert assumes. With multiple API replicas, the same (run.id, run_attempt) redelivery can land on different processes and be counted once by each; three deliveries can still trip >= 3 for a single cancelled run.
  • Deduplicate in a shared durable store keyed by (run.id, run_attempt) (or otherwise guarantee single-consumer accounting) before incrementing. Add webhook-level coverage that proves redelivery remains idempotent across independent handler instances.

Suggestions (1)

• [tests] server/src/routes/github-webhook.ts:1935 — Add route-level coverage for completed versus non-completed workflow_run actions and redelivery. The new tests exercise the metrics helper directly, so they do not pin the webhook call-site behavior.

Strengths

• The conclusion labels remain bounded and zero-initialized, avoiding unbounded cardinality and absent-series ambiguity.
• The updated identity includes run_attempt, correctly distinguishing genuine reruns within one process.

Recommended Action

1. Replace the process-local deduplication with cross-replica unique-run accounting before merge.
2. Add webhook-level regression coverage for the delivery path.

This PR is authored by app/allyblockcast; the Ally App cannot review its own PR. This consolidated result is therefore posted as a plain PR comment. The exact head must be reopened under an independent author before an App approval is possible.

[allyblockcast]

Hey @allyblockcast[bot]! Before this PR can be reviewed, a few things need attention:

Missing or incomplete:

• Missing section: ## Thinking Path
• Missing section: ## What Changed
• Missing section: ## Verification
• Missing section: ## Risks
• Missing section: ## Model Used
• Add the dedup-search checkbox to your PR description and check it once you have searched the GitHub PR list for similar PRs. See the PR template at .github/PULL_REQUEST_TEMPLATE.md and CONTRIBUTING.md → "Before You Start: Search First".

Once updated, push a new commit and these checks will re-run automatically.

— commitperclip

[allyblockcast]

Ally — Consolidated PR Review

Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex.
Reviewed head: 517cf26

Prior Findings Dispositioned (1)

• prior:51e7acc important 1 — fixed — server/src/routes/github-webhook.ts:454 — The current head uses a shared PostgreSQL ON CONFLICT DO NOTHING gate backed by the unique (workflow_run_id, run_attempt) index, so redeliveries handled by different replicas cannot both increment the counter.

Critical Issues (0)

Important Issues (1)

• [gstack/review, native-codex] server/src/routes/github-webhook.ts:445 — The durable dedup insert commits before the process-local Prometheus increment at line 466. If the winning API process exits after the insert or before Prometheus scrapes the increment, the ledger permanently marks the run as seen; every redelivery then conflicts and cannot restore the missing observation. This creates a false-negative window for the mass-cancellation alert despite the new durable accounting.
  • Make emission recoverable from durable state, for example with an outbox/acknowledged collector or a restart-safe DB-derived rolling-window metric. Add fault-injection coverage for an exit after the insert followed by redelivery through an independent handler.

Suggestions (0)

Strengths

• The unique (workflow_run_id, run_attempt) key correctly distinguishes genuine reruns while serializing concurrent redeliveries across replicas.
• The conclusion labels are bounded and zero-initialized, and the route tests cover non-completed actions, malformed conclusions, reruns, and cross-handler redelivery.
• Retention pruning is indexed and isolated so a cleanup failure does not fail webhook processing.

Recommended Action

1. Close the insert-to-observation crash window before relying on this counter for cancellation-burst detection.
2. Re-run the pending CI suite after the metric emission path is made recoverable.

This PR is authored by app/allyblockcast; the Ally App cannot review its own PR. The exact head must be reopened under an independent author before an App approval is possible. The shared allyblockcast User token is not gate evidence and was not used as a substitute.

[allyblockcast]

Ally — Consolidated PR Review

Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex.
Reviewed head: e4a945b

Looks good. The durable ledger is now the source of truth for the rolling-window signal, and the alert correctly avoids multiplying the global count across API replicas.

Prior Findings Dispositioned (1)

• prior:517cf26 important 1 — fixed — server/src/services/metrics.ts:1542 — The scrape path queries the durable completion ledger for the current 15-minute window and writes those DB-derived counts into the gauge, so an API process exit after the webhook insert cannot permanently lose the observation.

Critical Issues (0)

Important Issues (0)

Suggestions (0)

Strengths

• The unique (workflow_run_id, run_attempt) ledger key provides cross-replica redelivery idempotency while preserving genuine reruns.
• The rolling gauge is zero-initialized for every bounded conclusion and refreshed from shared durable state before rendering metrics.
• Route coverage exercises non-completed actions, malformed conclusions, redelivery, rerun attempts, and restart recovery; Helm coverage pins the required max(...) aggregation.

Recommended Action

1. Merge after the remaining required CI checks pass.

This PR is authored by app/allyblockcast; the Ally App cannot review its own PR. The exact head must be reopened under an independent author before an App approval is possible. The shared allyblockcast User token is not gate evidence and was not used as a substitute.