fix(heartbeat): escalate a repeated gateway allocation_missing fault instead of retrying forever (BLO-21803)

[allyblockcast]

Thinking Path

• Paperclip is the open source app people use to manage AI agents for work.
• The heartbeat service owns run finalization, automatic retry scheduling, and issue escalation when a run cannot make progress.
• BLO-19879 correctly treats a first allocation_missing gateway response as a hintless transient upstream fault so a brief routing gap can self-heal once.
• BLO-21803 showed the same org/provider/node allocation fault recurring on the scheduled retry itself, which means the system was preserving a stuck-looking issue while continuing a retry chain that could not succeed.
• This pull request separates gateway allocation fault detection from the broader hintless transient predicate so the finalize path can identify repeated allocation gaps specifically.
• The benefit is that a repeated BYOS allocation gap escalates to the existing blocked/stranded issue path with an actionable message instead of retrying indefinitely.

Linked Issues or Issue Description

Refs: BLO-21803

allocation_missing (BLO-19879's penstock gateway routing fault) was retried indefinitely instead of self-healing once. Two independent live reproductions showed the identical org_penstock / anthropic / blockcast-omar fault recurring on the next attempt of the same retry chain, days apart. That is a standing BYOS allocation gap, not a routing blip, so further retries only burn retry budget while leaving the issue looking alive.

What Changed

• Split isGatewayAllocationFault out of isHintlessTransientUpstreamFault without changing the first-fault transient classification.
• Tagged repeated gateway allocation failures on scheduled retry runs as allocation_missing_standing.
• Let repeated allocation gaps fall through to the existing untagged-failure escalation path (stranded_assigned_issue -> blocked) instead of scheduling another bounded retry.
• Added coverage for the gateway allocation matcher and for the no-retry behavior on allocation_missing_standing.

Verification

• pnpm exec tsc --noEmit in server/ completed with no new errors introduced; existing unbuilt-dependency errors were unchanged.
• pnpm exec vitest run src/__tests__/heartbeat-hintless-transient-upstream.test.ts passed 26/26, including the new matcher and retry-stop coverage.
• Searched GitHub PRs for BLO-21803 allocation_missing; only this PR matched.

Risks

Low-to-medium behavioral risk: the first allocation_missing response still gets the same bounded transient retry, but a repeated allocation failure on the scheduled retry now escalates to blocked. If a provider begins returning repeated allocation_missing during a longer-lived but recoverable outage, a human may need to unblock after the upstream allocation is repaired.

Not in scope / investigated and closed:

• Leaked slot-3 runtime reservation (BLO-21803 defect 2) was falsified live via Prometheus. The contextSnapshot shape that looked like a leak is a frozen run-creation snapshot artifact, and the live reservation gauges did not show the cited long-held reservation.
• Run 00fb7575 is no longer stuck: it ran and failed with a legitimate provider_throttled_no_progress fault, not allocation_missing, and has a live scheduled successor within its retry budget.
• The dashboard signal mentioned in BLO-21803, paperclip_queued_run_oldest_age_seconds, is not present in the metrics registry and is left for the owner of that acceptance criterion.

Model Used

Paperclip agent authored the branch. The exact model/provider was not recorded in the commit metadata. Codex GPT-5 was used for PR description cleanup and CI triage only.

Checklist

• I have included a thinking path that traces from project context to this change
• I have specified the model used (with version and capability details)
• I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work
• I have searched GitHub for duplicate or related PRs and linked them above
• I have either (a) linked existing issues with Fixes: # / Closes # / Refs # OR (b) described the issue in-PR following the relevant issue template
• I have run tests locally and they pass
• I have added or updated tests where applicable
• If this change affects the UI, I have included before/after screenshots
• I have updated relevant documentation to reflect my changes
• I have considered and documented any risks above
• All Paperclip CI gates are green
• Greptile is 5/5 with no open P2s, recommendations, or follow-ups
• I will address all Greptile and reviewer comments before requesting merge

[allyblockcast]

🔗 Paperclip issue: BLO-19879
🔗 Paperclip issue: BLO-20725
🔗 Paperclip issue: BLO-19411
🔗 Paperclip issue: BLO-21803

[allyblockcast]

🔗 Paperclip issue: BLO-19879
🔗 Paperclip issue: BLO-20725
🔗 Paperclip issue: BLO-19411
🔗 Paperclip issue: BLO-21803

[allyblockcast]

Hey @allyblockcast[bot]! Before this PR can be reviewed, a few things need attention:

Missing or incomplete:

• Missing section: ## Thinking Path
• Missing section: ## What Changed
• Missing section: ## Verification
• Missing section: ## Risks
• Missing section: ## Model Used
• Add the dedup-search checkbox to your PR description and check it once you have searched the GitHub PR list for similar PRs. See the PR template at .github/PULL_REQUEST_TEMPLATE.md and CONTRIBUTING.md → "Before You Start: Search First".

Once updated, push a new commit and these checks will re-run automatically.

— commitperclip

[allyblockcast]

Ally — Consolidated PR Review

Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex.
Reviewed head: b5fdab9

Important Issues (1)

• [code / gstack / native-codex] server/src/services/heartbeat.ts:21759 — scheduledRetryAttempt >= 1 proves only that this run is a retry, not that its predecessor failed with allocation_missing. A chain can begin with any retryable family (for example a 503 or provider throttle) and then encounter its first transient allocation-routing fault on attempt 1; this branch labels that first occurrence allocation_missing_standing, suppresses its intended self-heal retry, and sends the issue down the stranded/blocked path. The added unit test starts after the misclassification and therefore does not cover this mixed-family transition.
  • Persist or inspect a chain-specific allocation-fault marker from the immediately preceding run, and escalate only when both the predecessor and current run match the gateway allocation fault. Add an integration-level test for both allocation_missing -> allocation_missing and other transient -> allocation_missing chains.

Strengths

• The classifier extraction preserves the existing status gate and keeps allocation faults disjoint from rate-limit handling.
• The new terminal code and operator-facing message make a genuinely repeated provisioning gap actionable.

Recommended Action

1. Address the Important issue this cycle before merge.

This PR is authored by app/allyblockcast; the allyblockcast reviewer App cannot review its own PR. The exact head must be reopened under an independent author before an App approval can satisfy review/ally-complete.

[kkroo]

Reviewed the heartbeat finalization change. The split gateway allocation matcher preserves the first-attempt transient retry while preventing repeated scheduled-retry allocation_missing faults from rearming the bounded retry chain; focused tests cover the matcher and no-retry terminal code path.