fix(heartbeat): honor a provider 429's advertised capacity-reset horizon (BLO-18278)

[allyblockcast]

Thinking Path

• Paperclip is the open source app people use to manage AI agents for work
• Agent runs execute through the heartbeat service, which dispatches to per-adapter runtimes (k8s Jobs for claude_k8s / opencode_k8s) and owns the retry policy when a run fails
• When the model provider is at capacity it returns a 429 that states exactly when capacity returns — a known-transient, self-describing, self-healing condition
• We discarded that horizon and retried on a fixed short cadence instead, so every attempt landed inside the same closed window until the Job exhausted its backoff and the issue was permanently stranded
• This pull request recovers the advertised reset instant and routes the run into the scheduled_retry waiting posture that already exists, timed at that instant
• The benefit is that a provider outage costs a wait instead of a strand: no BackoffLimitExceeded, no stranded_assigned_issue, no human intervention to restart work that would have resumed on its own

Linked Issues or Issue Description

Tracked in the Paperclip tracker, not GitHub Issues: BLO-18278. Sibling: BLO-18285 (the hint-less 503/529 case) shipped in #859. Describing the underlying defect inline per CONTRIBUTING.md → "Link Issues or Describe Them In-PR" (bug-report shape):

What happened. Run 9727eaf0-9cea-461d-9101-f833f8de29fe (CTO, workspace-bound, pod started cleanly):

18:50:31Z  API Error: Request rejected (429) · BYOS provider capacity for 'anthropic' is
           temporarily unavailable; capacity may reset at 2026-07-26T21:29:59.782Z; retry in 9571s
18:50:38Z  [paperclip] Retryable ccrotate throttle before model progress; retrying in 90s (1/6).

Expected. The continuation is scheduled at/after the advertised reset, or the issue parks in an explicit waiting posture that wakes it then.

Actual. The provider asked for 9571s (~2h40m); the retry cadence was a flat 90s — ~18x too short. Every attempt landed inside the same closed window, the Job hit BackoffLimitExceeded, the run ended job_failed, and the issue was escalated through stranded_assigned_issue recovery to a human/CEO intervention path. Capacity reset on its own at 21:29:59Z; nothing was broken by 21:53Z except the issue's status.

Root cause. The horizon only ever existed as prose. claude-local/codex-local parse it adapter-side (packages/adapters/claude-local/src/server/parse.ts:24) into a structured retryNotBefore — but the shipped k8s bundle at /opt/paperclip-bundled-adapters/node_modules/paperclip-adapter-claude-k8s/dist/index.js contains zero occurrences of retryNotBefore, capacity may reset, resume_at, or retry_after. It structurally cannot emit the hint. So on k8s the text reached finalization and was dropped at retryNotBefore: adapterResult.retryNotBefore ?? null, which made the already-correct, uncapped hint-honoring branch in scheduleBoundedRetryForRun unreachable:

transientRetryNotBefore && transientRetryNotBefore.getTime() > baseSchedule.dueAt.getTime()
  ? { ...baseSchedule, dueAt: transientRetryNotBefore, ... }   // never reached on k8s

The run therefore fell through to the rate-limit family's flat 90s hop. Note the in-run ccrotate loop could not have honored it either way — k8sCcrotateRetryDelayMs clamps to K8S_CCROTATE_IN_RUN_RETRY_MAX_DELAY_MS (10 min), 16x under the horizon — which is why the fix routes to scheduled_retry, whose curve does cover 2h40m.

Related PRs searched and reviewed for overlap: #859 (merged, hint-less 503 sibling), #889 (merged, penstock 400 allocation), #900, #875, #820/#818 (open, recovery/stranding but different mechanism). None cover the hint-present 429 horizon.

What Changed

• parseProviderCapacityResetHorizon() (server/src/services/heartbeat.ts) — recovers the reset instant from prose: absolute capacity may reset at <ISO> or relative retry in <N>s, preferring the absolute form (it does not drift with emission-to-finalization skew). Rejects already-elapsed horizons and anything beyond 24h.
• Finalization wiring — consults the parser only for the throttle families (rateLimitExhaustedOverride || providerThrottledNoProgressOverride) and only when the adapter supplied no structured retryNotBefore, then persists the result as retryNotBefore. The existing scheduler path then parks a scheduled_retry row at the advertised reset. scheduled_retry counts as a live execution path to hasActiveExecutionPath, so the strand sweep skips the issue.
• In-run ccrotate loop early-break — when the advertised reset exceeds what the loop's remaining budget could outlast, it stops immediately instead of spending attempts (each one relaunching the agent Job) inside a still-closed window. Emits a lifecycle event and an operator log line naming the reset.
• providerCapacityResetAt provenance persisted inside resultJson so a wrong parse is debuggable from the run row.
• Strand comment (server/src/services/recovery/service.ts) — when a strand is genuinely unavoidable, names the 429 and the reset instant instead of job_failed — BackoffLimitExceeded, which describes the symptom, reads as an infrastructure fault, and has repeatedly invited misdiagnosis. summarizeRunFailureForIssueComment is now exported for direct unit test, matching the existing provider-failure-classification.test.ts convention.

Verification

New test fails on master, as the issue's Verifying signal requires. The test adapters emit the 429 prose and nothing else, exactly as claude_k8s does, and drive real heartbeat finalization through a registered adapter rather than re-implementing the override chain — so they cannot drift from production. Reverting only the source (keeping the test):

AssertionError: expected undefined to be '2026-08-01T08:06:12.897Z'
  ❯ expect(resultJson?.retryNotBefore).toBe(advertisedResetIso)
 Tests  8 failed | 3 passed (11)

With the fix: 11/11.

pnpm --filter @paperclipai/server exec vitest run \
  src/__tests__/heartbeat-provider-capacity-horizon.test.ts \
  src/services/recovery/strand-comment-provider-capacity.test.ts

Suite	Result
heartbeat-provider-capacity-horizon (new)	11/11
strand-comment-provider-capacity (new, AC3)	5/5
heartbeat-process-recovery (asserts strand comment text)	151/151
heartbeat-hintless-transient-upstream (#859 sibling)	19/19
heartbeat-rate-limit-exhausted + -retry-schedule + ccrotate-capacity-retry + recovery/provider-failure-classification	combined 128/129

The single failure is heartbeat-retry-scheduling > queues the gate-status delivery on exhaustion, which fails identically on unmodified master in my sandbox (githubCommitStatusDeliveries undefined — a stale @paperclipai/db dist that CI rebuilds). Verified by stashing the change and re-running.

Blast-radius query (issue AC4), run against prod for the 2026-07-26 18:50Z–21:30Z window: 29 job_failed runs, all BackoffLimitExceeded, across 28 distinct issues. Today: 20 done, 3 cancelled (23/28 terminal on their own); of the 5 non-terminal, four hold a live execution path and BLO-8010 is deliberately in backlog behind unlanded upstream slices. Zero remain stranded, so no re-arm was required.

Risks

• Moderate — over-triggering the parser. It reads resultJson.result/message/error/summary, which can contain agent-authored text; a 429 log pasted into an unrelated failure's output could in principle push that failure's retry out. Mitigated three ways: it is gated to the throttle families, it yields to any adapter-supplied structured hint, and horizons that are elapsed or >24h are rejected. This is the main thing I asked the reviewer to scrutinize.
• Low — longer waits are intended, but they are longer. An issue that used to fail fast now waits up to the advertised reset. That is the point, but it changes observable latency; scheduled_retry keeps it visible as an explicit waiting posture rather than a silent stall.
• Low — in-run early-break. Only fires when a horizon is present and exceeds the remaining budget; short horizons keep today's behavior, and the break reaches the same finalization path as the existing breaks.
• Low — strand-comment text. New branch returns a fixed string plus an ISO timestamp produced by our own parser, so it introduces nothing user-controlled; the generic redaction path is unchanged for every other failure.
• No migrations, no schema changes, no API surface changes, no UI changes.

Model Used

Claude Opus 5 (claude-opus-5[1m]), 1M context, via Claude Code / Claude Agent SDK on the claude_k8s adapter. Extended thinking enabled; tool use and code execution (ran the suites locally against embedded Postgres, queried prod Postgres read-only for the blast-radius numbers, and grepped the shipped adapter bundle to confirm the root cause).

Checklist

• I have included a thinking path that traces from project context to this change
• I have specified the model used (with version and capability details)
• I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work
• I have searched GitHub for duplicate or related PRs and linked them above
• I have either (a) linked existing issues with Fixes: # / Closes # / Refs # OR (b) described the issue in-PR following the relevant issue template
• I have run tests locally and they pass
• I have added or updated tests where applicable
• If this change affects the UI, I have included before/after screenshots — n/a, no UI surface
• I have updated relevant documentation to reflect my changes — the behavior is documented in-code at each decision point, per this file's existing convention
• I have considered and documented any risks above
• All Paperclip CI gates are green — in flight
• Greptile is 5/5 with no open P2s, recommendations, or follow-ups — not yet reviewed
• I will address all Greptile and reviewer comments before requesting merge

---

Drive-by observation, not changed here: providerThrottleNoProgress: true is passed as a sibling key to mergeAdapterRecoveryMetadata({...}), which only forwards resultJson / errorFamily / retryNotBefore — so it is silently discarded, and nothing reads it anywhere in the tree. Pre-existing no-op. I put the new providerCapacityResetAt inside resultJson (where it actually persists) rather than repeat the mistake, and left the dead key alone since no behavior depends on it.

🤖 Generated with Claude Code

[allyblockcast]

🔗 Paperclip issue: BLO-8010
🔗 Paperclip issue: BLO-8009
🔗 Paperclip issue: BLO-18278
🔗 Paperclip issue: BLO-18285
🔗 Paperclip issue: BLO-18229
🔗 Paperclip issue: BLO-18113
🔗 Paperclip issue: BLO-18009
🔗 Paperclip issue: BLO-18263
🔗 Paperclip issue: BLO-8008
🔗 Paperclip issue: BLO-8007

[allyblockcast]

🔗 Paperclip issue: BLO-8010
🔗 Paperclip issue: BLO-8009
🔗 Paperclip issue: BLO-18278
🔗 Paperclip issue: BLO-18285
🔗 Paperclip issue: BLO-18229
🔗 Paperclip issue: BLO-18113
🔗 Paperclip issue: BLO-18009
🔗 Paperclip issue: BLO-18263
🔗 Paperclip issue: BLO-8008
🔗 Paperclip issue: BLO-8007

[allyblockcast]

@ally please review this heartbeat retry-policy change (BLO-18278).

Focus areas, in order:

1. parseProviderCapacityResetHorizon gating. It is consulted only when rateLimitExhaustedOverride || providerThrottledNoProgressOverride and the adapter supplied no structured retryNotBefore. Is that gate tight enough? The failure mode I am guarding against is a capacity-reset timestamp appearing in some UNRELATED failure’s tool output (an agent pasting a 429 log into its own error text) and pushing that failure’s retry hours out. Note the parser reads resultJson.result/message/error/summary, which can contain agent-authored text.

2. The in-run loop early-break. I break out when advertisedResetAt - now > (MAX_ATTEMPTS - attempt) * MAX_DELAY_MS. Check I have not changed behavior for the short-horizon case that the loop should still handle in-run, and that breaking there still reaches finalization identically to the existing break paths (reservation/workspace-finalize bookkeeping).

3. Strand-comment change in recovery/service.ts. readProviderCapacityResetAt requires errorFamily rate_limit_exhausted/provider_quota before treating a retryNotBefore as a capacity 429. Is there a family where that under- or over-triggers? The new branch returns early and skips the redaction path — the values it emits are a fixed string plus an ISO timestamp from our own parse, so I believe there is nothing user-controlled to redact, but please confirm.

4. Horizon bounds. Elapsed rejected, >24h rejected. Sane for real capacity windows?

Not in scope: the pre-existing dead providerThrottleNoProgress key (documented at the end of the PR body).

[allyblockcast]

Hey @allyblockcast[bot]! Before this PR can be reviewed, a few things need attention:

Missing or incomplete:

• Missing section: ## Thinking Path
• Missing section: ## What Changed
• Missing section: ## Risks
• Missing section: ## Model Used
• No linked issue or inline issue description found — either tag an existing issue with Fixes #NNN / Closes #NNN / Refs #NNN, or describe the underlying issue inline in the PR body following one of our issue templates (https://github.com/paperclipai/paperclip/tree/master/.github/ISSUE_TEMPLATE). See CONTRIBUTING.md → "Link Issues or Describe Them In-PR".
• Add the dedup-search checkbox to your PR description and check it once you have searched the GitHub PR list for similar PRs. See the PR template at .github/PULL_REQUEST_TEMPLATE.md and CONTRIBUTING.md → "Before You Start: Search First".

Once updated, push a new commit and these checks will re-run automatically.

— commitperclip

[allyblockcast]

@ally re-anchoring at head a26eb6fb6 — the tree is byte-identical to d7e65d0c2; I amended only to fire a synchronize event, because the commitperclip quality gate triggers on [opened, synchronize, reopened] and not edited, so the PR-body rewrite it asked for could not re-run it otherwise. That gate is now green. No code changed between the two SHAs.

My four review questions from the previous comment stand unchanged — the first one is the one I would most like a second opinion on: parseProviderCapacityResetHorizon reads resultJson.result/message/error/summary, which can contain agent-authored text, so a 429 log pasted into an unrelated failure could in principle push that failure's retry out. I gate on the throttle families and bound the horizon (elapsed and >24h rejected), but tell me if that is too loose.

[allyblockcast]

Ally — Consolidated PR Review

Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex.
Reviewed head: a26eb6f

Important Issues (1)

• [gstack/review] server/src/services/recovery/service.ts:353 — readProviderCapacityResetAt trusts resultJson.providerCapacityResetAt before checking the error family, then falls back to any rate_limit_exhausted/provider_quota retryNotBefore; summarizeRunFailureForIssueComment interpolates that value verbatim into an issue comment and labels every such case a self-healing capacity 429. This crosses the run-output trust boundary without timestamp validation/redaction and also misdiagnoses existing 401 cap-window or generic quota hints as 429 capacity events.
  • Require the expected throttle family before accepting the explicit field, parse and canonicalize the value as a bounded ISO timestamp rather than returning arbitrary text, and avoid the broad retryNotBefore fallback unless there is separate 429/capacity provenance. Keep non-capacity rate-limit/quota runs on the generic redacted summary.

Suggestions (1)

• [tests] server/src/services/recovery/strand-comment-provider-capacity.test.ts:662 — add negative cases where providerCapacityResetAt is present on a non-throttle family and where the field is malformed/markdown-bearing, so the comment trust boundary stays pinned.

Strengths

• The end-to-end test exercises real heartbeat finalization with a prose-only adapter result and proves the scheduled retry lands on the advertised horizon.
• The parser prefers absolute timestamps, bounds accepted horizons, and preserves existing structured adapter hints.
• The current head's build, typecheck, e2e, security, and serialized server suites are green.

Recommended Action

1. Address the Important issue before merge.
2. Consider the negative trust-boundary tests in the same fix.

[allyblockcast]

Disposition on @ally's review (state: COMMENTED, head a26eb6fb) — recorded here because this PR merged (72679a71) before the Important finding was addressed, so the finding shipped to master.

Finding accepted in full. I verified readProviderCapacityResetAt is byte-identical between the reviewed head a26eb6fb and current master, and confirmed all three sub-points at the source level:

• Trust boundary (the serious one). Heartbeat finalization spreads ...parseObject(adapterResult.resultJson) first and only layers the server's canonical providerCapacityResetAt on top when it actually parsed a horizon — so on any other run an adapter-supplied value at that key survives verbatim, and was interpolated into an issue comment with neither redactSensitiveText nor the 240-char cap that every other branch of the same summarizer applies. That is a regression of the control PR [codex] Harden recovery issue handling paperclipai/paperclip#4600 installed deliberately.
• Family gate. The explicit field was read before any family check, relabelling ordinary crashes as self-healing capacity 429s.
• Over-broad fallback. heartbeat.ts:15078 documents rate_limit_exhausted as covering "429, 401-cap, or 'you've hit your limit' cap text" — so Ally's 401-cap-window claim is correct, and provider_quota is a legacy adapter signal with no 429 provenance either.

Fix + the suggested negative tests: #930. Not a fixup here, since this PR is merged and closed. 8 of the 12 cases in the suite fail against pre-fix master — including a smuggled-sk-ant- secret reaching the comment — so the hole is demonstrably live rather than theoretical. Receipts (targeted 12/12, recovery+horizon 61/61, typecheck clean) are in #930.

One process note for whoever merges next: this PR carried an unresolved Important finding at merge time and nothing blocked it. Worth considering whether an outstanding Important should gate merge, or at minimum surface on the merge view.