fix(heartbeat): dedupe concurrent process_lost reaps

[kkroo]

Thinking Path

• Paperclip is the app/control plane used to coordinate AI agent work and track each agent run through the heartbeat runtime.
• The server heartbeat service owns orphaned-run recovery, including the process_lost reaper and the Prometheus counter that pages on sustained loss.
• Production logs showed the same heartbeat run IDs being reaped multiple times within seconds while the alert numerator continued to climb.
• The metric was intentionally emitted after the status write, but the write used the unconditional setter, so concurrent reaper invocations could each finalize and count the same already-running row.
• The service already had a compare-and-swap helper for running-run finalization; the reaper needed to use it and skip terminal side effects when it loses that race.
• This pull request makes process_lost minting idempotent under concurrent reapers and adds a regression test that releases two reapers against the same run at once.
• The benefit is a more trustworthy process_lost alert and fewer duplicate retries/events from one underlying lost run.

Linked Issues or Issue Description

No standalone GitHub issue exists for this production incident. Bug details follow the bug-report template inline.

• Preflight: searched open/closed PRs for process_lost reaper dedupe and orphaned heartbeat runs process_lost; this PR is not an exact duplicate.
• What happened: paperclip-0 logged repeated reaped orphaned heartbeat runs messages for the same run IDs within seconds. The paperclip_process_lost_total numerator kept increasing even though each run should only be minted as process_lost once.
• Expected behavior: once one reaper invocation finalizes a running heartbeat run as terminal, other concurrent reapers must observe that they lost the transition and skip metrics, wakeup finalization, retries, run events, and issue-promotion side effects.
• Steps to reproduce: start two reapOrphanedRuns() calls, hold both after their active-run scan, release them together against the same dead local run, and observe that the old implementation reports/counts more than one reap.
• Paperclip version or commit: observed on Blockcast production around commit 769315676c7d1ee426ef38d1b2ff8d864d1e5db7; fixed by 3c0dded3711e1580307c04c7939e656938bfee44.
• Deployment mode: self-hosted server.
• Installation method: built from source / container image.
• Agent adapters involved: core bug; not adapter-specific.
• Database mode: external Postgres.
• Access context: system heartbeat worker.
• Relevant logs: reaped orphaned heartbeat runs repeated for 9ef9d480-1168-4bc4-958a-1dc7e1c96a3e and 936a3b6f-400e-4afb-87cf-a83a8478c27e within seconds.
• Related PRs: fix(worker): guard process death so crashes stop orphaning agent runs (BLO-19722) #891, fix(heartbeat): hold process_lost during cold-boot kube-blind window so live Jobs reattach across restart (BLO-12564) #686, fix(heartbeat): keep live k8s runs out of orphan reaper #510, fix(heartbeat): stop reaper from killing live pre-adapter runs (BLO-13176) #576, fix(heartbeat): reaper must ignore updatedAt churn for external-lifecycle runs (BLO-8827) #287.

What Changed

• Changed the orphan reaper process_lost finalization path to use setRunStatusIfRunning() instead of the unconditional status setter.
• Added a loser branch that clears local tracking for already-terminal rows and skips duplicate process_lost metrics, wakeup status changes, retries, run events, and issue-promotion side effects.
• Added a regression test that runs two concurrent reapers against the same local run and asserts only one reap/event is recorded.

Verification

• corepack pnpm --filter @paperclipai/server typecheck
• corepack pnpm exec vitest run server/src/__tests__/heartbeat-process-recovery.test.ts -t "deduplicates concurrent process_lost reaps" attempted locally; the file skipped because embedded Postgres support failed to initialize on this host with Postgres init script exited with code 1. The data directory might already exist. CI should execute the test on a healthy embedded-Postgres runner.
• git diff --check
• Production short-window metric check showed no fresh claude_k8s/opencode_k8s pre-adapter process_lost churn while this fix was prepared.

Risks

• Low migration risk: no schema or data migration.
• Behavioral risk is constrained to the orphan reaper's duplicate-race path. The first reaper to transition running -> failed still performs the existing terminal side effects.
• If a run remains running after a failed CAS unexpectedly, this patch skips that reaper pass; a later periodic reap can try again.

Model Used

OpenAI Codex, GPT-5 coding agent, tool-using mode with local shell, Kubernetes, Prometheus, GitHub CLI, and TypeScript typecheck verification.

Checklist

• I have included a thinking path that traces from project context to this change
• I have specified the model used (with version and capability details)
• I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work
• I have searched GitHub for duplicate or related PRs and linked them above
• I have either (a) linked existing issues with Fixes: # / Closes # OR (b) described the issue in-PR following the relevant issue template
• I have run tests locally and they pass
• I have added or updated tests where applicable
• If this change affects the UI, I have included before/after screenshots
• I have updated relevant documentation to reflect my changes
• I have considered and documented any risks above
• All Paperclip CI gates are green
• Greptile is 5/5 with no open P2s, recommendations, or follow-ups
• I will address all Greptile and reviewer comments before requesting merge

[cursor]

Bugbot is not enabled for your account, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.

[allyblockcast]

Hey @kkroo! Before this PR can be reviewed, a few things need attention:

Missing or incomplete:

• Missing section: ## Thinking Path
• Missing section: ## What Changed
• Missing section: ## Verification
• Missing section: ## Risks
• Missing section: ## Model Used
• No linked issue or inline issue description found — either tag an existing issue with Fixes #NNN / Closes #NNN / Refs #NNN, or describe the underlying issue inline in the PR body following one of our issue templates (https://github.com/paperclipai/paperclip/tree/master/.github/ISSUE_TEMPLATE). See CONTRIBUTING.md → "Link Issues or Describe Them In-PR".
• Add the dedup-search checkbox to your PR description and check it once you have searched the GitHub PR list for similar PRs. See the PR template at .github/PULL_REQUEST_TEMPLATE.md and CONTRIBUTING.md → "Before You Start: Search First".

Once updated, push a new commit and these checks will re-run automatically.

— commitperclip