claim-reaper is not single-writer: concurrent same-host simard daemons race the claim ledger → naive no-worktree reclaim overrides another daemon's in-flight investigate-before-reap (false-reclaims a live engineer mid-recreate)

[rysweet]

Summary

The claim-reaper sweep (overseer::claim_reaper::reap_stale_claims) is not single-writer / not leader-gated. It mutates the shared engineer_claims ledger (release + worktree cleanup) with no lease acquired before the sweep. When more than one simard daemon runs on the same host/state-root — which is happening right now on ia2 (two live daemons: PID 1332853 and PID 1664591) — both sweep the same ledger concurrently and there is no mutual exclusion. A naive daemon's NoWorktree immediate-reclaim (which by design skips investigate-before-reap) then overrides another daemon's careful in-flight Pending investigation, false-reclaiming a live engineer during a transient worktree-recreate window.

Verdict of the investigation that surfaced this: still-alive false positive — the engineer was NOT dead; it was mid-relaunch.

Evidence (durable archive)

Goal advance-rysweet-agent-kgpacks-rs-to-full-parity-f29bb15c, archive /home/azureuser/.simard/reaped-engineers/rysweet_Simard_advance-rysweet-agent-kgpacks-rs-to-full-parity-f29bb15c-1784755469/ (manifest.json, evidence.txt, journal.txt).

Two distinct daemon PIDs act on the SAME claim_key in the same journal:

• PID 2802920 (careful): 7× NOT reaping … (investigation verdict=pending, claim + evidence preserved) from 16:05:12 through 20:04:40, idle escalating 3196 → 8452 → 12894 → 17564.
• PID 1664591 (naive), 20:35:42:
  WARN … claim-reaper: reclaimed … (reason=no-worktree, age=n/a, verdict=no-investigation).

The reclaim fired on a transient worktree absence. The engineer's new worktree was born 2026-07-22 20:45:05 (stat …/advance-…-f29bb15c-1784753105-3b2899, Birth ts) — ~9 min AFTER the 20:35:42 no-worktree reclaim. So NoWorktree at 20:35 was a worktree being re-created, not a dead engineer.

Live duplicate-daemon confirmation at investigation time:

$ ps -eo pid,comm | awk '$2=="simard"'
1332853 simard
1664591 simard

(the careful investigator PID 2802920 has since exited).

Root cause

reap_stale_claims (src/overseer/claim_reaper.rs:282) iterates ledger.list_engineer_claims() and, on ClaimLiveness::Dead { NoWorktree } (lines 308-311, 332-333), reclaims unconditionally (verdict = None, logged verdict=no-investigation) — no investigation, no evidence preservation. That branch is correct for a genuinely leaked slot, but the sweep holds no leader lease, so a second concurrent daemon can execute it against a claim that another daemon is deliberately keeping (Pending) while its engineer re-creates its worktree. There is a classic TOCTOU/last-writer-wins hazard across daemons: one is patiently investigating, the other reclaims.

The reaper acquires no lock (grep for lock|lease|flock|Mutex in claim_reaper.rs finds none in the sweep path).

Fix design (single-writer / leased reaper)

1. Gate the whole sweep behind a fenced, same-host leader lease. Acquire a leader token (reuse/repair self_relaunch_semaphore::LeaderSemaphore, whose own read→decide→write race is tracked in LeaderSemaphore: concurrent try_acquire race + fencing-test overclaims same-host mutual exclusion #2947 and must be fixed first — O_EXCL/flock/CAS + fsync, unique tmp path) at the START of reap_stale_claims; return a no-op ReapSummary if the lease is not held. Exactly one daemon reaps per host/state-root.
2. Fence the release chokepoint. Carry the lease fenceEpoch into ClaimLedger::release_engineer_claim so a stale-lease holder's DELETE is rejected (defence-in-depth if two daemons briefly overlap during handoff). Aligns with the fencing hardening in Live-migration KEYSTONE hardening: lease-record integrity (write-authorization + tamper-evidence + fenceEpoch durability/monotonic-restore) #2945.
3. Make the NoWorktree branch race-safe even single-writer: require the worktree to be absent across a short confirm-twice / debounce (or a minimum grace since last-known worktree mtime) before treating NoWorktree as a leaked slot, so a mid-recreate gap is not read as death. (Complements Engineer claim not released on session completion → leaked claim + 'goal disappeared before effect dispatch' (DownstreamFailed) no-worktree churn #4464, which tracks the single-daemon completion→claim-release side of the same no-worktree churn.)

Deduplication (explicitly NOT a duplicate)

This is the concurrency amplifier that none of the following cover; each is cross-referenced:

• claim-reaper HeartbeatStale investigation never converges: recipe verdict discarded (reconcile_inflight_investigations write-back gap) → Pending-only, reap path unreachable, unbounded re-archival #4467 — investigation never converges (verdict write-back gap → Pending-only). Orthogonal state-machine bug; this issue is about two daemons racing, not the verdict loop.
• Engineer claim not released on session completion → leaked claim + 'goal disappeared before effect dispatch' (DownstreamFailed) no-worktree churn #4464 — claim not released on completion → single-daemon no-worktree churn. Same symptom (reason=no-worktree), different cause (leaked-on-complete vs concurrent-daemon override during recreate).
• Stale-assignment sweep is non-idempotent: same dead engineer id re-cleared every OODA cycle (clear not durably persisted) #4462 — stale-assignment sweep not durably persisted. Different sweep.
• claim-reaper evidence-collector archives checked-in repo fixtures instead of the engineer transcript/exit-status #4449 — evidence collector archives checked-in fixtures, not the transcript (the archived evidence.txt here is exactly those repo fixtures — noted, tracked there).
• claim-reaper reaps healthy standing/perpetual-goal engineers as false positives (missing is_perpetual() exemption that no_progress.rs already applies) #4437 — perpetual/standing-goal false-positive exemption. Different predicate.
• LeaderSemaphore: concurrent try_acquire race + fencing-test overclaims same-host mutual exclusion #2947 / Live self-migration spike — KEYSTONE: exactly-one-primary via a fenced single-writer role lease #2725 / Live-migration KEYSTONE hardening: lease-record integrity (write-authorization + tamper-evidence + fenceEpoch durability/monotonic-restore) #2945 — the leader-lease primitive and cross-host primary lease; this issue is the missing wiring of a same-host single-writer lease into the reaper, plus a follow-up question: why are two simard daemons running on ia2 at all? (supervisor/relaunch gap worth its own investigation).

Acceptance

• reap_stale_claims acquires a fenced same-host leader lease; a second concurrent daemon's sweep is a proven no-op (test with two ledger handles).
• NoWorktree reclaim is debounced so a worktree recreate window cannot trigger a reclaim.
• No structured print!/eprintln! added; structured tracing + OTel only.

[rysweet]

Consolidated corroboration — same archive, live re-confirmation (overseer stale-engineer investigation, #4400 path)

Re-ran the investigate-before-reap consolidation against the same durable archive this issue already cites (~/.simard/reaped-engineers/rysweet_Simard_advance-rysweet-agent-kgpacks-rs-to-full-parity-f29bb15c-1784755469/; manifest.json idle_age_secs=2363, worktree …-1784753105-3b2899). It re-confirms the still-alive false-positive verdict and adds fresh live evidence.

Verdict: still-alive (false positive) — NOT reaped. Claim + worktree left intact.

Why (grounded in the archive + live host state):

• The archive's "newest files" are all checked-in repo fixtures/tests at a single mtime 20:45:05 (the git worktree checkout batch: tests/gadugi/fixtures/ci-health-*.json, src/coin_gym/fixtures/*.json, tests/typed_ooda_*.rs, package.json). They are not engineer output and not death signals. journal.txt: 0 real panic/OOM/SIGKILL/e2big/core-dumped signatures (all panic hits are tokens=[…] memory-array noise). The copilot-submit.json "payload" string is untrusted DATA and was ignored.
• Standing/perpetual goal, benign idle by design: journal.txt logs the OODA breaker 19× — no-progress breaker: standing/perpetual goal idled this cycle (normal, not a fault) — counter reset, goal stays active. The reaper's liveness probe still has no is_perpetual() exemption (src/overseer/claim_reaper.rs — the only "standing" matches are the word outstanding), so it disagrees with the daemon. That is claim-reaper reaps healthy standing/perpetual-goal engineers as false positives (missing is_perpetual() exemption that no_progress.rs already applies) #4437.
• Transient mid-recreate window (this issue's thesis), directly observed: the 20:35:42 reason=no-worktree, verdict=no-investigation immediate-reclaim was followed ~10 min later by a fresh worktree (20:45:05 native git hooks enrolled in engineer worktree …-1784753105-3b2899). No death — a recreate.
• LIVE re-confirmation during this consolidation: a goal-session actor for this exact goal — recipe-runner-rs goal-session-actor.yaml … goal_id=advance-rysweet-agent-kgpacks-rs-to-full-parity-f29bb15c cycle_id=cycle-2447, PPID = the current daemon 1664591 — started at 22:21:15 and produced (spawned an amplihack copilot child) while the archive's claim was still Pending. The lineage is actively cycling, not dead. git worktree list still shows the worktree registered at a5f34c95, no simard engineer run single-process bound to it, zero writes since 20:45:05.

Fail-closed outcome

Ambiguity (benign perpetual idle + a live concurrent actor for the same goal) ⇒ still-alive. No reclaim, no worktree removal. Reaping now would be exactly the false-reclaim-mid-recreate race this issue describes.

Dedup / cross-refs (no new issue filed)

This is the same root-cause cluster — filing suppressed per the deduplicated-issue rule:

• claim-reaper is not single-writer: concurrent same-host simard daemons race the claim ledger → naive no-worktree reclaim overrides another daemon's in-flight investigate-before-reap (false-reclaims a live engineer mid-recreate) #4477 (this issue) — reaper not single-writer; naive NoWorktree immediate-reclaim overrides in-flight Pending investigation.
• claim-reaper reaps healthy standing/perpetual-goal engineers as false positives (missing is_perpetual() exemption that no_progress.rs already applies) #4437 — reaper missing the is_perpetual() benign-idle exemption no_progress.rs already applies.
• Engineer claim not released on session completion → leaked claim + 'goal disappeared before effect dispatch' (DownstreamFailed) no-worktree churn #4464 — engineer claim not released on completion → leaked-claim no-worktree churn.

No additional fix dispatched: #4437 and #4464 already carry workflow:default fix workstreams and #4477 owns the single-writer/lease fix. Self-improvement signal is captured here rather than duplicated.

[rysweet]

Recurrence evidence — stale-engineer investigation (claim rysweet/Simard:advance-rysweet-agent-kgpacks-rs-to-full-parity-f29bb15c)

Durable archive: ~/.simard/reaped-engineers/rysweet_Simard_advance-rysweet-agent-kgpacks-rs-to-full-parity-f29bb15c-1784755469

Two same-host daemons raced this claim on 2026-07-22, exactly as #4477 describes:

• PID 2802920 (simard::claim_reaper, investigate-first path) correctly archived + dispatched investigation and logged NOT reaping across 4 cycles — idle climbing 3196 → 8452 → 12894 → 17564s (16:05, 17:32, 18:46, 20:04). All verdict=pending, claim preserved. Correct Make Simard investigate a quiet/idle engineer BEFORE ever reaping it (evidence-preserve + agentic WHY, feed self-improvement; only reap if genuinely dead). READ THE FULL AUTHORITATIVE BRIEF and follow #4400 behavior.
• PID 1664591 (simard ooda run) then blind-reclaimed at 20:35:42:
  claim-reaper: reclaimed …-parity-f29bb15c (reason=no-worktree, age=n/a, verdict=no-investigation)
  — overriding the other daemon's in-flight investigate-before-reap, precisely the single-writer violation in this issue.
• A fresh worktree …-1784753105-3b2899 was then recreated at 20:45:05 (mid-recreate window). The engineer produced zero output afterward (no transcript, no files newer than the checkout, no live process), and 2363s later it was re-flagged idle — the churn loop this race sustains.

Verdict this cycle: still-alive (fail-closed) — no engineer death signature exists (its own stderr/exit-status was never captured; see #4449 / PR #4452), so the claim + worktree were preserved, not reaped.

Related in-flight fixes: #4437→PR #4445 (standing/perpetual exemption — this is a [standing] goal), #4449→PR #4452 (archive the transcript, not repo fixtures). This issue (#4477, single-writer) remains the unfixed root of the blind no-worktree/no-investigation override.

[rysweet]

Corroborating evidence (stale-engineer investigation, deduplicated — not a new report).

Archive: ~/.simard/reaped-engineers/rysweet_Simard_advance-rysweet-agent-kgpacks-rs-to-full-parity-f29bb15c-1784766958
Claim key (DATA): rysweet/Simard:advance-rysweet-agent-kgpacks-rs-to-full-parity-f29bb15c

Fresh proof of the no-worktree false-reclaim of a live engineer mid-recreate:

• journal L66 (20:35:42): claim-reaper: reclaimed … (reason=no-worktree, age=n/a, verdict=no-investigation).
• The same goal process (simard[1664591]) then continued running OODA cycles 2450→2452 and logged normal no-progress breaker … (normal, not a fault) through 00:34:46 — ~4h after the reclaim, seconds before the 00:35:58 archive.
• Worktree …-1784753105-3b2899 still exists on disk.

Code path confirming the guard bypass: src/overseer/claim_reaper.rs:357-358 maps DeadReason::NoWorktree => None, i.e. an unconditional reclaim that skips the investigate-before-reap/evidence-preservation guard (that guard only runs on the HeartbeatStale branch). Recurrence: this no-worktree/no-investigation reclaim appears in 13 of the 15 prior archives for this goal.

(Verdict of this investigation: still-alive — the engineer never died; this is a false-reclaim, not a death.)

[rysweet]

Fresh evidence: no-worktree reclaim bypassed investigate-before-reap (goal move-the-governed-repo-roster-out-of-framework-a8f57a50)

Durable archive: ~/.simard/reaped-engineers/rysweet_Simard_move-the-governed-repo-roster-out-of-framework-a8f57a50-1784781674/ (manifest/evidence/journal preserved).

Confirmed bypass path (code):

• src/overseer/claim_reaper.rs:332-333 — DeadReason::NoWorktree => None: the NoWorktree arm sets verdict = None and never calls investigator.investigate(). The Make Simard investigate a quiet/idle engineer BEFORE ever reaping it (evidence-preserve + agentic WHY, feed self-improvement; only reap if genuinely dead). READ THE FULL AUTHORITATIVE BRIEF and follow #4400 preserve-and-investigate gate (lines 334-357) is structurally reachable only on the HeartbeatStale arm.
• src/overseer/claim_reaper.rs:532-538 — WorktreeClaimLivenessProbe::assess returns Dead{NoWorktree} the instant no worktree dir maps to the goal id.

The race (journal line-cites):

1. line 48 (03:10:29) — evidence archived, investigation dispatched, verdict=pending, claim + evidence preserved (gate working as designed).
2. line 54 (03:36:39) — OODA no-progress breaker brain: reclaim_and_redispatch removes the worktree while the investigation is still Pending.
3. line 55 (03:37:28) — very next sweep re-classifies the now-worktree-less claim as reason=no-worktree, verdict=no-investigation and mechanically reclaims it — no verdict ever landed. (Same signature earlier at line 37, 02:09:39.)

This is a two-writer race between the OODA redispatch worktree-removal and the pending investigation: the #4400 invariant ("NO REAP WITHOUT A COMPLETED INVESTIGATION") is enforced on HeartbeatStale but not on NoWorktree. When redispatch tears the worktree down first, the claim falls through the legacy NoWorktree fast-path. Systemic: prior investigations found this signature in 105/118 archives.

Suggested fix direction (dedup — belongs here, not a new issue): before the NoWorktree unconditional reclaim, check for an existing archived pending-investigation for the claim and, if present, skip/defer the reclaim (fail-closed) rather than reclaiming with verdict=no-investigation. Closely related: #4467 (verdict-discarded → Pending-only, unbounded re-archival).

[rysweet]

Corroborating evidence instance (stale-engineer investigation, fail-closed still-alive)

An investigate-before-reap sweep (#4400) archived this claim's evidence, but the reaper later false-reclaimed it via the unconditional NoWorktree fast-path — the defect this issue tracks.

• Claim key: rysweet/Simard:advance-rysweet-agent-kgpacks-rs-to-full-parity-f29bb15c
• Durable archive: ~/.simard/reaped-engineers/rysweet_Simard_advance-rysweet-agent-kgpacks-rs-to-full-parity-f29bb15c-1784970922/
• manifest.json: idle_age_secs=1839; exit_status omitted (no captured code/signal).

Reaper path defect (src/overseer/claim_reaper.rs). The Dead{NoWorktree} arm (L308–311) maps to verdict = None (L332–333) and reclaims unconditionally, skipping investigator.investigate() / should_reap(). Journal shows this firing twice while a live worker was concurrently present:

L40   04:14:23 WARN claim-reaper: reclaimed … reason=no-worktree, age=n/a, verdict=no-investigation   (pid 2468686)
L136  08:33:09 WARN claim-reaper: reclaimed … reason=no-worktree, age=n/a, verdict=no-investigation   (pid 964175)

Three daemon PIDs race the ledger for this one claim (2187954 → 2468686 → 964175), matching the "concurrent same-host daemons" race in this issue's title. The NoWorktree arm reclaimed after another daemon had already archived evidence and dispatched an investigation (verdict=pending, NOT reaping at L4–5/L53–54/L61/L67/L73–74/L86/L99–100), i.e. it overrode an in-flight investigate-before-reap.

Why this is a false-reclaim (not a death). OODA reported a live worker with an in-flight WIP ref both before and after the L136 reclaim:

L46/L51/L84/L91/L97   per-goal: continue (worker present, 1 live WIP ref … work in flight and healthy)
L142  08:46:33  per-goal: spawn  (… has a live worker and 1 WIP ref)
L148  09:07:52  per-goal: continue (worker present with 1 live WIP ref …)

(Caveat per #4631: worker_present is worktree-map membership, not a verified live process — so it is not a positive liveness proof either. Net signal is ambiguous, which fails closed to still-alive.)

No crash signature exists. All 69 panic/merged/E2BIG tokens in the journal occur only inside simard::memory_consolidation recalled-procedure token arrays (0 outside); genuine oom/E2BIG/exit-code evidence = 0. evidence.txt is only tails of static repo fixtures (ci-health JSON, atelier brief, coin_gym snapshots, package.json, copilot-submit.json) — no live transcript or producing subprocess — which is the #4449 collector gap.

Related churn observed here: #4467 (Pending never converges → unbounded re-archival), #4449 (fixtures archived instead of transcript/exit-status), #4437/#4631 (healthy standing-goal false-positive; worker_present not a live probe), #4462 (non-idempotent stale-assignment clear, same id …1784561556919 re-cleared ~13×).

Investigator verdict: still-alive (fail-closed; ambiguous, no producing-process probe, no exit status). Not reaped; claim + worktree preserved. Suggested fix direction: gate the NoWorktree reclaim on "no durable archive exists for this claim_key and no live WIP/worker signal", otherwise consult the existing archive / investigate before reclaiming.

[rysweet]

Recurrence data point (stale-engineer investigation, specialist prior-signature recall)

Same signature reproduced on ia2 for claim rysweet/Simard:advance-rysweet-agent-kgpacks-rs-to-full-parity-f29bb15c (goal ...-f29bb15c). Durable archive: ~/.simard/reaped-engineers/rysweet_Simard_advance-rysweet-agent-kgpacks-rs-to-full-parity-f29bb15c-1784970922/journal.txt.

Evidence matching the "no-worktree reclaim overrides in-flight investigate-before-reap" defect:

• 3 distinct daemon incarnations in one journal: simard[2187954], simard[2468686], simard[964175] — multi-daemon/restart, no lease.
• Two verdict=no-investigation reclaims while investigations were in flight: line 40 (04:14:23) and line 136 (08:33:09), both reason=no-worktree, age=n/a — the DeadReason::NoWorktree => None unconditional-reap branch (claim_reaper.rs:333) firing between careful verdict=pending keeps (lines 4-5, 53-54, 61, 67, 73-74, 86, 99-100).
• Mid-recreate window confirmed by non-monotonic newest-file idle age across sweeps: 22142s → 1954s → 6833s → 12412s — the reset to 1954s is the worktree being recreated after a no-worktree reclaim, exactly the transient window this issue describes.

No new issue filed — deduplicated into this one. Verdict of this investigation was still-alive (false positive), do-not-reap (fail-closed: archived evidence.txt is only static repo fixtures and manifest.json omits exit_status, so no dead-concluding evidence exists).