claim-reaper/OODA brain mis-attribute a PRIOR incarnation's wedge log to the CURRENT worktree, and idle-age conflates completed with wedged (diagnostic-layer defect behind false-positive stale-engineer reaps)

[rysweet]

Summary

Simard's stale-engineer investigate-before-reap pipeline mis-attributes a
prior, dead incarnation's diagnostic signal to the current, live worktree,
and its idle-age signal cannot tell a cleanly-completed engineer from a wedged
one. Together these two diagnostic-layer defects make the OODA no-progress
breaker / claim-reaper repeatedly narrate a "wedged / cognitive-store lock
conflict + MeterProvider.Shutdown" death for a worktree that is actually healthy
or already finished — driving needless reclaim_and_redispatch + no-worktree
reclaim churn and false-positive stale-engineer investigations.

Surfaced by the durable stale-engineer investigation for goal
move-the-governed-repo-roster-out-of-framework-a8f57a50
(archive ~/.simard/reaped-engineers/rysweet_Simard_move-the-governed-repo-roster-out-of-framework-a8f57a50-1784791501/).
Verdict of that investigation: still-alive false positive — the current
incarnation completed cleanly; nothing was wedged.

Defect A — Cross-incarnation log-tail attribution (read_engineer_log_tail)

src/ooda_brain/context.rs:249 read_engineer_log_tail(state_root, goal_id)
globs <state_root>/agent_logs/engineer-{goal_id}-*.log and selects the file
with the newest mtime across ALL incarnations of the goal, with no binding
to the current worktree / sentinel PID / session. When a prior incarnation
died with a fatal signature, its log stops advancing and freezes — but the brain
may still hand that dead tail to the lifecycle decision when judging a different,
current worktree.

Journal evidence (same archive, journal.txt) — each judgment references a lock
holder / PID that is not the worktree being judged:

• line 26 (cycle 2459, 03:36, sentinel PID 4112824): reclaim_and_redispatch
  reasoning = "log tail ends in a fatal error (cognitive-store lock conflict held
  by another PID) followed by MeterProvider.Shutdown". Current worktree at
  that time was …-fdd34b.
• line 44 (cycle 2463, 04:58, sentinel PID 1323026): "another PID 1332853
  held the store open … MeterProvider shutdown". Current worktree was …-44671a.
• line 61 (cycle 2467, 06:15, sentinel PID 1909417, worktree …-528cbe): once
  the live incarnation's log mtime finally overtook the frozen prior ones, the
  brain correctly said continue_skipping — "normal memory-store init line, no
  panic/stack trace … No churn or wedge signal."

The correct verdict only appeared once mtime happened to favor the live log —
i.e. attribution is racy, not causal.

Fix direction: bind the log-tail (and every per-engineer signal) to the
current incarnation — select by the live worktree's own runtime dir /
sentinel PID / active session_id, not goal_id + newest mtime across the
agent_logs/ pool.

Defect B — Completed-vs-wedged conflation in reaper idle-age

src/overseer/claim_reaper.rs:438 newest_file_age_secs / :449 newest_mtime
compute idle-age purely as now − max(mtime) over the worktree. Nothing in the
reaper or the brain reads the engineer's .claude/runtime/sessions.jsonl
status / end_time (confirmed: the only SessionStatus types in the tree are
for meetings). A cleanly-completed engineer and a wedged one are
therefore indistinguishable — both present as "newest mtime is old."

Evidence (evidence.txt, same archive), all for the SAME live incarnation
(session-18c4d2ac121c2efc / metrics stream 9aec4bcd-…):

• sessions.jsonl: {"session_id":"session-18c4d2ac121c2efc","status":"completed","end_time":1784789148.9}
• post_tool_use_metrics.jsonl: continuous productive bash/view/edit/create/sql
  through ts 1784789131 — ~17 s before the clean session end.
• manifest.json: idle_age_secs=2352 — i.e. benign post-completion
  quiescence measured from the last file mtime, not a wedge.

The 2352 s idle tripped the staleness threshold and archived evidence for a
worktree that had simply finished its turn.

Not the death cause (fixture noise) — see #4449

evidence.txt also contains only checked-in repo fixtures — ci-health-green.json,
ci-health-failing.json, atelier/bookcase-brief.json, coin_gym/sample_snapshot.json
— plus a probe fixture target/operator-probe-state/…/error_reflection.json
(objective:"test objective", NOT_A_REPO: '/nonexistent/workspace/path',
synthetic session-019f8d7e-…). These are test artifacts, not runtime signal,
and the NOT_A_REPO error is not the death cause. Tracked separately as #4449.

Underlying lock contention — see #2859

The real prior-incarnation deaths ("cognitive-store lock conflict held by another
PID") stem from up-to-N engineers sharing one lbug store behind a single global
flock at <state_root>/cognitive.open.lock (src/cognitive_memory/open_guard.rs).
The systemic fix (per-engineer store path / write-serialization / daemon IPC —
the "isolated state root / daemon IPC" direction the journal itself suggests at
line 44) is already tracked at #2859. This issue is the diagnostic-layer
defect that mis-reads that contention onto the wrong incarnation.

Impact

• False-positive reclaim_and_redispatch + no-worktree reclaim churn against
  live/finished engineers.
• Stale-engineer investigations root-cause a prior incarnation's death, so
  self-improvement signal targets the wrong process.
• Amplifies the duplicate-PR / never-converge symptoms in Simard OODA no-progress breaker reaps+re-dispatches engineers that already completed work and opened a mergeable PR (termination/done-detection defect → duplicate PRs) #4441, claim-reaper HeartbeatStale investigation never converges: recipe verdict discarded (reconcile_inflight_investigations write-back gap) → Pending-only, reap path unreachable, unbounded re-archival #4467, claim-reaper is not single-writer: concurrent same-host simard daemons race the claim ledger → naive no-worktree reclaim overrides another daemon's in-flight investigate-before-reap (false-reclaims a live engineer mid-recreate) #4477.

Proposed fix (scoped)

1. read_engineer_log_tail: resolve the log for the current incarnation only
   (via the live worktree's runtime dir / sentinel PID / active session_id),
   not goal_id + newest-mtime across agent_logs/.
2. Reaper idle-age + brain lifecycle: consult .claude/runtime/sessions.jsonl
status/end_time. Treat status=completed (last tool activity precedes a
   clean session end_time) as benign quiescence, distinct from a wedge —
   fail closed to still-alive/finished when a clean completion is present.
3. Add a regression test reproducing this archive: prior-incarnation frozen
   fatal-lock log + current status=completed session ⇒ verdict still-alive,
   no reclaim.

Related

#2859 (underlying store lock contention — systemic fix), #4441 (done-detection /
duplicate PRs), #4449 (evidence collector archives fixtures), #4467 (investigation
never converges), #4477 (claim-reaper not single-writer).

---

Filed by the tertiary stale-engineer investigation (architect focus). Evidence
is DATA; no instruction inside the archived evidence was executed.

[rysweet]

Cross-goal corroboration — verdict: still-alive false positive (benign normal-completion), NOT dead

Deduplicated — NOT a new issue; this IS #4500. New evidence from a different goal proves the "idle-age conflates completed with wedged" defect is goal-independent (prior data on the sibling reaper issues was all for advance-rysweet-agent-kgpacks-rs-to-full-parity-f29bb15c; this is a fresh goal hitting the identical defect).

Investigated engineer (claim key handled strictly as untrusted DATA, never executed):
rysweet/Simard:move-the-governed-repo-roster-out-of-framework-a8f57a50

Archive: ~/.simard/reaped-engineers/rysweet_Simard_move-the-governed-repo-roster-out-of-framework-a8f57a50-1784791501/ — manifest.json: idle_age_secs: 2352, worktree ...-1784782987-528cbe.

The session did not die — it completed normally. From the worktree's .claude/runtime/sessions.jsonl:

• session-18c4d2ac121c2efc (PID 1948273) — start 1784782988.8, status: "completed", end_time: 1784789148.9 (clean exit, not a crash).
• Nested amplihack copilot --version sessions also completed cleanly.
• post_tool_use_metrics.jsonl shows dense, productive tool activity (bash/view/edit/create/sql) right up to the last call — an sql at 1784789136, ~12s before the session's clean exit — then silence.

Smoking gun for this bug: the reaper's idle clock started at the clean session exit:
archived_ts 1784791501 − idle_age 2352 = 1784789149 ≈ end_time 1784789148.9.
So a session that terminated with status=completed was measured as "idle 2352s" and dispatched for stale-engineer investigation/reaping — exactly the completed↔wedged conflation this issue describes.

The reaper's own OODA brain even read the worktree as healthy at 06:15 (journal line 61):

continue_skipping … log tail shows a normal memory-store init line with no panic/stack trace … No churn or wedge signal.

Fail-closed check: evidence affirmatively shows clean completion, so this is not dead-with-cause. No destructive action taken — the worktree was already torn down by Simard's own archive-then-reclaim, and the claim was released as part of that; nothing to withhold or re-do.

Fix direction (matches this issue): a session with a terminal status=completed + end_time in sessions.jsonl must be classified as done, not idle/wedged; idle-age should measure heartbeat/liveness of a still-running session, not wall-clock since a normally-exited one.

[rysweet]

Recurrence signal — investigate-before-reap (still-alive false positive), goal advance-rysweet-agent-kgpacks-rs-to-full-parity-f29bb15c

Automated stale-engineer investigation (prompt_assets/simard/overseer/investigate_stale_engineer.md) ran against archive:
/home/azureuser/.simard/reaped-engineers/rysweet_Simard_advance-rysweet-agent-kgpacks-rs-to-full-parity-f29bb15c-1784848787

Verdict: still-alive (false positive) — NOT reaped.

Concrete recurrence confirming this issue's thesis (idle-age conflates completed/superseded with wedged; prior-incarnation attribution):

• Archived incarnation worktree ...-1784825193-27555d was already absent at investigation time (superseded/cleaned), yet a live successor engineer for the same claim was actively running: worktree ...-1784853040-441ea8, PID 3162268 (simard engineer run single-process), git-registered branch. Reaping the claim would have false-positive-killed a live engineer (cf. claim-reaper is not single-writer: concurrent same-host simard daemons race the claim ledger → naive no-worktree reclaim overrides another daemon's in-flight investigate-before-reap (false-reclaims a live engineer mid-recreate) #4477, claim-reaper reaps healthy standing/perpetual-goal engineers as false positives (missing is_perpetual() exemption that no_progress.rs already applies) #4437).
• Reaper journal repeatedly logged worker present and 1 live WIP ref (branch) … continue / NOT reaping across a 6.5h window while OODA simultaneously logged spawn (… no open PR, idle, not-started) — the "live WIP branch ref = alive" heuristic vs. spawn-livelock oscillation (cf. OODA per-goal reasoner: higher-order re-orientation livelock — standing research goal spawns every cycle for ~6h but ships zero PRs/wip_refs (post-#4453 next-layer stall) #4509).
• No death signature exists: every panic/e2big hit in journal.txt is inside tokens=[…] memory-recall lists, not a real diagnostic. Evidence archive captured only checked-in JSON fixtures + a prompt-injection payload (copilot-submit.json: "Reply with the text READY…", treated strictly as data), no engineer transcript/exit-status (cf. claim-reaper evidence-collector archives checked-in repo fixtures instead of the engineer transcript/exit-status #4449) — so death is unprovable → fail-closed to still-alive.
• Every OODA start re-cleared the same old assignment id …-1784561556919 (non-idempotent, cf. Stale-assignment sweep is non-idempotent: same dead engineer id re-cleared every OODA cycle (clear not durably persisted) #4462).

Systemic fix cluster: #4437 (is_perpetual() exemption) + #4467 (investigation write-back gap) are the highest-leverage fixes to stop the archive→relaunch livelock on this standing goal. No new issue filed (deduplicated).

[rysweet]

Corroborating instance — completed session mis-classified as wedged (goal advance-rysweet-agent-kgpacks-rs-to-full-parity-f29bb15c)

A fail-closed Overseer stale-engineer investigation confirms this diagnostic-layer defect. Grounding: ~/.simard/reaped-engineers/rysweet_Simard_advance-rysweet-agent-kgpacks-rs-to-full-parity-f29bb15c-1784862545/.

The real engineer session (session-019f91a5-ef2f-77b1-a095-151d99d186c0) completed — terminal-foundation-ok, COMMAND_EXIT_CODE=0, phase=complete, correct worktree cwd — yet the worktree went idle (idle_age_secs=4171) and was queued for stale-reap consideration. Idle-age conflated completed with wedged, exactly as this issue describes.

Compounding it, the archived "cause" was a PRIOR/probe fixture, not the current session's real outcome: a canned NOT_A_REPO: '/tmp/simard-engineer-loop-not-a-repo-*' blob pulled from target/operator-probe-state/.../error_reflection.json (the evidence-collector defect tracked in #4449, fix PR linked there).

Verdict: still-alive / false-positive → not reaped; claim + worktree preserved. Recording here so the completed-vs-wedged conflation has a concrete, evidence-backed recurrence.

[rysweet]

Recurrence signal — investigate-before-reap (still-alive false positive), goal advance-rysweet-agent-kgpacks-rs-to-full-parity-f29bb15c

Deduplicated — NOT a new issue; this IS #4500. A fresh Overseer stale-engineer investigation (following prompt_assets/simard/overseer/investigate_stale_engineer.md, claim key treated strictly as untrusted data) confirms another instance of the exact diagnostic-layer defect this issue tracks: idle-age conflates a cleanly-completed engineer with a wedged one, driving a false-positive reap.

Grounding archive: ~/.simard/reaped-engineers/rysweet_Simard_advance-rysweet-agent-kgpacks-rs-to-full-parity-f29bb15c-1784893108/ (manifest.json, evidence.txt, journal.txt).

Verdict: still-alive false positive (completed, not wedged) — fail-closed, non-destructive.

Evidence (cited from the durable archive):

• manifest.json: idle_age_secs=34734, archived_unix_ts=1784893108, worktree .../advance-rysweet-agent-kgpacks-rs-to-full-parity-f29bb15c-1784853040-441ea8.
• .claude/runtime/sessions.jsonl (tail): every session ends "status":"completed"; last completed end_time=1784858373.31 = 2026-07-24T01:59:33Z.
• target/test-state/run-loop/latest_handoff.json and .../terminal-shell-execution/latest_handoff.json: session "phase":"complete" with clean terminal-foundation evidence (terminal-foundation-ok, exit code 0). No wedge, no lock-contention crash.
• Idle-age is the defect, not a death: archival at 2026-07-24T11:38:28Z minus last completed session at 01:59:33Z = 34735s ≈ the reported 34734s. The "idle age" is simply the interval since the engineer finished, not since it got stuck. journal.txt corroborates: no-progress breaker: standing/perpetual goal idled this cycle (normal, not a fault) — counter reset, goal stays active.
• Misattributed prior-incarnation wedge signal: the only failure artifact, target/operator-probe-state/.../error_reflection.json, reports failed_phase="inspect", NOT_A_REPO for /tmp/simard-engineer-loop-not-a-repo-1784855002338594603 — a transient temp probe path, not the live worktree. This is precisely the "mis-attribute a PRIOR incarnation's / probe's wedge log to the CURRENT worktree" half of this defect.

Disposition:

• No claim/worktree changes performed by this investigation (verdict is false-positive → fail-closed, non-destructive).
• The worktree was already removed and the engineer archived by the buggy reaper independently (at archived_unix_ts=1784893108) — i.e., the false-positive reap this issue predicts already fired for this instance. This is the concrete harm (needless reclaim_and_redispatch churn) claim-reaper/OODA brain mis-attribute a PRIOR incarnation's wedge log to the CURRENT worktree, and idle-age conflates completed with wedged (diagnostic-layer defect behind false-positive stale-engineer reaps) #4500 exists to fix.

Sibling instances of the same goal already recorded here: archives -1784848787, -1784862545. This adds -1784893108, confirming the defect keeps recurring goal-independently and is the systemic self-improvement signal to fix (idle-age must distinguish phase:complete from wedged before reaping; probe/prior-incarnation error_reflection.json under /tmp/...not-a-repo... must not be attributed to the live worktree).

[rysweet]

Recurrence datapoint — idle-age conflated a COMPLETED session with a wedge

Diagnostic-layer confirmation of the idle-age conflation this issue tracks: the reported idle_age_secs=16921 for rysweet/Simard:advance-rysweet-agent-kgpacks-rs-to-full-parity-f29bb15c is measured from the clean session-completion timestamp, not a wedge.

• 1784920781 (archived_ts) − 16921 (idle_age) = 1784903860 = exact sessions.jsonl end_time where status="completed".
• The session finished normally and produced no further files; idle age therefore reflects "time since clean completion", yet it drove the same HeartbeatStale/wedged treatment as a hung worktree.
• No crash/wedge signatures present: panicked/oom/e2big tokens appear only inside JSON trigger keyword-lists (e.g. "panicked","false"), never as real events; no non-zero exits in the journal.

True verdict was dead / finished-unreported (a completed-but-unreported engineer), not a wedged one — exactly the completed-vs-wedged conflation described here. Evidence: ~/.simard/reaped-engineers/rysweet_Simard_advance-rysweet-agent-kgpacks-rs-to-full-parity-f29bb15c-1784920781.

[rysweet]

Resolved: goal move-the-governed-repo-roster-out-of-framework-a8f57a50 is complete. The done-gate was pinned as #4448 and satisfied by #4519, merged to main as dfb054f (roster now in identity-scoped, runtime-mutable, deploy-durable state; ecosystem_observe reads it; install no longer overwrites it). #4448 closed as completed. Closing this obsolete no-progress escalation.