Rebootstrap native evals when recovery replaces a lease

[vincentkoc]

What happened

When a native fleet run recovered onto a replacement Crabbox lease, the run
retained bootstrapped_at_utc from the expired lease. The controller therefore
skipped hydration on the fresh machine and attempted to dispatch a runner path
that did not exist.

Expected behaviour

Bootstrap reuse must be tied to the exact lease identity. A replacement lease
must be hydrated before dispatch even when the run manifest contains an older
bootstrap timestamp.

Steps to reproduce

# Resume a recovery_required run with:
# - requested_lease_slug present
# - no active stored lease
# - bootstrapped_at_utc retained from the previous lease
python -m scripts.native_eval.fleet ... 

Relevant output

dispatch openclaw-gpt55-high-exploratory-4-r1-20260729 failed with exit 1:
fleet-dispatch: line 32: kill: (...) - No such process

/opt/shellbench-native/runner/scripts/native_eval/remote_run.sh:
No such file or directory

Environment

• Python version: 3.12
• OS: macOS controller, Ubuntu AWS lease
• ClawBench version / commit: 98301b2405070630ec61a331b11937e886688f61

[clawsweeper]

Codex review: keeping this open for maintainer follow-up; there is still a little grit to resolve. Reviewed August 23, 2026, 9:49 AM ET / 13:49 UTC.

Summary
Keep open: current main still skips hydration whenever an old bootstrap timestamp exists, even after lease recovery selects a replacement machine. The member-authored PR at #58 directly owns this bounded fix and should be reviewed rather than duplicated.

Reproducibility: yes. on current main, a recovered run retaining bootstrapped_at_utc reaches dispatch without hydration after a replacement lease is acquired; the guard at scripts/native_eval/fleet.py:566 establishes the path from the supplied manifest state.

Maintainer decision needed

Question	Recommendation
Should the member-authored replacement-lease repair at #58 be approved after its legacy-manifest compatibility behavior is reviewed?	Review and land the focused repair: Verify that legacy timestamp-only manifests rehydrate safely, then merge the existing lease-identity fix.

Why: The issue author is a repository member and the active PR owns the complete narrow repair, so this workflow must not close or replace either item automatically.

Next step
Do not open another repair branch: the active member-authored PR already owns this source-proven fix and needs normal maintainer review.

Review details

Best possible solution:

Persist the lease identity when hydration succeeds and only reuse bootstrap state when it matches the active lease, with regression coverage for legacy timestamp-only manifests.

Do we have a high-confidence way to reproduce the issue?

Yes: on current main, a recovered run retaining bootstrapped_at_utc reaches dispatch without hydration after a replacement lease is acquired; the guard at scripts/native_eval/fleet.py:566 establishes the path from the supplied manifest state.

Is this the best way to solve the issue?

Yes: the linked PR's lease-identity guard is the narrowest maintainable fix because hydration is machine-specific and timestamp-only state cannot establish that the active lease contains the runner.

AGENTS.md: not found in the target repository.

Remaining risk / open question:

• The linked repair changes persisted bootstrap state; review should confirm legacy manifests without a lease identity deliberately rehydrate once on recovery.

Codex review notes: model internal, reasoning high; reviewed against 884dd1bb5511.

Label changes

Label changes:

• add clawsweeper:needs-maintainer-review: Current issue advisory state selects this label.

Label justifications:

• P2: Replacement-lease recovery can fail native benchmark dispatches, but the defect is bounded to a recoverable evaluation-controller path.
• impact:other: The concrete impact is failed remote evaluation dispatch, outside the defined data, message, session, auth, or crash-loop labels.

Evidence reviewed

What I checked:

• Current-main defect: After obtaining a lease, dispatch is preceded by hydration only when bootstrapped_at_utc is absent; the condition does not compare bootstrap state with the active lease identity. (scripts/native_eval/fleet.py:561, 884dd1bb5511)
• Recovery path: Recovery entries without a stored lease are returned to planned state, where a replacement lease can be acquired before the timestamp-only bootstrap check runs. (scripts/native_eval/fleet.py:1403, 884dd1bb5511)
• Feature provenance: Native fleet implementation history identifies Vincent Koc as the author of the original matrix runner and its subsequent native-harness stabilization. (scripts/native_eval/fleet.py:541, b9acd9f7a010)
• Open fix owner: Provided GitHub context identifies an open, mergeable member-authored PR that explicitly fixes this issue, records bootstrapped_lease_id, and adds the replacement-lease regression case. (8949dc8328e9)

Likely related people:

• vincentkoc: Current-file history attributes the native matrix runner and native-harness stabilization to Vincent Koc; they also authored the linked repair. (role: native fleet feature author and recent area contributor; confidence: high; commits: 69f75c6629c4, b9acd9f7a010, 569b5c39c783; files: scripts/native_eval/fleet.py, tests/test_native_eval_fleet.py)

How this review workflow works

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.