OpenClaw runtime canary upgrade to v2026.4.27

[marcusglee11]

Objective

Create and execute a safe canary upgrade procedure for the OpenClaw runtime, moving from observed current version 2026.4.14 to available update 2026.4.25, with minimal operational risk.

Background

Current checks indicate:

• Current config validates cleanly
• Update is available via openclaw update
• This should be handled as a controlled rollout, not a blind in-place upgrade

Proposed runbook

A detailed runbook has been prepared here:

• upgrade-runtime-runbook-2026-04-27.md

Summary

1. Snapshot current state and back up ~/.openclaw
2. Validate baseline with openclaw status --deep and logs
3. Run a single-host canary upgrade to 2026.4.25
4. Restart and execute smoke tests
5. Observe for 5 to 20 minutes
6. Promote if healthy, otherwise roll back immediately

Smoke test checklist

• openclaw status
• main agent response works
• primary model invocation works
• memory search works
• critical channel delivery works
• cron / heartbeat execution still works
• no hard errors in logs after restart

Rollback

• Restore ~/.openclaw from backup
• Restart OpenClaw
• If required, reinstall previous runtime version 2026.4.14

Risks

• auth/provider regressions
• model ID drift
• plugin incompatibility
• config schema drift
• channel integration regressions

Action items

• Schedule maintenance window
• Execute canary upgrade on one host
• Record smoke test results
• Decide promote vs rollback

Notes

This change should not be combined with unrelated config edits in the same window unless required.

[marcusglee11]

Adding follow-up scope for execution:

Additional deliverable

Prepare a one-shot executable upgrade script that Hermes can run to perform the canary upgrade safely.

Script requirements

• create timestamped backup of ~/.openclaw
• capture pre-upgrade status and logs
• run openclaw update for target version 2026.4.25 if pinning is supported, otherwise upgrade to latest available release in the approved window
• restart OpenClaw or gateway as needed
• run post-upgrade smoke tests
• capture results to artifacts/log files
• print clear PASS / FAIL outcome
• include rollback helper path and commands

Smoke tests to automate

• openclaw status
• basic agent response path
• primary model invocation
• memory search sanity check
• critical channel sanity check if non-interactive verification is possible
• cron / heartbeat sanity check where safe
• log scan for hard errors after restart

Rollback hooks

• restore ~/.openclaw from the latest backup
• restart service
• optionally reinstall previous version 2026.4.14 if runtime rollback is needed

Operator notes for Hermes

• do not combine with unrelated config edits
• stop on first hard failure
• preserve all logs and outputs as evidence
• treat this as single-host canary execution first

[marcusglee11]

Deliverables complete

1. Executable upgrade script
scripts/upgrade/openclaw-canary-upgrade.sh

2. Operator runbook
artifacts/upgrades/openclaw-20260427/operator-runbook.md

3. PR #54 (code change to add these files to the repo)
#54

How to run

# Dry-run preview (recommended first)
cd ~/LifeOS
bash scripts/upgrade/openclaw-canary-upgrade.sh --dry-run

# Full execution
bash scripts/upgrade/openclaw-canary-upgrade.sh

What the script does (phases)

Phase	Action
0 - Preflight	Verify openclaw in PATH, version, update availability, systemd status
1 - Backup	~/.openclaw backup, systemd units, pre-upgrade status + logs
2 - Upgrade	openclaw update --yes --tag 2026.4.25, version verification
3 - Restart	Gateway health-wait (30s), bus watcher restart
4 - Smoke tests	status, health, doctor, cron, memory, log scan, channel, agent
5 - Summary	PASS/FAIL table with rollback commands, SUMMARY.md

Artifacts stored at

artifacts/upgrades/openclaw-20260427/<TIMESTAMP>/
  pre-upgrade-status.txt
  pre-upgrade-logs.txt
  upgrade-output.txt
  smoke-test-results.txt
  backup-manifest.txt
  SUMMARY.md

Backups go to ~/openclaw-upgrade-backups//dot-openclaw/.

Risks and assumptions

See operator-runbook.md for full details. Key assumptions:

• Single-host canary design only
• --tag 2026.4.25 pinning is supported (confirmed via --dry-run)
• pnpm is the package manager (confirmed)
• Gateway auto-restarts as part of openclaw update
• No critical dispatch in flight during execution

Script is not executed yet -- waiting for explicit operator approval.

[marcusglee11]

PR #54 (#54) contains the code changes for the script and runbook.

[marcusglee11]

Runtime investigation update — keep this open

I investigated the OpenClaw flakiness before closing this issue.

Findings:

• OpenClaw gateway is alive: openclaw health succeeds, and direct chat.history / sessions.list gateway calls succeed.
• The TUI watchdog line is client-side: it fires after 30s without stream delta updates; it is not by itself proof the gateway died.
• Local logs show the actual failure pattern: stuck TUI session, context overflow, auto-compaction, and a session write lock held for ~58s.
• The stuck session was still on openai-codex/gpt-5.4; current defaults/fallbacks are now github-copilot/gpt-5-mini.
• openclaw status and openclaw models status --check can still hang locally.
• openclaw agent --agent quick currently hits session-policy blocking and embedded fallback instability.

Action taken:

• Original PR feat: OpenClaw canary upgrade script + runbook for 2026.4.25 #54 was closed after renaming the invalid feat/ branch to the allowed fix/ prefix.
• Replacement PR opened: feat: OpenClaw canary upgrade script + runbook for 2026.4.25 #64
• feat: OpenClaw canary upgrade script + runbook for 2026.4.25 #64 includes the canary script/runbook and fixes the admin structure CI failure by allowing RECONCILIATION_RECEIPT_2026-04-27_v1.md in docs/11_admin/.
• Local verification on feat: OpenClaw canary upgrade script + runbook for 2026.4.25 #64 branch:
  • python3 -m pytest tests_doc/test_admin_structure.py tests_doc/test_full_doc_governance.py -q => 14 passed
  • python3 scripts/workflow/quality_gate.py check --scope changed --json => blocking checks passed; local mypy unavailable advisory waiver
  • python3 -m pytest -q => 3399 passed, 7 skipped, 7 warnings

Recommendation:

• Do not close OpenClaw runtime canary upgrade to v2026.4.27 #50 yet.
• Merge feat: OpenClaw canary upgrade script + runbook for 2026.4.25 #64 as tooling/runbook only if CI is green.
• Close OpenClaw runtime canary upgrade to v2026.4.27 #50 only after the canary script has actually run and its artifacts show clean enough OpenClaw runtime evidence.

[marcusglee11]

Provisional closure note — OpenClaw runtime upgrade/canary

Provisional close on CEO direction.

Current posture:

• This runtime-upgrade tracker is not the active bottleneck after the memory-track reprioritisation.
• OpenClaw remains less confidence-inspiring than Hermes and should not be treated as proven autonomous execution infrastructure on the basis of this closure.
• Treat OpenClaw as degraded / lower-confidence until a separate health probe proves otherwise.

Operational interpretation:

• Closing this issue does not certify OpenClaw as production-reliable.
• Closing this issue does not promote OpenClaw to active COO substrate.
• Future OpenClaw use should be gated by explicit smoke evidence: openclaw status, deep status/log scan, memory search, primary model invocation, critical channel delivery, and heartbeat/cron evidence.

Reopen or supersede with a fresh issue if:

• OpenClaw is needed for active autonomous work,
• silent failure recurs,
• runtime upgrade becomes necessary again,
• or a formal active/standby/demoted posture decision is required.

[marcusglee11]

Tooling PR merged; canary execution still open

Replacement PR #64 is merged and CI was green before merge.

Merged scope:

• OpenClaw 2026.4.25 canary upgrade script
• Operator runbook
• Branch-prefix CI repair
• Admin receipt allowlist repair
• Deterministic changed-file ordering repair for Python 3.11 CI

Verification before merge:

• Build Entry Gate: pass
• Quality Gate: pass
• Documentation Validation: pass
• Validate Artefact Index: pass
• Python 3.11 suite: pass
• Python 3.12 suite: pass
• Tool Invoke Linux/Windows: pass

Issue remains open intentionally. Remaining closure condition: execute the canary script and capture clean enough runtime artifacts/smoke evidence, including handling or explicitly accepting the current TUI watchdog/session flakiness.

[marcusglee11]

Corrective status — reopened pending canary execution

Reopening #50.

Reason:
PR #64 successfully landed the OpenClaw canary upgrade script and operator runbook, with CI green. That completes tooling preparation only.

This issue should remain open until the canary script is actually executed and runtime evidence is captured.

Remaining closure condition:

• run scripts/upgrade/openclaw-canary-upgrade.sh
• preserve generated artifacts under artifacts/upgrades/openclaw-20260427/<TIMESTAMP>/
• confirm smoke evidence is clean enough, or explicitly record accepted residual flakiness
• decide promote / rollback / defer

Current posture:
OpenClaw remains lower-confidence than Hermes. Do not treat #64 merge as proof of OpenClaw runtime reliability.

[marcusglee11]

Canary execution attempted on 2026-04-28 UTC after PR #64 landed.

Result: FAIL — keep this issue open.

Evidence:

• Script: ~/LifeOS/scripts/upgrade/openclaw-canary-upgrade.sh
• Full run artifact dir: ~/LifeOS/artifacts/upgrades/openclaw-20260427/20260428T055032Z
• Backup dir: ~/openclaw-upgrade-backups/20260428T055032Z
• Dry-run artifact dir: ~/LifeOS/artifacts/upgrades/openclaw-20260427/20260428T055009Z
• OpenClaw version before/after: OpenClaw 2026.4.25 (aa36ee6)
• openclaw update --yes --tag 2026.4.25 reported package update OK but returned failure because gateway did not become healthy after restart.

Key failure from upgrade-output.txt:

Update Result: OK
Before: 2026.4.25
After: 2026.4.25
Completion cache update failed: Error: spawnSync /home/linuxbrew/.linuxbrew/Cellar/node/25.2.1/bin/node ETIMEDOUT
Gateway did not become healthy after restart.
Gateway version mismatch: expected 2026.4.25, running gateway reported unavailable.
Service runtime: status=running, state=active, pid=361047, lastExit=0
Gateway port 18789 status: free.

Post-run probes:

• openclaw health later recovered and passed.
• direct gateway chat.history with 60s timeout passed.
• direct gateway sessions.list with 60s timeout passed.
• openclaw models status passed and shows default/fallback on github-copilot/gpt-5-mini.
• openclaw status still timed out under 30s.
• openclaw agent --agent quick --message 'smoke: reply with exactly OK' still timed out and printed: Gateway agent failed; falling back to embedded: GatewayClientRequestError: send blocked by session policy.

Relevant journal pattern after restart:

• gateway reaches [gateway] ready after ~14–17s,
• but CLI health/status paths can still hit websocket handshake timeouts,
• model-pricing fetches time out,
• bonjour watchdog/re-advertise loops continue,
• quick-agent invocation is blocked by session policy,
• session lock/facade activation errors appeared around embedded-agent paths.

Interpretation: the script is doing its job as a canary: package/runtime refresh is not enough. The gateway process can run and answer direct calls after warmup, but the CLI/TUI/session-policy paths remain flaky. Closure should wait until the canary script exits PASS or the runbook/script is amended to explicitly handle/waive these known runtime failure modes with stronger post-restart readiness logic.

[marcusglee11]

Runtime-readiness follow-up, 2026-04-28 UTC.

Result: still not closable. Keep issue open.

Findings:

• openclaw health and openclaw status still time out while direct gateway calls can pass.
• Current runtime is contaminated by openclaw-bus-watcher.service, which is repeatedly spawning local openclaw-agent processes from agent-comms-smoke; those agents hold quick-agent session locks and burn CPU.
• Evidence observed during investigation:
  • openclaw-bus-watcher.service restart counter reached 60+ and kept invoking openclaw agent --local --agent quick.
  • Gateway logs showed SessionWriteLockTimeoutError on quick-agent session files.
  • Direct chat.history and sessions.list passed after warm-up.
  • Fresh quick-agent smoke still timed out while gateway-side agent locks remained active.
• Separate OpenClaw runtime bug found and locally hot-patched for diagnosis only: gateway agent requests checked session.sendPolicy before checking whether deliver=true. That caused non-delivery CLI agent smokes to fail with send blocked by session policy when the session policy default is deny. This needs an upstream OpenClaw fix; the local bundle patch will not survive reinstall/update.

LifeOS canary/runbook patch prepared locally in worktree:

• /mnt/c/Users/cabra/Projects/lifeos-worktrees/issue50-runtime-readiness
• Branch: fix/issue50-runtime-readiness

Patch scope:

• scripts/upgrade/openclaw-canary-upgrade.sh
  • fails full canary preflight if openclaw-bus-watcher.service is active;
  • fails full canary preflight if active openclaw-agent processes exist;
  • keeps the watcher inactive during the canary window;
  • bounds CLI smoke commands with timeouts;
  • adds direct chat.history / sessions.list smoke gates;
  • uses a fresh quick-agent smoke session;
  • adds a readiness log-pattern gate for session-lock / policy / websocket / watchdog patterns.
• artifacts/upgrades/openclaw-20260427/operator-runbook.md
  • documents the watcher pause and no-active-agent prerequisite;
  • updates smoke expectations to match the actual closure criteria.

Verification on the LifeOS patch:

• bash -n scripts/upgrade/openclaw-canary-upgrade.sh: PASS
• bash scripts/upgrade/openclaw-canary-upgrade.sh --dry-run: PASS; warns that full canary would fail while watcher/agents are active
• bash scripts/upgrade/openclaw-canary-upgrade.sh: expected preflight FAIL with exit 64 because openclaw-bus-watcher.service is active
• python3 scripts/workflow/quality_gate.py check --scope changed --json: PASS; shellcheck unavailable locally, advisory only
• pytest runtime/tests -q: PASS — 3229 passed, 6 skipped, 7 warnings in 392.88s

Closure blockers remaining:

1. OpenClaw upstream/runtime fix needed for gateway agent send-policy gating: non-delivery agent calls must not be blocked by outbound delivery policy.
2. The issue-bus watcher/local-agent process cleanup problem must be resolved or the watcher must be paused for canary windows; otherwise it causes session locks and false readiness failures.
3. Full canary has not passed. Do not close OpenClaw runtime canary upgrade to v2026.4.27 #50.

[marcusglee11]

Clean-window canary rerun, 2026-04-28 UTC.

Result: still FAIL. Keep #50 open.

Maintenance window state before run:

• openclaw-bus-watcher.service: inactive/failed, not active
• openclaw-agent: no running local agent processes
• openclaw-gateway.service: active
• OpenClaw: 2026.4.25 (aa36ee6)

Artifact directory:

• /home/cabra/LifeOS/artifacts/upgrades/openclaw-20260427/20260428T072250Z

What improved:

• Preflight passed with bus watcher inactive.
• openclaw update --yes --tag 2026.4.25 completed package refresh successfully:
  • before: 2026.4.25
  • after: 2026.4.25
  • global update: PASS
  • plugin update: no updates needed
• The updater's built-in post-restart readiness gate still failed early, but the patched LifeOS canary correctly treated that as non-authoritative because package refresh succeeded and explicit canary readiness gates followed.
• Explicit gateway health recovered after 12s.
• Direct gateway calls passed:
  • chat.history: PASS
  • sessions.list: PASS
• Other subsystem checks passed:
  • openclaw health: PASS
  • openclaw cron list: PASS
  • openclaw memory index: PASS
  • Telegram channel health: PASS

Remaining failures:

• openclaw status: FAIL, timeout/failure
• openclaw doctor: FAIL
• log error scan: FAIL, 6 error lines
• fresh quick-agent invocation: FAIL
• readiness log pattern scan: FAIL, 1 suspect line

Key runtime evidence:

• Gateway log during the clean smoke window still reported:
  • errorCode=INVALID_REQUEST errorMessage=send blocked by session policy
• This happened after the canary update step refreshed the OpenClaw package and overwrote the local diagnostic hotfix, so it confirms the gateway agent send-policy bug is in the installed OpenClaw runtime, not just watcher contamination.

Conclusion:

• The LifeOS canary/runbook hardening is useful and should land: it now separates package-refresh success, gateway liveness, direct gateway calls, user-facing CLI readiness, quick-agent readiness, and watcher contamination.
• OpenClaw runtime canary upgrade to v2026.4.27 #50 is not closeable. The real remaining blocker is OpenClaw runtime behavior: user-facing status/doctor/quick-agent readiness still fails while lower-level gateway health and direct calls pass.
• Needed next fix: OpenClaw runtime should not enforce outbound session.sendPolicy=deny on non-delivery gateway agent invocations. Delivery policy should apply only when the request is actually delivering/sending externally.

[marcusglee11]

Upstream OpenClaw runtime bug filed for the remaining blocker:

• Gateway agent invocation blocked by session sendPolicy even when not delivering openclaw/openclaw#73381

This captures the clean-window evidence that gateway agent invocations are rejected with send blocked by session policy even when the request is not an external delivery/send. That is the runtime/package bug blocking #50 closure after watcher contamination was removed.