rysweet · rysweet · Jul 23, 2026 · Jul 23, 2026 · Jul 23, 2026 · Jul 23, 2026
diff --git a/Cargo.lock b/Cargo.lock
diff --git a/Cargo.toml b/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "simard"
-version = "0.36.0"
+version = "0.37.0"
 edition = "2024"
 default-run = "simard"
 

diff --git a/docs/howto/diagnose-typed-ooda-database-locked.md b/docs/howto/diagnose-typed-ooda-database-locked.md
@@ -0,0 +1,108 @@
+---
+title: 'How-to: diagnose a typed-OODA "database is locked" crash-loop'
+description: >
+  Confirm, diagnose, and clear the `typed outcome persistence failed: database
+  is locked` crash-loop in the typed-OODA ledger. Covers reading the
+  fail-visible tracing lines, verifying the WAL journal mode and 30s
+  busy_timeout are applied at open, and checking for the `-wal`/`-shm` sidecars,
+  so OODA cycles persist outcomes reliably.
+last_updated: 2026-07-23
+review_schedule: as-needed
+owner: simard
+doc_type: howto
+status: implemented
+related:
+  - ../reference/typed-ooda-ledger-concurrency.md
+  - ../reference/claim-reaper-api.md
+  - ../operations/cognitive-memory-durability.md
+  - ./diagnose-leaked-engineer-claims.md
+  - ./diagnose-and-recover-ooda-step-failures.md
+---
+
+# Diagnose a typed-OODA "database is locked" crash-loop
+
+> **Status: implemented (issue #4483).**
+> The concurrency hardening described here ships in
+> [`src/typed_ooda/ledger.rs`](https://github.com/rysweet/Simard/blob/main/src/typed_ooda/ledger.rs)
+> and [`src/typed_ooda/schema.rs`](https://github.com/rysweet/Simard/blob/main/src/typed_ooda/schema.rs).
+> Contract:
+> [Typed-OODA ledger concurrency hardening](../reference/typed-ooda-ledger-concurrency.md).
+
+## Symptom
+
+The daemon log repeats a persistence failure and OODA cycles stop making
+progress across many goals:
+
+```text
+typed outcome persistence failed: database is locked
+```
+
+On a hardened daemon this message should not recur. A single transient
+occurrence immediately followed by a successful retry is expected and benign;
+a **crash-loop** (the same message every cycle, no forward progress) means one
+of the concurrency settings below is not in effect — for example a ledger that
+was created before the fix and re-opened in rollback journal mode.
+
+## 1. Confirm it is the typed-OODA ledger
+
+The message originates in the ledger persistence path
+([`persistence`](https://github.com/rysweet/Simard/blob/main/src/typed_ooda/ledger.rs)
+error mapper). Confirm the failing writes are terminal-outcome / progress /
+effect-job persistence, and note whether the error clears on retry (benign) or
+loops (needs action). All lock diagnostics are structured `tracing` / OTel
+lines — there is no `print!`/`println!` output to grep for.
+
+## 2. Verify WAL and busy_timeout are applied
+
+WAL journal mode and the 30s `busy_timeout` are applied **at every connection
+open**, unconditionally — not only during a schema migration. Inspect the live
+ledger database:
+
+```bash
+sqlite3 <ledger.db> 'PRAGMA journal_mode;'   # expect: wal
+```
+
+If this reports `delete` (rollback mode), the ledger is running the
+pre-fix configuration. Restarting the daemon on the hardened binary re-opens the
+database and switches it to WAL; verify with the command above.
+
+## 3. Check the WAL sidecar files
+
+A WAL-mode ledger has two sidecar files next to the database:
+
+```bash
+ls -l <ledger-dir>/<ledger.db>-wal <ledger-dir>/<ledger.db>-shm
+```
+
+Both should be present and owned/permissioned like the ledger directory (not
+world-writable, not in a temp path). When taking a **cold** backup of the
+ledger, copy the `-wal` and `-shm` files alongside the main database, or
+checkpoint first — the same discipline used for the cognitive store
+([Cognitive Memory Durability](../operations/cognitive-memory-durability.md)).
+
+## 4. Confirm write transactions serialize, not fail
+
+Writers use `TransactionBehavior::Immediate`, so concurrent writers acquire the
+write lock at `BEGIN` and wait out the `busy_timeout` instead of racing and
+failing late. If you still see sustained lock errors after confirming WAL +
+busy_timeout, look for a writer holding a transaction open across slow work
+(network / agent I/O) — transaction bodies are meant to contain only
+bound-parameter SQL. A write that cannot acquire the lock within the
+`busy_timeout` surfaces the error to the log and metrics rather than looping
+forever; that surfaced error is the signal to investigate the slow holder.
+
+## 5. Rule out false reaps / leaked claims
+
+Persistent lock contention can coincide with engineer-claim lifecycle issues.
+The reaper lease-ownership guard that prevents false stale-engineer reaps is
+tracked separately (#4467/#4464/#4462/#4500) and is **not** part of this
+ledger-open hardening. To inspect leaked or reaped claims, follow
+[Diagnose and clear leaked engineer claims](./diagnose-leaked-engineer-claims.md).
+
+## Resolution checklist
+
+- [ ] Ledger reports `journal_mode = wal`.
+- [ ] `-wal` / `-shm` sidecars present with correct permissions.
+- [ ] `database is locked` no longer recurs every cycle (transient + retry OK).
+- [ ] OODA cycles persist terminal outcomes and progress again.
+- [ ] No false stale-engineer reaps or leaked `engineer_claims` rows.
diff --git a/docs/operations/index.md b/docs/operations/index.md
@@ -12,11 +12,20 @@ Simard deployment.
 | [Meeting REPL & Handoff Ingestion](meeting-handoffs.md) | Routing operator intent into the OODA loop |
 | [Progress-Evidence Kill Switch](progress-evidence-kill-switch.md) | `SIMARD_PROGRESS_EVIDENCE=off` and when to use it |
 
+Related reference pages:
+
+| Page | Topic |
+|---|---|
+| [Typed-OODA ledger concurrency hardening](../reference/typed-ooda-ledger-concurrency.md) | WAL + 30s busy_timeout applied at every ledger open, Immediate write txns, fail-visible lock propagation (#4483) |
+| [Deploy-gate canary unit-test stage](../reference/deploy-gate-unit-test-canary.md) | The self-deploy canary unit-test gate and the exit-101 red-canary root-cause fix (#4470/#4471/#4481/#4475) |
+| [Gym self-eval status wiring](../reference/gym-self-eval-status.md) | Real scenario count + non-idle self-eval in `simard status` |
+
 Related how-to guides:
 
 | Guide | Topic |
 |---|---|
 | [Diagnose handoff accumulation](../howto/diagnose-handoff-accumulation.md) | Detect, resolve, prevent handoff file buildup (#2268) |
+| [Diagnose a typed-OODA "database is locked" crash-loop](../howto/diagnose-typed-ooda-database-locked.md) | Confirm WAL + busy_timeout, clear the persistence crash-loop (#4483) |
 
 For contributor workflow (branching, merge policy, PR evidence
 requirements), see [`CONTRIBUTING.md`](https://github.com/rysweet/Simard/blob/main/CONTRIBUTING.md) at the

diff --git a/docs/reference/deploy-gate-unit-test-canary.md b/docs/reference/deploy-gate-unit-test-canary.md
@@ -0,0 +1,136 @@
+---
+title: "Reference: Deploy-gate canary unit-test stage"
+description: >
+  The contract for the self-deploy canary unit-test gate
+  (run_unit_test_gate in src/self_relaunch/gates.rs): how the gate invokes the
+  canary test suite, how a red canary (exit 101) blocks self-deploy, and the
+  root-cause fix that cleared the recurring exit-101 red canary so the running
+  daemon can self-deploy to merged main.
+last_updated: 2026-07-23
+review_schedule: as-needed
+owner: simard
+doc_type: reference
+status: implemented
+related:
+  - ./self-deploy-api.md
+  - ./overseer-deploy-canary-diagnostics.md
+  - ./typed-ooda-ledger-concurrency.md
+  - ../howto/enable-autonomous-self-merge-canary.md
+  - ../howto/verify-and-roll-back-a-self-deploy.md
+  - ../safe-self-update.md
+  - ../../src/self_relaunch/gates.rs
+  - ../../src/self_relaunch/types.rs
+---
+
+# Reference: Deploy-gate canary unit-test stage
+
+> **Status: implemented (issues #4470, #4471, #4481, #4475).** Present-tense
+> description of shipped behaviour. Primary source:
+> [`src/self_relaunch/gates.rs`](https://github.com/rysweet/Simard/blob/main/src/self_relaunch/gates.rs)
+> (`run_unit_test_gate`, `verify_canary`).
+>
+> This change root-causes and clears the recurring **red canary** — every
+> self-deploy was failing the `deploy_gate` unit-test stage with
+> `exit status: 101`, which blocked the running daemon from advancing to merged
+> `main` (`simard status`: *"running binary is 1 commit(s) behind merged main —
+> self-deploy required"*). The failing test lived in the typed-OODA concurrency
+> surface, so the root fix is delivered by the
+> [ledger concurrency hardening](./typed-ooda-ledger-concurrency.md); this page
+> documents the gate contract and the canary-green resolution.
+
+---
+
+## The canary gate sequence
+
+Before a freshly built candidate binary replaces the running daemon, the
+self-deploy path runs it through a sequence of gates via
+[`verify_canary`](https://github.com/rysweet/Simard/blob/main/src/self_relaunch/gates.rs).
+The sequence **does not short-circuit** — every gate runs so diagnostics report
+all failures, not just the first:
+
+| Gate (`RelaunchGate`) | What it proves |
+|---|---|
+| `Smoke` | The candidate binary starts and answers `--version`. |
+| `UnitTest` | The canary test suite passes (`cargo test`). |
+| `GymBaseline` | `gym list` succeeds against the candidate. |
+| `RpcHealth` | The candidate answers an RPC health probe within `health_timeout`. |
+
+A single failed gate produces a **red canary** and the self-deploy is aborted;
+the running binary stays in place. This is the intended fail-closed posture:
+**a red canary must never be papered over by disabling the gate.**
+
+---
+
+## The unit-test gate contract
+
+`run_unit_test_gate(config: &RelaunchConfig)` shells out to `cargo test` with
+**fixed arguments** (no `sh -c`, no dynamic interpolation of caller input):
+
+```text
+cargo test \
+  --manifest-path <RelaunchConfig.manifest_dir>/Cargo.toml \
+  --target-dir    <RelaunchConfig.canary_target_dir>
+```
+
+with `CARGO_BUILD_JOBS` set from
+[`cargo_jobs`](https://github.com/rysweet/Simard/blob/main/src/cargo_jobs.rs).
+The relevant [`RelaunchConfig`](https://github.com/rysweet/Simard/blob/main/src/self_relaunch/types.rs)
+fields are:
+
+| Field | Meaning |
+|---|---|
+| `manifest_dir` | Directory holding the candidate's `Cargo.toml` (default `.`). |
+| `canary_target_dir` | Isolated, PID-scoped `--target-dir` under the temp dir, so the canary build never clobbers the live target. |
+| `health_timeout` | Deadline for the RPC-health gate. |
+
+Result mapping:
+
+| `cargo test` outcome | `GateResult` |
+|---|---|
+| exit `0` | `passed: true`, detail `"all tests passed"`. |
+| non-zero (e.g. `exit 101` = test failures) | `passed: false`, detail `"tests failed (exit <status>): <truncated stderr>"`. |
+| failed to spawn | `passed: false`, detail `"cargo test failed to run: <err>"`. |
+
+Captured `stderr` is truncated (200 chars) before it is logged, and only the
+gate verdict and truncated detail are emitted through structured `tracing` /
+OTel — never full test output, tokens, or approval payloads.
+
+---
+
+## Root cause of the recurring exit-101 canary
+
+Exit status `101` is `cargo test`'s exit code for **test failures** (not a gate
+or harness bug). Reproducing the canary suite locally with the same fixed
+arguments surfaced the failing test in the typed-OODA persistence surface: the
+same `database is locked` contention described in
+[typed-OODA ledger concurrency hardening](./typed-ooda-ledger-concurrency.md)
+made the affected tests fail non-deterministically under the canary's parallel
+test execution.
+
+The resolution root-causes the defect rather than quarantining the symptom:
+
+- The underlying ledger concurrency defect is fixed at connection open (WAL +
+  30s busy_timeout), so the previously-failing tests now pass deterministically.
+- The gate itself is unchanged in posture — it is **not** disabled, weakened, or
+  made non-blocking.
+- Per issue #4471, deliberate quarantine remains available **only** for a test
+  proven obsolete/wrong, applied narrowly with a justification comment citing
+  #4471. It was not needed here.
+
+The fix lands on a fresh, non-conflicting branch, superseding the stale
+conflicting PRs #4480 / #4454 / #4436 / #4429 and coordinating with the
+root-cause/quarantine/hardening issues #4470 / #4471 / #4481 / #4475.
+
+---
+
+## Verifying a green canary
+
+After deploy, `simard status` no longer reports the running binary as behind
+merged `main`, and the deploy log records `deploy_gate: green canary` instead of
+the previous `red canary (gate unit-test: tests failed exit status: 101)`. To
+reproduce the gate manually, see
+[Enable the autonomous self-merge canary](../howto/enable-autonomous-self-merge-canary.md)
+and
+[Verify and roll back a self-deploy](../howto/verify-and-roll-back-a-self-deploy.md).
+For deep canary telemetry see
+[Overseer deploy-canary diagnostics](./overseer-deploy-canary-diagnostics.md).