Update .claude with 11 changed files (#4483)

.gitignore

@@ -1,6 +1,7 @@
 /target
 /target-shared
 /.claude/runtime/
+.claude/**/runtime/
 .cargo/config.toml
 __pycache__/
 *.pyc

docs/concepts/typed-outcome-ledger-shared-connection.md

@@ -0,0 +1,144 @@
+---
+title: "Typed-Outcome Ledger Shared Connection (one connection per file, not per handler)"
+description: >
+  Why the typed-outcome ledger uses a single, process-wide serialized SQLite
+  connection per ledger file instead of one connection per CapabilityHandler.
+  Explains the concurrent-writer burst that produced systemic
+  "typed outcome persistence failed: database is locked" errors (issue #4483),
+  why a per-handler Mutex could not prevent it, how a path-keyed shared
+  connection removes the race by construction, and why serializing the single
+  ledger writer is an acceptable trade.
+last_updated: 2026-07-23
+review_schedule: as-needed
+owner: simard
+doc_type: explanation
+status: design — not yet implemented
+related:
+  - ../reference/typed-outcome-ledger-connection-registry-api.md
+  - ../howto/diagnose-typed-outcome-database-is-locked.md
+  - ../reference/ooda-capability-api.md
+---
+
+# Typed-Outcome Ledger Shared Connection
+
+> **Spec-first (retcon) document.** This explains the *target* design for issue
+> **#4483**. The fix is **not yet landed** — today each `CapabilityHandler::open`
+> still builds its own `Mutex<Connection>`. The documentation and implementation
+> land in the **same pull request**; flip `status:` to `implemented` when that PR
+> merges.
+
+## The invariant
+
+There is exactly **one** typed-outcome ledger file per state root:
+
+```
+<state-root>/typed-ooda/outcomes.sqlite3
+```
+
+It is the durable audit trail of terminal OODA outcomes and the effect outbox.
+The invariant this design protects: **an outcome the system reported as recorded
+is durably present in that file, and a write never silently fails.**
+
+## What went wrong (issue #4483)
+
+Within a single goal session, several `CapabilityHandler` instances open that
+*same file*:
+
+- the **startup outbox worker** that drains pending effects on session start
+  (`OutboxWorker::drain_pending` in
+  `src/ooda_actions/advance_goal/typed_goal_session.rs`), and
+- the **route executor** that records terminal outcomes and enqueues effects.
+
+Concurrently running goal sessions add more openers of the same file.
+
+Before the fix, each `open` created an **independent** `Connection` and wrapped it
+in a **per-handler** `Mutex`:
+
+```rust
+// pre-fix
+connection: Mutex<Connection>,          // one per handler
+// ...
+connection: Mutex::new(connection),     // independent connection per open()
+```
+
+A `Mutex<Connection>` serializes access **inside one handler**. It does nothing
+between handlers: two handlers are two independent connections to the same file.
+When both attempt a write transaction at once, SQLite's file-level locking lets
+only one writer proceed and the other gets `SQLITE_BUSY` — surfaced as
+`"database is locked"`.
+
+`open` sets a `busy_timeout`, which retries for a few seconds. But when many
+writers converge in a **burst** — several goals reaching a terminal at the same
+moment, each with its startup drain firing — the timeout is exhausted and the
+busy error escapes. It is mapped through `persistence(..)` to
+`CapabilityErrorCode::PersistenceFailed`, which **aborts the record**. The
+terminal outcome that the loop believed it had persisted is dropped from the
+audit trail. That is the "systemic typed-outcome PersistenceFailed" of #4483.
+
+Concretely, the failure signature is several goals (say `<goal-a>` … `<goal-f>`,
+six distinct goals reaching a terminal in the same window) each logging
+`typed outcome persistence failed: ... database is locked` within the same
+few seconds — a contention *burst*, not a steady leak.
+
+## Why a bigger `busy_timeout` is not the fix
+
+Raising the timeout only widens the window before the error surfaces; under a
+true burst of independent connections the collision is structural, and a longer
+timeout trades a lost outcome for a stalled goal session. The problem is *having
+multiple independent writers to one file at all*, not how long each one waits.
+
+## The fix: one connection per file, shared
+
+The design collapses "one connection per handler" into **one connection per
+ledger file, per process**, held behind a process-global, path-keyed registry:
+
+```
+OnceLock<Mutex<HashMap<PathBuf, Arc<Mutex<Connection>>>>>
+```
+
+Every `CapabilityHandler` opened against a given file **clones the same**
+`Arc<Mutex<Connection>>`. The `Mutex` that used to serialize one handler now
+serializes **every** writer to that file across the whole process. There is only
+ever one connection issuing writes, so the cross-connection `SQLITE_BUSY` race
+cannot occur — it is removed by construction, not merely retried away.
+
+Opening a new file also applies durable pragmas once — WAL journaling,
+`busy_timeout = 5000`, and `foreign_keys = ON` — and a bounded
+`with_busy_retry` wraps writes as defense-in-depth against *external* processes
+touching the file. See the
+[connection registry API reference](../reference/typed-outcome-ledger-connection-registry-api.md)
+for the exact contract.
+
+## Why serializing the writer is acceptable
+
+The typed-outcome ledger is a **low-frequency, small-write** audit trail:
+terminal outcomes and outbox effect rows, written at OODA cycle boundaries — not
+a hot data-plane. Serializing its single writer costs nothing meaningful in
+throughput, and it buys a hard correctness guarantee: no dropped outcomes under
+contention. WAL additionally keeps **readers** from blocking the writer, so
+liveness/claim reads stay responsive while a write holds the connection.
+
+## Why path-keyed, not one global connection
+
+A single global connection would force *every* ledger file in the process to
+serialize against one lock — breaking test isolation (each test uses its own
+temp-dir ledger) and any future multi-tenant separation. Keying the registry by
+**canonical path** means only handlers for the *same* file share a connection;
+distinct files remain fully independent. This preserves the existing test
+suite's parallelism and keeps tenants' ledgers isolated.
+
+## What this does *not* change
+
+- The public `CapabilityHandler::open` / `with_engineer_liveness` API.
+- The schema and its `UNIQUE(outcome_id)` / `UNIQUE(session_id, cycle_id)`
+  invariants and foreign keys.
+- The fail-visible contract: a write either commits or the caller sees
+  `PersistenceFailed`. Sharing a connection changes *how many* writers exist, not
+  *whether* failures are surfaced.
+- Authorization, replay, and effect-lease semantics from the
+  [OODA capability API](../reference/ooda-capability-api.md).
+
+## See also
+
+- Reference: [Typed-outcome ledger connection registry API](../reference/typed-outcome-ledger-connection-registry-api.md).
+- Runbook: [Diagnose "typed outcome persistence failed: database is locked"](../howto/diagnose-typed-outcome-database-is-locked.md).

docs/howto/diagnose-typed-outcome-database-is-locked.md

@@ -0,0 +1,156 @@
+---
+title: Diagnose "typed outcome persistence failed: database is locked"
+description: >
+  Operator runbook for the systemic typed-outcome PersistenceFailed burst fixed
+  under issue #4483. Recognise the concurrent-writer "database is locked"
+  signature in the typed-ooda/outcomes.sqlite3 ledger, confirm the shared
+  path-keyed connection registry, WAL/busy_timeout/foreign_keys pragmas, and
+  bounded busy-retry are in effect, run the concurrency regression test, and
+  localise a recurrence to an external writer or a bypassed open() path.
+last_updated: 2026-07-23
+review_schedule: as-needed
+owner: simard
+doc_type: how-to
+status: design — not yet implemented
+related:
+  - ../concepts/typed-outcome-ledger-shared-connection.md
+  - ../reference/typed-outcome-ledger-connection-registry-api.md
+  - diagnose-and-recover-ooda-step-failures.md
+---
+
+# Diagnose "typed outcome persistence failed: database is locked"
+
+> **Spec-first (retcon) runbook.** This documents operating the fix for issue
+> **#4483**, which is **not yet landed**. The documentation and implementation
+> ship in the **same pull request**; flip `status:` to `implemented` on merge.
+> Until then, "in effect" checks below describe the state you are verifying once
+> the fix is present.
+
+## Symptom
+
+One or more terminal outcomes fail to persist, and the logs show:
+
+```
+typed outcome persistence failed: ... database is locked
+```
+
+(from `persistence(..)` → `CapabilityErrorCode::PersistenceFailed`). The
+distinguishing signature of issue #4483 is a **burst**: several distinct goals
+reaching a terminal in the same few-second window each emit the error, rather
+than a single steady failure. The affected file is the typed-outcome ledger:
+
+```
+<state-root>/typed-ooda/outcomes.sqlite3
+```
+
+## Why it happens (one line)
+
+Multiple `CapabilityHandler` instances opened against that one file used to hold
+**independent** connections; concurrent writers collided at SQLite's file lock
+and one got `SQLITE_BUSY`. See
+[the concept doc](../concepts/typed-outcome-ledger-shared-connection.md) for the
+full explanation.
+
+## Step 1 — Recognise the burst signature
+
+Confirm it is the concurrency burst and not an unrelated I/O error:
+
+```bash
+# Adjust the log source to your deployment.
+journalctl --user -u 'simard*' --since '15 min ago' \
+  | grep -E 'typed outcome persistence failed.*database is locked'
+```
+
+Look for **several distinct goal / session identifiers** clustered within the
+same few seconds. A lone occurrence spread over minutes is more likely an
+external writer (Step 4), disk pressure, or a permissions problem.
+
+## Step 2 — Confirm the shared connection registry is in effect
+
+The fix makes every handler for one file share a single connection. Verify the
+registry and the shared field type exist:
+
+```bash
+cd <repo-root>
+grep -n 'OnceLock<Mutex<HashMap<PathBuf' src/typed_ooda/ledger.rs
+grep -n 'connection:\s*Arc<Mutex<Connection>>\|SharedConn' src/typed_ooda/ledger.rs
+grep -n 'fn apply_pragmas\|fn with_busy_retry\|fn is_sqlite_busy' src/typed_ooda/ledger.rs
+```
+
+**Pre-fix (bug present)** you will instead see:
+
+```
+connection: Mutex<Connection>,          // per-handler, independent connections
+connection: Mutex::new(connection),
+```
+
+If you see the pre-fix form, the burst is expected under load — the fix has not
+landed on this build.
+
+## Step 3 — Confirm the durability pragmas and retry
+
+```bash
+grep -n 'journal_mode.*WAL\|WAL' src/typed_ooda/ledger.rs
+grep -n 'busy_timeout' src/typed_ooda/ledger.rs      # PRAGMA busy_timeout=5000 (== 5s)
+grep -n 'foreign_keys' src/typed_ooda/ledger.rs      # foreign_keys = ON
+```
+
+Then confirm WAL is actually active on a live ledger:
+
+```bash
+sqlite3 "<state-root>/typed-ooda/outcomes.sqlite3" 'PRAGMA journal_mode;'
+# expect: wal
+```
+
+A `-wal` / `-shm` sidecar file next to `outcomes.sqlite3` is the on-disk sign
+WAL is in use.
+
+## Step 4 — Localise a recurrence
+
+If the burst still appears **after** the fix is in effect, it must come from
+outside the in-process shared connection:
+
+1. **An external process** (a stray CLI, a manual `sqlite3` write session, a
+   backup tool holding a write lock) is touching the ledger. Check:
+   ```bash
+   fuser -v "<state-root>/typed-ooda/outcomes.sqlite3" 2>&1 || \
+     lsof "<state-root>/typed-ooda/outcomes.sqlite3"
+   ```
+   Close the external writer; `with_busy_retry` should absorb brief overlaps.
+2. **A bypassed `open` path** — some code constructed a raw `Connection` to the
+   ledger instead of going through `CapabilityHandler::open`, escaping the
+   registry. Search for it:
+   ```bash
+   grep -rn 'Connection::open' src/ | grep -i 'outcomes.sqlite3\|typed-ooda\|ledger'
+   ```
+   All ledger access must flow through `CapabilityHandler::open`.
+3. **Disk / filesystem** — a network filesystem with weak locking (NFS) can
+   break SQLite locking regardless of the registry. The ledger must live on a
+   local filesystem.
+
+## Step 5 — Run the concurrency regression test
+
+The fix ships with a regression that reproduces the #4483 burst — many handlers,
+one ledger path, concurrent writes, asserting zero `database is locked`
+failures:
+
+```bash
+cargo test -p <crate> typed_ooda -- --nocapture 2>&1 | grep -iE 'lock|busy|persist'
+# or target the specific test module for the ledger registry
+cargo test --test typed_ooda_contracts 2>&1 | tail -20
+```
+
+Green means the shared connection, pragmas, and retry are holding. If you can
+reproduce a burst in production but the test is green, you are almost certainly
+in a Step 4 case (external writer or bypassed `open`).
+
+## Escalation
+
+If none of the above localises it, capture:
+
+- the clustered log lines (Step 1) with goal/session IDs and timestamps,
+- `PRAGMA journal_mode;` output and the presence/absence of `-wal`/`-shm`,
+- `lsof` / `fuser` output for the ledger file,
+
+and attach them to a new issue referencing #4483 and the
+[connection registry reference](../reference/typed-outcome-ledger-connection-registry-api.md).

docs/index.md

@@ -27,6 +27,7 @@ Terminal sessions and repo-grounded engineer runs now bridge through one explici
 - [Typed-capability OODA architecture](./architecture/typed-ooda-loop.md) - Semantic/typed boundary, actor-session authority, durable terminals, effect outbox, and the explicit remaining migration boundary.
 - [OODA capability API](./reference/ooda-capability-api.md) - Terminal schemas, authorization, replay, effect leases, current limitations, errors, and policy configuration.
 - [Typed OODA goal-session deterministic rails](./reference/typed-ooda-goal-session-rails.md) — the two thin rail fixes that unblocked live OODA goals (#4076): propagating `AMPLIHACK_AGENT_BINARY` to the goal-session `recipe-runner-rs` subprocess (no silent `claude` fallback) and normalizing bare goal repo names to `rysweet/<name>` at spawn admission, plus the additive Act-loop failure-detail log.
+- [Typed-outcome ledger connection registry API](./reference/typed-outcome-ledger-connection-registry-api.md) — the path-keyed shared-connection registry (WAL + `busy_timeout` + `foreign_keys` pragmas, bounded busy-retry) that makes every handler for `typed-ooda/outcomes.sqlite3` share one serialized SQLite connection, closing the systemic "database is locked" `PersistenceFailed` burst (#4483). See the [why](./concepts/typed-outcome-ledger-shared-connection.md) and the [operator runbook](./howto/diagnose-typed-outcome-database-is-locked.md).
 - [Tutorial: Complete a typed OODA cycle](./tutorials/complete-a-typed-ooda-cycle.md) - Deterministic action, no-action, replay, and conflict examples.
 - [Tutorial: Run your first local session](./tutorials/run-your-first-local-session.md) - Exercise the local runtime through the primary CLI.
 - [Simard installer reference](./reference/simard-installer.md) - Shipped deployment contract for the binary, the owned `~/.local/bin/simard` PATH entrypoint and stale-orphan reconciliation, prompt assets, user systemd units, the post-deploy version-parity gate, rollback artifacts, and dry-run controls.