fix(typed-ooda): WAL-verified ledger open + gym self-eval status (#4480)

[rysweet]

Summary

Harden the typed-OODA ledger against the database is locked crash-loop and
surface a live gym self-eval state, with supporting docs. WAL journal mode is
now applied and verified on every ledger open() (not only during the
one-time schema migration), the busy-timeout absorbs cross-process contention,
and the status gym section reports enabled/idle instead of a dead active
value. This branch also folds in exe-mtime transient-failure resilience for the
OODA daemon helpers and the accompanying reference doc.

Issue

Closes #4480

Changed files (20 — matches git diff main...HEAD)

Source (7)

• src/typed_ooda/ledger.rs — unconditional WAL at open() + single-statement WAL read-back verification (warns if SQLite reports non-WAL), 30s busy-timeout, Immediate write txns, concurrency regression tests
• src/typed_ooda/schema.rs — drop the now-redundant WAL pragma from the migration branch
• src/status/provider.rs — gym self_eval_state → real scenario count + enabled/idle (no dead active)
• src/status/render.rs — align skipped-gym test fixture to idle
• src/self_relaunch/types.rs — supporting type changes
• src/operator_commands_ooda/daemon/helpers.rs — exe-mtime transient-failure resilience (retry/backoff) + io::Error::other clippy fix
• src/safe_update/tests_orchestrator.rs — test adjustments for the above

Docs (7)

• docs/howto/diagnose-typed-ooda-database-locked.md
• docs/reference/typed-ooda-ledger-concurrency.md
• docs/reference/deploy-gate-unit-test-canary.md
• docs/reference/gym-self-eval-status.md
• docs/reference/exe-mtime-transient-failure-resilience.md
• docs/operations/index.md
• docs/index.md

Config / build (4)

• Cargo.toml, Cargo.lock, mkdocs.yml, .github/hooks/amplihack-hooks.json

Tests (2)

• tests/adaptive_scaling.rs, tests/status_snapshot.rs

Diff stat (git diff main...HEAD)

 .github/hooks/amplihack-hooks.json                 |  12 +-
 Cargo.lock                                         |   2 +-
 Cargo.toml                                         |   2 +-
 docs/howto/diagnose-typed-ooda-database-locked.md  | 108 ++++++
 docs/index.md                                      |   2 +
 docs/operations/index.md                           |   9 +
 docs/reference/deploy-gate-unit-test-canary.md     | 151 +++++++++
 .../exe-mtime-transient-failure-resilience.md      | 364 +++++++++++++++++++++
 docs/reference/gym-self-eval-status.md             | 128 ++++++++
 docs/reference/typed-ooda-ledger-concurrency.md    | 180 ++++++++++
 mkdocs.yml                                         |   4 +
 src/operator_commands_ooda/daemon/helpers.rs       | 227 ++++++++++++-
 src/safe_update/tests_orchestrator.rs              |  14 +-
 src/self_relaunch/types.rs                         |  16 +
 src/status/provider.rs                             |  49 ++-
 src/status/render.rs                               |   2 +-
 src/typed_ooda/ledger.rs                           | 176 +++++++++-
 src/typed_ooda/schema.rs                           |   5 +-
 tests/adaptive_scaling.rs                          |   9 +-
 tests/status_snapshot.rs                           |   7 +
 20 files changed, 1446 insertions(+), 21 deletions(-)

Behavior

• Ledger concurrency (Fix the systemic 'typed outcome persistence failed: database is locked' crash-loop in Simard's typed-OODA persistence layer. Signature from journalctl --user -u simard-ooda (last 6h): 'typed goal-sess #4483): WAL journal mode applied unconditionally at
  every open() so a pre-existing v1 database still gets WAL; a read-back of
  PRAGMA journal_mode=WAL verifies the effective mode and emits a tracing
  warning if SQLite silently retained a non-WAL journal. busy_timeout raised
  5s→30s; writers use Immediate transactions.
• Gym self-eval status: assemble_gym reports the real
  benchmark_scenarios() count and an honest enabled/idle state.
• Exe-mtime resilience: daemon helpers retry transient exe-mtime read
  failures instead of failing the tick.

Review follow-ups (Step 17b)

• [1] Stale active naming — resolved: provider test/comment and
  render fixture now use enabled/idle.
• [2] Description ↔ diff mismatch — resolved by this description
  regeneration (now matches the 20-file main...HEAD diff exactly).
• [3] WAL switch not verified — resolved: open() reads back the
  effective journal mode and warns on non-WAL.

Verification

• Ledger concurrency tests pass (WAL-on-preexisting-v1, ≥30s busy-timeout,
  6×40 concurrent cross-connection writers).
• cargo fmt --check clean; cargo clippy --release -D warnings clean.

[rysweet]

Step 17b — Comprehensive Code Review

Verdict: Approve with minor, non-blocking notes. The core change is correct and well-tested: WAL is now applied unconditionally at every ledger open() (the real defect — pre-existing user_version==1 DBs skipped the migration-only WAL pragma and serialized on a whole-file lock), busy_timeout raised 5s→30s, writers use Immediate txns, and the honest gym self-eval wiring + P1 canary-gate guard are sound.

Verification performed (on PR branch feat/issue-4480-…, NODE_OPTIONS=--max-old-space-size=32768)

• ✅ Targeted tests (6/6): open_applies_wal_journal_mode_on_preexisting_v1_db, open_sets_busy_timeout_at_least_30s, concurrent_cross_connection_writers_never_hit_database_locked (6 writers × 40 iters, separate connections), assemble_gym_reports_real_scenarios_and_active_when_enabled, assemble_gym_reports_skip_flag_as_live_section, canary_default_gates_include_blocking_unit_test_gate.
• ✅ cargo fmt --check clean; cargo clippy --lib clean (no warnings).
• ✅ All internal .md links in the 4 new/changed docs resolve; referenced pages exist.
• ✅ Renderer (render.rs) passes self_eval_state through — no hardcoded consumer breaks on the new "enabled" value.
• ✅ schema::initialize has exactly one caller (CapabilityHandler::open), so dropping the WAL pragma from the migration branch is safe.
• ✅ No TODOs / stubs / swallowed errors introduced; lock errors are surfaced (fail-visible), not swallowed.

Findings

1. [Low] Stale "active" naming after the active→enabled rename.
src/status/render.rs:697 test fixture still sets self_eval_state: "active", and src/status/provider.rs test fn assemble_gym_reports_real_scenarios_and_active_when_enabled (+ its doc comment "an active state") reference the old value while the emitted/asserted value is now "enabled". Not a bug (fixture isn't asserted; renderer is pass-through), but the leftover naming is confusing. Suggest aligning fixture/test-name/comment to enabled.

2. [Low–Med] PR description does not match the actual diff.
The body's Changed files (24) and diff-stat (+849 / -1308, incl. gates.rs -418, overseer/deploy.rs -94, tests_deploy_drift.rs -238, source_prep.rs, operator_cli/mod.rs) do not match git diff main...HEAD (14 files, +822 / -7; those five files show zero changes vs main). Please regenerate the description so reviewers aren't misled about scope. (The canary-gate-convergence.md -380 deletion it lists also never existed on main and is referenced nowhere — no broken links, just stale metadata.)

3. [Low, optional hardening] WAL switch is not verified.
open() uses pragma_update(None,"journal_mode","WAL") but doesn't read back the effective mode. SQLite can silently retain the prior journal mode (e.g., on some network filesystems), which would let the exact crash-loop this fixes persist undetected. Since the how-to doc already instructs operators to check PRAGMA journal_mode manually, this is optional: consider reading the returned mode and emitting a tracing warning if it isn't wal.

Positives

Root-cause fix (not symptom quarantine); strong concurrency regression test; honest assemble_gym wiring to real benchmark_scenarios() count; the canary-gate guard test prevents a future silent gate removal; hermeticity fixes (scaler: None, XDG_DATA_HOME pin) are correct and clearly commented.

No blocking issues. Items 1–3 are follow-ups; 2 (description accuracy) is the most worth addressing before merge.

[rysweet]

Step 17c — Security Review (MANDATORY)

Verdict: APPROVE — no exploitable security vulnerabilities found.

Reviewed the actual 14-file diff (+822/-7); security-relevant surface is 4 source files (typed_ooda/ledger.rs, typed_ooda/schema.rs, status/provider.rs, self_relaunch/types.rs). Every concern from the security checklist was checked concretely and cleared.

Checklist evidence

• Injection (SQL/PRAGMA/command/path) — CLEAR

  • ledger.rs pragma_update(None, "journal_mode", "WAL") and the new PRAGMA journal_mode / PRAGMA busy_timeout query_row calls use constant literals only — no untrusted input reaches any PRAGMA/SQL string.
  • The write path exercised by the new concurrency tests, release_engineer_claim, uses a fully parameterized DELETE ... WHERE claim_key = ?1 (bound params![claim_key]) — no string concatenation, no injection via claim/goal identifiers.
  • Connection::open takes a typed Path; no shell invocation or dynamic path construction. Tests use tempfile::tempdir() (per-process, 0700).

• Sensitive-data handling — CLEAR

  • assemble_gym surfaces only benchmark_scenarios().len() (count of compiled-in curated scenarios) and a static "enabled"/"idle" string. No secrets, tokens, PII, paths, or runtime state exposed.
  • WAL -wal/-shm sidecars inherit the existing ledger dir's ownership/permissions — the main DB already lives there; no new info-disclosure surface, no relocation to a temp/world-readable path.

• Authentication / authorization — CLEAR (control strengthened, not weakened)

  • The new self_relaunch/types.rs test asserts default_gates() still contains the blocking RelaunchGate::UnitTest canary gate. This guards a security-relevant self-deploy invariant; it does not remove, disable, or make any gate non-blocking.

• New vulnerabilities / concurrency — none

  • busy_timeout 5s→30s and unconditional WAL are local-file SQLite tuning. The 30s wait is at most an availability/latency (DoS-class) concern, not attacker-triggerable across a trust boundary — contending processes (daemon, engineer worktree, reaper) are all first-party.
  • Journal-mode pragma_update propagates errors via .map_err(persistence)? (not swallowed). It ignores the returned mode string, so on an exotic FS WAL could remain rollback mode — an integrity/robustness edge only, and it fails toward stricter whole-file locking, never weaker isolation. Removing the redundant WAL pragma from schema.rs is safe because open() now asserts WAL on every connection before initialize().

• Supply chain — CLEAR

  • Cargo.toml/Cargo.lock: internal version bump 0.36.0 → 0.37.0 only. No new dependencies, no third-party version/source changes.

Verdict

Assessed against injection, crypto, access-control, credential, data-leak, misconfiguration, auth, integrity, SSRF, supply-chain, and prompt-injection categories. No finding meets the reporting threshold. Nothing to remediate from a security standpoint.

[rysweet]

Step 17d — Philosophy Guardian Review

Verdict: ✅ COMPLIANT. This PR is a model of the project philosophy: it replaces a stub with real behavior, fixes a real defect with a minimal change, and adds a guard against future BS — with no new abstractions or scope creep.

Compliance checklist

• Ruthless simplicity — The core defect fix is two lines: raise busy_timeout 5s→30s and apply journal_mode=WAL unconditionally at open(). No config knobs, no retry frameworks, no new types. The gym fix swaps a hardcoded None/"idle" for a real benchmark_scenarios().len() count.
• Bricks & studs / clean module boundaries — WAL responsibility is consolidated into a single owner (CapabilityHandler::open) and removed from the schema::initialize migration branch. Both sites carry a comment explaining the contract, so the seam is explicit rather than implicit. Changes stay within their modules (typed_ooda, status, self_relaunch).
• Zero-BS implementation — This is the standout. assemble_gym previously always returned an inert stub (configured_scenarios: None, self_eval_state: "idle"); it now surfaces the REAL scenario count when enabled. The canary guard test fails loudly if a future change drops the blocking UnitTest gate to make self-deploy "pass" — directly codifying "root-cause, don't paper over." No stubs, no faked APIs, no swallowed exceptions (all persistence errors propagate via .map_err(persistence)?).
• No over-engineering — 30s is a plain constant; WAL is asserted once. No speculative generality. Tests are behavior-specced (pre-existing-v1 WAL, busy_timeout floor, concurrent cross-connection writers) rather than mock-heavy.
• Honest semantics — self_eval_state value derives from !skip_gym and is named "enabled" (not a fake "active" implying an in-flight eval). The "idle"/skipped path is unchanged. This is truthful state reporting.

Minor, non-blocking observations

1. [Low] open() doesn't assert that PRAGMA journal_mode actually returned wal (SQLite can silently keep the prior mode if the DB is mid-transaction on another connection). Errors from the pragma statement itself are propagated; a follow-up tracing::warn! on an unexpected return value would fully close the Zero-BS loop. Already noted in the code review.
2. [Low] The assemble_gym_reports_real_scenarios_and_active_when_enabled test name still says "active" though the asserted value is "enabled" — cosmetic drift, no behavioral impact.

No philosophy blockers. The change strengthens Zero-BS compliance (stub → real data, guard against gaming gates) while holding the line on ruthless simplicity.

[rysweet]

Ready for Final Review

Workflow steps completed: requirements, design, implementation, tests, code review, philosophy compliance, cleanup, and quality audit.

Ready for merge approval.

[github-actions]

📊 Coverage Summary

Generated by cargo llvm-cov --workspace --summary-only (nightly, excluding test files)

Module	Lines	Covered	Coverage
Total	196315	165051	84.1%

_{Coverage data from CI run. Test files matching tests?/ are excluded from line counts.}