feat(watchdog): independent journal-driven restore path

[undeemed]

Intent

Implement the independent watchdog restore path for fpsmaxxing: a Linux-safe watchdog (library + binary in apps/watchdog) that recovers experiments abandoned when the gateway, workload, or agent dies, with no gateway/experiment-runner/LLM involvement, satisfying the safety invariant that the watchdog can restore state independently.

Detection and recovery: read the durable experiment journal owned by crates/control-plane (ADR 0002) and find two leak classes - unclosed experiments (a write-ahead apply-intent with no terminal completed/failed record = crash between mutation and rollback) and expired TTL leases (lease elapsed, no terminal record). Roll each back through its provider using the journaled snapshot, verifying the restore by re-reading provider state.

Key deliberate design decisions (so these are not mistaken for oversights in review): (1) The watchdog owns its OWN read-only journal module (apps/watchdog/src/journal.rs) and reads the SQLite journal directly, rather than adding read APIs to control-plane. This keeps control-plane changes at ZERO, preserves the independence invariant, and avoids overlap with a sibling crewmate concurrently adding an additive trial-journal API to control-plane. (2) The watchdog writes NOTHING to the journal except its own correlation-ID-linked 'watchdog-restore' records and the terminal 'failed' record that closes a restored experiment; it never creates or mutates the schema. (3) Idempotence by design: a successful restore writes a terminal record so the experiment is never selected again (no double-rollback); a failed rollback is left unclosed for a later pass to retry. Two policies: AllUnclosed (crash/reboot recovery) and ExpiredLeasesOnly (steady-state poll). (4) Lease age is computed with the journal's own SQL clock while lease_seconds is decoded from the write-ahead ChangeRequest in Rust, avoiding a JSON1 dependency; a missing/negative age fails safe to not-expired. (5) The binary provides an interval poll loop, a single --once pass (the unit a later Windows-service timer would drive), and --recover-all; Windows service registration is explicitly out of scope - the container is Linux-only, no Windows APIs, no real hardware.

Tests: 10 fault-injection E2E cases against sidecars/mock-provider covering crash mid-apply restoring defaults, real gateway/control-plane termination mid-apply (a panicking provider driven through the actual run_lifecycle write path), expired-lease reclaim, a live lease left for its owner, idempotence/no-double-rollback, failed and silently-corrupted rollback both retried, provider isolation, cleanly-closed experiments left untouched (real ControlPlane E2E), and a --once binary smoke test.

Docs: updated ARCHITECTURE watchdog section, README feature list, and an AGENTS repository rule pointing to apps/watchdog. Constraints honored: additive only, no control-plane schema or run_lifecycle changes; full local validation (cargo fmt --check, cargo check --workspace --all-targets, cargo clippy --workspace --all-targets -D warnings, cargo test --workspace, cargo deny check) passes.

What Changed

• Added a new apps/watchdog library and binary that recovers experiments abandoned when the gateway, workload, or agent dies, with no gateway/experiment-runner/LLM involvement. It ships its own read-only journal module (journal.rs) that reads the control-plane SQLite journal directly (zero control-plane changes), detects the two leak classes (unclosed apply-intents from a crash mid-apply, and expired TTL leases), rolls each back through its provider using the journaled snapshot, and verifies the restore by re-reading provider state. Restores are idempotent by writing a correlation-ID-linked terminal record; failed rollbacks are left unclosed for a later pass to retry.
• The binary drives recovery via an interval poll loop, a single --once pass, and --recover-all, selecting between the AllUnclosed (crash/reboot) and ExpiredLeasesOnly (steady-state) policies. Per review feedback, --recover-all is forced to a single pass so continuous polling always stays ExpiredLeasesOnly, and repeated restore failures are deduped to bound journal growth.
• Added 11 fault-injection E2E tests against the mock provider (crash mid-apply, real control-plane termination via a panicking provider, expired-lease reclaim, live lease left for its owner, idempotence, failed and corrupted rollback both retried, provider isolation, cleanly-closed experiments untouched, --once smoke) plus 2 binary-config unit tests, and updated the ARCHITECTURE watchdog section, README feature list and binary usage, and an AGENTS repository rule pointing to apps/watchdog.

Risk Assessment

✅ Low: Additive, well-isolated watchdog crate that only reads the control-plane journal and writes its own restore records; both prior-round findings were correctly resolved (single-pass --recover-all with unit tests, and retry-preserving failure dedup), and no new material bug was substantiated across the changed code, queries, or contracts.

Testing

Ran the watchdog crate's 11 fault-injection E2E tests and 2 config unit tests and the full cargo test --workspace suite (all pass) after fixing a build-environment issue where the sandbox's zig-cc compiler rejected the Rust target triple for bundled SQLite (resolved with ephemeral compiler wrappers, no source or global-config changes). Beyond the tests, I demonstrated the intent as an operator would experience it by running the real compiled watchdog binary against journals seeded to match the control-plane schema: it independently restored crash-leaked state and closed the experiment, was idempotent on re-run, and in steady-state poll reclaimed only the expired lease while leaving a live lease for its owner. Evidence captured as a CLI transcript with before/after journal state.

Evidence: Watchdog CLI transcript: crash-recovery + idempotence + expired-lease-only poll with before/after journal state

==========================================================================
DEMO A - crash recovery (owner died mid-experiment, lease still live)
==========================================================================

  A leaked experiment: baseline snapshot value=7, apply-intent
  wanted value=99, no terminal record (owner crashed). Lease 300s
  has NOT expired, but the owner is gone.

  journal state: BEFORE watchdog
  seq | exp | stage           | provider | payload
  ----+-----+-----------------+----------+----------------------------------------
    1 |   1 | snapshot        | mock     | {"value":7}
    2 |   1 | apply-intent    | mock     | params={"value":99} lease=300s

  $ fpsmaxxing-watchdog --once --recover-all --journal /tmp/wd-work/journal_a.sqlite
  fpsmaxxing-watchdog: restored experiment 1 (crash recovery, lease 300s)
  [exit 0]

  journal state: AFTER watchdog --recover-all
  seq | exp | stage           | provider | payload
  ----+-----+-----------------+----------+----------------------------------------
    1 |   1 | snapshot        | mock     | {"value":7}
    2 |   1 | apply-intent    | mock     | params={"value":99} lease=300s
    3 |   1 | watchdog-restore | mock     | {"reason":"crash-recovery","restored":true} snap={"value":7}
    4 |   1 | failed          | mock     | {"kind":"watchdog-reclaimed","stage":"watchdog","reason":"crash-recovery"}

  Re-run to prove idempotence (a closed experiment is never
  rolled back twice):

  $ fpsmaxxing-watchdog --once --recover-all --journal /tmp/wd-work/journal_a.sqlite
  (no output - nothing to restore)
  [exit 0]

  journal state: AFTER second pass (unchanged)
  seq | exp | stage           | provider | payload
  ----+-----+-----------------+----------+----------------------------------------
    1 |   1 | snapshot        | mock     | {"value":7}
    2 |   1 | apply-intent    | mock     | params={"value":99} lease=300s
    3 |   1 | watchdog-restore | mock     | {"reason":"crash-recovery","restored":true} snap={"value":7}
    4 |   1 | failed          | mock     | {"kind":"watchdog-reclaimed","stage":"watchdog","reason":"crash-recovery"}


==========================================================================
DEMO B - steady-state poll reclaims ONLY expired leases
==========================================================================

  Two leaked experiments owned by 'mock':
   - exp 1: intent 3600s old, lease 30s  -> lease EXPIRED
   - exp 2: intent fresh,     lease 300s -> still inside lease (LIVE)

  journal state: BEFORE watchdog
  seq | exp | stage           | provider | payload
  ----+-----+-----------------+----------+----------------------------------------
    1 |   1 | snapshot        | mock     | {"value":3}
    2 |   1 | apply-intent    | mock     | params={"value":60} lease=30s
    3 |   2 | snapshot        | mock     | {"value":4}
    4 |   2 | apply-intent    | mock     | params={"value":70} lease=300s

  Default poll policy is expired-leases-only, so a live lease is
  left for its rightful owner.

  $ fpsmaxxing-watchdog --once --journal /tmp/wd-work/journal_b.sqlite
  fpsmaxxing-watchdog: restored experiment 1 (lease expired, lease 30s)
  [exit 0]

  journal state: AFTER watchdog --once (default = expired leases only)
  seq | exp | stage           | provider | payload
  ----+-----+-----------------+----------+----------------------------------------
    1 |   1 | snapshot        | mock     | {"value":3}
    2 |   1 | apply-intent    | mock     | params={"value":60} lease=30s
    3 |   2 | snapshot        | mock     | {"value":4}
    4 |   2 | apply-intent    | mock     | params={"value":70} lease=300s
    5 |   1 | watchdog-restore | mock     | {"reason":"lease-expired","restored":true} snap={"value":3}
    6 |   1 | failed          | mock     | {"kind":"watchdog-reclaimed","stage":"watchdog","reason":"lease-expired"}

  => exp 1 (expired) closed by terminal record: True
  => exp 2 (live)    left untouched, still open : True

==========================================================================
Both demos used the real compiled fpsmaxxing-watchdog binary with no
gateway / experiment-runner / LLM present - the independence invariant.
==========================================================================

Evidence: Watchdog fault-injection E2E result

running 11 tests
test binary_once_reclaims_and_closes_a_leaked_experiment ... ok
test closed_experiments_are_never_touched ... ok
test corrupted_rollback_is_detected_by_the_verify_probe ... ok
test crash_mid_apply_restores_defaults ... ok
test expired_lease_is_reclaimed_by_the_steady_state_poll ... ok
test expired_leases_only_leaves_live_experiments_for_their_owner ... ok
test failed_rollback_leaves_the_experiment_for_a_later_pass ... ok
test gateway_termination_mid_apply_is_recovered ... ok
test other_providers_experiments_are_ignored ... ok
test reclaim_is_idempotent_and_does_not_double_rollback ... ok
test repeated_identical_failures_are_not_re_journaled_but_stay_retryable ... ok
test result: ok. 11 passed; 0 failed

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

🔧 **Review** - 1 issue found → auto-fixed (2) ✅

• ⚠️ apps/watchdog/src/main.rs:121 - --recover-all without --once runs the AllUnclosed policy on every poll interval (main loop lines 32-39 + Config::policy lines 120-126). AllUnclosed selects unclosed experiments regardless of lease deadline (lib.rs:99), so a continuous AllUnclosed poll will roll back an in-flight experiment that is still inside its lease as soon as any component holds a real lease (the experiment-runner scaffold, or a provider whose operation outlives a short lease), fighting the live owner and violating the 'only one provider may own a knob at a time' / 'leave live leases to their owner' invariants that ExpiredLeasesOnly is designed to protect. Harmless today only because the mock lifecycle is synchronous. Consider making --recover-all imply a single pass (or rejecting --recover-all combined with the loop) so continuous polling is always ExpiredLeasesOnly.

🔧 Fix: force --recover-all to a single pass in watchdog
2 issues (1 warning, 1 info) still open:

• ⚠️ apps/watchdog/src/main.rs:32 - In the steady-state loop (ExpiredLeasesOnly), a permanently-unrestorable expired-lease experiment is re-selected and re-journaled on every tick with no backoff or attempt cap. reclaim re-runs scan() each pass (lib.rs:180); a failed restore calls record_failure (journal.rs:148) which appends a new watchdog-restore row but no terminal record, so the same experiment stays selectable forever. A genuinely stuck knob (or the corrupt-rollback case that never verifies) therefore writes a fresh failure record every interval seconds (default 5s) indefinitely, growing the shared journal without bound and hammering provider.rollback/snapshot. Because the scan query does a full-table JOIN + NOT EXISTS, the accumulating rows also progressively slow every scan for both the watchdog and the control plane that share the file. Consider a bounded retry / exponential backoff, or a distinct non-selecting watchdog-abandoned record after N failures so a stuck experiment stops being retried at 5s cadence.
• ℹ️ apps/watchdog/src/lib.rs:182 - reclaim processes candidates in ascending experiment_id order (journal.rs:64 ORDER BY intent.experiment_id), rolling each back through the single shared provider instance. If two experiments on the same knob are simultaneously unclosed (a double-fault that also breaches the one-owner-per-knob invariant), the knob ends at the second-oldest experiment's snapshot rather than the true original baseline: e.g. exp1 snapshots A then leaks, exp2 snapshots the leaked B then leaks; reclaim sets knob->A (exp1) then knob->B (exp2), leaving B instead of A. Cannot occur on the current synchronous single-knob mock path and is guarded by the one-owner-per-knob invariant, but the watchdog is precisely the path that runs after that invariant may have been violated, so the recovery order is worth a deliberate decision.

🔧 Fix: dedup repeated watchdog restore failures; document reclaim order
✅ Re-checked - no issues remain.

✅ **Test** - passed

✅ No issues found.

• cargo test -p fpsmaxxing-watchdog — 11 fault-injection E2E tests (crash mid-apply, real control-plane gateway termination via PanicOnApply, expired-lease reclaim, live-lease left alone, idempotence, failed rollback retried, corrupted rollback caught by verify probe, provider isolation, cleanly-closed experiment untouched, --once binary smoke) + 2 binary-config unit tests, all pass
• cargo test --workspace — full suite green (contracts 8, control-plane 9, gateway mcp_e2e 2, mock-provider 1, watchdog 11+2)
• Manual CLI E2E: ran the real target/debug/fpsmaxxing-watchdog binary via --once --recover-all --journal <db> (crash recovery + idempotence re-run) and --once --journal <db> (default expired-leases-only poll) against seeded leaked journals, dumping journal rows before/after
• Setup fix: CC/CXX (zig-cc) rejected the Rust triple aarch64-unknown-linux-gnu; added ephemeral /tmp wrapper scripts translating the triple and exposing zig ar so the bundled-SQLite C build compiles

✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.