feat: durable persistence + verifiable audit log + operator CLI (#126, #127, #147, #124)

AGENTS.md

@@ -10,6 +10,8 @@ reference this file and add only tool-specific guidance.
 src/weaver_kernel/        — library source (one module per concern, ≤300 lines each)
   drivers/               — capability drivers (one file per driver)
   firewall/              — context firewall (redaction, summarization, budgets)
+  stores/                — pluggable persistence (SQLite/JSONL) + verifiable audit chain
+  cli/                   — `weaver-kernel` console entry point (audit, doctor)
 tests/                   — pytest suite (one test file per module)
 examples/                — runnable demos (prefer offline; network OK with fallback)
 docs/                    — reference documentation
@@ -139,6 +141,7 @@ See [docs/agent-context/review-checklist.md](docs/agent-context/review-checklist
 | Capability design conventions | [docs/capabilities.md](docs/capabilities.md) |
 | Context firewall details | [docs/context_firewall.md](docs/context_firewall.md) |
 | Action trace export contract | [docs/trace_export.md](docs/trace_export.md) |
+| Command-line interface (`weaver-kernel`) | [docs/cli.md](docs/cli.md) |
 
 ## Update policy
 

CHANGELOG.md

@@ -7,6 +7,48 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.11.0] - 2026-06-13
+
+### Added
+- **Pluggable persistence for the trace and revocation stores (#126).** New
+  `weaver_kernel.stores` package defining `TraceStoreProtocol`,
+  `RevocationStoreProtocol`, and `HandleStoreProtocol`, with stdlib-only durable
+  backends: `SQLiteTraceStore`, `JsonlTraceStore` (append-only), and
+  `SQLiteRevocationStore`. The in-memory `TraceStore` / `HMACTokenProvider`
+  remain the defaults — inject a backend via constructor to opt in. A token
+  revoked through a `SQLiteRevocationStore` stays revoked after a process
+  restart. No new runtime dependencies.
+- **Hash-chained, verifiable audit log (#127).** Persisted traces are wrapped in
+  a `prev_hash`/`record_hash` chain (HMAC-SHA256, keyed by
+  `WEAVER_KERNEL_SECRET`). `verify_chain()` (and `SQLiteTraceStore.verify_chain()`
+  / `JsonlTraceStore.verify_chain()`) detect mutation, interior insertion,
+  deletion, and reordering, reporting the first divergent record. Tail-truncation
+  and whole-log deletion are out of scope (no signed head anchor) — see
+  `docs/security.md`. `SQLiteTraceStore.prune()` drops old records while
+  preserving verifiability of the retained suffix via a checkpoint.
+  Tamper-evidence, not non-repudiation — see `docs/security.md`.
+- **`weaver-kernel` operator CLI.** `weaver-kernel audit list|show|verify|export`
+  inspects, filters, verifies, and exports a persisted trace store, with `--json`
+  on every subcommand and redaction-safe output by construction (#147).
+  `weaver-kernel doctor` preflight-checks the environment, optional extras, and
+  token / audit-chain self-test vectors (#124). Registered via
+  `[project.scripts]`; argparse, stdlib-only. See `docs/cli.md`.
+- `examples/persistent_audit_demo.py` — offline end-to-end demo of durable
+  hash-chained audit + tamper detection + durable revocation.
+
+### Changed
+- HMAC secret loading moved to `weaver_kernel._secrets` (shared by token signing
+  and audit-chain hashing). `HMACTokenProvider` now accepts a `revocation_store`
+  and delegates revocation to it; behaviour with the default in-memory store is
+  unchanged.
+- `Kernel(trace_store=...)` is now typed against `TraceStoreProtocol`, so any
+  conforming backend (including the durable ones) can be injected.
+- `SQLiteTraceStore` now remaps `sqlite3` errors to `AgentKernelError`: a
+  duplicate `action_id` (or seq) on `record()` and opening a non-SQLite file
+  (e.g. a JSONL store without `--format jsonl`) surface as typed errors instead
+  of leaking a traceback through the CLI. The durable trace stores document an
+  explicit single-writer constraint (durable revocation remains multi-process).
+
 ## [0.10.0] - 2026-06-07
 
 ### Changed

Makefile

@@ -26,5 +26,6 @@ example:
 	python examples/chainweaver_flow.py
 	python examples/evaluation_artifact_policy.py
 	python examples/trace_export_demo.py
+	python examples/persistent_audit_demo.py
 
 ci: fmt-check lint type test example

docs/architecture.md

@@ -158,6 +158,49 @@ Records every `ActionTrace`. `explain(action_id)` returns the full audit record.
 
 `export_action_trace` / `export_action_traces` serialise traces into a stable, versioned, JSON-serialisable shape for downstream analysis tools (distinct from the OpenTelemetry observability export); `Kernel.list_traces()` is the public accessor that feeds them the audit trail. See [trace_export.md](trace_export.md).
 
+## Persistence & durable stores
+
+The stateful stores are protocol-based seams (`weaver_kernel.stores`), mirroring
+the `Driver` / `PolicyEngine` pattern. The in-memory implementations are the
+defaults; durable backends are opt-in via constructor injection.
+
+| Protocol | Default (in-memory) | Durable backends | Injected via |
+|----------|--------------------|------------------|--------------|
+| `TraceStoreProtocol` | `TraceStore` | `SQLiteTraceStore`, `JsonlTraceStore` | `Kernel(trace_store=...)` |
+| `RevocationStoreProtocol` | `InMemoryRevocationStore` | `SQLiteRevocationStore` | `HMACTokenProvider(revocation_store=...)` |
+| `HandleStoreProtocol` | `HandleStore` | *(none yet — see below)* | `Kernel(handle_store=...)` |
+
+```python
+from weaver_kernel import Kernel, HMACTokenProvider
+from weaver_kernel.stores import SQLiteTraceStore, SQLiteRevocationStore
+
+kernel = Kernel(
+    registry,
+    token_provider=HMACTokenProvider(revocation_store=SQLiteRevocationStore("revoked.db")),
+    trace_store=SQLiteTraceStore("audit.db"),
+)
+```
+
+**Backend selection.** Use the in-memory defaults for ephemeral or single-process
+use. Use `SQLiteTraceStore` for a durable, queryable, hash-chained audit trail
+that survives restarts and supports retention pruning; use `JsonlTraceStore` for
+an append-only log that is easy to ship to a collector. Use
+`SQLiteRevocationStore` when `revoke()` / `revoke_all()` must outlive a process
+or apply across workers sharing a database file. All durable backends use only
+the standard library (`sqlite3`, `json`) — no new runtime dependency.
+
+**Verifiable audit chain.** Persisted traces are hash-chained
+(`prev_hash`/`record_hash`, HMAC-SHA256 keyed by `WEAVER_KERNEL_SECRET`).
+`verify_chain()` detects mutation, insertion, deletion, and reordering;
+`SQLiteTraceStore.prune(before=...)` enforces retention while keeping the
+retained suffix verifiable via a checkpoint. The integrity model and its limits
+are documented in [security.md](security.md#audit-log-integrity-hash-chain).
+
+**Handle persistence is intentionally not shipped yet.** `HandleStoreProtocol` is
+defined so a durable backend can be added without a breaking change, but handles
+are short-lived, TTL-bounded result caches whose durability matters far less than
+the audit trail's; only the in-memory `HandleStore` ships today.
+
 ### Adapters (`weaver_kernel.adapters`)
 Vendor-specific tool-format adapters that translate between `Capability` objects
 and the tool shapes used by LLM provider APIs:

docs/cli.md

@@ -0,0 +1,82 @@
+# Command-line interface
+
+agent-kernel installs a single console entry point, `weaver-kernel`, with two
+subcommands. Both are stdlib-only (argparse) and depend on no third-party
+packages.
+
+```text
+weaver-kernel audit   — inspect, filter, verify, and export persisted action traces
+weaver-kernel doctor  — preflight-check the local environment and self-test vectors
+```
+
+## `weaver-kernel audit`
+
+Operates on a **persisted** trace store — a `SQLiteTraceStore` database or a
+`JsonlTraceStore` file (see [architecture.md](architecture.md#persistence--durable-stores)).
+The format is inferred from the path suffix (`.jsonl` → JSONL, otherwise SQLite)
+and can be forced with `--format`. Pass `--secret` to verify a chain written with
+an explicit secret; by default the CLI uses `WEAVER_KERNEL_SECRET`.
+
+Output is redaction-safe by construction: the CLI renders only what an
+`ActionTrace` already holds. No flag surfaces raw driver output.
+
+> Traces describe **authorised** invocations only. A *denied* request never
+> produces an `ActionTrace` (policy gates before invocation, per I-02), so
+> filtering is by outcome (`--status succeeded|failed`), not by an
+> allow/deny/ask decision — the trace store does not record denials.
+
+### `audit list`
+
+Table (or `--json`) view with filters:
+
+```bash
+weaver-kernel audit list --store audit.db \
+  --principal u1 --capability billing.list_invoices \
+  --status succeeded --since 2026-01-01 --until 2026-02-01 --limit 50
+```
+
+### `audit show`
+
+Full redaction-safe detail for one action (the CLI face of `kernel.explain()`):
+
+```bash
+weaver-kernel audit show <ACTION_ID> --store audit.db
+```
+
+Exits non-zero with an error on stderr if the action id is unknown.
+
+### `audit verify`
+
+Runs chain verification and reports OK or the first divergent record. **Exits
+non-zero** when tampering is detected — suitable for a cron / CI integrity check:
+
+```bash
+weaver-kernel audit verify --store audit.db        # → "OK: Verified N record(s)."
+weaver-kernel audit verify --store audit.db --json  # → {"ok": true, ...}
+```
+
+### `audit export`
+
+Filtered export as JSONL (one redaction-safe trace per line), to stdout or a
+file. Uses the same filter flags as `list`:
+
+```bash
+weaver-kernel audit export --store audit.db --principal u1 --out u1-traces.jsonl
+```
+
+## `weaver-kernel doctor`
+
+Preflight checks for a local setup. Reports each check as `ok` / `warn` / `error`
+and **exits non-zero only when a check errors** (a broken build). A missing
+`WEAVER_KERNEL_SECRET` is a *warning* (insecure demo-only configuration), not a
+failure.
+
+```bash
+weaver-kernel doctor          # human-readable
+weaver-kernel doctor --json   # machine-readable list of checks
+```
+
+Checks: Python version; whether `WEAVER_KERNEL_SECRET` is set; availability of the
+optional `policy` / `mcp` / `otel` extras; a token sign/verify + tamper-detection
+self-test vector; and an audit-chain build/verify/mutate self-test vector. Secret
+material is never printed — only whether a secret is configured.

docs/security.md

@@ -81,6 +81,43 @@ strips payload-like fields (`payload`, `content`, `value`, `memory`, `text`,
 `memory.`. Non-sensitive metadata keys (`key`, `id`, `scope`, ...) are
 preserved so audit can still confirm an action took place.
 
+## Audit-log integrity (hash chain)
+
+When traces are persisted to a durable store (`SQLiteTraceStore`,
+`JsonlTraceStore`), each record is wrapped in a hash chain: `record_hash =
+HMAC-SHA256(secret, {seq, prev_hash, trace})`, where `prev_hash` is the previous
+record's hash (the first record links to a genesis value). `verify_chain()`
+recomputes every hash and checks the linkage, so it detects:
+
+- **mutation** of any persisted record (recomputed hash diverges),
+- **interior insertion, deletion, or reordering** (broken `prev_hash` linkage or a
+  non-contiguous `seq`),
+
+and reports the `seq` of the first divergent record. `SQLiteTraceStore.prune()`
+removes old records while preserving verifiability of the retained suffix by
+recording the last pruned record's hash as a checkpoint.
+
+**Truncation is the exception.** The chain stores no signed head/length anchor, so
+dropping the **most recent** records (tail truncation) — or deleting the whole
+store — leaves a self-consistent prefix that still verifies: there is no broken
+link or sequence gap to detect, and an empty store verifies vacuously. Detecting
+truncation requires anchoring the expected head out of band (a separately stored,
+signed record count + head hash); that is a planned follow-up. Until then, treat
+append-only durability (JSONL shipped to a write-once collector, or a SQLite file
+on append-only storage) as the truncation defense.
+
+**What this is — and is not.** This is **tamper-evidence**: anyone who does not
+hold `WEAVER_KERNEL_SECRET` cannot alter the log without `verify_chain()`
+detecting it. It is **not non-repudiation**: a host that controls the secret can
+forge a self-consistent chain, and the same secret signs tokens, so the audit
+log is only as trustworthy as secret custody. It does not encrypt trace contents
+at rest, and it does not anchor the chain to an external timestamping authority.
+The chain payload is the redaction-safe export shape — chaining adds no field the
+in-memory trace did not already hold and cannot widen the I-01 boundary.
+
+The CLI exposes verification to operators: `weaver-kernel audit verify --store
+audit.db` exits non-zero on any divergence (see [cli.md](cli.md)).
+
 ## Security disclaimers
 
 > **v0.1 is not production-hardened for real authentication.**

examples/persistent_audit_demo.py

@@ -0,0 +1,103 @@
+"""Durable, verifiable audit trail + durable revocation (issues #126, #127).
+
+Runs fully offline. Demonstrates:
+
+1. Recording invocations to a SQLite-backed, hash-chained trace store.
+2. Verifying chain integrity — and detecting tampering.
+3. Revocation that survives a fresh token-provider instance (as it would across
+   a process restart) via a SQLite revocation store.
+
+Run: ``python examples/persistent_audit_demo.py``
+"""
+
+from __future__ import annotations
+
+import asyncio
+import sqlite3
+import tempfile
+from pathlib import Path
+
+from weaver_kernel import (
+    Capability,
+    CapabilityRegistry,
+    CapabilityRequest,
+    HMACTokenProvider,
+    InMemoryDriver,
+    Kernel,
+    Principal,
+    SafetyClass,
+    StaticRouter,
+)
+from weaver_kernel.errors import TokenRevoked
+from weaver_kernel.stores import SQLiteRevocationStore, SQLiteTraceStore
+
+SECRET = "persistent-audit-demo-secret"  # demo only — set WEAVER_KERNEL_SECRET in production
+
+
+def _build_kernel(trace_db: Path, provider: HMACTokenProvider) -> Kernel:
+    registry = CapabilityRegistry()
+    registry.register(
+        Capability(
+            capability_id="billing.list_invoices",
+            name="List Invoices",
+            description="List invoices for a customer",
+            safety_class=SafetyClass.READ,
+        )
+    )
+    driver = InMemoryDriver(driver_id="billing")
+    driver.register_handler("billing.list_invoices", lambda ctx: [{"id": 1, "amount": 42}])
+    kernel = Kernel(
+        registry=registry,
+        token_provider=provider,
+        router=StaticRouter(routes={"billing.list_invoices": ["billing"]}),
+        trace_store=SQLiteTraceStore(trace_db, secret=SECRET),
+    )
+    kernel.register_driver(driver)
+    return kernel
+
+
+async def _main() -> None:
+    workdir = Path(tempfile.mkdtemp(prefix="weaver-audit-demo-"))
+    trace_db = workdir / "audit.db"
+    revoke_db = workdir / "revoked.db"
+
+    provider = HMACTokenProvider(secret=SECRET, revocation_store=SQLiteRevocationStore(revoke_db))
+    kernel = _build_kernel(trace_db, provider)
+    principal = Principal(principal_id="u1", roles=["reader"])
+    request = CapabilityRequest(capability_id="billing.list_invoices", goal="list invoices")
+
+    # 1. Record a couple of invocations into the durable, hash-chained store.
+    for _ in range(2):
+        token = kernel.get_token(request, principal, justification="month-end review")
+        await kernel.invoke(
+            token, principal=principal, args={"operation": "billing.list_invoices"}
+        )
+
+    store = SQLiteTraceStore(trace_db, secret=SECRET)
+    print(f"recorded {len(store.list_all())} traces")
+    print(f"verify (intact): {store.verify_chain().detail}")
+
+    # 2. Tamper directly with the database, then re-verify.
+    conn = sqlite3.connect(str(trace_db))
+    conn.execute('UPDATE traces SET payload = \'{"action_id":"x"}\' WHERE seq = 0')
+    conn.commit()
+    conn.close()
+    tampered = SQLiteTraceStore(trace_db, secret=SECRET).verify_chain()
+    print(f"verify (tampered): ok={tampered.ok}, first_bad_seq={tampered.first_bad_seq}")
+
+    # 3. Durable revocation: revoke a token, then prove a *fresh* provider
+    #    (as after a restart) still honours the revocation from disk.
+    token = kernel.get_token(request, principal, justification="will be revoked")
+    provider.revoke(token.token_id)
+    fresh = HMACTokenProvider(secret=SECRET, revocation_store=SQLiteRevocationStore(revoke_db))
+    try:
+        fresh.verify(
+            token, expected_principal_id="u1", expected_capability_id="billing.list_invoices"
+        )
+        print("ERROR: revoked token verified")
+    except TokenRevoked:
+        print("revoked token rejected by a fresh provider (survives restart)")
+
+
+if __name__ == "__main__":
+    asyncio.run(_main())

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "weaver-kernel"
-version = "0.10.0"
+version = "0.11.0"
 description = "Capability-based security kernel for AI agents operating in large tool ecosystems"
 readme = "README.md"
 license = { file = "LICENSE" }
@@ -40,6 +40,9 @@ dependencies = [
     "pydantic>=2",
 ]
 
+[project.scripts]
+weaver-kernel = "weaver_kernel.cli:main"
+
 [project.urls]
 Homepage = "https://github.com/dgenio/agent-kernel"
 Repository = "https://github.com/dgenio/agent-kernel"