diff --git a/docs/reference/cognitive-memory-provenance.md b/docs/reference/cognitive-memory-provenance.md index c5397c3a3..c36e0ebed 100644 --- a/docs/reference/cognitive-memory-provenance.md +++ b/docs/reference/cognitive-memory-provenance.md @@ -450,9 +450,61 @@ self-metrics they sit alongside. --- +## Observability: snapshot-dedup-hygiene self-metric + +Grounding coverage watches whether facts enter the graph *connected*; a sibling +self-metric watches whether the **goal-board snapshot layer stays lean**. +Goal-board snapshots are revisioned: each new revision `SUPERSEDES` the prior +one, and +`prune_superseded` (controlled forgetting) reclaims the archived revisions over +time. `graph_stats()` already reports two raw counts for this layer: + +| Field | Meaning | +|---|---| +| `snapshot_facts_total` | every goal-board snapshot revision still held (live + not-yet-pruned superseded) | +| `distinct_snapshot_caller_keys` | distinct logical goal-board snapshot streams behind them | + +The durable `goal_board_snapshot_dedup_ratio` self-metric is the hygiene *health* +signal derived from them. After each successful OODA cycle, when `graph_stats()` +succeeds, the daemon emits one sample to the `metrics.jsonl` series from the +**same** snapshot it already collects for the OTel edge gauges and the +grounding-coverage metric: + +``` +goal_board_snapshot_dedup_ratio = distinct_snapshot_caller_keys / snapshot_facts_total +``` + +- **What it measures.** The average *liveness* of goal-board snapshot streams, + in `[0.0, 1.0]`. `1.0` means every stream holds a single live revision; the + value falls toward `0` as superseded revisions pile up. When every snapshot + fact has a valid caller key, its inverse — total / distinct — is the mean + revisions retained per stream. +- **Why it matters.** That accumulation is exactly the monotonic-growth failure + controlled forgetting exists to prevent: if `prune_superseded` stops keeping + pace, archived revisions bloat semantic memory. Previously that was visible + only as the raw `graph_stats()` counts. The store-size-independent ratio adds + a durable, comparable history for operator analysis and future automated + regression detection; it does not currently create a Gym history signal. +- **Undefined on an empty goal-board snapshot layer.** When the store holds zero + goal-board snapshot facts, the ratio is *undefined* and **no** sample is + emitted (skip rather than + drag the series to a misleading `0.0`), mirroring the `fact_provenance_coverage` + convention. `distinct_snapshot_caller_keys` is clamped to `snapshot_facts_total` + defensively so a miscount can never yield a ratio above `1.0`. The emitter is + best-effort — a metrics-write failure is logged, never propagated — and pure + observation: it never changes memory state. + +The scoring is a pure function +(`cognitive_memory::metrics::goal_board_snapshot_dedup_ratio`) with the +per-cycle emitter (`record_goal_board_snapshot_dedup_ratio_metric`) beside +`record_provenance_coverage_metric`, so both graph-memory hygiene self-metrics +sit together. + +--- + ## Testing -The feature is covered by a TDD round-trip test (in +Provenance is covered by a TDD round-trip test (in `src/cognitive_memory/tests_provenance.rs`) that: 1. opens an in-memory backend via `LibraryCognitiveMemory::in_memory()` @@ -468,11 +520,29 @@ returns an empty list) and passes once the adapter records and traverses the `DERIVES_FROM` edge — proving the link is recallable end-to-end through Simard's own API. +Goal-board snapshot hygiene is covered by: + +1. ratio boundary tests in `src/cognitive_memory/metrics.rs`, +2. a hermetic injected-writer metric-entry construction test that asserts the + metric name, value, and serialized context (it does not exercise the real + JSONL storage path), and +3. a daemon wiring test that passes asymmetric `GraphStats` counts through + `record_graph_memory_self_metrics` and the goal-board emitter's injected + writer, then asserts the resulting metric name, `0.25` value, and serialized + `snapshot_facts` / `distinct_caller_keys` context. The asymmetric counts make + the test fail if the numerator and denominator are reversed. + +The process-boundary `memory stats --json` test in +`tests/bin_simard_memory_cli.rs` separately proves the same goal-board counts +are exposed to operators. + Run the relevant suites with: ```bash cargo test cognitive_memory cargo test memory_consolidation +cargo test graph_memory_metric_sweep_uses_goal_board_graph_stats_fields +cargo test --test bin_simard_memory_cli stats_shows_edges_and_dedup_section_via_direct_open ``` --- @@ -488,10 +558,13 @@ cargo test memory_consolidation `facts_with_provenance` / `facts_total` snapshot the coverage metric reads. - `src/cognitive_memory/metrics.rs` — `provenance_coverage()` (pure ratio) and `record_provenance_coverage_metric()` (per-cycle `fact_provenance_coverage` - emitter), plus the `GraphStats` snapshot type in `src/memory_cognitive.rs`. + emitter), plus the sibling `goal_board_snapshot_dedup_ratio()` / + `record_goal_board_snapshot_dedup_ratio_metric()` (per-cycle + `goal_board_snapshot_dedup_ratio` emitter), plus the `GraphStats` snapshot + type in `src/memory_cognitive.rs`. - `src/operator_commands_ooda/daemon/mod.rs` — the per-cycle sweep that reads - `graph_stats()` for the OTel edge gauges and emits the coverage self-metric - from the same snapshot. + `graph_stats()` for the OTel edge gauges and emits both graph-memory + self-metrics from the same snapshot. - `src/memory_consolidation/distillation.rs` — distillation writer that threads `source_episode_id` as `DERIVES_FROM` provenance. - `src/memory_consolidation/mod.rs` — `reflection_memory_operations` that diff --git a/docs/reference/telemetry-metrics.md b/docs/reference/telemetry-metrics.md index 6bf2f6da6..b4daab5a6 100644 --- a/docs/reference/telemetry-metrics.md +++ b/docs/reference/telemetry-metrics.md @@ -187,6 +187,17 @@ gauges the section renders `absent`, never a fabricated zero. > comparable and regressable, not just a raw count. See > [Cognitive-memory provenance § Observability](./cognitive-memory-provenance.md#observability-grounding-coverage-self-metric). +> **Goal-board snapshot hygiene.** A sibling durable +> **`goal_board_snapshot_dedup_ratio`** +> self-metric emits, from the same per-cycle `graph_stats()` snapshot, the +> average *liveness* of goal-board snapshot revisions +> (`distinct_snapshot_caller_keys / snapshot_facts_total` ∈ `[0, 1]`, higher is +> healthier). It falls when superseded snapshot revisions accumulate faster than +> controlled forgetting (`prune_superseded`) reclaims them, turning a pruning +> regression into a durable time series for operator and future automated +> analysis rather than only a raw count. See +> [Cognitive-memory provenance § Snapshot dedup hygiene](./cognitive-memory-provenance.md#observability-snapshot-dedup-hygiene-self-metric). + ### LLM usage — `simard.llm.*` Mirrored from `cost_tracking` (the ledger format is unchanged; these are diff --git a/src/cognitive_memory/metrics.rs b/src/cognitive_memory/metrics.rs index 8bccf6938..ed733177f 100644 --- a/src/cognitive_memory/metrics.rs +++ b/src/cognitive_memory/metrics.rs @@ -272,6 +272,105 @@ pub fn record_provenance_coverage_metric(facts_with_provenance: u64, facts_total } } +// ─────────────────────── graph-memory snapshot dedup hygiene ──────────────── +// +// A graph-memory *hygiene* signal complementary to grounding coverage above. +// Goal-board snapshot facts are revisioned: each new revision SUPERSEDES the +// prior one, and `prune_superseded` (controlled forgetting) reclaims archived +// revisions over time. `snapshot_facts_total` counts every goal-board snapshot +// revision the store still holds (live + not-yet-pruned superseded); +// `distinct_snapshot_caller_keys` counts the distinct logical streams behind +// them. Their ratio is the average *liveness* of the goal-board snapshot layer: +// 1.0 when +// every stream holds exactly one revision, falling toward 0 as superseded +// revisions accumulate faster than pruning reclaims them. That accumulation is +// exactly the monotonic-growth failure controlled forgetting exists to prevent, +// and — like grounding coverage — it was previously visible only as raw +// `graph_stats()` counts, never as a durable, comparable `metrics.jsonl` +// series. Emitting the ratio gives operators a store-size-independent history +// for manual and future automated regression analysis. + +/// Durable self-metric name for goal-board snapshot dedup hygiene, emitted to +/// `metrics.jsonl` after each successful OODA cycle when `graph_stats()` succeeds +/// by +/// [`record_goal_board_snapshot_dedup_ratio_metric`]. +pub const GOAL_BOARD_SNAPSHOT_DEDUP_RATIO_METRIC: &str = "goal_board_snapshot_dedup_ratio"; + +/// Average liveness of goal-board snapshot facts +/// (`distinct_snapshot_caller_keys / snapshot_facts_total`), in `[0.0, 1.0]`. +/// Higher is healthier: `1.0` means every goal-board stream holds a single live +/// revision; a value approaching `0` means superseded revisions have piled up +/// When every snapshot fact has a valid caller key, the inverse — total / +/// distinct — is the mean revisions retained per stream. +/// +/// Returns `None` (undefined, **not** `0.0`) when the store holds no goal-board +/// snapshot facts, so an empty goal-board snapshot layer contributes no +/// misleading `0.0` sample — the same "skip rather than drag the series to +/// zero" convention [`provenance_coverage`] and [`precision_at_k`] use. +/// `distinct_snapshot_caller_keys` is clamped to `snapshot_facts_total` +/// defensively (a stream always has ≥1 revision, so distinct ≤ total holds), so +/// a backend that miscounts can never yield a ratio above `1.0`. +pub fn goal_board_snapshot_dedup_ratio( + distinct_snapshot_caller_keys: u64, + snapshot_facts_total: u64, +) -> Option { + if snapshot_facts_total == 0 { + return None; + } + let distinct = distinct_snapshot_caller_keys.min(snapshot_facts_total); + Some(distinct as f64 / snapshot_facts_total as f64) +} + +/// Emit one durable [`GOAL_BOARD_SNAPSHOT_DEDUP_RATIO_METRIC`] sample (the +/// goal-board snapshot liveness ratio over the current `graph_stats()` snapshot) +/// to `metrics.jsonl`. +/// +/// Called after each successful OODA cycle when the daemon's metric sweep can +/// read `graph_stats()`, from the same block that records OpenTelemetry edge +/// gauges and [`record_provenance_coverage_metric`], so it adds no extra store +/// read. A snapshot-shaped metric (store state, not a per-cycle accumulator). +/// +/// No-op when the store holds no goal-board snapshot facts (undefined ratio — +/// see [`goal_board_snapshot_dedup_ratio`]), so the series carries signal only. +/// Best-effort: a metrics-write failure is logged, never propagated. +pub fn record_goal_board_snapshot_dedup_ratio_metric( + distinct_snapshot_caller_keys: u64, + snapshot_facts_total: u64, +) { + if cfg!(test) { + return; + } + if let Err(e) = record_goal_board_snapshot_dedup_ratio_metric_with( + distinct_snapshot_caller_keys, + snapshot_facts_total, + crate::self_metrics::record_metric, + ) { + tracing::warn!( + target: "simard::memory", + error = %e, + "failed to record goal_board_snapshot_dedup_ratio metric (memory unaffected)", + ); + } +} + +pub(crate) fn record_goal_board_snapshot_dedup_ratio_metric_with( + distinct_snapshot_caller_keys: u64, + snapshot_facts_total: u64, + writer: impl FnOnce(&str, f64, &str) -> Result<(), E>, +) -> Result<(), E> { + let Some(ratio) = + goal_board_snapshot_dedup_ratio(distinct_snapshot_caller_keys, snapshot_facts_total) + else { + return Ok(()); + }; + let context = serde_json::json!({ + "snapshot_facts": snapshot_facts_total, + "distinct_caller_keys": distinct_snapshot_caller_keys, + }) + .to_string(); + writer(GOAL_BOARD_SNAPSHOT_DEDUP_RATIO_METRIC, ratio, &context) +} + #[cfg(test)] mod tests { use super::*; @@ -415,4 +514,61 @@ mod tests { record_provenance_coverage_metric(3, 4); record_provenance_coverage_metric(0, 0); } + + // ── graph-memory snapshot dedup hygiene: pure math ────────────────────── + + #[test] + fn snapshot_dedup_ratio_is_distinct_streams_over_total_revisions() { + // Two streams, four revisions retained → each stream averages two + // revisions → liveness 0.5. One-revision-per-stream is a healthy 1.0. + assert_eq!(goal_board_snapshot_dedup_ratio(2, 4), Some(0.5)); + assert_eq!(goal_board_snapshot_dedup_ratio(4, 4), Some(1.0)); + assert_eq!(goal_board_snapshot_dedup_ratio(1, 8), Some(0.125)); + // Snapshot facts present but none carry a grouping key → distinct 0 over + // a nonzero total is a real, maximally-unhealthy 0.0 (emit it), NOT the + // undefined None reserved for an empty goal-board snapshot layer. + assert_eq!(goal_board_snapshot_dedup_ratio(0, 4), Some(0.0)); + } + + #[test] + fn snapshot_dedup_ratio_is_none_for_an_empty_snapshot_layer() { + // No snapshot facts → undefined ratio (skip, do NOT emit a misleading + // 0.0), matching the provenance_coverage / precision@k convention. A + // nonzero distinct count with a zero denominator is still None. + assert_eq!(goal_board_snapshot_dedup_ratio(0, 0), None); + assert_eq!(goal_board_snapshot_dedup_ratio(3, 0), None); + } + + #[test] + fn snapshot_dedup_ratio_clamps_overcount_to_one() { + // distinct ≤ total always holds (a stream has ≥1 revision); a backend + // that miscounts must never yield a ratio above 1.0. + assert_eq!(goal_board_snapshot_dedup_ratio(9, 4), Some(1.0)); + } + + #[test] + fn record_goal_board_snapshot_dedup_ratio_metric_builds_expected_entry() { + let mut recorded = None; + record_goal_board_snapshot_dedup_ratio_metric_with(2, 4, |name, value, context| { + recorded = Some((name.to_string(), value, context.to_string())); + Ok::<(), ()>(()) + }) + .expect("record metric through injected writer"); + + let (name, value, context) = recorded.expect("one metric entry"); + assert_eq!(name, GOAL_BOARD_SNAPSHOT_DEDUP_RATIO_METRIC); + assert_eq!(value, 0.5); + let context: serde_json::Value = + serde_json::from_str(&context).expect("metric context JSON"); + assert_eq!(context["snapshot_facts"], 4); + assert_eq!(context["distinct_caller_keys"], 2); + + let mut called = false; + record_goal_board_snapshot_dedup_ratio_metric_with(0, 0, |_, _, _| { + called = true; + Ok::<(), ()>(()) + }) + .expect("empty snapshot layer is a no-op"); + assert!(!called); + } } diff --git a/src/operator_commands_ooda/daemon/mod.rs b/src/operator_commands_ooda/daemon/mod.rs index 8b779a0f1..fc4cd4cf3 100644 --- a/src/operator_commands_ooda/daemon/mod.rs +++ b/src/operator_commands_ooda/daemon/mod.rs @@ -267,6 +267,17 @@ fn spawn_stats_snapshot_refresher(memory: std::sync::Weak, @@ -1754,34 +1765,37 @@ pub fn run_ooda_daemon( &[(names::ATTR_TYPE, "sensory")], ); } - if let Ok(g) = memories.memory.graph_stats() { - telemetry::gauge_set( - names::MEMORY_EDGES, - g.derives_from_edges as i64, - &[(names::ATTR_TYPE, "DERIVES_FROM")], - ); - telemetry::gauge_set( - names::MEMORY_EDGES, - g.similar_to_edges as i64, - &[(names::ATTR_TYPE, "SIMILAR_TO")], - ); - telemetry::gauge_set( - names::MEMORY_EDGES, - g.supersedes_edges as i64, - &[(names::ATTR_TYPE, "SUPERSEDES")], - ); - // Emit the durable graph-memory grounding-coverage - // self-metric from the SAME snapshot (no extra store - // read): fraction of semantic facts connected into the - // DERIVES_FROM provenance graph. Turns a grounding - // regression — facts entering semantic memory without a - // provenance edge — into a comparable, regressable - // `metrics.jsonl` series instead of only raw edge-count - // gauges. Best-effort; no-op on an empty store. - crate::cognitive_memory::metrics::record_provenance_coverage_metric( - g.facts_with_provenance, - g.facts_total, - ); + match memories.memory.graph_stats() { + Ok(g) => { + telemetry::gauge_set( + names::MEMORY_EDGES, + g.derives_from_edges as i64, + &[(names::ATTR_TYPE, "DERIVES_FROM")], + ); + telemetry::gauge_set( + names::MEMORY_EDGES, + g.similar_to_edges as i64, + &[(names::ATTR_TYPE, "SIMILAR_TO")], + ); + telemetry::gauge_set( + names::MEMORY_EDGES, + g.supersedes_edges as i64, + &[(names::ATTR_TYPE, "SUPERSEDES")], + ); + // Emit both durable graph-memory self-metrics from + // this same snapshot, with no additional store read. + record_graph_memory_self_metrics( + &g, + crate::cognitive_memory::metrics::record_goal_board_snapshot_dedup_ratio_metric, + ); + } + Err(error) => { + tracing::warn!( + target: "simard::memory", + error = %error, + "failed to read graph statistics; edge gauges and graph-memory self-metrics were not recorded", + ); + } } // Flush the metrics snapshot with the per-cycle enrichment // rollup section attached (issue #2942) so the dashboard's @@ -2409,6 +2423,39 @@ mod tests { use crate::rpc_transport::InMemoryRpcTransport; use serde_json::json; + #[test] + fn graph_memory_metric_sweep_uses_goal_board_graph_stats_fields() { + let stats = crate::memory_cognitive::GraphStats { + distinct_snapshot_caller_keys: 3, + snapshot_facts_total: 12, + ..Default::default() + }; + + let mut recorded = None; + record_graph_memory_self_metrics(&stats, |distinct_caller_keys, snapshot_facts_total| { + crate::cognitive_memory::metrics::record_goal_board_snapshot_dedup_ratio_metric_with( + distinct_caller_keys, + snapshot_facts_total, + |name, value, context| { + recorded = Some((name.to_string(), value, context.to_string())); + Ok::<(), ()>(()) + }, + ) + .expect("construct goal-board metric entry through injected writer"); + }); + + let (name, value, context) = recorded.expect("one goal-board metric entry"); + assert_eq!( + name, + crate::cognitive_memory::metrics::GOAL_BOARD_SNAPSHOT_DEDUP_RATIO_METRIC + ); + assert_eq!(value, 0.25); + let context: serde_json::Value = + serde_json::from_str(&context).expect("metric context JSON"); + assert_eq!(context["snapshot_facts"], 12); + assert_eq!(context["distinct_caller_keys"], 3); + } + fn mock_memory() -> Box { Box::new(CognitiveMemoryClient::new(Box::new( InMemoryRpcTransport::new("test-daemon-shutdown", |method, _params| match method { diff --git a/tests/bin_simard_memory_cli.rs b/tests/bin_simard_memory_cli.rs index a195523bc..e86f6c0eb 100644 --- a/tests/bin_simard_memory_cli.rs +++ b/tests/bin_simard_memory_cli.rs @@ -252,6 +252,23 @@ fn stats_shows_edges_and_dedup_section_via_direct_open() { Some(1), "the snapshot caller key must be grouped: {report}" ); + // The durable `goal_board_snapshot_dedup_ratio` self-metric uses these + // operator-visible goal-board counts. One stream holding one revision is a + // healthy liveness of 1.0. + let snapshot_facts = report["snapshot_dedup"]["snapshot_facts"] + .as_u64() + .expect("snapshot_facts must be numeric"); + let distinct_caller_keys = report["snapshot_dedup"]["distinct_caller_keys"] + .as_u64() + .expect("distinct_caller_keys must be numeric"); + assert_eq!( + simard::cognitive_memory::metrics::goal_board_snapshot_dedup_ratio( + distinct_caller_keys, + snapshot_facts, + ), + Some(1.0), + "goal_board_snapshot_dedup_ratio must derive from the operator-visible counts: {report}" + ); assert!( report.get("edges_note").is_none(), "direct open must compute the edges, not note them: {report}"