diff --git a/lore/1-tasks/active/0056_FEATURE_cloudwatch-alarms-push-freshness-mtls-notafter.md b/lore/1-tasks/archive/0056_FEATURE_cloudwatch-alarms-push-freshness-mtls-notafter/README.md similarity index 92% rename from lore/1-tasks/active/0056_FEATURE_cloudwatch-alarms-push-freshness-mtls-notafter.md rename to lore/1-tasks/archive/0056_FEATURE_cloudwatch-alarms-push-freshness-mtls-notafter/README.md index 6622c51..cfa90b9 100644 --- a/lore/1-tasks/active/0056_FEATURE_cloudwatch-alarms-push-freshness-mtls-notafter.md +++ b/lore/1-tasks/archive/0056_FEATURE_cloudwatch-alarms-push-freshness-mtls-notafter/README.md @@ -2,7 +2,7 @@ id: "0056" title: "CloudWatch alarms — sdex.last_push_at freshness + mTLS cert NotAfter" type: FEATURE -status: active +status: completed related_adr: ["0005", "0007"] related_tasks: ["0011", "0050", "0051", "0055", "0028", "0026"] tags: [layer-infra, priority-medium, effort-small, milestone-M1, observability, cloudwatch, alarms, sns, mtls, backfill] @@ -94,6 +94,24 @@ history: describe-slack-channel-configurations --region us-east-2` (Chatbot API is us-east-2-only). Task stays active: the freshness + mTLS fire-tests + notes/G-alarm-fire-test.md remain operational. + - date: 2026-07-08 + status: completed + who: okarcz + note: > + DONE. All operational fire-tests passed against the real metric path and + the last ACs closed. Finding A confirmed live (EventBridge diff shows no + drift → deployed probe IS the merged fix; runs clean every 15 min, no + Nullable(Int64) deser crash, publishes only for running+pushed streams, + alarm OK). Freshness fire-test: temporarily lowered sdexPushFreshnessSeconds + 604800->3600, alarm breached on live PushAgeSeconds 58211>3600 -> Slack + 12:09:28 UTC, restored -> recovery. mTLS fire-test: raised + mtlsNotAfterDaysThreshold 30->400 + manual probe invoke, alarm breached on + live MinDaysToNotAfter 350<400 -> Slack 12:27:09 UTC, restored -> recovery. + Both threshold edits reverted, git clean. Artifacts in + notes/G-alarm-fire-test.md. All 8 alarms (2 probes + 4 ledger-processor + + enrichment-backlog + mtls) route to #stellar-prices-api-bot with both + addAlarmAction + addOkAction. Fire-test artifacts committed on branch + docs/0056-alarm-fire-test-artifacts. --- # CloudWatch alarms — SDEX push freshness + mTLS NotAfter @@ -289,28 +307,40 @@ threshold trips. Restore the canonical cert post-test. the topic + both alarms + `SnsAction`; also back-wired the previously action-less enrichment-backlog alarm. Synth asserts topic + both alarms + metrics.* -- [ ] **(operational)** An ops address/alias is subscribed to - `prices-{env}-ops-alarms` and confirmed (not `PendingConfirmation`). *Topic - ships subscriber-less by design; see Step 3.5 for the one-time per-env - subscribe checklist. Prerequisite for the fire-test below to deliver.* -- [ ] **(operational)** Manual fire-test for the freshness alarm produces an - SNS delivery to the configured operator email (Tranche-1 AC #5). *Requires - a real deploy + a skipped push cycle + a subscribed address (Step 3.5).* +- [x] **(operational)** The ops topic has a confirmed subscriber. *Delivery is + **Slack via AWS Chatbot** to prices' own channel `#stellar-prices-api-bot` + (workspace `T83HLEDJN`, channel `C0BFWLMFQ9G`), not email — deployed + + smoke-tested 2026-07-08 (forced `ledger-processor-errors` ALARM→OK posted + both a breach and a recovery). See Step 3.5. Email subscribe is the + documented fallback if Slack is ever dropped.* +- [x] **(operational)** Manual fire-test for the freshness alarm produces a Slack + delivery (Tranche-1 AC #5). *2026-07-08: temporarily lowered + `sdexPushFreshnessSeconds` `604800→3600` and deployed, so the alarm breached + on the **real** `PushAgeSeconds` datapoint (`58211s @ 11:39 > 3600`) → 🚨 to + `#stellar-prices-api-bot` at `12:09:28 UTC`; restored `604800` → ✅ recovery. + Threshold reverted, `git status` clean. Artifacts in + [notes/G-alarm-fire-test.md](notes/G-alarm-fire-test.md).* - [x] Threshold for the freshness alarm is operator-tunable. *`config.opsAlarms.sdexPushFreshnessSeconds` (default 604800) + `mtlsNotAfterDaysThreshold` (30); validated in `validateConfig`. Per-env JSON, no code change.* -- [ ] **(operational)** `notes/G-alarm-fire-test.md` records the fire-test - timestamps + SNS message IDs for both alarms. *Produced during the deploy - fire-test above.* +- [x] **(operational)** `notes/G-alarm-fire-test.md` records the fire-test + timestamps for both alarms. *Both done 2026-07-08, real-metric path: freshness + (`PushAgeSeconds 58211>3600` → 🚨 `12:09:28`, restore → ✅) and mTLS NotAfter + (`MinDaysToNotAfter 350<400` → 🚨 `12:27:09`, restore → ✅), each breach + + recovery to `#stellar-prices-api-bot`. Slack cards don't surface the raw SNS + `MessageId` — note documents how to capture one if the AC strictly requires it.* - [x] **(from 0082, finding A)** `sdex-push-freshness` no longer false-fires in a live-only deployment. *Freshness probe `AGE_QUERY` now gates on `status='running' AND last_push_at IS NOT NULL` — live-only seed rows (running, NULL `last_push_at`) and the paused/completed seams publish no metric, so the `NOT_BREACHING` alarm stays OK; a running backfill that has pushed and then stalls still fires (AC #5). Supersedes the finding-#4 - `coalesce(…, started_at)` fallback. 4 unit tests. **Deploy-confirm** the - alarm sits OK in live-only is operational.* + `coalesce(…, started_at)` fallback. 4 unit tests. **Deploy-confirmed + 2026-07-08:** EventBridge `diff` shows no drift from the merged fix (the + deployed probe IS the finding-A build), the probe runs cleanly every 15 min + (no `Nullable(Int64)` deser crash), publishes `PushAgeSeconds` only for the + 2 running+pushed streams, and the alarm reads OK — no false-fire.* - [x] **(from 0082, finding B)** Ledger-processor lag + error alarms created. *Three alarms in `ObservabilityStack`, all → `prices-{env}-ops-alarms`, built from AWS-native metrics by deterministic name (no processor diff --git a/lore/1-tasks/archive/0056_FEATURE_cloudwatch-alarms-push-freshness-mtls-notafter/notes/G-alarm-fire-test.md b/lore/1-tasks/archive/0056_FEATURE_cloudwatch-alarms-push-freshness-mtls-notafter/notes/G-alarm-fire-test.md new file mode 100644 index 0000000..b6d3b82 --- /dev/null +++ b/lore/1-tasks/archive/0056_FEATURE_cloudwatch-alarms-push-freshness-mtls-notafter/notes/G-alarm-fire-test.md @@ -0,0 +1,102 @@ +--- +prefix: G +title: Alarm fire-test artifacts (freshness + mTLS NotAfter) +status: mature +spawned_from: "0056" +--- + +# G — Alarm fire-test artifacts + +Deploy-gated acceptance evidence for task 0056: each ops alarm, forced to +breach against real production data (or a controlled state), delivered a +notification to the ops channel `#stellar-prices-api-bot` via SNS → AWS +Chatbot → Slack. Timestamps are UTC. + +Environment: `production` · region `eu-central-1` · account `750702271865` · +ops topic `arn:aws:sns:eu-central-1:750702271865:prices-production-ops-alarms` · +Slack channel `#stellar-prices-api-bot` (`C0BFWLMFQ9G`, workspace `T83HLEDJN`). + +All alarms carry both `addAlarmAction` + `addOkAction` → the ops topic, so +each fire-test shows a breach **and** a recovery. + +--- + +## 1. SDEX push-freshness — `prices-production-sdex-push-freshness` (Tranche-1 AC #5) + +**Method (authentic, real-metric path).** Rather than wait 7 days for a real +skipped push, the operator-tunable threshold was temporarily lowered below the +live metric value, so the alarm breached on the **actual** published +`Prices/Backfill PushAgeSeconds` datapoint (not a forced `set-alarm-state`). +At test time both streams were `running` + pushed, with `last_push_at` frozen +~16 h prior (age ~58,211 s and climbing in lockstep with wall-clock), so the +metric was genuinely flowing. + +- Temp change: `config.opsAlarms.sdexPushFreshnessSeconds` `604800 → 3600` + (uncommitted working-tree edit), `make deploy-production-observability`. +- Metric: `Prices/Backfill` / `PushAgeSeconds`, dim `Stream=sdex_archive`, + `Environment=production`. + +| Transition | Datapoint (age_s @ time) | Threshold | Alarm state | Slack post (UTC) | +|---|---|---|---|---| +| 🚨 OK → ALARM | `58211.0 @ 2026-07-08 11:39` | `3600` | ALARM | **2026-07-08 12:09:28** | +| ✅ ALARM → OK | `58211.0 @ 2026-07-08 11:45` | `604800` (restored) | OK | 2026-07-08 ~12:1x | + +Slack breach text (verbatim): _"Threshold Crossed: 1 out of the last 1 +datapoints [58211.0 (08/07/26 11:39:00)] was greater than the threshold +(3600.0)…"_ with the alarm description _"sdex_archive.last_push_at has aged +past the Tranche-1 freshness threshold…"_. + +- Threshold restored to `604800` + redeployed; `git status` clean afterwards + (no residual config drift). Alarm returned to OK and posted the recovery. +- **Result: PASS.** Real-metric breach + recovery both delivered to Slack. + +> SNS message IDs: delivery confirmed visually in Slack (Chatbot does not +> surface the raw SNS `MessageId` in the channel card). If a raw `MessageId` +> is required for the AC, capture it from an `--protocol email`/SQS test +> subscription on the ops topic, or from CloudTrail `Publish` events. + +--- + +## 2. mTLS NotAfter — `prices-production-mtls-notafter` + +**Method (authentic, real-metric path).** Instead of issuing a short-lived test +cert (which touches secret material), the threshold was temporarily *raised +above* the live `MinDaysToNotAfter` value so the alarm breached on the **real** +cert metric. The `mtls-notafter-probe` runs daily (`rate(1 day)`), so after each +threshold change the probe was invoked manually to drop a fresh datapoint into +the alarm's current 1-day evaluation period (else `NOT_BREACHING` would hold it). + +- Temp change: `config.opsAlarms.mtlsNotAfterDaysThreshold` `30 → 400` + (uncommitted), `make deploy-production-observability`, then + `aws lambda invoke --function-name prices-production-mtls-notafter-probe`. +- Metric: `Prices/Mtls` / `MinDaysToNotAfter`, dim `Environment=production` + (aggregate min across the `ingestion` + `api` cert bundles). +- Live cert state at test time: both roles ~**350 days** to NotAfter + (`ingestion 350.053`, `api 350.053` — healthy, ~11.5 months of runway). + +| Transition | Datapoint (min_days @ time) | Threshold | Alarm state | Slack post (UTC) | +|---|---|---|---|---| +| 🚨 OK → ALARM | `350.053 @ 2026-07-07 12:27` | `400` | ALARM | **2026-07-08 12:27:09** | +| ✅ ALARM → OK | `350.051 @ 2026-07-07 12:30` | `30` (restored) | OK | 2026-07-08 ~12:3x | + +Slack breach text (verbatim): _"Threshold Crossed: 1 out of the last 1 datapoints +[350.05315972222223 (07/07/26 12:27:00)] was less than the threshold (400.0)…"_ +with the alarm description _"An mTLS client cert is within the Tranche-1 expiry +window…"_. + +- Threshold restored to `30` + redeployed + probe re-invoked; `git status` clean + afterwards (no residual config drift). Alarm returned to OK (`350 > 30`) and + posted the recovery. +- **Result: PASS.** Real-metric breach + recovery both delivered to Slack. + +> Same `MessageId` caveat as §1 — Slack cards don't surface the raw SNS ID. + +--- + +## Related smoke test (Slack routing, not a fire-test) + +Before the freshness test, the SNS → Chatbot → Slack path itself was proven by +forcing `prices-production-ledger-processor-errors` ALARM→OK via +`set-alarm-state`: both a 🚨 breach and ✅ recovery landed in +`#stellar-prices-api-bot` (2026-07-08). This validated the routing + the +`addOkAction` wiring independently of any real metric.