Skip to content

docs(0056): alarm fire-tests passed + task complete/archived #99

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
karczuRF merged 2 commits into from
Jul 8, 2026
Merged

docs(0056): alarm fire-tests passed + task complete/archived #99

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
91e0297
docs(lore-0056): record freshness fire-test, restructure task to dir
karczuRF Jul 8, 2026
8fbed68
docs(lore-0056): complete + archive — both alarm fire-tests passed
karczuRF Jul 8, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
56 changes: 43 additions & 13 deletions ...ch-alarms-push-freshness-mtls-notafter.md → ...ms-push-freshness-mtls-notafter/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
id: "0056"
title: "CloudWatch alarms — sdex.last_push_at freshness + mTLS cert NotAfter"
type: FEATURE
status: active
status: completed
related_adr: ["0005", "0007"]
related_tasks: ["0011", "0050", "0051", "0055", "0028", "0026"]
tags: [layer-infra, priority-medium, effort-small, milestone-M1, observability, cloudwatch, alarms, sns, mtls, backfill]
Expand Down Expand Up @@ -94,6 +94,24 @@ history:
describe-slack-channel-configurations --region us-east-2` (Chatbot API is
us-east-2-only). Task stays active: the freshness + mTLS fire-tests +
notes/G-alarm-fire-test.md remain operational.
- date: 2026-07-08
status: completed
who: okarcz
note: >
DONE. All operational fire-tests passed against the real metric path and
the last ACs closed. Finding A confirmed live (EventBridge diff shows no
drift → deployed probe IS the merged fix; runs clean every 15 min, no
Nullable(Int64) deser crash, publishes only for running+pushed streams,
alarm OK). Freshness fire-test: temporarily lowered sdexPushFreshnessSeconds
604800->3600, alarm breached on live PushAgeSeconds 58211>3600 -> Slack
12:09:28 UTC, restored -> recovery. mTLS fire-test: raised
mtlsNotAfterDaysThreshold 30->400 + manual probe invoke, alarm breached on
live MinDaysToNotAfter 350<400 -> Slack 12:27:09 UTC, restored -> recovery.
Both threshold edits reverted, git clean. Artifacts in
notes/G-alarm-fire-test.md. All 8 alarms (2 probes + 4 ledger-processor +
enrichment-backlog + mtls) route to #stellar-prices-api-bot with both
addAlarmAction + addOkAction. Fire-test artifacts committed on branch
docs/0056-alarm-fire-test-artifacts.
---

# CloudWatch alarms — SDEX push freshness + mTLS NotAfter
Expand Down Expand Up @@ -289,28 +307,40 @@ threshold trips. Restore the canonical cert post-test.
the topic + both alarms + `SnsAction`; also back-wired the previously
action-less enrichment-backlog alarm. Synth asserts topic + both alarms +
metrics.*
- [ ] **(operational)** An ops address/alias is subscribed to
`prices-{env}-ops-alarms` and confirmed (not `PendingConfirmation`). *Topic
ships subscriber-less by design; see Step 3.5 for the one-time per-env
subscribe checklist. Prerequisite for the fire-test below to deliver.*
- [ ] **(operational)** Manual fire-test for the freshness alarm produces an
SNS delivery to the configured operator email (Tranche-1 AC #5). *Requires
a real deploy + a skipped push cycle + a subscribed address (Step 3.5).*
- [x] **(operational)** The ops topic has a confirmed subscriber. *Delivery is
**Slack via AWS Chatbot** to prices' own channel `#stellar-prices-api-bot`
(workspace `T83HLEDJN`, channel `C0BFWLMFQ9G`), not email — deployed +
smoke-tested 2026-07-08 (forced `ledger-processor-errors` ALARM→OK posted
both a breach and a recovery). See Step 3.5. Email subscribe is the
documented fallback if Slack is ever dropped.*
- [x] **(operational)** Manual fire-test for the freshness alarm produces a Slack
delivery (Tranche-1 AC #5). *2026-07-08: temporarily lowered
`sdexPushFreshnessSeconds` `604800→3600` and deployed, so the alarm breached
on the **real** `PushAgeSeconds` datapoint (`58211s @ 11:39 > 3600`) → 🚨 to
`#stellar-prices-api-bot` at `12:09:28 UTC`; restored `604800` → ✅ recovery.
Threshold reverted, `git status` clean. Artifacts in
[notes/G-alarm-fire-test.md](notes/G-alarm-fire-test.md).*
- [x] Threshold for the freshness alarm is operator-tunable.
*`config.opsAlarms.sdexPushFreshnessSeconds` (default 604800) +
`mtlsNotAfterDaysThreshold` (30); validated in `validateConfig`. Per-env
JSON, no code change.*
- [ ] **(operational)** `notes/G-alarm-fire-test.md` records the fire-test
timestamps + SNS message IDs for both alarms. *Produced during the deploy
fire-test above.*
- [x] **(operational)** `notes/G-alarm-fire-test.md` records the fire-test
timestamps for both alarms. *Both done 2026-07-08, real-metric path: freshness
(`PushAgeSeconds 58211>3600` → 🚨 `12:09:28`, restore → ✅) and mTLS NotAfter
(`MinDaysToNotAfter 350<400` → 🚨 `12:27:09`, restore → ✅), each breach +
recovery to `#stellar-prices-api-bot`. Slack cards don't surface the raw SNS
`MessageId` — note documents how to capture one if the AC strictly requires it.*
- [x] **(from 0082, finding A)** `sdex-push-freshness` no longer false-fires in a
live-only deployment. *Freshness probe `AGE_QUERY` now gates on
`status='running' AND last_push_at IS NOT NULL` — live-only seed rows
(running, NULL `last_push_at`) and the paused/completed seams publish no
metric, so the `NOT_BREACHING` alarm stays OK; a running backfill that has
pushed and then stalls still fires (AC #5). Supersedes the finding-#4
`coalesce(…, started_at)` fallback. 4 unit tests. **Deploy-confirm** the
alarm sits OK in live-only is operational.*
`coalesce(…, started_at)` fallback. 4 unit tests. **Deploy-confirmed
2026-07-08:** EventBridge `diff` shows no drift from the merged fix (the
deployed probe IS the finding-A build), the probe runs cleanly every 15 min
(no `Nullable(Int64)` deser crash), publishes `PushAgeSeconds` only for the
2 running+pushed streams, and the alarm reads OK — no false-fire.*
- [x] **(from 0082, finding B)** Ledger-processor lag + error alarms created.
*Three alarms in `ObservabilityStack`, all → `prices-{env}-ops-alarms`,
built from AWS-native metrics by deterministic name (no processor
Expand Down
102 changes: 102 additions & 0 deletions ...ATURE_cloudwatch-alarms-push-freshness-mtls-notafter/notes/G-alarm-fire-test.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,102 @@
---
prefix: G
title: Alarm fire-test artifacts (freshness + mTLS NotAfter)
status: mature
spawned_from: "0056"
---

# G — Alarm fire-test artifacts

Deploy-gated acceptance evidence for task 0056: each ops alarm, forced to
breach against real production data (or a controlled state), delivered a
notification to the ops channel `#stellar-prices-api-bot` via SNS → AWS
Chatbot → Slack. Timestamps are UTC.

Environment: `production` · region `eu-central-1` · account `750702271865` ·
ops topic `arn:aws:sns:eu-central-1:750702271865:prices-production-ops-alarms` ·
Slack channel `#stellar-prices-api-bot` (`C0BFWLMFQ9G`, workspace `T83HLEDJN`).

All alarms carry both `addAlarmAction` + `addOkAction` → the ops topic, so
each fire-test shows a breach **and** a recovery.

---

## 1. SDEX push-freshness — `prices-production-sdex-push-freshness` (Tranche-1 AC #5)

**Method (authentic, real-metric path).** Rather than wait 7 days for a real
skipped push, the operator-tunable threshold was temporarily lowered below the
live metric value, so the alarm breached on the **actual** published
`Prices/Backfill PushAgeSeconds` datapoint (not a forced `set-alarm-state`).
At test time both streams were `running` + pushed, with `last_push_at` frozen
~16 h prior (age ~58,211 s and climbing in lockstep with wall-clock), so the
metric was genuinely flowing.

- Temp change: `config.opsAlarms.sdexPushFreshnessSeconds` `604800 → 3600`
(uncommitted working-tree edit), `make deploy-production-observability`.
- Metric: `Prices/Backfill` / `PushAgeSeconds`, dim `Stream=sdex_archive`,
`Environment=production`.

| Transition | Datapoint (age_s @ time) | Threshold | Alarm state | Slack post (UTC) |
|---|---|---|---|---|
| 🚨 OK → ALARM | `58211.0 @ 2026-07-08 11:39` | `3600` | ALARM | **2026-07-08 12:09:28** |
| ✅ ALARM → OK | `58211.0 @ 2026-07-08 11:45` | `604800` (restored) | OK | 2026-07-08 ~12:1x |

Slack breach text (verbatim): _"Threshold Crossed: 1 out of the last 1
datapoints [58211.0 (08/07/26 11:39:00)] was greater than the threshold
(3600.0)…"_ with the alarm description _"sdex_archive.last_push_at has aged
past the Tranche-1 freshness threshold…"_.

- Threshold restored to `604800` + redeployed; `git status` clean afterwards
(no residual config drift). Alarm returned to OK and posted the recovery.
- **Result: PASS.** Real-metric breach + recovery both delivered to Slack.

> SNS message IDs: delivery confirmed visually in Slack (Chatbot does not
> surface the raw SNS `MessageId` in the channel card). If a raw `MessageId`
> is required for the AC, capture it from an `--protocol email`/SQS test
> subscription on the ops topic, or from CloudTrail `Publish` events.

---

## 2. mTLS NotAfter — `prices-production-mtls-notafter`

**Method (authentic, real-metric path).** Instead of issuing a short-lived test
cert (which touches secret material), the threshold was temporarily *raised
above* the live `MinDaysToNotAfter` value so the alarm breached on the **real**
cert metric. The `mtls-notafter-probe` runs daily (`rate(1 day)`), so after each
threshold change the probe was invoked manually to drop a fresh datapoint into
the alarm's current 1-day evaluation period (else `NOT_BREACHING` would hold it).

- Temp change: `config.opsAlarms.mtlsNotAfterDaysThreshold` `30 → 400`
(uncommitted), `make deploy-production-observability`, then
`aws lambda invoke --function-name prices-production-mtls-notafter-probe`.
- Metric: `Prices/Mtls` / `MinDaysToNotAfter`, dim `Environment=production`
(aggregate min across the `ingestion` + `api` cert bundles).
- Live cert state at test time: both roles ~**350 days** to NotAfter
(`ingestion 350.053`, `api 350.053` — healthy, ~11.5 months of runway).

| Transition | Datapoint (min_days @ time) | Threshold | Alarm state | Slack post (UTC) |
|---|---|---|---|---|
| 🚨 OK → ALARM | `350.053 @ 2026-07-07 12:27` | `400` | ALARM | **2026-07-08 12:27:09** |
| ✅ ALARM → OK | `350.051 @ 2026-07-07 12:30` | `30` (restored) | OK | 2026-07-08 ~12:3x |

Slack breach text (verbatim): _"Threshold Crossed: 1 out of the last 1 datapoints
[350.05315972222223 (07/07/26 12:27:00)] was less than the threshold (400.0)…"_
with the alarm description _"An mTLS client cert is within the Tranche-1 expiry
window…"_.

- Threshold restored to `30` + redeployed + probe re-invoked; `git status` clean
afterwards (no residual config drift). Alarm returned to OK (`350 > 30`) and
posted the recovery.
- **Result: PASS.** Real-metric breach + recovery both delivered to Slack.

> Same `MessageId` caveat as §1 — Slack cards don't surface the raw SNS ID.

---

## Related smoke test (Slack routing, not a fire-test)

Before the freshness test, the SNS → Chatbot → Slack path itself was proven by
forcing `prices-production-ledger-processor-errors` ALARM→OK via
`set-alarm-state`: both a 🚨 breach and ✅ recovery landed in
`#stellar-prices-api-bot` (2026-07-08). This validated the routing + the
`addOkAction` wiring independently of any real metric.
Loading