docs(0056): alarm fire-tests passed + task complete/archived#99
Merged
Conversation
Freshness alarm fire-test passed against the real metric: temporarily lowered sdexPushFreshnessSeconds 604800->3600, deployed, and the alarm breached on the live PushAgeSeconds datapoint (58211s > 3600) -> Slack breach, then restored the threshold -> Slack recovery. Threshold reverted, no config drift. Convert the flat task file to directory form and add notes/G-alarm-fire-test.md capturing the freshness artifacts (mTLS NotAfter section pending its fire-test). Mark the freshness fire-test + ops-subscriber (Slack) + finding-A-live ACs done.
mTLS NotAfter fire-test done (authentic, real-metric): raised mtlsNotAfterDaysThreshold 30->400 + manual probe invoke, alarm breached on live MinDaysToNotAfter 350<400 -> Slack 12:27:09 UTC, restored 30 -> recovery. Threshold reverted, git clean. All ACs satisfied — both fire-tests captured in notes/G-alarm-fire-test.md, finding A confirmed live, finding B ledger-processor alarms deployed, ops delivery via Slack. Mark task completed and move active -> archive.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes out task 0056 (CloudWatch alarms — SDEX push-freshness + mTLS NotAfter): the last operational, deploy-gated acceptance items are done, verified against production, and archived.
What landed
set-alarm-state), each delivering a breach and a recovery to#stellar-prices-api-bot:sdexPushFreshnessSeconds604800→3600; alarm breached on livePushAgeSeconds 58211 > 3600→ 🚨12:09:28 UTC, restored → ✅.mtlsNotAfterDaysThreshold30→400+ manual probe invoke; alarm breached on liveMinDaysToNotAfter 350 < 400→ 🚨12:27:09 UTC, restored → ✅.Prices-production-EventBridgediff shows no drift, so the deployed freshness probe is the merged fix; it runs clean every 15 min (noNullable(Int64)deser crash) and publishes only for running+pushed streams.notes/G-alarm-fire-test.mdcapturing both fire-tests' timestamps and datapoints.active/ → archive/.Notes
MessageId; the G-note documents how to capture one (email/SQS test sub or CloudTrail) if strictly required.last_push_atfrozen ~16h (a combined backfill pushed once then stalled) — alarm is legitimately OK at the real 604800 threshold, would fire in ~6d if the stall persists. Flagged for 0053, not a 0056 concern.