fix(cron): persist settled failure-alert delivery outcome to job state … - #3
Open
warren-run-bot wants to merge 8 commits into
Open
fix(cron): persist settled failure-alert delivery outcome to job state …#3warren-run-bot wants to merge 8 commits into
warren-run-bot wants to merge 8 commits into
Conversation
…e and run history
… failure, fallback)
…run-history settlement
Fixes the P1 reviewer finding: with cooldownMs=0, two overlapping failure
alerts for the same job could both read "unknown" in the job-state and
run-history. The first alert's completion callback would scan for any
outstanding unknown row, selecting the NEWER run instead of its own.
Changes:
- tryRecordCronFailureNotificationDeliveryOutcome: when taskRunId is
provided, look up the exact row by runId instead of scanning for any
unknown-status row for the job. The scan path remains as a fallback for
callers without a taskRunId.
- recordFailureNotificationDeliveryOutcome: accepts an origin object
{taskRunId, alertAtMs}. The job-state mutate guard now also checks that
lastFailureAlertAtMs matches the originating alert's timestamp, so a
newer alert that has overwritten job state is not accidentally settled
by the older alert's callback.
- transportFailureAlert: captures taskRunId and alertAtMs at dispatch time
and forwards them to recordFailureNotificationDeliveryOutcome.
- emitFailureAlert / maybeEmitFailureAlert / maybeEmitDeliveryFailureAlert
/ finalizeCronFailureNotifications: all accept taskRunId and thread it
through to the transport.
- applyJobResult opts: adds taskRunId?; callers in
applyOutcomeToAuthoritativeJob and applyOutcomeToStoredJob forward
result.taskRunId. The skipped-alert maybeEmitFailureAlert call also
receives it.
- startup-run-repair: passes params.taskRunId to
finalizeCronFailureNotifications.
…n run-history row Two overlapping eligible alerts for one job settling out of order: - Each outcome lands on its own run-history row (identified by runAtMs). - Job state reflects the most recently started alert (B), not the older one (A) that happened to settle last. failure-alerts.persistence.test.ts: - Adds taskRunId? to finalizeAlertOutcome to propagate the stable run identity into TimedCronRunOutcome. - Adds 'settles each overlapping alert on its own run-history row in reverse settlement order': uses pre-generated taskRunIds, deferred transport resolvers, and per-runAtMs row lookup to assert exact settlement isolation. service.persists-delivered-status.test.ts: - Adds 'lands each overlapping alert's outcome on its own run-history row in reverse settlement order': exercises the same invariant through the full CronService API with two sequential error runs and cooldownMs=0, verifying job state is not clobbered by the later- settling A callback.
…ettlement identity; redact transport errors Finding 1: applyManualRunOutcome now passes prepared.taskRunId into applyJobResult so the alert callback targets the exact originating job-state slot and run-history row instead of falling back to the newest-unknown scan. Finding 2: transport catch handler applies redactSensitiveText(String(err)) before truncateUtf16Safe so credentials embedded in a channel rejection are never stored in job state or surfaced over the cron protocol. Finding 3: lastFailureAlertTaskRunId (persisted via state_json) replaces lastFailureAlertAtMs as the job-state ownership token. Both the DB-write guard and the in-memory resident update now compare origin.taskRunId to lastFailureAlertTaskRunId (strict equality; both-undefined path preserves backward compat). The newest-unknown fallback scan in tryRecordCronFailureNotificationDeliveryOutcome is deleted; taskRunId is the single identity mechanism.
… redaction failure-alerts.persistence.test.ts: - Add taskRunId to existing 'settled success/failure outcome' tests so their run-history rows update correctly after the fallback scan was removed. - New: 'two alerts at the same wall-clock millisecond settling correctly': proves taskRunId-based guard resolves the timestamp-collision bug (Finding 3). - New: 'redacts credentials in a transport rejection before persisting the error': registers a secret, asserts it is absent from persisted error (Finding 2); confirms the stored text is still informative (non-secret parts intact). service.persists-delivered-status.test.ts: - New: 'overlapping manual runs settling forward with distinct outcomes': verifies manual cron.run() calls thread taskRunId through applyManualRunOutcome into applyJobResult (Finding 1); proves forward-order settlement correctly routes each alert to its own run-state slot, and job state reflects the newest run's outcome once that alert settles.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
fix(cron): persist settled failure-alert delivery outcome to job state and run history
Run
run_ywv0dx8tb3scCommits (4)
Files changed
Prompt
Show prompt
🤖 Opened by warren run
run_ywv0dx8tb3sc