Hand recurring-step jobs back on shutdown instead of dying with them active#281
Merged
Merged
Conversation
…active A recurring pass interrupted by SIGTERM outlived the drain window and the process exited with its pg-boss job still ACTIVE. The stately singleton then blocked every process's chain for that step until job expiry — up to tens of minutes for the long-lease steps (agent chats, digests) after a routine deploy. The old tick loop had no durable blocker: a new process resumed the step immediately. Thread an abort signal from the worker's shutdown path into every chain: when draining begins, an in-flight pass is handed back (the handler resolves, pg-boss completes the job, the successor reschedules) and no new pass starts. The abandoned pass keeps running only until the process exits, so the residual overlap window is bounded by the drain timeout — the same window the tick loop had on deploys. The background registration retry also stops on a draining process.
Contributor
Author
|
@codex review |
|
Codex Review: Didn't find any major issues. More of your lovely PRs please. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #241.
A recurring pass interrupted by SIGTERM could outlive the drain window, so the process exited with its pg-boss job still active. The stately singleton then blocked that step's chain fleet-wide until job expiry — up to 20–30 minutes for the long-lease steps (agent chats, digests) after a routine deploy. The old tick loop had no durable blocker: a replacement process resumed the step immediately.
Fix: thread an
AbortSignalfrom the worker's shutdown path into every chain. When draining begins:startAfter;The abandoned pass keeps running only until the process exits, so the residual overlap window is bounded by the drain timeout (~2 min) — the same window the tick loop had on deploys, and far better than a 20–30 minute chain pause.
Testing
test:recurring(8): two new tests — shutdown hands a hung pass back and still reschedules; a draining process starts no new pass but keeps the chain alive.test:recurring-steps(4): new test — registration retries stop once shutdown begins.test:worker-tick(1),pnpm --filter @superlog/worker typecheck, biome all green.Summary by cubic
Prevents fleet-wide stalls by handing recurring-step jobs back on shutdown so
pg-bossjobs don’t stay ACTIVE. Draining processes stop starting new passes and stop registration retries, reducing deploy pauses to the drain window (~2 min).Bug Fixes
pg-bosscompletes the job and reschedules the successor.Migration
startRecurringStepsandregisterRecurringStepnow take an options object:{ logger?, retryDelayMs?, shutdown? }.Written for commit 92bbd6b. Summary will update on new commits.