Skip to content

Hand recurring-step jobs back on shutdown instead of dying with them active#281

Merged
arseniycodes merged 1 commit into
mainfrom
ash/recurring-shutdown-handoff
Jul 15, 2026
Merged

Hand recurring-step jobs back on shutdown instead of dying with them active#281
arseniycodes merged 1 commit into
mainfrom
ash/recurring-shutdown-handoff

Conversation

@arseniycodes

@arseniycodes arseniycodes commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

Follow-up to #241.

A recurring pass interrupted by SIGTERM could outlive the drain window, so the process exited with its pg-boss job still active. The stately singleton then blocked that step's chain fleet-wide until job expiry — up to 20–30 minutes for the long-lease steps (agent chats, digests) after a routine deploy. The old tick loop had no durable blocker: a replacement process resumed the step immediately.

Fix: thread an AbortSignal from the worker's shutdown path into every chain. When draining begins:

  • an in-flight pass is handed back — the handler resolves, pg-boss completes the job, and the successor reschedules normally with startAfter;
  • no new pass starts on the draining process (the job still completes and reschedules, so the successor runs on whichever process survives);
  • the background registration retry stops.

The abandoned pass keeps running only until the process exits, so the residual overlap window is bounded by the drain timeout (~2 min) — the same window the tick loop had on deploys, and far better than a 20–30 minute chain pause.

Testing

  • test:recurring (8): two new tests — shutdown hands a hung pass back and still reschedules; a draining process starts no new pass but keeps the chain alive.
  • test:recurring-steps (4): new test — registration retries stop once shutdown begins.
  • test:worker-tick (1), pnpm --filter @superlog/worker typecheck, biome all green.

Summary by cubic

Prevents fleet-wide stalls by handing recurring-step jobs back on shutdown so pg-boss jobs don’t stay ACTIVE. Draining processes stop starting new passes and stop registration retries, reducing deploy pauses to the drain window (~2 min).

  • Bug Fixes

    • Threaded an AbortSignal from worker shutdown into all recurring chains.
    • On drain, handlers resolve so pg-boss completes the job and reschedules the successor.
    • Draining processes don’t start new passes.
    • Registration retries stop once shutdown begins.
  • Migration

    • startRecurringSteps and registerRecurringStep now take an options object: { logger?, retryDelayMs?, shutdown? }.

Written for commit 92bbd6b. Summary will update on new commits.

Review in cubic

…active

A recurring pass interrupted by SIGTERM outlived the drain window and
the process exited with its pg-boss job still ACTIVE. The stately
singleton then blocked every process's chain for that step until job
expiry — up to tens of minutes for the long-lease steps (agent chats,
digests) after a routine deploy. The old tick loop had no durable
blocker: a new process resumed the step immediately.

Thread an abort signal from the worker's shutdown path into every
chain: when draining begins, an in-flight pass is handed back (the
handler resolves, pg-boss completes the job, the successor reschedules)
and no new pass starts. The abandoned pass keeps running only until the
process exits, so the residual overlap window is bounded by the drain
timeout — the same window the tick loop had on deploys. The background
registration retry also stops on a draining process.
@arseniycodes

Copy link
Copy Markdown
Contributor Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. More of your lovely PRs please.

Reviewed commit: 92bbd6ba29

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@arseniycodes
arseniycodes merged commit 477bbaf into main Jul 15, 2026
7 checks passed
@arseniycodes
arseniycodes deleted the ash/recurring-shutdown-handoff branch July 15, 2026 13:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant