Skip to content

Revive only dead recurring chains#282

Merged
arseniycodes merged 1 commit into
mainfrom
ash/recurring-reviver-dead-only
Jul 15, 2026
Merged

Revive only dead recurring chains#282
arseniycodes merged 1 commit into
mainfrom
ash/recurring-reviver-dead-only

Conversation

@arseniycodes

@arseniycodes arseniycodes commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

Follow-up to #241/#281.

The per-queue minute reviver cron seeded unconditionally. During a legitimately long pass (agent chats and digests may run 10–15 minutes before their warn deadline) the stately policy accepts that seed as a queued job next to the active one, where it sits for the pass's whole duration — aging past the stuck-queue alert threshold on a perfectly healthy queue — and its dedupe then swallows the successor's intended startAfter interval, so the next pass runs immediately instead of after the pause.

Fix:

  • Replace the per-queue crons with one shared recurring-reviver job (exclusive queue, minute cron) whose handler loads each chain's queued/active counts (reusing the queue-health loader) and seeds only chains with neither — a dead chain revives within a minute, a busy chain is left alone.
  • Step registration removes the legacy per-queue schedule rows (unschedule) so databases that booted an earlier revision don't keep the old unconditional seeding.
  • A counts-load failure logs and skips the round (never seed on unknown state); a reviver registration failure is non-fatal — healthy chains self-perpetuate without it.

Testing

  • test:recurring (10): reviver seeds only dead/missing chains and skips active/queued ones; counts-load failure seeds nothing; step registration creates no per-queue schedule and unschedules the legacy one.
  • test:recurring-steps (4): the shared reviver registers exactly once alongside the steps.
  • test:worker-tick, test:queue-health, worker typecheck, biome all green.

Summary by cubic

Replaced per-queue minute revivers with a single shared recurring-reviver that only seeds dead recurring chains. This prevents false stuck-queue alerts and keeps the intended pause between passes.

  • Bug Fixes
    • Added a shared exclusive recurring-reviver cron that seeds only chains with zero queued and zero active jobs (via queue-health counts).
    • Removed legacy per-queue schedules during step registration to stop unconditional minute seeds.
    • Safe failure modes: skip seeding if counts fail to load; reviver registration failure is non-fatal.
    • Tests updated to cover dead-only revival, single reviver registration, and unscheduling of legacy crons.

Written for commit a5faf5c. Summary will update on new commits.

Review in cubic

The per-queue minute cron seeded unconditionally. During a legitimately
long pass (agent chats and digests may run 10-15 minutes) the stately
policy accepts the seed as a queued job next to the active one, where it
ages for the pass's whole duration — tripping stuck-queue alerting on a
healthy queue — and its dedupe swallows the successor's intended
startAfter interval.

Replace the per-queue crons with one shared reviver job that checks each
chain's queued/active counts (via the queue-health loader) and seeds
only chains with neither. Registration also removes the legacy
per-queue schedule rows left in the database by earlier revisions.
@arseniycodes

Copy link
Copy Markdown
Contributor Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Can't wait for the next one!

Reviewed commit: a5faf5c828

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@arseniycodes arseniycodes merged commit e5f5c58 into main Jul 15, 2026
7 checks passed
@arseniycodes arseniycodes deleted the ash/recurring-reviver-dead-only branch July 15, 2026 13:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant