Skip to content

feat: implement keep-alive logic in catchup worker to process newly queued ranges - #108

Merged
anhthii merged 2 commits into
mainfrom
fix/catchup-keepalive-drained-ranges
Aug 5, 2026
Merged

feat: implement keep-alive logic in catchup worker to process newly queued ranges#108
anhthii merged 2 commits into
mainfrom
fix/catchup-keepalive-drained-ranges

Conversation

@vietddude

Copy link
Copy Markdown
Collaborator

No description provided.

@vietddude
vietddude requested a review from anhthii August 5, 2026 16:00
@vietddude

Copy link
Copy Markdown
Collaborator Author

Root cause

CatchupWorker.runCatchup() was a one-shot loop: it returned permanently once blockRanges drained to empty. The manager starts each worker's Start() exactly once and never restarts it.

Meanwhile the regular worker keeps queuing new catchup ranges at runtime — skipAheadIfLagging()queueCatchupRanges() writes ranges to Consul whenever it lags past maxLag. On BSC (400ms poll, fast blocks) the regular worker lags often, so:

  1. Startup catchup drains initial ranges → catchup goroutine exits for good.
  2. Regular worker later lags → writes new catchup_progress/* keys to Consul.
  3. Nobody processes them → keys pile up in Consul (observed), only REGULAR logs remain, no CATCHUP.

Other chains rarely hit it because they don't lag fast enough to queue ranges after the catchup loop has already exited.

Fix

  • runCatchup() now stays alive after draining: it waits one catchupIdleInterval (3s), reloads from the store, and loops — only exits on ctx.Done().
  • New reloadStoredRanges() reads only persisted ranges; the idle poll and post-pass reload use it instead of loadCatchupProgress(), whose head-vs-latest auto-create would otherwise spawn ranges overlapping the regular worker's in-flight blocks.
  • idleInterval field added so tests can shorten the wait.

Verification

UnitTestCatchupWorkerPicksUpRangesQueuedAfterDrain (passes with -race): drains, then a range queued afterward must be picked up and completed without the loop exiting.

Live (BSC, 40 seeded ranges) — reproduced end-to-end:

  • Loaded 40 ranges → processed in parallel → deleted from Consul down to 0.
  • Seeded 3 new ranges while idle → 3s later: Picked up newly queued catchup ranges ranges=3 → processed → completed. Consul back to 0.

Before this fix step 3 would be ignored forever — exactly the BSC symptom.

@anhthii
anhthii merged commit 760ee35 into main Aug 5, 2026
2 checks passed
@anhthii
anhthii deleted the fix/catchup-keepalive-drained-ranges branch August 5, 2026 16:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants