fix(progressive-sync): wall-clock cap on rclone + age-backstop dead-letter demotion (F-SOAK-4)#35
Merged
Conversation
…ry demotion
F-SOAK-4: one event whose blob download kept failing SLOWLY dammed its
route for 10+ hours. Two compounding gaps:
1. rclone --timeout is an IDLE timeout and Command::output() waits
forever, so a trickling-then-stalling transfer burned 15-30+ minutes
per attempt — the durable RETRY_BUDGET (5) accrued too slowly to ever
demote. run_wall_capped() now gives every rclone invocation a hard
wall deadline (2x idle timeout + 30s), draining stdout/stderr on
threads so a chatty child cannot deadlock the wait. A hang becomes a
countable failed attempt.
2. Count-only budget: attempts.json now stores {count, first_at}; an
event still failing RETRY_MAX_AGE_SECS (3600) after its first
recorded attempt dead-letters regardless of count (poison rule:
later events keep applying; reconcile/bisync heal the file). Legacy
count-only files migrate by adopting now() as first_at — never
fabricates instant demotion. Successful applies clear their record
so the file cannot grow unbounded.
4 new integration tests (age backstop unblocks route, budget demotion
clears attempts, legacy schema migration, success clears record) + 4
wall-cap unit tests. Workspace suite 367 green, clippy at baseline.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| ❌ Deployment failed View logs |
tytus-cli | 85f3cfd | Jul 06 2026, 09:35 AM |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Two changes that together kill head-of-line blocking by transient-forever blob failures:
run_wall_capped()— every rclone subprocess gets a hard wall-clock deadline (2× idle timeout + 30s slack; copyto = 630s). rclone's--timeoutis an idle timeout andCommand::output()waits forever, so a trickling transfer over a degraded path burned 15–30+ min per attempt. Output drained on threads (no pipe-buffer deadlock); on deadline the child is killed and the attempt fails countably.attempts.jsonupgraded to{count, first_at};should_demote = count ≥ RETRY_BUDGET || now − first_at ≥ 3600s. Slow-failing events can no longer ride below the count budget forever. Legacy count-only schema migrates safely (adopts now() as first_at). Successful applies clear their record.Why now
Live incident F-SOAK-4 (soak artifact 2026-07-06): one stuck blob pinned all 3 freelance routes' cursors from 21:28Z to the relay fix ~09:00Z next day — 14+ events queued behind it,
dead_letter_countstayed 0 the whole time. The WG relay is fixed (infra #20/#21) but the consumer must never again let one bad object dam a route.Tests
4 new pipeline integration tests + 4 wall-cap unit tests; workspace suite 367 green; clippy warning count identical to main.
🤖 Generated with Claude Code