fix(progressive-sync): durable gap grace + producer heartbeat + truthful status + wake resume [Phase 3/4]#34
Merged
Conversation
…l status, wake resume
Four surgical fixes for the Phase 3 consumer + tray scheduler
(sprint tytus-garagetytus-progressive-sync, consumer auto-halt hardening).
1. Persist gap grace across consumer rebuilds (HARD sprint exit condition).
The tray builds a fresh StateStore + BindingConsumer per poll, so the
in-memory RouteRuntime.gap_polls counter reset on every rebuild and the
sequence_gap halt could NEVER fire in production. The counter now lives
in a durable side file (gap_polls.json in store.dir, same atomic_write
discipline as attempts.json). Each entry is keyed by the cursor it was
observed at: any durable cursor advance (contiguous apply, dead-letter,
SkipSequence, Repair) invalidates the stale count implicitly, so a
counter left >= GAP_GRACE_POLLS by an adjudicated-and-cleared gap can
never instantly re-trip a NEW gap — this expresses the "reset when a
Repair/SkipSequence record is folded" requirement without hooking
fold() (pure state, no store access; Repair is appended by the tray).
Persistence is best effort (unlike attempts.json's fail-closed): a lost
count degrades to the old per-process grace, conservative by nature.
Halt semantics unchanged: increment per gap-observing poll, trip at
>= GAP_GRACE_POLLS, reset on empty prefix and contiguous apply.
2. Producer heartbeat staleness (producer-health-v1, dead-vs-quiet).
Per enabled route per poll, one small GET (existing S3Ops::cat_small)
of _tytus-sync/health/{remote_binding_id}/{route_id}.json. Pure rule
producer_health_from_heartbeat(): stale when now - written_at >
3*scan_interval_seconds + poll_interval (tray passes its real
POLL_INTERVAL_SECS); absent/unparsable/foreign-schema => "unknown"
(older producers must not alarm). Surfaced as PollOutcome
.producer_health and in the tray poll outcome JSON (served by both
/api/shared-folders/progressive/status and status v2). Status signal
only — never halts the consumer or touches repair_required. Includes a
no-dependency ISO 8601 UTC parser mirroring state::utc_now().
3. Status endpoint never lies with {}. status_snapshot() now merges the
in-memory map with the enabled-binding roster (discover(), mirroring
the scheduler's eligibility filter): enabled bindings the poller has
not reported yet appear as state="pending_first_poll" instead of being
absent — a fresh tray restart previously looked identical to "no
progressive sync configured". Entries gain an explicit lifecycle
marker (pending_first_poll | polling). Merge is a pure function
(snapshot_with_roster) so the invariant is unit-tested.
4. Sleep-poisoned backoff / wake resume. The scheduler tracks wall-clock
time across its 2 s tick; a wall delta beyond tick + 30 s can only mean
the machine slept, so pre-sleep Instant deadlines and doubled backoff
are poisoned. On detection (pure fn wake_detected, unit-tested): log
one line, clear the backoff map, reset every next_poll deadline to now
so all bindings re-poll immediately on wake. Self-contained — no OS
power-notification APIs.
No changes to event schema, journal format, or Repair/adjudication
semantics; repair_required stays #[serde(skip)] non-durable.
Tests added:
- pipeline.rs: sequence_gap_trips_across_consumer_rebuilds (3 separate
consumers, the tray lifecycle), contiguous_apply_resets_persisted_gap_counter,
repair_cleared_gap_counter_does_not_retrip_next_gap_instantly,
producer_heartbeat_fresh_reads_ok,
producer_heartbeat_old_written_at_reads_stale_without_halting,
producer_heartbeat_missing_or_unparsable_reads_unknown
- tray inline: snapshot_never_omits_enabled_binding,
wake_detected_only_on_wall_jump
Gates: cargo test -p tytus-progressive-sync (32 pipeline + 15 lib + 9
fixtures pass), cargo test -p tytus-tray (241 pass), clippy warning count
identical to main (13, all pre-existing). Also removed two pre-existing
unused imports in pipeline.rs and one pre-existing unused mut in apply.rs.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| ❌ Deployment failed View logs |
tytus-cli | 8c1b94d | Jul 05 2026, 05:47 PM |
…ng, per-route persistence - B1: parse_utc_epoch rejects impossible calendar dates (Feb 31, non-leap Feb 29) instead of normalizing through days-from-civil. - B2: fractional seconds must be 1+ ascii digits; '.fooZ'/'.Z' -> None. - B3: gap_polls.json entries for routes no longer authorized are pruned at consumer build (no pre-accumulated grace ambush on re-add). - B4: gap counter writes are read-modify-write scoped to the single route, so a concurrent consumer on the same store keeps its other-route entries. +2 regression tests (malformed-timestamp matrix incl. valid-fractional control; removed-route pruning). Suites: 58 crate + 241 tray green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
Author
|
Codex validation: round B NEEDS_FIX (impossible-date normalization, permissive fractional seconds, removed-route counter leak, whole-map write race) → all fixed in 8c1b94d with regression tests → round C VERDICT: SOLID. Suites: 58 progressive-sync + 241 tray green; clippy delta zero. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Phase 3/4 consumer deliverables of the shared-folders remediation sprint. Companion producer PR: traylinx/wannolot-infrastructure#19 (merged).
BindingConsumerevery poll, so the in-memorygap_pollscounter could never reachGAP_GRACE_POLLS: thesequence_gapauto-halt has never fired in production (Phase-1 finding Add Cloudflare Workers configuration #1). The counter now persists ingap_polls.json(same side-file discipline asattempts.json), keyed by the cursor it was observed at — any durable cursor advance (contiguous apply, dead-letter, SkipSequence, Repair) implicitly invalidates a stale count, so an adjudicated gap can't insta-retrip the next one. Persistence is deliberately best-effort (a lost write degrades to slower halt, never a spurious one) — rationale in code._tytus-sync/health/{binding}/{route}.json(producer-health-v1, onecat_smallper route per poll). Pure rule:stalewhennow − written_at > 3×scan_interval + poll_interval; absent/unparsable/foreign-schema →unknown(old producers must not alarm); future timestamps saturate took(clock skew must not fabricate death). Status signal only — never gates applies or touchesrepair_required.{}—status_snapshot()merges the in-memory poll map with the enabled-binding roster; unpolled bindings appear as explicitpending_first_pollentries. A fresh/restarted tray is no longer indistinguishable from "nothing configured".next_polldeadlines to now (macInstantfreezes during sleep; pre-sleep 300s backoffs described a network that no longer exists). Purewake_detecteddecision function, no OS power APIs.Tests (all green, independently re-run)
sequence_gap_trips_across_consumer_rebuilds— three separate consumers, each dropped after one poll (the tray's exact lifecycle) → third poll halts. This is the regression test production was missing (the existing gap test polls one long-lived consumer three times, which is why the bug was invisible).contiguous_apply_resets_persisted_gap_counter,repair_cleared_gap_counter_does_not_retrip_next_gap_instantlysnapshot_never_omits_enabled_binding,wake_detected_only_on_wall_jump(32s/33s boundary)repair_requiredstays non-durable.🤖 Generated with Claude Code