Skip to content

fix(protocol): stop internet providers flapping the relay under unreachable-peer backlogs - #420

Open
kivtxs wants to merge 1 commit into
mainfrom
fix/internet-provider-relay-flap
Open

fix(protocol): stop internet providers flapping the relay under unreachable-peer backlogs#420
kivtxs wants to merge 1 commit into
mainfrom
fix/internet-provider-relay-flap

Conversation

@kivtxs

@kivtxs kivtxs commented Aug 28, 2026

Copy link
Copy Markdown
Member

Problem

An internet-only provider that accumulates undelivered direct messages to peers that have gone offline gets disconnected by the relay's per-connection rate limiter and reconnects on a loop ("flapping", ~1–2×/min). It does not break live request/response — but it is a persistent connection-stability defect that grows with the size of a provider's unreachable-peer backlog, so any long-running provider serving many ephemeral peers hits it over time.

Root causes (three compounding defects, all always-on fixes)

  1. Backoff reset. A resend that had to register a fresh ACK (e.g. after an ACK-timeout re-queue) restarted the retry ladder at retry_count 0, pinning delay_for_retry at its 1s floor forever for a never-ACKing recipient. AckManager::set_retry_count now carries the retry-queue entry's accumulated count (retry_count + 1) onto the fresh ACK, so backoff keeps climbing. This was the dominant contributor.
  2. Confirmation-probe storm. An unconfirmed peer that vanished was re-probed every 5s indefinitely. The confirmation probe now escalates on the same 15s→600s ladder as the welcome lifecycle, resets on a reachability edge, and no-ops for non-probe-target peers.
  3. Dropped relay verdict (Python bridge). The Python bridge only logged the relay's recipient-keyed DeliveryError. It now correlates in-flight sends per recipient — a port of the iOS/Android RecipientInFlightTracker (record before the wire write, resolve on MessageSent, fail every live in-flight id on DeliveryError, feed internet_peer_presence(online=false)) — so unreachable peers are learned promptly.

These only reduce redundant relay traffic; they restore the SDK's own documented backoff, they don't change the delivery contract.

Opt-in flood control (default off)

Adds RetryConfig.edge_driven_unreachable_dm (default false). When enabled, a durably-unreachable DM stops being timed-probed after a bounded number of probes and rests in the outbox, re-driven only on a reachability edge (inbound frame / presence-online); a token bucket caps the core resend rate so a large backlog cannot burst past the relay limit. Intended for deployments whose peers always interact or advertise presence on return (e.g. a machine-to-machine capability exchange).

Default false reproduces the documented perpetual-probe behavior exactly, pinned by a test that fails if the default ever flips — so every existing native and third-party integration is unaffected. Plumbed through core config, the uniffi ProtocolConfig/RetryConfig records, the UDL, all three regenerated bindings (Swift/Kotlin/Python), and the RN TypeScript + native layers.

Tests & docs

Regression test for every mechanism — Rust: set_retry_count carry-forward, confirmation-probe escalation, edge-driven parking gate, resend rate cap, flag-defaults-off. Python: RecipientInFlightTracker record/resolve/drain/unrecord, record-before-write ordering, post-wire-failure keeps the entry. Docs updated (docs/configuration.md, docs/message-delivery.md); CHANGELOG entry added.

Validation

  • cargo test --workspace — 2623 pass / 0 fail (incl. doctests)
  • Python bridge suite — 56 pass
  • iOS bridge tests (swift test --package-path bindings/react-native/ios) — 252 pass / 0 fail; the iOS bridge typecheck over the excluded native sources (including the regenerated Generated/offline_protocol.swift) completes clean
  • Android bridge tests (gradle :offlineprotocol:testDebugUnitTest, RN Android CI harness) — 443 pass / 0 fail across 40 suites
  • cargo fmt --all --check clean; cargo clippy --workspace clean on changed crates (one pre-existing data_sync.rs:442 lint is unrelated to this change)
  • Empirical: the accumulated backlog that originally flapped now runs 0 relay disconnects and drains to 0/s steady state; the default path is byte-for-byte unchanged.

…chable-peer backlogs

An internet-only provider that accumulates undelivered direct messages to
peers that have gone offline gets disconnected by the relay's per-connection
rate limiter and reconnects on a loop ("flapping"). Three compounding defects
drive an unbounded resend rate to unreachable peers:

1. A resend that has to register a fresh ACK (e.g. after an ACK-timeout
   re-queue) restarted the backoff ladder at retry_count 0, pinning
   delay_for_retry at its 1s floor forever for a never-ACKing recipient.
   AckManager::set_retry_count now carries the retry-queue entry's accumulated
   count onto the fresh ACK so backoff keeps climbing.

2. An unconfirmed peer that vanished was re-probed every 5s indefinitely. The
   confirmation probe now escalates on the same 15s->600s ladder as the welcome
   lifecycle and resets on a reachability edge.

3. The Python relay bridge dropped the relay's recipient-keyed DeliveryError
   verdict. It now correlates in-flight sends per recipient (a port of the
   iOS/Android RecipientInFlightTracker: record before the wire write, resolve
   on MessageSent, fail every live in-flight id on DeliveryError, feed
   internet_peer_presence offline), so unreachable peers are learned promptly.

All three are always-on and only reduce redundant relay traffic; the default
delivery contract is unchanged.

Adds an opt-in RetryConfig.edge_driven_unreachable_dm (default false) for
deployments whose peers always interact or advertise presence on return (e.g.
a machine-to-machine capability exchange): after a bounded number of probes a
durably-unreachable DM stops being timed-probed and rests in the outbox,
re-driven only on a reachability edge, and the core resend rate is capped by a
token bucket so a large backlog cannot burst past the relay's rate limit.
Default false preserves the documented perpetual-probe behavior exactly, so
every existing native and third-party integration is unaffected.

Plumbed through core config, the uniffi ProtocolConfig and RetryConfig records,
the UDL, all three regenerated bindings (Swift/Kotlin/Python), and the React
Native TypeScript and native layers. Regression tests added for every mechanism
(Rust: set_retry_count carry-forward, confirmation-probe escalation, edge-driven
parking gate, resend rate cap, flag-defaults-off; Python: RecipientInFlightTracker
record/resolve/drain/unrecord and record-before-write ordering). Docs updated in
docs/configuration.md and docs/message-delivery.md; CHANGELOG entry added.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant