[wip] fix(iroh): survive home relay changes on relay-only connections#4435
Draft
Frando wants to merge 5 commits into
Draft
[wip] fix(iroh): survive home relay changes on relay-only connections#4435Frando wants to merge 5 commits into
Frando wants to merge 5 commits into
Conversation
|
Documentation for this PR has been generated and is available at: https://n0-computer.github.io/iroh/pr/4435/docs/iroh/ Last updated: 2026-07-24T12:14:56Z |
Frando
changed the title
relay_teardown_failover gives the peers disjoint relay maps: the server holds a fast and a slow relay (homing on the fast one), the client its own unrelated relay. The connection runs over the server's home relay, which is torn down mid-connection. Address lookup runs against an iroh-dns-server in the lab (new linux-only dev-dependency): the server publishes over pkarr HTTP, the client resolves the _iroh TXT records on port 53, the only UDP port its firewall allows. Util grows a named relay spawner and the dns-server spawner. The test needs the failover fixes in the next commits to pass.
When the home relay connection drops, incremental net reports run only QAD probes. If UDP is unavailable those measure nothing, so the report returns no relay latencies and no other configured relay is promoted to home; the endpoint retries the dead relay for up to the full-report interval (5 minutes) instead of failing over. The socket actor now watches the home relay connection. On loss it schedules a full net report, deferred by two seconds so a brief reconnect does not trigger one, and rate-limited so a still-down relay does not force back-to-back reports. A full report re-runs the HTTPS probes and lets a reachable relay be promoted.
The relay path of a connection runs over the remote's home relay; the peers do not share relay maps. When that relay dies, the remote fails over its home relay and republishes its endpoint info, but the client kept retrying the dead relay: nothing re-resolved the remote or opened a path over its new home relay. Opening a path over our own home relay cannot help, since the remote is not connected to it. Detection uses noq's PathEvent::Suspect: a validated path on an established connection goes suspect after two consecutive PTOs without an ACK. When a suspect path is the connection's only one, the remote-state actor forces an address lookup via trigger_address_lookup(true): not running starts it immediately; already running just ensures a retry is queued for once the current attempt finishes. The retry keeps going with backoff (AddressLookupRetry, 500ms up to 5s) until the remote's new home relay shows up and a path opens over it, the suspect path recovers on its own, or the connection closes - whichever comes first (client side only; servers wait for the client's path). No relay-specific state is tracked for this: it's a generic "keep retrying this lookup for as long as something still needs it," reusable for any future reason a lookup might need forcing. The path selector skips suspect paths, so detection scales with the path RTT and also covers the remote losing its relay while our own relay connections are fine. Once a replacement path is selected, apply_selected_path's existing redundant-path closing (previously IP-only, keeping one path per connection) now also covers relay paths: any other relay path on the same connection gets closed the same way a superseded IP path already was. This is needed because frames already queued on the dead path are only retransmitted elsewhere once the path is abandoned, and the peer cannot be told to abandon it over the dead relay itself. Depends on noq's Frando/path-suspect branch (git dep, PR pending upstream). Does not include noq's separate abandon-time retransmit fix (also PR pending, see Frando/abandon-retransmit there): without it, relay_teardown_failover is expected to be flaky, since an abandoned path's stray in-flight frame isn't retransmitted until noq's own PathDrained/idle-timeout machinery eventually frees it. Tracked as a follow-up once that PR lands.
active_relay_handle_for_endpoint queries every tracked active relay via HasEndpointRoute before creating a new relay connection. The outer reconnect backoff loop slept via a bare time::sleep that never polled prio_inbox, so a query sent to a relay mid-backoff (e.g. one that just died) sat unanswered until that unrelated backoff elapsed, directly gating unrelated new relay connections. sleep_backoff now answers HasEndpointRoute(false) immediately while still counting down the delay, mirroring run_dialing's existing pattern. Path establishment after a relay teardown is now consistently ~1-1.5s instead of up to 15s+ in traced runs.
Frando
force-pushed
the
fix/relay-failover
branch
from
July 24, 2026 12:08
22b841f to
a0f8a26
Compare
…timeouts relay_teardown_failover was flaky against Frando/path-suspect alone (close to 100% once the reconnect-backoff fix made path establishment fast): the client's post-teardown ping is reliably still in flight on the dying relay path when it gets abandoned, and without an immediate retransmit on abandon, that frame doesn't get resent until noq's own PathDrained/idle-timeout machinery eventually frees it - which can easily exceed a 30s budget (observed one recovery taking 45s, driven by a coalesced ack inflating the dead path's own RTT estimate, and thus its 3*PTO PathDrained deadline, well past the test timeout). Points noq's temporary git patch at Frando/feat-path-suspect, which combines both pending noq PRs (Frando/path-suspect, Frando/abandon-retransmit) for testing until they land upstream. Verified 12/12 clean runs, consistently ~13-16s. With the abandon-retransmit fix landing, the generous timeouts are no longer needed: recovery_timeout drops from 30s to 15s and timeout from 15s to 10s, both still comfortable margins over observed recovery time.
Frando
force-pushed
the
fix/relay-failover
branch
from
July 24, 2026 12:12
a0f8a26 to
dc90a4e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
The first commit adds a patchbay test that fails on main: On a relay-only connection with no working IP path, we destroy connectivity to the relay. The test fails on main. However, this doesn't have to be, we can make the connection survive: Breaking the relay connection could trigger a net report, chose a new home relay, and handle the new relay URL to the client over addr lookup, so that it can open a new path over the new relay.
This PR explores how to we can do this.
In its current form, it does this:
Depends on n0-computer/noq#769
Depends on n0-computer/noq#770
Breaking Changes
Notes & open questions
Draft. The fix was mostly done by an AI tool and still needs to be reviewed.
Change checklist
proposed change and wrote an as clear and concise description as
they could.
intented effect.