fix(runtime): open the client listener before dialing seeds - #66
Conversation
22e47ff to
dc5ff66
Compare
|
Reviewed the diff against current Checked the specifics:
Three things worth adjusting, none blocking. 1. The Results section double-counts. "e2e went from 13 cluster failures at bring-up to 580/584" was a pre-rebase run. 2. "~6 s per unreachable seed" understates the worst case. That holds when connect is refused outright (~200 ms per attempt × 30 ≈ 6 s). But if a seed accepts the TCP connection and then doesn't answer the handshake, 3. A consequence worth naming in the body. The node now accepts clients before it has joined any peer. Since every node self-commits ( No test. The PR changes startup ordering with nothing asserting it. The property is testable with the existing harness — start a node with a deliberately unreachable seed and assert the client port answers well before Also still open nearby, pre-existing and not this PR's job: |
A joiner dialed its seeds before acceptor.listen: each join attempt does a blocking connect plus a response read that can wait several seconds, retried up to 30 times per seed — so on a slow host the client listener could open tens of seconds late, and every readiness probe that only waits 10s reported the node dead (#54). Seed dialing now runs last, after all listeners are up; a joiner accepts clients even when its peers are slow or unreachable.
dc5ff66 to
0ca7d3c
Compare
What
Moves seed dialing from before
acceptor.listento the end ofRuntime.start, after every listener (raft, acceptor, metrics, dashboard) is up. The fix half of #54; the diagnosis half (log-tail on readiness failure) already landed in #55, so this PR's earlier harness hunk was dropped on rebase.Why
Every CI e2e run failed the same 13 cluster tests with the joiner never passing its 10 s readiness probe. The join handshake does a blocking connect plus a response read that can wait several seconds per attempt, retried 30× per seed — all before the client listener opened. A joiner must accept clients even when its peers are slow or unreachable.
Result (pre-rebase CI run of this change)
e2e went from 13 cluster failures at bring-up to 580/584, zero bring-up failures. The 4 remaining are the pre-existing replication-loss flakes (mget ×2, all-nodes-write, writes-after-leader-failure) — the best-effort-broadcast issue (#16), same set as the local macOS baseline.
Watch
start()returns — previously it served nothing during that window.