Skip to content

fix(runtime): open the client listener before dialing seeds - #66

Merged
oreofeolurin merged 1 commit into
devfrom
fix/flo-136-cluster-e2e-boot
Aug 30, 2026
Merged

fix(runtime): open the client listener before dialing seeds#66
oreofeolurin merged 1 commit into
devfrom
fix/flo-136-cluster-e2e-boot

Conversation

@oreofeolurin

@oreofeolurin oreofeolurin commented Aug 30, 2026

Copy link
Copy Markdown
Contributor

What

Moves seed dialing from before acceptor.listen to the end of Runtime.start, after every listener (raft, acceptor, metrics, dashboard) is up. The fix half of #54; the diagnosis half (log-tail on readiness failure) already landed in #55, so this PR's earlier harness hunk was dropped on rebase.

Why

Every CI e2e run failed the same 13 cluster tests with the joiner never passing its 10 s readiness probe. The join handshake does a blocking connect plus a response read that can wait several seconds per attempt, retried 30× per seed — all before the client listener opened. A joiner must accept clients even when its peers are slow or unreachable.

Result (pre-rebase CI run of this change)

e2e went from 13 cluster failures at bring-up to 580/584, zero bring-up failures. The 4 remaining are the pre-existing replication-loss flakes (mget ×2, all-nodes-write, writes-after-leader-failure) — the best-effort-broadcast issue (#16), same set as the local macOS baseline.

Watch

@oreofeolurin
oreofeolurin force-pushed the fix/flo-136-cluster-e2e-boot branch from 22e47ff to dc5ff66 Compare August 30, 2026 16:40
@oreofeolurin

Copy link
Copy Markdown
Contributor Author

Reviewed the diff against current dev. The change is correct and worth landing — moving seed dialing after the listeners is right regardless of what it does to the test numbers, because a joining node should serve clients even when its peers are slow or unreachable.

Checked the specifics:

  • Nothing between the old and new call site depends on peer connections (acceptor, metrics, dashboard are all independent), so the move is safe.
  • The gate changes from being inside if (clusterListenerWanted()) to if (self.raft_network) |rn|. Those are equivalent — raft_network is only ever set inside that block — and seeds can't be configured without the listener being wanted, since seeds.len > 0 forces it.
  • Bonus: it fixes a doc-comment indentation I left behind in fix(cluster): honor [cluster] enabled, implement cluster status, explain io_uring (#42 items 5, 6, 7) #47.

Three things worth adjusting, none blocking.

1. The Results section double-counts. "e2e went from 13 cluster failures at bring-up to 580/584" was a pre-rebase run. dev already got 13 → 4 from the O_NONBLOCK fix in #55 (setNonBlocking hardcoded macOS's 0x0004, so peer sockets stayed blocking on Linux and the seed's network thread blocked forever in read() after its first peer). This PR's current CI shows 584/588 with the same 4 — i.e. no delta, because the bring-up failures were already gone. The change is still worth having; the number just isn't its.

2. "~6 s per unreachable seed" understates the worst case. That holds when connect is refused outright (~200 ms per attempt × 30 ≈ 6 s). But if a seed accepts the TCP connection and then doesn't answer the handshake, connectToPeer reaches readExact, which polls with a 5 s timeout per attempt — so 30 × ~5.2 s ≈ 156 s, not 6. That is exactly the state I hit while debugging #54, before the O_NONBLOCK fix made seeds answer promptly. Unlikely now, but operators reading that line will size timeouts against it.

3. A consequence worth naming in the body. The node now accepts clients before it has joined any peer. Since every node self-commits (peer_count == 0), a write landing in that window commits locally and broadcasts to zero peers — and there is no repair path, so it never replicates. That widens the exposure in #16/#62 slightly. I don't think it argues against the change: blocking client readiness on peer latency is worse, and the real fix is the replication rework. But it is a real edge worth stating rather than discovering later.

No test. The PR changes startup ordering with nothing asserting it. The property is testable with the existing harness — start a node with a deliberately unreachable seed and assert the client port answers well before start() returns. Worth adding here or as a follow-up, since the next refactor of Runtime.start has nothing to stop it moving back.

Also still open nearby, pre-existing and not this PR's job: connectToPeer("127.0.0.1", seed_port) discards the seed's host, so cross-host clustering cannot work regardless of ordering.

A joiner dialed its seeds before acceptor.listen: each join attempt does
a blocking connect plus a response read that can wait several seconds,
retried up to 30 times per seed — so on a slow host the client listener
could open tens of seconds late, and every readiness probe that only
waits 10s reported the node dead (#54). Seed dialing now runs last,
after all listeners are up; a joiner accepts clients even when its
peers are slow or unreachable.
@oreofeolurin
oreofeolurin force-pushed the fix/flo-136-cluster-e2e-boot branch from dc5ff66 to 0ca7d3c Compare August 30, 2026 19:50
@oreofeolurin
oreofeolurin merged commit 7e01cb7 into dev Aug 30, 2026
4 of 5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant