Use case
Three membership transitions have no coverage anywhere, and all three are ordinary operational events rather than exotic failures.
1. A node restarts under the same address. This is what a rolling deploy does to every node in turn, and what a supervisor does after a crash. The cluster must distinguish "the node I knew at this address, still running" from "a new incarnation at the same address" — otherwise the returning node inherits the old one's membership state, including an unreachable mark it does not deserve, or its rejoin is rejected because the address is already present.
2. The first seed node restarts. Seed nodes are asymmetric: the first one bootstraps the cluster. When it restarts, it must join the cluster that formed rather than forming a second one. This is the failure mode #1176 already reports from a different direction — two same-name clusters that never merge after a heal, a fork that is permanent until one side restarts.
3. A node is marked unreachable and then comes back. A transient network problem or a long GC pause produces exactly this, and it is the common case, not the rare one. The current suites partition and then either heal within the test or let the minority self-down; none takes a node all the way to unreachable and then restores it.
tests/multi-node/ contains 21 suites. None covers any of these. Searching the tracker for "seed node restart", "rejoin" or "joins again" returns nothing.
Proposed shape
Three scenarios on the existing MultiNodeSpec, which already has crash, leave, partition and heal:
- Restart under the same address:
crash(role), then start a fresh system at the same address; assert it joins as a new incarnation, that the old incarnation is removed rather than resurrected, and that watchers of actors on the old node were notified.
- Seed restart: form a cluster from seeds,
crash the first seed, restart it; assert it joins the existing cluster and that member count returns to the original rather than a second cluster forming.
- Unreachable then reachable: partition long enough for the failure detector to mark unreachable, then heal before downing; assert the node returns to
up without a restart and without being downed.
The third needs care with timing — it is a race between the heal and the downing decision. That argues for driving it with the failure-detector configuration the suites already use to compress detection (heartbeatIntervalMs, unreachableAfterMs, downAfterMs), and for using virtual time where possible rather than a wall-clock window.
Acceptance
Verification status
Confirmed by reading and search. All 21 files in tests/multi-node/ were scanned for their fault calls and none constructs any of the three transitions. Tracker searches for seed node.*restart, rejoin|joins again over 712 open issues return zero matches. src/testkit/MultiNodeSpec.ts was read for the available primitives. Related: #1176.
Use case
Three membership transitions have no coverage anywhere, and all three are ordinary operational events rather than exotic failures.
1. A node restarts under the same address. This is what a rolling deploy does to every node in turn, and what a supervisor does after a crash. The cluster must distinguish "the node I knew at this address, still running" from "a new incarnation at the same address" — otherwise the returning node inherits the old one's membership state, including an
unreachablemark it does not deserve, or its rejoin is rejected because the address is already present.2. The first seed node restarts. Seed nodes are asymmetric: the first one bootstraps the cluster. When it restarts, it must join the cluster that formed rather than forming a second one. This is the failure mode #1176 already reports from a different direction — two same-name clusters that never merge after a heal, a fork that is permanent until one side restarts.
3. A node is marked unreachable and then comes back. A transient network problem or a long GC pause produces exactly this, and it is the common case, not the rare one. The current suites partition and then either heal within the test or let the minority self-down; none takes a node all the way to
unreachableand then restores it.tests/multi-node/contains 21 suites. None covers any of these. Searching the tracker for "seed node restart", "rejoin" or "joins again" returns nothing.Proposed shape
Three scenarios on the existing
MultiNodeSpec, which already hascrash,leave,partitionandheal:crash(role), then start a fresh system at the same address; assert it joins as a new incarnation, that the old incarnation is removed rather than resurrected, and that watchers of actors on the old node were notified.crashthe first seed, restart it; assert it joins the existing cluster and that member count returns to the original rather than a second cluster forming.upwithout a restart and without being downed.The third needs care with timing — it is a race between the heal and the downing decision. That argues for driving it with the failure-detector configuration the suites already use to compress detection (
heartbeatIntervalMs,unreachableAfterMs,downAfterMs), and for using virtual time where possible rather than a wall-clock window.Acceptance
upwithout being downed.Verification status
Confirmed by reading and search. All 21 files in
tests/multi-node/were scanned for their fault calls and none constructs any of the three transitions. Tracker searches forseed node.*restart,rejoin|joins againover 712 open issues return zero matches.src/testkit/MultiNodeSpec.tswas read for the available primitives. Related: #1176.