Skip to content

daemon: fold activeClusterTransport under the comms epoch guard - #6869

Open
psaab wants to merge 3 commits into
masterfrom
fix/6290-cluster-transport-lock
Open

daemon: fold activeClusterTransport under the comms epoch guard#6869
psaab wants to merge 3 commits into
masterfrom
fix/6290-cluster-transport-lock

Conversation

@psaab

@psaab psaab commented Aug 5, 2026

Copy link
Copy Markdown
Owner

activeClusterTransport was written by startClusterComms holding neither clusterCommsMu nor applySem, and read by applyTailReconciles step 20 under applySem.

The issue's "NOT a race today" is wrong

#6290 reasoned that the boot startClusterComms completes before the gRPC/HTTP servers accept commits, so the write is visible before any reader. That accounts for the wrong readers — applyConfig's own doc lists DHCP callbacks, config-poll, dynamic feeds, event engine, in-process CLI commits, CLI auto-rollback, cluster sync recv, none of which are the gRPC or HTTP servers.

The DHCP one is live before the write:

daemon_run_bringup.go  dhcp.New(..., d.onDHCPAddressChange, ...)   callback wired
daemon_run_bringup.go  boot d.applyConfig(cfg)
 -> daemon_apply_routing.go  reconcileDHCPClients(cfg)             client goroutines START
daemon_run.go          d.startClusterComms(ctx)                    the WRITE happens here

A lease arriving in that window runs onDHCPAddressChangeapplyConfig → step 20, which reads the field. That goroutine was created before the write, so goroutine creation orders them the wrong way and supplies no happens-before; applySem cannot bridge it because a semaphore only excludes participants that take it, and the boot writer does not. This is the mirror of #5113 on mgmtVRFInterfaces — written under applySem, read by this same callback without it — which daemon.go already documents as "a real Go data race".

Fix

setActiveTransportIfCurrent(gen, key), epoch-gated like publishSessionSyncIfCurrent rather than a bare locked store: the same window admits two concurrent startClusterComms calls (boot + a DHCP-driven restart from step 20), and an ungated store lets the loser land last, leaving the transport key describing a superseded epoch while clusterCommsGen names the live one. Step 20 now takes ONE activeTransport() snapshot for its comparison and its eight log fields.

Validation

Both new tests PASS under go test -race -count=1. Two mutations, each binding its own guard:

  • A — drop both mutex guards (master's exact shape) → WARNING: DATA RACE, write attributed to startClusterComms at daemon_ha_sync.go:706.
  • B — drop the gen != check, keep the mutex → TestSetActiveTransportDropsStaleEpoch FAILS without -race, while the race test still passes.

Neither test is satisfied by the other's guard.

go test ./pkg/... ./cmd/... exit 0 (59 packages, zero failures). go test -race ./pkg/daemon/ exit 0. go vet, gofmt, go test ./pkg/refactoraudit/ clean.

Not done

make test-failover — the required cluster smoke for this path per pkg/daemon/README.md — was not run; the lead schedules cluster work.

Note: daemon_ha_sync.go is now 1499 lines, one under the refactor-audit 1500 threshold. The gate passes, but the next line added to this file trips it.

Closes #6290

activeClusterTransport was written by startClusterComms holding neither
clusterCommsMu nor applySem, and read by applyTailReconciles step 20
under applySem. #6290 filed this as "NOT a race today", reasoning that
the boot startClusterComms completes before the gRPC/HTTP servers accept
commits, so the write is visible before any reader runs.

That accounts for the wrong readers. applyConfig's own doc lists its
callers: DHCP callbacks, config-poll, dynamic feeds, event engine,
in-process CLI commits, CLI auto-rollback and cluster sync recv — none of
which are the gRPC or HTTP servers. The DHCP one is already live before
the write: the boot applyConfig (daemon_run_bringup.go) starts the DHCP
clients via reconcileDHCPClients (daemon_apply_routing.go) with
onDHCPAddressChange already wired, and that callback re-enters
applyConfig — hence step 20 — on its own goroutine. Those goroutines are
created BEFORE the boot startClusterComms write, so goroutine creation
orders them the wrong way and establishes no happens-before, and applySem
cannot bridge it because a semaphore only excludes participants that take
it and the boot writer does not. A lease arriving in that window is an
unsynchronized concurrent read/write of a six-string struct: a real Go
data race. It is the mirror of #5113 on mgmtVRFInterfaces — written under
applySem, read by this same callback without it — which daemon.go already
documents as a real data race.

The write now goes through setActiveTransportIfCurrent(gen, key), which
is epoch-gated rather than a bare locked store, mirroring
publishSessionSyncIfCurrent. The same window admits two concurrent
startClusterComms calls (the boot one and a DHCP-driven restart from step
20); with an ungated store the loser of that ordering can land last,
leaving the transport key describing a superseded epoch while
clusterCommsGen names the live one, so the next commit compares against
the wrong baseline and either skips a needed comms restart or performs a
spurious one. Step 20 now takes ONE activeTransport() snapshot for both
its comparison and its eight log fields instead of up to six separate
reads a concurrent restart could interleave.

Validation: both new tests PASS under `go test -race -count=1`. Mutation
A — drop both mutex guards, i.e. master's exact shape — produces WARNING:
DATA RACE with the write attributed to startClusterComms at
daemon_ha_sync.go:706. Mutation B — drop the `gen !=` check but keep the
mutex — fails TestSetActiveTransportDropsStaleEpoch WITHOUT -race while
the race test still passes, so each guard is bound by its own test and
neither is satisfied by the other. Full `go test ./pkg/... ./cmd/...`
exit 0 (59 packages, zero failures); `go test -race ./pkg/daemon/` exit
0; go vet and gofmt clean; go test ./pkg/refactoraudit/ passes.

Not run: make test-failover, the required cluster smoke for this path
(pkg/daemon/README.md) — scheduled by the lead, not this lane.

Advances #6290.
Paul Saab added 2 commits August 6, 2026 03:16
Resolve the sole conflict, _Log.md, by union: every entry from both
sides is retained and none is rewritten. The file is no longer
append-ordered, so line counts and prefix checks say nothing useful
about the result; the resolution was verified structurally instead, by
confirming that each pre-merge side diffs into the merged file with
add-hunks only and no changed-or-deleted hunk on either side.

Every other path merged without conflict. Because a clean textual
auto-merge can still break compilation when a signature moves on one
side, that was confirmed by building rather than by inspection: go build
./... clean on the merged tree.

Advances #6869.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

HA: activeClusterTransport written in startClusterComms without clusterCommsMu (benign today via happens-before; fold under the #4958 epoch guard)

1 participant