daemon: fold activeClusterTransport under the comms epoch guard - #6869
Open
psaab wants to merge 3 commits into
Open
daemon: fold activeClusterTransport under the comms epoch guard#6869psaab wants to merge 3 commits into
psaab wants to merge 3 commits into
Conversation
activeClusterTransport was written by startClusterComms holding neither clusterCommsMu nor applySem, and read by applyTailReconciles step 20 under applySem. #6290 filed this as "NOT a race today", reasoning that the boot startClusterComms completes before the gRPC/HTTP servers accept commits, so the write is visible before any reader runs. That accounts for the wrong readers. applyConfig's own doc lists its callers: DHCP callbacks, config-poll, dynamic feeds, event engine, in-process CLI commits, CLI auto-rollback and cluster sync recv — none of which are the gRPC or HTTP servers. The DHCP one is already live before the write: the boot applyConfig (daemon_run_bringup.go) starts the DHCP clients via reconcileDHCPClients (daemon_apply_routing.go) with onDHCPAddressChange already wired, and that callback re-enters applyConfig — hence step 20 — on its own goroutine. Those goroutines are created BEFORE the boot startClusterComms write, so goroutine creation orders them the wrong way and establishes no happens-before, and applySem cannot bridge it because a semaphore only excludes participants that take it and the boot writer does not. A lease arriving in that window is an unsynchronized concurrent read/write of a six-string struct: a real Go data race. It is the mirror of #5113 on mgmtVRFInterfaces — written under applySem, read by this same callback without it — which daemon.go already documents as a real data race. The write now goes through setActiveTransportIfCurrent(gen, key), which is epoch-gated rather than a bare locked store, mirroring publishSessionSyncIfCurrent. The same window admits two concurrent startClusterComms calls (the boot one and a DHCP-driven restart from step 20); with an ungated store the loser of that ordering can land last, leaving the transport key describing a superseded epoch while clusterCommsGen names the live one, so the next commit compares against the wrong baseline and either skips a needed comms restart or performs a spurious one. Step 20 now takes ONE activeTransport() snapshot for both its comparison and its eight log fields instead of up to six separate reads a concurrent restart could interleave. Validation: both new tests PASS under `go test -race -count=1`. Mutation A — drop both mutex guards, i.e. master's exact shape — produces WARNING: DATA RACE with the write attributed to startClusterComms at daemon_ha_sync.go:706. Mutation B — drop the `gen !=` check but keep the mutex — fails TestSetActiveTransportDropsStaleEpoch WITHOUT -race while the race test still passes, so each guard is bound by its own test and neither is satisfied by the other. Full `go test ./pkg/... ./cmd/...` exit 0 (59 packages, zero failures); `go test -race ./pkg/daemon/` exit 0; go vet and gofmt clean; go test ./pkg/refactoraudit/ passes. Not run: make test-failover, the required cluster smoke for this path (pkg/daemon/README.md) — scheduled by the lead, not this lane. Advances #6290.
added 2 commits
August 6, 2026 03:16
# Conflicts: # _Log.md
Resolve the sole conflict, _Log.md, by union: every entry from both sides is retained and none is rewritten. The file is no longer append-ordered, so line counts and prefix checks say nothing useful about the result; the resolution was verified structurally instead, by confirming that each pre-merge side diffs into the merged file with add-hunks only and no changed-or-deleted hunk on either side. Every other path merged without conflict. Because a clean textual auto-merge can still break compilation when a signature moves on one side, that was confirmed by building rather than by inspection: go build ./... clean on the merged tree. Advances #6869.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
activeClusterTransportwas written bystartClusterCommsholding neitherclusterCommsMunorapplySem, and read byapplyTailReconcilesstep 20 underapplySem.The issue's "NOT a race today" is wrong
#6290 reasoned that the boot
startClusterCommscompletes before the gRPC/HTTP servers accept commits, so the write is visible before any reader. That accounts for the wrong readers —applyConfig's own doc lists DHCP callbacks, config-poll, dynamic feeds, event engine, in-process CLI commits, CLI auto-rollback, cluster sync recv, none of which are the gRPC or HTTP servers.The DHCP one is live before the write:
A lease arriving in that window runs
onDHCPAddressChange→applyConfig→ step 20, which reads the field. That goroutine was created before the write, so goroutine creation orders them the wrong way and supplies no happens-before;applySemcannot bridge it because a semaphore only excludes participants that take it, and the boot writer does not. This is the mirror of #5113 onmgmtVRFInterfaces— written underapplySem, read by this same callback without it — whichdaemon.goalready documents as "a real Go data race".Fix
setActiveTransportIfCurrent(gen, key), epoch-gated likepublishSessionSyncIfCurrentrather than a bare locked store: the same window admits two concurrentstartClusterCommscalls (boot + a DHCP-driven restart from step 20), and an ungated store lets the loser land last, leaving the transport key describing a superseded epoch whileclusterCommsGennames the live one. Step 20 now takes ONEactiveTransport()snapshot for its comparison and its eight log fields.Validation
Both new tests PASS under
go test -race -count=1. Two mutations, each binding its own guard:WARNING: DATA RACE, write attributed tostartClusterCommsatdaemon_ha_sync.go:706.gen !=check, keep the mutex →TestSetActiveTransportDropsStaleEpochFAILS without-race, while the race test still passes.Neither test is satisfied by the other's guard.
go test ./pkg/... ./cmd/...exit 0 (59 packages, zero failures).go test -race ./pkg/daemon/exit 0.go vet,gofmt,go test ./pkg/refactoraudit/clean.Not done
make test-failover— the required cluster smoke for this path perpkg/daemon/README.md— was not run; the lead schedules cluster work.Note:
daemon_ha_sync.gois now 1499 lines, one under the refactor-audit 1500 threshold. The gate passes, but the next line added to this file trips it.Closes #6290