csi: drive reconnect/guardian detection from the guardian's registry, not nvme list

[boddumanohar]

Summary

Alternative to #337, for the same root cause: SPDKCSI-RECONNECT-GUARDIAN e2e flakiness where the guardian never restarted a pod after total NVMe-oF path loss.

Investigation (k8s pod logs, SPDK target logs, and the raw ginkgo run — simplyBlockDeployGCP run 29477941151) pinned the failing run's timeline precisely:

• 07:25:42.147–07:25:42.284 — volume staged and connected
• 07:25:44.423 — test issues nvme disconnect -n <NQN> to induce total path loss
• 07:25:44.531–.688 — SPDK target confirms both paths' queues torn down

The device was alive for only ~2.3 seconds before losing all paths — shorter than the csi-node's MonitorConnection poll cadence (3s base + up to 500ms jitter). The loss was real and permanent, and the storage cluster was healthy throughout — not an infra flake.

Root cause: reconnectSubsystems (the MonitorConnection poll loop) discovered connections by scanning nvme list/nvme list-subsys and diffing successive snapshots against a package-level history map. That can only ever detect a lost connection if some earlier poll happened to observe it as present first. A volume that connects and loses all its paths faster than one poll interval was never recorded as present, so its disappearance had nothing to diff against and was silently missed forever. This isn't limited to this one timing window either — any gap that beats the poll cadence has the same failure mode (e.g. the csi-node pod itself restarting and losing its in-memory maps).

This PR's approach

Rather than patching the specific race (see #337 for that narrower fix), this replaces the discover-then-diff model entirely with one driven by the guardian's own registry:

• The guardian already tracks every lvol it has published on this node, synchronously, at NodeStageVolume/NodePublishVolume/NodeUnpublishVolume time (RegisterPublish/RegisterUnpublishByTargetPath) — no polling involved, no race window.
• That registry now also records the exact subsystem NQN each lvol was published under (the real wire value from the volume context — not reconstructed from clusterID+lvolID, which is wrong whenever a subsystem is shared across lvols).
• reconnectSubsystems now iterates the guardian's tracked lvols (Guardian.TrackedLvols()) and checks each one's live state directly against a single nvme list-subsys snapshot per poll: "should be connected" (guardian registry, race-free) vs. "is connected right now" (one live snapshot) — no historical diffing needed at all. A lvol missing from that snapshot is immediately actionable on the very first poll after the loss, regardless of how quickly it happened.
• This makes the isManagedLvol PV/PVC ownership filter (added previously to keep the old discovery loop from touching benchmark/foreign volumes) redundant, and removes it — the guardian's registry only ever contains lvols this driver's own NodePublishVolume handled.
• Removes the dead devicePath parameter threaded through recoverPathsWithANA/reconcileOptimizedPath/reconcileNonOptimizedPaths (already unused, flagged by the linter) and the device-path/sysfs-uuid discovery machinery (getNVMeDeviceInfos, logicalVolumeIdByDevicePath, devicePresentMap/deviceToLvolIDMap) that existed only to support the old discovery loop.

Trade-off vs. #337

This is a materially larger change than #337, and it's not free of downsides: reconnectSubsystems's proactive path-degradation recovery (independent of guardian restarts) now only covers lvols the guardian is tracking, which starts at NodePublishVolume time (needs a per-pod target path to derive a pod UID) rather than NodeStageVolume time (when the device actually connects, but with no pod context yet). In practice that window is milliseconds to low seconds — kubelet calls Stage then Publish back-to-back for the one pod using a volume — and it doesn't affect guardian-restart behavior at all (no pod exists yet to restart during that window either way, in old or new design). It could be closed by also registering with the guardian at NodeStageVolume time (clusterID/lvolID/NQN only, no pod UID yet) if full parity with the old recovery scope is wanted — happy to add that if this approach is preferred.

Test plan

• go build ./...
• go test ./pkg/... ./cmd/...
• golangci-lint run ./pkg/... — no new issues
• Added guardian_test.go covering TrackedLvols(): reflects RegisterPublish/RegisterUnpublishByTargetPath, skips entries missing an NQN, and reports AlreadyBroken after MarkBrokenLvol (removed reconnect_test.go, which tested the now-removed isManagedLvol)
• Re-run SPDKCSI-RECONNECT-GUARDIAN e2e a few times to confirm improved reliability (no existing unit test harness drives reconnectSubsystems end-to-end, since it shells out to real nvme state)

🤖 Generated with Claude Code