design(agent): runed shutdown detaches volumes out from under running containers

[oreofeolurin]

Summary

Restarting runed detaches every volume on the node while the containers using them keep running. Nothing coordinates the two, and nothing needs to have gone wrong for it to happen — it is the normal path, on every upgrade.

Split out of #270. The data-loss half is fixed in #271 (the filesystem is now flushed before the detach), so this is no longer silent corruption — but a running container is still left holding a filesystem whose block device has been taken away.

The sequence

1. Agent.shutdown (internal/agent/agent.go:317-325) stops subsystems in reverse registration order.
2. The volume subsystem's Stop drains every tracked mount (internal/agent/volumes/subsystem.go:287), which unmounts and detaches each volume.
3. Nothing stops the containers. There is no StopInstance/ContainerStop anywhere in the shutdown path — containers are owned by the Docker daemon and deliberately outlive runed. That is a feature: a control-plane restart should not take the workloads down.

So the unmount runs against a mount point a live container still has bound. umount(2) returns EBUSY, tearDown logs "Unmount failed; will still attempt Detach", and the detach proceeds so the volume is not stranded on a node that may be going away.

Consequences

• The container keeps running against a filesystem whose device is gone. Writes it makes after the detach have nowhere to go; behaviour from there is the kernel's, not ours (I/O errors, or a read-only remount).
• When runed comes back it re-attaches and re-mounts at the same host path. The container is not remounted — a bind captures the mount that existed when the container was created — so from then on the container and the host are looking at two different filesystems at the same path. The host sees a freshly mounted volume; the container sees the orphaned one. Nothing reports the divergence.
• I suspect this is why Rune: file contents written to a gce-pd volume are lost #270 observed stat returning 0 bytes from the host on a volume the workload believed it had written. I have not proven that specific chain, so treat it as a hypothesis, not a finding.

Why the fix is not "stop the containers first"

Containers surviving a runed restart is the desired property; taking every workload down to shut the agent down cleanly would be a much worse trade.

The better question is why shutdown detaches at all. A volume needs detaching when it is moving — the node is being drained or the volume is rebinding elsewhere — not because a process on the node restarted. The adopt-attached path (#173) already brings an attached volume back with no API calls, so leaving mounts in place across a restart is both cheaper and safer than tearing them down and rebuilding them.

Suggested direction

• Distinguish "this process is stopping" from "this node is giving up its volumes". On the former, leave mounts alone; on the latter, drain as today.
• If a drain must happen while instances are live, skip teardown for volumes still referenced by a running local instance and say so, rather than detaching underneath them.
• Whatever the policy, tearDown's "detach even though the unmount failed" should be an explicit, documented decision. It is defensible on a dying node and wrong on a healthy one; today it is unconditional and was never stated as a choice.

Related

• Rune: file contents written to a gce-pd volume are lost #270 — the data loss this ordering produced
• fix(storage): flush the filesystem before unmount/detach #271 — flushes before detach, so the data survives; does not change the ordering
• bug(storage): Unmount silently succeeds when the findmnt probe cannot run (expired/cancelled ctx), orphaning the mount #191 — the silent-unmount probe, which made the same path fail invisibly
• bug(agent): subsystem shutdown shares one budget sequentially, and three subsystems ignore it outright #189 — the shutdown budget, which decides whether the drain gets to run at all