Rune: file contents written to a gce-pd volume are lost

[jolaolurin]

Rune: first writes to a freshly-provisioned gce-pd volume are lost — files keep metadata, lose contents

Severity: a service that writes state on first boot and reads it back later
comes up permanently broken. Silent — nothing in rune logs or status indicates
a problem, and the service reports Running 1/1 right up until it is restarted.

Versions: rune server + CLI v0.0.1-dev.146 (8c34e706), single edge node
(e2-standard-2, europe-west2-a), gce-pd CSI driver, StorageClass
pd-ssd-euw2a (diskType: pd-ssd, fsType: ext4, reclaimPolicy: retain).

Summary

On the first boot after a gce-pd volume is newly provisioned, files written
by the service end up on disk with correct name, owner and mode but zero
length. Two unrelated files in our case (SYSTEM, flo.pid) — directory
entries and inodes persisted, file contents did not.

The service keeps running and reports healthy. The damage only surfaces on the
next restart, when it tries to read what it thinks it wrote.

Once the directory is manually cleared and the service restarted, the same
volume works correctly and survives subsequent restarts, including the
detach/attach cycle a rune restart performs. So this is specific to the
initial-provisioning window, not to ongoing use of the volume.

Evidence

Same cast, same configmap, same image (ghcr.io/floruntime/flo:0.1.0-dev.9),
changing only storageClassName:

storage	SYSTEM after first boot	outcome
pd-ssd-euw2a (gce-pd), freshly provisioned	0 bytes	dies on next boot
local, freshly provisioned	90 bytes	fine
pd-ssd-euw2a, after manual wipe + sync	90 bytes	fine, survives restarts

Control ruling out the workload: the same image with the same config file, run
directly under docker on the same host with a /tmp bind mount, writes 90
bytes and survives both docker stop (SIGTERM) and docker kill (SIGKILL) with
data intact. It is crash-consistent.

Reproduction

rune storageclass create -f storageclass-pd-ssd-euw2a.yaml   # once

# any service with a claimTemplate on that class that writes a file on boot
rune cast flo.yaml -n shared            # storageClassName: pd-ssd-euw2a
rune exec flo -n shared --no-tty -- stat -c '%s' /data/flo/SYSTEM
#   -> 0          (expected 90)

rune restart flo -n shared
rune logs flo -n shared --tail=5
#   Error starting runtime: error.UnexpectedEndOfInput      (crash-loops)

Swap to storageClassName: local and the same step returns 90.

Recovery — note that rune volume delete is not sufficient (see below):

rune stop flo -n shared
# on the host:
sudo rm -rf /var/lib/rune/mounts/flo-data-flo-0/{00000,SYSTEM,flo.pid} && sync
rune scale flo 1 -n shared
#   -> SYSTEM = 90 bytes, and restarts now work

On-disk state at failure

$ mount | grep flo-data
/dev/sdc on /var/lib/rune/mounts/flo-data-flo-0 type ext4 (rw,relatime)

$ sudo ls -la /var/lib/rune/mounts/flo-data-flo-0
drwxr-xr-x 4 ubuntu ubuntu  4096 Aug 29 00:00 00000
-rw-r--r-- 1 ubuntu ubuntu     0 Aug 29 00:00 SYSTEM      <-- 0 bytes
-rw-r--r-- 1 ubuntu ubuntu     0 Aug 29 00:00 flo.pid     <-- 0 bytes
drwx------ 2 ubuntu ubuntu 16384 Aug 29 00:00 lost+found

Mount options are unremarkable and identical to the MongoDB volume on the same
StorageClass. A disk snapshot of the failed state is preserved as
flo-corrupt-repro (source disk rune-shared-flo-data-flo-0) and can be shared.

What we have NOT established

Being explicit, because an earlier draft of this report guessed wrong: we first
suspected the volume detach was discarding unflushed page cache. That is
disproven — after the manual wipe, the same volume survives rune restart
(which detaches and reattaches) with contents intact.

The remaining suspicion is the provisioning path: format (mkfs.ext4),
mount, and hand-off to the container, where the earliest writes appear not to
reach the device. We have not instrumented that path and cannot say more from
outside. Worth checking whether the container is allowed to start before
mkfs/mount has fully settled, and whether there is a sync between format
and first use.

Secondary observations

1. rune volume delete does not give a clean slate under reclaimPolicy: retain. Deleting the volume row and letting the claimTemplate reprovision
   re-binds the same underlying GCE disk with its old contents, so the obvious
   recovery silently returns you to the broken state. This cost real debugging
   time. A warning on the delete path, or docs, would help.

2. rune cast on a stateful service can lose the PD attach race. Cast
   deletes and recreates the instance; the new one intermittently fails with
   LaunchFailed: failed to resolve volume mount "flo-data": context canceled
   while the old one is still detaching. The volume stays Available and bound
   and rune restart clears it, so nothing is lost — but a retry/backoff on
   attach would avoid an alarming failure mode. Roughly 1 in 3 casts.

3. rune volume delete has no --yes while rune release delete requires
   one. Minor inconsistency.

[oreofeolurin]

Confirmed, diagnosed and fixed in #271. Your suspected mechanism was exactly right.

Reproduced your on-disk state. On a loop-backed ext4: create the files, let the journal commit, write their contents without fsync, then snapshot the backing device — which is what a detached disk contains:

-rw-r--r-- 1 root root     0  SYSTEM
-rw-r--r-- 1 root root     0  flo.pid
drwx------ 2 root root 12288  lost+found

Same signature, down to lost+found. With a syncfs first, SYSTEM comes back at 90 bytes.

Why nothing flushed. umount(2) flushes on its own, so the happy path was safe. The loss is on the paths where the unmount does not happen — and the agent detaches anyway rather than strand a volume on a node that is going away:

1. The unmount fails with EBUSY because a container still holds the bind. Containers outlive runed, so restarting runed unmounts volumes out from under live containers.
2. The unmount is skipped. Every driver probed with findmnt via exec.CommandContext first, and exec cannot start a process on an expired or cancelled context — that failure was read as "not mounted", so Unmount returned success having done nothing. Teardown runs during shutdown, which is exactly where contexts expire.

Scope is wider than gce-pd. All four cloud drivers (do-volume, gce-pd, aws-ebs, hcloud-volume) carried byte-identical copies of that routine, so all four lose data the same way. local is unaffected, which matches your controlled comparison — nothing is detached there, so nothing discards the cache.

Your inference about the blast radius holds too: any service relying on ordinary write-back is exposed, and databases are protected only because they fsync their own journals. MongoDB on the same class is fine for that reason.

Fix: a shared mountsync package that calls syncfs(2) on the mount point before umount(2), and drops the findmnt probe entirely so a dead context can no longer skip the unmount. syncfs rather than sync so one volume's teardown does not stall every filesystem on the node inside a shutdown budget.

One thing #271 does not fix, worth tracking separately: nothing coordinates container teardown with volume teardown. runed shutdown will still unmount volumes while containers hold them and still fail with EBUSY — the data is now safe, but the volume is detached from under a running container. That needs a design answer rather than a patch.

Thank you for the plain-docker control and the host-side ls. The zero-length-with-correct-metadata detail is what identified the mechanism; without it this would have looked like an application bug.

[oreofeolurin]

Correction to my earlier comment. I said "your suspected mechanism was exactly right". That was wrong, and I apologise — I read a version of this report from before you added What we have NOT established, and answered a theory you had already retracted. You disproved detach-discards-cache yourself, and I asserted it back at you.

Where that leaves things:

#271 no longer says Fixes #270. It says Refs, so merging it will not close this. Your bug should stay open until someone reproduces the first-boot window on a gce-pd node.

What #271 does fix is real, and worse than I first described. Reviewers found my own explanation was backwards. umount(2) flushes only when it drops the last reference to the superblock — and a container holds a second one, because the runtime binds the mount into its own namespace. Measured on loop-backed ext4, 90 bytes written without fsync, reading back the raw device:

single mount, bare umount(2)         rc=0   90 bytes
second mount held, bare umount(2)    rc=0    0 bytes

So the agent's unmount returns success, flushes nothing, logs Volume unmounted, and detaches. That is a genuine data-loss path on every cloud driver, and it is now fixed and covered by a test that fails against the current code.

Whether it is your bug is unproven. There is a story where both are true — first-boot writes are seconds old and still dirty, while by a later rune restart writeback has already flushed them, which would explain why the wipe made it survivable — but that is an argument, not evidence, and I am not going to close your issue on it.

Your provisioning suspicion is the live lead and I have not investigated it yet: mkfs.ext4 → mount → hand-off, and whether the container is allowed to start before the mount has settled, or whether anything syncs between format and first use. That is the next thing to look at.

Two of your secondary observations look worth their own issues regardless of this — rune volume delete silently rebinding the same underlying disk under reclaimPolicy: retain (a recovery path that returns you to the broken state is its own bug), and the cast/attach race. Say the word and I will file them, or file them yourself if you would rather keep the detail.

Thank you for the correction in the report itself — and sorry for not re-reading before commenting.

[oreofeolurin]

Reopening — this was closed automatically by the merge of #271, which is not what that PR intended.

The cause was my own wording: #271 deliberately said Refs #270 rather than Fixes, but it also carried a section heading reading "Does this close #270?". GitHub parses close #270 as a closing keyword wherever it appears in the description, including inside a question written to say the opposite. The heading has been reworded.

To restate what #271 does and does not do: it fixes a real data-loss path in all four cloud volume drivers, where umount(2) returns success while flushing nothing because a container's bind mount holds a second reference to the superblock. That is not the mechanism you reported — you disproved detach-based loss yourself — so your bug remains open and unfixed. The provisioning path (mkfs.ext4 → mount → container hand-off) is still the live lead.