Daemon silently stands down after reboot when its recorded PID has been recycled (0.2.10-dev.83)

[jaded0]

Summary

After a reboot, happier daemon start-sync --takeover can conclude that another daemon
already owns the machine, exit 0, and leave the machine silently absent from the app.

The daemon records its PID in ~/.happier/servers/<relay-id>/daemon.state.json and a bare PID
in daemon.state.json.lock. Neither is reliably cleared on shutdown, so after a reboot the
recorded PID is a boot-era PID — and the kernel hands that same low PID to an unrelated
session service on the next boot. The takeover check only asks whether the PID is alive, so it
defers to dbus-daemon or wireplumber and stands down.

Two machines, same day, two different branches of the same bug:

Host	Stale PID	What actually held it after the reboot	Daemon output
workstation	1595	dbus-daemon --session	[CONTROL CLIENT] Refusing to force-kill PID 1595 (does not look like a happier daemon process)
home server	1262	wireplumber	Daemon already running

Note the asymmetry: the force-kill path does check /proc/<pid>/cmdline and correctly
refuses to kill dbus — then gives up rather than concluding the state file is garbage. The
"already running" path never checks at all.

The exit code is the worse half

Both paths exit 0. The generated systemd unit ships Restart=on-failure, so systemd logged
status=0/SUCCESS and never retried. systemctl --user status showed:

Active: inactive (dead) since Sun 2026-08-30 09:56:04 MDT; 12h ago
Duration: 1.017s
Main PID: 1586 (code=exited, status=0/SUCCESS)

No error, anywhere. The only tell is the one-second duration. Both machines sat unavailable
for 6–12 hours before anyone noticed the app was missing them.

Incidentally, renderSystemdServiceUnit defaults to restart: params.restart ?? 'always',
but the daemon service definition passes on-failure — so the supervisor safety net is
disabled in exactly the case where it would help.

Reproduction (no reboot required)

Any live PID that is not a happier process will do; PID 1 is convenient.

systemctl --user stop happier-daemon.default.service
D=~/.happier/servers/<relay-id>
printf '{\n  "pid": 1,\n  "httpPort": 40000\n}\n' > "$D/daemon.state.json"
echo 1 > "$D/daemon.state.json.lock"
systemctl --user start happier-daemon.default.service

systemctl --user is-active happier-daemon.default.service   # inactive
# journal: "Daemon already running" (or the force-kill refusal), exit 0, no retry

Why this is invisible in testing and near-deterministic in the field

The failure needs the recorded PID to be a boot-era PID, which only happens when the
daemon's last start was at boot. Daemon log filenames carry the PID, which makes the pattern
easy to see across three machines running the same build:

Host	PIDs the daemon recorded	Reboot cadence
shared lab box	2072 (at boot), then 37869, 75890, 306790, 378023	~monthly
workstation	1586, 1595, 1599, 1602	nightly
home server	1258, 1262, 1396	3× in one day

A long-uptime machine restarts the daemon mid-session (dev-channel upgrades) and ends up
recording a five- or six-digit PID. Nothing reaches 75890 during early boot, so the liveness
check correctly answers no and the stale state is discarded. That machine has never once hit
this bug.

Machines that reboot often record a boot-era PID instead, and those are close to
deterministic: one host's daemon drew 1586, 1595, 1599 and 1602 across four separate boots
— a 16-wide window, because systemd starts the same user units in the same order every time.
So last boot's PID is near-certain to be reissued to some neighbour in this boot's startup
cluster. It isn't bad luck; it's structural.

That's likely why this is under-reported: dev machines and CI containers with long uptimes
never recycle a boot-era PID, while users whose machines reboot daily hit it constantly.

An unclean shutdown is not required. One of the two hosts above was cleanly shut down
every night (last -x shows a proper shutdown system down before each boot). Its
daemon.state.json was removed correctly, but daemon.state.json.lock survived carrying a PID
from the previous morning's boot — and the lock alone is enough to trigger this.

Suggested fixes

1. Validate the process, not just the PID, in the "Daemon already running" path — the
   /proc/<pid>/cmdline check the force-kill path already performs. On mismatch, treat the
   state as stale, clear it, and continue starting.
2. Exit non-zero when standing down, so a supervisor can act. Possibly restore the
   template's own always default for the daemon unit.
3. Clear daemon.state.json.lock alongside the state file, including on graceful shutdown
   — it currently outlives clean stops and is independently sufficient to cause this.
4. Optionally record a boot id or process start time next to the PID; comparing that against
   /proc/<pid>/stat field 22 makes recycled PIDs unambiguous.

Workaround

A systemd ExecStartPre that runs the missing check before the daemon can misread the file.
Deliberately conservative — it leaves the state alone when a real daemon owns it, since
daemon.state.json carries that daemon's controlToken and httpPort, and deleting it would
orphan a live daemon on the relay under the same machineId:

looks_like_happier() {   # true only if the PID exists AND its cmdline names happier
    [ -r "/proc/$1/cmdline" ] || return 1
    case "$(tr '\0' ' ' < "/proc/$1/cmdline")" in *happier*) return 0;; *) return 1;; esac
}
# per server dir: read pid from daemon.state.json, else from the .lock;
# if ! looks_like_happier "$pid"; then rm -f daemon.state.json daemon.state.json.lock; fi
# always exit 0

With that plus Restart=always, both machines now recover from the planted-stale-state repro
above and re-register with the relay in about a second.

Environment

• CLI 0.2.10-dev.83, release channel dev
• Self-hosted relay happierdev/relay-server:dev-cce689463cf0 (server v0.2.10-dev.76),
  light flavor, sqlite
• Ubuntu 24.04, systemd user services with lingering enabled
• Observed 2026-08-30

Note: this was diagnosed from the shipped dev bundle rather than from source, so I can't point
at the exact lines — code search suggests apps/cli/src/daemon/, though the
Daemon already running string only surfaced for me in apps/stack/scripts/daemon.mjs.

---

Filed by an AI agent (Claude) working in my homelab, on my behalf and with my review.

[happier-bot]

Thanks @jaded0 for the detailed reboot reproduction. In particular, identifying that the recorded PID could now belong to dbus/wireplumber, and suggesting that startup distinguish an unrelated process from an unreadable one, directly helped isolate the ownership bug.

The root cause was confirmed: daemon lifecycle callers collapsed both a definitively unrelated process and an uninspectable process into the same null result. That made stale state and lock records remain fail-closed when the PID had actually been recycled.

The fix adds one canonical daemon lifecycle classification (daemon / not-daemon / unknown) and routes both daemon-state inspection and lock arbitration through it. A proven unrelated PID is now replaceable without being signaled; unknown ownership remains fail-closed; and an authenticated daemon ping remains stronger positive evidence.

Integrated source commits:

• dev/0.2: 5648fb5
• v0.3: 5c159a0

The real-process regression reproduced the incorrect starting verdict before the fix and passes afterward. The adjacent PID-safety, lock/startup, process-discovery, and force-stop suites also pass. The issue is now at source availability; because this was reported on the dev channel, please retry after the label advances to stage:dev. If it still reproduces then, the exact dev version and the same state/lock/process details would be the most useful follow-up.

cc: @leeroybrun