Skip to content

DIVE-3752: install the orphan watchdog in every plugin, and fix the clause that could not fire - #45

Merged
lodar merged 3 commits into
mainfrom
dive-3752-orphan-watchdog
Aug 26, 2026
Merged

DIVE-3752: install the orphan watchdog in every plugin, and fix the clause that could not fire#45
lodar merged 3 commits into
mainfrom
dive-3752-orphan-watchdog

Conversation

@5dive-bot

@5dive-bot 5dive-bot commented Aug 26, 2026

Copy link
Copy Markdown
Collaborator

DIVE-3752 — the orphan watchdog, installed everywhere, with its dead clause replaced

Items 1–3 of the row. Item 4 (rung-4 poller-dead restart) is in 5dive-cli and is filed
separately — see the bottom.

1. Installed in one plugin, needed in eight

DIVE-3486 compiled "bun run does not forward SIGTERM, so MCP servers orphan" on 2026-08-16 and
plugins/telegram/server.ts received the remedy. Nothing else did. Audited all eight plugins whose
server.ts ends in a long-lived timer:

plugin SIGTERM/INT SIGHUP stdin EOF ppid clause actually exits
telegram yes yes yes yes, but dead (§2) yes
buzz no no no no no
dashboard no no no no no
telegram-agy / -codex / -grok yes no yes no yes
telegram-pi / -opencode yes no no no noshutdown() stops the bot and returns

buzz is the measured leak: 22 reparented pollers on one seat, oldest six days old, all reaped by a
plain kill -TERM. Nothing was wedged — they were healthy processes nobody had asked to stop, which
is the signature of a missing handler rather than a hung poll. agent-don is a within-seat control:
same host, same launcher shape, ~22 restarts, both plugins spawned by the same parent — telegram 0
orphans, buzz 22. The variable is the plugin's own shutdown code.

All eight now install the same lifecycle.ts.

Why the file is duplicated per plugin directory: .claude-plugin/marketplace.json publishes
./plugins/<name> as the unit and Claude Code caches it at
<cache>/5dive-plugins/<plugin>/<version>/, so an import reaching outside the plugin directory
resolves in this repo and not on a customer box. Duplication is forced; drift between the copies
is not — a test asserts all eight are byte-identical.

2. The clause the watchdog existed for could not fire

Porting the snippet surfaced a defect in the original. Its ppid comparison is there on the stated
grounds that "stdin events don't reliably fire when the parent chain is severed" — so it is the
clause that covers the severed-parent case specifically. Under Bun it cannot fire. Measured on
this host: a grandchild whose parent was SIGKILLed, sampled every 500ms:

t=2s    cached=54284  realPpid=54284  bootAlive=true
t=2.5s  cached=54284  realPpid=1      bootAlive=false   <- parent killed here
t=5.5s  cached=54284  realPpid=1      bootAlive=false

process.ppid is captured at boot and never refreshed, so an orphan compares a dead pid against
itself forever. ps showed the real ppid was 1 the whole time. Telegram's zero-orphan record came
from its stdin handlers, not from that comparison
— which also means the wiki page crediting the
ppid clause for it is wrong on the mechanism, and that correction is compiled alongside this.

The module reads PPid: from /proc/self/status, with "is the boot parent still a process at all"
(kill(pid, 0), treating EPERM as alive) as the fallback for an unreadable /proc. Both readings
flipped at the instant of severance above. telegram's inline block is replaced rather than kept:
a broken implementation is not a proven one.

3. The && deafener, and giving the launcher a voice

"start": "bun install --no-summary && bun server.ts" in telegram, buzz and dashboard. Measured
under the launcher's own shell (bun run --shell=bun):

false && echo POLLER_RAN   ->  exit=1, nothing printed
false;  echo POLLER_RAN    ->  POLLER_RAN, exit=0

So a network-dependent step in front of a channel start is a deafener, and on 2026-08-26 three seats
including the coordinator were deaf for 2h33m with 9 human gates pending while the only signal
available to a human or to the DIVE-1434 canary was an absent heartbeat — which cannot say which
of three failures produced it.

The install is kept, not dropped. The row suggested dropping it on the grounds that
node_modules is vendored in every seat's plugin cache. It is vendored in the cache (92–99
entries on this seat) but it is gitignored in this repo, so deleting the install risks breaking
the first channel start on a cold box. && becomes ;: the install can still populate a cold
cache, and can no longer take the poller with it.

The remaining silence was server.ts throwing on import. A new start.ts is the entry point; it
imports node builtins and lifecycle.ts only, so it still loads — and can still write a record — in
exactly the case server.ts cannot. Start / exit / crash records with reasons land in
<state-dir>/lifecycle.log.

Honest limit: if Claude Code never spawns the launcher at all, nothing of ours runs and nothing
of ours can record it. That state is un-recordable from inside a plugin and stays outside this diff.
What is new is that "the poller is absent" now separates into a record exists with a reason versus
no record at all.

Checks

bun test933 pass, 0 fail, 3268 assertions across 35 files on the merged tree, including the
pre-existing test/parity.test.ts fork-drift suite.

test/lifecycle.test.ts adds 35 tests in three arms, because a single arm would have been vacuous
in a way this repo has been bitten by before:

  1. unit — the orphan decision and the record format, executed. Pure, so repo CI's bare
    bun test (no plugin deps installed) can actually run them.
  2. static — every plugin has the installLifecycle({...}) call, not merely the import: a
    pure module nobody calls is not installed. Comments are stripped before every assertion,
    because a bare "does it have a process.on" grep is satisfiable by a comment. Includes a
    regression arm that no plugin compares process.ppid to a boot snapshot again, and that no start
    script puts && in front of the server.
  3. end-to-end — spawns a real grandchild behind a real shell with a held-open stdin pipe (an
    ignored stdin would hit the EOF clause instantly and pass for the wrong reason), proves it is
    alive and still heartbeating a second later, then SIGKILLs the parent and requires the child
    to be gone.

Mutation control, because "it exited" is a claim that passes trivially: restoring the old
process.ppid clause makes the end-to-end arm sit alive for its full 15-second window and fail —
6 assertions pass first, so the positive control is what holds and the failure is the child's
survival. With the /proc read it exits in under a second.

The forks are generated, and the first push was red

bun generator/generate.ts --check is the parity workflow's second step, and I had run only
bun test locally. It reds the first push:

△ telegram-agy:  1 file(s) drift from committed:  only-in-committed  lifecycle.ts
△ telegram-grok: 1 file(s) drift from committed:  only-in-committed  lifecycle.ts

telegram-agy and telegram-grok are generated from the telegram base, so a new shared module
that is not in the generator's COPY_FILES exists in the committed fork and not in the generated
one. Fixed at the generator: lifecycle.ts joins tna.ts and banner.ts in the byte-exact
copy set — not name-swept, because the sweep would rewrite telegram inside the module's own
comments and break the byte-identity the parity arm asserts, and because those comments cite
plugins/telegram/server.ts by path as where the dead clause came from. A swept copy would claim
that history happened in a fork it never happened in.

The two gates now cover each other: the drift gate catches a fork the generator forgot, and the
parity arm's existsSync + byte-identity assertions catch a generator that drops the module.

Re-checked on the merged tree, unpiped exit codes: bun test 933 pass / 0 fail, exit 0;
generate.ts --check byte-exact on both forks, exit 0.

Versions

telegram 0.5.48 → 0.5.49, buzz 0.1.1 → 0.1.2, dashboard 0.4.0 → 0.4.1. A plugin change that
does not bump reaches no seat. The five telegram forks are not in marketplace.json and are not
version-distributed, so they are not bumped.

Not in this PR

Item 4 — let rung-4 poller-dead restart the seat — lives in 5dive-cli's recovery ladder
(src/cmd_supervisor.sh), whose verb set is nudge|resume|rotate|escalate|defer with no
restart verb to switch on
. Adding one is a new fleet-wide remedy that restarts seats, which
wants its own row, its own rate limit and its own smoke rather than riding a plugins PR. Filed
separately with olivia's recommendation and grounds carried forward intact.

🤖 Generated with Claude Code

lodar added 3 commits August 26, 2026 07:37
…lause that could not fire

The watchdog DIVE-3486 compiled reached exactly one plugin. buzz ended in a bare
setInterval with no process.on, no stdin handler and no exit, and leaked one live
poller per restart for six days; dashboard had the same gap, and telegram-pi and
telegram-opencode never call process.exit at all. All eight plugins that end in a
long-lived timer now install the same lifecycle.ts.

Porting it surfaced a second defect in the original: Bun caches process.ppid at
boot and never refreshes it, so `process.ppid !== bootPpid` — the clause that
exists precisely because stdin EOF is unreliable when the parent chain is severed
— cannot ever fire. Measured: an orphan reported its dead parent's pid for six
seconds while ps showed ppid=1. Read PPid: from /proc/self/status instead, with
boot-parent liveness as the fallback.

Also de-silence the channel start: `bun install && bun server.ts` exits 1 and
never starts the poller when the install fails, which is how three seats went
deaf for 2h33m with nine gates pending. `&&` becomes `;` (kept, not dropped —
node_modules is gitignored, so a fresh box still needs it), and a new start.ts
records start/exit/crash to <state-dir>/lifecycle.log, including the case where
server.ts throws on import and nothing else could speak.
The five telegram forks are generated, and `bun generator/generate.ts --check`
is the parity workflow's second step. It reds the first push: a new shared module
that is not in COPY_FILES is "only-in-committed" for telegram-agy and
telegram-grok.

lifecycle.ts joins tna.ts and banner.ts in the byte-exact copy set. Byte-exact
rather than name-swept for two reasons: the sweep would rewrite `telegram` inside
the module's own comments and break the byte-identity the parity arm asserts, and
those comments cite plugins/telegram/server.ts by path as the place the dead ppid
clause came from — a swept copy would claim that history happened in a fork it
never happened in.

bun test 933 pass / 0 fail; generate.ts --check byte-exact on both forks.
quinn's iteration-1 grade: plugins/telegram/server.ts did not parse — the new
installLifecycle import had been inserted INSIDE the msglog import's brace list.
Every gate on the PR was green: bun test never imports a server.ts (CI has no
plugin deps), the generator is a text transform that never parses what it
copies, and parity is the repo's only workflow. Shipped, the flagship plugin —
bumped to 0.5.49 and distributed via marketplace.json — would have deafened
every telegram seat on the next channel start.

1. Move the import out of the brace list. All 8 plugin entry points now
   transpile clean.

2. Add the gate that was missing. test/entrypoint-parse.test.ts sweeps every
   .ts under plugins/ with `bun build --no-bundle`, which resolves no import
   specifier and so needs no node_modules — verified on a worktree with none in
   any of the eight plugin dirs. Plus a named `entry points parse` step in
   parity.yml so the failure has a name in the checks list.

   The gate sweeps EVERY file rather than the declared entry point, and that is
   load-bearing: `start` now names start.ts, which imports only node builtins
   and lifecycle.ts and parses fine while the server.ts it imports does not.
   Mutation control: with the defect restored, exactly 1 of the 100 tests fires,
   and it is the sweep arm — the "the file each plugin actually launches parses"
   arm stays green throughout.

   Positive control inside the test: it constructs this exact defect in a temp
   file and requires the parser to reject it, alongside the well-formed twin, so
   a green sweep grades the tree and not a broken harness.

Checked on the merged tree with unpiped exit codes: bun test 1033 pass / 0 fail
/ 3386 assertions across 36 files, exit 0; bun generator/generate.ts --check
byte-exact on both committed forks, exit 0 (the forks are unchanged — the moved
line lands where the generator's block deletion already left it).

No further version bump: 0.5.49 is unreleased, introduced by this same branch.

Wiki: community/wiki/a-green-suite-that-never-imports-the-entry-point-cannot-fail.md
@lodar
lodar merged commit bc5a520 into main Aug 26, 2026
2 checks passed
@lodar
lodar deleted the dive-3752-orphan-watchdog branch August 26, 2026 09:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants