DIVE-3810: reload the box token when the control plane rejects it, and record the failure

[5dive-bot]

DIVE-3810 — pairing a phone rotated the box token and the dashboard channel kept the dead one

Pairing a phone rewrites /etc/5dive/connectord.env (shelld's /shell/rotate-token does the line
surgery) while the agent is running. plugins/dashboard/server.ts read that token once at
module scope, and it is the credential for all three control-plane calls — collect, ack, and the
agent's outbound reply. From the rotation onward the customer's dashboard chat was deaf and mute
until something restarted the agent. lodar hit it live twice on 2026-08-29 (glossy-flint): five
messages typed, silence, restart → 5 pending drained to 0 in 15 seconds.

The change

• TOKEN is mutable, and every control-plane call goes through one authedFetch. On a 401/403 it
  re-reads the file and retries once, and only if the token actually changed — a rejection that
  survives a reload is returned unchanged, so this cannot become a retry loop. Cost is one extra
  request on a request that was already failing; no watcher, no timer.
• The token path is CONNECTORD_ENV_FILE (default /etc/5dive/connectord.env) so the rotation is
  drivable by a test. An explicit CONNECTORD_TOKEN env override stays authoritative and is never
  reloaded — nothing rotates it.
• The failure now reaches a file on the box. The only previous signal was one stderr line that
  goes down the MCP stdio socket into the harness and is written nowhere. lifecycle.ts gains an
  auth event (a process that is healthy but 401ing is not start, exit or crash, and without
  an event of its own that state is recorded as nothing — the exact failure that file exists to
  refuse). One record per episode, one on recovery. All 8 copies of lifecycle.ts stay
  byte-identical; test/lifecycle.test.ts enforces it.
• The ack now checks res.ok. It previously printed healed N message(s) after a 401. Rows stay
  pending and redeliver exactly as before — only the log changes, from false to true. This does not
  touch DIVE-3809's ack/collect split, which is still owed.

How it was checked

test/dashboard-token-rotation.test.ts drives the real server.ts as a subprocess against a stub
control plane that 401s a stale bearer on every route, and rotates the token file underneath the
running process the way shelld does (line surgery, other lines kept):

1. inbound recovers with no restart — a message enqueued after the rotation is delivered, and
   the stub confirms it was rejected first (otherwise the test passes because the stale token still
   worked).
2. outbound recovers — the reply tool is called over MCP stdio after the rotation and the
   control plane receives the body. The mute half is half the defect; a fix graded only on collect
   would leave the customer talking to a wall.
3. the record — with a token that is stale on disk and rejected, lifecycle.log gets exactly
   one auth line naming the state, a second nudge adds none (a 5-minute poll must not write one
   line per poll), and when the on-disk token catches up the recovery is recorded and the held
   message arrives.
4. all three call sites are asserted against the source to go through authedFetch, and the
   bearer header is built in exactly one place. No single scenario reaches all three, and arming one
   call site is not arming the caller.

• Full repo suite: 1064 pass / 0 fail across 38 files (includes the lifecycle byte-identical
  parity arm and dashboard-collect-now).
• Mutation control: with the reload+retry deleted from authedFetch, tests 1–3 go red
  (20.2s / 20.4s / 22.5s timeouts) and only the structural arm stays green. The behavioural arms are
  load-bearing.

Who else caches this credential (row item 3, answered)

• 5dive-api/scripts/inc/shelld.mjs — already correct: let TOKEN with an in-memory swap on
  rotate (shelld.mjs:21-23, and the TOKEN = newToken in the rotate handler). It is the precedent
  this change follows.
• 5dive-cli — hooks/push-notify.sh, src/task/notify.sh read the file per invocation; no cache.
• src/cmd_agent_create.sh:2494 installs the dashboard channel with an empty token on purpose
  ("the plugin reads the box's connectord token itself"), so nothing is copied into the agent's
  .env and there is no second stale copy to chase.
• Conclusion: this plugin was the only long-lived reader that cached it.

Still open, deliberately not in this diff

• Should pairing rotate this token at all? (row item 4.) If the rotation is incidental, not
  rotating is smaller than making every consumer reload-safe. That is a control-plane call and it is
  independent of this fix — reload-safety is correct either way.
• Blast radius is unmeasured from this seat — no prod DATABASE_URL here. Boxes paired before
  this ships stay deaf until their agent restarts; restarting re-reads the file and is a safe interim
  sweep needing no deploy. The triage tell: healthy box, messages sitting in /pending with text
  intact, delivered_at NULL.

Iteration 2 — the version bump this fix needs to reach a box

The install path is version-keyed: a seat caches a plugin at
~/.claude/plugins/cache/5dive-plugins/<plugin>/<version>/. Merged unbumped, this change resolves
to already-installed, fetches nothing, and every deaf box stays deaf. Every prior dashboard
behaviour change bumped this file (0.1.0 -> 0.2.0 -> 0.2.1 -> 0.3.0 -> 0.4.0 -> 0.4.1) and no CI
gate catches its absence.

• plugins/dashboard/.claude-plugin/plugin.json 0.4.1 -> 0.4.2.
• CHANGES.md gains the matching ## Unreleased entry, naming that version.
• The other seven plugins are deliberately left at their current versions. Their whole diff is
  the LifecycleEvent union widening plus its comment - a type-only change with no runtime
  behaviour and no emitted auth record - so there is nothing for a bumped fetch to deliver. All
  eight copies of lifecycle.ts remain byte-identical (that arm still passes).

The .env override residual, stated in the code

server.ts's STATE_DIR/.env loader copies CONNECTORD_TOKEN into process.env before
loadConnectordToken runs, and an env-set token is deliberately never reloaded - so a seat holding
a non-empty CONNECTORD_TOKEN in ~/.claude/channels/dashboard/.env is permanently back in the
original bug with no signal. No live box is in that state: agent-create passes the dashboard channel
an empty token on purpose (5dive-cli/src/cmd_agent_create.sh, DIVE-841 arm), empty is falsy so it
falls through to the file, and the file is what pairing rotates. Nothing on the provision path
writes that key into the channel .env at all. Making the .env copy rotatable is a separate
decision; this is now written into the comment above loadConnectordToken rather than left implicit.

Full suite re-run on the new head: 1064 pass / 0 fail; test/dashboard-token-rotation.test.ts
4 pass / 0 fail.

🤖 Generated with Claude Code