Problem
outfit remote has three lifecycle verbs — start, pause, stop — with a
meaningful split the docs already teach: pause is done for now (stopping the
instance without terminating it, so a later start re-wakes it; the control
plane sweeps it only once it has been stopped beyond the retention window),
while stop is done for good (terminate; the next start launches fresh and
re-seeds).
The daemon has no third verb. A kind: remote fleet node can be paused
(remoteNode maps onto the control plane), but a kind: daemon node offers
only start and stop, so the fleet surface has no honest pause for its local
machines. The interactive dashboard (#59) makes this concrete: it drives
start/pause/stop from the keyboard on the selected node, with pause and stop
each asking for confirmation — and on a daemon node those two confirmations
would today be answering the same question.
Proposal
Give the daemon a pause that stops the engine but keeps the daemon
running: the node stays in the fleet, reachable on its control API, and a
start brings the engine back the way it does today. The fleet surface then
exposes the same trio for every node kind:
fleet pause <node> beside start/stop, one node at a time like them.- A
Pause on the fleet.Node contract, implemented by both kinds:
remoteNode onto the existing remote.Pause, daemonNode onto the daemon
control API (a new POST /v1/pause — a line in Routes(), a handler, a
path in docs/openapi.yaml — or onto the existing stop endpoint, depending
on the semantics decided below).
- Typed outcomes for a refused pause, consistent with the rest of the node
operations, so an unreachable or paused node is a row, not a failure.
Things to decide
- What separates pause from stop on a daemon node. Today's daemon
stop
already stops the engine and keeps the daemon running — the behaviour asked
for here — so the pair only differs if they gain different semantics. The
cloud contrast to lean on is done-for-now vs done-for-good: pause
preserves the warm state (weights on the boot disk; a start re-wakes),
stop tears down (terminate; re-provision and re-seed). Candidates for the
daemon split:
- pause keeps the node's resume-able state (stored deploy config and
engine key intact; a bare start re-wakes, the same way today's stop
leaves the node able to start again), and stop becomes the colder verb —
engine stopped and stored config and engine key discarded, so a
restart must be pushed again, the daemon-side mirror of a terminated
environment; or
- pause ≡ today's stop on the daemon (no daemon change at all; the
contract and command map daemon pause onto the existing stop endpoint),
keeping the done-for-now/done-for-good split for remote nodes only.
Option 1 mirrors the semantics the CLI already teaches and gives the
dashboard's two confirmations two truths; option 2 is smaller but leaves
pause and stop doing the same thing on half the fleet.
- Can "stop the engine" mean anything softer than killing the process?
llama.cpp has no in-process unload, so keeping the weights loaded while
paused is not expressible by the engine today; the daemon would at most be
able to mark a running node paused so routing stops choosing it. Noted so
the decision is made with the substrate in hand — the default assumption is
that pause terminates the engine process, exactly as stop does.
- Sweep. The control plane terminates a paused instance past the retention
window; a daemon node has no sweep — a paused daemon's engine is simply gone
and the operator decides when (or whether) it returns, matching the daemon's
report-and-never-act posture.
Notes
Problem
outfit remotehas three lifecycle verbs —start,pause,stop— with ameaningful split the docs already teach:
pauseis done for now (stopping theinstance without terminating it, so a later
startre-wakes it; the controlplane sweeps it only once it has been stopped beyond the retention window),
while
stopis done for good (terminate; the next start launches fresh andre-seeds).
The daemon has no third verb. A
kind: remotefleet node can be paused(
remoteNodemaps onto the control plane), but akind: daemonnode offersonly start and stop, so the
fleetsurface has no honest pause for its localmachines. The interactive dashboard (#59) makes this concrete: it drives
start/pause/stop from the keyboard on the selected node, with pause and stop
each asking for confirmation — and on a daemon node those two confirmations
would today be answering the same question.
Proposal
Give the daemon a pause that stops the engine but keeps the daemon
running: the node stays in the fleet, reachable on its control API, and a
start brings the engine back the way it does today. The fleet surface then
exposes the same trio for every node kind:
fleet pause <node>besidestart/stop, one node at a time like them.Pauseon thefleet.Nodecontract, implemented by both kinds:remoteNodeonto the existingremote.Pause,daemonNodeonto the daemoncontrol API (a new
POST /v1/pause— a line inRoutes(), a handler, apath in
docs/openapi.yaml— or onto the existing stop endpoint, dependingon the semantics decided below).
operations, so an unreachable or paused node is a row, not a failure.
Things to decide
stopalready stops the engine and keeps the daemon running — the behaviour asked
for here — so the pair only differs if they gain different semantics. The
cloud contrast to lean on is done-for-now vs done-for-good:
pausepreserves the warm state (weights on the boot disk; a start re-wakes),
stoptears down (terminate; re-provision and re-seed). Candidates for thedaemon split:
engine key intact; a bare start re-wakes, the same way today's stop
leaves the node able to start again), and stop becomes the colder verb —
engine stopped and stored config and engine key discarded, so a
restart must be pushed again, the daemon-side mirror of a terminated
environment; or
contract and command map daemon pause onto the existing stop endpoint),
keeping the done-for-now/done-for-good split for remote nodes only.
Option 1 mirrors the semantics the CLI already teaches and gives the
dashboard's two confirmations two truths; option 2 is smaller but leaves
pause and stop doing the same thing on half the fleet.
llama.cpp has no in-process unload, so keeping the weights loaded while
paused is not expressible by the engine today; the daemon would at most be
able to mark a running node paused so routing stops choosing it. Noted so
the decision is made with the substrate in hand — the default assumption is
that pause terminates the engine process, exactly as stop does.
window; a daemon node has no sweep — a paused daemon's engine is simply gone
and the operator decides when (or whether) it returns, matching the daemon's
report-and-never-act posture.
Notes
Pauseonfleet.Nodeis a change to thefleet-clientcontract, not anew capability: both implementations update in the same change, and
daemon-apigains the endpoint if the new-route option is chosen.fleet view/fleet dashboard): drive the fleet, not just watch it #59 (interactive fleet dashboard — the motivating consumer, whichpauses with confirmation), Unify the remote and fleet clients behind one node driver #99 /
unify-remote-fleet-node(the change thatmade a remote environment a fleet node, and whose contract this extends).