Skip to content

fix: terminal execution channel leaks and lost command output - #1564

Open
cplieger wants to merge 1 commit into
moghtech:mainfrom
cplieger:fix/terminal-execution-leaks
Open

fix: terminal execution channel leaks and lost command output#1564
cplieger wants to merge 1 commit into
moghtech:mainfrom
cplieger:fix/terminal-execution-leaks

Conversation

@cplieger

@cplieger cplieger commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Three independent lifecycle defects in terminal execution, found while investigating an Action on my own install that wedged twice in 26 hours.

1. Core leaks the response channel when a stream is dropped

ReceiverStream::cleanup only runs from poll_next, on END_OF_OUTPUT or a closed channel. There is no Drop, so a stream dropped part way through (client disconnect, cancelled Action) leaves its entry in connection.terminals forever while Periphery keeps forwarding to it.

Adds Drop, and tells Periphery to stop using the same message the interactive terminal already sends on client disconnect, so the existing terminal_channels().remove path cancels the forwarder. No new wire message. Also removes the map entry when the Begin send fails, since on that path no stream exists to drop.

2. ExecuteTerminal never registered for cancellation

ConnectTerminal registers a TerminalChannel with a CancellationToken and selects on it. ExecuteTerminal did neither, so nothing in the process could stop its forwarding task and it outlived Core's knowledge of the channel. This mirrors the sibling: register the channel, select on both tokens, deregister on every exit.

The loop also now ends the stream when the terminal dies rather than leaving Core's body open, bounds the wait for the Begin trigger, and removes the trigger entry when that wait gives up (TerminalTriggers::recv only removes it on success).

3. A killed command's output was thrown away

run_command drops the wait_with_output() future when the timeout or cancel branch wins, so everything already printed is lost and from_err_message reports stdout: "". The future is now pinned and re-awaited after the kill, capped at 1 MiB per stream so a runaway producer cannot outgrow the update document. This is what makes a timeout produce a usable log instead of one line of error text, and it applies to every existing caller that passes a timeout or cancel token, not just Actions.

What this does not do

It does not fix the No response channel found spam in #1392. That is Periphery's request keepalive against Core's responses map, which is the per-message RPC timeout never becoming a total one, i.e. your own second item on that issue. Untouched here, as is the absence of an alert for an update stuck InProgress.

Two unbounded waits also remain out of scope: the START_OF_OUTPUT wait in setup_execute_command_on_terminal, and the trigger wait in handle_terminal_forwarding.

One behaviour change worth calling out: on an abnormal end the execute stream now closes with END_OF_OUTPUT and no exit code, which your TS client already reports as Early exit without code, rather than an error item that aborts the HTTP body. The reason is also emitted as a line of output, because reacting to the missing exit code is optional for a caller and otherwise a truncated command can read as a complete one.

Verification

cargo build, cargo test and cargo fmt --check clean. Two new tests, each confirmed to fail without its fix:

  • timeout_preserves_output (lib/command) fails with stdout: "".
  • cancelled_terminal_still_forwards_buffered_output (bin/periphery) delivers 1 of 302 messages with the cancellation check at the top of the loop, and 104 of 302 without the unconstrained on the read. A cancelled token is not budget-aware, so mid-drain it beats a read that has spent the task's cooperative budget, and takes the exit code with it.

The Core and Periphery changes have no runtime test against a live pair; I have no second install to break. They rest on the code, on the symptoms above, and on that one unit test.

Separately, #1563 adds a user-set execution timeout for Actions. That is a control rather than a fix, and it depends on the third change here to produce a usable log when it fires.

Happy to split this into the lib/command half and the terminal half if you would rather review them separately.

Three lifecycle defects in terminal execution:

- Core's ReceiverStream had no Drop, so a stream dropped before
  END_OF_OUTPUT left its entry in connection.terminals forever, and
  Periphery kept forwarding to a channel Core no longer knew about.
  Cleanup now also tells Periphery to stop, reusing the message the
  interactive terminal already sends on client disconnect.

- ExecuteTerminal never registered a TerminalChannel or cancellation
  token, unlike ConnectTerminal, so nothing could stop its forwarding
  task. It now mirrors the sibling, ends the stream when the terminal
  dies, and bounds the wait for Core's begin trigger.

- run_command dropped the wait_with_output future when the timeout or
  cancel branch won, discarding everything the command had printed.
  The future is now re-awaited after the kill, capped per stream.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant