fix: terminal execution channel leaks and lost command output - #1564
Open
cplieger wants to merge 1 commit into
Open
fix: terminal execution channel leaks and lost command output#1564cplieger wants to merge 1 commit into
cplieger wants to merge 1 commit into
Conversation
Three lifecycle defects in terminal execution: - Core's ReceiverStream had no Drop, so a stream dropped before END_OF_OUTPUT left its entry in connection.terminals forever, and Periphery kept forwarding to a channel Core no longer knew about. Cleanup now also tells Periphery to stop, reusing the message the interactive terminal already sends on client disconnect. - ExecuteTerminal never registered a TerminalChannel or cancellation token, unlike ConnectTerminal, so nothing could stop its forwarding task. It now mirrors the sibling, ends the stream when the terminal dies, and bounds the wait for Core's begin trigger. - run_command dropped the wait_with_output future when the timeout or cancel branch won, discarding everything the command had printed. The future is now re-awaited after the kill, capped per stream.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three independent lifecycle defects in terminal execution, found while investigating an Action on my own install that wedged twice in 26 hours.
1. Core leaks the response channel when a stream is dropped
ReceiverStream::cleanuponly runs frompoll_next, onEND_OF_OUTPUTor a closed channel. There is noDrop, so a stream dropped part way through (client disconnect, cancelled Action) leaves its entry inconnection.terminalsforever while Periphery keeps forwarding to it.Adds
Drop, and tells Periphery to stop using the same message the interactive terminal already sends on client disconnect, so the existingterminal_channels().removepath cancels the forwarder. No new wire message. Also removes the map entry when theBeginsend fails, since on that path no stream exists to drop.2.
ExecuteTerminalnever registered for cancellationConnectTerminalregisters aTerminalChannelwith aCancellationTokenand selects on it.ExecuteTerminaldid neither, so nothing in the process could stop its forwarding task and it outlived Core's knowledge of the channel. This mirrors the sibling: register the channel, select on both tokens, deregister on every exit.The loop also now ends the stream when the terminal dies rather than leaving Core's body open, bounds the wait for the
Begintrigger, and removes the trigger entry when that wait gives up (TerminalTriggers::recvonly removes it on success).3. A killed command's output was thrown away
run_commanddrops thewait_with_output()future when the timeout or cancel branch wins, so everything already printed is lost andfrom_err_messagereportsstdout: "". The future is now pinned and re-awaited after the kill, capped at 1 MiB per stream so a runaway producer cannot outgrow the update document. This is what makes a timeout produce a usable log instead of one line of error text, and it applies to every existing caller that passes a timeout or cancel token, not just Actions.What this does not do
It does not fix the
No response channel foundspam in #1392. That is Periphery's request keepalive against Core'sresponsesmap, which is the per-message RPC timeout never becoming a total one, i.e. your own second item on that issue. Untouched here, as is the absence of an alert for an update stuckInProgress.Two unbounded waits also remain out of scope: the
START_OF_OUTPUTwait insetup_execute_command_on_terminal, and the trigger wait inhandle_terminal_forwarding.One behaviour change worth calling out: on an abnormal end the execute stream now closes with
END_OF_OUTPUTand no exit code, which your TS client already reports asEarly exit without code, rather than an error item that aborts the HTTP body. The reason is also emitted as a line of output, because reacting to the missing exit code is optional for a caller and otherwise a truncated command can read as a complete one.Verification
cargo build,cargo testandcargo fmt --checkclean. Two new tests, each confirmed to fail without its fix:timeout_preserves_output(lib/command) fails withstdout: "".cancelled_terminal_still_forwards_buffered_output(bin/periphery) delivers 1 of 302 messages with the cancellation check at the top of the loop, and 104 of 302 without theunconstrainedon the read. A cancelled token is not budget-aware, so mid-drain it beats a read that has spent the task's cooperative budget, and takes the exit code with it.The Core and Periphery changes have no runtime test against a live pair; I have no second install to break. They rest on the code, on the symptoms above, and on that one unit test.
Separately, #1563 adds a user-set execution timeout for Actions. That is a control rather than a fix, and it depends on the third change here to produce a usable log when it fires.
Happy to split this into the
lib/commandhalf and the terminal half if you would rather review them separately.