fix(flows): stop reporting what a script prints, on both sides - #970
fix(flows): stop reporting what a script prints, on both sides#970hubgan wants to merge 10 commits into
Conversation
A `script:` step's stdout and stderr no longer reach a report. The recorder returns none, a replay returns none, and there is no parameter to ask for them. `flow-add-script` drops `log`/`logTruncated`, the step report drops `scriptLog`/`scriptLogTruncated`, and the CLI and the MCP content builder drop the blocks that rendered them. Both surfaces land in an agent's context, and a script is a plausible place for a connection string, a token or customer data to be printed — so a parameter would not have been a control either, the agent being the one who would set it. What a step answers with is its verdict and its `reason`: a script with something to say has to `throw` it or return it in `output`. The capture goes with the channel rather than being hidden behind it. Nothing accumulates the text, so the per-step cap, the run-wide budget, the V8 frame collapser and the streaming half of the secret scrub have no subject left and are gone; `scrubSecretValues` absorbs the chunk walk it was the only caller of. Redaction still covers the two texts a script can put in front of a caller: the failure it reports and the document it returns. Two things the streams still owe. stderr is scanned for V8's heap-exhaustion banner — the only account of why an abort was an out-of-memory rather than a plain signal — over a sliding window that keeps no text. And both pipes are still drained: a paused one fills at 64 KiB and blocks the child from ever reaching its own time limit, which turns a passing step into a timeout. The exit-code reason said "no output was captured" of the output document. With the console channel gone that reads as "the script printed nothing", so it now says which it means.
`HeapFatalWatch.end()` scanned what `StringDecoder.end()` returns. That is either the empty string or a replacement character for a trailing incomplete sequence, and `V8_HEAP_FATAL_RE` matches ASCII only, so the scan could not match anything the earlier per-chunk scans had missed. The method it replaced had work to do — it flushed the decoders and the frame collapser into the captured log — and kept its call site after the capture was removed. Nothing else changes: the banner is matched as each chunk arrives, and the settle wait above is what puts the last chunks in before the streams are destroyed.
Six tests whose names or comments said more than they proved: - The only guard on stdout draining was an integration test that failed by suite timeout, over a flood sized against this host's pipe buffer. The executor's own suite now has one: 4 MiB is past any host's buffer, and the step's own limit is what fails it. - The straddle test did not straddle. The clamp landed 14 characters before the secret, so both message assertions passed with `partialSecretTail` stubbed out. The cut now falls about nine characters into the value. - The nested-secret test did not exercise longest-first ordering: `HOST` begins 16 characters into the `URL` value, so the walk meets `URL` first whatever the sort order. It pins containment and a standalone occurrence instead, which the prefix test beside it does not cover. - The stdout-close test described a write after the end that the fixture never made. It says what the fixture does, and writes its document a turn later so a run that stopped at the close would fail it. - The reworded exit-code reason had no assertion of its own. - The recorder's chatty script flooded 112 KiB, below the threshold its comment claimed to cover.
`argent flow --help` still promised the output this branch removed: a directory run printed "each script step's output", which it no longer does. It now names what that run really prints, and the flow-yaml reference, which listed failures alone, says the same. Both halves are already pinned by flow-script-render.test.ts; only the prose had drifted. `reason` was described as a failed script speaking for itself. For three of the nine failure kinds it is; for the other six the runner is, and on a pass it carries the executor's notes — a time limit clamped to the host's maximum among them, which matters because the step is recorded with the limit that was asked for. Both tool descriptions and the recorder's skill reference say so. Four comments in the executor described the subsystem this branch deleted: the settle wait explained itself with the logs it used to order, the drained streams were said to keep no text when the heap watch holds a window, `HeapFatalWatch` claimed to be the only reader when `resume()` reads too, and its doc comment carried a bad rewrap. `scrubSecretValues` lost the warning about truncation when its streaming half went; a failure message is cut before it arrives, so the comment points at the mitigation that handles it.
Twenty-four comment blocks added while the script log subsystem came out said
what the code beside them already said. The executor's `HeapFatalWatch` doc
described a sliding-window regex matcher in sixteen lines, over a class whose
pattern constant already explains why it is coarse. The rest were in tests:
fixture history ("a `console.log` used to be that proof"), restatements of the
assertion below them ("Every round, in order - three of them, not one"), and
intent the `it(...)` title already carries.
Four blocks stayed, for what the code cannot say:
- `child.stdout?.resume()` reads as a no-op. Only the comment says a paused
pipe blocks the child from reaching its own time limit.
- The settle wait before the destroys: IPC and stderr share no order, so a
terminal message routinely beats the banner the watch is scanning for.
- The failure-ceiling fixture's arithmetic. Where `- 41` and the trailing run
put the cut inside the value is not derivable from the numbers.
- Why the late secret is pushed from a later turn of the loop: pushing in the
same turn lands before `runOne` reads the set, so the obvious simplification
would pass against an implementation that snapshots it.
Comments only. No executable line changed.
Five places had gained a paragraph saying Argent discards what a script prints: `features/flows.mdx`, `reference/flow-yaml.mdx` and three `argent-create-flow` references. A reader who never saw the log channel learns nothing from being told it is absent, and both tool descriptions had grown the same sentence on top of the field list they needed to lose. All five files go back to their base content. What stays is deletion, not announcement. `flow-add-script` drops the `log`/`logTruncated` paragraph and keeps its `outputJson` sentence; `flow-run` drops "Its stdout and stderr come back on the step report"; `argent flow --help` drops "each script step's output" from what a directory run prints. Each of those named a field or a line that no longer exists, so leaving it there would be a promise the tool cannot keep. The exit-code failure message loses the same announcement. What it owed the reader was one disambiguation — "no output was captured" reads as "the script printed nothing" once the console channel is gone — so it names the output document and stops there, rather than going on to report the absence of a channel the reader never had. The batch-mode rewrite in `flow-yaml.mdx` goes back too. That sentence was equally imprecise before this branch, so correcting it here was churn this change does not own.
6eef318 to
10a0fb6
Compare
There was a problem hiding this comment.
Pull request overview
Removes the “script stdout/stderr as a report/recording channel” from the flow script: step end-to-end (executor → step report → recorder result → CLI/MCP renderers), while keeping stream draining for correctness and retaining stderr scanning solely for V8 heap-OOM classification.
Changes:
- Drop log capture (
log/logTruncated,scriptLog/scriptLogTruncated) and the associated per-step/run-wide log budgeting + streaming secret-scrub machinery. - Keep pipes drained (
stdout.resume()andstderrconsumption) and preserve heap-exhaustion detection via a small sliding-window watcher. - Update CLI + MCP content rendering and refactor/replace tests accordingly (including moving redaction coverage to verdict-controlled bytes).
Reviewed changes
Copilot reviewed 20 out of 20 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| packages/tool-server/src/tools/flows/script/flow-script-executor.ts | Removes log capture/budgets; drains stdout/stderr; keeps heap banner detection; updates exit-code messaging. |
| packages/tool-server/src/utils/secrets.ts | Reverts secret scrubbing helper back to whole-text usage; removes streaming chunk API and updates docs. |
| packages/tool-server/src/tools/flows/flow-script-step.ts | Stops attaching captured log fields to step outcomes; keeps verdict-only outcome surface. |
| packages/tool-server/src/tools/flows/flow-run.ts | Removes scriptLog* fields from step reports and drops run-scoped log budget plumbing; updates help text. |
| packages/tool-server/src/tools/flows/flow-add-script.ts | Removes log/logTruncated from tool result and tool description; updates error wording to match new surfaces. |
| packages/argent-cli/src/flow.ts | Removes script-log rendering and ensures batch/live output never includes script-printed bytes. |
| packages/argent-mcp/src/content.ts | Removes script-log blocks from MCP content rendering (status line only). |
| packages/tool-server/test/flows/script/flow-script-verdict.test.ts | Updates verdict fixture helper to match FlowScriptResult without log fields. |
| packages/tool-server/test/flows/script/flow-script-step-run.test.ts | Reworks integration tests to assert “printed output is not forwarded” and adds explicit drain coverage. |
| packages/tool-server/test/flows/script/flow-script-run.test.ts | Updates executor unit tests to assert absence of printed output and adds a stdout-flood drain test. |
| packages/tool-server/test/flows/script/flow-script-redaction.test.ts | New focused suite for redaction behavior on verdict-controlled bytes (failure message/stack + output doc). |
| packages/tool-server/test/flows/script/flow-script-record.test.ts | Updates recorder tests to assert no log fields returned and no printed output leaks. |
| packages/tool-server/test/flows/script/flow-script-protocol.test.ts | Updates protocol/robustness tests to no longer expect captured logs. |
| packages/tool-server/test/flows/script/flow-script-lifecycle.test.ts | Updates lifecycle/timeouts/cancellation tests to avoid asserting on captured logs; uses filesystem marks instead. |
| packages/tool-server/test/flows/script/flow-script-channel.test.ts | Updates “request never reaches runner” tests to assert script non-execution via a mark file, not logs. |
| packages/tool-server/test/flows/flow-concurrent-recording.test.ts | Updates a message assertion to remove “logs … are lost” wording. |
| packages/tool-server/test/flows/script/flow-script-logs.test.ts | Deletes the obsolete log-capture focused suite. |
| packages/argent-cli/test/flow.test.ts | Updates CLI flow tests to assert logs are never printed even if present “on the wire”. |
| packages/argent-cli/test/flow-script-render.test.ts | Removes script-log rendering tests and replaces them with “status-only” expectations. |
| packages/argent-mcp/test/content.test.ts | Updates MCP content tests to ensure script output is never rendered, even if present in input. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| projectRoot: request.projectRoot, | ||
| flowDir, | ||
| // No `secrets`: nothing resolves one into a script step yet, so there is | ||
| // nothing for the executor to redact out of the captured log. | ||
| ...(request.logBudget ? { logBudget: request.logBudget } : {}), | ||
| // nothing for the executor to redact out of the verdict it reports. | ||
| ...(request.signal ? { signal: request.signal } : {}), |
latekvo
left a comment
There was a problem hiding this comment.
[Diplomat, Opus 5]: Reviewed 10a0fb6e6 against its base a7e236acf, with both branches built and driven end to end rather than read.
What I ran:
- Both branches built and served: a tool-server from each on its own port, driven over HTTP (
flow-execute,flow-add-script,flow-start-recording/flow-finish-recording), plus the realargent flow runbinary across the single-flow, directory,--jsonand--helpsurfaces, andflowRunToMcpContentcalled on the server's own reports. - The leak question directly: fixtures printing
CANARY_STDOUT_LINE_ONE,DATABASE_URL=postgres://user:hunter2@db/prodandCANARY_STDERR_LINE sk-live-…on both pipes, on passing, failing, throwing, timed-out and cancelled steps. No script-printed byte reaches any of the four report surfaces on any status. - Drain correctness: 4 MiB and 256 MiB stdout floods both pass (256 MiB in 203 ms); a heap-exhausting script still classifies off the stderr window as "The script exceeded its 512 MiB heap limit."
- Mutation testing on the changed lines. Removing
child.stdout?.resume(), swapping the stderrheapWatchwiring for a bareresume(), droppingthis.tail = "", shrinkingHEAP_FATAL_WINDOW_CHARSto 4, reverting the exit-code wording, dropping the notes inscriptVerdict, and re-adding each deleted renderer are all killed by the suites in this PR. OnlySETTLE_TIMEOUT_MS = 0survives, which is a pre-existing timing race rather than anything this diff introduces. scrubSecretValuesbefore vs after, differentially fuzzed over 200,014 cases (marker-shaped text, nested and prefix values, lone surrogates, astral characters, empty and duplicate values): 0 divergences.- What outlives the call: fd count flat at 22 across 150 executions, and peak RSS 162 MB here against 207 MB on base.
prettier --check .,npm run knip,npm run typecheck:tests --workspaces, and the tool-server script suites (358 tests), argent-cli (115) and argent-mcp (36) are green.
The deleted flow-script-logs.test.ts checks out: everything dropped is about the removed channel, and the non-log pins moved to flow-script-redaction.test.ts. I also could not find a docs or skills page that mentions script stdout/stderr, so the "no docs update" call matches the tree.
Three notes below. None of them is about the leak this PR closes.
…p-logs # Conflicts: # packages/argent-cli/src/flow.ts # packages/argent-cli/test/flow-script-render.test.ts # packages/tool-server/src/tools/flows/flow-add-script.ts # packages/tool-server/src/tools/flows/flow-run.ts # packages/tool-server/src/tools/flows/flow-script-step.ts # packages/tool-server/test/flows/flow-concurrent-recording.test.ts
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Team Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
…p-logs The base branch condensed the prose across flow-add-script: the tool description, the append-failure message and their assertions were all rewritten. This branch had edited those same lines only to drop the `log`/`logTruncated` references from them. Resolved in favour of the base branch's wording in all three hunks - the condensed text names no log channel, so it already says what this branch was editing those lines to say. The log removals that stand on their own (the `log`/`logTruncated` result fields, the `logBudget` comment and the two `scriptLog` spreads) merged cleanly and are unchanged.
Dropping the script log channel took two first-party writers with it. `flow-script-runner.mjs` and the lifeline thread each announce their own failure to arm on the child's stderr, and stderr is now drained into a scan that keeps nothing — so a step that ran with the group kill unarmed came back byte-identical to a healthy one, and a descendant outliving a tool server that died afterwards had no account anywhere. stderr is scanned for those lines the way it is already scanned for V8's heap banner: a flag, never text. `HeapFatalWatch` becomes `StderrSignalWatch` and tracks both patterns, its window now dropped only once NEITHER is still looking — one shared `found` would have ended the scan at whichever banner arrived first. A match earns a sentence argent wrote, on `notes` beside the clamped-timeout and wrong-directory notes. The flag is forgeable, as the heap flag already was; what forging it must not buy is a way to put the script's own bytes back on the report, and it does not. The emission itself stays racy: `new Worker` rejects through an `error` event, so a script that returns immediately exits before it fires. Making that deterministic would mean awaiting both worker spawns ahead of every script, which is a cost on the hot path this does not take.
…annel The prose sweep reached the tool descriptions and the help text but not these, and each explains present-day code by machinery this branch deleted. `flow-script-protocol.ts` justified keeping logs off IPC by them landing "in one stream in written order" under a drain-time limit. `ScriptLogCapture` merged those pipes and both limits bounded it; all three are gone, so the one file that documents the executor↔runner contract was explaining its shape by a mechanism that no longer exists — and reading as an instruction not to move logging onto IPC on grounds nobody could check. The reason that actually holds is not about transport: the executor drains the pipes so console text never reaches the report at all. `finish` gave "would discard buffered log output" as the whole reason for waiting on both flushes. There is no log output to discard, but the wait still earns its place: stderr is scanned, for V8's heap banner and now for a watchdog that never armed, so a write dropped at exit costs a signal. The verdict is not at stake — `sendToParent` precedes the flush. `flushStream` said an unhandled `error` "would land in the step's own log". It would be an uncaught exception; what the guard buys is that a script closing its own stdout still passes, which is what the test beside it now pins.
#970 landed a docs/skills trim (from #940) and a `report a watchdog that never armed` fix while this branch was out; both touch the surfaces the bash step widened. Resolutions: - `flow-script-protocol.ts`, `flow-add-script.ts`, `flow-yaml.mdx`, `flows.mdx`, `tools.mdx` and the two `argent-create-flow` references take #970's trimmed prose and re-apply this branch's `.sh` facts on top of it. Nothing bash-specific was dropped: the working-directory and PATH rules the trim removed from the skill's prose move into the `Bash scripts` bullet list beside the CRLF and background-job ones. - #970 renamed `HeapFatalWatch` to `StderrSignalWatch` while this branch moved the watch into the `ChildRun` record. Both landed, so the field, the argument and the destructure carried the old name against the new class; they now read `stderrWatch`. `WATCHDOG_PROBLEM_RE` matches what this branch's runner and lifeline actually write. - Three bash tests asserted pre-trim wording. The mis-case pair now quotes the trimmed `Script path "..." has the wrong letter case` message, and the `mjs|sh` assertion is dropped because that message no longer quotes the pattern - `flow-script-step-parse.test.ts` pins the widened pattern directly. The `.sh` recording test asks for the trimmed "Check or restore its changes" instead of "nothing was rolled back".
The merge resolution folded it into `### Bash scripts` because #970's trim had removed the paragraph it lived in. It is not bash-specific: an `.mjs` reading `./fixtures/order.json` resolves it against the project root the same way, which is what the sentence was there to say. Only the `BASH_SOURCE` spelling of "the script's own directory" belongs under bash. The shipped reference never lost it — `flow-yaml.mdx` states it outside the `### Bash scripts` subsection — so this is the skill catching up.
Stacked on #940, which adds the recorder; that one is stacked on #865 (the
script:step) and #864 (the executor).Review this on top of #940 - the base branch here is
feat/flow-record-script-step.A
script:step used to bring its stdout and stderr back with it. The recorderreturned
log/logTruncated, the step report carriedscriptLog/scriptLogTruncated, and the CLI and the MCP content builder eachrendered them.
Both of those surfaces land in an agent's context, and a script is a plausible
place for a connection string, a bearer token or customer data to be printed -
by the script itself, or by a CLI it shells out to. Nothing in the flow file
declares what a script prints, so nothing can decide in advance whether that
text is safe to forward.
An opt-in parameter was the other option, and it is not a control: the agent is
the one who would set it. So the channel goes rather than being gated - the
recorder returns none, a replay returns none, and there is no parameter to ask
for them.
What a script step answers with now
Its verdict and its
reason.reasonis the script's own message for three ofthe nine failure kinds - a thrown error, a bad output document, a load failure -
and the runner's account of how the process stopped for the other six (an exit
code, a signal, a time limit, the heap limit, a cancellation). On a PASS it
carries the executor's notes, the clamped time limit among them.
A script with something to say has to
throwit or return it inoutput. Bothof those texts are still redacted against the flow's secrets; that half is
unchanged.
What goes with the channel
The capture is deleted rather than hidden behind a flag, so everything that
existed to bound it has no subject left: the 64 KiB per-step cap, the 256 KiB
run-wide budget, the V8 frame collapser, the partial-marker guard, and the
streaming half of the secret scrub.
scrubSecretValuesabsorbs the chunk walkit was the only caller of and goes back to being a whole-text function.
What the streams still owe
Three things, and only these:
account of why an abort was an out-of-memory rather than a plain signal.
StderrSignalWatchmatches over a sliding window and keeps no text beyond it.only the script's:
flow-script-runner.mjsand the lifeline thread eachreport their own failure to arm on it, and those lines are the only account of
a step that ran with the group kill unarmed. A match sets a flag and earns a
fixed sentence argent wrote, on
notesbeside the clamped-timeout andwrong-directory notes - never a byte the script chose, so a script forging the
banner buys itself a false note rather than a channel. The report is only as
available as the emission:
new Workerrejects through anerrorevent, so ascript that returns immediately can exit before it fires.
child from ever reaching its own time limit, which turns a passing step into a
timeout.
child.stdout?.resume()reads as a no-op and is not - see the A/B inVerification.
StderrSignalWatch.end()(thenHeapFatalWatch.end()) is dropped along the way: it scanned whatStringDecoder.end()returns, which is either the empty string or a replacementcharacter for a trailing incomplete sequence, and the banner regex is ASCII-only.
It could never match anything the per-chunk scans had missed. The settle wait
above it is what actually puts the last chunks in before the streams are
destroyed, and that stays.
One reworded reason
The exit-code failure said "no output was captured", meaning the output
document. With the console channel gone that reads as "the script printed
nothing", so it now names the document and points at
throw.Tests
flow-script-logs.test.ts(660 lines) goes; its redaction and heap coveragemoves to
flow-script-redaction.test.tsagainst the two texts that still carryscript-controlled bytes. Six tests whose names or comments claimed more than they
proved are fixed rather than deleted:
timeout, over a flood sized against this host's pipe buffer. The executor's own
suite now has one, at 4 MiB;
secret, so both assertions passed with
partialSecretTailstubbed out;The renderers keep
scriptLogon the wire in their fixtures on purpose: theassertion is that a report carrying one renders nothing from it.
Documentation
No docs update is needed.
packages/docsandpackages/skillsare untouchedby this PR. An earlier revision added a paragraph to five of them saying Argent
discards what a script prints; a reader who never saw the log channel learns
nothing from being told it is absent, so all five went back to their base content.
What did have to change is the prose that still promised the removed fields,
and there the edit is a deletion rather than a replacement:
flow-add-scriptdrops the
log/logTruncatedparagraph,flow-rundrops "Its stdout and stderrcome back on the step report", and
argent flow --helpdrops "each script step'soutput" from what a directory run prints.
Verification
npm run build,npx eslint . --max-warnings 0,npx prettier --check .,npm run typecheck:tests --workspaces,npm run knip,npm run test:scriptsandthe repo-hygiene checks are all green. (Only Tool Description Quality runs on
this PR — every other workflow is gated on
pull_request: branches: [main]— sothese were run locally.)
npm test -w @argent/tool-server: 378 files, 5025 passed, 1 skippednpm test -w @argent/cli: 27 files, 508 passednpm test -w @argent/mcp: 6 files, 82 passedExercised end to end against a tool server built from this branch, device-free
(a
script:step needs none), through all three entry points. Fixtures print aconnection string, a token-shaped value and a plain canary to stdout and stderr:
argent flow runover a flows directory - a passing script contributes nothing,and the three failures carry only their reason:
A single run still prints a PASS note -
f-clamp.yamlasks for a 166.7m limit:flow run --json, the raw report the CLI renders from, carries noscriptLog:{ "index": 0, "kind": "script", "flow": "a-chatty", "target": "../../scripts/chatty.mjs", "status": "pass" }flow-add-scriptover HTTP, recording the same chatty script - nolog, nologTruncated, on a pass and on a failure:{"status":"pass","durationMs":41, "message":"Script step added to \"scriptrec\" flow — it ran here exactly as it will at replay. …", "outputJson":"{\"seeded\":true,\"orderId\":\"ord_4711\"}", "stepCount":1,"recorded":"1. script: ../../scripts/chatty.mjs"} {"status":"fail","reason":"seed API returned 500","durationMs":119, "message":"The script \"../../scripts/throws.mjs\" failed — nothing was recorded in \"scriptrec\", …"}The MCP stdio adapter, replaying that recording - no
script output:block:None of
hunter2,sk-live-…,DATABASE_URLor any canary appears in the CLIoutput, the
--jsonreport, the HTTP payload or the MCP content blocks.The drain, A/B against the built bundle.
d-flood.yamlwrites 4 MiB tostdout - past any host's pipe buffer - under a 5s limit. With
child.stdout?.resume()patched out of the compiled tool-server it fails at thescript's own limit, and passes again the moment the line is restored:
The heap verdict in
c-oom.yamlis the matching proof for the other stream:stderr is still scanned even though nothing it carries is reported.