fix(cell): per-turn timeout + watchdog + drop post-summary wrap-up - #140
Merged
Conversation
Investigations could wedge at "thinking" forever when the LLM stream (or a stalled tool like a slow checkout) never returned — no per-turn timeout, so the turn never finished and nothing rescheduled. Also the model often emits a redundant wrap-up sentence beside the summary card despite the "STOP after present_investigation_summary" instruction. - runRealTurn: bound the whole turn (LLM stream + tool calls) with a 180s AbortSignal (combined with any caller signal). A hung model/tool aborts, the turn fails cleanly with a clear error, and the run parks/fails instead of hanging. Thread the signal through the code tools + checkout fetches so a stalled checkout aborts too. - Drop the assistant wrap-up text when the turn calls present_investigation_summary — the card IS the report. - InvestigationDO: set a durable watchdog alarm (240s) at turn start so a turn whose isolate is evicted mid-run gets re-run on a fresh isolate (no-ops if the row is no longer 'running'), instead of sitting 'running' until the coordinator's 20-min orphan reclaim. Cell-only — needs the cell redeploy. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0158jez4WfWj6ttmaxh3HEbT
coccyx
added a commit
that referenced
this pull request
Aug 15, 2026
Checking out a large polyglot repo (the OTel demo tarball is ~66 MB) wedged the InvestigationDO: store() walked every tar entry and did a synchronous SQLite INSERT per file — up to 6000 files / 48 MB — with no yield and no abort check. While that ran, the single-threaded DO could not service status/events, which started returning 502, and #140's per-turn timeout couldn't help (it aborts fetch/stream, not a sync loop). - Filter noise before decode: dependency/build/generated subtrees (node_modules, dist, target, gen, vendor, …) and asset/binary extensions (png, woff, pdf, wasm, lock, min.js, …). Source is all the agent reads. - Lower caps to 2500 files / 20 MB. - store() is now async: it yields to the request queue every 200 inserts (so status/events interleave mid-checkout) and checks the turn's abort signal (so a runaway checkout actually stops). - checkout_repo result reports what was skipped (no silent caps). - Unit tests for filtering, caps, and abort. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0158jez4WfWj6ttmaxh3HEbT
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes investigations wedging at "thinking" (observed live: a follow-up turn whose LLM stream hung for 8+ minutes with no output), and drops the redundant wrap-up text the model emits beside the summary card.
runRealTurnnow bounds the whole turn (LLM stream + tool calls) with a 180sAbortSignal(combined with any caller signal). A hung model or a stalled tool aborts, the turn fails cleanly with a clear error, and the run parks/fails instead of hanging. The signal is threaded through the code tools + checkout fetches, so a stalledcheckout_repoaborts too. This is the same root cause behind the "8-of-10 started-but-didn't-conclude" runs.InvestigationDOsets a durable 240s alarm at turn start. A DO alarm can't preempt a running handler (single-threaded), so the timeout above handles a live-isolate hang; this covers the other case — the isolate being evicted mid-turn (leaving the rowrunningwith no pending alarm). The watchdog re-fires on a fresh isolate and re-runs the turn (no-op if it's no longerrunning), instead of waiting on the coordinator's 20-min orphan reclaim.present_investigation_summary, suppress the accompanying assistant text at emit time. The seed prompt says STOP after the summary, but deepseek routinely adds a redundant sentence next to the card.Cell-only — needs the cell redeploy.
#139is already on master; this lands there too.🤖 Generated with Claude Code