DIVE-3786: stall alert stops naming an untested mechanism; re-arm kick verifies its own submit - #46
Merged
Conversation
…kick observe its own submit
detectStallCause() tested two pane patterns and reported everything else
as "listen loop wedged / agent is idle outside wait_for_message and won't
re-arm". It reads none of those things — the string was the else-branch,
shown to a paying customer as a diagnosis. It now says "not responding —
cause unknown" and prints the last lines of the agent's screen.
kickListenLoop() typed with `send-keys -l` (which APPENDS) and fired
Enter with the result discarded, under a comment noting the TUI drops
Enters. A dropped Enter stranded text in the composer and every later
kick concatenated onto it — the recovery path using the delivery path
that broke, which is why the customer saw the alert constantly. It now
clears with C-u, submits separately, and verifies by comparing the pane
before and after; one retry, then it sets rearmSubmitFailed.
That flag is the only wedged state we actually measure, so it is the only
one the alert names ("stuck at the input prompt").
The pane at alert time is written to last-stall-pane.txt next to the
existing stamp, so the next occurrence is diagnosable on the box.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
DIVE-3786 — stop reporting an untested mechanism, and make the re-arm kick observe its own submit
A customer reported seeing this constantly, from a codex box:
Defect 1 — that string was the else-branch
detectStallCause()captures the agent's tmux pane, takes the last 4000 chars, and tests exactly tworegexes (quota exhausted, auth expired). Everything else returned
listen loop wedged / agent is idle outside wait_for_message and won't re-arm— a named mechanism, in the customer's chat, from afunction that reads neither the listen loop, nor
wait_for_messagestate, nor re-arm state. Itsfrequency was evidence about the classifier's coverage, not about the listen loop.
Fix: the unmatched branch now returns
not responding — cause unknownplus the last 3 non-emptylines of the agent's screen (
paneTailSummary), so the owner sees what the agent is actually sittingon.
listen loop wedgedno longer exists in the repo.Defect 2 — the recovery path broke itself, which is why it fired constantly
Root cause found on the customer box: the agent had finished its turn cleanly and a line was sitting
unsubmitted in the codex composer.
kickListenLoop()was:send-keys -lappends, so each kick concatenated onto the stranded line instead of replacing it; andthe Enter was fire-and-forget, directly under a comment noting the TUI "occasionally drops an Enter".
Three unnoticed failures (
STALL_ESCALATE_AFTER) then fired the owner alert. Self-perpetuating byconstruction: the recovery mechanism used the delivery path that was broken.
Fix, per attempt:
C-u(clear composer) →send-keys -l <text>→ 400 ms → capture pane →send-keys Enter(own call) → 700 ms → capture pane → assert the pane changed. A landed Enterclears the composer and starts a turn, so byte-identical panes mean the keystroke went nowhere. One
retry, then
rearmSubmitFailed = true. A pane we cannot read counts as unconfirmed, not as success.rearmSubmitFailedis the only wedged state we actually measure, so it is the only one the alertnames: "stuck at the input prompt".
clearStallAlert()resets it on recovery.Also
sendStallAlert()writes<STATE_DIR>/last-stall-pane.txt(timestamp, agent, cause, full pane) nextto the existing stamp, so the next occurrence is diagnosable on the box rather than from a forwarded
screenshot.
Applied to all three forks —
telegram-codex,telegram-grok,telegram-agy— the code wasidentical in each, so a codex-only fix would have left the same string shipping on grok and
antigravity boxes.
Evidence
C-uclears a live codex composer — probed againstagent-andy(codex TUI, 2026-08-28): typeda probe string, composer showed it;
C-ureturned the composer to its placeholder hint. No Enterwas sent, nothing was submitted.
cat: text + Enter →pane changed (green when a submit lands); the same pane sampled twice with no input → byte-identical
(so the check can genuinely FIRE, not merely fail to false-alarm).
C-uthere took a stranded linefrom 1 occurrence to 0.
test/dive3786-honest-stall-cause.test.ts— 27 assertions across the 3 forks.Negative control: all 27 FAIL against
origin/main'sserver.ts; all 27 pass on this branch.bun build --external '*'parses all three patched servers. No repo-roottsconfig.json, so therewas no typecheck arm to run — not claimed.
Deliberately out of scope, follow-up owed
5dive-agent-start(repo5dive-cli, grok :1708 / antigravity :1733 cold-start kicks) has the sameunverified
-l→sleep 1→Entertwo-step. Left out: different repo, different ship path(deliberate
release-cut, and the boot path owes a smoke this host cannot run), it is the boot pathfor every agent, and defect 1 cannot bite there — the cold-start composer is empty by construction,
single-fire, so only the unverified-Enter half applies.
Separate finding while reading it: both CLI
_kickstrings contain "keep looping" and "call itagain immediately" — verbatim the model-facing phrases
test/rearm-loop-regression.test.tsbans asthe DIVE-1180 idle-token-burn cause. That guard scans only this repo, so the CLI copy is unguarded.
Worth its own row.