Skip to content

DIVE-3786: stall alert stops naming an untested mechanism; re-arm kick verifies its own submit - #46

Merged
lodar merged 1 commit into
mainfrom
dive-3786-dev
Aug 28, 2026
Merged

DIVE-3786: stall alert stops naming an untested mechanism; re-arm kick verifies its own submit#46
lodar merged 1 commit into
mainfrom
dive-3786-dev

Conversation

@5dive-bot

Copy link
Copy Markdown
Collaborator

DIVE-3786 — stop reporting an untested mechanism, and make the re-arm kick observe its own submit

A customer reported seeing this constantly, from a codex box:

can't respond right now — listen loop wedged / agent is idle outside wait_for_message and won't re-arm

Defect 1 — that string was the else-branch

detectStallCause() captures the agent's tmux pane, takes the last 4000 chars, and tests exactly two
regexes (quota exhausted, auth expired). Everything else returned listen loop wedged / agent is idle outside wait_for_message and won't re-arm — a named mechanism, in the customer's chat, from a
function that reads neither the listen loop, nor wait_for_message state, nor re-arm state. Its
frequency was evidence about the classifier's coverage, not about the listen loop.

Fix: the unmatched branch now returns not responding — cause unknown plus the last 3 non-empty
lines of the agent's screen (paneTailSummary), so the owner sees what the agent is actually sitting
on. listen loop wedged no longer exists in the repo.

Defect 2 — the recovery path broke itself, which is why it fired constantly

Root cause found on the customer box: the agent had finished its turn cleanly and a line was sitting
unsubmitted in the codex composer. kickListenLoop() was:

send-keys -l REARM_KICK_TEXT   // -l APPENDS to whatever is already in the composer
setTimeout(400) -> send-keys Enter, callback () => {}   // result discarded

send-keys -l appends, so each kick concatenated onto the stranded line instead of replacing it; and
the Enter was fire-and-forget, directly under a comment noting the TUI "occasionally drops an Enter".
Three unnoticed failures (STALL_ESCALATE_AFTER) then fired the owner alert. Self-perpetuating by
construction: the recovery mechanism used the delivery path that was broken.

Fix, per attempt: C-u (clear composer) → send-keys -l <text> → 400 ms → capture pane →
send-keys Enter (own call) → 700 ms → capture pane → assert the pane changed. A landed Enter
clears the composer and starts a turn, so byte-identical panes mean the keystroke went nowhere. One
retry, then rearmSubmitFailed = true. A pane we cannot read counts as unconfirmed, not as success.

rearmSubmitFailed is the only wedged state we actually measure, so it is the only one the alert
names: "stuck at the input prompt". clearStallAlert() resets it on recovery.

Also

sendStallAlert() writes <STATE_DIR>/last-stall-pane.txt (timestamp, agent, cause, full pane) next
to the existing stamp, so the next occurrence is diagnosable on the box rather than from a forwarded
screenshot.

Applied to all three forkstelegram-codex, telegram-grok, telegram-agy — the code was
identical in each, so a codex-only fix would have left the same string shipping on grok and
antigravity boxes.

Evidence

  • C-u clears a live codex composer — probed against agent-andy (codex TUI, 2026-08-28): typed
    a probe string, composer showed it; C-u returned the composer to its placeholder hint. No Enter
    was sent, nothing was submitted.
  • The pane-change check fires in both directions — real tmux session running cat: text + Enter →
    pane changed (green when a submit lands); the same pane sampled twice with no input → byte-identical
    (so the check can genuinely FIRE, not merely fail to false-alarm). C-u there took a stranded line
    from 1 occurrence to 0.
  • Full suite: 1060 pass / 0 fail across 37 files on the patched tree.
  • New guard test/dive3786-honest-stall-cause.test.ts — 27 assertions across the 3 forks.
    Negative control: all 27 FAIL against origin/main's server.ts; all 27 pass on this branch.
  • bun build --external '*' parses all three patched servers. No repo-root tsconfig.json, so there
    was no typecheck arm to run — not claimed.

Deliberately out of scope, follow-up owed

5dive-agent-start (repo 5dive-cli, grok :1708 / antigravity :1733 cold-start kicks) has the same
unverified -lsleep 1Enter two-step. Left out: different repo, different ship path
(deliberate release-cut, and the boot path owes a smoke this host cannot run), it is the boot path
for every agent, and defect 1 cannot bite there — the cold-start composer is empty by construction,
single-fire, so only the unverified-Enter half applies.

Separate finding while reading it: both CLI _kick strings contain "keep looping" and "call it
again immediately"
— verbatim the model-facing phrases test/rearm-loop-regression.test.ts bans as
the DIVE-1180 idle-token-burn cause. That guard scans only this repo, so the CLI copy is unguarded.
Worth its own row.

…kick observe its own submit

detectStallCause() tested two pane patterns and reported everything else
as "listen loop wedged / agent is idle outside wait_for_message and won't
re-arm". It reads none of those things — the string was the else-branch,
shown to a paying customer as a diagnosis. It now says "not responding —
cause unknown" and prints the last lines of the agent's screen.

kickListenLoop() typed with `send-keys -l` (which APPENDS) and fired
Enter with the result discarded, under a comment noting the TUI drops
Enters. A dropped Enter stranded text in the composer and every later
kick concatenated onto it — the recovery path using the delivery path
that broke, which is why the customer saw the alert constantly. It now
clears with C-u, submits separately, and verifies by comparing the pane
before and after; one retry, then it sets rearmSubmitFailed.

That flag is the only wedged state we actually measure, so it is the only
one the alert names ("stuck at the input prompt").

The pane at alert time is written to last-stall-pane.txt next to the
existing stamp, so the next occurrence is diagnosable on the box.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@lodar
lodar merged commit aba2817 into main Aug 28, 2026
2 checks passed
@lodar
lodar deleted the dive-3786-dev branch August 28, 2026 06:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants