fix: never queue speech into a mouth that cannot speak yet - #10
Open
FryD420 wants to merge 9 commits into
Open
Conversation
The mouth was one thread doing render-then-play-then-render, so every chunk boundary paid synth latency plus the 0.75s prebuffer in series (about a second of dead air per boundary) while the text had long since reached the screen. Now a synth thread renders LOOKAHEAD (2) chunks ahead of the one playing; the next chunk's audio is already finished when the previous one ends and plays with no gap. Both audio laws hold: still one long-lived OutputStream, still the prebuffer (satisfied instantly for pre-rendered chunks). A generation counter ties the two threads together for barge-in: shut_up() bumps it and anything ordered under the old generation is dropped wherever it's found, so nothing stale plays; a pending counter replaces "queue empty" for the speaking flag and wait_done(), since the text queue stopped being the whole story. The brain now logs one line per tool call ([tool] Read: <path>, [tool] Bash: <description>, ...) from the AssistantMessage that lands as the call runs, so the terminal shows what the agent is doing while the voice is quiet instead of a silent thinking loop. tests/test_mouth_lookahead.py drives the mouth with a fake synth and a fake real-time output device: boundary gaps under 80ms (measured ~0-1ms), a sub-prebuffer chunk completes, barge-in plays nothing stale and the mouth speaks fresh text afterwards.
…he session The Agent SDK's stream-json reader defaults to 1 MB per message. Reading a 1080p screenshot (~4 MB PNG, ~5 MB base64 on the wire) exceeded it and crashed the voice session. 16 MB gives ~3x headroom over 1080p and covers 4K. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…sh' verb When resume_last_session reattaches, the hidden warmup ping becomes a spoken turn: the agent says what was in flight and asks continue or start fresh. "start fresh" / "new session" / "start a new session" are added as synonyms for the clear verb so the answer is natural. Cold launches are unchanged. Docs and the spoken-console discipline text updated. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…other signal files Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…aunch right after a fresh start comes up cold Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…(.voice_activity, per tool call) so a face can tell thinking from dead Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ff-by-one) Background-task notifications (finished Bash jobs, Monitor events, timeouts) wake the model while the mic is quiet; its answer sits in the shared stream unread, and the next real question pairs with it — every reply one question late for the rest of the session. reset_turn can't catch it (_dirty is False: the turn wasn't ours). _drain_idle() pulls everything already buffered, non-blocking, logs the dropped text, and if a background turn is still mid-flight ask_stream waits (bounded, 30s) for its ResultMessage before sending. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Pasted paths, markup, and whole listing bodies were being read aloud (fences stripped to bare text by the old hygiene pass). _defence() splits chunks on ``` with fence state persisting across sentence chunks; the transcript keeps the full text, only the mouth mutes. Also: backtalk.json swaps the daily driver to claude-opus-5 with claude-fable-5 as the deep model (config is untracked; noted here). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Symptom: a relaunch where the recap appears as text in the log and the session is completely silent — greeting included. Not an audio fault. Kokoro takes ~30s to load on this machine, every launch (every `[mouth] voice ready` on record lands 25-32s after `[backtalk] up`). The greeting was queued at second zero and the resume recap at second nine, both into a pipeline that could not render a word, so they waited in the queue. Then the talk key was pressed one second after the voice finally loaded — and the PTT handler calls shut_up() unconditionally, which flushes the queue. Thirty-one seconds of waiting speech destroyed one second before it would have played. The fix is not a smarter flush. It is to never queue speech into a cold mouth: then a press during warm-up destroys nothing, because there is nothing queued yet to destroy. - mouth.py: a warm gate — is_warm(), wait_warm(timeout), start_warming(). warm() sets the event once the pipeline can genuinely render. - main.py: start_warming() is kicked explicitly right after Mouth(), because the render loop only triggered the load lazily on the first queued line — and nothing is queued until warm now, so that would never have fired. The greeting moves to a greet-when-warm thread and the resume recap awaits the gate before speaking. - The warm kick is deliberately NOT in Mouth.__init__: constructing a Mouth should not drag Kokoro into a unit test, and it did until this was moved out. - tests/test_warm_gate.py: nine checks, including a replay of the exact losing sequence — defer speech, fire shut_up() mid-warm-up, open the gate, assert the greeting still speaks. Deliberately unchanged: the ~30s load, and the unconditional flush on a talk-key press. Both are correct; only the queueing was wrong. test_mouth_lookahead.py and test_typed_clean.py still pass. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Owner
|
This is a proper piece of work and the warm-gate diagnosis is the best bug report I've had on this repo. Thirty-one seconds of queued speech destroyed one second before it would have played, and the conclusion that the flush is right and the queueing is wrong is the correct read. Same for the lookahead: you kept both audio laws and said so, which saved me the review. Two things before I can merge it, both mechanical. Line endings. Your editor rewrote every file as CRLF, so GitHub shows 2,253 changed lines. Ignoring that, the real change is 786 added and 70 removed, and 263 of those are your tests. Merging as-is would flip the whole repo to CRLF. Can you commit with LF? Rebase. You branched 13 commits back and main has moved a long way today, eleven bugs fixed, mostly Windows. We both touched main.py, mouth.py, brain.py and signals.py, so there will be conflicts. I'd rather you resolve them than me guess at your intent. Once it's rebased and on LF I can actually read it properly, and I'd like to. The thirty-second load on your machine is also worth its own issue, separately. That's a long time and I want to know why. #4 is going in as part of this, so I'm closing that one rather than merging both. Nothing of it is lost. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Symptom
A relaunch where the agent's reply shows up as text in the log and the session is completely silent — greeting included. It reads as broken audio. It isn't.
Cause
Kokoro takes ~30 s to load on my machine, every launch (every
[mouth] voice readyin my log lands 25–32 s after[backtalk] up). The greeting is queued at second zero and the resume recap at second nine — both into a pipeline that cannot render a word yet, so they sit in the queue.Then the talk key gets pressed, and the PTT handler calls
mouth.shut_up()unconditionally, which flushes the queue.In my case that press landed one second after the voice finally loaded. Thirty-one seconds of patiently waiting speech was destroyed one second before it would have played.
The fix
Not a smarter flush — the flush is correct. The bug is queueing speech into a cold mouth at all. Once nothing is queued until the voice is real, a press during warm-up destroys nothing, because there is nothing there to destroy.
mouth.py: a warm gate —is_warm(),wait_warm(timeout),start_warming().warm()sets the event once the pipeline can genuinely render.main.py:start_warming()is kicked explicitly right afterMouth(). The render loop only triggered the load lazily on the first queued line, and nothing is queued until warm now — so without this the load would never begin. The greeting moves to agreet-when-warmthread; the resume recap awaits the gate before speaking.Mouth.__init__: constructing aMouthshouldn't drag Kokoro into a unit test, and it did until I moved it out.Deliberately unchanged
The ~30 s model load, and the unconditional flush on a talk-key press. Both are right. Only the queueing was wrong.
Tests
tests/test_warm_gate.py— nine checks, including a replay of the exact losing sequence: defer speech, fireshut_up()mid-warm-up, open the gate, assert the greeting still speaks.test_mouth_lookahead.pypasses unchanged.Note on the base
This branches off
lookahead-and-tool-lines(PR #4), because the resume-recap path it gates was introduced there. If you'd rather have it againstmainon its own, say the word and I'll rework the greeting half to stand alone.