Per-agent Talk: live voice sessions for worker agents - #331
Open
aarjav wants to merge 3 commits into
Open
Conversation
Talk was god-only: connect() hard-coded the Michael persona and the hive-wide read+action tools, so dropping the toggle on a worker card would still have spun up the orchestrator. This parameterizes the loop by TARGET. - realtime/agentVoice.ts (new): a worker's persona built from its name, registry role and its own memory.md, plus self-scoped tools — read_my_memory, check_my_session, my_latest_reply, get_my_tasks, and ask_my_session, which relays the human's words into that agent's LIVE Claude Code session and reads the answer back. Routing is decided in code from the agent's status: idle → the renderer message queue (rides every delivery gate), busy → a steer note at the next hook. No action spine: a worker's voice cannot act on the floor. - realtime/session.ts: connect(target?) branches to connectWorker() for a worker and is otherwise untouched — god keeps MICHAEL_PERSONA, cedar, the read+action tools, and the floor/completion subscriptions, byte-identical. State carries the live target so each card knows whether the call is its own. - RealtimeMichaelToggle: optional `target`; a toggle that doesn't own the live call renders idle and switches the call to itself on click. - AgentCard/AgentStrip: worker cards get a compact Talk button on their note row. - test/agent-voice-scope.test.cjs pins both invariants (worker tools stay self-scoped; god's session is unchanged). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…the store
The relay worked in one direction only. ask_my_session delivered the human's
question into the agent's live session (visible in the terminal) and then waited
for the store fields `recentAssistantText` + `recentTextTs` to advance — but
nothing in the LIVE pipeline ever wrote `recentAssistantText`. Its only writer
was store/mockEvents.ts, which is why 558/558 passed while A3 failed on every
real worker: green in mocks, dead in the app. `hive:activity` carries status and
tool metadata, never the words, so the wait always ran out and the voice fell
back to "still working" for answers that had already landed.
Adds the missing PRODUCER; the read side is untouched.
main/transcript.ts readLatestAssistantText() — scans back from the tail of a
session transcript for the newest assistant record with
real text (thinking, tool_use and isSidechain records
skipped), returning {text, ts}. Memoized on size+mtime so
the poll is a stat in the steady state.
main/index.ts hive:agentLatestText, off the same hook-learned transcript
path as hive:agentContext.
preload/index.ts window.cth.agentLatestText + LatestAssistantText.
hooks/useHive.ts effect 2c-bis: 3s per-agent poll (not 15s — this is on the
critical path of a live call) writing both store fields,
and only when they actually moved.
Also bumps RELAY_WAIT_MS 25s → 90s: a tool-using Claude Code turn routinely
outlives 25s, so even a working relay would have reported "still working".
test/talk-relay-live-path.test.cjs is deliberately NON-mock: it runs the real
producer over real Claude Code JSONL on disk, drives agentVoice's exact wait
condition with the values the renderer poll derives, and pins the whole wire
(transcript → IPC → preload → store) plus "mockEvents is not the only writer".
Removing the producer fails it — the original bug could not ship green again.
Verified: typecheck + build clean, test:focused 559/559, and the producer read
back the correct last assistant message from a live transcript.
Contributor
|
✅ Evidence received. Before and after are both attached. Thanks — this is what makes a PR reviewable in one pass. |
aarjav
force-pushed
the
agent/worker-talk-per-agent-dwight
branch
from
August 26, 2026 02:25
c3955ce to
2522336
Compare
Author
|
Before and after screenshots. If a video is ABSOLUTELY needed I can upload one. 
|
Replace the single shared worker voice with voiceForAgent(target): Michael (god) keeps cedar; each known agent gets its own hand-picked voice so the familiar names sound as expected (pam=marin, jim=ash, dwight=verse, oscar=echo, angela=sage), and unknown agents hash their id into a cedar-free pool, stable across sessions. Worker scope guard untouched. Pins the map with a test; suite 559 -> 563. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this adds
Per-agent Talk — the live "Talk" voice button, until now available only on the orchestrator (god/Michael) card, now works on worker agent cards too. Clicking Talk on a worker card opens an OpenAI-realtime, speech-to-speech session that adopts that agent's identity: its name, its registry role, and a digest of its own
memory.md.marin; god keepscedar, so you can hear which agent picked up.ask_my_session,my_latest_reply,check_my_session,read_my_memory,get_my_tasks). They cannot hire, kill, pause, archive, dispatch other agents, or change settings — that stays god's console.session.tsonly reaches into the new per-agent module for non-god targets.How the voice ↔ session relay works
The voice session is the phone line to a working agent, not a separate brain.
ask_my_sessionrelays what the operator says into the agent's live Claude Code session over the two paths the app already trusts — the renderer message queue for an idle agent, a steer note for a busy one (chosen deterministically in code, never by the model) — then reads the session's reply back and speaks it.Bug caught in live QA and fixed here (A3)
The first live test surfaced a real gap: the operator's question reached the agent's session (visible in the terminal), but the session's answer never returned to the voice — the voice layer polled
recentAssistantText/recentTextTs, but nothing in the live pipeline ever wrote those fields (only mock data did, which is why unit tests were green while the live path was dead).The fix (commit
2522336) adds the missing producer:main/transcript.tssurfaces each agent's latest assistant message from its transcript;index.ts/preload);useHive's per-agent poll (~3s), sorecentAssistantTextnow has a live writer, not just mock;RELAY_WAIT_MSraised to 90s (real Claude Code turns routinely exceed 25s);Evidence
Before
Pre-feature build: the Talk button appears only on the orchestrator (BOSS / god) card. The worker cards — Jim and Pam — have no Talk button, so there is no way to hold a live voice conversation with a worker agent.
After
This build: worker cards — Jim and Pam — now show the Talk button, and a live worker voice session is active (god shown "listening"). Clicking Talk on a worker and asking "what are you working on?" returns the agent's answer spoken back in its own voice (
marin) within ~90s, sourced from its live session. god's Talk is unchanged (cedar).Testing
559 / 559tests pass, including the new non-mock live-path test.Known issues
marin) — so Jim and Pam sound identical to each other, though both remain clearly distinct from Michael/god (cedar). This is confirmed in code:AGENT_VOICEis one shared constant rather than a per-agent assignment. Per-worker voice differentiation is a planned follow-up, not part of this PR.Commits
bd7b1f2— feat(realtime): per-agent Talk — voice sessions for worker cards2522336— fix(realtime): per-agent Talk A3 — write the session's reply back to the store🤖 Generated with Claude Code