Skip to content

Add runtime-only voice plugin (STT + TTS) - #16

Open
devteapot wants to merge 7 commits into
mainfrom
feat/voice-plugin-stt-tts
Open

Add runtime-only voice plugin (STT + TTS)#16
devteapot wants to merge 7 commits into
mainfrom
feat/voice-plugin-stt-tts

Conversation

@devteapot

@devteapot devteapot commented May 29, 2026

Copy link
Copy Markdown
Owner

Context

Adds a first-party voice plugin exposing speech-to-text and text-to-speech as SLOP affordances, mirroring the LLM endpoint/adapter/credential layer from the #15 refactor. Scope is runtime-only: the plugin performs transcription/synthesis through the runtime's endpoint + credential config and exposes results over SLOP. Audio hardware (mic capture, playback) is a deliberately deferred surface concern.

What's here

  • src/stt/, src/tts/ — adapter interfaces + cloud (OpenAI, Deepgram, ElevenLabs) and local (faster-whisper, Kokoro, Piper) adapters, factories, default catalogs. Local vs cloud is just baseUrl + auth: { type: none }, exactly like the ollama LLM endpoint.
  • src/voice/profile-manager.ts — VoiceProfileManager resolving profile → endpoint → credential → adapter, reusing createCredentialStore() under a voice: key prefix, with per-modality readiness (ready | needs_credentials | not_configured) so a partial pipeline (STT-only or TTS-only) is first-class.
  • VoiceProvider — /session /stt /tts /approvals nodes; transcribe / synthesize / set_profile affordances. Audio crosses the SLOP boundary as base64.
  • Session plugin — mounts /voice and, when the active TTS profile has autospeak enabled, synthesizes each completed assistant turn in onTurnComplete and publishes the audio as a voice session extension for a future client to play.
  • voiceNetworkRule — requires approval before audio/text is sent to a non-local endpoint; local endpoints run without a prompt.
  • Config nested under plugins.voice; commented .sloppy/config.example.yaml; docs/18-voice-plugin.md.

Integration is one FIRST_PARTY_PLUGINS entry — no changes to the agent loop, turn coordinator, consumer/hub, or the TUI client surface.

Tests

31 new tests (config parse, adapters via mocked fetch, profile-manager readiness incl. env/secure_store/local-none, provider affordances via SlopConsumer, policy local-vs-remote, autospeak helper + /voice node + onTurnComplete extension publish).

  • bun test — 624 pass, 0 fail
  • bunx tsc --noEmit — clean
  • bunx biome check . — clean

Deferred (surface follow-ups, not built)

  • TUI push-to-talk mic capture + STT → composer.send_message re-entry
  • Client (TUI / native apps/sloppy-voice) playback of the /voice autospeak audio
  • Audio device selection; streaming STT/TTS
  • Content-ref handling for large audio blobs (currently inlined as base64)

🤖 Generated with Claude Code

@devteapot
devteapot force-pushed the feat/voice-plugin-stt-tts branch 2 times, most recently from a285189 to 6d35c91 Compare June 10, 2026 12:11
@devteapot
devteapot force-pushed the feat/voice-plugin-stt-tts branch 7 times, most recently from adb79cc to bfaee1d Compare July 15, 2026 14:25
devteapot and others added 7 commits July 22, 2026 06:02
@devteapot @claude
Add runtime-only voice plugin (STT + TTS)
fe206a6 
Adds a first-party `voice` plugin exposing speech-to-text and text-to-speech
as SLOP affordances, mirroring the LLM endpoint/adapter/credential layer.

- src/stt, src/tts: adapter interfaces + cloud (OpenAI, Deepgram, ElevenLabs)
  and local (faster-whisper, Kokoro, Piper) adapters, factories, catalogs.
  Local vs cloud is just baseUrl + auth:{type:none}, like the ollama endpoint.
- src/voice/profile-manager.ts: profile -> endpoint -> credential -> adapter
  resolution, reusing createCredentialStore() under a voice: key prefix, with
  per-modality readiness (ready | needs_credentials | not_configured).
- VoiceProvider: /session /stt /tts /approvals nodes; transcribe / synthesize /
  set_profile affordances. Audio crosses SLOP as base64.
- Session plugin: mounts /voice and, on autospeak, synthesizes completed
  assistant turns and publishes audio as a `voice` session extension for a
  future client to play.
- voiceNetworkRule gates audio/text egress to non-local endpoints via approval.
- Config nested under plugins.voice; example yaml + docs/18-voice-plugin.md.

Integration is one FIRST_PARTY_PLUGINS entry; no changes to the agent loop,
turn coordinator, consumer/hub, or the TUI client surface. Audio hardware
(mic capture, playback) is a deferred surface concern.

Tests: 31 new (config parse, adapters via mocked fetch, profile-manager
readiness, provider affordances, policy, autospeak). Full suite green;
tsc and biome clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@devteapot
--wip-- [skip ci]
9be4ebe
@devteapot @claude
Replace dual-path voice pipeline with streaming-only speech architecture
8e796f0 
Rewrite the voice pipeline around a runtime-owned streaming contract in
src/speech/: realtime STT over OpenAI-Realtime-like WebSocket dialects
(openai GA + vllm maps over one session core) and sentence-pipelined
streaming TTS over /v1/audio/speech with PCM output. Protocols are
registry-resolved strings so future providers can arrive as plugins.

Delete the batch path entirely: deepgram/elevenlabs/piper/openai-
transcriptions adapters, utterance capture, the transcribe/synthesize
affordances, and the autospeak plugin. The voice provider slims to a
profile/readiness surface with set_profile.

Conversation loop fixes folded into the rewrite:
- privacy boundary enforced: createSpeechNetworkRule gates the session
  provider's start_listening, checks both endpoints via the manager's
  own selection, matches ws/wss localhost; non-local continuous
  autostart parks in needs_approval
- realtime sessions emit a single closed event (local/remote/error);
  unexpected closes reconnect with exponential backoff
- live state (phase, throttled partial transcripts, connection health)
  publishes through a session extension record so /conversation
  actually refreshes
- streamed playback via a long-lived play process per reply; sample
  rates are per-endpoint config ({rate}-templated commands)
- synchronous start guard; stop during thinking suppresses the reply

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@devteapot @claude
Split speech layer into core mechanism and plugin protocol bindings
8d21ae2 
src/speech now contains zero vendor/protocol knowledge: the contract,
the (empty-at-start) protocol registry, the profile manager with
injected registry + endpoint defaults, the sentence segmenter, and new
generic streaming helpers (waitForOpen, closed-once guard,
AsyncChunkQueue) in streaming.ts.

The first-party bindings move to the voice plugin:
- voice/protocols/realtime-stt/{session,dialects}.ts and
  voice/protocols/openai-speech.ts (tracked as renames)
- voice/protocols/index.ts registers them idempotently from
  speechManagerFor() in catalog.ts, which all consumers route through
- voice/endpoints.ts holds the built-in endpoint defaults; the
  dgx-nemotron built-in is deleted entirely (site-specific endpoints
  are user config)

The plugin-boundary test now enforces both directions: plugins never
import session store internals, and src/speech never imports from the
plugin layer. Future speech providers are plugins that implement the
src/speech contract and call registry.register* — no core edits.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@devteapot @claude
Images registry: trail-resident images with a model-managed lifecycle
e0f9fc5 
Image bytes now live in an in-memory ImageRegistry exposed as the
first-party images provider (/gallery), not in conversation history.
Loaded images ride the per-turn state trail message as interleaved
caption+image blocks; since the trail is hard-replaced every request,
unloading an image truly removes it from context — history only keeps
tiny "/gallery/img-N" refs.

Ingestion: tool results with file:// image content_refs (e.g. the reachy
provider's capture_frame) register into the gallery, with the ref noted
at the tail of the result text; user [Image #N](file://...) attachments
register the same way, falling back to inline blocks when the plugin is
disabled. The previous tool-result image embedding (ToolResultContentBlock
.images + per-adapter emission) is deleted.

Lifecycle: images auto-load with a 1-turn TTL — glance once, describe,
drop the pixels. The model drives describe (one-liner that identifies an
unloaded image), load (optional ttl_turns), unload, pin (protects from
TTL/LRU/storage eviction without forcing attachment) and remove; auto
expiry plus maxLoaded LRU and maxStored caps backstop housekeeping, since
trail images re-bill at the full input rate every turn they stay loaded.

The collection node is "gallery" rather than "images": slop providers
strip a leading /<provider-id>/ from invoke paths, so a node named after
the provider id makes item affordances unresolvable.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@devteapot
Add inline emote markers to voice conversation
cc9253f
@devteapot
Refactor session runtime and provider plumbing
0be180c
@devteapot
devteapot force-pushed the feat/voice-plugin-stt-tts branch from bfaee1d to 0be180c Compare July 22, 2026 04:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@devteapot