A modern multi-provider speech gateway featuring Alibaba FunASR local STT, Gemini Live real-time voice chat, Gemini 3.1 Flash TTS, Hume AI Octave, and ElevenLabs voice cloning — built for embodied agents and humanoid-scale voice (wake → understand → fleet act → speak).
Good speech perception and reply are load-bearing for humanoids: hands-free commands, noisy environments, structured transcripts for planners, and local STT without per-minute cloud tax. Chinese open-weight industrial speech (FunASR, SenseVoice, CosyVoice, and related stacks) is shipping as deployable tooling — ModelScope/HuggingFace weights, ONNX edge, funasr-server — aligned with robotics and agent fleets at scale.
speech-mcp is the fleet voice layer: FunASR default STT, Voice Command Bus to fleet-agent (robot missions), optional cloud TTS/live for social quality.
Read the full thesis and architecture: docs/HUMANOID_VOICE.md
MCP Server — Advanced speech, RAG, and state management for agents and IDEs (Claude Desktop, Cursor, Windsurf).
Modern Webapp — A browser-based cockpit for real-time voice conversations, Creative Labs polyglot synthesis, voice clone management, and system monitoring.
Chinese open-weight speech stacks are ahead on industrial ASR: open weights on ModelScope/HuggingFace, published RTF benchmarks, and deploy paths from 234M ONNX to 7.7B GPU — without per-minute cloud STT billing. Among them, Alibaba FunASR is the integrated default in speech-mcp because it unifies VAD + ASR + punctuation + diarization in one AutoModel() call and ships a production toolkit (funasr-server, ONNX, Docker, native MCP in upstream v1.3.3+).
| What you get | Detail |
|---|---|
| Speed | Up to ~170× realtime on GPU, ~17× on CPU (vs ~13× for Whisper-large-v3 in published tables) |
| Models | Fun-ASR-MLT-Nano-2512 (multilingual, 31 langs incl. EN/JA/DE), SenseVoiceSmall, Paraformer family |
| Structured output | Timestamps, speaker labels; emotion/event tags via SenseVoice |
| Edge | PyTorch (cuda/cpu/mps), ONNX INT8, Windows runtime SDK, OpenAI-compatible sidecar on 10914 |
| MCP tools | transcribe_audio_file, transcribe_stream_chunk (default provider=funasr) |
| REST | POST /api/v1/transcribe?provider=funasr |
| Fleet voice bus | Post-wake utterance STT when FLEET_VOICE_DELEGATE=1 — auto-start listener, spoken wake greeting, sleep word, spoken replies, audible timer announcements (VOICE_COMMAND_BUS.md) |
| Papers | Fun-ASR (2509.12508), FunAudioLLM (2407.04051), SenseVoice (2401.04251) |
| Upstream | modelscope/FunASR |
uv sync --extra funasr
Copy-Item .env.example .env
# In .env:
# FUNASR_ENABLED=true
# FUNASR_MODEL=FunAudioLLM/Fun-ASR-MLT-Nano-2512
# FUNASR_DEVICE=cuda:0Sidecar (no torch in speech-mcp process):
uv sync --extra funasr
uv run python scripts/start_funasr_sidecar.py
# .env: FUNASR_OPENAI_URL=http://127.0.0.1:10914/v1Full guide (env matrix, benchmarks, licensing, sidecar API): docs/providers/funasr.md
speech-mcp is a gateway: cloud TTS/live voice plus local STT via FunASR. Other Chinese open models are documented for comparison and future providers — see docs/CHINESE_AI_RESEARCH.md.
| Project | Family | Primary role | speech-mcp | Learn more |
|---|---|---|---|---|
| FunASR | Alibaba / Tongyi | Industrial STT (VAD+punc+diarization) | Integrated (default STT) | funasr.md |
| SenseVoice | FunAudioLLM | Fast multilingual STT + emotion/events | Via FunASR hub models | CHINESE_AI_RESEARCH.md |
| Fun-ASR | FunAudioLLM | Nano ASR (2512), low-latency | Via FUNASR_MODEL |
funasr.md |
| CosyVoice 2/3 | FunAudioLLM | Zero-shot multilingual TTS, cloning | Future candidate | CHINESE_AI_RESEARCH.md |
| ChatTTS | 2noise | Conversational TTS (pauses, laughter) | Future candidate | CHINESE_AI_RESEARCH.md |
| GPT-SoVITS | Community | Few-shot voice cloning (ZH/JP strong) | Future candidate | local_voice_alternatives.md |
| Fish Speech | Fish Audio | Open TTS / voice clone | Not integrated | CHINESE_AI_RESEARCH.md |
| WeNet | Community | End-to-end ASR toolkit | Not integrated (FunASR preferred here) | CHINESE_AI_RESEARCH.md |
| FireRedASR | Xiaohongshu | Industrial Mandarin/English ASR | Not integrated | CHINESE_AI_RESEARCH.md |
Why FunASR over “just run Whisper”: one pipeline for agents (segments + speakers + punctuation), faster published RTF on Chinese/multilingual industrial sets, first-class MCP + OpenAI sidecar in this repo, and alignment with the FunAudioLLM model line (SenseVoice, CosyVoice) if you add local TTS later.
Offline duplex conversation (not batch STT): use kyutai-mcp (Moshi) alongside speech-mcp — see local_voice_alternatives.md.
| Provider | Mode | Quality | Key |
|---|---|---|---|
funasr |
Batch + chunk STT (local) | Highest local speed | FUNASR_ENABLED |
gemini_live |
Real-time conversation | Very good | GOOGLE_API_KEY |
gemini |
Batch TTS | Highest | GOOGLE_API_KEY |
gemma |
Batch TTS/STT | SOTA Local | None |
hume |
Batch TTS (Octave) | High | HUME_API_KEY |
elevenlabs |
Batch TTS + voice cloning | High | ELEVENLABS_API_KEY |
windows |
Batch TTS (SAPI5) | Low | None |
FunASR Local STT — Alibaba Fun-ASR-MLT-Nano integrated natively (multilingual: EN/JA/DE + 28 more). Structured transcripts with speakers and timestamps. OpenAI-compatible sidecar on port 10914. Streaming STT with barge-in (ja/en/de, CPU) via sherpa-onnx: docs/STREAMING_ASR.md. See FunASR guide.
Gemma 4 Native Multimodal — SOTA 2026 local engine integration. Features native audio/vision encoders for low-latency conversational reasoning. Supports prosody-aware interaction and local-first Zero-STT fallback. Optimized for A4B throughput (100+ t/s).
Gemini 3.1 Flash TTS — Highest-quality cloud synthesis (gemini-3.1-flash-tts-preview). 31 prebuilt voices, 100+ languages, expressive audio tags ([whispers], [excited], etc.).
Creative Labs — Polyglot synthesis demo with 19 languages (European, Slavic, Classical, Experimental, Global), literary samples, voice selection, prosody slider, and tongue-twister panel.
Voice Cloning — ElevenLabs Instant Voice Clone (IVC) via file upload. 5-second minimum audio sample. Cloned voices appear in the voice library immediately.
Offline Wake-Word — Privacy-first detection using openWakeWord (fully offline, Apache 2.0, no API key).
Fleet Voice Command Bus — With FLEET_VOICE_DELEGATE=1, wake → utterance STT → POST fleet-agent /api/voice/intent (e.g. "boomy go on patrol…"). Humanoid-scale pattern: docs/HUMANOID_VOICE.md. Ops: docs/VOICE_COMMAND_BUS.md.
RAG / Semantic Search — LanceDB + FastEmbed knowledge base over project docs. ask_docs tool uses Claude sampling for grounded Q&A.
Voice Intelligence (2026-08) — persistent voice memory (episodic diary), voice macros (spoken phrases → actions), translation bridge (FunASR → local LLM → TTS), sound-event detection, voice bank (named provider-routed voice profiles), speech analytics (real latency/cost telemetry), spoken fleet readouts, and a skill-first Chat page with personalities. Spec: docs/FEATURE_SPEC_2026.md.
Local AI — Ollama and LM Studio model discovery and grounded generation.
- Humanoid voice — thesis & fleet architecture ← why speech-mcp / FunASR / China open speech
- FunASR local STT (full guide) ← start here for STT
- Chinese FOSS speech landscape ← SenseVoice, CosyVoice, ChatTTS, GPT-SoVITS, …
- Installation — includes
uv sync --extra funasr - Configuration reference
- Tools reference —
transcribe_*MCP tools - Local voice alternatives ← kyutai-mcp / offline duplex
- Gemini Live voice chat
- Architecture
- openWakeWord
- Yahboom robot integration
- RAG technical overview
- Modern speech AI
git clone https://github.com/sandraschi/speech-mcp
cd speech-mcp
justThis opens an interactive dashboard showing all available commands. Run just bootstrap to install dependencies, then just serve or just dev to start.
If you don't have just installed:
git clone https://github.com/sandraschi/speech-mcp
cd speech-mcp
uv sync
cp .env.example .env
# Edit .env — add GOOGLE_API_KEY for cloud TTS; FUNASR_ENABLED=true for local STT
uv run python -m speech_mcp.webapp
cd web
npm install
npm run devBackend: http://localhost:10909 — Frontend: http://localhost:10908
For Claude Desktop MCP integration see docs/configuration.md.
| Channel | Command / artifact |
|---|---|
| Claude Desktop | Release .mcpb or just mcpb-pack |
| Windows app | Release NSIS/MSI or just build-native |
| Developers | just start (backend 10909 + Vite 10908) |
Fleet pattern: mcp-central-docs/standards/rules/tauri_godot_sota.md. Details: docs/DEVELOPMENT.md.
MIT — see LICENSE.
Contributors: @sandraschi. PRs welcome.