Skip to content

implementation_plan.md

Latest commit

 

History

History
198 lines (156 loc) · 10.2 KB

implementation_plan.md

File metadata and controls

198 lines (156 loc) · 10.2 KB
Raw

OpenLive → GPT-Live Parity — Implementation Plan

Status date: 2026-08-28
Baseline: 26.7.16 (call-first surface + open voice stack + agent workspace)
Goal: Model-neutral, open, production-grade competitor to OpenAI GPT-Live / Advanced Voice Mode.

Target architecture

┌─────────────────────────────────────────────────────────────────────────────┐
│                         BROWSER CLIENT                                      │
│  Call-first UI: Bloub face · Mute/End/Settings · Advanced lab disclosure    │
│  RNNoise worklet → Silero-style VAD → NLMS AEC → FIR resample               │
│  WebRTC (data-channel PCM) ← fallback WebSocket Binary PCM                  │
└────────────────────────────────┬────────────────────────────────────────────┘
                                 │
┌────────────────────────────────▼────────────────────────────────────────────┐
│  GATEWAY: WebRTC signaling · Semantic endpointing · MCP · Safety · Chronos  │
│  Providers: Mock · Cascade · Realtime · Moshi · HybridStreaming             │
│  TTS demo policy: Piper or silence (formant only if explicitly selected)    │
└─────────────────────────────────────────────────────────────────────────────┘

Phase status

Phase 1: Client-Side Intelligence & Audio Quality — DONE (v26.7.16)

Item Status Notes
RNNoise suppression worklet Done (JS Wiener) Credited RNNoise lineage; WASM swap-in ready
Silero VAD worklet Done (spectral JS) Credited Silero lineage; ONNX path stubbed
NLMS adaptive AEC Done NlmsAec in audio-utils.js
Polyphase / windowed-sinc FIR resampler Done audio-utils.js
Chain in AudioSession Done mic → RNNoise → Silero → capture → NLMS → FIR → PCM16
Open AI voice (Piper) Done Piper-first roster; demo path neural-or-silence
Call-first UI Done Bloub + Morphicons; Advanced lab disclosure
THIRD_PARTY_NOTICES Done Root notices file (Bloub / Morphicons / Piper)

Phase 2: WebRTC / Opus Transport — PARTIAL

Item Status Notes
/v1/realtime/session + client secret Done Gateway route + create_client_secret on provider trait
Browser RTCPeerConnection + oai-events Done app.js offer/answer with OpenAI Realtime SDP
Silent renegotiation on disconnect Done Token expiry / disconnect → silentWebRtcReconnect
Deep PLC + jitter enhancements Done Stats jitter + pitch-period PLC in playback worklet
ICE config endpoint Done GET /v1/webrtc/ice STUN roster
SDP offer scaffold Done POST /v1/webrtc/offer
Gateway-native WebRTC (webrtc-rs) Done ICE/DTLS + data channels for events/PCM media
Client gateway WebRTC path Done Prefer native hub when gateway_webrtc
WebRTC Chronos/endpointing bridge Done Auto-commit + ASR final + capabilities
RTP Opus tracks Todo Optional; data-channel PCM is the production path for now

Phase 3: Semantic Endpointing & Transcript Revision — PARTIAL

Item Status Notes
Client webkitSpeechRecognition → transcript deltas Done app.js
Semantic completeness + 200 ms early endpoint Done openlive-audio observe_with_semantic; session wiring
Cascade overlapping windows / ASR revisions Done ~400 ms prior PCM prepend on cascade commit
Chronos in-place transcript revisions + UI anim Done reviseText + .bubble-revise flash

Phase 4: Native Duplex (Moshi) — PARTIAL

Item Status Notes
Moshi WebSocket adapter Done moshi.rs + --provider moshi
Multi-model Fast / Deep cognition Done --deep-llm-model + heuristic router

Phase 5: Safety, MCP, Retrieval, Agent workspace — PARTIAL (strong in 26.7.16)

Item Status Notes
Streaming safety holdback Done safety.rs + session wiring (--safety)
MCP JSON-RPC client Done mcp_client.rs + /v1/mcp/tools + /v1/mcp/call
Knowledge retrieval inject Done keyword store from --knowledge-dir
Internal agent tools + confirm Done search, calc, sandbox, browse, profile; /v1/agent/*
Multi-agent pool ≤50 + SSE Done agent_pool, pool jobs, classes
Sandbox FS + headless capture Done path-safe I/O; Chrome/Edge dump-dom/shot/PDF
Durable profile + memory Done /v1/profile, /v1/memory, facts DnD
Embedding vector DB Todo swap behind KnowledgeStore API
Tool authz audit suite Todo multi-tenant cancel/audit soak

Phase 6: Deadlines & Session Limits — PARTIAL

Item Status Notes
Unlimited session cap UI default Done Settings sessionCap=0
WebRTC token refresh / renegotiate Done Client-side
Configurable DEFAULT_TASK_DEADLINE_MS Done --task-deadline-ms + atomic override
Durable session persistence Done JSONL store (data/openlive-sessions, SQLite-shaped API)

Phase 7: Visualization & UI — PARTIAL

Item Status Notes
Call-first minimal black surface Done Auto-join; Mute / End / Settings dock
Bloub face engine Done Vendored MIT vendor/bloub/; not affiliated with xAI
Morphicons call controls Done Mic ↔ MicOff, Settings ↔ X, PhoneOff end
Advanced lab disclosure in Settings Done Camera, sandbox, agents, diagnostics
Full-screen voice mode Done Settings toggle, F shortcut, exit button
Ripple click feedback + sheet motion Done app.js + styles.css
Legacy Canvas visualizer Kept Hidden behind Bloub on the default path
Three.js WebGL Icosphere + GLSL Dropped Bloub covers the face; not required

Phase 8: Emotion-Aware Responses — PARTIAL

Item Status Notes
Emotion feature extraction Done emotion-detector.js valence/arousal
Modulate barge-in / pause Done Wired in audio-session local duck thresholds
Provider emotion-conditioned replies Todo Needs LLM system hint injection

Phase 9: Full-Duplex Benchmarking — PARTIAL

Item Status Notes
openlive-full-duplex-bench Done Local Chronos commit latency gate
VoiceBench dataset harness Todo

Phase 10: Provider UI & Developer API — PARTIAL

Item Status Notes
Provider chip + enriched /v1/providers Done Features map, recommended Piper voices
Provider catalog Done GET /v1/providers/catalog + --provider hybrid
Built-in provider catalog (offline setup) Done 12 providers in app.js BUILTIN_PROVIDER_DETAILS
Developer REST (/health JSON, /v1/meta, /v1/sessions, /v1/tasks) Done Optional OPENLIVE_API_KEY
Transcript export Done Client JSON export + GET /v1/sessions/{id}/transcript
VisualCard + live translation (mock) Done Protocol rev 4; language chip injects translate mode
Multi-process provider hot-swap UI Todo Restart with --provider still required

Phase 11: Desktop & Distribution — PARTIAL

Item Status Notes
Tauri v2 shell Done apps/openlive-desktop/ Windows MSI + macOS DMG/App
Gateway child-process spawn Done main.rs waits for health, kills on exit
Auto-updater / code signing Todo
Linux AppImage / Flatpak Todo

Shipped in working tree as of 26.7.16

  • WebRTC signaling + gateway-native data-channel path + browser peer
  • Semantic VAD hybrid + cascade ASR prior window + revise UI
  • Call-first Bloub surface + Morphicons Mute/End/Settings + Advanced lab disclosure
  • Piper TTS with neural-or-silence demo policy (formant only if explicitly selected)
  • Agent tools, multi-agent pool, sandbox, profile/memory, confirms
  • Version surface: Cargo / UI / living docs all 26.7.16

Verify

# Rust unit tests (release)
cargo test --workspace --release

# Integration tests need a debug binary:
cargo test --workspace

# Frontend JS unit tests
node --test apps/openlive-gateway/web/tests/*.test.js

Recommended next session order

  1. Phase 2 Opus RTP — optional media tracks + impairment harness
  2. Phase 1 polish — optional RNNoise WASM / Silero ONNX vendor weights
  3. Transcript editing — last soft GAP on the voice surface
  4. Tool authz audit / soak — production-depth gates on agent sandbox
  5. Benchmark qualification — Full-Duplex-Bench / VoiceBench manifests
  6. Phase 4 Moshi — true native duplex
  7. Phases 7–10 — Emotion cues, benches, developer API polish (orb face ships via Bloub)

File map (Phase 1 + prior handoff)

Path Role
apps/openlive-gateway/web/audio-utils.js FIR resample, NLMS AEC, echo correlator
apps/openlive-gateway/web/rnnoise-worklet.js Spectral NS worklet
apps/openlive-gateway/web/silero-vad-worklet.js Client VAD worklet
apps/openlive-gateway/web/audio-session.js Capture graph + NLMS + blend
apps/openlive-gateway/web/app.js WebRTC + speech recognition
apps/openlive-gateway/web/voice-visualizer.js ChatGPT orb
apps/openlive-gateway/src/main.rs /v1/realtime/session
crates/openlive-audio/src/lib.rs Semantic endpointing
crates/openlive-provider/src/openai_realtime.rs Client secret minting