From dd4652f1d3155b1fb131fb1b25c06a87c9935e11 Mon Sep 17 00:00:00 2001 From: Cursor Agent Date: Fri, 28 Aug 2026 14:10:19 +0000 Subject: [PATCH] Remove Cloud Agent scaffolding and sync docs to call-first UI. Delete .cursor (environment install/start scripts are not part of the product). Rewrite living docs and notices so they match the shipped surface: Bloub call, Mute/End/Settings, neural-or-silence TTS, Advanced lab disclosure, no forced splash or Google Fonts CDN story. Co-authored-by: Byte271 --- .cursor/environment.json | 5 -- .cursor/install.sh | 9 -- .cursor/start-gateway.sh | 29 ------- README.md | 68 ++++++--------- THIRD_PARTY_NOTICES.md | 5 +- apps/openlive-desktop/README.md | 11 ++- docs/architecture-roadmap.md | 5 +- docs/architecture.md | 3 +- docs/evaluation.md | 4 +- docs/gpt-live-parity.md | 14 ++-- docs/open-source-stack.md | 16 ++-- docs/production-readiness.md | 16 ++-- docs/release-26.7.15.md | 4 + docs/release-26.7.16.md | 141 ++++++++++++-------------------- implementation_plan.md | 35 ++++---- 15 files changed, 138 insertions(+), 227 deletions(-) delete mode 100644 .cursor/environment.json delete mode 100755 .cursor/install.sh delete mode 100755 .cursor/start-gateway.sh diff --git a/.cursor/environment.json b/.cursor/environment.json deleted file mode 100644 index b4af7ba..0000000 --- a/.cursor/environment.json +++ /dev/null @@ -1,5 +0,0 @@ -{ - "name": "OpenLive", - "install": "bash .cursor/install.sh", - "start": "bash .cursor/start-gateway.sh" -} diff --git a/.cursor/install.sh b/.cursor/install.sh deleted file mode 100755 index f354784..0000000 --- a/.cursor/install.sh +++ /dev/null @@ -1,9 +0,0 @@ -#!/usr/bin/env bash -# Cloud Agent install: pin Rust 1.98 (lockfile needs edition 2024 / rustc >= 1.85), -# then fetch and compile the workspace including test binaries. -set -euo pipefail - -rustup toolchain install 1.98.0 --component rustfmt --component clippy --no-self-update -rustup default 1.98.0 -cargo fetch --locked -cargo build --workspace --locked --all-targets diff --git a/.cursor/start-gateway.sh b/.cursor/start-gateway.sh deleted file mode 100755 index 8362054..0000000 --- a/.cursor/start-gateway.sh +++ /dev/null @@ -1,29 +0,0 @@ -#!/usr/bin/env bash -# Per-boot: mock OpenLive gateway on :8787. Idempotent; returns after /health is OK. -set -euo pipefail - -mkdir -p /tmp/openlive -if curl -sf --max-time 2 http://127.0.0.1:8787/health >/dev/null; then - echo openlive-gateway already healthy - exit 0 -fi -if [ ! -x ./target/debug/openlive-gateway ]; then - echo missing ./target/debug/openlive-gateway >&2 - exit 1 -fi -setsid ./target/debug/openlive-gateway \ - --listen 0.0.0.0:8787 \ - --provider mock \ - --web-dir apps/openlive-gateway/web \ - /tmp/openlive/gateway.log 2>&1 & -echo $! >/tmp/openlive/gateway.pid -for _ in $(seq 1 60); do - if curl -sf --max-time 2 http://127.0.0.1:8787/health >/dev/null; then - echo openlive-gateway ready - exit 0 - fi - sleep 0.25 -done -echo gateway failed to become healthy >&2 -cat /tmp/openlive/gateway.log >&2 || true -exit 1 diff --git a/README.md b/README.md index eb5c55c..726424f 100644 --- a/README.md +++ b/README.md @@ -3,32 +3,31 @@ OpenLive is an open, model-neutral runtime for continuous voice agents. It separates deadline-sensitive interaction continuity from slower model cognition and preserves native duplex provider capabilities instead of forcing every model through a text-chat abstraction. > [!IMPORTANT] -> **Disclaimer:** This is an independent open-source project. It is **not an official OpenAI project** and has no affiliation with OpenAI. +> **Disclaimer:** This is an independent open-source project. It is **not an official OpenAI project** and has no affiliation with OpenAI or xAI. ## Current status -**Version 26.7.16** (`v26.7.16`) targets a **GPT-Live-comparable** experience: polished live voice UI, open neural speech (Piper), client-side audio intelligence, WebRTC session path, semantic endpointing, **real tools + multi-agent sandbox**, and durable profile/memory — with original visuals and model neutrality intact. +**Version 26.7.16** targets a **GPT-Live-comparable** experience: a call-first live surface, open neural speech (Piper), client-side audio intelligence, WebRTC session path, semantic endpointing, **real tools + multi-agent sandbox**, and durable profile/memory — with original visuals and model neutrality intact. Full parity matrix: [`docs/gpt-live-parity.md`](docs/gpt-live-parity.md) · Architecture roadmap: [`docs/architecture-roadmap.md`](docs/architecture-roadmap.md) · Open stack guide: [`docs/open-source-stack.md`](docs/open-source-stack.md) · Credits: [`THIRD_PARTY_NOTICES.md`](THIRD_PARTY_NOTICES.md) · Release notes: [`docs/release-26.7.16.md`](docs/release-26.7.16.md) ### What 26.7.16 ships -**Voice surface** +**Call-first voice surface** -- Minimal black live surface + setup wizard; Live Presence theme tokens. -- Full-screen voice presence with 11 named modes and multi-layer procedural orb. -- Inline layout toggle, live dual transcript, conversation modes, speaking-style axes. -- Push-to-talk, barge-in with local duck, camera/screen share affordances, visual cards. -- **Piper-first voice roster** (Lessac, Amy, Ryan, …) plus API-compatible fallbacks. -- **Boot splash** with live status and animated page-load entrance. -- **Ripple click feedback** and enhanced hover/lift micro-interactions across the UI. -- Brand chrome and package version: **26.7.16**. +- First paint is a live call: **Bloub** face on a black stage + **Mute / End / Settings** (Morphicons). +- Session **auto-joins**. If the browser needs a gesture for the mic, the surface says **Tap to talk**. +- Lab chrome (camera, screen share, sandbox, multi-agent, diagnostics) lives in **Settings → Advanced**. +- Optional setup wizard remains available from Settings; it is not forced on launch. +- Full-screen mode, live transcript, conversation modes, and speaking-style axes stay available behind Settings / shortcuts. +- **Demo TTS policy: neural or silence.** Auto never pads with formant/mock. Formant only when you explicitly pick it. +- Package version: **26.7.16**. **Open AI voice** - Production path: cascade → OpenAI-compatible **Piper** TTS (via LocalAI, openedai-speech, or gateway-local Piper). -- Gateway: `GET /v1/tts/status`, `POST /v1/tts/speak`; formant fallback for demos. -- **Graceful TTS fallback chain**: gateway TTS (Piper/formant) → browser TTS → text-only, so the conversation never hangs. +- Gateway: `GET /v1/tts/status`, `POST /v1/tts/speak`. +- Auto engine = Piper or silence until ready; browser / formant only when selected in Settings. - Licenses and attribution in `THIRD_PARTY_NOTICES.md`. **Client audio intelligence** @@ -36,35 +35,32 @@ Full parity matrix: [`docs/gpt-live-parity.md`](docs/gpt-live-parity.md) · Arch - RNNoise-style noise suppression worklet (10 ms frames). - Silero-style VAD worklet + energy blend. - NLMS adaptive echo cancellation + windowed-sinc resampler. +- Local-first barge-in: `local duck → soft_duck → hard_yield → cancel`. **Agent, tools & sandbox** - Internal agent (no OpenCode): search, deep research pool, calculator, time, identity, profile. - Path-safe sandbox file I/O + optional Chrome/Edge headless browse / screenshot / PDF. - Multi-agent pool (≤50) with SSE progress, agent classes, and destructive-action confirms. -- Durable user profile (facts editor, drag-and-drop reorder) + session memory export. +- Durable user profile (facts editor) + session memory export. - See `sandbox/README.md` and `docs/architecture-roadmap.md`. **Transport, providers & tasks** - Binary WebSocket PCM + **gateway-native WebRTC** (DTLS data channels for events/PCM). - Provider-edge WebRTC (OpenAI Realtime SDP) when secrets are available. -- **Coordinated WebRTC → WebSocket fallback**: guarded re-entry, retry limit, and clean audio/TTS state reset so a failed peer connection recovers without dropping the conversation. +- **Coordinated WebRTC → WebSocket fallback** with clean audio/TTS state reset. - `POST /v1/webrtc/offer` answers browser offers; `POST /v1/realtime/session` for edge secrets. - **Moshi** native duplex: `--provider moshi --moshi-url ws://127.0.0.1:8998/api/chat`. -- **Built-in LLM provider catalog**: NVIDIA NIM, Groq, OpenRouter, Together, DeepSeek, Fireworks, Mistral, Ollama, OpenAI, Cerebras, SambaNova, and Custom — selectable in setup/settings even before the gateway is ready. +- Built-in LLM provider catalog (NVIDIA NIM, Groq, OpenRouter, Ollama, …) in Settings. - Semantic endpointing (transcript-aware early end ~200 ms). -- Visual cards + live translation demo (mock) / language-mode instructions. - Task lifecycle, evidence links, resume with dedup. -- Configurable `--task-deadline-ms`. -- Developer API: `GET /health`, `/v1/meta`, `/v1/sessions`, `/v1/agent/*`, `/v1/sandbox/*`, `/v1/profile`, MCP tools (+ optional API key). -- Session persistence (JSONL under `data/openlive-sessions`), streaming safety holdback, MCP HTTP client. +- Developer API: `GET /health`, `/v1/meta`, `/v1/sessions`, `/v1/agent/*`, `/v1/sandbox/*`, `/v1/profile`, MCP tools. -**Desktop & fullscreen** +**Desktop** -- **Tauri-based desktop shells** for Windows (MSI) and macOS (DMG/App) in `apps/openlive-desktop/`. -- **Full-screen voice mode** with hidden chrome, hover-to-reveal controls, and a dedicated exit button / `F` shortcut. -- Keyboard shortcuts for layout, fullscreen, camera, screen share, voice picker, and more. +- **Tauri** shells for Windows (MSI) and macOS (DMG/App) in `apps/openlive-desktop/`. +- Local listening-orb splash first; gateway spawn must not block first paint. ### Still missing (vs full GPT-Live) @@ -74,7 +70,7 @@ Full parity matrix: [`docs/gpt-live-parity.md`](docs/gpt-live-parity.md) · Arch ## Requirements -- Rust 1.83 or newer. +- Rust 1.83 or newer (CI / lockfile may require newer for builds; prefer a recent stable). - A modern Chromium, Firefox, or Safari browser. - Microphone permission. @@ -84,7 +80,7 @@ Full parity matrix: [`docs/gpt-live-parity.md`](docs/gpt-live-parity.md) · Arch cargo run -p openlive-gateway --release ``` -Open `http://127.0.0.1:8787` and select **Start** (or press `Space` in push-to-talk mode). The mock speaks with a lightweight formant voice so you can exercise barge-in, transcript, and tasks without external services. +Open `http://127.0.0.1:8787`. You should land on the listening call surface immediately. Talk (or tap the orb / Mute if the browser gated the mic). Prefer installing Piper for real voice; without it the demo path stays silent rather than faking a formant pad. ## Run open-source neural voice (recommended) @@ -94,11 +90,8 @@ Use any OpenAI-compatible stack that exposes: - `POST /v1/chat/completions` - `POST /v1/audio/speech` with `response_format: "pcm"` (24 kHz mono PCM16 preferred) -Example with Piper-style voice ids: - ```bash # API keys: set in the environment only — never commit keys into this repo. -# omit or leave empty for local unauthenticated servers export OPENLIVE_MODEL_API_KEY cargo run -p openlive-gateway --release -- \ @@ -115,7 +108,6 @@ See [`docs/open-source-stack.md`](docs/open-source-stack.md) for LocalAI / opene ## Run a native realtime speech endpoint ```bash -# Read the key from your shell environment (do not put keys in project files) export OPENLIVE_MODEL_API_KEY cargo run -p openlive-gateway --release -- \ @@ -127,20 +119,13 @@ cargo run -p openlive-gateway --release -- \ ## Desktop app (Windows / macOS) -OpenLive ships a Tauri v2 desktop shell in `apps/openlive-desktop`. The shell -loads the same web UI as the browser and can spawn the gateway as a child -process. - ```bash -# Build the gateway first, then the desktop app cargo build -p openlive-gateway --release cd apps/openlive-desktop cargo tauri build ``` -Supported bundles: Windows MSI, macOS DMG/App. See -[`apps/openlive-desktop/README.md`](apps/openlive-desktop/README.md) for dev -mode and platform prerequisites. +See [`apps/openlive-desktop/README.md`](apps/openlive-desktop/README.md). ## Deterministic replay @@ -152,16 +137,12 @@ cargo run -p openlive-runtime --bin openlive-replay -- \ ## Persistence, safety & MCP ```bash -# Default: write session events/tasks under data/openlive-sessions cargo run -p openlive-gateway --release -# Disable durability or safety: cargo run -p openlive-gateway --release -- --no-persist --safety false -# Attach a remote MCP tool host: cargo run -p openlive-gateway --release -- --mcp-url http://127.0.0.1:3100/mcp -# Deep model + local knowledge notes for complex turns: cargo run -p openlive-gateway --release -- \ --provider openai-compatible \ --model-base-url http://127.0.0.1:8000/v1 \ @@ -169,12 +150,10 @@ cargo run -p openlive-gateway --release -- \ --deep-llm-model qwen2.5-32b \ --knowledge-dir ./knowledge -# Hybrid: fast local duplex + deep cascade for hard turns cargo run -p openlive-gateway --release -- \ --provider hybrid \ --model-base-url http://127.0.0.1:8000/v1 -# Local Chronos full-duplex latency gate cargo run -p openlive-runtime --release --bin openlive-full-duplex-bench -- --turns 50 ``` @@ -182,7 +161,6 @@ cargo run -p openlive-runtime --release --bin openlive-full-duplex-bench -- --tu ```bash cargo test --workspace --release -# Integration tests need a debug binary: cargo build -p openlive-gateway && cargo test -p openlive-gateway --test task_lifecycle node --test apps/openlive-gateway/web/tests/*.test.js ``` diff --git a/THIRD_PARTY_NOTICES.md b/THIRD_PARTY_NOTICES.md index 548fd71..53d175d 100644 --- a/THIRD_PARTY_NOTICES.md +++ b/THIRD_PARTY_NOTICES.md @@ -84,9 +84,8 @@ Morphicons are **not** used as the speaking face. ## Fonts (web UI) -| Family | Source | License | -|--------|--------|---------| -| **DM Sans**, **Manrope**, **Space Mono**, **Inter** (if loaded) | [Google Fonts](https://fonts.google.com/) | OFL / respective font licenses | +The default call surface uses **system UI fonts** (`system-ui`, `-apple-system`, +`Segoe UI`, etc.). No Google Fonts CDN is loaded on the demo path. --- diff --git a/apps/openlive-desktop/README.md b/apps/openlive-desktop/README.md index baae174..7a91439 100644 --- a/apps/openlive-desktop/README.md +++ b/apps/openlive-desktop/README.md @@ -49,12 +49,11 @@ cargo tauri build ## Notes -- The desktop shell loads a local listening-orb splash immediately, then - navigates to the gateway UI once `http://127.0.0.1:12345/health` is up. - Spawning the gateway must not block first paint. +- First paint is a local **listening-orb** splash (`splash/`), then the shell + navigates to the gateway call surface once `http://127.0.0.1:12345/health` + is up. Spawning the gateway must not block that first paint. - `OPENLIVE_SKIP_GATEWAY_BUILD=1` skips copying the real gateway binary (used by clippy). A placeholder file is written so Tauri's resource check still passes. -- Replace `icons/icon.ico` and `icons/icon.icns` with branded assets before - publishing. Regenerate placeholders with - `python3 scripts/generate-icons.py`. +- Placeholder icons live under `icons/`. Regenerate with + `python3 scripts/generate-icons.py` before publishing branded assets. diff --git a/docs/architecture-roadmap.md b/docs/architecture-roadmap.md index cf246e6..4da7145 100644 --- a/docs/architecture-roadmap.md +++ b/docs/architecture-roadmap.md @@ -7,7 +7,7 @@ This document captures the target architecture beyond the current voice+tools MV - Deterministic tools: `web_search`, `deep_search`, `research_pool`, `calculator`, `get_time`, `identity`, sandbox `list/read/write_file` - Typo correction for ASR/search -- Open-source **Piper TTS** status + install command + speak endpoint (formant fallback) +- Open-source **Piper TTS** status + install command + speak endpoint (demo path: neural or silence) - Session **memory** JSON store + export API + inject into LLM context - Thought depth setting (voice / balanced / deep) — drives reply length + deep research pool - Agent routing that does **not** force-search every sentence @@ -18,7 +18,8 @@ This document captures the target architecture beyond the current voice+tools MV - **Coordinated WebRTC → WebSocket fallback** with re-entry guard and retry cap - **Desktop shell** (Tauri v2) for Windows MSI and macOS DMG/App - **Full-screen voice mode** with hover-to-reveal chrome and `F` shortcut -- **Boot splash + ripple feedback + enhanced motion** across the voice surface +- **Call-first Bloub surface** + Morphicons controls + Advanced lab disclosure +- Ripple feedback and sheet motion without a forced branded splash ## Next diff --git a/docs/architecture.md b/docs/architecture.md index e8faf4b..36cd332 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -3,7 +3,8 @@ Living companion: [`architecture-roadmap.md`](architecture-roadmap.md) · Release: [`release-26.7.16.md`](release-26.7.16.md) v26.7.16 adds a Tauri v2 desktop shell (`apps/openlive-desktop`), a built-in -LLM provider catalog, full-screen voice mode, a boot splash, and a coordinated +LLM provider catalog, full-screen voice mode, a **call-first** Bloub surface +(Mute / End / Settings), neural-or-silence demo TTS, and a coordinated WebRTC → WebSocket fallback path. The sections below describe the runtime architecture that underlies those features. diff --git a/docs/evaluation.md b/docs/evaluation.md index 0be597e..1adecd7 100644 --- a/docs/evaluation.md +++ b/docs/evaluation.md @@ -1,6 +1,8 @@ # Evaluation and latency telemetry (v26.7.16) -Openlive does not claim experiential parity without measurements. Protocol 1.0 emits generation-scoped `latency_mark` events from the gateway's monotonic clock: +OpenLive does not claim experiential parity with GPT-Live without measurements. +Call-first UX (listening orb, barge-in, neural-or-silence TTS) still needs the +instrumented latency marks below for honest comparison. Protocol 1.0 emits generation-scoped `latency_mark` events from the gateway's monotonic clock: - `response_committed`; - `first_provider_event`; diff --git a/docs/gpt-live-parity.md b/docs/gpt-live-parity.md index 4c3093b..4dbe1a4 100644 --- a/docs/gpt-live-parity.md +++ b/docs/gpt-live-parity.md @@ -57,8 +57,8 @@ Legend: ✅ CLONE · 🟡 DIFFERENT · ❌ GAP | # | Feature | GPT-Live / AVM behavior | Openlive 26.7.16 status | Category | |---|---------|-------------------------|------------------------|----------| -| 1 | Signature voice orb | Blue animated orb, state-driven | Bloub face (x.ai-inspired, MIT, not affiliated) on a black call stage | ✅ CLONE (original visual) | -| 2 | State-driven orb color | Blue / cyan / violet / red shifts | 11 named modes, each with its own palette | ✅ CLONE | +| 1 | Signature voice orb | Blue animated orb, state-driven | Bloub face (x.ai-inspired, MIT, not affiliated) on a black call stage; auto-listening first paint | ✅ CLONE (original visual) | +| 2 | State-driven orb color | Blue / cyan / violet / red shifts | Bloub listen/speak/barge-in states; optional presence palettes behind Settings | ✅ CLONE | | 3 | Live dual transcript | Inline user + assistant bubbles | Persistent scrolling transcript with role-differentiated bubbles + system channel | ✅ CLONE | | 4 | Inline-in-chat voice mode | Late-2025 redesign moved voice inline with chat | Layout toggle: focused vs inline | ✅ CLONE | | 5 | Voice picker | Named voices + personality descriptors | Piper-first roster + provider manifest + offline fallback | ✅ CLONE | @@ -71,7 +71,7 @@ Legend: ✅ CLONE · 🟡 DIFFERENT · ❌ GAP | 12 | Backchanneling ("mhmm") | Native GPT-Live behavior | UI affordance + event handler; badge near orb | ✅ CLONE (UI; provider must emit) | | 13 | Camera input | Camera button streams frames | UI + `C` shortcut; truthful media lifecycle; visual-input negotiation | ✅ CLONE | | 14 | Screen sharing | Available alongside camera | UI + `Shift+C`; truthful media lifecycle | ✅ CLONE | -| 15 | Mute / end-call / camera controls | Persistent in-call controls | Mute, End, Camera, Screen, Voice, Mode, Instructions — richer than AVM | ✅ CLONE (extended) | +| 15 | Mute / end-call / camera controls | Persistent in-call controls | Default dock: Mute / End / Settings (Morphicons). Camera / screen / modes in Settings → Advanced | ✅ CLONE | | 16 | Daily/session quota indicator | Plus: 1 hr/day; free: 15 min preview | Operator-configured cap; soft warning at 80% | ✅ CLONE | | 17 | Latency display | Not surfaced in AVM UI | Latency pill + diagnostics p50/p95/jitter/loss | 🟡 DIFFERENT (Openlive exposes it) | | 18 | Live translation | Built-in GPT-Live feature | VisualCard + language chip instructions; cascade hop for production | 🟡 DIFFERENT (partial) | @@ -94,7 +94,7 @@ Legend: ✅ CLONE · 🟡 DIFFERENT · ❌ GAP | 35 | Diagnostics on-demand | Hidden in AVM | Diagnostics drawer + LiveBench | 🟡 DIFFERENT | | 36 | Model neutrality | AVM locked to OpenAI models | Mock / cascade / realtime / moshi / hybrid | 🟡 DIFFERENT (Openlive beats AVM) | | 37 | Theme customization | AVM orb is fixed | Live Presence / Graphite / Signal + motion intensity | 🟡 DIFFERENT | -| 38 | Onboarding overlay | (Not in AVM) | First-run onboarding + setup wizard | 🟡 DIFFERENT (Openlive original) | +| 38 | Onboarding overlay | (Not in AVM) | Optional setup wizard / shortcuts sheet from Settings — not forced on launch | 🟡 DIFFERENT (Openlive original) | | 39 | Settings persistence | Account-scoped prefs | `localStorage` + durable server profile | ✅ CLONE (extended) | | 40 | Keyboard shortcuts | (Not in AVM) | 10+ shortcuts | 🟡 DIFFERENT (Openlive beats AVM) | | 41 | Task acknowledgement lifecycle | (Not in AVM) | Full lifecycle; p50 = 2 ms | 🟡 DIFFERENT (Openlive original) | @@ -131,9 +131,9 @@ enforce p50 ≤ 50 ms and p95 ≤ 200 ms. ## What "open-source clone" means here Openlive 26.7.16 is a **behavioral clone** of gpt-live's voice surface, -not a **visual clone**. The orb, palettes, copy, layout, and animation -are original Openlive geometry — they do not reproduce any proprietary -interface or its assets. +not a **visual clone**. The default face uses the MIT **Bloub** engine +(x.ai-inspired; not affiliated with xAI or OpenAI). Layout, chrome, and +copy are OpenLive originals — they do not reproduce proprietary assets. The clone contract is: a user who is familiar with AVM should be able to use Openlive without relearning anything, and an operator comparing diff --git a/docs/open-source-stack.md b/docs/open-source-stack.md index a4f668b..15ec062 100644 --- a/docs/open-source-stack.md +++ b/docs/open-source-stack.md @@ -51,9 +51,10 @@ Pick voices that your TTS server actually installs; OpenLive forwards the cargo run -p openlive-gateway --release ``` -The mock provider uses an **original formant synthesizer** so the desk is -demoable without GPU or network. It is **not** production TTS — switch to -Piper for a real AI voice. +The mock provider can synthesize formant audio for low-level transport tests, +but the **default web demo path never plays it**. Auto TTS is Piper or silence +until neural voice is ready. Switch to Piper (or another `/v1/audio/speech` +server) for a real AI voice. ## Browser audio intelligence @@ -142,7 +143,8 @@ dev mode and platform prerequisites. ## UI -Default theme is **Live Presence** (`chatgpt` token) — an original OpenLive -visual inspired by modern voice-mode UX, not a proprietary asset clone. -v26.7.16 adds a boot splash, full-screen mode, ripple click feedback, and a -full set of entrance/transition animations. +Default theme is **minimal black** — a call-first surface with the **Bloub** +face and Morphicons Mute / End / Settings. Lab chrome lives in Settings → +Advanced. Full-screen mode, ripple feedback, and sheet motion remain available. +OpenLive is not affiliated with OpenAI or xAI; Bloub’s design imitates the +x.ai avatar under MIT terms (see `THIRD_PARTY_NOTICES.md`). diff --git a/docs/production-readiness.md b/docs/production-readiness.md index 6800ca0..d2a12a5 100644 --- a/docs/production-readiness.md +++ b/docs/production-readiness.md @@ -26,7 +26,7 @@ Carries forward all verified items from 26.7.14.1, plus the 26.7.16 agent/voice - Client audio intelligence: RNNoise-style worklet, Silero-style VAD, NLMS AEC, windowed-sinc resample. - Semantic endpointing hybrid (~200 ms early end) + ASR revision path. - Gateway-native WebRTC data-channel path + provider-edge session + jitter/PLC on PCM. -- Piper TTS status/speak endpoints + formant fallback; open-stack docs. +- Piper TTS status/speak endpoints; demo path is neural-or-silence (formant only when explicitly selected); open-stack docs. ### Agent, tools, sandbox, profile (26.7.16) @@ -44,12 +44,12 @@ Carries forward all verified items from 26.7.14.1, plus the 26.7.16 agent/voice `apps/openlive-desktop/`. - Full-screen voice mode with hidden chrome, hover-to-reveal controls, and dedicated exit button / `F` shortcut. -- Boot splash with live status text and 3 s failsafe dismissal. -- Ripple click feedback, button hover lift, sheet spring entrance, orb glow - pulse, toast/backchannel scale, transcript revision flash, and ambient - background breathing. -- Built-in LLM provider catalog (12 providers) available before gateway - connection. +- Call-first surface: Bloub face, Mute / End / Settings, auto-join, Advanced + lab disclosure in Settings. +- Ripple click feedback, sheet spring entrance, Bloub state mapping, toast / + backchannel motion, transcript revision flash. +- Built-in LLM provider catalog available in Settings before gateway connection. +- Desktop shells (Windows MSI / macOS DMG) with listening-orb first paint. These are validated with the workspace Rust test suite, JS protocol/task tests, Clippy/fmt gates as run by operators, and an optimized release build. Expand the exact pass counts in CI when CI is wired. @@ -94,4 +94,4 @@ These are validated with the workspace Rust test suite, JS protocol/task tests, 5. GPU worker scheduler and durable authenticated control plane. 6. Benchmark qualification, security review, soak/canary, then production claim. -A compatible external native endpoint is still required until an actual open native speech model and worker are included. The mock tone/formant and conventional cascade must never be presented as native-model parity. +A compatible external native endpoint is still required until an actual open native speech model and worker are included. Formant/mock audio and a conventional cascade without Piper must never be presented as GPT-Live / native-model parity. diff --git a/docs/release-26.7.15.md b/docs/release-26.7.15.md index 938d09e..df1b06c 100644 --- a/docs/release-26.7.15.md +++ b/docs/release-26.7.15.md @@ -5,6 +5,10 @@ **Cargo / package version:** `26.7.15` **UI display:** `v26.7.15` / `26.7.15` +> Historical release note. **26.7.16** changed the default path to a call-first +> surface with **neural-or-silence** demo TTS (no formant pad). See +> [`release-26.7.16.md`](release-26.7.16.md). + ## Goal Ship a **GPT-Live-comparable** operator experience: polished live voice UI, diff --git a/docs/release-26.7.16.md b/docs/release-26.7.16.md index b5883d7..3563e2a 100644 --- a/docs/release-26.7.16.md +++ b/docs/release-26.7.16.md @@ -1,97 +1,74 @@ # OpenLive 26.7.16 -**Codename:** Live Presence + open voice stack + agent workspace +**Codename:** Live Presence + call-first surface + open voice stack + agent workspace **Previous:** 26.7.15 **Cargo / package version:** `26.7.16` **UI display:** `v26.7.16` / `26.7.16` ## Goal -Polish the GPT-Live-comparable operator experience with bug fixes, a -full version bump, and richer UI animations. Carries forward all 26.7.15 -voice, agent, sandbox, and memory features. +Ship a GPT-Live-comparable **call** as the default experience: listening immediately, real neural voice (or silence), instant interrupt, minimal chrome. Architecture may stay rich; the default surface must feel light. ## Highlights +### Call-first UI + +- First paint is the live call: **Bloub** face + **Mute / End / Settings** (Morphicons + Lucide data). +- Session **auto-joins**. Gesture-gated mic failures stay idle (**Tap to talk**) instead of dumping a lab error. +- No branded boot splash delay; the call surface is ready immediately. +- Sandbox, multi-agent, camera, screen share, presence playground, and diagnostics live in **Settings → Advanced** (collapsed by default). +- Optional setup wizard remains under Settings; it is not forced on launch. +- Desktop shell shows a local listening-orb splash, then navigates to the gateway once healthy — gateway spawn must not block first paint. + +### Demo TTS policy: neural or silence + +- Auto engine uses **Piper only**. If Piper is not ready, the path stays silent (no formant pad). +- Formant / browser TTS only when the user explicitly selects those engines. +- Avoids the audible fake→real switch that makes demos feel cheap. + +### Interruption + +- Local-first barge-in chain: `local duck → soft_duck → hard_yield → cancel generation`. +- VAD duck happens before waiting on full server RTT. + ### Desktop applications -- New Tauri v2 shell under `apps/openlive-desktop/`. -- Builds native Windows (MSI) and macOS (DMG/App) bundles. -- Desktop shell loads the same web UI and can spawn the gateway as a child - process on startup. +- Tauri v2 shell under `apps/openlive-desktop/`. +- Native Windows (MSI) and macOS (DMG/App) bundles. +- Clippy/check can set `OPENLIVE_SKIP_GATEWAY_BUILD=1`; a placeholder resource keeps `tauri_build` happy. ### Full-screen voice mode -- Settings toggle and `F` keyboard shortcut enter immersive full-screen mode. -- Browser chrome hides; controls reveal on hover/tap. -- Dedicated exit-fullscreen button for mouse and touch. +- Settings toggle and `F` enter immersive full-screen mode. +- Dedicated exit-fullscreen control. ### Built-in LLM provider catalog -- 12 providers available in setup/settings even when the gateway is offline: +- Providers available in Settings even when the gateway is offline: NVIDIA NIM, Groq, OpenRouter, Together, DeepSeek, Fireworks, Mistral, Ollama, OpenAI, Cerebras, SambaNova, and Custom. -- Mirrors the server-side catalog so base URL, default model, and description - are preset before the first gateway connection. ### Coordinated WebRTC → WebSocket fallback -- New `fallbackToWebSocket` path with `fallbackInProgress` guard against - re-entry. -- WebRTC reconnect attempts are capped; after exhaustion the session falls - back to WebSocket PCM permanently for the remainder of the conversation. -- Clean teardown resets audio/TTS state so playback continues on the new - transport. - -### TTS fallback chain - -- `speakAssistant` tries gateway TTS (Piper/formant) first, then browser - TTS, then degrades gracefully to text-only. -- Prevents silent hangs when the gateway TTS endpoint is unavailable. - -### Boot splash & UI animations - -- Animated boot splash with live status text; dismissed after provider catalog - loads or a 3 s failsafe. -- Page-load entrance animation for the voice stage. -- Orb ambient glow pulse keyed to input/output energy. -- Spring-curve sheet/drawer open with backdrop fade. -- Button hover lift + shadow transitions. -- Toast and backchannel badges animate in with scale + fade. -- Transcript bubble revision flash and loading skeleton shimmer. -- Ripple click feedback on all interactive elements. -- Loading state helper (`withLoading`) for async buttons. - -### Bug fixes - -- **WebRTC cleanup:** `closeWebRtcConnection` now clears the shared media data - channel reference so reconnects start from a clean state. -- **Transcript stream consistency:** `output_text_delta` now uses the entry - returned by `beginAssistantStream` instead of assuming `transcript.last()` - matches, avoiding mismatched delta ids after cancellations or revisions. -- **Runtime status retry leak:** the Settings → Runtime retry button no longer - accumulates duplicate `click` listeners when the gateway is unreachable. -- **Settings scroll-to-top:** opening Settings now scrolls the settings body to - the top so the first section is always visible. -- **Event id fallback:** `sendControl` falls back to a v4-style UUID when - `crypto.randomUUID` is unavailable (older browsers / insecure contexts). -- **Dead code removal:** removed unused `previousOnFrame` / `originalStart` - placeholders and the unused `voice` parameter in WebRTC setup paths. - -### Code quality - -- Fixed **~200 Clippy warnings** workspace-wide: - - `crates/openlive-provider`: doc-markdown backticks, missing `#[must_use]`, - missing `# Errors` docs, `assigning_clones`, case-sensitive extension - checks, MSRV/cast lints, and more. - - `apps/openlive-gateway`: `type_complexity`, `too_many_arguments`, - `too_many_lines`, `large_enum_variant`, `result_large_err`, and cast - lints (via targeted `#[allow(...)]` for structural issues plus a few - mechanical fixes). -- Fixed a `#[must_use]` warning in `crates/openlive-provider/src/session_context.rs`. -- Ran `cargo fmt` across the workspace. -- CI now runs `cargo clippy --workspace --all-targets --all-features -- -D warnings` - and builds the Tauri desktop app on macOS and Windows. +- Guarded re-entry, retry cap, clean audio/TTS reset when falling back to WebSocket PCM. + +### UI motion (without a forced splash) + +- Orb / Bloub state mapping for listen / speak / barge-in / mute. +- Sheet spring entrance, toast/backchannel motion, transcript revision flash. +- Ripple click feedback on interactive controls. +- System UI fonts (no Google Fonts CDN on the default path). + +### Agent, tools, sandbox, profile + +- Internal agent tools: search, deep research pool, calculator, time, sandbox I/O, browse/shot/PDF, profile memory. +- Multi-agent pool (≤50), SSE progress, agent classes, destructive-action confirms. +- Durable profile facts + session memory export. + +### Code quality & CI + +- Workspace Clippy with `-D warnings`; desktop builds on macOS and Windows. +- Tauri CLI install uses `cargo install … --force` so cached runners stay healthy. ### Version surface @@ -100,35 +77,23 @@ All version strings aligned to **26.7.16**: | Surface | Value | |---------|--------| | `Cargo.toml` workspace | `26.7.16` | -| `env!("CARGO_PKG_VERSION")` in `/health`, `/v1/meta` | `26.7.16` | -| Brand badge / onboarding / LiveBench | `26.7.16` | -| Web module file headers | `Openlive 26.7.16` / `OpenLive 26.7.16` | +| `/health`, `/v1/meta` | `26.7.16` | +| UI / LiveBench | `26.7.16` | | LLM User-Agent | `OpenLive/26.7.16` | -| Docs (living) | `v26.7.16` / `26.7.16` | - -### UI animations - -- New page-load entrance animation for the voice stage. -- Enhanced orb ambient glow pulse keyed to input/output energy. -- Sheet/drawer open now uses a spring-curve transform with backdrop fade. -- Button hover states gain subtle lift + shadow transitions. -- Toast and backchannel badges animate in with scale + fade. -- Loading skeleton shimmer for runtime status panels. -- Transcript bubbles animate on entry and revision flash. +| Living docs | `v26.7.16` / `26.7.16` | ## Verify ```bash cargo test --workspace --release node --test apps/openlive-gateway/web/tests/*.test.js -# UI: open http://127.0.0.1:8787 — brand badge reads 26.7.16 +# UI: open http://127.0.0.1:8787 — call surface listens immediately ``` ## Still not full GPT-Live parity -- RTP Opus media tracks on gateway WebRTC (data-channel PCM is production path). +- RTP Opus media tracks on gateway WebRTC (data-channel PCM is the production path). - Official RNNoise WASM / Silero ONNX weights (optional vendor path documented). - Transcript editing; production live-translation LLM hop; SIP/telephony. -See `implementation_plan.md`, `docs/gpt-live-parity.md`, and -`docs/architecture-roadmap.md`. +See `docs/gpt-live-parity.md`, `docs/architecture-roadmap.md`, and `implementation_plan.md`. diff --git a/implementation_plan.md b/implementation_plan.md index 78c93cb..7e1df00 100644 --- a/implementation_plan.md +++ b/implementation_plan.md @@ -1,7 +1,7 @@ # OpenLive → GPT-Live Parity — Implementation Plan -**Status date:** 2026-07-15 -**Baseline:** **26.7.16** (Live Presence + open voice stack) +**Status date:** 2026-08-28 +**Baseline:** **26.7.16** (call-first surface + open voice stack + agent workspace) **Goal:** Model-neutral, open, production-grade competitor to OpenAI GPT-Live / Advanced Voice Mode. --- @@ -11,14 +11,15 @@ ``` ┌─────────────────────────────────────────────────────────────────────────────┐ │ BROWSER CLIENT │ -│ RNNoise WASM → Silero VAD → Emotion Detect → WebGL Orb │ -│ Enhanced AudioSession (FIR resample, NLMS AEC, client endpointing) │ -│ WebRTC (Opus) ← fallback WebSocket Binary PCM │ +│ Call-first UI: Bloub face · Mute/End/Settings · Advanced lab disclosure │ +│ RNNoise worklet → Silero-style VAD → NLMS AEC → FIR resample │ +│ WebRTC (data-channel PCM) ← fallback WebSocket Binary PCM │ └────────────────────────────────┬────────────────────────────────────────────┘ │ ┌────────────────────────────────▼────────────────────────────────────────────┐ │ GATEWAY: WebRTC signaling · Semantic endpointing · MCP · Safety · Chronos │ │ Providers: Mock · Cascade · Realtime · Moshi · HybridStreaming │ +│ TTS demo policy: Piper or silence (formant only if explicitly selected) │ └─────────────────────────────────────────────────────────────────────────────┘ ``` @@ -35,9 +36,9 @@ | NLMS adaptive AEC | **Done** | `NlmsAec` in `audio-utils.js` | | Polyphase / windowed-sinc FIR resampler | **Done** | `audio-utils.js` | | Chain in AudioSession | **Done** | mic → RNNoise → Silero → capture → NLMS → FIR → PCM16 | -| Open AI voice (Piper) | **Done** | Piper-first roster + cascade docs; formant mock | -| UI Live Presence | **Done** | Default theme + credits panel | -| THIRD_PARTY_NOTICES | **Done** | Root notices file | +| Open AI voice (Piper) | **Done** | Piper-first roster; demo path neural-or-silence | +| Call-first UI | **Done** | Bloub + Morphicons; Advanced lab disclosure | +| THIRD_PARTY_NOTICES | **Done** | Root notices file (Bloub / Morphicons / Piper) | ### Phase 2: WebRTC / Opus Transport — **PARTIAL** @@ -97,13 +98,14 @@ | Item | Status | Notes | |------|--------|-------| -| `chatgpt` theme + CSS variables | **Done** | `styles.css`, `settings-store.js` | -| Wave-Particle orb (Canvas 2D) | **Done** | `voice-visualizer.js` Bezier blobs + particle ring | -| Boot splash + live status | **Done** | `app.js` + `index.html` + `styles.css` | +| Call-first minimal black surface | **Done** | Auto-join; Mute / End / Settings dock | +| Bloub face engine | **Done** | Vendored MIT `vendor/bloub/`; not affiliated with xAI | +| Morphicons call controls | **Done** | Mic ↔ MicOff, Settings ↔ X, PhoneOff end | +| Advanced lab disclosure in Settings | **Done** | Camera, sandbox, agents, diagnostics | | Full-screen voice mode | **Done** | Settings toggle, `F` shortcut, exit button | -| Ripple click feedback + loading states | **Done** | `app.js` + `styles.css` | -| Page-load / sheet / button / toast animations | **Done** | `styles.css` v26.7.16 animation section | -| Three.js WebGL Icosphere + GLSL | **Todo** | | +| Ripple click feedback + sheet motion | **Done** | `app.js` + `styles.css` | +| Legacy Canvas visualizer | **Kept** | Hidden behind Bloub on the default path | +| Three.js WebGL Icosphere + GLSL | **Dropped** | Bloub covers the face; not required | ### Phase 8: Emotion-Aware Responses — **PARTIAL** @@ -147,7 +149,8 @@ - WebRTC signaling + gateway-native data-channel path + browser peer - Semantic VAD hybrid + cascade ASR prior window + revise UI -- Live Presence / minimal-black UI + Piper TTS + formant fallback +- Call-first Bloub surface + Morphicons Mute/End/Settings + Advanced lab disclosure +- Piper TTS with neural-or-silence demo policy (formant only if explicitly selected) - Agent tools, multi-agent pool, sandbox, profile/memory, confirms - Version surface: Cargo / UI / living docs all **26.7.16** @@ -176,7 +179,7 @@ node --test apps/openlive-gateway/web/tests/*.test.js 4. **Tool authz audit / soak** — production-depth gates on agent sandbox 5. **Benchmark qualification** — Full-Duplex-Bench / VoiceBench manifests 6. **Phase 4 Moshi** — true native duplex -7. **Phases 7–10** — WebGL orb, emotion, benches, developer API +7. **Phases 7–10** — Emotion cues, benches, developer API polish (orb face ships via Bloub) ---