Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 0 additions & 5 deletions .cursor/environment.json

This file was deleted.

9 changes: 0 additions & 9 deletions .cursor/install.sh

This file was deleted.

29 changes: 0 additions & 29 deletions .cursor/start-gateway.sh

This file was deleted.

68 changes: 23 additions & 45 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,68 +3,64 @@
OpenLive is an open, model-neutral runtime for continuous voice agents. It separates deadline-sensitive interaction continuity from slower model cognition and preserves native duplex provider capabilities instead of forcing every model through a text-chat abstraction.

> [!IMPORTANT]
> **Disclaimer:** This is an independent open-source project. It is **not an official OpenAI project** and has no affiliation with OpenAI.
> **Disclaimer:** This is an independent open-source project. It is **not an official OpenAI project** and has no affiliation with OpenAI or xAI.

## Current status

**Version 26.7.16** (`v26.7.16`) targets a **GPT-Live-comparable** experience: polished live voice UI, open neural speech (Piper), client-side audio intelligence, WebRTC session path, semantic endpointing, **real tools + multi-agent sandbox**, and durable profile/memory — with original visuals and model neutrality intact.
**Version 26.7.16** targets a **GPT-Live-comparable** experience: a call-first live surface, open neural speech (Piper), client-side audio intelligence, WebRTC session path, semantic endpointing, **real tools + multi-agent sandbox**, and durable profile/memory — with original visuals and model neutrality intact.

Full parity matrix: [`docs/gpt-live-parity.md`](docs/gpt-live-parity.md) · Architecture roadmap: [`docs/architecture-roadmap.md`](docs/architecture-roadmap.md) · Open stack guide: [`docs/open-source-stack.md`](docs/open-source-stack.md) · Credits: [`THIRD_PARTY_NOTICES.md`](THIRD_PARTY_NOTICES.md) · Release notes: [`docs/release-26.7.16.md`](docs/release-26.7.16.md)

### What 26.7.16 ships

**Voice surface**
**Call-first voice surface**

- Minimal black live surface + setup wizard; Live Presence theme tokens.
- Full-screen voice presence with 11 named modes and multi-layer procedural orb.
- Inline layout toggle, live dual transcript, conversation modes, speaking-style axes.
- Push-to-talk, barge-in with local duck, camera/screen share affordances, visual cards.
- **Piper-first voice roster** (Lessac, Amy, Ryan, …) plus API-compatible fallbacks.
- **Boot splash** with live status and animated page-load entrance.
- **Ripple click feedback** and enhanced hover/lift micro-interactions across the UI.
- Brand chrome and package version: **26.7.16**.
- First paint is a live call: **Bloub** face on a black stage + **Mute / End / Settings** (Morphicons).
- Session **auto-joins**. If the browser needs a gesture for the mic, the surface says **Tap to talk**.
- Lab chrome (camera, screen share, sandbox, multi-agent, diagnostics) lives in **Settings → Advanced**.
- Optional setup wizard remains available from Settings; it is not forced on launch.
- Full-screen mode, live transcript, conversation modes, and speaking-style axes stay available behind Settings / shortcuts.
- **Demo TTS policy: neural or silence.** Auto never pads with formant/mock. Formant only when you explicitly pick it.
- Package version: **26.7.16**.

**Open AI voice**

- Production path: cascade → OpenAI-compatible **Piper** TTS (via LocalAI, openedai-speech, or gateway-local Piper).
- Gateway: `GET /v1/tts/status`, `POST /v1/tts/speak`; formant fallback for demos.
- **Graceful TTS fallback chain**: gateway TTS (Piper/formant) → browser TTS → text-only, so the conversation never hangs.
- Gateway: `GET /v1/tts/status`, `POST /v1/tts/speak`.
- Auto engine = Piper or silence until ready; browser / formant only when selected in Settings.
- Licenses and attribution in `THIRD_PARTY_NOTICES.md`.

**Client audio intelligence**

- RNNoise-style noise suppression worklet (10 ms frames).
- Silero-style VAD worklet + energy blend.
- NLMS adaptive echo cancellation + windowed-sinc resampler.
- Local-first barge-in: `local duck → soft_duck → hard_yield → cancel`.

**Agent, tools & sandbox**

- Internal agent (no OpenCode): search, deep research pool, calculator, time, identity, profile.
- Path-safe sandbox file I/O + optional Chrome/Edge headless browse / screenshot / PDF.
- Multi-agent pool (≤50) with SSE progress, agent classes, and destructive-action confirms.
- Durable user profile (facts editor, drag-and-drop reorder) + session memory export.
- Durable user profile (facts editor) + session memory export.
- See `sandbox/README.md` and `docs/architecture-roadmap.md`.

**Transport, providers & tasks**

- Binary WebSocket PCM + **gateway-native WebRTC** (DTLS data channels for events/PCM).
- Provider-edge WebRTC (OpenAI Realtime SDP) when secrets are available.
- **Coordinated WebRTC → WebSocket fallback**: guarded re-entry, retry limit, and clean audio/TTS state reset so a failed peer connection recovers without dropping the conversation.
- **Coordinated WebRTC → WebSocket fallback** with clean audio/TTS state reset.
- `POST /v1/webrtc/offer` answers browser offers; `POST /v1/realtime/session` for edge secrets.
- **Moshi** native duplex: `--provider moshi --moshi-url ws://127.0.0.1:8998/api/chat`.
- **Built-in LLM provider catalog**: NVIDIA NIM, Groq, OpenRouter, Together, DeepSeek, Fireworks, Mistral, Ollama, OpenAI, Cerebras, SambaNova, and Custom — selectable in setup/settings even before the gateway is ready.
- Built-in LLM provider catalog (NVIDIA NIM, Groq, OpenRouter, Ollama, …) in Settings.
- Semantic endpointing (transcript-aware early end ~200 ms).
- Visual cards + live translation demo (mock) / language-mode instructions.
- Task lifecycle, evidence links, resume with dedup.
- Configurable `--task-deadline-ms`.
- Developer API: `GET /health`, `/v1/meta`, `/v1/sessions`, `/v1/agent/*`, `/v1/sandbox/*`, `/v1/profile`, MCP tools (+ optional API key).
- Session persistence (JSONL under `data/openlive-sessions`), streaming safety holdback, MCP HTTP client.
- Developer API: `GET /health`, `/v1/meta`, `/v1/sessions`, `/v1/agent/*`, `/v1/sandbox/*`, `/v1/profile`, MCP tools.

**Desktop & fullscreen**
**Desktop**

- **Tauri-based desktop shells** for Windows (MSI) and macOS (DMG/App) in `apps/openlive-desktop/`.
- **Full-screen voice mode** with hidden chrome, hover-to-reveal controls, and a dedicated exit button / `F` shortcut.
- Keyboard shortcuts for layout, fullscreen, camera, screen share, voice picker, and more.
- **Tauri** shells for Windows (MSI) and macOS (DMG/App) in `apps/openlive-desktop/`.
- Local listening-orb splash first; gateway spawn must not block first paint.

### Still missing (vs full GPT-Live)

Expand All @@ -74,7 +70,7 @@ Full parity matrix: [`docs/gpt-live-parity.md`](docs/gpt-live-parity.md) · Arch

## Requirements

- Rust 1.83 or newer.
- Rust 1.83 or newer (CI / lockfile may require newer for builds; prefer a recent stable).
- A modern Chromium, Firefox, or Safari browser.
- Microphone permission.

Expand All @@ -84,7 +80,7 @@ Full parity matrix: [`docs/gpt-live-parity.md`](docs/gpt-live-parity.md) · Arch
cargo run -p openlive-gateway --release
```

Open `http://127.0.0.1:8787` and select **Start** (or press `Space` in push-to-talk mode). The mock speaks with a lightweight formant voice so you can exercise barge-in, transcript, and tasks without external services.
Open `http://127.0.0.1:8787`. You should land on the listening call surface immediately. Talk (or tap the orb / Mute if the browser gated the mic). Prefer installing Piper for real voice; without it the demo path stays silent rather than faking a formant pad.

## Run open-source neural voice (recommended)

Expand All @@ -94,11 +90,8 @@ Use any OpenAI-compatible stack that exposes:
- `POST /v1/chat/completions`
- `POST /v1/audio/speech` with `response_format: "pcm"` (24 kHz mono PCM16 preferred)

Example with Piper-style voice ids:

```bash
# API keys: set in the environment only — never commit keys into this repo.
# omit or leave empty for local unauthenticated servers
export OPENLIVE_MODEL_API_KEY

cargo run -p openlive-gateway --release -- \
Expand All @@ -115,7 +108,6 @@ See [`docs/open-source-stack.md`](docs/open-source-stack.md) for LocalAI / opene
## Run a native realtime speech endpoint

```bash
# Read the key from your shell environment (do not put keys in project files)
export OPENLIVE_MODEL_API_KEY

cargo run -p openlive-gateway --release -- \
Expand All @@ -127,20 +119,13 @@ cargo run -p openlive-gateway --release -- \

## Desktop app (Windows / macOS)

OpenLive ships a Tauri v2 desktop shell in `apps/openlive-desktop`. The shell
loads the same web UI as the browser and can spawn the gateway as a child
process.

```bash
# Build the gateway first, then the desktop app
cargo build -p openlive-gateway --release
cd apps/openlive-desktop
cargo tauri build
```

Supported bundles: Windows MSI, macOS DMG/App. See
[`apps/openlive-desktop/README.md`](apps/openlive-desktop/README.md) for dev
mode and platform prerequisites.
See [`apps/openlive-desktop/README.md`](apps/openlive-desktop/README.md).

## Deterministic replay

Expand All @@ -152,37 +137,30 @@ cargo run -p openlive-runtime --bin openlive-replay -- \
## Persistence, safety & MCP

```bash
# Default: write session events/tasks under data/openlive-sessions
cargo run -p openlive-gateway --release

# Disable durability or safety:
cargo run -p openlive-gateway --release -- --no-persist --safety false

# Attach a remote MCP tool host:
cargo run -p openlive-gateway --release -- --mcp-url http://127.0.0.1:3100/mcp

# Deep model + local knowledge notes for complex turns:
cargo run -p openlive-gateway --release -- \
--provider openai-compatible \
--model-base-url http://127.0.0.1:8000/v1 \
--llm-model llama3.2 \
--deep-llm-model qwen2.5-32b \
--knowledge-dir ./knowledge

# Hybrid: fast local duplex + deep cascade for hard turns
cargo run -p openlive-gateway --release -- \
--provider hybrid \
--model-base-url http://127.0.0.1:8000/v1

# Local Chronos full-duplex latency gate
cargo run -p openlive-runtime --release --bin openlive-full-duplex-bench -- --turns 50
```

## Tests

```bash
cargo test --workspace --release
# Integration tests need a debug binary:
cargo build -p openlive-gateway && cargo test -p openlive-gateway --test task_lifecycle
node --test apps/openlive-gateway/web/tests/*.test.js
```
Expand Down
5 changes: 2 additions & 3 deletions THIRD_PARTY_NOTICES.md
Original file line number Diff line number Diff line change
Expand Up @@ -84,9 +84,8 @@ Morphicons are **not** used as the speaking face.

## Fonts (web UI)

| Family | Source | License |
|--------|--------|---------|
| **DM Sans**, **Manrope**, **Space Mono**, **Inter** (if loaded) | [Google Fonts](https://fonts.google.com/) | OFL / respective font licenses |
The default call surface uses **system UI fonts** (`system-ui`, `-apple-system`,
`Segoe UI`, etc.). No Google Fonts CDN is loaded on the demo path.

---

Expand Down
11 changes: 5 additions & 6 deletions apps/openlive-desktop/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,12 +49,11 @@ cargo tauri build

## Notes

- The desktop shell loads a local listening-orb splash immediately, then
navigates to the gateway UI once `http://127.0.0.1:12345/health` is up.
Spawning the gateway must not block first paint.
- First paint is a local **listening-orb** splash (`splash/`), then the shell
navigates to the gateway call surface once `http://127.0.0.1:12345/health`
is up. Spawning the gateway must not block that first paint.
- `OPENLIVE_SKIP_GATEWAY_BUILD=1` skips copying the real gateway binary
(used by clippy). A placeholder file is written so Tauri's resource
check still passes.
- Replace `icons/icon.ico` and `icons/icon.icns` with branded assets before
publishing. Regenerate placeholders with
`python3 scripts/generate-icons.py`.
- Placeholder icons live under `icons/`. Regenerate with
`python3 scripts/generate-icons.py` before publishing branded assets.
5 changes: 3 additions & 2 deletions docs/architecture-roadmap.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ This document captures the target architecture beyond the current voice+tools MV

- Deterministic tools: `web_search`, `deep_search`, `research_pool`, `calculator`, `get_time`, `identity`, sandbox `list/read/write_file`
- Typo correction for ASR/search
- Open-source **Piper TTS** status + install command + speak endpoint (formant fallback)
- Open-source **Piper TTS** status + install command + speak endpoint (demo path: neural or silence)
- Session **memory** JSON store + export API + inject into LLM context
- Thought depth setting (voice / balanced / deep) — drives reply length + deep research pool
- Agent routing that does **not** force-search every sentence
Expand All @@ -18,7 +18,8 @@ This document captures the target architecture beyond the current voice+tools MV
- **Coordinated WebRTC → WebSocket fallback** with re-entry guard and retry cap
- **Desktop shell** (Tauri v2) for Windows MSI and macOS DMG/App
- **Full-screen voice mode** with hover-to-reveal chrome and `F` shortcut
- **Boot splash + ripple feedback + enhanced motion** across the voice surface
- **Call-first Bloub surface** + Morphicons controls + Advanced lab disclosure
- Ripple feedback and sheet motion without a forced branded splash

## Next

Expand Down
3 changes: 2 additions & 1 deletion docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,8 @@
Living companion: [`architecture-roadmap.md`](architecture-roadmap.md) · Release: [`release-26.7.16.md`](release-26.7.16.md)

v26.7.16 adds a Tauri v2 desktop shell (`apps/openlive-desktop`), a built-in
LLM provider catalog, full-screen voice mode, a boot splash, and a coordinated
LLM provider catalog, full-screen voice mode, a **call-first** Bloub surface
(Mute / End / Settings), neural-or-silence demo TTS, and a coordinated
WebRTC → WebSocket fallback path. The sections below describe the runtime
architecture that underlies those features.

Expand Down
4 changes: 3 additions & 1 deletion docs/evaluation.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
# Evaluation and latency telemetry (v26.7.16)

Openlive does not claim experiential parity without measurements. Protocol 1.0 emits generation-scoped `latency_mark` events from the gateway's monotonic clock:
OpenLive does not claim experiential parity with GPT-Live without measurements.
Call-first UX (listening orb, barge-in, neural-or-silence TTS) still needs the
instrumented latency marks below for honest comparison. Protocol 1.0 emits generation-scoped `latency_mark` events from the gateway's monotonic clock:

- `response_committed`;
- `first_provider_event`;
Expand Down
14 changes: 7 additions & 7 deletions docs/gpt-live-parity.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,8 +57,8 @@ Legend: ✅ CLONE · 🟡 DIFFERENT · ❌ GAP

| # | Feature | GPT-Live / AVM behavior | Openlive 26.7.16 status | Category |
|---|---------|-------------------------|------------------------|----------|
| 1 | Signature voice orb | Blue animated orb, state-driven | Bloub face (x.ai-inspired, MIT, not affiliated) on a black call stage | ✅ CLONE (original visual) |
| 2 | State-driven orb color | Blue / cyan / violet / red shifts | 11 named modes, each with its own palette | ✅ CLONE |
| 1 | Signature voice orb | Blue animated orb, state-driven | Bloub face (x.ai-inspired, MIT, not affiliated) on a black call stage; auto-listening first paint | ✅ CLONE (original visual) |
| 2 | State-driven orb color | Blue / cyan / violet / red shifts | Bloub listen/speak/barge-in states; optional presence palettes behind Settings | ✅ CLONE |
| 3 | Live dual transcript | Inline user + assistant bubbles | Persistent scrolling transcript with role-differentiated bubbles + system channel | ✅ CLONE |
| 4 | Inline-in-chat voice mode | Late-2025 redesign moved voice inline with chat | Layout toggle: focused vs inline | ✅ CLONE |
| 5 | Voice picker | Named voices + personality descriptors | Piper-first roster + provider manifest + offline fallback | ✅ CLONE |
Expand All @@ -71,7 +71,7 @@ Legend: ✅ CLONE · 🟡 DIFFERENT · ❌ GAP
| 12 | Backchanneling ("mhmm") | Native GPT-Live behavior | UI affordance + event handler; badge near orb | ✅ CLONE (UI; provider must emit) |
| 13 | Camera input | Camera button streams frames | UI + `C` shortcut; truthful media lifecycle; visual-input negotiation | ✅ CLONE |
| 14 | Screen sharing | Available alongside camera | UI + `Shift+C`; truthful media lifecycle | ✅ CLONE |
| 15 | Mute / end-call / camera controls | Persistent in-call controls | Mute, End, Camera, Screen, Voice, Mode, Instructions — richer than AVM | ✅ CLONE (extended) |
| 15 | Mute / end-call / camera controls | Persistent in-call controls | Default dock: Mute / End / Settings (Morphicons). Camera / screen / modes in Settings → Advanced | ✅ CLONE |
| 16 | Daily/session quota indicator | Plus: 1 hr/day; free: 15 min preview | Operator-configured cap; soft warning at 80% | ✅ CLONE |
| 17 | Latency display | Not surfaced in AVM UI | Latency pill + diagnostics p50/p95/jitter/loss | 🟡 DIFFERENT (Openlive exposes it) |
| 18 | Live translation | Built-in GPT-Live feature | VisualCard + language chip instructions; cascade hop for production | 🟡 DIFFERENT (partial) |
Expand All @@ -94,7 +94,7 @@ Legend: ✅ CLONE · 🟡 DIFFERENT · ❌ GAP
| 35 | Diagnostics on-demand | Hidden in AVM | Diagnostics drawer + LiveBench | 🟡 DIFFERENT |
| 36 | Model neutrality | AVM locked to OpenAI models | Mock / cascade / realtime / moshi / hybrid | 🟡 DIFFERENT (Openlive beats AVM) |
| 37 | Theme customization | AVM orb is fixed | Live Presence / Graphite / Signal + motion intensity | 🟡 DIFFERENT |
| 38 | Onboarding overlay | (Not in AVM) | First-run onboarding + setup wizard | 🟡 DIFFERENT (Openlive original) |
| 38 | Onboarding overlay | (Not in AVM) | Optional setup wizard / shortcuts sheet from Settings — not forced on launch | 🟡 DIFFERENT (Openlive original) |
| 39 | Settings persistence | Account-scoped prefs | `localStorage` + durable server profile | ✅ CLONE (extended) |
| 40 | Keyboard shortcuts | (Not in AVM) | 10+ shortcuts | 🟡 DIFFERENT (Openlive beats AVM) |
| 41 | Task acknowledgement lifecycle | (Not in AVM) | Full lifecycle; p50 = 2 ms | 🟡 DIFFERENT (Openlive original) |
Expand Down Expand Up @@ -131,9 +131,9 @@ enforce p50 ≤ 50 ms and p95 ≤ 200 ms.
## What "open-source clone" means here

Openlive 26.7.16 is a **behavioral clone** of gpt-live's voice surface,
not a **visual clone**. The orb, palettes, copy, layout, and animation
are original Openlive geometry — they do not reproduce any proprietary
interface or its assets.
not a **visual clone**. The default face uses the MIT **Bloub** engine
(x.ai-inspired; not affiliated with xAI or OpenAI). Layout, chrome, and
copy are OpenLive originals — they do not reproduce proprietary assets.

The clone contract is: a user who is familiar with AVM should be able
to use Openlive without relearning anything, and an operator comparing
Expand Down
Loading
Loading