Free, Open-Source AI Video Dubbing & Voice Translation Platform — Powered by Google Gemini, Microsoft Edge-TTS, and Acoustic Vocal Isolation. Built for Idealabs Digital.
⚠️ Project Status Note: Edge-TTS is 100% fully functional, battle-tested, and recommended for production use. Secondary engines (F5-TTS and custom voice cloning models) are currently under active experimental development as the project is continuously updating.
- 100% Start-to-End Vocal Coverage: Ensures 100% of synthesized TTS speech plays from natural start to end without cutoffs, truncations, or premature shortening.
- 100% Background Music & FX Preservation: Uses Side-Channel Out-of-Phase Stereo (OOPS) phase subtraction and multi-stage bandstop filtering to cancel original vocals while keeping 100% of background music, instruments, guitars, score, and ambient sound effects completely untouched.
- Frame-Accurate 35s Chunking: Eliminates multi-second timestamp drift by processing audio in 35-second physical chunks with 2.0s overlap.
- Acoustic Trailing End-Time Extension: Uses spectral energy envelope analysis to extend segment
end_timeto natural speech release points. - Hybrid Acoustic Pitch & Timbre Diarization: Combines F0 pitch autocorrelation (Male < 165Hz, Female >= 165Hz) with Gemini dialogue context analysis to cleanly separate 3+ distinct speakers into Male and Female profiles.
- Cinematic Localization Engine: Translates foreign dialogue into idiomatic, natural spoken language suitable for professional dubbing and voice acting.
[YouTube Video URL]
│
▼ (yt-dlp)
┌───────────────┐
│ Original MP4 │
└──────┬────────┘
│
├─────────────────────────┐
▼ (ffmpeg) ▼ (Video Stream Copy)
┌───────────────┐ ┌───────────────┐
│ 16kHz WAV │ │ Video (No Aud)│
└──────┬────────┘ └───────┬───────┘
│ (35s Precision Chunking) │
▼ │
┌───────────────┐ │
│ Audio Chunks │ │
└──────┬────────┘ │
│ │
▼ (Gemini S2T + F0 Pitch) │
┌───────────────┐ │
│ Multi-Speaker │ │
│ Timed JSON │ │
└──────┬────────┘ │
│ │
▼ (Gemini Cinematic Trans) │
┌───────────────┐ │
│ Translated │ │
│ Timed JSON │ │
└──────┬────────┘ │
│ │
▼ (Edge-TTS / ElevenLabs) │
┌───────────────┐ │
│ Neural Vocals │ │
└──────┬────────┘ │
│ │
▼ (Acoustic End Align) │
┌───────────────┐ │
│ Extended & │ │
│ Padded Clips │ │
└──────┬────────┘ │
│ │
▼ (Side-Channel Mix) │
┌───────────────┐ │
│ Dubbed WAV │ │
└──────┬────────┘ │
│ │
└───────────┬──────────────┘
▼ (FFmpeg Lossless Muxing)
┌───────────────┐
│ Dubbed MP4 │
└───────────────┘
- Smart Side-Channel Vocal Isolation: Preserves 100% of stereo background music and environmental sound effects while cancelling center-panned spoken dialogue.
- Hybrid Acoustic Pitch & Timbre Diarization: Autocorrelates fundamental frequency (F0) to cluster 3+ distinct speakers into Male and Female profiles.
- Acoustic Trailing Release Boundary Extension: Dynamically extends segment
end_timeinto natural phrase release envelopes so speech never cuts off early. - Cinematic Dubbing Translation Engine: Translates dialogue into natural, idiomatic speech matching character intent and duration rhythm.
- Automatic Gender & Voice Matching: Maps Male and Female speakers to distinct gendered neural voices (
en-US-BrianNeural,en-US-EmmaMultilingualNeural,en-US-AvaNeural,en-US-AndrewNeural, etc.). - Smart Dynamic Sidechain Ducking: Gently ducks background music (-4dB default) during active speech with smooth crossfades so background music stays rich and audible.
- Speech Rate Auto-Pacing & Condensing: Shortens translations exceeding WPM speed thresholds and applies pitch-preserving time-stretching if required.
- Multi-Format Subtitle Exporter: Generates styled ASS, SRT, and WebVTT caption files with custom font sizes, colors, outlines, and exact frame timestamps.
- Voice Presets Manager & Preset Exporter: Saves, loads, exports, and deletes custom speaker-to-voice presets to JSON/CSV for recurring video projects.
- Automatic API Key Rotation Manager: Rotates through
GEMINI_API_KEY_1..10upon rate limit (429) hits with network latency health benchmarking. - Performance Profiler & HTML Diagnostics Reporter: Generates an HTML report containing timeline allocation, character estimates, and ASCII time charts.
- Zero-Dependency Waveform BMP & Animated Video Overlay: Generates visual waveform images and animated audio visualization video overlays using FFmpeg filters.
GeminiClientManager (config.py) implements automatic API key failover:
- Loads multiple keys from environment:
GEMINI_API_KEY_1,GEMINI_API_KEY_2,GEMINI_API_KEY_3, etc. - Upon encountering rate limit (
429) errors or quota limits, it rotates to the next available API key automatically.
- Python: version
3.9to3.12(tested on3.11.4). - FFmpeg: Must be installed and available in system PATH.
- Clone or extract this repository.
- Install dependencies:
pip install -r requirements.txt
- Create a
.envfile from.env.template:copy .env.template .env
- Fill in your Gemini API keys:
GEMINI_API_KEY_1=AIzaSy... GEMINI_API_KEY_2=AIzaSy... GEMINI_API_KEY_3=AIzaSy...
- Go to Google AI Studio.
- Sign in with your Google account.
- Click "Get API Key" in the left sidebar menu.
- Click "Create API Key in new project".
- Copy your generated API key (starts with
AIzaSy...). - Paste your key into your
.envfile asGEMINI_API_KEY_1. - (Optional for large videos): Create 2-3 free keys across Google projects and add them as
GEMINI_API_KEY_2,GEMINI_API_KEY_3in.envto enable automatic API key rotation!
Launch the FastAPI web interface at http://localhost:8000:
python main.py --webFeatures available in the dashboard:
- Live YouTube video ingestion and progress logging over WebSockets.
- Interactive segment editor table to edit translations before rendering.
- Voice Presets Manager to save, load, and delete speaker voice configurations.
- Subtitle downloader (SRT, WebVTT, ASS) and performance report viewer.
Run the automated unit test suite (46 tests):
python test_suite.pyRun the full integration test:
python run_integration.py