A single-character, real-time lip-sync system built on top of the FlashLips architecture — optimised to run at interactive framerates on medium- to high-grade consumer hardware, not just top-of-the-line server GPUs.
Based on: Zinonos et al., FlashLips: 100-FPS Mask-Free Latent Lip-Sync using Reconstruction Instead of Diffusion or GANs, arXiv:2512.20033v3 (Apr 2026). [paper] [HTML]
The original FlashLips paper trains on 100 000+ identities and targets H100-class GPUs. That generalisation has a real cost: a large dataset, a heavy frozen expression backbone (EmoPortraits) carried through training, and a model that has to be expressive enough to represent every face it might ever see.
I don't need any of that. I'm building a real-time streaming avatar for one fixed character. The character can still move around, change pose, and appear in different scenes — but it's always the same face, drawn from a known training distribution. Sometimes that's also one fixed voice; sometimes it's the same character driven by different speakers. Either way the identity axis collapses to a single point, and that opens up a specific set of optimisations:
- Pre-computed VAE latents — eliminates the largest training-time bottleneck. The frozen SDXL VAE encodes three tensors every batch (target, masked-target, reference). With a fixed corpus of frames those encodings never change, so I run them once during preprocessing and store a memory-mapped fp16 array. The dataset then indexes into it instead of re-encoding. Profiling showed this removed ~32% of batch time. The paper can't do this — it sees a new identity every minibatch.
- MediaPipe blendshapes instead of a frozen neural expression backbone. FlashLips uses EmoPortraits because it has to be identity-invariant across 100k+ faces. For one character I can pre-extract 52 ARKit blendshape coefficients with MediaPipe at data-prep time and store them as a flat NumPy array. They are geometry-only, identity-invariant by construction, and at training time the "encoder" is just an MLP over a 52-D vector — no neural backbone runs on the GPU at all.
- Smaller, character-specialised UNet. The paper's UNet has to model the appearance manifold of every identity in the training set. A single-character model only needs to represent variations of one face, so the same architecture has much more capacity per identity than it needs — which translates directly into either lower latency or smaller models at the same quality.
- Single-character-friendly reference selection. With one identity there's no need to randomly sample references from a video corpus — I can curate a small, diverse pool of reference frames once (eyes-open, eyes-closed, mouth-open, profile, etc.) and reuse it across training and inference.
Training on one identity also creates failure modes the paper doesn't have to worry about. I address them explicitly:
- Aggressive ColorJitter on the z_lips input. With 100k+ identities the same mouth shape appears naturally with many skin tones, so the network can't shortcut z_lips → colour. With one identity it absolutely can. I apply ColorJitter (hue 0.4, saturation 3.2, p=0.9 — the values from the paper's Appendix A.1) only to the frame feeding the lips encoder, while comparing the output to the un-jittered target. Same shape, different colour every time, forcing z_lips to encode geometry only.
- Cross-reconstruction training. Pure self-reconstruction on one identity lets the editor cheat by copying the reference frame's mouth wholesale, ignoring z_lips. I added a cross-recon strategy that extracts z_lips from frame A and paints it onto frame B, with a region-split loss (mouth region must match A, everything else must match B). An alternating strategy interleaves self-recon and cross-recon 1:1 per batch.
- Hardware-tiered training configs. Configs for RTX 4060 Ti, RTX 4090, RTX 5090, A6000, and H100 SXM, with batch size, dataloader workers,
torch.compile, and gradient checkpointing scaled per tier. The same training command works on a 16 GB consumer card and an 80 GB datacentre GPU. - Cloud training via RunPod. The
deployment/module provisions pods, launches training, streams logs, kills the right process tree, and downloads only the relevant checkpoints. Multiple named slots run experiments in parallel.
The system follows the two-stage FlashLips design:
A one-step latent-space editor. Given a masked target frame, a reference frame, and a lip-pose vector z_lips, it reconstructs the full frame in a single forward pass — no diffusion, no GANs.
Input to the UNet (52 channels, matching §A.3):
z_masked (4 ch) — SDXL VAE latent of the mouth-masked target frame
z̄_ref (16 ch) — reference latent projected by a small trainable 2-layer CNN
z_lips (32 ch) — M-dim lip-pose vector tiled spatially
The UNet predicts a residual Δz; the decoded image is VAE_decoder(z_masked + Δz).
Loss (Equation 9 from the paper — all weights implemented exactly):
0.1·L1_lat + 0.1·L1m_lat + 10·L1M_pix + 100·L1lips_pix + 50·VGG + 5·VGGFace
Lips Encoder:
| Component | Paper | This repo |
|---|---|---|
| Expression backbone | EmoPortraits (frozen neural) | MediaPipe 52-D blendshapes → MLP |
| Detail branch | Mouth-crop CNN → z_add | Same (trainable) |
| Output | z_lips = z_main + z_add (sum) | Same (M=32) |
| Distilled inference encoder | ResNet-34 head | Not yet implemented |
Once the reconstruction editor R_φ converges, I fine-tune a LipsChange editor L_θ on synthetic pseudo-pairs — following §3.1.2 and Appendix A.2 of the paper:
- Sample z_lips and synthesise a lip-altered variant S̃ = R_φ(S; z_lips).
- Form symmetric pairs: (S → S̃) with probability 2/3, (S̃ → S) with probability 1/3.
- Fine-tune L_θ (initialised from R_φ weights) without any mouth mask in the UNet input.
At inference, L_θ runs without explicit segmentation — it learns to localise edits on its own.
Predicts z_lips from speech using a flow-matching objective:
| Component | Paper | This repo |
|---|---|---|
| Audio features | wav2vec 2.0 | Mel spectrogram + 1D CNN |
| Network | Flow Matching Transformer (FMT) | Causal TCN |
| Emotion encoder e(a) | Yes | Not implemented |
| K source lip latents | Yes | Not implemented |
| Flow-matching objective | Yes | Implemented |
At inference, the predicted z_lips is fed directly into the Stage 1.5 LipsChange editor.
| Component | Status |
|---|---|
| Stage 1 — Visual Editor (reconstruction) | ✅ Done |
| Stage 1 — Lips Encoder (blendshape MLP + mouth CNN) | ✅ Done |
| Stage 1 — Losses (all 6 terms, paper-exact weights) | ✅ Done |
| Stage 1 — Cross-reconstruction + alternating training strategy | ✅ Done |
| Stage 1.5 — LipsChange mask-free self-refinement | ✅ Done |
| Stage 2 — Audio-to-Lips (flow matching, causal TCN) | ✅ Done |
| Preprocessing pipeline (frames, landmarks, blendshapes, latents) | ✅ Done |
| RunPod cloud training & deployment | ✅ Done |
| Distilled ResNet-34 inference encoder | ⏳ Pending |
1. Prepare character data
python -m flash_lips.scripts.preprocessing.prepare_character --video <path> --name <name>
2. Extract blendshapes
python -m flash_lips.scripts.preprocessing.extract_blendshapes --name <name>
3. Pre-compute VAE latents
python -m flash_lips.scripts.preprocessing.precompute_latents --name <name>
4. Train Stage 1 (Visual Editor)
python -m flash_lips.scripts.training.train_stage1 \
--config rtx_4090 run.character=<name>
5. Train Stage 1.5 (LipsChange — mask-free)
python -m flash_lips.scripts.training.train_lipschange \
--character <name> \
--reconstruction-checkpoint checkpoints/<name>/visual_editor_best.pt
6. Train Stage 2 (Audio-to-Lips)
python -m flash_lips.scripts.training.train_stage2 \
--character <name> --epochs 50
| GPU | VRAM | Config | Effective batch |
|---|---|---|---|
| RTX 4060 Ti | 16 GB | rtx_4060_ti |
8 |
| RTX 4090 | 24 GB | rtx_4090 |
8 |
| A6000 | 48 GB | a6000 |
24 |
| H100 SXM | 80 GB | h100_sxm |
32 |
| RTX 5090 | 32 GB | rtx_5090 |
— |
FlashLips U-Net runs at 109 FPS on an H100 SXM — 30× faster than KeySync (diffusion) and 19× faster than LatentSync, while matching or exceeding them on perceptual quality metrics.
This project is a personal re-implementation and extension of:
@article{zinonos2025flashlips,
title = {FlashLips: 100-FPS Mask-Free Latent Lip-Sync using Reconstruction
Instead of Diffusion or GANs},
author = {Zinonos, Andreas and Stypu{\l}kowski, Micha{\l} and Bigata, Antoni
and Petridis, Stavros and Pantic, Maja and Drobyshev, Nikita},
journal = {arXiv preprint arXiv:2512.20033},
year = {2025}
}Paper figures reproduced under CC BY-NC-SA 4.0 (the paper's licence).
This codebase is a personal research project and is not affiliated with the original authors.