full-duplex-speech-eval

Synthetic full-duplex dialogue generation with self-auditing evaluation.

Full-duplex voice systems (models that can listen while speaking) lack benchmarks that reward appropriate floor behavior (overlap, backchannels, interruption), and labelled conversational audio is expensive to produce. This project generates two-speaker conversations where the generator is the annotator: every label (turn spans, word timings, typed overlap regions, interruption events) falls out of a placement decision the pipeline already made, at zero labelling cost, with turn transitions resampled from 19,008 real floor-transfer offsets measured from the HCRC Map Task corpus. The generator is then audited by independent instruments it cannot influence: a pretrained Voice Activity Projection (VAP) turn-taking model and a paired neural-codec control. The audit's headline finding is negative.

What's built

The pipeline renders 12 stereo clips per batch (10 main + 2 on a held-out TTS engine), one speaker per channel, and emits with them the labels a benchmark needs: turn spans, word timings, typed overlap regions, backchannels, and interruption events. None of it is hand-annotated: each label is a record of a placement decision the generator already made.

Two design choices carry the weight:

• Turn timing is resampled, not fitted. Floor-transfer offsets are drawn directly from the empirical pool of 19,008 transitions measured from the HCRC Map Task timed-units annotations, not from a parametric summary of them, because summarising is exactly the step that produced the 54.5%-vs-27.4% error documented under Measurement lessons below.
• The held-out engine is the strong one. The 2 held-out clips use OpenAI TTS, the highest-quality engine available to the pipeline, so the holdout tests engine generalization rather than being trivially detectable.

Timing validation: real vs synthetic

Measured identically on both sides: the same IPU/FTO rule (Nguyen et al., TACL 2023) is applied to the Map Task timed-units annotations and to the pipeline's own published labels. From duplex-gen/out/README.md and duplex-gen/out_projection/README.md:

statistic	real (Map Task)	synthetic v0	synthetic projection
overlapping share of transitions	27.4%	34.5%	34.9%
FTO median	0.256 s	0.267 s	0.272 s
gap median	0.438 s	0.595 s	0.563 s
overlap median	0.328 s	0.581 s (1.8×)	0.401 s (1.2×)
n transitions	19,008	174	172

The overlap-median ratio ran v0 1.8× → bcfix 1.9× (0.581 → 0.619 s; the backchannel fix moved this aggregate the wrong way, consistent with the VAP finding below) → projection 1.2×. So the projection clamp improves this aggregate and worsens the VAP audit below: an aggregate timing statistic and a sequence-level predictability model disagree about the same intervention, which is why both instruments are kept.

A note on reading the per-batch reports: the labels/ and summary.json files are the run-time artifacts of each code state, and each batch's summary.json is the authoritative source for its numbers; the per-batch README.md reports were re-rendered at HEAD from those frozen summaries, so their prose reflects the final code state while their numbers are per-batch.

Generator quality checks

• ASR back-test (faster-whisper small, temperature pinned): median WER 7.7% across 24 stems (v0; 7.1% after the fixes); residual errors are dominated by text normalisation (two hundred dollar → $200), not intelligibility.
• Label error bar, measured where synthesizer ground truth exists: ASR-aligned word timings disagree with synthesizer timings by 46 ms mean, 101 ms p90 (n=387) (v0; p90 ~100 ms after the fixes); usable for distribution shape, not for fine per-gap claims.
• Invariant checks (interrupt leads in range, words inside turn spans, backchannels inside host turns) pass on every batch.

Hearing it

Three synthetic clips ship in samples/ (this project's own TTS output; the full listening protocol, including the real-corpus calibration excerpt, is in listen/INDEX.md):

• samples/01_best_d04_stereo.wav: baseline working, with backchannels landing inside the other speaker's turn (0.0% WER).
• samples/04_switched_v0_d07_stereo.wav: a one-word turn ("Switched.") completely masked by overlap.
• samples/05_switched_projection_d07_stereo.wav: the same moment after the projection clamp, audibly a partial fix.

Use headphones: one speaker per channel.

The audit caught what patching couldn't

The audit instruments showed my hand-tuned fixes made realism worse. Two successive interventions were built on top of the v0 pipeline: a backchannel level/prosody fix (bcfix) and a final-word overlap clamp (projection). Both were motivated by defects that are audible and real. Scored by a pretrained VAP model against 21 length-matched excerpts of real Map Task dialogue, every step moved the synthetic audio away from real on all three metrics:

• cumulative v0 → projection shift on vap_loss: −0.159 ± 0.108, with 10/10 clips agreeing in sign: synthetic dialogue became more predictable, while real dialogue sits at the less-predictable end of every metric;
• that shift is ≈24% of the 0.672 gap to real, in the wrong direction;
• the pattern is monotonic across both steps and all three metrics (H_bits, vap_loss, decisive).

The reading (full table in benchmarks/vap_delta_results.md): the deficiency of marginal-distribution sampling is structural. Patching backchannel placement and overlap projection at the margins is at or below the instrument's resolution and, where it registers at all, worsens the gap. The fixes each repaired a genuine local defect (you can hear both repairs in the samples above) while making the distribution less like real conversation. That is the argument for re-deriving the timing model rather than patching it again, and it is exactly the kind of thing an evaluation harness exists to catch before a demo-driven loop ships it.

VAP audit: levels and paired deltas

Pretrained stereo VAP model (Ekstedt & Skantze), 10 clips per batch paired by clip ID, real reference = 21 length-matched 68 s Map Task excerpts. Lower H_bits/vap_loss and higher decisive mean more predictable; real dialogue is the least predictable column. From benchmarks/vap_delta_results.md:

metric	v0	bcfix	projection	real (n=21)
H_bits	3.790 ± 0.130	3.781 ± 0.132	3.753 ± 0.127	4.282 ± 0.208
vap_loss	3.759 ± 0.384	3.666 ± 0.366	3.599 ± 0.311	4.431 ± 0.780
decisive	0.637 ± 0.031	0.643 ± 0.029	0.652 ± 0.029	0.505 ± 0.073

Paired per-clip deltas show v0→bcfix is a small but consistent regression (9/10 on vap_loss), bcfix→projection alone is a coin flip (5/10), and the cumulative v0→projection is 10/10 consistent at −0.159. A loudness-normalization control arm rules out audio level as the explanation.

Codec control: is the gap just acoustics?

The same 21 real excerpts scored before and after an EnCodec round trip (facebook/encodec_24khz, 24 kHz @ 6 kbps, each channel coded independently), so the only variable is the codec. From benchmarks/JOINT_RESULTS.md:

paired delta vs original (n=21)	value	sign agreement
codec H_bits	+0.026 ± 0.138	14/21 (indistinguishable from zero)
codec vap_loss	+0.124 ± 0.120	19/21
resample-only H_bits	−0.053 ± 0.058	18/21

An acoustic bottleneck manufactures essentially no predictability gap: against the synthetic-vs-real H_bits gap of −0.49, the codec accounts for roughly 5%. This reversed an earlier cross-corpus control (vocoded Fisher vs Map Task) that had attributed most of the gap to acoustics; the superseded section is kept verbatim and marked in JOINT_RESULTS.md, because the record of being wrong is part of the result. The defensible claim is a codec bottleneck does not manufacture this gap, not acoustics are exonerated: TTS departs from human speech in more ways than a codec round trip.

Reproducing

Generation (from duplex-gen/, requires uv; Python ≥3.12):

cd duplex-gen
uv run python generate.py               # ~4 min warm; seeded; writes to out_run/

Generation also requires ffmpeg on PATH (brew install ffmpeg) and is macOS-oriented: 3 of the 7 main voices use the macOS say synthesizer. Everything is seeded; a run rebuilds the clips with their labels, timing figure, and generated report in its own batch directory (generate.py mybatch → out_mybatch/; the script refuses to write into a directory that already contains a summary.json unless --force is passed, so the committed batches stay frozen).

Two honesty notes on what is and is not reproducible:

• Code state. The code at HEAD is the final state, with both interventions enabled (projection_clamp in duplex/timing.py). The committed out/, out_bcfix/, and out_projection/ directories are frozen artifacts of the three successive code states the audit compares: running generate.py today reproduces the projection-state output only. The earlier states survive as their committed labels, summaries, and reports.
• Labels vs audio. Every number in this README is backed by the committed labels, summaries, and result files. The audit scripts, however, consume audio, which is not committed: from a fresh clone only benchmarks/codec_control.py (after fetching Map Task) fully reproduces its table, while benchmarks/vap_delta.py and vap_delta2.py require the original synthetic audio (of which only the projection batch can be regenerated) and exit with a clear missing-audio message otherwise.

Stage 1 (scripts) needs no API key: the dialogue scripts are committed in assets/scripts.json. The 10 main clips use edge-tts (network, no key) and macOS say (local); only the 2 held-out clips call OpenAI TTS and need OPENAI_API_KEY; without a key they are skipped with a warning and the rest of the run completes. The reference FTO distribution is committed (assets/ref/maptask_fto.json) and is the supported path. Re-deriving it with duplex/reference.py requires the NXT annotation release (maptaskv2-1/Data/timed-units), a manual download from the Map Task site; scripts/fetch_maptask.sh fetches only the audio, not the annotations.

Auditing (from the repo root):

git clone https://github.com/ErikEkstedt/VAP benchmarks/VAP   # pretrained checkpoint included
scripts/fetch_maptask.sh                                      # real reference audio + excerpts
uv run --with torch --with torchaudio --with soundfile --with numpy \
       --with einops --with lightning --with transformers \
       python benchmarks/codec_control.py

benchmarks/README.md documents all five scripts and their inputs. The generated audio and third-party corpora are deliberately not committed; labels, summaries, and every number quoted here are.

Measurement lessons

The project's real subject is evaluation discipline. Three incidents, all documented in full where they happened:

1. The eval instrument was nondeterministic. faster-whisper falls back to temperature > 0 (which samples) when a greedy pass trips its compression-ratio or logprob threshold. One stem scored 74.0% WER during a run and 19.2% on recomputation from identical audio and identical code. Pinning temperature=[0.0] and condition_on_previous_text=False fixed it. An evaluation instrument that returns a different number for the same input is not an instrument (duplex-gen/out/README.md, "Four bugs that only running it caught").
2. A published summary statistic is not a substitute for the distribution it summarises. The first timing model targeted a 54.5% overlap share derived from per-minute event rates in a published figure; measuring the actual corpus gave 27.4%: the derivation had counted all overlapping speech against a transitions-only denominator. This is why the pipeline resamples the empirical FTO distribution instead of fitting a parametric summary to it.
3. We violated our own error bar one section after computing it. The projection fix was reported as clearing the previous speaker's endpoint by 0.07 s, but one of the two word timings involved is ASR-aligned, with a measured ±101 ms p90 error. The margin is inside the uncertainty of its own measurement, and a listener reports it does not clear. Stated plainly in out_projection/README.md because it is the failure mode this project exists to argue against.

Limitations

• Small n. 12 clips per batch (10 main + 2 held-out); 174 synthetic transitions against 19,008 real ones. The VAP paired deltas are over 10 clips.
• The headline deltas are not third-party re-scorable. The VAP paired deltas were scored on v0 and bcfix audio that is neither committed nor regenerable from HEAD, so the 10/10, −0.159 result cannot be recomputed from a fresh clone; only the projection batch and the codec control can be re-scored. The committed labels, summaries, and benchmarks/vap_delta_results.md are the record.
• One TTS family plus a 2-clip holdout. A classifier would separate the engines trivially; the held-out-engine clips exist to check engine fingerprinting and are far too few to check it properly.
• Synthetic scripts. LLM-authored customer-service dialogues; the reference corpus is British-English map-instruction dialogue. Real timing, wrong conversation type; the residual timing gap is deliberately left rather than tuned away.
• VAP is a proxy for realism, not a ground truth: it measures predictability under one pretrained model. vap_loss additionally depends on an energy-derived VAD.
• No floor competition. Two independent tracks placed on a shared timeline; the prosody does not react to being interrupted. The clips can teach what overlap looks like, not what causes it. (Full list: "What's fake about these clips" in duplex-gen/out/README.md.)

Attribution

• HCRC Map Task Corpus: Anderson et al., "The HCRC Map Task Corpus", Language and Speech 34(4):351–366, 1991; NXT annotations, Carletta et al. 2005. Freely available for research from the University of Edinburgh: https://groups.inf.ed.ac.uk/maptask/. This repo ships only measured floor-transfer offsets derived from the annotations (duplex-gen/assets/ref/maptask_fto.json), never the corpus audio; scripts/fetch_maptask.sh documents how to obtain it.
• VAP (Voice Activity Projection): Erik Ekstedt & Gabriel Skantze; pretrained model and code at https://github.com/ErikEkstedt/VAP.
• EnCodec: Défossez et al., "High Fidelity Neural Audio Compression", 2022; facebook/encodec_24khz via Hugging Face transformers.
• Full-Duplex-Bench: Lin et al.; https://github.com/DanielLin94144/Full-Duplex-Bench. Cited as the benchmark measuring the interruption-reaction behavior these clips are documented not to have.
• Comparison samples in benchmarks/JOINT_RESULTS.md: dGSLM (Nguyen et al., TACL 2023) and NTPP (Wang et al., ICLR 2025), from their public demo pages.

License

MIT; see LICENSE. Third-party corpora, models, and demo samples referenced above remain under their own licences and are not redistributed here.