title	Rubato
emoji	🎼
colorFrom	indigo
colorTo	gray
sdk	gradio
sdk_version	5.49.1
app_file	app.py
pinned	false
license	apache-2.0
short_description	Predict the silence as a distribution, then decide when to speak.
tags	turn-taking endpointing spoken-dialogue voice-agents survival-analysis realtime

Rubato

Predict the silence as a distribution, then decide when to speak.

Every voice agent you have used waits a fixed number of milliseconds before it answers. That one number is a bad trade forced on you: lower it and the agent talks over people who were only thinking; raise it and every reply feels slow. You get to pick a point on a curve, and the curve is the problem.

Rubato replaces the number with a distribution and a decision. At 50 Hz it predicts how much longer this particular silence will last, given who is speaking, what they just said, how they said it, and whether they just took a breath. Then a dynamic program picks the moment to start talking under an explicit price on the two errors — α for talking over someone, β for a second of dead air.

That moves the whole curve, not a point on it. Same audio, same recogniser: fewer collisions and faster replies, at every operating point.

Rubato is a model-agnostic middle layer. It has no ASR, no LLM, no TTS, and no opinion about what your agent says. Full-duplex speech models have settled the question of how to interleave audio; this is about when to start.

Both axes are minimised. Condition A is a curve because one threshold buys one point; Rubato's curve sits inside it, which is what "simultaneous improvement" means.

maptask (CC-BY-4.0) · 30,987 silences from 128 conversations and 64 speakers · 5,988 held out, speaker-disjoint (0 speakers shared between train and test).

	measured
covers the fixed-threshold frontier (A)	86% of its points
covers the semantic-completeness frontier (B)	100% of its points
latency saved at matched talk-over	84 ms vs A · 96 ms vs B
talk-over removed at matched latency	2.8 pp vs A · 2.1 pp vs B
hazard calibration (ECE)	0.0007
CRPS skill vs a covariate-free hazard	+0.173
layer cost per 20 ms frame	0.26 ms median, 0.49 ms p99

At the setting matched to a 1000 ms threshold: response latency 1000 → 772 ms (95 % CI 707–831) and talk-over 6.3% → 6.5% (95 % CI 5.6%–7.6%). Intervals are a conversation-level cluster bootstrap.

Ablations — where the improvement comes from

Mean height of each frontier over the latency range every condition can reach (0.30–2.00 s); lower is better.

condition	front area
marginal-DP	0.0996
A-fixed	0.0801
B-semantic	0.0744
C-greedy	0.0726
C-no-entrain	0.0661
C-timing-only	0.0609
C-rubato	0.0582
B-oracle	0.0057

marginal-DP is the control: the same dynamic program on a hazard with no covariates. Theory says it must reduce to a fixed threshold and therefore land exactly on A-fixed. Measured directly: its 26 settings collapse to 11 distinct onsets — as a fixed threshold must — and each sits within 0.07 pp of A's curve at the same latency (mean -0.007 pp). Its worse area is only that coarser sampling, not worse decisions. That is what makes the gap between A-fixed and C-rubato attributable to the prediction rather than to the optimiser.

B-oracle reads the ground-truth label and is unattainable; it bounds how much a better language model alone could buy.

Calibration, per-speaker robustness, and every arm on one plot

Sensitivity — does the result survive the protocol's assumptions?

Each row re-runs the whole benchmark with one assumption changed. The claim is only worth stating if it does not depend on the choices that could have been made otherwise.

variant	C covers A / B	latency saved vs A	talk-over removed vs A
headline settings	86% / 100%	84 ms	2.8 pp
right-censor a yielded floor at the partner's real onset (+slack) instead of assuming it stays open	86% / 100%	144 ms	3.8 pp
ASR lag = 0: the transcript arrives the instant the word ends	93% / 95%	112 ms	4.2 pp
conflict window 1.0 s (a short agent reply)	93% / 100%	97 ms	2.8 pp
conflict window 4.0 s (a long agent reply)	86% / 100%	76 ms	2.9 pp

Reproduce with make sensitivity.

What the numbers say, including where they are modest

• The headline holds. Rubato's frontier is inside the fixed-threshold frontier over the whole practical range, and the improvement is simultaneous: both readings — latency saved at matched collisions, collisions removed at matched latency — are positive. It is not a re-parameterised trade.
• The margin over a well-tuned semantic baseline is real but not enormous. Condition B, given the same linguistic features and 294 settings to choose from, gets most of the way there. The distribution and the dynamic program buy a further step; they do not make the LLM-completeness approach look silly, and the ablation table is where you should look rather than the headline.
• Each component earns its place, and one of them nearly did not. Ablating the acoustic cues, the entrainment, or the dynamic program each makes the frontier worse. The acoustic cues only help once the forecast decays transients correctly: a first implementation froze "in-breath 100 ms ago" across the whole three-second horizon, which told the policy the speaker was permanently about to resume and made those features worse than not having them at all. That is now DYNAMIC_DECAY, and tests/test_causality.py::test_transients_decay_in_the_forecast pins it.
• The control behaves as predicted. A dynamic program on a covariate-free hazard lands on condition A's curve, never below it. The gain is coming from the prediction, not from the optimiser.
• There is headroom left. A perfect completeness judge would answer at the ASR lag with almost no collisions. Nobody has one, but the gap to it says the ceiling here is set by knowing whether the turn is over, not by the decision layer.
• This is one corpus. Task-oriented, English, two-party, 64 speakers. The mechanism should generalise; the constants will not.

---

Three lines

from rubato import load_pretrained, CostWeights

taker = load_pretrained(weights=CostWeights.from_seconds_per_collision(2.0))
should_speak = taker.push_audio(chunk).should_speak     # one 20 ms frame of mono float audio

seconds_per_collision is the entire configuration surface, and it is written in the units of the decision you are actually making: how many seconds of extra latency is one talk-over worth to you? A customer-support bot might say 4; a brainstorming partner that should jump in might say 0.3.

Already using a framework:

# Pipecat — withholds the end-of-turn signal until the floor is genuinely free
from rubato.integrations.pipecat_processor import RubatoTurnGate
pipeline = Pipeline([transport.input(), stt, RubatoTurnGate(), llm, tts, transport.output()])

# LiveKit Agents — a per-utterance endpointing delay instead of a constant
from rubato.integrations.livekit_plugin import RubatoTurnDetector
session = AgentSession(stt=..., llm=..., tts=..., turn_detection=RubatoTurnDetector())

---

How it fits

flowchart LR
    MIC([microphone]) --> VAD[VAD + prosody<br/>causal, backward windows only]
    MIC --> ASR[ASR<br/>partial transcript]

    subgraph RUBATO ["Rubato — the timing layer"]
        direction TB
        PM["<b>PauseDistributionModel</b><br/>discrete-time hazard h<sub>k</sub><br/>streaming, calibrated"]
        EN["<b>OnlineEntrainment</b><br/>this partner's tempo<br/>and pause habits"]
        DP["<b>DecisionPolicy</b><br/>argmin α·P(collision) + β·delay<br/>backward dynamic program"]
        SG["<b>SilenceGeneration</b><br/>how long <i>we</i> should pause<br/>given what we're about to say"]
        EN -- "speaker posterior" --> PM
        PM -- "survival curve S(t)" --> DP
        EN -- "tempo match" --> SG
    end

    VAD -- "energy, F0, breath" --> PM
    ASR -- "completeness<br/>(gated by ASR lag)" --> PM
    DP -- "START / WAIT" --> LLM[LLM]
    LLM --> SG
    SG -- "onset delay" --> TTS[TTS]
    TTS --> SPK([speaker])

    style RUBATO fill:#eef4ff,stroke:#4a7fd4,stroke-width:2px
    style DP fill:#dceafe,stroke:#2f6fd0
    style PM fill:#dceafe,stroke:#2f6fd0

Loading

Rubato sits between the detector and the generator. It never sees a waveform it has to understand and never emits one.

---

The four pieces

1 · PauseDistributionModel — the silence as a survival curve

At frame k of an ongoing silence it emits

$$h_k = P(\text{partner resumes in frame } k{+}1 \mid \text{silent through } k,\ x_{1:k}), \qquad S_k=\prod_{j\le k}(1-h_j)$$

from a one-hidden-layer network over a strictly causal feature vector: a radial basis expansion of log elapsed silence (the difference between 100 ms and 300 ms of silence is pragmatically enormous; between 2.1 s and 2.3 s it is nothing), the duration and internal pause count of the turn so far, transcript completeness, terminal prosody, per-frame acoustic precursors like in-breaths, and a running posterior over this partner.

Two things are load-bearing and both are tested:

• Causality. Frame k reads only frames ≤ k. The forecast of future hazards is the model asked what it would say later if the evidence froze — never a peek. tests/test_causality.py rewrites the future of the signal and asserts the already-emitted decisions are bit-identical.
• Calibration. Among the frames where it says 5 %, the partner must resume about 5 % of the time — because the policy multiplies these numbers by costs. A ranking-good, probability-bad model picks the wrong stopping time at every operating point.

2 · DecisionPolicy — optimal stopping under an asymmetric cost

Talking over someone and answering late are both errors, but not the same error, and their exchange rate is a product decision rather than a fact about speech. So it is a parameter. A backward dynamic program over the forecast survival curve solves for the stopping time exactly:

$$C_\text{speak}(j)=\alpha,P(T\le j{+}W\mid T>j)+\beta,(k{+}j)\Delta,\qquad V(j)=\min!\big(C_\text{speak}(j),;(1-h_j),V(j{+}1)\big)$$

Being pre-empted costs zero. If the human carries on while we are still silent, that is the outcome we wanted: no collision, and no answer was owed yet. That single asymmetry produces the behaviour people recognise — when a resumption looks likely, waiting is nearly free, so it waits; when the floor is clearly open, every frame is pure cost, so it starts immediately, including at zero gap, which no fixed threshold can ever do.

Why a fixed threshold is the special case, not the competitor. Take a hazard with no covariates. Then $h_j$ is identical in every episode, the dynamic program returns the same stopping frame every time — and that is a fixed threshold. Fixed thresholds are exactly the optimal policies under a distribution that knows nothing. Conditioning can only do better. The benchmark ships this as a control: marginal-DP runs the same solver on a covariate-free hazard and must land on top of condition A's curve. tests/test_pareto.py asserts both halves. If the control did not collapse, the measured gain would be an artefact of the optimiser and the result would mean nothing.

3 · SilenceGeneration — the agent's own pauses

The other three components decide when it is safe to speak. This one decides when it is right to. A confirmation 700 ms late sounds slow; an instant answer to a hard question sounds like it did not listen; an instant reply to something painful sounds callous. People read response latency as evidence of processing, and answering as fast as possible throws that channel away.

A base delay per dialogue act, scaled by content difficulty, emotional weight, whether the response is dispreferred, and the partner's measured tempo. scripts/blind_eval.py runs the three-condition listening test — fixed, random, Rubato — in randomised order. The random arm draws from the same marginal delay distribution as Rubato, which is what separates "variation sounds better" from "variation about the right things sounds better".

4 · OnlineEntrainment — adapting to this person

Conjugate posteriors (Normal-Inverse-Gamma over log pause and gap durations, Beta over the hold rate) plus an EMA of speech rate, all shrinking to a population prior so a new partner is handled exactly as well as the average one. A system tuned to the population mean is, by construction, too impatient with people who pause to think — which is why the benchmark reports the slow-speaker gap as a metric in its own right and not just an aggregate.

---

The benchmark

Offline replay over recorded conversations: the agent takes one participant's place, sees the recording as a causal stream, and every condition is scored by the same code path — rubato.streaming.replay_episode, which is the definition of the metrics.

	condition
A	Fixed threshold — what almost everything deployed does. Swept over 29 thresholds; 800 / 1000 / 1200 ms get labelled markers.
B	Semantic completeness — a completeness judge picks between a short and a long threshold. Trained on the same split, given the same linguistic features Rubato gets, and swept over 294 (p*, fast, slow) combinations so its frontier is as good as that family gets.
B*	Perfect completeness judge — reads the ground-truth label. Unattainable, drawn dashed, never scored as a competitor. It exists to answer "wouldn't a stronger LLM close the gap?" with a number instead of an opinion.
C	Rubato — distribution + asymmetric-cost dynamic program + entrainment.

Ablations: −entrainment, −prosody/breath (a separately trained timing-only model, so the ablation measures the feature and not a train/test mismatch), −dynamic program (greedy threshold on the same distribution), and the marginal-DP control.

Metrics

1. Talk-over rate × response latency, as a Pareto front — the headline.
2. False barge-in rate — collisions restricted to turn-relevant holds, where the agent actually took someone's turn away. Reported separately because a policy can lower the marginal rate by being timid everywhere while still stealing turns at the moments that matter.
3. Calibration — reliability diagram, ECE, and censoring-aware CRPS, each with a skill score against the covariate-free hazard. (Calibration alone is cheap: always predicting the base rate is perfectly calibrated and useless.)
4. Per-speaker spread, especially the gap between the slowest and fastest third of talkers.
5. The layer's own cost — must stay under 10 ms per 20 ms frame.

Uncertainty is a conversation-level cluster bootstrap. Episodes inside one dialogue come from the same two people; an episode-level bootstrap would give intervals several times too narrow.

Assumptions, stated rather than buried

These are the three places the protocol could be accused of tilting the field, so each one is named, defaulted defensibly, and swept.

• The open-floor counterfactual. When the partner takes the floor after a gap, we assume the first speaker had finished and the floor would have stayed open for max_wait_s. The alternative — right-censoring at the partner's real onset — is available (--counterfactual censor) and is worse, because that censoring is informative: the partner speaks precisely when the first speaker sounds finished, which biases the estimated hazard sharply upward.
• The conflict window W (default 2.0 s) is the assumed length of the agent's reply: how long it occupies the floor once it starts. It is swept (--conflict-window), and two window-free companions — hold_intrusion_rate and false_barge_in_rate — are reported alongside.
• ASR lag (default 250 ms) gates every transcript-derived feature, for condition B and for Rubato. A benchmark that hands a method the last word the instant it ends flatters every semantic approach including this one. Swept with --asr-lag 0.

make sensitivity re-runs the whole thing under each alternative.

Splits

Speaker-disjoint, by union-find over the (conversation, speaker) graph. Map Task participants appear in several dialogues, and the entrainment claim is precisely a claim about learning an individual's habits — a leaked participant would make it untestable. The benchmark prints the train/test speaker overlap; it should be zero.

Reproducing

make install
make bench          # downloads 12 MB of CC BY 4.0 annotations, no audio
make figures

---

Data

corpus	licence	why
HCRC Map Task (primary)	CC BY 4.0	Genuinely dyadic. Silences come from forced alignment as explicit <sil> spans, so within-turn pauses are measured, not inferred from transcriber segment edges — 31 % of them are under 200 ms, and a corpus that could not see those would make an impatient threshold look far better than it is. It also annotates in-breaths and lip smacks, the acoustic precursors Rubato's dynamic features exist to use.
AMI Meeting Corpus	CC BY 4.0	Secondary. Four-party, and its manual word timings tile contiguously inside transcriber segments, so its effective pause resolution is coarser. Episodes are restricted to locally dyadic stretches.
Synthetic dyads	CC0	Offline tests, CI, and exercising the audio path end to end. Never reported as a headline result. Its generative process — hierarchical latent traits, categorical clause endings, mixture-of-lognormal pauses — is deliberately not the model's parameterisation, so fitting it requires generalising.

Only annotations are downloaded. No audio is fetched, and none is redistributed. Prosodic features are available if you point --audio-root at recordings you have obtained yourself under their terms. Datasets built by scripts/build_dataset.py carry the source licence in the card's front matter, in prose, and on every record; the script refuses to push a corpus marked non-redistributable.

Attribution, as both licences require:

Anderson, A. H. et al. (1991). The HCRC Map Task Corpus. Language and Speech 34(4), 351–366. Annotations v2.1 © 2007 HCRC, University of Edinburgh & University of Glasgow. CC BY 4.0.

Carletta, J. et al. (2006). The AMI Meeting Corpus: A Pre-Announcement. MLMI. CC BY 4.0.

---

Responsible use

This project assumes the system using it discloses that it is an AI.

Rubato exists to make assistants less irritating to talk to. The same capability makes a synthetic voice harder to tell from a person, and that is a use this project is not for. Concretely:

• Disclose, in the modality the person is attending to. Spoken, at the start — not buried in a settings page or a terms-of-service link.
• Do not use it for impersonation. Not to make an automated caller pass as human, not to imitate a specific person, and not in any deployment whose value depends on the listener not knowing. Several jurisdictions require disclosure for automated calls; treat that as a floor, not a target.
• Do not use it to time interruptions strategically — to cut someone off at a moment calculated to stop them finishing a thought. The asymmetric cost makes that trivially configurable, which is exactly why it is named here.
• Treat entrainment state as personal data. It is a behavioural profile of an individual's speech. Keep it in session scope unless you have a reason and consent to persist it.
• Speech data. Do not redistribute recordings of identifiable people without consent and a licence. This repository downloads no audio and publishes none; keep it that way in anything you build on it.

The layer has no way to enforce any of this. It is stated because the people who build on it are the ones who can.

---

Repository

rubato/
  types.py            timing constants and the episode/decision data model
  features.py         causal feature extraction, ASR-lag gating, completion judge
  pause_model.py      streaming discrete-time hazard + calibration + fast oracle
  policy.py           the asymmetric-cost dynamic program
  silence_gen.py      the agent's own content-sensitive pauses
  entrainment.py      per-partner conjugate posteriors
  streaming.py        the one causal loop; offline replay = the metric definitions
  replay.py           exact fast paths for the sweeps
  calibration.py      reliability, censoring-aware CRPS, skill scores
  baselines.py        conditions A, B and the oracle reference
  data/               maptask.py, ami.py, synthetic.py, schema.py, audio.py, loader.py
  integrations/       pipecat_processor.py, livekit_plugin.py
  cli.py
benchmarks/           run.py, metrics.py, pareto.py, figures/, results/
scripts/              push_model.py, build_dataset.py, blind_eval.py, update_readme.py
tests/                causality, calibration, pareto, latency, replay-equivalence,
                      policy, entrainment, silence-gen, data, streaming
app.py                the Gradio Space

make test          # 94 tests, offline, ~45 s
make bench         # the full benchmark
make sensitivity   # every protocol assumption, swept
make listening     # blind A/B/C stimuli for the silence-generation study
make app           # the Space, locally

Every public function's docstring names which claim it exists to demonstrate: simultaneous-improvement, calibration, low-latency, or entrainment.

Known limitations

• English lexical features. The temporal and acoustic half transfers; the completeness features do not.
• Task-oriented training data. Map Task participants are following a route on a map; open conversation has longer and more variable gaps.
• Two-party only. Multi-party floor management is a different problem.
• The layer cannot know you paused because you were about to say something difficult. It models the surface of turn-taking, not the intention behind it.

Citation

@software{rubato2026,
  title  = {Rubato: predicting silence distributions for spoken-dialogue turn-taking},
  year   = {2026},
  note   = {Apache-2.0}
}