Skip to content

Add batched Whisper inference with configurable batch size - #49

Open
Suijiku wants to merge 2 commits into
ObscureAintSecure:masterfrom
Suijiku:batched-inference
Open

Add batched Whisper inference with configurable batch size#49
Suijiku wants to merge 2 commits into
ObscureAintSecure:masterfrom
Suijiku:batched-inference

Conversation

@Suijiku

@Suijiku Suijiku commented Jul 9, 2026

Copy link
Copy Markdown

Closes #46.

Summary

TalkTrack decodes sequentially. faster-whisper >= 1.1 ships BatchedInferencePipeline
(same model, VAD-chunked parallel decode), typically several times faster on long
recordings. It still returns a lazy segment generator, so the existing
for segment in segments_gen: progress.emit(...) loop keeps working (segments just
arrive in bursts) and batched segments still carry avg_logprob/start/end/text.

A new Batch size setting (Settings > Transcription, range 1–16, default 8) is
persisted in config and passed through to the worker.

Safety hatch: batch_size = 1 keeps the classic sequential path
(model.transcribe), which retains condition_on_previous_text. Any value > 1 uses
BatchedInferencePipeline. This makes the change strictly non-regressive at batch size 1.

Changes

  • app/transcription/transcriber.pybatch_size on TranscriptionWorker; batched vs
    sequential path selection.
  • app/ui/settings_dialog.py — "Batch size" QSpinBox (load/save).
  • app/utils/config.pytranscription.batch_size default (8).
  • app/main_window.py — pass batch_size from config to the worker.
  • requirements.txt / pyproject.tomlfaster-whisper>=1.1.0; uv.lock re-synced.
  • tests/test_transcriber.py — batched vs sequential path selection, VAD enabled, segment
    mapping.

Trade-off (stated honestly)

Batching drops cross-chunk condition_on_previous_text — a minor consistency cost — in
exchange for speed and robustness against hallucination-repetition loops on long
recordings (the same trade WhisperX makes). batch_size = 1 is available for anyone who
prefers the classic behavior.

Testing

  • python -m pytest tests/ -v — full suite green, including new tests that assert the
    batched pipeline is used for batch_size > 1, the sequential path for batch_size == 1,
    VAD is enabled, and segments map correctly (avg_logprob → confidence).
  • uv lock --check — resolves; resolved faster-whisper version unchanged (1.2.1).
  • App launches and transcribes end-to-end on the batched path (Windows 11).
  • Verified against faster-whisper 1.2.1 that BatchedInferencePipeline.transcribe accepts
    batch_size/vad_filter/language and that batched segments still carry
    avg_logprob/start/end/text.

The ~4x figure is faster-whisper's documented batched-vs-sequential speedup, not a number
I measured here. Happy to add machine-specific wall-clock + peak-RAM numbers (sequential vs
batch 4 vs 8 on the same recording) if useful — just let me know the target hardware.

Suijiku added 2 commits July 9, 2026 16:16
Wrap the cached WhisperModel in faster-whisper's BatchedInferencePipeline for
VAD-chunked parallel decoding (typically several times faster on long
recordings). batch_size is configurable in Settings > Transcription (1-16,
default 8) and persisted in config. batch_size == 1 keeps the classic
sequential path as a safety hatch (retains condition_on_previous_text); >1
uses the batched pipeline. The segment loop is unchanged — batched segments
still carry avg_logprob/start/end/text. Bumps faster-whisper floor to >=1.1.0
(BatchedInferencePipeline) in requirements.txt and pyproject.toml.
Regenerated after bumping the requirement floor for BatchedInferencePipeline.
Resolved version is unchanged (1.2.1); only the recorded specifier updates.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add batched inference (BatchedInferencePipeline) for faster transcription

1 participant