Meter STT usage on audio streamed to the provider - #9
Open
abhinav-t41 wants to merge 5 commits into
Open
Conversation
STT providers bill on audio streamed to them (silence included), but we only measured VAD-detected turn audio, systematically underestimating real usage. Add SttUsageMeterMixin that counts bytes at the run_stt seam — the exact point audio leaves for the provider — on both instrumented STT services, and emit a usage.stt trace event at session teardown with streamed_seconds (billing truth), speech_seconds (VAD turn audio, kept for analytics), and provider_reported_seconds (Deepgram's own duration from ListenV1Metadata, for calibration). Also documents the broader cross-provider cost-comparison design in docs/cost-metering-design.md. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
TTS providers bill on characters sent over the wire, not characters the user hears: Pipecat pushes each sentence to the provider as the LLM streams it, so on interruption Deepgram has already received (and bills for) text whose audio is discarded by Clear. Pipecat's built-in TTSUsageMetricsData metric cannot measure this — the Deepgram websocket service never emits it, and the base-class fallback drops accumulated text on interruption, which is exactly where billing diverges from the transcript. Add TtsUsageMeterMixin counting characters at the run_tts seam — the exact prepared text sent to the provider — on both instrumented TTS services, and emit a usage.tts trace event (provider, model, voice, sent_characters) at session teardown next to usage.stt. Also rewrite docs/cost-metering-design.md §6 with the verified billing semantics: sent = billed for Deepgram; ElevenLabs deducts credits on successful generation so sent_characters is a tight upper bound there; per-provider differences belong in the pricing catalog, not the meter. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Meter TTS usage on characters sent to the provider
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
STT providers bill on audio streamed to them — silence included — but we only measured VAD-detected turn audio (
AudioBufferProcessorturns), systematically underestimating real usage. A 5-minute call with 40 seconds of speech still streams ~5 minutes of audio to the provider.Fix
SttUsageMeterMixincounts bytes at therun_sttseam — the exact point audio leaves for the provider, after all mute/reconnect/empty-frame guards — on bothInstrumentedDeepgramSTTServiceandInstrumentedElevenLabsSTTService. Seconds = bytes ÷ (sample_rate × 2) for 16-bit mono PCM.usage.stttrace event is recorded per run:streamed_seconds— the billed quantity (should match provider dashboards within ~1s)speech_seconds— VAD turn audio, kept for analyticsprovider_reported_seconds— Deepgram's ownMetadata.duration, accumulated across reconnects, as a built-in calibration check (null for ElevenLabs)session_duration_secaccessor onSttEvaluationSession; the existing turn-audio capture is unchanged.Docs & tests
docs/cost-metering-design.md— the broader cross-provider cost-comparison design this is step 2 of.server/tests/test_stt_usage_meter.py— byte→seconds math, zero guards, per-instance state.Testing
uv run pytest -q: 29 passed; the 2test_agent_config.pyfailures also fail on cleanmain(pre-existing, unrelated).ruff check: no new violations (4 pre-existing onmain).select payload from trace_events where event_type = 'usage.stt' order by created_at desc limit 1;— for Deepgram sessions
streamed_seconds≈provider_reported_seconds; for ElevenLabs compare against the dashboard usage.