Expanded LLM/STT/TTS usage tracking across pre-prod flows and some fixes

[MSami625]

What Changed?

• Expanded LLM/STT/TTS usage tracking across pre-prod flows: evaluator results, Metrics Studio runs, persona generation, playground voice calls, and test-agent LLM-to-LLM simulation
• Added external_agent_usage.py to extract and record provider usage from Vapi, Retell, ElevenLabs, and Smallest call payloads
• Wired usage recording into playground polling, evaluator result processing, and test-agent bridge/simulation paths (with dedup via external_usage_recorded)
• Added LLM-to-LLM evaluator simulation for voice-bundle-only agents (no external voice provider)
• Fixed Celery run_evaluator asyncio event loop handling for voice-bridge runs
• Improved playground polling (skip duplicate processing, Vapi polls on refresh only)
• Relaxed CSP connect-src / worker-src for voice provider WebRTC connections
• Added tests for pre-prod usage, simulation usage, LLM-to-LLM simulation, and CSP voice provider rules

Why?

Pre-prod testing (evaluators, playground, personas, Metrics Studio) was not consistently attributing LLM/STT/TTS usage, and external voice provider token usage from call payloads was not being captured. This gives accurate per-workspace usage/cost tracking across synthetic testing and live provider calls.

How to Test?

1. pytest tests/test_services/test_usage/test_pre_prod_usage.py tests/test_services/test_usage/test_test_agent_simulation_usage.py tests/test_services/test_testing/test_llm_to_llm_evaluator_simulation.py tests/test_core/test_security_headers_middleware.py -v
2. Run an evaluator with voice bundle only → confirm LLM-to-LLM simulation completes and usage is recorded under TEST_AGENT
3. Run an evaluator with voice bridge → confirm bridge call works (no asyncio event loop error) and usage is tracked
4. Make a playground web call (Vapi/Retell/ElevenLabs) → confirm provider LLM/STT/TTS usage is recorded once
5. Run a Metrics Studio batch → confirm usage is attributed to the metric studio run
6. Verify playground loads and connects to voice providers without CSP errors

Release Label

• major - breaking change, next release bumps major version
• minor - backward-compatible feature, next release bumps minor version
• fix - backward-compatible bug fix, next release bumps patch version
• No label (defaults to patch release)

Checklist

• [ x]] I have read the CONTRIBUTING.md guide.
• [ x]] My code follows the project's style guidelines.
• I have added tests that prove my fix is effective or my feature works.
• [ x]] I have updated documentation where needed.

[greptile-apps]

Greptile Summary

The PR expands attribution and billing telemetry across evaluator, playground, persona, Metrics Studio, and test-agent workflows while adding internal LLM-to-LLM voice simulation.

• Adds external-provider LLM/STT/TTS extraction and playground post-call processing.
• Propagates richer usage context through API and worker flows.
• Adds Flexprice product-event wiring and setup support.
• Updates voice-provider connectivity policy, authentication session handling, and related tests.

Confidence Score: 3/5

The PR does not yet appear safe to merge because playground retries can duplicate captured provider usage and evaluator simulation retries can rebill completed turns.

Provider counters are durably incremented before the playground dedup marker commits, while LLM-to-LLM simulation records additive usage per turn but retries from the beginning without durable progress; both paths can inflate workspace usage and cost.

Files Needing Attention: app/services/playground/post_call_processing.py, app/workers/tasks/run_evaluator.py, app/services/testing/llm_to_llm_evaluator_simulation.py

Important Files Changed

Filename	Overview
app/services/playground/post_call_processing.py	Adds row-locked post-call deduplication, but usage capture and marker persistence still cross independent transactional boundaries.
app/api/v1/routes/playground.py	Routes completed provider calls through locked usage and evaluator-result claims, resolving the previously reported overlapping-poll race.
app/services/usage/external_agent_usage.py	Extracts normalized provider usage and reports capture failures to callers across supported external voice platforms.
app/workers/tasks/run_evaluator.py	Adds voice-bundle simulation and safer event-loop handling, while task-level retry still replays uncheckpointed simulation work.
app/services/testing/llm_to_llm_evaluator_simulation.py	Implements per-turn synthetic conversations with usage attribution, but persists transcript progress only after the full simulation succeeds.
app/services/billing/flexprice_service.py	Expands idempotent external product-event metering and associated event metadata.
app/services/usage/context.py	Extends usage-context dimensions and propagation for additional pre-production workflows.

_{Reviews (13): Last reviewed commit: "feat(playground): enhance call data proc..." | Re-trigger Greptile}

[TEJASNARAYANS]

@greptile-apps