[DRAFT EVIDENCE] competing API-cost outbox implementation - #988
Closed
groupthinking wants to merge 16 commits into
Closed
[DRAFT EVIDENCE] competing API-cost outbox implementation#988groupthinking wants to merge 16 commits into
groupthinking wants to merge 16 commits into
Conversation
…-outbox-17376027973963893260 # Conflicts: # .github/workflows/coverage.yml
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
…o the first/most-expensive pricing key (`gemini-3-pro`), overcounting cost by ~16x
This commit fixes the issue reported at src/youtube_extension/services/ai/hybrid_processor_service.py:334
## Bug
`calculate_cost` in `src/youtube_extension/backend/services/api_cost_monitor.py` had this fallback for unknown models:
```python
service_costs = self.COST_MODELS[service]
if model not in service_costs:
# Use average cost for unknown models
model = list(service_costs.keys())[0]
```
The comment says "average cost", but the code picks the **first** key. For Google that is `gemini-3-pro` (`input 0.000875 / output 0.0035`) — the **most expensive** Google model.
### Concrete trigger
`_track_gemini_usage` (hybrid_processor_service.py:330) records usage with `model=result.model_name`. `GeminiResult.model_name` defaults to `DEFAULT_GEMINI_MODEL = os.getenv("GEMINI_MODEL", "gemini-3.5-flash")` (gemini_service.py:60,283).
`COST_MODELS["google"]` only contains: `gemini-3-pro`, `gemini-3-flash`, `gemini-1.5-pro`, `gemini-1.5-flash`. Since `gemini-3.5-flash` (and the test's `gemini-2.0-flash`) are absent, the fallback selects `gemini-3-pro`.
Intended flash tier `gemini-3-flash` is `input 0.000052 / output 0.00021`, so `0.000875 / 0.000052 ≈ 16.8x` and `0.0035 / 0.00021 ≈ 16.7x`. **Every tracked Gemini call is overcounted by ~16x.** This directly undermines the budget-alert/outbox feature: alerts trip far too early and dashboards inflate.
## Fix
Rewrote the fallback into a tier-aware resolver `_fallback_model_cost`:
1. Tokenizes the model name on non-alphanumeric separators (crucial: naive substring matching would match `mini` inside `gemini`).
2. Matches a shared pricing tier keyword (`flash`, `pro`, `mini`, `haiku`, etc.), cheapest-tier-first, so `gemini-3.5-flash` → `gemini-3-flash` and `gemini-2.0-flash` → `gemini-3-flash`.
3. If no tier matches, uses the **true average** across known models (matching the original comment's intent) instead of the most-expensive key.
Also moved the `youtube` quota branch above the lookup (its result is independent of the model key) and guarded against `None`.
### Verification
* `gemini-3.5-flash`, `gemini-2.0-flash`, `gemini-2.5-flash-lite` → flash pricing (`0.000052 / 0.00021`).
* `gemini-3-pro-preview` → pro pricing.
* Fully unknown names → averaged pricing.
* Exact-match models (e.g. the `test_anthropic_migration.py` `claude-*` keys) hit `service_costs.get(model)` directly and are unchanged.
`re` and `Any` were already imported in the file, so no new imports were needed.
Co-authored-by: Vercel <vercel[bot]@users.noreply.github.com>
Co-authored-by: groupthinking <garveyht@gmail.com>
Contributor
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Contributor
|
Important Review skippedAuto reviews are limited based on label configuration. 🏷️ Required labels (at least one) (1)
Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Repository YAML (base), Repository UI (inherited), Organization UI (inherited) Review profile: ASSERTIVE Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
✨ Simplify code
Comment |
🔍 PR Validation
|
github-actions
Bot
added
documentation
Dependency Review✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.Snapshot WarningsEnsure that dependencies are being submitted on PR branches and consider enabling retry-on-snapshot-warnings. See the documentation for more information and troubleshooting advice. Scanned FilesNone |
Agent Completion Truth Gate: BLOCKEDReasons: Machine-readable verdict{
"details": {
"invalid_fields": [
"issue.number",
"policy.agent_login",
"policy.run_id"
]
},
"reasons": [
"invalid_payload"
],
"verdict": "blocked"
} |
groupthinking
marked this pull request as draft
July 25, 2026 00:43
groupthinking
changed the title
groupthinking
added
ci/cd
duplicate
ChatGPT Codex Connector
Owner
Author
Duplicate disposition receipt
#988 has no focused child issue or valid execution manifest and directly competes with the existing canonical API-cost outbox unit. #869's exact head has the full CI/security/PostgreSQL suite passing and zero unresolved review threads. Closing #988 without merge as duplicate evidence. The branch and all commits remain preserved; no implementation progress is credited. |
Contributor
There was a problem hiding this comment.
Pull request overview
Hardens API-cost accounting and webhook delivery with transactional usage tracking, durable retries, and Gemini usage metadata.
Changes:
- Adds atomic outbox claims, retry scheduling, recovery, and idempotency headers.
- Tracks Gemini usage metadata and expands pricing entries.
- Adds lifecycle, concurrency, migration, and cost-accounting tests.
Reviewed changes
Copilot reviewed 11 out of 11 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
src/youtube_extension/backend/services/api_cost_monitor.py |
Implements transactional accounting and durable outbox processing. |
src/youtube_extension/services/ai/gemini_service.py |
Preserves provider usage metadata. |
src/youtube_extension/services/ai/hybrid_processor_service.py |
Records Gemini usage and isolates telemetry failures. |
tests/unit/test_api_cost_outbox_worker.py |
Tests outbox durability, concurrency, retries, and lifecycle. |
tests/unit/test_api_cost_monitor.py |
Tests pricing and atomic alert staging. |
tests/unit/test_api_cost_database_substrate.py |
Updates transaction-boundary expectations. |
tests/unit/test_hybrid_processor_service.py |
Tests Gemini usage recording. |
tests/unit/test_gemini_service.py |
Tests metadata preservation. |
tests/unit/test_gh_aw_workflow_governance.py |
Updates the coverage-floor assertion. |
pyproject.toml |
Adjusts the measured coverage baseline. |
docs/reports/PR869_OUTBOX_PROOFS.md |
Records historical verification evidence and remaining gates. |
| }, | ||
| "google": { | ||
| # Current generation (routable from GeminiService) | ||
| "gemini-3.5-flash": {"input": 0.0001, "output": 0.0004}, |
Comment on lines
+321
to
+322
| input_tokens = int(getattr(usage, "prompt_token_count", 0) or 0) | ||
| output_tokens = int(getattr(usage, "candidates_token_count", 0) or 0) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Canonical issue
Closes #
Outcome
Describe the user or operational result this PR produces.
Scope
Risk
Verification
List exact automated and manual checks, tied to the current head SHA.
Production evidence
Provide the Vercel preview, production deployment, runtime evidence, or state why production evidence is not applicable.
Agent handoff
Agent provenance
Human-authored pull requests may delete this section. Agent-authored pull requests must replace agent-lock-example with agent-lock-manifest and fill the values. Scope and test paths remain authoritative in the linked issue.
The declared agent publishes a result comment on the linked issue or PR with the exact run ID and current 40-character head SHA. Replace
agent-lock-event-examplewithagent-lock-eventonly when publishing real evidence.