fix: route embeddings through JSON parser so cost and tokens are captured - #133
Open
kyto-agent wants to merge 1 commit into
Open
fix: route embeddings through JSON parser so cost and tokens are captured#133kyto-agent wants to merge 1 commit into
kyto-agent wants to merge 1 commit into
Conversation
…ured Three changes: 1. general.ts: Change the non-streaming condition from `!body.stream && endpoint !== 'embeddings'` to `!body.stream || endpoint === 'embeddings'`. Embeddings return plain JSON (not SSE), so they need the JSON branch where resolveUsage() extracts usage.cost from the response. Previously the SSE parser found no data: lines and logged cost=0 / tokens=0. 2. shared.ts: estimateUpstreamCost() now accepts an endpoint parameter and calls fetchEmbeddingModels() for embedding requests. Previously it only queried fetchLanguageModels(), so embedding models were never found and every call fell back to a flat $0.05 reservation. 3. stats.ts: Add OR cost > 0 to the model stats HAVING clause so models with non-zero cost but zero tokens still appear in Usage by Model.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Embedding model requests were being forced through the SSE streaming parser even though the
/v1/embeddingsendpoint returns plain JSON (not Server-Sent Events). This causedresolveUsage()to never receive actual data, so every embedding request was logged with:cost: "0"→ dashboard shows "Free"promptTokens: 0,totalTokens: 0→ dashboard shows "0 in 0 out"totalTokens > 0HAVING clause)Additionally,
estimateUpstreamCost()only calledfetchLanguageModels(), so embedding models were never found and every embedding call fell back to a flat $0.05 reservation regardless of actual input size.Fix
1. Route embeddings through the JSON branch (
general.ts)Changed the condition from:
to:
Embeddings always return plain JSON, so they need the non-streaming branch where
resolveUsage(data)can extractusage.costandusage.total_tokensfrom the response.2. Use the right model fetcher for cost estimation (
shared.ts)estimateUpstreamCost()now accepts anendpointparameter and callsfetchEmbeddingModels()for embedding requests. For embeddings, only input tokens are estimated (no completion side). This gives an accurate reservation instead of the flat $0.05 fallback.3. Include zero-token models in stats when they have cost (
stats.ts)Added
OR COALESCE(SUM(cost), 0) > 0to the HAVING clause so models with non-zero cost but potentially zero tokens (some embedding/image edge cases) still appear in "Usage by Model".Files changed
src/routes/proxy/v1/general.ts— routing condition + pass endpoint to cost estimatorsrc/routes/proxy/shared.ts— importfetchEmbeddingModels, branch cost estimation by endpointsrc/lib/stats.ts— relax HAVING clause to include costly zero-token models