Skip to content

fix: route embeddings through JSON parser so cost and tokens are captured - #133

Open
kyto-agent wants to merge 1 commit into
hackclub:mainfrom
kyto-agent:fix/embedding-cost-display
Open

fix: route embeddings through JSON parser so cost and tokens are captured#133
kyto-agent wants to merge 1 commit into
hackclub:mainfrom
kyto-agent:fix/embedding-cost-display

Conversation

@kyto-agent

Copy link
Copy Markdown

Problem

Embedding model requests were being forced through the SSE streaming parser even though the /v1/embeddings endpoint returns plain JSON (not Server-Sent Events). This caused resolveUsage() to never receive actual data, so every embedding request was logged with:

  • cost: "0" → dashboard shows "Free"
  • promptTokens: 0, totalTokens: 0 → dashboard shows "0 in 0 out"
  • Embedding models filtered out of the global Usage by Model table (due to the totalTokens > 0 HAVING clause)

Additionally, estimateUpstreamCost() only called fetchLanguageModels(), so embedding models were never found and every embedding call fell back to a flat $0.05 reservation regardless of actual input size.

Fix

1. Route embeddings through the JSON branch (general.ts)

Changed the condition from:

if (!body.stream && endpoint !== "embeddings")

to:

if (!body.stream || endpoint === "embeddings")

Embeddings always return plain JSON, so they need the non-streaming branch where resolveUsage(data) can extract usage.cost and usage.total_tokens from the response.

2. Use the right model fetcher for cost estimation (shared.ts)

estimateUpstreamCost() now accepts an endpoint parameter and calls fetchEmbeddingModels() for embedding requests. For embeddings, only input tokens are estimated (no completion side). This gives an accurate reservation instead of the flat $0.05 fallback.

3. Include zero-token models in stats when they have cost (stats.ts)

Added OR COALESCE(SUM(cost), 0) > 0 to the HAVING clause so models with non-zero cost but potentially zero tokens (some embedding/image edge cases) still appear in "Usage by Model".

Files changed

  • src/routes/proxy/v1/general.ts — routing condition + pass endpoint to cost estimator
  • src/routes/proxy/shared.ts — import fetchEmbeddingModels, branch cost estimation by endpoint
  • src/lib/stats.ts — relax HAVING clause to include costly zero-token models

…ured

Three changes:

1. general.ts: Change the non-streaming condition from
   `!body.stream && endpoint !== 'embeddings'` to
   `!body.stream || endpoint === 'embeddings'`. Embeddings return plain
   JSON (not SSE), so they need the JSON branch where resolveUsage()
   extracts usage.cost from the response. Previously the SSE parser
   found no data: lines and logged cost=0 / tokens=0.

2. shared.ts: estimateUpstreamCost() now accepts an endpoint parameter
   and calls fetchEmbeddingModels() for embedding requests. Previously it
   only queried fetchLanguageModels(), so embedding models were never
   found and every call fell back to a flat $0.05 reservation.

3. stats.ts: Add OR cost > 0 to the model stats HAVING clause so models
   with non-zero cost but zero tokens still appear in Usage by Model.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant