sdk: launch QA — eight more defects found against the live gateway - #5
Merged
Conversation
Follow-on to the error-vocabulary fix (#4), from the same QA pass. Every defect below was found by calling api.conifer.build rather than by reading our own tests, and every one was invisible to the offline suites — their fixtures encoded the same wrong assumptions the code did. SILENT FAILURES · `requestId` was inert. The gateway derives its request id from the first of `idempotency-key` then `x-request-id`, and the SDK always sends the former, so a caller's trace id was NEVER consulted. · `chat({defer:true})` returned an empty completion for a turn that had been accepted AND DEBITED — at the call site, indistinguishable from a model that answered with nothing. · Transient 409s were treated as terminal, then (once retried) given 0.75s to resolve a cross-replica convergence. · `key_spend_cap_exceeded` had no class, though it is a THIRD distinct 402 whose remedy is unique: the account may be fully funded and every other key working, so adding credit does nothing. ALL THREE MIGRATION SHIMS DROPPED CONSTRAINTS Checked against the vendors' published schemas instead of our own card — the comparison the existing tests structurally could not make. · OpenRouter dropped five request fields, plus `X-OpenRouter-Categories`. · Vercel dropped every `providerOptions.gateway` key but two — including `zdr` and `dataCollection`, which are PRIVACY constraints. · Helicone dropped `Helicone-Omit-Request`/`-Response` (same failure), `Helicone-Auth` and `Helicone-Retry-Enabled`. This violated the portability card's first law inside the very feature written to uphold it. A dropped `zdr` means the request succeeds, nothing errors, and a promise the caller made to THEIR OWN users has quietly lapsed. All three now fail closed, with a catch-all so a control the vendor ships tomorrow refuses rather than vanishing, and an anti-drift gate driven from the VENDOR's field list in both languages. A FRESH PYTHON VENV COULD NOT MAKE ITS FIRST CALL python.org installs have an empty CA trust store and no certifi to fall back on. Added a `tls` extra, kept optional so zero runtime dependencies stays true. SURFACE ADDED · Embeddings — base64 on the wire (~3x less network), decoded transparently. Applied silently only because it was MEASURED lossless. An unrecognized encoding decodes to an EMPTY vector rather than a guess: a wrong vector sails through a cosine similarity and returns nonsense rankings forever. · Deferred jobs — defer(), jobs.status/result/cancel, and a wait() that stops on terminal states, backs off, and never cancels paid work on a timeout. · ReceiptCollector / SpendBudget — receipts for the client you ALREADY use. Our differentiator rides response headers, which every mainstream client discards, so following our own advice made it invisible. Reads headers only, so the body stays the caller's and streaming is untouched. Verified against the real `openai` and `anthropic` packages. · emptyReason — a reasoning model spends maxTokens on its thinking block FIRST, so a tight budget returns "" plus a bill. That string looks like a refusal, a filter, or a broken SDK; now it explains itself. Also settled, with evidence: /v1/responses and /v1/messages need no code here. The vendor SDKs speak them correctly and all three doors carry identical receipt headers, so ReceiptCollector already covers them. A live QA harness (scripts/live-qa.mjs, python/scripts/live_qa.py) now exercises every surface against production in both languages. It found two of the bugs above. TS 73 -> 145 tests Python 35 -> 99 tests live 18/18 TS, 17/17 Python
Found while inspecting the built artifacts rather than trusting the build's exit code: `readme = "README.md"` in python/pyproject.toml named a file that does not exist in python/, and setuptools resolved it to NOTHING without a warning. The build succeeded; the wheel's METADATA carried `Description-Content-Type: text/markdown` and an EMPTY body — a blank PyPI project page. On launch day the first thing most Python users would have seen of this SDK is nothing at all. Fixed with a symlink to the repo README (mode 120000, so a clone gets the link). Not a copy — two versions of a 400-line document drift, and the stale one is always the one a user reads on PyPI. Not `../README.md` either: setuptools refuses to read outside the package root and fails the build loudly, which is how that route was ruled out. The description now ships at ~24000 bytes instead of 0. Added RELEASING.md: the full pre-publish gate (offline in both languages, both CI routes, the live QA harness, artifact inspection, and a fresh-install consumer check), the `--access public` trap that publishes a scoped npm package private by default, and the reminder to update the README's "not on a registry yet" note. Three new packaging tests, each pinning something that fails SILENTLY: the Python package ships the README it declares and it has not drifted; the two manifests carry the same version; and the README cannot both claim "not on npm" and show a registry install. Neither name is taken yet (@conifer/sdk, the @conifer scope, and conifer-sdk all 404), and publishing stays a human action — it is public and irreversible per version.
OpenRouter puts the settled cost in the response BODY as `usage.cost`. Conifer
discloses it on `x-conifer-cost-nanousd`, a response HEADER. That difference is
invisible until it isn't: every logging pipeline, request recorder,
LangChain/LiteLLM callback and JSON-dumping debug line keeps the body and
discards the headers. A team migrating from OpenRouter loses their cost column,
and the fix — reach for a header — is somewhere they are not looking.
It matters more on Conifer than elsewhere: a normal caller CANNOT read their
usage history back (`/admin/usage/*` is owner-only), so the receipt attached to
the turn is their only record of what they spent.
`usage` now carries `cost` (decimal USD, the field OpenRouter uses) and
`cost_nanousd` (the exact integer billed). Verified live in both languages: the
body figure equals the header figure.
Two invariants, both tested:
· ADDITIVE ONLY — a `cost` the gateway sends itself always wins.
· ABSENT STAYS ABSENT — a stream's head carries no cost, so nothing is added;
a 0 would tell every dashboard the turn was free. Checked against a real
stream, not a fixture.
The receipt remains the authority; the float is a compatibility convenience.
TS 152 pass, Python 103 pass, live 19/19 TS and 18/18 Python
The MCP server predates `emptyReason` and never learned it. Measured live through the real stdio server: `conifer_complete` against a reasoning model with `max_tokens: 16` returns an empty string, a real bill, and no reason. An agent receiving that will usually RETRY, spending the caller's money a second time on a turn that fails identically. The cause — the thinking block consuming the budget before the visible answer starts — is a budget problem the agent could fix, but only if something tells it. `conifer_complete` and `conifer_compare` now carry `empty_reason`. Absent when there is text, and absent for a tool call where empty text IS the answer. The cost is still reported alongside: the turn was billed, and hiding that would be worse than the silence being fixed. On `conifer_compare` it matters twice, since that tool sorts cheapest-first — a model returning nothing would otherwise read as the cheap winner. TS 154 pass, Python 103, live 19/19
`cards/sdk.output.card.json` names three consumers — the MCP server, the Python twin, the website docs. That is a stated relationship, and NOTHING checked it. The only enforcement was somebody remembering to look, and all three had drifted, each at a real cost: docs telling migrating users to keep embeddings on a competitor (a door served since 2026-08-26), a documented Python install that dies on the first call, an MCP server handing agents an empty answer and a bill with no reason (so the agent retries and pays twice), and Python missing `attribution_from_openrouter` entirely. tests/consumers.test.ts now checks every TS helper has a Python twin, every error class exists both sides, the MCP spending tools report cost AND explain an empty answer, the README covers the surface, and RELEASING.md names the traps. Verified by re-introducing two of the real bugs; each fails with the specific name and the remedy. The parity check encodes the naming conventions rather than assuming plain snake_case, because three differ ON PURPOSE: proper nouns stay whole (`from_openrouter`), units follow the language (`backoffMs` vs `backoff_seconds` — TS matches setTimeout, Python matches time.sleep), and a free function may be a property (`textOf()` vs `.text`). Also fixed a two-route difference this exposed: tests that GREP a source file must resolve from the repo root, since on the compile route the suite runs from `.test-build/tests/` where `src/` and `mcp/` are compiled `.js`. It passed on one route and ENOENT'd on the other. TS 160 pass, both routes; Python 103; live 19/19
Found by auditing the SDK against `public/llms.txt` — the file AI agents read to learn Conifer. It tells agents to size a vector column from the catalog's `embedding_dimensions` "before spending a token". Good advice, and the SDK was the one place that field was reachable only through the untyped `raw` hatch. A pgvector column is declared `vector(1536)` BEFORE the first call; getting it wrong means a migration on a populated table. So the field is now typed in both languages, and absent on a chat seat — a different statement from a width of 0, which would size a column to nothing. Verified live across all seven embedding seats, and the live QA harness now asserts the identity that matters: the width the catalog ADVERTISES is the width the vector comes back at. If those diverged, the failure would land in someone's database schema rather than their code. TS 162 pass, both routes; Python 105; live 20/20
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-on to #4, from the same QA pass against
api.conifer.build. Every defect below was found by calling the gateway rather than by reading our own tests — and every one was invisible to the offline suites, because their fixtures encoded the same wrong assumptions the code did.Silent failures
requestIdwas inert. The gateway derives its request id from the first ofidempotency-keythenx-request-id. The SDK always sends the former, so a caller's trace id was never once consulted — no way to correlate a support question with their own logs.chat({defer:true})returned an empty completion for a turn that had been accepted and debited. At the call site, indistinguishable from a model that answered with nothing.key_spend_cap_exceededhad no class, though it is a third distinct 402 with a unique remedy: the account may be fully funded and every other key working, so adding credit does nothing.All three migration shims dropped constraints
Checked against the vendors' published schemas instead of our own card — the comparison the existing portability tests were structurally unable to make.
X-OpenRouter-CategoriesproviderOptions.gatewaykey but two — includingzdranddataCollectionHelicone-Omit-Request/-Response,Helicone-Auth,Helicone-Retry-EnabledThis violated the portability card's first law inside the feature written to uphold it. The privacy ones are the worst case: a dropped
zdrmeans the request succeeds, nothing errors, and a promise the caller made to their own users has quietly lapsed.All three now fail closed, each with a catch-all so a control the vendor ships tomorrow refuses rather than vanishing. Both suites drive an anti-drift gate from the vendor's field list; I verified it by re-introducing the bug in each language.
A fresh Python venv could not make its first call
python.org installs have an empty CA trust store and no
certifito fall back on. Added atlsextra, kept optional so zero runtime dependencies stays true.Surface added
defer(),jobs.status/result/cancel, and await()that stops on terminal states, backs off, and never cancels paid work on a timeout.ReceiptCollector/SpendBudget— receipts for the client you already use. Our differentiator rides response headers, which every mainstream client discards, so following our own advice made it invisible. Reads headers only, so the body stays the caller's and streaming is untouched. Verified against the realopenaiandanthropicpackages.emptyReason— a reasoning model spendsmaxTokenson its thinking block first, so a tight budget returns""plus a bill. That string looks like a refusal, a filter, or a broken SDK; now it explains itself.Settled with evidence
/v1/responsesand/v1/messagesneed no code here. The vendor SDKs speak them correctly and all three doors carry identical receipt headers, soReceiptCollectoralready covers them — reimplementing someone else's wire would be a third implementation to keep in sync.Verification
A live QA harness (
scripts/live-qa.mjs,python/scripts/live_qa.py) exercises every surface against production in both languages. It found two of the bugs above. Also ran the packed tarball and the Python package into clean projects and used them as a consumer does — which is how the venv and 409-backoff bugs surfaced.