diff --git a/CHANGELOG.md b/CHANGELOG.md index 083d632d3..1835330a6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,32 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +## [0.7.1] - 2026-05-06 + +A focused pass on provider error handling, surfaced by a 5-persona pre-release review. + +### Fixed + +- **Provider 4xx errors now show the inner error message instead of a raw JSON dump.** When any provider returned the standard `{error: {message, type, code}}` shape (OpenAI, Azure OpenAI, OpenRouter, etc.), `parseAPICallError`'s extraction chain short-circuited on the truthy parent `error` object, the `typeof errMsg === "string"` guard rejected it, and the parser fell through to dumping the raw response body — which appeared as `APIError: Bad Request: {?:?}` after telemetry redaction collapsed string values to `?`. Telemetry caught users retrying broken model selections 3+ times in the same session because the surfaced error gave no clue about the cause. Users now see actionable text such as `APIError: Bad Request: The model 'gpt-5-codex' does not exist or you do not have access to it.` The OR-chain is replaced with explicit-typeof ternaries that mirror `parseStreamError`'s pattern, so a truthy non-string at any tier cannot block a valid string further down the chain. (#789, closes #788) +- **Bedrock / AWS Lambda `errorMessage` shape is now extracted.** AWS APIs that return `{errorMessage: "...", errorType: "..."}` (Lambda style) previously fell through the OpenAI/Anthropic-shaped chain to a raw-body dump. Added `body.errorMessage` to the extraction ladder in both `parseAPICallError` and `parseStreamError`. +- **Streaming error path no longer dumps `Unknown: {"type":"error",...}` for non-OpenAI codes.** `parseStreamError` previously handled only 4 OpenAI error codes (`context_length_exceeded`, `insufficient_quota`, `usage_not_included`, `invalid_prompt`); everything else fell through to `JSON.stringify(e)`. Added a default fallback that runs the same string-typeof chain as `parseAPICallError`, so any extractable provider message becomes a clean api_error. +- **`model_not_found` no longer triggers a silent retry storm.** OpenAI 404s are forced retryable in general (some legitimate models 404 transiently), but `error.code === "model_not_found"` now short-circuits to `isRetryable: false` — the user sees the actionable error on attempt 1 instead of after 5 silent retries. + +### Added + +- **`altimate models` discoverability hint on model-not-found errors.** When `error.code === "model_not_found"`, the surfaced message now ends with `Run \`altimate models\` to see available models.` so the next step is one command away. +- **Provider-API-Errors troubleshooting reference** at `docs/docs/reference/troubleshooting.md` covering model-not-found, unauthorized, rate-limited, context-overflow, and HTML-page error classes. + +### Privacy + +- **`Telemetry.maskString` now redacts email addresses and internal hostnames.** Pre-fix, the JSON-quote masking rule incidentally collapsed everything inside provider error JSON to `?`. The provider-error fix unwraps that JSON, which means provider-side identifiers (caller emails, internal `*.local` / `*.internal` / RFC1918 / IPv6 loopback / ULA / link-local / AWS IMDS endpoints) now flow as plain English. Added explicit redaction patterns so they're masked before reaching telemetry, the share backend, or local session storage. The masker is kept in sync with `parseAPICallError`'s `maskInternalHost` (same internal-endpoint coverage); query-string and fragment characters (`+`, `#`, `,`, `;`) are inside the trailing char class so secrets past the `` marker don't survive. `sk-…` and `Bearer …` token redaction is unchanged. +- **`metadata.url` on `MessageV2.APIError` masks internal hosts and strips basic-auth userinfo.** When `error.url` points at `localhost`, `*.local`, `*.internal`, an RFC1918 IPv4, IPv6 loopback / ULA / link-local, or the AWS IMDS address (`169.254.169.254`), the host is rewritten to `internal-host.redacted` before the URL lands on the parsed error. Basic-auth userinfo (`user:pass@…`) is stripped on **every** URL — internal or public — since a credential in a public-host URL is at least as risky as one in an internal proxy. Public-host URLs are otherwise preserved verbatim for debugging. +- **`responseBody` is capped at 4KB** at the `parseAPICallError` boundary. Without this, a hostile or verbose gateway could persist a 100KB+ body into local storage and (for shared sessions) the share backend. + +### Testing + +- 46 adversarial tests covering JSON-scalar bodies, prototype-pollution attempts, 100KB error messages, malformed JSON, every-tier null/numeric extraction, Bedrock `errorMessage` precedence, the `parseStreamError` fallback for unknown codes, the `model_not_found` retry-storm carve-out, the `altimate models` hint, the responseBody cap, the metadata.url internal-host masking (incl. IPv6 loopback/ULA/link-local, AWS IMDS, public-host basic-auth userinfo strip, RFC1918 boundary checks, lookalike-hostname guards), and the new email / internal-host `maskString` patterns (incl. IMDS, IPv6, and query-fragment leak guards). + ## [0.7.0] - 2026-05-03 ### Changed diff --git a/docs/docs/reference/troubleshooting.md b/docs/docs/reference/troubleshooting.md index ec02ad718..052e76f9d 100644 --- a/docs/docs/reference/troubleshooting.md +++ b/docs/docs/reference/troubleshooting.md @@ -30,6 +30,30 @@ altimate --print-logs --log-level DEBUG 3. If behind a proxy, set `HTTPS_PROXY` (see [Network](network.md)) 4. Try a different provider to isolate the issue +### Provider API Errors + +**Symptoms:** `APIError: : ` shown in chat output. Common forms: + +- `APIError: Bad Request: The model 'foo' does not exist or you do not have access to it.` +- `APIError: Unauthorized: Invalid API key` +- `APIError: Rate limit exceeded` + +As of v0.7.1, altimate-code surfaces the **inner provider message** instead of dumping the raw JSON body. The status prefix (`Bad Request:`, `Unauthorized:`, etc.) comes from the provider's HTTP status code; everything after the colon is the provider's text verbatim. + +**Solutions by error class:** + +1. **Model not found** (`APIError: Bad Request: The model '' does not exist...`) — list the models your provider currently exposes and re-run with one of them: + ```bash + altimate models + ``` + `model_not_found` errors no longer auto-retry; the message you see is the first attempt, not the fifth. +2. **Unauthorized / 401** — re-run `altimate auth login ` and re-issue the request. +3. **Rate limited / 429** — altimate-code automatically retries on rate-limit responses (including plain-text 429s from Alibaba/DashScope). If you keep hitting rate limits, lower `parallel_tool_calls` or switch to a less-saturated model. +4. **Context overflow** — switch to a larger-context model or trim earlier turns with `/compact`. Detection covers Anthropic, Bedrock, OpenAI, Gemini, xAI, Groq, OpenRouter, DeepSeek, Copilot, llama.cpp, LM Studio, MiniMax, Kimi, Moonshot, Azure OpenAI, and HTTP 413. +5. **HTML page returned** — usually a gateway/proxy error. The CLI returns a friendly hint pointing at `altimate auth login` rather than dumping the raw HTML. + +**Privacy note:** error messages flow through the same redaction layer as everything else (`sk-…`, `Bearer …`, email addresses, and `*.local` / `*.internal` / RFC1918 / IPv6 loopback / ULA / link-local / AWS IMDS hostnames are masked before reaching telemetry). Internal-host URLs in `metadata.url` are also redacted before they reach local storage or shared sessions, and basic-auth userinfo (`user:pass@…`) is stripped from every URL regardless of whether the host is internal. + ### Tool Execution Errors **Symptoms:** "No native handler" or tool execution failures for data engineering tools. diff --git a/packages/opencode/src/altimate/telemetry/index.ts b/packages/opencode/src/altimate/telemetry/index.ts index de104aed9..b13837597 100644 --- a/packages/opencode/src/altimate/telemetry/index.ts +++ b/packages/opencode/src/altimate/telemetry/index.ts @@ -1060,6 +1060,26 @@ export namespace Telemetry { return s .replace(/sk-(?:ant-)?[A-Za-z0-9_-]{20,}/g, "sk-***") .replace(/Bearer\s+[A-Za-z0-9._-]{20,}/gi, "Bearer ***") + // Email addresses — providers occasionally echo caller identity in error text. + .replace(/[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}/g, "") + // Internal hostnames in URLs — keeps parity with `parseAPICallError`'s + // `maskInternalHost` so an error message containing the same URL doesn't + // leak through telemetry while metadata.url is masked. Covers: + // *.local / *.internal / *.localhost + // RFC1918 IPv4: 10/8, 172.16/12, 192.168/16, plus 127/8 loopback + // AWS IMDS / link-local IPv4: 169.254/16 + // IPv6 in brackets: [::1] loopback, [fc??::/[fd??:: ULA, [fe80:: link-local + // Char class includes `+`, `#`, `,`, `;` so secrets in query/fragment + // don't survive past the redaction marker. Over-masking is the correct + // failure mode here. + .replace( + // `(?:[^\/\s@]+@)?` allows optional basic-auth userinfo + // (`user:pass@`) before the host so URLs like + // `https://admin:hunter2@10.0.0.5/x` are still recognized as internal + // and redacted whole. The credential goes with the host into . + /\bhttps?:\/\/(?:[^\/\s@]+@)?(?:localhost|127\.\d+\.\d+\.\d+|10\.\d+\.\d+\.\d+|192\.168\.\d+\.\d+|172\.(?:1[6-9]|2\d|3[01])\.\d+\.\d+|169\.254\.\d+\.\d+|0\.0\.0\.0|\[(?:::1|fc[0-9a-f]{2}:[^\]]*|fd[0-9a-f]{2}:[^\]]*|fe80:[^\]]*)\]|[A-Za-z0-9.-]+\.(?:local|internal|localhost))(?::\d+)?[\w/.?=&%+#,;~!*'()@:-]*/gi, + "", + ) .replace(/'(?:[^'\\]|\\.)*'/g, "?") .replace(/"(?:[^"\\]|\\.)*"/g, "?") .replace(/\s+/g, " ") diff --git a/packages/opencode/src/provider/error.ts b/packages/opencode/src/provider/error.ts index 691de93bd..c30251702 100644 --- a/packages/opencode/src/provider/error.ts +++ b/packages/opencode/src/provider/error.ts @@ -28,6 +28,21 @@ export namespace ProviderError { function isOpenAiErrorRetryable(e: APICallError) { const status = e.statusCode if (!status) return e.isRetryable + // altimate_change start — upstream_fix: don't retry-storm on model_not_found. + // OpenAI 404s are forced retryable below because some legitimate models 404 + // transiently, but `model_not_found` will never recover; retrying 5x just + // delays the user seeing the (now-readable) error message. + if (status === 404) { + try { + const body = e.responseBody ? JSON.parse(e.responseBody) : null + if (body?.error?.code === "model_not_found") return false + } catch { + // Malformed JSON on a 404 falls through to "force retryable" below — + // intentional; some providers emit non-JSON 404 bodies for transient + // model availability blips and those should still retry. + } + } + // altimate_change end // openai sometimes returns 404 for models that are actually available return status === 404 || e.isRetryable } @@ -61,19 +76,27 @@ export namespace ProviderError { try { const body = JSON.parse(e.responseBody) - // altimate_change start — upstream_fix: OpenAI errors use {error: {message}} shape; - // the original `body.message || body.error || body.error?.message` short-circuits on - // the parent object, fails the typeof string guard, and dumps the raw body. Use an - // explicit-typeof ternary so a truthy non-string at any level can't block a valid - // string further down the chain (matches parseStreamError's pattern below). + // altimate_change start — upstream_fix: extract provider error messages + // across the four shapes in the wild: + // 1. {error: {message: "..."}} — OpenAI / Azure OpenAI / OpenRouter + // 2. {message: "..."} — Anthropic-style top-level + // 3. {errorMessage: "..."} — Bedrock / AWS Lambda + // 4. {error: "..."} — legacy plain-string shape + // The original `body.message || body.error || body.error?.message` short- + // circuited on a truthy parent object, failed the `typeof === "string"` + // guard, and dumped the raw body. Use an explicit-typeof ternary so a + // truthy non-string at any tier can't block a valid string further down + // the chain (matches parseStreamError's pattern below). const errMsg = typeof body.error?.message === "string" ? body.error.message : typeof body.message === "string" ? body.message - : typeof body.error === "string" - ? body.error - : undefined + : typeof body.errorMessage === "string" + ? body.errorMessage + : typeof body.error === "string" + ? body.error + : undefined if (errMsg) return `${msg}: ${errMsg}` // altimate_change end } catch {} @@ -161,6 +184,32 @@ export namespace ProviderError { responseBody, } } + + // altimate_change start — upstream_fix: extend extraction to non-OpenAI error + // codes. The switch above only handles 4 OpenAI shapes; everything else fell + // through to `JSON.stringify(e)` in the caller (session/message-v2.ts), which + // showed users `Unknown: {"type":"error",...}`. Apply the same string-typeof + // chain we use in parseAPICallError so any extractable provider message lands + // as a clean api_error. + const fallbackMsg = + typeof body?.error?.message === "string" + ? body.error.message + : typeof body?.message === "string" + ? body.message + : typeof body?.errorMessage === "string" + ? body.errorMessage + : typeof body?.error === "string" + ? body.error + : undefined + if (fallbackMsg) { + return { + type: "api_error", + message: fallbackMsg, + isRetryable: false, + responseBody, + } + } + // altimate_change end } export type ParsedAPICallError = @@ -179,6 +228,67 @@ export namespace ProviderError { metadata?: Record } + // altimate_change start — cap responseBody at 4KB before it lands on a + // MessageV2.APIError. Without this cap, a hostile gateway returning a 100KB + // body (or just verbose providers like LiteLLM) would inflate local storage, + // share-backend uploads, and diagnostic dumps. + const RESPONSE_BODY_CAP = 4096 + function capResponseBody(body: string | undefined): string | undefined { + if (!body) return body + if (body.length <= RESPONSE_BODY_CAP) return body + return body.slice(0, RESPONSE_BODY_CAP) + `…[truncated ${body.length - RESPONSE_BODY_CAP} chars]` + } + // altimate_change end + + // altimate_change start — sanitize metadata.url before it lands on the + // parsed error. Two transforms are applied: + // (1) basic-auth userinfo (`user:pass@…`) is stripped on every URL, + // internal or public — a credential in a misconfigured proxy URL + // must not flow into telemetry / local storage / share regardless + // of where the URL points. + // (2) the hostname is rewritten to `internal-host.redacted` if it + // matches an internal endpoint (RFC1918, *.local, *.internal, + // localhost, *.localhost, IPv6 loopback / ULA / link-local, or + // the AWS IMDS address 169.254.169.254). Public provider URLs + // are otherwise preserved for debugging. + function maskInternalHost(url: string): string { + try { + const u = new URL(url) + // u.hostname keeps IPv6 brackets (e.g. "[::1]"); strip for regex match. + const host = u.hostname.replace(/^\[|\]$/g, "") + const hadCredentials = u.username !== "" || u.password !== "" + // Always clear userinfo — the credential is the riskier part of the URL. + u.username = "" + u.password = "" + const isInternal = + host === "localhost" || + host === "0.0.0.0" || // any-interface bind, often misconfigured proxy + host.endsWith(".local") || + host.endsWith(".internal") || + host.endsWith(".localhost") || + /^127\./.test(host) || + /^10\./.test(host) || + /^192\.168\./.test(host) || + /^172\.(1[6-9]|2\d|3[01])\./.test(host) || + /^169\.254\./.test(host) || // AWS IMDS / link-local IPv4 + host === "::1" || // IPv6 loopback + /^fc[0-9a-f]{2}:/i.test(host) || // IPv6 ULA (RFC4193 fc00::/8) + /^fd[0-9a-f]{2}:/i.test(host) || // IPv6 ULA (RFC4193 fd00::/8) + /^fe80:/i.test(host) // IPv6 link-local + if (isInternal) { + u.hostname = "internal-host.redacted" + return u.toString() + } + // No host change but we may have removed credentials — re-serialize + // only if userinfo was present, otherwise return the original string + // so URLs round-trip untouched (preserves trailing slashes, casing). + return hadCredentials ? u.toString() : url + } catch { + return url + } + } + // altimate_change end + export function parseAPICallError(input: { providerID: ProviderID; error: APICallError }): ParsedAPICallError { const m = message(input.providerID, input.error) // Check responseBody for context_length_exceeded code (e.g., OpenAI-style errors) @@ -188,20 +298,38 @@ export namespace ProviderError { return { type: "context_overflow", message: m, - responseBody: input.error.responseBody, + // altimate_change start — cap responseBody on context_overflow path + responseBody: capResponseBody(input.error.responseBody), + // altimate_change end } } - const metadata = input.error.url ? { url: input.error.url } : undefined + // altimate_change start — append a `models` discoverability hint when the + // error code is model_not_found. Pairs with the retry-storm carve-out in + // isOpenAiErrorRetryable so the user sees the hint on the first attempt + // instead of after 5 silent retries. + let finalMessage = m + if (codeFromBody === "model_not_found") { + finalMessage = `${m} Run \`altimate models\` to see available models.` + } + // altimate_change end + + // altimate_change start — mask internal hostnames in metadata.url + const metadata = input.error.url ? { url: maskInternalHost(input.error.url) } : undefined + // altimate_change end return { type: "api_error", - message: m, + // altimate_change start — finalMessage carries the optional /models hint + message: finalMessage, + // altimate_change end statusCode: input.error.statusCode, isRetryable: input.providerID.startsWith("openai") ? isOpenAiErrorRetryable(input.error) : input.error.isRetryable, responseHeaders: input.error.responseHeaders, - responseBody: input.error.responseBody, + // altimate_change start — cap responseBody on api_error path + responseBody: capResponseBody(input.error.responseBody), + // altimate_change end metadata, } } diff --git a/packages/opencode/test/altimate/tracing-display-crash.test.ts b/packages/opencode/test/altimate/tracing-display-crash.test.ts index db22573c1..0eda918b3 100644 --- a/packages/opencode/test/altimate/tracing-display-crash.test.ts +++ b/packages/opencode/test/altimate/tracing-display-crash.test.ts @@ -169,7 +169,9 @@ describe("flushSync — crash recovery", () => { model: "anthropic/claude-sonnet-4-20250514", agent: "builder", }) - await new Promise((r) => setTimeout(r, 50)) + // Deterministic wait for the startTrace snapshot — `await sleep(50)` + // races on slow CI runners (this test failed on CI run 25448250105). + await tracer.flush() tracer.logStepStart({ id: "1" }) tracer.logToolCall({ @@ -181,8 +183,8 @@ describe("flushSync — crash recovery", () => { id: "1", reason: "tool_calls", cost: 0.005, tokens: { input: 1000, output: 200, reasoning: 50, cache: { read: 100, write: 25 } }, }) - // Wait for logStepFinish snapshot - await new Promise((r) => setTimeout(r, 50)) + // Deterministic wait for the logStepFinish snapshot. + await tracer.flush() tracer.logStepStart({ id: "2" }) // Crash mid-generation diff --git a/packages/opencode/test/provider/error.test.ts b/packages/opencode/test/provider/error.test.ts index e5387b06a..4d887c6d6 100644 --- a/packages/opencode/test/provider/error.test.ts +++ b/packages/opencode/test/provider/error.test.ts @@ -76,11 +76,29 @@ describe("ProviderError.parseStreamError: SSE error classification", () => { expect(ProviderError.parseStreamError({ type: "content", text: "hello" })).toBeUndefined() }) - test("returns undefined for unknown error codes", () => { + test("falls back to api_error with extracted message for unknown error codes (v0.7.1+)", () => { + // Behavior change in v0.7.1: previously returned undefined, which caused the + // caller to fall through to JSON.stringify(e). Now extracts the message via + // the same string-typeof chain used in parseAPICallError so users see a + // clean api_error instead of `Unknown: {"type":"error",...}`. + const result = ProviderError.parseStreamError({ + type: "error", + error: { code: "unknown_code", message: "weird" }, + }) + expect(result?.type).toBe("api_error") + if (result && result.type === "api_error") { + expect(result.message).toBe("weird") + expect(result.isRetryable).toBe(false) + } + }) + + test("returns undefined when no extractable message exists for unknown code", () => { + // Last-resort behavior: extractor finds no string anywhere — caller falls + // back to JSON.stringify(e), which is at least visible if not friendly. expect( ProviderError.parseStreamError({ type: "error", - error: { code: "unknown_code", message: "weird" }, + error: { code: "unknown_code" }, }), ).toBeUndefined() }) diff --git a/packages/opencode/test/skill/release-v0.7.1-adversarial.test.ts b/packages/opencode/test/skill/release-v0.7.1-adversarial.test.ts new file mode 100644 index 000000000..dcebfb9b4 --- /dev/null +++ b/packages/opencode/test/skill/release-v0.7.1-adversarial.test.ts @@ -0,0 +1,800 @@ +/** + * Adversarial tests for v0.7.1 release. + * + * Release content: + * 1. ProviderError.parseAPICallError — extract OpenAI nested error.message + * instead of dumping raw body when typeof guard rejects body.error + * (#789, closes #788) + * + * Focus here is the surface a chaos engineer / provider-spec abuser would + * actually throw at parseAPICallError post-fix: malformed JSON shapes, + * boundary scalar-typed bodies, prototype-pollution attempts, and very large + * error strings. The original fix's regression-guard tests live in + * test/provider/error.test.ts; this file pins the edges they don't cover. + */ + +import { describe, test, expect } from "bun:test" +import { ProviderError } from "../../src/provider/error" +import { Telemetry } from "../../src/altimate/telemetry" +import { APICallError } from "ai" + +function makeAPICallError(opts: { + message?: string + statusCode?: number + responseBody?: string + url?: string +}): APICallError { + return new APICallError({ + message: opts.message ?? "", + statusCode: opts.statusCode, + responseBody: opts.responseBody, + isRetryable: false, + url: opts.url ?? "", + requestBodyValues: {}, + }) +} + +// --------------------------------------------------------------------------- +// Boundary — scalar / null / array bodies that JSON.parse can produce +// --------------------------------------------------------------------------- + +describe("parseAPICallError — JSON-scalar and non-object bodies", () => { + test("body parses to null — does not crash, falls through to raw body dump", () => { + const result = ProviderError.parseAPICallError({ + providerID: "openai" as any, + error: makeAPICallError({ + message: "Bad Request", + statusCode: 400, + responseBody: "null", + }), + }) + expect(result.type).toBe("api_error") + if (result.type === "api_error") { + // Optional chaining on null returns undefined for body.error?.message + // and reading body.message / body.error on null throws — the try/catch + // around JSON.parse + the extraction must absorb that and fall through. + expect(result.message).toContain("Bad Request") + } + }) + + test("body parses to a number scalar — does not crash", () => { + const result = ProviderError.parseAPICallError({ + providerID: "openai" as any, + error: makeAPICallError({ + message: "Bad Request", + statusCode: 400, + responseBody: "42", + }), + }) + expect(result.type).toBe("api_error") + if (result.type === "api_error") { + expect(result.message).toContain("Bad Request") + } + }) + + test("body parses to a string scalar — does not crash", () => { + const result = ProviderError.parseAPICallError({ + providerID: "openai" as any, + error: makeAPICallError({ + message: "Bad Request", + statusCode: 400, + responseBody: '"a plain string"', + }), + }) + expect(result.type).toBe("api_error") + if (result.type === "api_error") { + expect(result.message).toContain("Bad Request") + } + }) + + test("body parses to an array — does not crash, no value extracted", () => { + const result = ProviderError.parseAPICallError({ + providerID: "openai" as any, + error: makeAPICallError({ + message: "Bad Request", + statusCode: 400, + responseBody: '[{"error":{"message":"buried"}}]', + }), + }) + expect(result.type).toBe("api_error") + if (result.type === "api_error") { + // Array.error is undefined — extractor finds nothing, falls back to raw. + expect(result.message).toContain("Bad Request") + } + }) +}) + +// --------------------------------------------------------------------------- +// Prototype-pollution attempt — must not pollute Object.prototype +// --------------------------------------------------------------------------- + +describe("parseAPICallError — prototype pollution attempts", () => { + test("__proto__ in body does not pollute Object.prototype", () => { + // Important: write the JSON as a literal string. `JSON.stringify({__proto__: ...})` + // produces `{}` because in object-literal syntax `__proto__` is the prototype + // setter (it sets [[Prototype]]) rather than an own enumerable property, and + // JSON.stringify only walks own enumerables. Building the JSON by hand puts + // the malicious key on the wire so JSON.parse in parseAPICallError actually + // sees it — which is the surface a hostile gateway would exploit. + // try/finally so a regression doesn't leak prototype pollution into the + // rest of the suite (cascading-failure containment). + const before = (Object.prototype as any).polluted + try { + ProviderError.parseAPICallError({ + providerID: "openai" as any, + error: makeAPICallError({ + message: "Bad Request", + statusCode: 400, + responseBody: '{"__proto__":{"polluted":"yes"},"error":{"message":"harmless surface"}}', + }), + }) + expect((Object.prototype as any).polluted).toBe(before) + // Modern V8 makes __proto__ a regular property post-JSON.parse since 2019, + // but if a future refactor ever switches to Object.assign / spread we want + // a regression guard. + } finally { + if (before === undefined) delete (Object.prototype as any).polluted + else (Object.prototype as any).polluted = before + } + }) + + test("constructor.prototype injection does not pollute", () => { + const before = (Object.prototype as any).injected + try { + ProviderError.parseAPICallError({ + providerID: "openai" as any, + error: makeAPICallError({ + message: "Bad Request", + statusCode: 400, + responseBody: JSON.stringify({ + constructor: { prototype: { injected: "yes" } }, + error: { message: "harmless" }, + }), + }), + }) + expect((Object.prototype as any).injected).toBe(before) + } finally { + if (before === undefined) delete (Object.prototype as any).injected + else (Object.prototype as any).injected = before + } + }) +}) + +// --------------------------------------------------------------------------- +// Large strings — extractor returns input as-is, no truncation expected here +// (consumers handle truncation), but it must not crash. +// --------------------------------------------------------------------------- + +describe("parseAPICallError — large message bodies", () => { + test("100KB error.message is returned without crashing", () => { + const huge = "x".repeat(100_000) + const result = ProviderError.parseAPICallError({ + providerID: "openai" as any, + error: makeAPICallError({ + message: "Bad Request", + statusCode: 400, + responseBody: JSON.stringify({ error: { message: huge } }), + }), + }) + expect(result.type).toBe("api_error") + if (result.type === "api_error") { + expect(result.message.length).toBeGreaterThan(99_000) + expect(result.message.startsWith("Bad Request:")).toBe(true) + } + }) +}) + +// --------------------------------------------------------------------------- +// Malformed JSON — parser must absorb and fall back to status text or body +// --------------------------------------------------------------------------- + +describe("parseAPICallError — malformed JSON bodies", () => { + test("unparseable JSON does not crash, falls through to raw body append", () => { + const result = ProviderError.parseAPICallError({ + providerID: "openai" as any, + error: makeAPICallError({ + message: "Bad Request", + statusCode: 400, + responseBody: '{"error": {"message": "unterminated', + }), + }) + expect(result.type).toBe("api_error") + if (result.type === "api_error") { + expect(result.message).toContain("Bad Request") + } + }) + + test("empty-string body — falls back to raw (which is empty), preserves status", () => { + const result = ProviderError.parseAPICallError({ + providerID: "openai" as any, + error: makeAPICallError({ + message: "Bad Request", + statusCode: 400, + responseBody: "", + }), + }) + expect(result.type).toBe("api_error") + if (result.type === "api_error") { + expect(result.message).toBe("Bad Request") + } + }) +}) + +// --------------------------------------------------------------------------- +// Type-confusion ladder — null at any tier of the OR chain must not crash +// --------------------------------------------------------------------------- + +describe("parseAPICallError — null and missing fields at every tier", () => { + test("body.error explicitly null — falls through to body.message", () => { + const result = ProviderError.parseAPICallError({ + providerID: "openai" as any, + error: makeAPICallError({ + message: "Bad Request", + statusCode: 400, + responseBody: JSON.stringify({ error: null, message: "fallback" }), + }), + }) + expect(result.type).toBe("api_error") + if (result.type === "api_error") { + expect(result.message).toContain("fallback") + } + }) + + test("body.error.message explicitly null — falls through to body.message", () => { + const result = ProviderError.parseAPICallError({ + providerID: "openai" as any, + error: makeAPICallError({ + message: "Bad Request", + statusCode: 400, + responseBody: JSON.stringify({ + error: { message: null, code: "x" }, + message: "fallback", + }), + }), + }) + expect(result.type).toBe("api_error") + if (result.type === "api_error") { + expect(result.message).toContain("fallback") + } + }) + + test("body.error.message numeric — falls through to body.message", () => { + // typeof 42 === "number", not "string" — must not assign to errMsg. + const result = ProviderError.parseAPICallError({ + providerID: "openai" as any, + error: makeAPICallError({ + message: "Bad Request", + statusCode: 400, + responseBody: JSON.stringify({ + error: { message: 42 }, + message: "fallback", + }), + }), + }) + expect(result.type).toBe("api_error") + if (result.type === "api_error") { + expect(result.message).toContain("fallback") + expect(result.message).not.toContain("42") + } + }) +}) + +// --------------------------------------------------------------------------- +// Fix A — Bedrock / AWS Lambda errorMessage shape +// --------------------------------------------------------------------------- + +describe("parseAPICallError — body.errorMessage extraction (Bedrock/Lambda)", () => { + test("extracts body.errorMessage when no error.message or top-level message", () => { + const result = ProviderError.parseAPICallError({ + providerID: "amazon-bedrock" as any, + error: makeAPICallError({ + message: "Bad Request", + statusCode: 400, + responseBody: JSON.stringify({ + errorMessage: "ValidationException: model ID is required", + errorType: "ValidationException", + }), + }), + }) + expect(result.type).toBe("api_error") + if (result.type === "api_error") { + expect(result.message).toContain("ValidationException: model ID is required") + } + }) + + test("body.error.message wins over body.errorMessage when both present", () => { + const result = ProviderError.parseAPICallError({ + providerID: "amazon-bedrock" as any, + error: makeAPICallError({ + message: "Bad Request", + statusCode: 400, + responseBody: JSON.stringify({ + error: { message: "nested wins" }, + errorMessage: "lambda style", + }), + }), + }) + expect(result.type).toBe("api_error") + if (result.type === "api_error") { + expect(result.message).toContain("nested wins") + expect(result.message).not.toContain("lambda style") + } + }) +}) + +// --------------------------------------------------------------------------- +// Fix C — parseStreamError fallback for non-OpenAI codes +// --------------------------------------------------------------------------- + +describe("parseStreamError — fallback for codes not in the switch", () => { + test("unknown code with body.error.message returns api_error, not undefined", () => { + const result = ProviderError.parseStreamError({ + type: "error", + error: { code: "anthropic_overloaded", message: "Provider is overloaded, try again." }, + }) + expect(result).toBeDefined() + expect(result?.type).toBe("api_error") + if (result && result.type === "api_error") { + expect(result.message).toBe("Provider is overloaded, try again.") + expect(result.isRetryable).toBe(false) + } + }) + + test("unknown code with no extractable string returns undefined", () => { + // Last-resort behavior: caller falls back to JSON.stringify(e). + const result = ProviderError.parseStreamError({ + type: "error", + error: { code: "weird_code" }, + }) + expect(result).toBeUndefined() + }) + + test("unknown code with body.errorMessage (Bedrock-style) returns api_error", () => { + const result = ProviderError.parseStreamError({ + type: "error", + errorMessage: "ValidationException from streaming endpoint", + }) + expect(result).toBeDefined() + if (result && result.type === "api_error") { + expect(result.message).toContain("ValidationException") + } + }) + + test("documented OpenAI codes still hit the switch (regression guard)", () => { + const result = ProviderError.parseStreamError({ + type: "error", + error: { code: "insufficient_quota", message: "ignored" }, + }) + expect(result?.type).toBe("api_error") + if (result && result.type === "api_error") { + // Pre-existing literal — switch must take precedence over fallback. + expect(result.message).toContain("Quota exceeded") + } + }) +}) + +// --------------------------------------------------------------------------- +// Fix D — /models hint on model_not_found +// --------------------------------------------------------------------------- + +describe("parseAPICallError — /models hint on model_not_found", () => { + test("appends `altimate-code models` hint when error.code === model_not_found", () => { + const result = ProviderError.parseAPICallError({ + providerID: "openai" as any, + error: makeAPICallError({ + message: "Bad Request", + statusCode: 400, + responseBody: JSON.stringify({ + error: { + message: "The model 'gpt-99' does not exist or you do not have access to it.", + type: "invalid_request_error", + code: "model_not_found", + }, + }), + }), + }) + expect(result.type).toBe("api_error") + if (result.type === "api_error") { + expect(result.message).toContain("does not exist") + // Pin the canonical hint text exactly so docs / changelog / code can't + // diverge silently. This string is the single source of truth. + expect(result.message).toContain("Run `altimate models` to see available models.") + } + }) + + test("does NOT append hint when error.code is something else", () => { + const result = ProviderError.parseAPICallError({ + providerID: "openai" as any, + error: makeAPICallError({ + message: "Bad Request", + statusCode: 400, + responseBody: JSON.stringify({ + error: { message: "Invalid request", code: "invalid_request_error" }, + }), + }), + }) + expect(result.type).toBe("api_error") + if (result.type === "api_error") { + expect(result.message).not.toContain("altimate models") + } + }) +}) + +// --------------------------------------------------------------------------- +// Fix E — model_not_found is not retryable (carve-out from OpenAI 404 logic) +// --------------------------------------------------------------------------- + +describe("parseAPICallError — model_not_found skips retry-storm", () => { + test("OpenAI 404 with code=model_not_found has isRetryable=false", () => { + const result = ProviderError.parseAPICallError({ + providerID: "openai" as any, + error: makeAPICallError({ + message: "Not Found", + statusCode: 404, + responseBody: JSON.stringify({ + error: { message: "model gone", code: "model_not_found" }, + }), + }), + }) + expect(result.type).toBe("api_error") + if (result.type === "api_error") { + expect(result.isRetryable).toBe(false) + } + }) + + test("OpenAI 404 without model_not_found preserves retryable=true (regression guard)", () => { + // Existing behavior: OpenAI 404 is force-retried for transient model availability blips. + const result = ProviderError.parseAPICallError({ + providerID: "openai" as any, + error: makeAPICallError({ + message: "Not Found", + statusCode: 404, + responseBody: JSON.stringify({ + error: { message: "transient", code: "service_unavailable" }, + }), + }), + }) + expect(result.type).toBe("api_error") + if (result.type === "api_error") { + expect(result.isRetryable).toBe(true) + } + }) + + test("malformed body on 404 falls back to retryable=true (preserves transient-blip handling)", () => { + const result = ProviderError.parseAPICallError({ + providerID: "openai" as any, + error: makeAPICallError({ + message: "Not Found", + statusCode: 404, + responseBody: "{not valid json", + }), + }) + expect(result.type).toBe("api_error") + if (result.type === "api_error") { + expect(result.isRetryable).toBe(true) + } + }) +}) + +// --------------------------------------------------------------------------- +// Fix G — responseBody capped at 4KB +// --------------------------------------------------------------------------- + +describe("parseAPICallError — responseBody cap", () => { + test("100KB responseBody is truncated to exactly 4096 chars + truncation marker", () => { + const huge = "a".repeat(100_000) + const result = ProviderError.parseAPICallError({ + providerID: "openai" as any, + error: makeAPICallError({ + message: "Bad Request", + statusCode: 400, + responseBody: huge, + }), + }) + expect(result.type).toBe("api_error") + if (result.type === "api_error") { + // Pin the boundary EXACTLY. A regression from 4096 → e.g. 8192 would + // still pass `toBeLessThan(5000)` for shorter bodies; pin the prefix + // length and the appended marker so the cap is the load-bearing + // assertion, not the upper bound. + const prefix = "a".repeat(4096) + expect(result.responseBody).toBe(`${prefix}…[truncated 95904 chars]`) + } + }) + + test("small responseBody passes through untouched", () => { + const small = "small body" + const result = ProviderError.parseAPICallError({ + providerID: "openai" as any, + error: makeAPICallError({ + message: "Bad Request", + statusCode: 400, + responseBody: small, + }), + }) + expect(result.type).toBe("api_error") + if (result.type === "api_error") { + expect(result.responseBody).toBe(small) + } + }) +}) + +// --------------------------------------------------------------------------- +// Fix H — metadata.url internal-host masking +// --------------------------------------------------------------------------- + +describe("parseAPICallError — metadata.url masking for internal hosts", () => { + test("public provider URL is preserved verbatim", () => { + const result = ProviderError.parseAPICallError({ + providerID: "openai" as any, + error: makeAPICallError({ + message: "Bad Request", + statusCode: 400, + url: "https://api.openai.com/v1/chat/completions", + }), + }) + expect(result.type).toBe("api_error") + if (result.type === "api_error") { + expect(result.metadata?.url).toBe("https://api.openai.com/v1/chat/completions") + } + }) + + test(".internal hostname is redacted", () => { + const result = ProviderError.parseAPICallError({ + providerID: "openai" as any, + error: makeAPICallError({ + message: "Bad Request", + statusCode: 400, + url: "https://llm-gateway.bigbank.internal/v1/chat", + }), + }) + expect(result.type).toBe("api_error") + if (result.type === "api_error") { + expect(result.metadata?.url).not.toContain("bigbank.internal") + expect(result.metadata?.url).toContain("internal-host.redacted") + } + }) + + test("RFC1918 10.x IP host is redacted", () => { + const result = ProviderError.parseAPICallError({ + providerID: "openai" as any, + error: makeAPICallError({ + message: "Bad Request", + statusCode: 400, + url: "http://10.20.30.40:8080/v1/chat", + }), + }) + expect(result.type).toBe("api_error") + if (result.type === "api_error") { + expect(result.metadata?.url).not.toContain("10.20.30.40") + expect(result.metadata?.url).toContain("internal-host.redacted") + } + }) + + test("malformed URL falls back to verbatim (does not crash)", () => { + const result = ProviderError.parseAPICallError({ + providerID: "openai" as any, + error: makeAPICallError({ + message: "Bad Request", + statusCode: 400, + url: "not a url", + }), + }) + expect(result.type).toBe("api_error") + if (result.type === "api_error") { + expect(result.metadata?.url).toBe("not a url") + } + }) + + test("internal URL with basic-auth userinfo redacts BOTH host and credentials", () => { + // Regression guard: u.toString() preserves u.username/u.password by default, + // so naively rewriting only u.hostname leaks a basic-auth password through + // metadata.url. Must clear both before serializing. + const result = ProviderError.parseAPICallError({ + providerID: "openai" as any, + error: makeAPICallError({ + message: "Bad Request", + statusCode: 400, + url: "https://admin:hunter2@10.20.30.40/secret", + }), + }) + expect(result.type).toBe("api_error") + if (result.type === "api_error") { + expect(result.metadata?.url).not.toContain("hunter2") + expect(result.metadata?.url).not.toContain("admin:") + expect(result.metadata?.url).not.toContain("10.20.30.40") + expect(result.metadata?.url).toContain("internal-host.redacted") + } + }) + + test("IPv6 loopback host is redacted", () => { + const result = ProviderError.parseAPICallError({ + providerID: "openai" as any, + error: makeAPICallError({ + message: "Bad Request", + statusCode: 400, + url: "http://[::1]:8080/admin", + }), + }) + expect(result.type).toBe("api_error") + if (result.type === "api_error") { + expect(result.metadata?.url).toContain("internal-host.redacted") + } + }) + + test("IPv6 ULA (fc00::/7) host is redacted", () => { + const result = ProviderError.parseAPICallError({ + providerID: "openai" as any, + error: makeAPICallError({ + message: "Bad Request", + statusCode: 400, + url: "http://[fc00::1]/v1", + }), + }) + expect(result.type).toBe("api_error") + if (result.type === "api_error") { + expect(result.metadata?.url).toContain("internal-host.redacted") + } + }) + + test("AWS IMDS endpoint (169.254.169.254) is redacted", () => { + const result = ProviderError.parseAPICallError({ + providerID: "openai" as any, + error: makeAPICallError({ + message: "Bad Request", + statusCode: 400, + url: "http://169.254.169.254/latest/meta-data/", + }), + }) + expect(result.type).toBe("api_error") + if (result.type === "api_error") { + expect(result.metadata?.url).not.toContain("169.254.169.254") + expect(result.metadata?.url).toContain("internal-host.redacted") + } + }) + + test("RFC1918 boundary: 172.15 (NOT private) and 172.32 (NOT private) preserved", () => { + const a = ProviderError.parseAPICallError({ + providerID: "openai" as any, + error: makeAPICallError({ + message: "Bad Request", + statusCode: 400, + url: "http://172.15.0.1/v1", + }), + }) + if (a.type === "api_error") expect(a.metadata?.url).toContain("172.15.0.1") + + const b = ProviderError.parseAPICallError({ + providerID: "openai" as any, + error: makeAPICallError({ + message: "Bad Request", + statusCode: 400, + url: "http://172.32.0.1/v1", + }), + }) + if (b.type === "api_error") expect(b.metadata?.url).toContain("172.32.0.1") + }) + + test("lookalike hostname `attacker-localhost.com` is NOT redacted", () => { + // Defends against substring-match regression — must use boundary check. + const result = ProviderError.parseAPICallError({ + providerID: "openai" as any, + error: makeAPICallError({ + message: "Bad Request", + statusCode: 400, + url: "https://attacker-localhost.com/exfil", + }), + }) + expect(result.type).toBe("api_error") + if (result.type === "api_error") { + expect(result.metadata?.url).toBe("https://attacker-localhost.com/exfil") + } + }) + + test("basic-auth userinfo on a PUBLIC host is also stripped", () => { + // Pre cubic-bot review: userinfo was only cleared for internal hosts. + // Credentials in a public URL are arguably more dangerous (they're real + // keys, not just a misconfigured gateway), so userinfo strip runs + // regardless of internal/public classification. + const result = ProviderError.parseAPICallError({ + providerID: "openai" as any, + error: makeAPICallError({ + message: "Bad Request", + statusCode: 400, + url: "https://user:hunter2@api.openai.com/v1/chat", + }), + }) + expect(result.type).toBe("api_error") + if (result.type === "api_error") { + expect(result.metadata?.url).not.toContain("hunter2") + expect(result.metadata?.url).not.toContain("user:") + // Public host preserved; only userinfo redacted. + expect(result.metadata?.url).toContain("api.openai.com") + } + }) +}) + +// --------------------------------------------------------------------------- +// Fix F — maskString email + internal-host masking +// --------------------------------------------------------------------------- + +describe("Telemetry.maskString — email and internal-host patterns", () => { + test("email addresses are masked to ", () => { + const out = Telemetry.maskString("user@bigbank.com is not authorized") + expect(out).not.toContain("user@bigbank.com") + expect(out).toContain("") + }) + + test("internal .local hostname URL is masked", () => { + const out = Telemetry.maskString("Cannot reach https://llm-gw.fortune500.local/v1/chat") + expect(out).not.toContain("fortune500.local") + expect(out).toContain("") + }) + + test("RFC1918 10.x URL is masked", () => { + const out = Telemetry.maskString("Connection refused at http://10.20.30.40:8080/v1") + expect(out).not.toContain("10.20.30.40") + expect(out).toContain("") + }) + + test("public URL is left alone", () => { + const out = Telemetry.maskString("https://api.openai.com/v1/chat returned 500") + expect(out).toContain("api.openai.com") + }) + + test("api key still masked (regression guard)", () => { + const out = Telemetry.maskString("Auth failed with sk-abcdefghij1234567890XX") + expect(out).not.toContain("sk-abcdefghij1234567890XX") + expect(out).toContain("sk-***") + }) + + test("AWS IMDS URL (169.254.169.254) is masked", () => { + const out = Telemetry.maskString("Cannot reach http://169.254.169.254/latest/meta-data/") + expect(out).not.toContain("169.254.169.254") + expect(out).toContain("") + }) + + test("IPv6 loopback URL is masked", () => { + const out = Telemetry.maskString("Connection refused at http://[::1]:8080/admin") + expect(out).not.toContain("[::1]") + expect(out).toContain("") + }) + + test("IPv6 ULA (fc00::) URL is masked", () => { + const out = Telemetry.maskString("Backend down: http://[fc00::1]/v1") + expect(out).not.toContain("fc00") + expect(out).toContain("") + }) + + test("IPv6 link-local (fe80::) URL is masked", () => { + const out = Telemetry.maskString("Probe failed http://[fe80::1%25eth0]/x") + expect(out).not.toContain("fe80") + expect(out).toContain("") + }) + + test("query-string with `+` and `#` does not leak past internal-host marker", () => { + // Char class previously omitted +/#/,/; — secrets after `?` survived. + const out = Telemetry.maskString("Failed http://10.0.0.1/x?token=foo+bar#frag") + expect(out).not.toContain("foo+bar") + expect(out).not.toContain("#frag") + expect(out).toContain("") + }) + + test("internal URL with basic-auth userinfo is fully redacted (cubic P1 regression)", () => { + // Pre-fix: regex started with the host alternation, missing `user:pass@` + // prefix, so `https://admin:hunter2@10.0.0.5/x` did NOT match — basic-auth + // creds + internal host both leaked. Now matches via the optional + // `(?:[^\/\s@]+@)?` group. + const out = Telemetry.maskString("Cannot reach https://admin:hunter2@10.0.0.5/secret") + expect(out).not.toContain("hunter2") + expect(out).not.toContain("admin:") + expect(out).not.toContain("10.0.0.5") + expect(out).toContain("") + }) + + test("0.0.0.0 (any-interface bind) is masked", () => { + const out = Telemetry.maskString("Probe failed http://0.0.0.0:8080/") + expect(out).not.toContain("0.0.0.0") + expect(out).toContain("") + }) +})