diff --git a/README.md b/README.md index 97a7513..a64c329 100644 --- a/README.md +++ b/README.md @@ -1,8 +1,9 @@ # ai-kit **The AI layer of an app, in one install.** Which model to call, what to do when -the vendor deletes it, what to do when you're going too fast, how to share a free -tier fairly between users, and how to fill a form from plain language. +the vendor deletes it, how to walk the fallback and know when none of it worked, +what to do when you're going too fast, how to share a free tier fairly between +users, and how to fill a form from plain language. ```bash npm install ai-kit @@ -46,11 +47,7 @@ import { freeChain, usableChain, chainFrom } from 'ai-kit'; const providers = freeChain('MYAPP'); // groq → openrouter const links = usableChain(providers, process.env); // drops vendors with no key - -for (const { provider, model } of chainFrom(process.env.MYAPP_MODEL, links)) { - // POST `${provider.baseUrl}/chat/completions` with `model` - // on failure, continue — that is the whole point -} +const chain = chainFrom(process.env.MYAPP_MODEL, links); ``` Falling back to a **smaller model at the same vendor buys nothing**: it draws on @@ -61,6 +58,45 @@ the same org-wide daily budget, so when the day runs dry every link in that via a text tool protocol, not native `tool_calls`. A native-only client would have silently lost most of the chain. +### Is it up? — walk the chain, and know when none of it worked + +A chain nobody walks is a list, not a fallback. This was found sitting unused +next to a single-shot caller in an app this package's `freeChain` had already +saved from a retired model — the list existed, and nothing tried link two. + +```ts +import { tryChain, createHealthTracker } from 'ai-kit'; + +const llmHealth = createHealthTracker(); // one per process; see below + +const { text } = await tryChain(chain, { + health: llmHealth, + attempt: async ({ provider, model }) => { + // POST `${provider.baseUrl}/chat/completions` with `model` — your own + // fetch, your own retries. Throw to demote to the next link. + return callVendor(provider, model); + }, +}); +``` + +No HTTP client here either — `attempt` makes the real request; `tryChain` only +decides which link goes next and throws `ChainExhaustedError` (naming every +link's failure, not just the last) when none of them work. + +`createHealthTracker()` is a factory, not a global: a single-process app gets +the old "shared state everywhere" behaviour for free by making exactly one and +exporting it — + +```ts +// lib/llm-health.ts +export const llmHealth = createHealthTracker(); +``` + +— and a health route reports `llmHealth.getHealth()` instead of only ever +checking the database. That gap is not hypothetical: an app's `/health` reported +"healthy" while its only configured key was returning 401 and every chat route +was answering a friendly, silent, hardcoded apology. HTTP 200 is not evidence. + ### Still there? — catch a retirement before a user does ```ts @@ -78,7 +114,7 @@ never *gone*. Treating "I could not look" as "nothing is there" marks every mode retired and invents an outage someone then acts on. > This fleet runs it daily across every repo from -> [`dotfiles/scripts/ci/model-pin-audit.mjs`](https://github.com/catomean/dotfiles). +> [`fleet/scripts/ci/model-pin-audit.mjs`](https://github.com/bitbaum/fleet). ### Too fast? — the three kinds of 429 @@ -124,7 +160,7 @@ import { useAiForm } from 'ai-kit/react'; import { createFormAssistHandler } from 'ai-kit/server'; ``` -Re-exported from [`ai-forms`](https://github.com/catomean/ai-forms), which +Re-exported from [`ai-forms`](https://github.com/bitbaum/ai-forms), which stays its own package — it works, four apps run it, and it is useful well outside this fleet. Swallowing it would have broken those four for the sake of a filing system. @@ -163,9 +199,9 @@ locally. | Package | For | |---|---| -| [`ai-forms`](https://github.com/catomean/ai-forms) | Form filling on its own, without the model layer | -| [`threadkit`](https://github.com/catomean/threadkit) | Messages between people, and who may see them | -| [`limitkit`](https://github.com/catomean/limitkit) | Stopping someone doing something too often | +| [`ai-forms`](https://github.com/bitbaum/ai-forms) | Form filling on its own, without the model layer | +| [`threadkit`](https://github.com/bitbaum/threadkit) | Messages between people, and who may see them | +| [`limitkit`](https://github.com/bitbaum/limitkit) | Stopping someone doing something too often | `threadkit` and `limitkit` are **not** merged in here, on purpose: neither has anything to do with AI. An app that throttles its login form should not install a diff --git a/package.json b/package.json index d4f0d20..3e2910f 100644 --- a/package.json +++ b/package.json @@ -1,22 +1,24 @@ { "name": "ai-kit", - "version": "0.4.0", - "description": "One install for the AI layer of an app: which model to call and what to do when the vendor retires it, how to read the three kinds of 429, a fair daily budget across users, and headless AI form filling.", + "version": "0.5.0", + "description": "One install for the AI layer of an app: which model to call, what to do when the vendor retires it, how to walk the fallback chain and know when none of it worked, how to read the three kinds of 429, a fair daily budget across users, and headless AI form filling.", "license": "MIT", "author": "Mao Nakamoto", - "homepage": "https://github.com/catomean/ai-kit#readme", + "homepage": "https://github.com/bitbaum/ai-kit#readme", "repository": { "type": "git", - "url": "git+https://github.com/catomean/ai-kit.git" + "url": "git+https://github.com/bitbaum/ai-kit.git" }, "bugs": { - "url": "https://github.com/catomean/ai-kit/issues" + "url": "https://github.com/bitbaum/ai-kit/issues" }, "keywords": [ "ai", "llm", "form-fill", "fallback", + "failover", + "health-check", "free-tier", "rate-limit", "quota", diff --git a/src/attempt.ts b/src/attempt.ts new file mode 100644 index 0000000..b3fc3c5 --- /dev/null +++ b/src/attempt.ts @@ -0,0 +1,82 @@ +/** + * Walk a chain, never own the fetch. + * + * `usableChain`/`chainFrom` (chain.ts) already answer WHICH links exist and in + * what order. What was still missing — in every app that hand-rolled it, and + * inconsistently — is the loop that tries link one, and on failure tries link + * two, rather than picking the first link and calling it once. A chain nobody + * walks is a list, not a fallback: it was found sitting unused next to a + * single-shot caller in the same app that this package's `freeChain` already + * protected from a retired model but not from a dead key, because ordering the + * links and walking them were still two different jobs and only one had a + * home. + * + * This still ships no HTTP client. `attempt` is supplied by the caller and + * does the actual request; this only decides which link goes next, and + * records the outcome if a `HealthTracker` is given. + */ + +import type { Link } from "./chain.js"; +import type { HealthTracker } from "./health.js"; + +export interface ChainAttemptFailure { + link: Link; + message: string; +} + +/** Every link in the chain was tried and failed (or the chain was empty). */ +export class ChainExhaustedError extends Error { + readonly failures: ChainAttemptFailure[]; + + constructor(failures: ChainAttemptFailure[]) { + super( + failures.length === 0 + ? "No usable link in the chain — every provider is missing its key, or has no models configured." + : `All ${failures.length} link(s) failed — ${failures + .map((f) => `${f.link.provider.id}/${f.link.model}: ${f.message}`) + .join("; ")}`, + ); + this.name = "ChainExhaustedError"; + this.failures = failures; + } +} + +export interface TryChainOptions { + /** Makes the actual call for one link. Throw to demote to the next link. */ + attempt: (link: Link) => Promise; + /** Records one success or one failure for the WHOLE walk, not per link. */ + health?: HealthTracker; + /** Called on each link's failure, before moving to the next — e.g. to log it. */ + onLinkFailure?: (link: Link, error: unknown) => void; +} + +/** + * Try each link in order; return the first success. + * + * Health is recorded once per call — a success on link two is still a success + * for the app, and a health check that flagged it "degraded" because the FIRST + * link failed would be reporting its own fallback working as a problem. + * + * Throws `ChainExhaustedError` (carrying every link's failure) when none + * succeed, so a caller can log exactly what was tried rather than only the + * last error — the failure that matters is often not the last one. + */ +export async function tryChain(chain: Link[], options: TryChainOptions): Promise { + const failures: ChainAttemptFailure[] = []; + + for (const link of chain) { + try { + const result = await options.attempt(link); + options.health?.recordSuccess(); + return result; + } catch (error) { + const message = error instanceof Error ? error.message : String(error); + failures.push({ link, message }); + options.onLinkFailure?.(link, error); + } + } + + const exhausted = new ChainExhaustedError(failures); + options.health?.recordFailure(exhausted); + throw exhausted; +} diff --git a/src/health.ts b/src/health.ts new file mode 100644 index 0000000..1c8ca74 --- /dev/null +++ b/src/health.ts @@ -0,0 +1,92 @@ +/** + * Observed health of an AI feature — was the last generation actually usable? + * + * This exists because of a failure that "the chain never fails" hides just as + * well as a single pin does. On 2026-08-28 an app's only configured key + * started returning 401 and the chain (correctly) had nowhere else to go — the + * routes caught the error and answered HTTP 200 with an apology, and the + * app's own `/health` reported "healthy" because it only ever checked the + * database. A total outage of the product's core feature was invisible to + * every automated check, for as long as nobody happened to try it by hand. + * + * A tracker records what actually happened, so a health route can report it + * and a strict check can refuse to call the app "up" while its AI is down. + * + * FACTORY, NOT A SINGLETON. A module-level global would force a shared state + * shape on every consumer and make the transitions untestable without mutating + * process-wide state between tests. `createHealthTracker()` returns an + * isolated instance — a single-process app gets the old singleton behaviour + * for free by creating exactly one and exporting it from its own module: + * + * // lib/llm-health.ts + * export const llmHealth = createHealthTracker(); + * + * If the app scales horizontally, this state is per-instance and wants a + * shared store — that migration is app-specific and out of scope here. + */ + +export type HealthStatus = "unknown" | "ok" | "degraded" | "down"; + +export interface Health { + status: HealthStatus; + consecutiveFailures: number; + lastError: string | null; + /** Epoch milliseconds. Format at the API boundary, not here. */ + lastSuccessAt: number | null; + /** Epoch milliseconds. Format at the API boundary, not here. */ + lastFailureAt: number | null; +} + +export interface HealthTrackerOptions { + /** Consecutive failures before status flips from "degraded" to "down". Default 3. */ + downAfter?: number; + /** Clock injection point for tests. Default `Date.now`. */ + now?: () => number; +} + +export interface HealthTracker { + /** Call after a generation that produced usable content. */ + recordSuccess(): void; + /** Call when generation threw, or returned nothing usable. */ + recordFailure(error: unknown): void; + getHealth(): Health; + /** Test seam — also useful for an app-triggered "recheck now". */ + reset(): void; +} + +export function createHealthTracker(options: HealthTrackerOptions = {}): HealthTracker { + const downAfter = options.downAfter ?? 3; + const now = options.now ?? Date.now; + + let consecutiveFailures = 0; + let lastError: string | null = null; + let lastSuccessAt: number | null = null; + let lastFailureAt: number | null = null; + + return { + recordSuccess() { + consecutiveFailures = 0; + lastError = null; + lastSuccessAt = now(); + }, + recordFailure(error: unknown) { + consecutiveFailures += 1; + lastFailureAt = now(); + lastError = error instanceof Error ? error.message : String(error ?? "unknown error"); + }, + getHealth(): Health { + let status: HealthStatus; + if (consecutiveFailures >= downAfter) status = "down"; + else if (consecutiveFailures > 0) status = "degraded"; + else if (lastSuccessAt !== null) status = "ok"; + else status = "unknown"; + return { status, consecutiveFailures, lastError, lastSuccessAt, lastFailureAt }; + }, + reset() { + consecutiveFailures = 0; + lastError = null; + lastSuccessAt = null; + lastFailureAt = null; + }, + }; +} diff --git a/src/index.ts b/src/index.ts index 57286f9..b757092 100644 --- a/src/index.ts +++ b/src/index.ts @@ -3,10 +3,10 @@ * * WHAT IT IS FOR * -------------- - * An app that wants an AI feature needs four unrelated-looking decisions to go - * right, and getting any one wrong looks identical from the outside: the - * assistant is broken. This package holds all four, so adding AI is one - * decision instead of four. + * An app that wants an AI feature needs several unrelated-looking decisions to + * go right, and getting any one wrong looks identical from the outside: the + * assistant is broken. This package holds all of them, so adding AI is one + * decision instead of many. * * which model — a fallback list ACROSS VENDORS, because a single pinned free * model is a scheduled outage, and a smaller model at the same @@ -14,6 +14,10 @@ * still there? — has the vendor retired an id we still ask for? The list is * itself a list of pins, so it rots too. Zero tokens, so it can * run on a schedule instead of being remembered. + * walk it — a chain nobody walks is a list, not a fallback. `tryChain` + * tries each link and stops at the first success; a + * `HealthTracker` records whether the WHOLE chain came back + * empty, so a health route can say so before a user does. * too fast? — tell the three kinds of 429 apart. They share a status code * and need opposite responses; only the body distinguishes them. * who gets it — divide a fixed daily pool across active users, so the person @@ -32,9 +36,11 @@ * * STILL NOT INCLUDED: an HTTP client. Every app has its own calling conventions, * retries and logging, and replacing those is a rewrite rather than an adoption. - * This supplies the decisions; the caller keeps the fetch. That rule is under - * review — `ai-forms`, the most-adopted package in this fleet, is the one that - * broke it by shipping a route factory and a hook. + * This supplies the decisions; the caller keeps the fetch — `tryChain` is an + * orchestrator, not a client: the caller's own `attempt` function makes the + * actual request. That rule is under review — `ai-forms`, the most-adopted + * package in this fleet, is the one that broke it by shipping a route factory + * and a hook. */ export { @@ -62,6 +68,21 @@ export { catalogReport, } from "./catalog.js"; +export { + type ChainAttemptFailure, + type TryChainOptions, + ChainExhaustedError, + tryChain, +} from "./attempt.js"; + +export { + type HealthStatus, + type Health, + type HealthTrackerOptions, + type HealthTracker, + createHealthTracker, +} from "./health.js"; + export { type RateLimitKind, classifyRateLimit, diff --git a/test/attempt.test.js b/test/attempt.test.js new file mode 100644 index 0000000..6af8756 --- /dev/null +++ b/test/attempt.test.js @@ -0,0 +1,103 @@ +/** + * `tryChain`'s entire reason to exist: try the next link on failure, instead + * of stopping at the first one — and never own the actual request. + */ +import { test } from 'node:test'; +import assert from 'node:assert/strict'; + +import { tryChain, ChainExhaustedError, createHealthTracker } from 'ai-kit'; + +function link(providerId, model) { + return { provider: { id: providerId, baseUrl: 'https://example.invalid', keyEnv: 'X', models: [model], dailyTokens: 0 }, model }; +} + +test('returns the first link\'s result without trying the rest', async () => { + const tried = []; + const result = await tryChain([link('groq', 'a'), link('openrouter', 'b')], { + attempt: async (l) => { + tried.push(l.model); + return `ok:${l.model}`; + }, + }); + assert.equal(result, 'ok:a'); + assert.deepEqual(tried, ['a']); +}); + +test('demotes to the next link on failure — the whole point of a chain', async () => { + const tried = []; + const result = await tryChain([link('groq', 'dead'), link('openrouter', 'alive')], { + attempt: async (l) => { + tried.push(l.model); + if (l.model === 'dead') throw new Error('401 unauthorized'); + return `ok:${l.model}`; + }, + }); + assert.equal(result, 'ok:alive'); + assert.deepEqual(tried, ['dead', 'alive']); +}); + +test('every link failing throws ChainExhaustedError naming every failure, not just the last', async () => { + await assert.rejects( + tryChain([link('groq', 'a'), link('openrouter', 'b')], { + attempt: async (l) => { + throw new Error(`${l.model} refused`); + }, + }), + (err) => { + assert.ok(err instanceof ChainExhaustedError); + assert.equal(err.failures.length, 2); + assert.match(err.message, /a refused/); + assert.match(err.message, /b refused/); + return true; + }, + ); +}); + +test('an empty chain is a distinct, honest failure — not silently "succeeds with nothing"', async () => { + await assert.rejects( + tryChain([], { attempt: async () => 'unreachable' }), + (err) => { + assert.ok(err instanceof ChainExhaustedError); + assert.match(err.message, /no key|No usable link/i); + return true; + }, + ); +}); + +test('a success on link two still records a SUCCESS, not degraded — the fallback working is not a problem', async () => { + const health = createHealthTracker(); + await tryChain([link('groq', 'dead'), link('openrouter', 'alive')], { + health, + attempt: async (l) => { + if (l.model === 'dead') throw new Error('401'); + return 'ok'; + }, + }); + assert.equal(health.getHealth().status, 'ok'); + assert.equal(health.getHealth().consecutiveFailures, 0); +}); + +test('exhausting the chain records exactly ONE failure on the tracker, not one per link', async () => { + const health = createHealthTracker({ downAfter: 2 }); + await assert.rejects( + tryChain([link('groq', 'a'), link('openrouter', 'b')], { + health, + attempt: async () => { + throw new Error('down'); + }, + }), + ); + assert.equal(health.getHealth().consecutiveFailures, 1, 'one exhausted walk is one data point, not two'); +}); + +test('onLinkFailure fires per demoted link, for callers that want to log each attempt', async () => { + const seen = []; + await tryChain([link('groq', 'a'), link('openrouter', 'b')], { + onLinkFailure: (l, err) => seen.push(`${l.provider.id}:${err.message}`), + attempt: async (l) => { + if (l.provider.id === 'groq') throw new Error('boom'); + return 'ok'; + }, + }); + assert.deepEqual(seen, ['groq:boom']); +}); diff --git a/test/exports.test.js b/test/exports.test.js index 734055c..525bb55 100644 --- a/test/exports.test.js +++ b/test/exports.test.js @@ -17,6 +17,10 @@ test('the package exports its public surface through the exports map', () => { const expected = [ // chain 'providerModels', 'withEnvPrefix', 'freeChain', 'dayCapacityTokens', 'usableChain', 'chainFrom', + // attempt + 'tryChain', 'ChainExhaustedError', + // health + 'createHealthTracker', // limits 'classifyRateLimit', 'retryAfterSeconds', 'humanizeWait', 'rateLimitMessage', // fair-share diff --git a/test/health.test.js b/test/health.test.js new file mode 100644 index 0000000..a0d14ad --- /dev/null +++ b/test/health.test.js @@ -0,0 +1,90 @@ +/** + * The tracker's whole job is turning a run of successes/failures into a + * status a health route can trust — pinned here, transition by transition. + */ +import { test } from 'node:test'; +import assert from 'node:assert/strict'; + +import { createHealthTracker } from 'ai-kit'; + +test('starts unknown — no evidence either way yet', () => { + const tracker = createHealthTracker(); + assert.equal(tracker.getHealth().status, 'unknown'); +}); + +test('one success is ok, not merely "not down"', () => { + const tracker = createHealthTracker(); + tracker.recordSuccess(); + const health = tracker.getHealth(); + assert.equal(health.status, 'ok'); + assert.equal(health.consecutiveFailures, 0); + assert.equal(health.lastError, null); +}); + +test('failures short of the threshold are degraded, not down', () => { + const tracker = createHealthTracker({ downAfter: 3 }); + tracker.recordFailure(new Error('401')); + assert.equal(tracker.getHealth().status, 'degraded'); + tracker.recordFailure(new Error('401')); + assert.equal(tracker.getHealth().status, 'degraded'); +}); + +test('the Nth consecutive failure flips it down, exactly at the threshold', () => { + const tracker = createHealthTracker({ downAfter: 3 }); + tracker.recordFailure(new Error('a')); + tracker.recordFailure(new Error('b')); + tracker.recordFailure(new Error('c')); + const health = tracker.getHealth(); + assert.equal(health.status, 'down'); + assert.equal(health.consecutiveFailures, 3); + assert.equal(health.lastError, 'c'); +}); + +test('a single success recovers a down tracker to ok, not degraded', () => { + const tracker = createHealthTracker({ downAfter: 2 }); + tracker.recordFailure(new Error('x')); + tracker.recordFailure(new Error('y')); + assert.equal(tracker.getHealth().status, 'down'); + + tracker.recordSuccess(); + const health = tracker.getHealth(); + assert.equal(health.status, 'ok'); + assert.equal(health.consecutiveFailures, 0, 'the streak must reset, not merely drop below threshold'); +}); + +test('a non-Error failure still records a readable message', () => { + const tracker = createHealthTracker(); + tracker.recordFailure('plain string failure'); + assert.equal(tracker.getHealth().lastError, 'plain string failure'); +}); + +test('reset returns to unknown, clearing every field', () => { + const tracker = createHealthTracker(); + tracker.recordFailure(new Error('boom')); + tracker.reset(); + assert.deepEqual(tracker.getHealth(), { + status: 'unknown', + consecutiveFailures: 0, + lastError: null, + lastSuccessAt: null, + lastFailureAt: null, + }); +}); + +test('the clock is injectable, not read from a global', () => { + let now = 1000; + const tracker = createHealthTracker({ now: () => now }); + tracker.recordSuccess(); + assert.equal(tracker.getHealth().lastSuccessAt, 1000); + now = 2000; + tracker.recordFailure(new Error('later')); + assert.equal(tracker.getHealth().lastFailureAt, 2000); +}); + +test('two independent trackers never share state — no hidden singleton', () => { + const a = createHealthTracker(); + const b = createHealthTracker(); + a.recordFailure(new Error('only a')); + assert.equal(a.getHealth().status, 'degraded'); + assert.equal(b.getHealth().status, 'unknown'); +});