feat: tryChain + createHealthTracker — walk the chain, know when it's down

README.md

@@ -1,8 +1,9 @@
 # ai-kit
 
 **The AI layer of an app, in one install.** Which model to call, what to do when
-the vendor deletes it, what to do when you're going too fast, how to share a free
-tier fairly between users, and how to fill a form from plain language.
+the vendor deletes it, how to walk the fallback and know when none of it worked,
+what to do when you're going too fast, how to share a free tier fairly between
+users, and how to fill a form from plain language.
 
 ```bash
 npm install ai-kit
@@ -46,11 +47,7 @@ import { freeChain, usableChain, chainFrom } from 'ai-kit';
 
 const providers = freeChain('MYAPP');               // groq → openrouter
 const links = usableChain(providers, process.env);   // drops vendors with no key
-
-for (const { provider, model } of chainFrom(process.env.MYAPP_MODEL, links)) {
-  // POST `${provider.baseUrl}/chat/completions` with `model`
-  // on failure, continue — that is the whole point
-}
+const chain = chainFrom(process.env.MYAPP_MODEL, links);
 ```
 
 Falling back to a **smaller model at the same vendor buys nothing**: it draws on
@@ -61,6 +58,45 @@ the same org-wide daily budget, so when the day runs dry every link in that
 via a text tool protocol, not native `tool_calls`. A native-only client would
 have silently lost most of the chain.
 
+### Is it up? — walk the chain, and know when none of it worked
+
+A chain nobody walks is a list, not a fallback. This was found sitting unused
+next to a single-shot caller in an app this package's `freeChain` had already
+saved from a retired model — the list existed, and nothing tried link two.
+
+```ts
+import { tryChain, createHealthTracker } from 'ai-kit';
+
+const llmHealth = createHealthTracker(); // one per process; see below
+
+const { text } = await tryChain(chain, {
+  health: llmHealth,
+  attempt: async ({ provider, model }) => {
+    // POST `${provider.baseUrl}/chat/completions` with `model` — your own
+    // fetch, your own retries. Throw to demote to the next link.
+    return callVendor(provider, model);
+  },
+});
+```
+
+No HTTP client here either — `attempt` makes the real request; `tryChain` only
+decides which link goes next and throws `ChainExhaustedError` (naming every
+link's failure, not just the last) when none of them work.
+
+`createHealthTracker()` is a factory, not a global: a single-process app gets
+the old "shared state everywhere" behaviour for free by making exactly one and
+exporting it —
+
+```ts
+// lib/llm-health.ts
+export const llmHealth = createHealthTracker();
+```
+
+— and a health route reports `llmHealth.getHealth()` instead of only ever
+checking the database. That gap is not hypothetical: an app's `/health` reported
+"healthy" while its only configured key was returning 401 and every chat route
+was answering a friendly, silent, hardcoded apology. HTTP 200 is not evidence.
+
 ### Still there? — catch a retirement before a user does
 
 ```ts
@@ -78,7 +114,7 @@ never *gone*. Treating "I could not look" as "nothing is there" marks every mode
 retired and invents an outage someone then acts on.
 
 > This fleet runs it daily across every repo from
-> [`dotfiles/scripts/ci/model-pin-audit.mjs`](https://github.com/catomean/dotfiles).
+> [`fleet/scripts/ci/model-pin-audit.mjs`](https://github.com/bitbaum/fleet).
 
 ### Too fast? — the three kinds of 429
 
@@ -124,7 +160,7 @@ import { useAiForm } from 'ai-kit/react';
 import { createFormAssistHandler } from 'ai-kit/server';
 ```
 
-Re-exported from [`ai-forms`](https://github.com/catomean/ai-forms), which
+Re-exported from [`ai-forms`](https://github.com/bitbaum/ai-forms), which
 stays its own package — it works, four apps run it, and it is useful well outside
 this fleet. Swallowing it would have broken those four for the sake of a filing
 system.
@@ -163,9 +199,9 @@ locally.
 
 | Package | For |
 |---|---|
-| [`ai-forms`](https://github.com/catomean/ai-forms) | Form filling on its own, without the model layer |
-| [`threadkit`](https://github.com/catomean/threadkit) | Messages between people, and who may see them |
-| [`limitkit`](https://github.com/catomean/limitkit) | Stopping someone doing something too often |
+| [`ai-forms`](https://github.com/bitbaum/ai-forms) | Form filling on its own, without the model layer |
+| [`threadkit`](https://github.com/bitbaum/threadkit) | Messages between people, and who may see them |
+| [`limitkit`](https://github.com/bitbaum/limitkit) | Stopping someone doing something too often |
 
 `threadkit` and `limitkit` are **not** merged in here, on purpose: neither has
 anything to do with AI. An app that throttles its login form should not install a

package.json

@@ -1,22 +1,24 @@
 {
   "name": "ai-kit",
-  "version": "0.4.0",
-  "description": "One install for the AI layer of an app: which model to call and what to do when the vendor retires it, how to read the three kinds of 429, a fair daily budget across users, and headless AI form filling.",
+  "version": "0.5.0",
+  "description": "One install for the AI layer of an app: which model to call, what to do when the vendor retires it, how to walk the fallback chain and know when none of it worked, how to read the three kinds of 429, a fair daily budget across users, and headless AI form filling.",
   "license": "MIT",
   "author": "Mao Nakamoto",
-  "homepage": "https://github.com/catomean/ai-kit#readme",
+  "homepage": "https://github.com/bitbaum/ai-kit#readme",
   "repository": {
     "type": "git",
-    "url": "git+https://github.com/catomean/ai-kit.git"
+    "url": "git+https://github.com/bitbaum/ai-kit.git"
   },
   "bugs": {
-    "url": "https://github.com/catomean/ai-kit/issues"
+    "url": "https://github.com/bitbaum/ai-kit/issues"
   },
   "keywords": [
     "ai",
     "llm",
     "form-fill",
     "fallback",
+    "failover",
+    "health-check",
     "free-tier",
     "rate-limit",
     "quota",

src/attempt.ts

@@ -0,0 +1,82 @@
+/**
+ * Walk a chain, never own the fetch.
+ *
+ * `usableChain`/`chainFrom` (chain.ts) already answer WHICH links exist and in
+ * what order. What was still missing — in every app that hand-rolled it, and
+ * inconsistently — is the loop that tries link one, and on failure tries link
+ * two, rather than picking the first link and calling it once. A chain nobody
+ * walks is a list, not a fallback: it was found sitting unused next to a
+ * single-shot caller in the same app that this package's `freeChain` already
+ * protected from a retired model but not from a dead key, because ordering the
+ * links and walking them were still two different jobs and only one had a
+ * home.
+ *
+ * This still ships no HTTP client. `attempt` is supplied by the caller and
+ * does the actual request; this only decides which link goes next, and
+ * records the outcome if a `HealthTracker` is given.
+ */
+
+import type { Link } from "./chain.js";
+import type { HealthTracker } from "./health.js";
+
+export interface ChainAttemptFailure {
+  link: Link;
+  message: string;
+}
+
+/** Every link in the chain was tried and failed (or the chain was empty). */
+export class ChainExhaustedError extends Error {
+  readonly failures: ChainAttemptFailure[];
+
+  constructor(failures: ChainAttemptFailure[]) {
+    super(
+      failures.length === 0
+        ? "No usable link in the chain — every provider is missing its key, or has no models configured."
+        : `All ${failures.length} link(s) failed — ${failures
+            .map((f) => `${f.link.provider.id}/${f.link.model}: ${f.message}`)
+            .join("; ")}`,
+    );
+    this.name = "ChainExhaustedError";
+    this.failures = failures;
+  }
+}
+
+export interface TryChainOptions<T> {
+  /** Makes the actual call for one link. Throw to demote to the next link. */
+  attempt: (link: Link) => Promise<T>;
+  /** Records one success or one failure for the WHOLE walk, not per link. */
+  health?: HealthTracker;
+  /** Called on each link's failure, before moving to the next — e.g. to log it. */
+  onLinkFailure?: (link: Link, error: unknown) => void;
+}
+
+/**
+ * Try each link in order; return the first success.
+ *
+ * Health is recorded once per call — a success on link two is still a success
+ * for the app, and a health check that flagged it "degraded" because the FIRST
+ * link failed would be reporting its own fallback working as a problem.
+ *
+ * Throws `ChainExhaustedError` (carrying every link's failure) when none
+ * succeed, so a caller can log exactly what was tried rather than only the
+ * last error — the failure that matters is often not the last one.
+ */
+export async function tryChain<T>(chain: Link[], options: TryChainOptions<T>): Promise<T> {
+  const failures: ChainAttemptFailure[] = [];
+
+  for (const link of chain) {
+    try {
+      const result = await options.attempt(link);
+      options.health?.recordSuccess();
+      return result;
+    } catch (error) {
+      const message = error instanceof Error ? error.message : String(error);
+      failures.push({ link, message });
+      options.onLinkFailure?.(link, error);
+    }
+  }
+
+  const exhausted = new ChainExhaustedError(failures);
+  options.health?.recordFailure(exhausted);
+  throw exhausted;
+}

src/health.ts

@@ -0,0 +1,92 @@
+/**
+ * Observed health of an AI feature — was the last generation actually usable?
+ *
+ * This exists because of a failure that "the chain never fails" hides just as
+ * well as a single pin does. On 2026-08-28 an app's only configured key
+ * started returning 401 and the chain (correctly) had nowhere else to go — the
+ * routes caught the error and answered HTTP 200 with an apology, and the
+ * app's own `/health` reported "healthy" because it only ever checked the
+ * database. A total outage of the product's core feature was invisible to
+ * every automated check, for as long as nobody happened to try it by hand.
+ *
+ * A tracker records what actually happened, so a health route can report it
+ * and a strict check can refuse to call the app "up" while its AI is down.
+ *
+ * FACTORY, NOT A SINGLETON. A module-level global would force a shared state
+ * shape on every consumer and make the transitions untestable without mutating
+ * process-wide state between tests. `createHealthTracker()` returns an
+ * isolated instance — a single-process app gets the old singleton behaviour
+ * for free by creating exactly one and exporting it from its own module:
+ *
+ *     // lib/llm-health.ts
+ *     export const llmHealth = createHealthTracker();
+ *
+ * If the app scales horizontally, this state is per-instance and wants a
+ * shared store — that migration is app-specific and out of scope here.
+ */
+
+export type HealthStatus = "unknown" | "ok" | "degraded" | "down";
+
+export interface Health {
+  status: HealthStatus;
+  consecutiveFailures: number;
+  lastError: string | null;
+  /** Epoch milliseconds. Format at the API boundary, not here. */
+  lastSuccessAt: number | null;
+  /** Epoch milliseconds. Format at the API boundary, not here. */
+  lastFailureAt: number | null;
+}
+
+export interface HealthTrackerOptions {
+  /** Consecutive failures before status flips from "degraded" to "down". Default 3. */
+  downAfter?: number;
+  /** Clock injection point for tests. Default `Date.now`. */
+  now?: () => number;
+}
+
+export interface HealthTracker {
+  /** Call after a generation that produced usable content. */
+  recordSuccess(): void;
+  /** Call when generation threw, or returned nothing usable. */
+  recordFailure(error: unknown): void;
+  getHealth(): Health;
+  /** Test seam — also useful for an app-triggered "recheck now". */
+  reset(): void;
+}
+
+export function createHealthTracker(options: HealthTrackerOptions = {}): HealthTracker {
+  const downAfter = options.downAfter ?? 3;
+  const now = options.now ?? Date.now;
+
+  let consecutiveFailures = 0;
+  let lastError: string | null = null;
+  let lastSuccessAt: number | null = null;
+  let lastFailureAt: number | null = null;
+
+  return {
+    recordSuccess() {
+      consecutiveFailures = 0;
+      lastError = null;
+      lastSuccessAt = now();
+    },
+    recordFailure(error: unknown) {
+      consecutiveFailures += 1;
+      lastFailureAt = now();
+      lastError = error instanceof Error ? error.message : String(error ?? "unknown error");
+    },
+    getHealth(): Health {
+      let status: HealthStatus;
+      if (consecutiveFailures >= downAfter) status = "down";
+      else if (consecutiveFailures > 0) status = "degraded";
+      else if (lastSuccessAt !== null) status = "ok";
+      else status = "unknown";
+      return { status, consecutiveFailures, lastError, lastSuccessAt, lastFailureAt };
+    },
+    reset() {
+      consecutiveFailures = 0;
+      lastError = null;
+      lastSuccessAt = null;
+      lastFailureAt = null;
+    },
+  };
+}

src/index.ts

@@ -3,17 +3,21 @@
  *
  * WHAT IT IS FOR
  * --------------
- * An app that wants an AI feature needs four unrelated-looking decisions to go
- * right, and getting any one wrong looks identical from the outside: the
- * assistant is broken. This package holds all four, so adding AI is one
- * decision instead of four.
+ * An app that wants an AI feature needs several unrelated-looking decisions to
+ * go right, and getting any one wrong looks identical from the outside: the
+ * assistant is broken. This package holds all of them, so adding AI is one
+ * decision instead of many.
  *
  *   which model  — a fallback list ACROSS VENDORS, because a single pinned free
  *                  model is a scheduled outage, and a smaller model at the same
  *                  vendor draws on the same exhausted daily budget.
  *   still there? — has the vendor retired an id we still ask for? The list is
  *                  itself a list of pins, so it rots too. Zero tokens, so it can
  *                  run on a schedule instead of being remembered.
+ *   walk it      — a chain nobody walks is a list, not a fallback. `tryChain`
+ *                  tries each link and stops at the first success; a
+ *                  `HealthTracker` records whether the WHOLE chain came back
+ *                  empty, so a health route can say so before a user does.
  *   too fast?    — tell the three kinds of 429 apart. They share a status code
  *                  and need opposite responses; only the body distinguishes them.
  *   who gets it  — divide a fixed daily pool across active users, so the person
@@ -32,9 +36,11 @@
  *
  * STILL NOT INCLUDED: an HTTP client. Every app has its own calling conventions,
  * retries and logging, and replacing those is a rewrite rather than an adoption.
- * This supplies the decisions; the caller keeps the fetch. That rule is under
- * review — `ai-forms`, the most-adopted package in this fleet, is the one that
- * broke it by shipping a route factory and a hook.
+ * This supplies the decisions; the caller keeps the fetch — `tryChain` is an
+ * orchestrator, not a client: the caller's own `attempt` function makes the
+ * actual request. That rule is under review — `ai-forms`, the most-adopted
+ * package in this fleet, is the one that broke it by shipping a route factory
+ * and a hook.
  */
 
 export {
@@ -62,6 +68,21 @@ export {
   catalogReport,
 } from "./catalog.js";
 
+export {
+  type ChainAttemptFailure,
+  type TryChainOptions,
+  ChainExhaustedError,
+  tryChain,
+} from "./attempt.js";
+
+export {
+  type HealthStatus,
+  type Health,
+  type HealthTrackerOptions,
+  type HealthTracker,
+  createHealthTracker,
+} from "./health.js";
+
 export {
   type RateLimitKind,
   classifyRateLimit,