Skip to content

Llm OpenAI shape #581

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
fretchen merged 3 commits into from
Jul 28, 2026
Merged

Llm OpenAI shape #581

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
216b4a3
try the switch
fretchen Jul 28, 2026
28b3048
Update sc_llm_x402_buyer.ipynb
fretchen Jul 28, 2026
4c368ce
Clean up
fretchen Jul 28, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion scw_js/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,7 @@ Generates AI images using Black Forest Labs API with USDC payment via x402 proto

LLM chat paid via x402 batch-settlement USDC payment channels — no bearer token, the payment voucher itself proves wallet control. `llmx402` handles chat requests (deposit/voucher/402 flow); `llmx402cron` claims and settles accumulated channels on a 12h schedule. See [`assistent_plan.md`](../assistent_plan.md) at the repo root for the full design record (deposit/voucher/claim/settle lifecycle, pricing model, gotchas).

**`llm/v1` contract.** [`openapi.llm.json`](./openapi.llm.json) declares `x-service-type: "llm/v1"` — the interchangeable-agent contract. It is defined entirely by that document: the request/response schema (`LLMChatRequest`/`LLMChatResponse`) plus the `x-interop-floor` (a compatible agent must advertise ≥1 `accepts[]` entry with USDC on Base `eip155:8453`, scheme `batch-settlement`). A `v2` aligning to the OpenAI chat-completions shape is the intended evolution, not yet built.
**`llm/v1` contract.** [`openapi.llm.json`](./openapi.llm.json) declares `x-service-type: "llm/v1"` — the interchangeable-agent contract. It is defined entirely by that document: the request/response body is the **OpenAI chat-completions shape** (`{ model, messages }` in → an OpenAI `chat.completion` object out — `LLMChatRequest`/`LLMChatResponse`), plus the `x-interop-floor` (a compatible agent must advertise ≥1 `accepts[]` entry with USDC on Base `eip155:8453`, scheme `batch-settlement`). Payment stays x402 batch-settlement, so the OpenAI shape is for **body legibility, not drop-in OpenAI-SDK use** — a stock SDK sends `Authorization: Bearer` and can't satisfy the 402. Streaming is not supported. `model` is validated against the advertised id(s) (`mistral-large-latest`; more can be added later).

### `growth_api.ts` - Growth Agent Draft Approval

Expand Down
99 changes: 81 additions & 18 deletions scw_js/llm_service.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,14 +49,52 @@ export interface LLMMessage {
content: string;
}

interface LLMResponse {
content: string;
usage: {
prompt_tokens: number;
completion_tokens: number;
total_tokens: number;
};
export interface LLMUsage {
prompt_tokens: number;
completion_tokens: number;
total_tokens: number;
}

/**
* An OpenAI chat-completion response object. Mistral (and the other OpenAI-compatible
* upstreams in LLM_PROVIDERS) already return this shape, so we forward it largely as-is
* rather than unwrapping it — the endpoint's public contract is the OpenAI chat-completions
* shape. `usage` is load-bearing: the batch-settlement handler prices the settled amount
* from it (see getSettleAmount in sc_llm_x402.ts).
*/
export interface LLMResponse {
id: string;
object: "chat.completion";
created: number;
model: string;
choices: Array<{
index: number;
message: { role: string; content: string };
finish_reason: string | null;
}>;
usage: LLMUsage;
}

/**
* Resolve an OpenAI-style `model` id (as sent by a caller in the request body) to the
* internal provider whose config serves it. Returns `null` for an unknown/unsupported model
* so the handler can answer with an OpenAI-shaped `model_not_found` error. Today only
* Mistral's `mistral-large-latest` is advertised; exposing IONOS later is a one-line change
* (add its `defaultModel` here) — the registry (LLM_PROVIDERS) already carries the ids.
*/
export function resolveModel(modelId: string): { provider: string } | null {
for (const [provider, config] of Object.entries(LLM_PROVIDERS)) {
if (config.defaultModel === modelId) {
return { provider };
}
}
return null;
}

/** The model ids this endpoint currently advertises + serves (for OpenAPI enum + validation). */
export function advertisedModelIds(): string[] {
// Only Mistral is live today (see sc_llm_x402.ts's LLM_PROVIDER); IONOS stays internal.
return [LLM_PROVIDERS.mistral.defaultModel];
}

export async function callLLMAPI(
Expand All @@ -66,9 +104,18 @@ export async function callLLMAPI(
): Promise<LLMResponse> {
if (dummy) {
return {
content: "I am a placeholder for the LLM response",
id: "chatcmpl-mock",
object: "chat.completion",
created: Math.floor(Date.now() / 1000),
model: "placeholder-model",
choices: [
{
index: 0,
message: { role: "assistant", content: "I am a placeholder for the LLM response" },
finish_reason: "stop",
},
],
usage: { prompt_tokens: 5, completion_tokens: 15, total_tokens: 15 },
model: "placeholder model",
};
}
const config = getLLMProviderConfig(provider);
Expand Down Expand Up @@ -104,19 +151,35 @@ export async function callLLMAPI(
);
}

const data = (await response.json()) as {
choices: Array<{ message: { content: string } }>;
usage: LLMResponse["usage"];
model: string;
// Upstream is OpenAI-compatible; forward its chat-completion envelope, synthesizing only
// the fields a given provider might omit (id/created/object) so our response is a
// well-formed OpenAI chat.completion regardless of upstream quirks.
const data = (await response.json()) as Partial<LLMResponse> & {
choices?: Array<{
index?: number;
message: { role?: string; content: string };
finish_reason?: string | null;
}>;
usage?: LLMUsage;
model?: string;
};
const firstChoice = data.choices[0];
if (!firstChoice) {
throw new Error(`LLM API returned empty choices (model: ${data.model})`);
const firstChoice = data.choices?.[0];
if (!firstChoice || !data.usage) {
throw new Error(
`LLM API returned an incomplete completion (model: ${data.model ?? config.defaultModel})`,
);
}
return {
content: firstChoice.message.content,
id: data.id ?? `chatcmpl-${Date.now()}`,
object: "chat.completion",
created: data.created ?? Math.floor(Date.now() / 1000),
model: data.model ?? config.defaultModel,
choices: data.choices!.map((c, i) => ({
index: c.index ?? i,
message: { role: c.message.role ?? "assistant", content: c.message.content },
finish_reason: c.finish_reason ?? "stop",
})),
usage: data.usage,
model: data.model,
};
}

Expand Down
162 changes: 93 additions & 69 deletions scw_js/notebooks/sc_llm_x402_buyer.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -106,15 +106,15 @@
},
{
"cell_type": "code",
"execution_count": null,
"execution_count": 2,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"🚨 REAL MONEY on Base Mainnet\n",
" eip155:8453 • USDC USD Coin @ 0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913\n",
"🧪 Base Sepolia (Testnet)\n",
" eip155:84532 • USDC USDC @ 0x036CbD53842c5426634e7929541eC2318f3dCF7e\n",
" Service: http://localhost:8085\n"
]
}
Expand Down Expand Up @@ -174,9 +174,17 @@
},
{
"cell_type": "code",
"execution_count": null,
"execution_count": 3,
"metadata": {},
"outputs": [],
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"✅ buyer client ready, registered for eip155:84532\n"
]
}
],
"source": [
"import { x402Client, wrapFetchWithPayment, x402HTTPClient } from \"npm:@x402/fetch@^2.17.0\";\n",
"import { toClientEvmSigner } from \"npm:@x402/evm@^2.17.0\";\n",
Expand Down Expand Up @@ -263,10 +271,19 @@
},
{
"cell_type": "code",
"execution_count": null,
"execution_count": 4,
"id": "aae724d6",
"metadata": {},
"outputs": [],
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"💵 Buyer USDC: 18.6445 (estimated deposit ≈ 0.5)\n",
" ✅ enough USDC to open the channel\n"
]
}
],
"source": [
"import { formatUnits } from \"npm:viem@2\";\n",
"\n",
Expand Down Expand Up @@ -330,36 +347,27 @@
"name": "stdout",
"output_type": "stream",
"text": [
"📡 status=200 elapsed=12901ms\n",
"📨 body: {\n",
" \"content\": \"The capital of France is **Paris**.\",\n",
" \"usage\": {\n",
" \"prompt_tokens\": 10,\n",
" \"total_tokens\": 19,\n",
" \"completion_tokens\": 9,\n",
" \"prompt_tokens_details\": {\n",
" \"cached_tokens\": 0\n",
" }\n",
" },\n",
" \"model\": \"mistral-large-latest\"\n",
"}\n",
"📡 status=200 elapsed=3418ms\n",
"📨 reply: I am a placeholder for the LLM response\n",
"🧾 settlement receipt: {\n",
" \"success\": true,\n",
" \"payer\": \"0x553179556FC2A39e535D65b921e01fA995E79101\",\n",
" \"transaction\": \"0xcf5f89bae9626c199793dd75fef91afc7b097050d674347bc1b7b851c19c10ff\",\n",
" \"network\": \"eip155:8453\",\n",
" \"payer\": \"0x553179556fc2a39e535d65b921e01fa995e79101\",\n",
" \"transaction\": \"\",\n",
" \"network\": \"eip155:84532\",\n",
" \"amount\": \"\",\n",
" \"extra\": {\n",
" \"channelState\": {\n",
" \"channelId\": \"0xbd47699c24d5fbd16eb316de548ab39e36931a953ee33aa30dc9b28ff5507446\",\n",
" \"balance\": \"15000\",\n",
" \"channelId\": \"0x6697fd8be2d5899f6799d31c5e70411db50f1b5b0c0a0f7cbe76c4bb4498a517\",\n",
" \"balance\": \"0\",\n",
" \"totalClaimed\": \"0\",\n",
" \"withdrawRequestedAt\": 0,\n",
" \"refundNonce\": \"0\",\n",
" \"chargedCumulativeAmount\": \"18\"\n",
" \"chargedCumulativeAmount\": \"130\"\n",
" },\n",
" \"chargedAmount\": \"18\"\n",
" \"chargedAmount\": \"25\"\n",
" }\n",
"}\n"
"}\n",
"✅ msg1: OpenAI reply received (39 chars), usage present\n"
]
}
],
Expand All @@ -378,15 +386,17 @@
" // failure either way). Keeping this try/catch as defense-in-depth, not because the crash is\n",
" // expected anymore.\n",
" //\n",
" // No `useDummyData` field — matches the real client (useX402Chat.ts) exactly. Mock-vs-real\n",
" // is decided entirely server-side by NETWORK (see the network-selection cell): testnet always\n",
" // mocks, mainnet always calls the real Mistral API. There is nothing else to set here.\n",
" // OpenAI chat-completions request body ({ model, messages }) — matches the real client\n",
" // (useX402Chat.ts) exactly. `model` must be one the server advertises (mistral-large-latest).\n",
" // No `useDummyData` field: mock-vs-real is decided entirely server-side by NETWORK (see the\n",
" // network-selection cell): testnet always mocks, mainnet always calls the real Mistral API.\n",
" // The response is a standard OpenAI chat.completion — the reply is body.choices[0].message.content.\n",
" let response: Response;\n",
" try {\n",
" response = await fetchWithPayment(SERVICE_URL, {\n",
" method: \"POST\",\n",
" headers: { \"Content-Type\": \"application/json\" },\n",
" body: JSON.stringify({ data: { prompt: [{ role: \"user\", content }] } }),\n",
" body: JSON.stringify({ model: \"mistral-large-latest\", messages: [{ role: \"user\", content }] }),\n",
" });\n",
" } catch (err) {\n",
" const elapsedMs = Math.round(performance.now() - started);\n",
Expand All @@ -401,7 +411,7 @@
" try { body = JSON.parse(bodyText); } catch { body = bodyText; }\n",
"\n",
" console.log(`📡 status=${response.status} elapsed=${elapsedMs}ms`);\n",
" console.log(\"📨 body:\", JSON.stringify(body, null, 2).slice(0, 600));\n",
" console.log(\"📨 reply:\", body?.choices?.[0]?.message?.content ?? JSON.stringify(body, null, 2).slice(0, 600));\n",
"\n",
" let receipt: unknown = null;\n",
" try {\n",
Expand All @@ -414,7 +424,28 @@
" return { response, body, receipt, elapsedMs, threw: null };\n",
"}\n",
"\n",
"const msg1 = await sendChatMessage(\"What is the capital of France?\");"
"// Minimal viable check (not a full test suite — a notebook `throw` fails the cell loudly).\n",
"// Asserts the three things this notebook otherwise only eyeballs: (1) the x402 flow succeeded\n",
"// (HTTP 200, not a 402/500), (2) the message round-tripped (a non-empty reply came back), and\n",
"// (3) the response is the correct OpenAI chat.completion shape with `usage` present (the field\n",
"// settlement is priced from). On testnet the reply is the server's placeholder mock, so this\n",
"// proves shape + transmission + payment, NOT real inference quality (that costs mainnet money).\n",
"function assertOk(label: string, r: { response: Response | null; body: any }) {\n",
" if (!r.response || r.response.status !== 200) {\n",
" throw new Error(`${label}: expected HTTP 200, got ${r.response?.status ?? \"no response\"}`);\n",
" }\n",
" const reply = r.body?.choices?.[0]?.message?.content;\n",
" if (typeof reply !== \"string\" || reply.length === 0) {\n",
" throw new Error(`${label}: no OpenAI-shaped reply (choices[0].message.content) in body`);\n",
" }\n",
" if (!r.body?.usage) {\n",
" throw new Error(`${label}: response missing 'usage' (batch-settlement pricing depends on it)`);\n",
" }\n",
" console.log(`✅ ${label}: OpenAI reply received (${reply.length} chars), usage present`);\n",
"}\n",
"\n",
"const msg1 = await sendChatMessage(\"What is the capital of France?\");\n",
"assertOk(\"msg1\", msg1);"
]
},
{
Expand Down Expand Up @@ -442,63 +473,56 @@
"name": "stdout",
"output_type": "stream",
"text": [
"📡 status=200 elapsed=5345ms\n",
"📨 body: {\n",
" \"content\": \"The capital of Germany is **Berlin**. It has been the capital since the reunification of Germany in 1990, following the fall of the Berlin Wall in 1989. Before that, Bonn served as the capital of West Germany (Federal Republic of Germany) during the Cold War.\",\n",
" \"usage\": {\n",
" \"prompt_tokens\": 11,\n",
" \"total_tokens\": 74,\n",
" \"completion_tokens\": 63,\n",
" \"prompt_tokens_details\": {\n",
" \"cached_tokens\": 0\n",
" }\n",
" },\n",
" \"model\": \"mistral-large-latest\"\n",
"}\n",
"📡 status=200 elapsed=5216ms\n",
"📨 reply: I am a placeholder for the LLM response\n",
"🧾 settlement receipt: {\n",
" \"success\": true,\n",
" \"payer\": \"0x553179556fc2a39e535d65b921e01fa995e79101\",\n",
" \"transaction\": \"\",\n",
" \"network\": \"eip155:8453\",\n",
" \"amount\": \"\",\n",
" \"payer\": \"0x553179556FC2A39e535D65b921e01fA995E79101\",\n",
" \"transaction\": \"0xbb6abd3e35362ed66de255294e830a2db1139836ae1357c3f59e06b218c99da5\",\n",
" \"network\": \"eip155:84532\",\n",
" \"extra\": {\n",
" \"channelState\": {\n",
" \"channelId\": \"0xbd47699c24d5fbd16eb316de548ab39e36931a953ee33aa30dc9b28ff5507446\",\n",
" \"balance\": \"0\",\n",
" \"totalClaimed\": \"0\",\n",
" \"channelId\": \"0x6697fd8be2d5899f6799d31c5e70411db50f1b5b0c0a0f7cbe76c4bb4498a517\",\n",
" \"balance\": \"1507100\",\n",
" \"totalClaimed\": \"30\",\n",
" \"withdrawRequestedAt\": 0,\n",
" \"refundNonce\": \"0\",\n",
" \"chargedCumulativeAmount\": \"118\"\n",
" \"chargedCumulativeAmount\": \"155\"\n",
" },\n",
" \"chargedAmount\": \"100\"\n",
" \"chargedAmount\": \"25\"\n",
" }\n",
"}\n",
"📡 status=200 elapsed=23975ms\n",
"📨 body: {\n",
" \"content\": \"Italy is a fascinating country with a rich history, vibrant culture, and significant global influence. Here’s a quick overview of key aspects:\\n\\n### **1. Geography & Regions**\\n- **Location**: Southern Europe, shaped like a boot, surrounded by the Mediterranean Sea (Adriatic, Ionian, Tyrrhenian, and Ligurian Seas).\\n- **Regions**: 20 regions, including iconic ones like **Tuscany** (Florence, Siena), **Lombardy** (Milan), **Veneto** (Venice), **Lazio** (Rome), **Campania** (Naples, Pompeii), and **Sicily** (Palermo).\\n- **Landmarks**: The Alps (north), Apennine Mountains (spine\n",
"✅ msg2: OpenAI reply received (39 chars), usage present\n",
"📡 status=200 elapsed=1741ms\n",
"📨 reply: I am a placeholder for the LLM response\n",
"🧾 settlement receipt: {\n",
" \"success\": true,\n",
" \"payer\": \"0x553179556FC2A39e535D65b921e01fA995E79101\",\n",
" \"transaction\": \"0xebadfc6c97ee7988d69cb1c696649d96053f9a63f8da4ae7058ddf8ca798b673\",\n",
" \"network\": \"eip155:8453\",\n",
" \"payer\": \"0x553179556fc2a39e535d65b921e01fa995e79101\",\n",
" \"transaction\": \"\",\n",
" \"network\": \"eip155:84532\",\n",
" \"amount\": \"\",\n",
" \"extra\": {\n",
" \"channelState\": {\n",
" \"channelId\": \"0xbd47699c24d5fbd16eb316de548ab39e36931a953ee33aa30dc9b28ff5507446\",\n",
" \"balance\": \"30000\",\n",
" \"totalClaimed\": \"0\",\n",
" \"channelId\": \"0x6697fd8be2d5899f6799d31c5e70411db50f1b5b0c0a0f7cbe76c4bb4498a517\",\n",
" \"balance\": \"1507100\",\n",
" \"totalClaimed\": \"30\",\n",
" \"withdrawRequestedAt\": 0,\n",
" \"refundNonce\": \"0\",\n",
" \"chargedCumulativeAmount\": \"1871\"\n",
" \"chargedCumulativeAmount\": \"180\"\n",
" },\n",
" \"chargedAmount\": \"1753\"\n",
" \"chargedAmount\": \"25\"\n",
" }\n",
"}\n"
"}\n",
"✅ msg3: OpenAI reply received (39 chars), usage present\n"
]
}
],
"source": [
"const msg2 = await sendChatMessage(\"And what is the capital of Germany?\");\n",
"const msg3 = await sendChatMessage(\"And Italy?\");"
"assertOk(\"msg2\", msg2);\n",
"\n",
"const msg3 = await sendChatMessage(\"And Italy?\");\n",
"assertOk(\"msg3\", msg3);"
]
},
{
Expand Down
Loading
Loading