llama component example

llama.cpp, running in the browser as a WebAssembly component, driving the GPU through wasi:webgpu.

How it works

llama.cpp has no knowledge of the web platform. It is compiled against WASI interfaces (wasi:webgpu, wasi:filesystem, …) and packaged as a component behind a small WIT API. jco then transpiles that component into an ES module:

import { api } from './generated/llama.js';
const { Model, Context, Sampler } = api;

const model = await new Model(bytes);
const tokens = await model.tokenize(prompt, true);
const token = await sampler.sample(ctx);

Jco derives classes, promises, typed arrays, etc. from the WIT types, so the API is idiomatic JavaScript rather than a C FFI. The JavaScript side never touches C++, the C++ side never touches JavaScript, and the entire contract between them is a few lines of WIT.

Build & run

npm install
npm run fetch-llama-wit   # pull the compiled component from ghcr.io
npm run build             # generate types + transpile to src/generated
npx http-server .         # any static file server works

The app depends on WebGPU and JSPI.

Firefox is slow

Firefox polls GPU devices on a 100ms timer, so every GPU sync costs ~96ms. Fix in progress upstream (bugzilla#1870699).

WIP

• wasi:webgpu for preview 3 is not merged yet: wasi-gfx/wasi-webgpu-headers#34, demo branch at MendyBerger/wasi-webgpu-headers.
• dawn_wasi_webgpu_cpp: Dawn's C++ WebGPU wrapper adapted to sit on top of webgpu.h over wasi:webgpu. A standard C++ wrapper in webgpu-headers (webgpu-native/webgpu-headers#596) would make this adaptation unnecessary.
• The WIT API itself is a moving target: cosmonic-labs/llama.wit, branch wasm-cg-demo.
• The wasi:webgpu browser shim in src/shims/ is vendored for now; its home will be wasi-gfx/wasi-gfx-shim.
• The remaining WASI shims (wasi:cli, wasi:clocks, wasi:random, wasi:filesystem) are to be upstreamed into jco.