llmkit-rs

Unified multi-provider async LLM client for Rust. One trait, one streaming API, one Tower middleware stack — across OpenAI, Anthropic, and local Ollama models, with zero provider lock-in.

[dependencies]
llmkit-rs = "0.1"   # published as `llmkit-rs`; imported as `llmkit`
tokio = { version = "1", features = ["full"] }

Quick start

use llmkit::prelude::*;
use std::time::Duration;

#[tokio::main]
async fn main() -> LlmResult<()> {
    let client = LlmClientBuilder::new()
        .provider(AnthropicProvider::from_env()?.model("claude-opus-4-8"))
        .fallback(OpenAiProvider::from_env()?.model("gpt-4o-mini"))
        .layer(TracingLayer::new())
        .layer(CostTrackingLayer::with_budget(10.00))
        .layer(RateLimitLayer::token_bucket(60_000, Duration::from_secs(60)))
        .layer(RetryLayer::exponential(3, Duration::from_millis(200)))
        .build()?;

    let resp = client
        .chat(ChatRequest::builder().user("What's the weather in Karachi?").build())
        .await?;

    println!("{}", resp.text().unwrap_or_default());
    Ok(())
}

Features

• Multi-provider — OpenAI (GPT-4o, o1), Anthropic (Claude), Ollama (local Llama/Mistral)
• Unified LlmProvider trait — one call site regardless of backend
• Streaming via tokio async Stream, with provider-specific SSE/NDJSON normalised into one StreamDelta
• Type-safe tools — #[derive(ToolSchema)] generates JSON Schema; automatic multi-turn tool loop
• Embeddings normalised across providers
• Tower middleware — retry (exponential backoff), rate limiting (token bucket), cost tracking (per-request + session budget), tracing
• Provider fallback — primary → secondary on error/timeout
• Model aliasing — "fast" → gpt-4o-mini, "smart" → claude-sonnet-4-6
• Configurable connection pooling and timeouts via reqwest

Workspace

Crate	Purpose
llmkit	Public facade: builder, client, re-exports
llmkit-core	Traits, types, errors — no I/O
llmkit-openai	OpenAI adapter
llmkit-anthropic	Anthropic Messages API adapter
llmkit-ollama	Ollama local adapter
llmkit-tower	Tower middleware layers
llmkit-macros	#[derive(ToolSchema)]

Most users only depend on llmkit. Provider adapters are feature-gated (openai, anthropic, ollama; all on by default).

Examples

ANTHROPIC_API_KEY=... cargo run --example basic_chat
ANTHROPIC_API_KEY=... cargo run --example streaming
ANTHROPIC_API_KEY=... cargo run --example tool_calling
cargo run --example multi_provider     # needs OPENAI_API_KEY + ANTHROPIC_API_KEY
cargo run --example cost_tracking

License

Licensed under either of Apache-2.0 or MIT at your option.