diff --git a/README.md b/README.md index 2c59b83c..34030364 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,35 @@ # inference-cache -A Kubernetes-native cache plane for LLM inference. +A **vendor-neutral, Kubernetes-native cache-policy control plane for LLM inference.** + +inference-cache makes routing **cache-aware**: it tracks which replica already holds a +prompt's prefix warm and returns that as a routing *hint*, so a gateway can reuse +KV/prefill instead of recomputing it — cutting time-to-first-token and cost. It +**orchestrates** existing KV-cache technology (LMCache, Mooncake); it is **not** a new +distributed cache and **not** the data-plane gateway. Guiding principle — **"we decide +routing; the gateway follows"**: all routing intelligence lives in the server, and the +gateway simply tokenizes → calls `LookupRoute` → routes to the returned replica → +round-robins on `NO_HINT`. + +**One request, end to end:** an engine serves a prompt and emits KV-cache events → a +subscriber sidecar reports them to the server → the in-memory index records +`(tenant, model, hash_scheme, prefix_hash) → replica` → a later `LookupRoute` returns the +ranked replicas holding that prefix warm. The plane moves *metadata about where the KV +lives* — never KV tensors or prompt text. See [`docs/concepts/`](docs/concepts/) for depth. + +## Engines and backends + +The `(engine, backend)` pair selects a runtime adapter, so supporting a new engine or +cache backend is adding an adapter — the core API and gRPC contract don't change. + +- **Supported today:** the **vLLM** engine with a managed **LMCache** backend (the + default; validated end-to-end), or an **External** backend (bring your own endpoint via + `spec.type: External` + `spec.externalEndpoint`). +- **In active development:** the **SGLang** engine (LMCache multiprocess mode) and the + **Mooncake** backend (a peer-to-peer transfer-engine mesh). + +`spec.integration.engine` is the runtime id (`vllm` default, or `sglang`); `spec.type` is +the backend (`LMCache` default, `External`, `Mooncake`, …). ## Repository layout @@ -8,7 +37,7 @@ One operator, split across two control-plane binaries (controller + server) plus the operator CLI and the CRDs. **CRDs — the API** -- `api/v1alpha1/` — Go types (`CacheBackend`; `CachePolicy`, `CacheTenant`, `PromptTemplate`, `PDTopology`, `CacheIndex` as they land) + generated deepcopy +- `api/v1alpha1/` — Go types for the six v1alpha1 CRDs (`CacheBackend`, `CachePolicy`, `CacheTenant`, `CacheIndex`, `PromptTemplate`, `PDTopology`) + generated deepcopy - `config/` — generated CRD, RBAC, and sample manifests **`inferencecache-controller`** (`cmd/controller`) — watches CRDs and provisions cache backends @@ -64,9 +93,10 @@ an operator who forgets the flag from accidentally shipping unauthenticated ## Cluster Prerequisites The controller serves admission webhooks — defaulting + validation for -`CacheBackend`, plus a mutating Pod webhook that auto-injects the LMCache -engine configuration into pods labeled to match a `CacheBackend`'s -`spec.engineSelector` — over TLS, so deploying it requires [cert-manager][cm] +`CacheBackend`, plus a mutating Pod webhook that auto-injects the cache +engine configuration (chosen by the matched runtime adapter) into pods +labeled to match a `CacheBackend`'s `spec.engineSelector` — over TLS, so +deploying it requires [cert-manager][cm] v1.0+ in the target cluster. The default install (`config/default`) provisions a self-signed `Issuer` plus a `Certificate` for the webhook serving cert, and relies on cert-manager's `cert-manager.io/inject-ca-from` @@ -227,11 +257,14 @@ make dev-cluster KIND_CLUSTER=cache-dev KIND_NODE_IMAGE=kindest/node:v1.31.0 ## Documentation -Design docs live under [`docs/`](docs/): +Docs live under [`docs/`](docs/): -- [`docs/design/cachebackend-api.md`](docs/design/cachebackend-api.md) — the `CacheBackend` CRD contract -- [`docs/design/grpc-contract.md`](docs/design/grpc-contract.md) — the `InferenceCache` gRPC service contract (B4) -- [`docs/design/policy-crds.md`](docs/design/policy-crds.md) — policy CRDs (`CachePolicy`, `CacheTenant`, `PromptTemplate`, `PDTopology`, `CacheIndex`) +- **Concepts** — [`docs/concepts/`](docs/concepts/): engine binding, engine overrides, policy tuning, tenant identity/quota, and the CacheIndex cluster aggregate. +- **Design** — [`docs/design/`](docs/design/): the [`CacheBackend` CRD contract](docs/design/cachebackend-api.md), the [`InferenceCache` gRPC contract](docs/design/grpc-contract.md), [gRPC TLS](docs/design/grpc-tls.md), [policy CRDs](docs/design/policy-crds.md), [LookupRoute ranking](docs/design/lookuproute-ranking.md), and [KV-event subscriber wiring](docs/design/kvevent-subscriber-wiring.md). +- **Reference** — [`docs/reference/`](docs/reference/): the [metric surface](docs/reference/metrics.md) and gRPC [reason codes](docs/reference/reason-codes.md). +- **Operations & observability** — [`docs/operations/`](docs/operations/), the alert runbooks in [`docs/observability/`](docs/observability/), and the [`inferencecache doctor`](docs/cli/doctor.md) preflight CLI. +- **Reference stack** — [`docs/reference-stack/`](docs/reference-stack/): runnable manifests + a GPU runbook for a full vLLM/LMCache stack. +- **Quick start** — [`docs/quickstart.md`](docs/quickstart.md). Contributor guide: [`CONTRIBUTING.md`](CONTRIBUTING.md) (layout, naming rule, push/PR gates).