Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
51 changes: 42 additions & 9 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,14 +1,43 @@
# inference-cache

A Kubernetes-native cache plane for LLM inference.
A **vendor-neutral, Kubernetes-native cache-policy control plane for LLM inference.**

inference-cache makes routing **cache-aware**: it tracks which replica already holds a
prompt's prefix warm and returns that as a routing *hint*, so a gateway can reuse
KV/prefill instead of recomputing it — cutting time-to-first-token and cost. It
**orchestrates** existing KV-cache technology (LMCache, Mooncake); it is **not** a new
distributed cache and **not** the data-plane gateway. Guiding principle — **"we decide
routing; the gateway follows"**: all routing intelligence lives in the server, and the
gateway simply tokenizes → calls `LookupRoute` → routes to the returned replica →
round-robins on `NO_HINT`.

**One request, end to end:** an engine serves a prompt and emits KV-cache events → a
subscriber sidecar reports them to the server → the in-memory index records
`(tenant, model, hash_scheme, prefix_hash) → replica` → a later `LookupRoute` returns the
ranked replicas holding that prefix warm. The plane moves *metadata about where the KV
lives* — never KV tensors or prompt text. See [`docs/concepts/`](docs/concepts/) for depth.

## Engines and backends

The `(engine, backend)` pair selects a runtime adapter, so supporting a new engine or
cache backend is adding an adapter — the core API and gRPC contract don't change.

- **Supported today:** the **vLLM** engine with a managed **LMCache** backend (the
default; validated end-to-end), or an **External** backend (bring your own endpoint via
`spec.type: External` + `spec.externalEndpoint`).
- **In active development:** the **SGLang** engine (LMCache multiprocess mode) and the
**Mooncake** backend (a peer-to-peer transfer-engine mesh).

`spec.integration.engine` is the runtime id (`vllm` default, or `sglang`); `spec.type` is
the backend (`LMCache` default, `External`, `Mooncake`, …).

## Repository layout

One operator, split across two control-plane binaries (controller + server) plus
the operator CLI and the CRDs.

**CRDs — the API**
- `api/v1alpha1/` — Go types (`CacheBackend`; `CachePolicy`, `CacheTenant`, `PromptTemplate`, `PDTopology`, `CacheIndex` as they land) + generated deepcopy
- `api/v1alpha1/` — Go types for the six v1alpha1 CRDs (`CacheBackend`, `CachePolicy`, `CacheTenant`, `CacheIndex`, `PromptTemplate`, `PDTopology`) + generated deepcopy
- `config/` — generated CRD, RBAC, and sample manifests

**`inferencecache-controller`** (`cmd/controller`) — watches CRDs and provisions cache backends
Expand Down Expand Up @@ -64,9 +93,10 @@ an operator who forgets the flag from accidentally shipping unauthenticated
## Cluster Prerequisites

The controller serves admission webhooks — defaulting + validation for
`CacheBackend`, plus a mutating Pod webhook that auto-injects the LMCache
engine configuration into pods labeled to match a `CacheBackend`'s
`spec.engineSelector` — over TLS, so deploying it requires [cert-manager][cm]
`CacheBackend`, plus a mutating Pod webhook that auto-injects the cache
engine configuration (chosen by the matched runtime adapter) into pods
labeled to match a `CacheBackend`'s `spec.engineSelector` — over TLS, so
deploying it requires [cert-manager][cm]
v1.0+ in the target cluster. The default install (`config/default`)
provisions a self-signed `Issuer` plus a `Certificate` for the webhook
serving cert, and relies on cert-manager's `cert-manager.io/inject-ca-from`
Expand Down Expand Up @@ -227,11 +257,14 @@ make dev-cluster KIND_CLUSTER=cache-dev KIND_NODE_IMAGE=kindest/node:v1.31.0

## Documentation

Design docs live under [`docs/`](docs/):
Docs live under [`docs/`](docs/):

- [`docs/design/cachebackend-api.md`](docs/design/cachebackend-api.md) — the `CacheBackend` CRD contract
- [`docs/design/grpc-contract.md`](docs/design/grpc-contract.md) — the `InferenceCache` gRPC service contract (B4)
- [`docs/design/policy-crds.md`](docs/design/policy-crds.md) — policy CRDs (`CachePolicy`, `CacheTenant`, `PromptTemplate`, `PDTopology`, `CacheIndex`)
- **Concepts** — [`docs/concepts/`](docs/concepts/): engine binding, engine overrides, policy tuning, tenant identity/quota, and the CacheIndex cluster aggregate.
- **Design** — [`docs/design/`](docs/design/): the [`CacheBackend` CRD contract](docs/design/cachebackend-api.md), the [`InferenceCache` gRPC contract](docs/design/grpc-contract.md), [gRPC TLS](docs/design/grpc-tls.md), [policy CRDs](docs/design/policy-crds.md), [LookupRoute ranking](docs/design/lookuproute-ranking.md), and [KV-event subscriber wiring](docs/design/kvevent-subscriber-wiring.md).
- **Reference** — [`docs/reference/`](docs/reference/): the [metric surface](docs/reference/metrics.md) and gRPC [reason codes](docs/reference/reason-codes.md).
- **Operations & observability** — [`docs/operations/`](docs/operations/), the alert runbooks in [`docs/observability/`](docs/observability/), and the [`inferencecache doctor`](docs/cli/doctor.md) preflight CLI.
- **Reference stack** — [`docs/reference-stack/`](docs/reference-stack/): runnable manifests + a GPU runbook for a full vLLM/LMCache stack.
- **Quick start** — [`docs/quickstart.md`](docs/quickstart.md).

Contributor guide: [`CONTRIBUTING.md`](CONTRIBUTING.md) (layout, naming rule, push/PR gates).

Expand Down
Loading