From 6604bcc7469aa5f114dd59d6cd71438aa0450fd7 Mon Sep 17 00:00:00 2001 From: Ed Sun Date: Thu, 23 Jul 2026 18:49:53 -0700 Subject: [PATCH] docs: sync index-key definitions with the LoRA adapter partition MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The content/prefix key gained `adapter` in #159 — (tenant, model, hash_scheme, adapter, prefix_hash), and the chain/block key likewise — but several design/ops/reference docs still showed the pre-adapter 4-tuple. Update lookuproute-ranking, cachebackend-api, index-sizing, cachetenant-identity-and-quota, metrics, and reason-codes; add brief cardinality/quota notes where adapters multiply distinct keys. The (tenant, model, hash_scheme) serving SCOPE stays deliberately adapter-blind (TENANT_HOT / AFFINITY_HINT / diagnostics) — only the content key gains adapter. Docs only; no code change. --- docs/concepts/cachetenant-identity-and-quota.md | 7 +++++-- docs/design/cachebackend-api.md | 2 +- docs/design/lookuproute-ranking.md | 14 +++++++++----- docs/operations/index-sizing.md | 12 +++++++++--- docs/reference/metrics.md | 2 +- docs/reference/reason-codes.md | 4 ++-- 6 files changed, 27 insertions(+), 14 deletions(-) diff --git a/docs/concepts/cachetenant-identity-and-quota.md b/docs/concepts/cachetenant-identity-and-quota.md index d13ce5cf..d2fa212b 100644 --- a/docs/concepts/cachetenant-identity-and-quota.md +++ b/docs/concepts/cachetenant-identity-and-quota.md @@ -20,8 +20,11 @@ These are the only spec fields — do not expect others. | `isolationMode` | enum (`Fairness`) | `Fairness` | Only value implemented today. | | `crypto` | object | empty | Reserved for future cryptographic isolation; carries no fields today. | -The quota unit is the **distinct key** `(tenant, model, hash_scheme, prefix_hash)`. +The quota unit is the **distinct key** `(tenant, model, hash_scheme, adapter, prefix_hash)`. A prefix held warm on several replicas counts **once**, not once per replica. +Because `adapter` is part of the key (`""` for base-model / non-LoRA traffic), +the *same* prompt content served under N LoRA adapters counts as N distinct keys +— serving many adapters multiplies quota consumption accordingly. ## Status and printer columns @@ -43,7 +46,7 @@ Columns: **Tenant** (`spec.tenantID`), **Entries** (`status.indexEntries`), ## Identity isolation is structural -The index keys every entry by `(tenant, model, hash_scheme, prefix_hash)`, so +The index keys every entry by `(tenant, model, hash_scheme, adapter, prefix_hash)`, so tenant A's hint records can never collide with tenant B's, and `LookupRoute` is tenant-scoped: a lookup carrying `tenant_id = A` only ever sees A's prefixes. This isolation is a property of **our** data structure, not of the engine — it diff --git a/docs/design/cachebackend-api.md b/docs/design/cachebackend-api.md index 30c20bbc..9062cb72 100644 --- a/docs/design/cachebackend-api.md +++ b/docs/design/cachebackend-api.md @@ -187,7 +187,7 @@ Deliberately **not** injected for SGLang (a real engine difference, not an omiss The two override surfaces are **separate, and neither reaches the other**: the MP tunables above are read from `spec.backendConfig` and shape the **worker** sidecar, while `spec.integration.engineOverrides` edits the **engine** container's args/env only. So `engineOverrides` cannot retune `l1SizeGB` or `mpPort` (set them in `backendConfig`), and it cannot reach the worker's command line at all; what it *can* do is add or suppress engine args/env outside the reserved set above. -**KV-event source & `hash_scheme: "sglang"`.** SGLang adopted vLLM's KV-event wire wholesale — `--kv-events-config` drives a ZMQ `ZmqEventPublisher` emitting the same msgspec array-like `BlockStored` / `BlockRemoved` / `AllBlocksCleared` tuples (`BlockStored` carries `token_ids`, so the subscriber derives the same in-pod content fingerprint it does for vLLM). The shipped `kvevent-subscriber` binary therefore decodes SGLang's stream **unchanged**; the only difference is the adapter pins the sidecar's `--hash-scheme=sglang`. The index keys on `(tenant, model, hash_scheme, prefix_hash)`, so SGLang prefixes occupy a **domain disjoint** from vLLM's: a request hashed under one scheme never false-hits a bytewise-identical entry recorded under the other, even when both engines tokenize the same text to the same token ids and the content fingerprints collide. As with vLLM, the operator must launch the engine with `--kv-events-config '{"publisher":"zmq","endpoint":"tcp://*:5557","topic":"kv-events"}'` for the publisher to be active — the adapter wires the cache offload, not the event publisher. +**KV-event source & `hash_scheme: "sglang"`.** SGLang adopted vLLM's KV-event wire wholesale — `--kv-events-config` drives a ZMQ `ZmqEventPublisher` emitting the same msgspec array-like `BlockStored` / `BlockRemoved` / `AllBlocksCleared` tuples (`BlockStored` carries `token_ids`, so the subscriber derives the same in-pod content fingerprint it does for vLLM). The shipped `kvevent-subscriber` binary therefore decodes SGLang's stream **unchanged**; the only difference is the adapter pins the sidecar's `--hash-scheme=sglang`. The index keys on `(tenant, model, hash_scheme, adapter, prefix_hash)`, so SGLang prefixes occupy a **domain disjoint** from vLLM's: a request hashed under one scheme never false-hits a bytewise-identical entry recorded under the other, even when both engines tokenize the same text to the same token ids and the content fingerprints collide. As with vLLM, the operator must launch the engine with `--kv-events-config '{"publisher":"zmq","endpoint":"tcp://*:5557","topic":"kv-events"}'` for the publisher to be active — the adapter wires the cache offload, not the event publisher. > **Block-size alignment for server-derived lookups (operational note).** The subscriber-ingested path is block-size-safe by construction: the subscriber derives each prefix fingerprint in-pod using the `block_size` carried on the engine's own `BlockStored` event (SGLang's `--page-size`, often 64), so SGLang entries land in the index at SGLang's block size with no server involvement. The **server-derived** lookup path is the catch: when a gateway calls `LookupRoute` with raw `token_ids` / `prompt_text` (rather than a pre-computed `prefix_hash` / `block_hashes` chain), the server fingerprints them with its single global `--engine-block-size` (default 16, vLLM's). For SGLang those server-side hashes only line up with SGLang-ingested entries when `--engine-block-size` is set to SGLang's page size. Because the flag is **one global value**, a single server cannot serve raw-`token_ids` lookups for both a vLLM (16) and an SGLang (64) deployment at once — in a mixed-engine cluster, have gateways send the pre-computed `prefix_hash` / `block_hashes` (the subscriber/fingerprint path, which is block-size-correct per engine) for the raw-token path, or run a server per block size. Making `--engine-block-size` per-`hash_scheme` is a server change tracked as a follow-up; it is the first concrete latently-vLLM-centric assumption this second engine surfaced. diff --git a/docs/design/lookuproute-ranking.md b/docs/design/lookuproute-ranking.md index 45c365a6..3402ab72 100644 --- a/docs/design/lookuproute-ranking.md +++ b/docs/design/lookuproute-ranking.md @@ -26,11 +26,15 @@ response is ever an error on the hot path. Fail open. ## 2. The baseline strategy — `PREFIX_MATCH` -The index keys cached prefixes by `(tenant, model, hash_scheme, prefix_hash)` +The index keys cached prefixes by `(tenant, model, hash_scheme, adapter, prefix_hash)` → the set of replicas that hold it, with per-replica `token_count` and a -`last_seen` timestamp. A baseline lookup: +`last_seen` timestamp. `adapter` is the resolved LoRA identity (`""` for +base-model / non-LoRA traffic); it partitions the content, never the fingerprint, +so identical tokens under different adapters share a hash but not an entry. The +`(tenant, model, hash_scheme)` *scope* used by the fallbacks below deliberately +stays adapter-blind. A baseline lookup: -1. Look up the request's `(tenant, model, hash_scheme, prefix_hash)`. +1. Look up the request's `(tenant, model, hash_scheme, adapter, prefix_hash)`. 2. For each replica that holds it, compute ``` @@ -71,7 +75,7 @@ collapses back to `matched_tokens × freshness`. ### Why the exact-full-hash path leaves hits on the table -§2 looks up the request's `(tenant, model, hash_scheme, prefix_hash)` as +§2 looks up the request's `(tenant, model, hash_scheme, adapter, prefix_hash)` as **one opaque blob**. Two requests that share the first N KV blocks of a prefix but diverge after — common with a shared system prompt plus per-request RAG context — hash to different `prefix_hash` values and miss @@ -116,7 +120,7 @@ slip into a non-additive change. ### How the lookup walks the chain For each block `block_hashes[i]` the request carries, the index already -holds a `(tenant, model, hash_scheme, block_hash)` → `{replicaID → +holds a `(tenant, model, hash_scheme, adapter, block_hash)` → `{replicaID → {tokenCount, lastSeen}}` entry — populated either from a single chain `PrefixEntry` (the index *expands* the chain into N per-block entries) or from N legacy single-blob `PrefixEntry` reports (one per block, the diff --git a/docs/operations/index-sizing.md b/docs/operations/index-sizing.md index 78e420af..ddbdb48e 100644 --- a/docs/operations/index-sizing.md +++ b/docs/operations/index-sizing.md @@ -1,11 +1,17 @@ # Index sizing & memory tuning The inferencecache-server holds a soft-state cache-state index in process memory: -distinct prefix entries per `(tenant, model, hash_scheme, prefix_hash)`, replica +distinct prefix entries per `(tenant, model, hash_scheme, adapter, prefix_hash)`, replica stats per `(tenant, model, replica_id)`, plus a few secondary indices. This guide gives operators the numbers and dials to size the server pod and the per-namespace / per-tenant knobs that bound the index footprint. +> **LoRA note.** `adapter` (`""` for base-model / non-LoRA traffic) is part of the +> prefix key, so identical prompt content served under N adapters is N distinct +> entries. A multi-adapter workload multiplies key cardinality — and therefore +> memory and eviction pressure — by roughly the number of actively-cached +> adapters; size `maxIndexEntries` and pod memory for that fan-out. + **Audience.** Cluster operators picking pod resource limits, `CachePolicy.spec.evictionTTL`, and `CacheTenant.spec.quota.maxIndexEntries` for their workload. @@ -54,7 +60,7 @@ so reaching for a larger footprint needs a recompile today. **Two units that get conflated.** The CRD field name `maxIndexEntries`, the snapshot field `tenants[].indexEntries`, and the metric `inferencecache_index_entries` (with a `model` label, e.g. `inferencecache_index_entries{model="meta-llama/Llama-3"}`) all -count **distinct prefix keys** — one per `(tenant, model, hash_scheme, prefix_hash)`, +count **distinct prefix keys** — one per `(tenant, model, hash_scheme, adapter, prefix_hash)`, regardless of how many replicas hold it. The internal `pkg/index.DefaultMaxEntries` cap, by contrast, counts **total storage entries** — one per `(prefix_key, replica)` tuple. A single prefix held by R replicas is 1 "indexEntries" but R "storage entries". When @@ -86,7 +92,7 @@ table — the rule of thumb is for back-of-envelope work, not a tight predictor. A single entry is a chain of small Go allocations under two nested `map`s: - outer `map[prefixKey]…` bucket header + the prefix-key strings (tenant, model, - hash_scheme, prefix_hash); + hash_scheme, adapter, prefix_hash); - inner `map[replicaID]*replicaEntry` header + the replica-id string; - the `replicaEntry` struct (token count + lastSeen `time.Time` + `atomic.Int64` LFU counter). diff --git a/docs/reference/metrics.md b/docs/reference/metrics.md index 41f1427b..59d19d01 100644 --- a/docs/reference/metrics.md +++ b/docs/reference/metrics.md @@ -48,7 +48,7 @@ silently. |---|---|---|---| | `inferencecache_server_up` | *(none)* | `1` if the cache policy server is serving requests, `0` otherwise. | Server starts (→`1`) / shuts down (→`0`). Liveness signal. | | `inferencecache_server_grpc_tls_enabled` | *(none)* | `1` if the gRPC server (`:9090`) is terminating TLS, `0` if serving plaintext. | Set once at startup from `--tls-cert-file`/`--tls-key-file` (both set → `1`, both empty → `0`). Confirms the prod wire posture from Prometheus. See `docs/design/grpc-tls.md`. | -| `inferencecache_index_entries` | `model` | **Distinct prefix entries** the in-memory `CacheIndex` currently holds for that model, **excluding reserved-tenant (`inferencecache.io/probe`) entries**. One entry = one unique `(tenant, model, hash_scheme, prefix_hash)` tuple, regardless of how many replicas hold it. | Rises on new `(scheme, hash)` from `ReportCacheState`; falls on `AllBlocksCleared` / TTL eviction / max-entries cap. A `BlockRemoved` only drops the entry in **single-tier (non-L2) mode**, where the subscriber forwards it as `PREFIX_EVICTED`; in **L2/Offload mode** a `BlockRemoved` **retains** the entry — the subscriber re-reports it at tier T2 rather than deleting it, so the count holds (the entry ages out only via TTL). Idempotent re-reports and T1↔T2 tier changes on an existing `(scheme, hash)` do **not** move it. The probe's synthetic state IS in the index during a Run but is excluded from this gauge so a scrape that races Stage C cannot transiently surface a probe-tenant count on a real model bucket — see `WithReservedTenants` in `pkg/index/index.go`. | +| `inferencecache_index_entries` | `model` | **Distinct prefix entries** the in-memory `CacheIndex` currently holds for that model, **excluding reserved-tenant (`inferencecache.io/probe`) entries**. One entry = one unique `(tenant, model, hash_scheme, adapter, prefix_hash)` tuple, regardless of how many replicas hold it. | Rises on new `(scheme, hash)` from `ReportCacheState`; falls on `AllBlocksCleared` / TTL eviction / max-entries cap. A `BlockRemoved` only drops the entry in **single-tier (non-L2) mode**, where the subscriber forwards it as `PREFIX_EVICTED`; in **L2/Offload mode** a `BlockRemoved` **retains** the entry — the subscriber re-reports it at tier T2 rather than deleting it, so the count holds (the entry ages out only via TTL). Idempotent re-reports and T1↔T2 tier changes on an existing `(scheme, hash)` do **not** move it. The probe's synthetic state IS in the index during a Run but is excluded from this gauge so a scrape that races Stage C cannot transiently surface a probe-tenant count on a real model bucket — see `WithReservedTenants` in `pkg/index/index.go`. | ### Counters diff --git a/docs/reference/reason-codes.md b/docs/reference/reason-codes.md index 0764220b..f9e17fd0 100644 --- a/docs/reference/reason-codes.md +++ b/docs/reference/reason-codes.md @@ -34,10 +34,10 @@ module — until then, treat `NO_HINT` as the only `LookupPDRoute` answer. | Code | Status | RPC | When the server emits it | Response shape | What the gateway does | |---|---|---|---|---|---| -| `PREFIX_MATCH` | **shipped** | LookupRoute only | The index has at least one replica holding the request's `(tenant, model, hash_scheme)` prefix — either the exact `prefix_hash` (legacy single-blob path) OR the leading run of `block_hashes[0..k]` (chain longest-prefix path; see [`../design/lookuproute-ranking.md`](../design/lookuproute-ranking.md) §2.5) — the ranker returned a non-empty set, AND at least one replica's realized `matched_tokens` cleared the per-namespace `minimumMatchedTokens` floor (default 64) AND the top surviving replica's score cleared the per-namespace `routingFloorScore` floor (default `0.1`). The score includes the distinguishing-power factor `1 − num_matching_replicas / total_replicas` so an overlap held by every replica (chat-template framing, RAG corpus headers, custom system prompts) collapses to score 0 and is filtered by the score floor. Both floors can downgrade independently. | `replica_scores` non-empty, ranked best-first by `matched_tokens × freshness × pressure_factor × slo_bias × distinguishing_power`. The pressure / SLO / distinguishing-power factors collapse appropriately when their inputs are absent (no stats → pressure_factor = 1; no SLO hint → slo_bias = 1; single-replica deployment → distinguishing_power = 1). Every qualifying replica is returned today (no top-K limit); the gateway typically uses the top entry. | Route to the top-ranked replica → prefix-cache hit; lower TTFT. | +| `PREFIX_MATCH` | **shipped** | LookupRoute only | The index has at least one replica holding the request's `(tenant, model, hash_scheme, adapter)` prefix — either the exact `prefix_hash` (legacy single-blob path) OR the leading run of `block_hashes[0..k]` (chain longest-prefix path; see [`../design/lookuproute-ranking.md`](../design/lookuproute-ranking.md) §2.5) — the ranker returned a non-empty set, AND at least one replica's realized `matched_tokens` cleared the per-namespace `minimumMatchedTokens` floor (default 64) AND the top surviving replica's score cleared the per-namespace `routingFloorScore` floor (default `0.1`). The score includes the distinguishing-power factor `1 − num_matching_replicas / total_replicas` so an overlap held by every replica (chat-template framing, RAG corpus headers, custom system prompts) collapses to score 0 and is filtered by the score floor. Both floors can downgrade independently. | `replica_scores` non-empty, ranked best-first by `matched_tokens × freshness × pressure_factor × slo_bias × distinguishing_power`. The pressure / SLO / distinguishing-power factors collapse appropriately when their inputs are absent (no stats → pressure_factor = 1; no SLO hint → slo_bias = 1; single-replica deployment → distinguishing_power = 1). Every qualifying replica is returned today (no top-K limit); the gateway typically uses the top entry. | Route to the top-ranked replica → prefix-cache hit; lower TTFT. | | `NO_HINT` | **shipped** | both | The fail-open default. **`LookupPDRoute`**: every call (the handler is a stub). **`LookupRoute`**: the prefix is novel under matching contract keys AND no usable affinity fallback fired (`affinityRouting: Disabled` on the per-namespace policy, OR no replica known to serve the `(tenant, model, hash_scheme)` engine domain, OR no usable seed — empty `block_hashes` and empty `prefix_hash`, OR structurally malformed input — empty `hash_scheme` or chain arrays of mismatched length); the ranker found nothing AND the same affinity-disabled / no-replica / no-seed / malformed clause holds; any of `tenant_id`, `model_id`, or `hash_scheme` was unspecified (a contract violation — set-but-wrong values surface as the matching `UNKNOWN_*` code instead); the index is globally empty (cold-start carve-out); the request was policy-gated below `minimumPrefixTokens` AND `affinityRouting: Disabled` (with `affinityRouting: Enabled` — the default — the same gate surfaces as `AFFINITY_HINT`); **every replica that held the prefix matched fewer tokens than `minimumMatchedTokens` (result-side per-replica floor, default 64) AND `affinityRouting: Disabled`** (with affinity Enabled the downgrade surfaces as `AFFINITY_HINT`); **the top per-replica score from the distinguishing-power-aware ranker fell below `routingFloorScore` (result-side score floor, default `0.1`) AND `affinityRouting: Disabled`** (with affinity Enabled the downgrade surfaces as `AFFINITY_HINT`); or an index-disabled state. | `replica_scores` **empty**. Not an error. | Route per the gateway's default policy (round-robin, least-loaded, …). The cache plane is invisible to this request. | | `POLICY_REQUIRES_CHAIN` | **shipped** | LookupRoute only | The tenant's `CachePolicy.spec.strategy.requireChain` is `true`, but the request did not carry a valid `block_hashes` + `block_token_counts` chain. The server returns before touching the index. This is a policy-gated empty result, separated from `NO_HINT` so operators can see legacy/exact callers hitting a chain-only namespace in `inferencecache_lookup_route_calls_total{reason_code="POLICY_REQUIRES_CHAIN"}`. | Empty `replica_scores`. | Treat as `NO_HINT`; update the caller to send the chain if it should benefit from the cache plane in this namespace. | -| `TENANT_HOT` | **shipped** | LookupRoute only | No exact prefix match for `(tenant, model, hash_scheme, prefix_hash)`, `CachePolicy.spec.strategy.enableTenantHot` is not `false`, and the tenant has at least one replica that (a) has reported stats recently (within ~5 minutes by default), (b) has a `hit_rate` above a small floor (default 0.1), AND (c) currently has **at least one prefix entry in the requested `(tenant, model, hash_scheme)` in the index** — proving the replica serves the requested engine domain. The "in the index" check is sweep-driven (an entry past TTL stays counted until the next sweep removes it), so for at most one sweep interval a recently-stale entry can briefly still satisfy the check; per soft-state semantics that yields at worst a soft hint that turns into a cache miss, never a wrong answer. A coarser locality signal than `PREFIX_MATCH` — useful when the prefix is novel but the tenant already has servers warm in the cache rotation. When the same fallback is found but the policy disables tenant-hot, the handler downgrades it to `NO_HINT`. | `replica_scores` non-empty (tenant-hot ranked); `matched_tokens` is **0** because there is no prefix overlap (the gateway must rely on `reason_code`, not `matched_tokens`, to recognize this branch). Shape otherwise unchanged. | Treat as a softer hint than `PREFIX_MATCH`; gateway free to use or ignore. | +| `TENANT_HOT` | **shipped** | LookupRoute only | No exact prefix match for `(tenant, model, hash_scheme, adapter, prefix_hash)`, `CachePolicy.spec.strategy.enableTenantHot` is not `false`, and the tenant has at least one replica that (a) has reported stats recently (within ~5 minutes by default), (b) has a `hit_rate` above a small floor (default 0.1), AND (c) currently has **at least one prefix entry in the requested `(tenant, model, hash_scheme)` in the index** — proving the replica serves the requested engine domain. The "in the index" check is sweep-driven (an entry past TTL stays counted until the next sweep removes it), so for at most one sweep interval a recently-stale entry can briefly still satisfy the check; per soft-state semantics that yields at worst a soft hint that turns into a cache miss, never a wrong answer. A coarser locality signal than `PREFIX_MATCH` — useful when the prefix is novel but the tenant already has servers warm in the cache rotation. When the same fallback is found but the policy disables tenant-hot, the handler downgrades it to `NO_HINT`. | `replica_scores` non-empty (tenant-hot ranked); `matched_tokens` is **0** because there is no prefix overlap (the gateway must rely on `reason_code`, not `matched_tokens`, to recognize this branch). Shape otherwise unchanged. | Treat as a softer hint than `PREFIX_MATCH`; gateway free to use or ignore. | | `TIMEOUT` | **shipped** | LookupRoute only | The lookup deadline expired before the index could rank — either the caller's context was already past its deadline on arrival or the per-tenant `CachePolicy.spec.lookupTimeoutMs` budget elapsed during the lookup. Gateway clients also synthesize this locally when *they* cancel a slow `LookupRoute` RPC. | Server: empty `replica_scores`. Client-side synth: same. | Treat as `NO_HINT`. | | `UNKNOWN_TENANT` | **shipped** | LookupRoute only | After a prefix miss (and the `TENANT_HOT` fallback when it applies — non-chain requests; chain requests skip `TENANT_HOT` by design and classify directly), AND the index is **not globally empty**: the request supplied a non-empty `tenant_id` and the index has **zero prefix entries for that tenant** across every model and hash scheme. **Cold-start carve-out:** a globally empty index (server just started, no `ReportCacheState` yet) stays on `NO_HINT` so a fresh deployment does not flood gateways with `UNKNOWN_TENANT`; the diagnostic resumes the moment any replica has reported state. Canonical asymmetric shape: a gateway-SDK querying with `tenant_id="default"` while the producer (kvevent-subscriber sidecar) is publishing under `tenant_id=$(POD_NAMESPACE)`. | Empty `replica_scores`. | Treat as `NO_HINT` for routing (still fail-open — the cache plane is hint-only); surface as a configuration error (log line / metric / SDK warning). **Do not retry under a different key** — the cache plane will not change between calls. | | `UNKNOWN_MODEL` | **shipped** | LookupRoute only | Same precondition as `UNKNOWN_TENANT` above. The tenant is known but the `(tenant_id, model_id)` pair has **zero entries**. The model has never served traffic in this tenant, or the model identifier disagrees between producer and consumer. | Empty `replica_scores`. | Same as `UNKNOWN_TENANT`: fail-open, surface as configuration error. |