feat: partition the routing index by LoRA adapter so cache hints can't alias

cmd/kvevent-subscriber/main.go

@@ -50,16 +50,24 @@ func main() {
 		cacheTier          = flag.String("cache-tier", "auto", `which vLLM cache-usage gauge to read: "auto" (kv→gpu→cpu fallback) | "kv" | "gpu" | "cpu"`)
 		engineModel        = flag.String("engine-model-name", "", `value of the engine's `+"`model_name`"+` Prometheus label to filter /metrics by (e.g. "Qwen/Qwen2.5-0.5B-Instruct"). Distinct from --model-id (the cache-plane index key). Empty = no label filter (aggregates every series — fine when the engine serves one model).`)
 		ignoreBlockRemoved = flag.Bool("ignore-block-removed", false, `declare that this engine is paired with an L2 offload tier (e.g. LMCache): a BlockRemoved is re-reported as a T2 (reload-from-L2) entry instead of a PREFIX_EVICTED delete, so the routing hint survives the HBM eviction but is honestly tagged colder. Default off for single-tier deployments (BlockRemoved → PREFIX_EVICTED). Name kept for compatibility. See docs/design/kvevent-subscriber-wiring.md "L2 cache tier semantics".`)
+		adapterNames       = flag.String("lora-adapter-names", "", `comma-separated `+"`id=name`"+` map from the engine's internal LoRA id (BlockStored.lora_id, assigned in --lora-modules load order) to the stable adapter identity used as the index partition — the same string the gateway sends as LookupRouteRequest.adapter_id (e.g. "1=sql-lora,2=chat-lora"). An unmapped non-nil lora_id is FAIL-CLOSED: its blocks are dropped (not cached) rather than indexed under a replica-local "lora:<id>" that could alias different adapters across replicas — so LoRA caching REQUIRES this map. Omit ONLY for base-model / non-LoRA traffic (nil lora_id → the default "" partition). The identity mapped here MUST match the gateway's LookupRoute query adapter_id end-to-end or every lookup silently misses (the same agreement --hash-scheme requires).`)
 	)
 	flag.Parse()
 
 	logger := slog.New(slog.NewJSONHandler(os.Stderr, nil))
 
+	names, err := engine.ParseAdapterNames(*adapterNames)
+	if err != nil {
+		logger.Error("invalid --lora-adapter-names", "value", *adapterNames, "err", err)
+		os.Exit(2)
+	}
+
 	cfg := engine.Config{
-		ReplicaID:  *replica,
-		ModelID:    *model,
-		TenantID:   *tenant,
-		HashScheme: *scheme,
+		ReplicaID:    *replica,
+		ModelID:      *model,
+		TenantID:     *tenant,
+		HashScheme:   *scheme,
+		AdapterNames: names,
 	}
 	if err := cfg.Validate(); err != nil {
 		logger.Error("invalid config", "err", err)

docs/design/kvevent-subscriber-wiring.md

@@ -192,6 +192,59 @@ Other adapters (e.g. plain vLLM, or future runtimes with no L2 tier) leave
 the flag off for the same reason as `EventsOnly` — their stored prefixes stay
 T1 and a `BlockRemoved` deletes them.
 
+## LoRA adapter identity — `--lora-adapter-names`
+
+The routing key is a **content fingerprint over token IDs only**, so two prompts
+with identical tokens under different LoRA adapters hash identically. The
+subscriber therefore reads `lora_id` off each `BlockStored` event and stamps the
+resolved adapter identity on every reported `PrefixEntry`, so the server can put
+them in disjoint **index partitions** rather than one aliased entry (see
+[`grpc-contract.md`](grpc-contract.md) "Update — adapter (LoRA) index
+partition"). The adapter never enters the hash.
+
+`lora_id` is vLLM's **internal load-order integer**, not a name: the same integer
+can mean different adapters on two replicas whose `--lora-modules` order differs,
+and the index is shared across replicas. `--lora-adapter-names` maps it to the
+stable identity the gateway sends as `LookupRouteRequest.adapter_id`:
+
+```
+--lora-adapter-names "1=sql-lora,2=chat-lora"
+```
+
+- **Unmapped id** → **dropped (fail-closed)**. A non-nil `lora_id` with no
+  mapping has only a replica-local load-order integer, which could alias
+  different adapters across replicas whose load order differs — so its blocks are
+  dropped (not indexed) rather than partitioned under a hazardous `lora:<id>`.
+  The adapter gets no routing hint (a cache miss, never a wrong replica) until it
+  is mapped; supply the map — for adapters **known at startup** — and make the
+  gateway send the matching `adapter_id`.
+- **No LoRA at all** (the default): every event carries a nil `lora_id`, so
+  every entry and eviction lands in the default (`""`) partition — byte-for-byte
+  the pre-adapter behavior. Nothing to configure.
+
+### Limitations of the static flag
+
+`--lora-adapter-names` is resolved once at **startup**, so an adapter it does not
+name is **fail-closed** — its blocks are dropped (uncached), never aliased. That
+trades cache benefit for safety in two cases, both a tracked follow-up:
+
+- **Runtime loads.** An adapter loaded or reloaded *after* the subscriber starts
+  whose id isn't in the map is dropped — a static list can't name it. Covering
+  arbitrary runtime loads needs the subscriber to resolve adapter identity
+  **dynamically** from the engine's adapter registry.
+- **Managed injection.** On the `CacheBackend`-driven path the subscriber
+  sidecar is **injected**, so operators cannot pass the flag on a container they
+  do not define; the explicit-set guidance below applies only to a
+  **self-managed** subscriber. Configuring the mapping through the managed path
+  needs a `CacheBackend` field the webhook forwards.
+
+Neither is a correctness regression — a dropped adapter gets no routing hint (a
+cache miss, never a wrong replica), strictly safer than the alias a replica-local
+`lora:<id>` fallback would risk. For a self-managed subscriber, set
+`--lora-adapter-names` on the container directly: the controller has no view of
+the engine's `--lora-modules` ordering, so multi-adapter engines must supply the
+map explicitly.
+
 ## What this unblocks
 
 * The runbook / demo path no longer needs `port-forward` + a hand-launched

docs/reference-stack/scripts/default_install_smoke.sh

@@ -1340,6 +1340,118 @@ until has_reason_code "$policy_high_resp" "PREFIX_MATCH" \
 done
 log "CachePolicy push adopted: above-threshold lookup hit; below-request-gate lookup returned AFFINITY_HINT; trivial-match (matched_tokens<floor) lookup returned AFFINITY_HINT — both minimum-token policy knobs enforced end-to-end (the non-PREFIX_MATCH outcomes prove the gates fired; the affinity fallback then surfaces the single known replica)"
 
+# --- adapter (LoRA) index-partition probe ----------------------------------
+# The content fingerprint is derived from token IDs ALONE, so two requests with
+# identical tokens under different LoRA adapters produce the SAME prefix_hash.
+# adapter_id partitions the INDEX so those identical keys cannot alias — without
+# it, a lookup for adapter A could be handed a replica holding only adapter B's
+# KV. This drives the gRPC surface the way an operator would and needs no engine
+# traffic: seed ONE prefix under adapter "smoke-lora-a" via ReportCacheState,
+# then look the identical hash up three ways.
+#
+#   - same adapter      → PREFIX_MATCH (the partition is a partition, not a
+#                         hash change — matching inside it is unaffected)
+#   - different adapter → NOT PREFIX_MATCH (the alias this partition prevents)
+#   - no adapter at all → NOT PREFIX_MATCH (the default partition, where this
+#                         adapter-scoped content was never ingested)
+#
+# The two negative cases land on AFFINITY_HINT rather than NO_HINT because
+# cachepolicy-sample leaves affinityRouting at its Enabled default and the
+# replica is known to the scope; the assertion is on "not PREFIX_MATCH", which
+# is the property that matters (no cross-adapter cache claim).
+log "seeding an adapter-scoped prefix and asserting LookupRoute cannot alias it across adapter_id"
+adapter_model="install-smoke-adapter"
+adapter_replica="adapter-smoke-replica"
+adapter_hash_b64="YWRhcHRlci1wcmVmaXg=" # base64("adapter-prefix") — stored at tokenCount=64, clears both policy gates
+
+# No stats in this update: a warm hit_rate would let the negative probes below
+# qualify for TENANT_HOT, making the expected fail-open code ambiguous between
+# TENANT_HOT and AFFINITY_HINT. Without stats the non-matching adapter partitions
+# fall straight through to the affinity fallback, so the expected code is
+# deterministically AFFINITY_HINT (the replica still SERVES the adapter-blind
+# scope via the prefix ingest, so affinity has a candidate). PREFIX_MATCH on the
+# same adapter needs only the prefix entry, not stats.
+adapter_report_payload="$(cat <<EOF
+{"replicaId":"$adapter_replica","modelId":"$adapter_model","tenantId":"$POLICY_SMOKE_NS","hashScheme":"vllm","prefixes":[{"prefixHash":"$adapter_hash_b64","tokenCount":64,"adapterId":"smoke-lora-a"}]}
+EOF
+)"
+adapter_report_resp="$(grpcurl_report_cache_state "$adapter_report_payload" "$LOG_DIR/grpcurl-adapter-report.err")" || {
+  cat "$LOG_DIR/grpcurl-adapter-report.err" >&2 || true
+  fail "grpcurl ReportCacheState did not accept the adapter-scoped smoke prefix"
+}
+log "ReportCacheState (adapter-scoped) response: $adapter_report_resp"
+
+adapter_same_payload="$(cat <<EOF
+{"modelId":"$adapter_model","tenantId":"$POLICY_SMOKE_NS","hashScheme":"vllm","prefixHash":"$adapter_hash_b64","prefixTokenCount":64,"adapterId":"smoke-lora-a"}
+EOF
+)"
+adapter_other_payload="$(cat <<EOF
+{"modelId":"$adapter_model","tenantId":"$POLICY_SMOKE_NS","hashScheme":"vllm","prefixHash":"$adapter_hash_b64","prefixTokenCount":64,"adapterId":"smoke-lora-b"}
+EOF
+)"
+adapter_none_payload="$(cat <<EOF
+{"modelId":"$adapter_model","tenantId":"$POLICY_SMOKE_NS","hashScheme":"vllm","prefixHash":"$adapter_hash_b64","prefixTokenCount":64}
+EOF
+)"
+
+deadline=$(($(date +%s) + POLICY_PUSH_TIMEOUT))
+adapter_same_resp=""
+until has_reason_code "$adapter_same_resp" "PREFIX_MATCH"; do
+  adapter_same_resp="$(grpcurl_lookup_route "$adapter_same_payload" "$LOG_DIR/grpcurl-adapter-same.err")" || true
+  if has_reason_code "$adapter_same_resp" "PREFIX_MATCH"; then
+    break
+  fi
+  if [ "$(date +%s)" -ge "$deadline" ]; then
+    echo "same-adapter LookupRoute response (want PREFIX_MATCH):" >&2
+    echo "$adapter_same_resp" >&2
+    if [ -s "$LOG_DIR/grpcurl-adapter-same.err" ]; then
+      cat "$LOG_DIR/grpcurl-adapter-same.err" >&2
+    fi
+    fail "LookupRoute with the ingesting adapter_id did not return PREFIX_MATCH within ${POLICY_PUSH_TIMEOUT}s — the adapter partition must not break matching inside it"
+  fi
+  sleep 2
+done
+
+# assert_adapter_no_alias runs one NEGATIVE adapter probe. The property under
+# test is "the identical prefix hash does NOT match in a different partition,"
+# but a transport error mustn't masquerade as that: grpcurl failing or returning
+# an empty body would leave has_reason_code false and silently pass. So require
+# the RPC to SUCCEED and return a non-empty body, then assert the expected
+# fail-open reason_code — AFFINITY_HINT, because affinityRouting is Enabled for
+# this tenant (the policy block above proved it) and the replica serves the
+# adapter-blind (tenant, model, hash_scheme) scope, so a non-matching adapter
+# partition downgrades to a stable affinity pick rather than a prefix hit. An
+# RPC error is therefore a red, not a green.
+assert_adapter_no_alias() {
+  local payload="$1" err_file="$2" label="$3" alias_msg="$4"
+  local resp
+  if ! resp="$(grpcurl_lookup_route "$payload" "$err_file")" || [ -z "$resp" ]; then
+    if [ -s "$err_file" ]; then
+      cat "$err_file" >&2
+    fi
+    echo "$label LookupRoute response: ${resp:-<empty>}" >&2
+    fail "$label LookupRoute returned no response — an RPC/transport error must fail the adapter-partition probe, not silently pass it as a non-match"
+  fi
+  if has_reason_code "$resp" "PREFIX_MATCH"; then
+    echo "$label LookupRoute response (want NOT PREFIX_MATCH):" >&2
+    echo "$resp" >&2
+    fail "$alias_msg"
+  fi
+  if ! has_reason_code "$resp" "AFFINITY_HINT"; then
+    echo "$label LookupRoute response (want fail-open AFFINITY_HINT):" >&2
+    echo "$resp" >&2
+    fail "$label LookupRoute did not return the expected fail-open AFFINITY_HINT — a non-matching adapter partition must downgrade to a stable affinity pick, never error and never PREFIX_MATCH"
+  fi
+}
+
+assert_adapter_no_alias "$adapter_other_payload" "$LOG_DIR/grpcurl-adapter-other.err" \
+  "different-adapter" \
+  "LookupRoute for adapter_id=smoke-lora-b matched a prefix ingested under smoke-lora-a — identical token content aliased across adapters, which would route to a replica holding a DIFFERENT adapter's KV"
+assert_adapter_no_alias "$adapter_none_payload" "$LOG_DIR/grpcurl-adapter-none.err" \
+  "no-adapter" \
+  "LookupRoute without adapter_id matched an adapter-scoped prefix — adapter-scoped ingest must stay out of the default partition"
+log "adapter partition enforced end-to-end: same adapter_id → PREFIX_MATCH; a different adapter_id and an absent adapter_id each returned the fail-open AFFINITY_HINT (RPC succeeded, off the prefix-match path) on the identical prefix hash"
+
 # --- routingFloorScore end-to-end probe ------------------------------------
 # Proves the new field flows CR → controller flatten → /policy push → server
 # resolver → buildLookupResponse downgrade. The same 64-token prefix that
@@ -3643,4 +3755,4 @@ log "Mooncake engine pod: hostNetwork=true, dnsPolicy=ClusterFirstWithHostNet, w
 
 kubectl delete namespace "$MOONCAKE_SMOKE_NS" --ignore-not-found --wait=false >/dev/null 2>&1 || true
 
-log "PASS — install bundle came up, CacheIndex + CacheTenant status writing, PromptTemplate + PDTopology schema-only surfaces, server HTTP surface, CachePolicy push adoption, gRPC fail-open (plaintext default), CacheBackend ↔ engine-pod binding signals + drift cadence, spec.resources defaults + thread-through, External backend end-to-end, /snapshot + /policy + /probe unauth rejection, audience binding on all three endpoints, the opt-in gRPC TLS overlay (incl. the existing LookupRoute call pattern over TLS), kernel-check injection shape + report-only FAIL condition path (EngineKernelsHealthy=False/KernelLoadFailed), the operator 'inferencecache doctor' CLI against the live install, the managed Mooncake backend provisioning contract (stand-in master reaches Available on hostNetwork behind a headless Service, Recreate strategy, mooncakestore:// RPC endpoint in status) plus the engineHostNetwork opt-in end-to-end (warning fires only without it; a matched engine pod is admitted onto hostNetwork with ClusterFirstWithHostNet and the mooncakestore:// connector, while a non-Mooncake engine pod stays on the pod network; real engine KV transfer is NOT exercised here), and every config/samples/ manifest applies cleanly — all work"
+log "PASS — install bundle came up, CacheIndex + CacheTenant status writing, PromptTemplate + PDTopology schema-only surfaces, server HTTP surface, CachePolicy push adoption, gRPC fail-open (plaintext default), adapter (LoRA) index partitioning on LookupRoute, CacheBackend ↔ engine-pod binding signals + drift cadence, spec.resources defaults + thread-through, External backend end-to-end, /snapshot + /policy + /probe unauth rejection, audience binding on all three endpoints, the opt-in gRPC TLS overlay (incl. the existing LookupRoute call pattern over TLS), kernel-check injection shape + report-only FAIL condition path (EngineKernelsHealthy=False/KernelLoadFailed), the operator 'inferencecache doctor' CLI against the live install, the managed Mooncake backend provisioning contract (stand-in master reaches Available on hostNetwork behind a headless Service, Recreate strategy, mooncakestore:// RPC endpoint in status) plus the engineHostNetwork opt-in end-to-end (warning fires only without it; a matched engine pod is admitted onto hostNetwork with ClusterFirstWithHostNet and the mooncakestore:// connector, while a non-Mooncake engine pod stays on the pod network; real engine KV transfer is NOT exercised here), and every config/samples/ manifest applies cleanly — all work"