diff --git a/config/observability/alerting-rules.yaml b/config/observability/alerting-rules.yaml index c12f8c51..39e886b0 100644 --- a/config/observability/alerting-rules.yaml +++ b/config/observability/alerting-rules.yaml @@ -76,12 +76,16 @@ groups: # BOTH. Upstream docs list the unsuffixed names at # https://docs.vllm.ai/en/latest/usage/metrics/; the series # have been present since vLLM ~0.18 (the release that added - # LMCache-style offload support). This operator does not - # currently scrape them - # (pkg/adapters/engine/metrics_scraper.go only reads - # vllm:prefix_cache_{hits,queries} for T1 plus - # vllm:*_cache_usage_perc), so the alert binds directly to - # vLLM's exposition. Operators should sanity-check the metric + # LMCache-style offload support). The operator's scraper + # (pkg/adapters/engine/metrics_scraper.go) now ALSO reads these + # external counters and projects them per-CacheBackend as the + # inferencecache_backend_t2_hit_rate gauge + the T2Degraded + # condition + the BackendT2Degraded alert in the + # inferencecache-cachebackend group below — prefer that for clean + # per-CacheBackend attribution. This per-pod alert still binds + # directly to vLLM's exposition for pod-level granularity and to + # stay useful where the operator's scrape path isn't wired. + # Operators should sanity-check the metric # exists on at least one engine pod before relying on the # alert: see the runbook's "verify the metric is exposed" # step. @@ -240,6 +244,55 @@ groups: summary: "Synthetic CacheBackend probe failing on {{ $labels.backend }} in namespace {{ $labels.namespace }} (stage={{ $labels.stage }})." runbook_url: "https://github.com/cachebox-project/inference-cache/blob/main/docs/observability/alerts.md#serverprobefail" + - alert: BackendT2Degraded + # Per-CacheBackend tier-2 (external offload) health, read from the + # controller-emitted `inferencecache_backend_t2_hit_rate` gauge (the + # CacheIndex poller projects it from status.indexParticipation.t2HitRate, + # computed from the engine's vllm:external_prefix_cache_{hits,queries} + # counters). A value of 0 means the tier was queried but has served ZERO + # reloads since it was first exercised — the silent-degradation signal + # (offload store/connection failure, an under-sized offload server, or a + # scheduler/worker hash mismatch). + # + # This is the per-CacheBackend counterpart to LMCacheT2NoHits above. That + # alert reads vLLM's per-POD exposition directly and so needs a vLLM + # PodMonitor scoped to cache-bound pods (see its SCRAPE-SCOPING note); + # this one binds to the operator's own gauge, whose `backend` label is the + # canonical / — so attribution is per-CacheBackend with + # no scrape-scoping caveat. It is the Alertmanager-side companion of the + # advisory T2Degraded CR condition (CacheBackend.status, not scraped) — + # but BROADER: the poller projects these metrics for ANY exercised backend, + # so this alert also fires for External backends, whose T2Degraded + # condition the controller does not manage. The condition is the + # managed-backend status view; this alert is the Prometheus view over + # every exercised backend. + # + # The query-token rate is the ACTIVITY GATE: it keeps a backend that took + # a few cold misses and then went idle from paging — its query counter goes + # flat (rate ~0, below the floor), so only a backend with sustained tier-2 + # traffic AND zero reloads alerts. The 1000 tokens/sec floor matches the + # per-pod LMCacheT2NoHits (the counters are TOKENS, not requests — one + # ~1000-token prefixed request/sec clears it; idle replicas stay under). + # `max by` collapses the controller pod/instance labels (and any HA-replica + # duplicates), leaving a clean (namespace, backend) alert; `namespace` is + # the install scope Prometheus injects from the controller scrape target, + # so installs stay isolated. (inferencecache_backend_t2_query_tokens_total + # is a MONOTONIC counter the controller builds by accumulating only the + # positive per-tick deltas of the per-backend aggregate, so a replica/tenant + # drain or engine restart is clamped out and rate() never sees a phantom + # reset.) + expr: | + (max by (namespace, backend) (rate(inferencecache_backend_t2_query_tokens_total[10m])) > 1000) + and on (namespace, backend) + (max by (namespace, backend) (inferencecache_backend_t2_hit_rate) == 0) + for: 5m + labels: + severity: warning + component: inference-cache + annotations: + summary: "Tier-2 (external offload) on CacheBackend {{ $labels.backend }} is serving queries but zero reloads — silently degraded (store/connection failure, under-sized offload server, or scheduler/worker hash mismatch)." + runbook_url: "https://github.com/cachebox-project/inference-cache/blob/main/docs/observability/alerts.md#backendt2degraded" + # ---------------------------------------------------------------------------- # DEFERRED — the metrics these alerts consume are not yet exposed on # /metrics. Enable each rule in the same change that ships its metric. diff --git a/config/observability/prometheus-rules-tests.yaml b/config/observability/prometheus-rules-tests.yaml index 3af6dff6..74ed1f54 100644 --- a/config/observability/prometheus-rules-tests.yaml +++ b/config/observability/prometheus-rules-tests.yaml @@ -1,8 +1,9 @@ -# Unit tests for the alerting rules in alerting-rules.yaml. Each Stage 1 -# alert has both a "should fire" and a "should not fire" case; together they -# pin the threshold and confirm the alert is not spuriously firing on healthy -# baselines. Run with `promtool test rules prometheus-rules-tests.yaml` (or -# via `make verify-prometheus`). +# Unit tests for the alerting rules in alerting-rules.yaml. Each active alert +# — the server-side Stage 1 alerts plus the controller-side ServerProbeFail and +# BackendT2Degraded — has "should fire" and "should not fire" cases; together +# they pin the threshold and confirm the alert is not spuriously firing on +# healthy baselines. Run with `promtool test rules prometheus-rules-tests.yaml` +# (or via `make verify-prometheus`). # # Synthetic time-series notation: `N+Mx20` is "start at N, add M every step, # produce 21 samples total". The default step is `interval:` (30s here). For @@ -479,3 +480,116 @@ tests: - eval_time: 15m alertname: ServerProbeFail exp_alerts: [] + + # -------------------------------------------------------------------------- + # BackendT2Degraded (for: 5m, warning) + # + # Per-CacheBackend tier-2 health from two controller series: + # inferencecache_backend_t2_query_tokens_total (cumulative; the ACTIVITY GATE via + # rate(); backend=/) and + # inferencecache_backend_t2_hit_rate (the ratio; == 0 means zero reloads). + # `namespace` is the install scope Prometheus injects from the controller + # scrape. Fires only on rate(query_tokens) > 1000 tokens/sec AND hit_rate == 0. + # query_tokens 0+60000x40 = 2000 tokens/sec (above the floor); 40 samples at + # 30s = 20m, past the [10m] rate window plus the 5m `for:` dwell. + # -------------------------------------------------------------------------- + - interval: 30s + name: BackendT2Degraded fires when a queried backend serves zero reloads + input_series: + - series: 'inferencecache_backend_t2_query_tokens_total{namespace="inference-cache-system",backend="team-a/sample"}' + values: '0+60000x40' + - series: 'inferencecache_backend_t2_hit_rate{namespace="inference-cache-system",backend="team-a/sample"}' + values: '0+0x40' + alert_rule_test: + # Before the 5m `for:` dwell elapses the alert is only pending, not firing, + # even though the rate gate is already satisfied — this pins the dwell. + - eval_time: 3m + alertname: BackendT2Degraded + exp_alerts: [] + - eval_time: 20m + alertname: BackendT2Degraded + exp_alerts: + - exp_labels: + severity: warning + component: inference-cache + namespace: inference-cache-system + backend: team-a/sample + exp_annotations: + summary: "Tier-2 (external offload) on CacheBackend team-a/sample is serving queries but zero reloads — silently degraded (store/connection failure, under-sized offload server, or scheduler/worker hash mismatch)." + runbook_url: "https://github.com/cachebox-project/inference-cache/blob/main/docs/observability/alerts.md#backendt2degraded" + + - interval: 30s + name: BackendT2Degraded does not fire when the backend is serving reloads (hit-rate > 0) + input_series: + - series: 'inferencecache_backend_t2_query_tokens_total{namespace="inference-cache-system",backend="team-a/sample"}' + values: '0+60000x40' + - series: 'inferencecache_backend_t2_hit_rate{namespace="inference-cache-system",backend="team-a/sample"}' + values: '0.75+0x40' + alert_rule_test: + - eval_time: 20m + alertname: BackendT2Degraded + exp_alerts: [] + + - interval: 30s + # THE activity gate: a backend that took a few cold misses (hit-rate 0) and + # then went idle (query gauge flat -> rate 0, below the 1000/sec floor) must + # NOT page. This is the false positive a cumulative-only `== 0` form would hit. + name: BackendT2Degraded does not fire on an idle backend even with hit-rate zero + input_series: + - series: 'inferencecache_backend_t2_query_tokens_total{namespace="inference-cache-system",backend="team-a/sample"}' + values: '500+0x40' + - series: 'inferencecache_backend_t2_hit_rate{namespace="inference-cache-system",backend="team-a/sample"}' + values: '0+0x40' + alert_rule_test: + - eval_time: 20m + alertname: BackendT2Degraded + exp_alerts: [] + + - interval: 30s + # Coverage for the 1000 tokens/sec floor: a low-traffic backend (500/sec, + # below the floor) with zero reloads must NOT page. + name: BackendT2Degraded does not fire below the query-rate floor + input_series: + - series: 'inferencecache_backend_t2_query_tokens_total{namespace="inference-cache-system",backend="team-a/sample"}' + values: '0+15000x40' + - series: 'inferencecache_backend_t2_hit_rate{namespace="inference-cache-system",backend="team-a/sample"}' + values: '0+0x40' + alert_rule_test: + - eval_time: 20m + alertname: BackendT2Degraded + exp_alerts: [] + + - interval: 30s + # Two installs sharing one Prometheus, each with an actively-queried zero-hit + # backend of the same name in a different install namespace. Both alert + # independently -- `max by (namespace, backend)` preserves the install scope. + name: BackendT2Degraded keeps installs separate when backend names collide + input_series: + - series: 'inferencecache_backend_t2_query_tokens_total{namespace="install-a",backend="team-a/cache"}' + values: '0+60000x40' + - series: 'inferencecache_backend_t2_hit_rate{namespace="install-a",backend="team-a/cache"}' + values: '0+0x40' + - series: 'inferencecache_backend_t2_query_tokens_total{namespace="install-b",backend="team-a/cache"}' + values: '0+60000x40' + - series: 'inferencecache_backend_t2_hit_rate{namespace="install-b",backend="team-a/cache"}' + values: '0+0x40' + alert_rule_test: + - eval_time: 20m + alertname: BackendT2Degraded + exp_alerts: + - exp_labels: + severity: warning + component: inference-cache + namespace: install-a + backend: team-a/cache + exp_annotations: + summary: "Tier-2 (external offload) on CacheBackend team-a/cache is serving queries but zero reloads — silently degraded (store/connection failure, under-sized offload server, or scheduler/worker hash mismatch)." + runbook_url: "https://github.com/cachebox-project/inference-cache/blob/main/docs/observability/alerts.md#backendt2degraded" + - exp_labels: + severity: warning + component: inference-cache + namespace: install-b + backend: team-a/cache + exp_annotations: + summary: "Tier-2 (external offload) on CacheBackend team-a/cache is serving queries but zero reloads — silently degraded (store/connection failure, under-sized offload server, or scheduler/worker hash mismatch)." + runbook_url: "https://github.com/cachebox-project/inference-cache/blob/main/docs/observability/alerts.md#backendt2degraded" diff --git a/config/observability/prometheus-rules.yaml b/config/observability/prometheus-rules.yaml index 5f768f4b..596f57fd 100644 --- a/config/observability/prometheus-rules.yaml +++ b/config/observability/prometheus-rules.yaml @@ -165,6 +165,28 @@ spec: summary: "Synthetic CacheBackend probe failing on {{ $labels.backend }} in namespace {{ $labels.namespace }} (stage={{ $labels.stage }})." runbook_url: "https://github.com/cachebox-project/inference-cache/blob/main/docs/observability/alerts.md#serverprobefail" + - alert: BackendT2Degraded + # Per-CacheBackend tier-2 (external offload) silent-degradation signal: + # the controller's inferencecache_backend_t2_hit_rate gauge is 0, i.e. + # the tier was queried but has served zero reloads. Per-CacheBackend + # counterpart to the per-pod LMCacheT2NoHits (no scrape-scoping caveat); + # companion to the advisory T2Degraded CR condition (broader: also fires + # for External backends, whose condition the controller does not manage). + # The query-token rate is + # the activity gate (an idle backend's flat counter → rate ~0 → no page). + # See the flat alerting-rules.yaml for the full rationale. + expr: | + (max by (namespace, backend) (rate(inferencecache_backend_t2_query_tokens_total[10m])) > 1000) + and on (namespace, backend) + (max by (namespace, backend) (inferencecache_backend_t2_hit_rate) == 0) + for: 5m + labels: + severity: warning + component: inference-cache + annotations: + summary: "Tier-2 (external offload) on CacheBackend {{ $labels.backend }} is serving queries but zero reloads — silently degraded (store/connection failure, under-sized offload server, or scheduler/worker hash mismatch)." + runbook_url: "https://github.com/cachebox-project/inference-cache/blob/main/docs/observability/alerts.md#backendt2degraded" + # ------------------------------------------------------------------------ # DEFERRED — the metrics these alerts consume are not yet exposed on # /metrics. Enable each rule in the same change that ships its metric. diff --git a/docs/observability/alerts.md b/docs/observability/alerts.md index c529d6ef..89d27dfd 100644 --- a/docs/observability/alerts.md +++ b/docs/observability/alerts.md @@ -29,12 +29,13 @@ There are two distribution shapes, same rule set, drift-gated by favor of explicit `ServiceMonitor` / `PodMonitor` CRs. 2. A [`PodMonitor`](../../config/observability/podmonitor.yaml) that tells Prometheus to scrape the controller pod's `:8080/metrics`. - Required for the controller-side alerts (`ServerProbeFail` reads - `inferencecache_backend_probe_result_total`, which the - CacheBackend reconciler emits; the existing - `inferencecache_backend_server_restart_cascades_total` is also - controller-emitted). Without this, those rules load but never - have a series to evaluate. + Required for the controller-side alerts: `ServerProbeFail` reads + `inferencecache_backend_probe_result_total` and `BackendT2Degraded` + reads `inferencecache_backend_t2_hit_rate` + + `inferencecache_backend_t2_query_tokens_total` — all controller-emitted + (as is the not-yet-alerted-on + `inferencecache_backend_server_restart_cascades_total`). Without this, + those rules load but never have a series to evaluate. 3. The [`PrometheusRule`](../../config/observability/prometheus-rules.yaml) carrying the alerts. @@ -79,9 +80,10 @@ There are two distribution shapes, same rule set, drift-gated by `inference-cache-server` pod (server-side series: index, lookup, auth) AND the `inference-cache-controller-manager` pod (controller-side series: per-stage probe-result counter, cache-server restart-cascade - counter). Server-only scrape leaves the controller-side alerts - (`ServerProbeFail` today) loaded but inert — they read - `inferencecache_backend_probe_result_total` which is controller-emitted. + counter, and the tier-2 hit-rate gauge + query-token counter). Server-only scrape leaves the controller-side alerts + (`ServerProbeFail`, `BackendT2Degraded`) loaded but inert — they read + controller-emitted series (`inferencecache_backend_probe_result_total`, + `inferencecache_backend_t2_hit_rate`, `inferencecache_backend_t2_query_tokens_total`). To keep the alerts' per-install scoping working, both scrapes must inject a `namespace` label. Two valid shapes: 1. **Recommended** — Kubernetes service discovery @@ -106,8 +108,8 @@ There are two distribution shapes, same rule set, drift-gated by shapes (1) and (2) require you to wire BOTH scrape entries explicitly when you are not on prometheus-operator. -Both files contain the same six active alerts (five Stage 1 alerts plus -the controller-side `ServerProbeFail`) plus commented-out placeholders for +Both files contain the same seven active alerts (five Stage 1 alerts plus +two controller-side, `ServerProbeFail` and `BackendT2Degraded`) plus commented-out placeholders for two more that depend on metrics not yet exposed (see [Deferred alerts](#deferred-alerts) below). @@ -158,11 +160,11 @@ alerts](#deferred-alerts) below). > interval: 30s > ``` > -> The other five alerts work as-is once this bundle is applied — they +> The other six alerts work as-is once this bundle is applied — they > only read `inferencecache_*` series, which the shipped ServiceMonitor > (server-side: `IndexEmpty`, `LookupRouteDegenerate`, > `LookupRouteHighTimeout`, `IndexEvictionsSpike`) and PodMonitor -> (controller-side: `ServerProbeFail`) cover between them. +> (controller-side: `ServerProbeFail`, `BackendT2Degraded`) cover between them. > > **The alerts rely on a `namespace` label per install.** Both the shipped > `ServiceMonitor` for `inference-cache-server` and any prometheus-operator @@ -332,12 +334,24 @@ the series have been present since vLLM 0.18 (the first release tagged in the upstream v0.18 docs page). Our alert and triage queries accept both the unsuffixed and `_total` forms via `{__name__=~"...(_total)?"}`. -This operator has no in-process scrape of those upstream metrics — its -own scraper (`pkg/adapters/engine/metrics_scraper.go`) only reads the T1 -`vllm:prefix_cache_{hits,queries}` plus `vllm:*_cache_usage_perc`. That -means the alert binds directly to vLLM's exposition, and an upstream -rename, deprecation, or version skew can silently make the alert inert -while `promtool test rules` (which uses synthetic series) still passes. +The operator's own scraper (`pkg/adapters/engine/metrics_scraper.go`) +now ALSO reads these external counters (alongside the T1 +`vllm:prefix_cache_{hits,queries}` and `vllm:*_cache_usage_perc`) and +projects them per-CacheBackend as the `inferencecache_backend_t2_hit_rate` +gauge and the `T2Degraded` condition — see +[`BackendT2Degraded`](#backendt2degraded), the per-CacheBackend +counterpart to this alert. This per-POD alert still binds directly to +vLLM's exposition: it gives pod-level granularity and keeps working even +where the operator's index/scrape path isn't wired. The tradeoff is that +an upstream rename, deprecation, or version skew can silently make THIS +alert inert while `promtool test rules` (which uses synthetic series) +still passes — prefer `BackendT2Degraded` for clean per-CacheBackend +attribution where it's available. **But do not retire this per-pod alert:** +`BackendT2Degraded` reads the *lifetime-cumulative* hit-rate, so its `== 0` +only catches a tier that has **never** served a reload; a mid-life regression +(served reloads before, now zero under continuing queries) keeps that cumulative +ratio > 0 and is caught *only* by this windowed, rate-based `LMCacheT2NoHits`. +Run both. **Operator responsibility:** before enabling the alert in production, confirm at least one engine pod publishes the series: @@ -700,6 +714,108 @@ sum by (backend, stage, result) ( --- +### `BackendT2Degraded` + +- **Severity**: `warning` +- **For**: 5 minutes +- **Source metrics** (both controller-emitted, `backend` = canonical + `/`; Prometheus injects the install `namespace` from the + controller scrape target): + - `inferencecache_backend_t2_query_tokens_total` — cumulative tier-2 query tokens; + `rate()` over it is the **activity gate**. + - `inferencecache_backend_t2_hit_rate` — the reload hit-rate (`== 0` = zero + reloads), projected from `status.indexParticipation.t2HitRate` (computed + from the engine's `vllm:external_prefix_cache_{hits,queries}` counters). +- **Expr**: fires when `rate(query_tokens[10m]) > 1000` tokens/sec **and** + `hit_rate == 0`. + +A CacheBackend's tier-2 (external offload, e.g. LMCache) is actively +being queried but recalling **zero** reloads — its hit-rate gauge is sitting at +exactly 0 while tier-2 query traffic continues. The offload tier looks "wired" +(the CacheBackend is `Ready`, stores are landing) but nothing is ever recalled: +every offload `put` is wasted work and the engine refills T2 forever without +benefiting from it. This is the per-CacheBackend, controller-attributed +counterpart to [`LMCacheT2NoHits`](#lmcachet2nohits) (which reads vLLM's per-pod +exposition directly and needs a scoped vLLM PodMonitor); it is the +Alertmanager-side companion of the advisory `T2Degraded` condition on +`CacheBackend.status` (which Prometheus does not scrape) — but **broader**: the +poller projects these metrics for *any* exercised CacheBackend, so this alert +also fires for `External`-type backends, whose `T2Degraded` condition the +controller does not manage. The condition is the managed-backend status view; +this alert is the Prometheus view over every exercised backend. + +> **Why the activity gate matters.** The hit-rate gauge is cumulative, so a +> backend that took a few cold misses and then went idle keeps exporting `0` +> long after it stopped serving traffic — `hit_rate == 0` alone would page it as +> "degraded." The `rate(query_tokens) > 1000` tokens/sec clause is the gate: an +> idle backend's query gauge goes flat (`rate ≈ 0`), so only a backend with +> sustained tier-2 traffic *and* zero reloads alerts. (The 1000 tokens/sec floor +> matches `LMCacheT2NoHits` — the counters are tokens, not requests.) A backend +> that has never been queried exports no series and cannot trip the alert; an +> engine restart resets its cumulative counters, which `rate()` handles. + +#### Likely causes + +The same failure class as [`LMCacheT2NoHits`](#lmcachet2nohits), surfaced +per-CacheBackend: + +1. **Scheduler/worker hash mismatch under tensor parallelism** — e.g. + `PYTHONHASHSEED` unset on a TP>1 engine, so the scheduler's lookup hashes + never match the workers' stored hashes. Stores succeed, reloads are always 0. +2. **Offload server under-provisioned / OOMKilled** — a large model's KV working + set exceeds the standalone offload server's memory; stores fail (broken pipe) + and nothing is recallable. +3. **Client/server version skew** in the offload subsystem (old client library + vs newer server image, or vice versa) — the wire handshake succeeds but + `put`/`get` opcodes diverge. +4. **External backend down / wrong endpoint** for an `External`-type CacheBackend. + +#### First-response runbook + +```bash +# 1. Identify the degraded backend — the alert's `backend` label is /. +kubectl get cachebackend -A + +# 2. (Managed backends) Confirm the symptom on the CR — the advisory T2Degraded +# condition mirrors the gauge for controller-managed backends: +kubectl get cachebackend -n \ + -o jsonpath='{range .status.conditions[?(@.type=="T2Degraded")]}{.status} {.reason}{"\n"}{end}' +# True / T2ZeroHitRate == the tier was queried but served zero reloads. +# External-type backends have NO T2Degraded condition (the controller does not +# manage their status) — confirm those via the gauge / PromQL in "Triage queries" +# below; a missing condition there is expected, not a sign the alert is bogus. + +# 3. For a managed LMCache backend, check the engine pods for the TP>1 hash +# mismatch (cause 1) and the offload server for OOM (cause 2): +kubectl get pod -l \ + -o jsonpath='{.items[*].spec.containers[0].env[?(@.name=="PYTHONHASHSEED")].value}' +kubectl get pod -l \ + -o jsonpath='{range .items[*]}{.status.containerStatuses[0].lastState.terminated.reason}{"\n"}{end}' +# OOMKilled on the server == cause 2 (raise its memory request/limit). +``` + +Triage queries: + +```promql +# The degraded backends right now — the alert's own expr (actively queried AND +# recalling zero reloads): +(max by (namespace, backend) (rate(inferencecache_backend_t2_query_tokens_total[10m])) > 1000) +and on (namespace, backend) +(max by (namespace, backend) (inferencecache_backend_t2_hit_rate) == 0) + +# Hit-rate and query-token rate across exercised backends, for context: +max by (namespace, backend) (inferencecache_backend_t2_hit_rate) +max by (namespace, backend) (rate(inferencecache_backend_t2_query_tokens_total[10m])) +``` + +The alert clears the moment the offload tier serves reloads again: a successful +reload lifts `inferencecache_backend_t2_hit_rate` above 0, dropping the `== 0` +clause — no engine restart or counter reset required. It also clears if tier-2 +query traffic falls below the rate floor, or if the backend is decommissioned and +its series drain out of the index. + +--- + ## Deferred alerts Two more alerts are scoped to ship as part of the same observability @@ -718,14 +834,14 @@ metric. ## How alerts compose -The five Stage 1 alerts are not independent — they map onto a small set +These active alerts are not independent — they map onto a small set of recurring failure modes: | Failure mode | Alerts that fire | Where to look first | |---|---|---| | Subscriber sidecar not injected | `IndexEmpty` (critical) | controller flags + webhook config | | Engine prefix-cache off | `IndexEmpty` (critical) | engine flags `--enable-prefix-caching` + `--kv-events-config` | -| Offload tier version skew | `LMCacheT2NoHits` (warning) + maybe `LookupRouteDegenerate` if T2 is the only cache path | offload client/server image tags | +| Offload tier silently degraded (version skew, server OOM, or `PYTHONHASHSEED` hash mismatch) | `BackendT2Degraded` (warning, per-CacheBackend) + `LMCacheT2NoHits` (warning, per-pod) + maybe `LookupRouteDegenerate` if T2 is the only cache path | `CacheBackend.status` `T2Degraded` condition; offload client/server image tags; offload server memory | | Tenant or `hash_scheme` mismatch | `LookupRouteDegenerate` (warning) alongside `inferencecache_index_entries > 0` | gateway-side request shape | | Server overload | `LookupRouteHighTimeout` (warning) + maybe `LookupRouteDegenerate` | `inferencecache_lookup_route_latency_seconds` p99, `process_resident_memory_bytes`, server's global `MaxEntries` | | Working set outgrew config | `IndexEvictionsSpike` (info) | server's global `MaxEntries` vs. observed working-set size; `CacheTenant.spec.quota.maxIndexEntries` per tenant |