Skip to content

feat(controller): surface a sizing warning when the lmcache-server is OOMKilled#126

Open
EdHasNoLife wants to merge 1 commit into
mainfrom
cac-204-server-oom-warning
Open

feat(controller): surface a sizing warning when the lmcache-server is OOMKilled#126
EdHasNoLife wants to merge 1 commit into
mainfrom
cac-204-server-oom-warning

Conversation

@EdHasNoLife

Copy link
Copy Markdown
Collaborator

Surface a sizing warning when the standalone lmcache-server is OOMKilled — under-provisioning was previously a silent break (the 8Gi spec.resources default OOMs on large models; the 32B/TP=2 dig needed 120Gi).

When a managed backend is ReplicasUnavailable, the reconciler checks whether a cache-server container was OOMKilled (current or last termination state, scoped to the owned Deployment's containers) and folds an actionable diagnostic into the Ready/Degraded condition message — "cache-server … OOMKilled … raise spec.resources.limits.memory". It rides the existing BackendDegraded Warning event (no new event/reason/rate-limiter); the reason stays ReplicasUnavailable, so the Degraded gate + Progressing derivation are untouched.

  • detectServerOOM helper — best-effort (a list error returns no diagnostic), no pod List on the healthy path, scoped to the owned Deployment's container names (a crashed service-mesh sidecar is not misread)
  • unit test (current/last termination state, sidecar-scoping, healthy) + an end-to-end test (ReplicasUnavailable + OOM pod → enriched Ready/Degraded message)
  • doc: the Degraded condition note in cachebackend-api.md

The "surface a warning" half of the server-memory sizing sub-ask of the tier-2-health work — the prevention complement to the BackendT2Degraded alert's detection.

… OOMKilled

The standalone lmcache-server's memory is spec.resources (defaulted to a 4Gi
request / 8Gi limit). For a large model the KV working set far exceeds 8Gi, so
the server is silently OOMKilled under T2 write load — an under-provisioning
break that previously looked identical to a healthy-but-idle backend.

When a managed backend's workload is ReplicasUnavailable, the reconciler now
checks whether a cache-server container was OOMKilled (current or last
termination state, scoped to the owned Deployment's containers) and folds an
actionable diagnostic into the Ready/Degraded condition message — "cache-server
... OOMKilled ... raise spec.resources.limits.memory". It rides the existing
BackendDegraded Warning event (no new event/reason/limiter), so under-
provisioning is loud and diagnosable instead of silent.

- detectServerOOM helper (best-effort; no pod List on the healthy path)
- message enrichment at the managedReadiness call site (reason unchanged, so the
  Degraded gate + Progressing derivation are untouched)
- unit test (detection: current/last state, sidecar-scoping, healthy) + an
  end-to-end test (ReplicasUnavailable + OOM pod -> enriched Ready/Degraded msg)
@linear-code

linear-code Bot commented Jun 29, 2026

Copy link
Copy Markdown

CAC-204

@github-actions

Copy link
Copy Markdown

Codex review

Files reviewed

  • docs/design/
    • docs/design/cachebackend-api.md [reviewed]
  • internal/controller/
    • internal/controller/cachebackend_controller.go [reviewed]
    • internal/controller/cachebackend_controller_test.go [reviewed]
    • internal/controller/cachebackend_server_restart.go [reviewed]
    • internal/controller/cachebackend_server_restart_test.go [reviewed]

Findings

Blocking

None.

Should-fix

internal/controller/cachebackend_server_restart.go:934detectServerOOM attributes any pod with the backend selector labels to this backend, but does not verify the pod is actually owned by the owned Deployment. A foreign/manual/stale pod in the namespace with copied labels and a matching container name can falsely enrich Ready/Degraded and the BackendDegraded event with an OOM sizing diagnosis; this should reuse the existing pod -> ReplicaSet -> Deployment ownership check or otherwise authenticate the pod set.

Nit

None.

Per-category coverage

  1. Vendor-neutral naming: Clean
  2. Contract & spec / docs sync: Clean
  3. gRPC fail-open semantics: Clean
  4. Quality (tests / error wrapping / package placement / generated drift): findings above at: internal/controller/cachebackend_server_restart.go:934

Verdict

changes-requested

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant