feat(controller): surface a sizing warning when the lmcache-server is OOMKilled#126
Open
EdHasNoLife wants to merge 1 commit into
Open
feat(controller): surface a sizing warning when the lmcache-server is OOMKilled#126EdHasNoLife wants to merge 1 commit into
EdHasNoLife wants to merge 1 commit into
Conversation
… OOMKilled The standalone lmcache-server's memory is spec.resources (defaulted to a 4Gi request / 8Gi limit). For a large model the KV working set far exceeds 8Gi, so the server is silently OOMKilled under T2 write load — an under-provisioning break that previously looked identical to a healthy-but-idle backend. When a managed backend's workload is ReplicasUnavailable, the reconciler now checks whether a cache-server container was OOMKilled (current or last termination state, scoped to the owned Deployment's containers) and folds an actionable diagnostic into the Ready/Degraded condition message — "cache-server ... OOMKilled ... raise spec.resources.limits.memory". It rides the existing BackendDegraded Warning event (no new event/reason/limiter), so under- provisioning is loud and diagnosable instead of silent. - detectServerOOM helper (best-effort; no pod List on the healthy path) - message enrichment at the managedReadiness call site (reason unchanged, so the Degraded gate + Progressing derivation are untouched) - unit test (detection: current/last state, sidecar-scoping, healthy) + an end-to-end test (ReplicasUnavailable + OOM pod -> enriched Ready/Degraded msg)
Codex reviewFiles reviewed
FindingsBlockingNone. Should-fix
NitNone. Per-category coverage
Verdictchanges-requested |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Surface a sizing warning when the standalone lmcache-server is OOMKilled — under-provisioning was previously a silent break (the 8Gi
spec.resourcesdefault OOMs on large models; the 32B/TP=2 dig needed 120Gi).When a managed backend is
ReplicasUnavailable, the reconciler checks whether a cache-server container was OOMKilled (current or last termination state, scoped to the owned Deployment's containers) and folds an actionable diagnostic into theReady/Degradedcondition message — "cache-server … OOMKilled … raisespec.resources.limits.memory". It rides the existingBackendDegradedWarning event (no new event/reason/rate-limiter); the reason staysReplicasUnavailable, so the Degraded gate + Progressing derivation are untouched.detectServerOOMhelper — best-effort (a list error returns no diagnostic), no pod List on the healthy path, scoped to the owned Deployment's container names (a crashed service-mesh sidecar is not misread)Ready/Degradedmessage)Degradedcondition note incachebackend-api.mdThe "surface a warning" half of the server-memory sizing sub-ask of the tier-2-health work — the prevention complement to the
BackendT2Degradedalert's detection.