Skip to content

Add StatefulSet CacheBackend support #112

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
heymrbox wants to merge 11 commits into
base: main
Choose a base branch
from
Open

Add StatefulSet CacheBackend support #112

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
1c72d00
Add StatefulSet CacheBackend support
heymrbox Jun 12, 2026
a560a88
Surface immutable StatefulSet storage drift
heymrbox Jun 12, 2026
f5fe047
Warn on retained shared PVCs
heymrbox Jun 12, 2026
f9bfbc1
Normalize StatefulSet PVC drift checks
heymrbox Jun 12, 2026
ed1b9e3
Reconcile StatefulSet retention drift
heymrbox Jun 12, 2026
e6b4535
Clear status on workload replacement failure
heymrbox Jun 12, 2026
2484b08
Cascade StatefulSet server restarts
heymrbox Jun 12, 2026
49c82e5
Clarify StatefulSet storage mutability docs
heymrbox Jun 12, 2026
9564625
Keep StatefulSet drift maintenance active
heymrbox Jun 12, 2026
88509f4
Validate StatefulSet PVC names
heymrbox Jun 12, 2026
b0e082a
Document StatefulSet PVC validation
heymrbox Jun 12, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
92 changes: 47 additions & 45 deletions api/v1alpha1/cachebackend_types.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,22 +46,24 @@ const (
// (spec.autoscaling) are both reconciled today. The autoscaling spec drives a
// HorizontalPodAutoscaler. On a managed Deployment backend
// (deploymentKind=Deployment, the default), spec.storage.pvc provisions a
// PersistentVolumeClaim owner-referenced to the CacheBackend (reclaimed only
// when the CacheBackend is deleted), mounts it into the cache-server pod at the
// runtime adapter's declared data path, and reports the bound PVC's actual size
// in status.capacity.
// single PersistentVolumeClaim owner-referenced to the CacheBackend (reclaimed
// only when the CacheBackend is deleted), mounts it into the cache-server pod at
// the runtime adapter's declared data path, and reports the bound PVC's actual
// size in status.capacity. On a managed StatefulSet backend, spec.storage.pvc
// renders a volumeClaimTemplate instead, giving each replica its own
// ReadWriteOnce PVC.
//
// Two scoping rules apply:
//
// - Only a managed Deployment backend provisions storage. A StatefulSet
// backend is routed to the unmanaged path today (see DeploymentKind), so it
// provisions nothing, storage included; per-replica volumeClaimTemplates are
// a follow-up. An External backend provisions no workload at all, so
// spec.storage.pvc is a no-op there.
// - A persistent backend must be single-replica. A ReadWriteOnce PVC cannot be
// multi-attached, so replicas (or an autoscaling ceiling) > 1 is surfaced
// Ready=False/InvalidStorageConfiguration rather than provisioned;
// per-replica persistent storage via StatefulSet is a separate follow-up.
// - Managed Deployment backends provision one shared PVC. Managed StatefulSet
// backends provision per-replica PVCs via volumeClaimTemplates. External
// backends provision no workload at all, so spec.storage.pvc is a no-op
// there.
// - A persistent Deployment backend must be single-replica. A ReadWriteOnce
// PVC cannot be multi-attached, so replicas (or an autoscaling ceiling) > 1
// is surfaced Ready=False/InvalidStorageConfiguration rather than
// provisioned. Use deploymentKind=StatefulSet for multi-replica persistent
// storage.
//
// NOTE: provisioning + mounting the PVC does not by itself make the cache server
// spill KV to it — switching the LMCache server to a disk-backed storage device
Expand All @@ -79,9 +81,9 @@ type CacheBackendSpec struct {
Type CacheBackendType `json:"type,omitempty"`

// DeploymentKind identifies whether a managed backend is reconciled as a
// Deployment or StatefulSet. Defaults to Deployment — the only kind the
// Phase-1 reconciler templates; StatefulSet is reserved for future
// per-replica-PVC topologies and is a no-op today.
// Deployment or StatefulSet. Defaults to Deployment. StatefulSet uses the
// same adapter-rendered pod and service shape, but can back
// spec.storage.pvc with per-replica volumeClaimTemplates.
// +optional
// +kubebuilder:default=Deployment
DeploymentKind CacheBackendDeploymentKind `json:"deploymentKind,omitempty"`
Expand Down Expand Up @@ -563,12 +565,13 @@ type CacheBackendStatus struct {
Endpoint string `json:"endpoint,omitempty"`

// Capacity is a human-readable summary of the backend's provisioned
// capacity: the bound PersistentVolumeClaim's actual capacity when
// spec.storage.pvc is set and the PVC has bound (the real provisioned size,
// which may exceed the request), or empty for an ephemeral backend or while
// the PVC is still pending (e.g. a WaitForFirstConsumer StorageClass that
// binds only once the pod schedules). It is informational; clients must not
// parse it.
// capacity: for Deployment-backed persistent storage, the bound shared
// PersistentVolumeClaim's actual capacity when the PVC has bound (the real
// provisioned size, which may exceed the request). Empty for an ephemeral
// backend, a StatefulSet backend (per-replica PVCs have no aggregate
// capacity projection yet), or while the Deployment PVC is still pending
// (e.g. a WaitForFirstConsumer StorageClass that binds only once the pod
// schedules). It is informational; clients must not parse it.
// +optional
Capacity string `json:"capacity,omitempty"`

Expand Down Expand Up @@ -623,42 +626,41 @@ type CacheBackendStatus struct {
FirstKVEventObservedAt *metav1.Time `json:"firstKVEventObservedAt,omitempty"`

// FirstAvailableAt latches the first time the managed cache-backend
// workload was observed Available — the stable anchor for the
// firstEventTimeout clock. It is deliberately a latched timestamp rather
// than the live Deployment's Available condition LastTransitionTime: that
// condition resets on an availability flap, which would restart the
// workload was observed serving (Deployment Available or StatefulSet Ready)
// — the stable anchor for the firstEventTimeout clock. It is deliberately
// a latched timestamp rather than the live workload transition time: that
// transition resets on an availability flap, which would restart the
// timeout window and let a backend that already breached the timeout
// (Degraded / NoKVEventsObserved) bounce back to AwaitingFirstKVEvent
// without any KV event — contradicting the "once Degraded, stays Degraded
// until an event arrives" contract. Anchoring on this write-once value
// keeps the elapsed window monotonic, so Degraded is sticky. Written
// write-once when the workload first reports Available and never cleared
// write-once when the workload first reports serving and never cleared
// (inert while the backend is not managed). A genuinely recreated managed
// Deployment keeps the prior anchor; the gate re-evaluates from it, which
// is safe because the engine event source is unchanged by a cache-server
// workload keeps the prior anchor; the gate re-evaluates from it, which is
// safe because the engine event source is unchanged by a cache-server
// restart.
// +optional
FirstAvailableAt *metav1.Time `json:"firstAvailableAt,omitempty"`

// ObservedServerInstance is the controller's cascade-decision
// baseline — a stable identifier for the Ready cache-server pod
// set the controller last anchored against. NOT a live current-
// pod-set view: the controller intentionally pins this through
// transient rolling-update midpoints and through no-Ready
// windows so the cascade does not fire on rollbacks or transient
// outages. For the live pod inventory, operators should consult
// ObservedServerInstance is the controller's cascade-decision baseline — a
// stable identifier for the Ready cache-server pod set the controller last
// anchored against. NOT a live current-pod-set view: the controller
// intentionally pins this through transient rolling-update midpoints and
// through no-Ready windows so the cascade does not fire on rollbacks or
// transient outages. For the live pod inventory, operators should consult
// status.matchedEnginePods (engine side) and `kubectl get pod`
// (cache-server side).
//
// Shape: `<pod-uid>:<restart-sum>` per Ready pod, comma-joined
// and lex-sorted by pod name. restart-sum is the per-pod
// containerStatuses[].RestartCount summed across cache-server
// containers (the names from the owned Deployment's pod
// template; foreign sidecars are excluded). Inert and cleared
// for External backends, unsupported-runtime backends, and on
// the InvalidStorageConfiguration gate (a persistent
// multi-replica spec the controller refuses to provision until
// the operator scales to 1 or removes spec.storage.pvc).
// Shape: `<pod-uid>:<restart-sum>` per Ready pod, comma-joined and
// lex-sorted by pod name. restart-sum is the per-pod
// containerStatuses[].RestartCount summed across cache-server containers
// (the names from the owned workload's pod template; foreign sidecars are
// excluded). Inert and cleared for External backends, unsupported-runtime
// backends, and on the InvalidStorageConfiguration gate (a persistent
// multi-replica Deployment spec the controller refuses to provision until
// the operator scales to 1, removes spec.storage.pvc, or switches to
// StatefulSet).
//
// Operator-side recovery for the upstream LMCache
// LMServerConnector EPIPE-on-restart bug. See
Expand Down
92 changes: 47 additions & 45 deletions config/crd/bases/inferencecache.io_cachebackends.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,22 +68,24 @@ spec:
(spec.autoscaling) are both reconciled today. The autoscaling spec drives a
HorizontalPodAutoscaler. On a managed Deployment backend
(deploymentKind=Deployment, the default), spec.storage.pvc provisions a
PersistentVolumeClaim owner-referenced to the CacheBackend (reclaimed only
when the CacheBackend is deleted), mounts it into the cache-server pod at the
runtime adapter's declared data path, and reports the bound PVC's actual size
in status.capacity.
single PersistentVolumeClaim owner-referenced to the CacheBackend (reclaimed
only when the CacheBackend is deleted), mounts it into the cache-server pod at
the runtime adapter's declared data path, and reports the bound PVC's actual
size in status.capacity. On a managed StatefulSet backend, spec.storage.pvc
renders a volumeClaimTemplate instead, giving each replica its own
ReadWriteOnce PVC.

Two scoping rules apply:

- Only a managed Deployment backend provisions storage. A StatefulSet
backend is routed to the unmanaged path today (see DeploymentKind), so it
provisions nothing, storage included; per-replica volumeClaimTemplates are
a follow-up. An External backend provisions no workload at all, so
spec.storage.pvc is a no-op there.
- A persistent backend must be single-replica. A ReadWriteOnce PVC cannot be
multi-attached, so replicas (or an autoscaling ceiling) > 1 is surfaced
Ready=False/InvalidStorageConfiguration rather than provisioned;
per-replica persistent storage via StatefulSet is a separate follow-up.
- Managed Deployment backends provision one shared PVC. Managed StatefulSet
backends provision per-replica PVCs via volumeClaimTemplates. External
backends provision no workload at all, so spec.storage.pvc is a no-op
there.
- A persistent Deployment backend must be single-replica. A ReadWriteOnce
PVC cannot be multi-attached, so replicas (or an autoscaling ceiling) > 1
is surfaced Ready=False/InvalidStorageConfiguration rather than
provisioned. Use deploymentKind=StatefulSet for multi-replica persistent
storage.

NOTE: provisioning + mounting the PVC does not by itself make the cache server
spill KV to it — switching the LMCache server to a disk-backed storage device
Expand Down Expand Up @@ -152,9 +154,9 @@ spec:
default: Deployment
description: |-
DeploymentKind identifies whether a managed backend is reconciled as a
Deployment or StatefulSet. Defaults to Deployment — the only kind the
Phase-1 reconciler templates; StatefulSet is reserved for future
per-replica-PVC topologies and is a no-op today.
Deployment or StatefulSet. Defaults to Deployment. StatefulSet uses the
same adapter-rendered pod and service shape, but can back
spec.storage.pvc with per-replica volumeClaimTemplates.
enum:
- Deployment
- StatefulSet
Expand Down Expand Up @@ -2133,12 +2135,13 @@ spec:
capacity:
description: |-
Capacity is a human-readable summary of the backend's provisioned
capacity: the bound PersistentVolumeClaim's actual capacity when
spec.storage.pvc is set and the PVC has bound (the real provisioned size,
which may exceed the request), or empty for an ephemeral backend or while
the PVC is still pending (e.g. a WaitForFirstConsumer StorageClass that
binds only once the pod schedules). It is informational; clients must not
parse it.
capacity: for Deployment-backed persistent storage, the bound shared
PersistentVolumeClaim's actual capacity when the PVC has bound (the real
provisioned size, which may exceed the request). Empty for an ephemeral
backend, a StatefulSet backend (per-replica PVCs have no aggregate
capacity projection yet), or while the Deployment PVC is still pending
(e.g. a WaitForFirstConsumer StorageClass that binds only once the pod
schedules). It is informational; clients must not parse it.
type: string
conditions:
description: Conditions describe the latest observations of the backend.
Expand Down Expand Up @@ -2214,19 +2217,19 @@ spec:
firstAvailableAt:
description: |-
FirstAvailableAt latches the first time the managed cache-backend
workload was observed Available — the stable anchor for the
firstEventTimeout clock. It is deliberately a latched timestamp rather
than the live Deployment's Available condition LastTransitionTime: that
condition resets on an availability flap, which would restart the
workload was observed serving (Deployment Available or StatefulSet Ready)
— the stable anchor for the firstEventTimeout clock. It is deliberately
a latched timestamp rather than the live workload transition time: that
transition resets on an availability flap, which would restart the
timeout window and let a backend that already breached the timeout
(Degraded / NoKVEventsObserved) bounce back to AwaitingFirstKVEvent
without any KV event — contradicting the "once Degraded, stays Degraded
until an event arrives" contract. Anchoring on this write-once value
keeps the elapsed window monotonic, so Degraded is sticky. Written
write-once when the workload first reports Available and never cleared
write-once when the workload first reports serving and never cleared
(inert while the backend is not managed). A genuinely recreated managed
Deployment keeps the prior anchor; the gate re-evaluates from it, which
is safe because the engine event source is unchanged by a cache-server
workload keeps the prior anchor; the gate re-evaluates from it, which is
safe because the engine event source is unchanged by a cache-server
restart.
format: date-time
type: string
Expand Down Expand Up @@ -2314,25 +2317,24 @@ spec:
type: integer
observedServerInstance:
description: |-
ObservedServerInstance is the controller's cascade-decision
baseline — a stable identifier for the Ready cache-server pod
set the controller last anchored against. NOT a live current-
pod-set view: the controller intentionally pins this through
transient rolling-update midpoints and through no-Ready
windows so the cascade does not fire on rollbacks or transient
outages. For the live pod inventory, operators should consult
ObservedServerInstance is the controller's cascade-decision baseline — a
stable identifier for the Ready cache-server pod set the controller last
anchored against. NOT a live current-pod-set view: the controller
intentionally pins this through transient rolling-update midpoints and
through no-Ready windows so the cascade does not fire on rollbacks or
transient outages. For the live pod inventory, operators should consult
status.matchedEnginePods (engine side) and `kubectl get pod`
(cache-server side).

Shape: `<pod-uid>:<restart-sum>` per Ready pod, comma-joined
and lex-sorted by pod name. restart-sum is the per-pod
containerStatuses[].RestartCount summed across cache-server
containers (the names from the owned Deployment's pod
template; foreign sidecars are excluded). Inert and cleared
for External backends, unsupported-runtime backends, and on
the InvalidStorageConfiguration gate (a persistent
multi-replica spec the controller refuses to provision until
the operator scales to 1 or removes spec.storage.pvc).
Shape: `<pod-uid>:<restart-sum>` per Ready pod, comma-joined and
lex-sorted by pod name. restart-sum is the per-pod
containerStatuses[].RestartCount summed across cache-server containers
(the names from the owned workload's pod template; foreign sidecars are
excluded). Inert and cleared for External backends, unsupported-runtime
backends, and on the InvalidStorageConfiguration gate (a persistent
multi-replica Deployment spec the controller refuses to provision until
the operator scales to 1, removes spec.storage.pvc, or switches to
StatefulSet).

Operator-side recovery for the upstream LMCache
LMServerConnector EPIPE-on-restart bug. See
Expand Down
2 changes: 2 additions & 0 deletions config/rbac/role.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,7 @@ rules:
- apps
resources:
- deployments
- statefulsets
verbs:
- create
- delete
Expand All @@ -60,6 +61,7 @@ rules:
resources:
- deployments/status
- replicasets
- statefulsets/status
verbs:
- get
- apiGroups:
Expand Down
2 changes: 1 addition & 1 deletion config/samples/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ before applying. All but `recipe-gpu-production` run without a GPU.
| [`recipe-external-cache.yaml`](recipe-external-cache.yaml) | `type: External` — point the operator at a cache server you manage yourself; the controller provisions nothing. |
| [`recipe-multi-tenant.yaml`](recipe-multi-tenant.yaml) | Two CacheTenants + two CacheBackends across two namespaces — isolated cache identity and entry-count quotas; separate engines for per-tenant memory isolation. |
| [`recipe-tuning.yaml`](recipe-tuning.yaml) | CPU-dev shape plus a meaningful `engineOverrides` block (tune `LMCACHE_CHUNK_SIZE`, add `LMCACHE_LOG_LEVEL=DEBUG`). |
| [`recipe-persistent-cache.yaml`](recipe-persistent-cache.yaml) | `spec.storage.pvc` — provisions a PVC owner-referenced to the CacheBackend and mounts it into the cache-server pod (single replica; ReadWriteOnce). Adopt-and-keep on removal. (Server-side disk-backed KV is a separate follow-up; the volume is mounted but the server still keeps KV in memory.) |
| [`recipe-persistent-cache.yaml`](recipe-persistent-cache.yaml) | `spec.storage.pvc` on `deploymentKind: Deployment` — provisions one owner-referenced ReadWriteOnce PVC and mounts it into the cache-server pod. Use `deploymentKind: StatefulSet` for per-replica PVCs. (Server-side disk-backed KV is a separate follow-up; the volume is mounted but the server still keeps KV in memory.) |

**Prerequisites per recipe.** Most recipes are self-contained. One has an
external dependency: `recipe-external-cache.yaml` needs a cache server already
Expand Down
Loading
Loading