From 5f12dd32e848502ff8946ea93070849c84f455ce Mon Sep 17 00:00:00 2001 From: Ed Sun Date: Thu, 23 Jul 2026 13:29:07 -0700 Subject: [PATCH 1/5] docs(sglang): rewrite the reference stack to the MP data plane MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The reference manifest + README still described the pre-MP lm:// model with a 'KNOWN LIMITATION / caches nothing / do not use as a working offload' narrative — now false since the MP data plane shipped. Rewrite deployment.yaml to the MP topology (Redis L2 + node-local MP-worker native sidecar + --lmcache-config-file, no lm:// env) and the README to match; refresh the VERSIONS.md note that flagged the manifest as lagging. Derived from the GPU-validated adapter render (redis_l2.go + sglang_mp.go), structurally validated (kubectl dry-run); needs a GPU re-run before it's golden. --- docs/reference-stack/VERSIONS.md | 18 +- .../manifests/sglang-lmcache/README.md | 393 +++++++++--------- .../manifests/sglang-lmcache/deployment.yaml | 268 +++++++----- 3 files changed, 364 insertions(+), 315 deletions(-) diff --git a/docs/reference-stack/VERSIONS.md b/docs/reference-stack/VERSIONS.md index 94dd40e9..981194d3 100644 --- a/docs/reference-stack/VERSIONS.md +++ b/docs/reference-stack/VERSIONS.md @@ -44,16 +44,14 @@ The SGLang reference pins **two** images: (a) the **derived engine image** (whic the MP worker also runs by default), and (b) the **Redis L2** store. It does **not** pin an lmcache-server — SGLang never dials one. -> **Note — the reference *manifest* is not yet updated to this topology.** -> `manifests/sglang-lmcache/deployment.yaml` still ships the earlier -> lmcache-server-based placeholder (a non-applyable digest); rewriting it to the MP -> topology — derived engine image + a Redis L2 + the MP-worker sidecar — is part of -> the increment-3 reference-stack leg (see the phased delivery in -> [`sglang-lmcache-mp-mode.md`](../design/sglang-lmcache-mp-mode.md)). The **managed -> path the controller renders** already matches this section (that is what -> `redis_l2.go` + the SGLang adapter produce, GPU-validated); the pins below are -> authoritative for that path. Until the manifest leg lands, this section is ahead of -> the hand-written reference YAML — follow the pins here, not that manifest. +> **Note — the reference manifest now matches this topology.** +> `manifests/sglang-lmcache/deployment.yaml` renders the MP topology: a **Redis L2** + +> the **MP-worker native sidecar** + the derived engine image (still a non-applyable +> placeholder digest — substitute your own build). It is **derived from the +> GPU-validated adapter render** (`redis_l2.go` + the SGLang adapter) and structurally +> checked (`kubectl apply --dry-run`); re-run it on a GPU before treating it as golden. +> The pins below are authoritative for **both** the manifest and the controller-rendered +> managed path. (a) The **derived** SGLang engine image with the lmcache client baked in (the base `lmsysorg/sglang` does not bundle it). This image is used **twice**: as the diff --git a/docs/reference-stack/manifests/sglang-lmcache/README.md b/docs/reference-stack/manifests/sglang-lmcache/README.md index f11d9f7b..e8c9019f 100644 --- a/docs/reference-stack/manifests/sglang-lmcache/README.md +++ b/docs/reference-stack/manifests/sglang-lmcache/README.md @@ -1,101 +1,98 @@ -# SGLang + LMCache reference stack +# SGLang + LMCache (MP mode) reference stack The second-engine sibling of the top-level [vLLM + LMCache reference](../../README.md): -a **SGLang** deployment publishing **KV-cache events over ZMQ** (the intended -reference leg — usable on its own once `--enable-lmcache` is dropped; see the -status caveat below). Its **LMCache offload** leg is a **non-working historical -template** — GPU validation (2026-07) found it wired to the wrong LMCache mode -(see the KNOWN LIMITATION below); treat it as a record of the shipped topology, -not a serving cache. It is the hand-built -reference the `(sglang, LMCache)` runtime adapter -(`pkg/adapters/runtime/sglang`) was templated from: in [`deployment.yaml`](deployment.yaml) -the SGLang container's **LMCache wiring** (`--enable-lmcache` + the `LMCACHE_*` / -`INFERENCECACHE_FAIL_OPEN` env) mirrors what the adapter auto-injects. The rest -of the manifest — image, resources, the lmcache-server, `--kv-events-config` — -is operator-owned scaffolding the adapter assumes is already present, so the -file as a whole is **not** byte-for-byte adapter output. - -> **KNOWN LIMITATION — the `lm://` LMCache offload wiring here is wrong for SGLang (GPU-validated 2026-07).** The `LMCACHE_REMOTE_URL` / standalone `lmcache-server` wiring below mirrors vLLM+LMCache, but SGLang drives LMCache in **multiprocess (MP) mode**: config from the **`--lmcache-config-file` flag** (the `LMCACHE_*` env is ignored) and a **node-local** worker addressed by `mp_host`/`mp_port` over shared memory — not a cluster-reachable `lm://` server. Applied as written (env only, **no** `--lmcache-config-file`) the SGLang engine **refuses to start** — `MP mode requires --lmcache-config-file`. (A *separate* case: supplying a `--lmcache-config-file` that carries `remote_url: lm://…` instead **hangs** the engine — that config is not what this manifest ships.) Either way it does not offload. Use this manifest to understand the KV-event path and the injection surface, **not** as a working LMCache offload. The correct MP-mode wiring (`--lmcache-config-file` + a per-node worker) is a pending fix; see `docs/design/cachebackend-api.md` (SGLang engine support). The "Status" note below predates this validation. - -> **Status.** The **LMCache-offload leg** HAS since been GPU-validated (2026-07) and -> found **broken** — see the KNOWN LIMITATION above. The **KV-event leg** below -> remains an **unvalidated template**: no GPU run has exercised it end-to-end. -> Treat that leg as a **template**, not a known-good run. Applied, it is -> *designed to* stand up the real SGLang → ZMQ **KV-event** path, and the -> standalone flow's intended check is only that the publisher **starts** (from -> engine logs) — it does **not** consume or assert `BlockStored` frames (there is -> no standalone consumer). The one thing that **is** actually executed (off-GPU) -> is SGLang's exact wire *shape*, via the Go test -> (`go test ./pkg/adapters/engine/ -run SGLang`). The **LMCache offload** leg -> (`LMCACHE_REMOTE_URL`) was GPU-validated and found **broken** — SGLang uses -> LMCache MP mode, not this `lm://` env wiring (see -> [Wire-test caveat — resolved](#wire-test-caveat-resolved)). +a **SGLang** deployment that publishes **KV-cache events over ZMQ** and **offloads +KV to a shared Redis L2** via LMCache **multiprocess (MP) mode**. It is the hand-built +reference the `(sglang, LMCache)` runtime adapter (`pkg/adapters/runtime/sglang`) +mirrors: [`deployment.yaml`](deployment.yaml) stands up the same shape the adapter +auto-injects — a Redis L2 store, the engine with `--enable-lmcache` + +`--lmcache-config-file`, and a **node-local MP-worker native sidecar** that offloads +to that Redis. The engine image / `--model-path` / resources / `--kv-events-config` +are operator-owned scaffolding the adapter assumes is present, so the file as a whole +is **not** byte-for-byte adapter output. + +> **Validation status.** This manifest is **derived from the GPU-validated adapter +> render** (`sglang_mp.go` + `redis_l2.go`; the controller-rendered managed path was +> validated store→flush→retrieve end-to-end in the MP-mode increment) and is +> **structurally checked** (`kubectl apply --dry-run`). It has **not** been +> independently re-run end-to-end on a GPU in this exact hand shape — run it on a GPU +> host (below) before treating it as a golden reference. All pins live in +> [`../../VERSIONS.md`](../../VERSIONS.md). + +> **Kubernetes ≥ 1.29 required.** The MP worker is a **native sidecar** (an +> `initContainers` entry with `restartPolicy: Always`), which older apiservers do not +> understand. ## Why this exists (and what's already validated) SGLang adopted vLLM's KV-event wire wholesale: `--kv-events-config` drives a ZMQ -`ZmqEventPublisher` emitting the **same** msgspec `BlockStored` / `BlockRemoved` -/ `AllBlocksCleared` event structs vLLM does (the batch envelope adds a trailing +`ZmqEventPublisher` emitting the **same** msgspec `BlockStored` / `BlockRemoved` / +`AllBlocksCleared` event structs vLLM does (the batch envelope adds a trailing `attn_dp_rank` the decoder ignores). Two consequences: 1. **The event-decode path is engine-agnostic and already covered by tests.** The - shipped `kvevent-subscriber` decodes SGLang's stream unchanged; the only - difference is the `--hash-scheme=sglang` tag. The Go decoder is exercised - against a synthetic SGLang-shaped frame in - `pkg/adapters/engine/sglang_wire_test.go`, and the cross-engine isolation - (`hash_scheme` keeps SGLang and vLLM prefixes disjoint) in + shipped `kvevent-subscriber` decodes SGLang's stream unchanged; the only difference + is the `--hash-scheme=sglang` tag. The Go decoder is exercised against a synthetic + SGLang-shaped frame in `pkg/adapters/engine/sglang_wire_test.go`, and the + cross-engine isolation (`hash_scheme` keeps SGLang and vLLM prefixes disjoint) in `pkg/index` (`TestNoCrossEngineFalseHitVLLMvsSGLang`). -2. **You can validate off-GPU** (below): the Go test covers SGLang's exact wire - shape; the Python synthetic tooling covers the shared decode/redaction logic. - -What this reference stack is *intended* to add on top of those tests is the -**real engine → ZMQ events** path on a GPU — a live SGLang pod publishing real -`BlockStored`/`BlockRemoved` frames a subscriber can read. **As shipped this is -blocked:** with `--enable-lmcache` but no `--lmcache-config-file` the engine -**refuses to start** (`MP mode requires --lmcache-config-file`) and never reaches -serving/event publishing — drop `--enable-lmcache` to exercise the event path today. Extending -that all the way to **index + `LookupRoute`** additionally requires the cache plane +2. **You can validate the wire off-GPU** (below): the Go test covers SGLang's exact + wire shape; the Python synthetic tooling covers the shared decode/redaction logic. + +What this reference adds on top of those tests is the **real engine → ZMQ events + +LMCache offload** path on a GPU: a live SGLang pod publishing real +`BlockStored`/`BlockRemoved` frames while the MP worker offloads evicted KV to Redis. +Extending that to **index + `LookupRoute`** additionally requires the cache plane installed and a `CacheBackend` (`engine: sglang`, `type: LMCache`) whose -`engineSelector` matches these pods and whose `backendConfig.model` is set, so -the controller auto-attaches the `kvevent-subscriber` sidecar (see -`config/samples/cachebackend-sglang.yaml` and the install docs). The raw -`kubectl apply` below is *intended* to stand up the engine + lmcache-server, so -that the verifiable outcome would be the engine serving traffic with its -**KV-event publisher started** (asserted from the engine logs below). **As shipped -with `--enable-lmcache` but no `--lmcache-config-file` the engine refuses to start -(`MP mode requires --lmcache-config-file`, KNOWN LIMITATION), so none of that is -reached** — drop `--enable-lmcache` to observe the engine + publisher. Even -when it runs, this flow shows neither the consumed event stream (the standalone -manifest ships no consumer) nor the -populated index; the index/`LookupRoute` criterion below assumes the controller -+ that `CacheBackend` are also present. +`engineSelector` matches these pods and whose `backendConfig.model` is set, so the +controller auto-attaches the `kvevent-subscriber` sidecar (see +`config/samples/cachebackend-sglang.yaml` and the install docs). The standalone +manifest ships no event consumer, so on its own it shows the engine serving + the +publisher started + KV offloading to Redis, **not** a populated index. + +## How the MP data plane fits together + +``` + ┌─────────────────────── engine pod ───────────────────────┐ + │ [initContainers] │ + │ lmcache-mp-worker (native sidecar, restartPolicy:Always)│ + │ writes /etc/lmcache/config.yaml (mp_host/mp_port) │ + │ runs the LMCache MP server on 127.0.0.1:5555 │──resp──▶ redis-l2 + │ ▲ CUDA-IPC + /dev/shm (L1) │ (ClusterIP, + │ [containers] │ shared L2) + │ sglang --enable-lmcache --lmcache-config-file │ + │ --kv-events-config (ZMQ :5557) ────────────────────────┼──▶ kvevent-subscriber + └────────────────────────────────────────────────────────────┘ (managed path) +``` + +The engine dials the **local** worker over `mp_host`/`mp_port` (never the Redis +endpoint directly); the worker holds the L1 in `/dev/shm` and offloads its shared +tier to Redis over the `resp` `--l2-adapter`. `lm://` is **not** a valid MP +`--l2-adapter` type, which is why the shared tier is Redis, not the standalone +`lmcache-server` the vLLM path uses. ## Engine-side differences from the vLLM reference | | vLLM | SGLang | |---|---|---| | LMCache on | `--kv-transfer-config '{"kv_connector":"LMCacheConnectorV1",…}'` | `--enable-lmcache` (bare flag) + `LMCACHE_USE_EXPERIMENTAL=True` | +| LMCache config source | `LMCACHE_*` env (`LMCACHE_REMOTE_URL`, …) | **`--lmcache-config-file`** (written by the MP worker); the `LMCACHE_*` env is ignored | +| Shared tier | standalone `lm://` `lmcache-server` | **Redis L2** behind a node-local **MP worker** (`resp` `--l2-adapter`) | | vLLM-only env | `VLLM_USE_V1=1`, `PYTHONHASHSEED=0` | *(neither — no v1 codepath; SGLang sha256-hashes, independent of `PYTHONHASHSEED`)* | | Default HTTP port | 8000 | 30000 | | KV-event wire | ZMQ `BlockStored`/`BlockRemoved`/`AllBlocksCleared` | same event structs; batch envelope adds a trailing `attn_dp_rank` the decoder ignores | -| LMCACHE_* tunables | read from env (`LMCACHE_REMOTE_URL` etc.) | **ignored** — SGLang reads config only from `--lmcache-config-file` (KNOWN LIMITATION) | ## Deploy and test on a GPU > Needs an NVIDIA GPU host (or a managed GPU cluster advertising `nvidia.com/gpu`) -> and a Hugging Face token for the gated reference model. Size the GPU the same -> way as the vLLM path — see [`../../GPU-RUNBOOK.md`](../../GPU-RUNBOOK.md); the -> 8B reference model fits on a single 24 GB card. - -This manifest is a **standalone** reference showing the hand-built shape the -adapter was templated from, with **no controller or `CacheBackend` in the loop** -(the `LMCACHE_REMOTE_URL` env points at the bundled in-namespace `lmcache-server`). -**As shipped it does NOT come up:** with `--enable-lmcache` but no -`--lmcache-config-file` the SGLang engine refuses to start -(`MP mode requires --lmcache-config-file`, KNOWN LIMITATION), so it neither serves -nor emits ZMQ events. The commands below are the intended shape, -not a runnable flow — drop `--enable-lmcache` to bring up the engine + event path, -and await the MP-mode fix for a working LMCache offload. +> and a Hugging Face token for the gated reference model. Size the GPU the same way +> as the vLLM path — see [`../../GPU-RUNBOOK.md`](../../GPU-RUNBOOK.md); the 8B +> reference model fits on a single 24 GB card. + +This manifest is a **standalone** reference showing the hand-built shape the adapter +mirrors, with **no controller or `CacheBackend` in the loop** — the engine reads its +MP config from the worker's `--lmcache-config-file`, and the worker offloads to the +in-namespace `redis-l2` Service. Run the commands below from this directory (`docs/reference-stack/manifests/sglang-lmcache/`) — the relative paths @@ -107,47 +104,36 @@ kind create cluster --name inference-cache-substrate --config ../../kind/cluster helm repo add nvdp https://nvidia.github.io/k8s-device-plugin helm install nvdp nvdp/nvidia-device-plugin -n kube-system -# 2. Fix BOTH placeholder images in deployment.yaml: replace the ENTIRE SGLang -# engine image reference `example.invalid/sglang-lmcache@sha256:0000...` with -# your real derived image (repo AND digest), and swap the lmcache-server's -# all-zero digest for its real one — see the deployment.yaml header + -# VERSIONS.md. Then create the namespace + HF token secret (idempotent so the -# runbook re-runs cleanly). +# 2. Fix the SGLang image in deployment.yaml: replace the ENTIRE placeholder +# reference `example.invalid/sglang-lmcache@sha256:0000...` (in BOTH the +# lmcache-mp-worker init container and the sglang engine container) with your +# real DERIVED image (repo AND digest) — the base sglang image does not bundle +# the lmcache client, so you must build one (`pip install lmcache==0.5.1` onto the +# GPU-validated cu13 base). See the deployment.yaml header + VERSIONS.md. Redis is +# a normal pullable tag (digest-pin for production). Then create the namespace + +# HF token secret (idempotent so the runbook re-runs cleanly). kubectl create namespace cache-substrate --dry-run=client -o yaml | kubectl apply -f - kubectl -n cache-substrate create secret generic hf-token --from-literal=token="$HF_TOKEN" \ --dry-run=client -o yaml | kubectl apply -f - -# 3. Deploy the lmcache-server + SGLang engine (the manifest wires them together). -# -# !! WILL NOT COMPLETE AS-IS (GPU-validated 2026-07) !! As shipped (--enable-lmcache -# + env, no --lmcache-config-file) the SGLang engine REFUSES TO START -# (`MP mode requires --lmcache-config-file`; see "Wire-test caveat (resolved)"), -# so the sglang rollout below never goes green and steps 4+ (serving) cannot -# pass. These commands are kept as the intended SHAPE; they become runnable only -# after the MP-mode adapter fix (--lmcache-config-file + a per-node worker). Do -# NOT run this expecting a working LMCache offload today. -# -# (Once fixed: wait for BOTH rollouts — fail-open means the engine can serve even -# if lmcache-server is unready, so a green engine rollout alone would hide a -# broken cache server, i.e. offload silently never happening.) +# 3. Deploy the Redis L2 + the SGLang engine (with its MP-worker sidecar). +# Wait for BOTH rollouts. The engine rollout only goes green once the MP worker's +# startupProbe passes (the ZMQ server is listening on 127.0.0.1:5555) — SGLang has +# no cacheless fallback while --enable-lmcache is on, so the worker is a serving +# prerequisite, not a soft dependency. kubectl apply -f deployment.yaml -kubectl -n cache-substrate rollout status deploy/lmcache-server --timeout=5m +kubectl -n cache-substrate rollout status deploy/redis-l2 --timeout=5m kubectl -n cache-substrate rollout status deploy/sglang-lmcache-llama-8b --timeout=20m # 4. Drive the SAME long-prefix prompt twice (first warms, second reuses). This -# exercises the engine and triggers its KV-event publisher on :5557; -# CONSUMING/verifying the frames is the managed path below — there is no -# standalone printer for SGLang's 3-tuple stream in this repo. Swap the -# `prefix = ...` line below for a REAL prompt long enough to span several KV -# blocks (>> the engine's --page-size in tokens); a short literal won't -# reliably produce BlockStored / prefix reuse. -# Build the request body with python3 (no jq dependency); the long shared -# prefix is what triggers block storage + reuse. +# exercises the engine, triggers its KV-event publisher on :5557, and — on HBM +# eviction — offloads KV to Redis via the MP worker. Swap the `prefix = ...` line +# for a REAL prompt long enough to span several KV blocks (>> the engine's +# --page-size in tokens); a short literal won't reliably produce BlockStored / +# prefix reuse. BODY="$(python3 - <<'PY' import json prefix = "You are a helpful assistant. " * 200 # ~1k+ tokens of shared prefix -# Tiny, greedy generation — the point is prefix reuse (block storage), not the -# output; a short deterministic decode keeps the smoke fast and reproducible. print(json.dumps({"model": "meta-llama/Meta-Llama-3-8B-Instruct", "messages": [{"role": "user", "content": prefix}], "max_tokens": 16, "temperature": 0})) @@ -155,8 +141,6 @@ PY )" kubectl -n cache-substrate port-forward svc/sglang-lmcache-llama-8b 30000:30000 & pf=$!; trap 'kill "$pf" 2>/dev/null' EXIT # clean up the port-forward (frees :30000) -# Wait for the port-forward to serve /health, but don't spin forever if it dies -# (e.g. the pod never became ready) — bound the wait and bail if $pf has exited. ready="" for _ in $(seq 60); do kill -0 "$pf" 2>/dev/null || { echo "FAIL: port-forward exited early (is the pod running?)"; exit 1; } @@ -167,8 +151,6 @@ done rc=0 for i in 1 2; do # -sfS: fail (non-zero) on HTTP 4xx/5xx so a rejected request doesn't look served. - # Capture that failure — otherwise the trailing (exit-0) kill masks a rejected - # request and the whole block still "passes". curl -sfS localhost:30000/v1/chat/completions -H 'content-type: application/json' -d "$BODY" >/dev/null \ || { echo "FAIL: request $i was not served (HTTP error)"; rc=1; break; } done @@ -176,103 +158,100 @@ kill "$pf" 2>/dev/null; trap - EXIT [ "$rc" -eq 0 ] || exit 1 ``` -### What success looks like (standalone) — publisher-start smoke only +### What success looks like (standalone) -- **The verifiable standalone outcome is: the engine serves both requests AND - its KV-event publisher started.** Assert the publisher concretely from the - engine logs (so a mis-configured `--kv-events-config` fails the check rather - than silently passing): +- **The engine serves both requests AND its KV-event publisher started.** Assert the + publisher concretely from the engine logs (so a mis-configured `--kv-events-config` + fails the check rather than silently passing): ```bash # Match a publisher-STARTUP line, not a config/arg echo: SGLang echoes # "--kv-events-config"/"kv-events" at boot, so grepping for "kv-events" would - # false-pass on the echo alone. If your SGLang build's startup log phrases the - # publisher differently, adjust the pattern below (confirm it against a real - # run's logs the first time). + # false-pass on the echo alone. Adjust the pattern to your SGLang build's phrasing + # (confirm against a real run's logs the first time). kubectl -n cache-substrate logs deploy/sglang-lmcache-llama-8b -c sglang \ | grep -iE 'zmq.*publish|publisher thread|start(ing|ed).*publisher' \ || { echo "FAIL: KV-event publisher did not start (check --kv-events-config)"; exit 1; } ``` - Full stream verification stops there standalone: this flow does **not** consume - or assert the `BlockStored` frames themselves, **nor does it prove LMCache - offload works** — and in fact SGLang does **not** store KV into `lmcache-server` - over `LMCACHE_REMOTE_URL` at all: that offload wiring is broken (SGLang uses - LMCache MP mode; see [Wire-test caveat — resolved](#wire-test-caveat-resolved)), - so do not expect stores to arrive there. Observing the actual event stream needs a - consumer (the managed Go sidecar below; this repo ships no standalone 3-tuple - printer), and the decoder's correctness against SGLang's exact wire is covered - offline by the Go test. End-to-end frame→index→`LookupRoute` - verification requires the **managed controller/sidecar path** (see below), not - this standalone deployment — even on a GPU, the standalone manifest has no - consumer, so it never populates the index. This is an inherent limit of the - consumer-less standalone manifest, not a gap to paper over. +- **KV offloaded to Redis.** After driving traffic that spans an HBM eviction, the L2 + keyspace is non-empty — a quick check via the MP worker container: + + ```bash + # The MP worker offloads to redis-l2 over `resp`. A non-zero dbsize after eviction + # is the standalone signal that offload is actually happening. + kubectl -n cache-substrate exec deploy/redis-l2 -c redis-l2 -- redis-cli dbsize + ``` + + (Whether an eviction occurs depends on model size vs. HBM; a larger working set or a + smaller `--mem-fraction-static` forces it. On a small model with plenty of HBM the + L1 may hold everything and Redis stays empty — that is not a failure, just no + eviction. Full frame→index→`LookupRoute` verification needs the **managed path** + below; the standalone manifest ships no event consumer, so it never populates the + index.) + - **Privacy boundary — the RAW ZMQ frames carry `token_ids`.** Both vLLM's and - SGLang's `BlockStored` wire includes the block's token ids; the - "metadata-only, never token content" guarantee is about what the IC - `kvevent-subscriber` *reports to the policy server* — it hashes the token_ids - in-pod into the content fingerprint and forwards only hashes + counts (over - `127.0.0.1`). That report-level guarantee holds regardless of how the publisher - binds. The **raw ZMQ frames** are a separate matter: the manifest binds - `tcp://*:5557` (the operator contract in + SGLang's `BlockStored` wire includes the block's token ids; the "metadata-only, + never token content" guarantee is about what the IC `kvevent-subscriber` *reports to + the policy server* — it hashes the token_ids in-pod into the content fingerprint and + forwards only hashes + counts (over `127.0.0.1`). That report-level guarantee holds + regardless of how the publisher binds. The **raw ZMQ frames** are a separate matter: + the manifest binds `tcp://*:5557` (the operator contract in [`docs/design/cachebackend-api.md`](../../../design/cachebackend-api.md), kept - identical to the vLLM reference), so the raw, token-bearing frames **are** - reachable by any in-cluster pod that knows this pod's IP. Two things bound that - exposure: the Service deliberately exposes **only** the HTTP API (never - `:5557`), and — if in-cluster raw-frame access is a concern — you can add a - **NetworkPolicy** restricting `:5557` to this pod (the in-pod subscriber reaches - the publisher over `127.0.0.1`, so it is unaffected). If you must inspect the - raw stream during dev, `kubectl port-forward` `:5557` yourself rather than - adding it to the Service. + identical to the vLLM reference), so the raw, token-bearing frames **are** reachable + by any in-cluster pod that knows this pod's IP. Two things bound that exposure: the + Service deliberately exposes **only** the HTTP API (never `:5557`), and — if + in-cluster raw-frame access is a concern — you can add a **NetworkPolicy** restricting + `:5557` to this pod (the in-pod subscriber reaches the publisher over `127.0.0.1`, so + it is unaffected). If you must inspect the raw stream during dev, `kubectl + port-forward` `:5557` yourself rather than adding it to the Service. + - **Why the two `scripts/` helpers don't verify SGLang here:** - `scripts/kv_events_subscriber.py` decodes only vLLM's 2-tuple synthetic frames - (it would print `UNDECODED` on SGLang's 3-tuple), and - `scripts/prefix_cache_hit_test.sh` reads vLLM's `vllm:prefix_cache_hits` - counter SGLang doesn't emit (use it only as a request driver). The shipped - live SGLang consumer is the managed Go sidecar below. + `scripts/kv_events_subscriber.py` decodes only vLLM's 2-tuple synthetic frames (it + would print `UNDECODED` on SGLang's 3-tuple), and `scripts/prefix_cache_hit_test.sh` + reads vLLM's `vllm:prefix_cache_hits` counter SGLang doesn't emit (use it only as a + request driver). The shipped live SGLang consumer is the managed Go sidecar below. ### The managed path (what the adapter automates) -In a real install you do **not** hand-write this manifest. You create a -`CacheBackend` (`engine: sglang`, `type: LMCache`; see +In a real install you do **not** hand-write this manifest. You create a `CacheBackend` +(`engine: sglang`, `type: LMCache`; see [`config/samples/cachebackend-sglang.yaml`](../../../../config/samples/cachebackend-sglang.yaml)) -whose `engineSelector` matches your SGLang pods, and the controller renders its -own lmcache-server, injects the engine's LMCache env (overwriting the manual -`LMCACHE_REMOTE_URL` with the managed endpoint), and — with -`--kvevent-subscriber-image` set — auto-attaches the Go `kvevent-subscriber` -sidecar (tagged `--hash-scheme=sglang`) that reports to the index, enabling -`LookupRoute` to return SGLang replicas (never a vLLM replica on the same prefix -bytes — the `hash_scheme` tag keeps them disjoint). Two caveats the manual -manifest sidesteps but the managed path must honor: the CacheBackend must live -in the **engine pods' namespace** (the Pod webhook matches per-namespace), and -because injection is **create-time only**, the CacheBackend's -`status.endpoint` must be **published** before the engine pod is created, or the -pod admits unwired and must be recreated. (The precondition is specifically -`status.endpoint`, **not** `Ready` — managed `Ready` is gated on the first KV -event observed *from these very pods*, so waiting for `Ready` first would be -circular.) The served model, the CacheBackend's -`backendConfig.model`, and the request's `model` must all agree, or the index -keys per-model and `LookupRoute` silently misses. **Block-size caveat:** for -raw-`token_ids`/`prompt_text` lookups the server fingerprints at its single -global `--engine-block-size` (default 16, vLLM's); SGLang's page size (e.g. 64) -must match it, or gateways must send pre-computed `prefix_hash`/`block_hashes` — -otherwise `LookupRoute` silently misses even with events flowing (see the -"Block-size alignment" note in [`docs/design/cachebackend-api.md`](../../../design/cachebackend-api.md)). -This managed wiring is -exercised by the **controller/webhook envtests** — the SGLang pod-injection, -reserved-override, and admission tests — not reproduced step-by-step here. (The -install-smoke gate's all-samples backstop additionally admits the -`config/samples/cachebackend-sglang.yaml` shape against a real-cluster webhook, -but it does not drive the full inject→index→`LookupRoute` flow.) +whose `engineSelector` matches your SGLang pods, and the controller renders the **Redis +L2** store, injects the **MP-worker sidecar + `--enable-lmcache` + `--lmcache-config-file`** +onto the engine pod, and — with `--kvevent-subscriber-image` set — auto-attaches the Go +`kvevent-subscriber` sidecar (tagged `--hash-scheme=sglang`) that reports to the index, +enabling `LookupRoute` to return SGLang replicas (never a vLLM replica on the same +prefix bytes — the `hash_scheme` tag keeps them disjoint). Two caveats the manual +manifest sidesteps but the managed path must honor: the CacheBackend must live in the +**engine pods' namespace** (the Pod webhook matches per-namespace), and because +injection is **create-time only**, the CacheBackend's `status.endpoint` (the Redis L2 +address) must be **published** before the engine pod is created, or the pod admits +unwired and must be recreated. (The precondition is specifically `status.endpoint`, +**not** `Ready` — managed `Ready` is gated on the first KV event observed *from these +very pods*, so waiting for `Ready` first would be circular.) The served model, the +CacheBackend's `backendConfig.model`, and the request's `model` must all agree, or the +index keys per-model and `LookupRoute` silently misses. **Block-size caveat:** for +raw-`token_ids`/`prompt_text` lookups the server fingerprints at its single global +`--engine-block-size` (default 16, vLLM's); SGLang's page size (e.g. 64) must match it, +or gateways must send pre-computed `prefix_hash`/`block_hashes` — otherwise +`LookupRoute` silently misses even with events flowing (see the "Block-size alignment" +note in [`docs/design/cachebackend-api.md`](../../../design/cachebackend-api.md)). This +managed wiring is exercised by the **controller/webhook envtests** — the SGLang +pod-injection, reserved-override, and admission tests — not reproduced step-by-step +here. (The install-smoke gate additionally admits the +`config/samples/cachebackend-sglang.yaml` shape against a real-cluster webhook via its +all-samples backstop, though it does not yet drive the SGLang reconcile or the full +inject→index→`LookupRoute` flow.) ## Validate the event wire WITHOUT a GPU Two complementary off-GPU checks, with an important scope distinction: 1. **The shared decode + token-redaction path** (Python, no image/cluster). The - `kv_events_synthetic_publisher.py` emits the **2-field** `[ts, events]` - EventBatch envelope (vLLM's shape), so it exercises the decode + - token-redaction logic common to both engines — not SGLang's exact envelope: + `kv_events_synthetic_publisher.py` emits the **2-field** `[ts, events]` EventBatch + envelope (vLLM's shape), so it exercises the decode + token-redaction logic common + to both engines — not SGLang's exact envelope: ```bash pip install -r ../../scripts/requirements.txt @@ -283,33 +262,31 @@ Two complementary off-GPU checks, with an important scope distinction: kill "$pub" 2>/dev/null; trap - EXIT ``` -2. **SGLang's exact wire shape** (Go, run from the repo root). SGLang's real - envelope is the **3-tuple** `[ts, events, attn_dp_rank]` with a 6-field - `BlockStored`; that shape — and that the subscriber's decoder tolerates the - trailing `attn_dp_rank` and tags reports `hash_scheme=sglang` — is asserted - by the Go fixture the shipped subscriber actually uses (from the repo root): +2. **SGLang's exact wire shape** (Go, run from the repo root). SGLang's real envelope + is the **3-tuple** `[ts, events, attn_dp_rank]` with a 6-field `BlockStored`; that + shape — and that the subscriber's decoder tolerates the trailing `attn_dp_rank` and + tags reports `hash_scheme=sglang` — is asserted by the Go fixture the shipped + subscriber actually uses (from the repo root): `go test ./pkg/adapters/engine/ -run SGLang` - (`TestDecodeSGLangEventBatch` + `TestReporterTagsSGLangScheme`). The Python - synthetic path above does **not** cover the 3-tuple; rely on the Go test for - the SGLang-specific envelope. - -## Wire-test caveat (resolved) - -GPU validation (2026-07) answered this: the `LMCACHE_REMOTE_URL` / remote -`lmcache-server` topology this manifest wires is **wrong for SGLang, and must not -be applied as a working config.** SGLang ignores the `LMCACHE_*` env, reads config -only from the `--lmcache-config-file` flag, and drives LMCache in **MP -(multiprocess) mode** — a node-local worker reached over `mp_host`/`mp_port` -(default `5555`) + shared memory, not a cluster-reachable `lm://` server. A config -with `remote_url: lm://…` does **not** offload — it **hangs the engine at -startup**; with no config file the engine refuses to start -(`MP mode requires --lmcache-config-file`). - -**Do not** mount a `remote_url: lm://…` config expecting it to work. The correct -wiring — a `--lmcache-config-file` carrying `mp_host`/`mp_port` plus a **per-node -LMCache MP worker** (which may itself carry a `remote_url` for a shared tier) — is -the tracked adapter follow-up. See `docs/design/cachebackend-api.md` (SGLang -engine support). + (`TestDecodeSGLangEventBatch` + `TestReporterTagsSGLangScheme`). The Python synthetic + path above does **not** cover the 3-tuple; rely on the Go test for the + SGLang-specific envelope. + +## LMCache MP mode (how the offload works) + +SGLang drives LMCache in **MP (multiprocess) mode**, not the `lm://` remote-server +model the vLLM path uses. It ignores the `LMCACHE_*` env and reads config only from the +`--lmcache-config-file` flag, which points at the file the **MP worker** writes +(`mp_host: 127.0.0.1`, `mp_port: 5555`). The worker runs the LMCache MP server on +loopback, holds the L1 in `/dev/shm`, and offloads its shared tier to the Redis L2 over +the `resp` `--l2-adapter` (`lm://` is not a valid MP `--l2-adapter` type). The engine +attaches to the local worker over CUDA-IPC + shared memory; because they speak the MP +wire to each other, the worker defaults to the **same image** as the engine (same +lmcache version) — `backendConfig.workerImage` overrides it, at which point keeping the +two lmcache versions aligned is yours. Authoritative design + GPU-validation evidence: +[`docs/design/sglang-lmcache-mp-mode.md`](../../../design/sglang-lmcache-mp-mode.md); +key/field reference: [`docs/design/cachebackend-api.md`](../../../design/cachebackend-api.md) +(SGLang engine support). ## Teardown diff --git a/docs/reference-stack/manifests/sglang-lmcache/deployment.yaml b/docs/reference-stack/manifests/sglang-lmcache/deployment.yaml index afb6bbbd..fffc7b25 100644 --- a/docs/reference-stack/manifests/sglang-lmcache/deployment.yaml +++ b/docs/reference-stack/manifests/sglang-lmcache/deployment.yaml @@ -1,89 +1,100 @@ -# SGLang + LMCache reference deployment — the second-engine sibling of the -# top-level vLLM + LMCache reference (manifests/deployment.yaml). +# SGLang + LMCache (MP mode) reference deployment — the second-engine sibling of +# the top-level vLLM + LMCache reference (manifests/deployment.yaml). # -# This is the hand-built reference the (sglang, LMCache) runtime adapter was -# templated from: a standalone lmcache-server + an SGLang engine wired to it, -# publishing KV-cache events over ZMQ. The SGLang engine container's LMCache -# wiring (the --enable-lmcache arg + the LMCACHE_* / INFERENCECACHE_FAIL_OPEN -# env) mirrors what the adapter's auto-injection produces, so an operator can -# read one manifest to understand the engine-side contract the controller -# applies on their behalf. The rest — the image, resources, the lmcache-server, -# and --kv-events-config — is operator-owned scaffolding the adapter assumes is -# already present (it wires the cache offload, not the event publisher or the -# pod shape), so the full manifest is NOT byte-for-byte adapter output. +# SGLang drives LMCache in MULTIPROCESS (MP) mode, so this stack differs from the +# vLLM reference on BOTH halves of the data plane: +# - the shared tier is a Redis L2 store (below), NOT a standalone lm:// server +# (lm:// is not a valid MP --l2-adapter type), and +# - the engine attaches to a node-local MP-worker sidecar over 127.0.0.1 +# (mp_host/mp_port) and reads its config from --lmcache-config-file; the old +# lm:// LMCACHE_REMOTE_URL env is gone (MP mode ignores it). +# This mirrors what the (sglang, LMCache) runtime adapter auto-injects on a matched +# engine pod (pkg/adapters/runtime/sglang + .../internal/enginewire/sglang_mp.go + +# redis_l2.go), which is GPU-validated end to end. The engine image / --model-path / +# resources / --kv-events-config remain operator-owned scaffolding the adapter +# assumes is already present, so this is NOT byte-for-byte adapter output. # -# KNOWN LIMITATION (GPU-validated 2026-07, fix pending): the lm:// LMCache -# offload wiring here (LMCACHE_REMOTE_URL + the standalone lmcache-server) is -# WRONG for SGLang. SGLang drives LMCache in multiprocess (MP) mode — config via -# the --lmcache-config-file flag (the LMCACHE_* env is ignored), a node-local -# worker over mp_host/mp_port, not a cluster-reachable lm:// server. Applied as -# written (env only, no --lmcache-config-file) the engine REFUSES TO START -# (MP mode requires --lmcache-config-file); a config file carrying remote_url -# instead would hang. Either way this manifest shows the KV-event path and -# injection surface, not a working LMCache offload. See the README's KNOWN -# LIMITATION note and docs/design/cachebackend-api.md (SGLang engine support). +# VALIDATION STATUS: this hand manifest is derived from the GPU-validated adapter +# render (store->flush->retrieve reuses KV, via the controller-rendered managed +# path, #149) and is structurally checked (kubectl apply --dry-run). It has not been +# independently re-run end-to-end on a GPU in this shape — run the SGLang leg of the +# smoke / a GPU host before treating it as golden. Pins: docs/reference-stack/VERSIONS.md. # -# >> You need an NVIDIA GPU. << SGLang loads weights on CUDA, and the LMCache -# connector offloads KV. Unlike vLLM there is no first-class CPU build to fall -# back to here, so this manifest is GPU-only. SGLang's exact KV-event wire (the -# 3-tuple EventBatch envelope) is covered off-GPU by the Go fixture -# `go test ./pkg/adapters/engine/ -run SGLang`; the Python scripts/ tooling -# additionally exercises the shared decode + token-redaction logic (it models +# >> You need an NVIDIA GPU. << SGLang loads weights on CUDA and the LMCache MP +# worker moves KV over CUDA-IPC. There is no first-class CPU fallback here, so this +# manifest is GPU-only. SGLang's exact KV-event wire (the 3-tuple EventBatch +# envelope) is covered off-GPU by `go test ./pkg/adapters/engine/ -run SGLang`; the +# Python scripts/ tooling exercises the shared decode + token-redaction (it models # vLLM's 2-tuple envelope). See this directory's README. # -# IMAGE PINNING: the SGLang image ships as a **non-applyable placeholder digest** -# (the all-zero `@sha256:` below won't pull — same convention as the vLLM -# reference manifest, see VERSIONS.md). Substitute a real digest before any run, -# and ensure the lmcache client is present in the image (the base SGLang image -# does not bundle it — bake `pip install lmcache` into a derived image or an init -# step, version-aligned with the lmcache-server tag below). +# KUBERNETES >= 1.29 REQUIRED: the MP worker is a native sidecar (an initContainers +# entry with restartPolicy: Always), which older apiservers do not understand. # -# WIRE-TEST CAVEAT — RESOLVED (GPU-validated 2026-07): the LMCACHE_REMOTE_URL -# wiring below is WRONG for SGLang (env only + no --lmcache-config-file makes the -# engine refuse to start — see the KNOWN LIMITATION above). SGLang reads config -# only from --lmcache-config-file and uses LMCache MP -# mode, not this remote lm:// server. Do not treat the env wiring as a working -# offload. Correct MP-mode wiring is the tracked follow-up; see the README. +# IMAGE PINNING (see VERSIONS.md): the SGLang engine image ships as a non-applyable +# placeholder digest (the all-zero @sha256: below won't pull) — the base sglang +# image does NOT bundle the lmcache client, so you must build a DERIVED image +# (`pip install lmcache==0.5.1` onto the GPU-validated cu13 base) and substitute its +# real digest. The MP worker defaults to the SAME image (engine <-> worker must run +# the same lmcache version — they speak the MP wire). Redis is a normal pullable +# tag; digest-pin it for production per VERSIONS.md. apiVersion: apps/v1 kind: Deployment metadata: - name: lmcache-server + name: redis-l2 namespace: cache-substrate labels: - app: lmcache-server + app: redis-l2 spec: + # SINGLETON: the Redis L2 holds no replicated state, so >1 replica would shard KV + # across independent keyspaces and silently partition the tier. Keep replicas: 1 + # (the managed path's admission enforces this for a (sglang, LMCache) backend). replicas: 1 selector: matchLabels: - app: lmcache-server + app: redis-l2 template: metadata: labels: - app: lmcache-server + app: redis-l2 spec: containers: - - name: lmcache-server - # Non-applyable placeholder digest, like the engine image above — pin - # both before a run. = lmcache/standalone:v0.4.7 (the adapter's default, - # a fixed non-floating version); resolve its real digest per VERSIONS.md - # and substitute here. Must be wire-compatible with the lmcache *client* - # baked into the SGLang image (VERSIONS.md "version alignment"). - image: lmcache/standalone@sha256:0000000000000000000000000000000000000000000000000000000000000000 + - name: redis-l2 + # The shared L2 the MP worker offloads to (its `resp` --l2-adapter). A + # major.minor-alpine tag is stable in wire/config but mutable within its + # patch line — digest-pin for production (VERSIONS.md). Redis speaks RESP + # and needs NO lmcache-version alignment, so it moves independently of the + # engine/lmcache tuple. + image: docker.io/library/redis:7.4-alpine imagePullPolicy: IfNotPresent - command: ["lmcache_server"] - args: ["0.0.0.0", "65432", "cpu"] + # No `command`: keep the image ENTRYPOINT (docker-entrypoint.sh), which + # drops root -> the unprivileged `redis` user. `redis-server` leads args + # (replaces CMD, keeps ENTRYPOINT). --save ""/--appendonly no: ephemeral + # cache tier, no persistence/PVC (a restart starts cold — soft state). + # --maxmemory + allkeys-lru bound the dataset so write load can't OOM the + # cgroup; the managed path derives ~80% of the memory limit (here ~6Gi of + # the 8Gi limit). --protected-mode no: the worker dials over the ClusterIP + # (a non-loopback client). + args: + - "redis-server" + - "--save" + - "" + - "--appendonly" + - "no" + - "--protected-mode" + - "no" + - "--maxmemory" + - "6gb" + - "--maxmemory-policy" + - "allkeys-lru" ports: - - { name: lmcache, containerPort: 65432 } + - { name: redis, containerPort: 6379 } readinessProbe: - tcpSocket: { port: lmcache } - initialDelaySeconds: 5 + tcpSocket: { port: redis } + initialDelaySeconds: 3 periodSeconds: 10 failureThreshold: 6 - # The lmcache-server keeps KV in memory; bound it by the cgroup so - # tier-2 (offload) write load can't OOM the node. Mirrors the managed adapter's - # spec.resources CRD default (4Gi request / 8Gi limit) — size to your - # working set (a large model's KV is tens of GB; see the version- - # alignment note in docs/design/cachebackend-api.md). + # The memory LIMIT both bounds the tier and sizes --maxmemory above. Size + # to your working set (a large model's KV is tens of GB). resources: requests: { memory: 4Gi } limits: { memory: 8Gi } @@ -91,16 +102,16 @@ spec: apiVersion: v1 kind: Service metadata: - name: lmcache-server + name: redis-l2 namespace: cache-substrate labels: - app: lmcache-server + app: redis-l2 spec: type: ClusterIP selector: - app: lmcache-server + app: redis-l2 ports: - - { name: lmcache, port: 65432, targetPort: lmcache } + - { name: redis, port: 6379, targetPort: redis } --- apiVersion: apps/v1 kind: Deployment @@ -110,7 +121,7 @@ metadata: labels: app: sglang-lmcache-llama-8b # The cache plane's mutating Pod webhook matches engine pods by label - # (CacheBackend.spec.engineSelector). This mirrors config/samples/cachebackend-sglang.yaml. + # (CacheBackend.spec.engineSelector). Mirrors config/samples/cachebackend-sglang.yaml. app.kubernetes.io/name: sglang spec: replicas: 1 @@ -123,15 +134,80 @@ spec: app: sglang-lmcache-llama-8b app.kubernetes.io/name: sglang spec: + # The MP worker is a NATIVE SIDECAR: an initContainers entry with + # restartPolicy: Always, so it starts (and its startupProbe gates the engine) + # before the engine container, and keeps running for the pod's life. Requires + # Kubernetes >= 1.29. + initContainers: + - name: lmcache-mp-worker + restartPolicy: Always + # Defaults to the ENGINE image so the worker and engine run the same + # lmcache version (they speak the MP wire). Same non-applyable placeholder + # digest as the engine — substitute your derived image (VERSIONS.md). + image: example.invalid/sglang-lmcache@sha256:0000000000000000000000000000000000000000000000000000000000000000 + imagePullPolicy: IfNotPresent + # Write the engine's MP config file (mp_host/mp_port), then exec the + # LMCache MP server on loopback, offloading to the Redis L2 via `resp`. + # This is the shape the adapter renders; keep chunk_size/mp_port/l1-size-gb + # in sync with the config the engine reads (below) and the shm sizeLimit. + command: ["sh", "-c"] + args: + - | + set -e + printf 'chunk_size: 256\nmp_host: "127.0.0.1"\nmp_port: 5555\n' > /etc/lmcache/config.yaml + exec python3 -m lmcache.v1.multiprocess.server \ + --host 127.0.0.1 --port 5555 \ + --chunk-size 256 --l1-size-gb 4 --eviction-policy LRU \ + --l2-adapter '{"type":"resp","host":"redis-l2","port":6379}' + env: + # The GPU-less worker must SEE the engine's GPU to CUDA-IPC its KV. It + # holds no nvidia.com/gpu request (consumes none of the node's + # allocatable), but the visibility is real — on a shared node it can see + # every GPU. GPU-VALIDATED as required: revoking it kills the worker + # ("Device UUID ... not found") and the engine never readies. See the + # GPU-visibility note in docs/design/cachebackend-api.md. + - { name: NVIDIA_VISIBLE_DEVICES, value: "all" } + volumeMounts: + - { name: lmcache-config, mountPath: /etc/lmcache } + - { name: shm, mountPath: /dev/shm } # shared L1 tmpfs with the engine + # The MP server binds mp_port on loopback, which a pod-IP probe can't + # reach — exec a loopback check so the engine only starts once the worker + # is listening. Gating the engine on the worker is deliberate: SGLang has + # no cacheless fallback while --enable-lmcache is on (see "Fail-open + # semantics" in docs/design/sglang-lmcache-mp-mode.md). + startupProbe: + exec: + command: + - python3 + - -c + - "import socket; socket.create_connection(('127.0.0.1',5555),1)" + periodSeconds: 3 + failureThreshold: 40 + # The L1 lives in the memory-backed /dev/shm tmpfs, charged to this + # container's cgroup, so it MUST carry a matching memory request+limit + # (l1-size-gb + ~1Gi headroom) or the L1 is invisible to the scheduler. + resources: + requests: { memory: 5Gi } + limits: { memory: 5Gi } + # Restricted-compatible: this mutation lands before Pod Security admission, + # so the worker must carry the container-only Restricted requirements + # itself or it would get the whole engine pod rejected in a restricted + # namespace. No added capabilities (KV moves over CUDA-IPC + /dev/shm). + securityContext: + allowPrivilegeEscalation: false + capabilities: + drop: ["ALL"] + seccompProfile: + type: RuntimeDefault containers: # Container name MUST be `sglang` so the adapter's InjectEngineConfig / # engineOverrides target the right container (EngineContainerName()). - name: sglang - # Non-applyable placeholder for a DERIVED image: base lmsysorg/sglang - # with the lmcache client baked in (the base does NOT bundle it — see - # header + VERSIONS.md for the build/pin steps). The repo name is - # deliberately not `lmsysorg/sglang` so that substituting only the - # digest can't accidentally pull the lmcache-less upstream base. + # Non-applyable placeholder for a DERIVED image: base lmsysorg/sglang (the + # GPU-validated cu13 tag) with the lmcache client baked in (the base does + # NOT bundle it — see header + VERSIONS.md). The repo name is deliberately + # not `lmsysorg/sglang` so substituting only the digest can't accidentally + # pull the lmcache-less upstream base. image: example.invalid/sglang-lmcache@sha256:0000000000000000000000000000000000000000000000000000000000000000 imagePullPolicy: IfNotPresent command: ["python", "-m", "sglang.launch_server"] @@ -144,38 +220,31 @@ spec: - "--model-path=meta-llama/Meta-Llama-3-8B-Instruct" - "--host=0.0.0.0" - "--port=30000" - # SGLang's LMCache connector (store_true flag — no value). + # SGLang's LMCache connector (store_true flag — no value). Replaces + # vLLM's --kv-transfer-config. - "--enable-lmcache" + # Points the engine at the MP config file the worker writes (mp_host/ + # mp_port). MP mode aborts at startup WITHOUT this flag. + - "--lmcache-config-file" + - "/etc/lmcache/config.yaml" # KV-cache event publisher — SGLang's ZmqEventPublisher, same event - # structs as vLLM (3-tuple batch envelope). The consumer is the Go - # kvevent-subscriber (managed path); the Python scripts/kv_events_subscriber.py - # demo decodes only vLLM's 2-tuple shape (UNDECODED on SGLang). - # Binds tcp://*:5557 — the operator contract in - # docs/design/cachebackend-api.md (kept identical to the vLLM - # reference). NOTE: these raw frames carry token_ids, and tcp://* means - # the port is reachable by any pod that knows this pod's IP. The - # metadata-only guarantee is REPORT-level (the in-pod subscriber - # forwards only hashes + counts over 127.0.0.1); to also keep the RAW - # frames off the cluster network, this Service omits :5557 and you can - # add a NetworkPolicy restricting :5557 to this pod. See the README - # privacy note. + # structs as vLLM (3-tuple batch envelope). Consumer is the Go + # kvevent-subscriber (managed path). Binds tcp://*:5557 — the operator + # contract in docs/design/cachebackend-api.md (kept identical to the + # vLLM reference). These raw frames carry token_ids; the Service below + # omits :5557 so they stay off the cluster network (the in-pod + # subscriber reads them over 127.0.0.1). See the README privacy note. - "--kv-events-config" - '{"publisher":"zmq","endpoint":"tcp://*:5557","buffer_steps":10000,"topic":"kv-events"}' env: # Gates SGLang's experimental LMCache integration — required for # --enable-lmcache to engage the connector. - { name: LMCACHE_USE_EXPERIMENTAL, value: "True" } - # The remote lmcache-server, lm:// scheme — what the adapter injects. - # KNOWN-BROKEN for SGLang (ignored env; SGLang needs MP mode — see the - # KNOWN LIMITATION header). Kept to document the shipped wiring. - - { name: LMCACHE_REMOTE_URL, value: "lm://lmcache-server:65432" } - - { name: LMCACHE_REMOTE_SERDE, value: "naive" } - - { name: LMCACHE_CHUNK_SIZE, value: "256" } - - { name: LMCACHE_LOCAL_CPU, value: "False" } - - { name: LMCACHE_MAX_LOCAL_CPU_SIZE, value: "20" } # Mirror of spec.integration.failOpen (defaults true) — the adapter # injects this as a reserved env so the engine layer can honor the - # fail-open contract. Part of the adapter-injected set. + # fail-open contract. (The old lm:// env — LMCACHE_REMOTE_URL / SERDE / + # CHUNK_SIZE / LOCAL_CPU — is intentionally absent: MP mode ignores it, + # and the MP worker's tunables live in its config file / CLI above.) - { name: INFERENCECACHE_FAIL_OPEN, value: "true" } # Gated model — needs a Hugging Face token (see README step 2). - name: HF_TOKEN @@ -194,14 +263,20 @@ spec: nvidia.com/gpu: "1" volumeMounts: - { name: cache-home, mountPath: /root/.cache/huggingface } - - { name: shm, mountPath: /dev/shm } + - { name: shm, mountPath: /dev/shm } # shared L1 tmpfs with the MP worker + - { name: lmcache-config, mountPath: /etc/lmcache } # reads the worker's config.yaml volumes: - name: cache-home emptyDir: {} + # The MP L1 tier lives here; shared by the engine and the worker. Sized to + # cover the engine's own /dev/shm use PLUS the L1 budget (l1-size-gb=4). - name: shm emptyDir: medium: Memory sizeLimit: 8Gi + # Carries the MP config file the worker writes and the engine reads. + - name: lmcache-config + emptyDir: {} --- apiVersion: v1 kind: Service @@ -217,8 +292,7 @@ spec: # Only the HTTP API is node-exposed (for driving traffic); reuses the kind # cluster's 30080 mapping (../../kind/cluster.yaml), shared with the vLLM # reference — deploy ONE engine reference at a time. The KV-event port (:5557) - # is deliberately NOT in this Service: those raw frames carry token_ids, and - # the consumer is the in-pod subscriber sidecar reading 127.0.0.1 (managed - # path) — there is no need to expose token-bearing frames cluster-wide. + # is deliberately NOT in this Service: those raw frames carry token_ids, and the + # consumer is the in-pod subscriber sidecar reading 127.0.0.1 (managed path). ports: - { name: http, port: 30000, targetPort: http, nodePort: 30080 } From 877e5cb38e416310a6c7f9b5eafb89d4682bfe13 Mon Sep 17 00:00:00 2001 From: Ed Sun Date: Thu, 23 Jul 2026 13:38:36 -0700 Subject: [PATCH 2/5] docs(sglang): assert dbsize>0 (write-through); fix stale VERSIONS note (review) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit LMCache MP is write-through — KV lands in Redis on store, not only on HBM eviction — so the standalone check asserts dbsize>0 after requests rather than making it eviction-conditional. Also drop the VERSIONS.md parenthetical that still called the manifest rewrite a future increment (it landed in this PR). --- docs/reference-stack/VERSIONS.md | 5 +++-- .../manifests/sglang-lmcache/README.md | 22 +++++++++---------- 2 files changed, 14 insertions(+), 13 deletions(-) diff --git a/docs/reference-stack/VERSIONS.md b/docs/reference-stack/VERSIONS.md index 981194d3..5bcaafc2 100644 --- a/docs/reference-stack/VERSIONS.md +++ b/docs/reference-stack/VERSIONS.md @@ -103,8 +103,9 @@ replaces that placeholder.) Concretely: build steps above. The **reference manifest** still ships a non-applyable placeholder digest under an `example.invalid/sglang-lmcache` name (all-zero `@sha256:`) — substitute your derived image's real digest before the GPU run. - (Updating that manifest to the validated tuple is part of the reference-stack - increment; the controller-rendered managed path does not read it.) + (The manifest already renders the MP topology — Redis L2 + MP-worker sidecar; only + substituting your operator-built image digest remains. The controller-rendered + managed path does not read this manifest.) - **RESOLVED — the concrete `(sglang-tag, lmcache-version)` tuple is `(lmsysorg/sglang:nightly-dev-cu13-20260711-7de33ce8, lmcache 0.5.1)`**, validated end-to-end on an A100: the MP worker registers the engine's KV cache over CUDA-IPC diff --git a/docs/reference-stack/manifests/sglang-lmcache/README.md b/docs/reference-stack/manifests/sglang-lmcache/README.md index e8c9019f..99d82472 100644 --- a/docs/reference-stack/manifests/sglang-lmcache/README.md +++ b/docs/reference-stack/manifests/sglang-lmcache/README.md @@ -174,21 +174,21 @@ kill "$pf" 2>/dev/null; trap - EXIT || { echo "FAIL: KV-event publisher did not start (check --kv-events-config)"; exit 1; } ``` -- **KV offloaded to Redis.** After driving traffic that spans an HBM eviction, the L2 - keyspace is non-empty — a quick check via the MP worker container: +- **KV offloaded to Redis.** LMCache MP mode is **write-through**: a stored prefix + lands in the L2 immediately, not only on HBM eviction (GPU-validated — a 3760-token + prompt took Redis `DBSIZE` 0→14, i.e. 14 chunks of the 256-token `chunk_size`). So + after driving the prompts above, assert the L2 keyspace is non-empty: ```bash - # The MP worker offloads to redis-l2 over `resp`. A non-zero dbsize after eviction - # is the standalone signal that offload is actually happening. - kubectl -n cache-substrate exec deploy/redis-l2 -c redis-l2 -- redis-cli dbsize + # The MP worker offloads to redis-l2 over `resp`. dbsize > 0 after requests is the + # standalone signal that offload is actually happening. + n=$(kubectl -n cache-substrate exec deploy/redis-l2 -c redis-l2 -- redis-cli dbsize | tr -dc '0-9') + [ "${n:-0}" -gt 0 ] || { echo "FAIL: Redis L2 empty after requests — offload not happening"; exit 1; } + echo "Redis L2 holds $n KV chunks" ``` - (Whether an eviction occurs depends on model size vs. HBM; a larger working set or a - smaller `--mem-fraction-static` forces it. On a small model with plenty of HBM the - L1 may hold everything and Redis stays empty — that is not a failure, just no - eviction. Full frame→index→`LookupRoute` verification needs the **managed path** - below; the standalone manifest ships no event consumer, so it never populates the - index.) + (Full frame→index→`LookupRoute` verification needs the **managed path** below; the + standalone manifest ships no event consumer, so it never populates the index.) - **Privacy boundary — the RAW ZMQ frames carry `token_ids`.** Both vLLM's and SGLang's `BlockStored` wire includes the block's token ids; the "metadata-only, From 51d7bb0ccbad7fa8635bad2f0813f5d63e1460d7 Mon Sep 17 00:00:00 2001 From: Ed Sun Date: Thu, 23 Jul 2026 13:44:07 -0700 Subject: [PATCH 3/5] docs(sglang): byte-exact Redis maxmemory; kill write-through contradiction; precise privacy note (review r2) - --maxmemory 6gb (6e9 bytes) understated the managed ~80%-of-8Gi render; set the byte-exact 6871947674 (~6.4Gi). - step-4 said offload happens 'on HBM eviction', contradicting the write-through DBSIZE check; reword to write-through. - Service :5557 comment no longer overstates the privacy guarantee (pod IP is still reachable without a NetworkPolicy). - Enumerate what a GPU run must confirm before this is treated as golden. --- .../manifests/sglang-lmcache/README.md | 5 +++-- .../manifests/sglang-lmcache/deployment.yaml | 22 +++++++++++-------- 2 files changed, 16 insertions(+), 11 deletions(-) diff --git a/docs/reference-stack/manifests/sglang-lmcache/README.md b/docs/reference-stack/manifests/sglang-lmcache/README.md index 99d82472..5a21677f 100644 --- a/docs/reference-stack/manifests/sglang-lmcache/README.md +++ b/docs/reference-stack/manifests/sglang-lmcache/README.md @@ -126,8 +126,9 @@ kubectl -n cache-substrate rollout status deploy/redis-l2 --timeout=5m kubectl -n cache-substrate rollout status deploy/sglang-lmcache-llama-8b --timeout=20m # 4. Drive the SAME long-prefix prompt twice (first warms, second reuses). This -# exercises the engine, triggers its KV-event publisher on :5557, and — on HBM -# eviction — offloads KV to Redis via the MP worker. Swap the `prefix = ...` line +# exercises the engine, triggers its KV-event publisher on :5557, and — because +# MP mode is write-through — offloads stored KV to Redis immediately (not only on +# HBM eviction; see the DBSIZE check below). Swap the `prefix = ...` line # for a REAL prompt long enough to span several KV blocks (>> the engine's # --page-size in tokens); a short literal won't reliably produce BlockStored / # prefix reuse. diff --git a/docs/reference-stack/manifests/sglang-lmcache/deployment.yaml b/docs/reference-stack/manifests/sglang-lmcache/deployment.yaml index fffc7b25..d07bef52 100644 --- a/docs/reference-stack/manifests/sglang-lmcache/deployment.yaml +++ b/docs/reference-stack/manifests/sglang-lmcache/deployment.yaml @@ -16,9 +16,11 @@ # # VALIDATION STATUS: this hand manifest is derived from the GPU-validated adapter # render (store->flush->retrieve reuses KV, via the controller-rendered managed -# path, #149) and is structurally checked (kubectl apply --dry-run). It has not been -# independently re-run end-to-end on a GPU in this shape — run the SGLang leg of the -# smoke / a GPU host before treating it as golden. Pins: docs/reference-stack/VERSIONS.md. +# path, #149) and is structurally checked (kubectl apply --dry-run). It has NOT been +# independently re-run end-to-end on a GPU in this exact shape — before treating it +# as a golden reference, a GPU run should confirm: engine + MP-worker startup, the +# config-file handoff, CUDA-IPC attachment, and a Redis store/retrieve (README step 4). +# Pins: docs/reference-stack/VERSIONS.md. # # >> You need an NVIDIA GPU. << SGLang loads weights on CUDA and the LMCache MP # worker moves KV over CUDA-IPC. There is no first-class CPU fallback here, so this @@ -71,9 +73,10 @@ spec: # (replaces CMD, keeps ENTRYPOINT). --save ""/--appendonly no: ephemeral # cache tier, no persistence/PVC (a restart starts cold — soft state). # --maxmemory + allkeys-lru bound the dataset so write load can't OOM the - # cgroup; the managed path derives ~80% of the memory limit (here ~6Gi of - # the 8Gi limit). --protected-mode no: the worker dials over the ClusterIP - # (a non-loopback client). + # cgroup; the managed path derives ~80% of the memory LIMIT (below: the + # byte-exact value for an 8Gi limit, 8589934592 - 8589934592/5 = + # 6871947674 ≈ 6.4Gi). --protected-mode no: the worker dials over the + # ClusterIP (a non-loopback client). args: - "redis-server" - "--save" @@ -83,7 +86,7 @@ spec: - "--protected-mode" - "no" - "--maxmemory" - - "6gb" + - "6871947674" # bytes = ~80% of the 8Gi limit below (matches the managed render) - "--maxmemory-policy" - "allkeys-lru" ports: @@ -232,8 +235,9 @@ spec: # kvevent-subscriber (managed path). Binds tcp://*:5557 — the operator # contract in docs/design/cachebackend-api.md (kept identical to the # vLLM reference). These raw frames carry token_ids; the Service below - # omits :5557 so they stay off the cluster network (the in-pod - # subscriber reads them over 127.0.0.1). See the README privacy note. + # omits :5557, so they aren't Service-exposed — but the pod IP is still + # directly reachable in-cluster unless a NetworkPolicy restricts it (the + # in-pod subscriber reads over 127.0.0.1). See the README privacy note. - "--kv-events-config" - '{"publisher":"zmq","endpoint":"tcp://*:5557","buffer_steps":10000,"topic":"kv-events"}' env: From c0f38d498d43839dc6e47ae1bd29e683df77467b Mon Sep 17 00:00:00 2001 From: Ed Sun Date: Thu, 23 Jul 2026 13:52:01 -0700 Subject: [PATCH 4/5] docs(sglang): prove L2 retrieval (flush+reload), not just writes; fix VERSIONS Redis row (review r3) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - DBSIZE>0 only proves write-through offload; add a flush_cache + re-request step asserting the worker's 'Retrieved' log — the store->flush->retrieve cycle the managed path was GPU-validated on. - The Redis L2 row said 'not a reference manifest / code default only', now contradicted by the manifest this PR added; list it in Where + distinguish the pullable-tag default from a reproducible pin. --- docs/reference-stack/VERSIONS.md | 2 +- .../manifests/sglang-lmcache/README.md | 29 +++++++++++++++---- 2 files changed, 24 insertions(+), 7 deletions(-) diff --git a/docs/reference-stack/VERSIONS.md b/docs/reference-stack/VERSIONS.md index 5bcaafc2..8fc5c66a 100644 --- a/docs/reference-stack/VERSIONS.md +++ b/docs/reference-stack/VERSIONS.md @@ -19,7 +19,7 @@ on a GPU host, then propagate to any automation that templates these manifests. | Model | `meta-llama/Llama-3.1-8B-Instruct` | `manifests/deployment.yaml` | Gated on HF; needs `HF_TOKEN`. Small enough for a single A10/L40S-class GPU. Swap freely. | | SGLang engine + MP worker | base `docker.io/lmsysorg/sglang:nightly-dev-cu13-20260711-7de33ce8` + **lmcache 0.5.1** → derive + `@sha256:` | `manifests/sglang-lmcache/deployment.yaml`; the controller-rendered worker defaults to the **engine's own image** (`backendConfig.workerImage` overrides) | **GPU-validated tuple** (store→flush→retrieve reuses KV). The engine and the MP worker MUST run the same lmcache version — they speak the MP wire to each other — which is why the worker defaults to the engine image; overriding `workerImage` makes that alignment yours to maintain. Base `lmsysorg/sglang` does **not** bundle lmcache: derive an image (`pip install lmcache==0.5.1`) and pin its digest. **cu13 is load-bearing**: lmcache 0.5.1 needs `libcudart.so.13`, so a cu12 base fails to align, and stock `v0.5.1.post2-cu126` is too old to have `--enable-lmcache`. See [SGLang derived image reproducibility](#sglang-derived-image-reproducibility). GPU-only. | | lmcache-server image | `lmcache/standalone:v0.4.7` → `@sha256:` | `manifests/deployment.yaml` (vLLM) | **vLLM only.** The `lm://` standalone server, correct and in use for the **vLLM+LMCache** managed-backend default. **Not part of the SGLang wire in any form**: SGLang is MP-only, and `lm://` is not a valid MP `--l2-adapter` type, so it is not reachable behind the MP worker either — the SGLang shared tier is the Redis L2 row below. (An earlier revision of this file predicted the MP fix would reuse this server behind a per-node worker; GPU validation disproved that, and the prediction is retired rather than left to mislead.) | -| Redis L2 store (SGLang) | `docker.io/library/redis:7.4-alpine` → `@sha256:` | `pkg/adapters/runtime/redis_l2.go` (`defaultRedisImage`), overridable per-CR via `backendConfig.redisImage` | The shared **L2** the SGLang LMCache **MP worker** offloads to (its `resp` `--l2-adapter`) — the SGLang analogue of the `lm://` lmcache-server, which SGLang cannot use (`lm://` is not a valid MP `--l2-adapter` type). Rendered by the controller, not a reference manifest, so this is a **code default** rather than a manifest pin. `7.4-alpine` is a minor-version tag: stable in wire protocol and config surface, but **mutable within its patch line**. Per the image-pin policy in [`sglang-lmcache-mp-mode.md`](../design/sglang-lmcache-mp-mode.md), production **must** override `backendConfig.redisImage` with an exact release or `@sha256:` digest — the default is a convenience for dev/smoke, not a reproducible pin. Redis itself needs no lmcache version alignment (the MP worker speaks RESP), so this pin is independent of the engine/lmcache tuple above. | +| Redis L2 store (SGLang) | `docker.io/library/redis:7.4-alpine` → `@sha256:` | `pkg/adapters/runtime/redis_l2.go` (`defaultRedisImage`, overridable per-CR via `backendConfig.redisImage`) **and** `manifests/sglang-lmcache/deployment.yaml` (the reference manifest ships this pullable tag) | The shared **L2** the SGLang LMCache **MP worker** offloads to (its `resp` `--l2-adapter`) — the SGLang analogue of the `lm://` lmcache-server, which SGLang cannot use (`lm://` is not a valid MP `--l2-adapter` type). Both the controller default and the reference manifest use this tag as a **convenience default**, not a reproducible pin (unlike the GPU images above, `redis:7.4-alpine` is pullable, so the manifest ships the tag rather than a placeholder digest). `7.4-alpine` is a minor-version tag: stable in wire protocol and config surface, but **mutable within its patch line**. Per the image-pin policy in [`sglang-lmcache-mp-mode.md`](../design/sglang-lmcache-mp-mode.md), production **must** override `backendConfig.redisImage` with an exact release or `@sha256:` digest — the default is a convenience for dev/smoke, not a reproducible pin. Redis itself needs no lmcache version alignment (the MP worker speaks RESP), so this pin is independent of the engine/lmcache tuple above. | | SGLang model | `meta-llama/Meta-Llama-3-8B-Instruct` | `manifests/sglang-lmcache/deployment.yaml` | Served model for the SGLang reference, kept equal to `config/samples/cachebackend-sglang.yaml`'s `backendConfig.model` so the managed-path docs line up. Gated on HF; needs `HF_TOKEN`. Swap freely, but keep the engine `--model-path`, the CacheBackend `backendConfig.model`, and request `model` identical. | | CPU image | `vllm/vllm-openai-cpu:latest-{x86_64,arm64}` | `manifests/cpu-local/deployment.yaml` | vLLM's dedicated CPU build (arch-tagged). Runs the v1 engine on CPU (vLLM >= ~0.21), incl. the KV-event publisher. Verified on `0.21.0` (arm64): prefix-cache hit + real ZMQ events. Needs adequate RAM (CPU baseline ~5 GiB + KV). | | CPU model | `Qwen/Qwen2.5-0.5B-Instruct` | `manifests/cpu-local/deployment.yaml` | Ungated, tiny, CPU-runnable. | diff --git a/docs/reference-stack/manifests/sglang-lmcache/README.md b/docs/reference-stack/manifests/sglang-lmcache/README.md index 5a21677f..e2eea103 100644 --- a/docs/reference-stack/manifests/sglang-lmcache/README.md +++ b/docs/reference-stack/manifests/sglang-lmcache/README.md @@ -175,19 +175,36 @@ kill "$pf" 2>/dev/null; trap - EXIT || { echo "FAIL: KV-event publisher did not start (check --kv-events-config)"; exit 1; } ``` -- **KV offloaded to Redis.** LMCache MP mode is **write-through**: a stored prefix - lands in the L2 immediately, not only on HBM eviction (GPU-validated — a 3760-token - prompt took Redis `DBSIZE` 0→14, i.e. 14 chunks of the 256-token `chunk_size`). So - after driving the prompts above, assert the L2 keyspace is non-empty: +- **KV *written* to Redis (write-through).** LMCache MP mode is write-through: a + stored prefix lands in the L2 immediately, not only on HBM eviction (GPU-validated — + a 3760-token prompt took Redis `DBSIZE` 0→14, i.e. 14 chunks of the 256-token + `chunk_size`). This proves the offload **write** path — assert the keyspace is + non-empty after the requests above: ```bash - # The MP worker offloads to redis-l2 over `resp`. dbsize > 0 after requests is the - # standalone signal that offload is actually happening. + # dbsize > 0 proves the MP worker wrote KV to redis-l2 over `resp`. n=$(kubectl -n cache-substrate exec deploy/redis-l2 -c redis-l2 -- redis-cli dbsize | tr -dc '0-9') [ "${n:-0}" -gt 0 ] || { echo "FAIL: Redis L2 empty after requests — offload not happening"; exit 1; } echo "Redis L2 holds $n KV chunks" ``` +- **KV *reloaded* from Redis (the read path).** `DBSIZE > 0` only proves writes — a + repeat request could still be served from the engine's own GPU radix / L1 cache + without touching L2. To prove **retrieval**, flush the engine's local cache first, + then re-request, and look for the worker's reload signal (this is the + store→flush→retrieve cycle the managed path was GPU-validated on): + + ```bash + kubectl -n cache-substrate port-forward svc/sglang-lmcache-llama-8b 30000:30000 & pf=$!; sleep 3 + curl -sfS -X POST localhost:30000/flush_cache >/dev/null # clear the engine's GPU radix cache + curl -sfS localhost:30000/v1/chat/completions -H 'content-type: application/json' -d "$BODY" >/dev/null + kill "$pf" 2>/dev/null + # The MP worker logs a retrieval when it serves the prefix back from L1+L2: + kubectl -n cache-substrate logs deploy/sglang-lmcache-llama-8b -c lmcache-mp-worker \ + | grep -iE 'Retrieved [0-9]+ tokens|Prefetch .*L2' \ + || { echo "FAIL: no L2 retrieval after flush — reload path not working"; exit 1; } + ``` + (Full frame→index→`LookupRoute` verification needs the **managed path** below; the standalone manifest ships no event consumer, so it never populates the index.) From 4f9d94207ed22fab71e00a97f5a8bb7801a998cd Mon Sep 17 00:00:00 2001 From: Ed Sun Date: Thu, 23 Jul 2026 13:55:33 -0700 Subject: [PATCH 5/5] docs(sglang): scope L2-retrieval grep to post-flush (--since-time) + check curls; --dry-run=client (review r4) The retrieval check grepped the whole worker log, so a warm-up retrieval could false-pass; scope it with --since-time and fail the flush/request curls explicitly. Also spell out --dry-run=client (bare --dry-run is deprecated). --- docs/reference-stack/VERSIONS.md | 2 +- .../manifests/sglang-lmcache/README.md | 14 +++++++++----- .../manifests/sglang-lmcache/deployment.yaml | 2 +- 3 files changed, 11 insertions(+), 7 deletions(-) diff --git a/docs/reference-stack/VERSIONS.md b/docs/reference-stack/VERSIONS.md index 8fc5c66a..159d4fa8 100644 --- a/docs/reference-stack/VERSIONS.md +++ b/docs/reference-stack/VERSIONS.md @@ -49,7 +49,7 @@ pin an lmcache-server — SGLang never dials one. > the **MP-worker native sidecar** + the derived engine image (still a non-applyable > placeholder digest — substitute your own build). It is **derived from the > GPU-validated adapter render** (`redis_l2.go` + the SGLang adapter) and structurally -> checked (`kubectl apply --dry-run`); re-run it on a GPU before treating it as golden. +> checked (`kubectl apply --dry-run=client`); re-run it on a GPU before treating it as golden. > The pins below are authoritative for **both** the manifest and the controller-rendered > managed path. diff --git a/docs/reference-stack/manifests/sglang-lmcache/README.md b/docs/reference-stack/manifests/sglang-lmcache/README.md index e2eea103..ebc7a9d7 100644 --- a/docs/reference-stack/manifests/sglang-lmcache/README.md +++ b/docs/reference-stack/manifests/sglang-lmcache/README.md @@ -14,7 +14,7 @@ is **not** byte-for-byte adapter output. > **Validation status.** This manifest is **derived from the GPU-validated adapter > render** (`sglang_mp.go` + `redis_l2.go`; the controller-rendered managed path was > validated store→flush→retrieve end-to-end in the MP-mode increment) and is -> **structurally checked** (`kubectl apply --dry-run`). It has **not** been +> **structurally checked** (`kubectl apply --dry-run=client`). It has **not** been > independently re-run end-to-end on a GPU in this exact hand shape — run it on a GPU > host (below) before treating it as a golden reference. All pins live in > [`../../VERSIONS.md`](../../VERSIONS.md). @@ -195,12 +195,16 @@ kill "$pf" 2>/dev/null; trap - EXIT store→flush→retrieve cycle the managed path was GPU-validated on): ```bash + ts=$(date -u +%Y-%m-%dT%H:%M:%SZ) # only count retrievals logged AFTER this point kubectl -n cache-substrate port-forward svc/sglang-lmcache-llama-8b 30000:30000 & pf=$!; sleep 3 - curl -sfS -X POST localhost:30000/flush_cache >/dev/null # clear the engine's GPU radix cache - curl -sfS localhost:30000/v1/chat/completions -H 'content-type: application/json' -d "$BODY" >/dev/null + curl -sfS -X POST localhost:30000/flush_cache >/dev/null \ + || { kill "$pf" 2>/dev/null; echo "FAIL: flush_cache request failed"; exit 1; } # clear GPU radix cache + curl -sfS localhost:30000/v1/chat/completions -H 'content-type: application/json' -d "$BODY" >/dev/null \ + || { kill "$pf" 2>/dev/null; echo "FAIL: post-flush request not served"; exit 1; } kill "$pf" 2>/dev/null - # The MP worker logs a retrieval when it serves the prefix back from L1+L2: - kubectl -n cache-substrate logs deploy/sglang-lmcache-llama-8b -c lmcache-mp-worker \ + # A retrieval logged AFTER the flush (scoped by --since-time so a stale warm-up + # retrieval can't false-pass) proves the L1/L2 reload: + kubectl -n cache-substrate logs deploy/sglang-lmcache-llama-8b -c lmcache-mp-worker --since-time="$ts" \ | grep -iE 'Retrieved [0-9]+ tokens|Prefetch .*L2' \ || { echo "FAIL: no L2 retrieval after flush — reload path not working"; exit 1; } ``` diff --git a/docs/reference-stack/manifests/sglang-lmcache/deployment.yaml b/docs/reference-stack/manifests/sglang-lmcache/deployment.yaml index d07bef52..617a3641 100644 --- a/docs/reference-stack/manifests/sglang-lmcache/deployment.yaml +++ b/docs/reference-stack/manifests/sglang-lmcache/deployment.yaml @@ -16,7 +16,7 @@ # # VALIDATION STATUS: this hand manifest is derived from the GPU-validated adapter # render (store->flush->retrieve reuses KV, via the controller-rendered managed -# path, #149) and is structurally checked (kubectl apply --dry-run). It has NOT been +# path, #149) and is structurally checked (kubectl apply --dry-run=client). It has NOT been # independently re-run end-to-end on a GPU in this exact shape — before treating it # as a golden reference, a GPU run should confirm: engine + MP-worker startup, the # config-file handoff, CUDA-IPC attachment, and a Redis store/retrieve (README step 4).