docs(sglang): rewrite the reference stack to the MP data plane

[EdHasNoLife]

The SGLang reference stack still described the pre-MP lm:// model with a pervasive "KNOWN LIMITATION / caches nothing / do not use as a working offload / drop --enable-lmcache" narrative — now false: the MP data plane shipped (#149), so SGLang genuinely offloads. An operator following the reference was told a working feature is broken.

Rewrites the reference stack to the shipped MP topology:

• manifests/sglang-lmcache/deployment.yaml — replaces the standalone lm:// lmcache-server + LMCACHE_REMOTE_URL env with a Redis L2 store + a node-local MP-worker native sidecar (an initContainers entry with restartPolicy: Always) that writes the --lmcache-config-file and offloads to Redis via the resp --l2-adapter; the engine gets --enable-lmcache + --lmcache-config-file, and the lm:// env is dropped.
• manifests/sglang-lmcache/README.md — rewritten to the working MP flow (deploy Redis L2 + engine + worker, drive traffic, assert publisher-start + a non-empty Redis L2 after eviction), keeping the still-valid off-GPU wire tests, privacy boundary, managed-path, and block-size sections.
• VERSIONS.md — refreshes the note that flagged the manifest as lagging (it now matches). The pins themselves were already MP.

Validation. Derived from the GPU-validated adapter render (redis_l2.go + internal/enginewire/sglang_mp.go; the controller-rendered managed path was validated store→flush→retrieve in the MP increment) and structurally validated (kubectl apply --dry-run=client passes all four objects; native-sidecar restartPolicy recognized). It has not been independently re-run end-to-end on a GPU in this hand shape — the manifest + README + VERSIONS all say so and recommend a GPU run before treating it as golden.

Still deferred under CAC-251 (not in this PR): the MP smoke assertion. It modifies the shared 3,600-line per-PR install-smoke gate, which I can't run locally — adding an unvalidated assertion there is high-blast-radius (a wrong resource name / timing budget breaks CI for the whole repo), so it should be added with a real kind run, not blind. (Also corrected while here: cachebackend-api.md and VERSIONS.md's pins were already MP — my earlier "still open" list over-counted; see the CAC-251 comments.)

CAC-251.

[github-actions]

Codex review

Files reviewed

• docs/reference-stack/
  • [reviewed] VERSIONS.md
• docs/reference-stack/manifests/sglang-lmcache/
  • [reviewed] README.md
  • [reviewed] deployment.yaml

Findings

Blocking

None.

Should-fix

• docs/reference-stack/manifests/sglang-lmcache/README.md:177 — The documented Redis verification neither asserts a non-zero DBSIZE nor reliably produces the claimed eviction. Moreover, the design records Redis as write-through, so the runbook should assert the parsed value is greater than zero after requests instead of making success conditional on HBM eviction.

• docs/reference-stack/VERSIONS.md:106 — The text still says updating the reference manifest to the validated tuple is a future reference-stack increment. This contradicts lines 47–54 and the purpose of this PR; clarify that only replacing the placeholder with the operator-built digest remains.

Nit

None.

Per-category coverage

1. Vendor-neutral naming: Clean
2. Contract & spec / docs sync: findings above at: docs/reference-stack/manifests/sglang-lmcache/README.md:177, docs/reference-stack/VERSIONS.md:106
3. gRPC fail-open semantics: Clean
4. Quality (tests / error wrapping / package placement / generated drift): findings above at: docs/reference-stack/manifests/sglang-lmcache/README.md:177

Verdict

changes-requested

[linear-code]

CAC-251

[github-actions]

Codex review

Files reviewed

• docs/reference-stack/
  • VERSIONS.md [reviewed]
• docs/reference-stack/manifests/sglang-lmcache/
  • README.md [reviewed]
  • deployment.yaml [reviewed]

Findings

Blocking

None.

Should-fix

• docs/reference-stack/manifests/sglang-lmcache/deployment.yaml:85 — The hand-written Redis limit does not match the managed render it claims to mirror: 6gb is 6,000,000,000 bytes, while 80% of an 8 GiB limit is approximately 6.4 GiB (6,871,947,674 bytes). This materially reduces cache capacity and makes the “same shape” and “~6Gi” documentation inaccurate.
• docs/reference-stack/manifests/sglang-lmcache/deployment.yaml:17 — Verify: the rewritten reference stack is presented as working, but this exact manifest has only received client-side schema validation and has never been run end-to-end. A GPU run should confirm container startup, config-file handoff, CUDA-IPC attachment, and Redis store/retrieve before replacing the previous warning with a working-reference claim.
• docs/reference-stack/manifests/sglang-lmcache/README.md:128 — The traffic instructions say Redis offload occurs “on HBM eviction,” while lines 177–180 state that MP mode is write-through and populates Redis immediately. These conflicting descriptions leave operators unsure what the DBSIZE assertion actually proves.

Nit

• docs/reference-stack/manifests/sglang-lmcache/deployment.yaml:234 — Omitting port 5557 from the Service does not make the wildcard-bound publisher stay “off the cluster network”; other pods can still connect directly to the pod IP. The README correctly explains this at lines 193–207, so the manifest comment should use the same qualification.

Per-category coverage

1. Vendor-neutral naming: Clean
2. Contract & spec / docs sync: findings above at: docs/reference-stack/manifests/sglang-lmcache/deployment.yaml:85, docs/reference-stack/manifests/sglang-lmcache/README.md:128
3. gRPC fail-open semantics: Clean
4. Quality (tests / error wrapping / package placement / generated drift): findings above at: docs/reference-stack/manifests/sglang-lmcache/deployment.yaml:17, docs/reference-stack/manifests/sglang-lmcache/deployment.yaml:234

Verdict

changes-requested

[github-actions]

Codex review

Files reviewed

• docs/reference-stack/
  • [reviewed] VERSIONS.md
• docs/reference-stack/manifests/sglang-lmcache/
  • [reviewed] README.md
  • [reviewed] deployment.yaml

Findings

Blocking

None.

Should-fix

• docs/reference-stack/manifests/sglang-lmcache/README.md:128 — The validation never proves retrieval from Redis L2 — Repeating the prompt can hit the engine’s existing HBM/L1 cache, while DBSIZE > 0 only proves writes. Evict or restart with a fresh local cache before the second request and assert an L2 retrieval signal.
• docs/reference-stack/VERSIONS.md:22 — The Redis row still says Redis is “Rendered by the controller, not a reference manifest” and omits the new manifest from Where — This directly contradicts the new Redis deployment and the topology note at lines 47–54; list manifests/sglang-lmcache/deployment.yaml and distinguish its pin from the managed default.

Nit

None.

Per-category coverage

1. Vendor-neutral naming: Clean
2. Contract & spec / docs sync: findings above at: docs/reference-stack/VERSIONS.md:22
3. gRPC fail-open semantics: Clean
4. Quality (tests / error wrapping / package placement / generated drift): findings above at: docs/reference-stack/manifests/sglang-lmcache/README.md:128

Verdict

changes-requested

[github-actions]

Codex review

Files reviewed

• docs/reference-stack/
  • [reviewed] VERSIONS.md
• docs/reference-stack/manifests/sglang-lmcache/
  • [reviewed] README.md
  • [reviewed] deployment.yaml

Findings

Blocking

None.

Should-fix

• docs/reference-stack/manifests/sglang-lmcache/README.md:198 — The L2-retrieval assertion can false-pass — it greps the worker’s entire log history, so a retrieval emitted before flush_cache satisfies the check; the fixed three-second port-forward delay and unchecked flush also make the sequence unreliable. Capture a timestamp/log offset and require a new retrieval after successful flush and request calls.
• docs/reference-stack/manifests/sglang-lmcache/README.md:14 — Verify: the exact hand-written reference topology has not been exercised end-to-end — this PR changes the documented stack from known-broken to working, but only the related controller render was GPU-tested. Run the supplied manifest through store→flush→retrieve on a GPU before presenting it as the working reference.
• docs/reference-stack/manifests/sglang-lmcache/deployment.yaml:17 — Verify: the exact manifest’s config handoff, CUDA-IPC attachment, and Redis read/write path remain untested — structural dry-run validation cannot catch image/runtime or data-plane incompatibilities. This is the core behavior introduced by the PR and needs a GPU smoke run.

Nit

• docs/reference-stack/manifests/sglang-lmcache/README.md:17 — The stated validation command, kubectl apply --dry-run, is incomplete for current kubectl — document the actual --dry-run=client invocation so readers can reproduce it.
• docs/reference-stack/VERSIONS.md:38 — The same abbreviated kubectl apply --dry-run claim is not reproducible as written — use kubectl apply --dry-run=client.

Per-category coverage

1. Vendor-neutral naming: Clean
2. Contract & spec / docs sync: findings above at: docs/reference-stack/manifests/sglang-lmcache/README.md:14, docs/reference-stack/manifests/sglang-lmcache/deployment.yaml:17
3. gRPC fail-open semantics: Clean
4. Quality (tests / error wrapping / package placement / generated drift): findings above at: docs/reference-stack/manifests/sglang-lmcache/README.md:17, docs/reference-stack/manifests/sglang-lmcache/README.md:198, docs/reference-stack/manifests/sglang-lmcache/deployment.yaml:17, docs/reference-stack/VERSIONS.md:38

Verdict

changes-requested

[github-actions]

Codex review

Files reviewed

• docs/reference-stack/
  • [reviewed] VERSIONS.md
• docs/reference-stack/manifests/sglang-lmcache/
  • [reviewed] README.md
  • [reviewed] deployment.yaml

Findings

Blocking

None.

Should-fix

• docs/reference-stack/manifests/sglang-lmcache/deployment.yaml:144 — Add the mandatory install-smoke assertion for the new MP worker injection surface. — CONTRIBUTING.md requires operator-facing manifest changes to extend default_install_smoke.sh; without it, regressions in the documented native-sidecar wiring can merge undetected.
• docs/reference-stack/manifests/sglang-lmcache/README.md:173 — Verify and pin the actual publisher-startup log emitted by the documented SGLang image. — The speculative regex and instruction to “adjust the pattern” can fail a working deployment or pass an unrelated publisher message, so the reference’s stated success assertion is not reproducible.

Nit

None.

Per-category coverage

1. Vendor-neutral naming: Clean
2. Contract & spec / docs sync: Clean
3. gRPC fail-open semantics: Clean
4. Quality (tests / error wrapping / package placement / generated drift): findings above at: docs/reference-stack/manifests/sglang-lmcache/deployment.yaml:144, docs/reference-stack/manifests/sglang-lmcache/README.md:173

Verdict

changes-requested

[EdHasNoLife]

Dispositioning the two remaining Codex findings (PR is otherwise green/mergeable):

• Install-smoke assertion — deferred to CAC-251, not added here. This PR is docs-only: it changes a GPU-only reference manifest with non-applyable placeholder digests the smoke gate can't apply (the gate installs config/default + config/samples/, not docs/reference-stack/manifests/). The MP-worker injection surface itself landed in feat(sglang): LMCache MP-mode engine wire — the working data plane (phase 2, increment 2) #149; its managed-path reconcile smoke assertion is the tracked CAC-251 item. Adding it means editing the shared ~3,600-line per-PR install-smoke gate, which wants a real kind run to validate — deliberately not done blind.
• Pin the exact publisher-startup log line — can't be pinned without a GPU run of the specific SGLang image to observe the real phrasing; the "confirm against your build's first-run logs" caveat (pre-existing) is the honest handling.

CAC-251.