weight sync: add complete CPU-prepared delta updates on SGLang v0.5.16

[nanjiangwill]

Summary

• pin rollout images to SGLang v0.5.16 plus a clean, responsibility-separated weight-sync fork
• remove tensor-sparsity-dependent partial reloads in favor of complete, general delta target preparation
• expose explicit low-RAM disk and short-pause CPU delta modes; FULL publications and resets always use disk
• migrate Modal Volumes to v2, use the shared sglang-cache, and reserve CPU memory with (request, limit) tuples
• replace legacy reload probes with one reusable profiler per model and document the validated benchmark matrix

Design

Disk mode reconstructs and verifies a complete mutable checkpoint on host-local NVMe during /pull_weights; /update_weights_from_disk then performs a complete quantized reload.

CPU mode retains one canonical checkpoint-layout image per host plus one complete rank-local runtime image per TP rank. /pull_weights stages the compressed delta once, applies XOR/overwrite operations and verifies publisher checksums in canonical tensor space, then runs every target through SGLang's ordinary loader and quantization hooks while generation continues. /update_weights_from_cpu only copies the prepared rank images into stable GPU storages.

Checksums fail loudly before live GPU weights are touched. Every runtime storage is prepared and committed; the recorded 686 changed Kimi tensors describe this delta and are not an implementation sparsity assumption.

Post-load layout work is capability-gated per quantization method. GLM FP8 performs almost all layout work in host memory. ModelOpt NVFP4 retains bounded CUDA staging because its device transform is materially faster. MXFP4 has independent reload-safe layout handling.

SGLang fork

• image: lmsysorg/sglang:v0.5.16
• branch: modal-projects/sglang:stitch-sglang-v0.5.16
• pin: 47561f9f2d7ead3a74d54cf0175e5937fb81207c
• includes the upstream-style no-GDS fastsafetensors configuration and page-cache release from Support fastsafetensors no-GDS loading and page-cache release sgl-project/sglang#31859

The nine fork commits separate fastsafetensors transport, safe quantized disk reload, verified disk lineage materialization, CPU image preparation, async APIs, distributed hardening, checksum-only reseeding, MXFP4 reload safety, and host-vs-device postprocessing.

Validation

Local:

• uv run pytest -q — 54 passed
• uv run python -m compileall -q cookbook tools/profiling src
• git diff --check
• SGLang weight-sync/NVFP4/MXFP4 tests — 29 passed plus 21 distributed subtests
• SGLang pre-commit run --all-files — passed
• B300 SM 10.3 MXFP4 smoke — native and hybrid repeated reloads preserved every storage pointer and reproduced target runtime bytes exactly

GPU matrix, HiCache disabled, with fresh TP4 engines:

Model	Mode	Runtime	n	Pause range	Complete loop range
GLM-4.5-Air FP8	CPU	gVisor	2	1.002–1.005 s	34.8–36.4 s
GLM-4.5-Air FP8	CPU	runc	2	0.990–0.999 s	19.1–21.9 s
GLM-4.5-Air FP8	disk	gVisor	2	21.89–25.41 s	88.29–94.21 s
GLM-4.5-Air FP8	disk	runc	2	20.44 s	78.93–84.64 s
Kimi K2.6 NVFP4	CPU	gVisor	2	3.131–4.155 s	118.59–126.94 s
Kimi K2.6 NVFP4	CPU	runc	2	2.754–2.805 s	56.35–61.96 s
Kimi K2.6 NVFP4	disk	gVisor	3	138.95–482.49 s	171.01–534.77 s
Kimi K2.6 NVFP4	disk	runc	3	194.49–203.23 s	207.67–216.63 s

Every sample generated before, throughout preparation, and after commit; served the target version; changed from v0; and reproduced an exact deterministic token/text/logprob fingerprint after the update.

The CPU path meets the pause target: GLM commits in about one second and Kimi in 2.75–4.15 seconds. Kimi's complete CPU loop is 56–62 seconds with runc and 119–127 seconds with the default runtime. Disk reload remains dominated by rereading and transforming the complete checkpoint during the pause.