diff --git a/FEATURES.md b/FEATURES.md
index 6052a34a..876c06ab 100644
--- a/FEATURES.md
+++ b/FEATURES.md
@@ -4,17 +4,17 @@
 > Part of the [TensorSharp](README.md) documentation.
 
 
-- **Multi-architecture support** -- DeepSeek V4 Flash, GLM 5.x, Gemma 4, Gemma 3, DiffusionGemma, Qwen 3, Qwen 3.5/3.6-family, GPT OSS, Nemotron-H, Mistral 3, Muse-Glimmer, Qwen-Image-Edit (image editing), MiniMax-H3 (video with native 32 kHz stereo audio), and Wan 2.1/2.2 (video only)
-- **Multimodal inference** -- image, video, and audio inputs (Gemma 4); images for Gemma 3 / Qwen 3.5/3.6-family / Mistral 3 / Muse-Glimmer / Nemotron-H Omni. Audio input is Gemma 4 only. `--pdf` is architecture-agnostic: a born-digital PDF's text layer is inlined into the prompt for any model, and only scanned PDFs fall back to page images (which then need a vision model). Generated media is its own axis: Qwen-Image-Edit emits an image, Wan 2.1/2.2 emit an H.264 MP4, and MiniMax-H3 is the one family whose output is **audio as well as video** — a 32 kHz stereo track denoised jointly with the picture and written as a sidecar `.wav` beside the MP4
-- **Thinking / reasoning mode** -- structured chain-of-thought output with `<think>` / `<|channel>thought` / `<|channel>analysis` / `to=self` tags (Qwen 3, Qwen 3.5/3.6-family, Gemma 4, GPT OSS, Nemotron-H, Muse-Glimmer, DeepSeek V4, GLM 5.x)
+- **Multi-architecture support** -- DeepSeek V4 Flash, GLM 5.x (GLM-5.2 `glm-dsa` and GLM-5.3-Flash `glm5next`), Gemma 4, Gemma 3, DiffusionGemma, Qwen 3, Qwen 3.5/3.6-family, Qwen 3.8 Flash Next (`qwen4exp`), GPT OSS, Nemotron-H, Mistral 3, Muse-Glimmer, Qwen-Image-Edit (image editing), MiniMax-H3 (video with native 32 kHz stereo audio), and Wan 2.1/2.2 (video only)
+- **Multimodal inference** -- image, video, and audio inputs (Gemma 4); images for Gemma 3 / Qwen 3.5/3.6-family / Qwen 3.8 Flash Next / GLM-5.3-Flash / Mistral 3 / Muse-Glimmer / Nemotron-H Omni, each through its own `mmproj` tower. Audio input is Gemma 4 only. `--pdf` is architecture-agnostic: a born-digital PDF's text layer is inlined into the prompt for any model, and only scanned PDFs fall back to page images (which then need a vision model). Generated media is its own axis: Qwen-Image-Edit emits an image, Wan 2.1/2.2 emit an H.264 MP4, and MiniMax-H3 is the one family whose output is **audio as well as video** — a 32 kHz stereo track denoised jointly with the picture and written as a sidecar `.wav` beside the MP4
+- **Thinking / reasoning mode** -- structured chain-of-thought output with `<think>` / `<|channel>thought` / `<|channel>analysis` / `to=self` tags (Qwen 3, Qwen 3.5/3.6-family, Qwen 3.8 Flash Next, Gemma 4, GPT OSS, Nemotron-H, Muse-Glimmer, DeepSeek V4, GLM 5.x)
 - **Tool calling / function calling** -- models can invoke user-defined tools; multi-turn tool-call conversations supported across all three API styles
 - **Quantized model support** -- loads GGUF files with Q4_K_M, Q8_0, F16, MXFP4, and other quantization formats; performs native quantized matmul without dequantizing to FP32, including memory-efficient pure C# CPU loading for large GGUFs
 - **GPU-accelerated** -- GGML Metal on macOS, GGML CUDA on Windows/Linux with NVIDIA GPUs, GGML Vulkan on Windows/Linux with AMD/Intel/NVIDIA GPUs, a direct CUDA/cuBLAS backend with PTX kernels, and an MLX backend for Apple Silicon (mlx-c / Metal), all with CPU fallbacks for unsupported ops
 - **Optimized pure C# CPU backend** -- managed GEMM fast paths plus fused SIMD kernels for RMSNorm, RoPE, softmax, fused activations, and other inference hot paths
-- **Continuous batching & paged KV cache** -- vLLM-style block-paged KV pool with block-hash prefix sharing across requests, iteration-level scheduler that admits / preempts sequences mid-batch, optional SSD-backed tier for very large KV working sets, and a native fused paged-attention kernel (`TSGgml_PagedAttentionForward`) that drives `ggml_flash_attn_ext` on Metal/CUDA/Vulkan. Enabled by default in `TensorSharp.Server`; opt-out with `--no-continuous-batching`. See [docs/PAGED_ATTENTION_AND_CONTINUOUS_BATCHING.md](docs/PAGED_ATTENTION_AND_CONTINUOUS_BATCHING.md). GLM 5.x is the exception: MLA with weight absorption (one 576-wide cache row per token per layer) and the DSA lightning indexer have no paged layout, so concurrency is served by native per-sequence **slots** instead — each request owns its MLA and indexer caches and its own `n_past`, and binding a request switches the active slot without moving KV bytes.
+- **Continuous batching & paged KV cache** -- vLLM-style block-paged KV pool with block-hash prefix sharing across requests, iteration-level scheduler that admits / preempts sequences mid-batch, optional SSD-backed tier for very large KV working sets, and a native fused paged-attention kernel (`TSGgml_PagedAttentionForward`) that drives `ggml_flash_attn_ext` on Metal/CUDA/Vulkan. Enabled by default in `TensorSharp.Server`; opt-out with `--no-continuous-batching`. See [docs/PAGED_ATTENTION_AND_CONTINUOUS_BATCHING.md](docs/PAGED_ATTENTION_AND_CONTINUOUS_BATCHING.md). GLM 5.x is the exception: MLA with weight absorption (one 576-wide cache row per token per layer) and the DSA lightning indexer have no paged layout, so concurrency is served by native per-sequence **slots** instead — each request owns its MLA and indexer caches and its own `n_past`, and binding a request switches the active slot without moving KV bytes. Qwen 3.8 Flash Next (`qwen4exp`) is the same shape for the same reason — its GatedDeltaNet, PLE and QSA-indexer state have no paged layout either — and is served through per-sequence **state holders**: each in-flight request owns its attention KV and indexer caches, its GDN conv + delta-net state and its PLE history, the native kernel keys its device-resident recurrent state by the holder, so switching requests is a reference swap, and the engine round-robins sequences through their own captured fused decode graphs.
 - **Speculative decoding** -- a pluggable algorithm layer (`--spec-type`: `auto` / `draft-head` / `block` / `ngram`) over a shared draft-verify-rollback runtime; the weight-free `ngram` speculator works on every model, learned draft heads accelerate solo (non-concurrent) decode. Qwen 3.6 and GLM 5.2 ship their NextN block fused into the trunk GGUF; Gemma 4 loads a separate EAGLE-style `gemma4-assistant` draft GGUF via `--spec-draft-model` whose draft layers attend the target's own KV cache. The draft proposes up to `--spec-draft` tokens per step (kept while draft confidence ≥ `--spec-pmin`) and the trunk verifies them in a single batched forward; the request's own sampler — penalties included — drives both drafting and verification, so output is identical to standard decode. Opt in with `--spec` on either host (off by default). On `TensorSharp.Cli` it engages on every single-sequence path — `--input`, `--multi-turn-jsonl` and `--interactive`. On ggml backends fused multi-token-verify / draft-step kernels make it a clear win; the direct `cuda` backend runs a fully GPU-resident per-op verify/draft and is also a win. CPU / GGML CPU / MLX stay on standard decode. Env: `TS_SPEC_*` (shared; the legacy `TS_MTP_*` spellings still work) and `TS_GMTP_*` (Gemma 4 tuning).
 - **Batched / parallel inference** -- `IBatchedPagedModel.ForwardBatch` implementations for Mistral 3, Gemma 4, GPT OSS, Qwen 3, Qwen 3.5/3.6-family, and Nemotron-H all run by default and pack N sequences into a single forward pass with paged K/V scatter and per-sequence attention via the native kernel. Gemma 4, Qwen 3.5/3.6, GPT OSS, and Nemotron-H expose a per-family `TS_<FAMILY>_BATCHED=0` escape hatch (`TS_GEMMA4_BATCHED=0`, `TS_QWEN35_BATCHED=0`, `TS_GPTOSS_BATCHED=0`, `TS_NEMOTRON_BATCHED=0`) to fall back to the per-sequence KV-swap path for A/B comparison or regression isolation; Qwen 3 and Mistral 3 have no per-family switch — use the global `TS_SCHED_DISABLE_BATCHED=1`. GLM 5.x has no paged `ForwardBatch`; instead an opt-in batched fused decode (`TS_BATCHED_FUSED_DECODE=1`) runs one graph with one token per sequence so the weights are read once — 1.81x aggregate decode at 4 concurrent requests. It stays off by default because batching changes the GEMM shapes and a 2-bit MoE amplifies that into different expert picks.
-- **Tensor parallelism & distributed inference** -- split a model across multiple GPUs (Megatron-LM column/row-parallel pattern) with `--tp N` on both `TensorSharp.Cli` and `TensorSharp.Server` (or `TENSORSHARP_TP_DEGREE`), and extend across machines with peer-to-peer TCP clustering (`--tp-node-id` / `--tp-peers`). Hierarchical AllReduce minimizes inter-node traffic. Runs on the direct `cuda` backend and on the GGML CUDA / Vulkan backends, where each rank owns a ggml backend, weight shards, and KV cache on its own GPU. Supports all autoregressive architectures (Qwen 3, Mistral 3, Gemma 3/4, Qwen 3.5/3.6-family, GPT OSS, Nemotron-H, GLM 5.x — GGML backends only, Muse-Glimmer — `--tp 2` max there, 2 KV heads) with architecture-specific strategies for MoE expert parallelism / expert slicing, GatedDeltaNet per-rank V-head ownership, and Mamba2 replication. Fused per-rank graphs make `--tp 2` decode faster than a single GPU (Gemma 4 E4B 51.7 vs 37.3 tok/s) and run models that do not fit one card. Note that TP is not the only way a model reaches several GPUs: DeepSeek V4 and GLM 5.x **layer-split across every visible GPU by default, with no flag** (their whole-model executors bin-pack whole layers against each device's free VRAM), and `--tp` switches GLM 5.x from that to Megatron sharding within each layer, while on DeepSeek V4 it only caps how many GPUs the layer split uses (the same thing `TS_DSV4_NGPU` sets). Every other architecture uses a single GPU unless `--tp` is passed. Optional Redis-backed KV cache and Responses API store for shared state. → [Tensor Parallelism](USAGE.md#tensor-parallelism--distributed-inference)
+- **Tensor parallelism & distributed inference** -- split a model across multiple GPUs (Megatron-LM column/row-parallel pattern) with `--tp N` on both `TensorSharp.Cli` and `TensorSharp.Server` (or `TENSORSHARP_TP_DEGREE`), and extend across machines with peer-to-peer TCP clustering (`--tp-node-id` / `--tp-peers`). Hierarchical AllReduce minimizes inter-node traffic. Runs on the direct `cuda` backend and on the GGML CUDA / Vulkan backends, where each rank owns a ggml backend, weight shards, and KV cache on its own GPU. Supports all autoregressive architectures (Qwen 3, Mistral 3, Gemma 3/4, Qwen 3.5/3.6-family, GPT OSS, Nemotron-H, GLM 5.x — GGML backends only, and GLM-5.2 `glm-dsa` only, Muse-Glimmer — `--tp 2` max there, 2 KV heads) with architecture-specific strategies for MoE expert parallelism / expert slicing, GatedDeltaNet per-rank V-head ownership, and Mamba2 replication. Fused per-rank graphs make `--tp 2` decode faster than a single GPU (Gemma 4 E4B 51.7 vs 37.3 tok/s) and run models that do not fit one card. Note that TP is not the only way a model reaches several GPUs — there are two multi-GPU modes and they are not the same thing. **Tensor parallelism** shards the weights *inside* every layer and pays a collective per layer to reconverge, so it can buy latency as well as capacity. A **layer split** gives each GPU a contiguous run of *whole* layers: nothing is sharded, no collective is issued, and it is a **capacity** feature — it is how a model that does not fit one card runs at all, not a way to make it faster. DeepSeek V4 and GLM 5.x **layer-split across every visible GPU by default, with no flag** (their whole-model executors bin-pack whole layers against each device's free VRAM); `--tp` switches GLM-5.2 from that to Megatron sharding within each layer, GLM-5.3-Flash (`glm5next`) refuses `--tp` cleanly and stays on the layer split, and on DeepSeek V4 `--tp` only caps how many GPUs the layer split uses (the same thing `TS_DSV4_NGPU` sets). On Qwen 3.8 Flash Next (`qwen4exp`), `--tp N` *is* a layer split — the architecture shards no weights, and this is the same (and only) multi-GPU mode llama.cpp offers it, since `-sm row` refuses to load it. Measured on 2x A100-80GB with Qwen3.8-Flash-Next-UD-Q2_K_XL (73.4 GiB): greedy output **byte-identical** between the 1-GPU and 2-GPU runs (same SHA-256), VRAM 24.2 GB + 26.2 GB instead of all of it on one card, and throughput unchanged (prefill ~1520-1550 t/s, decode ~56 t/s either way). Startup prints which mode ran and the per-GPU layer/byte split. Every other architecture uses a single GPU unless `--tp` is passed, and one that supports neither tensor parallelism nor a layer split now says so on stderr and runs on one GPU instead of silently leaving the extra GPUs idle. Optional Redis-backed KV cache and Responses API store for shared state. → [Tensor Parallelism](USAGE.md#tensor-parallelism--distributed-inference)
 - **Ollama & OpenAI API compatibility** -- drop-in replacement endpoints for existing tooling
 - **Configurable sampling** -- temperature, top-k, top-p, min-p, repetition/presence/frequency penalties, seed, stop sequences
 - **Structured outputs** -- the OpenAI `response_format` JSON schema is compiled to a grammar and enforced by grammar-constrained decoding: any token that would break the schema is removed from the distribution before sampling, so the response is structurally valid by construction rather than repaired afterwards. Supported: `type`, `enum`, `const`, `properties`, `required`, `additionalProperties`, `items`, `prefixItems`, `min/maxItems`, `anyOf`, `oneOf`, `allOf`, `$ref`/`$defs` (recursive included), `min/maxLength`, `pattern`, the date/time/date-time/uuid formats, and integer `minimum`/`maximum`. Keywords a CFG cannot express (`not`, `if`/`then`/`else`, `dependentSchemas`, `dependentRequired`, `multipleOf`, `patternProperties`) are refused up front. `TS_JSON_GRAMMAR=0` falls back to prompt-and-repair.
@@ -28,7 +28,7 @@
 - **Video generation, video-only (Wan 2.1 text-to-video, Wan 2.2 text/image-to-video)** -- a prompt (plus an optional first-frame image on the Wan 2.2 models) produces an H.264 MP4 with no audio track. The loaded `wan` GGUF is the Wan DiT — Wan 2.1 T2V, Wan 2.2 TI2V-5B (48-channel 16×16×4 latent, 24 fps) and Wan 2.2 A14B (two 14B experts switched at a timestep boundary, second GGUF auto-resolved) are auto-detected; TensorSharp resolves the companions alongside it — the UMT5-XXL text encoder GGUF (prompt → 512×4096 conditioning, exact unigram-Viterbi SentencePiece tokenization) and the matching causal 3D video VAE (`wan_2.1_vae.safetensors` / `Wan2.2_VAE.safetensors`). The FlowMatch CFG denoise (UniPC or Euler) runs the whole DiT (self-attention with 3D RoPE + flash attention over F16 keys/values, cross-attention, AdaLN time modulation — per-token-timestep for TI2V image-to-video) as ONE resident-weight ggml graph per step, CUDA-graph-captured per shape (`TSGgml_WanDitForward`); the video VAE decodes all temporal chunks in one graph with the causal feature cache carried in-graph (`TSGgml_WanVaeDecode`) -- convs go through MPSGraph on Metal (a 736x544x81f decode: 159 s -> 80 s, 1.99x, numerics unchanged at 93.9 dB PSNR; `TS_WAN_VAE_MPS_CONV=0` restores ggml's im2col+GEMM lowering) and through a banded im2col+GEMM path elsewhere, with the im2col budget and the tiling threshold now sized from free device memory instead of a fixed 16 GB card's budget, so large-memory devices decode a 720p plane whole (565 s vs 655 s banded, peak RSS 4.85 vs 5.37 GB) while small cards still tile; and image-to-video conditioning encodes the first frame through the causal VAE encoder in one graph (`TSGgml_WanVaeEncode`). Each stage releases its VRAM before the next, so TI2V-5B 81-frame 480p image-to-video and both A14B Q4_K_M experts fit a 16 GB GPU. Generation runs on every backend except MLX: the GGML paths (`ggml_cuda`, `ggml_metal`, `ggml_vulkan`, `ggml_cpu`) share the whole-graph kernels, while `--backend cuda` and `--backend cpu` run a ggml-independent direct implementation (`WanDirect*`: resident-quantized linears on TensorSharp's MMQ/dp4a/cuBLAS routing with streaming online-softmax attention kernels on CUDA, parallel SIMD GEMM/attention on CPU, and a channels-last banded-im2col causal video VAE shared by both). **Step-distilled checkpoints are auto-detected from the DiT file name** (`Turbo`, `distill`, `Lightning`, `lightx2v`, `FastWan`, `-dmd`, or an explicit `…-4steps-…`) and are by far the biggest speed lever: the official 50-step x CFG recipe costs 100 DiT passes, a 4-step distilled checkpoint costs 4, and the pipeline switches to that step count with guidance off automatically (`--diffusion-steps` / `--cfg` override). Measured on an M5 Pro at 1088x832x121f = 27 404 tokens, `ggml_metal`, Wan2.2-TI2V-5B Q8_0: the base checkpoint runs 100 passes at 120.2 s for ~3 h 30 m end to end, and the identical request on a Turbo checkpoint runs 4 passes for **17 m 30 s** -- only the `--model` path differs. On base checkpoints `--cfg-cache-stride 2` / `3` reuses the guidance direction between steps for a further 1.30x / 1.43x. Numerics verified against diffusers (DiT cosine > 0.995, VAE encoders > 0.999, decoders 59.9 dB / >35 dB PSNR) and across backends (final-latent cosine ≥ 0.999 on identical seeds); the F16 attention keys/values that make one 27 k-token self-attention 2.02x faster (~1.7x per DiT pass together with the VAE work) score the same 0.999964 DiT cosine as F32. Driven from C# via `WanVideoModel.GenerateVideo(prompt, WanVideoParams)`, the CLI (`--prompt`, `--image`, `--video-frames`, `--fps`, `--flow-shift`, `--negative-prompt`), the server API (`/v1/videos/generations` with base64 `image`, `/api/video-generate[/stream]` with `imagePath`), and the Web UI chat (type a prompt — with an attached image for image-to-video — and get the video with live progress: per-pass timings, a running ETA and a 30 s heartbeat, since one pass over a 5-second 720p latent is minutes of GPU work). → [Wan card](docs/models/wan.md)
 - **Hybrid SSM-Transformer** -- Nemotron-H mixes Mamba2 SSM layers, attention-only layers, and MoE FFN layers in a single model. The Mamba2 step has both a per-sequence native kernel and a batched native kernel (`TSGgml_NemotronMamba2BatchedStepF32`, NEON SIMD + GCD parallelism) used by the batched path. On GGML backends the attention layers decode through the device-side flash-attention kernel against the resident KV cache (`TS_NEMOTRON_FLASH_DECODE=0` restores the host path), so decode no longer degrades with context length.
 - **Hybrid Attention-Recurrent** -- Qwen 3.5/3.6-family models mix full-attention layers with GatedDeltaNet recurrent layers; the batched path keeps recurrent running state in a per-slot recurrent-state pool
-- **Mixture of Experts** -- Gemma 4 MoE variants (e.g. gemma-4-26B-A4B), GPT OSS MoE (e.g. gpt-oss-20b), Qwen 3.5/3.6-family MoE (`qwen35moe` / `qwen3next` variants such as Qwen3.5-35B-A3B), Nemotron-H MoE FFN layers, and GLM 5.2 (744B-A40B: 256 routed experts at top-8 plus one shared expert, sigmoid-gated routing with a selection-only bias and a x2.5 routed scale, after 3 leading dense SwiGLU layers)
+- **Mixture of Experts** -- Gemma 4 MoE variants (e.g. gemma-4-26B-A4B), GPT OSS MoE (e.g. gpt-oss-20b), Qwen 3.5/3.6-family MoE (`qwen35moe` / `qwen3next` variants such as Qwen3.5-35B-A3B), Nemotron-H MoE FFN layers, and GLM 5.2 (744B-A40B: 256 routed experts at top-8 plus one shared expert, sigmoid-gated routing with a selection-only bias and a x2.5 routed scale, after 3 leading dense SwiGLU layers), GLM-5.3-Flash (320B: 288 routed experts at top-8 plus one shared expert and the same x2.5 routed scale, with a SwiGLU clamp limit of 10 on every FFN), and Qwen 3.8 Flash Next (512 experts, 10 used per token, interleaved with GatedDeltaNet recurrent layers)
 - **MoE CPU offload** -- `--n-cpu-moe N` / `--cpu-moe` (llama.cpp's `-ncmoe` / `-cmoe` equivalent) keeps the routed expert weights of the first N layers in system RAM and multiplies them on the host, leaving attention, the norms, the router and the always-active shared expert on the accelerator. The offloaded layers stay inside the fused whole-model graph on every architecture that has one (Qwen 3.5/3.6, Gemma 4 MoE, GPT OSS, DiffusionGemma) — the accelerator pauses after each offloaded layer's router, the host multiplies the selected experts straight out of the GGUF mmap, and the result is handed back before the next segment — so only ~8 KB of activation crosses the bus per layer at decode. Gemma 4 MoE and Qwen 3.5/3.6 segment their prefill graphs the same way, where the host side becomes a real GEMM over the whole prompt chunk. It also composes with tensor parallelism: under `--tp N` the seams merge into the ranks' own AllReduce segment schedule, so the fused multi-rank graph is kept and each offloaded layer is evaluated once on the host over the unsharded expert stack (Qwen3.5-35B-A3B `--tp 2`: 17.4 GB of resident weights across 2 GPUs falls to 3.2 GB; gemma-4-26B-A4B: 12.9 GB falls to 2.4 GB, byte-identical output). Measured on a 16 GB RTX 3080 Laptop: Qwen3.6-35B-A3B 13.4 -> 4.6 GB at `--cpu-moe`; gemma-4-26B-A4B 16.1 -> 4.8 GB (decode 39.7 -> 17.7 tok/s, and 38.6 tok/s at `--n-cpu-moe 8` for 3 GB back); gpt-oss-20b 16.2 -> 2.9 GB, which takes it off the WDDM spill cliff and turns 0.3 tok/s into 25.4 at `--n-cpu-moe 12`. That is what makes these models fit beside a long-context KV cache on a 12-16 GB GPU. DeepSeek V4 Flash uses the same seam on the GPU backends: it is 91% routed-expert bytes, and the loader sizes its layer split against each device's *actual* free VRAM. Offload stays opt-in there too -- a checkpoint that does not fit is refused at load with the exact `--n-cpu-moe N` that would make it fit, rather than silently trading away decode throughput -- and with that flag 3x48 GB RTX A6000 hosts the UD-Q8_K_XL checkpoint at 10 tok/s decode / 126 tok/s prefill. GLM 5.x offloads the same way (92% of that checkpoint is routed-expert bytes) and its host-resident experts are served straight from the GGUF mapping rather than copied. Note that offload is for *fitting*, not speed: on 3x RTX PRO 6000 where GLM-5.2 already fits, `--n-cpu-moe 30` costs pp2048 915.9 -> 94.7 and tg64 43.9 -> 16.4 tok/s. On GLM 5.2 the host-resident experts are multiplied straight out of the GGUF mapping with no private copy, and offload composes with `--tp N` — host-resident layers keep their experts whole and rank 0 evaluates them. It also nearly doubles the context the loader can size there, 342,272 -> 646,400 tokens. -> [MoE CPU offload](USAGE.md#mixture-of-experts-cpu-offload---n-cpu-moe)
 - **Batched GPU MoE** -- a single fused GGML graph dispatch handles all selected experts (plus the optional shared expert and residual add) for Qwen 3.5/3.6-family and Nemotron-H decode, eliminating per-expert round-trips
 - **Whole-model fused decode graphs** -- Gemma 4 (dense and MoE), Qwen 3.5/3.6 and GPT OSS run an entire decode token — every layer, the MoE router and experts, the final norm and the LM head — as ONE GGML graph dispatch instead of one submission per layer, so the GPU is never left waiting on the host between layers. On CUDA/Vulkan the graph is built once with stable tensor addresses and replayed (`ggml_set_rows` KV write with the row as an I64 input, a stride-padded attention window with an F16 mask input), which is what lets ggml-cuda capture it as a CUDA graph. GPT OSS decode: 24 → 154 tok/s on an A40, and flat in context length (133 tok/s at 16K) where the per-layer path collapsed to 2.3. Disable per model with `TS_GPTOSS_MODEL_DECODE=0` / `TS_GEMMA4_FD_PERSIST=0` / `TS_QWEN35_FD_PERSIST=0`.
@@ -41,7 +41,7 @@
 
 ## Thinking / Reasoning Mode
 
-Models that support thinking mode (Qwen 3, Qwen 3.5/3.6-family, Gemma 4, GPT OSS, Nemotron-H, Muse-Glimmer, DeepSeek V4, GLM 5.x) can produce structured chain-of-thought reasoning before generating the final answer. The thinking content is separated from the main response and can be displayed or hidden by the client.
+Models that support thinking mode (Qwen 3, Qwen 3.5/3.6-family, Qwen 3.8 Flash Next, Gemma 4, GPT OSS, Nemotron-H, Muse-Glimmer, DeepSeek V4, GLM 5.x) can produce structured chain-of-thought reasoning before generating the final answer. The thinking content is separated from the main response and can be displayed or hidden by the client.
 
 - **Qwen 3 / Qwen 3.5/3.6-family / Nemotron-H:** uses `<think>...</think>` tags
 - **Gemma 4:** uses `<|channel>thought\n...<channel|>` tags
@@ -55,11 +55,13 @@ Enable via `--think` (console), `"think": true` (Ollama API), or the thinking to
 
 DeepSeek V4 ships **DSpark** ("Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation") as a support module in the checkpoint: three DSV4 blocks that read the trunk's hidden states and propose a whole BLOCK of tokens per step instead of one, a Markov head that conditions each block position on the token before it, and a confidence head that predicts each position's acceptance probability. TensorSharp loads it as a separate drafter GGUF (`--draft-model`, built with `eng/dsv4-dspark-to-gguf.py`) and runs it on both GPU engines (`--backend cuda` and `--backend ggml_cuda`) for greedy single-sequence generation — on ggml the drafter is three extra graph layers whose key ring the trunk graph commits itself, so speculation costs no host round trips; the trunk verifies each block in one batched forward and keeps only the prefix it would have produced anyway. Measured **1.3-1.4x decode** on 4xA40 with the drafter's cumulative-confidence gate at its default; the gate matters because each extra verify row pulls a fresh set of MoE experts through VRAM. See the [DeepSeek V4 card](docs/models/deepseek4.md#dspark-speculative-decoding).
 
-## DFlash Block Speculative Decoding (Muse-Glimmer)
+## DFlash / DFlash2 Block Speculative Decoding (Muse-Glimmer, Qwen 3.8)
 
-Muse-Glimmer has its own block drafter, **DFlash**: a separate 5-layer GGUF (`general.architecture = dflash`) that proposes the whole speculative window in one forward. It borrows the target's token embedding and LM head, keeps its own sliding-window KV ring, and runs three passes per step — *encode* the trunk's per-layer input residuals at `dflash.target_layers` into one wide row, *inject* that row as the K/V of every draft layer, then *draft* `[anchor, MASK x (block-1)]` through the five blocks and score it with the target's LM head. The trunk verifies the block in one batched forward and keeps only the prefix it would have produced anyway, so **the emitted token stream is the plain-greedy stream**.
+Muse-Glimmer and Qwen 3.8 both have a block drafter, **DFlash**: a separate 5-layer GGUF (`general.architecture = dflash`) that proposes the whole speculative window in one forward. It is architecture-agnostic on the TensorSharp side - a target model gets it by tapping the per-layer residuals the drafter's encoder reads, and nothing else. It borrows the target's token embedding and LM head, keeps its own sliding-window KV ring, and runs three passes per step — *encode* the trunk's per-layer input residuals at `dflash.target_layers` into one wide row, *inject* that row as the K/V of every draft layer, then *draft* `[anchor, MASK x (block-1)]` through the five blocks and score it with the target's LM head. The trunk verifies the block in one batched forward and keeps only the prefix it would have produced anyway, so **the emitted token stream is the plain-greedy stream**.
 
-Both halves are fused native graphs that CUDA-graph-capture and replay, and the draft block finishes with an on-device `argmax`, so the 202048-wide probability block never crosses PCIe. A runtime cost governor measures speculation against plain decoding and parks the drafter while it is measurably slower — speculation can therefore only help, but it needs a few hundred generated tokens to settle. Load it with `--draft-model` (CLI) or `TS_MUSE_GLIMMER_DFLASH`. Sampling composes — verification draws each token from a trunk row with the run's own sampler — but note that a block drafter proposes its whole block in one pass, so penalties are not applied to the proposal and acceptance falls as a penalized history grows. Measured against llama.cpp's own DFlash on one RTX PRO 6000 Blackwell (Q8_0, greedy, 60-token prompt): 50.9 tok/s vs llama.cpp's 45.5 and 35.0 plain. See the [Muse-Glimmer card](docs/models/muse-glimmer.md#3-dflash-speculative-decoding).
+**DFlash2** is the same backbone plus two additions, both keyed off the GGUF so one code path serves either generation: a grouped dynamic depthwise convolution around every attention and every FFN sublayer (which gives a block-diffusion draft a local left-to-right signal without a second forward), and a candidate selector that scores the top-K candidates of adjacent positions pairwise through two low-rank `[vocab, r]` codebooks and reads the block off as a walk through that lattice - so position *i+1* is no longer chosen without knowing what *i* chose. Attach either with `--draft-model`; the file says which it is. See [speculative_decoding.md](docs/speculative_decoding.md#dflash-and-dflash2).
+
+Both halves are fused native graphs that CUDA-graph-capture and replay, and the draft block finishes with an on-device `argmax` (or, for DFlash2, a ~7 KB lattice), so the 202048-wide probability block never crosses PCIe. A runtime cost governor measures speculation against plain decoding and parks the drafter while it is measurably slower — speculation can therefore only help, but it needs a few hundred generated tokens to settle. Load it with `--draft-model` (CLI) or `TS_MUSE_GLIMMER_DFLASH`. Sampling composes — verification draws each token from a trunk row with the run's own sampler — but note that a block drafter proposes its whole block in one pass, so penalties are not applied to the proposal and acceptance falls as a penalized history grows. Measured against llama.cpp's own DFlash on one RTX PRO 6000 Blackwell (Q8_0, greedy, 60-token prompt): 50.9 tok/s vs llama.cpp's 45.5 and 35.0 plain. See the [Muse-Glimmer card](docs/models/muse-glimmer.md#3-dflash-speculative-decoding).
 
 ## Speculative Decoding
 
@@ -73,7 +75,7 @@ Four algorithms ship today, selected with `--spec-type`:
 |---|---|---|
 | `auto` *(default)* | whatever drafter the checkpoint carries | — |
 | `draft-head` | one token per pass through a NextN/MTP head, chaining its own hidden state (Qwen 3.6, GLM 5.2, Gemma 4's separate assistant GGUF) | yes, per target model |
-| `block` | a whole block per pass with a confidence head (DeepSeek V4 DSpark, Muse-Glimmer DFlash) | yes, per target model |
+| `block` | a whole block per pass with a confidence head (DeepSeek V4 DSpark, DFlash / DFlash2 on Muse-Glimmer and Qwen 3.8) | yes, per target model |
 | `ngram` | suffix match over the sequence's own tokens — where did these last few tokens occur before, and what followed? | **no** |
 
 `ngram` is the model-agnostic one: it works on **every** checkpoint, including those that ship no speculator at all, and is strong wherever the answer quotes its input — summarizing, editing, translating or answering about a document, repetitive structured output, code with repeated identifiers, agentic tool loops. Measured on Qwen3.5-9B (Q8_0, ggml_metal, M5 Pro), which ships no draft head: **45.2 tok/s vs 31.4 plain (1.44x)** on a reproduce-this-config prompt, with byte-identical output. On free-form prose it finds nothing, every step degrades to a plain decode, and the runtime cost governor keeps that cheap.
@@ -118,6 +120,27 @@ computation, and row-parallel projections (output, down) followed by an AllReduc
 that reconverges the hidden state. Norms, embeddings, and the LM head are
 replicated.
 
+**Two multi-GPU modes, and they are not the same thing.** Tensor parallelism
+shards the weights *inside* each layer and pays a collective (one or two
+AllReduces per layer) to reconverge, so every rank does part of every layer's
+work and the mode can buy latency as well as capacity. A **layer split**
+instead gives each GPU a contiguous run of *whole* layers: nothing is sharded,
+no collective is issued, and the GPUs take a token in turn rather than working
+on it together. A layer split is therefore a **capacity** feature — it is how a
+model that does not fit one card runs at all — and should not be expected to
+raise throughput. Which mode each architecture uses:
+
+| Architecture | Multi-GPU mode |
+|---|---|
+| Qwen 3, Mistral 3, Gemma 3/4, Qwen 3.5/3.6-family, GPT OSS, Nemotron-H, Muse-Glimmer (`--tp 2` max) | Tensor parallelism, opt in with `--tp N` |
+| GLM-5.2 (`glm-dsa`) | Layer split across every visible GPU by default; `--tp N` switches it to tensor parallelism |
+| GLM-5.3-Flash (`glm5next`) | Layer split across every visible GPU by default; `--tp` is refused cleanly |
+| DeepSeek V4 Flash | Layer split across every visible GPU by default; `--tp N` only caps how many GPUs the split uses (as `TS_DSV4_NGPU` does) |
+| Qwen 3.8 Flash Next (`qwen4exp`) | Layer split, opt in with `--tp N` (GGML CUDA / Vulkan) |
+
+Startup prints which mode actually ran, so it never has to be inferred from
+`nvidia-smi`.
+
 **Local TP** runs within a single process. On the direct `cuda` backend one
 thread issues commands to all GPUs and CUDA streams provide the parallelism; on
 the GGML backends a rank worker pool drives the GPUs concurrently, because a
@@ -143,7 +166,7 @@ Architecture-specific strategies handle heterogeneous layers:
 | MoE on GGML (Gemma 4) | Megatron split *inside* each expert (gate/up column-parallel, down row-parallel) so the fused whole-model MoE trunk kernel keeps working with global expert ids; the expert sum becomes a third row-parallel AllReduce per layer. `TS_GEMMA4_TP_FUSED_MOE=0` falls back to the whole-expert per-op path |
 | GatedDeltaNet SSM (Qwen 3.5/3.6) | Block-cyclic V-head assignment — each rank runs its own packed GDN kernel on its V-head subset with independent delta/conv state, resident on its GPU; no cross-rank communication for the recurrent path |
 | Mamba2 SSM (Nemotron-H) | Replicated on rank 0, result broadcast to all ranks |
-| MLA + sparse-attention MoE on GGML (GLM 5.x) | Attention heads column-parallel (`attn_q_b` / `attn_k_b` / `attn_v_b`) with row-parallel `attn_output`; the 256 routed experts are Megatron-split *inside every expert* (gate/up column-parallel, down row-parallel) rather than partitioned by expert id, because `ggml_mul_mat_id` requires a token's selected expert ids to stay distinct. Router, norms, lightning indexer, shared expert and the 3 dense layers are replicated — two AllReduces per layer. `TS_GLM_TP_SHARD` picks which halves are split (1 heads, 2 experts, 3 both); `TS_GLM_TP_OVERSUBSCRIBE=1` packs several ranks onto one GPU for testing |
+| MLA + sparse-attention MoE on GGML (GLM-5.2 `glm-dsa`; GLM-5.3-Flash `glm5next` refuses `--tp` and stays on the layer split) | Attention heads column-parallel (`attn_q_b` / `attn_k_b` / `attn_v_b`) with row-parallel `attn_output`; the 256 routed experts are Megatron-split *inside every expert* (gate/up column-parallel, down row-parallel) rather than partitioned by expert id, because `ggml_mul_mat_id` requires a token's selected expert ids to stay distinct. Router, norms, lightning indexer, shared expert and the 3 dense layers are replicated — two AllReduces per layer. `TS_GLM_TP_SHARD` picks which halves are split (1 heads, 2 experts, 3 both); `TS_GLM_TP_OVERSUBSCRIBE=1` packs several ranks onto one GPU for testing |
 
 TP runs on the `cuda` backend and on the GGML CUDA / Vulkan backends
 (`ggml_cuda`, `ggml_vulkan`); MLX is single-device. On the GGML backends each
@@ -183,6 +206,35 @@ it changes the reduction order, a 2-bit MoE reproduces 3 of the 6 recorded
 llama.cpp goldens where the layer split reproduces 5 of 6. (Against llama.cpp
 running on the same backend, the layer split is 6/6.)
 
+**Layer split on Qwen 3.8 Flash Next.** `--tp N` on `qwen4exp` runs a layer
+split, not tensor parallelism: none of
+its weights are sharded, its decode is one persisted single-device GGML graph
+per token, and its GDN/PLE recurrent state lives in device buffers owned by a
+single backend. It is also the same (and only) multi-GPU mode llama.cpp offers
+this architecture — `-sm row` refuses to load it.
+
+Measured on 2x A100-80GB, Qwen3.8-Flash-Next-UD-Q2_K_XL (73.4 GiB):
+
+- greedy output is **byte-identical** between the 1-GPU and the 2-GPU run
+  (same SHA-256);
+- VRAM 24.2 GB + 26.2 GB — roughly half the model on each card instead of all
+  of it on one;
+- throughput unchanged: prefill ~1520-1550 t/s and decode ~56 t/s either way.
+  For reference, llama.cpp on the same box: 1 GPU pp1536 1094 / tg128 61.2;
+  2 GPUs `-sm layer` 1200 / 61.5 — so llama.cpp also gains ~10% prefill and
+  ~0 decode from the second card.
+
+`TS_Q4E_LAYER_SPLIT=20,28` overrides the automatic balance with explicit layer
+counts per GPU (llama.cpp's `--tensor-split` in spirit) and throws rather than
+silently ignoring a value it cannot honour — useful because the automatic
+balance prices weights and cannot see the vision tower, which loads later and
+lands on GPU 0. Details: [Qwen 3.8 Flash Next card](docs/models/qwen38-flash-next.md).
+
+Architectures that support neither tensor parallelism nor a layer split now say
+so on stderr and run on one GPU, instead of accepting `--tp N`, printing a
+tensor-parallelism banner and leaving the extra GPUs holding a CUDA context and
+NCCL buffers while idle.
+
 Batched/continuous-batching forward under TP is implemented for Qwen 3
 and Mistral 3; MoE models fall back to per-sequence forward under TP.
 
@@ -238,6 +290,30 @@ Gemma 3 supports PNG, JPEG, and HEIC/HEIF image inputs. The non-gated example ab
 
 All Qwen 3.5/3.6-family variants (`qwen35`, `qwen35moe`, and `qwen3next`) load through the same `Qwen35Model` implementation. Image inputs are supported via the dynamic-resolution `Qwen35VisionEncoder`; pass the selected repository's projector explicitly (for the 9B and Qwen 3.6 examples, `mmproj-F16.gguf`). The MoE variants (e.g. Qwen3.5-35B-A3B and Qwen3.6-35B-A3B GGUFs that report the same architecture keys) additionally enable a fused `MoEExpertsSwiGLUResidual` GGML kernel during decode that runs all selected experts, the optional shared expert, and the residual add in a single GPU graph dispatch.
 
+### Qwen 3.8 Flash Next
+
+Qwen3.8-Flash-Next (`qwen4exp`) supports image input through the Qwen3.5-VL
+vision tower with (T, H, W) IMRoPE positions; put the repository's
+`mmproj-BF16.gguf` beside the model to enable it. Multi-image prompts and
+multi-turn image sessions both work, with KV reuse across turns — extend-only,
+because the GatedDeltaNet recurrence cannot rewind, so a cached prefix is
+reused only when the new prompt extends it exactly.
+
+- **Images:** PNG, JPEG, HEIC/HEIF
+
+### GLM-5.3-Flash
+
+GLM-5.3-Flash (`glm5next`) supports image input through the GLM-OCR ViT in the
+repository's `mmproj-BF16.gguf` — RMS norms, fused QKV, per-head q/k RMS norms,
+2D vision RoPE, a SwiGLU-clamp MLP and a 2x2 conv merger, with all 24 blocks
+running as one device-resident GGML graph. The projected embeddings override
+the `<|image|>` placeholder rows inside the native executor; the text tower is
+NoPE, so image tokens need no MRoPE bookkeeping. `--image`, multi-image prompts
+and multi-turn image sessions are all supported. GLM-5.2 (`glm-dsa`) is
+text-only.
+
+- **Images:** PNG, JPEG, HEIC/HEIF
+
 ### Mistral 3
 
 Mistral 3 supports image inputs via the Pixtral vision encoder. The example repository uses `mmproj-mistralai_Mistral-Small-3.1-24B-Instruct-2503-f16.gguf`; pass it explicitly with `--mmproj`.
diff --git a/FEATURES_zh-cn.md b/FEATURES_zh-cn.md
index a07c53e2..5a74b79c 100644
--- a/FEATURES_zh-cn.md
+++ b/FEATURES_zh-cn.md
@@ -4,16 +4,16 @@
 > [TensorSharp](README_zh-cn.md) 文档的一部分。
 
 
-- **多架构支持** —— DeepSeek V4 Flash、GLM 5.x、Gemma 4、Gemma 3、DiffusionGemma、Qwen 3、Qwen 3.5/3.6-family、GPT OSS、Nemotron-H、Mistral 3、Muse-Glimmer、Qwen-Image-Edit（图像编辑）、MiniMax-H3（视频 + 原生 32 kHz 立体声音频），以及 Wan 2.1/2.2（仅视频）
-- **多模态推理** —— 图像、视频和音频输入（Gemma 4）；图像输入（Gemma 3 / Qwen 3.5/3.6-family / Mistral 3 / Muse-Glimmer / Nemotron-H Omni）。音频输入仅 Gemma 4 支持。`--pdf` 与架构无关：原生数字 PDF 的文本层会被内联进任意模型的提示词，只有扫描件才回退为页面图像（此时需要视觉模型）。生成的媒体是另一条轴：Qwen-Image-Edit 输出图像，Wan 2.1/2.2 输出 H.264 MP4，而 MiniMax-H3 是唯一**连音频一起输出**的家族——32 kHz 立体声音轨与画面联合去噪，并作为旁挂 `.wav` 写在 MP4 旁边
-- **思维链 / 推理模式** —— 通过 `<think>` / `<|channel>thought` / `<|channel>analysis` 标签输出结构化的思维链推理（Qwen 3、Qwen 3.5/3.6-family、Gemma 4、GPT OSS、Nemotron-H、Muse-Glimmer、DeepSeek V4、GLM 5.x）
+- **多架构支持** —— DeepSeek V4 Flash、GLM 5.x（GLM-5.2 `glm-dsa` 与 GLM-5.3-Flash `glm5next`）、Gemma 4、Gemma 3、DiffusionGemma、Qwen 3、Qwen 3.5/3.6-family、Qwen 3.8 Flash Next（`qwen4exp`）、GPT OSS、Nemotron-H、Mistral 3、Muse-Glimmer、Qwen-Image-Edit（图像编辑）、MiniMax-H3（视频 + 原生 32 kHz 立体声音频），以及 Wan 2.1/2.2（仅视频）
+- **多模态推理** —— 图像、视频和音频输入（Gemma 4）；图像输入（Gemma 3 / Qwen 3.5/3.6-family / Qwen 3.8 Flash Next / GLM-5.3-Flash / Mistral 3 / Muse-Glimmer / Nemotron-H Omni，各自通过自己的 `mmproj` 视觉塔）。音频输入仅 Gemma 4 支持。`--pdf` 与架构无关：原生数字 PDF 的文本层会被内联进任意模型的提示词，只有扫描件才回退为页面图像（此时需要视觉模型）。生成的媒体是另一条轴：Qwen-Image-Edit 输出图像，Wan 2.1/2.2 输出 H.264 MP4，而 MiniMax-H3 是唯一**连音频一起输出**的家族——32 kHz 立体声音轨与画面联合去噪，并作为旁挂 `.wav` 写在 MP4 旁边
+- **思维链 / 推理模式** —— 通过 `<think>` / `<|channel>thought` / `<|channel>analysis` 标签输出结构化的思维链推理（Qwen 3、Qwen 3.5/3.6-family、Qwen 3.8 Flash Next、Gemma 4、GPT OSS、Nemotron-H、Muse-Glimmer、DeepSeek V4、GLM 5.x）
 - **工具调用 / 函数调用** —— 模型可调用用户定义的工具；所有三种 API 风格均支持多轮工具调用对话
 - **量化模型支持** —— 加载 Q4_K_M、Q8_0、F16、MXFP4 等量化格式的 GGUF 文件；执行原生量化矩阵乘法（matmul），无需反量化到 FP32，并且纯 C# CPU 后端在加载大型 GGUF 时也会保持量化权重压缩状态
 - **GPU 加速** —— 通过 GGML 支持 Apple Metal（macOS）、GGML CUDA（Windows/Linux + NVIDIA）和 GGML Vulkan（Windows/Linux + AMD/Intel/NVIDIA），并提供 Direct CUDA/cuBLAS 后端（含 PTX 内核与未覆盖算子的 CPU 回退），以及面向 Apple Silicon 的 MLX 后端（mlx-c / Metal）
 - **优化后的纯 C# CPU 后端** —— 为 GEMM、RMSNorm、RoPE、softmax、融合激活等推理热点路径提供托管快速路径和 SIMD 内核
-- **连续批处理 & 分页 KV 缓存** —— vLLM 风格的分页 KV 块池，跨请求的块级哈希前缀共享，迭代级调度器（可在批内动态加入/抢占序列），可选的 SSD 冷层用于超大 KV 工作集，原生融合分页注意力内核（`TSGgml_PagedAttentionForward`，在 Metal/CUDA/Vulkan 上驱动 `ggml_flash_attn_ext`）。`TensorSharp.Server` 默认启用，可用 `--no-continuous-batching` 关闭。详见 [docs/PAGED_ATTENTION_AND_CONTINUOUS_BATCHING_zh-cn.md](docs/PAGED_ATTENTION_AND_CONTINUOUS_BATCHING_zh-cn.md)。GLM 5.x 是个例外：带权重吸收的 MLA（每层每 token 只占一行 576 宽的缓存）与 DSA lightning indexer 没有分页布局，因此那里的并发靠原生的按序列**槽位**来承载——每个请求拥有自己的 MLA 与索引器缓存以及自己的 `n_past`，绑定请求只是切换活跃槽位，不搬运任何 KV 字节。
+- **连续批处理 & 分页 KV 缓存** —— vLLM 风格的分页 KV 块池，跨请求的块级哈希前缀共享，迭代级调度器（可在批内动态加入/抢占序列），可选的 SSD 冷层用于超大 KV 工作集，原生融合分页注意力内核（`TSGgml_PagedAttentionForward`，在 Metal/CUDA/Vulkan 上驱动 `ggml_flash_attn_ext`）。`TensorSharp.Server` 默认启用，可用 `--no-continuous-batching` 关闭。详见 [docs/PAGED_ATTENTION_AND_CONTINUOUS_BATCHING_zh-cn.md](docs/PAGED_ATTENTION_AND_CONTINUOUS_BATCHING_zh-cn.md)。GLM 5.x 是个例外：带权重吸收的 MLA（每层每 token 只占一行 576 宽的缓存）与 DSA lightning indexer 没有分页布局，因此那里的并发靠原生的按序列**槽位**来承载——每个请求拥有自己的 MLA 与索引器缓存以及自己的 `n_past`，绑定请求只是切换活跃槽位，不搬运任何 KV 字节。Qwen 3.8 Flash Next（`qwen4exp`）出于同样的原因是同样的形状——它的 GatedDeltaNet、PLE 与 QSA 索引器状态同样没有分页布局——因此靠按序列的**状态持有器**承载：每个在飞请求拥有自己的注意力 KV 与索引器缓存、GDN 卷积 + delta-net 状态以及 PLE 历史，原生内核把驻留设备的递归状态按持有器做键，因此切换请求只是一次引用交换；引擎则让各序列轮转跑各自捕获的融合 decode 图。
 - **投机解码** —— 在共享的"起草—验证—回滚"运行时之上，架了一层可插拔的算法（`--spec-type`：`auto` / `draft-head` / `block` / `ngram`）；无权重的 `ngram` 投机器对**所有**模型都可用，训练出来的草稿头则加速单序列（无并发）decode。Qwen 3.6 与 GLM 5.2 将 NextN 块内嵌在主干 GGUF 中；Gemma 4 通过 `--spec-draft-model` 加载独立的 EAGLE 风格 `gemma4-assistant` 草稿 GGUF，其草稿层读取目标模型自身的 KV 缓存。草稿每步最多提议 `--spec-draft` 个 token（草稿置信度 ≥ `--spec-pmin` 时保留），主干用一次批量前向完成验证；起草与验证均由该请求自己的采样器（含惩罚项）驱动，因此输出与标准 decode 完全一致。CLI 与服务端均通过 `--spec` 启用（默认关闭）。在 `TensorSharp.Cli` 上，它在所有单序列路径上生效——`--input`、`--multi-turn-jsonl` 与 `--interactive`。ggml 后端有融合的多 token 验证 / 草稿步内核，是明确收益；Direct `cuda` 后端运行完全驻留 GPU 的逐算子验证 / 草稿，同样有收益；CPU / GGML CPU / MLX 保持标准 decode。环境变量：`TS_SPEC_*`（通用；旧的 `TS_MTP_*` 拼法仍然有效）与 `TS_GMTP_*`（Gemma 4 调优）。
-- **张量并行与分布式推理** —— 用 `--tp N`（`TensorSharp.Cli` 与 `TensorSharp.Server` 均支持，也可用 `TENSORSHARP_TP_DEGREE`）把一个模型按 Megatron-LM 列/行并行范式切分到多张 GPU 上，再用点对点 TCP 集群（`--tp-node-id` / `--tp-peers`）扩展到多台机器。分层 AllReduce 把跨网络流量降到最低。可运行在 Direct `cuda` 后端以及 GGML CUDA / Vulkan 后端上——后者每个 rank 在自己的 GPU 上拥有独立的 ggml 后端、权重分片与 KV 缓存。支持全部自回归架构（Qwen 3、Mistral 3、Gemma 3/4、Qwen 3.5/3.6-family、GPT OSS、Nemotron-H、GLM 5.x、Muse-Glimmer——因为只有 2 个 KV 头，并行度上限为 `--tp 2`），并针对 MoE 专家并行 / 专家切分、GatedDeltaNet 按 rank V-head 归属、Mamba2 复制等异构层提供各自的策略。融合的按 rank 计算图使 `--tp 2` 的 decode 快于单卡（Gemma 4 E4B 51.7 对 37.3 tok/s），也让单卡装不下的模型得以运行。注意 TP 并不是模型用上多张 GPU 的唯一途径：DeepSeek V4 与 GLM 5.x **不加任何开关就会按层切分到所有可见 GPU**（它们的整模型执行器会按每张卡的空闲显存对整层做装箱），`--tp` 在 GLM 5.x 上会把这种切分换成层内部的 Megatron 切分，在 DeepSeek V4 上则只是限制按层切分使用几张卡（等同 `TS_DSV4_NGPU`）；其他所有架构在不加 `--tp` 时只用一张 GPU。服务端还可选用 Redis 支撑的共享 KV 缓存与 Responses API 存储。→ [张量并行](USAGE_zh-cn.md#张量并行与分布式推理)
+- **张量并行与分布式推理** —— 用 `--tp N`（`TensorSharp.Cli` 与 `TensorSharp.Server` 均支持，也可用 `TENSORSHARP_TP_DEGREE`）把一个模型按 Megatron-LM 列/行并行范式切分到多张 GPU 上，再用点对点 TCP 集群（`--tp-node-id` / `--tp-peers`）扩展到多台机器。分层 AllReduce 把跨网络流量降到最低。可运行在 Direct `cuda` 后端以及 GGML CUDA / Vulkan 后端上——后者每个 rank 在自己的 GPU 上拥有独立的 ggml 后端、权重分片与 KV 缓存。支持全部自回归架构（Qwen 3、Mistral 3、Gemma 3/4、Qwen 3.5/3.6-family、GPT OSS、Nemotron-H、GLM 5.x（仅 GGML 后端，且仅 GLM-5.2 `glm-dsa`）、Muse-Glimmer——因为只有 2 个 KV 头，并行度上限为 `--tp 2`），并针对 MoE 专家并行 / 专家切分、GatedDeltaNet 按 rank V-head 归属、Mamba2 复制等异构层提供各自的策略。融合的按 rank 计算图使 `--tp 2` 的 decode 快于单卡（Gemma 4 E4B 51.7 对 37.3 tok/s），也让单卡装不下的模型得以运行。注意 TP 并不是模型用上多张 GPU 的唯一途径——现在产品里有两种不同的多卡模式。**张量并行**把每一层*内部*的权重切片，并为此每层付出集合通信的代价来重新汇聚，因此它可能同时买到容量与延迟。**按层切分**则是每张 GPU 拿一段连续的*整层*：不切分任何权重，不发起任何集合通信；它是一项**容量**特性——它解决的是“单卡装不下的模型怎么跑”，而不是让它更快。DeepSeek V4 与 GLM 5.x **不加任何开关就会按层切分到所有可见 GPU**（它们的整模型执行器会按每张卡的空闲显存对整层做装箱）；`--tp` 在 GLM-5.2 上会把这种切分换成层内部的 Megatron 切分，GLM-5.3-Flash（`glm5next`）会干净地拒绝 `--tp` 并继续用按层切分，而在 DeepSeek V4 上 `--tp` 只是限制按层切分使用几张卡（等同 `TS_DSV4_NGPU`）。在 Qwen 3.8 Flash Next（`qwen4exp`）上，`--tp N` 本身*就是*按层切分——该架构不切分任何权重，而这也是 llama.cpp 对这个架构提供的同一种（也是唯一一种）多卡模式，因为 `-sm row` 拒绝加载它。在 2x A100-80GB 上用 Qwen3.8-Flash-Next-UD-Q2_K_XL（73.4 GiB）实测：单卡与双卡的贪心输出**逐字节一致**（SHA-256 相同），显存为 24.2 GB + 26.2 GB 而不是全部压在一张卡上，吞吐则基本不变（prefill 约 1520-1550 t/s，decode 约 56 t/s，两边都一样）。启动时会打印实际跑的是哪种模式，以及每张 GPU 的层数 / 字节分配。其他所有架构在不加 `--tp` 时只用一张 GPU；而既不支持张量并行、也不支持按层切分的架构现在会在 stderr 上明确说明并只用一张 GPU，而不是默默地把其余 GPU 扔在那里闲置。服务端还可选用 Redis 支撑的共享 KV 缓存与 Responses API 存储。→ [张量并行](USAGE_zh-cn.md#张量并行与分布式推理)
 - **批处理 / 并行推理** —— 已为 Mistral 3、Gemma 4、GPT OSS、Qwen 3、Qwen 3.5/3.6-family、Nemotron-H 默认启用 `IBatchedPagedModel.ForwardBatch`，能在一次前向传播中打包 N 个序列，使用 `slotMapping` 进行分页 K/V 写入，并通过原生内核做按序列注意力。Gemma 4、Qwen 3.5/3.6、GPT OSS 与 Nemotron-H 提供各自的 `TS_<FAMILY>_BATCHED=0` 兜底开关；Qwen 3 与 Mistral 3 没有家族专属开关，请用全局 `TS_SCHED_DISABLE_BATCHED=1` 强制回到按序列 KV-swap 路径。GLM 5.x 没有分页版 `ForwardBatch`；取而代之的是一条可选的批处理融合解码（`TS_BATCHED_FUSED_DECODE=1`）：一张图、每个序列一个 token，权重只读一次——4 路并发下总解码吞吐 1.81 倍。默认关闭，因为批处理会改变 GEMM 形状，而 2-bit MoE 会把这点差异放大成不同的专家选择。
 - **兼容 Ollama 与 OpenAI API** —— 可作为现有工具链的即插即用替代端点
 - **可配置采样** —— temperature、top-k、top-p、min-p、重复/存在/频率惩罚、seed、停止序列
@@ -28,7 +28,7 @@
 - **视频生成，仅视频（Wan 2.1 文生视频，Wan 2.2 文/图生视频）** —— 提示词（Wan 2.2 模型可再加一张首帧图片）生成 H.264 MP4 视频（无音轨）。所加载的 `wan` GGUF 是 Wan DiT —— 自动识别 Wan 2.1 T2V、Wan 2.2 TI2V-5B（48 通道 16×16×4 潜空间、24 fps）与 Wan 2.2 A14B（两个 14B 专家按时间步边界切换，第二个 GGUF 自动配对）；TensorSharp 在其旁解析伴随模型——UMT5-XXL 文本编码器 GGUF（提示词 → 512×4096 条件，精确的 unigram-Viterbi SentencePiece 分词）与对应的因果 3D 视频 VAE（`wan_2.1_vae.safetensors` / `Wan2.2_VAE.safetensors`）。FlowMatch CFG 去噪（UniPC 或 Euler）每步将整个 DiT（带 3D RoPE + flash 注意力的自注意力、交叉注意力、AdaLN 时间调制——TI2V 图生视频为逐 token 时间步）作为单个常驻权重 ggml 图运行，按形状 CUDA 图捕获（`TSGgml_WanDitForward`）；视频 VAE 在单个图内解码全部时序块（`TSGgml_WanVaeDecode`）——Metal 上卷积走 MPSGraph（736x544x81f 的一次解码从 159 秒降到 80 秒，1.99×，数值不变，PSNR 93.9 dB；`TS_WAN_VAE_MPS_CONV=0` 可恢复 ggml 的 im2col+GEMM 下降路径），其他后端走带状 im2col+GEMM；im2col 预算与分块阈值现在按设备可用显存推导，而不再固定按 16 GB 显卡的预算，因此大显存设备可整幅解码 720p 平面（565 秒 / 峰值 RSS 4.85 GB，对比分成两带的 655 秒 / 5.37 GB），小显存设备仍然分块。图生视频的首帧经因果 VAE 编码器单图编码（`TSGgml_WanVaeEncode`）。各阶段在进入下一阶段前释放各自 VRAM，因此 TI2V-5B 81 帧 480p 图生视频与两个 A14B Q4_K_M 专家均可在 16 GB GPU 上运行。**步数蒸馏检查点会按 DiT 文件名自动识别**（`Turbo`、`distill`、`Lightning`、`lightx2v`、`FastWan`、`-dmd`，或显式的 `…-4steps-…`），这是最大的提速手段：官方 50 步 × CFG 配方需要 100 次 DiT 前向，而 4 步蒸馏检查点只需 4 次，管线会自动切换到该步数并关闭引导（`--diffusion-steps` / `--cfg` 可覆盖）。在 M5 Pro、`ggml_metal`、Wan2.2-TI2V-5B Q8_0、1088×832×121f = 27 404 token 上实测：基础检查点 100 次前向、每次 120.2 秒，端到端约 3 小时 30 分；同一请求换成 Turbo 检查点只需 4 次前向，端到端 **17 分 30 秒**——只有 `--model` 路径不同。基础检查点上还可用 `--cfg-cache-stride 2` / `3` 复用引导方向，再快 1.30× / 1.43×。数值已对照 diffusers 验证（DiT 余弦 > 0.995，VAE 编码器 > 0.999，解码器 59.9 dB / >35 dB PSNR）；让单次 27k token 自注意力快 2.02×（连同 VAE 的改动，每次 DiT 前向约 1.7×）的 F16 注意力键值，其 DiT 余弦与 F32 同为 0.999964。可从 C# 通过 `WanVideoModel.GenerateVideo(prompt, WanVideoParams)`、CLI（`--prompt`、`--image`、`--video-frames`、`--fps`、`--flow-shift`、`--negative-prompt`）、服务器 API（`/v1/videos/generations` 支持 base64 `image`，`/api/video-generate[/stream]` 支持 `imagePath`）以及 Web UI 聊天（输入提示词——附图即为图生视频——获得带实时进度的视频）驱动。→ [Wan 卡片](docs/models/wan_zh-cn.md)
 - **混合 SSM-Transformer** —— Nemotron-H 在单个模型中混合 Mamba2 SSM 层、纯注意力层和 MoE FFN 层；Mamba2 步现在同时提供单序列原生内核与批处理原生内核（`TSGgml_NemotronMamba2BatchedStepF32`，NEON SIMD + GCD 并行）。在 GGML 后端上，注意力层直接用设备侧 flash-attention 内核对常驻 KV 缓存做 decode（`TS_NEMOTRON_FLASH_DECODE=0` 恢复主机路径），decode 速度不再随上下文长度衰减。
 - **混合注意力-递归网络** —— Qwen 3.5/3.6-family 在同一模型中混合全注意力层与 GatedDeltaNet 递归层；批处理路径下递归运行状态保存在每槽位的递归状态池中
-- **专家混合（MoE）** —— 支持 Gemma 4 MoE 变体（例如 gemma-4-26B-A4B）、GPT OSS MoE（例如 gpt-oss-20b）、Qwen 3.5/3.6-family MoE（`qwen35moe` / `qwen3next` 变体，例如 Qwen3.5-35B-A3B）、Nemotron-H MoE FFN 层，以及 GLM 5.2（744B-A40B：256 个路由专家 top-8 加 1 个共享专家，sigmoid 门控路由带一个只影响选择的 bias 与 x2.5 的路由缩放，前面还有 3 个稠密 SwiGLU 层）
+- **专家混合（MoE）** —— 支持 Gemma 4 MoE 变体（例如 gemma-4-26B-A4B）、GPT OSS MoE（例如 gpt-oss-20b）、Qwen 3.5/3.6-family MoE（`qwen35moe` / `qwen3next` 变体，例如 Qwen3.5-35B-A3B）、Nemotron-H MoE FFN 层，以及 GLM 5.2（744B-A40B：256 个路由专家 top-8 加 1 个共享专家，sigmoid 门控路由带一个只影响选择的 bias 与 x2.5 的路由缩放，前面还有 3 个稠密 SwiGLU 层）、GLM-5.3-Flash（320B：288 个路由专家 top-8 加 1 个共享专家，同样的 x2.5 路由缩放，所有 FFN 均带 SwiGLU 限幅 10）、以及 Qwen 3.8 Flash Next（512 个专家、每 token 用 10 个，与 GatedDeltaNet 递归层交错排列）
 - **MoE 专家 CPU 卸载** —— `--n-cpu-moe N` / `--cpu-moe`（对应 llama.cpp 的 `-ncmoe` / `-cmoe`，环境变量 `TS_N_CPU_MOE`）把前 N 层的路由专家权重留在系统内存并在主机侧相乘，注意力、各处 norm、router 与常驻共享专家仍留在加速器上。在所有具备整模型融合图的架构（Qwen 3.5/3.6、Gemma 4 MoE、GPT OSS、DiffusionGemma）上，被卸载的层仍留在同一张融合图内——加速器在每个被卸载层的 router 之后暂停，主机直接从 GGUF mmap 中取出被选中的专家做乘法，再把结果交回下一段，因此 decode 时每层只有约 8 KB 激活跨总线。它同样能与张量并行组合：`--tp N` 下这些接缝会并入各 rank 的 AllReduce 分段计划（Qwen3.5-35B-A3B `--tp 2`：两卡上 17.4 GB 常驻权重降到 3.2 GB；gemma-4-26B-A4B：12.9 GB 降到 2.4 GB，输出逐字节一致）。在 16 GB 的 RTX 3080 Laptop 上实测：Qwen3.6-35B-A3B `--cpu-moe` 后显存 13.4 → 4.6 GB；gemma-4-26B-A4B 16.1 → 4.8 GB（decode 39.7 → 17.7 tok/s，若只用 `--n-cpu-moe 8` 则为 38.6 tok/s 并让出 3 GB）；gpt-oss-20b 16.2 → 2.9 GB，从而避开 WDDM 溢出悬崖，`--n-cpu-moe 12` 时把 0.3 tok/s 变成 25.4。所有架构（含 DeepSeek V4 Flash）默认都是 0：装不下的模型会在加载时直接拒绝并给出所需的 `--n-cpu-moe N`，而不是悄悄牺牲 decode 吞吐。GLM 5.x 走同一条路径（该 checkpoint 92% 的字节是路由专家），并且主机常驻的专家直接由 GGUF 映射提供，不做私有拷贝。要记住 offload 是为了"装得下"而不是为了快：在 GLM-5.2 本来就放得下的 3× RTX PRO 6000 上，`--n-cpu-moe 30` 会把 pp2048 从 915.9 拉到 94.7、tg64 从 43.9 拉到 16.4 tok/s。→ [MoE CPU 卸载（英文）](USAGE.md#mixture-of-experts-cpu-offload---n-cpu-moe) GLM 5.2 在它的原生 `glm-dsa` 执行器上使用同一条接缝：`--cpu-moe` / `--n-cpu-moe N` 把路由专家留在系统内存里、直接从 GGUF 映射中取用做乘法（不做私有拷贝），并且能与 `--tp N` 组合——驻留主机的层保留完整专家，由 rank 0 求值。在 3x RTX PRO 6000 上，`--n-cpu-moe 30` 用吞吐换空间（pp2048 94.7 / tg64 16.4，对比全部常驻时的 915.9 / 43.9），把加载器能定下的上下文从 342,272 抬到 646,400 token，几乎翻倍。
 - **批量 GPU MoE** —— Qwen 3.5/3.6-family 与 Nemotron-H 在 decode 时通过单次融合的 GGML 计算图调度处理所有被选中的专家（Qwen 3.5-family 还包括可选的 shared expert 与残差加法），消除每个专家的 CPU-GPU 往返
 - **整模型融合 decode 计算图** —— Gemma 4（dense 与 MoE）、Qwen 3.5/3.6 与 GPT OSS 把一个 decode token 的全部计算——每一层、MoE 路由与专家、最终 norm 与 LM head——作为**一次** GGML 计算图调度提交，而不是每层提交一次，GPU 因此不会在层与层之间空等主机。在 CUDA/Vulkan 上该图只构建一次、张量地址保持稳定后反复重放（KV 写入用 `ggml_set_rows`、行号作为 I64 输入，注意力窗口按 stride 补齐、掩码作为 F16 输入），这正是 ggml-cuda 能把它捕获成 CUDA 图的前提。GPT OSS decode 在 A40 上从 24 → 154 tok/s，且随上下文长度基本持平（16K 时仍有 133 tok/s，而逐层路径已跌到 2.3）。可按模型用 `TS_GPTOSS_MODEL_DECODE=0` / `TS_GEMMA4_FD_PERSIST=0` / `TS_QWEN35_FD_PERSIST=0` 关闭。
@@ -41,7 +41,7 @@
 
 ## 思维链 / 推理模式
 
-支持思维链模式的模型（Qwen 3、Qwen 3.5/3.6-family、Gemma 4、GPT OSS、Nemotron-H、DeepSeek V4、GLM 5.x）可以在生成最终答案之前产出结构化的思维链推理内容。思维内容与主要回复分开，客户端可选择显示或隐藏。
+支持思维链模式的模型（Qwen 3、Qwen 3.5/3.6-family、Qwen 3.8 Flash Next、Gemma 4、GPT OSS、Nemotron-H、DeepSeek V4、GLM 5.x）可以在生成最终答案之前产出结构化的思维链推理内容。思维内容与主要回复分开，客户端可选择显示或隐藏。
 
 - **Qwen 3 / Qwen 3.5/3.6-family / Nemotron-H：** 使用 `<think>...</think>` 标签
 - **Gemma 4：** 使用 `<|channel>thought\n...<channel|>` 标签
@@ -119,6 +119,23 @@ TensorSharp 支持**张量并行（TP）**——按 Megatron-LM 列/行并行范
 随后由一次 AllReduce 把隐藏状态重新汇聚。归一化层、词嵌入与 LM head 在各 rank
 上复制。
 
+**两种多卡模式，二者并不是一回事。** 张量并行把每一层*内部*的权重切片，并为此每层
+付出一到两次 AllReduce 来重新汇聚：每个 rank 都参与每一层的一部分计算，因此这种模式
+可能同时买到容量与延迟。**按层切分**则是给每张 GPU 一段连续的*整层*：不切分任何权重，
+不发起任何集合通信，各张卡是轮流处理同一个 token 而不是一起处理它。因此按层切分是一项
+**容量**特性——它解决的是“单卡装不下的模型怎么跑得起来”——不应指望它提升吞吐。各架构
+分别用哪一种：
+
+| 架构 | 多卡模式 |
+|---|---|
+| Qwen 3、Mistral 3、Gemma 3/4、Qwen 3.5/3.6-family、GPT OSS、Nemotron-H、Muse-Glimmer（上限 `--tp 2`） | 张量并行，用 `--tp N` 显式开启 |
+| GLM-5.2（`glm-dsa`） | 默认按层切分到所有可见 GPU；`--tp N` 会把它换成张量并行 |
+| GLM-5.3-Flash（`glm5next`） | 默认按层切分到所有可见 GPU；`--tp` 会被干净地拒绝 |
+| DeepSeek V4 Flash | 默认按层切分到所有可见 GPU；`--tp N` 只限制这次切分用几张卡（与 `TS_DSV4_NGPU` 相同） |
+| Qwen 3.8 Flash Next（`qwen4exp`） | 按层切分，用 `--tp N` 显式开启（GGML CUDA / Vulkan） |
+
+启动时会打印实际跑的是哪一种模式，因此不必从 `nvidia-smi` 去猜。
+
 **本地 TP** 在单个进程内运行。在 Direct `cuda` 后端上，由一个线程向所有 GPU 下发
 命令，真正的并行由 CUDA stream 提供；在 GGML 后端上则由一个 rank 工作线程池并发
 驱动各张 GPU——因为 GGML 的一次算子调用同时完成提交与同步。用 `--tp N`
@@ -142,11 +159,33 @@ TensorSharp 支持**张量并行（TP）**——按 Megatron-LM 列/行并行范
 | GGML 上的 MoE（Gemma 4） | 在**每个专家内部**按 Megatron 方式切分（gate/up 列并行、down 行并行），使融合的整模 MoE 主干内核仍能使用全局专家 id；专家求和成为该层的第三个行并行 AllReduce 点。`TS_GEMMA4_TP_FUSED_MOE=0` 可回退到逐算子的整专家路径 |
 | GatedDeltaNet SSM（Qwen 3.5/3.6） | 块循环 V-head 分配——各 rank 在自己的 V-head 子集上运行常驻本卡的打包 GDN 内核，delta/conv 状态相互独立；循环路径无需跨 rank 通信 |
 | Mamba2 SSM（Nemotron-H） | 在 rank 0 上复制计算，结果广播给所有 rank |
-| GGML 上的 MLA + 稀疏注意力 MoE（GLM 5.x） | 注意力头列并行（`attn_q_b` / `attn_k_b` / `attn_v_b`）配行并行 `attn_output`；256 个路由专家不是按专家 id 划分，而是在**每个专家内部**按 Megatron 方式切分（gate/up 列并行、down 行并行），因为 `ggml_mul_mat_id` 要求同一 token 选中的专家 id 互不相同。router、各处 norm、lightning indexer、共享专家与 3 个稠密层均为复制；每层两次 AllReduce。`TS_GLM_TP_SHARD` 选择切哪一半（1 注意力头、2 专家、3 两者都切），`TS_GLM_TP_OVERSUBSCRIBE=1` 可把多个 rank 挤在同一张卡上做测试 |
+| GGML 上的 MLA + 稀疏注意力 MoE（GLM-5.2 `glm-dsa`；GLM-5.3-Flash `glm5next` 会拒绝 `--tp`，继续用按层切分） | 注意力头列并行（`attn_q_b` / `attn_k_b` / `attn_v_b`）配行并行 `attn_output`；256 个路由专家不是按专家 id 划分，而是在**每个专家内部**按 Megatron 方式切分（gate/up 列并行、down 行并行），因为 `ggml_mul_mat_id` 要求同一 token 选中的专家 id 互不相同。router、各处 norm、lightning indexer、共享专家与 3 个稠密层均为复制；每层两次 AllReduce。`TS_GLM_TP_SHARD` 选择切哪一半（1 注意力头、2 专家、3 两者都切），`TS_GLM_TP_OVERSUBSCRIBE=1` 可把多个 rank 挤在同一张卡上做测试 |
 
 TP 可运行在 `cuda` 后端以及 GGML CUDA / Vulkan 后端（`ggml_cuda`、`ggml_vulkan`）上；MLX 为单设备。在 GGML 后端上，每个 rank 拥有自己 GPU 上的 ggml 后端、权重分片与 KV 缓存，跨 GPU AllReduce 走 ggml-cuda 的集合通信（可用时用 NCCL），小载荷则在主机内存中归约。**TP 下 CUDA 图捕获保持开启**——一个张量并行 token 是几十次按 rank 的小提交，重放它们值约 45% 的 decode 吞吐（4×A40：Qwen 3.5-9B `--tp 4` 从 88 → 128.5 tok/s，Qwen 3.5-35B-A3B `--tp 2` 从 71.3 → 104.1，后者正是 TP 输给还是赢过单卡的分界线）。用 `TS_GGML_TP_CUDA_GRAPHS=0` 关闭。集合通信的选择靠实测而非能力标志位：启动时该组会验证所宣称的设备对之间的 peer copy 是否真的把数据送到，以及一次真实的 NCCL AllReduce 能否完成，然后选出通过检验的最快传输。有些主机（常见于虚拟化云实例）宣称支持 peer access 却从不兑现，此时会保留 NCCL 集合通信但禁用 peer 传输，而不是干脆放弃它——这在超过两张卡时尤为重要，因为那里用不上 pinned-host 流水线，替代方案是每个层边界都经主机内存归约（4×A40 实测：Qwen 3.5-9B Q8_0 decode 53.5 → 75.1 tok/s）。GGML 上的 TP 同时带来**容量**与**延迟**收益：融合的按 rank block 计算图（注意力、稠密 FFN、MoE 主干、GatedDeltaNet）取代了逐算子前向，在 2× RTX 2000 Ada 上 `--tp 2` 的 decode 达到单卡的 **1.39×**（Gemma 4 E4B Q8_0，51.7 对 37.3 tok/s）与 **1.06×**（Qwen 3.5-9B Q8_0），且 Gemma 4 的输出与单卡逐字节一致；单卡装不下的模型则只能靠 TP 运行（Qwen 3.5-35B-A3B IQ4_XS 共 16.6 GB，拆到两张 16 GB 卡上，prefill 184 tok/s、decode 18 tok/s）。完整测量数据见 `TENSOR_PARALLELISM_PLAN.md`（Stage 1b 与 1c）。这能推广多远，取决于互连带宽以及一层里究竟有多少能拆：在没有 NVLink 的主机上，每层两次 AllReduce 会成为瓶颈——GLM-5.2 UD-IQ2_XXS 在 3× RTX PRO 6000（PCIe）上 `--tp 3` 为 pp2048 505.6 / tg64 17.6 tok/s，而单机按层切分是 915.9 / 43.9，并且每个 rank 都要各自持有一份全长缓存，能装下的上下文从 342,272 掉到 91,136 token。那里的 TP 是**容量**特性而非延迟特性；它还改变了归约顺序，因此在 2-bit MoE 上，对着录制的 llama.cpp 金标准，按层切分复现 5/6 条提示，而 `--tp 3` 只复现 3/6。TP 下的批处理 /
 连续批处理前向目前实现于 Qwen 3 与 Mistral 3；MoE 模型在 TP 下回退到按序列前向。
 
+**Qwen 3.8 Flash Next 上的按层切分。** `qwen4exp` 上的 `--tp N` 跑的是按层切分而不是
+张量并行：它的权重一个都不切分，它的
+decode 是每个 token 一张持久化的单设备 GGML 图，而它的 GDN / PLE 递归状态存放在由单个
+后端持有的设备缓冲里。这也是 llama.cpp 对这个架构提供的同一种（也是唯一一种）多卡模式
+——`-sm row` 拒绝加载它。
+
+在 2x A100-80GB 上用 Qwen3.8-Flash-Next-UD-Q2_K_XL（73.4 GiB）实测：
+
+- 单卡与双卡的贪心输出**逐字节一致**（SHA-256 相同）；
+- 显存 24.2 GB + 26.2 GB——大致每张卡放一半模型，而不是全部压在一张卡上；
+- 吞吐不变：prefill 约 1520-1550 t/s，decode 约 56 t/s，两边都一样。作为参照，同一台机器
+  上的 llama.cpp：单卡 pp1536 1094 / tg128 61.2，双卡 `-sm layer` 1200 / 61.5——也就是说
+  llama.cpp 从第二张卡上同样只拿到约 10% 的 prefill 收益、decode 基本为 0。
+
+`TS_Q4E_LAYER_SPLIT=20,28` 可用显式的每卡层数覆盖自动均衡（相当于 llama.cpp 的
+`--tensor-split`），且在无法满足给定值时直接抛错而不是默默忽略——这很有用，因为自动均衡
+只按权重定价，看不到稍后才加载、且会落在 GPU 0 上的视觉塔。详见
+[Qwen 3.8 Flash Next 卡片](docs/models/qwen38-flash-next_zh-cn.md)。
+
+既不支持张量并行、也不支持按层切分的架构现在会在 stderr 上明确说明并只用一张 GPU，而不是
+接受 `--tp N`、打印一条张量并行横幅，然后让其余 GPU 拿着 CUDA 上下文与 NCCL 缓冲闲置。
+
 本地集合通信优先使用 CUDA 点对点（P2P）DMA，但启动时会对每一对支持 P2P 的设备
 做一次往返自检，任何回读数据损坏的设备对（在部分 L4 PCIe 拓扑上出现过）都会被
 永久降级；因此在 P2P 不可用的主机（A16 vGPU 配置、大多数消费级显卡）上会自动改
@@ -194,6 +233,25 @@ Gemma 3 支持 PNG、JPEG 与 HEIC/HEIF 图像输入。上文非 gated 示例使
 
 所有 Qwen 3.5/3.6-family 变体（`qwen35`、`qwen35moe` 与 `qwen3next`）共用同一个 `Qwen35Model` 实现。图像输入通过支持动态分辨率的 `Qwen35VisionEncoder` 处理；请显式传入所选仓库的投影器（上文 9B 与 Qwen 3.6 示例均为 `mmproj-F16.gguf`）。MoE 变体（例如 Qwen3.5-35B-A3B，以及使用同一架构标识的 Qwen3.6-35B-A3B GGUF）在 decode 时还会启用融合的 `MoEExpertsSwiGLUResidual` GGML 内核，将所有被选中的专家、可选的 shared expert 与残差加法合并到一次 GPU 计算图调度中执行。
 
+### Qwen 3.8 Flash Next
+
+Qwen3.8-Flash-Next（`qwen4exp`）通过 Qwen3.5-VL 视觉塔支持图像输入，位置编码为 (T, H, W)
+的 IMRoPE；把仓库中的 `mmproj-BF16.gguf` 放在模型旁边即可启用。多图提示与多轮图像会话都
+可用，并且跨轮复用 KV——但只能“继续追加”，因为 GatedDeltaNet 的递归无法回退，所以只有当
+新提示恰好是缓存前缀的延长时才会复用。
+
+- **图像：** PNG、JPEG、HEIC/HEIF
+
+### GLM-5.3-Flash
+
+GLM-5.3-Flash（`glm5next`）通过仓库中 `mmproj-BF16.gguf` 里的 GLM-OCR ViT 支持图像输入：
+RMS norm、融合 QKV、按头的 q/k RMS norm、2D 视觉 RoPE、带 SwiGLU 限幅的 MLP 与 2x2 卷积
+merger，全部 24 个块作为一张驻留设备的 GGML 图运行。投影后的 embedding 在原生执行器内部
+直接覆盖 `<|image|>` 占位行；由于文本塔是 NoPE，图像 token 不需要任何 MRoPE 记账。
+`--image`、多图提示与多轮图像会话均受支持。GLM-5.2（`glm-dsa`）仅支持文本。
+
+- **图像：** PNG、JPEG、HEIC/HEIF
+
 ### Mistral 3
 
 Mistral 3 通过 Pixtral 视觉编码器支持图像输入。示例仓库使用 `mmproj-mistralai_Mistral-Small-3.1-24B-Instruct-2503-f16.gguf`；请通过 `--mmproj` 显式传入。
diff --git a/InferenceWeb.Tests/DFlash2MathTests.cs b/InferenceWeb.Tests/DFlash2MathTests.cs
new file mode 100644
index 00000000..f3a36677
--- /dev/null
+++ b/InferenceWeb.Tests/DFlash2MathTests.cs
@@ -0,0 +1,240 @@
+// Copyright (c) Zhongkai Fu. All rights reserved.
+// https://github.com/zhongkaifu/TensorSharp
+//
+// This file is part of TensorSharp.
+//
+// TensorSharp is licensed under the BSD-3-Clause license found in the LICENSE file in the root directory of this source tree.
+//
+// TensorSharp is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the BSD-3-Clause License for more details.
+using System;
+using System.Collections.Generic;
+using TensorSharp.Models;
+using Xunit;
+
+namespace InferenceWeb.Tests;
+
+/// <summary>
+/// The two pieces of arithmetic DFlash2 adds on top of DFlash, checked against
+/// independent references written from the algorithm rather than from the
+/// implementation:
+///
+///   * the grouped dynamic depthwise convolution that wraps every attention and
+///     every FFN sublayer, and
+///   * the greedy walk through the candidate selector's transition lattice.
+///
+/// These run without weights, a GPU, or a native library, so they hold on every
+/// machine - unlike the end-to-end parity runs, which need the checkpoints.
+/// </summary>
+public class DFlash2MathTests
+{
+    /// <summary>
+    /// out[r][c] = sum_t (base[side][t][c] + delta[r][t][c / groupSize]) * x[r-t][c],
+    /// tap t masked off for r &lt; t. Written the slow, obvious way.
+    /// </summary>
+    private static float[] ReferenceConvolve(
+        float[] x, float[] baseKernel, float[] delta,
+        int rows, int hidden, int taps, int groupSize, int deltaStride, int deltaOffset, int side)
+    {
+        int groups = hidden / groupSize;
+        var outp = new float[rows * hidden];
+        for (int r = 0; r < rows; r++)
+        {
+            for (int c = 0; c < hidden; c++)
+            {
+                float acc = 0f;
+                for (int t = 0; t < taps; t++)
+                {
+                    if (r - t < 0)
+                        continue;                     // the block-boundary mask
+                    float b = baseKernel[(side * taps + t) * hidden + c];
+                    float d = delta[r * deltaStride + deltaOffset + t * groups + (c / groupSize)];
+                    acc += (b + d) * x[(r - t) * hidden + c];
+                }
+                outp[r * hidden + c] = acc;
+            }
+        }
+        return outp;
+    }
+
+    [Theory]
+    [InlineData(1, 8, 2, 2)]      // one row: only tap 0 survives the mask
+    [InlineData(8, 16, 2, 4)]     // the shipped shape (2 taps)
+    [InlineData(5, 12, 3, 3)]     // a wider kernel than anything shipped
+    [InlineData(16, 64, 2, 16)]   // the shipped group size
+    public void GroupedConvolve_MatchesReference(int rows, int hidden, int taps, int groupSize)
+    {
+        int groups = hidden / groupSize;
+        int deltaStride = 2 * taps * groups;   // both sides, as the projection emits them
+        var rng = new Random(1234 + rows * 31 + hidden);
+
+        float[] Rand(int n)
+        {
+            var a = new float[n];
+            for (int i = 0; i < n; i++)
+                a[i] = (float)(rng.NextDouble() * 2 - 1);
+            return a;
+        }
+
+        float[] x = Rand(rows * hidden);
+        float[] baseKernel = Rand(2 * taps * hidden);
+        float[] delta = Rand(rows * deltaStride);
+
+        for (int side = 0; side < 2; side++)
+        {
+            int deltaOffset = side * taps * groups;
+            float[] expected = ReferenceConvolve(x, baseKernel, delta, rows, hidden, taps, groupSize, deltaStride, deltaOffset, side);
+            float[] actual = ModelBase.DFlashGroupedConvolve(x, baseKernel, delta, rows, hidden, taps, groupSize, deltaStride, deltaOffset, side);
+
+            Assert.Equal(expected.Length, actual.Length);
+            for (int i = 0; i < expected.Length; i++)
+                Assert.True(Math.Abs(expected[i] - actual[i]) < 1e-4f, $"side {side} element {i}: {expected[i]} != {actual[i]}");
+        }
+    }
+
+    /// <summary>
+    /// A checkpoint's conv_base starts (and stays close to) the identity - tap 0 all
+    /// ones, later taps zero - so an all-zero delta must leave the rows untouched.
+    /// That is the property that makes the convolution safe to add to a trained
+    /// backbone, and the one a layout mistake would break first.
+    /// </summary>
+    [Fact]
+    public void GroupedConvolve_IdentityKernelIsAPassThrough()
+    {
+        const int rows = 8, hidden = 32, taps = 2, groupSize = 8;
+        int groups = hidden / groupSize;
+        int deltaStride = 2 * taps * groups;
+
+        var x = new float[rows * hidden];
+        for (int i = 0; i < x.Length; i++)
+            x[i] = i + 1;
+
+        var baseKernel = new float[2 * taps * hidden];
+        for (int side = 0; side < 2; side++)
+            for (int c = 0; c < hidden; c++)
+                baseKernel[(side * taps + 0) * hidden + c] = 1f;   // tap 0 = identity
+
+        var delta = new float[rows * deltaStride];                  // no dynamic part
+
+        float[] actual = ModelBase.DFlashGroupedConvolve(x, baseKernel, delta, rows, hidden, taps, groupSize, deltaStride, 0, side: 0);
+        Assert.Equal(x, actual);
+    }
+
+    /// <summary>Tap 1 must never reach across the block boundary: row 0 sees only
+    /// itself no matter what the tap-1 coefficient is.</summary>
+    [Fact]
+    public void GroupedConvolve_FirstRowIgnoresTheShiftedTap()
+    {
+        const int rows = 3, hidden = 4, taps = 2, groupSize = 2;
+        int groups = hidden / groupSize;
+        int deltaStride = 2 * taps * groups;
+
+        var x = new float[] { 1, 2, 3, 4, 10, 20, 30, 40, 100, 200, 300, 400 };
+        var baseKernel = new float[2 * taps * hidden];
+        for (int c = 0; c < hidden; c++)
+        {
+            baseKernel[(0 * taps + 0) * hidden + c] = 1f;   // tap 0 identity
+            baseKernel[(0 * taps + 1) * hidden + c] = 5f;   // tap 1 deliberately huge
+        }
+        var delta = new float[rows * deltaStride];
+
+        float[] actual = ModelBase.DFlashGroupedConvolve(x, baseKernel, delta, rows, hidden, taps, groupSize, deltaStride, 0, side: 0);
+
+        // Row 0: x[0] only. Row r>0: x[r] + 5*x[r-1].
+        Assert.Equal(new float[] { 1, 2, 3, 4 }, actual[0..4]);
+        Assert.Equal(new float[] { 10 + 5 * 1, 20 + 5 * 2, 30 + 5 * 3, 40 + 5 * 4 }, actual[4..8]);
+        Assert.Equal(new float[] { 100 + 5 * 10, 200 + 5 * 20, 300 + 5 * 30, 400 + 5 * 40 }, actual[8..12]);
+    }
+
+    [Fact]
+    public void TopK_SelectsTheLargestValuesRegardlessOfOrder()
+    {
+        var rng = new Random(7);
+        var row = new float[4096];
+        for (int i = 0; i < row.Length; i++)
+            row[i] = (float)(rng.NextDouble() * 100 - 50);
+
+        const int k = 16;
+        var ids = new int[k];
+        var vals = new float[k];
+        ModelBase.DFlashTopK(row, k, ids, vals, 0);
+
+        // Every reported (id, value) pair must be consistent...
+        for (int i = 0; i < k; i++)
+            Assert.Equal(row[ids[i]], vals[i]);
+        // ...the ids distinct...
+        Assert.Equal(k, new HashSet<int>(ids).Count);
+        // ...and the set must be exactly the k largest (the top-k is unsorted, so
+        // compare as sets against a sorted reference).
+        var sorted = (float[])row.Clone();
+        Array.Sort(sorted);
+        float cutoff = sorted[^k];
+        foreach (float v in vals)
+            Assert.True(v >= cutoff, $"{v} < cutoff {cutoff}");
+    }
+
+    /// <summary>
+    /// The lattice walk: position 0 takes the argmax of the anchor row, and every
+    /// later position takes the argmax of the row selected by the PREVIOUS
+    /// position's choice. Getting that indirection wrong is invisible in the output
+    /// (the trunk still corrects every token) and shows up only as lost acceptance,
+    /// so it is worth pinning exactly.
+    /// </summary>
+    [Fact]
+    public void WalkLattice_FollowsThePreviousChoicesRow()
+    {
+        const int gamma = 3, k = 4;
+        // scores layout: [k] anchor row, then one [k(pred), k(cand)] matrix per
+        // following position, candidate-fastest.
+        var scores = new float[k + k * k * (gamma - 1)];
+
+        // Anchor row: candidate 2 wins.
+        scores[0] = 0.1f; scores[1] = 0.2f; scores[2] = 9.0f; scores[3] = 0.3f;
+
+        // Position 1, predecessor rows. Only row 2 should ever be read; make the
+        // others point somewhere else so a wrong index is caught.
+        float[,] p1 = { { 5, 0, 0, 0 }, { 0, 5, 0, 0 }, { 0, 0, 0, 7 }, { 0, 0, 5, 0 } };
+        for (int p = 0; p < k; p++)
+            for (int c = 0; c < k; c++)
+                scores[k + (p * k) + c] = p1[p, c];
+
+        // Position 2: only predecessor row 3 matters (position 1 chose candidate 3).
+        float[,] p2 = { { 1, 0, 0, 0 }, { 0, 1, 0, 0 }, { 0, 0, 1, 0 }, { 0, 8, 0, 0 } };
+        for (int p = 0; p < k; p++)
+            for (int c = 0; c < k; c++)
+                scores[k + k * k + (p * k) + c] = p2[p, c];
+
+        // Distinct token ids so the gather is checked too.
+        var cand = new int[gamma * k];
+        for (int e = 0; e < gamma; e++)
+            for (int c = 0; c < k; c++)
+                cand[e * k + c] = 1000 * (e + 1) + c;
+
+        var draft = new int[gamma];
+        var conf = new float[gamma];
+        int n = ModelBase.DFlashWalkLattice(gamma, k, scores, cand, draft, conf);
+
+        Assert.Equal(gamma, n);
+        Assert.Equal(new[] { 1002, 2003, 3001 }, draft);
+        // Each confidence is the softmax of the row it was chosen from.
+        Assert.Equal(ModelBase.DFlashSoftmaxAt(new[] { 0.1f, 0.2f, 9.0f, 0.3f }, 2), conf[0], 5);
+        Assert.Equal(ModelBase.DFlashSoftmaxAt(new[] { 0f, 0f, 0f, 7f }, 3), conf[1], 5);
+        Assert.Equal(ModelBase.DFlashSoftmaxAt(new[] { 0f, 8f, 0f, 0f }, 1), conf[2], 5);
+    }
+
+    /// <summary>A one-position block has no transitions at all: the walk must read
+    /// only the anchor row and never index past it.</summary>
+    [Fact]
+    public void WalkLattice_SinglePositionUsesOnlyTheAnchorRow()
+    {
+        const int gamma = 1, k = 3;
+        var scores = new float[] { 1f, 4f, 2f };
+        var cand = new[] { 11, 22, 33 };
+        var draft = new int[gamma];
+        var conf = new float[gamma];
+
+        Assert.Equal(1, ModelBase.DFlashWalkLattice(gamma, k, scores, cand, draft, conf));
+        Assert.Equal(22, draft[0]);
+        Assert.Equal(ModelBase.DFlashSoftmaxAt(new[] { 1f, 4f, 2f }, 1), conf[0], 5);
+    }
+}
diff --git a/InferenceWeb.Tests/Gemma4ConcurrentThenSoloRegressionTests.cs b/InferenceWeb.Tests/Gemma4ConcurrentThenSoloRegressionTests.cs
new file mode 100644
index 00000000..a2bf4371
--- /dev/null
+++ b/InferenceWeb.Tests/Gemma4ConcurrentThenSoloRegressionTests.cs
@@ -0,0 +1,189 @@
+// Copyright (c) Zhongkai Fu. All rights reserved.
+// https://github.com/zhongkaifu/TensorSharp
+//
+// This file is part of TensorSharp.
+//
+// TensorSharp is licensed under the BSD-3-Clause license found in the LICENSE file in the root directory of this source tree.
+//
+// Regression: "two prompts in parallel, then New Chat, then a prompt -> <pad>
+// forever".
+//
+// Mechanism. Gemma 4 serves N>=2 concurrent requests from per-request KV-cache
+// holders. On the first multi-sequence step the executor calls
+// AdoptPrimaryCacheToFused, which hands the live PRIMARY cache to the running
+// request's holder and mints a REPLACEMENT primary for later single-stream use.
+// That replacement is the cache the next N==1 request runs on - and it was
+// allocated without being zeroed, while its sibling CreateFreshHolder zeroed
+// its own allocation for exactly this reason.
+//
+// It matters because the fused decode graph reads a FIXED 256-padded attention
+// window: rows past the written length are masked with -inf, which cancels a
+// FINITE score but turns a non-finite one into NaN, and one NaN takes the whole
+// softmax row -> every logit NaN -> argmax returns index 0 -> Gemma's <pad>,
+// forever. ModelBase.InitializeCacheTensor skips the fill on GgmlCuda/Mlx and
+// the pool hands back recycled (not zeroed) blocks, so on those backends the
+// padding was live activation garbage.
+//
+// The fix moved the zero-fill INTO AllocateKvCacheArrays so no allocation site
+// can skip it. This test drives the exact user-visible sequence.
+//
+// Opt-in, and it needs TWO env vars:
+//   TS_TEST_MODEL_DIR=<dir containing gemma-4-E4B*.gguf>
+//   TS_TEST_GGML_BACKEND=cuda
+// The second one matters. GGML allows exactly ONE backend per process and
+// GgmlBackendTestInitializer pins CPU by default, so without it this test can
+// only report "cannot load on GgmlCuda ... Skipping". And CPU would prove
+// nothing: the defect exists only where cache allocation is not zero-filled.
+//
+// Verified to catch the bug: with the zero-fill reverted this fails with
+// "solo request after the concurrent burst emitted 47 <pad> tokens".
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using System.Threading.Tasks;
+using Microsoft.Extensions.Logging.Abstractions;
+using TensorSharp;
+using TensorSharp.Runtime;
+using TensorSharp.Runtime.Scheduling;
+using Xunit;
+using Xunit.Abstractions;
+
+namespace InferenceWeb.Tests;
+
+public class Gemma4ConcurrentThenSoloRegressionTests
+{
+    private const string EnvModelDir = "TS_TEST_MODEL_DIR";
+    private readonly ITestOutputHelper _output;
+
+    public Gemma4ConcurrentThenSoloRegressionTests(ITestOutputHelper output) { _output = output; }
+
+    [ModelFact("TS_TEST_MODEL_DIR", "gemma-4-e4b")]
+    public async Task SoloRequestAfterConcurrentBurst_DoesNotDegenerate()
+    {
+        string modelPath = FindGemma4();
+        if (modelPath == null) { _output.WriteLine("no gemma-4-e4b model; skipping"); return; }
+
+        // The defect only exists on a backend whose cache allocation is NOT
+        // zero-filled, so a green run on ggml_cpu would prove nothing. Try the GPU
+        // backend and treat "cannot create" as a skip rather than a false pass.
+        BackendType backend = OperatingSystem.IsMacOS() ? BackendType.GgmlMetal : BackendType.GgmlCuda;
+        TensorSharp.Models.ModelBase model;
+        try
+        {
+            model = TensorSharp.Models.ModelBase.Create(modelPath, backend);
+        }
+        catch (Exception ex)
+        {
+            _output.WriteLine($"cannot load on {backend} ({ex.GetType().Name}: {ex.Message}); "
+                              + "this defect only reproduces on a non-zero-filling GPU backend. Skipping.");
+            return;
+        }
+        using var _model = model;
+        var renderer = new KVCachePromptRenderer(new GgufPromptRenderer());
+        const int blockSize = 256;
+        var cfg = new SchedulerConfig
+        {
+            MaxNumBatchedTokens = 4096,
+            MaxNumRunningSequences = 4,
+            MaxPrefillChunkSize = 1024,
+            NumBlocks = 128,
+            BlockSize = blockSize,
+            EnablePrefixCaching = true,
+            DecodeQuantumTokens = blockSize,
+        };
+        using var engine = new InferenceEngine(model, cfg, NullLogger.Instance);
+
+        // 1. Two requests in flight together. This is what makes the executor
+        //    hand the primary cache away and mint the replacement.
+        var a = Generate(engine, model, renderer, "Describe the game Final Fantasy VII in detail.", "par-a", 48);
+        await Task.Delay(300);   // let A start before B joins, as two browser tabs would
+        var b = Generate(engine, model, renderer, "Describe the book A Brief History of Time in detail.", "par-b", 48);
+        string outA = await a, outB = await b;
+        _output.WriteLine($"[parallel A] {Trim(outA)}");
+        _output.WriteLine($"[parallel B] {Trim(outB)}");
+
+        // 2. Both finished -> holders released -> the replacement primary becomes
+        //    the active cache. "New Chat" then sends a plain single-stream request,
+        //    which is the one that used to emit nothing but <pad>.
+        string solo = await Generate(engine, model, renderer,
+            "Describe the game Final Fantasy VII in detail.", "solo-after", 48);
+        _output.WriteLine($"[solo after burst] {Trim(solo)}");
+
+        AssertNotDegenerate(outA, "first parallel request");
+        AssertNotDegenerate(outB, "second parallel request");
+        AssertNotDegenerate(solo, "solo request after the concurrent burst");
+    }
+
+    /// <summary>Degenerate == the sampler is being fed dead logits: the reply is
+    /// empty, is one token repeated, or is mostly Gemma's &lt;pad&gt;.</summary>
+    private static void AssertNotDegenerate(string text, string what)
+    {
+        Assert.False(string.IsNullOrWhiteSpace(text), $"{what} produced no text at all.");
+
+        int pads = CountOccurrences(text, "<pad>");
+        Assert.True(pads <= 2,
+            $"{what} emitted {pads} <pad> tokens - the NaN-logits signature. Text: {Trim(text)}");
+
+        // A healthy 48-token reply has many distinct words; a fixed-point loop has
+        // one or two. This also catches placeholder-token streams (<unusedNN>).
+        var words = text.Split(new[] { ' ', '\n', '\r', '\t' }, StringSplitOptions.RemoveEmptyEntries);
+        if (words.Length >= 8)
+        {
+            int distinct = words.Distinct(StringComparer.Ordinal).Count();
+            Assert.True(distinct > 3,
+                $"{what} repeated the same {distinct} token(s) for its whole reply. Text: {Trim(text)}");
+        }
+    }
+
+    private static int CountOccurrences(string haystack, string needle)
+    {
+        int n = 0, i = 0;
+        while ((i = haystack.IndexOf(needle, i, StringComparison.Ordinal)) >= 0) { n++; i += needle.Length; }
+        return n;
+    }
+
+    private static string Trim(string s)
+        => s.Length <= 200 ? s : s.Substring(0, 200) + "...";
+
+    private static async Task<string> Generate(
+        InferenceEngine engine, TensorSharp.Models.ModelBase model,
+        KVCachePromptRenderer renderer, string prompt, string reqId, int maxNewTokens)
+    {
+        var history = new List<ChatMessage> { new() { Role = "user", Content = prompt } };
+        var tokens = renderer.RenderToTokens(
+            model.Tokenizer,
+            model.Config?.ChatTemplate,
+            history,
+            model.Config?.Architecture ?? string.Empty,
+            addGenerationPrompt: true,
+            tools: null,
+            enableThinking: false);
+
+        var seq = new SequenceState(reqId, tokens, maxNewTokens, 256, SamplingConfig.Default);
+        var handle = engine.SubmitRequest(seq);
+        var sb = new System.Text.StringBuilder();
+        try
+        {
+            await foreach (var tok in handle.Tokens.ReadAllAsync())
+                sb.Append(model.Tokenizer.Decode(new List<int> { tok }));
+        }
+        catch (Exception ex)
+        {
+            throw new Xunit.Sdk.XunitException($"generation for {reqId} failed: {ex}");
+        }
+        await handle.Completion;
+        return sb.ToString();
+    }
+
+    private static string FindGemma4()
+    {
+        string dir = Environment.GetEnvironmentVariable(EnvModelDir);
+        if (string.IsNullOrEmpty(dir) || !Directory.Exists(dir)) return null;
+        return Directory.GetFiles(dir, "*.gguf").FirstOrDefault(p =>
+        {
+            var n = Path.GetFileName(p).ToLowerInvariant();
+            return n.Contains("gemma-4-e4b") && !n.Contains("mmproj") && !n.Contains("assistant");
+        });
+    }
+}
diff --git a/InferenceWeb.Tests/Qwen35TokenDecodeContractTests.cs b/InferenceWeb.Tests/Qwen35TokenDecodeContractTests.cs
index cffdabd3..6b8bfaa6 100644
--- a/InferenceWeb.Tests/Qwen35TokenDecodeContractTests.cs
+++ b/InferenceWeb.Tests/Qwen35TokenDecodeContractTests.cs
@@ -188,13 +188,17 @@ public void Qwen35LayerDescriptor_HasExpectedBlittableLayout()
         FieldInfo[] fields = typeof(Qwen35LayerDecodeArgs).GetFields(
             BindingFlags.Public | BindingFlags.Instance);
 
-        // 24 int32s: the 23 original scalars plus CpuMoe, the per-layer MoE CPU
-        // offload flag (--n-cpu-moe). It is appended at the END of the int32 run
-        // so the struct stays a pointers / int64 / int32 sequence and the native
-        // TSGgmlQwen35LayerDesc keeps the same offsets for every field before it.
-        Assert.Equal(33, fields.Count(field => field.FieldType == typeof(IntPtr)));
-        Assert.Equal(48, fields.Count(field => field.FieldType == typeof(long)));
-        Assert.Equal(24, fields.Count(field => field.FieldType == typeof(int)));
+        // Every addition is appended at the END of its own run, so the struct stays
+        // a pointers / int64 / int32 sequence and the native TSGgmlQwen35LayerDesc
+        // keeps the same offsets for every field before it. The two most recent:
+        //   CpuMoe                    per-layer MoE CPU offload (--n-cpu-moe)
+        //   FfnGateW / FfnUpW (+ their shapes and types)
+        //                             the dense FFN of a mixed-quant "UD" layer whose
+        //                             ffn_gate and ffn_up have different GGML types
+        //                             and so cannot be fused into one tensor
+        Assert.Equal(35, fields.Count(field => field.FieldType == typeof(IntPtr)));
+        Assert.Equal(54, fields.Count(field => field.FieldType == typeof(long)));
+        Assert.Equal(26, fields.Count(field => field.FieldType == typeof(int)));
         Assert.All(fields, field => Assert.True(
             field.FieldType == typeof(IntPtr) ||
             field.FieldType == typeof(long) ||
@@ -204,10 +208,10 @@ public void Qwen35LayerDescriptor_HasExpectedBlittableLayout()
         static long Align(long value, int alignment)
             => (value + alignment - 1) / alignment * alignment;
 
-        long int64Start = Align(33L * IntPtr.Size, sizeof(long));
-        long int32Start = int64Start + 48L * sizeof(long);
+        long int64Start = Align(35L * IntPtr.Size, sizeof(long));
+        long int32Start = int64Start + 54L * sizeof(long);
         long expectedSize = Align(
-            int32Start + 24L * sizeof(int),
+            int32Start + 26L * sizeof(int),
             Math.Max(IntPtr.Size, sizeof(long)));
 
         Assert.Equal(0, Marshal.OffsetOf<Qwen35LayerDecodeArgs>(
@@ -218,11 +222,24 @@ static long Align(long value, int alignment)
             nameof(Qwen35LayerDecodeArgs.StructBytes)).ToInt64());
         Assert.Equal(expectedSize, Marshal.SizeOf<Qwen35LayerDecodeArgs>());
 
-        // CpuMoe must stay last: the native side reads it by name, but the
-        // struct_bytes handshake only catches a size change, not a reordering
-        // of the int32 run.
+        // The int32 run's ORDER is what the struct_bytes handshake cannot check -
+        // it only catches a size change - so pin the tail explicitly. CpuMoe kept
+        // its slot when the split-gate/up pair was appended after it.
         Assert.Equal(
             int32Start + 23L * sizeof(int),
             Marshal.OffsetOf<Qwen35LayerDecodeArgs>(nameof(Qwen35LayerDecodeArgs.CpuMoe)).ToInt64());
+        Assert.Equal(
+            int32Start + 24L * sizeof(int),
+            Marshal.OffsetOf<Qwen35LayerDecodeArgs>(nameof(Qwen35LayerDecodeArgs.FfnGateType)).ToInt64());
+        Assert.Equal(
+            int32Start + 25L * sizeof(int),
+            Marshal.OffsetOf<Qwen35LayerDecodeArgs>(nameof(Qwen35LayerDecodeArgs.FfnUpType)).ToInt64());
+        // Same for the pointer and int64 runs.
+        Assert.Equal(
+            34L * IntPtr.Size,
+            Marshal.OffsetOf<Qwen35LayerDecodeArgs>(nameof(Qwen35LayerDecodeArgs.FfnUpW)).ToInt64());
+        Assert.Equal(
+            int64Start + 48L * sizeof(long),
+            Marshal.OffsetOf<Qwen35LayerDecodeArgs>(nameof(Qwen35LayerDecodeArgs.FfnGateNe0)).ToInt64());
     }
 }
diff --git a/InferenceWeb.Tests/Qwen4ExpLayerSplitTests.cs b/InferenceWeb.Tests/Qwen4ExpLayerSplitTests.cs
new file mode 100644
index 00000000..f4459745
--- /dev/null
+++ b/InferenceWeb.Tests/Qwen4ExpLayerSplitTests.cs
@@ -0,0 +1,183 @@
+// Copyright (c) Zhongkai Fu. All rights reserved.
+// https://github.com/zhongkaifu/TensorSharp
+//
+// This file is part of TensorSharp.
+//
+// TensorSharp is licensed under the BSD-3-Clause license found in the LICENSE file in the root directory of this source tree.
+//
+// The layer -> GPU assignment behind qwen4exp's multi-GPU layer split.
+//
+// The properties asserted here are not cosmetic. Each device boundary inside a
+// token is a span cut and a residual hand-off through host memory, so:
+//   * runs must be CONTIGUOUS - an interleaved map costs a seam per interleave;
+//   * device indices must be MONOTONIC - the residual only travels forwards;
+//   * no device may be left empty - that is a seam bought for nothing.
+// And the map must be stable for the model's lifetime: it decides which GPU each
+// weight is uploaded to, and the preload frees the host copy right afterwards.
+using System.Linq;
+using TensorSharp.Models;
+using Xunit;
+
+namespace InferenceWeb.Tests;
+
+public class Qwen4ExpLayerSplitTests
+{
+    private static long[] Uniform(int n, long bytes) => Enumerable.Repeat(bytes, n).ToArray();
+
+    [Fact]
+    public void SingleDevice_PutsEverythingOnGpu0()
+    {
+        int[] map = Qwen4ExpModel.PackLayersOntoDevices(Uniform(48, 1000), 0, 0, 1);
+        Assert.All(map, d => Assert.Equal(0, d));
+    }
+
+    [Theory]
+    [InlineData(2)]
+    [InlineData(3)]
+    [InlineData(4)]
+    public void RunsAreContiguousMonotonicAndCoverEveryDevice(int devices)
+    {
+        int[] map = Qwen4ExpModel.PackLayersOntoDevices(Uniform(48, 1000), 0, 0, devices);
+
+        Assert.Equal(48, map.Length);
+        Assert.Equal(0, map[0]);
+        for (int i = 1; i < map.Length; i++)
+        {
+            // Monotonic, and never skipping a device: both are what makes the
+            // number of seams exactly devices-1.
+            Assert.True(map[i] == map[i - 1] || map[i] == map[i - 1] + 1,
+                $"layer {i} jumped from device {map[i - 1]} to {map[i]}");
+        }
+        Assert.Equal(devices - 1, map[^1]);
+        Assert.Equal(devices, map.Distinct().Count());
+    }
+
+    [Fact]
+    public void SharedBytesShiftLayersOffDevice0()
+    {
+        // Device 0 also carries the token embedding (the head group is charged to
+        // the LAST device instead - see below; the vision tower is not modelled at
+        // all, see BuildLayerDeviceMap). Charging those bytes is the whole point: an
+        // equal LAYER count would make gpu0 the one that runs out of VRAM.
+        int[] even = Qwen4ExpModel.PackLayersOntoDevices(Uniform(48, 1000), 0, 0, 2);
+        int[] loaded = Qwen4ExpModel.PackLayersOntoDevices(Uniform(48, 1000), 12_000, 0, 2);
+
+        int evenOnGpu0 = even.Count(d => d == 0);
+        int loadedOnGpu0 = loaded.Count(d => d == 0);
+        Assert.True(loadedOnGpu0 < evenOnGpu0,
+            $"shared bytes should move layers off gpu0, but it kept {loadedOnGpu0} vs {evenOnGpu0}");
+        Assert.True(loadedOnGpu0 >= 1, "gpu0 must still run at least one layer - it holds the embedding.");
+    }
+
+    [Fact]
+    public void UnevenLayerCostsStillBalanceBytes()
+    {
+        // Real models are not uniform: qwen4exp's dense layers are far smaller than
+        // its 512-expert MoE layers. Balance is on BYTES, not layer count.
+        var bytes = new long[8];
+        for (int i = 0; i < 8; i++) bytes[i] = i < 4 ? 100 : 900;
+
+        int[] map = Qwen4ExpModel.PackLayersOntoDevices(bytes, 0, 0, 2);
+
+        long gpu0 = 0, gpu1 = 0;
+        for (int i = 0; i < 8; i++) { if (map[i] == 0) gpu0 += bytes[i]; else gpu1 += bytes[i]; }
+        long total = gpu0 + gpu1;
+        // Each side within 40% of half: a layer is indivisible, so exact halves are
+        // not reachable, but a count-based split (4/4 = 400/3600) would be far worse.
+        Assert.True(gpu0 > total * 0.1 && gpu1 > total * 0.1,
+            $"bytes are badly balanced: gpu0={gpu0} gpu1={gpu1}");
+    }
+
+    [Fact]
+    public void MoreDevicesThanLayers_LeavesNoDeviceWithoutWork()
+    {
+        // Degenerate but reachable (--tp 4 on a tiny test model). Every device that
+        // appears in the map must own at least one layer; the packing must not
+        // advance past a device it never used.
+        int[] map = Qwen4ExpModel.PackLayersOntoDevices(Uniform(3, 1000), 0, 0, 4);
+        Assert.Equal(3, map.Length);
+        foreach (int d in map.Distinct())
+            Assert.Contains(d, map);
+        for (int i = 1; i < map.Length; i++)
+            Assert.True(map[i] == map[i - 1] || map[i] == map[i - 1] + 1);
+    }
+
+
+    [Fact]
+    public void HeadBytesShiftLayersOffTheLastDevice()
+    {
+        // The final mixer + LM head ride the LAST span, so they are charged to the
+        // LAST device. Charging them to device 0 (as the first cut of this code did)
+        // made the last GPU the one that ran out of VRAM.
+        int[] even = Qwen4ExpModel.PackLayersOntoDevices(Uniform(48, 1000), 0, 0, 2);
+        int[] headed = Qwen4ExpModel.PackLayersOntoDevices(Uniform(48, 1000), 0, 12_000, 2);
+
+        int evenOnLast = even.Count(d => d == 1);
+        int headedOnLast = headed.Count(d => d == 1);
+        Assert.True(headedOnLast < evenOnLast,
+            $"head bytes should move layers off the last gpu, but it kept {headedOnLast} vs {evenOnLast}");
+        Assert.True(headedOnLast >= 1, "the last gpu must still run at least one layer - it holds the head.");
+    }
+
+    [Fact]
+    public void SharedAndHeadBytesPullFromOppositeEnds()
+    {
+        // Both ends loaded: device 0 carries the embedding, the last carries the head,
+        // so the middle devices should take more layers than either end.
+        int[] map = Qwen4ExpModel.PackLayersOntoDevices(Uniform(60, 1000), 15_000, 15_000, 3);
+        int first = map.Count(d => d == 0), mid = map.Count(d => d == 1), last = map.Count(d => d == 2);
+        Assert.Equal(60, first + mid + last);
+        Assert.True(mid > first && mid > last,
+            $"middle device should take the most layers: {first}/{mid}/{last}");
+    }
+
+
+    // ---- TS_Q4E_LAYER_SPLIT override ----------------------------------------
+
+    [Fact]
+    public void Override_Unset_UsesTheAutomaticBalance()
+    {
+        Assert.Null(Qwen4ExpModel.ParseLayerSplitOverride(null, 48, 2));
+        Assert.Null(Qwen4ExpModel.ParseLayerSplitOverride("", 48, 2));
+        Assert.Null(Qwen4ExpModel.ParseLayerSplitOverride("   ", 48, 2));
+    }
+
+    [Fact]
+    public void Override_IgnoredWithoutASplit()
+    {
+        Assert.Null(Qwen4ExpModel.ParseLayerSplitOverride("48", 48, 1));
+    }
+
+    [Fact]
+    public void Override_AssignsExactlyTheRequestedCounts()
+    {
+        int[] map = Qwen4ExpModel.ParseLayerSplitOverride("20,28", 48, 2);
+        Assert.NotNull(map);
+        Assert.Equal(20, map.Count(d => d == 0));
+        Assert.Equal(28, map.Count(d => d == 1));
+        // Still contiguous and monotonic - the seam count must not change.
+        for (int i = 1; i < map.Length; i++)
+            Assert.True(map[i] == map[i - 1] || map[i] == map[i - 1] + 1);
+    }
+
+    [Theory]
+    [InlineData("20,29", 48, 2)]      // counts do not sum to the layer count
+    [InlineData("20", 48, 2)]         // too few devices
+    [InlineData("10,20,18", 48, 2)]   // too many devices
+    [InlineData("0,48", 48, 2)]       // a GPU with no layers is a seam for nothing
+    [InlineData("-1,49", 48, 2)]      // negative
+    [InlineData("twenty,28", 48, 2)]  // unparseable
+    public void Override_RejectsAnythingItCannotHonour(string spec, int layers, int devices)
+    {
+        // Throwing matters: silently falling back to the automatic balance would let
+        // an operator believe they had placed the layers when they had not.
+        Assert.Throws<System.ArgumentException>(
+            () => Qwen4ExpModel.ParseLayerSplitOverride(spec, layers, devices));
+    }
+
+    [Fact]
+    public void EmptyModel_DoesNotThrow()
+    {
+        Assert.Empty(Qwen4ExpModel.PackLayersOntoDevices(System.Array.Empty<long>(), 0, 0, 2));
+    }
+}
diff --git a/InferenceWeb.Tests/ServerOptionsBuilderTests.cs b/InferenceWeb.Tests/ServerOptionsBuilderTests.cs
index ebb058e8..cd247feb 100644
--- a/InferenceWeb.Tests/ServerOptionsBuilderTests.cs
+++ b/InferenceWeb.Tests/ServerOptionsBuilderTests.cs
@@ -9,8 +9,11 @@
 // MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the BSD-3-Clause License for more details.
 
 using System;
+using System.Collections.Generic;
 using System.IO;
+using TensorSharp.Runtime;
 using TensorSharp.Runtime.Scheduling;
+using TensorSharp.Runtime.Speculative;
 using TensorSharp.Server.Hosting;
 
 namespace InferenceWeb.Tests;
@@ -486,6 +489,7 @@ public void ServerUsage_PrintUsage_DocumentsEveryKnownFlag()
             "--paged-kv", "--paged-kv-block-size", "--paged-kv-ram-mb",
             "--paged-kv-ssd-dir", "--paged-kv-ssd-mb", "--paged-kv-quant-bits",
             "--continuous-batching", "--prefill-chunk-size",
+            "--spec", "--spec-draft", "--spec-pmin", "--spec-draft-model",
             "--mtp-spec", "--mtp-draft", "--mtp-pmin", "--mtp-draft-model", "--draft-model",
             "--qwen-image-vae", "--qwen-image-vl", "--qwen-image-mmproj", "--qwen-image-lora",
             "--wan-vae", "--wan-te", "--wan-dit2",
@@ -1277,4 +1281,175 @@ public void Build_NoWebUiFlag_OverridesEnvVarZero()
         var options = ServerOptionsBuilder.Build(new[] { "--no-webui" }, _baseDir);
         Assert.False(options.WebUiEnabled);
     }
+
+    // ---- Usage page vs parser: the whole class of "documented but rejected" ----
+    //
+    // The server applies several flag families in passes that run BEFORE
+    // ServerOptionsBuilder.Build and that READ argv without removing anything.
+    // Build then walks the same argv and throws "Unknown option" for whatever it
+    // does not explicitly recognise. Every such family therefore needs an entry in
+    // Build's skip list, and twice one was missed: --wan-vae/--wan-te (fixed with a
+    // one-off test below), then EVERY --spec* spelling, which made
+    //   TensorSharp.Server --model m.gguf --draft-model d.gguf --mtp-spec --spec-draft 3
+    // die with `Unknown option '--spec-draft'` even though --help documents it.
+    //
+    // These tests close the class instead of the instance: they enumerate the usage
+    // page itself, so a flag can never again be documented-but-rejected, or
+    // accepted-but-unsuggestible, without a red test.
+
+    /// <summary>A plausible value for a flag, keyed on the placeholder its usage
+    /// entry declares. Files must exist because several appliers stat them.</summary>
+    private string[] SampleArgsFor(string flag, string usage)
+    {
+        string filePath = Path.Combine(_baseDir, "sample.gguf");
+        if (!File.Exists(filePath)) File.WriteAllBytes(filePath, new byte[] { 1, 2, 3, 4 });
+
+        // Value flags are the ones the usage page renders as "<flag> <placeholder>".
+        int idx = usage.IndexOf(flag + " <", StringComparison.Ordinal);
+        if (idx < 0)
+            return new[] { flag };                       // bare switch
+
+        int lt = idx + flag.Length + 1;
+        int gt = usage.IndexOf('>', lt);
+        string placeholder = gt > lt ? usage.Substring(lt + 1, gt - lt - 1) : string.Empty;
+
+        string value = placeholder switch
+        {
+            "path" or "path|none" or "dir" => filePath,
+            "url" => "localhost:6379",
+            "t" => "f16",
+            "name" => "ngram",
+            "type" => "ggml_cpu",
+            "mode" => "ref",
+            "config|request" => "config",
+            "list" => "10.0.0.1:9500",
+            "text" => "</s>",
+            "address" => "127.0.0.1",
+            "urls" => "http://0.0.0.0:18099",
+            "f" or "p" or "x" => "0.5",
+            _ => "1",
+        };
+        return new[] { flag, value };
+    }
+
+    [Fact]
+    public void Build_AcceptsEveryFlagOnTheUsagePage()
+    {
+        var sw = new StringWriter();
+        ServerUsage.PrintUsage(sw);
+        string usage = sw.ToString();
+
+        var rejected = new List<string>();
+        int checkedFlags = 0;
+        foreach (string flag in ServerUsage.DocumentedFlags())
+        {
+            // --config is consumed and REMOVED by ConfigFileArgs.Expand before
+            // Build ever sees it, so Build legitimately does not know it.
+            if (flag == "--config") continue;
+            checkedFlags++;
+
+            using var scope = new EnvScope();
+            string[] args = SampleArgsFor(flag, usage);
+            Exception ex = Record.Exception(() =>
+            {
+                ServerOptionsBuilder.ApplySpeculativeCliFlags(args);
+                ServerOptionsBuilder.Build(args, _baseDir);
+            });
+            if (ex is ArgumentException ae &&
+                ae.Message.StartsWith("Unknown option", StringComparison.Ordinal))
+            {
+                rejected.Add(flag + " -> " + ae.Message);
+            }
+        }
+
+        // Guard against a vacuous pass: an accessor that yielded nothing would
+        // otherwise make this test green while checking nothing at all.
+        Assert.True(checkedFlags > 40, $"DocumentedFlags() yielded only {checkedFlags} flags.");
+        Assert.True(rejected.Count == 0,
+            "These flags are on the --help page but ServerOptionsBuilder.Build rejects them:\n  "
+            + string.Join("\n  ", rejected));
+    }
+
+    [Fact]
+    public void SuggestFlagCorrection_KnowsEverySpeculativeSpelling()
+    {
+        // A typo near a real flag must suggest THAT flag. Before the fix "--spe"
+        // suggested "--seed" (Levenshtein 2) because no --spec* name was in the
+        // known-flag table at all - an actively misleading hint.
+        foreach (string flag in SpeculativeCliFlags.SwitchFlags)
+        {
+            string typo = flag.Substring(0, flag.Length - 1);
+            var ex = Assert.Throws<ArgumentException>(
+                () => ServerOptionsBuilder.Build(new[] { typo }, _baseDir));
+            Assert.Contains("Did you mean '" + flag + "'", ex.Message);
+        }
+    }
+
+    [Theory]
+    [InlineData("--spec")]
+    [InlineData("--no-spec")]
+    [InlineData("--mtp-spec")]
+    [InlineData("--no-mtp-spec")]
+    [InlineData("--spec-draft")]
+    [InlineData("--mtp-draft")]
+    [InlineData("--spec-type")]
+    [InlineData("--mtp-type")]
+    [InlineData("--spec-pmin")]
+    [InlineData("--mtp-pmin")]
+    [InlineData("--spec-draft-model")]
+    [InlineData("--mtp-draft-model")]
+    [InlineData("--draft-model")]
+    public void Build_SpeculativeFlags_SurviveBothPasses(string flag)
+    {
+        string draft = Path.Combine(_baseDir, "draft.gguf");
+        File.WriteAllBytes(draft, new byte[] { 1, 2, 3, 4 });
+
+        string[] args = flag switch
+        {
+            "--spec" or "--no-spec" or "--mtp-spec" or "--no-mtp-spec" => new[] { flag },
+            "--spec-draft" or "--mtp-draft" => new[] { flag, "3" },
+            "--spec-type" or "--mtp-type" => new[] { flag, "ngram" },
+            "--spec-pmin" or "--mtp-pmin" => new[] { flag, "0.6" },
+            _ => new[] { flag, draft },
+        };
+
+        // Pass 1: the applier consumes it (--draft-model is server-local and
+        // handled by ApplySpeculativeCliFlags' own extra branch, not the tables).
+        ServerOptionsBuilder.ApplySpeculativeCliFlags(args);
+
+        // Pass 2: the unknown-arg trap must not trip on that very same argv.
+        var ex = Record.Exception(() => ServerOptionsBuilder.Build(args, _baseDir));
+        Assert.False(
+            ex is ArgumentException ae && ae.Message.StartsWith("Unknown option", StringComparison.Ordinal),
+            flag + " was consumed by ApplySpeculativeCliFlags but rejected by Build: " + ex?.Message);
+
+        // And the "=" spelling, which takes TryReadOne's prefix branch.
+        if (args.Length == 2)
+        {
+            string[] eqArgs = { flag + "=" + args[1] };
+            ServerOptionsBuilder.ApplySpeculativeCliFlags(eqArgs);
+            var ex2 = Record.Exception(() => ServerOptionsBuilder.Build(eqArgs, _baseDir));
+            Assert.False(
+                ex2 is ArgumentException ae2 && ae2.Message.StartsWith("Unknown option", StringComparison.Ordinal),
+                flag + "=VALUE was rejected by Build: " + ex2?.Message);
+        }
+    }
+
+    [Fact]
+    public void ConfigFile_SpecKeys_StartTheServer()
+    {
+        // A --config file naming the current spellings must work end to end:
+        // ConfigFileArgs.Expand turns {"spec-draft": 3} into --spec-draft 3, which
+        // then has to survive Build. This is the surface the shipped configs use.
+        string cfg = Path.Combine(_baseDir, "spec.json");
+        File.WriteAllText(cfg, "{ \"spec\": true, \"spec-draft\": 3, \"spec-pmin\": 0.6 }");
+
+        string[] expanded = ConfigFileArgs.Expand(new[] { "--config", cfg });
+        ServerOptionsBuilder.ApplySpeculativeCliFlags(expanded);
+        var ex = Record.Exception(() => ServerOptionsBuilder.Build(expanded, _baseDir));
+        Assert.False(
+            ex is ArgumentException ae && ae.Message.StartsWith("Unknown option", StringComparison.Ordinal),
+            "config-file spec keys were rejected: " + ex?.Message);
+    }
+
 }
diff --git a/InferenceWeb.Tests/TensorParallelSupportGateTests.cs b/InferenceWeb.Tests/TensorParallelSupportGateTests.cs
new file mode 100644
index 00000000..6ca4cb1f
--- /dev/null
+++ b/InferenceWeb.Tests/TensorParallelSupportGateTests.cs
@@ -0,0 +1,144 @@
+// Copyright (c) Zhongkai Fu. All rights reserved.
+// https://github.com/zhongkaifu/TensorSharp
+//
+// This file is part of TensorSharp.
+//
+// TensorSharp is licensed under the BSD-3-Clause license found in the LICENSE file in the root directory of this source tree.
+//
+// How `--tp N` is resolved per architecture.
+//
+// Two regressions live here. First: `--tp 2` on an architecture with no
+// tensor-parallel implementation used to be accepted in full silence - a real
+// multi-GPU context and NCCL group were built, the banner announced "Tensor
+// parallelism: 2 GPUs", and then every weight was uploaded through rank 0 and
+// the model ran on GPU 0, because sharding is opt-in per model class and
+// qwen4exp never opted in. Second: refusing outright then threw the second GPU
+// away for an architecture that CAN use it - just not by sharding. qwen4exp now
+// resolves --tp N to a LAYER SPLIT (each GPU holds a contiguous run of whole
+// layers), which is the same and only multi-GPU mode llama.cpp offers for it.
+using System;
+using TensorSharp;
+using Xunit;
+
+namespace InferenceWeb.Tests;
+
+public class TensorParallelSupportGateTests
+{
+    private static int Resolve(string arch, BackendType backend, int tpDegree,
+        ref ITensorParallelGroup group, out int layerSplit)
+        => TensorSharp.Models.ModelBase.ResolveTensorParallelSupport(
+            arch, backend, tpDegree, ref group, out layerSplit);
+
+    [Fact]
+    public void LayerSplitArchitecture_ResolvesToASplit_NotTensorParallelism()
+    {
+        ITensorParallelGroup group = null;
+        int tp = Resolve("qwen4exp", BackendType.GgmlCuda, 2, ref group, out int layerSplit);
+
+        // No tensor-parallel group: IsTensorParallel gates weight sharding and the
+        // AllReduce machinery, none of which a layer split uses.
+        Assert.Equal(1, tp);
+        Assert.Null(group);
+        // ...but both GPUs are used, by layers.
+        Assert.Equal(2, layerSplit);
+    }
+
+    [Fact]
+    public void LayerSplit_OnlyOnBackendsThatHaveSeveralDevices()
+    {
+        // ggml_cpu exposes one device; there is nothing to split across, so this
+        // must fall back to the loud single-GPU degrade rather than claim a split.
+        ITensorParallelGroup group = null;
+        int tp = Resolve("qwen4exp", BackendType.GgmlCpu, 2, ref group, out int layerSplit);
+        Assert.Equal(1, tp);
+        Assert.Equal(1, layerSplit);
+    }
+
+    [Fact]
+    public void DistributedTpOnUnsupportedArchitecture_Throws()
+    {
+        // A distributed group cannot be downgraded on one node: the other nodes
+        // would still be waiting on collectives this rank will never issue. A
+        // layer split is single-process, so it is not an answer here either.
+        ITensorParallelGroup group = new StubTpGroup();
+        var ex = Assert.Throws<NotSupportedException>(
+            () => Resolve("qwen4exp", BackendType.GgmlCuda, 2, ref group, out _));
+        Assert.Contains("qwen4exp", ex.Message, StringComparison.OrdinalIgnoreCase);
+    }
+
+    [Theory]
+    [InlineData("qwen35")]
+    [InlineData("gemma4")]
+    [InlineData("muse-glimmer")]
+    [InlineData("glm-dsa")]
+    [InlineData("deepseek4")]   // multi-GPU through its own executor, not the TP group
+    public void TpCapableArchitectures_AreUntouched(string arch)
+    {
+        ITensorParallelGroup group = null;
+        Assert.Equal(4, Resolve(arch, BackendType.GgmlCuda, 4, ref group, out int layerSplit));
+        Assert.Equal(1, layerSplit);
+    }
+
+    [Fact]
+    public void NoTpRequested_IsAlwaysAPassthrough()
+    {
+        // The gate must not fire on ordinary single-GPU runs of the very
+        // architectures it knows about.
+        ITensorParallelGroup group = null;
+        Assert.Equal(1, Resolve("qwen4exp", BackendType.GgmlCuda, 1, ref group, out int layerSplit));
+        Assert.Equal(1, layerSplit);
+        Assert.Null(group);
+    }
+
+    [Fact]
+    public void UnknownArchitecture_IsNotBlocked()
+    {
+        // The table is a deny-list of known-unsupported architectures, not an
+        // allow-list: a new arch must not be refused just for being absent.
+        ITensorParallelGroup group = null;
+        Assert.Equal(2, Resolve("brand-new-arch", BackendType.GgmlCuda, 2, ref group, out _));
+    }
+
+    [Fact]
+    public void EveryEntryExplainsItself()
+    {
+        // The message is the whole value of the gate - it is what tells the
+        // operator why the second GPU is idle. An empty one is a bug.
+        Assert.NotEmpty(TensorSharp.Models.ModelBase.ArchitecturesWithoutTensorParallel);
+        foreach (var kv in TensorSharp.Models.ModelBase.ArchitecturesWithoutTensorParallel)
+        {
+            Assert.False(string.IsNullOrWhiteSpace(kv.Value), $"'{kv.Key}' has no explanation.");
+            Assert.Contains(kv.Key, kv.Value, StringComparison.OrdinalIgnoreCase);
+        }
+    }
+
+    [Fact]
+    public void EveryLayerSplitArchitectureIsAlsoDeclaredNonTensorParallel()
+    {
+        // The split list is consulted only after the no-TP list matches. An arch in
+        // one and not the other would silently never split.
+        foreach (string arch in TensorSharp.Models.ModelBase.ArchitecturesWithLayerSplit)
+        {
+            Assert.True(TensorSharp.Models.ModelBase.ArchitecturesWithoutTensorParallel.ContainsKey(arch),
+                $"'{arch}' can layer-split but is not listed as lacking tensor parallelism, so the "
+                + "split branch is unreachable for it.");
+        }
+    }
+
+    /// <summary>Minimal live group: the gate only reads whether one exists.</summary>
+    private sealed class StubTpGroup : ITensorParallelGroup
+    {
+        public int Degree => 2;
+        public bool IsActive => true;
+        public int GlobalDegree => 2;
+        public int GlobalRankOffset => 0;
+        public int NodeCount => 2;
+        public IAllocator GetAllocator(int rank) => throw new NotSupportedException();
+        public void AllReduce(Tensor[] tensors) => throw new NotSupportedException();
+        public void Synchronize() { }
+        public void Barrier() { }
+        public void BroadcastControl(int op, int[] payload) => throw new NotSupportedException();
+        public (int op, int[] payload) ReceiveControl() => throw new NotSupportedException();
+        public void Dispose() { }
+    }
+}
diff --git a/MODEL_DOWNLOADS.md b/MODEL_DOWNLOADS.md
index 4796e04d..ef9b7e0f 100644
--- a/MODEL_DOWNLOADS.md
+++ b/MODEL_DOWNLOADS.md
@@ -19,14 +19,16 @@ TensorSharp loads models in GGUF format. Below are verified Hugging Face repos f
 | Qwen 3.5 / 3.6 family | Qwen3.5-9B | [unsloth/Qwen3.5-9B-GGUF](https://huggingface.co/unsloth/Qwen3.5-9B-GGUF) — mmproj `mmproj-F16.gguf` in the same repo |
 | Qwen 3.5 / 3.6 family | Qwen3.5-35B-A3B (MoE) | [ggml-org/Qwen3.5-35B-A3B-GGUF](https://huggingface.co/ggml-org/Qwen3.5-35B-A3B-GGUF) — mmproj `mmproj-Qwen3.5-35B-A3B-Q8_0.gguf` in the same repo |
 | Qwen 3.5 / 3.6 family | Qwen3.6-35B-A3B (MoE, embedded NextN MTP) | [unsloth/Qwen3.6-35B-A3B-MTP-GGUF](https://huggingface.co/unsloth/Qwen3.6-35B-A3B-MTP-GGUF) — these GGUFs retain the NextN block for the server's `--mtp-spec`; mmproj `mmproj-F16.gguf` in the same repo. The base repo [unsloth/Qwen3.6-35B-A3B-GGUF](https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF) ships the same file names with NextN stripped — those load fine but silently fall back to standard decode |
+| Qwen 3.8 Flash Next | Qwen3.8-Flash-Next (hybrid MoE, image-capable) | [unsloth/Qwen3.8-Flash-Next-GGUF](https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF) — one subdirectory per quant (`UD-Q2_K_XL/`, …), each a multi-shard set; point `--model` at the `-00001-of-` shard. `mmproj-BF16.gguf` beside the model enables image input, multi-image prompts and multi-turn image sessions included. `general.architecture` = `qwen4exp`. On a multi-GPU box `--tp N` runs a **layer split** — whole layers per GPU, the same (and only) multi-GPU mode llama.cpp offers this architecture — which buys capacity, not speed; see [USAGE.md](USAGE.md#tensor-parallelism--distributed-inference) |
 | GPT OSS | gpt-oss-20b (MoE) | [ggml-org/gpt-oss-20b-GGUF](https://huggingface.co/ggml-org/gpt-oss-20b-GGUF) — `gpt-oss-20b-MXFP4.gguf` (note the uppercase `MXFP4`), text only, no companion files |
 | Nemotron-H | Nemotron-H-8B-Reasoning-128K | [bartowski/nvidia_Nemotron-H-8B-Reasoning-128K-GGUF](https://huggingface.co/bartowski/nvidia_Nemotron-H-8B-Reasoning-128K-GGUF) |
 | Nemotron-H | Nemotron-H-47B-Reasoning-128K | [bartowski/nvidia_Nemotron-H-47B-Reasoning-128K-GGUF](https://huggingface.co/bartowski/nvidia_Nemotron-H-47B-Reasoning-128K-GGUF) |
 | Nemotron-H | Nemotron 3 Nano Omni 30B-A3B (image-capable) | [unsloth/NVIDIA-Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF](https://huggingface.co/unsloth/NVIDIA-Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF) — mmproj `mmproj-BF16.gguf` (same repo) is required for image input. Audio is preprocessed only: real audio inference needs a Parakeet audio mmproj these GGUFs do not ship |
 | Mistral 3 | Mistral-Small-3.1-24B-Instruct-2503 | [bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF](https://huggingface.co/bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF) — Pixtral mmproj `mmproj-mistralai_Mistral-Small-3.1-24B-Instruct-2503-f16.gguf` in the same repo |
-| Muse-Glimmer | Muse-Glimmer-30B (dense, image-capable) | [unsloth/Muse-Glimmer-30B-GGUF](https://huggingface.co/unsloth/Muse-Glimmer-30B-GGUF) — e.g. `Muse-Glimmer-30B-UD-Q4_K_XL.gguf` or `Muse-Glimmer-30B-Q8_0.gguf`; `general.architecture` = `muse-glimmer` / `muse_glimmer`. Image input requires `mmproj-Muse-Glimmer-30B-Q8_0.gguf` (same repo) passed **explicitly** with `--mmproj` — this is the one family with no mmproj auto-detection. Optional speed artifact: the DFlash block drafter `dflash-kquant.gguf` (same repo) loaded with `--draft-model` for lossless speculative decoding — pass no sampler flags, it needs plain greedy |
+| Muse-Glimmer | Muse-Glimmer-30B (dense, image-capable) | [unsloth/Muse-Glimmer-30B-GGUF](https://huggingface.co/unsloth/Muse-Glimmer-30B-GGUF) — e.g. `Muse-Glimmer-30B-UD-Q4_K_XL.gguf` or `Muse-Glimmer-30B-Q8_0.gguf`; `general.architecture` = `muse-glimmer` / `muse_glimmer`. Image input requires `mmproj-Muse-Glimmer-30B-Q8_0.gguf` (same repo) passed **explicitly** with `--mmproj` — this is the one family with no mmproj auto-detection. Optional speed artifacts: the DFlash block drafter `dflash-kquant.gguf` (same repo) or the newer DFlash2 drafter [z-lab/Muse-Glimmer-30B-DFlash2-GGUF](https://huggingface.co/z-lab/Muse-Glimmer-30B-DFlash2-GGUF) (prefer `-Q4_K_M` on a 16 GB card — see the note on drafter size in [speculative_decoding.md](docs/speculative_decoding.md#what-to-expect)), loaded with `--draft-model` for lossless speculative decoding — pass no sampler flags, it needs plain greedy |
 | DeepSeek V4 | DeepSeek-V4-Flash-0731 (284B MoE) | [unsloth/DeepSeek-V4-Flash-0731-GGUF](https://huggingface.co/unsloth/DeepSeek-V4-Flash-0731-GGUF) — one subdirectory per quant (`UD-Q8_K_XL/`, `UD-IQ4_XS/`, `UD-IQ1_S/`, …), each a multi-shard set; point `--model` at the `-00001-of-` shard. Text only |
-| GLM 5.x | GLM-5.2 (744B-A40B MoE, embedded NextN MTP) | [unsloth/GLM-5.2-GGUF](https://huggingface.co/unsloth/GLM-5.2-GGUF) — one subdirectory per quant (`UD-Q4_K_XL/`, `UD-IQ2_XXS/`, …), each a multi-shard set; point `--model` at the `-00001-of-` shard. Text only. These GGUFs already carry the NextN block for the server's `--mtp-spec` — unlike Qwen 3.6 there is no separate MTP repo to pick |
+| GLM 5.x | GLM-5.2 (744B-A40B MoE, embedded NextN MTP) | [unsloth/GLM-5.2-GGUF](https://huggingface.co/unsloth/GLM-5.2-GGUF) — one subdirectory per quant (`UD-Q4_K_XL/`, `UD-IQ2_XXS/`, …), each a multi-shard set; point `--model` at the `-00001-of-` shard. **Text only** — GLM-5.3-Flash in the next row is the one that takes images. These GGUFs already carry the NextN block for the server's `--mtp-spec` — unlike Qwen 3.6 there is no separate MTP repo to pick |
+| GLM 5.x | GLM-5.3-Flash (320B, 288 routed experts, text + image) | [unsloth/GLM-5.3-Flash-GGUF](https://huggingface.co/unsloth/GLM-5.3-Flash-GGUF) — one subdirectory per quant (`UD-Q2_K_XL/`, …), each a multi-shard set; point `--model` at the `-00001-of-` shard. `general.architecture` = `glm5next`, and it loads through the same native executor as GLM-5.2. Unlike 5.2 it **takes images**: `mmproj-BF16.gguf` (the GLM-OCR ViT, same repo) enables `--image`, multi-image prompts and multi-turn image sessions. Its NextN block is not wired up yet, so there is no `--mtp-spec` here, and `--tp` is cleanly refused — use the default layer split across every visible GPU |
 | DeepSeek V4 | DSpark speculative drafters (optional — speed only) | see [DSpark drafters](#dspark-drafters) below — a separate GGUF loaded with `--draft-model` for ~1.3-1.4x decode |
 | DiffusionGemma | diffusiongemma-26B-A4B-it | [unsloth/diffusiongemma-26B-A4B-it-GGUF](https://huggingface.co/unsloth/diffusiongemma-26B-A4B-it-GGUF) (`general.architecture` = `diffusion-gemma`) |
 | Qwen-Image-Edit | MMDiT DiT (the `--model` GGUF) | [unsloth/Qwen-Image-Edit-2511-GGUF](https://huggingface.co/unsloth/Qwen-Image-Edit-2511-GGUF) (e.g. `qwen-image-edit-2511-Q4_K_M.gguf`; `general.architecture` = `qwen_image`) |
diff --git a/MODEL_DOWNLOADS_zh-cn.md b/MODEL_DOWNLOADS_zh-cn.md
index d0f874fb..9dda128e 100644
--- a/MODEL_DOWNLOADS_zh-cn.md
+++ b/MODEL_DOWNLOADS_zh-cn.md
@@ -17,13 +17,15 @@ TensorSharp 使用 GGUF 格式模型文件。以下是各架构对应的已核
 | Qwen 3.5 | Qwen3.5-9B | [unsloth/Qwen3.5-9B-GGUF](https://huggingface.co/unsloth/Qwen3.5-9B-GGUF)，投影器 `mmproj-F16.gguf` |
 | Qwen 3.5 | Qwen3.5-35B-A3B | [ggml-org/Qwen3.5-35B-A3B-GGUF](https://huggingface.co/ggml-org/Qwen3.5-35B-A3B-GGUF)，投影器 `mmproj-Qwen3.5-35B-A3B-Q8_0.gguf` |
 | Qwen 3.6 | Qwen3.6-35B-A3B（保留 NextN） | [unsloth/Qwen3.6-35B-A3B-MTP-GGUF](https://huggingface.co/unsloth/Qwen3.6-35B-A3B-MTP-GGUF)，投影器 `mmproj-F16.gguf`。**注意不要下载基础仓库** [unsloth/Qwen3.6-35B-A3B-GGUF](https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF)：它的文件名完全相同，但剥离了 NextN 块，`--mtp-spec` 会静默回落到普通解码 |
+| Qwen 3.8 Flash Next | Qwen3.8-Flash-Next（混合 MoE，支持图像） | [unsloth/Qwen3.8-Flash-Next-GGUF](https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF)；每种量化一个子目录（`UD-Q2_K_XL/` 等），均为多分片，`--model` 指向 `-00001-of-` 分片。模型旁的 `mmproj-BF16.gguf` 启用图像输入，多图提示与多轮图像会话都可用。`general.architecture` 为 `qwen4exp`。多卡机器上 `--tp N` 走的是**按层切分**——整层落在单卡，也是 llama.cpp 对这个架构唯一提供的多卡模式——买到的是容量而不是速度，见 [USAGE_zh-cn.md](USAGE_zh-cn.md#张量并行与分布式推理) |
 | GPT OSS | gpt-oss-20b（MoE） | [ggml-org/gpt-oss-20b-GGUF](https://huggingface.co/ggml-org/gpt-oss-20b-GGUF)，文件 `gpt-oss-20b-MXFP4.gguf`（注意 `MXFP4` 为大写）；纯文本，无伴随文件 |
 | Nemotron-H | Nemotron-H-8B / 47B Reasoning | [8B](https://huggingface.co/bartowski/nvidia_Nemotron-H-8B-Reasoning-128K-GGUF) / [47B](https://huggingface.co/bartowski/nvidia_Nemotron-H-47B-Reasoning-128K-GGUF) |
 | Nemotron-H | Nemotron 3 Nano Omni 30B-A3B | [unsloth/NVIDIA-Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF](https://huggingface.co/unsloth/NVIDIA-Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF)，图像输入需 `mmproj-BF16.gguf`；仓库未附真实音频推理需要的 Parakeet mmproj |
 | Mistral 3 | Mistral-Small-3.1-24B-Instruct | [bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF](https://huggingface.co/bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF)，Pixtral 投影器 `mmproj-mistralai_Mistral-Small-3.1-24B-Instruct-2503-f16.gguf` |
 | Muse-Glimmer | Muse-Glimmer-30B（稠密，支持图像） | [unsloth/Muse-Glimmer-30B-GGUF](https://huggingface.co/unsloth/Muse-Glimmer-30B-GGUF)，如 `Muse-Glimmer-30B-UD-Q4_K_XL.gguf` 或 `Muse-Glimmer-30B-Q8_0.gguf`；`general.architecture` 为 `muse-glimmer` / `muse_glimmer`。图像输入需同仓库的 `mmproj-Muse-Glimmer-30B-Q8_0.gguf`，且必须**显式**用 `--mmproj` 指定——这是唯一没有 mmproj 自动探测的系列。可选提速产物：同仓库的 DFlash 分块 draft `dflash-kquant.gguf`，用 `--draft-model` 加载即可无损推测解码——不要传任何采样参数，它只在纯贪心下生效 |
 | DeepSeek V4 | DeepSeek-V4-Flash-0731（284B MoE） | [unsloth/DeepSeek-V4-Flash-0731-GGUF](https://huggingface.co/unsloth/DeepSeek-V4-Flash-0731-GGUF)；每种量化一个子目录（`UD-Q8_K_XL/`、`UD-IQ4_XS/` 等），均为多分片，`--model` 指向 `-00001-of-` 分片。仅文本 |
-| GLM 5.x | GLM-5.2（744B-A40B MoE，内嵌 NextN MTP） | [unsloth/GLM-5.2-GGUF](https://huggingface.co/unsloth/GLM-5.2-GGUF)；每种量化一个子目录（`UD-Q4_K_XL/`、`UD-IQ2_XXS/` 等），均为多分片，`--model` 指向 `-00001-of-` 分片。仅文本。这些 GGUF 已带有服务端 `--mtp-spec` 所需的 NextN 块——与 Qwen 3.6 不同，不存在需要挑选的独立 MTP 仓库 |
+| GLM 5.x | GLM-5.2（744B-A40B MoE，内嵌 NextN MTP） | [unsloth/GLM-5.2-GGUF](https://huggingface.co/unsloth/GLM-5.2-GGUF)；每种量化一个子目录（`UD-Q4_K_XL/`、`UD-IQ2_XXS/` 等），均为多分片，`--model` 指向 `-00001-of-` 分片。**仅文本**——下一行的 GLM-5.3-Flash 才是支持图像的那个。这些 GGUF 已带有服务端 `--mtp-spec` 所需的 NextN 块——与 Qwen 3.6 不同，不存在需要挑选的独立 MTP 仓库 |
+| GLM 5.x | GLM-5.3-Flash（320B，288 个路由专家，文本 + 图像） | [unsloth/GLM-5.3-Flash-GGUF](https://huggingface.co/unsloth/GLM-5.3-Flash-GGUF)；每种量化一个子目录（`UD-Q2_K_XL/` 等），均为多分片，`--model` 指向 `-00001-of-` 分片。`general.architecture` 为 `glm5next`，与 GLM-5.2 走同一个原生执行器。与 5.2 不同，它**支持图像**：同仓库的 `mmproj-BF16.gguf`（GLM-OCR ViT）启用 `--image`、多图提示与多轮图像会话。它的 NextN 块尚未接入，因此这里没有 `--mtp-spec`；`--tp` 也会被明确拒绝——请用默认的按层切分摊到所有可见 GPU |
 | DeepSeek V4 | DSpark 推测解码 draft（可选，仅提速） | 见下方 [DSpark draft 模型](#dspark-draft-模型)，用 `--draft-model` 加载，解码约 1.3-1.4 倍 |
 | DiffusionGemma | diffusiongemma-26B-A4B-it | [unsloth/diffusiongemma-26B-A4B-it-GGUF](https://huggingface.co/unsloth/diffusiongemma-26B-A4B-it-GGUF)，如 `diffusiongemma-26B-A4B-it-Q4_K_M.gguf` |
 | Qwen-Image-Edit | MMDiT DiT（必需） | [unsloth/Qwen-Image-Edit-2511-GGUF](https://huggingface.co/unsloth/Qwen-Image-Edit-2511-GGUF)，如 `qwen-image-edit-2511-Q4_K_M.gguf` |
diff --git a/README.md b/README.md
index 7877fdcc..a47e2ade 100644
--- a/README.md
+++ b/README.md
@@ -26,8 +26,10 @@
 - **🚀 Continuous batching & paged KV cache.** vLLM-style paged KV pool with block-hash prefix sharing and an iteration-level scheduler, on by default in the server. → [deep dive](docs/PAGED_ATTENTION_AND_CONTINUOUS_BATCHING.md)
 - **🧬 DeepSeek V4 Flash (284B MoE) with three whole-model executors.** The compressed-sparse-attention 1M-context architecture runs on a direct-CUDA engine (`--backend cuda`), the native ggml executor (`--backend ggml_cuda` / `ggml_vulkan`), *and* a 100% pure-C# CPU executor (`--backend cpu`, no native dependencies). Weights layer-split automatically across every visible GPU, so a model far larger than one card still runs; the server hosts it with per-sequence slots and continuous batching. → [DeepSeek V4 card](docs/models/deepseek4.md)
 - **🧠 GLM-5.2 (744B-A40B MoE) with tensor parallelism and CPU MoE offload.** Multi-head Latent Attention plus a DeepSeek Sparse Attention "lightning indexer" that picks which 2048 cached tokens each query may attend to. `--tp N` runs every layer on every GPU (heads column/row-parallel, every expert split row-wise) and `--cpu-moe` keeps the routed experts — 92% of the checkpoint — in system RAM. The default layer split and `--cpu-moe` reproduce llama.cpp token-for-token on the same backend; `--tp` sums per-rank partials, so on a 2-bit MoE its last-bit difference reaches the top-8 router and the near-tied tokens can differ. Head-to-head on 3x RTX PRO 6000: **pp2048 918.9 vs llama.cpp's 763.1 tok/s**, tg64 43.7 vs 42.2. The advertised 1M context (~93 GiB of KV) is a ceiling rather than a promise — once the weights land the loader sizes the context to the VRAM actually free and logs its pick (342,272 tokens on the layer split, 646,400 with `--n-cpu-moe 30`); `MAX_CONTEXT` makes a specific length a hard requirement instead. → [GLM card](docs/models/glm.md)
+- **🧠 GLM-5.3-Flash (320B MoE) decodes 2.0× llama.cpp.** The hybrid successor — 288 routed experts, KDA linear attention on 34 of 45 trunk layers, NoPE MLA + DSA on the other 11 with a *pooled* indexer (top-k 2048 over 4-cell pools, then expanded to their members), Sinkhorn hyper-connections ×4 — loads through the same native executor and the same `GlmDsaModel` as GLM-5.2. On 2× RTX PRO 6000 Blackwell (96 GB), GLM-5.3-Flash-UD-Q2_K_XL (101 GiB), layer split, both engines at `n_ubatch` 2048 and back to back: **tg64 73.5 vs llama.cpp's 36.6 tok/s**, with prefill within a few percent either way (pp2048 2014 vs 2070, pp16384 1692 vs 1690, pp32768 1446 vs 1483). Vision through `mmproj-BF16.gguf` (the GLM-OCR ViT): `--image`, multi-image and multi-turn image sessions. The layer split across every visible GPU, `--cpu-moe` / `--n-cpu-moe` and per-sequence native slots all work; `--tp` tensor parallelism is cleanly refused (use the layer split) and NextN/MTP speculation is not implemented yet. → [GLM card](docs/models/glm.md)
+- **⚡ Qwen 3.8 Flash Next — the whole token as one graph.** A hybrid MoE: GatedDeltaNet recurrent layers on 36 of 48 layers interleaved with full-attention layers (some behind Qwen Sparse Attention's indexer), a PLE n-gram embedding block, ×4 hyper-connection streams and 512 experts with 10 used. Embedding, in-graph PLE, all 48 layers, the final mixer and the LM head run as (almost) **one captured graph per token**, from a shape-keyed cache; vision rides the Qwen3.5-VL tower with (T,H,W) IMRoPE, including multi-image and multi-turn image sessions with KV reuse across turns (extend-only — the GDN recurrence cannot rewind). `--tp N` runs a **layer split** here, not tensor parallelism: on 2× A100-80GB, Qwen3.8-Flash-Next-UD-Q2_K_XL (73.4 GiB) lands 24.2 GB + 26.2 GB across the two cards with prefill ~1520–1550 t/s and decode ~56 tok/s either way, and greedy output byte-identical to the 1-GPU run — capacity, not speed. → [Qwen 3.8 Flash Next card](docs/models/qwen38-flash-next.md)
 - **🔮 Speculative decoding — four algorithms over one draft-verify runtime.** Multi-token-prediction draft heads accelerate solo decode on Qwen 3.6 (NextN block embedded in the trunk — use an MTP-retaining GGUF such as [unsloth/Qwen3.6-35B-A3B-MTP-GGUF](https://huggingface.co/unsloth/Qwen3.6-35B-A3B-MTP-GGUF); the base repo ships the same file names with the block stripped), **GLM 5.2** (its NextN block ships in the stock checkpoint — **~1.3× decode**, 94% draft acceptance, on 2× RTX PRO 6000 with `--n-cpu-moe 20`) and Gemma 4 (separate `gemma4-assistant` draft GGUF, `--spec-draft-model`); DeepSeek V4 adds **DSpark** block drafting (`--draft-model`), which proposes a whole block of tokens per step for **1.3–1.4× decode** (up to 2.0× on multi-turn chat). A fourth algorithm needs no trained weights at all: `--spec-type ngram` matches the sequence's own suffix against the tokens it has already seen, works on **every** checkpoint, and measured **45.2 tok/s against 31.4 plain (1.44×)** on Qwen3.5-9B (Q8_0, `ggml_metal`, M5 Pro) — a model that ships no draft head — with byte-identical output. In every case the draft proposes, the trunk verifies in one batched forward, and the output matches standard decode. Off by default; opt in with `--spec` on either host (the historical `--mtp-*` spellings are accepted as aliases). → [Speculative decoding](FEATURES.md#speculative-decoding)
-- **🔗 Tensor parallelism & distributed clustering.** Split a model across multiple GPUs with `--tp N` — on the direct `cuda` backend **and** on GGML CUDA / Vulkan — and extend across machines with peer-to-peer TCP clustering (`--tp-node-id` / `--tp-peers`). Megatron-LM column/row-parallel pattern with hierarchical AllReduce; MoE expert parallelism and per-rank GatedDeltaNet kernels on GGML. Fused per-rank execution makes `--tp 2` decode **1.39×** a single GPU on Gemma 4 E4B and **1.57×** on Muse-Glimmer 30B (which also gains **1.34×** prefill — the one model that beats a single GPU on both phases), and runs models that do not fit one card at all (Qwen 3.5-35B-A3B; Muse-Glimmer 30B Q8_0 at 28.2 GB on 24 GB cards). Optional Redis-backed KV cache and Responses API store. → [Tensor Parallelism](USAGE.md#tensor-parallelism--distributed-inference)
+- **🔗 Tensor parallelism & distributed clustering.** Split a model across multiple GPUs with `--tp N` — on the direct `cuda` backend **and** on GGML CUDA / Vulkan — and extend across machines with peer-to-peer TCP clustering (`--tp-node-id` / `--tp-peers`). Megatron-LM column/row-parallel pattern with hierarchical AllReduce; MoE expert parallelism and per-rank GatedDeltaNet kernels on GGML. Fused per-rank execution makes `--tp 2` decode **1.39×** a single GPU on Gemma 4 E4B and **1.57×** on Muse-Glimmer 30B (which also gains **1.34×** prefill — the one model that beats a single GPU on both phases), and runs models that do not fit one card at all (Qwen 3.5-35B-A3B; Muse-Glimmer 30B Q8_0 at 28.2 GB on 24 GB cards). Architectures that shard no weights take the same `--tp N` as a **layer split** instead — each GPU holds a contiguous run of whole layers (Qwen 3.8 Flash Next; DeepSeek V4 and GLM 5.x layer-split by default) — which buys capacity rather than speed, and an architecture that supports neither now says so on stderr and runs on one GPU instead of silently leaving the others idle. Optional Redis-backed KV cache and Responses API store. → [Tensor Parallelism](USAGE.md#tensor-parallelism--distributed-inference)
 - **🎨 Qwen-Image-Edit image editing.** Prompt + input image → edited image, driving a 60-block MMDiT with a Qwen-Image VAE and Qwen2.5-VL-7B text encoder. CUDA-graph-captured DiT, FlowMatch-Euler true-CFG denoise, live Web UI previews, and a [Lightning distillation LoRA](https://huggingface.co/lightx2v/Qwen-Image-Edit-2511-Lightning) fast path (`--qwen-image-lora`, applied as a runtime side-path over the untouched quantized weights) that takes the default 30 steps × CFG — 60 DiT forwards — down to **4**. Beat `stable-diffusion.cpp` **1.19×** on a warm 4-step edit. → [Qwen-Image-Edit card](docs/models/qwenimage.md)
 - **🎬🔊 MiniMax-H3 joint audio-video generation.** Prompt → video **with a native 32 kHz stereo soundtrack**, generated together rather than dubbed on: one 19.3B diffusion transformer denoises a packed video+audio latent in a single token sequence, up to 15 s at 24 fps. Text-to-video, image-to-video (the photo becomes the first frame and the prompt drives the motion), first/last-frame morphing, and reference-to-video on the separate Ref2VA checkpoint — up to nine references in any mix of stills (`--ref-image`), clips (`--ref-video`, with `--ref-video-audio` for a clip's own soundtrack) and standalone audio (`--ref-audio`), each taking its own stretch of the shared timeline before the generated clip, so the person or product carries over while camera, background and composition come entirely from the prompt. All of it CFG-free at 4–8 steps against a 20-step default. Seven native ggml graphs — a 50-layer Qwen3-VL-32B text encoder with its 27-block vision tower and DeepStack taps, the packed-latent DiT with its learned AdaLN curve table and 3-axis float RoPE, a pure-transformer video VAE (36 blocks, no deconvolutions), and an alias-free BigVGAN audio VAE. Frame counts snap to a `17k+5` grid (5, 22, 39, 56, 73, 90 …) and **any grid length decodes correctly** — the video VAE runs 5 latent frames at a time with a 2-frame look-ahead and cross-fades the seams, while `h3_attend` pre-scales V by a power of two derived from the key count so that a long clip's unmasked bidirectional attention (8646 packed tokens at 107 frames, against 2364 at 22) stays finite in ggml's FP16 flash-attention accumulator; before that fix a 107-frame clip came back with every pixel black and the audio clamped. Runs **2.4×** faster end-to-end than `stable-diffusion.cpp` at 256×256 and **1.7×** at 640×384 on an M5 Pro (`ggml_metal`); on a 16 GB RTX 3080 Laptop (`ggml_cuda`) `stable-diffusion.cpp` takes the end-to-end win instead — 1.15× at 256×256, 1.07× at 640×384 — while TensorSharp stays ahead *per denoise step* (3.325 s vs 3.338 s), the gap being fixed setup cost of which ~3 s is H.264 encoding and .NET startup rather than inference. Every network verified against the reference: text encoder cos 0.999999, DiT cos 0.998, both VAEs cos 1.000000/0.99999. CLI (`--image`, `--end-image`, `--ref-image`, `--video-mode`, `--no-audio`; the soundtrack is written as a sidecar `.wav` next to the MP4), `/api/video-generate`, `/v1/videos/generations`, and two auto-downloading configs — [`config/minimax-h3-fl2va.json`](config/minimax-h3-fl2va.json) and [`config/minimax-h3-ref2va.json`](config/minimax-h3-ref2va.json) — that fetch all four networks (~33.5 GB; only the denoiser differs between the two) and load them one at a time, so peak VRAM is the largest of them rather than their sum. → [MiniMax-H3 card](docs/models/minimax-h3.md)
 - **🎬 Wan 2.1 / 2.2 video generation — video only (text → video and image → video).** The video-only alternative to MiniMax-H3, and the home of the repository's single biggest speed lever. Prompt → H.264 MP4; on the Wan 2.2 models (TI2V-5B, I2V-A14B) an uploaded image becomes the video's first frame while the prompt drives motion, camera and scene changes. One resident-weight ggml graph per denoise step (CUDA-graph-captured, flash attention, per-token-timestep modulation for TI2V i2v), causal 3D video VAE encode+decode each as a single graph, A14B's two 14B experts hot-swapped at the timestep boundary, stagewise VRAM handoff — TI2V-5B generates 81-frame 480p image-to-video on a 16 GB GPU in under 8 min, and Wan 2.1 runs **6.0×** faster end-to-end than `stable-diffusion.cpp` on the identical workload. **Step-distilled checkpoints are auto-detected from the DiT file name** (`Turbo` / `distill` / `Lightning` / `lightx2v` / `FastWan` / `…-4steps-…`) and switch to that step count with guidance off — 4 DiT passes instead of the official recipe's 100, which took the same 1088×832×121-frame image-to-video from **3 h 30 m to 17 m 30 s** on an M5 Pro. It is the single biggest speed lever in the repository and needs no flag, only a different `--model` file. Numerics verified against diffusers (DiT cos > 0.995, VAE encoders cos > 0.999, decode 59.9 dB / >35 dB PSNR). CLI (`--image`), `/v1/videos/generations`, and Web UI chat with image upload. → [Wan card](docs/models/wan.md)
@@ -79,7 +81,9 @@ dotnet run --project TensorSharp.Cli -c Release -p:TensorSharpSkipMlxNative=true
 
 Tensor parallelism splits one model across N GPUs. It runs on the direct
 `cuda` backend and on the GGML CUDA / Vulkan backends (`--backend ggml_cuda`,
-`ggml_vulkan`). Install the CUDA toolkit first, then:
+`ggml_vulkan`). On architectures that shard no weights — Qwen 3.8 Flash Next,
+DeepSeek V4, GLM 5.x — the same flag runs a layer split instead: one
+contiguous run of whole layers per GPU. Install the CUDA toolkit first, then:
 
 ```bash
 # On RunPod's Ubuntu 24.04 images, point the loader at the CUDA compat libraries first:
@@ -160,7 +164,8 @@ Implemented and exercised by the test/benchmark matrix. Pick a quantization that
 | Family | Example model (GGUF) | Image / Video / Audio | Thinking | Tools | Card |
 |---|---|---|---|---|---|
 | DeepSeek V4 Flash | [DeepSeek-V4-Flash-0731](https://huggingface.co/unsloth/DeepSeek-V4-Flash-0731-GGUF) (284B MoE, split GGUF) | — / — / — | ✅ | ✅ | [deepseek4.md](docs/models/deepseek4.md) |
-| GLM 5.x | [GLM-5.2](https://huggingface.co/unsloth/GLM-5.2-GGUF) (744B-A40B MoE, split GGUF) | — / — / — | ✅ | ✅ | [glm.md](docs/models/glm.md) |
+| GLM 5.x | [GLM-5.2](https://huggingface.co/unsloth/GLM-5.2-GGUF) (744B-A40B MoE, split GGUF), [GLM-5.3-Flash](https://huggingface.co/unsloth/GLM-5.3-Flash-GGUF) (320B MoE, split GGUF, + mmproj) | ✅ (5.3-Flash) / — / — | ✅ | ✅ | [glm.md](docs/models/glm.md) |
+| Qwen 3.8 Flash Next | [Qwen3.8-Flash-Next](https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF) (hybrid GDN + attention MoE, 512 experts, split GGUF, + mmproj) | ✅ / — / — | ✅ | ✅ | [qwen38-flash-next.md](docs/models/qwen38-flash-next.md) |
 | Gemma 4 | [gemma-4-E4B-it](https://huggingface.co/ggml-org/gemma-4-E4B-it-GGUF) (also 31B, 26B-A4B MoE) | ✅ / ✅ / ✅ | ✅ | ✅ | [gemma4.md](docs/models/gemma4.md) |
 | Qwen 3.5 / 3.6 | [Qwen3.5-9B](https://huggingface.co/unsloth/Qwen3.5-9B-GGUF) (also 35B-A3B MoE) | ✅ / — / — | ✅ | ✅ | [qwen35.md](docs/models/qwen35.md) |
 | Qwen 3 | [Qwen3-4B](https://huggingface.co/Qwen/Qwen3-4B-GGUF) | — / — / — | ✅ | ✅ | [qwen3.md](docs/models/qwen3.md) |
@@ -192,7 +197,7 @@ Several families have a *fast lane* — a different artifact to download, or one
 | **Qwen 3.6** | An MTP-retaining GGUF — [unsloth/Qwen3.6-35B-A3B-MTP-GGUF](https://huggingface.co/unsloth/Qwen3.6-35B-A3B-MTP-GGUF), not the base repo — plus `--spec` | Enables NextN speculative decode on solo sequences. The base repo ships the same file names with the block stripped and silently falls back. |
 | **Gemma 4** | `--spec-draft-model` with the matching [`gemma4-assistant` draft GGUF](https://huggingface.co/AtomicChat/gemma-4-26B-A4B-it-assistant-GGUF) plus `--spec` (server) | Speculative decode on GGML backends and the direct `cuda` backend. Draft and target hidden sizes must match, or startup fails. |
 | Any MoE that does not fit the card | `--n-cpu-moe N` / `--cpu-moe` | gpt-oss-20b 16.2 → 2.9 GB VRAM on a 16 GB laptop card, turning the WDDM spill cliff's 0.3 tok/s into **25.4** at `--n-cpu-moe 12`. |
-| Multi-GPU | `--tp N` | Gemma 4 E4B decode **1.39×** a single GPU, Muse-Glimmer 30B **1.57×** decode / **1.34×** prefill — and it runs models that fit on no single card. |
+| Multi-GPU | `--tp N` | Gemma 4 E4B decode **1.39×** a single GPU, Muse-Glimmer 30B **1.57×** decode / **1.34×** prefill — and it runs models that fit on no single card. On architectures that shard no weights the same flag is a layer split, i.e. capacity and not speed: Qwen 3.8 Flash Next UD-Q2_K_XL splits 24.2 + 26.2 GB over 2× A100-80GB with throughput unchanged and byte-identical greedy output (`TS_Q4E_LAYER_SPLIT=20,28` overrides the automatic balance). |
 | Every family | Pick the right backend: `ggml_cuda` on NVIDIA, `ggml_metal` on Apple Silicon, `ggml_cpu` (not `cpu`) without a GPU | Gemma 4 26B-A4B decodes 78.7 tok/s on `ggml_cuda` vs 35.3 on the direct `cuda` backend; on Apple Silicon Muse-Glimmer 30B prefills 413.6 tok/s on `ggml_metal` vs 29.0 on MLX. |
 
 Per-family detail, including the numbers behind every row: [MiniMax-H3](docs/models/minimax-h3.md) · [Wan](docs/models/wan.md) · [Qwen-Image-Edit](docs/models/qwenimage.md) · [DeepSeek V4](docs/models/deepseek4.md) · [Muse-Glimmer](docs/models/muse-glimmer.md) · [Features](FEATURES.md).
@@ -202,7 +207,8 @@ Per-family detail, including the numbers behind every row: [MiniMax-H3](docs/mod
 | Architecture | GGUF arch keys | Example Models | Multimodal | Thinking | Tools | MTP spec | Card |
 |---|---|---|---|---|---|---|---|
 | DeepSeek V4 Flash | `deepseek4` | DeepSeek-V4-Flash (284B MoE, 256 experts, compressed sparse attention, 1M context) | Text only | Yes | Yes (DSML) | Yes (DSpark block drafter, separate GGUF) | [deepseek4.md](docs/models/deepseek4.md) |
-| GLM 5.x | `glm-dsa` | GLM-5.2 (744B-A40B MoE, 256 experts, MLA + DeepSeek Sparse Attention, 1M context) | Text only | Yes | Yes (XML tool calls) | Yes (embedded NextN block) | [glm.md](docs/models/glm.md) |
+| GLM 5.x | `glm-dsa`, `glm5next` | GLM-5.2 (744B-A40B MoE, 256 experts, MLA + DeepSeek Sparse Attention, 1M context), GLM-5.3-Flash (320B MoE, 288 experts, KDA linear attention + NoPE MLA with a pooled indexer) | Text only (5.2), Image (5.3-Flash) | Yes | Yes (XML tool calls) | Yes on GLM-5.2 (embedded NextN block) | [glm.md](docs/models/glm.md) |
+| Qwen 3.8 Flash Next | `qwen4exp` | Qwen3.8-Flash-Next (hybrid MoE, 512 experts / 10 used, GatedDeltaNet on 36 of 48 layers interleaved with QSA-indexed full attention, PLE n-gram block, ×4 hyper-connections) | Image | Yes | Yes | — | [qwen38-flash-next.md](docs/models/qwen38-flash-next.md) |
 | Gemma 4 | `gemma4` | gemma-4-E4B, gemma-4-31B, gemma-4-26B-A4B (MoE) | Image, Video, Audio | Yes | Yes | Yes (separate draft GGUF) | [gemma4.md](docs/models/gemma4.md) |
 | Gemma 3 | `gemma3` | gemma-3-4b | Image | No | No | — | [gemma3.md](docs/models/gemma3.md) |
 | Qwen 3 | `qwen3`, `qwen2`, `qwen2vl`, `qwen2_vl` | Qwen3-4B (Qwen2 / Qwen2.5-VL GGUFs also load, as text-only chat) | Text only | Yes | Yes | — | [qwen3.md](docs/models/qwen3.md) |
@@ -263,13 +269,13 @@ New here? The sections above are all you need to get running. Everything else is
 
 | Area | Status |
 |---|---|
-| Model families | DeepSeek V4 Flash (`deepseek4`), GLM 5.x (`glm-dsa`), Gemma 3/4, DiffusionGemma, Qwen 3, Qwen 3.5/3.6-family (`qwen35`, `qwen35moe`, `qwen3next`), GPT OSS, Nemotron-H (incl. Nemotron 3 Nano Omni), Mistral 3, Muse-Glimmer (`muse-glimmer`, `muse_glimmer`). Image editing via Qwen-Image-Edit (`qwen_image`, `qwen-image` MMDiT); joint video-and-audio generation via MiniMax-H3 (`minimax-h3`, `minimax_h3`) and video-only generation via Wan 2.1 / 2.2 (`wan`, `wan2.1`, `wan2.2`). |
+| Model families | DeepSeek V4 Flash (`deepseek4`), GLM 5.x (`glm-dsa`, `glm5next`), Gemma 3/4, DiffusionGemma, Qwen 3, Qwen 3.5/3.6-family (`qwen35`, `qwen35moe`, `qwen3next`), Qwen 3.8 Flash Next (`qwen4exp`), GPT OSS, Nemotron-H (incl. Nemotron 3 Nano Omni), Mistral 3, Muse-Glimmer (`muse-glimmer`, `muse_glimmer`). Image editing via Qwen-Image-Edit (`qwen_image`, `qwen-image` MMDiT); joint video-and-audio generation via MiniMax-H3 (`minimax-h3`, `minimax_h3`) and video-only generation via Wan 2.1 / 2.2 (`wan`, `wan2.1`, `wan2.2`). |
 | Inference hosts | CLI, interactive REPL, ASP.NET Core web UI, Ollama-style API, OpenAI Chat Completions-style API. |
 | Backends | Pure C# CPU, direct CUDA/cuBLAS (`cuda`), MLX Metal (`mlx`), GGML CPU, GGML Metal, GGML CUDA, GGML Vulkan. DeepSeek V4 additionally has three whole-model executors of its own — direct-CUDA, native ggml, and a pure-C# CPU one — each layer-splitting the weights across every visible GPU (`--tp N` / `TS_DSV4_NGPU` caps the count). Among the video families, Wan is the one that restricts its backends: it runs on the GGML backends and on the direct `cuda` / pure-C# `cpu` ones, but not on MLX. |
-| Multimodal | Gemma 4 image/video/audio; Gemma 3, Qwen 3.5-family, Mistral 3, Nemotron-H Omni, Muse-Glimmer image input; PDF documents (CLI `--pdf` + Web UI). Media *out*: Qwen-Image-Edit (image), MiniMax-H3 (H.264 MP4 **plus a 32 kHz stereo `.wav` sidecar**, generated together in one packed latent), and Wan 2.1 / 2.2 (H.264 MP4 video only, text→video and image→video). |
-| Continuous batching | vLLM-style paged KV cache, block-hash prefix sharing, iteration-level scheduler (default on; opt-out `--no-continuous-batching`). DeepSeek V4 and GLM 5.x serve through their own native per-sequence slots on the same engine — a compressed MLA cache row per token has no paged layout to page — and GLM adds an opt-in batched fused decode (`TS_BATCHED_FUSED_DECODE=1`, 1.81x aggregate at 4 concurrent requests). |
+| Multimodal | Gemma 4 image/video/audio; Gemma 3, Qwen 3.5-family, Qwen 3.8 Flash Next, GLM-5.3-Flash, Mistral 3, Nemotron-H Omni, Muse-Glimmer image input; PDF documents (CLI `--pdf` + Web UI). Media *out*: Qwen-Image-Edit (image), MiniMax-H3 (H.264 MP4 **plus a 32 kHz stereo `.wav` sidecar**, generated together in one packed latent), and Wan 2.1 / 2.2 (H.264 MP4 video only, text→video and image→video). |
+| Continuous batching | vLLM-style paged KV cache, block-hash prefix sharing, iteration-level scheduler (default on; opt-out `--no-continuous-batching`). DeepSeek V4 and GLM 5.x serve through their own native per-sequence slots on the same engine — a compressed MLA cache row per token has no paged layout to page — and GLM adds an opt-in batched fused decode (`TS_BATCHED_FUSED_DECODE=1`, 1.81x aggregate at 4 concurrent requests). Qwen 3.8 Flash Next uses per-sequence state holders for the same reason — its GatedDeltaNet, PLE and indexer state has no paged layout either. |
 | Speculative decoding | MTP / NextN draft heads on Qwen 3.6 and GLM 5.2 (both embedded in the checkpoint) and Gemma 4 (separate draft GGUF); DSpark block drafting on DeepSeek V4 (`cuda` / `ggml_cuda` only) and DFlash block drafting on Muse-Glimmer, both loading a separate drafter GGUF via `--draft-model`; plus a weight-free n-gram (prompt-lookup) speculator that needs no drafter at all and therefore works on every checkpoint, selected with `--spec-type ngram`. Every emitted token is drawn from a trunk row with the run's own sampler, so the emitted stream is the one plain decoding would have produced. Off by default; opt in with `--spec` on either host (`--mtp-spec` still accepted), or by passing `--draft-model` for a block drafter. |
-| Tensor parallelism | Megatron-LM column/row-parallel TP on the direct `cuda` backend and on GGML CUDA / Vulkan (`--tp N` / `TENSORSHARP_TP_DEGREE`, CLI and server); distributed multi-node TP via peer-to-peer TCP (`--tp-node-id` / `--tp-peers`), with hierarchical AllReduce and automatic host-staging fallback when CUDA P2P is unavailable. All autoregressive architectures; MoE expert parallelism and fused per-rank decode/prefill graphs for Gemma 4 and Qwen 3.5/3.6 on GGML. Optional Redis-backed KV cache and Responses API store. |
+| Tensor parallelism | Megatron-LM column/row-parallel TP on the direct `cuda` backend and on GGML CUDA / Vulkan (`--tp N` / `TENSORSHARP_TP_DEGREE`, CLI and server); distributed multi-node TP via peer-to-peer TCP (`--tp-node-id` / `--tp-peers`), with hierarchical AllReduce and automatic host-staging fallback when CUDA P2P is unavailable. All autoregressive architectures; MoE expert parallelism and fused per-rank decode/prefill graphs for Gemma 4 and Qwen 3.5/3.6 on GGML. Architectures that shard no weights take the same `--tp N` as a layer split — a contiguous run of whole layers per GPU, as with DeepSeek V4 and GLM 5.x; on Qwen 3.8 Flash Next (`qwen4exp`) `TS_Q4E_LAYER_SPLIT=20,28` overrides the automatic balance and throws rather than ignoring a split it cannot honour. Startup prints which mode ran and the per-GPU layer/byte split, and an architecture that supports neither mode says so on stderr and runs on one GPU. Optional Redis-backed KV cache and Responses API store. |
 | Server model scope | One explicitly hosted GGUF via `--model`; optional explicit projector via `--mmproj`; no directory scanning. |
 | Observability | Structured per-turn logs, queue status, and KV-cache reuse metrics across Web UI, Ollama, and OpenAI shapes. |
 
diff --git a/README_zh-cn.md b/README_zh-cn.md
index eab4e836..89312eab 100644
--- a/README_zh-cn.md
+++ b/README_zh-cn.md
@@ -26,8 +26,10 @@ Zhongkai Fu 所著的 **[From Tensors to Tokens: Building a Multimodal LLM Infer
 - **🚀 连续批处理 & 分页 KV 缓存。** vLLM 风格的分页 KV 池，支持基于内容哈希的前缀共享与迭代级调度器，服务端默认启用。→ [深入文档](docs/PAGED_ATTENTION_AND_CONTINUOUS_BATCHING_zh-cn.md)
 - **🧬 DeepSeek V4 Flash（284B MoE），三套整模型执行器。** 这套压缩稀疏注意力、1M 上下文的架构可运行在 Direct CUDA 引擎（`--backend cuda`）、原生 ggml 执行器（`--backend ggml_cuda` / `ggml_vulkan`），以及 **100% 纯 C# 的 CPU 执行器**（`--backend cpu`，零原生依赖）上。权重会自动按层切分到所有可见 GPU，因此远大于单卡显存的模型依然跑得起来；服务端以原生 per-sequence slot + 连续批处理托管它。→ [DeepSeek V4 卡片](docs/models/deepseek4.md)
 - **🧠 GLM-5.2（744B-A40B MoE），支持张量并行与 CPU MoE offload。** Multi-head Latent Attention，外加一个 DeepSeek 稀疏注意力的 "lightning indexer"，由它挑出每个 query 可以看的那 2048 个已缓存 token。`--tp N` 让每一层都跑在每张 GPU 上（head 按列/行并行，每个专家按行切开），`--cpu-moe` 则把路由专家——占 checkpoint 的 92%——留在系统内存里。默认的按层切分与 `--cpu-moe` 在同后端下与 llama.cpp 逐 token 一致；`--tp` 是把各 rank 的局部和相加，在 2 bit MoE 上这点最后一位的差别会传到 top-8 路由，几乎并列的 token 可能不同。3× RTX PRO 6000 上的正面对比：**pp2048 918.9，llama.cpp 为 763.1 tok/s**；tg64 43.7 对 42.2。自报的 1M 上下文（约 93 GiB 的 KV）是上限而非承诺——权重落盘之后，加载器会按实际空闲的显存来定上下文并打印它的选择（按层切分 342,272 token，`--n-cpu-moe 30` 为 646,400）；设 `MAX_CONTEXT` 则把某个长度变成硬性要求。→ [GLM 卡片](docs/models/glm_zh-cn.md)
+- **🧠 GLM-5.3-Flash（320B MoE），decode 达 llama.cpp 的 2.0×。** 混合架构的后继者——288 个路由专家，45 层主干中有 34 层用 KDA 线性注意力，另外 11 层是 NoPE MLA + DSA 并配一个*池化*索引器（先在 4 格一池上取 top-k 2048，再展开到池内成员），以及 Sinkhorn 超连接 ×4——它通过与 GLM-5.2 完全相同的原生执行器与同一个 `GlmDsaModel` 加载。2× RTX PRO 6000 Blackwell（96 GB）、GLM-5.3-Flash-UD-Q2_K_XL（101 GiB）、按层切分、两个引擎均设 `n_ubatch` 2048，背靠背实测：**tg64 73.5 tok/s，llama.cpp 为 36.6**；prefill 双方相差不过几个百分点（pp2048 2014 对 2070，pp16384 1692 对 1690，pp32768 1446 对 1483）。视觉由 `mmproj-BF16.gguf`（GLM-OCR ViT）提供：`--image`、多图与多轮图像会话。跨所有可见 GPU 的按层切分、`--cpu-moe` / `--n-cpu-moe` 与原生 per-sequence slot 均可用；`--tp` 张量并行会被明确拒绝（请改用按层切分），NextN/MTP 投机解码则尚未实现。→ [GLM 卡片](docs/models/glm_zh-cn.md)
+- **⚡ Qwen 3.8 Flash Next——整个 token 一张图。** 一个混合 MoE：48 层中有 36 层是 GatedDeltaNet 递归层，与全注意力层交错（其中一部分还要过 Qwen 稀疏注意力的索引器），外加 PLE n-gram 嵌入块、×4 超连接流，以及 512 个专家（每步激活 10 个）。嵌入、图内 PLE、全部 48 层、最后的 mixer 与 LM head 以（几乎）**每个 token 一张捕获图**的方式运行，取自按形状索引的图缓存；视觉沿用 Qwen3.5-VL 视觉塔与 (T,H,W) IMRoPE，支持多图与多轮图像会话，并在轮次之间复用 KV（只能向后延伸——GDN 的递归状态无法回退）。`--tp N` 在这里跑的是**按层切分**而不是张量并行：2× A100-80GB 上，Qwen3.8-Flash-Next-UD-Q2_K_XL（73.4 GiB）分别占用两张卡的 24.2 GB 与 26.2 GB，prefill 约 1520–1550 t/s、decode 约 56 tok/s，与单卡持平，贪心输出也与单卡逐字节一致——买到的是容量而不是速度。→ [Qwen 3.8 Flash Next 卡片](docs/models/qwen38-flash-next_zh-cn.md)
 - **🔮 投机解码——四种算法共用一套草稿-验证运行时。** 多 token 预测草稿头加速单序列 decode：Qwen 3.6（NextN 块内嵌于主干——需使用保留该块的 GGUF，例如 [unsloth/Qwen3.6-35B-A3B-MTP-GGUF](https://huggingface.co/unsloth/Qwen3.6-35B-A3B-MTP-GGUF)；基础仓库的同名文件已剥离该块）、**GLM 5.2**（NextN 块已随官方 checkpoint 一同分发，无需额外下载；2× RTX PRO 6000 + `--n-cpu-moe 20` 实测 decode **约 1.3×**，草稿接受率 94%）与 Gemma 4（独立 `gemma4-assistant` 草稿 GGUF，`--spec-draft-model`）；DeepSeek V4 则新增 **DSpark** 块级起草（`--draft-model`），每步提议一整块 token，decode 提速 **1.3–1.4×**（多轮对话最高 2.0×）。第四种算法完全不需要训练权重：`--spec-type ngram` 用序列自身的后缀去匹配它已经见过的 token，因而在**任何**检查点上都可用——在本身不带草稿头的 Qwen3.5-9B 上实测 **45.2 tok/s 对普通 decode 的 31.4（1.44×）**（Q8_0、`ggml_metal`、M5 Pro），输出逐字节一致。以上都是草稿提议、主干一次批量前向验证，输出与标准 decode 一致。默认关闭；两端均可用 `--spec` 开启（历史拼写 `--mtp-*` 仍作为别名接受）。→ [投机解码](FEATURES_zh-cn.md#投机解码)
-- **🔗 张量并行与分布式集群。** 用 `--tp N` 把一个模型切分到多张 GPU 上——Direct `cuda` 后端**以及** GGML CUDA / Vulkan 后端均支持——再用点对点 TCP 集群（`--tp-node-id` / `--tp-peers`）扩展到多台机器。采用 Megatron-LM 列/行并行范式与分层 AllReduce；GGML 上提供 MoE 专家并行与按 rank 的 GatedDeltaNet 融合内核。融合式按 rank 执行让 Gemma 4 E4B 上 `--tp 2` 的 decode 达到单卡的 **1.39×**、Muse-Glimmer 30B 达到 **1.57×**（其 prefill 同时提升 **1.34×**——是这里唯一在两个阶段都超过单卡的模型），也让单卡装不下的模型（Qwen 3.5-35B-A3B；24 GB 卡上 28.2 GB 的 Muse-Glimmer 30B Q8_0）得以运行。可选 Redis 支撑的 KV 缓存与 Responses API 存储。→ [张量并行](USAGE_zh-cn.md#张量并行与分布式推理)
+- **🔗 张量并行与分布式集群。** 用 `--tp N` 把一个模型切分到多张 GPU 上——Direct `cuda` 后端**以及** GGML CUDA / Vulkan 后端均支持——再用点对点 TCP 集群（`--tp-node-id` / `--tp-peers`）扩展到多台机器。采用 Megatron-LM 列/行并行范式与分层 AllReduce；GGML 上提供 MoE 专家并行与按 rank 的 GatedDeltaNet 融合内核。融合式按 rank 执行让 Gemma 4 E4B 上 `--tp 2` 的 decode 达到单卡的 **1.39×**、Muse-Glimmer 30B 达到 **1.57×**（其 prefill 同时提升 **1.34×**——是这里唯一在两个阶段都超过单卡的模型），也让单卡装不下的模型（Qwen 3.5-35B-A3B；24 GB 卡上 28.2 GB 的 Muse-Glimmer 30B Q8_0）得以运行。本身不切分任何权重的架构则把同一个 `--tp N` 当作**按层切分**来跑——每张 GPU 拿一段连续的整层（Qwen 3.8 Flash Next；DeepSeek V4 与 GLM 5.x 默认就是按层切分）——买到的是容量而不是速度；两种模式都不支持的架构现在会在 stderr 上明说，并改用单卡运行，而不是默不作声地让其余 GPU 闲着。可选 Redis 支撑的 KV 缓存与 Responses API 存储。→ [张量并行](USAGE_zh-cn.md#张量并行与分布式推理)
 - **🎨 Qwen-Image-Edit 图像编辑。** 提示词 + 输入图像 → 编辑后的图像，驱动 60 块 MMDiT，配以 Qwen-Image VAE 与 Qwen2.5-VL-7B 文本编码器。CUDA 图捕获的整 DiT、FlowMatch-Euler true-CFG 去噪、Web UI 实时预览，以及 [Lightning 蒸馏 LoRA](https://huggingface.co/lightx2v/Qwen-Image-Edit-2511-Lightning) 快速路径（`--qwen-image-lora`，以运行期旁路的形式挂在原封不动的量化权重旁），把默认的 30 步 × CFG——即 60 次 DiT 前向——降到 **4** 次。热态 4 步编辑比 `stable-diffusion.cpp` 快 **1.19×**。→ [Qwen-Image-Edit 卡片](docs/models/qwenimage_zh-cn.md)
 - **🎬🔊 MiniMax-H3 音视频联合生成。** 提示词 → 视频 **+ 原生 32 kHz 立体声音轨**，两者一起生成而不是事后配音：同一个 193 亿参数的扩散 Transformer 在一条 token 序列里对打包的“视频+音频”潜变量去噪，最长 15 秒、24 fps。支持文生视频、图生视频（照片成为第一帧，提示词驱动运动）、首尾帧变换，以及独立的 Ref2VA 检查点上的参考生视频——最多九个参考，可任意混合静图（`--ref-image`）、片段（`--ref-video`，片段自带的音轨用 `--ref-video-audio`）与独立音轨（`--ref-audio`），每个参考在共享时间轴上占据生成片段之前的一段，因此人物或产品的特征保留下来，而机位、背景和构图完全由提示词决定。以上全部无需 CFG，相对 20 步的默认值只需 4–8 步。七张原生 ggml 整图——50 层 Qwen3-VL-32B 文本编码器及其 27 层视觉塔与 DeepStack 注入、打包潜变量 DiT（含习得的 AdaLN 曲线表与三轴浮点 RoPE）、纯 Transformer 视频 VAE（36 层，无反卷积）、以及无混叠 BigVGAN 音频 VAE。帧数对齐到 `17k+5` 网格（5、22、39、56、73、90……），且**任意网格长度都能正确解码**——视频 VAE 每次跑 5 个潜变量帧、带 2 帧前瞻并对接缝做交叉淡化；同时 `h3_attend` 会按 key 数取一个 2 的幂预先缩放 V，使长片段那条无掩码的双向注意力（107 帧时 8646 个打包 token，22 帧时为 2364）在 ggml 的 FP16 flash-attention 累加器里保持有限——在此修复之前，107 帧的片段会返回全黑画面与被削平的音频。在 M5 Pro（`ggml_metal`）上端到端比 `stable-diffusion.cpp` 快 **2.4 倍**（256×256）与 **1.7 倍**（640×384）；而在 16 GB 的 RTX 3080 Laptop（`ggml_cuda`）上端到端反过来由 `stable-diffusion.cpp` 领先——256×256 快 1.15×、640×384 快 1.07×——但*逐去噪步*仍是 TensorSharp 更快（3.325 秒对 3.338 秒），差距全在固定的启动开销上，其中约 3 秒是 H.264 编码与 .NET 进程启动，而不是推理。每个网络都对着参考实现校验：文本编码器 cos 0.999999、DiT cos 0.998、两个 VAE cos 1.000000 / 0.99999。CLI（`--image`、`--end-image`、`--ref-image`、`--video-mode`、`--no-audio`；音轨作为旁挂 `.wav` 写在 MP4 旁边）、`/api/video-generate`、`/v1/videos/generations`，以及两份自动下载的配置——[`config/minimax-h3-fl2va.json`](config/minimax-h3-fl2va.json) 与 [`config/minimax-h3-ref2va.json`](config/minimax-h3-ref2va.json)——它们会取回全部四个网络（约 33.5 GB；两份配置之间只有去噪器不同）并逐个加载、逐个释放，因此显存峰值是其中最大的一个而不是四者之和。→ [MiniMax-H3 模型卡](docs/models/minimax-h3_zh-cn.md)
 - **🎬 Wan 2.1 / 2.2 视频生成——仅视频（文本→视频、图像→视频）。** MiniMax-H3 之外的纯视频选择，也是本仓库最大单项提速手段的所在。提示词 → H.264 MP4；Wan 2.2（TI2V-5B、I2V-A14B）上上传的图像作为首帧，提示词驱动运动、镜头与场景变化。每个去噪步一张常驻权重的 ggml 图（CUDA 图捕获、flash attention），因果 3D 视频 VAE 编/解码各一张图，A14B 的两个 14B 专家在时间步边界热切换，分阶段显存交接——TI2V-5B 在 16 GB GPU 上 8 分钟内生成 81 帧 480p 图生视频，Wan 2.1 端到端比 `stable-diffusion.cpp` 快 **6.0×**。**步数蒸馏检查点会按 DiT 文件名自动识别**（`Turbo` / `distill` / `Lightning` / `lightx2v` / `FastWan` / `…-4steps-…`），并切换到该步数、关闭引导——4 次 DiT 前向而不是官方配方的 100 次，同一个 1088×832×121 帧的图生视频请求因此从 **3 小时 30 分降到 17 分 30 秒**（M5 Pro）。这是本仓库中最大的单项提速手段，不需要任何参数，只是换一个 `--model` 文件。数值已对照 diffusers 验证（DiT 余弦 > 0.995，VAE 编码器余弦 > 0.999，解码 59.9 dB / >35 dB PSNR）。支持 CLI（`--image`）、`/v1/videos/generations` 与可上传图片的 Web UI 聊天。→ [Wan 卡片](docs/models/wan_zh-cn.md)
@@ -78,7 +80,9 @@ dotnet run --project TensorSharp.Cli -c Release -p:TensorSharpSkipMlxNative=true
 **Linux（Ubuntu）+ 多张 NVIDIA GPU —— 张量并行**
 
 张量并行把一个模型切分到 N 张 GPU 上，可运行在 Direct `cuda` 后端以及 GGML CUDA /
-Vulkan 后端（`--backend ggml_cuda`、`ggml_vulkan`）。请先安装 CUDA 工具包，然后：
+Vulkan 后端（`--backend ggml_cuda`、`ggml_vulkan`）。本身不切分权重的架构
+（Qwen 3.8 Flash Next、DeepSeek V4、GLM 5.x）则把同一个参数当作按层切分：
+每张 GPU 拿一段连续的整层。请先安装 CUDA 工具包，然后：
 
 ```bash
 # 在 RunPod 的 Ubuntu 24.04 镜像上，需要先让动态链接器找到 CUDA 兼容库：
@@ -155,7 +159,8 @@ dotnet run --project TensorSharp.Server -c Release -- --help
 | 家族 | 示例模型（GGUF） | 图像 / 视频 / 音频 | 思维链 | 工具 | 卡片 |
 |---|---|---|---|---|---|
 | DeepSeek V4 Flash | [DeepSeek-V4-Flash-0731](https://huggingface.co/unsloth/DeepSeek-V4-Flash-0731-GGUF)（284B MoE，分片 GGUF） | — / — / — | ✅ | ✅ | [deepseek4](docs/models/deepseek4_zh-cn.md) |
-| GLM 5.x | [GLM-5.2](https://huggingface.co/unsloth/GLM-5.2-GGUF)（744B-A40B MoE，分片 GGUF） | — / — / — | ✅ | ✅ | [glm](docs/models/glm_zh-cn.md) |
+| GLM 5.x | [GLM-5.2](https://huggingface.co/unsloth/GLM-5.2-GGUF)（744B-A40B MoE，分片 GGUF）、[GLM-5.3-Flash](https://huggingface.co/unsloth/GLM-5.3-Flash-GGUF)（320B MoE，分片 GGUF，+ mmproj） | ✅（5.3-Flash） / — / — | ✅ | ✅ | [glm](docs/models/glm_zh-cn.md) |
+| Qwen 3.8 Flash Next | [Qwen3.8-Flash-Next](https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF)（GDN + 注意力混合 MoE，512 专家，分片 GGUF，+ mmproj） | ✅ / — / — | ✅ | ✅ | [qwen38-flash-next](docs/models/qwen38-flash-next_zh-cn.md) |
 | Gemma 4 | [gemma-4-E4B-it](https://huggingface.co/ggml-org/gemma-4-E4B-it-GGUF)（另有 31B、26B-A4B MoE） | ✅ / ✅ / ✅ | ✅ | ✅ | [gemma4](docs/models/gemma4_zh-cn.md) |
 | Qwen 3.5 / 3.6 | [Qwen3.5-9B](https://huggingface.co/unsloth/Qwen3.5-9B-GGUF)（另有 35B-A3B MoE） | ✅ / — / — | ✅ | ✅ | [qwen35](docs/models/qwen35_zh-cn.md) |
 | Qwen 3 | [Qwen3-4B](https://huggingface.co/Qwen/Qwen3-4B-GGUF) | — / — / — | ✅ | ✅ | [qwen3](docs/models/qwen3_zh-cn.md) |
@@ -187,7 +192,7 @@ dotnet run --project TensorSharp.Server -c Release -- --help
 | **Qwen 3.6** | 保留 MTP 块的 GGUF——[unsloth/Qwen3.6-35B-A3B-MTP-GGUF](https://huggingface.co/unsloth/Qwen3.6-35B-A3B-MTP-GGUF)，而非基础仓库——再加 `--spec` | 启用单序列上的 NextN 投机解码。基础仓库文件名相同但已剥离该块，会静默回退到普通 decode。 |
 | **Gemma 4** | `--spec-draft-model` 加载配套的 [`gemma4-assistant` 草稿 GGUF](https://huggingface.co/AtomicChat/gemma-4-26B-A4B-it-assistant-GGUF)，并加 `--spec`（服务端） | 在 GGML 各后端与 Direct `cuda` 后端上启用投机解码。草稿与目标的 hidden size 必须一致，否则启动即失败。 |
 | 显存装不下的 MoE | `--n-cpu-moe N` / `--cpu-moe` | 16 GB 笔记本显卡上 gpt-oss-20b 显存从 16.2 GB 降到 2.9 GB，把 WDDM 换页造成的 0.3 tok/s 变成 `--n-cpu-moe 12` 下的 **25.4 tok/s**。 |
-| 多 GPU | `--tp N` | Gemma 4 E4B decode 达单卡的 **1.39×**，Muse-Glimmer 30B decode **1.57×** / prefill **1.34×**——并且能跑单卡装不下的模型。 |
+| 多 GPU | `--tp N` | Gemma 4 E4B decode 达单卡的 **1.39×**，Muse-Glimmer 30B decode **1.57×** / prefill **1.34×**——并且能跑单卡装不下的模型。本身不切分权重的架构上，同一个参数是按层切分，买到的是容量而不是速度：Qwen 3.8 Flash Next UD-Q2_K_XL 在 2× A100-80GB 上分成 24.2 + 26.2 GB，吞吐与单卡持平，贪心输出逐字节一致（`TS_Q4E_LAYER_SPLIT=20,28` 可覆盖自动均衡）。 |
 | 所有家族 | 选对后端：NVIDIA 用 `ggml_cuda`，Apple Silicon 用 `ggml_metal`，无 GPU 用 `ggml_cpu`（而非 `cpu`） | Gemma 4 26B-A4B 在 `ggml_cuda` 上 decode 78.7 tok/s，Direct `cuda` 后端仅 35.3；Apple Silicon 上 Muse-Glimmer 30B 的 prefill 在 `ggml_metal` 上为 413.6 tok/s，MLX 上为 29.0。 |
 
 每一行背后的完整数据与逐家族细节：[MiniMax-H3](docs/models/minimax-h3_zh-cn.md) · [Wan](docs/models/wan_zh-cn.md) · [Qwen-Image-Edit](docs/models/qwenimage_zh-cn.md) · [DeepSeek V4](docs/models/deepseek4_zh-cn.md) · [Muse-Glimmer](docs/models/muse-glimmer_zh-cn.md) · [功能特性](FEATURES_zh-cn.md)。
@@ -197,7 +202,8 @@ dotnet run --project TensorSharp.Server -c Release -- --help
 | 架构 | GGUF 架构标识 | 示例模型 | 多模态 | 思维链 | 工具调用 | MTP 投机 | 卡片 |
 |---|---|---|---|---|---|---|---|
 | DeepSeek V4 Flash | `deepseek4` | DeepSeek-V4-Flash（284B MoE，256 专家，压缩稀疏注意力，1M 上下文） | 仅文本 | 支持 | 支持（DSML） | 支持（DSpark 块级草稿，独立 GGUF） | [deepseek4](docs/models/deepseek4_zh-cn.md) |
-| GLM 5.x | `glm-dsa` | GLM-5.2（744B-A40B MoE，256 专家，MLA + DeepSeek 稀疏注意力，1M 上下文） | 仅文本 | 支持 | 支持（XML 工具调用） | 支持（内嵌 NextN 块） | [glm](docs/models/glm_zh-cn.md) |
+| GLM 5.x | `glm-dsa`、`glm5next` | GLM-5.2（744B-A40B MoE，256 专家，MLA + DeepSeek 稀疏注意力，1M 上下文）、GLM-5.3-Flash（320B MoE，288 专家，KDA 线性注意力 + NoPE MLA 与池化索引器） | 仅文本（5.2）、图像（5.3-Flash） | 支持 | 支持（XML 工具调用） | GLM-5.2 支持（内嵌 NextN 块） | [glm](docs/models/glm_zh-cn.md) |
+| Qwen 3.8 Flash Next | `qwen4exp` | Qwen3.8-Flash-Next（混合 MoE，512 专家 / 激活 10 个，48 层中 36 层为 GatedDeltaNet 并与 QSA 索引的全注意力层交错，PLE n-gram 块，×4 超连接） | 图像 | 支持 | 支持 | — | [qwen38-flash-next](docs/models/qwen38-flash-next_zh-cn.md) |
 | Gemma 4 | `gemma4` | gemma-4-E4B、gemma-4-31B、gemma-4-26B-A4B（MoE） | 图像、视频、音频 | 支持 | 支持 | 支持（独立草稿 GGUF） | [gemma4](docs/models/gemma4_zh-cn.md) |
 | Gemma 3 | `gemma3` | gemma-3-4b | 图像 | 不支持 | 不支持 | — | [gemma3](docs/models/gemma3_zh-cn.md) |
 | Qwen 3 | `qwen3`、`qwen2`、`qwen2vl`、`qwen2_vl` | Qwen3-4B（Qwen2 / Qwen2.5-VL 的 GGUF 也能加载，按纯文本对话运行） | 仅文本 | 支持 | 支持 | — | [qwen3](docs/models/qwen3_zh-cn.md) |
@@ -258,13 +264,13 @@ TensorSharp 在 CUDA 的 prefill / 首 token 延迟上明显领先（多轮 pref
 
 | 范围 | 状态 |
 |---|---|
-| 模型家族 | DeepSeek V4 Flash（`deepseek4`）、GLM 5.x（`glm-dsa`）、Gemma 3/4、DiffusionGemma、Qwen 3、Qwen 3.5/3.6-family（`qwen35`、`qwen35moe`、`qwen3next`）、GPT OSS、Nemotron-H（含 Nemotron 3 Nano Omni）、Mistral 3、Muse-Glimmer（`muse-glimmer`、`muse_glimmer`）。图像编辑通过 Qwen-Image-Edit（`qwen_image`、`qwen-image` MMDiT）；音视频联合生成通过 MiniMax-H3（`minimax-h3`、`minimax_h3`），纯视频生成通过 Wan 2.1 / 2.2（`wan`、`wan2.1`、`wan2.2`）。 |
+| 模型家族 | DeepSeek V4 Flash（`deepseek4`）、GLM 5.x（`glm-dsa`、`glm5next`）、Gemma 3/4、DiffusionGemma、Qwen 3、Qwen 3.5/3.6-family（`qwen35`、`qwen35moe`、`qwen3next`）、Qwen 3.8 Flash Next（`qwen4exp`）、GPT OSS、Nemotron-H（含 Nemotron 3 Nano Omni）、Mistral 3、Muse-Glimmer（`muse-glimmer`、`muse_glimmer`）。图像编辑通过 Qwen-Image-Edit（`qwen_image`、`qwen-image` MMDiT）；音视频联合生成通过 MiniMax-H3（`minimax-h3`、`minimax_h3`），纯视频生成通过 Wan 2.1 / 2.2（`wan`、`wan2.1`、`wan2.2`）。 |
 | 推理宿主 | CLI、交互式 REPL、ASP.NET Core Web UI、Ollama 风格 API、OpenAI Chat Completions 风格 API。 |
 | 后端 | 纯 C# CPU、Direct CUDA/cuBLAS（`cuda`）、MLX Metal（`mlx`）、GGML CPU、GGML Metal、GGML CUDA、GGML Vulkan。DeepSeek V4 另有三套专属的整模型执行器——Direct CUDA、原生 ggml 与纯 C# CPU——都会把权重按层切分到所有可见 GPU（`--tp N` / `TS_DSV4_NGPU` 限定卡数）。视频家族中，Wan 是对后端有限制的那一个：它可运行于各 GGML 后端以及 Direct `cuda` / 纯 C# `cpu` 后端，但不支持 MLX。 |
-| 多模态 | Gemma 4 图像/视频/音频；Gemma 3、Qwen 3.5-family、Mistral 3、Nemotron-H Omni、Muse-Glimmer 图像输入；PDF（CLI `--pdf` + Web UI）。媒体*输出*：Qwen-Image-Edit（图像）、MiniMax-H3（H.264 MP4 **外加一份 32 kHz 立体声 `.wav` 旁挂文件**，两者在同一份打包潜变量里一起生成），以及 Wan 2.1 / 2.2（仅 H.264 MP4 视频，文本→视频与图像→视频）。 |
-| 连续批处理 | vLLM 风格分页 KV 缓存、基于内容哈希的前缀共享、迭代级调度器（默认启用，`--no-continuous-batching` 关闭）。DeepSeek V4 与 GLM 5.x 在同一引擎上通过各自原生的 per-sequence slot 提供服务——压缩后的 MLA 每 token 只有一行缓存，没有可分页的布局——GLM 还提供可选的批处理融合解码（`TS_BATCHED_FUSED_DECODE=1`，4 路并发下总吞吐 1.81 倍）。 |
+| 多模态 | Gemma 4 图像/视频/音频；Gemma 3、Qwen 3.5-family、Qwen 3.8 Flash Next、GLM-5.3-Flash、Mistral 3、Nemotron-H Omni、Muse-Glimmer 图像输入；PDF（CLI `--pdf` + Web UI）。媒体*输出*：Qwen-Image-Edit（图像）、MiniMax-H3（H.264 MP4 **外加一份 32 kHz 立体声 `.wav` 旁挂文件**，两者在同一份打包潜变量里一起生成），以及 Wan 2.1 / 2.2（仅 H.264 MP4 视频，文本→视频与图像→视频）。 |
+| 连续批处理 | vLLM 风格分页 KV 缓存、基于内容哈希的前缀共享、迭代级调度器（默认启用，`--no-continuous-batching` 关闭）。DeepSeek V4 与 GLM 5.x 在同一引擎上通过各自原生的 per-sequence slot 提供服务——压缩后的 MLA 每 token 只有一行缓存，没有可分页的布局——GLM 还提供可选的批处理融合解码（`TS_BATCHED_FUSED_DECODE=1`，4 路并发下总吞吐 1.81 倍）。Qwen 3.8 Flash Next 出于同样的原因使用逐序列状态持有者——它的 GatedDeltaNet、PLE 与索引器状态同样没有可分页的布局。 |
 | 投机解码 | Qwen 3.6 与 GLM 5.2（两者均内嵌于 checkpoint）以及 Gemma 4（独立草稿 GGUF）的 MTP / NextN 草稿头；DeepSeek V4 的 DSpark 块级起草（仅 `cuda` / `ggml_cuda`）与 Muse-Glimmer 的 DFlash 块级起草，两者都通过 `--draft-model` 加载独立的草稿 GGUF；此外还有一个不需要任何草稿权重的 n-gram（prompt-lookup）投机器，用 `--spec-type ngram` 选择，因而在任何检查点上都能用。每个输出 token 都取自主干的一行 logits，并由本次运行自身配置的采样器抽出，因此输出流与普通 decode 产生的完全相同。默认关闭；CLI 与服务端两端均可用 `--spec` 启用（`--mtp-spec` 仍被接受），块级起草则传 `--draft-model`。 |
-| 张量并行 | Direct `cuda` 后端与 GGML CUDA / Vulkan 后端上的 Megatron-LM 列/行并行 TP（`--tp N` / `TENSORSHARP_TP_DEGREE`，CLI 与服务端均支持）；通过点对点 TCP 的多节点分布式 TP（`--tp-node-id` / `--tp-peers`），采用分层 AllReduce，CUDA P2P 不可用时自动回退到主机中转。覆盖全部自回归架构；GGML 上 Gemma 4 与 Qwen 3.5/3.6 使用 MoE 专家并行与融合的按 rank decode/prefill 计算图。可选 Redis 支撑的 KV 缓存与 Responses API 存储。 |
+| 张量并行 | Direct `cuda` 后端与 GGML CUDA / Vulkan 后端上的 Megatron-LM 列/行并行 TP（`--tp N` / `TENSORSHARP_TP_DEGREE`，CLI 与服务端均支持）；通过点对点 TCP 的多节点分布式 TP（`--tp-node-id` / `--tp-peers`），采用分层 AllReduce，CUDA P2P 不可用时自动回退到主机中转。覆盖全部自回归架构；GGML 上 Gemma 4 与 Qwen 3.5/3.6 使用 MoE 专家并行与融合的按 rank decode/prefill 计算图。本身不切分权重的架构把同一个 `--tp N` 当作按层切分——每张 GPU 拿一段连续的整层，与 DeepSeek V4、GLM 5.x 一致；Qwen 3.8 Flash Next（`qwen4exp`）上可用 `TS_Q4E_LAYER_SPLIT=20,28` 覆盖自动均衡，遇到无法满足的切分会直接报错而不是静默忽略。启动时会打印实际采用的模式与每张 GPU 的层数/字节分配；两种模式都不支持的架构会在 stderr 上明说，并改用单卡运行。可选 Redis 支撑的 KV 缓存与 Responses API 存储。 |
 | 服务端模型范围 | 通过 `--model` 显式托管单个 GGUF；可通过 `--mmproj` 显式指定投影器；不扫描目录。 |
 | 可观测性 | 结构化每轮日志、队列状态，以及 Web UI / Ollama / OpenAI 中的 KV 缓存复用指标。 |
 
diff --git a/TensorSharp.Backends.GGML/GgmlBasicOps.cs b/TensorSharp.Backends.GGML/GgmlBasicOps.cs
index 07b09f13..430940bc 100644
--- a/TensorSharp.Backends.GGML/GgmlBasicOps.cs
+++ b/TensorSharp.Backends.GGML/GgmlBasicOps.cs
@@ -1272,6 +1272,67 @@ public static unsafe void FusedVisionAttention(
         /// each). Weight arrays are indexed by block; all blocks share identical shapes.
         /// Returns false on any failure so the caller falls back to the per-block path.
         /// </summary>
+        /// <summary>
+        /// Whole GLM-5.3-Flash vision encoder (24 GLM-OCR ViT blocks) as one
+        /// device-resident GGML graph: RMS norms, fused qkv + per-head q/k RMS
+        /// norms, 2D vision RoPE, SDPA, and the SwiGLU-clamp MLP per block.
+        /// Weights are cached device-resident across encodes.
+        /// </summary>
+        public static unsafe bool GlmVisionEncoder(
+            Tensor hidden, float eps, float attnScale, float swigluLimit,
+            int numPatches, int numHeads, int headDim, int halfDim,
+            float[] cosTable, float[] sinTable,
+            Tensor[] ln1W, Tensor[] qkvW, Tensor[] qkvB,
+            Tensor[] qnW, Tensor[] knW,
+            Tensor[] outW, Tensor[] outB, Tensor[] ln2W,
+            Tensor[] gateW, Tensor[] gateB,
+            Tensor[] upW, Tensor[] upB, Tensor[] downW, Tensor[] downB)
+        {
+            if (!HasNativeBufferStorage(hidden))
+                return false;
+            if (!TryCreateStandardView(hidden, out GgmlTensorView2D hiddenView))
+                return false;
+
+            int blockCount = ln1W.Length;
+            if (blockCount == 0)
+                return false;
+
+            int lnDim = (int)ln1W[0].ElementCount();
+            int qkvNe0 = (int)qkvW[0].Sizes[qkvW[0].DimensionCount - 1];
+            int qkvNe1 = (int)qkvW[0].Sizes[0];
+            long qkvBytes = qkvW[0].ElementCount() * sizeof(float);
+            int outNe0 = (int)outW[0].Sizes[outW[0].DimensionCount - 1];
+            int outNe1 = (int)outW[0].Sizes[0];
+            long outBytes = outW[0].ElementCount() * sizeof(float);
+            int ffnNe0 = (int)upW[0].Sizes[upW[0].DimensionCount - 1];
+            int ffnNe1 = (int)upW[0].Sizes[0];
+            long ffnUpBytes = upW[0].ElementCount() * sizeof(float);
+            long ffnDownBytes = downW[0].ElementCount() * sizeof(float);
+
+            IntPtr[] Ptrs(Tensor[] ws)
+            {
+                var a = new IntPtr[blockCount];
+                for (int i = 0; i < blockCount; i++)
+                    a[i] = GetBufferStart(ws[i]);
+                return a;
+            }
+
+            fixed (float* cosPtr = cosTable, sinPtr = sinTable)
+            {
+                return GgmlNative.GlmVisionEncoder(hiddenView,
+                    blockCount, eps, attnScale, swigluLimit,
+                    numPatches, numHeads, headDim, halfDim,
+                    (IntPtr)cosPtr, (IntPtr)sinPtr,
+                    Ptrs(ln1W), Ptrs(qkvW), Ptrs(qkvB), Ptrs(qnW), Ptrs(knW),
+                    Ptrs(outW), Ptrs(outB), Ptrs(ln2W),
+                    Ptrs(gateW), Ptrs(gateB), Ptrs(upW), Ptrs(upB), Ptrs(downW), Ptrs(downB),
+                    lnDim,
+                    qkvNe0, qkvNe1, qkvBytes,
+                    outNe0, outNe1, outBytes,
+                    ffnNe0, ffnNe1, ffnUpBytes, ffnDownBytes);
+            }
+        }
+
         public static unsafe bool Qwen35VisionEncoder(
             Tensor hidden, float eps, float attnScale,
             int numPatches, int numHeads, int headDim, int halfDim,
@@ -1626,7 +1687,9 @@ public static bool DFlashInject(
                 vArr, vTypeArr, vNe0Arr, vNe1Arr, vBytesArr,
                 kNormArr, ringKArr, ringVArr, ringDtype);
 
-        /// <summary>DFlash PASS C (block draft + borrowed LM head + softmax + on-device top-1) in one GGML graph.</summary>
+        /// <summary>DFlash PASS C in one GGML graph: the block draft, then either the
+        /// borrowed LM head + softmax + on-device top-1 (plain DFlash) or the DFlash2
+        /// candidate lattice the caller walks (selRank &gt; 0).</summary>
         public static bool DFlashDraftBlock(
             int[] blockIds, int blockLen, int[] positions,
             int numLayers, int hiddenSize, int headDim, int numHeads, int numKvHeads, int ringRows,
@@ -1646,7 +1709,19 @@ public static bool DFlashDraftBlock(
             IntPtr outNormData,
             IntPtr tokEmbdData, int tokEmbdType, long tokEmbdNe0, long tokEmbdNe1, long tokEmbdBytes,
             IntPtr lmHeadData, int lmHeadType, long lmHeadNe0, long lmHeadNe1, long lmHeadBytes,
-            int vocabSize, int[] idsOut, float[] confOut)
+            int vocabSize, int[] idsOut, float[] confOut,
+            int convTaps, int convGroupSize, int convNumGroups,
+            IntPtr[] attnConvBaseArr,
+            IntPtr[] attnConvProjArr, int[] attnConvProjTypeArr,
+            long[] attnConvProjNe0Arr, long[] attnConvProjNe1Arr, long[] attnConvProjBytesArr,
+            IntPtr[] ffnConvBaseArr,
+            IntPtr[] ffnConvProjArr, int[] ffnConvProjTypeArr,
+            long[] ffnConvProjNe0Arr, long[] ffnConvProjNe1Arr, long[] ffnConvProjBytesArr,
+            int selRank, int selTopK, float selLogitScale, float selLogitSoftcap,
+            IntPtr selHiddenData, int selHiddenType, long selHiddenNe0, long selHiddenNe1, long selHiddenBytes,
+            IntPtr selPredData, int selPredType, long selPredNe0, long selPredNe1, long selPredBytes,
+            IntPtr selSuccData, int selSuccType, long selSuccNe0, long selSuccNe1, long selSuccBytes,
+            float[] selScoresOut, int[] selCandOut)
             => GgmlNative.DFlashDraftBlock(blockIds, blockLen, positions,
                 numLayers, hiddenSize, headDim, numHeads, numKvHeads, ringRows,
                 eps, ropeBase, ropeFreqScale, kqScale, ringSlotPos, slidingWindow,
@@ -1663,7 +1738,17 @@ public static bool DFlashDraftBlock(
                 ringKArr, ringVArr, ringDtype, outNormData,
                 tokEmbdData, tokEmbdType, tokEmbdNe0, tokEmbdNe1, tokEmbdBytes,
                 lmHeadData, lmHeadType, lmHeadNe0, lmHeadNe1, lmHeadBytes,
-                vocabSize, idsOut, confOut);
+                vocabSize, idsOut, confOut,
+                convTaps, convGroupSize, convNumGroups,
+                attnConvBaseArr, attnConvProjArr, attnConvProjTypeArr,
+                attnConvProjNe0Arr, attnConvProjNe1Arr, attnConvProjBytesArr,
+                ffnConvBaseArr, ffnConvProjArr, ffnConvProjTypeArr,
+                ffnConvProjNe0Arr, ffnConvProjNe1Arr, ffnConvProjBytesArr,
+                selRank, selTopK, selLogitScale, selLogitSoftcap,
+                selHiddenData, selHiddenType, selHiddenNe0, selHiddenNe1, selHiddenBytes,
+                selPredData, selPredType, selPredNe0, selPredNe1, selPredBytes,
+                selSuccData, selSuccType, selSuccNe0, selSuccNe1, selSuccBytes,
+                selScoresOut, selCandOut);
 
         /// <summary>Drop the persistent DFlash graphs.</summary>
         public static void DFlashResetCaches() => GgmlNative.DFlashResetCaches();
@@ -2405,7 +2490,12 @@ public static bool Qwen35ModelDecodeToken(
         /// tokens of one sequence as a single graph. Outputs per-row logits
         /// [vocab, N] and post-norm hidden [hidden, N] (normedOut), advancing each
         /// recurrent layer's GDN state from ConvStateIn/DeltaStateIn to
-        /// ConvStateOut/DeltaStateOut. Returns false on an unsupported shape.</summary>
+        /// ConvStateOut/DeltaStateOut. Returns false on an unsupported shape.
+        ///
+        /// captureLayers/captureData additionally tap the residual ENTERING each
+        /// named layer into captureCount consecutive [hidden, N] blocks - what a
+        /// DFlash drafter's encoder consumes, and the reason speculation on this
+        /// trunk does not have to fall back to the op-by-op loop.</summary>
         public static bool Qwen35ModelVerify(
             Qwen35LayerDecodeArgs[] layers, int numLayers,
             IntPtr hidden, int hiddenSize, int startPos, int numTokens,
@@ -2419,7 +2509,10 @@ public static bool Qwen35ModelVerify(
             IntPtr lmHead, int lmHeadType, long lmHeadNe0, long lmHeadNe1, long lmHeadBytes,
             IntPtr finalNorm, IntPtr normedOut, int nLogitRows = -1,
             int[] mropePos = null, int[] mropeSections = null,
-            int tpDegree = 1, IntPtr[] tpPlanOut = null)
+            int tpDegree = 1, IntPtr[] tpPlanOut = null,
+            IntPtr captureData = default, int[] captureLayers = null, int captureCount = 0,
+            int stateSnapshots = 1, IntPtr stateSnapshotsUsed = default,
+            bool deviceStateCurrent = false, bool deferStateDownload = false)
         {
             return GgmlNative.Qwen35ModelVerify(
                 layers, numLayers, hidden, hiddenSize, startPos, numTokens,
@@ -2432,9 +2525,26 @@ public static bool Qwen35ModelVerify(
                 logits, vocabSize,
                 lmHead, lmHeadType, lmHeadNe0, lmHeadNe1, lmHeadBytes,
                 finalNorm, normedOut, nLogitRows, mropePos, mropeSections,
-                tpDegree, tpPlanOut);
+                tpDegree, tpPlanOut, captureData, captureLayers, captureCount, stateSnapshots,
+                stateSnapshotsUsed, deviceStateCurrent, deferStateDownload);
         }
 
+        /// <summary>Commit one recurrent-state snapshot into the live device state
+        /// (see TSGgml_Qwen35CommitStateSnapshot).</summary>
+        public static bool Qwen35CommitStateSnapshot(int slot, int numRecurrentLayers)
+            => GgmlNative.Qwen35CommitStateSnapshot(slot, numRecurrentLayers);
+
+        /// <summary>Read the live device recurrent state back into the host mirrors
+        /// (see TSGgml_Qwen35DrainDeviceState).</summary>
+        public static bool Qwen35DrainDeviceState(IntPtr[] convOut, IntPtr[] deltaOut, int numRecurrentLayers)
+            => GgmlNative.Qwen35DrainDeviceState(convOut, deltaOut, numRecurrentLayers);
+
+        /// <summary>Pull one per-token recurrent-state snapshot out of the verify that
+        /// just ran (see TSGgml_Qwen35FetchStateSnapshot).</summary>
+        public static bool Qwen35FetchStateSnapshot(int slot, IntPtr[] convOut, IntPtr[] deltaOut,
+            int numRecurrentLayers)
+            => GgmlNative.Qwen35FetchStateSnapshot(slot, convOut, deltaOut, numRecurrentLayers);
+
         /// <summary>Release every rank's parked tensor-parallel prefill graph
         /// (see TSGgml_Qwen35ReleaseVerifyTpGraphs).</summary>
         public static void Qwen35ReleaseVerifyTpGraphs() => GgmlNative.Qwen35ReleaseVerifyTpGraphs();
@@ -2857,12 +2967,108 @@ public static void Qwen35AttentionLayerDecode(
         /// <param name="ssmNormWData">Per-D RMSNorm weights [headDim].</param>
         /// <param name="chunkSize">Chunk size (must be a positive power of two).</param>
         /// <param name="eps">Epsilon used for L2Norm and RMSNorm.</param>
+        /// <summary>
+        /// Qwen3.8-Flash-Next fused FFN half-layer: the hyper-connection mixer, the
+        /// 512-expert MoE and the scatter back into the 4-wide residual as ONE graph.
+        /// Returns false when the backend declines the shape, so the caller can fall
+        /// back to the op-by-op path.
+        /// </summary>
+        public static bool Qwen4ExpFfnBlock(ref Qwen4ExpFfnArgs args, IntPtr resData,
+            int nEmbd, int hc, int hcLowRank, int nTokens,
+            int nExpert, int nExpertUsed, int nFf, int nFfShared, float eps, int cacheSlot,
+            bool resResident = false)
+        {
+            return GgmlNative.Qwen4ExpFfnBlock(ref args, resData, nEmbd, hc, hcLowRank,
+                nTokens, nExpert, nExpertUsed, nFf, nFfShared, eps, cacheSlot, resResident);
+        }
+
+        /// <summary>
+        /// Qwen3.8-Flash-Next fused attention half-layer: mixer, joint query|gate
+        /// projection, Q/K norm, partial rotary, KV append, gated attention and the
+        /// scatter, as ONE graph.
+        /// </summary>
+        public static bool Qwen4ExpAttnBlock(ref Qwen4ExpAttnArgs args, IntPtr resData, IntPtr maskData,
+            int nEmbd, int hc, int hcLowRank, int nTokens,
+            int headDim, int nHead, int nHeadKv, int kvCapacity, int nKv, int position,
+            int nRot, float ropeBase, float ropeFreqScale, float attnScale,
+            float eps, int cacheSlot, bool resResident = false)
+        {
+            return GgmlNative.Qwen4ExpAttnBlock(ref args, resData, maskData, nEmbd, hc, hcLowRank,
+                nTokens, headDim, nHead, nHeadKv, kvCapacity, nKv, position,
+                nRot, ropeBase, ropeFreqScale, attnScale, eps, cacheSlot, resResident);
+        }
+
+        /// <summary>Copy the 4-wide residual into the device-resident buffer the fused
+        /// kernels chain through, and back out again.</summary>
+        public static bool Qwen4ExpResUpload(IntPtr data, long bytes)
+            => GgmlNative.Qwen4ExpResUpload(data, bytes);
+
+        public static bool Qwen4ExpResDownload(IntPtr data, long bytes)
+            => GgmlNative.Qwen4ExpResDownload(data, bytes);
+
+        /// <summary>
+        /// Qwen3.8-Flash-Next fused recurrent half-layer: mixer, projections, causal
+        /// depthwise conv, the Gated DeltaNet recurrence and the scatter, as ONE graph.
+        /// The conv and recurrent state are updated in place inside it.
+        /// </summary>
+        public static bool Qwen4ExpGdnBlock(ref Qwen4ExpGdnArgs args, IntPtr resData,
+            int nEmbd, int hc, int hcLowRank, int nTokens,
+            int headKDim, int headVDim, int nKHeads, int nVHeads, int dConv,
+            float eps, int cacheSlot, bool resResident = false)
+        {
+            return GgmlNative.Qwen4ExpGdnBlock(ref args, resData, nEmbd, hc, hcLowRank, nTokens,
+                headKDim, headVDim, nKHeads, nVHeads, dConv, eps, cacheSlot, resResident);
+        }
+
+        /// <summary>Drop every cached qwen4exp FFN graph (they pin weight bindings).</summary>
+        /// <summary>
+        /// Qwen3.8-Flash-Next token span: layers [layerBegin, layerEnd) - the
+        /// recurrent-or-attention half AND the FFN half of each - as ONE persisted
+        /// GGML graph. With the PLE layer the only host interruption, a token is two
+        /// of these calls instead of 96 per-layer ones.
+        /// </summary>
+        public static bool Qwen4ExpTokenSpan(
+            IntPtr ffn, IntPtr gdn, IntPtr attn, IntPtr kinds,
+            int layerBegin, int layerEnd,
+            IntPtr resData, IntPtr maskData,
+            int nEmbd, int hc, int hcLowRank, int nTokens,
+            int headKDim, int headVDim, int nKHeads, int nVHeads, int dConv,
+            int headDim, int nHead, int nHeadKv, int kvCapacity, int nKv, int position,
+            int nRot, float ropeBase, float ropeFreqScale, float attnScale,
+            int nExpert, int nExpertUsed, int nFf, int nFfSh,
+            float eps, int cacheSlot, bool firstFfnOnly = false,
+            IntPtr head = default, IntPtr logitsOut = default,
+            IntPtr ple = default, int pleLayer = -1, IntPtr pleEmb = default,
+            IntPtr mropePos = default, IntPtr mropeSections = default,
+            int ropePosition = -1,
+            // GPU this span's layers live on (layer split). -1 / 0 = the current
+            // rank, which is the only rank on a single-GPU run.
+            int device = 0)
+        {
+            return GgmlNative.Qwen4ExpTokenSpan(ffn, gdn, attn, kinds, layerBegin, layerEnd,
+                resData, maskData, nEmbd, hc, hcLowRank, nTokens,
+                headKDim, headVDim, nKHeads, nVHeads, dConv,
+                headDim, nHead, nHeadKv, kvCapacity, nKv, position,
+                nRot, ropeBase, ropeFreqScale, attnScale,
+                nExpert, nExpertUsed, nFf, nFfSh, eps, cacheSlot, firstFfnOnly,
+                head, logitsOut, ple, pleLayer, pleEmb, mropePos, mropeSections, ropePosition,
+                device);
+        }
+
+        public static void Qwen4ExpResetFfnCache() => GgmlNative.Qwen4ExpResetFfnCache();
+
+        public static void Qwen4ExpInvalidateSeqState(IntPtr key) => GgmlNative.Qwen4ExpInvalidateSeqState(key);
+
+        public static void Qwen4ExpReleaseAllSeqState() => GgmlNative.Qwen4ExpReleaseAllSeqState();
+
+        public static void Qwen4ExpReleaseSeqState(IntPtr[] keys) => GgmlNative.Qwen4ExpReleaseSeqState(keys);
+
         public static void GatedDeltaNetChunked(
             Tensor q, Tensor k, Tensor v, Tensor z,
             Tensor alpha, Tensor beta,
             Tensor state, Tensor gatedOut,
             IntPtr dtBiasData, IntPtr aLogData, IntPtr ssmNormWData,
-            int chunkSize, float eps)
+            int chunkSize, float eps, int gateMode = 0)
         {
             if (q == null || k == null || v == null || z == null
                 || alpha == null || beta == null || state == null || gatedOut == null)
@@ -2886,7 +3092,7 @@ public static void GatedDeltaNetChunked(
                 qView, kView, vView, zView,
                 alphaView, betaView, stateView, gatedOutView,
                 dtBiasData, aLogData, ssmNormWData,
-                chunkSize, eps);
+                chunkSize, eps, gateMode);
         }
 
         /// <summary>
diff --git a/TensorSharp.Backends.GGML/GgmlContext.cs b/TensorSharp.Backends.GGML/GgmlContext.cs
index 2d9b2fb1..90b8fc85 100644
--- a/TensorSharp.Backends.GGML/GgmlContext.cs
+++ b/TensorSharp.Backends.GGML/GgmlContext.cs
@@ -17,6 +17,17 @@ public sealed class GgmlContext
         internal GgmlMemoryPool MemoryPool { get; }
 
         public GgmlContext(int[] deviceIds, GgmlBackendType backendType)
+            : this(deviceIds, backendType, enableCollectives: true)
+        {
+        }
+
+        /// <param name="enableCollectives">
+        /// False for a LAYER SPLIT: bring up one backend per GPU but create no
+        /// cross-device collective. Each GPU runs a contiguous run of layers and
+        /// the only thing that crosses a boundary is the residual, handed over
+        /// through host memory, so there is nothing to AllReduce.
+        /// </param>
+        public GgmlContext(int[] deviceIds, GgmlBackendType backendType, bool enableCollectives)
         {
             if (deviceIds == null || deviceIds.Length == 0)
             {
@@ -32,20 +43,30 @@ public GgmlContext(int[] deviceIds, GgmlBackendType backendType)
 
             if (deviceIds.Length > 1)
             {
-                // Tensor parallelism: bring up one ggml backend per GPU. Ops then
-                // select a rank with GgmlNative.SetActiveDevice; tensors carry
-                // their rank through GgmlAllocator.DeviceId.
+                // Several GPUs: bring up one ggml backend per GPU. Ops then select a
+                // rank with GgmlNative.SetActiveDevice; tensors carry their rank
+                // through GgmlAllocator.DeviceId. Used both by tensor parallelism
+                // (every GPU holds a shard of every weight, collectives on) and by a
+                // layer split (each GPU holds a run of whole layers, collectives off).
                 if (backendType != GgmlBackendType.Cuda && backendType != GgmlBackendType.Vulkan)
                 {
                     throw new NotSupportedException(
-                        $"The GGML {backendType} backend exposes a single device; tensor parallelism requires the CUDA or Vulkan backend.");
+                        $"The GGML {backendType} backend exposes a single device; multi-GPU requires the CUDA or Vulkan backend.");
                 }
                 // The native side needs to know whether the ranks will be driven
                 // concurrently: that decides whether ggml-cuda's graph capture
                 // (which is process-wide and breaks under concurrent CUDA calls)
                 // has to be turned off for the run.
-                GgmlNative.TensorParallelInit(backendType, DeviceIds, GgmlTensorParallelGroup.ParallelRanks);
-                HasDeviceAllReduce = GgmlNative.TensorParallelHasDeviceAllReduce();
+                if (enableCollectives)
+                {
+                    GgmlNative.TensorParallelInit(backendType, DeviceIds, GgmlTensorParallelGroup.ParallelRanks);
+                    HasDeviceAllReduce = GgmlNative.TensorParallelHasDeviceAllReduce();
+                }
+                else
+                {
+                    GgmlNative.MultiDeviceInit(backendType, DeviceIds);
+                    HasDeviceAllReduce = false;
+                }
             }
             OpRegistry.RegisterAssembly(Assembly.GetExecutingAssembly());
 
diff --git a/TensorSharp.Backends.GGML/GgmlGlmNative.cs b/TensorSharp.Backends.GGML/GgmlGlmNative.cs
index 39e88dcd..f9481ea8 100644
--- a/TensorSharp.Backends.GGML/GgmlGlmNative.cs
+++ b/TensorSharp.Backends.GGML/GgmlGlmNative.cs
@@ -71,6 +71,12 @@ private static extern int TSGgml_GlmForwardBatchedDecode(IntPtr handle, int n, i
         [DllImport(DllName, CallingConvention = Conv)]
         private static extern void TSGgml_GlmFree(IntPtr handle);
 
+        [DllImport(DllName, CallingConvention = Conv)]
+        private static extern unsafe int TSGgml_GlmQueueVisionRows(IntPtr handle, float* rows, int nRows, int index);
+
+        [DllImport(DllName, CallingConvention = Conv)]
+        private static extern void TSGgml_GlmClearVisionRows(IntPtr handle);
+
         // ---- NextN/MTP speculative decoding -------------------------------
 
         [DllImport(DllName, CallingConvention = Conv)]
@@ -131,6 +137,21 @@ public static unsafe bool Forward(IntPtr handle, int[] tokens, float[] logitsOut
 
         public static void Reset(IntPtr handle) => TSGgml_GlmReset(handle);
 
+        /// <summary>Queue projected vision-embedding rows (glm5next) to override the
+        /// token embeddings of image-placeholder positions in the NEXT Forward call.
+        /// <paramref name="index"/> is the first placeholder's position within that
+        /// call's token array. The queue is consumed by the forward.</summary>
+        public static unsafe bool QueueVisionRows(IntPtr handle, float[] rows, int nRows, int index)
+        {
+            fixed (float* r = rows)
+            {
+                return TSGgml_GlmQueueVisionRows(handle, r, nRows, index) != 0;
+            }
+        }
+
+        /// <summary>Drop queued vision rows (a cancelled or re-planned prompt).</summary>
+        public static void ClearVisionRows(IntPtr handle) => TSGgml_GlmClearVisionRows(handle);
+
         /// <summary>Drop the KV of the tokens after <paramref name="nPast"/>.</summary>
         public static bool Rewind(IntPtr handle, int nPast) => TSGgml_GlmRewind(handle, nPast) != 0;
 
diff --git a/TensorSharp.Backends.GGML/GgmlNative.cs b/TensorSharp.Backends.GGML/GgmlNative.cs
index 87bdba5d..55623681 100644
--- a/TensorSharp.Backends.GGML/GgmlNative.cs
+++ b/TensorSharp.Backends.GGML/GgmlNative.cs
@@ -352,6 +352,91 @@ public struct Gemma4MoELayerDecodeArgs
     public float LayerOutputScale;
 }
 
+    /// <summary>
+    /// Mirrors TSGgmlQwen4ExpPleArgs in ggml_ops_qwen4exp.cpp - the PLE block run
+    /// inside the span; only the n-gram hash and the table gather stay host-side.
+    /// </summary>
+    [StructLayout(LayoutKind.Sequential)]
+    public struct Qwen4ExpPleArgs
+    {
+        public IntPtr KeyW, ValueW, NormKey, NormQuery, NormConv, Conv1dT, ConvState;
+        public long KeyBytes, ValueBytes;
+        public int KeyType, ValueType, Kern, Dil;
+    }
+
+    /// <summary>
+    /// Mirrors TSGgmlQwen4ExpHeadArgs in ggml_ops_qwen4exp.cpp - the final
+    /// hyper-connection mixer (which IS the output norm) plus the LM head.
+    /// </summary>
+    [StructLayout(LayoutKind.Sequential)]
+    public struct Qwen4ExpHeadArgs
+    {
+        public IntPtr HcNorm, HcDown, HcUp, Head;
+        public long HcDownBytes, HcUpBytes, HeadBytes;
+        public int HcDownType, HcUpType, HeadType;
+        public int Vocab;
+    }
+
+    /// <summary>
+    /// Mirrors TSGgmlQwen4ExpAttnArgs in ggml_ops_qwen4exp.cpp - the full-attention
+    /// half of a qwen4exp layer. Pointers first, then int64, then int32.
+    /// </summary>
+    [StructLayout(LayoutKind.Sequential)]
+    public struct Qwen4ExpAttnArgs
+    {
+        public IntPtr HcNorm, HcDown, HcUp, HcInject;
+        public IntPtr Wq, Wk, Wv, Wo;
+        public IntPtr QNorm, KNorm;
+        public IntPtr KCache, VCache;
+
+        public long HcDownBytes, HcUpBytes, HcInjectBytes;
+        public long WqBytes, WkBytes, WvBytes, WoBytes;
+        public long KvBytes;
+
+        public int HcDownType, HcUpType, HcInjectType;
+        public int WqType, WkType, WvType, WoType;
+        public int KvType;
+    }
+
+    /// <summary>
+    /// Mirrors TSGgmlQwen4ExpGdnArgs in ggml_ops_qwen4exp.cpp - the recurrent half
+    /// of a qwen4exp layer. Pointers first, then int64, then int32.
+    /// </summary>
+    [StructLayout(LayoutKind.Sequential)]
+    public struct Qwen4ExpGdnArgs
+    {
+        public IntPtr HcNorm, HcDown, HcUp, HcInject;
+        public IntPtr Qkv, Gate, Beta, Alpha;
+        public IntPtr Conv1d, SsmDt, SsmA, SsmNorm, OutProj;
+        public IntPtr ConvState, SsmState;
+
+        public long HcDownBytes, HcUpBytes, HcInjectBytes;
+        public long QkvBytes, GateBytes, BetaBytes, AlphaBytes, OutProjBytes;
+
+        public int HcDownType, HcUpType, HcInjectType;
+        public int QkvType, GateType, BetaType, AlphaType, OutProjType;
+    }
+
+    /// <summary>
+    /// Mirrors TSGgmlQwen4ExpFfnArgs in ggml_ops_qwen4exp.cpp. Pointers first,
+    /// then int64, then int32 - append within a run rather than reordering.
+    /// </summary>
+    [StructLayout(LayoutKind.Sequential)]
+    public struct Qwen4ExpFfnArgs
+    {
+        public IntPtr HcNorm, HcDown, HcUp, HcInject;
+        public IntPtr Router, GateExps, UpExps, DownExps;
+        public IntPtr ShGateInp, ShGate, ShUp, ShDown;
+
+        public long HcDownBytes, HcUpBytes, HcInjectBytes;
+        public long RouterBytes, GateExpsBytes, UpExpsBytes, DownExpsBytes;
+        public long ShGateBytes, ShUpBytes, ShDownBytes;
+
+        public int HcDownType, HcUpType, HcInjectType;
+        public int RouterType, GateExpsType, UpExpsType, DownExpsType;
+        public int ShGateType, ShUpType, ShDownType;
+    }
+
 // Descriptor for the Qwen3.5/3.6 full-model decode kernel
 // (TSGgml_Qwen35ModelDecode). Field order/types MUST match the native
 // TSGgmlQwen35LayerDesc struct EXACTLY: 23 pointers, then 27 int64, then 13 int32.
@@ -392,6 +477,12 @@ public struct Qwen35LayerDecodeArgs
     public IntPtr ShexpUpW;
     public IntPtr ShexpDownW;
     public IntPtr ShexpGateInpW;
+    /// <summary>Dense FFN with gate and up UNFUSED, for the mixed-quant "UD"
+    /// layers where the two tensors have different GGML types and no
+    /// imatrix-free requantization can bring them together. Non-zero exactly
+    /// when <see cref="GuW"/> is zero; the graph then runs two matmuls.</summary>
+    public IntPtr FfnGateW;
+    public IntPtr FfnUpW;
 
     // int64 weight shapes
     public long QkvNe0, QkvNe1, QkvBytes;
@@ -410,6 +501,8 @@ public struct Qwen35LayerDecodeArgs
     public long ShexpGateNe0, ShexpGateNe1, ShexpGateBytes;
     public long ShexpUpNe0, ShexpUpNe1, ShexpUpBytes;
     public long ShexpDownNe0, ShexpDownNe1, ShexpDownBytes;
+    public long FfnGateNe0, FfnGateNe1, FfnGateBytes;
+    public long FfnUpNe0, FfnUpNe1, FfnUpBytes;
 
     // int32 scalars
     public int StructBytes;
@@ -425,6 +518,8 @@ public struct Qwen35LayerDecodeArgs
     /// <summary>Non-zero keeps this layer's routed experts in system RAM and runs its
     /// MoE FFN on the host (MoeCpuOffloadConfig / --n-cpu-moe).</summary>
     public int CpuMoe;
+    public int FfnGateType;
+    public int FfnUpType;
 }
 
 // Descriptor for the fused DiffusionGemma decode-layer kernel
@@ -1746,6 +1841,26 @@ private static partial int TSGgml_Qwen35VisionEncoderF32(
             int upNe0, int upNe1, long upBytes, int upBDim,
             int downNe0, int downNe1, long downBytes, int downBDim);
 
+        [LibraryImport(DllName)]
+        [UnmanagedCallConv(CallConvs = new[] { typeof(CallConvCdecl) })]
+        private static partial int TSGgml_GlmVisionEncoderF32(
+            GgmlTensorView2D hidden,
+            int blockCount, float eps, float attnScale, float swigluLimit,
+            int numPatches, int numHeads, int headDim, int halfDim,
+            IntPtr cosTable, IntPtr sinTable,
+            IntPtr[] ln1W,
+            IntPtr[] qkvW, IntPtr[] qkvB,
+            IntPtr[] qnW, IntPtr[] knW,
+            IntPtr[] outW, IntPtr[] outB,
+            IntPtr[] ln2W,
+            IntPtr[] gateW, IntPtr[] gateB,
+            IntPtr[] upW, IntPtr[] upB,
+            IntPtr[] downW, IntPtr[] downB,
+            int lnDim,
+            int qkvNe0, int qkvNe1, long qkvBytes,
+            int outNe0, int outNe1, long outBytes,
+            int ffnNe0, int ffnNe1, long ffnUpBytes, long ffnDownBytes);
+
         [LibraryImport(DllName)]
         [UnmanagedCallConv(CallConvs = new[] { typeof(CallConvCdecl) })]
         private static partial int TSGgml_FusedGemma4VisionBlockF32(
@@ -2513,7 +2628,23 @@ private static partial int TSGgml_DFlashDraftBlock(
             IntPtr outNormData,
             IntPtr tokEmbdData, int tokEmbdType, long tokEmbdNe0, long tokEmbdNe1, long tokEmbdBytes,
             IntPtr lmHeadData, int lmHeadType, long lmHeadNe0, long lmHeadNe1, long lmHeadBytes,
-            int vocabSize, int[] idsOut, float[] confOut);
+            int vocabSize, int[] idsOut, float[] confOut,
+            // DFlash2 grouped dynamic convolution. convTaps == 0 disables it and
+            // every array below may be null (a first-generation drafter).
+            int convTaps, int convGroupSize, int convNumGroups,
+            IntPtr[] attnConvBaseArr,
+            IntPtr[] attnConvProjArr, int[] attnConvProjTypeArr,
+            long[] attnConvProjNe0Arr, long[] attnConvProjNe1Arr, long[] attnConvProjBytesArr,
+            IntPtr[] ffnConvBaseArr,
+            IntPtr[] ffnConvProjArr, int[] ffnConvProjTypeArr,
+            long[] ffnConvProjNe0Arr, long[] ffnConvProjNe1Arr, long[] ffnConvProjBytesArr,
+            // DFlash2 candidate selector. selRank == 0 disables it; when it is on,
+            // idsOut/confOut are left untouched and the lattice comes back instead.
+            int selRank, int selTopK, float selLogitScale, float selLogitSoftcap,
+            IntPtr selHiddenData, int selHiddenType, long selHiddenNe0, long selHiddenNe1, long selHiddenBytes,
+            IntPtr selPredData, int selPredType, long selPredNe0, long selPredNe1, long selPredBytes,
+            IntPtr selSuccData, int selSuccType, long selSuccNe0, long selSuccNe1, long selSuccBytes,
+            float[] selScoresOut, int[] selCandOut);
 
         [LibraryImport(DllName)]
         [UnmanagedCallConv(CallConvs = new[] { typeof(CallConvCdecl) })]
@@ -2538,7 +2669,18 @@ public static bool DFlashInject(
                 vArr, vTypeArr, vNe0Arr, vNe1Arr, vBytesArr,
                 kNormArr, ringKArr, ringVArr, ringDtype) != 0;
 
-        /// <summary>DFlash PASS C in one graph, returning the on-device argmax id and its softmax probability per row.</summary>
+        /// <summary>
+        /// DFlash PASS C in one graph.
+        ///
+        /// Plain DFlash returns the on-device argmax id and its softmax probability
+        /// per block row. A DFlash2 drafter (selRank &gt; 0) instead returns the
+        /// candidate ids and the transition lattice the caller walks:
+        /// selCandOut is [selTopK, gamma] and selScoresOut holds the anchor row
+        /// (selTopK floats, block position 0 scored against the verified anchor)
+        /// followed by one [selTopK(pred), selTopK(cand)] matrix per following
+        /// position, candidate-fastest. That is ~7 KB per step against the 12.9 MB
+        /// a [vocab, block] readback would cost.
+        /// </summary>
         public static bool DFlashDraftBlock(
             int[] blockIds, int blockLen, int[] positions,
             int numLayers, int hiddenSize, int headDim, int numHeads, int numKvHeads, int ringRows,
@@ -2558,7 +2700,23 @@ public static bool DFlashDraftBlock(
             IntPtr outNormData,
             IntPtr tokEmbdData, int tokEmbdType, long tokEmbdNe0, long tokEmbdNe1, long tokEmbdBytes,
             IntPtr lmHeadData, int lmHeadType, long lmHeadNe0, long lmHeadNe1, long lmHeadBytes,
-            int vocabSize, int[] idsOut, float[] confOut)
+            int vocabSize, int[] idsOut, float[] confOut,
+            // DFlash2 grouped dynamic convolution. convTaps == 0 disables it and
+            // every array below may be null (a first-generation drafter).
+            int convTaps, int convGroupSize, int convNumGroups,
+            IntPtr[] attnConvBaseArr,
+            IntPtr[] attnConvProjArr, int[] attnConvProjTypeArr,
+            long[] attnConvProjNe0Arr, long[] attnConvProjNe1Arr, long[] attnConvProjBytesArr,
+            IntPtr[] ffnConvBaseArr,
+            IntPtr[] ffnConvProjArr, int[] ffnConvProjTypeArr,
+            long[] ffnConvProjNe0Arr, long[] ffnConvProjNe1Arr, long[] ffnConvProjBytesArr,
+            // DFlash2 candidate selector. selRank == 0 disables it; when it is on,
+            // idsOut/confOut are left untouched and the lattice comes back instead.
+            int selRank, int selTopK, float selLogitScale, float selLogitSoftcap,
+            IntPtr selHiddenData, int selHiddenType, long selHiddenNe0, long selHiddenNe1, long selHiddenBytes,
+            IntPtr selPredData, int selPredType, long selPredNe0, long selPredNe1, long selPredBytes,
+            IntPtr selSuccData, int selSuccType, long selSuccNe0, long selSuccNe1, long selSuccBytes,
+            float[] selScoresOut, int[] selCandOut)
             => TSGgml_DFlashDraftBlock(blockIds, blockLen, positions,
                 numLayers, hiddenSize, headDim, numHeads, numKvHeads, ringRows,
                 eps, ropeBase, ropeFreqScale, kqScale, ringSlotPos, slidingWindow,
@@ -2575,7 +2733,17 @@ public static bool DFlashDraftBlock(
                 ringKArr, ringVArr, ringDtype, outNormData,
                 tokEmbdData, tokEmbdType, tokEmbdNe0, tokEmbdNe1, tokEmbdBytes,
                 lmHeadData, lmHeadType, lmHeadNe0, lmHeadNe1, lmHeadBytes,
-                vocabSize, idsOut, confOut) != 0;
+                vocabSize, idsOut, confOut,
+                convTaps, convGroupSize, convNumGroups,
+                attnConvBaseArr, attnConvProjArr, attnConvProjTypeArr,
+                attnConvProjNe0Arr, attnConvProjNe1Arr, attnConvProjBytesArr,
+                ffnConvBaseArr, ffnConvProjArr, ffnConvProjTypeArr,
+                ffnConvProjNe0Arr, ffnConvProjNe1Arr, ffnConvProjBytesArr,
+                selRank, selTopK, selLogitScale, selLogitSoftcap,
+                selHiddenData, selHiddenType, selHiddenNe0, selHiddenNe1, selHiddenBytes,
+                selPredData, selPredType, selPredNe0, selPredNe1, selPredBytes,
+                selSuccData, selSuccType, selSuccNe0, selSuccNe1, selSuccBytes,
+                selScoresOut, selCandOut) != 0;
 
         /// <summary>Drop the persistent DFlash graphs (ring reallocation / KV reset).</summary>
         public static void DFlashResetCaches() => TSGgml_DFlashResetCaches();
@@ -3551,7 +3719,10 @@ private static partial int TSGgml_Qwen35ModelVerify(
             IntPtr lmHead, int lmHeadType, long lmHeadNe0, long lmHeadNe1, long lmHeadBytes,
             IntPtr finalNorm, IntPtr normedOut, int nLogitRows,
             int[] mropePos, int[] mropeSections,
-            int tpDegree, IntPtr[] tpPlanOut);
+            int tpDegree, IntPtr[] tpPlanOut,
+            IntPtr captureData, int[] captureLayers, int captureCount,
+            int stateSnapshots, IntPtr stateSnapshotsUsed, int deviceStateCurrent,
+            int deferStateDownload);
 
         public static bool Qwen35ModelVerify(
             Qwen35LayerDecodeArgs[] layers, int numLayers,
@@ -3566,7 +3737,10 @@ public static bool Qwen35ModelVerify(
             IntPtr lmHead, int lmHeadType, long lmHeadNe0, long lmHeadNe1, long lmHeadBytes,
             IntPtr finalNorm, IntPtr normedOut, int nLogitRows,
             int[] mropePos = null, int[] mropeSections = null,
-            int tpDegree = 1, IntPtr[] tpPlanOut = null)
+            int tpDegree = 1, IntPtr[] tpPlanOut = null,
+            IntPtr captureData = default, int[] captureLayers = null, int captureCount = 0,
+            int stateSnapshots = 1, IntPtr stateSnapshotsUsed = default,
+            bool deviceStateCurrent = false, bool deferStateDownload = false)
         {
             return TSGgml_Qwen35ModelVerify(
                 layers, numLayers, hidden, hiddenSize, startPos, numTokens,
@@ -3579,9 +3753,53 @@ public static bool Qwen35ModelVerify(
                 logits, vocabSize,
                 lmHead, lmHeadType, lmHeadNe0, lmHeadNe1, lmHeadBytes,
                 finalNorm, normedOut, nLogitRows, mropePos, mropeSections,
-                tpDegree, tpPlanOut) != 0;
+                tpDegree, tpPlanOut, captureData, captureLayers, captureCount,
+                stateSnapshots, stateSnapshotsUsed, deviceStateCurrent ? 1 : 0,
+                deferStateDownload ? 1 : 0) != 0;
         }
 
+        [LibraryImport(DllName)]
+        [UnmanagedCallConv(CallConvs = new[] { typeof(CallConvCdecl) })]
+        private static partial int TSGgml_Qwen35CommitStateSnapshot(int slot, int numRecurrentLayers);
+
+        /// <summary>
+        /// Commit one recurrent-state snapshot into the live device state, without a
+        /// host round trip. The next verify can then skip its state upload, which is
+        /// the point: that upload plus the matching download was the largest per-step
+        /// cost of speculative decoding on a Qwen 3.5/3.8 hybrid trunk.
+        ///
+        /// <paramref name="slot"/> counts back from the end of the verified batch;
+        /// -1 means the post-window state, which is what a single-row step commits.
+        /// </summary>
+        public static bool Qwen35CommitStateSnapshot(int slot, int numRecurrentLayers)
+            => TSGgml_Qwen35CommitStateSnapshot(slot, numRecurrentLayers) != 0;
+
+        [LibraryImport(DllName)]
+        [UnmanagedCallConv(CallConvs = new[] { typeof(CallConvCdecl) })]
+        private static partial int TSGgml_Qwen35DrainDeviceState(
+            IntPtr[] convOut, IntPtr[] deltaOut, int numRecurrentLayers);
+
+        /// <summary>Read the live device recurrent state back into the host mirrors,
+        /// for anything that has to run the op-by-op recurrent path.</summary>
+        public static bool Qwen35DrainDeviceState(IntPtr[] convOut, IntPtr[] deltaOut, int numRecurrentLayers)
+            => TSGgml_Qwen35DrainDeviceState(convOut, deltaOut, numRecurrentLayers) != 0;
+
+        [LibraryImport(DllName)]
+        [UnmanagedCallConv(CallConvs = new[] { typeof(CallConvCdecl) })]
+        private static partial int TSGgml_Qwen35FetchStateSnapshot(
+            int slot, IntPtr[] convOut, IntPtr[] deltaOut, int numRecurrentLayers);
+
+        /// <summary>
+        /// Pull ONE per-token recurrent-state snapshot out of the verify that just
+        /// ran, counting <paramref name="slot"/> tokens back from the end of that
+        /// batch. False when there is nothing to pull (no snapshotting verify has
+        /// run, or the slot is out of range), and the caller keeps its old
+        /// restore-and-re-forward path.
+        /// </summary>
+        public static bool Qwen35FetchStateSnapshot(int slot, IntPtr[] convOut, IntPtr[] deltaOut,
+            int numRecurrentLayers)
+            => TSGgml_Qwen35FetchStateSnapshot(slot, convOut, deltaOut, numRecurrentLayers) != 0;
+
         [LibraryImport(DllName)]
         [UnmanagedCallConv(CallConvs = new[] { typeof(CallConvCdecl) })]
         private static partial void TSGgml_Qwen35ReleaseVerifyTpGraphs();
@@ -3696,7 +3914,159 @@ private static partial int TSGgml_GatedDeltaNetChunkedF32(
             IntPtr aLogData,
             IntPtr ssmNormWData,
             int chunkSize,
-            float eps);
+            float eps,
+            int gateMode);
+
+        [LibraryImport(DllName)]
+        private static partial int TSGgml_Qwen4ExpFfnBlock(
+            ref Qwen4ExpFfnArgs args,
+            IntPtr resData,
+            int nEmbd, int hc, int hcLowRank, int nTokens,
+            int nExpert, int nExpertUsed, int nFf, int nFfShared,
+            float eps, int cacheSlot, int resResident);
+
+        [LibraryImport(DllName)]
+        private static partial int TSGgml_Qwen4ExpGdnBlock(
+            ref Qwen4ExpGdnArgs args,
+            IntPtr resData,
+            int nEmbd, int hc, int hcLowRank, int nTokens,
+            int headKDim, int headVDim, int nKHeads, int nVHeads, int dConv,
+            float eps, int cacheSlot, int resResident);
+
+        [LibraryImport(DllName)]
+        internal static partial void TSGgml_Qwen4ExpResetFfnCache();
+
+        [LibraryImport(DllName)]
+        internal static partial void TSGgml_Qwen4ExpInvalidateSeqState(IntPtr key);
+
+        [LibraryImport(DllName)]
+        internal static partial void TSGgml_Qwen4ExpReleaseAllSeqState();
+
+        [LibraryImport(DllName)]
+        internal static unsafe partial void TSGgml_Qwen4ExpReleaseSeqState(IntPtr* keys, int n);
+
+        /// <summary>Re-arm the one-time seed upload for one recurrent-state entry
+        /// (keyed by its host seed pointer); the next graph build re-uploads from
+        /// the host copy. Used after the managed reset zeroes that copy.</summary>
+        public static void Qwen4ExpInvalidateSeqState(IntPtr key) => TSGgml_Qwen4ExpInvalidateSeqState(key);
+
+        /// <summary>Free every native sequence-state entry and cached graph
+        /// (model dispose).</summary>
+        public static void Qwen4ExpReleaseAllSeqState() => TSGgml_Qwen4ExpReleaseAllSeqState();
+
+        /// <summary>Free the device recurrent-state entries of a released sequence
+        /// holder and drop every cached graph (surviving holders rebuild and
+        /// re-bind their own still-alive entries).</summary>
+        public static unsafe void Qwen4ExpReleaseSeqState(IntPtr[] keys)
+        {
+            if (keys == null || keys.Length == 0) return;
+            fixed (IntPtr* k = keys)
+            {
+                TSGgml_Qwen4ExpReleaseSeqState(k, keys.Length);
+            }
+        }
+
+        /// <summary>
+        /// One graph for the hyper-connection mixer, the 512-expert MoE and the
+        /// scatter back into the wide residual. Returns false when the backend
+        /// declines the shape, so the caller falls back to the op-by-op path.
+        /// </summary>
+        public static bool Qwen4ExpFfnBlock(ref Qwen4ExpFfnArgs args, IntPtr resData,
+            int nEmbd, int hc, int hcLowRank, int nTokens,
+            int nExpert, int nExpertUsed, int nFf, int nFfShared, float eps, int cacheSlot,
+            bool resResident)
+        {
+            return TSGgml_Qwen4ExpFfnBlock(ref args, resData,
+                nEmbd, hc, hcLowRank, nTokens,
+                nExpert, nExpertUsed, nFf, nFfShared, eps, cacheSlot,
+                resResident ? 1 : 0) != 0;
+        }
+
+        public static bool Qwen4ExpGdnBlock(ref Qwen4ExpGdnArgs args, IntPtr resData,
+            int nEmbd, int hc, int hcLowRank, int nTokens,
+            int headKDim, int headVDim, int nKHeads, int nVHeads, int dConv,
+            float eps, int cacheSlot, bool resResident)
+        {
+            return TSGgml_Qwen4ExpGdnBlock(ref args, resData, nEmbd, hc, hcLowRank, nTokens,
+                headKDim, headVDim, nKHeads, nVHeads, dConv, eps, cacheSlot,
+                resResident ? 1 : 0) != 0;
+        }
+
+        [LibraryImport(DllName)]
+        private static partial int TSGgml_Qwen4ExpAttnBlock(
+            ref Qwen4ExpAttnArgs args,
+            IntPtr resData,
+            IntPtr maskData,
+            int nEmbd, int hc, int hcLowRank, int nTokens,
+            int headDim, int nHead, int nHeadKv, int kvCapacity, int nKv, int position,
+            int nRot, float ropeBase, float ropeFreqScale, float attnScale,
+            float eps, int cacheSlot, int resResident);
+
+        public static bool Qwen4ExpAttnBlock(ref Qwen4ExpAttnArgs args, IntPtr resData, IntPtr maskData,
+            int nEmbd, int hc, int hcLowRank, int nTokens,
+            int headDim, int nHead, int nHeadKv, int kvCapacity, int nKv, int position,
+            int nRot, float ropeBase, float ropeFreqScale, float attnScale,
+            float eps, int cacheSlot, bool resResident)
+        {
+            return TSGgml_Qwen4ExpAttnBlock(ref args, resData, maskData, nEmbd, hc, hcLowRank,
+                nTokens, headDim, nHead, nHeadKv, kvCapacity, nKv, position,
+                nRot, ropeBase, ropeFreqScale, attnScale, eps, cacheSlot,
+                resResident ? 1 : 0) != 0;
+        }
+
+        [LibraryImport(DllName)]
+        private static partial int TSGgml_Qwen4ExpTokenSpan(
+            IntPtr ffn, IntPtr gdn, IntPtr attn, IntPtr kinds,
+            int layerBegin, int layerEnd,
+            IntPtr resData, IntPtr maskData,
+            int nEmbd, int hc, int hcLowRank, int nTokens,
+            int headKDim, int headVDim, int nKHeads, int nVHeads, int dConv,
+            int headDim, int nHead, int nHeadKv, int kvCapacity, int nKv, int position,
+            int nRot, float ropeBase, float ropeFreqScale, float attnScale,
+            int nExpert, int nExpertUsed, int nFf, int nFfSh,
+            float eps, int cacheSlot, int firstFfnOnly,
+            IntPtr head, IntPtr logitsOut,
+            IntPtr ple, int pleLayer, IntPtr pleEmb,
+            IntPtr mropePos, IntPtr mropeSections, int ropePosition,
+            int device);
+
+        public static bool Qwen4ExpTokenSpan(
+            IntPtr ffn, IntPtr gdn, IntPtr attn, IntPtr kinds,
+            int layerBegin, int layerEnd,
+            IntPtr resData, IntPtr maskData,
+            int nEmbd, int hc, int hcLowRank, int nTokens,
+            int headKDim, int headVDim, int nKHeads, int nVHeads, int dConv,
+            int headDim, int nHead, int nHeadKv, int kvCapacity, int nKv, int position,
+            int nRot, float ropeBase, float ropeFreqScale, float attnScale,
+            int nExpert, int nExpertUsed, int nFf, int nFfSh,
+            float eps, int cacheSlot, bool firstFfnOnly,
+            IntPtr head, IntPtr logitsOut,
+            IntPtr ple, int pleLayer, IntPtr pleEmb,
+            IntPtr mropePos, IntPtr mropeSections, int ropePosition, int device)
+        {
+            return TSGgml_Qwen4ExpTokenSpan(ffn, gdn, attn, kinds, layerBegin, layerEnd,
+                resData, maskData, nEmbd, hc, hcLowRank, nTokens,
+                headKDim, headVDim, nKHeads, nVHeads, dConv,
+                headDim, nHead, nHeadKv, kvCapacity, nKv, position,
+                nRot, ropeBase, ropeFreqScale, attnScale,
+                nExpert, nExpertUsed, nFf, nFfSh, eps, cacheSlot,
+                firstFfnOnly ? 1 : 0, head, logitsOut, ple, pleLayer, pleEmb,
+                mropePos, mropeSections, ropePosition, device) != 0;
+        }
+
+        [LibraryImport(DllName)]
+        private static partial int TSGgml_Qwen4ExpResUpload(IntPtr data, long bytes);
+
+        [LibraryImport(DllName)]
+        private static partial int TSGgml_Qwen4ExpResDownload(IntPtr data, long bytes);
+
+        public static bool Qwen4ExpResUpload(IntPtr data, long bytes)
+            => TSGgml_Qwen4ExpResUpload(data, bytes) != 0;
+
+        public static bool Qwen4ExpResDownload(IntPtr data, long bytes)
+            => TSGgml_Qwen4ExpResDownload(data, bytes) != 0;
+
+        public static void Qwen4ExpResetFfnCache() => TSGgml_Qwen4ExpResetFfnCache();
 
         // Mirrors NemoMamba2BatchedSeqDesc in ggml_ops_mamba2.cpp; same 32-byte
         // POD layout on 64-bit (two ints, two padding ints, two pointers).
@@ -4467,6 +4837,37 @@ public static bool Qwen35VisionEncoder(
             return rc != 0;
         }
 
+        public static bool GlmVisionEncoder(
+            GgmlTensorView2D hidden,
+            int blockCount, float eps, float attnScale, float swigluLimit,
+            int numPatches, int numHeads, int headDim, int halfDim,
+            IntPtr cosTable, IntPtr sinTable,
+            IntPtr[] ln1W,
+            IntPtr[] qkvW, IntPtr[] qkvB,
+            IntPtr[] qnW, IntPtr[] knW,
+            IntPtr[] outW, IntPtr[] outB,
+            IntPtr[] ln2W,
+            IntPtr[] gateW, IntPtr[] gateB,
+            IntPtr[] upW, IntPtr[] upB,
+            IntPtr[] downW, IntPtr[] downB,
+            int lnDim,
+            int qkvNe0, int qkvNe1, long qkvBytes,
+            int outNe0, int outNe1, long outBytes,
+            int ffnNe0, int ffnNe1, long ffnUpBytes, long ffnDownBytes)
+        {
+            int rc = TSGgml_GlmVisionEncoderF32(hidden,
+                blockCount, eps, attnScale, swigluLimit,
+                numPatches, numHeads, headDim, halfDim,
+                cosTable, sinTable,
+                ln1W, qkvW, qkvB, qnW, knW, outW, outB, ln2W,
+                gateW, gateB, upW, upB, downW, downB,
+                lnDim,
+                qkvNe0, qkvNe1, qkvBytes,
+                outNe0, outNe1, outBytes,
+                ffnNe0, ffnNe1, ffnUpBytes, ffnDownBytes);
+            return rc != 0;
+        }
+
         public static void FusedGemma4VisionBlock(
             GgmlTensorView2D hidden, float eps,
             IntPtr ln1W,
@@ -5451,12 +5852,13 @@ public static void GatedDeltaNetChunked(
             IntPtr aLogData,
             IntPtr ssmNormWData,
             int chunkSize,
-            float eps)
+            float eps,
+            int gateMode = 0)
         {
             CheckResult(TSGgml_GatedDeltaNetChunkedF32(
                 q, k, v, z, alpha, beta, state, gatedOut,
                 dtBiasData, aLogData, ssmNormWData,
-                chunkSize, eps), "gated_delta_net_chunked");
+                chunkSize, eps, gateMode), "gated_delta_net_chunked");
         }
 
         // Batched per-token Nemotron Mamba2 step. Runs all (seq, token) pairs
diff --git a/TensorSharp.Backends.GGML/GgmlTensorParallel.cs b/TensorSharp.Backends.GGML/GgmlTensorParallel.cs
index 3b1a0598..200f0ae0 100644
--- a/TensorSharp.Backends.GGML/GgmlTensorParallel.cs
+++ b/TensorSharp.Backends.GGML/GgmlTensorParallel.cs
@@ -31,6 +31,10 @@ internal static partial class GgmlNative
         [UnmanagedCallConv(CallConvs = new[] { typeof(CallConvCdecl) })]
         private static partial int TSGgml_TensorParallelInit(int backendType, int[] deviceIndices, int count, int concurrentRanks);
 
+        [LibraryImport(DllName)]
+        [UnmanagedCallConv(CallConvs = new[] { typeof(CallConvCdecl) })]
+        private static partial int TSGgml_MultiDeviceInit(int backendType, int[] deviceIndices, int count);
+
         [LibraryImport(DllName)]
         [UnmanagedCallConv(CallConvs = new[] { typeof(CallConvCdecl) })]
         private static partial int TSGgml_SetActiveDevice(int rank);
@@ -141,6 +145,30 @@ public static void TensorParallelInit(GgmlBackendType backendType, int[] deviceI
             s_cachedRankValid = false;
         }
 
+        /// <summary>
+        /// Bring up one ggml backend per listed GPU for a LAYER SPLIT: same device
+        /// setup as <see cref="TensorParallelInit"/>, but no cross-device collective
+        /// is created.
+        ///
+        /// A layer split never reduces across devices - each GPU owns a contiguous
+        /// run of layers and only the residual crosses a boundary, through host
+        /// memory - so initialising NCCL/P2P would spend the startup time and take
+        /// on the lying-P2P first-collective hang risk for machinery that is never
+        /// used.
+        /// </summary>
+        public static void MultiDeviceInit(GgmlBackendType backendType, int[] deviceIndices)
+        {
+            if (deviceIndices == null || deviceIndices.Length == 0)
+                throw new ArgumentException("At least one device index is required.", nameof(deviceIndices));
+
+            if (TSGgml_MultiDeviceInit((int)backendType, deviceIndices, deviceIndices.Length) == 0)
+            {
+                throw new InvalidOperationException(GetLastErrorMessage(
+                    $"Failed to initialize {deviceIndices.Length} GGML device(s) for a layer split."));
+            }
+            s_cachedRankValid = false;
+        }
+
         /// <summary>Number of ranks the native bridge currently has initialized.</summary>
         public static int TensorParallelDegree()
         {
diff --git a/TensorSharp.Cli/CliUsage.cs b/TensorSharp.Cli/CliUsage.cs
index b0995229..e2e74877 100644
--- a/TensorSharp.Cli/CliUsage.cs
+++ b/TensorSharp.Cli/CliUsage.cs
@@ -126,7 +126,9 @@ private static readonly (string Section, OptionHelp[] Options)[] Sections =
                     "Split the model across N GPUs on this machine (tensor parallelism): each GPU holds 1/N of " +
                     "every weight and the shards cooperate on every token. Use it when a model does not fit on one " +
                     "GPU. Range: 1 to the number of local GPUs. Applies to the cuda, ggml_cuda, and ggml_vulkan " +
-                    "backends. Default: 1 — no splitting (TENSORSHARP_TP_DEGREE env var overrides).",
+                    "backends. " +
+                    "Multi-GPU is implemented PER ARCHITECTURE, not per backend, and in two forms. Architectures that shard weights run true tensor parallelism. qwen4exp (Qwen3.8-Flash-Next) shards nothing, so --tp N runs it as a LAYER SPLIT instead - each GPU holds a contiguous run of whole layers, which is the same and only multi-GPU mode llama.cpp offers for it. That is a CAPACITY feature: it lets a model, context or resident-weight set that one GPU cannot hold fit across several, and is not expected to raise tok/s. The startup line says which mode actually ran. An architecture that supports neither says so on stderr and runs on one GPU rather than silently leaving the others idle. " +
+                    "Default: 1 — no splitting (TENSORSHARP_TP_DEGREE env var overrides).",
                     "--backend ggml_cuda --tp 2"),
                 new OptionHelp("--tp-node-id <N>",
                     "This node's 0-based ID for multi-node (distributed) tensor parallelism over TCP. Node 0 is " +
@@ -201,7 +203,7 @@ private static readonly (string Section, OptionHelp[] Options)[] Sections =
                 new OptionHelp("--spec-type <name>",
                     "Which speculation ALGORITHM to draft with. 'auto' (default) uses whatever drafter the " +
                     "checkpoint carries: a per-token NextN/MTP head (GLM-5.2, Qwen 3.6, Gemma 4's separate " +
-                    "assistant GGUF) or a block drafter (DeepSeek V4 DSpark, Muse-Glimmer DFlash). " +
+                    "assistant GGUF) or a block drafter (DeepSeek V4 DSpark, DFlash / DFlash2 on Muse-Glimmer and Qwen 3.8). " +
                     "'draft-head' and 'block' pin one of those explicitly. 'ngram' needs NO trained weights at " +
                     "all - it drafts by finding where the last few tokens occurred earlier in the context and " +
                     "proposing what followed, so it works on every model and is strong on summarizing, editing, " +
@@ -229,7 +231,8 @@ private static readonly (string Section, OptionHelp[] Options)[] Sections =
                     "--spec --spec-draft-model gemma-4-12B-it-Q4_0-MTP.gguf"),
                 new OptionHelp("--draft-model <path>",
                     "Block drafter GGUF that has to be resident before the model's layer split runs (DeepSeek " +
-                    "V4's DSpark support module, Muse-Glimmer's DFlash). The drafter proposes a whole block of " +
+                    "V4's DSpark support module, the DFlash / DFlash2 drafters for Muse-Glimmer and Qwen 3.8). "
+                    + "The drafter proposes a whole block of " +
                     "tokens per step and the trunk verifies it in one batched forward. Naming the file IS the " +
                     "request - such a drafter needs no --spec. Every emitted token is still drawn from a trunk " +
                     "row - with argmax under a greedy config, with your sampler otherwise - so output is " +
diff --git a/TensorSharp.Cli/Program.cs b/TensorSharp.Cli/Program.cs
index 5881e43a..53cfad56 100644
--- a/TensorSharp.Cli/Program.cs
+++ b/TensorSharp.Cli/Program.cs
@@ -1,4 +1,4 @@
-// Copyright (c) Zhongkai Fu. All rights reserved.
+// Copyright (c) Zhongkai Fu. All rights reserved.
 // https://github.com/zhongkaifu/TensorSharp
 //
 // This file is part of TensorSharp.
@@ -734,6 +734,25 @@ static void MainCore(string[] args)
                     model.MultimodalInjector.LoadProjectors(autoMmproj);
                 }
             }
+            else if (imagePath != null &&
+                     (model.Config.Architecture == "qwen4exp" || model.Config.Architecture == "glm5next"))
+            {
+                // Any mmproj companion beside the model (the published file is
+                // mmproj-BF16.gguf).
+                string modelDir = Path.GetDirectoryName(modelPath);
+                string autoMmproj = null;
+                if (modelDir != null && Directory.Exists(modelDir))
+                {
+                    foreach (string candidate in Directory.GetFiles(modelDir, "*mmproj*.gguf"))
+                    { autoMmproj = candidate; break; }
+                }
+                if (autoMmproj != null)
+                {
+                    _log.LogInformation(LogEventIds.HostConfiguration,
+                        "Auto-loading vision encoder: {MmProj}", autoMmproj);
+                    model.MultimodalInjector.LoadProjectors(autoMmproj);
+                }
+            }
             else if (imagePath != null &&
                      (model.Config.Architecture == "qwen35" ||
                       model.Config.Architecture == "qwen35moe" ||
@@ -1023,9 +1042,22 @@ static void MainCore(string[] args)
                     _log.LogError(LogEventIds.CliFailed, "Image file not found: {ImagePath}", imagePath);
                     return;
                 }
-                imagePaths = new List<string> { imagePath };
+                // Every --image in order; imagePath alone would keep only the last.
+                imagePaths = imagePathList.Count > 0
+                    ? new List<string>(imagePathList)
+                    : new List<string> { imagePath };
+                foreach (string ip in imagePaths)
+                {
+                    if (!File.Exists(ip))
+                    {
+                        _log.LogError(LogEventIds.CliFailed, "Image file not found: {ImagePath}", ip);
+                        return;
+                    }
+                }
                 if (!hasUserInput)
-                    rawText = "What is in this image? Please describe it.";
+                    rawText = imagePaths.Count > 1
+                        ? "What is in these images? Please describe each."
+                        : "What is in this image? Please describe it.";
                 _log.LogInformation(LogEventIds.UploadReceived,
                     "Image input: {ImagePath} ({Bytes})",
                     imagePath, LoggingExtensions.FormatBytes(new FileInfo(imagePath).Length));
@@ -1183,6 +1215,7 @@ static void RunMultiTurnTest(ModelBase model, string jsonlPath, int maxTokens,
                 string userMsg;
                 int turnMaxTokens = maxTokens;
                 bool forceReset = false;
+                List<string> turnImages = null;
                 try
                 {
                     var doc = JsonDocument.Parse(line);
@@ -1199,13 +1232,22 @@ static void RunMultiTurnTest(ModelBase model, string jsonlPath, int maxTokens,
                         turnMaxTokens = mt.GetInt32();
                     if (root.TryGetProperty("force_reset", out var fr))
                         forceReset = fr.GetBoolean();
+                    if (root.TryGetProperty("images", out var imgs) && imgs.ValueKind == JsonValueKind.Array)
+                    {
+                        turnImages = new List<string>();
+                        foreach (var im in imgs.EnumerateArray())
+                        {
+                            string ip = im.GetString();
+                            if (!string.IsNullOrEmpty(ip)) turnImages.Add(ip);
+                        }
+                    }
                 }
                 catch
                 {
                     userMsg = line;
                 }
 
-                history.Add(new ChatMessage { Role = "user", Content = userMsg });
+                history.Add(new ChatMessage { Role = "user", Content = userMsg, ImagePaths = turnImages });
                 _log.LogInformation(LogEventIds.ChatStarted,
                     "multi-turn turn={Turn}/{TotalTurns} user=\"{User}\"",
                     turn + 1, lines.Length, LoggingExtensions.SanitizeForLog(userMsg));
@@ -1226,6 +1268,12 @@ static void RunMultiTurnTest(ModelBase model, string jsonlPath, int maxTokens,
                     addGenerationPrompt: true,
                     enableThinking: enableThinking);
 
+                // Expand image placeholders, prepare (cached) vision embeddings and
+                // the IMRoPE position table over the WHOLE conversation so far.
+                bool anyImages = history.Exists(m => m.ImagePaths != null && m.ImagePaths.Count > 0);
+                if (anyImages)
+                    inputTokens = model.MultimodalInjector.ProcessPromptTokens(history, inputTokens);
+
                 _log.LogInformation(LogEventIds.ChatStarted,
                     "multi-turn prompt tokens={PromptTokens}", inputTokens.Count);
 
@@ -1237,7 +1285,7 @@ static void RunMultiTurnTest(ModelBase model, string jsonlPath, int maxTokens,
                 double decodeMs;
 
                 var turnDecoder = SpeculativeDecodingOptions.TryCreate(
-                    model, specSettings, hasMediaAttachments: false, out string turnDeclineReason,
+                    model, specSettings, hasMediaAttachments: anyImages, out string turnDeclineReason,
                     multiTurnDecoder);
                 if (turnDecoder != null)
                     multiTurnDecoder = turnDecoder;
@@ -1367,10 +1415,19 @@ static float[] ApplyReusePlan(ModelBase model, KVCache kvCache, ReusePlan plan,
                 case ReusePlanKind.PartialReuse:
                 {
                     int reused = plan.ReusedPrefixLength;
-                    int suffixLength = plan.TokensToForward;
+                    // A reuse boundary inside an image span would truncate half an
+                    // injection; the injector pulls it back to the span start.
+                    int clamped = model.MultimodalInjector.ClampReusablePrefix(reused);
+                    if (clamped != reused)
+                        reused = clamped;
+                    int suffixLength = inputTokens.Count - reused;
                     model.TruncateKVCache(reused);
                     kvCache.TruncateTo(reused);
 
+                    // Vision embeddings and the IMRoPE slice for the tokens being
+                    // forwarded, offset by the reused prefix.
+                    model.MultimodalInjector.QueuePromptEmbeddingsForSlice(reused, suffixLength);
+
                     var suffix = new int[suffixLength];
                     for (int i = 0; i < suffixLength; i++)
                         suffix[i] = inputTokens[reused + i];
@@ -1384,6 +1441,7 @@ static float[] ApplyReusePlan(ModelBase model, KVCache kvCache, ReusePlan plan,
                 {
                     model.ResetKVCache();
                     kvCache.Reset();
+                    model.MultimodalInjector.QueuePromptEmbeddingsForSlice(0, inputTokens.Count);
                     var allTokens = inputTokens.ToArray();
                     float[] logits = model.Forward(allTokens);
                     kvCache.RecordAppend(allTokens, logits);
@@ -1713,6 +1771,18 @@ internal static void ResolveChatSamplingDefaults(
             if (!pinned.HasFlag(SamplingFields.TopP)) cfg.TopP = 0.95f;
             if (!pinned.HasFlag(SamplingFields.MinP)) cfg.MinP = 0.05f;
             if (!pinned.HasFlag(SamplingFields.PenaltyLastN)) cfg.PenaltyLastN = 64;
+            // NOTE: RepetitionPenalty is deliberately NOT defaulted here. The chat
+            // config is seeded from SamplingConfig.Greedy, which sets it to 1.0
+            // (disabled), and PenaltyLastN=64 above therefore penalises nothing -
+            // which looks like an oversight but is exactly llama.cpp's default pair
+            // (common/common.h: penalty_last_n=64, penalty_repeat=1.0), and matching
+            // that chain is this method's stated contract. A 1.1 default was tried
+            // while chasing an endless-repetition report on Qwen3.8-Flash-Next; the
+            // real cause turned out to be a mid-sequence fused-path fallback that
+            // reset the recurrent state (see Qwen4ExpModel.WarnIfQsaBudgetExceeded),
+            // and once that was fixed the seed that looped no longer did. Operators
+            // who want a penalty pass --repeat-penalty; a GGUF can also ask for one
+            // via general.sampling.penalty_repeat, applied just below.
 
             // 2. The model's own recommendation wins over our generic defaults.
             string fromModel = model?.Config?.RecommendedSampling?.ApplyTo(cfg, pinned) ?? string.Empty;
@@ -2356,6 +2426,51 @@ static string RunInference(ModelBase model, string rawText, List<string> imagePa
                             "No vision encoder loaded. Use --mmproj to specify the vision encoder GGUF.");
                     }
                 }
+                else if (model is Qwen4ExpModel q4eVision)
+                {
+                    // The injector owns the whole qwen4exp pipeline: image-pad
+                    // expansion, embedding cache and the (T,H,W) IMRoPE table the
+                    // token-span kernel rotates image positions with.
+                    if (q4eVision.VisionEncoder != null)
+                    {
+                        var mmHistory = new List<ChatMessage>
+                        {
+                            new ChatMessage { Role = "user", Content = rawText ?? "", ImagePaths = imagePaths }
+                        };
+                        inputTokens = model.MultimodalInjector.ProcessPromptTokens(mmHistory, inputTokens);
+                        model.MultimodalInjector.QueuePromptEmbeddingsForSlice(0, inputTokens.Count);
+                        _log.LogInformation(LogEventIds.HostConfiguration,
+                            "qwen4exp vision: prompt expanded to {Tokens} tokens for {Images} image(s)",
+                            inputTokens.Count, imagePaths.Count);
+                    }
+                    else
+                    {
+                        _log.LogWarning(LogEventIds.HostConfiguration,
+                            "No vision encoder loaded. Use --mmproj to specify the vision encoder GGUF.");
+                    }
+                }
+                else if (model is GlmDsaModel glmVision)
+                {
+                    // The injector owns the glm5next pipeline: <|image|> expansion
+                    // and the embedding-override spans the native executor applies.
+                    if (glmVision.VisionEncoder != null)
+                    {
+                        var mmHistory = new List<ChatMessage>
+                        {
+                            new ChatMessage { Role = "user", Content = rawText ?? "", ImagePaths = imagePaths }
+                        };
+                        inputTokens = model.MultimodalInjector.ProcessPromptTokens(mmHistory, inputTokens);
+                        model.MultimodalInjector.QueuePromptEmbeddingsForSlice(0, inputTokens.Count);
+                        _log.LogInformation(LogEventIds.HostConfiguration,
+                            "glm5next vision: prompt expanded to {Tokens} tokens for {Images} image(s)",
+                            inputTokens.Count, imagePaths.Count);
+                    }
+                    else
+                    {
+                        _log.LogWarning(LogEventIds.HostConfiguration,
+                            "No vision encoder loaded. Use --mmproj to specify the vision encoder GGUF.");
+                    }
+                }
                 else
                 {
                     int imagePadId = model.Tokenizer.LookupToken("<|image_pad|>");
@@ -2877,6 +2992,14 @@ bool OnToken(int t)
                     decoder.TokensDrafted, decoder.TokensAccepted,
                     decoder.AcceptanceRate, decoder.VerifySteps, decoder.PlainSteps, decoder.RollbackSteps,
                     decoder.ParkedSteps, decoder.PlainMsPerToken, decoder.SpecMsPerToken);
+                // Where a speculative step actually goes. Cheap (one timestamp per
+                // phase per step) and the only way to tell a slow DRAFTER from a slow
+                // verify or an expensive rollback without a profiler.
+                _log.LogInformation(LogEventIds.CliBenchmark,
+                    "cli.inference speculative timing: draftMs={DraftMs:F0} verifyMs={VerifyMs:F0} "
+                    + "snapshotMs={SnapMs:F0} rollbackMs={RollMs:F0} catchUpMs={CatchMs:F0} plainMs={PlainMs:F0}",
+                    decoder.Stats.DraftMs, decoder.Stats.VerifyMs, decoder.Stats.SnapshotMs,
+                    decoder.Stats.RollbackMs, decoder.Stats.CatchUpMs, decoder.Stats.PlainMs);
                 _log.LogInformation(LogEventIds.ChatCompleted,
                     "cli.inference finishReason={FinishReason} tokens={Tokens}",
                     trimmedAtStop != null ? "stop_sequence" : hitEos ? "eos" : "max_tokens",
diff --git a/TensorSharp.Cli/SpeculativeDecodingOptions.cs b/TensorSharp.Cli/SpeculativeDecodingOptions.cs
index e2d12bb5..24beac35 100644
--- a/TensorSharp.Cli/SpeculativeDecodingOptions.cs
+++ b/TensorSharp.Cli/SpeculativeDecodingOptions.cs
@@ -48,6 +48,11 @@ internal readonly struct Settings
             /// <summary>Cap on tokens drafted per step. Always positive.</summary>
             public int MaxDraftTokens { get; init; }
 
+            /// <summary>True when the operator actually named that cap, so a model
+            /// that prefers a narrower DEFAULT window
+            /// (ISpeculativeTarget.SpecPreferredDraftWindow) leaves it alone.</summary>
+            public bool MaxDraftTokensExplicit { get; init; }
+
             /// <summary>Draft-confidence gate, or null to let the ALGORITHM apply its own
             /// default — 0.75 for a per-token head, 0.35 for a block drafter, 0 for
             /// n-gram. They threshold different quantities, so there is no shared
@@ -70,6 +75,7 @@ internal readonly struct Settings
                     ? SpeculatorRegistry.Auto
                     : SpeculatorName,
                 MaxDraftTokens = MaxDraftTokens,
+                MaxDraftTokensExplicit = MaxDraftTokensExplicit,
                 MinDraftProb = MinDraftProb,
             };
         }
@@ -122,6 +128,7 @@ internal static Settings Resolve(int specDraftMax, float specDraftConfMin)
                 Requested = cfg.Speculation.Enabled,
                 SpeculatorName = cfg.Speculation.SpeculatorName,
                 MaxDraftTokens = specDraftMax > 0 ? specDraftMax : Math.Max(1, cfg.Speculation.MaxDraftTokens),
+                MaxDraftTokensExplicit = specDraftMax > 0 || cfg.Speculation.MaxDraftTokensExplicit,
                 MinDraftProb = specDraftConfMin >= 0f ? specDraftConfMin : cfg.Speculation.MinDraftProb,
                 AnyExplicit = cfg.Speculation.Enabled || specDraftMax > 0 || specDraftConfMin >= 0f
                               || cfg.Speculation.MinDraftProb.HasValue
diff --git a/TensorSharp.GGML.Native/CMakeLists.txt b/TensorSharp.GGML.Native/CMakeLists.txt
index 0af11c25..8a50a105 100644
--- a/TensorSharp.GGML.Native/CMakeLists.txt
+++ b/TensorSharp.GGML.Native/CMakeLists.txt
@@ -233,6 +233,7 @@ set(TSG_GGMLOPS_SOURCES
     ggml_ops_transformer.cpp
     ggml_ops_transformer_prefill.cpp
     ggml_ops_gptoss_decode.cpp
+    ggml_ops_qwen4exp.cpp
     ggml_ops_gptoss_prefill.cpp
     ggml_ops_qwen35_decode.cpp
     ggml_ops_qwen35_verify.cpp
diff --git a/TensorSharp.GGML.Native/ggml_ops_core.cpp b/TensorSharp.GGML.Native/ggml_ops_core.cpp
index e72bc867..86f1ad1f 100644
--- a/TensorSharp.GGML.Native/ggml_ops_core.cpp
+++ b/TensorSharp.GGML.Native/ggml_ops_core.cpp
@@ -155,10 +155,23 @@ namespace tsg
              std::strcmp(value, "ON") == 0);
     }
 
+    // ggml's DEBUG channel is where the CUDA backend reports whether a graph is
+    // being CUDA-graph-captured ("CUDA graph warmup complete" / "... reset"),
+    // which is not otherwise observable and is worth ~19 ms per replay on a
+    // 3765-node graph under WDDM. Off by default because it is chatty.
+    static bool ggml_debug_log_enabled()
+    {
+        static const bool v = []{
+            const char* e = std::getenv("TS_GGML_LOG_DEBUG");
+            return is_truthy_env(e);
+        }();
+        return v;
+    }
+
     static void filtered_ggml_log(enum ggml_log_level level, const char* text, void* user_data)
     {
         (void) user_data;
-        if (level == GGML_LOG_LEVEL_DEBUG)
+        if (level == GGML_LOG_LEVEL_DEBUG && !ggml_debug_log_enabled())
             return;
         std::fputs(text, stderr);
         std::fflush(stderr);
diff --git a/TensorSharp.GGML.Native/ggml_ops_dflash.cpp b/TensorSharp.GGML.Native/ggml_ops_dflash.cpp
index 705a0ce0..adec222d 100644
--- a/TensorSharp.GGML.Native/ggml_ops_dflash.cpp
+++ b/TensorSharp.GGML.Native/ggml_ops_dflash.cpp
@@ -45,6 +45,23 @@ using namespace tsg;
 // slot holding position p is masked from a block query at position qp when
 // qp - p >= n_swa or p > qp; the block's own b columns are never masked, because
 // DFlash drafts with llama_set_causal_attn(ctx_dft, false).
+//
+// ----------------------------------------------------------------------------
+// DFlash2 (conv_taps > 0 / sel_rank > 0) adds two things to the draft graph:
+//
+//   GROUPED DYNAMIC CONVOLUTION around every attention and every FFN sublayer.
+//   One projection of the sublayer INPUT yields both filters; tap t of channel c
+//   at block position r is (base[side][t][c] + delta[r][t][c / group_size]) and
+//   multiplies x[r-t][c], zeroed for r < t. The shift is a get_rows with a
+//   constant index vector and the boundary mask a constant [1,1,b] multiply -
+//   both baked once, because the block layout never changes between steps.
+//
+//   CANDIDATE SELECTOR instead of the per-row argmax. The LM head runs over the
+//   gamma = b-1 PROPOSAL rows only (not the anchor's), a top-k keeps the
+//   candidates, and the pairwise transition scores come back to the host as
+//   k + k*k*(gamma-1) floats - ~7 KB, against 12.9 MB for a [vocab, b] readback.
+//   The walk itself is gamma steps over k candidates and stays on the host: it is
+//   inherently sequential and the data is already small.
 // ============================================================================
 
 namespace
@@ -66,6 +83,11 @@ namespace
         ggml_tensor* ring_v = nullptr;
         ggml_tensor* k_cpy = nullptr;
         ggml_tensor* v_cpy = nullptr;
+        // DFlash2 only.
+        ggml_tensor* attn_conv_base = nullptr;
+        ggml_tensor* attn_conv_proj = nullptr;
+        ggml_tensor* ffn_conv_base = nullptr;
+        ggml_tensor* ffn_conv_proj = nullptr;
     };
 
     // Persistent graph cache. Both entry points key on
@@ -83,6 +105,9 @@ namespace
         ggml_tensor* mask = nullptr;         // draft only (shared across layers)
         ggml_tensor* out = nullptr;
         ggml_tensor* out_conf = nullptr;   // draft only
+        ggml_tensor* out_s0 = nullptr;     // DFlash2 selector: [k] anchor row
+        ggml_tensor* out_scores = nullptr; // DFlash2 selector: [k, k, gamma-1]
+        ggml_tensor* out_cand = nullptr;   // DFlash2 selector: [k, gamma] I32
         const void* sig = nullptr;
         const void* sig_ring = nullptr;
         int n_rows = 0;
@@ -94,6 +119,7 @@ namespace
             if (ctx != nullptr) { ggml_free(ctx); ctx = nullptr; }
             graph = nullptr; valid = false;
             in_main = pos = kv_index = mask = out = out_conf = nullptr;
+            out_s0 = out_scores = out_cand = nullptr;
             sig = sig_ring = nullptr;
             n_rows = out_count = 0;
         }
@@ -184,6 +210,53 @@ namespace
                 ggml_backend_tensor_set(u.t, resolve_upload_source(u.data), 0, u.bytes);
         }
     };
+
+    // DFlash2 grouped dynamic depthwise convolution over one block.
+    //
+    //   out[r][c] = sum_t (base[side][t][c] + delta[r][t][c / group_size]) * x[r-t][c]
+    //
+    // with tap t masked off for the first t rows of the block. `x` is [hidden, n]
+    // and `coef` the [2 * taps * groups, n] projection of the sublayer input, laid
+    // out side-major then tap then group (the order the checkpoint exports).
+    // shift_idx[t] / tap_mask[t] are the constant row-shift and boundary mask for
+    // tap t (both null at t == 0, which is never shifted and never masked).
+    ggml_tensor* df_grouped_conv(
+        ggml_context* ctx, ggml_tensor* x, ggml_tensor* coef, ggml_tensor* base_w,
+        int side, int taps, int groups, int group_size, int hidden, int n,
+        ggml_tensor** shift_idx, ggml_tensor** tap_mask)
+    {
+        ggml_tensor* x3 = ggml_reshape_3d(ctx, x, group_size, groups, n);   // [S, G, n]
+        ggml_tensor* out = nullptr;
+        for (int tap = 0; tap < taps; tap++)
+        {
+            // Static, per-channel half of the kernel: base[side][tap] as [S, G, 1].
+            const std::size_t base_off =
+                (static_cast<std::size_t>(side) * taps + tap) * static_cast<std::size_t>(hidden) * sizeof(float);
+            ggml_tensor* b3 = ggml_view_3d(ctx, base_w, group_size, groups, 1,
+                                           static_cast<std::size_t>(group_size) * sizeof(float),
+                                           static_cast<std::size_t>(hidden) * sizeof(float), base_off);
+
+            // Dynamic, per-token per-group half: the (side, tap) slice of coef.
+            // Materialized rather than viewed because the slice is strided and the
+            // add below broadcasts it over the group's channels.
+            const std::size_t coef_off =
+                (static_cast<std::size_t>(side) * taps + tap) * static_cast<std::size_t>(groups) * sizeof(float);
+            ggml_tensor* d2 = ggml_cont(ctx, ggml_view_2d(ctx, coef, groups, n, coef->nb[1], coef_off));
+            ggml_tensor* d3 = ggml_reshape_3d(ctx, d2, 1, groups, n);
+
+            ggml_tensor* kern = ggml_add(ctx, ggml_repeat(ctx, b3, x3), d3);  // [S, G, n]
+
+            ggml_tensor* xs = x3;
+            if (tap > 0)
+                xs = ggml_reshape_3d(ctx, ggml_get_rows(ctx, x, shift_idx[tap]), group_size, groups, n);
+
+            ggml_tensor* term = ggml_mul(ctx, kern, xs);
+            if (tap > 0)
+                term = ggml_mul(ctx, term, tap_mask[tap]);                    // [1, 1, n]
+            out = (out == nullptr) ? term : ggml_add(ctx, out, term);
+        }
+        return ggml_reshape_2d(ctx, out, hidden, n);
+    }
 }
 
 // ---------------------------------------------------------------------------
@@ -391,7 +464,21 @@ TSG_EXPORT int TSGgml_DFlashDraftBlock(
     const void* out_norm_data,
     const void* tok_embd_data, int tok_embd_type, std::int64_t tok_embd_ne0, std::int64_t tok_embd_ne1, std::int64_t tok_embd_bytes,
     const void* lm_head_data, int lm_head_type, std::int64_t lm_head_ne0, std::int64_t lm_head_ne1, std::int64_t lm_head_bytes,
-    int vocab_size, int* ids_out, float* conf_out)
+    int vocab_size, int* ids_out, float* conf_out,
+    // ---- DFlash2 grouped dynamic convolution (conv_taps == 0 disables) ----
+    int conv_taps, int conv_group_size, int conv_num_groups,
+    void** attn_conv_base_arr,
+    void** attn_conv_proj_arr, int* attn_conv_proj_type_arr,
+    std::int64_t* attn_conv_proj_ne0_arr, std::int64_t* attn_conv_proj_ne1_arr, std::int64_t* attn_conv_proj_bytes_arr,
+    void** ffn_conv_base_arr,
+    void** ffn_conv_proj_arr, int* ffn_conv_proj_type_arr,
+    std::int64_t* ffn_conv_proj_ne0_arr, std::int64_t* ffn_conv_proj_ne1_arr, std::int64_t* ffn_conv_proj_bytes_arr,
+    // ---- DFlash2 candidate selector (sel_rank == 0 disables) ----
+    int sel_rank, int sel_top_k, float sel_logit_scale, float sel_logit_softcap,
+    const void* sel_hidden_data, int sel_hidden_type, std::int64_t sel_hidden_ne0, std::int64_t sel_hidden_ne1, std::int64_t sel_hidden_bytes,
+    const void* sel_pred_data, int sel_pred_type, std::int64_t sel_pred_ne0, std::int64_t sel_pred_ne1, std::int64_t sel_pred_bytes,
+    const void* sel_succ_data, int sel_succ_type, std::int64_t sel_succ_ne0, std::int64_t sel_succ_ne1, std::int64_t sel_succ_bytes,
+    float* sel_scores_out, int* sel_cand_out)
 {
     try
     {
@@ -404,6 +491,28 @@ TSG_EXPORT int TSGgml_DFlashDraftBlock(
         const int q_dim = num_heads * head_dim;
         const int out_count = b;
 
+        const bool use_conv = conv_taps > 0 && conv_group_size > 0 && conv_num_groups > 0
+            && attn_conv_base_arr != nullptr && attn_conv_proj_arr != nullptr
+            && ffn_conv_base_arr != nullptr && ffn_conv_proj_arr != nullptr;
+        const bool use_selector = sel_rank > 0 && sel_top_k > 0
+            && sel_hidden_data != nullptr && sel_pred_data != nullptr && sel_succ_data != nullptr
+            && sel_scores_out != nullptr && sel_cand_out != nullptr;
+        const int gamma = b - 1;               // proposal rows (row 0 is the anchor)
+        if (use_selector && (gamma < 1 || sel_top_k > vocab_size))
+        {
+            set_last_error("DFlash draft: selector needs at least one proposal row and top_k <= vocab.");
+            return 0;
+        }
+        if (use_conv && conv_taps > b)
+        {
+            set_last_error("DFlash draft: conv_kernel_size exceeds the block width.");
+            return 0;
+        }
+        const std::size_t sel_scores_floats = use_selector
+            ? static_cast<std::size_t>(sel_top_k)
+              + static_cast<std::size_t>(sel_top_k) * sel_top_k * (gamma - 1)
+            : 0;
+
         const void* sig = attn_norm_arr[0];
         const void* sig_ring = ring_k_arr[0];
         // Metal is included for the same reason as the inject graph above.
@@ -459,8 +568,23 @@ TSG_EXPORT int TSGgml_DFlashDraftBlock(
             // output, silently collapsed acceptance).
             if (g_backend_type == BACKEND_TYPE_METAL)
                 ggml_backend_synchronize(g_backend);
-            ggml_backend_tensor_get(dc->out, ids_out, 0, static_cast<std::size_t>(b) * sizeof(std::int32_t));
-            finalize_compute_with_download(dc->out_conf, conf_out, static_cast<std::size_t>(b) * sizeof(float));
+            if (use_selector)
+            {
+                ggml_backend_tensor_get(dc->out_cand, sel_cand_out, 0,
+                    static_cast<std::size_t>(sel_top_k) * gamma * sizeof(std::int32_t));
+                ggml_backend_tensor_get(dc->out_s0, sel_scores_out, 0,
+                    static_cast<std::size_t>(sel_top_k) * sizeof(float));
+                if (dc->out_scores != nullptr)
+                {
+                    finalize_compute_with_download(dc->out_scores, sel_scores_out + sel_top_k,
+                        (sel_scores_floats - sel_top_k) * sizeof(float));
+                }
+            }
+            else
+            {
+                ggml_backend_tensor_get(dc->out, ids_out, 0, static_cast<std::size_t>(b) * sizeof(std::int32_t));
+                finalize_compute_with_download(dc->out_conf, conf_out, static_cast<std::size_t>(b) * sizeof(float));
+            }
             host_read_barrier();
             clear_last_error();
             return 1;
@@ -488,9 +612,40 @@ TSG_EXPORT int TSGgml_DFlashDraftBlock(
         ggml_tensor* mask_t = ggml_new_tensor_4d(ctx, GGML_TYPE_F16, kv_len, b, 1, 1);
         ggml_set_input(ids_t); ggml_set_input(pos_t); ggml_set_input(mask_t);
 
+        // Constant per-tap row shift and block-boundary mask for the DFlash2
+        // convolution. They depend only on the block width, which is part of the
+        // cache key, so they are written once at build time and survive replay.
+        std::vector<ggml_tensor*> conv_shift(use_conv ? conv_taps : 0, nullptr);
+        std::vector<ggml_tensor*> conv_mask(use_conv ? conv_taps : 0, nullptr);
+        std::vector<std::vector<std::int32_t>> conv_shift_data(use_conv ? conv_taps : 0);
+        std::vector<std::vector<float>> conv_mask_data(use_conv ? conv_taps : 0);
+        for (int tap = 1; tap < (use_conv ? conv_taps : 0); tap++)
+        {
+            conv_shift[tap] = ggml_new_tensor_1d(ctx, GGML_TYPE_I32, b);
+            conv_mask[tap] = ggml_new_tensor_3d(ctx, GGML_TYPE_F32, 1, 1, b);
+            ggml_set_input(conv_shift[tap]);
+            ggml_set_input(conv_mask[tap]);
+            conv_shift_data[tap].resize(b);
+            conv_mask_data[tap].resize(b);
+            for (int r = 0; r < b; r++)
+            {
+                conv_shift_data[tap][r] = r >= tap ? r - tap : 0;
+                conv_mask_data[tap][r] = r >= tap ? 1.0f : 0.0f;
+            }
+        }
+
         ggml_tensor* tok_t = ggml_new_tensor_2d(ctx, static_cast<ggml_type>(tok_embd_type), tok_embd_ne0, tok_embd_ne1);
         ggml_tensor* out_norm_t = ggml_new_tensor_1d(ctx, GGML_TYPE_F32, hidden_size);
         ggml_tensor* lm_head_t = ggml_new_tensor_2d(ctx, static_cast<ggml_type>(lm_head_type), lm_head_ne0, lm_head_ne1);
+        ggml_tensor* sel_hidden_t = nullptr;
+        ggml_tensor* sel_pred_t = nullptr;
+        ggml_tensor* sel_succ_t = nullptr;
+        if (use_selector)
+        {
+            sel_hidden_t = ggml_new_tensor_2d(ctx, static_cast<ggml_type>(sel_hidden_type), sel_hidden_ne0, sel_hidden_ne1);
+            sel_pred_t   = ggml_new_tensor_2d(ctx, static_cast<ggml_type>(sel_pred_type),   sel_pred_ne0,   sel_pred_ne1);
+            sel_succ_t   = ggml_new_tensor_2d(ctx, static_cast<ggml_type>(sel_succ_type),   sel_succ_ne0,   sel_succ_ne1);
+        }
 
         // llama.cpp's dflash graph feeds build_inp_embd straight in: no embedding
         // scale and (unlike the Muse-Glimmer trunk) no weightless input RMSNorm.
@@ -513,6 +668,15 @@ TSG_EXPORT int TSGgml_DFlashDraftBlock(
             lt.down_w = ggml_new_tensor_2d(ctx, static_cast<ggml_type>(down_type_arr[l]), down_ne0_arr[l], down_ne1_arr[l]);
             lt.ring_k = ggml_new_tensor_3d(ctx, static_cast<ggml_type>(ring_dtype), head_dim, ring_rows, num_kv_heads);
             lt.ring_v = ggml_new_tensor_3d(ctx, static_cast<ggml_type>(ring_dtype), head_dim, ring_rows, num_kv_heads);
+            if (use_conv)
+            {
+                lt.attn_conv_base = ggml_new_tensor_3d(ctx, GGML_TYPE_F32, hidden_size, conv_taps, 2);
+                lt.ffn_conv_base = ggml_new_tensor_3d(ctx, GGML_TYPE_F32, hidden_size, conv_taps, 2);
+                lt.attn_conv_proj = ggml_new_tensor_2d(ctx, static_cast<ggml_type>(attn_conv_proj_type_arr[l]),
+                                                       attn_conv_proj_ne0_arr[l], attn_conv_proj_ne1_arr[l]);
+                lt.ffn_conv_proj = ggml_new_tensor_2d(ctx, static_cast<ggml_type>(ffn_conv_proj_type_arr[l]),
+                                                      ffn_conv_proj_ne0_arr[l], ffn_conv_proj_ne1_arr[l]);
+            }
         }
 
         for (int l = 0; l < num_layers; l++)
@@ -520,6 +684,19 @@ TSG_EXPORT int TSGgml_DFlashDraftBlock(
             auto& lt = layers[l];
 
             ggml_tensor* h = ggml_mul(ctx, ggml_rms_norm(ctx, inpL, eps), lt.attn_norm_w);
+
+            // DFlash2: one projection of the sublayer input carries both filters,
+            // so the output-side coefficients are computed here and held across the
+            // attention - they are keyed on the INPUT, not on what attention made.
+            ggml_tensor* attn_conv_coef = nullptr;
+            if (use_conv)
+            {
+                attn_conv_coef = ggml_mul_mat(ctx, lt.attn_conv_proj, h);   // [2*taps*G, b]
+                h = df_grouped_conv(ctx, h, attn_conv_coef, lt.attn_conv_base, /*side=*/0,
+                                    conv_taps, conv_num_groups, conv_group_size, hidden_size, b,
+                                    conv_shift.data(), conv_mask.data());
+            }
+
             ggml_tensor* q = ggml_mul_mat(ctx, lt.q_w, h);
             ggml_tensor* k = ggml_mul_mat(ctx, lt.k_w, h);
             ggml_tensor* v = ggml_mul_mat(ctx, lt.v_w, h);
@@ -561,23 +738,119 @@ TSG_EXPORT int TSGgml_DFlashDraftBlock(
             }
 
             ggml_tensor* attn_out = ggml_mul_mat(ctx, lt.o_w, attn_flat);
+            if (use_conv)
+            {
+                attn_out = df_grouped_conv(ctx, attn_out, attn_conv_coef, lt.attn_conv_base, /*side=*/1,
+                                           conv_taps, conv_num_groups, conv_group_size, hidden_size, b,
+                                           conv_shift.data(), conv_mask.data());
+            }
             ggml_tensor* ffn_inp = ggml_add(ctx, attn_out, inpL);
 
             ggml_tensor* fh = ggml_mul(ctx, ggml_rms_norm(ctx, ffn_inp, eps), lt.ffn_norm_w);
+            ggml_tensor* ffn_conv_coef = nullptr;
+            if (use_conv)
+            {
+                ffn_conv_coef = ggml_mul_mat(ctx, lt.ffn_conv_proj, fh);
+                fh = df_grouped_conv(ctx, fh, ffn_conv_coef, lt.ffn_conv_base, /*side=*/0,
+                                     conv_taps, conv_num_groups, conv_group_size, hidden_size, b,
+                                     conv_shift.data(), conv_mask.data());
+            }
             ggml_tensor* gate = ggml_mul_mat(ctx, lt.gate_w, fh);
             ggml_tensor* up   = ggml_mul_mat(ctx, lt.up_w, fh);
             ggml_tensor* act  = ggml_mul(ctx, ggml_silu(ctx, gate), up);
             ggml_tensor* down = ggml_mul_mat(ctx, lt.down_w, act);
+            if (use_conv)
+            {
+                down = df_grouped_conv(ctx, down, ffn_conv_coef, lt.ffn_conv_base, /*side=*/1,
+                                       conv_taps, conv_num_groups, conv_group_size, hidden_size, b,
+                                       conv_shift.data(), conv_mask.data());
+            }
             inpL = ggml_add(ctx, down, ffn_inp);
         }
 
         ggml_tensor* cur = ggml_mul(ctx, ggml_rms_norm(ctx, inpL, eps), out_norm_t);
+
+        ggml_tensor* sel_s0 = nullptr;
+        ggml_tensor* sel_scores = nullptr;
+        ggml_tensor* sel_cand = nullptr;
+        if (use_selector)
+        {
+            // Only the gamma PROPOSAL rows reach the head; row 0 is the anchor's
+            // own prediction, which the selector never consumes.
+            ggml_tensor* pred_h = ggml_view_2d(ctx, cur, hidden_size, gamma, cur->nb[1], cur->nb[1]);
+            ggml_tensor* sel_logits = ggml_mul_mat(ctx, lm_head_t, pred_h);        // [vocab, gamma]
+            sel_cand = ggml_top_k(ctx, sel_logits, sel_top_k);                     // [k, gamma] I32
+            if (!backend_supports_op(sel_cand))
+            {
+                set_last_error("DFlash draft: this backend has no top-k over the vocabulary.");
+                if (can_persist) ggml_free(ctx);
+                return 0;
+            }
+            ggml_set_output(sel_cand);
+
+            // unary[e][c]: the head logit of candidate c at position e.
+            ggml_tensor* logits3 = ggml_reshape_3d(ctx, sel_logits, 1, vocab_size, gamma);
+            ggml_tensor* unary = ggml_get_rows(ctx, logits3, sel_cand);            // [1, k, gamma]
+            ggml_tensor* unary_kg = ggml_reshape_3d(ctx, unary, sel_top_k, 1, gamma);
+            // The target's LM-head transform. Applied here, after the top-k, because
+            // both halves are monotonic (the candidate set cannot change) and this
+            // touches k*gamma values instead of vocab*gamma. Without it the unary
+            // term enters the lattice at the wrong scale and swamps the transition
+            // scores it is meant to compete with.
+            if (sel_logit_scale != 1.0f)
+                unary_kg = ggml_scale(ctx, unary_kg, sel_logit_scale);
+            if (sel_logit_softcap > 0.0f)
+            {
+                unary_kg = ggml_scale(ctx,
+                    ggml_tanh(ctx, ggml_scale(ctx, unary_kg, 1.0f / sel_logit_softcap)),
+                    sel_logit_softcap);
+            }
+
+            ggml_tensor* ph = ggml_mul_mat(ctx, sel_hidden_t, pred_h);             // [r, gamma]
+            ggml_tensor* cand_flat = ggml_reshape_1d(ctx, sel_cand, static_cast<std::int64_t>(sel_top_k) * gamma);
+            ggml_tensor* keys = ggml_get_rows(ctx, sel_succ_t, cand_flat);         // [r, k*gamma]
+            ggml_tensor* keys3 = ggml_reshape_3d(ctx, keys, sel_rank, sel_top_k, gamma);
+
+            // Position 0's predecessor is the verified anchor, which is block_ids[0]
+            // - the same tensor the embedding lookup already reads.
+            ggml_tensor* anchor_id = ggml_view_1d(ctx, ids_t, 1, 0);
+            ggml_tensor* a0 = ggml_get_rows(ctx, sel_pred_t, anchor_id);           // [r, 1]
+            ggml_tensor* ph0 = ggml_view_2d(ctx, ph, sel_rank, 1, ph->nb[1], 0);
+            ggml_tensor* m0 = ggml_mul(ctx, a0, ph0);                              // [r, 1]
+            ggml_tensor* keys0 = ggml_view_2d(ctx, keys3, sel_rank, sel_top_k, keys3->nb[1], 0);
+            sel_s0 = ggml_add(ctx, ggml_mul_mat(ctx, keys0, m0),
+                              ggml_view_2d(ctx, unary_kg, sel_top_k, 1, unary_kg->nb[1], 0));
+            ggml_set_output(sel_s0);                                               // [k, 1]
+
+            if (gamma > 1)
+            {
+                // Position e's predecessors are position e-1's candidates, so the
+                // predecessor ids are the same tensor shifted by one slot.
+                ggml_tensor* prev_flat = ggml_view_1d(ctx, sel_cand,
+                    static_cast<std::int64_t>(sel_top_k) * (gamma - 1), 0);
+                ggml_tensor* preds = ggml_get_rows(ctx, sel_pred_t, prev_flat);
+                ggml_tensor* preds3 = ggml_reshape_3d(ctx, preds, sel_rank, sel_top_k, gamma - 1);
+                ggml_tensor* ph_rest = ggml_view_3d(ctx, ph, sel_rank, 1, gamma - 1,
+                                                    ph->nb[1], ph->nb[1], ph->nb[1]);
+                ggml_tensor* m = ggml_mul(ctx, preds3, ph_rest);                   // [r, k, gamma-1]
+                ggml_tensor* keys_rest = ggml_view_3d(ctx, keys3, sel_rank, sel_top_k, gamma - 1,
+                                                      keys3->nb[1], keys3->nb[2], keys3->nb[2]);
+                // [k(candidate), k(predecessor), gamma-1] -- candidate fastest, which
+                // is the row the host walk scans.
+                sel_scores = ggml_mul_mat(ctx, keys_rest, m);
+                ggml_tensor* u_rest = ggml_view_3d(ctx, unary_kg, sel_top_k, 1, gamma - 1,
+                                                   unary_kg->nb[1], unary_kg->nb[2], unary_kg->nb[2]);
+                sel_scores = ggml_add(ctx, sel_scores, u_rest);
+                ggml_set_output(sel_scores);
+            }
+        }
+
         // The TARGET's LM head, with NEITHER logit_scale NOR the tanh softcap:
         // llama.cpp's dflash graph ends at build_lora_mm(output, cur).
-        ggml_tensor* logits = ggml_mul_mat(ctx, lm_head_t, cur);        // [vocab, b]
+        ggml_tensor* logits = use_selector ? nullptr : ggml_mul_mat(ctx, lm_head_t, cur);        // [vocab, b]
         // Softmax on device: argmax is invariant under it, and the winning
         // probability IS the confidence the executor multiplies cumulatively.
-        ggml_tensor* probs = ggml_soft_max(ctx, logits);
+        ggml_tensor* probs = use_selector ? nullptr : ggml_soft_max(ctx, logits);
 
         // Reduce to (argmax id, winning probability) ON DEVICE. llama.cpp pulls the
         // whole [vocab, b] block back every draft step -- 202048*16*4 = 12.9 MB of
@@ -585,20 +858,37 @@ TSG_EXPORT int TSGgml_DFlashDraftBlock(
         // large share of its per-step cost. Two b-element tensors carry everything
         // the caller needs: argmax is invariant under softmax, and the winning
         // probability IS the confidence the executor multiplies cumulatively.
-        ggml_tensor* am = ggml_argmax(ctx, probs);                            // [b] I32
-        ggml_tensor* am2 = ggml_reshape_2d(ctx, am, 1, b);                    // [1, b]
-        ggml_tensor* pr3 = ggml_reshape_3d(ctx, probs, 1, vocab_size, b);     // [1, vocab, b]
-        ggml_tensor* mp = ggml_get_rows(ctx, pr3, am2);                       // [1, 1, b] F32
-        ggml_tensor* out = ggml_new_tensor_1d(ctx, GGML_TYPE_I32, b);
-        ggml_tensor* out_conf = ggml_new_tensor_1d(ctx, GGML_TYPE_F32, b);
-        ggml_tensor* out_node = ggml_cpy(ctx, am, out);
-        ggml_tensor* conf_node = ggml_cpy(ctx, ggml_reshape_1d(ctx, mp, b), out_conf);
-        ggml_set_output(out_node);
-        ggml_set_output(conf_node);
-
-        ggml_cgraph* graph = ggml_new_graph_custom(ctx, static_cast<std::size_t>(num_layers) * 128 + 512, false);
-        ggml_build_forward_expand(graph, out_node);
-        ggml_build_forward_expand(graph, conf_node);
+        ggml_tensor* out = nullptr;
+        ggml_tensor* out_conf = nullptr;
+        ggml_tensor* out_node = nullptr;
+        ggml_tensor* conf_node = nullptr;
+        if (!use_selector)
+        {
+            ggml_tensor* am = ggml_argmax(ctx, probs);                            // [b] I32
+            ggml_tensor* am2 = ggml_reshape_2d(ctx, am, 1, b);                    // [1, b]
+            ggml_tensor* pr3 = ggml_reshape_3d(ctx, probs, 1, vocab_size, b);     // [1, vocab, b]
+            ggml_tensor* mp = ggml_get_rows(ctx, pr3, am2);                       // [1, 1, b] F32
+            out = ggml_new_tensor_1d(ctx, GGML_TYPE_I32, b);
+            out_conf = ggml_new_tensor_1d(ctx, GGML_TYPE_F32, b);
+            out_node = ggml_cpy(ctx, am, out);
+            conf_node = ggml_cpy(ctx, ggml_reshape_1d(ctx, mp, b), out_conf);
+            ggml_set_output(out_node);
+            ggml_set_output(conf_node);
+        }
+
+        ggml_cgraph* graph = ggml_new_graph_custom(ctx, static_cast<std::size_t>(num_layers) * 256 + 1024, false);
+        if (use_selector)
+        {
+            ggml_build_forward_expand(graph, sel_cand);
+            ggml_build_forward_expand(graph, sel_s0);
+            if (sel_scores != nullptr)
+                ggml_build_forward_expand(graph, sel_scores);
+        }
+        else
+        {
+            ggml_build_forward_expand(graph, out_node);
+            ggml_build_forward_expand(graph, conf_node);
+        }
 
         DfBinder binder;
         binder.dev = ggml_backend_get_device(g_backend);
@@ -621,6 +911,26 @@ TSG_EXPORT int TSGgml_DFlashDraftBlock(
             binder.bind(lt.k_norm_w,    k_norm_arr[l],    head_norm_bytes, true);
             binder.bind(lt.ring_k, ring_k_arr[l], ring_bytes, true, GGML_BACKEND_BUFFER_USAGE_COMPUTE);
             binder.bind(lt.ring_v, ring_v_arr[l], ring_bytes, true, GGML_BACKEND_BUFFER_USAGE_COMPUTE);
+            if (use_conv)
+            {
+                const std::size_t conv_base_bytes =
+                    2u * static_cast<std::size_t>(conv_taps) * hidden_size * sizeof(float);
+                binder.bind(lt.attn_conv_base, attn_conv_base_arr[l], conv_base_bytes, true);
+                binder.bind(lt.ffn_conv_base, ffn_conv_base_arr[l], conv_base_bytes, true);
+                binder.bind(lt.attn_conv_proj, attn_conv_proj_arr[l],
+                            static_cast<std::size_t>(attn_conv_proj_bytes_arr[l]), true);
+                binder.bind(lt.ffn_conv_proj, ffn_conv_proj_arr[l],
+                            static_cast<std::size_t>(ffn_conv_proj_bytes_arr[l]), true);
+            }
+        }
+        if (use_selector)
+        {
+            binder.bind(sel_hidden_t, const_cast<void*>(sel_hidden_data),
+                        static_cast<std::size_t>(sel_hidden_bytes), true);
+            binder.bind(sel_pred_t, const_cast<void*>(sel_pred_data),
+                        static_cast<std::size_t>(sel_pred_bytes), true);
+            binder.bind(sel_succ_t, const_cast<void*>(sel_succ_data),
+                        static_cast<std::size_t>(sel_succ_bytes), true);
         }
         binder.bind(out_norm_t, const_cast<void*>(out_norm_data), norm_bytes, true);
         binder.bind(tok_t, const_cast<void*>(tok_embd_data), static_cast<std::size_t>(tok_embd_bytes), true);
@@ -656,6 +966,13 @@ TSG_EXPORT int TSGgml_DFlashDraftBlock(
         ggml_backend_tensor_set(ids_t, block_ids, 0, static_cast<std::size_t>(b) * sizeof(std::int32_t));
         ggml_backend_tensor_set(pos_t, positions, 0, static_cast<std::size_t>(b) * sizeof(std::int32_t));
         ggml_backend_tensor_set(mask_t, mask_data.data(), 0, mask_data.size() * sizeof(ggml_fp16_t));
+        for (int tap = 1; tap < (use_conv ? conv_taps : 0); tap++)
+        {
+            ggml_backend_tensor_set(conv_shift[tap], conv_shift_data[tap].data(), 0,
+                static_cast<std::size_t>(b) * sizeof(std::int32_t));
+            ggml_backend_tensor_set(conv_mask[tap], conv_mask_data[tap].data(), 0,
+                static_cast<std::size_t>(b) * sizeof(float));
+        }
 
         if (ggml_backend_graph_compute(g_backend, graph) != GGML_STATUS_SUCCESS)
         {
@@ -666,10 +983,25 @@ TSG_EXPORT int TSGgml_DFlashDraftBlock(
         // See the replay path above: Metal must drain before reading the ids.
         if (g_backend_type == BACKEND_TYPE_METAL)
             ggml_backend_synchronize(g_backend);
-        ggml_backend_tensor_get(out, ids_out, 0, static_cast<std::size_t>(b) * sizeof(std::int32_t));
-        finalize_compute_with_download(out_conf, conf_out, static_cast<std::size_t>(b) * sizeof(float));
-        // Unconditional: conf_out is the caller's host confidence array and on
-        // Metal async mode the download above is only QUEUED.
+        if (use_selector)
+        {
+            ggml_backend_tensor_get(sel_cand, sel_cand_out, 0,
+                static_cast<std::size_t>(sel_top_k) * gamma * sizeof(std::int32_t));
+            ggml_backend_tensor_get(sel_s0, sel_scores_out, 0,
+                static_cast<std::size_t>(sel_top_k) * sizeof(float));
+            if (sel_scores != nullptr)
+            {
+                finalize_compute_with_download(sel_scores, sel_scores_out + sel_top_k,
+                    (sel_scores_floats - sel_top_k) * sizeof(float));
+            }
+        }
+        else
+        {
+            ggml_backend_tensor_get(out, ids_out, 0, static_cast<std::size_t>(b) * sizeof(std::int32_t));
+            finalize_compute_with_download(out_conf, conf_out, static_cast<std::size_t>(b) * sizeof(float));
+        }
+        // Unconditional: the outputs above land in caller host arrays and on Metal
+        // async mode the download is only QUEUED.
         host_read_barrier();
 
         if (can_persist && slot != nullptr)
@@ -677,6 +1009,7 @@ TSG_EXPORT int TSGgml_DFlashDraftBlock(
             slot->ctx = ctx; slot->buffer = persist_buf; slot->graph = graph;
             slot->in_main = ids_t; slot->pos = pos_t; slot->mask = mask_t;
             slot->out = out; slot->out_conf = out_conf;
+            slot->out_s0 = sel_s0; slot->out_scores = sel_scores; slot->out_cand = sel_cand;
             slot->sig = sig; slot->sig_ring = sig_ring; slot->n_rows = b; slot->out_count = out_count;
             slot->valid = true;
         }
diff --git a/TensorSharp.GGML.Native/ggml_ops_fused.cpp b/TensorSharp.GGML.Native/ggml_ops_fused.cpp
index a90467f8..e809eb45 100644
--- a/TensorSharp.GGML.Native/ggml_ops_fused.cpp
+++ b/TensorSharp.GGML.Native/ggml_ops_fused.cpp
@@ -3081,6 +3081,298 @@ static int fused_qwen35_vision_encoder_f32_impl(
     return 1;
 }
 
+
+// ---------------------------------------------------------------------------
+// GLM-5.3-Flash (glm5next) vision encoder: the GLM-OCR ViT as ONE graph.
+// Differences from the Qwen3.5 tower above: RMS norms without biases, per-head
+// RMS q/k norms (one [head_dim] weight shared by every head), and a SwiGLU-clamp
+// MLP (gate <= L, up in [-L, L]) instead of the GELU 2-layer MLP. The rope, the
+// attention core and the whole-graph mechanics are shared.
+// ---------------------------------------------------------------------------
+
+static ggml_tensor* build_glm_vision_attn_subgraph(
+    ggml_context* ctx, ggml_tensor* cur,
+    ggml_tensor* ln_w_t, float eps,
+    ggml_tensor* qkv_w_t, ggml_tensor* qkv_b_t,
+    ggml_tensor* qn_w_t, ggml_tensor* kn_w_t,
+    ggml_tensor* out_w_t, ggml_tensor* out_b_t,
+    ggml_tensor* cos_t, ggml_tensor* sin_t,
+    int rows, int hidden, int num_heads, int head_dim, int half_dim,
+    float attn_scale)
+{
+    const int triple_hidden = 3 * hidden;
+    ggml_tensor* inp = ggml_cont(ctx, cur);
+
+    // RMS norm (no bias)
+    ggml_tensor* ln_out = ggml_mul(ctx, ggml_rms_norm(ctx, inp, eps), ln_w_t);
+
+    ggml_tensor* qkv = ggml_mul_mat(ctx, qkv_w_t, ln_out);
+    ggml_tensor* qkv_biased = ggml_add(ctx, qkv, ggml_repeat(ctx, ggml_reshape_2d(ctx, qkv_b_t, triple_hidden, 1), qkv));
+
+    std::size_t row_bytes = static_cast<std::size_t>(triple_hidden) * sizeof(float);
+    std::size_t d_bytes = static_cast<std::size_t>(hidden) * sizeof(float);
+    ggml_tensor* q_raw = ggml_cont(ctx, ggml_view_2d(ctx, qkv_biased, hidden, rows, row_bytes, 0));
+    ggml_tensor* k_raw = ggml_cont(ctx, ggml_view_2d(ctx, qkv_biased, hidden, rows, row_bytes, d_bytes));
+    ggml_tensor* v_raw = ggml_cont(ctx, ggml_view_2d(ctx, qkv_biased, hidden, rows, row_bytes, 2 * d_bytes));
+
+    ggml_tensor* q_3d = ggml_reshape_3d(ctx, q_raw, head_dim, num_heads, rows);
+    ggml_tensor* k_3d = ggml_reshape_3d(ctx, k_raw, head_dim, num_heads, rows);
+
+    // per-head RMS q/k norms: rms_norm normalizes dim 0 (= head_dim) and the
+    // [head_dim] weight broadcasts across heads and rows.
+    q_3d = ggml_mul(ctx, ggml_rms_norm(ctx, q_3d, eps), qn_w_t);
+    k_3d = ggml_mul(ctx, ggml_rms_norm(ctx, k_3d, eps), kn_w_t);
+
+    ggml_tensor* cos_3d = ggml_reshape_3d(ctx, cos_t, half_dim, 1, rows);
+    ggml_tensor* sin_3d = ggml_reshape_3d(ctx, sin_t, half_dim, 1, rows);
+
+    std::size_t head_row_bytes = static_cast<std::size_t>(head_dim) * sizeof(float);
+    std::size_t half_bytes_local = static_cast<std::size_t>(half_dim) * sizeof(float);
+    auto apply_rope = [&](ggml_tensor* x_3d) -> ggml_tensor* {
+        ggml_tensor* x_lo = ggml_view_3d(ctx, x_3d, half_dim, num_heads, rows,
+            head_row_bytes, head_row_bytes * num_heads, 0);
+        ggml_tensor* x_hi = ggml_view_3d(ctx, x_3d, half_dim, num_heads, rows,
+            head_row_bytes, head_row_bytes * num_heads, half_bytes_local);
+        ggml_tensor* lo_c = ggml_cont(ctx, x_lo);
+        ggml_tensor* hi_c = ggml_cont(ctx, x_hi);
+        ggml_tensor* out_lo = ggml_sub(ctx, ggml_mul(ctx, lo_c, cos_3d), ggml_mul(ctx, hi_c, sin_3d));
+        ggml_tensor* out_hi = ggml_add(ctx, ggml_mul(ctx, lo_c, sin_3d), ggml_mul(ctx, hi_c, cos_3d));
+        return ggml_concat(ctx, out_lo, out_hi, 0);
+    };
+
+    // The q/k-norm outputs are non-contiguous mul results over reshaped views;
+    // cont them so the rope's views land on plain layouts.
+    ggml_tensor* q_roped = apply_rope(ggml_cont(ctx, q_3d));
+    ggml_tensor* k_roped = apply_rope(ggml_cont(ctx, k_3d));
+    ggml_tensor* v_3d = ggml_reshape_3d(ctx, v_raw, head_dim, num_heads, rows);
+
+    ggml_tensor* q_perm = ggml_permute(ctx, q_roped, 0, 2, 1, 3);
+    ggml_tensor* k_perm = ggml_permute(ctx, k_roped, 0, 2, 1, 3);
+
+    ggml_tensor* attn_out = build_vision_attention(ctx, q_perm, k_perm, v_3d,
+        rows, num_heads, head_dim, attn_scale);
+    ggml_tensor* attn_flat = ggml_reshape_2d(ctx, ggml_cont(ctx, attn_out), hidden, rows);
+
+    ggml_tensor* out_proj = ggml_mul_mat(ctx, out_w_t, attn_flat);
+    ggml_tensor* out_biased = ggml_add(ctx, out_proj, ggml_repeat(ctx, ggml_reshape_2d(ctx, out_b_t, hidden, 1), out_proj));
+
+    return ggml_add(ctx, inp, out_biased);
+}
+
+static ggml_tensor* build_glm_vision_mlp_subgraph(
+    ggml_context* ctx, ggml_tensor* cur,
+    ggml_tensor* ln_w_t, float eps,
+    ggml_tensor* gate_w_t, ggml_tensor* gate_b_t,
+    ggml_tensor* up_w_t, ggml_tensor* up_b_t,
+    ggml_tensor* down_w_t, ggml_tensor* down_b_t,
+    int rows, int hidden, int dff, float swiglu_limit)
+{
+    ggml_tensor* inp = ggml_cont(ctx, cur);
+    ggml_tensor* ln_out = ggml_mul(ctx, ggml_rms_norm(ctx, inp, eps), ln_w_t);
+
+    ggml_tensor* gate = ggml_mul_mat(ctx, gate_w_t, ln_out);
+    gate = ggml_add(ctx, gate, ggml_repeat(ctx, ggml_reshape_2d(ctx, gate_b_t, dff, 1), gate));
+    ggml_tensor* up = ggml_mul_mat(ctx, up_w_t, ln_out);
+    up = ggml_add(ctx, up, ggml_repeat(ctx, ggml_reshape_2d(ctx, up_b_t, dff, 1), up));
+
+    if (swiglu_limit > 0.0f)
+    {
+        gate = ggml_clamp(ctx, gate, -INFINITY, swiglu_limit);
+        up = ggml_clamp(ctx, up, -swiglu_limit, swiglu_limit);
+    }
+    ggml_tensor* h = ggml_mul(ctx, ggml_silu(ctx, gate), up);
+
+    ggml_tensor* down = ggml_mul_mat(ctx, down_w_t, h);
+    down = ggml_add(ctx, down, ggml_repeat(ctx, ggml_reshape_2d(ctx, down_b_t, hidden, 1), down));
+
+    return ggml_add(ctx, inp, down);
+}
+
+static int fused_glm_vision_encoder_f32_impl(
+    const TensorView2DDesc& hidden_desc,
+    int block_count, float eps, float attn_scale, float swiglu_limit,
+    int num_patches, int num_heads, int head_dim, int half_dim,
+    const float* cos_table, const float* sin_table,
+    const float* const* ln1_w,
+    const float* const* qkv_w, const float* const* qkv_b,
+    const float* const* qn_w, const float* const* kn_w,
+    const float* const* out_w, const float* const* out_b,
+    const float* const* ln2_w,
+    const float* const* gate_w, const float* const* gate_b,
+    const float* const* up_w,  const float* const* up_b,
+    const float* const* down_w, const float* const* down_b,
+    int ln_dim,
+    int qkv_ne0, int qkv_ne1, std::size_t qkv_bytes,
+    int out_ne0, int out_ne1, std::size_t out_bytes,
+    int ffn_ne0, int ffn_ne1, std::size_t ffn_up_bytes, std::size_t ffn_down_bytes)
+{
+    if (!ensure_backend()) return 0;
+    if (!validate_desc(hidden_desc, "hidden")) return 0;
+    if (block_count <= 0 || block_count > 128) { set_last_error("glm_vision_encoder: bad block_count"); return 0; }
+
+    const int rows = hidden_desc.dim0;
+    const int hidden = hidden_desc.dim1;
+    const int dff = ffn_ne1;
+    const std::size_t cos_sin_elems = static_cast<std::size_t>(num_patches) * half_dim;
+
+    const std::size_t ctx_size = static_cast<std::size_t>(32) * 1024 * 1024;
+    PooledContextHandle context;
+    if (!context.init(ctx_size)) { set_last_error("glm_vision_encoder: ctx init failed."); return 0; }
+    ggml_context* ctx = context.value;
+
+    std::vector<BufferHandle> host_ptr_buffers;
+    bool use_zero_copy = can_map_standard_view(hidden_desc);
+    TensorBinding hidden_binding;
+    if (use_zero_copy)
+    {
+        ggml_backend_buffer_t buf = nullptr;
+        if (!create_binding_from_host_ptr_2d(ctx, g_backend, hidden_desc, hidden_binding, buf))
+        {
+            use_zero_copy = false;
+            hidden_binding = create_standard_binding(ctx, hidden_desc);
+        }
+        else
+            host_ptr_buffers.emplace_back(buf);
+    }
+    else
+        hidden_binding = create_standard_binding(ctx, hidden_desc);
+    if (!hidden_binding.storage) { set_last_error("glm_vision_encoder: hidden bind failed."); return 0; }
+
+    ggml_tensor* cos_t = ggml_new_tensor_1d(ctx, GGML_TYPE_F32, cos_sin_elems);
+    ggml_tensor* sin_t = ggml_new_tensor_1d(ctx, GGML_TYPE_F32, cos_sin_elems);
+    if (!cos_t || !sin_t) { set_last_error("glm_vision_encoder: cos/sin alloc failed."); return 0; }
+    ggml_set_input(cos_t);
+    ggml_set_input(sin_t);
+
+    ggml_backend_dev_t dev = ggml_backend_get_device(g_backend);
+    struct HostBinding { ggml_tensor* tensor; const void* data; std::size_t bytes; };
+    std::vector<HostBinding> upload_list;
+    auto bind_w = [&](ggml_tensor* t, const void* data, std::size_t bytes) {
+        if (t == nullptr || data == nullptr) return;
+        if (bytes >= 4096 && dev != nullptr)
+        {
+            ggml_backend_buffer_t buf = nullptr; void* addr = nullptr; bool need = false;
+            if (try_get_cacheable_tensor_buffer(g_backend, dev, t, const_cast<void*>(data), bytes, buf, addr, need))
+            {
+                if (ggml_backend_tensor_alloc(buf, t, addr) == GGML_STATUS_SUCCESS)
+                {
+                    if (need) upload_list.push_back({ t, data, bytes });
+                    return;
+                }
+                invalidate_cached_buffer(const_cast<void*>(data));
+            }
+        }
+        ggml_set_input(t);
+        upload_list.push_back({ t, data, bytes });
+    };
+
+    ggml_tensor* cur = hidden_binding.tensor;
+    for (int b = 0; b < block_count; b++)
+    {
+        ggml_tensor* ln1w_t = ggml_new_tensor_1d(ctx, GGML_TYPE_F32, ln_dim);
+        ggml_tensor* qkvw_t = ggml_new_tensor_2d(ctx, GGML_TYPE_F32, qkv_ne0, qkv_ne1);
+        ggml_tensor* qkvb_t = ggml_new_tensor_1d(ctx, GGML_TYPE_F32, qkv_ne1);
+        ggml_tensor* qnw_t = ggml_new_tensor_1d(ctx, GGML_TYPE_F32, head_dim);
+        ggml_tensor* knw_t = ggml_new_tensor_1d(ctx, GGML_TYPE_F32, head_dim);
+        ggml_tensor* outw_t = ggml_new_tensor_2d(ctx, GGML_TYPE_F32, out_ne0, out_ne1);
+        ggml_tensor* outb_t = ggml_new_tensor_1d(ctx, GGML_TYPE_F32, out_ne1);
+        ggml_tensor* ln2w_t = ggml_new_tensor_1d(ctx, GGML_TYPE_F32, ln_dim);
+        ggml_tensor* gatew_t = ggml_new_tensor_2d(ctx, GGML_TYPE_F32, ffn_ne0, ffn_ne1);
+        ggml_tensor* gateb_t = ggml_new_tensor_1d(ctx, GGML_TYPE_F32, ffn_ne1);
+        ggml_tensor* upw_t = ggml_new_tensor_2d(ctx, GGML_TYPE_F32, ffn_ne0, ffn_ne1);
+        ggml_tensor* upb_t = ggml_new_tensor_1d(ctx, GGML_TYPE_F32, ffn_ne1);
+        ggml_tensor* downw_t = ggml_new_tensor_2d(ctx, GGML_TYPE_F32, ffn_ne1, ffn_ne0);
+        ggml_tensor* downb_t = ggml_new_tensor_1d(ctx, GGML_TYPE_F32, ffn_ne0);
+        if (!ln1w_t || !qkvw_t || !outw_t || !ln2w_t || !gatew_t || !upw_t || !downw_t)
+        { set_last_error("glm_vision_encoder: block tensor alloc failed."); return 0; }
+
+        bind_w(ln1w_t, ln1_w[b], static_cast<std::size_t>(ln_dim) * sizeof(float));
+        bind_w(qkvw_t, qkv_w[b], qkv_bytes);
+        bind_w(qkvb_t, qkv_b[b], static_cast<std::size_t>(qkv_ne1) * sizeof(float));
+        bind_w(qnw_t, qn_w[b], static_cast<std::size_t>(head_dim) * sizeof(float));
+        bind_w(knw_t, kn_w[b], static_cast<std::size_t>(head_dim) * sizeof(float));
+        bind_w(outw_t, out_w[b], out_bytes);
+        bind_w(outb_t, out_b[b], static_cast<std::size_t>(out_ne1) * sizeof(float));
+        bind_w(ln2w_t, ln2_w[b], static_cast<std::size_t>(ln_dim) * sizeof(float));
+        bind_w(gatew_t, gate_w[b], ffn_up_bytes);
+        bind_w(gateb_t, gate_b[b], static_cast<std::size_t>(ffn_ne1) * sizeof(float));
+        bind_w(upw_t, up_w[b], ffn_up_bytes);
+        bind_w(upb_t, up_b[b], static_cast<std::size_t>(ffn_ne1) * sizeof(float));
+        bind_w(downw_t, down_w[b], ffn_down_bytes);
+        bind_w(downb_t, down_b[b], static_cast<std::size_t>(ffn_ne0) * sizeof(float));
+
+        cur = build_glm_vision_attn_subgraph(ctx, cur, ln1w_t, eps,
+            qkvw_t, qkvb_t, qnw_t, knw_t, outw_t, outb_t, cos_t, sin_t,
+            rows, hidden, num_heads, head_dim, half_dim, attn_scale);
+        cur = build_glm_vision_mlp_subgraph(ctx, cur, ln2w_t, eps,
+            gatew_t, gateb_t, upw_t, upb_t, downw_t, downb_t, rows, hidden, dff, swiglu_limit);
+    }
+
+    ggml_tensor* output = ggml_cpy(ctx, cur, hidden_binding.tensor);
+    if (!output) { set_last_error("glm_vision_encoder: output cpy failed."); return 0; }
+    ggml_set_output(output);
+
+    ggml_cgraph* graph = ggml_new_graph_custom(ctx, static_cast<std::size_t>(block_count) * 220 + 512, false);
+    if (!graph) { set_last_error("glm_vision_encoder: graph creation failed."); return 0; }
+    ggml_build_forward_expand(graph, output);
+
+    ggml_gallocr_t galloc = ggml_gallocr_new(ggml_backend_get_default_buffer_type(g_backend));
+    if (galloc == nullptr || !ggml_gallocr_alloc_graph(galloc, graph))
+    {
+        if (galloc != nullptr) ggml_gallocr_free(galloc);
+        set_last_error("glm_vision_encoder: gallocr allocation failed.");
+        return 0;
+    }
+
+    if (!use_zero_copy)
+        upload_binding(hidden_binding, hidden_desc.data, hidden_binding.raw_bytes);
+    for (auto& u : upload_list)
+        ggml_backend_tensor_set(u.tensor, resolve_upload_source(u.data), 0, u.bytes);
+    ggml_backend_tensor_set(cos_t, cos_table, 0, cos_sin_elems * sizeof(float));
+    ggml_backend_tensor_set(sin_t, sin_table, 0, cos_sin_elems * sizeof(float));
+
+    ggml_status status = ggml_backend_graph_compute(g_backend, graph);
+    if (status != GGML_STATUS_SUCCESS) { ggml_gallocr_free(galloc); set_last_error("glm_vision_encoder: graph compute failed."); return 0; }
+    finalize_compute(use_zero_copy, hidden_binding.storage, hidden_desc.data, hidden_binding.raw_bytes);
+
+    ggml_gallocr_free(galloc);
+    clear_last_error();
+    return 1;
+}
+
+TSG_EXPORT int TSGgml_GlmVisionEncoderF32(
+    TensorView2DDesc hidden,
+    int block_count, float eps, float attn_scale, float swiglu_limit,
+    int num_patches, int num_heads, int head_dim, int half_dim,
+    const float* cos_table, const float* sin_table,
+    const float* const* ln1_w,
+    const float* const* qkv_w, const float* const* qkv_b,
+    const float* const* qn_w, const float* const* kn_w,
+    const float* const* out_w, const float* const* out_b,
+    const float* const* ln2_w,
+    const float* const* gate_w, const float* const* gate_b,
+    const float* const* up_w, const float* const* up_b,
+    const float* const* down_w, const float* const* down_b,
+    int ln_dim,
+    int qkv_ne0, int qkv_ne1, int64_t qkv_bytes,
+    int out_ne0, int out_ne1, int64_t out_bytes,
+    int ffn_ne0, int ffn_ne1, int64_t ffn_up_bytes, int64_t ffn_down_bytes)
+{
+    try
+    {
+        return fused_glm_vision_encoder_f32_impl(hidden, block_count, eps, attn_scale, swiglu_limit,
+            num_patches, num_heads, head_dim, half_dim, cos_table, sin_table,
+            ln1_w, qkv_w, qkv_b, qn_w, kn_w, out_w, out_b, ln2_w,
+            gate_w, gate_b, up_w, up_b, down_w, down_b,
+            ln_dim,
+            qkv_ne0, qkv_ne1, static_cast<std::size_t>(qkv_bytes),
+            out_ne0, out_ne1, static_cast<std::size_t>(out_bytes),
+            ffn_ne0, ffn_ne1, static_cast<std::size_t>(ffn_up_bytes), static_cast<std::size_t>(ffn_down_bytes));
+    }
+    catch (const std::exception& e) { set_last_error(e.what()); return 0; }
+    catch (...) { set_last_error("glm_vision_encoder: unknown error"); return 0; }
+}
+
 TSG_EXPORT int TSGgml_Qwen35VisionEncoderF32(
     TensorView2DDesc hidden,
     int block_count, float eps, float attn_scale,
diff --git a/TensorSharp.GGML.Native/ggml_ops_gated_delta_net.cpp b/TensorSharp.GGML.Native/ggml_ops_gated_delta_net.cpp
index 32c8968c..3ab211f7 100644
--- a/TensorSharp.GGML.Native/ggml_ops_gated_delta_net.cpp
+++ b/TensorSharp.GGML.Native/ggml_ops_gated_delta_net.cpp
@@ -172,7 +172,9 @@ namespace
 //      with cross-chunk recurrent state propagation. qGExp = q * gExp is
 //      pre-computed once over the full chunked layout so the per-chunk loop
 //      body contains four mul_mat / two broadcast mul / two add ops only.
-//   6. Runs RMSNorm and gates by silu(z).
+//   6. Runs RMSNorm and gates by silu(z), or by sigmoid(z) when gate_mode is 1.
+//      Qwen 3.5 / Qwen3-Next use silu; qwen4exp (Qwen3.8-Flash-Next) is identical
+//      apart from this one activation, so it rides the same kernel.
 //   7. Writes the output back to gated_out and the updated recurrent state
 //      back to the state tensor.
 TSG_EXPORT int TSGgml_GatedDeltaNetChunkedF32(
@@ -188,7 +190,8 @@ TSG_EXPORT int TSGgml_GatedDeltaNetChunkedF32(
     void* a_log_data,
     void* ssm_norm_w_data,
     int chunk_size,
-    float eps)
+    float eps,
+    int gate_mode)
 {
     try
     {
@@ -514,8 +517,8 @@ TSG_EXPORT int TSGgml_GatedDeltaNetChunkedF32(
 
             // z permute (D, H, T, 1) -> (D, T, H, 1), silu, multiply.
             ggml_tensor* z_p     = ggml_cont(ctx, ggml_permute(ctx, z, 0, 2, 1, 3));
-            ggml_tensor* z_silu  = ggml_silu(ctx, z_p);
-            ggml_tensor* gated   = ggml_mul(ctx, attn_rms, z_silu);                          // (D, T, H, 1)
+            ggml_tensor* z_act   = gate_mode == 1 ? ggml_sigmoid(ctx, z_p) : ggml_silu(ctx, z_p);
+            ggml_tensor* gated   = ggml_mul(ctx, attn_rms, z_act);                           // (D, T, H, 1)
 
             // Permute back to (D, H, T, 1) and copy into output binding.
             ggml_tensor* gated_out = ggml_cont(ctx, ggml_permute(ctx, gated, 0, 2, 1, 3));
diff --git a/TensorSharp.GGML.Native/ggml_ops_glm_dsa.cpp b/TensorSharp.GGML.Native/ggml_ops_glm_dsa.cpp
index 2dba0e68..dcaf336d 100644
--- a/TensorSharp.GGML.Native/ggml_ops_glm_dsa.cpp
+++ b/TensorSharp.GGML.Native/ggml_ops_glm_dsa.cpp
@@ -132,6 +132,23 @@ struct glm_hparams
     int32_t indexer_head_size = 0;
     int32_t indexer_top_k = 0;
     std::vector<uint8_t> indexer_full;   // per trunk layer
+
+    // --- GLM-5.3-Flash (glm5next) ------------------------------------------
+    // The hybrid successor: 34 of 45 trunk layers are KDA linear attention, the
+    // MLA+DSA layers are NoPE (n_rot 0), the indexer scores 4-cell pools, and
+    // every residual crossing goes through Sinkhorn-constrained hyper-connections.
+    bool    g5n = false;
+    int32_t kda_head_dim = 0;            // 128
+    int32_t kda_n_head = 0;              // n_head on KDA layers (64)
+    float   kda_gate_lb = -5.0f;         // multiplicative gate lower bound
+    int32_t d_conv = 0;                  // short conv kernel (4)
+    int32_t indexer_kpool = 0;           // cells per pool (4); 0 = unpooled (5.2)
+    int32_t hc_mult = 0;                 // hyper-connection stream count (4)
+    int32_t hc_sinkhorn = 20;
+    float   hc_eps = 1e-6f;
+    float   swiglu_clamp = 0.0f;         // 0 = no clamp
+    float   norm_eps = 0.0f;             // indexer k_norm LayerNorm eps (1e-6; 0 for glm-dsa)
+    std::vector<uint8_t> is_recr;        // per trunk layer: KDA (1) vs MLA+DSA (0)
     float   rms_eps = 1e-5f;
     float   rope_freq_base = 10000.0f;
     int32_t n_ctx_train = 0;
@@ -175,6 +192,35 @@ struct glm_layer_weights
     ggml_tensor * idx_k_norm_b = nullptr;
     ggml_tensor * idx_proj = nullptr;
 
+    // --- glm5next: KDA linear attention ----------------------------------
+    ggml_tensor * kda_wq = nullptr;      // [n_embd, d_inner]
+    ggml_tensor * kda_wk = nullptr;
+    ggml_tensor * kda_wv = nullptr;
+    ggml_tensor * kda_wo = nullptr;      // [d_inner, n_embd]
+    ggml_tensor * kda_conv_q = nullptr;  // [d_conv, 1, d_inner]
+    ggml_tensor * kda_conv_k = nullptr;
+    ggml_tensor * kda_conv_v = nullptr;
+    ggml_tensor * kda_f_a = nullptr;     // [n_embd, 128]
+    ggml_tensor * kda_f_b = nullptr;     // [128, d_inner]
+    ggml_tensor * kda_dt_b = nullptr;    // [d_inner]
+    ggml_tensor * kda_a = nullptr;       // [n_head] (-exp(A_log))
+    ggml_tensor * kda_beta = nullptr;    // [n_embd, n_head]
+    ggml_tensor * kda_g_a = nullptr;     // [n_embd, 128]
+    ggml_tensor * kda_g_b = nullptr;     // [128, d_inner]
+    ggml_tensor * kda_o_norm = nullptr;  // [head_dim]
+
+    // --- glm5next: Sinkhorn hyper-connections ----------------------------
+    ggml_tensor * hc_attn_fn = nullptr;    // [hc*n_embd, (2+hc)*hc]
+    ggml_tensor * hc_attn_scale = nullptr; // [3]
+    ggml_tensor * hc_attn_base = nullptr;  // [(2+hc)*hc]
+    ggml_tensor * hc_ffn_fn = nullptr;
+    ggml_tensor * hc_ffn_scale = nullptr;
+    ggml_tensor * hc_ffn_base = nullptr;
+
+    // --- glm5next: pooled indexer compressor -----------------------------
+    ggml_tensor * idx_comp_gate = nullptr; // [n_embd, d_idx]
+    ggml_tensor * idx_comp_ape = nullptr;  // [d_idx, kpool]
+
     // NextN/MTP wiring. Only the trailing draft block carries these; the eh_proj
     // is replicated on every rank (it runs before the head split) and the two
     // optional tensors are absent from GLM-5.2, which shares the trunk's
@@ -195,6 +241,7 @@ struct glm_layer
     bool cpu_moe = false;     // routed experts live in host RAM
     bool indexer_full = false;
     bool is_moe = false;
+    bool recurrent = false;   // glm5next: KDA linear attention instead of MLA
 
     // Shard boundaries under tensor parallelism (rank r owns [first[r], first[r+1])).
     int head_first[MAX_GPUS + 1] = {};
@@ -244,6 +291,12 @@ struct glm_slot
     // across the split and no collective is needed to share them.
     std::vector<ggml_tensor *> kv_k[MAX_GPUS];    // [n_kv_row, n_ctx] F16
     std::vector<ggml_tensor *> idx_k[MAX_GPUS];   // [indexer_head_size, n_ctx] F16 (or null)
+                                                  // glm5next: [d_idx, 2, n_ctx] key|gate pairs
+
+    // glm5next KDA recurrent state, F32, updated in place by the graph:
+    //   conv: [d_conv-1, 3*d_inner]   ssm: [head_dim, head_dim, n_head]
+    std::vector<ggml_tensor *> kda_conv[MAX_GPUS];
+    std::vector<ggml_tensor *> kda_ssm[MAX_GPUS];
 };
 
 struct tensor_source
@@ -267,6 +320,18 @@ struct graph_inputs
     ggml_tensor * kq_mask[MAX_GPUS + 1] = {};  // F16/F32 [n_kv, nt]
     ggml_tensor * lid_mask[MAX_GPUS + 1] = {}; // F16 [n_kv, nt] (sparse layers only)
     ggml_tensor * kv_idxs[MAX_GPUS + 1] = {};  // I64 [nt] destination cache rows
+    // glm5next pooled indexer (sparse graphs only): the pool->cell map, the
+    // per-query pool visibility bias, and the always-attended trailing cells of
+    // each query's own (incomplete) pool with their write values (0 for a real
+    // tail cell, -inf for an unused lane - a no-op write on the -inf canvas).
+    ggml_tensor * pool_cells[MAX_GPUS + 1] = {};   // I32 [kpool * n_pools]
+    ggml_tensor * pool_bias[MAX_GPUS + 1] = {};    // F16 (fused) / F32 [n_pools, nt]
+    ggml_tensor * trail_cells[MAX_GPUS + 1] = {};  // I32 [kpool, nt]
+    ggml_tensor * trail_vals[MAX_GPUS + 1] = {};   // F32 [1, kpool, nt]
+    // glm5next vision: embedding rows that replace the token embeddings of
+    // image-placeholder positions (embedding device only).
+    ggml_tensor * embd_rows = nullptr;             // F32 [n_embd, n_ovr]
+    ggml_tensor * embd_idx = nullptr;              // I64 [n_ovr]
     ggml_tensor * out_ids = nullptr;           // I32 [n_out]
     /// NextN/MTP only: the trunk hidden state of the token preceding each row,
     /// F32 [n_embd, nt]. Lives on the embedding device like inp.tokens.
@@ -297,10 +362,18 @@ struct bd_token
     ggml_tensor * kv_idx[MAX_GPUS + 1] = {};    // I64 [1]  destination cache row
     ggml_tensor * kq_mask[MAX_GPUS + 1] = {};   // F16 [n_kv, 1]
     ggml_tensor * lid_mask[MAX_GPUS + 1] = {};  // F16 [n_kv, 1]
+    // glm5next pooled indexer (sparse tokens only): this token's pool map,
+    // pool-visibility bias and always-attended trailing-pool lanes.
+    ggml_tensor * pool_cells[MAX_GPUS + 1] = {};  // I32 [n_kv]
+    ggml_tensor * pool_bias[MAX_GPUS + 1] = {};   // F16 (fused) / F32 [n_pools, 1]
+    ggml_tensor * trail_cells[MAX_GPUS + 1] = {}; // I32 [kpool, 1]
+    ggml_tensor * trail_vals[MAX_GPUS + 1] = {};  // F32 [1, kpool, 1]
 };
 
 struct graph_build_result
 {
+    int64_t n_ovr = 0;   // glm5next vision-override rows this graph expects
+
     ggml_context * ctx = nullptr;
     ggml_cgraph * gf = nullptr;
     ggml_backend_sched_t sched = nullptr;
@@ -407,6 +480,11 @@ struct glm_model
     bool flash_attn = false;
     bool fused_lid = false;      // ggml_lightning_indexer has a kernel here
 
+    // ggml_dsv4_hc_pre/post have a kernel on this backend (glm5next only).
+    // Probed at load; when false the graph builds the equivalent batched
+    // mul_mat instead of bouncing the residual stream through the CPU backend.
+    bool hc_native = true;
+
     // ggml_backend_sched's "a higher-priority backend wants this op" heuristic.
     // It is a win when a host-resident tensor is small enough to stream, and a
     // hard failure when it is not: with the routed experts offloaded, a
@@ -431,6 +509,14 @@ struct glm_model
 
     std::vector<float> logits;
 
+    // glm5next vision: embedding rows queued by the managed side to OVERRIDE the
+    // token embeddings of image-placeholder positions in the NEXT prompt
+    // forward. `index` is the row's position within that forward call's token
+    // array; the chunking wrapper slices per ubatch. Consumed (cleared) by the
+    // wrapper after the prompt completes.
+    struct embd_override { int64_t index; std::vector<float> rows; };
+    std::vector<embd_override> embd_ovr;
+
     // host scratch, refilled per ubatch
     std::vector<int32_t> h_tokens;
     std::vector<int32_t> h_pos;
@@ -505,6 +591,41 @@ static bool kv_bool(gguf_context * g, const char * key, bool * out)
     return true;
 }
 
+static bool kv_arr_u32_as_u8(gguf_context * g, const char * key, std::vector<uint8_t> & out)
+{
+    const int64_t i = gguf_find_key(g, key);
+    if (i < 0 || gguf_get_kv_type(g, i) != GGUF_TYPE_ARRAY) return false;
+    const gguf_type et = gguf_get_arr_type(g, i);
+    const size_t n = gguf_get_arr_n(g, i);
+    out.resize(n);
+    const void * data = gguf_get_arr_data(g, i);
+    for (size_t j = 0; j < n; j++)
+    {
+        int64_t v = 0;
+        switch (et)
+        {
+            case GGUF_TYPE_UINT8:  v = ((const uint8_t  *) data)[j]; break;
+            case GGUF_TYPE_INT8:   v = ((const int8_t   *) data)[j]; break;
+            case GGUF_TYPE_UINT16: v = ((const uint16_t *) data)[j]; break;
+            case GGUF_TYPE_INT16:  v = ((const int16_t  *) data)[j]; break;
+            case GGUF_TYPE_UINT32: v = ((const uint32_t *) data)[j]; break;
+            case GGUF_TYPE_INT32:  v = ((const int32_t  *) data)[j]; break;
+            default: return false;
+        }
+        out[j] = v != 0 ? 1 : 0;
+    }
+    return true;
+}
+
+static bool kv_arr_f32_first(gguf_context * g, const char * key, float * out)
+{
+    const int64_t i = gguf_find_key(g, key);
+    if (i < 0 || gguf_get_kv_type(g, i) != GGUF_TYPE_ARRAY) return false;
+    if (gguf_get_arr_type(g, i) != GGUF_TYPE_FLOAT32 || gguf_get_arr_n(g, i) == 0) return false;
+    *out = ((const float *) gguf_get_arr_data(g, i))[0];
+    return true;
+}
+
 static bool kv_arr_u8(gguf_context * g, const char * key, std::vector<uint8_t> & out)
 {
     int64_t id = gguf_find_key(g, key);
@@ -950,6 +1071,8 @@ static glm_slot * slot_alloc(glm_model & m)
     {
         slot->kv_k[r].assign((size_t) n_cache_layers, nullptr);
         slot->idx_k[r].assign((size_t) n_cache_layers, nullptr);
+        slot->kda_conv[r].assign((size_t) n_cache_layers, nullptr);
+        slot->kda_ssm[r].assign((size_t) n_cache_layers, nullptr);
     }
 
     // Cache tensors live on the device that reads them, so attention never
@@ -957,20 +1080,40 @@ static glm_slot * slot_alloc(glm_model & m)
     std::vector<ggml_context *> ctxs((size_t) m.n_gpu + 1, nullptr);
     for (int d = 0; d <= m.n_gpu; d++)
     {
-        ggml_init_params p = { (size_t) (4 * m.tp * n_cache_layers + 16) * ggml_tensor_overhead(), nullptr, true };
+        ggml_init_params p = { (size_t) (8 * m.tp * n_cache_layers + 16) * ggml_tensor_overhead(), nullptr, true };
         ctxs[d] = ggml_init(p);
     }
 
     for (int il = 0; il < n_cache_layers; il++)
     {
+        const bool recr = il < m.hp.n_layer && m.layers[il].recurrent;
         for (int r = 0; r < m.tp; r++)
         {
             const int d = m.tp > 1 ? m.rank_device(r) : m.layers[il].device;
+            if (recr)
+            {
+                // KDA linear attention keeps a fixed-size recurrent state per
+                // sequence instead of KV rows: the short-conv tail and the
+                // per-head delta-net state, both updated in place by the graph.
+                const int64_t d_inner = (int64_t) m.hp.kda_head_dim * m.hp.kda_n_head;
+                slot->kda_conv[r][il] = ggml_new_tensor_2d(ctxs[d], GGML_TYPE_F32,
+                        m.hp.d_conv - 1, 3 * d_inner);
+                slot->kda_ssm[r][il] = ggml_new_tensor_3d(ctxs[d], GGML_TYPE_F32,
+                        m.hp.kda_head_dim, m.hp.kda_head_dim, m.hp.kda_n_head);
+                ggml_format_name(slot->kda_conv[r][il], "slot%d_kconv.%d.%d", slot->id, r, il);
+                ggml_format_name(slot->kda_ssm[r][il], "slot%d_kssm.%d.%d", slot->id, r, il);
+                continue;
+            }
             slot->kv_k[r][il] = ggml_new_tensor_2d(ctxs[d], GGML_TYPE_F16, m.hp.n_kv_row, m.n_ctx);
             ggml_format_name(slot->kv_k[r][il], "slot%d_kv.%d.%d", slot->id, r, il);
             if (il < m.hp.n_layer && m.layers[il].indexer_full)
             {
-                slot->idx_k[r][il] = ggml_new_tensor_2d(ctxs[d], GGML_TYPE_F16, m.hp.indexer_head_size, m.n_ctx);
+                // glm5next caches an indexer key AND a compressor gate per cell,
+                // packed [key | gate] into one row so a pool's members are
+                // gathered once.
+                const int64_t idx_row = m.hp.indexer_kpool > 0 ? 2 * m.hp.indexer_head_size
+                                                               : m.hp.indexer_head_size;
+                slot->idx_k[r][il] = ggml_new_tensor_2d(ctxs[d], GGML_TYPE_F16, idx_row, m.n_ctx);
                 ggml_format_name(slot->idx_k[r][il], "slot%d_idx.%d.%d", slot->id, r, il);
             }
         }
@@ -1008,6 +1151,25 @@ static glm_slot * slot_alloc(glm_model & m)
     return raw;
 }
 
+/// Zero a slot's KDA recurrent state (glm5next): a new conversation must not
+/// inherit the previous one's conv tail or delta-net state. The MLA rows need
+/// no such wipe - they are rewritten before anything reads them.
+static void slot_clear_recurrent(glm_model & m, glm_slot & slot)
+{
+    std::vector<uint8_t> zeros;
+    auto wipe = [&](ggml_tensor * t) {
+        if (!t) return;
+        const size_t nb = ggml_nbytes(t);
+        if (zeros.size() < nb) zeros.assign(nb, 0);
+        ggml_backend_tensor_set(t, zeros.data(), 0, nb);
+    };
+    for (int r = 0; r < m.tp; r++)
+    {
+        for (size_t il = 0; il < slot.kda_conv[r].size(); il++) wipe(slot.kda_conv[r][il]);
+        for (size_t il = 0; il < slot.kda_ssm[r].size(); il++) wipe(slot.kda_ssm[r][il]);
+    }
+}
+
 static void slot_free(glm_model & m, int slot_id)
 {
     // Graphs bake this slot's cache addresses into their nodes; drop them first.
@@ -1041,6 +1203,14 @@ static void slot_free(glm_model & m, int slot_id)
 /// second is a ceiling the loader may cap to whatever the VRAM actually holds,
 /// because a 1M-token advertisement is not a promise that 1M tokens of KV fit
 /// beside the weights.
+static std::string akey(const char * arch, const char * suffix)
+{
+    std::string k(arch);
+    k += ".";
+    k += suffix;
+    return k;
+}
+
 static glm_model * glm_load(const char * gguf_path, int n_gpu_req, int n_ctx, int n_ubatch,
                             int n_threads, int n_cpu_moe_req, const char * backend_name, int tp_req,
                             bool ctx_is_hard_limit, bool load_mtp)
@@ -1141,48 +1311,88 @@ static glm_model * glm_load(const char * gguf_path, int n_gpu_req, int n_ctx, in
     shard_files shards = resolve_shards(gguf_path, split_count);
 
     glm_hparams & hp = m->hp;
+    // GLM-5.3-Flash ("glm5next") loads through this same executor: it keeps the
+    // MLA+DSA+MoE core and adds KDA layers, pooled indexing and Sinkhorn
+    // hyper-connections on top.
+    {
+        const int64_t ai = gguf_find_key(g0, "general.architecture");
+        if (ai >= 0 && gguf_get_kv_type(g0, ai) == GGUF_TYPE_STRING)
+            hp.g5n = strcmp(gguf_get_val_str(g0, ai), "glm5next") == 0;
+    }
+    const char * AP = hp.g5n ? "glm5next" : "glm-dsa";
     bool ok = true;
-    ok &= kv_u32(g0, "glm-dsa.block_count", &hp.n_layer_all);
-    ok &= kv_u32(g0, "glm-dsa.embedding_length", &hp.n_embd);
-    ok &= kv_u32(g0, "glm-dsa.attention.head_count", &hp.n_head);
-    ok &= kv_u32(g0, "glm-dsa.rope.dimension_count", &hp.n_rot);
-    ok &= kv_u32(g0, "glm-dsa.attention.q_lora_rank", &hp.q_lora_rank);
-    ok &= kv_u32(g0, "glm-dsa.attention.kv_lora_rank", &hp.kv_lora_rank);
-    ok &= kv_f32(g0, "glm-dsa.attention.layer_norm_rms_epsilon", &hp.rms_eps);
-    ok &= kv_u32(g0, "glm-dsa.expert_count", &hp.n_expert);
-    ok &= kv_u32(g0, "glm-dsa.expert_used_count", &hp.n_expert_used);
-    ok &= kv_u32(g0, "glm-dsa.expert_feed_forward_length", &hp.n_ff_exp);
-    ok &= kv_u32(g0, "glm-dsa.attention.indexer.head_count", &hp.indexer_n_head);
-    ok &= kv_u32(g0, "glm-dsa.attention.indexer.key_length", &hp.indexer_head_size);
-    ok &= kv_u32(g0, "glm-dsa.attention.indexer.top_k", &hp.indexer_top_k);
-
-    kv_u32(g0, "glm-dsa.feed_forward_length", &hp.n_ff);
-    kv_u32(g0, "glm-dsa.leading_dense_block_count", &hp.n_dense_lead);
-    kv_u32(g0, "glm-dsa.nextn_predict_layers", &hp.n_layer_nextn);
-    kv_u32(g0, "glm-dsa.expert_shared_count", &hp.n_expert_shared);
-    kv_f32(g0, "glm-dsa.expert_weights_scale", &hp.expert_weights_scale);
-    kv_bool(g0, "glm-dsa.expert_weights_norm", &hp.expert_weights_norm);
-    kv_u32(g0, "glm-dsa.expert_gating_func", &hp.expert_gating_func);
-    kv_f32(g0, "glm-dsa.rope.freq_base", &hp.rope_freq_base);
-    kv_u32(g0, "glm-dsa.context_length", &hp.n_ctx_train);
+    ok &= kv_u32(g0, akey(AP, "block_count").c_str(), &hp.n_layer_all);
+    ok &= kv_u32(g0, akey(AP, "embedding_length").c_str(), &hp.n_embd);
+    ok &= kv_u32(g0, akey(AP, "attention.head_count").c_str(), &hp.n_head);
+    ok &= kv_u32(g0, akey(AP, "rope.dimension_count").c_str(), &hp.n_rot);
+    ok &= kv_u32(g0, akey(AP, "attention.q_lora_rank").c_str(), &hp.q_lora_rank);
+    ok &= kv_u32(g0, akey(AP, "attention.kv_lora_rank").c_str(), &hp.kv_lora_rank);
+    ok &= kv_f32(g0, akey(AP, "attention.layer_norm_rms_epsilon").c_str(), &hp.rms_eps);
+    ok &= kv_u32(g0, akey(AP, "expert_count").c_str(), &hp.n_expert);
+    ok &= kv_u32(g0, akey(AP, "expert_used_count").c_str(), &hp.n_expert_used);
+    ok &= kv_u32(g0, akey(AP, "expert_feed_forward_length").c_str(), &hp.n_ff_exp);
+    ok &= kv_u32(g0, akey(AP, "attention.indexer.head_count").c_str(), &hp.indexer_n_head);
+    ok &= kv_u32(g0, akey(AP, "attention.indexer.key_length").c_str(), &hp.indexer_head_size);
+    ok &= kv_u32(g0, akey(AP, "attention.indexer.top_k").c_str(), &hp.indexer_top_k);
+
+    kv_u32(g0, akey(AP, "feed_forward_length").c_str(), &hp.n_ff);
+    kv_u32(g0, akey(AP, "leading_dense_block_count").c_str(), &hp.n_dense_lead);
+    kv_u32(g0, akey(AP, "nextn_predict_layers").c_str(), &hp.n_layer_nextn);
+    kv_u32(g0, akey(AP, "expert_shared_count").c_str(), &hp.n_expert_shared);
+    kv_f32(g0, akey(AP, "expert_weights_scale").c_str(), &hp.expert_weights_scale);
+    kv_bool(g0, akey(AP, "expert_weights_norm").c_str(), &hp.expert_weights_norm);
+    kv_u32(g0, akey(AP, "expert_gating_func").c_str(), &hp.expert_gating_func);
+    kv_f32(g0, akey(AP, "rope.freq_base").c_str(), &hp.rope_freq_base);
+    kv_u32(g0, akey(AP, "context_length").c_str(), &hp.n_ctx_train);
 
     int32_t key_len = 0, val_len = 0;
-    kv_u32(g0, "glm-dsa.attention.key_length", &key_len);
-    kv_u32(g0, "glm-dsa.attention.value_length", &val_len);
+    kv_u32(g0, akey(AP, "attention.key_length").c_str(), &key_len);
+    kv_u32(g0, akey(AP, "attention.value_length").c_str(), &val_len);
     hp.n_embd_head_k = key_len;
     hp.n_embd_head_v = val_len;
-    kv_u32(g0, "glm-dsa.attention.key_length_mla", &hp.n_embd_head_k);
-    kv_u32(g0, "glm-dsa.attention.value_length_mla", &hp.n_embd_head_v);
+    kv_u32(g0, akey(AP, "attention.key_length_mla").c_str(), &hp.n_embd_head_k);
+    kv_u32(g0, akey(AP, "attention.value_length_mla").c_str(), &hp.n_embd_head_v);
 
     int32_t expert_groups = 1, expert_groups_used = 1;
-    kv_u32(g0, "glm-dsa.expert_group_count", &expert_groups);
-    kv_u32(g0, "glm-dsa.expert_group_used_count", &expert_groups_used);
+    kv_u32(g0, akey(AP, "expert_group_count").c_str(), &expert_groups);
+    kv_u32(g0, akey(AP, "expert_group_used_count").c_str(), &expert_groups_used);
 
     hp.n_layer = hp.n_layer_all - hp.n_layer_nextn;
     hp.n_nope = hp.n_embd_head_k - hp.n_rot;
     hp.n_kv_row = hp.kv_lora_rank + hp.n_rot;
     if (hp.expert_gating_func == 0) hp.expert_gating_func = 2;
 
+    if (hp.g5n)
+    {
+        ok &= kv_u32(g0, akey(AP, "kda.head_dim").c_str(), &hp.kda_head_dim);
+        ok &= kv_u32(g0, akey(AP, "ssm.conv_kernel").c_str(), &hp.d_conv);
+        ok &= kv_u32(g0, akey(AP, "attention.indexer.kpool").c_str(), &hp.indexer_kpool);
+        ok &= kv_u32(g0, akey(AP, "hyper_connection.count").c_str(), &hp.hc_mult);
+        kv_f32(g0, akey(AP, "kda.gate_lower_bound").c_str(), &hp.kda_gate_lb);
+        kv_u32(g0, akey(AP, "hyper_connection.sinkhorn_iterations").c_str(), &hp.hc_sinkhorn);
+        kv_f32(g0, akey(AP, "hyper_connection.epsilon").c_str(), &hp.hc_eps);
+        kv_arr_f32_first(g0, akey(AP, "swiglu_clamp_exp").c_str(), &hp.swiglu_clamp);
+        kv_f32(g0, akey(AP, "attention.layer_norm_epsilon").c_str(), &hp.norm_eps);
+
+        // per-layer type: attention.head_count_kv is 0 on KDA layers, 1 on MLA
+        std::vector<uint8_t> kvh;
+        if (kv_arr_u32_as_u8(g0, akey(AP, "attention.head_count_kv").c_str(), kvh))
+        {
+            hp.is_recr.assign((size_t) hp.n_layer, 0);
+            for (int il = 0; il < hp.n_layer && il < (int) kvh.size(); il++)
+                hp.is_recr[(size_t) il] = kvh[(size_t) il] == 0 ? 1 : 0;
+        }
+        else
+        {
+            ok = false;
+        }
+        hp.kda_n_head = hp.n_head;
+        if (hp.kda_head_dim <= 0 || hp.d_conv <= 1 || hp.indexer_kpool <= 0 || hp.hc_mult <= 0)
+            ok = false;
+        if (hp.indexer_top_k % (hp.indexer_kpool > 0 ? hp.indexer_kpool : 1) != 0)
+            ok = false;
+    }
+
     resolve_indexer_types(hp, g0);
     gguf_free(g0);
     if (meta0) { ggml_free(meta0); meta0 = nullptr; }
@@ -1197,7 +1407,19 @@ static glm_model * glm_load(const char * gguf_path, int n_gpu_req, int n_ctx, in
         fprintf(stderr, "[glm] %d expert groups (top-%d) are not supported\n", expert_groups, expert_groups_used);
         return nullptr;
     }
-    if (hp.indexer_full.empty() || hp.indexer_full[0] == 0)
+    if (hp.g5n && m->tp > 1)
+    {
+        fprintf(stderr, "[glm] glm5next tensor parallelism is not wired yet; use the layer split (omit --tp)\n");
+        return nullptr;
+    }
+    if (hp.g5n)
+    {
+        // glm5next: the indexer is full on every MLA layer and absent on KDA ones.
+        hp.indexer_full.assign((size_t) hp.n_layer, 0);
+        for (int il = 0; il < hp.n_layer; il++)
+            hp.indexer_full[(size_t) il] = hp.is_recr[(size_t) il] ? 0 : 1;
+    }
+    else if (hp.indexer_full.empty() || hp.indexer_full[0] == 0)
     {
         fprintf(stderr, "[glm] layer 0 must carry a full DSA indexer\n");
         return nullptr;
@@ -1277,7 +1499,14 @@ static glm_model * glm_load(const char * gguf_path, int n_gpu_req, int n_ctx, in
     // --mtp-spec before the model loads, and the managed side forwards that as
     // `load_mtp`. A checkpoint that declares nextn_predict_layers but ships no
     // MTP tensors (a trunk-only re-quantization) loads normally without one.
-    if (load_mtp)
+    if (load_mtp && hp.g5n)
+    {
+        // The glm5next NextN block is a full DSA decoder layer wrapped in its
+        // own hyper-connections; llama.cpp asserts its graph unimplemented and
+        // this executor does not build it yet either.
+        fprintf(stderr, "[glm] glm5next NextN/MTP speculation is not implemented; serving standard decode\n");
+    }
+    if (load_mtp && !hp.g5n)
     {
         auto has_src = [&](const char * suffix) {
             char nm[256];
@@ -1339,6 +1568,7 @@ static glm_model * glm_load(const char * gguf_path, int n_gpu_req, int n_ctx, in
     {
         m->layers[il].indexer_full = hp.indexer_full[il] != 0;
         m->layers[il].is_moe = il >= hp.n_dense_lead;
+        m->layers[il].recurrent = hp.g5n && hp.is_recr[(size_t) il] != 0;
     }
     if (m->has_mtp)
     {
@@ -1352,8 +1582,15 @@ static glm_model * glm_load(const char * gguf_path, int n_gpu_req, int n_ctx, in
     // slot allocation instead of at load.
     auto layer_cache_bytes = [&](int il) -> size_t
     {
+        // A glm5next KDA layer keeps a fixed-size recurrent state instead of
+        // per-position rows; an MLA layer's indexer cache holds [key | gate]
+        // pairs when the indexer is pooled.
+        if (hp.g5n && il < hp.n_layer && hp.is_recr[(size_t) il])
+            return (size_t) (hp.d_conv - 1) * 3 * hp.kda_head_dim * hp.kda_n_head * 4
+                 + (size_t) hp.kda_head_dim * hp.kda_head_dim * hp.kda_n_head * 4 + 4 * 256;
         size_t b = (size_t) hp.n_kv_row * m->n_ctx * 2;
-        if (hp.indexer_full[il]) b += (size_t) hp.indexer_head_size * m->n_ctx * 2;
+        if (hp.indexer_full[il])
+            b += (size_t) (hp.indexer_kpool > 0 ? 2 : 1) * hp.indexer_head_size * m->n_ctx * 2;
         return b + 4 * 256;
     };
 
@@ -1721,11 +1958,46 @@ static glm_model * glm_load(const char * gguf_path, int n_gpu_req, int n_ctx, in
             // rows and the top-k selection identical on every rank without a
             // collective.
             W.attn_norm = WL.full(d, true, "blk.%d.attn_norm.weight", il);
+            W.ffn_norm  = WL.full(d, true, "blk.%d.ffn_norm.weight", il);
+
+            if (hp.g5n)
+            {
+                // Every glm5next layer crosses the residual through Sinkhorn
+                // hyper-connections, twice.
+                W.hc_attn_fn    = WL.full(d, true, "blk.%d.hc_attn_fn.weight", il);
+                W.hc_attn_scale = WL.full(d, true, "blk.%d.hc_attn_scale.weight", il);
+                W.hc_attn_base  = WL.full(d, true, "blk.%d.hc_attn_base.weight", il);
+                W.hc_ffn_fn     = WL.full(d, true, "blk.%d.hc_ffn_fn.weight", il);
+                W.hc_ffn_scale  = WL.full(d, true, "blk.%d.hc_ffn_scale.weight", il);
+                W.hc_ffn_base   = WL.full(d, true, "blk.%d.hc_ffn_base.weight", il);
+            }
+
+            if (L.recurrent)
+            {
+                // KDA linear attention: the whole layer half is these tensors and
+                // nothing from the MLA set exists in the file.
+                W.kda_wq     = WL.full(d, true, "blk.%d.attn_q.weight", il);
+                W.kda_wk     = WL.full(d, true, "blk.%d.attn_k.weight", il);
+                W.kda_wv     = WL.full(d, true, "blk.%d.attn_v.weight", il);
+                W.kda_wo     = WL.full(d, true, "blk.%d.attn_output.weight", il);
+                W.kda_conv_q = WL.full(d, true, "blk.%d.ssm_conv1d_q.weight", il);
+                W.kda_conv_k = WL.full(d, true, "blk.%d.ssm_conv1d_k.weight", il);
+                W.kda_conv_v = WL.full(d, true, "blk.%d.ssm_conv1d_v.weight", il);
+                W.kda_f_a    = WL.full(d, true, "blk.%d.ssm_f_a.weight", il);
+                W.kda_f_b    = WL.full(d, true, "blk.%d.ssm_f_b.weight", il);
+                W.kda_dt_b   = WL.full(d, true, "blk.%d.ssm_dt.bias", il);
+                W.kda_a      = WL.full(d, true, "blk.%d.ssm_a", il);
+                W.kda_beta   = WL.full(d, true, "blk.%d.ssm_beta.weight", il);
+                W.kda_g_a    = WL.full(d, true, "blk.%d.ssm_g_a.weight", il);
+                W.kda_g_b    = WL.full(d, true, "blk.%d.ssm_g_b.weight", il);
+                W.kda_o_norm = WL.full(d, true, "blk.%d.ssm_norm.weight", il);
+            }
+            else
+            {
             W.wq_a      = WL.full(d, true, "blk.%d.attn_q_a.weight", il);
             W.q_a_norm  = WL.full(d, true, "blk.%d.attn_q_a_norm.weight", il);
             W.wkv_a_mqa = WL.full(d, true, "blk.%d.attn_kv_a_mqa.weight", il);
             W.kv_a_norm = WL.full(d, true, "blk.%d.attn_kv_a_norm.weight", il);
-            W.ffn_norm  = WL.full(d, true, "blk.%d.ffn_norm.weight", il);
 
             if (L.indexer_full)
             {
@@ -1734,6 +2006,11 @@ static glm_model * glm_load(const char * gguf_path, int n_gpu_req, int n_ctx, in
                 W.idx_k_norm_w = WL.full(d, true, "blk.%d.indexer.k_norm.weight", il);
                 W.idx_k_norm_b = WL.full(d, true, "blk.%d.indexer.k_norm.bias", il);
                 W.idx_proj     = WL.full(d, true, "blk.%d.indexer.proj.weight", il);
+                if (hp.indexer_kpool > 0)
+                {
+                    W.idx_comp_gate = WL.full(d, true, "blk.%d.indexer_compressor_gate.weight", il);
+                    W.idx_comp_ape  = WL.full(d, true, "blk.%d.indexer_compressor_ape.weight", il);
+                }
             }
 
             // --- attention, sharded by head ------------------------------------
@@ -1758,6 +2035,7 @@ static glm_model * glm_load(const char * gguf_path, int n_gpu_req, int n_ctx, in
                 W.wv_b = WL.full(d, true, "blk.%d.attn_v_b.weight", il);
                 W.wo   = WL.full(d, true, "blk.%d.attn_output.weight", il);
             }
+            }   // !L.recurrent
 
             // --- FFN ------------------------------------------------------------
             if (!L.is_moe)
@@ -1833,7 +2111,18 @@ static glm_model * glm_load(const char * gguf_path, int n_gpu_req, int n_ctx, in
                 }
             }
 
-            if (!W.attn_norm || !W.wq_a || !W.wq_b || !W.wkv_a_mqa || !W.wk_b || !W.wv_b || !W.wo || !W.ffn_norm)
+            bool complete = W.attn_norm && W.ffn_norm;
+            if (L.recurrent)
+                complete = complete && W.kda_wq && W.kda_wk && W.kda_wv && W.kda_wo
+                        && W.kda_conv_q && W.kda_conv_k && W.kda_conv_v
+                        && W.kda_f_a && W.kda_f_b && W.kda_dt_b && W.kda_a && W.kda_beta
+                        && W.kda_g_a && W.kda_g_b && W.kda_o_norm;
+            else
+                complete = complete && W.wq_a && W.wq_b && W.wkv_a_mqa && W.wk_b && W.wv_b && W.wo;
+            if (hp.g5n && il < hp.n_layer)
+                complete = complete && W.hc_attn_fn && W.hc_attn_scale && W.hc_attn_base
+                        && W.hc_ffn_fn && W.hc_ffn_scale && W.hc_ffn_base;
+            if (!complete)
             {
                 fprintf(stderr, "[glm] layer %d rank %d is incomplete\n", il, r);
                 return nullptr;
@@ -2059,7 +2348,10 @@ static glm_model * glm_load(const char * gguf_path, int n_gpu_req, int n_ctx, in
             ggml_init_params pp = { 16 * ggml_tensor_overhead() + 4096, nullptr, true };
             ggml_context * pctx = ggml_init(pp);
             ggml_tensor * q = ggml_new_tensor_4d(pctx, GGML_TYPE_F32, hp.indexer_head_size, hp.indexer_n_head, 1, 1);
-            ggml_tensor * k = ggml_new_tensor_4d(pctx, GGML_TYPE_F16, hp.indexer_head_size, 1, 256, 1);
+            // glm5next scores POOLS whose keys come out of ggml_get_rows in
+            // F32; glm-dsa scores the F16 cache directly.
+            ggml_tensor * k = ggml_new_tensor_4d(pctx, hp.g5n ? GGML_TYPE_F32 : GGML_TYPE_F16,
+                                                 hp.indexer_head_size, 1, 256, 1);
             ggml_tensor * w = ggml_new_tensor_4d(pctx, GGML_TYPE_F32, hp.indexer_n_head, 1, 1, 1);
             ggml_tensor * mask = ggml_new_tensor_4d(pctx, GGML_TYPE_F16, 256, 1, 1, 1);
             ggml_tensor * li = ggml_lightning_indexer(pctx, q, k, w, mask);
@@ -2068,6 +2360,25 @@ static glm_model * glm_load(const char * gguf_path, int n_gpu_req, int n_ctx, in
         }
     }
 
+    if (hp.g5n)
+    {
+        ggml_init_params pp = { 16 * ggml_tensor_overhead() + 4096, nullptr, true };
+        ggml_context * pctx = ggml_init(pp);
+        const int64_t hcn = hp.hc_mult;
+        ggml_tensor * x = ggml_new_tensor_3d(pctx, GGML_TYPE_F32, hp.n_embd, hcn, 1);
+        ggml_tensor * w = ggml_new_tensor_2d(pctx, GGML_TYPE_F32, hcn, 1);
+        ggml_tensor * c = ggml_new_tensor_3d(pctx, GGML_TYPE_F32, hcn, hcn, 1);
+        ggml_tensor * xf = ggml_new_tensor_2d(pctx, GGML_TYPE_F32, hp.n_embd, 1);
+        ggml_tensor * hpre = ggml_dsv4_hc_pre(pctx, x, w);
+        ggml_tensor * hpost = ggml_dsv4_hc_post(pctx, xf, x, w, c);
+        m->hc_native = ggml_backend_supports_op(m->backends[0], hpre)
+                    && ggml_backend_supports_op(m->backends[0], hpost);
+        ggml_free(pctx);
+        if (const char * e = getenv("TS_GLM_HC_NATIVE")) m->hc_native = atoi(e) != 0;
+        fprintf(stderr, "[glm] hyper-connection ops: %s\n",
+                m->hc_native ? "native" : "decomposed (backend has no fused kernel)");
+    }
+
     m->op_offload = (n_cpu_moe == 0);
     if (const char * e = getenv("TS_GLM_OP_OFFLOAD")) m->op_offload = atoi(e) != 0;
 
@@ -2316,6 +2627,22 @@ struct graph_builder
         const int64_t N = nt;
 
         ggml_tensor * q = ggml_mul_mat(ctx, LW.wq_b, qr);
+        ggml_tensor * kv_pe = ggml_mul_mat(ctx, LW.wkv_a_mqa, cur);
+
+        ggml_tensor * Qcur = nullptr;
+        ggml_tensor * Kcur = nullptr;
+        if (hp.n_rot == 0)
+        {
+            // glm5next: nope-only, the latent IS the cache row.
+            ggml_tensor * kv_cmpr = rms(kv_pe, LW.kv_a_norm);
+            ggml_tensor * q3 = ggml_reshape_3d(ctx, q, hp.n_embd_head_k, n_head, N);
+            ggml_tensor * q_nope_p = ggml_permute(ctx, q3, 0, 2, 1, 3);
+            ggml_tensor * q_abs = ggml_mul_mat(ctx, LW.wk_b, q_nope_p);          // [kv_lora, N, n_head]
+            Qcur = ggml_cont(ctx, ggml_permute(ctx, q_abs, 0, 2, 1, 3));         // [kv_lora, n_head, N]
+            Kcur = ggml_cont(ctx, ggml_reshape_3d(ctx, kv_cmpr, hp.kv_lora_rank, 1, N));
+        }
+        else
+        {
         ggml_tensor * q_nope = ggml_view_3d(ctx, q, hp.n_nope, n_head, N,
                                             ggml_row_size(q->type, hp.n_embd_head_k),
                                             ggml_row_size(q->type, hp.n_embd_head_k) * n_head, 0);
@@ -2324,7 +2651,6 @@ struct graph_builder
                                           ggml_row_size(q->type, hp.n_embd_head_k) * n_head,
                                           ggml_row_size(q->type, hp.n_nope));
 
-        ggml_tensor * kv_pe = ggml_mul_mat(ctx, LW.wkv_a_mqa, cur);
         const size_t row = ggml_row_size(kv_pe->type, hp.n_kv_row);
         ggml_tensor * kv_cmpr = ggml_view_2d(ctx, kv_pe, hp.kv_lora_rank, N, row, 0);
         ggml_tensor * k_pe = ggml_view_3d(ctx, kv_pe, hp.n_rot, 1, N, row, row,
@@ -2338,9 +2664,10 @@ struct graph_builder
         ggml_tensor * q_abs = ggml_mul_mat(ctx, LW.wk_b, q_nope_p);              // [kv_lora, N, n_head]
         q_abs = ggml_permute(ctx, q_abs, 0, 2, 1, 3);                           // [kv_lora, n_head, N]
 
-        ggml_tensor * Qcur = ggml_cont(ctx, ggml_concat(ctx, q_abs, q_pe, 0));  // [n_kv_row, n_head, N]
-        ggml_tensor * Kcur = ggml_cont(ctx, ggml_concat(ctx,
+        Qcur = ggml_cont(ctx, ggml_concat(ctx, q_abs, q_pe, 0));                // [n_kv_row, n_head, N]
+        Kcur = ggml_cont(ctx, ggml_concat(ctx,
                 ggml_reshape_3d(ctx, kv_cmpr, hp.kv_lora_rank, 1, N), k_pe, 0)); // [n_kv_row, 1, N]
+        }
 
         ggml_tensor * cat = nullptr;
         for (int64_t i = 0; i < N; i++)
@@ -2385,6 +2712,356 @@ struct graph_builder
         return ggml_mul_mat(ctx, LW.wo, flat);
     }
 
+
+    // =====================================================================
+    // glm5next batched decode: one graph, N tokens, N sequences. The shared
+    // parts (projections, hyper-connections, router, experts, LM head) run
+    // once over the batch; the per-token parts (KDA recurrence against each
+    // slot's own state, cache writes, pooled scoring, attention) are built per
+    // token because each token sees a different slot's history.
+    // =====================================================================
+
+    /// KDA for a batch of single tokens: the six projections run over the whole
+    /// batch, then each token runs its own conv/delta-net step against its
+    /// slot's persistent state.
+    ggml_tensor * build_kda_bd(int il, ggml_tensor * cur)
+    {
+        const glm_layer_weights & LW = m.layers[il].w[0];
+        const int64_t hd = hp.kda_head_dim;
+        const int64_t H = hp.kda_n_head;
+        const int64_t d_inner = hd * H;
+        const int64_t dc = hp.d_conv;
+        const int64_t N = nt;
+
+        ggml_tensor * qp = ggml_mul_mat(ctx, LW.kda_wq, cur);
+        ggml_tensor * kp = ggml_mul_mat(ctx, LW.kda_wk, cur);
+        ggml_tensor * vp = ggml_mul_mat(ctx, LW.kda_wv, cur);
+        ggml_tensor * qkv = ggml_concat(ctx, ggml_concat(ctx, qp, kp, 0), vp, 0);   // [3*d_inner, N]
+
+        ggml_tensor * conv_w = ggml_concat(ctx,
+                ggml_concat(ctx,
+                    ggml_reshape_2d(ctx, LW.kda_conv_q, dc, d_inner),
+                    ggml_reshape_2d(ctx, LW.kda_conv_k, dc, d_inner), 1),
+                ggml_reshape_2d(ctx, LW.kda_conv_v, dc, d_inner), 1);               // [dc, 3*d_inner]
+
+        ggml_tensor * g_pre = ggml_mul_mat(ctx, LW.kda_f_b, ggml_mul_mat(ctx, LW.kda_f_a, cur));
+        g_pre = ggml_add(ctx, g_pre, LW.kda_dt_b);                                  // [d_inner, N]
+        ggml_tensor * beta_all = ggml_sigmoid(ctx, ggml_mul_mat(ctx, LW.kda_beta, cur));   // [H, N]
+        ggml_tensor * gate_all = ggml_mul_mat(ctx, LW.kda_g_b, ggml_mul_mat(ctx, LW.kda_g_a, cur)); // [d_inner, N]
+
+        ggml_tensor * cat = nullptr;
+        for (int64_t i = 0; i < N; i++)
+        {
+            bd_token & B = res.bd[(size_t) i];
+            glm_slot & sl = *m.slots.at(B.slot_id);
+            ggml_tensor * conv_state = sl.kda_conv[0][(size_t) il];                  // [dc-1, 3*d_inner]
+            ggml_tensor * ssm_state = sl.kda_ssm[0][(size_t) il];                    // [hd, hd, H]
+
+            ggml_tensor * col = ggml_cont(ctx, ggml_view_2d(ctx, qkv, 3 * d_inner, 1,
+                    qkv->nb[1], (size_t) i * qkv->nb[1]));
+            ggml_tensor * col_t = ggml_reshape_3d(ctx, ggml_cont(ctx, ggml_transpose(ctx, col)),
+                    1, 3 * d_inner, 1);
+            ggml_tensor * conv_in = ggml_concat(ctx,
+                    ggml_reshape_3d(ctx, conv_state, dc - 1, 3 * d_inner, 1), col_t, 0);  // [dc, 3*d_inner, 1]
+            ggml_tensor * conv_out = ggml_silu(ctx, ggml_ssm_conv(ctx, conv_in, conv_w)); // [3*d_inner, 1]
+
+            ggml_tensor * tail = ggml_view_3d(ctx, conv_in, dc - 1, 3 * d_inner, 1,
+                    conv_in->nb[1], conv_in->nb[2], conv_in->nb[0]);
+            ggml_build_forward_expand(gf, ggml_cpy(ctx, tail, conv_state));
+
+            const size_t rs_hd = ggml_row_size(conv_out->type, hd);
+            ggml_tensor * qc = ggml_view_3d(ctx, conv_out, hd, H, 1, rs_hd, conv_out->nb[1], 0);
+            ggml_tensor * kc = ggml_view_3d(ctx, conv_out, hd, H, 1, rs_hd, conv_out->nb[1],
+                    ggml_row_size(conv_out->type, d_inner));
+            ggml_tensor * vc = ggml_view_3d(ctx, conv_out, hd, H, 1, rs_hd, conv_out->nb[1],
+                    ggml_row_size(conv_out->type, 2 * d_inner));
+
+            qc = ggml_reshape_4d(ctx, ggml_l2_norm(ctx, ggml_cont(ctx, qc), 1e-6f), hd, H, 1, 1);
+            kc = ggml_reshape_4d(ctx, ggml_l2_norm(ctx, ggml_cont(ctx, kc), 1e-6f), hd, H, 1, 1);
+            vc = ggml_reshape_4d(ctx, ggml_cont(ctx, vc), hd, H, 1, 1);
+
+            ggml_tensor * g = ggml_cont(ctx, ggml_view_2d(ctx, g_pre, d_inner, 1,
+                    g_pre->nb[1], (size_t) i * g_pre->nb[1]));
+            g = ggml_reshape_3d(ctx, g, hd, H, 1);
+            g = ggml_mul(ctx, g, ggml_reshape_3d(ctx, LW.kda_a, 1, H, 1));
+            g = ggml_sigmoid(ctx, ggml_scale(ctx, g, -1.0f));
+            g = ggml_scale(ctx, g, hp.kda_gate_lb);
+            ggml_tensor * g4 = ggml_reshape_4d(ctx, g, hd, H, 1, 1);
+
+            ggml_tensor * b4 = ggml_reshape_4d(ctx, ggml_cont(ctx, ggml_view_2d(ctx, beta_all, H, 1,
+                    beta_all->nb[1], (size_t) i * beta_all->nb[1])), 1, H, 1, 1);
+
+            ggml_tensor * s4 = ggml_reshape_4d(ctx, ssm_state, hd, hd, H, 1);
+            ggml_tensor * gdn = ggml_gated_delta_net(ctx, qc, kc, vc, g4, b4, s4, 1);
+
+            ggml_tensor * core = ggml_view_3d(ctx, gdn, hd, H, 1,
+                    ggml_row_size(gdn->type, hd), ggml_row_size(gdn->type, hd * H), 0);
+            ggml_tensor * new_state = ggml_view_3d(ctx, gdn, hd, hd, H,
+                    ggml_row_size(gdn->type, hd), ggml_row_size(gdn->type, hd * hd),
+                    ggml_row_size(gdn->type, hd * H));
+            ggml_build_forward_expand(gf, ggml_cpy(ctx, new_state, ssm_state));
+
+            ggml_tensor * gate = ggml_cont(ctx, ggml_view_2d(ctx, gate_all, d_inner, 1,
+                    gate_all->nb[1], (size_t) i * gate_all->nb[1]));
+            gate = ggml_reshape_3d(ctx, gate, hd, H, 1);
+            ggml_tensor * normed = ggml_mul(ctx, ggml_rms_norm(ctx, ggml_cont(ctx, core), hp.rms_eps),
+                                            LW.kda_o_norm);
+            ggml_tensor * gated = ggml_mul(ctx, normed, ggml_sigmoid(ctx, gate));
+            ggml_tensor * out_i = ggml_reshape_2d(ctx, ggml_cont(ctx, gated), d_inner, 1);
+
+            cat = cat ? ggml_concat(ctx, cat, out_i, 1) : out_i;                    // [d_inner, N]
+        }
+
+        return ggml_mul_mat(ctx, LW.kda_wo, cat);                                   // [n_embd, N]
+    }
+
+    /// Pooled indexer for a batch of single tokens: shared projections, then a
+    /// per-token cache write and (when past the dense limit) per-token pooled
+    /// scoring against that token's own slot.
+    void build_indexer_g5n_bd(int il, ggml_tensor * cur, ggml_tensor * qr,
+                              std::vector<ggml_tensor *> & top_k)
+    {
+        const glm_layer & L = m.layers[il];
+        const glm_layer_weights & LW = L.w[0];
+        const int dev = device_of(il, 0);
+        const int64_t D = hp.indexer_head_size;
+        const int64_t H = hp.indexer_n_head;
+        const int64_t r = hp.indexer_kpool;
+        const int64_t N = nt;
+
+        ggml_tensor * ik = ggml_mul_mat(ctx, LW.idx_attn_k, cur);                   // [D, N]
+        ik = ggml_norm(ctx, ik, hp.norm_eps);
+        ik = ggml_mul(ctx, ik, LW.idx_k_norm_w);
+        ik = ggml_add(ctx, ik, LW.idx_k_norm_b);
+        ggml_tensor * gate = ggml_mul_mat(ctx, LW.idx_comp_gate, cur);              // [D, N]
+        ggml_tensor * packed = ggml_concat(ctx,
+                ggml_reshape_3d(ctx, ik, D, 1, N),
+                ggml_reshape_3d(ctx, gate, D, 1, N), 1);                            // [D, 2, N]
+        packed = ggml_cont(ctx, packed);
+
+        ggml_tensor * iq = ggml_mul_mat(ctx, LW.idx_attn_q_b, qr);                  // [D*H, N]
+        iq = ggml_reshape_3d(ctx, iq, D, H, N);
+        ggml_tensor * w = ggml_mul_mat(ctx, LW.idx_proj, cur);                      // [H, N]
+        ggml_mul_mat_set_prec(w, GGML_PREC_F32);
+        w = ggml_scale(ctx, w, 1.0f / sqrtf((float) (D * H)));
+
+        ggml_tensor * ape = ggml_cont(ctx, ggml_transpose(ctx, LW.idx_comp_ape));   // [r, D]
+
+        for (int64_t i = 0; i < N; i++)
+        {
+            bd_token & B = res.bd[(size_t) i];
+            glm_slot & sl = *m.slots.at(B.slot_id);
+            ggml_tensor * cache = sl.idx_k[0][(size_t) il];                          // [2D, n_ctx]
+
+            ggml_tensor * pk = ggml_cont(ctx, ggml_view_2d(ctx, packed, 2 * D, 1,
+                    (size_t) 2 * D * packed->nb[0], (size_t) i * packed->nb[2]));
+            ggml_build_forward_expand(gf, ggml_set_rows(ctx, cache, pk, B.kv_idx[dev]));
+
+            if (!B.sparse) { top_k[(size_t) i] = nullptr; continue; }
+
+            const int64_t n_pools = B.n_kv / r;
+            ggml_tensor * kg = ggml_view_2d(ctx, cache, 2 * D, B.n_kv, cache->nb[1], 0);
+            ggml_tensor * members = ggml_get_rows(ctx, kg, B.pool_cells[dev]);       // [2D, n_kv] F32
+
+            const size_t nb_mem = members->nb[1];
+            ggml_tensor * mem_k = ggml_view_3d(ctx, members, D, r, n_pools, nb_mem, nb_mem * r, 0);
+            ggml_tensor * mem_g = ggml_view_3d(ctx, members, D, r, n_pools, nb_mem, nb_mem * r,
+                                               (size_t) D * members->nb[0]);
+            ggml_tensor * keys_t = ggml_cont(ctx, ggml_permute(ctx, mem_k, 1, 0, 2, 3));
+            ggml_tensor * gate_t = ggml_cont(ctx, ggml_permute(ctx, mem_g, 1, 0, 2, 3));
+            gate_t = ggml_add(ctx, gate_t, ggml_reshape_3d(ctx, ape, r, D, 1));
+            ggml_tensor * probs = ggml_soft_max(ctx, gate_t);
+            ggml_tensor * pool_k = ggml_sum_rows(ctx, ggml_mul(ctx, keys_t, probs)); // [1, D, n_pools]
+            pool_k = ggml_reshape_2d(ctx, ggml_cont(ctx, pool_k), D, n_pools);
+
+            ggml_tensor * qi = ggml_cont(ctx, ggml_view_3d(ctx, iq, D, H, 1,
+                    iq->nb[1], iq->nb[2], (size_t) i * iq->nb[2]));
+            ggml_tensor * wi = ggml_cont(ctx, ggml_view_2d(ctx, w, H, 1,
+                    w->nb[1], (size_t) i * w->nb[1]));
+
+            ggml_tensor * score = nullptr;
+            if (m.fused_lid)
+            {
+                ggml_tensor * pool_kf = ggml_reshape_3d(ctx, pool_k, D, 1, n_pools);
+                score = ggml_lightning_indexer(ctx, qi, pool_kf, wi, B.pool_bias[dev]);   // [n_pools, 1]
+            }
+            else
+            {
+                ggml_tensor * qp2 = ggml_permute(ctx, qi, 0, 2, 1, 3);              // [D, 1, H]
+                ggml_tensor * kq = ggml_mul_mat(ctx, pool_k, qp2);                  // [n_pools, 1, H]
+                kq = ggml_cont(ctx, ggml_permute(ctx, kq, 2, 1, 0, 3));             // [H, 1, n_pools]
+                ggml_tensor * sc = ggml_relu(ctx, kq);
+                sc = ggml_mul(ctx, sc, wi);
+                sc = ggml_sum_rows(ctx, sc);                                        // [1, 1, n_pools]
+                sc = ggml_cont(ctx, ggml_permute(ctx, sc, 2, 1, 0, 3));             // [n_pools, 1, 1]
+                score = ggml_add(ctx, sc, B.pool_bias[dev]);
+            }
+
+            const int64_t select_k = std::min<int64_t>(n_pools, hp.indexer_top_k / r);
+            ggml_tensor * sel = ggml_cont(ctx, ggml_top_k(ctx, score, (int) select_k));
+            ggml_tensor * pc2 = ggml_reshape_2d(ctx, B.pool_cells[dev], r, n_pools);
+            ggml_tensor * sel_flat = ggml_reshape_1d(ctx, sel, select_k);
+            ggml_tensor * cells = ggml_get_rows(ctx, pc2, sel_flat);                // I32 [r, select_k]
+            top_k[(size_t) i] = ggml_reshape_2d(ctx, cells, r * select_k, 1);
+        }
+    }
+
+    /// Single-token variant of build_topk_mask_g5n.
+    ggml_tensor * build_topk_mask_g5n_1(int dev, ggml_tensor * kq_mask, ggml_tensor * top_k,
+                                        ggml_tensor * trail_cells, ggml_tensor * trail_vals, int64_t nkv)
+    {
+        const int64_t n_top_k = top_k->ne[0];
+        const int64_t r = trail_cells->ne[0];
+
+        ggml_tensor * base = ggml_fill(ctx, kq_mask, -INFINITY);                     // [nkv, 1]
+        ggml_set_output(base);
+        ggml_tensor * all = ggml_view_3d(ctx, base, 1, nkv, 1, base->nb[0], base->nb[1], 0);
+
+        ggml_tensor * t_idx = ggml_view_3d(ctx, trail_cells, r, 1, 1,
+                                           trail_cells->nb[1], trail_cells->nb[1], 0);
+        ggml_tensor * with_tail = ggml_set_rows(ctx, all, trail_vals, t_idx);
+
+        ggml_tensor * idx = ggml_view_3d(ctx, top_k, n_top_k, 1, 1, top_k->nb[1], top_k->nb[1], 0);
+        ggml_tensor * zeros = ggml_fill(ctx, ggml_new_tensor_3d(ctx, GGML_TYPE_F32, 1, n_top_k, 1), 0.0f);
+        ggml_tensor * unmasked = ggml_set_rows(ctx, with_tail, zeros, idx);
+
+        ggml_tensor * masked = ggml_view_2d(ctx, unmasked, nkv, 1, unmasked->nb[2], 0);
+        ggml_tensor * out = ggml_add(ctx, masked, kq_mask);
+        pin(base, dev);
+        pin(zeros, dev);
+        pin(with_tail, dev);
+        pin(unmasked, dev);
+        pin(out, dev);
+        return out;
+    }
+
+    /// Whole graph for one glm5next batched decode step.
+    void build_batched_g5n()
+    {
+        graph_inputs & inp = res.inp;
+        const int64_t N = nt;
+        const int64_t hcm = hp.hc_mult;
+        const int64_t r = hp.indexer_kpool;
+
+        bool dev_used[MAX_GPUS + 1] = {};
+        bool dev_mla[MAX_GPUS + 1] = {};
+        for (int il = 0; il < hp.n_layer; il++)
+        {
+            dev_used[m.layers[il].device] = true;
+            if (!m.layers[il].recurrent) dev_mla[m.layers[il].device] = true;
+        }
+
+        for (int d = 0; d <= m.n_gpu; d++)
+        {
+            if (!dev_used[d]) continue;
+            char nb[64];
+            snprintf(nb, sizeof(nb), "inp_tokens.%d", d);
+            inp.tokens[d] = new_input(GGML_TYPE_I32, N, 0, nb, d);
+            if (!dev_mla[d]) continue;
+            for (int64_t i = 0; i < N; i++)
+            {
+                bd_token & B = res.bd[(size_t) i];
+                snprintf(nb, sizeof(nb), "bd%lld_kv_idx.%d", (long long) i, d);
+                B.kv_idx[d] = new_input(GGML_TYPE_I64, 1, 0, nb, d);
+                snprintf(nb, sizeof(nb), "bd%lld_kq_mask.%d", (long long) i, d);
+                B.kq_mask[d] = new_input(m.flash_attn ? GGML_TYPE_F16 : GGML_TYPE_F32, B.n_kv, 1, nb, d);
+                if (B.sparse)
+                {
+                    snprintf(nb, sizeof(nb), "bd%lld_pool_cells.%d", (long long) i, d);
+                    B.pool_cells[d] = new_input(GGML_TYPE_I32, B.n_kv, 0, nb, d);
+                    snprintf(nb, sizeof(nb), "bd%lld_pool_bias.%d", (long long) i, d);
+                    B.pool_bias[d] = new_input(m.fused_lid ? GGML_TYPE_F16 : GGML_TYPE_F32,
+                                               B.n_kv / r, 1, nb, d);
+                    snprintf(nb, sizeof(nb), "bd%lld_trail_cells.%d", (long long) i, d);
+                    B.trail_cells[d] = new_input(GGML_TYPE_I32, r, 1, nb, d);
+                    snprintf(nb, sizeof(nb), "bd%lld_trail_vals.%d", (long long) i, d);
+                    B.trail_vals[d] = new_input_3d(GGML_TYPE_F32, 1, r, 1, nb, d);
+                }
+            }
+        }
+
+        ggml_tensor * emb = ggml_get_rows(ctx, m.tok_embd, inp.tokens[m.layers[0].device]);
+        ggml_tensor * inpL = ggml_repeat_4d(ctx, ggml_reshape_3d(ctx, emb, hp.n_embd, 1, N),
+                                            hp.n_embd, hcm, N, 1);
+
+        std::vector<ggml_tensor *> top_k((size_t) N, nullptr);
+        std::vector<ggml_tensor *> masks((size_t) N, nullptr);
+
+        for (int il = 0; il < hp.n_layer; il++)
+        {
+            const glm_layer & L = m.layers[il];
+            const glm_layer_weights & LW = L.w[0];
+            const int dev = L.device;
+
+            if (il > 0 && dev != m.layers[il - 1].device)
+                pin(inpL, dev);
+
+            ggml_tensor * residual = inpL;
+            ggml_tensor * post = nullptr;
+            ggml_tensor * comb = nullptr;
+            ggml_tensor * cur = build_hc_pre(inpL, LW.hc_attn_fn, LW.hc_attn_scale, LW.hc_attn_base,
+                                             &post, &comb);
+            ggml_build_forward_expand(gf, residual);
+            ggml_build_forward_expand(gf, post);
+            ggml_build_forward_expand(gf, comb);
+            cur = rms(cur, LW.attn_norm);
+
+            ggml_tensor * attn = nullptr;
+            if (L.recurrent)
+            {
+                attn = build_kda_bd(il, cur);
+            }
+            else
+            {
+                ggml_tensor * qr = rms(ggml_mul_mat(ctx, LW.wq_a, cur), LW.q_a_norm);
+                build_indexer_g5n_bd(il, cur, qr, top_k);
+                for (int64_t i = 0; i < N; i++)
+                {
+                    bd_token & B = res.bd[(size_t) i];
+                    masks[(size_t) i] = (B.sparse && top_k[(size_t) i])
+                        ? build_topk_mask_g5n_1(dev, B.kq_mask[dev], top_k[(size_t) i],
+                                                B.trail_cells[dev], B.trail_vals[dev], B.n_kv)
+                        : B.kq_mask[dev];
+                }
+                attn = build_attention_bd(il, cur, qr, nullptr, masks);
+            }
+
+            inpL = build_hc_post(attn, residual, post, comb);
+
+            residual = inpL;
+            cur = build_hc_pre(inpL, LW.hc_ffn_fn, LW.hc_ffn_scale, LW.hc_ffn_base, &post, &comb);
+            ggml_build_forward_expand(gf, residual);
+            ggml_build_forward_expand(gf, post);
+            ggml_build_forward_expand(gf, comb);
+            cur = rms(cur, LW.ffn_norm);
+
+            ggml_tensor * ffn = nullptr;
+            if (!L.is_moe)
+            {
+                ffn = build_dense_ffn(il, 0, cur);
+            }
+            else
+            {
+                ffn = build_moe(il, 0, cur);
+                ggml_tensor * sh = build_shexp(il, 0, cur);
+                if (sh) ffn = ggml_add(ctx, ffn, sh);
+            }
+            inpL = build_hc_post(ffn, residual, post, comb);
+        }
+
+        // every token's logits
+        ggml_tensor * flat = ggml_reshape_2d(ctx, inpL, hcm * hp.n_embd, N);
+        ggml_tensor * x3 = ggml_reshape_3d(ctx, flat, hp.n_embd, hcm, N);
+        ggml_tensor * cur = hc_mean(x3);
+        cur = rms(cur, m.output_norm);
+        cur = ggml_mul_mat(ctx, m.output, cur);                                      // [vocab, N]
+        ggml_set_output(cur);
+        ggml_set_name(cur, "logits");
+        res.logits = cur;
+        ggml_build_forward_expand(gf, cur);
+        bd_log("g5n built, %d nodes\n", ggml_graph_n_nodes(gf));
+    }
+
     /// Whole graph for one batched decode step.
     void build_batched()
     {
@@ -2692,7 +3369,24 @@ struct graph_builder
                                                           : hp.n_head;
 
         ggml_tensor * q = ggml_mul_mat(ctx, LW.wq_b, qr);                       // [n_head*head_k, nt]
+        ggml_tensor * kv_pe = ggml_mul_mat(ctx, LW.wkv_a_mqa, cur);              // [kv_lora + rope, nt]
 
+        ggml_tensor * Qcur = nullptr;
+        ggml_tensor * Kcur = nullptr;
+        if (hp.n_rot == 0)
+        {
+            // glm5next MLA is nope-only: no rope half anywhere, so the latent IS
+            // the whole cache row and the absorbed query needs no concat - but
+            // it does need the cont deepseek-style graphs get from theirs.
+            ggml_tensor * kv_cmpr = rms(kv_pe, LW.kv_a_norm);
+            ggml_tensor * q3 = ggml_reshape_3d(ctx, q, hp.n_embd_head_k, n_head, nt);
+            ggml_tensor * q_nope_p = ggml_permute(ctx, q3, 0, 2, 1, 3);          // [head_k, nt, n_head]
+            ggml_tensor * q_abs = ggml_mul_mat(ctx, LW.wk_b, q_nope_p);          // [kv_lora, nt, n_head]
+            Qcur = ggml_cont(ctx, ggml_permute(ctx, q_abs, 0, 2, 1, 3));         // [kv_lora, n_head, nt]
+            Kcur = ggml_reshape_3d(ctx, kv_cmpr, hp.kv_lora_rank, 1, nt);
+        }
+        else
+        {
         ggml_tensor * q_nope = ggml_view_3d(ctx, q, hp.n_nope, n_head, nt,
                                             ggml_row_size(q->type, hp.n_embd_head_k),
                                             ggml_row_size(q->type, hp.n_embd_head_k) * n_head, 0);
@@ -2701,7 +3395,6 @@ struct graph_builder
                                           ggml_row_size(q->type, hp.n_embd_head_k) * n_head,
                                           ggml_row_size(q->type, hp.n_nope));
 
-        ggml_tensor * kv_pe = ggml_mul_mat(ctx, LW.wkv_a_mqa, cur);              // [kv_lora + rope, nt]
         const size_t row = ggml_row_size(kv_pe->type, hp.n_kv_row);
         ggml_tensor * kv_cmpr = ggml_view_2d(ctx, kv_pe, hp.kv_lora_rank, nt, row, 0);
         ggml_tensor * k_pe = ggml_view_3d(ctx, kv_pe, hp.n_rot, 1, nt, row, row,
@@ -2718,10 +3411,11 @@ struct graph_builder
         q_abs = ggml_permute(ctx, q_abs, 0, 2, 1, 3);                           // [kv_lora, n_head, nt]
 
         // rope goes last so an in-place context shift stays possible
-        ggml_tensor * Qcur = ggml_concat(ctx, q_abs, q_pe, 0);                  // [n_kv_row, n_head, nt]
+        Qcur = ggml_concat(ctx, q_abs, q_pe, 0);                                // [n_kv_row, n_head, nt]
 
-        ggml_tensor * Kcur = ggml_concat(ctx, ggml_reshape_3d(ctx, kv_cmpr, hp.kv_lora_rank, 1, nt),
-                                         k_pe, 0);                              // [n_kv_row, 1, nt]
+        Kcur = ggml_concat(ctx, ggml_reshape_3d(ctx, kv_cmpr, hp.kv_lora_rank, 1, nt),
+                           k_pe, 0);                                            // [n_kv_row, 1, nt]
+        }
 
         ggml_build_forward_expand(gf,
             ggml_set_rows(ctx, slot.kv_k[rank][il], ggml_reshape_2d(ctx, Kcur, hp.n_kv_row, nt), kv_idxs));
@@ -2767,12 +3461,26 @@ struct graph_builder
     }
 
     // ---- FFN --------------------------------------------------------------
+
+    /// silu(gate) * up, with glm5next's clamp: up into [-L, L], gate into
+    /// (-inf, L], BEFORE the activation - the reference routes the dense
+    /// layers, the shared expert and the routed experts all through it.
+    ggml_tensor * swiglu(ggml_tensor * gate, ggml_tensor * up)
+    {
+        if (hp.swiglu_clamp > 0.0f)
+        {
+            up = ggml_clamp(ctx, up, -hp.swiglu_clamp, hp.swiglu_clamp);
+            gate = ggml_clamp(ctx, gate, -INFINITY, hp.swiglu_clamp);
+        }
+        return ggml_mul(ctx, ggml_silu(ctx, gate), up);
+    }
+
     ggml_tensor * build_dense_ffn(int il, int rank, ggml_tensor * cur)
     {
         const glm_layer_weights & LW = m.layers[il].w[rank];
         ggml_tensor * gate = ggml_mul_mat(ctx, LW.ffn_gate, cur);
         ggml_tensor * up = ggml_mul_mat(ctx, LW.ffn_up, cur);
-        ggml_tensor * h = ggml_mul(ctx, ggml_silu(ctx, gate), up);
+        ggml_tensor * h = swiglu(gate, up);
         return ggml_mul_mat(ctx, LW.ffn_down, h);
     }
 
@@ -2782,7 +3490,7 @@ struct graph_builder
         if (!LW.ffn_gate_shexp) return nullptr;
         ggml_tensor * gate = ggml_mul_mat(ctx, LW.ffn_gate_shexp, cur);
         ggml_tensor * up = ggml_mul_mat(ctx, LW.ffn_up_shexp, cur);
-        ggml_tensor * h = ggml_mul(ctx, ggml_silu(ctx, gate), up);
+        ggml_tensor * h = swiglu(gate, up);
         return ggml_mul_mat(ctx, LW.ffn_down_shexp, h);
     }
 
@@ -2834,7 +3542,7 @@ struct graph_builder
 
         ggml_tensor * up = ggml_mul_mat_id(ctx, LW.ffn_up_exps, cur3, selected);    // [n_ff_exp, n_used, nt]
         ggml_tensor * gate = ggml_mul_mat_id(ctx, LW.ffn_gate_exps, cur3, selected);
-        ggml_tensor * h = ggml_mul(ctx, ggml_silu(ctx, gate), up);
+        ggml_tensor * h = swiglu(gate, up);
         ggml_tensor * experts = ggml_mul_mat_id(ctx, LW.ffn_down_exps, h, selected); // [n_embd, n_used, nt]
 
         trace("moe_sel", il, rank, selected);
@@ -2856,8 +3564,548 @@ struct graph_builder
         return moe_out;
     }
 
+
+    // =====================================================================
+    // glm5next (GLM-5.3-Flash): KDA linear attention, pooled DSA indexing and
+    // Sinkhorn hyper-connections around every residual crossing. The MLA
+    // attention, the MoE and the graph plumbing are shared with glm-dsa above.
+    // =====================================================================
+
+    // ---- hyper-connections ----
+    // The residual stream is [n_embd, hc, nt]. Each crossing computes, from an
+    // RMS-normed flattening of the stream, hc pre-weights (which stream mix the
+    // sublayer consumes), hc post-weights (how the sublayer output is written
+    // back per stream) and an [hc, hc] Sinkhorn-normalized mixing matrix.
+    std::map<ggml_tensor *, ggml_tensor *> hc_t_cache;
+
+    ggml_tensor * hc_affine(ggml_tensor * x, ggml_tensor * scale, ggml_tensor * base)
+    {
+        return ggml_add(ctx, ggml_mul(ctx, x, scale), base);
+    }
+
+    ggml_tensor * view_row_1d(ggml_tensor * t, int64_t ne0, int64_t i0)
+    {
+        return ggml_view_1d(ctx, t, ne0, ggml_row_size(t->type, i0));
+    }
+
+    ggml_tensor * view_row_2d(ggml_tensor * t, int64_t ne0, int64_t ne1, int64_t i0)
+    {
+        return ggml_view_2d(ctx, t, ne0, ne1, t->nb[1], ggml_row_size(t->type, i0));
+    }
+
+    /// The residual stream with the stream axis moved into dim 0, for the
+    /// decomposed mul_mat forms. Memoized: hc_post takes the same `residual`
+    /// its hc_pre took as `x`, so a layer pays for the transpose once.
+    ggml_tensor * hc_streams_first(ggml_tensor * x)
+    {
+        auto it = hc_t_cache.find(x);
+        if (it != hc_t_cache.end()) return it->second;
+        ggml_tensor * t = ggml_cont(ctx, ggml_permute(ctx, x, 1, 0, 2, 3));   // [hc, n_embd, nt]
+        hc_t_cache[x] = t;
+        return t;
+    }
+
+    ggml_tensor * build_hc_pre_op(ggml_tensor * x, ggml_tensor * w)
+    {
+        if (m.hc_native)
+            return ggml_dsv4_hc_pre(ctx, x, w);
+        const int64_t hcm = x->ne[1];
+        const int64_t n = x->ne[2];
+        ggml_tensor * xt = hc_streams_first(x);                       // [hc, n_embd, n]
+        ggml_tensor * w3 = ggml_reshape_3d(ctx, w, hcm, 1, n);        // [hc, 1, n]
+        ggml_tensor * out = ggml_mul_mat(ctx, xt, w3);                // [n_embd, 1, n]
+        return ggml_reshape_2d(ctx, out, x->ne[0], n);
+    }
+
+    ggml_tensor * build_hc_pre(ggml_tensor * x, ggml_tensor * fn, ggml_tensor * hc_scale, ggml_tensor * hc_base,
+                               ggml_tensor ** post, ggml_tensor ** comb)
+    {
+        const int64_t hcm = hp.hc_mult;
+        const int64_t hc_dim = hcm * hp.n_embd;
+        const int64_t n = x->ne[2];
+
+        ggml_tensor * flat = ggml_reshape_2d(ctx, x, hc_dim, n);
+        ggml_tensor * flat_norm = ggml_rms_norm(ctx, flat, hp.rms_eps);
+        ggml_tensor * mixes = ggml_mul_mat(ctx, fn, flat_norm);       // [(2+hc)*hc, n]
+
+        ggml_tensor * scale_pre = view_row_1d(hc_scale, 1, 0);
+        ggml_tensor * scale_post = view_row_1d(hc_scale, 1, 1);
+        ggml_tensor * base_pre = view_row_1d(hc_base, hcm, 0);
+        ggml_tensor * base_post = view_row_1d(hc_base, hcm, hcm);
+
+        ggml_tensor * pre = view_row_2d(mixes, hcm, n, 0);
+        pre = hc_affine(pre, scale_pre, base_pre);
+        pre = ggml_sigmoid(ctx, pre);
+        pre = ggml_scale_bias(ctx, pre, 1.0f, hp.hc_eps);
+
+        *post = view_row_2d(mixes, hcm, n, hcm);
+        *post = hc_affine(*post, scale_post, base_post);
+        *post = ggml_sigmoid(ctx, *post);
+        *post = ggml_scale(ctx, *post, 2.0f);
+
+        *comb = ggml_dsv4_hc_comb(ctx, mixes, hc_scale, hc_base, hp.hc_eps, hp.hc_sinkhorn);
+
+        return build_hc_pre_op(x, pre);
+    }
+
+    ggml_tensor * build_hc_post(ggml_tensor * x, ggml_tensor * residual, ggml_tensor * post, ggml_tensor * comb)
+    {
+        if (m.hc_native)
+            return ggml_dsv4_hc_post(ctx, x, residual, post, comb);
+
+        const int64_t n_embd = x->ne[0];
+        const int64_t n = x->ne[1];
+        const int64_t hcm = residual->ne[1];
+
+        // rank-1 term: an outer product per token, expressed as a mul_mat whose
+        // contracted dimension is 1.
+        ggml_tensor * x1 = ggml_reshape_3d(ctx, x, 1, n_embd, n);
+        ggml_tensor * p1 = ggml_reshape_3d(ctx, post, 1, hcm, n);
+        ggml_tensor * outer = ggml_mul_mat(ctx, x1, p1);              // [n_embd, hc, n]
+
+        ggml_tensor * rt = hc_streams_first(residual);                // [hc(src), n_embd, n]
+        ggml_tensor * ct = ggml_cont(ctx, ggml_permute(ctx, comb, 1, 0, 2, 3));   // [hc(src), hc(dst), n]
+        ggml_tensor * mixed = ggml_mul_mat(ctx, rt, ct);              // [n_embd, hc(dst), n]
+
+        return ggml_add(ctx, outer, mixed);
+    }
+
+    /// Mean over the hyper-connection streams: [n_embd, hc, n] -> [n_embd, n].
+    /// glm5next's head is this unweighted mean, not DSV4's learned gated one.
+    ggml_tensor * hc_mean(ggml_tensor * x)
+    {
+        const int64_t hcm = x->ne[1];
+        const int64_t n = x->ne[2];
+        ggml_tensor * acc = nullptr;
+        for (int64_t c = 0; c < hcm; c++)
+        {
+            ggml_tensor * v = ggml_cont(ctx, ggml_view_2d(ctx, x, hp.n_embd, n, x->nb[2], (size_t) c * x->nb[1]));
+            acc = acc ? ggml_add(ctx, acc, v) : v;
+        }
+        return ggml_scale(ctx, acc, 1.0f / (float) hcm);
+    }
+
+    // ---- KDA linear attention ----
+    // Short conv over concatenated q/k/v with a persistent (d_conv-1)-column
+    // tail, l2-normed q/k, a per-CHANNEL decay gate bounded below
+    // multiplicatively, and the fused gated-delta-net recurrence whose state
+    // lives in the slot and is committed in-graph. f, g and beta read the layer
+    // input, not the convolved q/k/v.
+    ggml_tensor * build_kda(int il, ggml_tensor * cur)
+    {
+        const glm_layer_weights & LW = m.layers[il].w[0];
+        const int64_t hd = hp.kda_head_dim;
+        const int64_t H = hp.kda_n_head;
+        const int64_t d_inner = hd * H;
+        const int64_t dc = hp.d_conv;
+
+        ggml_tensor * qp = ggml_mul_mat(ctx, LW.kda_wq, cur);
+        ggml_tensor * kp = ggml_mul_mat(ctx, LW.kda_wk, cur);
+        ggml_tensor * vp = ggml_mul_mat(ctx, LW.kda_wv, cur);
+        ggml_tensor * qkv = ggml_concat(ctx, ggml_concat(ctx, qp, kp, 0), vp, 0);   // [3*d_inner, nt]
+        trace("kda_qkv", il, 0, qkv);
+
+        // stored separately in the file, stacked back into one kernel
+        ggml_tensor * conv_w = ggml_concat(ctx,
+                ggml_concat(ctx,
+                    ggml_reshape_2d(ctx, LW.kda_conv_q, dc, d_inner),
+                    ggml_reshape_2d(ctx, LW.kda_conv_k, dc, d_inner), 1),
+                ggml_reshape_2d(ctx, LW.kda_conv_v, dc, d_inner), 1);               // [dc, 3*d_inner]
+
+        ggml_tensor * conv_state = slot.kda_conv[0][(size_t) il];                    // [dc-1, 3*d_inner]
+        ggml_tensor * qkv_t = ggml_reshape_3d(ctx, ggml_cont(ctx, ggml_transpose(ctx, qkv)), nt, 3 * d_inner, 1);
+        ggml_tensor * conv_in = ggml_concat(ctx, ggml_reshape_3d(ctx, conv_state, dc - 1, 3 * d_inner, 1),
+                                            qkv_t, 0);                               // [dc-1+nt, 3*d_inner, 1]
+        // SiLU on the conv output, not on the projections
+        ggml_tensor * conv_out = ggml_silu(ctx, ggml_ssm_conv(ctx, conv_in, conv_w));   // [3*d_inner, nt]
+        trace("kda_conv", il, 0, conv_out);
+
+        // keep the last dc-1 columns for the next ubatch. The concat above has
+        // materialized conv_in, so overwriting the state here cannot disturb it.
+        ggml_tensor * tail = ggml_view_3d(ctx, conv_in, dc - 1, 3 * d_inner, 1,
+                conv_in->nb[1], conv_in->nb[2], (size_t) nt * conv_in->nb[0]);
+        ggml_build_forward_expand(gf, ggml_cpy(ctx, tail, conv_state));
+
+        const size_t rs_hd = ggml_row_size(conv_out->type, hd);
+        ggml_tensor * qc = ggml_view_3d(ctx, conv_out, hd, H, nt, rs_hd, conv_out->nb[1], 0);
+        ggml_tensor * kc = ggml_view_3d(ctx, conv_out, hd, H, nt, rs_hd, conv_out->nb[1],
+                ggml_row_size(conv_out->type, d_inner));
+        ggml_tensor * vc = ggml_view_3d(ctx, conv_out, hd, H, nt, rs_hd, conv_out->nb[1],
+                ggml_row_size(conv_out->type, 2 * d_inner));
+
+        // 1e-6 is the reference's own constant, not the model's norm eps. The
+        // 1/sqrt(hd) query scale is applied inside the fused op.
+        qc = ggml_l2_norm(ctx, ggml_cont(ctx, qc), 1e-6f);
+        kc = ggml_l2_norm(ctx, ggml_cont(ctx, kc), 1e-6f);
+        qc = ggml_reshape_4d(ctx, qc, hd, H, nt, 1);
+        kc = ggml_reshape_4d(ctx, kc, hd, H, nt, 1);
+        vc = ggml_reshape_4d(ctx, ggml_cont(ctx, vc), hd, H, nt, 1);
+
+        // forget gate: g = lower_bound * sigmoid(exp(A_log) * (f_b(f_a(x)) + dt_bias)),
+        // per channel; ssm_a holds -exp(A_log), so exp(A_log)*y == -(y * ssm_a).
+        ggml_tensor * g = ggml_mul_mat(ctx, LW.kda_f_b, ggml_mul_mat(ctx, LW.kda_f_a, cur));
+        g = ggml_add(ctx, g, LW.kda_dt_b);
+        g = ggml_reshape_3d(ctx, g, hd, H, nt);
+        g = ggml_mul(ctx, g, ggml_reshape_3d(ctx, LW.kda_a, 1, H, 1));
+        g = ggml_sigmoid(ctx, ggml_scale(ctx, g, -1.0f));
+        g = ggml_scale(ctx, g, hp.kda_gate_lb);
+        ggml_tensor * g4 = ggml_reshape_4d(ctx, g, hd, H, nt, 1);
+        trace("kda_gate", il, 0, g4);
+
+        ggml_tensor * beta = ggml_sigmoid(ctx, ggml_mul_mat(ctx, LW.kda_beta, cur));
+        ggml_tensor * b4 = ggml_reshape_4d(ctx, beta, 1, H, nt, 1);
+
+        ggml_tensor * state = slot.kda_ssm[0][(size_t) il];                          // [hd, hd, H]
+        ggml_tensor * s4 = ggml_reshape_4d(ctx, state, hd, hd, H, 1);
+
+        ggml_tensor * gdn = ggml_gated_delta_net(ctx, qc, kc, vc, g4, b4, s4, 1);
+
+        const int64_t attn_elems = hd * H * nt;
+        ggml_tensor * core = ggml_view_3d(ctx, gdn, hd, H, nt,
+                ggml_row_size(gdn->type, hd), ggml_row_size(gdn->type, hd * H), 0);
+        ggml_tensor * new_state = ggml_view_3d(ctx, gdn, hd, hd, H,
+                ggml_row_size(gdn->type, hd), ggml_row_size(gdn->type, hd * hd),
+                ggml_row_size(gdn->type, attn_elems));
+        ggml_build_forward_expand(gf, ggml_cpy(ctx, new_state, state));
+        trace("kda_scan_out", il, 0, core);
+
+        // low-rank output gate; RMS over hd with one weight shared by every
+        // head, then a plain sigmoid gate (not FusedRMSNormGated's SiLU).
+        ggml_tensor * gate = ggml_mul_mat(ctx, LW.kda_g_b, ggml_mul_mat(ctx, LW.kda_g_a, cur));
+        gate = ggml_reshape_3d(ctx, gate, hd, H, nt);
+        ggml_tensor * normed = ggml_mul(ctx, ggml_rms_norm(ctx, ggml_cont(ctx, core), hp.rms_eps), LW.kda_o_norm);
+        ggml_tensor * gated = ggml_mul(ctx, normed, ggml_sigmoid(ctx, gate));
+        trace("kda_normed", il, 0, gated);
+
+        ggml_tensor * out2 = ggml_reshape_2d(ctx, ggml_cont(ctx, gated), d_inner, nt);
+        ggml_tensor * proj = ggml_mul_mat(ctx, LW.kda_wo, out2);
+        trace("kda_out", il, 0, proj);
+        return proj;
+    }
+
+    // ---- pooled lightning indexer ----
+    // Caches this layer's indexer key AND compressor gate ([key | gate] in one
+    // cell row - stored unconditionally, exactly like glm-dsa's keys), then,
+    // when scoring, compresses each kpool-cell pool with a softmax over the
+    // cached gates (plus a per-slot additive position embedding), scores the
+    // POOLS, takes the top-k over pools and expands the winners back to their
+    // member cells. Cell-level top-k is NOT equivalent: ReLU drives most pool
+    // scores to exactly 0, tie groups span pools, and an unordered top-k then
+    // truncates a pool mid-way.
+    //
+    // No rope (the tower is nope-only) and no Hadamard: the rotation exists to
+    // help fp8 kernels, scoring in F32 here it would only cost accuracy.
+    ggml_tensor * build_indexer_g5n(int il, int rank, ggml_tensor * cur, ggml_tensor * qr, bool score_now)
+    {
+        const glm_layer & L = m.layers[il];
+        const glm_layer_weights & LW = L.w[rank];
+        const int dev = device_of(il, rank);
+        const int64_t D = hp.indexer_head_size;
+        const int64_t H = hp.indexer_n_head;
+        const int64_t r = hp.indexer_kpool;
+
+        // a genuine LayerNorm: weight AND bias, at eps 1e-6 (from the GGUF).
+        ggml_tensor * ik = ggml_mul_mat(ctx, LW.idx_attn_k, cur);          // [D, nt]
+        ik = ggml_norm(ctx, ik, hp.norm_eps);
+        ik = ggml_mul(ctx, ik, LW.idx_k_norm_w);
+        ik = ggml_add(ctx, ik, LW.idx_k_norm_b);
+        trace("indexer_k", il, rank, ik);
+
+        // the pooling gate is a SECOND, INDEPENDENT projection of the hidden
+        // state; it must be cached beside the key because a pool is only built
+        // once its members have left the batch.
+        ggml_tensor * gate = ggml_mul_mat(ctx, LW.idx_comp_gate, cur);     // [D, nt]
+
+        ggml_tensor * packed = ggml_concat(ctx,
+                ggml_reshape_3d(ctx, ik, D, 1, nt),
+                ggml_reshape_3d(ctx, gate, D, 1, nt), 1);                  // [D, 2, nt]
+        ggml_build_forward_expand(gf,
+            ggml_set_rows(ctx, slot.idx_k[rank][il], ggml_reshape_2d(ctx, packed, 2 * D, nt),
+                          res.inp.kv_idxs[dev]));
+
+        if (!score_now)
+            return nullptr;
+
+        ggml_tensor * kbuf = slot.idx_k[rank][il];                          // [2D, n_ctx] F16
+        const int64_t n_pools = n_kv / r;
+
+        // gather each pool's members: one row per cell, key and gate adjacent,
+        // so the members are fetched once. ggml_get_rows yields F32, so the
+        // compression runs in F32 even though the cache is F16.
+        ggml_tensor * kg = ggml_view_2d(ctx, kbuf, 2 * D, n_kv, kbuf->nb[1], 0);
+        ggml_tensor * members = ggml_get_rows(ctx, kg, res.inp.pool_cells[dev]);   // [2D, r*n_pools]
+
+        const size_t nb_mem = members->nb[1];
+        ggml_tensor * mem_k = ggml_view_3d(ctx, members, D, r, n_pools, nb_mem, nb_mem * r, 0);
+        ggml_tensor * mem_g = ggml_view_3d(ctx, members, D, r, n_pools, nb_mem, nb_mem * r,
+                                           (size_t) D * members->nb[0]);
+
+        // D independent r-way softmaxes over the SLOT axis; ape is added
+        // pre-softmax and is indexed by logical slot (position % kpool), which
+        // pool_cells' position order matches by construction.
+        ggml_tensor * keys_t = ggml_cont(ctx, ggml_permute(ctx, mem_k, 1, 0, 2, 3));   // [r, D, n_pools]
+        ggml_tensor * gate_t = ggml_cont(ctx, ggml_permute(ctx, mem_g, 1, 0, 2, 3));
+        ggml_tensor * ape = ggml_cont(ctx, ggml_transpose(ctx, LW.idx_comp_ape));      // [r, D]
+        gate_t = ggml_add(ctx, gate_t, ggml_reshape_3d(ctx, ape, r, D, 1));
+        ggml_tensor * probs = ggml_soft_max(ctx, gate_t);
+
+        // per-channel weighted average over the pool members -> [D, n_pools]
+        ggml_tensor * pool_k = ggml_sum_rows(ctx, ggml_mul(ctx, keys_t, probs));       // [1, D, n_pools]
+        pool_k = ggml_reshape_2d(ctx, ggml_cont(ctx, pool_k), D, n_pools);
+        trace("indexer_pool_k", il, rank, pool_k);
+
+        ggml_tensor * q = ggml_mul_mat(ctx, LW.idx_attn_q_b, qr);           // [D*H, nt]
+        q = ggml_reshape_3d(ctx, q, D, H, nt);
+
+        // sign-unconstrained head weights, both scale constants folded in on
+        // the small tensor. F32 is not cosmetic: a bf16 head gate moves logits
+        // by ~1e-2, enough to swap near-tied pools under a hard top-k cut.
+        ggml_tensor * w = ggml_mul_mat(ctx, LW.idx_proj, cur);              // [H, nt]
+        ggml_mul_mat_set_prec(w, GGML_PREC_F32);
+        w = ggml_scale(ctx, w, 1.0f / sqrtf((float) (D * H)));
+
+        ggml_tensor * score = nullptr;
+        if (m.fused_lid)
+        {
+            // pool_k stays F32 so the kernel takes its F32 vector path; the
+            // pool-visibility bias rides in as the mask.
+            ggml_tensor * pool_kf = ggml_reshape_3d(ctx, pool_k, D, 1, n_pools);
+            score = ggml_lightning_indexer(ctx, q, pool_kf, w, res.inp.pool_bias[dev]);   // [n_pools, nt]
+        }
+        else
+        {
+            ggml_tensor * qp2 = ggml_permute(ctx, q, 0, 2, 1, 3);           // [D, nt, H]
+            ggml_tensor * kq = ggml_mul_mat(ctx, pool_k, qp2);              // [n_pools, nt, H]
+            // the ReLU sits BETWEEN the per-head dot product and the head
+            // weighting; the weights are sign-free, so moving it is a
+            // different function.
+            kq = ggml_cont(ctx, ggml_permute(ctx, kq, 2, 1, 0, 3));         // [H, nt, n_pools]
+            ggml_tensor * sc = ggml_relu(ctx, kq);
+            sc = ggml_mul(ctx, sc, w);
+            sc = ggml_sum_rows(ctx, sc);                                    // [1, nt, n_pools]
+            sc = ggml_cont(ctx, ggml_permute(ctx, sc, 2, 1, 0, 3));         // [n_pools, nt, 1]
+            score = ggml_add(ctx, sc, res.inp.pool_bias[dev]);
+        }
+        trace("indexer_pool_score", il, rank, score);
+
+        // top-k over POOLS, then expand each winner to its member cells.
+        const int64_t select_k = std::min<int64_t>(n_pools, hp.indexer_top_k / r);
+        ggml_tensor * sel = ggml_cont(ctx, ggml_top_k(ctx, score, (int) select_k));    // I32 [select_k, nt]
+        ggml_tensor * pc2 = ggml_reshape_2d(ctx, res.inp.pool_cells[dev], r, n_pools);
+        ggml_tensor * sel_flat = ggml_reshape_1d(ctx, sel, select_k * nt);
+        ggml_tensor * cells = ggml_get_rows(ctx, pc2, sel_flat);            // I32 [r, select_k*nt]
+        ggml_tensor * out = ggml_reshape_2d(ctx, cells, r * select_k, nt);
+        trace("indexer_top_k", il, rank, out);
+        return out;
+    }
+
+    /// build_topk_mask plus the always-attended trailing cells of each query's
+    /// own incomplete pool: -inf canvas, write the trail lane values (0 for a
+    /// real tail cell, -inf for an unused lane - a no-op on the canvas), then
+    /// unmask the selected pools' cells, then add the causal mask back so a
+    /// selected-but-future cell stays masked. The trail write runs FIRST so a
+    /// pool scatter landing on the same cell wins.
+    ggml_tensor * build_topk_mask_g5n(int dev, ggml_tensor * kq_mask, ggml_tensor * top_k,
+                                      ggml_tensor * trail_cells, ggml_tensor * trail_vals)
+    {
+        const int64_t n_top_k = top_k->ne[0];
+        const int64_t r = trail_cells->ne[0];
+        trace("topk", trace_il, trace_rank, top_k);
+
+        ggml_tensor * base = ggml_fill(ctx, kq_mask, -INFINITY);               // [n_kv, nt]
+        ggml_set_output(base);
+        ggml_tensor * all = ggml_view_3d(ctx, base, 1, n_kv, nt, base->nb[0], base->nb[1], 0);
+
+        ggml_tensor * t_idx = ggml_view_3d(ctx, trail_cells, r, nt, 1,
+                                           trail_cells->nb[1], trail_cells->nb[2], 0);
+        ggml_tensor * with_tail = ggml_set_rows(ctx, all, trail_vals, t_idx);   // [1, n_kv, nt]
+
+        ggml_tensor * idx = ggml_view_3d(ctx, top_k, n_top_k, nt, 1,
+                                         top_k->nb[1], top_k->nb[2], 0);
+        ggml_tensor * zeros = ggml_fill(ctx, ggml_new_tensor_3d(ctx, GGML_TYPE_F32, 1, n_top_k, nt), 0.0f);
+        ggml_tensor * unmasked = ggml_set_rows(ctx, with_tail, zeros, idx);
+
+        ggml_tensor * masked = ggml_view_2d(ctx, unmasked, n_kv, nt, unmasked->nb[2], 0);
+        ggml_tensor * out = ggml_add(ctx, masked, kq_mask);
+
+        pin(base, dev);
+        pin(zeros, dev);
+        pin(with_tail, dev);
+        pin(unmasked, dev);
+        pin(out, dev);
+        return out;
+    }
+
+    // ---- the glm5next trunk ----
+    void build_g5n()
+    {
+        graph_inputs & inp = res.inp;
+        const int64_t hcm = hp.hc_mult;
+        const int64_t r = hp.indexer_kpool;
+        const int64_t n_pools = n_kv / r;
+
+        std::vector<uint8_t> used_dev((size_t) m.n_gpu + 1, 0);
+        for (int il = 0; il < hp.n_layer; il++) used_dev[(size_t) m.layers[il].device] = 1;
+
+        // Only MLA layers read the mask, the cache indices and the pool inputs;
+        // a KDA-only device gets none (an input no node reads is never
+        // allocated, and writing to it would fault).
+        const ggml_type mask_type = m.flash_attn ? GGML_TYPE_F16 : GGML_TYPE_F32;
+        std::vector<uint8_t> dev_mla((size_t) m.n_gpu + 1, 0);
+        for (int il = 0; il < hp.n_layer; il++)
+            if (!m.layers[il].recurrent) dev_mla[(size_t) m.layers[il].device] = 1;
+
+        for (int d = 0; d <= m.n_gpu; d++)
+        {
+            if (!used_dev[(size_t) d] || !dev_mla[(size_t) d]) continue;
+            char nb[64];
+            snprintf(nb, sizeof(nb), "kq_mask.%d", d);
+            inp.kq_mask[d] = new_input(mask_type, n_kv, nt, nb, d);
+            snprintf(nb, sizeof(nb), "kv_idxs.%d", d);
+            inp.kv_idxs[d] = new_input(GGML_TYPE_I64, nt, 0, nb, d);
+            if (sparse)
+            {
+                snprintf(nb, sizeof(nb), "pool_cells.%d", d);
+                inp.pool_cells[d] = new_input(GGML_TYPE_I32, r * n_pools, 0, nb, d);
+                snprintf(nb, sizeof(nb), "pool_bias.%d", d);
+                inp.pool_bias[d] = new_input(m.fused_lid ? GGML_TYPE_F16 : GGML_TYPE_F32, n_pools, nt, nb, d);
+                snprintf(nb, sizeof(nb), "trail_cells.%d", d);
+                inp.trail_cells[d] = new_input(GGML_TYPE_I32, r, nt, nb, d);
+                snprintf(nb, sizeof(nb), "trail_vals.%d", d);
+                inp.trail_vals[d] = new_input_3d(GGML_TYPE_F32, 1, r, nt, nb, d);
+            }
+        }
+
+        const int dev_embd = m.layers[0].device;
+        {
+            char nb[64];
+            snprintf(nb, sizeof(nb), "inp_tokens.%d", dev_embd);
+            inp.tokens[dev_embd] = new_input(GGML_TYPE_I32, nt, 0, nb, dev_embd);
+        }
+        const int dev_last = m.layers[(size_t) hp.n_layer - 1].device;
+        if (res.want_logits)
+            inp.out_ids = new_input(GGML_TYPE_I32, res.n_out, 0, "inp_out_ids", dev_last);
+
+        // hc_mult exact copies of the embedding: no scaling, no one-hot.
+        ggml_tensor * emb = ggml_get_rows(ctx, m.tok_embd, inp.tokens[dev_embd]);
+        if (res.n_ovr > 0)
+        {
+            // Vision rows: the projected image embeddings replace the token
+            // embeddings of the placeholder positions before the stream fans
+            // out. get_rows leaves `emb` contiguous, so set_rows may write
+            // straight through it.
+            inp.embd_rows = new_input(GGML_TYPE_F32, hp.n_embd, res.n_ovr, "embd_rows", dev_embd);
+            inp.embd_idx = new_input(GGML_TYPE_I64, res.n_ovr, 0, "embd_idx", dev_embd);
+            emb = ggml_set_rows(ctx, emb, inp.embd_rows, inp.embd_idx);
+        }
+        ggml_tensor * inpL = ggml_repeat_4d(ctx, ggml_reshape_3d(ctx, emb, hp.n_embd, 1, nt),
+                                            hp.n_embd, hcm, nt, 1);
+
+        for (int il = 0; il < hp.n_layer; il++)
+        {
+            const glm_layer & L = m.layers[il];
+            const glm_layer_weights & LW = L.w[0];
+            const int dev = L.device;
+
+            if (il > 0 && dev != m.layers[il - 1].device)
+                pin(inpL, dev);
+
+            // ---- attention crossing ----
+            ggml_tensor * residual = inpL;
+            ggml_tensor * post = nullptr;
+            ggml_tensor * comb = nullptr;
+            ggml_tensor * cur = build_hc_pre(inpL, LW.hc_attn_fn, LW.hc_attn_scale, LW.hc_attn_base,
+                                             &post, &comb);
+            ggml_build_forward_expand(gf, residual);
+            ggml_build_forward_expand(gf, post);
+            ggml_build_forward_expand(gf, comb);
+            trace("hc_attn_pre", il, 0, cur);
+
+            cur = rms(cur, LW.attn_norm);
+            trace("attn_norm", il, 0, cur);
+
+            ggml_tensor * attn = nullptr;
+            if (L.recurrent)
+            {
+                attn = build_kda(il, cur);
+            }
+            else
+            {
+                ggml_tensor * qr = rms(ggml_mul_mat(ctx, LW.wq_a, cur), LW.q_a_norm);
+                trace("qr", il, 0, qr);
+                ggml_tensor * mask = inp.kq_mask[dev];
+                if (L.indexer_full)
+                {
+                    ggml_tensor * tk = build_indexer_g5n(il, 0, cur, qr, sparse);
+                    trace_il = il; trace_rank = 0;
+                    if (sparse && tk)
+                        mask = build_topk_mask_g5n(dev, inp.kq_mask[dev], tk,
+                                                   inp.trail_cells[dev], inp.trail_vals[dev]);
+                }
+                attn = build_attention(il, 0, cur, qr, inp.pos[dev], mask, inp.kv_idxs[dev]);
+            }
+            trace("attn_out", il, 0, attn);
+
+            inpL = build_hc_post(attn, residual, post, comb);
+            trace("hc_attn_post", il, 0, inpL);
+
+            // ---- FFN crossing ----
+            residual = inpL;
+            cur = build_hc_pre(inpL, LW.hc_ffn_fn, LW.hc_ffn_scale, LW.hc_ffn_base, &post, &comb);
+            // expand before the sublayer so op offload cannot pull the mHC
+            // state onto the expert weights' backend
+            ggml_build_forward_expand(gf, residual);
+            ggml_build_forward_expand(gf, post);
+            ggml_build_forward_expand(gf, comb);
+
+            cur = rms(cur, LW.ffn_norm);
+            trace("ffn_norm", il, 0, cur);
+
+            ggml_tensor * ffn = nullptr;
+            if (!L.is_moe)
+            {
+                ffn = build_dense_ffn(il, 0, cur);
+            }
+            else
+            {
+                ffn = build_moe(il, 0, cur);
+                ggml_tensor * sh = build_shexp(il, 0, cur);
+                if (sh) ffn = ggml_add(ctx, ffn, sh);
+            }
+            trace("ffn_out", il, 0, ffn);
+
+            inpL = build_hc_post(ffn, residual, post, comb);
+            trace("l_out", il, 0, inpL);
+        }
+
+        if (!res.want_logits)
+        {
+            // A non-final prefill chunk only has to leave its caches and
+            // recurrent state behind.
+            ggml_build_forward_expand(gf, inpL);
+            return;
+        }
+
+        // select the output rows BEFORE the mean: one token's streams are one
+        // contiguous row of the flattened stream tensor.
+        ggml_tensor * flat = ggml_reshape_2d(ctx, inpL, hcm * hp.n_embd, nt);
+        ggml_tensor * selr = ggml_get_rows(ctx, flat, inp.out_ids);
+        ggml_tensor * x3 = ggml_reshape_3d(ctx, selr, hp.n_embd, hcm, res.n_out);
+
+        // unweighted mean over the streams, then the ordinary norm + head.
+        ggml_tensor * cur = hc_mean(x3);
+        cur = rms(cur, m.output_norm);
+        cur = ggml_mul_mat(ctx, m.output, cur);
+        ggml_set_output(cur);
+        ggml_set_name(cur, "logits");
+        res.logits = cur;
+        ggml_build_forward_expand(gf, cur);
+    }
+
     void build()
     {
+        if (hp.g5n) { build_g5n(); return; }
+
         graph_inputs & inp = res.inp;
 
         // Per-token inputs are duplicated per participating device so the
@@ -3235,7 +4483,10 @@ static graph_build_result * acquire_batched_graph(glm_model & m, int n, const in
         B.slot_id = slot_ids[i];
         B.p = positions[i];
         B.n_kv = plan_n_kv(m, B.p + 1);
-        B.sparse = topk_enabled && (B.p + 1) > m.hp.indexer_top_k;
+        const int64_t n_select = m.hp.indexer_kpool > 0
+            ? (int64_t) m.hp.indexer_top_k + m.hp.indexer_kpool - 1
+            : (int64_t) m.hp.indexer_top_k;
+        B.sparse = topk_enabled && (B.p + 1) > n_select;
         if (B.p + 1 > m.n_ctx) return nullptr;
     }
 
@@ -3282,7 +4533,8 @@ static graph_build_result * acquire_batched_graph(glm_model & m, int n, const in
 
     glm_slot & any = *m.slots.at(slot_ids[0]);
     graph_builder gb(m, *entry, any, n, 0, 0, false);
-    gb.build_batched();
+    if (m.hp.g5n) gb.build_batched_g5n();
+    else          gb.build_batched();
 
     if (!ggml_backend_sched_alloc_graph(entry->sched, entry->gf))
     {
@@ -3374,6 +4626,44 @@ static bool forward_batched_decode(glm_model & m, int n, const int32_t * slot_id
                 }
                 ggml_backend_tensor_set(B.lid_mask[d], m16.data(), 0, (size_t) B.n_kv * 2);
             }
+            if (live(B.pool_cells[d]))
+            {
+                const int64_t r = m.hp.indexer_kpool;
+                const int64_t n_pools = B.n_kv / r;
+                std::vector<int32_t> pc((size_t) B.n_kv);
+                for (int64_t j = 0; j < B.n_kv; j++) pc[(size_t) j] = (int32_t) j;
+                ggml_backend_tensor_set(B.pool_cells[d], pc.data(), 0, pc.size() * sizeof(int32_t));
+
+                const int64_t bo_vis = std::min<int64_t>((p + 1) / r, n_pools);
+                if (B.pool_bias[d] && B.pool_bias[d]->buffer)
+                {
+                    if (B.pool_bias[d]->type == GGML_TYPE_F16)
+                    {
+                        std::vector<uint16_t> pb((size_t) n_pools, 0xFC00);
+                        for (int64_t b = 0; b < bo_vis; b++) pb[(size_t) b] = 0;
+                        ggml_backend_tensor_set(B.pool_bias[d], pb.data(), 0, pb.size() * 2);
+                    }
+                    else
+                    {
+                        std::vector<float> pb((size_t) n_pools, -INFINITY);
+                        for (int64_t b = 0; b < bo_vis; b++) pb[(size_t) b] = 0.0f;
+                        ggml_backend_tensor_set(B.pool_bias[d], pb.data(), 0, pb.size() * 4);
+                    }
+                }
+                const int64_t ts = (p + 1) / r * r;
+                const int64_t n_tail = p + 1 - ts;
+                std::vector<int32_t> tc((size_t) r);
+                std::vector<float> tv((size_t) r);
+                for (int64_t j = 0; j < r; j++)
+                {
+                    tc[(size_t) j] = (int32_t) std::min<int64_t>(ts + j, B.n_kv - 1);
+                    tv[(size_t) j] = j < n_tail ? 0.0f : -INFINITY;
+                }
+                if (live(B.trail_cells[d]))
+                    ggml_backend_tensor_set(B.trail_cells[d], tc.data(), 0, tc.size() * sizeof(int32_t));
+                if (live(B.trail_vals[d]))
+                    ggml_backend_tensor_set(B.trail_vals[d], tv.data(), 0, tv.size() * sizeof(float));
+            }
         }
     }
 
@@ -3396,19 +4686,25 @@ static bool forward_batched_decode(glm_model & m, int n, const int32_t * slot_id
 
 static graph_build_result * acquire_graph(glm_model & m, glm_slot & slot, int64_t nt, int64_t p0,
                                           int64_t n_out, bool want_logits, bool * out_reused,
-                                          bool want_h = false, int kind = 0)
+                                          bool want_h = false, int kind = 0, int64_t n_ovr = 0)
 {
     const int64_t n_kv = plan_n_kv(m, p0 + nt);
     static const bool topk_enabled = []() { const char * e = getenv("TS_GLM_TOPK"); return !(e && atoi(e) == 0); }();
     // The draft block attends densely, so the indexer's sparsity never applies
     // to it and must not enter its cache key.
-    const bool sparse = kind == 0 && topk_enabled && (p0 + nt) > m.hp.indexer_top_k;
+    // glm5next selects whole pools plus the query's own trailing pool; below
+    // top_k + kpool - 1 resident positions the selection cannot drop anything.
+    const int64_t n_select = m.hp.indexer_kpool > 0
+        ? (int64_t) m.hp.indexer_top_k + m.hp.indexer_kpool - 1
+        : (int64_t) m.hp.indexer_top_k;
+    const bool sparse = kind == 0 && topk_enabled && (p0 + nt) > n_select;
 
     for (auto it = m.graph_cache.begin(); it != m.graph_cache.end(); ++it)
     {
         graph_build_result * e = it->get();
         if (e->nt == nt && e->n_kv == n_kv && e->n_out == n_out && e->want_logits == want_logits &&
-            e->sparse == sparse && e->slot_id == slot.id && e->want_h == want_h && e->kind == kind)
+            e->sparse == sparse && e->slot_id == slot.id && e->want_h == want_h && e->kind == kind &&
+            e->n_ovr == n_ovr)
         {
             auto entry = std::move(*it);
             m.graph_cache.erase(it);
@@ -3428,6 +4724,7 @@ static graph_build_result * acquire_graph(glm_model & m, glm_slot & slot, int64_
     entry->want_h = want_h;
     entry->kind = kind;
     entry->slot_id = slot.id;
+    entry->n_ovr = n_ovr;
 
     // Node budget. A GLM layer costs ~110 nodes per rank (MoE alone is ~40, the
     // DSA indexer another ~20 on a full layer), so 256 per layer per rank leaves
@@ -3478,7 +4775,8 @@ static graph_build_result * acquire_graph(glm_model & m, glm_slot & slot, int64_
 /// row instead of only the last, which is what makes one verify pass over a
 /// speculative window cost one trunk forward instead of K+1 of them.
 static bool forward_ubatch(glm_model & m, const int32_t * tokens, int64_t nt, bool want_logits, float * logits_out,
-                           float * h_out = nullptr, bool all_logits_rows = false)
+                           float * h_out = nullptr, bool all_logits_rows = false,
+                           const float * ovr_rows = nullptr, const int64_t * ovr_idx = nullptr, int64_t n_ovr = 0)
 {
     glm_slot & slot = *m.active_slot;
     const int64_t p0 = slot.n_past;
@@ -3491,7 +4789,7 @@ static bool forward_ubatch(glm_model & m, const int32_t * tokens, int64_t nt, bo
     const bool want_h = h_out != nullptr;
     const int64_t n_out = (want_logits && all_logits_rows) ? nt : 1;
     bool reused = false;
-    graph_build_result * gr = acquire_graph(m, slot, nt, p0, n_out, want_logits, &reused, want_h, /*kind=*/0);
+    graph_build_result * gr = acquire_graph(m, slot, nt, p0, n_out, want_logits, &reused, want_h, /*kind=*/0, n_ovr);
     if (!gr) return false;
 
     const int64_t n_kv = gr->n_kv;
@@ -3553,6 +4851,73 @@ static bool forward_ubatch(glm_model & m, const int32_t * tokens, int64_t nt, bo
     }
     set_input_i32(gr->inp.out_ids, m.h_out_ids.data(), (size_t) n_out);
 
+    if (n_ovr > 0)
+    {
+        if (live(gr->inp.embd_rows))
+            ggml_backend_tensor_set(gr->inp.embd_rows, ovr_rows, 0,
+                                    (size_t) n_ovr * m.hp.n_embd * sizeof(float));
+        if (live(gr->inp.embd_idx))
+            ggml_backend_tensor_set(gr->inp.embd_idx, ovr_idx, 0, (size_t) n_ovr * sizeof(int64_t));
+    }
+
+    // glm5next pooled-indexer inputs. Pools are position-aligned windows of
+    // kpool cells over the padded cache window (256 % kpool == 0, so the map is
+    // the identity); a pool is scoreable for a query only when its LAST member
+    // is visible, which also drops the query's own trailing pool - those cells
+    // ride in through trail_cells/trail_vals instead, unconditionally.
+    if (m.hp.g5n && gr->sparse)
+    {
+        const int64_t r = m.hp.indexer_kpool;
+        const int64_t n_pools = n_kv / r;
+        std::vector<int32_t> pcells((size_t) (r * n_pools));
+        for (int64_t j = 0; j < r * n_pools; j++) pcells[(size_t) j] = (int32_t) j;
+
+        std::vector<float> pbias_f32;
+        std::vector<uint16_t> pbias_f16;
+        bool want_f16 = false, want_f32 = false;
+        for (int d = 0; d <= m.n_gpu; d++)
+            if (live(gr->inp.pool_bias[d]))
+                (gr->inp.pool_bias[d]->type == GGML_TYPE_F16 ? want_f16 : want_f32) = true;
+        if (want_f16) pbias_f16.assign((size_t) (n_pools * nt), 0xFC00);
+        if (want_f32) pbias_f32.assign((size_t) (n_pools * nt), -INFINITY);
+        for (int64_t t = 0; t < nt; t++)
+        {
+            const int64_t bo_vis = std::min<int64_t>((p0 + t + 1) / r, n_pools);
+            if (want_f16) for (int64_t b = 0; b < bo_vis; b++) pbias_f16[(size_t) (t * n_pools + b)] = 0;
+            if (want_f32) for (int64_t b = 0; b < bo_vis; b++) pbias_f32[(size_t) (t * n_pools + b)] = 0.0f;
+        }
+
+        std::vector<int32_t> tcells((size_t) (r * nt));
+        std::vector<float> tvals((size_t) (r * nt));
+        for (int64_t t = 0; t < nt; t++)
+        {
+            const int64_t q = p0 + t;
+            const int64_t ts = (q + 1) / r * r;
+            const int64_t n_tail = q + 1 - ts;
+            for (int64_t j = 0; j < r; j++)
+            {
+                tcells[(size_t) (t * r + j)] = (int32_t) std::min<int64_t>(ts + j, n_kv - 1);
+                tvals[(size_t) (t * r + j)] = j < n_tail ? 0.0f : -INFINITY;
+            }
+        }
+
+        for (int d = 0; d <= m.n_gpu; d++)
+        {
+            if (live(gr->inp.pool_cells[d]))
+                ggml_backend_tensor_set(gr->inp.pool_cells[d], pcells.data(), 0, pcells.size() * sizeof(int32_t));
+            if (live(gr->inp.pool_bias[d]))
+            {
+                if (gr->inp.pool_bias[d]->type == GGML_TYPE_F16)
+                    ggml_backend_tensor_set(gr->inp.pool_bias[d], pbias_f16.data(), 0, pbias_f16.size() * 2);
+                else
+                    ggml_backend_tensor_set(gr->inp.pool_bias[d], pbias_f32.data(), 0, pbias_f32.size() * 4);
+            }
+            if (live(gr->inp.trail_cells[d]))
+                ggml_backend_tensor_set(gr->inp.trail_cells[d], tcells.data(), 0, tcells.size() * sizeof(int32_t));
+            if (live(gr->inp.trail_vals[d]))
+                ggml_backend_tensor_set(gr->inp.trail_vals[d], tvals.data(), 0, tvals.size() * sizeof(float));
+        }
+    }
 
     // _async, not the synchronous entry point: the graph is already allocated
     // above (which the synchronous one asserts against), and the explicit
@@ -3714,17 +5079,76 @@ TSG_EXPORT int TSGgml_GlmForward(void * handle, const int32_t * tokens, int n_to
 
     const int ub = m->n_ubatch > 0 ? m->n_ubatch : 512;
     int done = 0;
+    // Per-ubatch slice of the queued vision-override rows. Row indices are
+    // relative to THIS call's token array; inside a ubatch they become
+    // row-in-ubatch offsets for the graph's set_rows.
+    std::vector<float> ovr_rows;
+    std::vector<int64_t> ovr_idx;
     while (done < n_tokens)
     {
         const int take = std::min(ub, n_tokens - done);
         const bool last = (done + take) == n_tokens;
-        if (!forward_ubatch(*m, tokens + done, take, last && logits_out != nullptr, logits_out))
+        ovr_rows.clear();
+        ovr_idx.clear();
+        if (!m->embd_ovr.empty())
+        {
+            const int64_t ne = m->hp.n_embd;
+            for (const auto & span : m->embd_ovr)
+            {
+                const int64_t rows = (int64_t) (span.rows.size() / (size_t) ne);
+                for (int64_t r = 0; r < rows; r++)
+                {
+                    const int64_t gi = span.index + r;
+                    if (gi < done || gi >= done + take) continue;
+                    ovr_idx.push_back(gi - done);
+                    ovr_rows.insert(ovr_rows.end(),
+                                    span.rows.begin() + (size_t) (r * ne),
+                                    span.rows.begin() + (size_t) ((r + 1) * ne));
+                }
+            }
+        }
+        if (!forward_ubatch(*m, tokens + done, take, last && logits_out != nullptr, logits_out,
+                            nullptr, false,
+                            ovr_rows.empty() ? nullptr : ovr_rows.data(),
+                            ovr_idx.empty() ? nullptr : ovr_idx.data(),
+                            (int64_t) ovr_idx.size()))
+        {
+            m->embd_ovr.clear();
             return 0;
+        }
         done += take;
     }
+    m->embd_ovr.clear();
     return 1;
 }
 
+/// Queue projected vision-embedding rows to override the token embeddings of
+/// image-placeholder positions in the NEXT TSGgml_GlmForward call. `index` is
+/// the first placeholder's position within that call's token array. Cleared
+/// after the forward (successful or not).
+TSG_EXPORT int TSGgml_GlmQueueVisionRows(void * handle, const float * rows, int n_rows, int index)
+{
+    glm_model * m = (glm_model *) handle;
+    if (!m || !rows || n_rows <= 0 || index < 0) return 0;
+    if (!m->hp.g5n)
+    {
+        fprintf(stderr, "[glm] vision embedding rows are only supported for glm5next\n");
+        return 0;
+    }
+    glm_model::embd_override span;
+    span.index = index;
+    span.rows.assign(rows, rows + (size_t) n_rows * m->hp.n_embd);
+    m->embd_ovr.push_back(std::move(span));
+    return 1;
+}
+
+/// Drop queued vision rows (a cancelled or re-planned prompt).
+TSG_EXPORT void TSGgml_GlmClearVisionRows(void * handle)
+{
+    glm_model * m = (glm_model *) handle;
+    if (m) m->embd_ovr.clear();
+}
+
 /// One decode step for `n` sequences at once (one token each). Declines — by
 /// returning 0 without touching any state — whenever it cannot serve the batch,
 /// leaving the caller on the per-sequence path.
@@ -3758,12 +5182,19 @@ TSG_EXPORT void TSGgml_GlmReset(void * handle)
     glm_model * m = (glm_model *) handle;
     if (!m || !m->active_slot) return;
     m->active_slot->n_past = 0;
+    // glm5next: a new conversation must not inherit the KDA recurrent state.
+    if (m->hp.g5n) slot_clear_recurrent(*m, *m->active_slot);
 }
 
 TSG_EXPORT int TSGgml_GlmRewind(void * handle, int n_past)
 {
     glm_model * m = (glm_model *) handle;
     if (!m || !m->active_slot || n_past < 0 || n_past > m->active_slot->n_past) return 0;
+    // The KDA recurrence cannot be rewound to an earlier position: a cached
+    // prefix is reusable only when the new prompt EXTENDS it (no rewind), and
+    // anything else must restart from zero with a cleared state.
+    if (m->hp.g5n && n_past != m->active_slot->n_past && n_past != 0) return 0;
+    if (m->hp.g5n && n_past == 0) slot_clear_recurrent(*m, *m->active_slot);
     m->active_slot->n_past = n_past;
     return 1;
 }
diff --git a/TensorSharp.GGML.Native/ggml_ops_qwen35_decode.cpp b/TensorSharp.GGML.Native/ggml_ops_qwen35_decode.cpp
index 0615f657..38e8fa88 100644
--- a/TensorSharp.GGML.Native/ggml_ops_qwen35_decode.cpp
+++ b/TensorSharp.GGML.Native/ggml_ops_qwen35_decode.cpp
@@ -987,6 +987,7 @@ namespace
             // ffn (dense)
             ggml_tensor* post_attn_norm_w;
             ggml_tensor* gu_w;
+            ggml_tensor* ffn_gate_w; ggml_tensor* ffn_up_w;
             ggml_tensor* down_w;
             // ffn (MoE)
             ggml_tensor* gate_inp_w;
@@ -1044,7 +1045,15 @@ namespace
             // FFN
             if (d.is_moe == 0)
             {
-                t.gu_w = ggml_new_tensor_2d(ctx, static_cast<ggml_type>(d.gu_type), d.gu_ne0, d.gu_ne1);
+                if (d.gu_w != nullptr)
+                {
+                    t.gu_w = ggml_new_tensor_2d(ctx, static_cast<ggml_type>(d.gu_type), d.gu_ne0, d.gu_ne1);
+                }
+                else
+                {
+                    t.ffn_gate_w = ggml_new_tensor_2d(ctx, static_cast<ggml_type>(d.ffn_gate_type), d.ffn_gate_ne0, d.ffn_gate_ne1);
+                    t.ffn_up_w   = ggml_new_tensor_2d(ctx, static_cast<ggml_type>(d.ffn_up_type),   d.ffn_up_ne0,   d.ffn_up_ne1);
+                }
                 t.down_w = ggml_new_tensor_2d(ctx, static_cast<ggml_type>(d.down_type), d.down_ne0, d.down_ne1);
             }
             else
@@ -1474,8 +1483,18 @@ namespace
             {
                 // Dense SwiGLU over the packed gate/up projection is faster than
                 // splitting it into two Metal matmuls for this quantized model.
-                ggml_tensor* gu = ggml_mul_mat(ctx, t.gu_w, ffn_normed_2d);
-                ggml_tensor* act_2d = ggml_swiglu(ctx, gu);
+                ggml_tensor* act_2d;
+                if (t.gu_w != nullptr)
+                {
+                    act_2d = ggml_swiglu(ctx, ggml_mul_mat(ctx, t.gu_w, ffn_normed_2d));
+                }
+                else
+                {
+                    // Unfused mixed-quant gate/up: two matmuls, same arithmetic.
+                    ggml_tensor* g = ggml_mul_mat(ctx, t.ffn_gate_w, ffn_normed_2d);
+                    ggml_tensor* u = ggml_mul_mat(ctx, t.ffn_up_w, ffn_normed_2d);
+                    act_2d = ggml_mul(ctx, ggml_silu(ctx, g), u);
+                }
                 ggml_tensor* down_mm = ggml_mul_mat(ctx, t.down_w, act_2d);
                 ffn_down = ggml_reshape_1d(ctx, down_mm, H);
                 if (tp_mode) { tp_partial.push_back(down_mm); tp_boundary.push_back(ffn_down); }
@@ -1804,7 +1823,15 @@ namespace
             bind_or_mark(t.post_attn_norm_w, d.post_attn_norm_w, static_cast<std::size_t>(H) * sizeof(float), true);
             if (d.is_moe == 0)
             {
-                bind_or_mark(t.gu_w, d.gu_w, static_cast<std::size_t>(d.gu_bytes), true);
+                if (t.gu_w != nullptr)
+                {
+                    bind_or_mark(t.gu_w, d.gu_w, static_cast<std::size_t>(d.gu_bytes), true);
+                }
+                else
+                {
+                    bind_or_mark(t.ffn_gate_w, d.ffn_gate_w, static_cast<std::size_t>(d.ffn_gate_bytes), true);
+                    bind_or_mark(t.ffn_up_w,   d.ffn_up_w,   static_cast<std::size_t>(d.ffn_up_bytes),   true);
+                }
                 bind_or_mark(t.down_w, d.down_w, static_cast<std::size_t>(d.down_bytes), true);
             }
             else
@@ -2499,6 +2526,7 @@ namespace
             ggml_tensor* conv_state_in; ggml_tensor* delta_state_in;
             ggml_tensor* conv_state_out; ggml_tensor* delta_state_out;
             ggml_tensor* gu_w; ggml_tensor* down_w;
+            ggml_tensor* ffn_gate_w; ggml_tensor* ffn_up_w;
             ggml_tensor* gate_inp_w; ggml_tensor* gate_exps; ggml_tensor* up_exps; ggml_tensor* down_exps;
             ggml_tensor* shexp_gate_w; ggml_tensor* shexp_up_w; ggml_tensor* shexp_down_w; ggml_tensor* shexp_gate_inp_w;
         };
@@ -2542,7 +2570,15 @@ namespace
             }
             if (d.is_moe == 0)
             {
-                t.gu_w = ggml_new_tensor_2d(ctx, static_cast<ggml_type>(d.gu_type), d.gu_ne0, d.gu_ne1);
+                if (d.gu_w != nullptr)
+                {
+                    t.gu_w = ggml_new_tensor_2d(ctx, static_cast<ggml_type>(d.gu_type), d.gu_ne0, d.gu_ne1);
+                }
+                else
+                {
+                    t.ffn_gate_w = ggml_new_tensor_2d(ctx, static_cast<ggml_type>(d.ffn_gate_type), d.ffn_gate_ne0, d.ffn_gate_ne1);
+                    t.ffn_up_w   = ggml_new_tensor_2d(ctx, static_cast<ggml_type>(d.ffn_up_type),   d.ffn_up_ne0,   d.ffn_up_ne1);
+                }
                 t.down_w = ggml_new_tensor_2d(ctx, static_cast<ggml_type>(d.down_type), d.down_ne0, d.down_ne1);
             }
             else
@@ -2713,9 +2749,21 @@ namespace
             if (d.is_moe == 0)
             {
                 const std::int64_t ffDense = d.ff_dense;
-                ggml_tensor* gu = ggml_mul_mat(ctx, t.gu_w, ffn_normed); // [2*ffDense, T]
-                ggml_tensor* g_part = ggml_cont(ctx, ggml_view_2d(ctx, gu, ffDense, T, gu->nb[1], 0));
-                ggml_tensor* u_part = ggml_cont(ctx, ggml_view_2d(ctx, gu, ffDense, T, gu->nb[1], static_cast<std::size_t>(ffDense) * sizeof(float)));
+                ggml_tensor* g_part;
+                ggml_tensor* u_part;
+                if (t.gu_w != nullptr)
+                {
+                    ggml_tensor* gu = ggml_mul_mat(ctx, t.gu_w, ffn_normed); // [2*ffDense, T]
+                    g_part = ggml_cont(ctx, ggml_view_2d(ctx, gu, ffDense, T, gu->nb[1], 0));
+                    u_part = ggml_cont(ctx, ggml_view_2d(ctx, gu, ffDense, T, gu->nb[1], static_cast<std::size_t>(ffDense) * sizeof(float)));
+                }
+                else
+                {
+                    // Unfused mixed-quant gate/up: two matmuls, and the halves are
+                    // already dense so the two conts above are not needed either.
+                    g_part = ggml_mul_mat(ctx, t.ffn_gate_w, ffn_normed);
+                    u_part = ggml_mul_mat(ctx, t.ffn_up_w, ffn_normed);
+                }
                 ggml_tensor* act = ggml_mul(ctx, ggml_silu(ctx, g_part), u_part); // [ffDense, T]
                 ffn_out = ggml_mul_mat(ctx, t.down_w, act); // [H, T]
             }
@@ -2823,7 +2871,15 @@ namespace
             bind_or_mark(t.post_attn_norm_w, d.post_attn_norm_w, static_cast<std::size_t>(H) * sizeof(float), true);
             if (d.is_moe == 0)
             {
-                bind_or_mark(t.gu_w, d.gu_w, static_cast<std::size_t>(d.gu_bytes), true);
+                if (t.gu_w != nullptr)
+                {
+                    bind_or_mark(t.gu_w, d.gu_w, static_cast<std::size_t>(d.gu_bytes), true);
+                }
+                else
+                {
+                    bind_or_mark(t.ffn_gate_w, d.ffn_gate_w, static_cast<std::size_t>(d.ffn_gate_bytes), true);
+                    bind_or_mark(t.ffn_up_w,   d.ffn_up_w,   static_cast<std::size_t>(d.ffn_up_bytes),   true);
+                }
                 bind_or_mark(t.down_w, d.down_w, static_cast<std::size_t>(d.down_bytes), true);
             }
             else
diff --git a/TensorSharp.GGML.Native/ggml_ops_qwen35_verify.cpp b/TensorSharp.GGML.Native/ggml_ops_qwen35_verify.cpp
index 7b6e1947..a27ac626 100644
--- a/TensorSharp.GGML.Native/ggml_ops_qwen35_verify.cpp
+++ b/TensorSharp.GGML.Native/ggml_ops_qwen35_verify.cpp
@@ -74,7 +74,28 @@ namespace
         ggml_tensor* mask_t = nullptr;
         ggml_tensor* logits_out = nullptr;
         ggml_tensor* normed_out = nullptr;
+        /// DFlash residual taps: one [H, N] block per requested layer, holding the
+        /// residual ENTERING it. Empty unless the caller asked for them.
+        std::vector<ggml_tensor*> capture_out;
+        int capture_count = 0;
         std::vector<ggml_tensor*> conv_in, delta_in, conv_out, delta_out;
+        /// Per-token recurrent-state snapshots, one entry per RECURRENT layer.
+        /// conv_snaps is [convDim * conv_dim, K] and delta_snaps [D, K], slot s
+        /// holding the state as it stood s tokens before the end of the batch.
+        /// They exist so a partially-rejected verify does not have to restore a
+        /// pre-verify snapshot and re-forward the accepted prefix: the state it
+        /// would have recomputed is already sitting in slot (N-1-accepted).
+        std::vector<ggml_tensor*> conv_snaps, delta_snaps;
+        /// Per (recurrent layer, slot) views of the two above, shaped exactly like
+        /// the live conv_state_in / delta_state_in so one slot can be committed with
+        /// a device-to-device tensor copy. Index [i * n_snapshots + slot].
+        std::vector<ggml_tensor*> conv_snap_slots, delta_snap_slots;
+        int n_snapshots = 0;
+        /// The post-window state was NOT downloaded; the caller commits it (or one
+        /// snapshot slot) into the live slices on the device instead. True for every
+        /// step of a speculative session, single-row plain steps included - which is
+        /// the point: the 151 MB state then never crosses PCIe at all.
+        bool deferred_state = false;
         std::size_t buffer_bytes = 0;
         std::uint64_t lru = 0;
         void reset()
@@ -84,6 +105,10 @@ namespace
             graph = nullptr; valid = false;
             hidden_t = pos_t = kv_index = mask_t = logits_out = normed_out = nullptr;
             conv_in.clear(); delta_in.clear(); conv_out.clear(); delta_out.clear();
+            conv_snaps.clear(); delta_snaps.clear();
+            conv_snap_slots.clear(); delta_snap_slots.clear(); n_snapshots = 0;
+            deferred_state = false;
+            capture_out.clear(); capture_count = 0;
             n = window = num_layers = out_vocab = n_logits = 0; has_normed = false; sig = nullptr;
             buffer_bytes = 0;
         }
@@ -91,6 +116,14 @@ namespace
     Q35VerifyCache g_q35vc[16];
     std::uint64_t g_q35vc_clock = 0;
 
+    // The entry whose snapshots are live, i.e. the one the most recent verify
+    // computed or replayed. TSGgml_Qwen35FetchStateSnapshot reads from it, and it
+    // is cleared whenever a call runs without snapshots so a stale fetch cannot
+    // silently return the wrong sequence's state.
+    Q35VerifyCache* g_q35vc_last_snap = nullptr;
+    std::size_t g_q35vc_snap_conv_bytes = 0;
+    std::size_t g_q35vc_snap_delta_bytes = 0;
+
     // Total-VRAM budget for the resident persist verify graphs. Each entry is a
     // whole-model graph in its own alloc_ctx buffer (own slots, needed for CUDA-graph
     // capture), sized ~0.2 GB (N=1) to ~0.8 GB (N=maxDraft+1). Without a cap, a
@@ -246,8 +279,13 @@ namespace
         const void* lm_head_data, int lm_head_type, std::int64_t lm_head_ne0, std::int64_t lm_head_ne1, std::int64_t lm_head_bytes,
         const void* final_norm_data, void* normed_out, int n_logit_rows,
         const std::int32_t* mrope_pos, const std::int32_t* mrope_sections,
-        int tp_degree, void** tp_plan_out)
+        int tp_degree, void** tp_plan_out,
+        float* capture_data, const int* capture_layers, int capture_count,
+        int state_snapshots, int* state_snapshots_used, int device_state_current,
+        int defer_state_download)
     {
+        if (state_snapshots_used != nullptr)
+            *state_snapshots_used = 1;
         if (!ensure_backend())
             return 0;
         if (layers == nullptr || num_layers <= 0 || hidden_data == nullptr || num_tokens < 1)
@@ -338,6 +376,18 @@ namespace
         bool any_cpu_moe = false;
         for (int l = 0; l < num_layers; l++)
             if (layers[l].is_moe != 0 && layers[l].cpu_moe != 0) { any_cpu_moe = true; break; }
+        // The DFlash drafter's encoder reads the residual entering a handful of
+        // trunk layers. Tapping them here keeps speculation on the fused trunk
+        // instead of forcing the op-by-op loop just to observe them.
+        const int cap_count = (capture_data != nullptr && capture_layers != nullptr && capture_count > 0)
+            ? capture_count : 0;
+        // Per-token recurrent-state snapshots for a rollback-free partial accept.
+        // Only on the persist path, because the fetch reads the graph's own output
+        // tensors after the fact and only a persisted graph still owns them; only in
+        // host state mode, because resident mode updates the state in place; and only
+        // when the caller asked for at least two (one is what the plain path already
+        // keeps). TS_Q35_VERIFY_SNAPSHOTS=0 forces the old snapshot/re-forward path.
+        static const bool fv_snapshots_cfg = []{ const char* e = std::getenv("TS_Q35_VERIFY_SNAPSHOTS"); return e == nullptr || e[0] != '0'; }();
         static const bool fv_persist_cfg = []{ const char* e = std::getenv("TS_Q35_VERIFY_PERSIST"); return e == nullptr || e[0] != '0'; }();
         // Multimodal MRoPE: per-axis positions (T/H/W/E axis-concatenated, [4N] I32)
         // route the attention RoPE through ggml_rope_multi (interleaved MRoPE, the
@@ -361,7 +411,17 @@ namespace
         // TP always takes the non-persist path: the plan executes after this
         // call returns, so the context is parked in g_q35v_tp instead, and a
         // prefill-sized graph never repeats its exact shape anyway.
-        const bool fv_persist = fv_persist_cfg && (n_logits >= N) && !use_mrope && num_layers > 1 && !tp_mode && !any_cpu_moe;
+        // num_layers == 1 is the MTP draft block. It was pinned to the non-persist
+        // path because its captured graph used to deadlock the stream on the third
+        // replay; TS_Q35_MTP_DRAFT_PERSIST=1 re-tests that on the current ggml,
+        // because rebuilding a graph per draft call costs ~3.7 ms of the 6.2 ms a
+        // draft step takes.
+        static const bool mtp_draft_persist = []{
+            const char* e = std::getenv("TS_Q35_MTP_DRAFT_PERSIST");
+            return e != nullptr && e[0] == '1';
+        }();
+        const bool fv_persist = fv_persist_cfg && (n_logits >= N) && !use_mrope
+            && (num_layers > 1 || mtp_draft_persist) && !tp_mode && !any_cpu_moe;
 
         const std::size_t convStateBytes = static_cast<std::size_t>(convDim) * conv_dim * sizeof(float);
         const std::size_t deltaStateBytes = static_cast<std::size_t>(head_k_dim) * head_v_dim * num_v_heads * sizeof(float);
@@ -402,6 +462,20 @@ namespace
             }
         }
 
+        // Deferring the state download needs a graph whose output tensors outlive
+        // the call, which is exactly the persist path; resident mode has nothing to
+        // defer (it updates the state in place).
+        const bool defer_state = fv_snapshots_cfg && defer_state_download != 0 && fv_persist && !resident_state;
+        const int n_snap = (defer_state && state_snapshots > 1 && state_snapshots <= N)
+            ? state_snapshots : 1;
+        // What the caller has to do next, and getting it wrong silently decodes from
+        // a stale recurrent state:
+        //    0 -> deferred with no snapshots; commit slot -1 (the post-window state)
+        //    1 -> downloaded, as it always used to be; nothing to do
+        //   >1 -> deferred with N snapshots; commit slot (N-1-accepted)
+        if (state_snapshots_used != nullptr)
+            *state_snapshots_used = defer_state ? (n_snap > 1 ? n_snap : 0) : 1;
+
         // ===== Persist reuse fast-path: upload the per-call inputs + replay =====
         if (fv_persist)
         {
@@ -410,7 +484,10 @@ namespace
                 if (!c.valid || c.n != N || c.window != window || c.sig != sig ||
                     c.num_layers != num_layers || c.out_vocab != vocab_size ||
                     c.n_logits != n_logits ||
-                    c.has_normed != (normed_out != nullptr))
+                    c.has_normed != (normed_out != nullptr) ||
+                    c.capture_count != cap_count ||
+                    c.n_snapshots != n_snap ||
+                    c.deferred_state != defer_state)
                     continue;
                 // llama.cpp pattern (llama-context.cpp): before re-setting the inputs of
                 // a REUSED graph we must fully synchronize, else we overwrite input
@@ -429,7 +506,7 @@ namespace
                 ggml_backend_tensor_set(c.mask_t, mk.data(), 0, mk.size() * sizeof(ggml_fp16_t));
                 // Host mode uploads the per-call GDN state; resident keeps it device-
                 // resident (cacheable, in-place), so no upload/download here.
-                if (!resident_state)
+                if (!resident_state && device_state_current == 0)
                 {
                     int gi = 0;
                     for (int l = 0; l < num_layers; l++)
@@ -440,8 +517,15 @@ namespace
                         gi++;
                     }
                 }
-                if (ggml_backend_graph_compute(g_backend, c.graph) != GGML_STATUS_SUCCESS) { c.reset(); break; }
-                if (!resident_state)
+                // device_state_current: the live state slices already hold what the
+                // caller wants (a previous verify's snapshot was committed into them
+                // on the device), so the ~300 MB round trip this upload and the
+                // matching download used to cost per step is simply not paid.
+                // Profiled (a plain compute unless TS_GGML_NODE_PROFILE is set): this
+                // is the hot path - every warm speculative verify replays here - and
+                // it was the one graph the node profiler could not see.
+                if (tsg::graph_compute_profiled(g_backend, c.graph, "qwen35 verify replay") != GGML_STATUS_SUCCESS) { c.reset(); break; }
+                if (!resident_state && !c.deferred_state)
                 {
                     int gi = 0;
                     for (int l = 0; l < num_layers; l++)
@@ -452,8 +536,19 @@ namespace
                         gi++;
                     }
                 }
+                // Deferred: the state stays on the device until the caller commits it.
+                g_q35vc_last_snap = c.deferred_state ? &c : nullptr;
+                g_q35vc_snap_conv_bytes = convStateBytes;
+                g_q35vc_snap_delta_bytes = deltaStateBytes;
                 if (normed_out != nullptr && c.normed_out != nullptr)
                     finalize_compute_with_download(c.normed_out, normed_out, static_cast<std::size_t>(H) * N * sizeof(float));
+                for (int ci = 0; ci < c.capture_count; ci++)
+                {
+                    if (c.capture_out[ci] == nullptr) continue;
+                    finalize_compute_with_download(c.capture_out[ci],
+                        capture_data + static_cast<std::size_t>(ci) * H * N,
+                        static_cast<std::size_t>(H) * N * sizeof(float));
+                }
                 finalize_compute_with_download(c.logits_out, logits_data, static_cast<std::size_t>(vocab_size) * n_logits * sizeof(float));
                 host_read_barrier();
                 c.lru = ++g_q35vc_clock;
@@ -517,6 +612,11 @@ namespace
             ggml_tensor* conv_state_out; ggml_tensor* delta_state_out;
             // ffn
             ggml_tensor* post_attn_norm_w; ggml_tensor* gu_w; ggml_tensor* down_w;
+            ggml_tensor* ffn_gate_w; ggml_tensor* ffn_up_w;
+            ggml_tensor* conv_snaps = nullptr;   // [convDim * conv_dim, n_snap]
+            ggml_tensor* delta_snaps = nullptr;  // [D, n_snap] (a view of the GDN output)
+            std::vector<ggml_tensor*> conv_snap_slots;   // per slot, shaped like conv_state_in
+            std::vector<ggml_tensor*> delta_snap_slots;  // per slot, shaped like delta_state_in
             ggml_tensor* gate_inp_w; ggml_tensor* gate_exps; ggml_tensor* up_exps; ggml_tensor* down_exps;
             ggml_tensor* shexp_gate_w; ggml_tensor* shexp_up_w; ggml_tensor* shexp_down_w; ggml_tensor* shexp_gate_inp_w;
         };
@@ -640,7 +740,15 @@ namespace
             }
             if (d.is_moe == 0)
             {
-                t.gu_w = ggml_new_tensor_2d(ctx, static_cast<ggml_type>(d.gu_type), d.gu_ne0, d.gu_ne1);
+                if (d.gu_w != nullptr)
+                {
+                    t.gu_w = ggml_new_tensor_2d(ctx, static_cast<ggml_type>(d.gu_type), d.gu_ne0, d.gu_ne1);
+                }
+                else
+                {
+                    t.ffn_gate_w = ggml_new_tensor_2d(ctx, static_cast<ggml_type>(d.ffn_gate_type), d.ffn_gate_ne0, d.ffn_gate_ne1);
+                    t.ffn_up_w   = ggml_new_tensor_2d(ctx, static_cast<ggml_type>(d.ffn_up_type),   d.ffn_up_ne0,   d.ffn_up_ne1);
+                }
                 t.down_w = ggml_new_tensor_2d(ctx, static_cast<ggml_type>(d.down_type), d.down_ne0, d.down_ne1);
             }
             else
@@ -735,15 +843,35 @@ namespace
         const std::size_t graph_size = static_cast<std::size_t>(num_layers) * (260 + 2 * num_kv_heads) + 1024;
         ggml_cgraph* graph = ggml_new_graph_custom(ctx, graph_size, false);
         ggml_tensor* hidden = hidden_t;
-        // llama.cpp feeds strided head/token views directly to Metal's norm, CPY,
-        // and gated-delta-net kernels. Keep the established materialized layout on
-        // CUDA/Vulkan, whose capture/replay paths were validated with those CONTs.
-        const bool metal_strided_views = g_backend_type == BACKEND_TYPE_METAL;
+        // llama.cpp feeds strided head/token views straight to the norm, CPY and
+        // gated-delta-net kernels rather than materializing them, and ggml-cuda
+        // consumes the strides just as ggml-metal does. Doing the same here removes
+        // ~170 CONT nodes and their copies from a 64-layer graph. Metal was already
+        // on this path; CUDA/Vulkan joined it after the outputs were checked
+        // byte-identical. TS_Q35_VERIFY_STRIDED_VIEWS=0 restores the materializing
+        // layout if a backend ever disagrees.
+        static const bool strided_views_cfg = []{
+            const char* e = std::getenv("TS_Q35_VERIFY_STRIDED_VIEWS");
+            return e == nullptr || e[0] != '0';
+        }();
+        const bool metal_strided_views = strided_views_cfg
+            && (g_backend_type == BACKEND_TYPE_METAL || g_backend_type == BACKEND_TYPE_CUDA);
+        std::vector<ggml_tensor*> capture_out(cap_count, nullptr);
         for (int l = 0; l < num_layers; l++)
         {
             const TSGgmlQwen35LayerDesc& d = layers[l];
             LayerTensors& t = lt[l];
 
+            // The residual ENTERING layer l, copied out before the block runs -
+            // exactly res->t_layer_inp[l] in llama.cpp's DFlash capture.
+            for (int ci = 0; ci < cap_count; ci++)
+            {
+                if (capture_layers[ci] != l) continue;
+                ggml_tensor* dst = ggml_new_tensor_2d(ctx, GGML_TYPE_F32, H, N);
+                capture_out[ci] = ggml_cpy(ctx, hidden, dst);
+                ggml_set_output(capture_out[ci]);
+            }
+
             ggml_tensor* normed = ggml_mul(ctx, ggml_rms_norm(ctx, hidden, eps), t.attn_norm_w); // [H, N]
             ggml_tensor* block_out;
 
@@ -944,6 +1072,27 @@ namespace
                 // out slice.
                 t.conv_state_out = ggml_cpy(ctx, new_conv, resident_state ? t.conv_state_in : t.conv_state_out);
 
+                if (n_snap > 1)
+                {
+                    // The conv state after row r is simply rows [r+1, r+1+convDim) of
+                    // conv_input - the same window new_conv takes at r = N-1 - so every
+                    // snapshot is already in a tensor the graph built. Slot s is s
+                    // tokens back from the end, matching the GDN op's slot order.
+                    t.conv_snaps = ggml_new_tensor_2d(ctx, GGML_TYPE_F32,
+                        static_cast<std::int64_t>(convDim) * conv_dim, n_snap);
+                    ggml_set_output(t.conv_snaps);
+                    for (int ssl = 0; ssl < n_snap; ssl++)
+                    {
+                        ggml_tensor* src = ggml_view_2d(ctx, conv_input, convDim, conv_dim,
+                            conv_input->nb[1], static_cast<std::size_t>(N - ssl) * conv_input->nb[0]);
+                        ggml_tensor* dst = ggml_view_2d(ctx, t.conv_snaps, convDim, conv_dim,
+                            static_cast<std::size_t>(convDim) * sizeof(float),
+                            static_cast<std::size_t>(ssl) * convDim * conv_dim * sizeof(float));
+                        ggml_build_forward_expand(graph, ggml_cpy(ctx, src, dst));
+                        t.conv_snap_slots.push_back(dst);
+                    }
+                }
+
                 // l2-norm over head_k_dim. q/k keep num_k_heads heads: the fused
                 // gated_delta_net kernel broadcasts each v-head h to k-head (h % num_k_heads)
                 // internally (kernel iq1 = h_idx % neqk1, neqk1 = q->ne[1]), so pre-tiling
@@ -1004,7 +1153,10 @@ namespace
                 // token outputs (rows [0,N)) + ONLY the FINAL state snapshot (we roll
                 // back via host snapshot/re-forward, not the per-prefix snapshots, so
                 // requesting K=N would waste ~19 MB/layer of VRAM on unused states).
-                ggml_tensor* gdn = ggml_gated_delta_net(ctx, q4, k4, v4, g4, beta4, state4, 1);
+                // n_snap slots, most-recent first: slot 0 is the post-window state
+                // (all the plain path needs) and slot s the state s tokens earlier,
+                // which is what a partial accept rolls back to.
+                ggml_tensor* gdn = ggml_gated_delta_net(ctx, q4, k4, v4, g4, beta4, state4, n_snap);
                 // Per-token outputs occupy the first N rows ([S_v*H] each).
                 ggml_tensor* gdn_out = ggml_view_2d(ctx, gdn, value_dim, N, ggml_row_size(gdn->type, value_dim), 0);
                 // Final state snapshot (slot 0, most-recent) at offset N * (S_v*H).
@@ -1018,6 +1170,26 @@ namespace
                 // slice (downloaded after compute) — NOT in-place, so the persist
                 // replay's captured CUDA graph stays valid across re-uploads.
                 t.delta_state_out = ggml_cpy(ctx, new_state, resident_state ? state4 : t.delta_state_out);
+                if (n_snap > 1)
+                {
+                    const std::int64_t d_elems =
+                        static_cast<std::int64_t>(head_k_dim) * head_v_dim * num_v_heads;
+                    t.delta_snaps = ggml_view_2d(ctx, gdn, d_elems, n_snap,
+                        ggml_row_size(gdn->type, d_elems),
+                        ggml_row_size(gdn->type, value_dim) * static_cast<std::size_t>(N));
+                    ggml_set_output(t.delta_snaps);
+                    for (int ssl = 0; ssl < n_snap; ssl++)
+                    {
+                        // Shaped like delta_state_in so committing a slot is one
+                        // device-to-device tensor copy rather than a host round trip.
+                        t.delta_snap_slots.push_back(ggml_view_3d(ctx, gdn,
+                            head_k_dim, head_v_dim, num_v_heads,
+                            ggml_row_size(gdn->type, head_k_dim),
+                            ggml_row_size(gdn->type, head_k_dim * head_v_dim),
+                            ggml_row_size(gdn->type, value_dim) * static_cast<std::size_t>(N)
+                                + ggml_row_size(gdn->type, d_elems) * static_cast<std::size_t>(ssl)));
+                    }
+                }
 
                 // gated RMSNorm with z, per token: rms_norm(out) * ssm_norm * silu(z).
                 ggml_tensor* out_2d = ggml_reshape_2d(ctx,
@@ -1044,8 +1216,18 @@ namespace
             ggml_tensor* ffn_out;
             if (d.is_moe == 0)
             {
-                ggml_tensor* gu = ggml_mul_mat(ctx, t.gu_w, ffn_normed);
-                ggml_tensor* act = ggml_swiglu(ctx, gu);
+                ggml_tensor* act;
+                if (t.gu_w != nullptr)
+                {
+                    act = ggml_swiglu(ctx, ggml_mul_mat(ctx, t.gu_w, ffn_normed));
+                }
+                else
+                {
+                    // Unfused mixed-quant gate/up: two matmuls, same arithmetic.
+                    ggml_tensor* g = ggml_mul_mat(ctx, t.ffn_gate_w, ffn_normed);
+                    ggml_tensor* u = ggml_mul_mat(ctx, t.ffn_up_w, ffn_normed);
+                    act = ggml_mul(ctx, ggml_silu(ctx, g), u);
+                }
                 ffn_out = ggml_mul_mat(ctx, t.down_w, act); // [H, N]
                 if (tp_mode) { tp_partial.push_back(ffn_out); tp_boundary.push_back(ffn_out); }
             }
@@ -1257,9 +1439,22 @@ namespace
         ggml_tensor* logits_out_t = ggml_new_tensor_2d(ctx, GGML_TYPE_F32, vocab_size, n_logits);
         ggml_tensor* logits_cpy = ggml_cpy(ctx, logits, logits_out_t);
         ggml_set_output(logits_cpy);
+        // A capture whose layer never ran would leave an unwritten output; refuse
+        // rather than hand the drafter uninitialised residuals.
+        for (int ci = 0; ci < cap_count; ci++)
+        {
+            if (capture_out[ci] == nullptr)
+            {
+                set_last_error("Qwen3.5 model verify: a DFlash capture layer is outside the trunk.");
+                if (fv_persist) ggml_free(ctx);
+                return 0;
+            }
+        }
 
         if (normed_cpy != nullptr)
             ggml_build_forward_expand(graph, normed_cpy);
+        for (int ci = 0; ci < cap_count; ci++)
+            ggml_build_forward_expand(graph, capture_out[ci]);
         ggml_build_forward_expand(graph, logits_cpy);
 
         // Turn the recorded seams into node cut points (see
@@ -1304,7 +1499,15 @@ namespace
             bind_or_mark(t.post_attn_norm_w, d.post_attn_norm_w, static_cast<std::size_t>(H) * sizeof(float), true);
             if (d.is_moe == 0)
             {
-                bind_or_mark(t.gu_w, d.gu_w, static_cast<std::size_t>(d.gu_bytes), true);
+                if (t.gu_w != nullptr)
+                {
+                    bind_or_mark(t.gu_w, d.gu_w, static_cast<std::size_t>(d.gu_bytes), true);
+                }
+                else
+                {
+                    bind_or_mark(t.ffn_gate_w, d.ffn_gate_w, static_cast<std::size_t>(d.ffn_gate_bytes), true);
+                    bind_or_mark(t.ffn_up_w,   d.ffn_up_w,   static_cast<std::size_t>(d.ffn_up_bytes),   true);
+                }
                 bind_or_mark(t.down_w, d.down_w, static_cast<std::size_t>(d.down_bytes), true);
             }
             else
@@ -1427,10 +1630,12 @@ namespace
         if (uses_dynamic_kv_index)
             ggml_backend_tensor_set(kv_index, kv_index_data.data(), 0, static_cast<std::size_t>(N) * sizeof(std::int64_t));
         ggml_backend_tensor_set(attn_mask, attn_mask_data.data(), 0, attn_mask_data.size() * sizeof(ggml_fp16_t));
-        if (!resident_state)
+        if (!resident_state && device_state_current == 0)
         {
             // Host mode: upload the per-call GDN state. Resident mode skips this — the
-            // state is device-resident (cacheable), seeded only on invalidation.
+            // state is device-resident (cacheable), seeded only on invalidation; so
+            // does a caller whose last snapshot commit already left the live slices
+            // correct on the device.
             for (int l = 0; l < num_layers; l++)
             {
                 if (layers[l].is_recurrent == 0) continue;
@@ -1469,6 +1674,10 @@ namespace
             if (normed_out != nullptr && normed_out_t != nullptr)
                 pending.plan.extra_out.push_back({ normed_cpy, normed_out,
                     static_cast<std::size_t>(H) * N * sizeof(float) });
+            for (int ci = 0; ci < cap_count; ci++)
+                pending.plan.extra_out.push_back({ capture_out[ci],
+                    capture_data + static_cast<std::size_t>(ci) * H * N,
+                    static_cast<std::size_t>(H) * N * sizeof(float) });
             for (int l = 0; l < num_layers; l++)
             {
                 const TSGgmlQwen35LayerDesc& d = layers[l];
@@ -1533,8 +1742,10 @@ namespace
         }
 
         // Download the post-window GDN state (per recurrent layer) + outputs. Resident
-        // mode skips the state download (it stays device-resident, updated in-place).
-        if (!resident_state)
+        // mode skips the state download (it stays device-resident, updated in-place);
+        // so does snapshot mode, where the caller fetches exactly one slot once it
+        // knows how much of the draft the sampler accepted.
+        if (!resident_state && !defer_state)
         {
             for (int l = 0; l < num_layers; l++)
             {
@@ -1548,6 +1759,10 @@ namespace
         }
         if (normed_out != nullptr && normed_out_t != nullptr)
             finalize_compute_with_download(normed_out_t, normed_out, static_cast<std::size_t>(H) * N * sizeof(float));
+        for (int ci = 0; ci < cap_count; ci++)
+            finalize_compute_with_download(capture_out[ci],
+                capture_data + static_cast<std::size_t>(ci) * H * N,
+                static_cast<std::size_t>(H) * N * sizeof(float));
         finalize_compute_with_download(logits_out_t, logits_data, static_cast<std::size_t>(vocab_size) * n_logits * sizeof(float));
         host_read_barrier();
 
@@ -1569,6 +1784,26 @@ namespace
             slot->num_layers = num_layers; slot->out_vocab = vocab_size;
             slot->n_logits = n_logits;
             slot->has_normed = (normed_out != nullptr);
+            slot->capture_count = cap_count;
+            slot->capture_out = capture_out;
+            slot->n_snapshots = n_snap;
+            slot->deferred_state = defer_state;
+            slot->conv_snaps.clear(); slot->delta_snaps.clear();
+            slot->conv_snap_slots.clear(); slot->delta_snap_slots.clear();
+            if (n_snap > 1)
+            {
+                for (int l = 0; l < num_layers; l++)
+                {
+                    if (layers[l].is_recurrent == 0) continue;
+                    slot->conv_snaps.push_back(lt[l].conv_snaps);
+                    slot->delta_snaps.push_back(lt[l].delta_snaps);
+                    for (int ssl = 0; ssl < n_snap; ssl++)
+                    {
+                        slot->conv_snap_slots.push_back(lt[l].conv_snap_slots[ssl]);
+                        slot->delta_snap_slots.push_back(lt[l].delta_snap_slots[ssl]);
+                    }
+                }
+            }
             slot->ctx = ctx; slot->buffer = persist_buf; slot->graph = graph;
             slot->buffer_bytes = persist_buf != nullptr ? ggml_backend_buffer_get_size(persist_buf) : 0;
             slot->hidden_t = hidden_t; slot->pos_t = pos_tensor;
@@ -1584,16 +1819,25 @@ namespace
                 slot->delta_out.push_back(lt[l].delta_state_out);
             }
             slot->lru = ++g_q35vc_clock;
+            g_q35vc_last_snap = defer_state ? slot : nullptr;
+            g_q35vc_snap_conv_bytes = convStateBytes;
+            g_q35vc_snap_delta_bytes = deltaStateBytes;
             // Bound the resident persist-graph total: evict LRU entries (never the one
             // just built) so the cache never re-overcommits VRAM across many N shapes.
             q35_verify_cache_evict_to_budget(0, slot);
         }
+        else if (defer_state)
+        {
+            // A deferring call that did not persist cannot be committed from.
+            g_q35vc_last_snap = nullptr;
+        }
         clear_last_error();
         return 1;
     }
 
     void reset_qwen35_verify_cache()
     {
+        g_q35vc_last_snap = nullptr;
         for (auto& c : g_q35vc) c.reset();
     }
 }
@@ -1611,7 +1855,10 @@ TSG_EXPORT int TSGgml_Qwen35ModelVerify(
     const void* lm_head_data, int lm_head_type, std::int64_t lm_head_ne0, std::int64_t lm_head_ne1, std::int64_t lm_head_bytes,
     const void* final_norm_data, void* normed_out, int n_logit_rows,
     const std::int32_t* mrope_pos, const std::int32_t* mrope_sections,
-    int tp_degree, void** tp_plan_out)
+    int tp_degree, void** tp_plan_out,
+    float* capture_data, const int* capture_layers, int capture_count,
+    int state_snapshots, int* state_snapshots_used, int device_state_current,
+    int defer_state_download)
 {
     try
     {
@@ -1626,13 +1873,188 @@ TSG_EXPORT int TSGgml_Qwen35ModelVerify(
             logits_data, vocab_size,
             lm_head_data, lm_head_type, lm_head_ne0, lm_head_ne1, lm_head_bytes,
             final_norm_data, normed_out, n_logit_rows, mrope_pos, mrope_sections,
-            tp_degree, tp_plan_out);
+            tp_degree, tp_plan_out, capture_data, capture_layers, capture_count,
+            state_snapshots, state_snapshots_used, device_state_current,
+            defer_state_download);
         return r;
     }
     catch (const std::exception& ex) { set_last_error(ex.what()); return 0; }
     catch (...) { set_last_error("Unknown error in Qwen3.5 model verify."); return 0; }
 }
 
+// Fetch ONE per-token recurrent-state snapshot from the verify that just ran.
+//
+// This is the whole point of the snapshots: a partially-rejected draft used to
+// restore a pre-verify copy of the recurrent state and re-forward the accepted
+// prefix through the entire trunk - a second whole-model forward, plus the state
+// crossing PCIe twice - because the state after row m simply did not exist
+// anywhere. It does now: the gated-delta-net op emits it, and the conv state after
+// row m is a window of a tensor the graph already built. `slot` counts BACK from
+// the end of the batch, so slot 0 is the post-window state and slot (N-1-accepted)
+// is the one a partial accept wants.
+//
+// Returns 0 when there is nothing to fetch (no snapshotting verify has run, a
+// non-persist call intervened, or the slot is out of range), and the caller keeps
+// the old restore-and-re-forward path.
+TSG_EXPORT int TSGgml_Qwen35FetchStateSnapshot(
+    int slot, void** conv_out_arr, void** delta_out_arr, int num_recurrent_layers)
+{
+    try
+    {
+        Q35VerifyCache* c = g_q35vc_last_snap;
+        if (c == nullptr || !c->valid || c->n_snapshots <= 1)
+            return 0;
+        if (slot < 0 || slot >= c->n_snapshots)
+            return 0;
+        if (conv_out_arr == nullptr || delta_out_arr == nullptr)
+            return 0;
+        if (static_cast<int>(c->conv_snaps.size()) != num_recurrent_layers
+            || static_cast<int>(c->delta_snaps.size()) != num_recurrent_layers)
+        {
+            set_last_error("Qwen3.5 state snapshot: recurrent layer count mismatch.");
+            return 0;
+        }
+
+        host_read_barrier();
+        for (int i = 0; i < num_recurrent_layers; i++)
+        {
+            if (c->conv_snaps[i] == nullptr || c->delta_snaps[i] == nullptr)
+                return 0;
+            if (conv_out_arr[i] != nullptr)
+            {
+                ggml_backend_tensor_get(c->conv_snaps[i], conv_out_arr[i],
+                    static_cast<std::size_t>(slot) * g_q35vc_snap_conv_bytes,
+                    g_q35vc_snap_conv_bytes);
+            }
+            if (delta_out_arr[i] != nullptr)
+            {
+                ggml_backend_tensor_get(c->delta_snaps[i], delta_out_arr[i],
+                    static_cast<std::size_t>(slot) * g_q35vc_snap_delta_bytes,
+                    g_q35vc_snap_delta_bytes);
+            }
+        }
+        host_read_barrier();
+        clear_last_error();
+        return 1;
+    }
+    catch (const std::exception& ex) { set_last_error(ex.what()); return 0; }
+    catch (...) { set_last_error("Unknown error in Qwen3.5 state snapshot fetch."); return 0; }
+}
+
+namespace
+{
+    /// ggml_backend_tensor_copy's precondition, without reaching into ggml-impl.h.
+    bool q35v_same_layout(const ggml_tensor* a, const ggml_tensor* b)
+    {
+        if (a->type != b->type)
+            return false;
+        for (int i = 0; i < GGML_MAX_DIMS; i++)
+        {
+            if (a->ne[i] != b->ne[i] || a->nb[i] != b->nb[i])
+                return false;
+        }
+        return true;
+    }
+}
+
+// Commit ONE recurrent-state snapshot into the LIVE state, entirely on the device.
+//
+// The live conv_state_in / delta_state_in slices live in one shared buffer that
+// EVERY cached verify graph binds, so writing them here is visible to the next
+// verify whatever shape it runs at - which is what lets that verify skip its
+// ~300 MB state upload, and this step skip the matching download. That round trip
+// was the single largest per-step cost of speculative decoding on this trunk.
+//
+// `slot` counts back from the end of the verified batch: 0 is the post-window
+// state, (N-1-accepted) the state the accepted prefix ends in.
+TSG_EXPORT int TSGgml_Qwen35CommitStateSnapshot(int slot, int num_recurrent_layers)
+{
+    try
+    {
+        Q35VerifyCache* c = g_q35vc_last_snap;
+        if (c == nullptr || !c->valid || !c->deferred_state)
+            return 0;
+        // slot -1 is the post-window state in the *_state_out slices, which is what a
+        // single-row step (and a fully-accepted verify with no snapshots) commits.
+        const bool from_out = slot < 0;
+        if (!from_out && (c->n_snapshots <= 1 || slot >= c->n_snapshots))
+            return 0;
+        if (static_cast<int>(c->conv_in.size()) != num_recurrent_layers
+            || static_cast<int>(c->delta_in.size()) != num_recurrent_layers
+            || static_cast<int>(c->conv_out.size()) != num_recurrent_layers
+            || static_cast<int>(c->delta_out.size()) != num_recurrent_layers
+            || (!from_out
+                && (static_cast<int>(c->conv_snap_slots.size()) != num_recurrent_layers * c->n_snapshots
+                    || static_cast<int>(c->delta_snap_slots.size()) != num_recurrent_layers * c->n_snapshots)))
+        {
+            set_last_error("Qwen3.5 state commit: recurrent layer count mismatch.");
+            return 0;
+        }
+
+        for (int i = 0; i < num_recurrent_layers; i++)
+        {
+            const int idx = i * c->n_snapshots + (from_out ? 0 : slot);
+            ggml_tensor* csrc = from_out ? c->conv_out[i] : c->conv_snap_slots[idx];
+            ggml_tensor* dsrc = from_out ? c->delta_out[i] : c->delta_snap_slots[idx];
+            if (csrc == nullptr || dsrc == nullptr || c->conv_in[i] == nullptr || c->delta_in[i] == nullptr)
+                return 0;
+            // ggml_backend_tensor_copy is void and ASSERTS same-layout; the slot
+            // views were built with exactly the live slices' shapes, so a mismatch
+            // is a build-side bug - but check rather than abort the process.
+            if (!q35v_same_layout(csrc, c->conv_in[i]) || !q35v_same_layout(dsrc, c->delta_in[i]))
+            {
+                set_last_error("Qwen3.5 state commit: snapshot/live layout mismatch.");
+                return 0;
+            }
+            ggml_backend_tensor_copy(csrc, c->conv_in[i]);
+            ggml_backend_tensor_copy(dsrc, c->delta_in[i]);
+        }
+        clear_last_error();
+        return 1;
+    }
+    catch (const std::exception& ex) { set_last_error(ex.what()); return 0; }
+    catch (...) { set_last_error("Unknown error in Qwen3.5 state commit."); return 0; }
+}
+
+// Read the LIVE recurrent state back to the host. The device copy is authoritative
+// while a speculative session keeps committing snapshots into it; anything that has
+// to run the op-by-op recurrent path (a prefill chunk, an unsupported shape, a
+// backend without the fused verify) needs the host mirror to catch up first.
+TSG_EXPORT int TSGgml_Qwen35DrainDeviceState(
+    void** conv_out_arr, void** delta_out_arr, int num_recurrent_layers)
+{
+    try
+    {
+        Q35VerifyCache* c = g_q35vc_last_snap;
+        if (c == nullptr || !c->valid)
+            return 0;
+        if (conv_out_arr == nullptr || delta_out_arr == nullptr)
+            return 0;
+        if (static_cast<int>(c->conv_in.size()) != num_recurrent_layers
+            || static_cast<int>(c->delta_in.size()) != num_recurrent_layers)
+        {
+            set_last_error("Qwen3.5 state drain: recurrent layer count mismatch.");
+            return 0;
+        }
+
+        host_read_barrier();
+        for (int i = 0; i < num_recurrent_layers; i++)
+        {
+            if (c->conv_in[i] == nullptr || c->delta_in[i] == nullptr)
+                return 0;
+            if (conv_out_arr[i] != nullptr)
+                ggml_backend_tensor_get(c->conv_in[i], conv_out_arr[i], 0, g_q35vc_snap_conv_bytes);
+            if (delta_out_arr[i] != nullptr)
+                ggml_backend_tensor_get(c->delta_in[i], delta_out_arr[i], 0, g_q35vc_snap_delta_bytes);
+        }
+        host_read_barrier();
+        clear_last_error();
+        return 1;
+    }
+    catch (const std::exception& ex) { set_last_error(ex.what()); return 0; }
+    catch (...) { set_last_error("Unknown error in Qwen3.5 state drain."); return 0; }
+}
+
 // Release every rank's parked tensor-parallel prefill graph. The C# side calls
 // this when the model is torn down (the parked contexts hold pooled memory and
 // the plans reference per-rank gallocr buffers).
diff --git a/TensorSharp.GGML.Native/ggml_ops_qwen4exp.cpp b/TensorSharp.GGML.Native/ggml_ops_qwen4exp.cpp
new file mode 100644
index 00000000..9ca6eeac
--- /dev/null
+++ b/TensorSharp.GGML.Native/ggml_ops_qwen4exp.cpp
@@ -0,0 +1,2392 @@
+// Copyright (c) Zhongkai Fu. All rights reserved.
+// https://github.com/zhongkaifu/TensorSharp
+//
+// This file is part of TensorSharp.
+//
+// TensorSharp is licensed under the BSD-3-Clause license found in the LICENSE file in the root directory of this source tree.
+//
+// TensorSharp is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the BSD-3-Clause License for more details.
+#include "ggml_ops_internal.h"
+#include "ggml-impl.h"
+#include <cstdlib>
+#include <cstdio>
+
+#include <cmath>
+#include <cstring>
+#include <cstdint>
+#include <vector>
+#include <unordered_map>
+
+using namespace tsg;
+
+// ============================================================================
+// Qwen3.8-Flash-Next (qwen4exp) fused layers.
+//
+// Three per-layer half-kernels (FFN, GDN, attention), and above them ONE graph
+// for a whole run of layers - which in practice means one graph per token, cut
+// only where the PLE layer has to read the residual on the host.
+//
+// Why. A qwen4exp decode step is dispatch bound, not arithmetic bound: the
+// op-by-op path issued roughly 850 GGML submissions per token; the per-layer
+// fused kernels cut that to 96, but 96 is still 96 graph launches, 96 stream
+// drains and 96 host round trips of the residual where llama.cpp has exactly
+// one of each. The token-span kernel is that same shape: one graph, the
+// residual crossing the PCIe bus twice per token instead of 192 times, and one
+// CUDA graph for ggml-cuda to capture instead of 96 alternating ones.
+//
+// The per-layer entry points remain as the fallback - QSA over budget, foreign
+// shapes, debugging - and share the node builders with the span, so there is a
+// single source of truth for the graph each half builds.
+// ============================================================================
+namespace
+{
+    constexpr const char* kQwen4ExpFfnKernel = "qwen4exp fused FFN block";
+    constexpr const char* kQwen4ExpGdnKernel = "qwen4exp fused GDN block";
+    constexpr const char* kQwen4ExpAttnKernel = "qwen4exp fused attention block";
+    constexpr const char* kQwen4ExpSpanKernel = "qwen4exp fused token span";
+    constexpr int kQwen4ExpMaxSlots = 128;
+    constexpr int kQwen4ExpSpanSlots = 128;
+    // A 48-layer span builds ~150 nodes a layer; 16384 leaves headroom.
+    constexpr int kQwen4ExpSpanGraphSize = 16384;
+
+    // A weight binding resolved through the resident cache, remembered so a
+    // REPLAY can re-resolve it. The cache can move or re-create a device copy
+    // (large allocations elsewhere churn it), and a persisted graph would keep
+    // reading the old address forever - the graph-uid stamp even tells ggml-cuda
+    // to skip the staleness walk that might have noticed.
+    struct Q4eCachedBind
+    {
+        ggml_tensor* tensor;
+        void* data;
+        std::size_t bytes;
+        ggml_backend_buffer_usage usage;
+    };
+
+    // One built graph per layer (or per span), kept across tokens.
+    //
+    // Building the graph is most of a decode step's cost here: the arithmetic for
+    // one token is trivial next to a ggml context, a gallocr plan and a fresh
+    // topology 48 times a token. Holding the graph makes a decode step an upload of
+    // the residual, a replay and a download - and keeps the topology byte-identical
+    // token to token, which is what lets ggml-cuda's graph capture engage.
+    struct Qwen4ExpFfnCache
+    {
+        bool valid = false;
+        ggml_context* ctx = nullptr;
+        ggml_cgraph* graph = nullptr;
+        ggml_gallocr_t alloc = nullptr;
+        ggml_tensor* res_in = nullptr;
+        ggml_tensor* res_out = nullptr;
+        int n_tokens = 0;
+        int hc_dim = 0;
+        const void* sig = nullptr;      // the descriptor this graph was built from
+        // Span graphs are keyed on all three descriptor arrays and the layer range.
+        const void* sig2 = nullptr;
+        const void* sig3 = nullptr;
+        const void* sig4 = nullptr;      // head descriptor, or null
+        const void* sig5 = nullptr;      // PLE descriptor, or null
+        ggml_tensor* logits = nullptr;
+        ggml_tensor* ple_emb_in = nullptr;
+        int layer_begin = -1;
+        int layer_end = -1;
+        int kv_capacity = -1;
+        // Span only: the first layer contributes only its FFN half (its attention
+        // half already ran through the per-layer kernel).
+        int first_ffn_only = 0;
+        // Whether the pos tensor carries the 4 IMRoPE sections (image prompts).
+        int use_mrope = 0;
+        // Recurrent state, read through *_in and written through *_out; the caller
+        // copies out -> in after every compute. The span writes the state in place
+        // instead and only uses the buffer.
+        ggml_tensor* conv_in = nullptr;
+        ggml_tensor* conv_out = nullptr;
+        ggml_tensor* ssm_in = nullptr;
+        ggml_tensor* ssm_out = nullptr;
+        ggml_backend_buffer_t state_buf = nullptr;
+        bool state_ready = false;
+        // Whether res_in is bound to the shared device buffer. A graph built one way
+        // cannot be replayed the other: a non-resident graph owns a private res_in, so
+        // replaying it in resident mode chains nothing and the layers stop composing.
+        int res_resident = -1;
+        // Span only: state write-backs issued host-side after each compute when
+        // TS_Q4E_SPAN_STATE=host - (src, dst) pairs, src a graph output, dst the
+        // persistent state tensor.
+        std::vector<std::pair<ggml_tensor*, ggml_tensor*>> span_copies;
+        std::vector<Q4eCachedBind> rebinds;
+        unsigned rebind_tick = 0;
+        // Span attention inputs, one set per in-span attention layer. Private per
+        // layer, exactly as the per-layer kernels have them.
+        std::vector<ggml_tensor*> span_masks;
+        std::vector<ggml_tensor*> span_pos;
+        std::vector<ggml_tensor*> span_kvidx;
+        std::vector<ggml_tensor*> gdn_probe;
+        // Attention only: the mask is an input and the graph shape follows n_kv.
+        ggml_tensor* mask = nullptr;
+        ggml_tensor* pos = nullptr;
+        ggml_tensor* kv_idx = nullptr;
+        int n_kv = -1;
+
+        // Drop the graph but KEEP the recurrent state: a shape change does this.
+        void reset_graph()
+        {
+            if (alloc) { ggml_gallocr_free(alloc); alloc = nullptr; }
+            if (ctx) { ggml_free(ctx); ctx = nullptr; }
+            graph = nullptr; res_in = nullptr; res_out = nullptr;
+            conv_in = conv_out = ssm_in = ssm_out = nullptr;
+            valid = false; n_tokens = 0; hc_dim = 0; sig = nullptr; res_resident = -1;
+            sig2 = nullptr; sig3 = nullptr; sig4 = nullptr; sig5 = nullptr;
+            logits = nullptr; ple_emb_in = nullptr;
+            layer_begin = -1; layer_end = -1; kv_capacity = -1; first_ffn_only = 0;
+            use_mrope = 0;
+            mask = nullptr; pos = nullptr; kv_idx = nullptr; n_kv = -1;
+            span_copies.clear(); rebinds.clear(); gdn_probe.clear();
+            span_masks.clear(); span_pos.clear(); span_kvidx.clear();
+        }
+
+        // Drop everything including the state: a KV reset does this.
+        void reset()
+        {
+            reset_graph();
+            if (state_buf) { ggml_backend_buffer_free(state_buf); state_buf = nullptr; }
+            state_ready = false;
+        }
+    };
+
+    // The PLE conv history: one persistent device buffer (one PLE layer in the
+    // shipped checkpoint). ready=false re-seeds from the host on the next build.
+    // ---- per-sequence recurrent-state store -------------------------------
+    // GDN conv+ssm state (one entry per layer per sequence) and the PLE conv
+    // history live in device buffers KEYED BY THE HOST SEED POINTER the C# side
+    // passes in the descriptor (each sequence holder owns its own pinned seed
+    // arrays, so the key is per-sequence per-layer for free). The buffer base
+    // is baked into every persisted graph that binds it; the graphs are keyed
+    // on the (per-holder) descriptor addresses, so a graph only ever binds its
+    // own holder's state entry. `ready` gates the one-time seed upload: a
+    // rebuild binds the existing buffer WITHOUT re-seeding (the device copy is
+    // authoritative; the host seed is stale after the first forward).
+    struct Q4eSeqStateEntry
+    {
+        ggml_backend_buffer_t buf = nullptr;
+        std::size_t bytes = 0;
+        bool ready = false;
+    };
+    // ---- per-device executor state ---------------------------------------
+    //
+    // A layer split puts a contiguous run of layers on each GPU, so the same
+    // process drives several devices within one token. Every field below either
+    // IS device memory or bakes a device pointer into a persisted graph, so a
+    // single shared copy would hand device 1 device 0's buffers.
+    //
+    // Indexed by tsg::g_active_rank, which ScopedRank sets around each span.
+    struct Q4eDeviceState
+    {
+        Qwen4ExpFfnCache ffn[kQwen4ExpMaxSlots];
+        Qwen4ExpFfnCache gdn[kQwen4ExpMaxSlots];
+        Qwen4ExpFfnCache attn[kQwen4ExpMaxSlots];
+        Qwen4ExpFfnCache span[kQwen4ExpSpanSlots];
+
+        // GDN conv+ssm state and the PLE conv history, keyed by the HOST SEED
+        // POINTER from the descriptors. Per device as well as per key: with a
+        // layer split, layer L's recurrent state must live on layer L's GPU, and
+        // the same holder's seed pointers are used for layers on both.
+        std::unordered_map<const void*, Q4eSeqStateEntry> seq_state;
+
+        // The 4-wide residual held on the device across a span.
+        ggml_backend_buffer_t res_buf = nullptr;
+        std::size_t res_capacity = 0;
+        ggml_context* res_ctx = nullptr;
+        ggml_tensor* res = nullptr;
+    };
+
+    Q4eDeviceState g_q4e_devs[tsg::TSG_MAX_DEVICES];
+    inline Q4eDeviceState& q4e_dev() { return g_q4e_devs[tsg::g_active_rank]; }
+
+#define g_q4e_ffn            (q4e_dev().ffn)
+#define g_q4e_gdn            (q4e_dev().gdn)
+#define g_q4e_attn           (q4e_dev().attn)
+#define g_q4e_span           (q4e_dev().span)
+#define g_q4e_seq_state      (q4e_dev().seq_state)
+#define g_q4e_res_buf        (q4e_dev().res_buf)
+#define g_q4e_res_capacity   (q4e_dev().res_capacity)
+#define g_q4e_res_ctx        (q4e_dev().res_ctx)
+#define g_q4e_res            (q4e_dev().res)
+
+    Q4eSeqStateEntry* q4e_seq_state(const void* key, std::size_t bytes)
+    {
+        if (key == nullptr) return nullptr;
+        Q4eSeqStateEntry& e = g_q4e_seq_state[key];
+        if (e.buf != nullptr && e.bytes < bytes)
+        {
+            ggml_backend_buffer_free(e.buf);
+            e.buf = nullptr;
+            e.ready = false;
+        }
+        if (e.buf == nullptr)
+        {
+            e.buf = ggml_backend_buft_alloc_buffer(
+                    ggml_backend_get_default_buffer_type(g_backend), bytes);
+            e.bytes = bytes;
+            e.ready = false;
+            if (e.buf == nullptr) return nullptr;
+        }
+        return &e;
+    }
+
+    // ggml-cuda's flash attention takes F16 K/V and one of a fixed set of head sizes;
+    // for head_dim 256 the only other condition is V->ne[0] == K->ne[0], which holds
+    // here. TS_Q4E_FLASH_ATTN=0 falls back to the soft_max path.
+    bool q4e_flash_attn_ok(int kv_type, int head_dim)
+    {
+        static const bool enabled = []{
+            const char* e = std::getenv("TS_Q4E_FLASH_ATTN");
+            return !(e != nullptr && e[0] == '0');
+        }();
+        if (!enabled || kv_type != GGML_TYPE_F16) return false;
+        switch (head_dim)
+        {
+            case 64: case 80: case 96: case 112: case 128: case 256: return true;
+            default: return false;
+        }
+    }
+
+    // ggml-cuda picks its GQA-optimised flash-attention kernel only when
+    // K->ne[1] % FATTN_KQ_STRIDE == 0, so the window is padded to that and the pad
+    // masked off. This is what llama.cpp's get_n_kv rounds to, for the same reason.
+    // The padding is only worth it under flash attention: the soft_max path pays for
+    // every padded column instead of skipping the block.
+    constexpr int kQwen4ExpKvStride = 256;
+
+    // The span writes the GDN state in place inside the graph - cpy(tail ->
+    // conv_state) expanded after every node that reads the state, so node order
+    // sequences the write behind the read. This is the DEFAULT: no host-issued
+    // copies and no extra synchronize per span. The historical "in-place writes
+    // do not take effect" failures - including an earlier note in this file
+    // declaring them measurably wrong - were the gallocr leaf-free bug corrupting
+    // the small gate weights, not the write-back; with uploaded leafs
+    // OUTPUT-flagged the in-place dataflow verifies clean at every length.
+    // TS_Q4E_SPAN_STATE=host restores the copied-out dataflow for comparison.
+    // TS_Q4E_SPAN_REBUILD=1 disables the span replay path entirely - every call
+    // rebuilds the graph. Diagnosis only: separates a wrong-graph bug from a
+    // wrong-replay one.
+    bool q4e_span_force_rebuild()
+    {
+        static const bool v = []{
+            const char* e = std::getenv("TS_Q4E_SPAN_REBUILD");
+            return e != nullptr && e[0] == '1';
+        }();
+        return v;
+    }
+
+    // TS_Q4E_SPAN_FA_MAX=N: only the first N attention layers in a span use flash
+    // attention; the rest run the soft_max path over the SAME padded window.
+    // Diagnosis only - N=0 separates "the pad poisons the output" from "the flash
+    // attention node does".
+    int q4e_span_fa_max()
+    {
+        static const int v = []{
+            const char* e = std::getenv("TS_Q4E_SPAN_FA_MAX");
+            return (e != nullptr && *e != 0) ? std::atoi(e) : 1 << 30;
+        }();
+        return v;
+    }
+
+    // TS_Q4E_SPAN_TRACE=1 prints the residual L2 norm after every layer of every
+    // span call. Diagnosis only: diffing a good run against a bad one names the
+    // first layer whose output moves.
+    bool q4e_span_trace()
+    {
+        static const bool v = []{
+            const char* e = std::getenv("TS_Q4E_SPAN_TRACE");
+            return e != nullptr && e[0] == '1';
+        }();
+        return v;
+    }
+
+    bool q4e_span_state_in_graph()
+    {
+        static const bool v = []{
+            const char* e = std::getenv("TS_Q4E_SPAN_STATE");
+            return !(e != nullptr && e[0] == 'h');
+        }();
+        return v;
+    }
+
+    // Re-resolve a persisted graph's cache-bound weights before a replay. If the
+    // resident cache moved (or re-created) any device copy, the graph's captured
+    // pointers are stale: report it so the caller rebuilds. Content refreshes
+    // (needs_upload with an unmoved pointer) are handled in place.
+    bool q4e_refresh_bindings(Qwen4ExpFfnCache* slot, ggml_backend_dev_t dev)
+    {
+        // The full walk is a few hundred hash lookups; a moved device copy has
+        // never been observed (the guard exists as insurance), so sample it. A
+        // rebuild always re-binds everything regardless.
+        if (++slot->rebind_tick % 32 != 1)
+            return true;
+        for (const Q4eCachedBind& cb : slot->rebinds)
+        {
+            void* before = cb.tensor->data;
+            bool needs_upload = false;
+            if (!try_bind_cached_tensor(g_backend, dev, cb.tensor, cb.data, cb.bytes,
+                                        needs_upload, cb.usage))
+                return false;
+            if (cb.tensor->data != before)
+            {
+                fprintf(stderr, "[q4e] cached weight moved (%p -> %p, %zu bytes); rebuilding the graph%c",
+                        before, cb.tensor->data, cb.bytes, 10);
+                return false;
+            }
+            if (needs_upload)
+            {
+                // Same redirect as Q4eBinder::flush: for a quantized weight cb.data
+                // is a CacheKey (a GCHandle value), not memory.
+                ggml_backend_tensor_set(cb.tensor, resolve_upload_source(cb.data), 0, cb.bytes);
+            }
+        }
+        return true;
+    }
+
+    // Print the L2 of the probed GDN nodes. Diagnosis only.
+    void q4e_trace_probe(Qwen4ExpFfnCache* slot, const char* tag, int position)
+    {
+        if (slot->gdn_probe.empty()) return;
+        static const char* names[] = { "mixed", "qkv", "convout", "gdnout", "proj" };
+        fprintf(stderr, "[q4e-gdn0] %s pos=%d:", tag, position);
+        std::vector<float> buf;
+        for (std::size_t i = 0; i < slot->gdn_probe.size() && i < 5; ++i)
+        {
+            ggml_tensor* t = slot->gdn_probe[i];
+            buf.resize((std::size_t)ggml_nelements(t));
+            ggml_backend_tensor_get(t, buf.data(), 0, ggml_nbytes(t));
+            double n2 = 0.0;
+            for (float f : buf) n2 += (double)f * f;
+            fprintf(stderr, " %s=%.9e", names[i], std::sqrt(n2));
+        }
+        fprintf(stderr, "%c", 10);
+    }
+
+    // Print the L2 of every state tensor a span carries. Diagnosis only.
+    void q4e_trace_state(Qwen4ExpFfnCache* slot, const char* tag, int position)
+    {
+        if (!q4e_span_trace() || slot->span_copies.empty()) return;
+        fprintf(stderr, "[q4e-state] %s pos=%d:", tag, position);
+        std::vector<float> buf;
+        for (std::size_t i = 0; i < slot->span_copies.size(); ++i)
+        {
+            ggml_tensor* st = slot->span_copies[i].second;
+            buf.resize((std::size_t)ggml_nelements(st));
+            ggml_backend_tensor_get(st, buf.data(), 0, ggml_nbytes(st));
+            double n2 = 0.0;
+            for (float f : buf) n2 += (double)f * f;
+            fprintf(stderr, " %.9e", std::sqrt(n2));
+        }
+        fprintf(stderr, "%c", 10);
+    }
+
+    int q4e_pad_kv(int n_kv, int kv_capacity, bool use_flash)
+    {
+        const int stride = use_flash ? kQwen4ExpKvStride : 1;
+        int n_kv_pad = ((n_kv + stride - 1) / stride) * stride;
+        if (n_kv_pad > kv_capacity) n_kv_pad = kv_capacity;
+        return n_kv_pad;
+    }
+
+    // Fill the two index inputs: RoPE positions and the KV rows this step writes.
+    // Values change per token, shapes do not - which is what lets the graph persist.
+    // A multimodal graph's pos tensor holds 4 sections (T|H|W|zero, IMRoPE order);
+    // mrope3 is the per-token (t,h,w) table for image prompts, null for text where
+    // every component is the scalar position.
+    // position indexes the KV cache rows; rope_position is the rotary position of
+    // the first token, which falls BEHIND the cache index once an image has been
+    // compacted into the position stream (IMRoPE gives an HxW image max(H,W)
+    // positions, not HxW). llama.cpp's mtmd advances n_past the same way.
+    void q4e_set_attn_indices(ggml_tensor* pos, ggml_tensor* kv_idx, int T, int position,
+                              const int32_t* mrope3 = nullptr, int rope_position = -1)
+    {
+        if (rope_position < 0) rope_position = position;
+        std::vector<int64_t> k((std::size_t)T);
+        for (int i = 0; i < T; ++i) k[i] = position + i;
+        ggml_backend_tensor_set(kv_idx, k.data(), 0, (std::size_t)T * sizeof(int64_t));
+        if (pos == nullptr) return;
+        const int comps = (int)(pos->ne[0] / T);
+        std::vector<int32_t> p((std::size_t)comps * T);
+        if (comps == 1)
+        {
+            for (int i = 0; i < T; ++i) p[i] = rope_position + i;
+        }
+        else
+        {
+            for (int i = 0; i < T; ++i)
+            {
+                const int32_t t = mrope3 ? mrope3[3 * i + 0] : rope_position + i;
+                const int32_t h = mrope3 ? mrope3[3 * i + 1] : rope_position + i;
+                const int32_t w = mrope3 ? mrope3[3 * i + 2] : rope_position + i;
+                p[i] = t; p[T + i] = h; p[2 * T + i] = w; p[3 * T + i] = 0;
+            }
+        }
+        ggml_backend_tensor_set(pos, p.data(), 0, p.size() * sizeof(int32_t));
+    }
+
+    // TS_Q4E_LOG=1 reports how often each kernel rebuilt its graph rather than
+    // replaying it.
+    struct Q4eStat { long builds = 0; long replays = 0; };
+    Q4eStat g_q4e_stat[4];   // 0 = FFN, 1 = GDN, 2 = ATTN, 3 = SPAN
+    bool q4e_log_enabled()
+    {
+        static const bool v = []{
+            const char* e = std::getenv("TS_Q4E_LOG");
+            return e != nullptr && e[0] == '1';
+        }();
+        return v;
+    }
+    void q4e_note(int k, bool build)
+    {
+        if (!q4e_log_enabled()) return;
+        if (build) g_q4e_stat[k].builds++; else g_q4e_stat[k].replays++;
+        long total = 0;
+        for (const Q4eStat& st : g_q4e_stat) total += st.builds + st.replays;
+        if (total % 2000 == 0 || (k == 3 && (g_q4e_stat[3].builds + g_q4e_stat[3].replays) % 200 == 0))
+            fprintf(stderr, "[q4e] ffn b=%ld r=%ld | gdn b=%ld r=%ld | attn b=%ld r=%ld | span b=%ld r=%ld\n",
+                    g_q4e_stat[0].builds, g_q4e_stat[0].replays,
+                    g_q4e_stat[1].builds, g_q4e_stat[1].replays,
+                    g_q4e_stat[2].builds, g_q4e_stat[2].replays,
+                    g_q4e_stat[3].builds, g_q4e_stat[3].replays);
+    }
+
+    // Stamp every persisted graph with a stable non-zero id.
+    //
+    // ggml_new_graph leaves uid at 0, and ggml-cuda treats 0 as "unknown", so on every
+    // replay it re-walks the nodes comparing a copy of each tensor struct and its
+    // sources to decide whether the captured CUDA graph is still valid. With a stable
+    // id it recognises the graph and skips that walk.
+    // TS_Q4E_GRAPH_UID=0 leaves uid at 0 so ggml-cuda re-checks node properties on
+    // every replay instead of trusting the id.
+    bool q4e_graph_uid_enabled()
+    {
+        static const bool v = []{
+            const char* e = std::getenv("TS_Q4E_GRAPH_UID");
+            return !(e != nullptr && e[0] == '0');
+        }();
+        return v;
+    }
+
+    // TS_Q4E_PHASE=1 prints wall times for the span build phases at T>1.
+    bool q4e_phase_log()
+    {
+        static const bool v = []{
+            const char* e = std::getenv("TS_Q4E_PHASE");
+            return e != nullptr && e[0] == '1';
+        }();
+        return v;
+    }
+    double q4e_now_ms()
+    {
+        return (double)ggml_time_us() / 1000.0;
+    }
+
+    uint64_t q4e_next_graph_uid()
+    {
+        static uint64_t next = 1;
+        return next++;
+    }
+
+    // The 4-wide residual, held on the DEVICE for the whole forward.
+    //
+    // Used by the per-layer fallback's residency experiment; the token span does not
+    // need it - inside one graph the residual never exists on the host at all.
+    // Clamp a caller-supplied device index to an initialized rank. -1 (or an
+    // out-of-range value from a host that predates the layer split) means "the
+    // current rank", which is 0 on every single-GPU run.
+    int q4e_resolve_device(int device)
+    {
+        if (device < 0) return tsg::g_active_rank;
+        const int ndev = tsg::g_device_count.load(std::memory_order_acquire);
+        if (device >= ndev || device >= tsg::TSG_MAX_DEVICES) return tsg::g_active_rank;
+        return device;
+    }
+
+    // Ensure the shared residual tensor exists and is at least `bytes` big.
+    // Per device: see Q4eDeviceState.
+    bool q4e_res_ensure(std::size_t bytes)
+    {
+        if (g_q4e_res != nullptr && g_q4e_res_capacity >= bytes)
+            return true;
+        // Every persisted graph binds its res_in to this buffer's base, so a realloc
+        // here strands every one of them. Loud on purpose.
+        if (g_q4e_res != nullptr)
+            fprintf(stderr, "[q4e] residual buffer REALLOC %zu -> %zu bytes\n",
+                    g_q4e_res_capacity, bytes);
+        if (g_q4e_res_ctx) { ggml_free(g_q4e_res_ctx); g_q4e_res_ctx = nullptr; }
+        if (g_q4e_res_buf) { ggml_backend_buffer_free(g_q4e_res_buf); g_q4e_res_buf = nullptr; }
+        g_q4e_res = nullptr;
+
+        ggml_init_params ip{};
+        ip.mem_size = ggml_tensor_overhead() * 4;
+        ip.mem_buffer = nullptr;
+        ip.no_alloc = true;
+        g_q4e_res_ctx = ggml_init(ip);
+        if (g_q4e_res_ctx == nullptr) return false;
+
+        g_q4e_res = ggml_new_tensor_1d(g_q4e_res_ctx, GGML_TYPE_F32, (int64_t)(bytes / sizeof(float)));
+        g_q4e_res_buf = ggml_backend_buft_alloc_buffer(
+                ggml_backend_get_default_buffer_type(g_backend), bytes);
+        if (g_q4e_res_buf == nullptr) return false;
+        if (ggml_backend_tensor_alloc(g_q4e_res_buf, g_q4e_res,
+                ggml_backend_buffer_get_base(g_q4e_res_buf)) != GGML_STATUS_SUCCESS)
+            return false;
+        g_q4e_res_capacity = bytes;
+        return true;
+    }
+
+    // One place for the weight-binding policy the three block builders shared as a
+    // copy-pasted lambda each. Collecting the uploads here rather than binding
+    // immediately is what lets several layers build into one graph: everything is
+    // bound before a single ggml_gallocr_alloc_graph runs over the lot.
+    struct Q4eHostBinding { ggml_tensor* tensor; void* data; std::size_t bytes; };
+
+    struct Q4eBinder
+    {
+        ggml_backend_dev_t dev = nullptr;
+        std::vector<Q4eHostBinding> upload_list;
+        std::vector<Q4eCachedBind> cached;
+
+        void add(ggml_tensor* tgt, void* data, std::size_t bytes,
+                 ggml_backend_buffer_usage usage = GGML_BACKEND_BUFFER_USAGE_WEIGHTS)
+        {
+            if (tgt == nullptr || data == nullptr) return;
+            if (bytes >= 4096)
+            {
+                bool needs_upload = false;
+                if (try_bind_cached_tensor(g_backend, dev, tgt, data, bytes, needs_upload, usage))
+                {
+                    cached.push_back({tgt, data, bytes, usage});
+                    if (needs_upload) upload_list.push_back({tgt, data, bytes});
+                    return;
+                }
+                ggml_backend_buffer_t buf = nullptr;
+                if (try_get_host_ptr_buffer(g_backend, dev, data, bytes, true, buf))
+                {
+                    if (ggml_backend_tensor_alloc(buf, tgt, data) == GGML_STATUS_SUCCESS)
+                        return;
+                }
+                // A weight this size normally cache-binds; falling through here means
+                // the resident cache could not take it (VRAM pressure). It will live
+                // in the graph's own allocation instead - flagged below - and that is
+                // worth being able to see.
+                fprintf(stderr, "[q4e] weight (%zu bytes) fell out of the resident cache into the graph allocation%c",
+                        bytes, 10);
+            }
+            // The tensor becomes a gallocr-owned leaf, uploaded once at build time.
+            // It MUST carry the OUTPUT flag: ggml_gallocr_free_node only exempts
+            // outputs ("graph outputs are never freed") - the INPUT flag controls
+            // early allocation but the free path ignores it - so an unprotected
+            // leaf is freed after its last consumer and its memory reused by later
+            // intermediates. The FIRST compute reads the weight correctly and then
+            // overwrites it in place; every REPLAY of the persisted graph reads
+            // whatever activations landed there, an error that scales with their
+            // magnitude. A build works exactly once - precisely the difference
+            // between a rebuilt-per-token graph that stays correct and a persisted
+            // one that decays. The INPUT flag stays for the early allocation.
+            ggml_set_input(tgt);
+            ggml_set_output(tgt);
+            upload_list.push_back({tgt, data, bytes});
+        }
+
+        void flush()
+        {
+            // resolve_upload_source, not the raw pointer. For a QUANTIZED weight the
+            // "host pointer" C# passes is a CacheKey - a GCHandle value, not memory -
+            // and the redirect turns it back into the real bytes. Every other fused
+            // executor in this repo does this (dflash, qwen35, gemma4, gptoss,
+            // muse_glimmer); qwen4exp got away without it only because rank 0 always
+            // had every weight preloaded, so needs_upload was never true here.
+            // A layer split makes a misplaced weight reachable, and without this the
+            // symptom is a cudaMemcpy from a handle value. It also counts the
+            // redirect (reported by TS_GGML_LOG_VRAM=1), so a misplacement shows up
+            // as a number instead of a crash.
+            for (const Q4eHostBinding& hb : upload_list)
+                ggml_backend_tensor_set(hb.tensor, resolve_upload_source(hb.data), 0, hb.bytes);
+            upload_list.clear();
+        }
+    };
+}
+
+extern "C"
+{
+
+// Per-layer weights. Pointers first, then int64, then int32 - the layout the
+// C# side mirrors; append within a run rather than reordering.
+struct TSGgmlQwen4ExpFfnArgs
+{
+    // hyper-connection mixer
+    void* hc_norm;          // f32 [hc_dim], gamma folded to (1 + w)
+    void* hc_down;          // [hc_dim, hc_low_rank]
+    void* hc_up;            // [hc_low_rank, hc_dim]
+    void* hc_inject;        // [hc_dim, hc]
+    // MoE
+    void* router;           // [n_embd, n_expert]
+    void* gate_exps;        // [n_embd, n_ff, n_expert]
+    void* up_exps;          // [n_embd, n_ff, n_expert]
+    void* down_exps;        // [n_ff, n_embd, n_expert]
+    // shared expert
+    void* sh_gate_inp;      // f32 [n_embd] - one sigmoid scalar per token
+    void* sh_gate;          // [n_embd, n_ff_sh]
+    void* sh_up;            // [n_embd, n_ff_sh]
+    void* sh_down;          // [n_ff_sh, n_embd]
+
+    long long hc_down_bytes, hc_up_bytes, hc_inject_bytes;
+    long long router_bytes, gate_exps_bytes, up_exps_bytes, down_exps_bytes;
+    long long sh_gate_bytes, sh_up_bytes, sh_down_bytes;
+
+    int hc_down_type, hc_up_type, hc_inject_type;
+    int router_type, gate_exps_type, up_exps_type, down_exps_type;
+    int sh_gate_type, sh_up_type, sh_down_type;
+};
+
+// Per-layer weights for the recurrent (Gated DeltaNet) half of a layer.
+struct TSGgmlQwen4ExpGdnArgs
+{
+    // hyper-connection mixer
+    void* hc_norm;
+    void* hc_down;
+    void* hc_up;
+    void* hc_inject;
+    // delta net
+    void* qkv;              // [n_embd, conv_dim]
+    void* gate;             // [n_embd, value_dim]
+    void* beta;             // [n_embd, n_v_heads]
+    void* alpha;            // [n_embd, n_v_heads]
+    void* conv1d;           // f32 [d_conv, conv_dim]
+    void* ssm_dt;           // f32 [n_v_heads]
+    void* ssm_a;            // f32 [n_v_heads], pre-negated
+    void* ssm_norm;         // f32 [head_v_dim]
+    void* out_proj;         // [value_dim, n_embd]
+    // state, updated in place
+    void* conv_state;       // f32 [d_conv-1, conv_dim]
+    void* ssm_state;        // f32 [head_v_dim, head_v_dim, n_v_heads]
+
+    long long hc_down_bytes, hc_up_bytes, hc_inject_bytes;
+    long long qkv_bytes, gate_bytes, beta_bytes, alpha_bytes, out_proj_bytes;
+
+    int hc_down_type, hc_up_type, hc_inject_type;
+    int qkv_type, gate_type, beta_type, alpha_type, out_proj_type;
+};
+
+// Per-layer weights for the full-attention half of a layer.
+struct TSGgmlQwen4ExpAttnArgs
+{
+    void* hc_norm;
+    void* hc_down;
+    void* hc_up;
+    void* hc_inject;
+    void* wq;               // [n_embd, head_dim * n_head * 2]  (query|gate interleaved)
+    void* wk;               // [n_embd, head_dim * n_head_kv]
+    void* wv;
+    void* wo;               // [head_dim * n_head, n_embd]
+    void* q_norm;           // f32 [head_dim]
+    void* k_norm;           // f32 [head_dim]
+    void* k_cache;          // f16/f32 [head_dim, capacity, n_head_kv]
+    void* v_cache;
+
+    long long hc_down_bytes, hc_up_bytes, hc_inject_bytes;
+    long long wq_bytes, wk_bytes, wv_bytes, wo_bytes;
+    long long kv_bytes;     // per cache
+
+    int hc_down_type, hc_up_type, hc_inject_type;
+    int wq_type, wk_type, wv_type, wo_type;
+    int kv_type;
+};
+
+// ============================================================================
+// Node builders. Each appends one half-layer to ctx and returns the new
+// residual; the entry points and the token span share them, so the graph a
+// half builds has a single source of truth.
+// ============================================================================
+
+// FFN half: hyper-connection mixer -> routed experts + gated shared expert ->
+// hyper-connection scatter. No side effects; expands nothing.
+static ggml_tensor* q4e_nodes_ffn(
+    ggml_context* ctx, Q4eBinder& bnd,
+    const TSGgmlQwen4ExpFfnArgs* a, ggml_tensor* res_in,
+    int n_embd, int hc, int hc_low_rank, int T,
+    int n_expert, int n_expert_used, int n_ff, int n_ff_sh, float eps)
+{
+    const int hc_dim = hc * n_embd;
+    ggml_tensor* w_norm = ggml_new_tensor_1d(ctx, GGML_TYPE_F32, hc_dim);
+    ggml_tensor* w_down = ggml_new_tensor_2d(ctx, (ggml_type)a->hc_down_type, hc_dim, hc_low_rank);
+    ggml_tensor* w_up = ggml_new_tensor_2d(ctx, (ggml_type)a->hc_up_type, hc_low_rank, hc_dim);
+    ggml_tensor* w_inject = ggml_new_tensor_2d(ctx, (ggml_type)a->hc_inject_type, hc_dim, hc);
+    ggml_tensor* w_router = ggml_new_tensor_2d(ctx, (ggml_type)a->router_type, n_embd, n_expert);
+    ggml_tensor* w_gate_e = ggml_new_tensor_3d(ctx, (ggml_type)a->gate_exps_type, n_embd, n_ff, n_expert);
+    ggml_tensor* w_up_e = ggml_new_tensor_3d(ctx, (ggml_type)a->up_exps_type, n_embd, n_ff, n_expert);
+    ggml_tensor* w_down_e = ggml_new_tensor_3d(ctx, (ggml_type)a->down_exps_type, n_ff, n_embd, n_expert);
+    ggml_tensor* w_sh_gi = ggml_new_tensor_2d(ctx, GGML_TYPE_F32, n_embd, 1);
+    ggml_tensor* w_sh_g = ggml_new_tensor_2d(ctx, (ggml_type)a->sh_gate_type, n_embd, n_ff_sh);
+    ggml_tensor* w_sh_u = ggml_new_tensor_2d(ctx, (ggml_type)a->sh_up_type, n_embd, n_ff_sh);
+    ggml_tensor* w_sh_d = ggml_new_tensor_2d(ctx, (ggml_type)a->sh_down_type, n_ff_sh, n_embd);
+
+    // ---- hyper-connection mixer ------------------------------------------
+    // Grouped RMS norm: normalise over ONE residual stream, then scale the
+    // whole hc-wide row by the gamma vector.
+    ggml_tensor* res3 = ggml_reshape_3d(ctx, res_in, n_embd, hc, T);
+    ggml_tensor* xn = ggml_rms_norm(ctx, res3, eps);
+    xn = ggml_reshape_2d(ctx, xn, hc_dim, T);
+    xn = ggml_mul(ctx, xn, w_norm);
+
+    ggml_tensor* lo = ggml_mul_mat(ctx, w_down, xn);
+    lo = ggml_silu(ctx, ggml_scale(ctx, lo, 1.0f / (float)hc));
+    ggml_tensor* gate = ggml_sigmoid(ctx, ggml_mul_mat(ctx, w_up, lo));
+
+    ggml_tensor* gated = ggml_mul(ctx, xn, gate);
+    gated = ggml_reshape_3d(ctx, gated, n_embd, hc, T);
+
+    // Collapse the streams by their mean.
+    ggml_tensor* mixed = ggml_cont(ctx, ggml_view_2d(ctx, gated, n_embd, T,
+            ggml_row_size(gated->type, n_embd) * hc, 0));
+    for (int c = 1; c < hc; ++c)
+    {
+        ggml_tensor* s = ggml_view_2d(ctx, gated, n_embd, T,
+                ggml_row_size(gated->type, n_embd) * hc,
+                ggml_row_size(gated->type, n_embd) * c);
+        mixed = ggml_add(ctx, mixed, s);
+    }
+    mixed = ggml_scale(ctx, mixed, 1.0f / (float)hc);
+
+    ggml_tensor* inject = ggml_mul_mat(ctx, w_inject, xn);   // [hc, T]
+
+    // ---- routed experts ---------------------------------------------------
+    // Softmax over every expert, top-k, then renormalise the selected
+    // weights - llama.cpp's build_moe_ffn with norm_w.
+    ggml_tensor* logits = ggml_mul_mat(ctx, w_router, mixed);      // [n_expert, T]
+    ggml_tensor* probs = ggml_soft_max(ctx, logits);
+    // ggml_argsort_top_k, not ggml_top_k: this is the exact node shape llama.cpp's
+    // build_moe_ffn emits, and ggml-cuda's topk_moe fusion matches on the node
+    // sequence rather than on intent.
+    ggml_tensor* sel = ggml_argsort_top_k(ctx, probs, n_expert_used);  // [n_used, T] i32
+
+    ggml_tensor* w_sel = ggml_get_rows(ctx,
+            ggml_reshape_3d(ctx, probs, 1, n_expert, T), sel);     // [1, n_used, T]
+    w_sel = ggml_reshape_2d(ctx, w_sel, n_expert_used, T);
+    ggml_tensor* w_sum = ggml_sum_rows(ctx, w_sel);                // [1, T]
+    w_sel = ggml_div(ctx, w_sel, w_sum);
+    w_sel = ggml_reshape_3d(ctx, w_sel, 1, n_expert_used, T);
+
+    ggml_tensor* moe_in = ggml_reshape_3d(ctx, mixed, n_embd, 1, T);
+    ggml_tensor* e_up = ggml_mul_mat_id(ctx, w_up_e, moe_in, sel);      // [n_ff, n_used, T]
+    ggml_tensor* e_gate = ggml_mul_mat_id(ctx, w_gate_e, moe_in, sel);
+    ggml_tensor* par = ggml_mul(ctx, ggml_silu(ctx, e_gate), e_up);
+    ggml_tensor* experts = ggml_mul_mat_id(ctx, w_down_e, par, sel);    // [n_embd, n_used, T]
+    experts = ggml_mul(ctx, experts, w_sel);
+
+    ggml_tensor* moe_out = ggml_view_2d(ctx, experts, n_embd, T,
+            experts->nb[2], 0);
+    for (int k = 1; k < n_expert_used; ++k)
+    {
+        ggml_tensor* s = ggml_view_2d(ctx, experts, n_embd, T,
+                experts->nb[2], (std::size_t)k * experts->nb[1]);
+        moe_out = ggml_add(ctx, moe_out, s);
+    }
+
+    // ---- shared expert, behind its own sigmoid scalar ---------------------
+    ggml_tensor* sg = ggml_mul_mat(ctx, w_sh_g, mixed);
+    ggml_tensor* su = ggml_mul_mat(ctx, w_sh_u, mixed);
+    ggml_tensor* sh = ggml_mul_mat(ctx, w_sh_d, ggml_mul(ctx, ggml_silu(ctx, sg), su));
+    ggml_tensor* s_gate = ggml_sigmoid(ctx, ggml_mul_mat(ctx, w_sh_gi, mixed)); // [1, T]
+    ggml_tensor* ffn_out = ggml_add(ctx, moe_out, ggml_mul(ctx, sh, s_gate));
+
+    // ---- hyper-connection scatter ----------------------------------------
+    // 2*sigmoid centres the weights on 1, so an untrained injection matrix
+    // reproduces a plain residual add.
+    ggml_tensor* wsc = ggml_scale(ctx, ggml_sigmoid(ctx,
+            ggml_scale(ctx, inject, 1.0f / (float)hc)), 2.0f);
+    wsc = ggml_reshape_3d(ctx, wsc, 1, hc, T);
+
+    ggml_tensor* b = ggml_reshape_3d(ctx, ffn_out, n_embd, 1, T);
+    b = ggml_repeat_4d(ctx, b, n_embd, hc, T, 1);
+
+    ggml_tensor* res_out = ggml_add(ctx, res3, ggml_mul(ctx, b, wsc));
+    res_out = ggml_reshape_2d(ctx, res_out, hc_dim, T);
+
+    bnd.add(w_norm, a->hc_norm, (std::size_t)hc_dim * sizeof(float));
+    bnd.add(w_down, a->hc_down, (std::size_t)a->hc_down_bytes);
+    bnd.add(w_up, a->hc_up, (std::size_t)a->hc_up_bytes);
+    bnd.add(w_inject, a->hc_inject, (std::size_t)a->hc_inject_bytes);
+    bnd.add(w_router, a->router, (std::size_t)a->router_bytes);
+    bnd.add(w_gate_e, a->gate_exps, (std::size_t)a->gate_exps_bytes);
+    bnd.add(w_up_e, a->up_exps, (std::size_t)a->up_exps_bytes);
+    bnd.add(w_down_e, a->down_exps, (std::size_t)a->down_exps_bytes);
+    bnd.add(w_sh_gi, a->sh_gate_inp, (std::size_t)n_embd * sizeof(float));
+    bnd.add(w_sh_g, a->sh_gate, (std::size_t)a->sh_gate_bytes);
+    bnd.add(w_sh_u, a->sh_up, (std::size_t)a->sh_up_bytes);
+    bnd.add(w_sh_d, a->sh_down, (std::size_t)a->sh_down_bytes);
+
+    return res_out;
+}
+
+// GDN half: hyper-connection mixer -> projections -> causal conv ->
+// gated delta net -> sigmoid-gated norm -> out proj -> scatter.
+//
+// The caller owns conv_state / ssm_state (created in ctx, allocated into the
+// layer's persistent state buffer) and decides how the write-back reaches
+// them: the per-layer entry copies out -> in after the compute exactly as it
+// always has; the span expands cpy(tail -> conv_state) into the graph AFTER
+// the nodes that read the state, so node order sequences the write behind the
+// read. The write-back sources come out through `wb`.
+struct Q4eGdnWriteback { ggml_tensor* tail; ggml_tensor* new_state; };
+
+static ggml_tensor* q4e_nodes_gdn(
+    ggml_context* ctx, Q4eBinder& bnd,
+    const TSGgmlQwen4ExpGdnArgs* a, ggml_tensor* res_in,
+    ggml_tensor* conv_state, ggml_tensor* ssm_state,
+    int n_embd, int hc, int hc_low_rank, int T,
+    int head_k_dim, int head_v_dim, int n_k_heads, int n_v_heads, int d_conv,
+    float eps, Q4eGdnWriteback* wb,
+    std::vector<ggml_tensor*>* probe = nullptr)
+{
+    const int hc_dim = hc * n_embd;
+    const int key_dim = head_k_dim * n_k_heads;
+    const int value_dim = head_v_dim * n_v_heads;
+    const int conv_dim = key_dim * 2 + value_dim;
+    const int hist = d_conv - 1;
+
+    ggml_tensor* w_norm    = ggml_new_tensor_1d(ctx, GGML_TYPE_F32, hc_dim);
+    ggml_tensor* w_down    = ggml_new_tensor_2d(ctx, (ggml_type)a->hc_down_type, hc_dim, hc_low_rank);
+    ggml_tensor* w_up      = ggml_new_tensor_2d(ctx, (ggml_type)a->hc_up_type, hc_low_rank, hc_dim);
+    ggml_tensor* w_inject  = ggml_new_tensor_2d(ctx, (ggml_type)a->hc_inject_type, hc_dim, hc);
+    ggml_tensor* w_qkv     = ggml_new_tensor_2d(ctx, (ggml_type)a->qkv_type, n_embd, conv_dim);
+    ggml_tensor* w_gate    = ggml_new_tensor_2d(ctx, (ggml_type)a->gate_type, n_embd, value_dim);
+    ggml_tensor* w_beta    = ggml_new_tensor_2d(ctx, (ggml_type)a->beta_type, n_embd, n_v_heads);
+    ggml_tensor* w_alpha   = ggml_new_tensor_2d(ctx, (ggml_type)a->alpha_type, n_embd, n_v_heads);
+    ggml_tensor* w_conv    = ggml_new_tensor_2d(ctx, GGML_TYPE_F32, d_conv, conv_dim);
+    ggml_tensor* w_dt      = ggml_new_tensor_1d(ctx, GGML_TYPE_F32, n_v_heads);
+    ggml_tensor* w_a       = ggml_new_tensor_1d(ctx, GGML_TYPE_F32, n_v_heads);
+    ggml_tensor* w_ssmnorm = ggml_new_tensor_1d(ctx, GGML_TYPE_F32, head_v_dim);
+    ggml_tensor* w_out     = ggml_new_tensor_2d(ctx, (ggml_type)a->out_proj_type, value_dim, n_embd);
+
+    // ---- hyper-connection mixer ----
+    ggml_tensor* res3 = ggml_reshape_3d(ctx, res_in, n_embd, hc, T);
+    ggml_tensor* xn = ggml_rms_norm(ctx, res3, eps);
+    xn = ggml_reshape_2d(ctx, xn, hc_dim, T);
+    xn = ggml_mul(ctx, xn, w_norm);
+
+    ggml_tensor* lo = ggml_silu(ctx, ggml_scale(ctx, ggml_mul_mat(ctx, w_down, xn), 1.0f / (float)hc));
+    ggml_tensor* gt = ggml_sigmoid(ctx, ggml_mul_mat(ctx, w_up, lo));
+    ggml_tensor* gated = ggml_reshape_3d(ctx, ggml_mul(ctx, xn, gt), n_embd, hc, T);
+
+    ggml_tensor* mixed = ggml_cont(ctx, ggml_view_2d(ctx, gated, n_embd, T,
+            ggml_row_size(gated->type, n_embd) * hc, 0));
+    for (int c = 1; c < hc; ++c)
+    {
+        mixed = ggml_add(ctx, mixed, ggml_view_2d(ctx, gated, n_embd, T,
+                ggml_row_size(gated->type, n_embd) * hc,
+                ggml_row_size(gated->type, n_embd) * c));
+    }
+    mixed = ggml_scale(ctx, mixed, 1.0f / (float)hc);
+    ggml_tensor* inject = ggml_mul_mat(ctx, w_inject, xn);
+
+    // ---- projections ----
+    ggml_tensor* qkv = ggml_mul_mat(ctx, w_qkv, mixed);        // [conv_dim, T]
+    ggml_tensor* z = ggml_mul_mat(ctx, w_gate, mixed);         // [value_dim, T]
+    ggml_tensor* beta_raw = ggml_mul_mat(ctx, w_beta, mixed);  // [n_v_heads, T]
+    ggml_tensor* alpha_raw = ggml_mul_mat(ctx, w_alpha, mixed);
+
+    // ---- causal depthwise conv over the ring history ----
+    // conv_state is [hist, conv_dim]; qkv transposed is [T, conv_dim].
+    ggml_tensor* qkv_t = ggml_reshape_3d(ctx, ggml_cont(ctx, ggml_transpose(ctx, qkv)),
+            T, conv_dim, 1);
+    ggml_tensor* conv_in = ggml_concat(ctx, conv_state, qkv_t, 0); // [hist + T, conv_dim, 1]
+    ggml_tensor* conv_out = ggml_silu(ctx, ggml_ssm_conv(ctx, conv_in, w_conv)); // [conv_dim, T, 1]
+
+    // keep the last `hist` columns for the next token
+    ggml_tensor* tail = ggml_cont(ctx, ggml_view_3d(ctx, conv_in, hist, conv_dim, 1,
+            conv_in->nb[1], conv_in->nb[2], ggml_row_size(conv_in->type, T)));
+
+    // ---- delta net ----
+    ggml_tensor* q = ggml_view_3d(ctx, conv_out, head_k_dim, n_k_heads, T,
+            ggml_row_size(conv_out->type, head_k_dim), conv_out->nb[1], 0);
+    ggml_tensor* k = ggml_view_3d(ctx, conv_out, head_k_dim, n_k_heads, T,
+            ggml_row_size(conv_out->type, head_k_dim), conv_out->nb[1],
+            ggml_row_size(conv_out->type, key_dim));
+    ggml_tensor* v = ggml_view_3d(ctx, conv_out, head_v_dim, n_v_heads, T,
+            ggml_row_size(conv_out->type, head_v_dim), conv_out->nb[1],
+            ggml_row_size(conv_out->type, 2 * key_dim));
+
+    q = ggml_l2_norm(ctx, ggml_cont(ctx, q), eps);
+    k = ggml_l2_norm(ctx, ggml_cont(ctx, k), eps);
+
+    // Repeat q/k up to the value-head count. ggml_repeat TILES (head h reads
+    // h % n_k_heads), which is the convention Qwen 3.5's kernel and llama.cpp's
+    // non-fused path both use; leaving it to the op's own broadcast produced
+    // fluent-looking noise.
+    if (n_k_heads != n_v_heads)
+    {
+        q = ggml_repeat_4d(ctx, q, head_k_dim, n_v_heads, T, 1);
+        k = ggml_repeat_4d(ctx, k, head_k_dim, n_v_heads, T, 1);
+    }
+    q = ggml_reshape_4d(ctx, ggml_cont(ctx, q), head_k_dim, n_v_heads, T, 1);
+    k = ggml_reshape_4d(ctx, ggml_cont(ctx, k), head_k_dim, n_v_heads, T, 1);
+    v = ggml_reshape_4d(ctx, ggml_cont(ctx, v), head_v_dim, n_v_heads, T, 1);
+
+    // The op scales q internally (llama.cpp passes it unscaled), and a uniform
+    // scale here would be absorbed by the RMS norm below in any case.
+
+    ggml_tensor* b4 = ggml_reshape_4d(ctx, ggml_sigmoid(ctx, beta_raw), 1, n_v_heads, T, 1);
+    ggml_tensor* g4 = ggml_reshape_4d(ctx,
+            ggml_mul(ctx, ggml_softplus(ctx, ggml_add(ctx, alpha_raw, w_dt)), w_a),
+            1, n_v_heads, T, 1);
+    ggml_tensor* s4 = ggml_reshape_4d(ctx, ssm_state, head_v_dim, head_v_dim, n_v_heads, 1);
+
+    ggml_tensor* gdn_out = ggml_gated_delta_net(ctx, q, k, v, g4, b4, s4, 1);
+
+    const int64_t attn_elems = (int64_t)head_v_dim * n_v_heads * T;
+    ggml_tensor* core = ggml_view_3d(ctx, gdn_out, head_v_dim, n_v_heads, T,
+            ggml_row_size(gdn_out->type, head_v_dim),
+            ggml_row_size(gdn_out->type, head_v_dim * n_v_heads), 0);
+    ggml_tensor* new_state = ggml_view_3d(ctx, gdn_out, head_v_dim, head_v_dim, n_v_heads,
+            ggml_row_size(gdn_out->type, head_v_dim),
+            ggml_row_size(gdn_out->type, head_v_dim * head_v_dim),
+            ggml_row_size(gdn_out->type, attn_elems));
+
+    // qwen4exp closes with a SIGMOID gate, where Qwen 3.5 uses SiLU.
+    ggml_tensor* normed = ggml_mul(ctx, ggml_rms_norm(ctx, ggml_cont(ctx, core), eps), w_ssmnorm);
+    ggml_tensor* zg = ggml_sigmoid(ctx, ggml_reshape_3d(ctx, z, head_v_dim, n_v_heads, T));
+    ggml_tensor* out2 = ggml_reshape_2d(ctx, ggml_mul(ctx, normed, zg), value_dim, T);
+    ggml_tensor* proj = ggml_mul_mat(ctx, w_out, out2);        // [n_embd, T]
+
+    // ---- hyper-connection scatter ----
+    ggml_tensor* wsc = ggml_reshape_3d(ctx, ggml_scale(ctx,
+            ggml_sigmoid(ctx, ggml_scale(ctx, inject, 1.0f / (float)hc)), 2.0f), 1, hc, T);
+    ggml_tensor* bexp = ggml_repeat_4d(ctx, ggml_reshape_3d(ctx, proj, n_embd, 1, T),
+            n_embd, hc, T, 1);
+    ggml_tensor* res_out = ggml_reshape_2d(ctx,
+            ggml_add(ctx, res3, ggml_mul(ctx, bexp, wsc)), hc_dim, T);
+
+    bnd.add(w_norm, a->hc_norm, (std::size_t)hc_dim * sizeof(float));
+    bnd.add(w_down, a->hc_down, (std::size_t)a->hc_down_bytes);
+    bnd.add(w_up, a->hc_up, (std::size_t)a->hc_up_bytes);
+    bnd.add(w_inject, a->hc_inject, (std::size_t)a->hc_inject_bytes);
+    bnd.add(w_qkv, a->qkv, (std::size_t)a->qkv_bytes);
+    bnd.add(w_gate, a->gate, (std::size_t)a->gate_bytes);
+    bnd.add(w_beta, a->beta, (std::size_t)a->beta_bytes);
+    bnd.add(w_alpha, a->alpha, (std::size_t)a->alpha_bytes);
+    bnd.add(w_conv, a->conv1d, (std::size_t)d_conv * conv_dim * sizeof(float));
+    bnd.add(w_dt, a->ssm_dt, (std::size_t)n_v_heads * sizeof(float));
+    bnd.add(w_a, a->ssm_a, (std::size_t)n_v_heads * sizeof(float));
+    bnd.add(w_ssmnorm, a->ssm_norm, (std::size_t)head_v_dim * sizeof(float));
+    bnd.add(w_out, a->out_proj, (std::size_t)a->out_proj_bytes);
+
+    if (probe != nullptr)
+    {
+        ggml_tensor* nodes[] = { mixed, qkv, conv_out, gdn_out, proj };
+        for (ggml_tensor* t : nodes) { ggml_set_output(t); probe->push_back(t); }
+    }
+
+    wb->tail = tail;
+    wb->new_state = new_state;
+    return res_out;
+}
+
+// Attention half: hyper-connection mixer -> joint query|gate -> Q/K norm ->
+// partial rotary -> KV append -> (flash) attention -> sigmoid gate -> out
+// proj -> scatter.
+//
+// The KV write is expanded into `graph` HERE, before the caller expands the
+// returned residual: nothing in the residual's tree depends on the write -
+// k_full is a plain view of the cache and ggml does not treat view aliasing as
+// an edge - so node order is the only thing sequencing the write against the
+// read, and this token has to be able to attend to itself.
+static ggml_tensor* q4e_nodes_attn(
+    ggml_context* ctx, ggml_cgraph* graph, Q4eBinder& bnd,
+    const TSGgmlQwen4ExpAttnArgs* a, ggml_tensor* res_in,
+    ggml_tensor* mask, ggml_tensor* pos, ggml_tensor* kv_idx,
+    int n_embd, int hc, int hc_low_rank, int T,
+    int head_dim, int n_head, int n_head_kv, int kv_capacity, int n_kv_pad,
+    int n_rot, float rope_base, float rope_freq_scale, float attn_scale,
+    float eps, bool use_flash,
+    std::vector<ggml_tensor*>* kv_out = nullptr,
+    std::vector<ggml_tensor*>* probe = nullptr,
+    const int32_t* mrope_sections = nullptr)
+{
+    const int hc_dim = hc * n_embd;
+    const int q_dim = head_dim * n_head;
+
+    ggml_tensor* w_norm   = ggml_new_tensor_1d(ctx, GGML_TYPE_F32, hc_dim);
+    ggml_tensor* w_down   = ggml_new_tensor_2d(ctx, (ggml_type)a->hc_down_type, hc_dim, hc_low_rank);
+    ggml_tensor* w_up     = ggml_new_tensor_2d(ctx, (ggml_type)a->hc_up_type, hc_low_rank, hc_dim);
+    ggml_tensor* w_inject = ggml_new_tensor_2d(ctx, (ggml_type)a->hc_inject_type, hc_dim, hc);
+    ggml_tensor* wq       = ggml_new_tensor_2d(ctx, (ggml_type)a->wq_type, n_embd, q_dim * 2);
+    ggml_tensor* wk       = ggml_new_tensor_2d(ctx, (ggml_type)a->wk_type, n_embd, head_dim * n_head_kv);
+    ggml_tensor* wv       = ggml_new_tensor_2d(ctx, (ggml_type)a->wv_type, n_embd, head_dim * n_head_kv);
+    ggml_tensor* wo       = ggml_new_tensor_2d(ctx, (ggml_type)a->wo_type, q_dim, n_embd);
+    ggml_tensor* q_norm_w = ggml_new_tensor_1d(ctx, GGML_TYPE_F32, head_dim);
+    ggml_tensor* k_norm_w = ggml_new_tensor_1d(ctx, GGML_TYPE_F32, head_dim);
+    ggml_tensor* k_cache  = ggml_new_tensor_3d(ctx, (ggml_type)a->kv_type, head_dim, kv_capacity, n_head_kv);
+    ggml_tensor* v_cache  = ggml_new_tensor_3d(ctx, (ggml_type)a->kv_type, head_dim, kv_capacity, n_head_kv);
+
+    // ---- hyper-connection mixer ----
+    ggml_tensor* res3 = ggml_reshape_3d(ctx, res_in, n_embd, hc, T);
+    ggml_tensor* xn = ggml_mul(ctx,
+            ggml_reshape_2d(ctx, ggml_rms_norm(ctx, res3, eps), hc_dim, T), w_norm);
+    ggml_tensor* lo = ggml_silu(ctx, ggml_scale(ctx, ggml_mul_mat(ctx, w_down, xn), 1.0f / (float)hc));
+    ggml_tensor* gt = ggml_sigmoid(ctx, ggml_mul_mat(ctx, w_up, lo));
+    ggml_tensor* gated = ggml_reshape_3d(ctx, ggml_mul(ctx, xn, gt), n_embd, hc, T);
+    ggml_tensor* mixed = ggml_cont(ctx, ggml_view_2d(ctx, gated, n_embd, T,
+            ggml_row_size(gated->type, n_embd) * hc, 0));
+    for (int c = 1; c < hc; ++c)
+        mixed = ggml_add(ctx, mixed, ggml_view_2d(ctx, gated, n_embd, T,
+                ggml_row_size(gated->type, n_embd) * hc, ggml_row_size(gated->type, n_embd) * c));
+    mixed = ggml_scale(ctx, mixed, 1.0f / (float)hc);
+    ggml_tensor* inject = ggml_mul_mat(ctx, w_inject, xn);
+
+    // ---- q | gate, interleaved per head ----
+    ggml_tensor* qg = ggml_mul_mat(ctx, wq, mixed);                 // [q_dim*2, T]
+    const std::size_t esz = ggml_element_size(qg);
+    ggml_tensor* q = ggml_view_3d(ctx, qg, head_dim, n_head, T,
+            esz * head_dim * 2, esz * head_dim * 2 * n_head, 0);
+    ggml_tensor* gate = ggml_cont(ctx, ggml_view_3d(ctx, qg, head_dim, n_head, T,
+            esz * head_dim * 2, esz * head_dim * 2 * n_head, esz * head_dim));
+
+    q = ggml_mul(ctx, ggml_rms_norm(ctx, ggml_cont(ctx, q), eps), q_norm_w);
+    ggml_tensor* k = ggml_reshape_3d(ctx, ggml_mul_mat(ctx, wk, mixed), head_dim, n_head_kv, T);
+    k = ggml_mul(ctx, ggml_rms_norm(ctx, k, eps), k_norm_w);
+    ggml_tensor* v = ggml_reshape_3d(ctx, ggml_mul_mat(ctx, wv, mixed), head_dim, n_head_kv, T);
+
+    // Partial rotary over the first n_rot dims. IMRoPE reduces to NEOX when every
+    // position component is equal, which it is for text - so text graphs keep the
+    // exact NEOX path they have always had (byte-stable), and only a graph whose
+    // pos tensor carries the 4 IMRoPE sections takes the multi-axis rotation, the
+    // same op llama.cpp's qwen4exp runs.
+    if (mrope_sections != nullptr)
+    {
+        int sect[4] = { mrope_sections[0], mrope_sections[1], mrope_sections[2], mrope_sections[3] };
+        q = ggml_rope_multi(ctx, q, pos, nullptr, n_rot, sect, GGML_ROPE_TYPE_IMROPE,
+                            0, rope_base, rope_freq_scale, 0.0f, 1.0f, 0.0f, 0.0f);
+        k = ggml_rope_multi(ctx, k, pos, nullptr, n_rot, sect, GGML_ROPE_TYPE_IMROPE,
+                            0, rope_base, rope_freq_scale, 0.0f, 1.0f, 0.0f, 0.0f);
+    }
+    else
+    {
+        q = ggml_rope_ext(ctx, q, pos, nullptr, n_rot, 2, 0, rope_base, rope_freq_scale,
+                          0.0f, 1.0f, 0.0f, 0.0f);
+        k = ggml_rope_ext(ctx, k, pos, nullptr, n_rot, 2, 0, rope_base, rope_freq_scale,
+                          0.0f, 1.0f, 0.0f, 0.0f);
+    }
+
+    // ---- append to the cache ----
+    ggml_tensor* k_write = ggml_cont(ctx, ggml_permute(ctx, k, 0, 2, 1, 3)); // [hd, T, kvH]
+    ggml_tensor* v_write = ggml_cont(ctx, ggml_permute(ctx, v, 0, 2, 1, 3));
+    ggml_tensor* k_full = ggml_view_3d(ctx, k_cache, head_dim, n_kv_pad, n_head_kv,
+            k_cache->nb[1], k_cache->nb[2], 0);
+    ggml_tensor* v_full = ggml_view_3d(ctx, v_cache, head_dim, n_kv_pad, n_head_kv,
+            v_cache->nb[1], v_cache->nb[2], 0);
+
+    // The KV write goes into the graph FIRST - see the function comment.
+    ggml_build_forward_expand(graph, ggml_set_rows(ctx, k_cache, k_write, kv_idx));
+    ggml_build_forward_expand(graph, ggml_set_rows(ctx, v_cache, v_write, kv_idx));
+
+    // ---- attention ----
+    ggml_tensor* q_attn = ggml_cont(ctx, ggml_permute(ctx, q, 0, 2, 1, 3));  // [hd, T, nH]
+    ggml_tensor* attn = nullptr;
+    if (use_flash)
+    {
+        // One fused kernel in place of mul_mat -> soft_max -> cont(permute(V)) ->
+        // mul_mat -> cont(permute). It never materialises the [n_kv, T, n_head]
+        // scores and never copies the whole V window. This is also what llama.cpp
+        // runs here. Result lands as [hd, nH, T] - already the layout the gate and
+        // the output projection want.
+        attn = ggml_flash_attn_ext(ctx, q_attn, k_full, v_full, mask,
+                                   attn_scale, 0.0f, 0.0f);
+        ggml_flash_attn_ext_set_prec(attn, GGML_PREC_F32);
+    }
+    else
+    {
+        ggml_tensor* scores = ggml_mul_mat(ctx, k_full, q_attn);             // [n_kv, T, nH]
+        ggml_mul_mat_set_prec(scores, GGML_PREC_F32);
+        ggml_tensor* probs = ggml_soft_max_ext(ctx, scores, mask, attn_scale, 0.0f);
+        ggml_tensor* v_perm = ggml_cont(ctx, ggml_permute(ctx, v_full, 1, 0, 2, 3));
+        attn = ggml_mul_mat(ctx, v_perm, probs);                             // [hd, T, nH]
+        attn = ggml_cont(ctx, ggml_permute(ctx, attn, 0, 2, 1, 3));          // [hd, nH, T]
+        if (probe != nullptr)
+        {
+            ggml_set_output(scores); ggml_set_output(probs); ggml_set_output(attn);
+            probe->push_back(scores); probe->push_back(probs); probe->push_back(attn);
+        }
+    }
+
+    // qwen4exp gates the attention output before the output projection.
+    attn = ggml_mul(ctx, attn, ggml_sigmoid(ctx, gate));
+    ggml_tensor* proj = ggml_mul_mat(ctx, wo, ggml_reshape_2d(ctx, attn, q_dim, T));
+
+    // ---- hyper-connection scatter ----
+    ggml_tensor* wsc = ggml_reshape_3d(ctx, ggml_scale(ctx,
+            ggml_sigmoid(ctx, ggml_scale(ctx, inject, 1.0f / (float)hc)), 2.0f), 1, hc, T);
+    ggml_tensor* bexp = ggml_repeat_4d(ctx, ggml_reshape_3d(ctx, proj, n_embd, 1, T),
+            n_embd, hc, T, 1);
+    ggml_tensor* res_out = ggml_reshape_2d(ctx,
+            ggml_add(ctx, res3, ggml_mul(ctx, bexp, wsc)), hc_dim, T);
+
+    bnd.add(w_norm, a->hc_norm, (std::size_t)hc_dim * sizeof(float));
+    bnd.add(w_down, a->hc_down, (std::size_t)a->hc_down_bytes);
+    bnd.add(w_up, a->hc_up, (std::size_t)a->hc_up_bytes);
+    bnd.add(w_inject, a->hc_inject, (std::size_t)a->hc_inject_bytes);
+    bnd.add(wq, a->wq, (std::size_t)a->wq_bytes);
+    bnd.add(wk, a->wk, (std::size_t)a->wk_bytes);
+    bnd.add(wv, a->wv, (std::size_t)a->wv_bytes);
+    bnd.add(wo, a->wo, (std::size_t)a->wo_bytes);
+    bnd.add(q_norm_w, a->q_norm, (std::size_t)head_dim * sizeof(float));
+    bnd.add(k_norm_w, a->k_norm, (std::size_t)head_dim * sizeof(float));
+    // The caches are read AND written, so they need a device buffer that outlives
+    // the graph rather than a weights binding.
+    bnd.add(k_cache, a->k_cache, (std::size_t)a->kv_bytes, GGML_BACKEND_BUFFER_USAGE_ANY);
+    bnd.add(v_cache, a->v_cache, (std::size_t)a->kv_bytes, GGML_BACKEND_BUFFER_USAGE_ANY);
+    if (kv_out != nullptr) { kv_out->push_back(k_cache); kv_out->push_back(v_cache); }
+
+    return res_out;
+}
+
+// ============================================================================
+// Per-layer entry points (the fallback path).
+// ============================================================================
+
+// res_data is the 4-wide residual, [hc * n_embd, n_tokens] row-major on the
+// host, read and written in place.
+TSG_EXPORT int TSGgml_Qwen4ExpFfnBlock(
+    const TSGgmlQwen4ExpFfnArgs* a,
+    void* res_data,
+    int n_embd, int hc, int hc_low_rank, int n_tokens,
+    int n_expert, int n_expert_used, int n_ff, int n_ff_sh,
+    float eps, int cache_slot, int res_resident)
+{
+    try
+    {
+        if (a == nullptr || res_data == nullptr)
+        {
+            set_last_error("qwen4exp FFN block: null args.");
+            return 0;
+        }
+        if (!ensure_backend())
+            return 0;
+
+        const int hc_dim = hc * n_embd;
+        const int T = n_tokens;
+        const std::size_t res_bytes = (std::size_t)hc_dim * T * sizeof(float);
+
+        // Replay: same layer, same shape, same weights.
+        Qwen4ExpFfnCache* slot = (cache_slot >= 0 && cache_slot < kQwen4ExpMaxSlots)
+            ? &g_q4e_ffn[cache_slot] : nullptr;
+        if (slot != nullptr && slot->valid && slot->n_tokens == T && slot->hc_dim == hc_dim
+            && slot->sig == (const void*)a && slot->res_resident == res_resident
+            && q4e_refresh_bindings(slot, ggml_backend_get_device(g_backend)))
+        {
+            if (!res_resident) ggml_backend_tensor_set(slot->res_in, res_data, 0, res_bytes);
+            q4e_note(0, false);
+            if (graph_compute_profiled(g_backend, slot->graph, kQwen4ExpFfnKernel) != GGML_STATUS_SUCCESS)
+            {
+                slot->reset();
+                set_last_error("qwen4exp FFN block: replay failed.");
+                return 0;
+            }
+            if (res_resident)
+            {
+                // Chain on the device: the next layer reads what this one wrote.
+                ggml_backend_tensor_copy(slot->res_out, slot->res_in);
+            }
+            else
+            {
+                ggml_backend_tensor_get(slot->res_out, res_data, 0, res_bytes);
+            }
+            return 1;
+        }
+        if (slot != nullptr) slot->reset();
+
+        ggml_init_params ip{};
+        ip.mem_size = ggml_tensor_overhead() * 512 + ggml_graph_overhead();
+        ip.mem_buffer = nullptr;
+        ip.no_alloc = true;
+        ggml_context* ctx = ggml_init(ip);
+        if (ctx == nullptr)
+        {
+            set_last_error("qwen4exp FFN block: ggml_init failed.");
+            return 0;
+        }
+
+        ggml_tensor* res_in = ggml_new_tensor_2d(ctx, GGML_TYPE_F32, hc_dim, T);
+        ggml_set_input(res_in);
+        if (res_resident)
+        {
+            if (!q4e_res_ensure(res_bytes) ||
+                ggml_backend_tensor_alloc(g_q4e_res_buf, res_in,
+                        ggml_backend_buffer_get_base(g_q4e_res_buf)) != GGML_STATUS_SUCCESS)
+            {
+                ggml_free(ctx);
+                set_last_error("qwen4exp FFN block: failed to bind the residual buffer.");
+                return 0;
+            }
+        }
+
+        Q4eBinder binder{ggml_backend_get_device(g_backend)};
+        ggml_tensor* res_out = q4e_nodes_ffn(ctx, binder, a, res_in,
+                n_embd, hc, hc_low_rank, T, n_expert, n_expert_used, n_ff, n_ff_sh, eps);
+        if (res_out == nullptr)
+        {
+            ggml_free(ctx);
+            set_last_error("qwen4exp FFN block: failed to build the graph.");
+            return 0;
+        }
+        ggml_set_output(res_out);
+
+        ggml_cgraph* graph = ggml_new_graph(ctx);
+        if (q4e_graph_uid_enabled()) graph->uid = q4e_next_graph_uid();
+        ggml_build_forward_expand(graph, res_out);
+
+        ggml_gallocr_t alloc = ggml_gallocr_new(ggml_backend_get_default_buffer_type(g_backend));
+        if (alloc == nullptr || !ggml_gallocr_alloc_graph(alloc, graph))
+        {
+            if (alloc) ggml_gallocr_free(alloc);
+            ggml_free(ctx);
+            set_last_error("qwen4exp FFN block: failed to allocate graph tensors.");
+            return 0;
+        }
+
+        binder.flush();
+        if (!res_resident) ggml_backend_tensor_set(res_in, res_data, 0, res_bytes);
+
+        q4e_note(0, true);
+        if (graph_compute_profiled(g_backend, graph, kQwen4ExpFfnKernel) != GGML_STATUS_SUCCESS)
+        {
+            ggml_gallocr_free(alloc);
+            ggml_free(ctx);
+            set_last_error("qwen4exp FFN block: graph compute failed.");
+            return 0;
+        }
+
+        if (res_resident)
+        {
+            ggml_backend_tensor_copy(res_out, res_in);
+        }
+        else
+        {
+            ggml_backend_tensor_get(res_out, res_data, 0, res_bytes);
+        }
+
+        if (slot != nullptr)
+        {
+            // Hand the graph to the cache. The weight bindings and the gallocr plan
+            // stay valid as long as the descriptor and the shape do, which the
+            // replay check above enforces.
+            slot->ctx = ctx;
+            slot->graph = graph;
+            slot->alloc = alloc;
+            slot->res_in = res_in;
+            slot->res_out = res_out;
+            slot->n_tokens = T;
+            slot->hc_dim = hc_dim;
+            slot->sig = (const void*)a;
+            slot->res_resident = res_resident;
+            slot->rebinds = std::move(binder.cached);
+            slot->valid = true;
+            return 1;
+        }
+
+        ggml_gallocr_free(alloc);
+        ggml_free(ctx);
+        return 1;
+    }
+    catch (const std::exception& e)
+    {
+        set_last_error(std::string("qwen4exp FFN block: ") + e.what());
+        return 0;
+    }
+    catch (...)
+    {
+        set_last_error("qwen4exp FFN block: unknown error.");
+        return 0;
+    }
+}
+
+TSG_EXPORT int TSGgml_Qwen4ExpGdnBlock(
+    const TSGgmlQwen4ExpGdnArgs* a,
+    void* res_data,
+    int n_embd, int hc, int hc_low_rank, int n_tokens,
+    int head_k_dim, int head_v_dim, int n_k_heads, int n_v_heads, int d_conv,
+    float eps, int cache_slot, int res_resident)
+{
+    try
+    {
+        if (a == nullptr || res_data == nullptr) { set_last_error("qwen4exp GDN block: null args."); return 0; }
+        if (!ensure_backend()) return 0;
+
+        const int hc_dim = hc * n_embd;
+        const int T = n_tokens;
+        const int key_dim = head_k_dim * n_k_heads;
+        const int value_dim = head_v_dim * n_v_heads;
+        const int conv_dim = key_dim * 2 + value_dim;
+        const int hist = d_conv - 1;
+        const std::size_t res_bytes = (std::size_t)hc_dim * T * sizeof(float);
+
+        Qwen4ExpFfnCache* slot = (cache_slot >= 0 && cache_slot < kQwen4ExpMaxSlots)
+            ? &g_q4e_gdn[cache_slot] : nullptr;
+        if (slot != nullptr && slot->valid && slot->n_tokens == T && slot->hc_dim == hc_dim
+            && slot->sig == (const void*)a
+            && q4e_refresh_bindings(slot, ggml_backend_get_device(g_backend)))
+        {
+            ggml_backend_tensor_set(slot->res_in, res_data, 0, res_bytes);
+            q4e_note(1, false);
+            if (graph_compute_profiled(g_backend, slot->graph, kQwen4ExpGdnKernel) != GGML_STATUS_SUCCESS)
+            { slot->reset(); set_last_error("qwen4exp GDN block: replay failed."); return 0; }
+            ggml_backend_synchronize(g_backend);
+            if (slot->conv_out != nullptr)
+            {
+                ggml_backend_tensor_copy(slot->conv_out, slot->conv_in);
+                ggml_backend_tensor_copy(slot->ssm_out, slot->ssm_in);
+            }
+            if (res_resident) ggml_backend_tensor_copy(slot->res_out, slot->res_in);
+            else ggml_backend_tensor_get(slot->res_out, res_data, 0, res_bytes);
+            return 1;
+        }
+        // Rebuild the graph; the state buffer below is untouched by that.
+        if (slot != nullptr) slot->reset_graph();
+
+        ggml_init_params ip{};
+        ip.mem_size = ggml_tensor_overhead() * 512 + ggml_graph_overhead();
+        ip.mem_buffer = nullptr;
+        ip.no_alloc = true;
+        ggml_context* ctx = ggml_init(ip);
+        if (ctx == nullptr) { set_last_error("qwen4exp GDN block: ggml_init failed."); return 0; }
+
+        ggml_tensor* res_in = ggml_new_tensor_2d(ctx, GGML_TYPE_F32, hc_dim, T);
+        ggml_set_input(res_in);
+        if (res_resident)
+        {
+            if (!q4e_res_ensure(res_bytes) ||
+                ggml_backend_tensor_alloc(g_q4e_res_buf, res_in,
+                        ggml_backend_buffer_get_base(g_q4e_res_buf)) != GGML_STATUS_SUCCESS)
+            {
+                ggml_free(ctx);
+                set_last_error("qwen4exp GDN block: failed to bind the residual buffer.");
+                return 0;
+            }
+        }
+
+        // The state is read through *_in and written through a SEPARATE *_out, copied
+        // back device-to-device after the graph runs - the per-layer path's proven
+        // dataflow, kept as is. (The token span writes the state in place instead,
+        // with node order sequencing the write behind the read.)
+        ggml_tensor* conv_state = ggml_new_tensor_3d(ctx, GGML_TYPE_F32, hist, conv_dim, 1);
+        ggml_tensor* ssm_state  = ggml_new_tensor_3d(ctx, GGML_TYPE_F32, head_v_dim, head_v_dim, n_v_heads);
+        ggml_tensor* conv_state_out = ggml_new_tensor_3d(ctx, GGML_TYPE_F32, hist, conv_dim, 1);
+        ggml_tensor* ssm_state_out  = ggml_new_tensor_3d(ctx, GGML_TYPE_F32, head_v_dim, head_v_dim, n_v_heads);
+        ggml_set_input(conv_state);      ggml_set_input(ssm_state);
+        ggml_set_output(conv_state_out); ggml_set_output(ssm_state_out);
+
+        Q4eBinder binder{ggml_backend_get_device(g_backend)};
+        Q4eGdnWriteback wb{};
+        ggml_tensor* res_out = q4e_nodes_gdn(ctx, binder, a, res_in,
+                conv_state, ssm_state,
+                n_embd, hc, hc_low_rank, T,
+                head_k_dim, head_v_dim, n_k_heads, n_v_heads, d_conv, eps, &wb);
+        ggml_set_output(res_out);
+
+        ggml_cgraph* graph = ggml_new_graph(ctx);
+        if (q4e_graph_uid_enabled()) graph->uid = q4e_next_graph_uid();
+        ggml_build_forward_expand(graph, res_out);
+        // state write-back rides the same graph
+        ggml_build_forward_expand(graph, ggml_cpy(ctx, wb.tail, conv_state_out));
+        ggml_build_forward_expand(graph, ggml_cpy(ctx, wb.new_state, ssm_state_out));
+
+        // Give the two *_in state tensors their OWN device buffer so gallocr never
+        // owns them and a graph rebuild cannot disturb them. Carrying the state across
+        // a rebuild by copying was the alternative and it did not survive contact:
+        // one fused layer was enough to derail the model.
+        const std::size_t conv_bytes = (std::size_t)hist * conv_dim * sizeof(float);
+        const std::size_t ssm_bytes = (std::size_t)head_v_dim * head_v_dim * n_v_heads * sizeof(float);
+        const std::size_t ssm_off = (conv_bytes + 255) & ~(std::size_t)255;
+        bool zero_state = true;
+        Q4eSeqStateEntry* st = nullptr;
+        if (slot != nullptr)
+        {
+            st = q4e_seq_state(a->conv_state, ssm_off + ssm_bytes);
+            if (st == nullptr)
+            {
+                ggml_free(ctx);
+                set_last_error("qwen4exp GDN block: failed to allocate the state buffer.");
+                return 0;
+            }
+            zero_state = !st->ready;
+            std::uint8_t* base = (std::uint8_t*)ggml_backend_buffer_get_base(st->buf);
+            if (ggml_backend_tensor_alloc(st->buf, conv_state, base) != GGML_STATUS_SUCCESS ||
+                ggml_backend_tensor_alloc(st->buf, ssm_state, base + ssm_off) != GGML_STATUS_SUCCESS)
+            {
+                ggml_free(ctx);
+                set_last_error("qwen4exp GDN block: failed to bind the state buffer.");
+                return 0;
+            }
+        }
+
+        // Seed the state only when the buffer is new; a rebuild keeps what is there.
+        if (zero_state)
+        {
+            binder.upload_list.push_back({conv_state, a->conv_state, conv_bytes});
+            binder.upload_list.push_back({ssm_state, a->ssm_state, ssm_bytes});
+        }
+
+        ggml_gallocr_t alloc = ggml_gallocr_new(ggml_backend_get_default_buffer_type(g_backend));
+        if (alloc == nullptr || !ggml_gallocr_alloc_graph(alloc, graph))
+        {
+            if (alloc) ggml_gallocr_free(alloc);
+            ggml_free(ctx);
+            set_last_error("qwen4exp GDN block: failed to allocate graph tensors.");
+            return 0;
+        }
+
+        binder.flush();
+        if (st != nullptr) st->ready = true;
+        if (!res_resident) ggml_backend_tensor_set(res_in, res_data, 0, res_bytes);
+
+        q4e_note(1, true);
+        if (graph_compute_profiled(g_backend, graph, kQwen4ExpGdnKernel) != GGML_STATUS_SUCCESS)
+        {
+            ggml_gallocr_free(alloc); ggml_free(ctx);
+            set_last_error("qwen4exp GDN block: graph compute failed.");
+            return 0;
+        }
+
+        ggml_backend_synchronize(g_backend);
+        ggml_backend_tensor_copy(conv_state_out, conv_state);
+        ggml_backend_tensor_copy(ssm_state_out, ssm_state);
+        if (res_resident) ggml_backend_tensor_copy(res_out, res_in);
+        else ggml_backend_tensor_get(res_out, res_data, 0, res_bytes);
+
+        if (slot != nullptr)
+        {
+            slot->ctx = ctx; slot->graph = graph; slot->alloc = alloc;
+            slot->res_in = res_in; slot->res_out = res_out;
+            slot->conv_in = conv_state; slot->conv_out = conv_state_out;
+            slot->ssm_in = ssm_state; slot->ssm_out = ssm_state_out;
+            slot->n_tokens = T; slot->hc_dim = hc_dim; slot->sig = (const void*)a;
+            slot->res_resident = res_resident;
+            slot->rebinds = std::move(binder.cached);
+            slot->valid = true;
+            return 1;
+        }
+        ggml_gallocr_free(alloc); ggml_free(ctx);
+        return 1;
+    }
+    catch (const std::exception& e)
+    { set_last_error(std::string("qwen4exp GDN block: ") + e.what()); return 0; }
+    catch (...)
+    { set_last_error("qwen4exp GDN block: unknown error."); return 0; }
+}
+
+// mask_data is [n_kv_pad, T] F16, built host-side: 0 where token t may attend to
+// cell j, -inf otherwise (the C# side pads with the same predicate and stride).
+TSG_EXPORT int TSGgml_Qwen4ExpAttnBlock(
+    const TSGgmlQwen4ExpAttnArgs* a,
+    void* res_data,
+    const void* mask_data,
+    int n_embd, int hc, int hc_low_rank, int n_tokens,
+    int head_dim, int n_head, int n_head_kv, int kv_capacity, int n_kv, int position,
+    int n_rot, float rope_base, float rope_freq_scale, float attn_scale,
+    float eps, int cache_slot, int res_resident)
+{
+    try
+    {
+        if (a == nullptr || res_data == nullptr) { set_last_error("qwen4exp attn block: null args."); return 0; }
+        if (!ensure_backend()) return 0;
+
+        const int hc_dim = hc * n_embd;
+        const int T = n_tokens;
+        const std::size_t res_bytes = (std::size_t)hc_dim * T * sizeof(float);
+        const bool use_flash = q4e_flash_attn_ok(a->kv_type, head_dim);
+        const int n_kv_pad = q4e_pad_kv(n_kv, kv_capacity, use_flash);
+        const std::size_t mask_bytes = (std::size_t)n_kv_pad * T * sizeof(uint16_t);
+
+        Qwen4ExpFfnCache* slot = (cache_slot >= 0 && cache_slot < kQwen4ExpMaxSlots)
+            ? &g_q4e_attn[cache_slot] : nullptr;
+
+        // Keyed on the PADDED width, so the topology only moves once every stride
+        // tokens instead of every token. Position and the KV write row reach the graph
+        // as inputs, so their values change without the shape moving.
+        if (slot != nullptr && slot->valid && slot->n_tokens == T && slot->hc_dim == hc_dim
+            && slot->sig == (const void*)a && slot->res_resident == res_resident
+            && slot->n_kv == n_kv_pad
+            && q4e_refresh_bindings(slot, ggml_backend_get_device(g_backend)))
+        {
+            if (!res_resident) ggml_backend_tensor_set(slot->res_in, res_data, 0, res_bytes);
+            ggml_backend_tensor_set(slot->mask, mask_data, 0, mask_bytes);
+            q4e_set_attn_indices(slot->pos, slot->kv_idx, T, position);
+            q4e_note(2, false);
+            if (graph_compute_profiled(g_backend, slot->graph, kQwen4ExpAttnKernel) != GGML_STATUS_SUCCESS)
+            { slot->reset_graph(); set_last_error("qwen4exp attn block: replay failed."); return 0; }
+            if (res_resident) ggml_backend_tensor_copy(slot->res_out, slot->res_in);
+            else ggml_backend_tensor_get(slot->res_out, res_data, 0, res_bytes);
+            return 1;
+        }
+        if (slot != nullptr) slot->reset_graph();
+
+        ggml_init_params ip{};
+        ip.mem_size = ggml_tensor_overhead() * 512 + ggml_graph_overhead();
+        ip.mem_buffer = nullptr;
+        ip.no_alloc = true;
+        ggml_context* ctx = ggml_init(ip);
+        if (ctx == nullptr) { set_last_error("qwen4exp attn block: ggml_init failed."); return 0; }
+
+        ggml_tensor* res_in = ggml_new_tensor_2d(ctx, GGML_TYPE_F32, hc_dim, T);
+        ggml_set_input(res_in);
+        if (res_resident)
+        {
+            if (!q4e_res_ensure(res_bytes) ||
+                ggml_backend_tensor_alloc(g_q4e_res_buf, res_in,
+                        ggml_backend_buffer_get_base(g_q4e_res_buf)) != GGML_STATUS_SUCCESS)
+            {
+                ggml_free(ctx);
+                set_last_error("qwen4exp attn block: failed to bind the residual buffer.");
+                return 0;
+            }
+        }
+
+        ggml_tensor* mask = ggml_new_tensor_2d(ctx, GGML_TYPE_F16, n_kv_pad, T);
+        ggml_set_input(mask);
+        ggml_tensor* pos = ggml_new_tensor_1d(ctx, GGML_TYPE_I32, T);
+        ggml_set_input(pos);
+        // The rows this step writes, as an INPUT rather than a baked view offset.
+        ggml_tensor* kv_idx = ggml_new_tensor_1d(ctx, GGML_TYPE_I64, T);
+        ggml_set_input(kv_idx);
+
+        // The builder expands the KV write into the graph before we expand the
+        // residual below, so the graph exists first.
+        ggml_cgraph* graph = ggml_new_graph(ctx);
+        if (q4e_graph_uid_enabled()) graph->uid = q4e_next_graph_uid();
+
+        Q4eBinder binder{ggml_backend_get_device(g_backend)};
+        std::vector<ggml_tensor*> kv_tensors;
+        ggml_tensor* res_out = q4e_nodes_attn(ctx, graph, binder, a, res_in,
+                mask, pos, kv_idx,
+                n_embd, hc, hc_low_rank, T,
+                head_dim, n_head, n_head_kv, kv_capacity, n_kv_pad,
+                n_rot, rope_base, rope_freq_scale, attn_scale, eps, use_flash,
+                &kv_tensors);
+        ggml_set_output(res_out);
+        ggml_build_forward_expand(graph, res_out);
+
+        ggml_gallocr_t alloc = ggml_gallocr_new(ggml_backend_get_default_buffer_type(g_backend));
+        if (alloc == nullptr || !ggml_gallocr_alloc_graph(alloc, graph))
+        {
+            if (alloc) ggml_gallocr_free(alloc);
+            ggml_free(ctx);
+            set_last_error("qwen4exp attn block: failed to allocate graph tensors.");
+            return 0;
+        }
+
+        binder.flush();
+        // Same pad-reads-uninitialised-memory hazard as the span; see the comment
+        // there. Zero the device K/V at the start of the sequence.
+        if (position == 0)
+            for (ggml_tensor* t : kv_tensors)
+                ggml_backend_tensor_memset(t, 0, 0, ggml_nbytes(t));
+        if (!res_resident) ggml_backend_tensor_set(res_in, res_data, 0, res_bytes);
+        ggml_backend_tensor_set(mask, mask_data, 0, mask_bytes);
+        q4e_set_attn_indices(pos, kv_idx, T, position);
+
+        q4e_note(2, true);
+        if (graph_compute_profiled(g_backend, graph, kQwen4ExpAttnKernel) != GGML_STATUS_SUCCESS)
+        {
+            ggml_gallocr_free(alloc); ggml_free(ctx);
+            set_last_error("qwen4exp attn block: graph compute failed.");
+            return 0;
+        }
+
+        if (res_resident) ggml_backend_tensor_copy(res_out, res_in);
+        else ggml_backend_tensor_get(res_out, res_data, 0, res_bytes);
+
+        if (slot != nullptr)
+        {
+            slot->ctx = ctx; slot->graph = graph; slot->alloc = alloc;
+            slot->res_in = res_in; slot->res_out = res_out; slot->mask = mask;
+            slot->pos = pos; slot->kv_idx = kv_idx;
+            slot->n_tokens = T; slot->hc_dim = hc_dim; slot->sig = (const void*)a;
+            slot->res_resident = res_resident; slot->n_kv = n_kv_pad;
+            slot->rebinds = std::move(binder.cached);
+            slot->valid = true;
+            return 1;
+        }
+        ggml_gallocr_free(alloc); ggml_free(ctx);
+        return 1;
+    }
+    catch (const std::exception& e)
+    { set_last_error(std::string("qwen4exp attn block: ") + e.what()); return 0; }
+    catch (...)
+    { set_last_error("qwen4exp attn block: unknown error."); return 0; }
+}
+
+// ============================================================================
+// The token span: layers [layer_begin, layer_end) - both halves each - as ONE
+// graph. With the PLE layer the only host interruption, a decode token is two
+// of these calls instead of 96 per-layer ones: one residual upload, one graph
+// launch, one download per span, and a single stable topology for ggml-cuda's
+// CUDA graph capture instead of 96 alternating ones.
+//
+// ffn/gdn/attn are the BASE pointers of the full per-layer descriptor arrays
+// (pinned on the C# side), indexed here by absolute layer id. kinds[il] != 0
+// marks a recurrent (GDN) layer.
+//
+// GDN state lives in the per-layer slots' state buffers (g_q4e_gdn[il]), so
+// the span and the per-layer fallback read and write the SAME state and a
+// fallback mid-sequence stays coherent. The span writes the state in place -
+// cpy(tail -> conv_state) expanded after the nodes that read conv_state, so
+// node order sequences the write behind the read.
+// ============================================================================
+// The PLE block riding inside the span. Only the n-gram hash and the gather
+// from the ~320M-row host table stay on the CPU; the gathered rows arrive as a
+// graph input and the projections, norms, gating, dilated depthwise conv and
+// the residual add all run on the device. The conv history is persistent
+// device state, written in place like the GDN state.
+struct TSGgmlQwen4ExpPleArgs
+{
+    void* key_w;            // [n_embd, hc_dim]
+    void* value_w;          // [n_embd, n_embd]
+    void* norm_key;         // f32 [hc_dim]
+    void* norm_query;       // f32 [hc_dim]
+    void* norm_conv;        // f32 [hc_dim]
+    void* conv1d_t;         // f32 [hc_dim, kern] - tap-major transpose of ple_conv1d
+    void* conv_state;       // f32 [hc_dim, hist] seed (host layout matches)
+
+    long long key_bytes, value_bytes;
+
+    int key_type, value_type;
+    int kern;               // conv kernel taps
+    int dil;                // dilation (the n-gram size)
+};
+
+// The output stage: the final hyper-connection mixer (which IS the output norm -
+// qwen4exp ships no separate one) and the LM head, riding the tail of the last
+// span. The mixer runs on the LAST token only - at prefill the managed path used
+// to mix every position and throw all but one away.
+struct TSGgmlQwen4ExpHeadArgs
+{
+    void* hc_norm;          // f32 [hc_dim]
+    void* hc_down;          // [hc_dim, hc_low_rank]
+    void* hc_up;            // [hc_low_rank, hc_dim]
+    void* head;             // [n_embd, vocab]
+
+    long long hc_down_bytes, hc_up_bytes, head_bytes;
+
+    int hc_down_type, hc_up_type, head_type;
+    int vocab;
+};
+
+TSG_EXPORT int TSGgml_Qwen4ExpTokenSpan(
+    const TSGgmlQwen4ExpFfnArgs* ffn,
+    const TSGgmlQwen4ExpGdnArgs* gdn,
+    const TSGgmlQwen4ExpAttnArgs* attn,
+    const unsigned char* kinds,
+    int layer_begin, int layer_end,
+    void* res_data,
+    const void* mask_data,
+    int n_embd, int hc, int hc_low_rank, int n_tokens,
+    int head_k_dim, int head_v_dim, int n_k_heads, int n_v_heads, int d_conv,
+    int head_dim, int n_head, int n_head_kv, int kv_capacity, int n_kv, int position,
+    int n_rot, float rope_base, float rope_freq_scale, float attn_scale,
+    int n_expert, int n_expert_used, int n_ff, int n_ff_sh,
+    float eps, int cache_slot, int first_ffn_only,
+    const TSGgmlQwen4ExpHeadArgs* head, void* logits_out,
+    const TSGgmlQwen4ExpPleArgs* ple, int ple_layer, const void* ple_emb,
+    const int* mrope_pos, const int* mrope_sections, int rope_position,
+    int device)
+{
+    try
+    {
+        if (ffn == nullptr || gdn == nullptr || attn == nullptr || kinds == nullptr
+            || res_data == nullptr || layer_begin < 0 || layer_end <= layer_begin
+            || layer_end > kQwen4ExpMaxSlots)
+        {
+            set_last_error("qwen4exp token span: bad args.");
+            return 0;
+        }
+        // LAYER SPLIT: run this span's layers on their own GPU. Everything the
+        // span touches - the persisted graph slot, the resident weight copies,
+        // the KV device copies, the GDN/PLE state buffers, the residual buffer -
+        // is selected by the active rank, so the scope is the whole mechanism.
+        tsg::ScopedRank q4e_rank(q4e_resolve_device(device));
+        if (!ensure_backend()) return 0;
+        if ((head != nullptr) != (logits_out != nullptr))
+        {
+            set_last_error("qwen4exp token span: head and logits_out come together.");
+            return 0;
+        }
+        const bool has_ple = ple != nullptr && ple_layer >= layer_begin && ple_layer < layer_end;
+        const bool use_mrope = mrope_pos != nullptr && mrope_sections != nullptr;
+        if (has_ple && ple_emb == nullptr)
+        {
+            set_last_error("qwen4exp token span: the PLE block needs the gathered rows.");
+            return 0;
+        }
+
+        const int hc_dim = hc * n_embd;
+        const int T = n_tokens;
+        const std::size_t res_bytes = (std::size_t)hc_dim * T * sizeof(float);
+
+        bool has_attn = false;
+        for (int il = layer_begin; il < layer_end; ++il)
+        {
+            if (il == layer_begin && first_ffn_only != 0) continue;
+            if (kinds[il] == 0) { has_attn = true; break; }
+        }
+        if (has_attn && mask_data == nullptr)
+        {
+            set_last_error("qwen4exp token span: an attention layer needs the mask.");
+            return 0;
+        }
+
+        // One padded window for every attention layer in the span; they share one
+        // mask, one position tensor and one write-row tensor.
+        bool use_flash = false;
+        int n_kv_pad = 0;
+        if (has_attn)
+        {
+            int first_attn = (first_ffn_only != 0) ? layer_begin + 1 : layer_begin;
+            while (kinds[first_attn] != 0) ++first_attn;
+            use_flash = q4e_flash_attn_ok(attn[first_attn].kv_type, head_dim);
+            n_kv_pad = q4e_pad_kv(n_kv, kv_capacity, use_flash);
+        }
+        const std::size_t mask_bytes = (std::size_t)n_kv_pad * T * sizeof(uint16_t);
+
+        Qwen4ExpFfnCache* slot = (cache_slot >= 0 && cache_slot < kQwen4ExpSpanSlots)
+            ? &g_q4e_span[cache_slot] : nullptr;
+        if (slot == nullptr)
+        {
+            set_last_error("qwen4exp token span: bad cache slot.");
+            return 0;
+        }
+
+        // Replay: same span, same shape, same descriptors, same padded window - and
+        // every cache-bound weight still where the graph believes it is.
+        if (!q4e_span_force_rebuild()
+            && slot->valid
+            && q4e_refresh_bindings(slot, ggml_backend_get_device(g_backend))
+            && slot->n_tokens == T && slot->hc_dim == hc_dim
+            && slot->sig == (const void*)ffn && slot->sig2 == (const void*)gdn
+            && slot->sig3 == (const void*)attn
+            && slot->layer_begin == layer_begin && slot->layer_end == layer_end
+            && slot->kv_capacity == kv_capacity && slot->n_kv == n_kv_pad
+            && slot->first_ffn_only == first_ffn_only
+            && slot->sig4 == (const void*)head
+            && slot->sig5 == (const void*)(has_ple ? ple : nullptr)
+            && slot->use_mrope == (use_mrope ? 1 : 0))
+        {
+            ggml_backend_tensor_set(slot->res_in, res_data, 0, res_bytes);
+            if (slot->ple_emb_in != nullptr)
+                ggml_backend_tensor_set(slot->ple_emb_in, ple_emb, 0,
+                        (std::size_t)n_embd * T * sizeof(float));
+            for (ggml_tensor* m : slot->span_masks)
+                ggml_backend_tensor_set(m, mask_data, 0, mask_bytes);
+            for (std::size_t i = 0; i < slot->span_pos.size(); ++i)
+                q4e_set_attn_indices(slot->span_pos[i], slot->span_kvidx[i], T, position,
+                        use_mrope ? (const int32_t*)mrope_pos : nullptr, rope_position);
+            q4e_note(3, false);
+            if (q4e_span_trace() && !slot->span_copies.empty() && T == 1)
+            {
+                // What the graph will actually read as its recurrent state, sampled
+                // immediately before the compute.
+                std::vector<float> pb;
+                fprintf(stderr, "[q4e-prestate] slot%d pos=%d:", cache_slot, position);
+                for (std::size_t i = 0; i < slot->span_copies.size(); ++i)
+                {
+                    ggml_tensor* st = slot->span_copies[i].second;
+                    pb.resize((std::size_t)ggml_nelements(st));
+                    ggml_backend_tensor_get(st, pb.data(), 0, ggml_nbytes(st));
+                    double n2 = 0.0;
+                    for (float f : pb) n2 += (double)f * f;
+                    fprintf(stderr, " %.9e", std::sqrt(n2));
+                }
+                fprintf(stderr, "%c", 10);
+            }
+            if (graph_compute_profiled(g_backend, slot->graph, kQwen4ExpSpanKernel) != GGML_STATUS_SUCCESS)
+            {
+                slot->reset_graph();
+                set_last_error("qwen4exp token span: replay failed.");
+                return 0;
+            }
+            if (slot->logits != nullptr)
+            {
+                for (const auto& c : slot->span_copies)
+                    ggml_backend_tensor_copy(c.first, c.second);
+                ggml_backend_tensor_get(slot->logits, logits_out, 0,
+                        (std::size_t)head->vocab * sizeof(float));
+                q4e_trace_state(slot, "replay", position);
+                return 1;
+            }
+            // The synchronize is LOAD-BEARING: ggml_backend_tensor_copy issues a
+            // legacy-stream memcpy, and ggml-cuda's compute stream is non-blocking,
+            // so without the drain the copy can read conv_out/ssm_out while the
+            // graph is still writing them. That race is timing-dependent - short
+            // contexts kept the GPU caught up and hid it; long contexts queue
+            // deeper and the copy wins, which corrupted the recurrent state from
+            // the first replay after a long prefill.
+            if (!slot->span_copies.empty())
+                ggml_backend_synchronize(g_backend);
+            for (const auto& c : slot->span_copies)
+                ggml_backend_tensor_copy(c.first, c.second);
+            ggml_backend_tensor_get(slot->res_out, res_data, 0, res_bytes);
+            q4e_trace_probe(slot, "replay", position);
+            q4e_trace_state(slot, "replay", position);
+            return 1;
+        }
+        slot->reset_graph();
+
+        const double t0 = q4e_phase_log() ? q4e_now_ms() : 0.0;
+        const int n_layers = layer_end - layer_begin;
+        ggml_init_params ip{};
+        ip.mem_size = ggml_tensor_overhead() * ((std::size_t)n_layers * 256 + 1024)
+                    + ggml_graph_overhead_custom(kQwen4ExpSpanGraphSize, false);
+        ip.mem_buffer = nullptr;
+        ip.no_alloc = true;
+        ggml_context* ctx = ggml_init(ip);
+        if (ctx == nullptr) { set_last_error("qwen4exp token span: ggml_init failed."); return 0; }
+
+        ggml_tensor* res_in = ggml_new_tensor_2d(ctx, GGML_TYPE_F32, hc_dim, T);
+        ggml_set_input(res_in);
+
+        // ONE mask, position and write-row tensor shared by every attention layer
+        // in the span - same values for all, so three uploads a replay instead of
+        // three dozen. (Private per-layer copies were briefly used to bisect the
+        // leaf-free bug; the shared tensors were never at fault.)
+        ggml_tensor* mask = nullptr;
+        ggml_tensor* pos = nullptr;
+        ggml_tensor* kv_idx = nullptr;
+        if (has_attn)
+        {
+            mask = ggml_new_tensor_2d(ctx, GGML_TYPE_F16, n_kv_pad, T);
+            ggml_set_input(mask);
+            pos = ggml_new_tensor_1d(ctx, GGML_TYPE_I32, use_mrope ? 4 * T : T);
+            ggml_set_input(pos);
+            kv_idx = ggml_new_tensor_1d(ctx, GGML_TYPE_I64, T);
+            ggml_set_input(kv_idx);
+            slot->span_masks.push_back(mask);
+            slot->span_pos.push_back(pos);
+            slot->span_kvidx.push_back(kv_idx);
+        }
+
+        ggml_tensor* ple_emb_in = nullptr;
+        if (has_ple)
+        {
+            ple_emb_in = ggml_new_tensor_2d(ctx, GGML_TYPE_F32, n_embd, T);
+            ggml_set_input(ple_emb_in);
+        }
+
+        ggml_cgraph* graph = ggml_new_graph_custom(ctx, kQwen4ExpSpanGraphSize, false);
+        if (q4e_graph_uid_enabled()) graph->uid = q4e_next_graph_uid();
+
+        Q4eBinder binder{ggml_backend_get_device(g_backend)};
+
+        const std::size_t conv_dim = (std::size_t)(head_k_dim * n_k_heads) * 2
+                                   + (std::size_t)(head_v_dim * n_v_heads);
+        const std::size_t conv_bytes = (std::size_t)(d_conv - 1) * conv_dim * sizeof(float);
+        const std::size_t ssm_bytes = (std::size_t)head_v_dim * head_v_dim * n_v_heads * sizeof(float);
+        const std::size_t ssm_off = (conv_bytes + 255) & ~(std::size_t)255;
+
+        std::vector<Qwen4ExpFfnCache*> seeded;              // unused since the seq-state map; kept for shape
+        std::vector<Q4eSeqStateEntry*> seeded_states;
+        std::vector<ggml_tensor*> kv_tensors;
+        std::vector<ggml_tensor*> trace_res;
+        std::vector<ggml_tensor*> probe_nodes;
+        int attn_seen = 0;
+        ggml_tensor* res = res_in;
+        bool failed = false;
+        const char* fail_what = nullptr;
+
+        for (int il = layer_begin; il < layer_end && !failed; ++il)
+        {
+            if (has_ple && il == ple_layer)
+            {
+                // ---- the PLE block, ahead of this layer's halves ----
+                const int hc_dim2 = hc_dim;
+                const int kern = ple->kern;
+                const int dil = ple->dil;
+                const int hist = (kern - 1) * dil;
+
+                ggml_tensor* w_key   = ggml_new_tensor_2d(ctx, (ggml_type)ple->key_type, n_embd, hc_dim2);
+                ggml_tensor* w_value = ggml_new_tensor_2d(ctx, (ggml_type)ple->value_type, n_embd, n_embd);
+                ggml_tensor* w_nk    = ggml_new_tensor_1d(ctx, GGML_TYPE_F32, hc_dim2);
+                ggml_tensor* w_nq    = ggml_new_tensor_1d(ctx, GGML_TYPE_F32, hc_dim2);
+                ggml_tensor* w_nc    = ggml_new_tensor_1d(ctx, GGML_TYPE_F32, hc_dim2);
+                ggml_tensor* w_ct    = ggml_new_tensor_2d(ctx, GGML_TYPE_F32, hc_dim2, kern);
+
+                // conv history: persistent device state, like the GDN state.
+                ggml_tensor* conv_state = nullptr;
+                Q4eSeqStateEntry* ple_st = nullptr;
+                if (hist > 0)
+                {
+                    const std::size_t st_bytes = (std::size_t)hist * hc_dim2 * sizeof(float);
+                    ple_st = q4e_seq_state(ple->conv_state, st_bytes);
+                    if (ple_st == nullptr)
+                    { failed = true; fail_what = "ple state alloc"; break; }
+                    conv_state = ggml_new_tensor_2d(ctx, GGML_TYPE_F32, hc_dim2, hist);
+                    ggml_set_input(conv_state);
+                    if (ggml_backend_tensor_alloc(ple_st->buf, conv_state,
+                            ggml_backend_buffer_get_base(ple_st->buf)) != GGML_STATUS_SUCCESS)
+                    { failed = true; fail_what = "ple state bind"; break; }
+                    if (!ple_st->ready)
+                    {
+                        binder.upload_list.push_back({conv_state, ple->conv_state,
+                                (std::size_t)hist * hc_dim2 * sizeof(float)});
+                        seeded_states.push_back(ple_st);
+                    }
+                }
+
+                // key/query grouped norms: normalise each stream, scale the full row.
+                auto gnorm = [&](ggml_tensor* x, ggml_tensor* w) {
+                    ggml_tensor* x3 = ggml_reshape_3d(ctx, x, n_embd, hc, T);
+                    ggml_tensor* nx = ggml_reshape_2d(ctx, ggml_rms_norm(ctx, x3, eps), hc_dim2, T);
+                    return ggml_mul(ctx, nx, w);
+                };
+
+                ggml_tensor* keyn = gnorm(ggml_mul_mat(ctx, w_key, ple_emb_in), w_nk);   // [hc_dim, T]
+                ggml_tensor* qryn = gnorm(res, w_nq);
+
+                // Per-stream dot, scaled, signed-sqrt, sigmoid: the PLE gate.
+                ggml_tensor* prod = ggml_reshape_3d(ctx, ggml_mul(ctx, keyn, qryn), n_embd, hc, T);
+                ggml_tensor* sdot = ggml_scale(ctx, ggml_sum_rows(ctx, prod),
+                        1.0f / std::sqrt((float)n_embd));                                 // [1, hc, T]
+                ggml_tensor* sg   = ggml_sgn(ctx, sdot);
+                ggml_tensor* mag  = ggml_sqrt(ctx, ggml_clamp(ctx, ggml_mul(ctx, sdot, sg),
+                        1e-6f, 3.0e38f));
+                ggml_tensor* gate = ggml_sigmoid(ctx, ggml_mul(ctx, sg, mag));            // [1, hc, T]
+
+                ggml_tensor* val  = ggml_mul_mat(ctx, w_value, ple_emb_in);               // [n_embd, T]
+                ggml_tensor* v3   = ggml_repeat_4d(ctx,
+                        ggml_reshape_3d(ctx, val, n_embd, 1, T), n_embd, hc, T, 1);
+                ggml_tensor* gated = ggml_reshape_2d(ctx, ggml_mul(ctx, v3, gate), hc_dim2, T);
+
+                // Dilated causal depthwise conv over the conv-normed gate output.
+                ggml_tensor* normc = gnorm(gated, w_nc);
+                ggml_tensor* padded = (conv_state != nullptr)
+                        ? ggml_concat(ctx, conv_state, normc, 1)                          // [hc_dim, hist+T]
+                        : normc;
+                ggml_tensor* acc = nullptr;
+                for (int kk = 0; kk < kern; ++kk)
+                {
+                    // tap kk reads (kern-1-kk) dilated positions back: with hist rows
+                    // of history in front, that is a plain offset of kk*dil rows.
+                    ggml_tensor* slice = ggml_view_2d(ctx, padded, hc_dim2, T,
+                            padded->nb[1], (std::size_t)(kk * dil) * padded->nb[1]);
+                    ggml_tensor* wk = ggml_view_2d(ctx, w_ct, hc_dim2, 1,
+                            w_ct->nb[1], (std::size_t)kk * w_ct->nb[1]);
+                    ggml_tensor* term = ggml_mul(ctx, slice, wk);
+                    acc = (acc == nullptr) ? term : ggml_add(ctx, acc, term);
+                }
+                ggml_tensor* conv = ggml_silu(ctx, acc);                                  // [hc_dim, T]
+
+                // res += gated + conv
+                res = ggml_add(ctx, ggml_add(ctx, res, gated), conv);
+                ggml_build_forward_expand(graph, res);
+                if (conv_state != nullptr)
+                {
+                    // Keep the last `hist` rows for the next batch; ordered after
+                    // every read of the state by the expand above.
+                    ggml_tensor* tail = ggml_view_2d(ctx, padded, hc_dim2, hist,
+                            padded->nb[1], (std::size_t)T * padded->nb[1]);
+                    ggml_build_forward_expand(graph, ggml_cpy(ctx, tail, conv_state));
+                }
+
+                binder.add(w_key, ple->key_w, (std::size_t)ple->key_bytes);
+                binder.add(w_value, ple->value_w, (std::size_t)ple->value_bytes);
+                binder.add(w_nk, ple->norm_key, (std::size_t)hc_dim2 * sizeof(float));
+                binder.add(w_nq, ple->norm_query, (std::size_t)hc_dim2 * sizeof(float));
+                binder.add(w_nc, ple->norm_conv, (std::size_t)hc_dim2 * sizeof(float));
+                binder.add(w_ct, ple->conv1d_t, (std::size_t)hc_dim2 * kern * sizeof(float));
+            }
+
+            if (il == layer_begin && first_ffn_only != 0)
+            {
+                // The attention half of this layer already ran per-layer; the span
+                // picks up from its FFN half below.
+            }
+            else if (kinds[il] != 0)
+            {
+                // ---- recurrent half ----
+                Q4eSeqStateEntry* gst = q4e_seq_state(gdn[il].conv_state, ssm_off + ssm_bytes);
+                if (gst == nullptr)
+                { failed = true; fail_what = "state buffer alloc"; break; }
+                ggml_tensor* conv_state = ggml_new_tensor_3d(ctx, GGML_TYPE_F32,
+                        d_conv - 1, (int64_t)conv_dim, 1);
+                ggml_tensor* ssm_state = ggml_new_tensor_3d(ctx, GGML_TYPE_F32,
+                        head_v_dim, head_v_dim, n_v_heads);
+                ggml_set_input(conv_state);
+                ggml_set_input(ssm_state);
+                std::uint8_t* base = (std::uint8_t*)ggml_backend_buffer_get_base(gst->buf);
+                if (ggml_backend_tensor_alloc(gst->buf, conv_state, base) != GGML_STATUS_SUCCESS ||
+                    ggml_backend_tensor_alloc(gst->buf, ssm_state, base + ssm_off) != GGML_STATUS_SUCCESS)
+                { failed = true; fail_what = "state buffer bind"; break; }
+                if (!gst->ready)
+                {
+                    binder.upload_list.push_back({conv_state, gdn[il].conv_state, conv_bytes});
+                    binder.upload_list.push_back({ssm_state, gdn[il].ssm_state, ssm_bytes});
+                    seeded_states.push_back(gst);
+                }
+
+                Q4eGdnWriteback wb{};
+                res = q4e_nodes_gdn(ctx, binder, &gdn[il], res,
+                        conv_state, ssm_state,
+                        n_embd, hc, hc_low_rank, T,
+                        head_k_dim, head_v_dim, n_k_heads, n_v_heads, d_conv, eps, &wb,
+                        (q4e_span_trace() && il == layer_begin && cache_slot == 0 && T == 1)
+                            ? &slot->gdn_probe : nullptr);
+                // The residual first: its tree holds every node that READS the state
+                // (the concat, the delta-net input). Then the write-back, so node
+                // order puts the write strictly after the read.
+                ggml_build_forward_expand(graph, res);
+                if (q4e_span_state_in_graph())
+                {
+                    ggml_build_forward_expand(graph, ggml_cpy(ctx, wb.tail, conv_state));
+                    ggml_build_forward_expand(graph, ggml_cpy(ctx, wb.new_state, ssm_state));
+                }
+                else
+                {
+                    ggml_tensor* conv_out = ggml_new_tensor_3d(ctx, GGML_TYPE_F32,
+                            d_conv - 1, (int64_t)conv_dim, 1);
+                    ggml_tensor* ssm_out = ggml_new_tensor_3d(ctx, GGML_TYPE_F32,
+                            head_v_dim, head_v_dim, n_v_heads);
+                    ggml_set_output(conv_out);
+                    ggml_set_output(ssm_out);
+                    ggml_build_forward_expand(graph, ggml_cpy(ctx, wb.tail, conv_out));
+                    ggml_build_forward_expand(graph, ggml_cpy(ctx, wb.new_state, ssm_out));
+                    slot->span_copies.push_back({conv_out, conv_state});
+                    slot->span_copies.push_back({ssm_out, ssm_state});
+                }
+            }
+            else
+            {
+                // ---- attention half (expands its own KV write first) ----
+                bool fa_here = use_flash && (attn_seen < q4e_span_fa_max());
+                ++attn_seen;
+                res = q4e_nodes_attn(ctx, graph, binder, &attn[il], res,
+                        mask, pos, kv_idx,
+                        n_embd, hc, hc_low_rank, T,
+                        head_dim, n_head, n_head_kv, kv_capacity, n_kv_pad,
+                        n_rot, rope_base, rope_freq_scale, attn_scale, eps, fa_here,
+                        &kv_tensors,
+                        (q4e_span_trace() && attn_seen == 1 && T == 1) ? &probe_nodes : nullptr,
+                        use_mrope ? (const int32_t*)mrope_sections : nullptr);
+                ggml_build_forward_expand(graph, res);
+            }
+            if (q4e_span_trace()) { ggml_set_output(res); trace_res.push_back(res); }
+
+            // ---- FFN half ----
+            res = q4e_nodes_ffn(ctx, binder, &ffn[il], res,
+                    n_embd, hc, hc_low_rank, T,
+                    n_expert, n_expert_used, n_ff, n_ff_sh, eps);
+            ggml_build_forward_expand(graph, res);
+            if (q4e_span_trace()) { ggml_set_output(res); trace_res.push_back(res); }
+        }
+
+        if (failed)
+        {
+            ggml_free(ctx);
+            set_last_error(std::string("qwen4exp token span: ") + (fail_what ? fail_what : "build") + " failed.");
+            return 0;
+        }
+
+        const double t_nodes = q4e_phase_log() ? q4e_now_ms() : 0.0;
+        ggml_tensor* res_out = res;
+        ggml_tensor* logits = nullptr;
+        if (head != nullptr)
+        {
+            // ---- final mixer on the LAST token only, then the LM head ----
+            ggml_tensor* w_fnorm = ggml_new_tensor_1d(ctx, GGML_TYPE_F32, hc_dim);
+            ggml_tensor* w_fdown = ggml_new_tensor_2d(ctx, (ggml_type)head->hc_down_type, hc_dim, hc_low_rank);
+            ggml_tensor* w_fup   = ggml_new_tensor_2d(ctx, (ggml_type)head->hc_up_type, hc_low_rank, hc_dim);
+            ggml_tensor* w_head  = ggml_new_tensor_2d(ctx, (ggml_type)head->head_type, n_embd, head->vocab);
+
+            ggml_tensor* last = ggml_view_2d(ctx, res_out, hc_dim, 1,
+                    res_out->nb[1], (std::size_t)(T - 1) * res_out->nb[1]);
+            ggml_tensor* res3f = ggml_reshape_3d(ctx, ggml_cont(ctx, last), n_embd, hc, 1);
+            ggml_tensor* xnf = ggml_mul(ctx,
+                    ggml_reshape_2d(ctx, ggml_rms_norm(ctx, res3f, eps), hc_dim, 1), w_fnorm);
+            ggml_tensor* lof = ggml_silu(ctx, ggml_scale(ctx,
+                    ggml_mul_mat(ctx, w_fdown, xnf), 1.0f / (float)hc));
+            ggml_tensor* gtf = ggml_sigmoid(ctx, ggml_mul_mat(ctx, w_fup, lof));
+            ggml_tensor* gatedf = ggml_reshape_3d(ctx, ggml_mul(ctx, xnf, gtf), n_embd, hc, 1);
+            ggml_tensor* mixedf = ggml_cont(ctx, ggml_view_2d(ctx, gatedf, n_embd, 1,
+                    ggml_row_size(gatedf->type, n_embd) * hc, 0));
+            for (int c = 1; c < hc; ++c)
+                mixedf = ggml_add(ctx, mixedf, ggml_view_2d(ctx, gatedf, n_embd, 1,
+                        ggml_row_size(gatedf->type, n_embd) * hc,
+                        ggml_row_size(gatedf->type, n_embd) * c));
+            mixedf = ggml_scale(ctx, mixedf, 1.0f / (float)hc);
+
+            logits = ggml_mul_mat(ctx, w_head, mixedf);       // [vocab, 1]
+            ggml_set_output(logits);
+            ggml_build_forward_expand(graph, logits);
+
+            binder.add(w_fnorm, head->hc_norm, (std::size_t)hc_dim * sizeof(float));
+            binder.add(w_fdown, head->hc_down, (std::size_t)head->hc_down_bytes);
+            binder.add(w_fup, head->hc_up, (std::size_t)head->hc_up_bytes);
+            binder.add(w_head, head->head, (std::size_t)head->head_bytes);
+        }
+        else
+        {
+            ggml_set_output(res_out);
+        }
+
+        const double t_pregal = q4e_phase_log() ? q4e_now_ms() : 0.0;
+        ggml_gallocr_t alloc = ggml_gallocr_new(ggml_backend_get_default_buffer_type(g_backend));
+        if (alloc == nullptr || !ggml_gallocr_alloc_graph(alloc, graph))
+        {
+            if (alloc) ggml_gallocr_free(alloc);
+            ggml_free(ctx);
+            set_last_error("qwen4exp token span: failed to allocate graph tensors.");
+            return 0;
+        }
+        const double t_gal = q4e_phase_log() ? q4e_now_ms() : 0.0;
+
+        binder.flush();
+        for (Q4eSeqStateEntry* e : seeded_states) e->ready = true;
+        if (ple_emb_in != nullptr)
+            ggml_backend_tensor_set(ple_emb_in, ple_emb, 0,
+                    (std::size_t)n_embd * T * sizeof(float));
+
+        // A fresh sequence starts with a KV cache whose device copy holds whatever
+        // was already in that memory - the host buffer it uploads from is a pooled
+        // allocation with no zero guarantee. The window this graph reads is padded
+        // past the rows any token has written, and a masked-off column contributes
+        // nothing only if it is FINITE: an Inf in a never-written K row makes its
+        // score Inf, Inf plus the -inf mask is NaN, and one NaN takes the whole
+        // softmax row (flash attention alike). Zero the device copies at the start
+        // of the sequence so the pad always reads zeros.
+        if (position == 0)
+            for (ggml_tensor* t : kv_tensors)
+                ggml_backend_tensor_memset(t, 0, 0, ggml_nbytes(t));
+
+        ggml_backend_tensor_set(res_in, res_data, 0, res_bytes);
+        for (ggml_tensor* m : slot->span_masks)
+            ggml_backend_tensor_set(m, mask_data, 0, mask_bytes);
+        for (std::size_t i = 0; i < slot->span_pos.size(); ++i)
+            q4e_set_attn_indices(slot->span_pos[i], slot->span_kvidx[i], T, position,
+                    use_mrope ? (const int32_t*)mrope_pos : nullptr, rope_position);
+
+        const double t_up = q4e_phase_log() ? q4e_now_ms() : 0.0;
+        q4e_note(3, true);
+        if (graph_compute_profiled(g_backend, graph, kQwen4ExpSpanKernel) != GGML_STATUS_SUCCESS)
+        {
+            ggml_gallocr_free(alloc); ggml_free(ctx);
+            set_last_error("qwen4exp token span: graph compute failed.");
+            return 0;
+        }
+        if (q4e_phase_log() && T > 1)
+        {
+            ggml_backend_synchronize(g_backend);
+            const double t_cmp = q4e_now_ms();
+            fprintf(stderr, "[q4e-phase] span %d..%d T=%d: nodes=%.1fms binder=%.1fms gallocr=%.1fms uploads=%.1fms compute=%.1fms%c",
+                    layer_begin, layer_end, T,
+                    t_nodes - t0, t_pregal - t_nodes, t_gal - t_pregal, t_up - t_gal, t_cmp - t_up, 10);
+        }
+
+        // See the replay path: the drain before the copies is load-bearing.
+        if (!slot->span_copies.empty())
+            ggml_backend_synchronize(g_backend);
+        for (const auto& c : slot->span_copies)
+            ggml_backend_tensor_copy(c.first, c.second);
+        if (logits != nullptr)
+            ggml_backend_tensor_get(logits, logits_out, 0, (std::size_t)head->vocab * sizeof(float));
+        else
+            ggml_backend_tensor_get(res_out, res_data, 0, res_bytes);
+        q4e_trace_probe(slot, "build", position);
+        q4e_trace_state(slot, "build", position);
+
+        if (!probe_nodes.empty())
+        {
+            static const char* names[] = { "scores", "probs", "attn" };
+            for (std::size_t i = 0; i < probe_nodes.size() && i < 3; ++i)
+            {
+                ggml_tensor* t = probe_nodes[i];
+                std::vector<float> pb((std::size_t)ggml_nelements(t));
+                ggml_backend_tensor_get(t, pb.data(), 0, ggml_nbytes(t));
+                double n2 = 0.0; float amax = 0.0f; std::size_t iamax = 0; bool bad = false;
+                for (std::size_t j = 0; j < pb.size(); ++j)
+                {
+                    float f = pb[j];
+                    n2 += (double)f * f;
+                    if (std::fabs(f) > amax) { amax = std::fabs(f); iamax = j; }
+                    if (!std::isfinite(f)) bad = true;
+                }
+                fprintf(stderr, "[q4e-probe] %s ne=[%d,%d,%d] l2=%.9e amax=%.6e@%zu head=%.6e %.6e %.6e%s%c",
+                        names[i], (int)t->ne[0], (int)t->ne[1], (int)t->ne[2],
+                        std::sqrt(n2), amax, iamax,
+                        pb.size() > 0 ? pb[0] : 0.0f, pb.size() > 1 ? pb[1] : 0.0f,
+                        pb.size() > 52 ? pb[52] : 0.0f, bad ? " NAN" : "", 10);
+            }
+        }
+        if (q4e_span_trace() && has_attn && T == 1)
+        {
+            // Read back the K/V pad rows this graph could see. A masked-off pad
+            // column only contributes nothing if its K row is exactly zero.
+            std::vector<uint16_t> kv_buf;
+            for (std::size_t t = 0; t < kv_tensors.size(); ++t)
+            {
+                ggml_tensor* kc = kv_tensors[t];
+                const std::size_t total = ggml_nbytes(kc);
+                kv_buf.resize(total / 2);
+                ggml_backend_tensor_get(kc, kv_buf.data(), 0, total);
+                // [head_dim, capacity, kvH]; pad rows are n_kv..n_kv_pad of dim 1.
+                int max_bits = 0; long nonzero = 0;
+                for (int64_t h = 0; h < kc->ne[2]; ++h)
+                    for (int64_t r = n_kv; r < n_kv_pad; ++r)
+                    {
+                        const uint16_t* row = kv_buf.data()
+                            + (h * kc->ne[1] + r) * kc->ne[0];
+                        for (int64_t c = 0; c < kc->ne[0]; ++c)
+                        {
+                            int m = row[c] & 0x7FFF;   // f16 magnitude bits
+                            if (m != 0) { ++nonzero; if (m > max_bits) max_bits = m; }
+                        }
+                    }
+                if (nonzero > 0)
+                    fprintf(stderr, "[q4e-kvpad] tensor %zu: %ld nonzero pad values, max f16 bits 0x%04x (rows %d..%d)%c",
+                            t, nonzero, (unsigned)max_bits, n_kv, n_kv_pad, 10);
+            }
+            fprintf(stderr, "[q4e-kvpad] scan done (%zu tensors)%c", kv_tensors.size(), 10);
+        }
+        if (q4e_span_trace())
+        {
+            std::vector<float> buf((std::size_t)hc_dim * T);
+            fprintf(stderr, "[q4e-trace] span %d..%d T=%d pos=%d:", layer_begin, layer_end, T, position);
+            for (std::size_t i = 0; i < trace_res.size(); ++i)
+            {
+                ggml_backend_tensor_get(trace_res[i], buf.data(), 0, buf.size() * sizeof(float));
+                double n2 = 0.0; bool bad = false;
+                for (float f : buf) { n2 += (double)f * f; if (!std::isfinite(f)) bad = true; }
+                fprintf(stderr, " %zu:%.4e%s", i, std::sqrt(n2), bad ? "!NAN" : "");
+            }
+            fprintf(stderr, "\n");
+        }
+
+        slot->ctx = ctx; slot->graph = graph; slot->alloc = alloc;
+        slot->res_in = res_in; slot->res_out = res_out;
+        slot->n_tokens = T; slot->hc_dim = hc_dim;
+        slot->sig = (const void*)ffn; slot->sig2 = (const void*)gdn; slot->sig3 = (const void*)attn;
+        slot->sig4 = (const void*)head; slot->logits = logits;
+        slot->sig5 = (const void*)(has_ple ? ple : nullptr);
+        slot->ple_emb_in = ple_emb_in;
+        slot->layer_begin = layer_begin; slot->layer_end = layer_end;
+        slot->kv_capacity = kv_capacity; slot->n_kv = n_kv_pad;
+        slot->first_ffn_only = first_ffn_only;
+        slot->use_mrope = use_mrope ? 1 : 0;
+        slot->rebinds = std::move(binder.cached);
+        slot->res_resident = 0;
+        slot->valid = true;
+        return 1;
+    }
+    catch (const std::exception& e)
+    { set_last_error(std::string("qwen4exp token span: ") + e.what()); return 0; }
+    catch (...)
+    { set_last_error("qwen4exp token span: unknown error."); return 0; }
+}
+
+// Copy the residual to / from the device-resident buffer. The op-by-op attention
+// half still works on the host, so it brackets itself with these.
+TSG_EXPORT int TSGgml_Qwen4ExpResUpload(const void* data, long long bytes)
+{
+    try
+    {
+        if (!ensure_backend() || data == nullptr || bytes <= 0) return 0;
+        if (!q4e_res_ensure((std::size_t)bytes)) return 0;
+        ggml_backend_tensor_set(g_q4e_res, data, 0, (std::size_t)bytes);
+        return 1;
+    }
+    catch (...) { set_last_error("qwen4exp residual upload failed."); return 0; }
+}
+
+TSG_EXPORT int TSGgml_Qwen4ExpResDownload(void* data, long long bytes)
+{
+    try
+    {
+        if (!ensure_backend() || data == nullptr || bytes <= 0) return 0;
+        if (g_q4e_res == nullptr || g_q4e_res_capacity < (std::size_t)bytes) return 0;
+        ggml_backend_synchronize(g_backend);
+        ggml_backend_tensor_get(g_q4e_res, data, 0, (std::size_t)bytes);
+        return 1;
+    }
+    catch (...) { set_last_error("qwen4exp residual download failed."); return 0; }
+}
+
+TSG_EXPORT void TSGgml_Qwen4ExpResetFfnCache()
+{
+    // Sweep every initialized device. Under a layer split the token's layers are
+    // spread across GPUs, each with its own graph slots and residual buffer;
+    // resetting only the active rank would leave live graphs elsewhere pointing
+    // at state the caller believes it has dropped.
+    const int ndev = tsg::g_device_count.load(std::memory_order_acquire);
+    for (int d = 0; d < ndev && d < tsg::TSG_MAX_DEVICES; ++d)
+    {
+        tsg::ScopedRank rank(d);
+        for (int i = 0; i < kQwen4ExpMaxSlots; ++i)
+        {
+            g_q4e_ffn[i].reset();
+            g_q4e_gdn[i].reset();
+            g_q4e_attn[i].reset();
+        }
+        for (int i = 0; i < kQwen4ExpSpanSlots; ++i)
+            g_q4e_span[i].reset();
+        if (g_q4e_res_ctx) { ggml_free(g_q4e_res_ctx); g_q4e_res_ctx = nullptr; }
+        if (g_q4e_res_buf) { ggml_backend_buffer_free(g_q4e_res_buf); g_q4e_res_buf = nullptr; }
+        g_q4e_res = nullptr; g_q4e_res_capacity = 0;
+    }
+}
+
+/// Mark one sequence-state entry (keyed by its host seed pointer) as needing a
+/// re-seed on the next graph build. The buffer and its baked addresses stay
+/// valid; only the one-time upload re-arms. Used by the managed reset, whose
+/// host copies are freshly zeroed.
+TSG_EXPORT void TSGgml_Qwen4ExpInvalidateSeqState(const void* key)
+{
+    const int ndev = tsg::g_device_count.load(std::memory_order_acquire);
+    for (int d = 0; d < ndev && d < tsg::TSG_MAX_DEVICES; ++d)
+    {
+        tsg::ScopedRank rank(d);
+        auto it = g_q4e_seq_state.find(key);
+        if (it != g_q4e_seq_state.end()) it->second.ready = false;
+    }
+}
+
+/// Free EVERY sequence-state entry and drop every cached graph. Called on
+/// model dispose so a later model load in the same process can never collide
+/// with stale entries keyed on recycled host addresses.
+TSG_EXPORT void TSGgml_Qwen4ExpReleaseAllSeqState()
+{
+    const int ndev = tsg::g_device_count.load(std::memory_order_acquire);
+    for (int d = 0; d < ndev && d < tsg::TSG_MAX_DEVICES; ++d)
+    {
+        tsg::ScopedRank rank(d);
+        for (auto& kv : g_q4e_seq_state)
+            if (kv.second.buf) ggml_backend_buffer_free(kv.second.buf);
+        g_q4e_seq_state.clear();
+    }
+    TSGgml_Qwen4ExpResetFfnCache();
+}
+
+/// Free the sequence-state entries for a released sequence holder and drop
+/// every cached graph (graphs bake state-buffer addresses; surviving holders
+/// rebuild on their next token and re-bind their still-alive entries without
+/// re-seeding).
+TSG_EXPORT void TSGgml_Qwen4ExpReleaseSeqState(const void* const* keys, int n)
+{
+    bool freed = false;
+    const int ndev = tsg::g_device_count.load(std::memory_order_acquire);
+    for (int d = 0; d < ndev && d < tsg::TSG_MAX_DEVICES; ++d)
+    {
+        tsg::ScopedRank rank(d);
+        for (int i = 0; i < n; ++i)
+        {
+            auto it = g_q4e_seq_state.find(keys[i]);
+            if (it == g_q4e_seq_state.end()) continue;
+            if (it->second.buf) ggml_backend_buffer_free(it->second.buf);
+            g_q4e_seq_state.erase(it);
+            freed = true;
+        }
+    }
+    if (freed)
+        TSGgml_Qwen4ExpResetFfnCache();
+}
+
+} // extern "C"
diff --git a/TensorSharp.GGML.Native/ggml_ops_tensor_parallel.cpp b/TensorSharp.GGML.Native/ggml_ops_tensor_parallel.cpp
index 9e22fcf4..e80a6f2a 100644
--- a/TensorSharp.GGML.Native/ggml_ops_tensor_parallel.cpp
+++ b/TensorSharp.GGML.Native/ggml_ops_tensor_parallel.cpp
@@ -1310,7 +1310,8 @@ TSG_EXPORT int TSGgml_SetNativeEnvironmentVariable(const char* name, const char*
 // Bring up `count` backends on the given physical device indices. Rank 0 reuses
 // the already-initialized singleton when its device matches, so a TP run does
 // not pay for a second context on the main GPU. Returns 1 on success.
-TSG_EXPORT int TSGgml_TensorParallelInit(int backendType, const int* deviceIndices, int count, int concurrentRanks)
+static int tsg_multi_device_init(int backendType, const int* deviceIndices, int count,
+                                 int concurrentRanks, bool enableCollectives)
 {
     try
     {
@@ -1442,8 +1443,11 @@ TSG_EXPORT int TSGgml_TensorParallelInit(int backendType, const int* deviceIndic
 
         tsg::g_device_count.store(count, std::memory_order_release);
         // Resolve the collective backend eagerly so an NCCL/P2P init failure is
-        // reported at startup rather than mid-decode.
-        if (count > 1)
+        // reported at startup rather than mid-decode. A LAYER SPLIT skips this:
+        // it never reduces across devices, and bringing NCCL up anyway would
+        // spend the startup time and take on the lying-P2P hang risk for a
+        // collective that is never issued.
+        if (count > 1 && enableCollectives)
         {
             const bool device_ar = tsg::tp_comm_ensure();
             std::fprintf(stderr,
@@ -1451,6 +1455,12 @@ TSG_EXPORT int TSGgml_TensorParallelInit(int backendType, const int* deviceIndic
                 count, device_ar ? "device (backend collective)" : "host");
             std::fflush(stderr);
         }
+        else if (count > 1)
+        {
+            std::fprintf(stderr,
+                "[GGML] multi-device (layer split): %d device(s), no collectives\n", count);
+            std::fflush(stderr);
+        }
         return 1;
     }
     catch (const std::exception& ex)
@@ -1460,6 +1470,22 @@ TSG_EXPORT int TSGgml_TensorParallelInit(int backendType, const int* deviceIndic
     }
 }
 
+TSG_EXPORT int TSGgml_TensorParallelInit(int backendType, const int* deviceIndices, int count, int concurrentRanks)
+{
+    return tsg_multi_device_init(backendType, deviceIndices, count, concurrentRanks, true);
+}
+
+// Bring up one backend per device WITHOUT a cross-device collective, for the
+// layer-split path: each GPU owns a contiguous run of layers and the only thing
+// that crosses a device boundary is the residual, which the caller hands over
+// through host memory. Same device bring-up as TensorParallelInit in every other
+// respect, so ranks, the per-device resident weight caches and the VRAM budgets
+// all work identically.
+TSG_EXPORT int TSGgml_MultiDeviceInit(int backendType, const int* deviceIndices, int count)
+{
+    return tsg_multi_device_init(backendType, deviceIndices, count, /*concurrentRanks*/ 1, false);
+}
+
 // 1 when the fused tensor-parallel path can run (several ranks spanning every
 // initialized device). The per-layer partials reduce with the backend's device
 // collective when it has one (ggml-cuda: NCCL / P2P) and through host staging
diff --git a/TensorSharp.GGML.Native/ggml_ops_transformer_common.h b/TensorSharp.GGML.Native/ggml_ops_transformer_common.h
index dd7cfd97..db22fbb3 100644
--- a/TensorSharp.GGML.Native/ggml_ops_transformer_common.h
+++ b/TensorSharp.GGML.Native/ggml_ops_transformer_common.h
@@ -480,6 +480,14 @@ struct TSGgmlQwen35LayerDesc
     void* shexp_up_w;        // [hidden, shared_ff]
     void* shexp_down_w;      // [shared_ff, hidden]
     void* shexp_gate_inp_w;  // [hidden] F32 (shared-expert sigmoid gate)
+    // Dense FFN with gate and up UNFUSED. A mixed-quant "UD"/dynamic GGUF can
+    // store ffn_gate and ffn_up in different types (IQ2_XS vs IQ2_S, ...), which
+    // a single fused tensor cannot represent and which no imatrix-free
+    // requantization can reconcile. Those layers keep both tensors as they were
+    // quantized and the graph runs two matmuls instead of one. Non-null exactly
+    // when gu_w is null.
+    void* ffn_gate_w;        // ffn_gate [hidden, ff_dense]
+    void* ffn_up_w;          // ffn_up   [hidden, ff_dense]
 
     // --- int64 weight shapes/bytes ---
     std::int64_t qkv_ne0, qkv_ne1, qkv_bytes;
@@ -498,6 +506,8 @@ struct TSGgmlQwen35LayerDesc
     std::int64_t shexp_gate_ne0, shexp_gate_ne1, shexp_gate_bytes;
     std::int64_t shexp_up_ne0, shexp_up_ne1, shexp_up_bytes;
     std::int64_t shexp_down_ne0, shexp_down_ne1, shexp_down_bytes;
+    std::int64_t ffn_gate_ne0, ffn_gate_ne1, ffn_gate_bytes;
+    std::int64_t ffn_up_ne0, ffn_up_ne1, ffn_up_bytes;
 
     // --- int32 scalars ---
     std::int32_t struct_bytes;
@@ -514,6 +524,7 @@ struct TSGgmlQwen35LayerDesc
     // in system RAM. The whole-model decode graph then omits their mul_mat_id
     // chain, pauses after the router, and lets the host multiply them.
     std::int32_t cpu_moe;
+    std::int32_t ffn_gate_type, ffn_up_type;
 };
 
 // Per-layer descriptor for the GPT-OSS whole-model decode kernel
diff --git a/TensorSharp.Models/ModelBase.cs b/TensorSharp.Models/ModelBase.cs
index e213c8a0..0b3bdc06 100644
--- a/TensorSharp.Models/ModelBase.cs
+++ b/TensorSharp.Models/ModelBase.cs
@@ -83,6 +83,30 @@ private QuantizedWeight(IntPtr data, long rawBytes, int ggmlType, long ne0, long
             _ownerToken = ownerToken;
         }
 
+        /// <summary>
+        /// A non-owning view over ONE expert of a stacked expert tensor
+        /// (<c>ffn_gate_exps</c> and friends), so a model whose GGUF stacks its
+        /// experts can still drive the per-expert linear paths that unstacked files
+        /// get for free. The view borrows the stack's buffer - it must not free it -
+        /// and holds the stack as its owner token so the memory outlives it.
+        /// </summary>
+        public static QuantizedWeight CreateExpertView(StackedExpertWeights stacked, int expert)
+        {
+            if (stacked == null)
+                throw new ArgumentNullException(nameof(stacked));
+            if (expert < 0 || expert >= stacked.NumExperts)
+                throw new ArgumentOutOfRangeException(nameof(expert));
+
+            return new QuantizedWeight(
+                stacked.Data + (nint)(expert * stacked.PerExpertRawBytes),
+                stacked.PerExpertRawBytes,
+                stacked.GgmlType,
+                stacked.PerExpertNe0,
+                stacked.PerExpertNe1,
+                ownsBuffer: false,
+                ownerToken: stacked);
+        }
+
         public void Dispose()
         {
             ReleaseHostData();
@@ -370,7 +394,7 @@ public StackedExpertWeights(
         }
     }
 
-    public abstract class ModelBase : IModelArchitecture
+    public abstract partial class ModelBase : IModelArchitecture
     {
         public ModelConfig Config { get; protected set; }
         public ITokenizer Tokenizer { get; protected set; }
@@ -516,8 +540,23 @@ public virtual void PrepareForPrefill(int requiredContextTokens) { }
         protected int _forwardCount;
         protected Stopwatch _forwardSw = new Stopwatch();
 
-        protected ModelBase(string ggufPath, BackendType backend, int tpDegree = 1, ITensorParallelGroup tpGroup = null)
+        /// <summary>
+        /// Number of GPUs this model spreads its LAYERS across (1 = single GPU).
+        ///
+        /// This is llama.cpp's <c>--split-mode layer</c>, not tensor parallelism:
+        /// each GPU owns a contiguous run of whole layers, nothing is sharded and
+        /// no collective is ever issued - only the residual crosses a device
+        /// boundary, and it does so through host memory. It is a CAPACITY feature
+        /// (measured on 2xA100 with llama.cpp: +10% prefill, +0.5% decode when the
+        /// model already fits on one GPU), so the win is running a model, context
+        /// or resident-weight set that one GPU cannot hold.
+        /// </summary>
+        protected int LayerSplitDegree { get; }
+
+        protected ModelBase(string ggufPath, BackendType backend, int tpDegree = 1,
+            ITensorParallelGroup tpGroup = null, int layerSplitDegree = 1)
         {
+            LayerSplitDegree = Math.Max(1, layerSplitDegree);
             _backend = backend;
             // The pure-C# CPU backend must never touch native (ggml P/Invoke) dequant — route
             // every dequant/row-size through the managed implementation (bit-exact vs native,
@@ -549,9 +588,25 @@ protected ModelBase(string ggufPath, BackendType backend, int tpDegree = 1, ITen
                     // A caller-supplied group (multi-node) already owns the
                     // multi-GPU context; reuse it rather than initializing the
                     // devices a second time.
-                    _ggmlContext = FindGgmlContext(_tpGroup) ?? CreateGgmlContext(ggmlType, tpDegree);
-                    _tpGroup ??= CreateGgmlTpGroup(_ggmlContext);
-                    _allocator = _tpGroup != null ? _tpGroup.GetAllocator(0) : new GgmlAllocator(_ggmlContext, 0);
+                    if (LayerSplitDegree > 1)
+                    {
+                        // LAYER SPLIT: one backend per GPU, NO tensor-parallel group.
+                        // _tpGroup must stay null - IsTensorParallel gates the weight
+                        // sharding and AllReduce machinery, none of which applies here,
+                        // and leaving it set would also make the startup banner claim a
+                        // transport that is never used.
+                        _ggmlContext = CreateGgmlContext(ggmlType, LayerSplitDegree, enableCollectives: false);
+                        _allocator = new GgmlAllocator(_ggmlContext, 0);
+                    }
+                    else
+                    {
+                        // A caller-supplied group (multi-node) already owns the
+                        // multi-GPU context; reuse it rather than initializing the
+                        // devices a second time.
+                        _ggmlContext = FindGgmlContext(_tpGroup) ?? CreateGgmlContext(ggmlType, tpDegree);
+                        _tpGroup ??= CreateGgmlTpGroup(_ggmlContext);
+                        _allocator = _tpGroup != null ? _tpGroup.GetAllocator(0) : new GgmlAllocator(_ggmlContext, 0);
+                    }
                     break;
                 }
                 case BackendType.Cuda:
@@ -578,7 +633,8 @@ protected ModelBase(string ggufPath, BackendType backend, int tpDegree = 1, ITen
         /// TENSORSHARP_TP_DEVICES (e.g. "0,2") to pick specific GPUs, which is how
         /// you avoid a display-attached or otherwise busy card.
         /// </summary>
-        private static GgmlContext CreateGgmlContext(GgmlBackendType backendType, int tpDegree)
+        private static GgmlContext CreateGgmlContext(GgmlBackendType backendType, int tpDegree,
+            bool enableCollectives = true)
         {
             if (tpDegree <= 1)
                 return new GgmlContext(new[] { 0 }, backendType);
@@ -588,9 +644,9 @@ private static GgmlContext CreateGgmlContext(GgmlBackendType backendType, int tp
             if (available < tpDegree)
             {
                 throw new InvalidOperationException(
-                    $"Requested tensor-parallel degree {tpDegree} but the GGML {backendType} backend sees only {available} GPU(s).");
+                    $"Requested {tpDegree} GPU(s) but the GGML {backendType} backend sees only {available}.");
             }
-            return new GgmlContext(devices, backendType);
+            return new GgmlContext(devices, backendType, enableCollectives);
         }
 
         private static int[] ParseTpDevices(int tpDegree)
@@ -930,11 +986,25 @@ internal static int ResolvePrefillWarmupTargetLength(
 
         protected void InitializeCacheTensor(Tensor tensor)
         {
-            // First allocation still zero-fills on every backend that keeps a host
-            // copy (including Vulkan/Metal): the fused kernels' flash-padding may
-            // read never-written cache rows, which must be finite.
-            if (tensor != null && (ShouldZeroFillCacheTensors ||
-                _backend == BackendType.GgmlVulkan || _backend == BackendType.GgmlMetal))
+            // ALLOCATION-time zero, on every backend that can do it. The fused
+            // decode kernels read a flash/attention window padded past the rows any
+            // token has written; a masked-off column contributes nothing only if
+            // its K row is FINITE, and an Inf in never-written memory plus the
+            // -inf mask is NaN - which takes the whole softmax row, then every
+            // logit, then argmax, which returns token 0 forever.
+            //
+            // GgmlCuda used to be excluded here (via ShouldZeroFillCacheTensors)
+            // and got recycled, uncleared pool blocks instead. That is the same
+            // defect in ten model families at once: every KV grow past the initial
+            // capacity, and every freshly-allocated per-request cache, could come
+            // up non-finite. The perf argument for skipping the fill belongs to
+            // ResetCacheTensor (per REQUEST, potentially multi-GB), not here -
+            // this runs once per cache allocation, next to the allocation itself.
+            //
+            // Mlx stays excluded: its Fill goes through MlxNative.Full, whose
+            // behaviour for block-quantized KV dtypes is unverified on this
+            // machine, and no Mlx-specific failure of this kind has been observed.
+            if (tensor != null && _backend != BackendType.Mlx)
                 Ops.Fill(tensor, 0f);
         }
 
@@ -1532,6 +1602,7 @@ protected void PrepareCudaQuantizedWeightsForInference()
                     mappedHostViews++;
             }
 
+            int activeRank = 0;
             foreach (var kv in _quantWeights)
             {
                 string weightName = kv.Key;
@@ -1549,6 +1620,23 @@ protected void PrepareCudaQuantizedWeightsForInference()
                 if (!ShouldPreloadCudaQuantWeightToDevice(weightName))
                     continue;
 
+                // LAYER SPLIT: upload this weight to the GPU that owns its layer.
+                // Exactly one rank, because ReleaseHostData() below frees the host
+                // copy - a second preload elsewhere would upload from freed memory.
+                // Weights that are NOT preloaded (the stacked experts, vetoed by
+                // ShouldPreloadCudaQuantWeightToDevice) keep their host views and are
+                // bound lazily by the native binder on whichever rank is active when
+                // their layer runs, so they distribute across the GPUs for free.
+                if (LayerSplitDegree > 1)
+                {
+                    int rank = PreloadRankForWeight(weightName);
+                    if (rank != activeRank)
+                    {
+                        GgmlBasicOps.SetActiveRank(rank);
+                        activeRank = rank;
+                    }
+                }
+
                 // llama.cpp keeps token_embd on the host (its CPU_Mapped model
                 // buffer): embedding lookup is a row gather, and when the quant
                 // type has no device get_rows kernel Embedding() always serves it
@@ -1587,6 +1675,9 @@ protected void PrepareCudaQuantizedWeightsForInference()
                 }
             }
 
+            if (activeRank != 0)
+                GgmlBasicOps.SetActiveRank(0);
+
             if (mappedHostViews == 0)
                 _gguf?.Dispose();
             _cudaQuantWeightsPrepared = true;
@@ -1958,7 +2049,7 @@ private void PrepareDirectCudaQuantizedWeightsForInference()
         private static readonly bool s_retainAllHostQuantWeights =
             Environment.GetEnvironmentVariable("TS_GGML_RETAIN_HOST_WEIGHTS") == "1";
 
-        private static bool ShouldRetainCudaHostQuantWeight(string weightName)
+        protected virtual bool ShouldRetainCudaHostQuantWeight(string weightName)
         {
             return s_retainAllHostQuantWeights ||
                 string.Equals(weightName, "token_embd.weight", StringComparison.Ordinal) ||
@@ -1977,6 +2068,14 @@ private static bool ShouldRetainCudaHostQuantWeight(string weightName)
         /// expert belonging to a <c>--n-cpu-moe</c> layer is multiplied on the host
         /// and uploading it would spend exactly the VRAM the flag exists to save.
         /// </summary>
+        /// <summary>
+        /// GPU that should hold <paramref name="weightName"/> under a layer split.
+        /// Default 0 (everything on the first GPU); a model that splits overrides
+        /// this to return the rank owning the weight's layer. Only consulted when
+        /// <see cref="LayerSplitDegree"/> &gt; 1.
+        /// </summary>
+        protected virtual int PreloadRankForWeight(string weightName) => 0;
+
         protected virtual bool ShouldPreloadCudaQuantWeightToDevice(string weightName)
             => !MoeCpuOffloadConfig.IsOffloadedExpertWeightName(weightName);
 
@@ -2105,12 +2204,30 @@ protected unsafe void PopulateQuantizedRows(Tensor result, QuantizedWeight weigh
             InvalidateTensorDeviceCache(result);
         }
 
+        /// <summary>
+        /// True when this model's FFN can run <c>ffn_gate</c> and <c>ffn_up</c> as
+        /// two separate projections - i.e. when <see cref="FuseGateUpWeights"/> is
+        /// allowed to leave a layer unfused.
+        ///
+        /// Default false, deliberately: most families look up
+        /// <c>blk.N.ffn_gate_up.weight</c> unconditionally and would either throw
+        /// or (worse) bind a null weight and produce silent garbage. Mixed-IQ "UD"
+        /// GGUFs make that reachable, so a family that has not implemented the
+        /// split path must say so at load time rather than fail later.
+        /// </summary>
+        protected virtual bool SupportsSplitGateUpFfn => false;
+
         protected unsafe void FuseGateUpWeights(int numLayers = 0)
         {
             if (numLayers <= 0)
                 numLayers = Config.NumLayers;
             int fused = 0;
             int requantized = 0;
+            // "layer:gateType+upType" - the type pair is the only place the
+            // mismatch is ever surfaced, and it is what a future GGUF tripping a
+            // DIFFERENT combination has to be diagnosed from.
+            var splitLayers = new List<string>();
+            var requantLayers = new List<string>();
             for (int l = 0; l < numLayers; l++)
             {
                 string gateName = $"blk.{l}.ffn_gate.weight";
@@ -2132,18 +2249,31 @@ protected unsafe void FuseGateUpWeights(int numLayers = 0)
                         requant = TryRequantizeForFusion(gw, uw, out bool requantIsGate);
                         if (requant == null)
                         {
-                            Console.WriteLine(
-                                $"  WARNING: layer {l} ffn_gate ({(Runtime.GgmlTensorType)(uint)gw.GgmlType}) and ffn_up " +
-                                $"({(Runtime.GgmlTensorType)(uint)uw.GgmlType}) quant types differ and requantization is " +
-                                "unavailable; gate/up left unfused.");
+                            // ggml refuses to quantize INTO IQ2_XXS / IQ2_XS /
+                            // IQ1_S without an importance matrix
+                            // (ggml_quantize_requires_imatrix), so a layer whose gate
+                            // and up are both such types cannot be brought to a
+                            // common type at load time. Collect and report ONCE below
+                            // - ten per-layer WARNINGs read like ten problems - and
+                            // let the report say whether this family can actually run
+                            // the two projections separately (SupportsSplitGateUpFfn).
+                            splitLayers.Add(
+                                $"{l}:{(Runtime.GgmlTensorType)(uint)gw.GgmlType}+" +
+                                $"{(Runtime.GgmlTensorType)(uint)uw.GgmlType}");
                             continue;
                         }
                         if (requantIsGate) gateSrc = requant; else upSrc = requant;
                         requantized++;
+                        requantLayers.Add(
+                            $"{l}:{(Runtime.GgmlTensorType)(uint)gw.GgmlType}+" +
+                            $"{(Runtime.GgmlTensorType)(uint)uw.GgmlType}->" +
+                            $"{(Runtime.GgmlTensorType)(uint)requant.GgmlType}");
                     }
 
-                    // Gate-up fusion must always succeed: model FFN code expects
-                    // a single fused tensor at guName. If MLX view-fusion fails
+                    // Where fusion IS possible it must produce a tensor at guName.
+                    // (It is not always possible - see the split path above - and the
+                    // FFN of every model that can load such a GGUF handles a missing
+                    // guName by running gate and up separately.) If MLX view-fusion fails
                     // (gate/up not contiguous in the GGUF file), fall back to a
                     // copy. Cost is bounded — 2 tensors × per-layer, host memory
                     // released after the MLX device upload.
@@ -2175,6 +2305,37 @@ protected unsafe void FuseGateUpWeights(int numLayers = 0)
                 Console.WriteLine(requantized > 0
                     ? $"  Fused projections: {fused} Gate+Up ({requantized} mixed-quant layers requantized to a common type)"
                     : $"  Fused projections: {fused} Gate+Up");
+            if (requantLayers.Count > 0)
+            {
+                // A dequantize+requantize of already-lossy weights is a real
+                // (small) quality and VRAM change. It used to happen silently.
+                Console.WriteLine(
+                    $"    Requantized to fuse: {string.Join(", ", requantLayers)}");
+            }
+            if (splitLayers.Count > 0)
+            {
+                // ggml refuses to quantize INTO IQ2_XXS / IQ2_XS / IQ1_S without an
+                // importance matrix, so a layer whose gate and up are BOTH such
+                // types cannot be brought to a common type at load time.
+                if (SupportsSplitGateUpFfn)
+                {
+                    Console.WriteLine(
+                        $"  Split projections: {splitLayers.Count} of {numLayers} layers keep separate " +
+                        "ffn_gate/ffn_up (mixed IQ quant types that would need an importance matrix to " +
+                        "requantize). This model runs them as two matmuls instead of one, with identical " +
+                        "output - no action needed.");
+                }
+                else
+                {
+                    Console.Error.WriteLine(
+                        $"  WARNING: {splitLayers.Count} of {numLayers} layers have mixed-IQ ffn_gate/ffn_up that " +
+                        "cannot be fused (requantizing into IQ2_XXS/IQ2_XS/IQ1_S needs an importance matrix), and " +
+                        $"this architecture ({Config?.Architecture ?? "unknown"}) has no split-FFN path. Those layers " +
+                        "have no usable FFN weight and generation will be wrong or will fail. Use a GGUF whose " +
+                        "ffn_gate and ffn_up share a quant type - most non-UD quants do.");
+                }
+                Console.WriteLine($"    Layers: {string.Join(", ", splitLayers)}");
+            }
         }
 
         /// <summary>
@@ -6294,8 +6455,9 @@ public static ModelBase Create(string ggufPath, BackendType backend, int tpDegre
             arch ??= "qwen3";
 
             ApplyArchitectureNativeTunables(arch, backend, probe);
+            tpDegree = ResolveTensorParallelSupport(arch, backend, tpDegree, ref tpGroup, out int layerSplit);
 
-            return arch switch
+            ModelBase model = arch switch
             {
                 // qwen2vl is Qwen2/Qwen2.5-VL. Its language model is Qwen3's block
                 // with a QKV bias and no QK norm, both of which Qwen3Model detects
@@ -6303,7 +6465,7 @@ public static ModelBase Create(string ggufPath, BackendType backend, int tpDegre
                 // RoPE when the t/h/w position components are equal, which they are
                 // for text tokens, so the vision tower (mmproj) is not required.
                 "qwen3" or "qwen2" or "qwen2vl" or "qwen2_vl" => new Qwen3Model(ggufPath, backend, tpDegree, tpGroup),
-                "qwen35" or "qwen35moe" or "qwen3next" => new Qwen35Model(ggufPath, backend, tpDegree, tpGroup),
+                "qwen35" or "qwen35moe" or "qwen3next" => new Qwen35Model(ggufPath, backend, tpDegree, tpGroup, draftModelPath),
                 "gemma3" => new Gemma3Model(ggufPath, backend, tpDegree, tpGroup),
                 "gemma4" => new Gemma4Model(ggufPath, backend, tpDegree, tpGroup),
                 "diffusion-gemma" or "diffusion_gemma" => new DiffusionGemmaModel(ggufPath, backend),
@@ -6315,10 +6477,127 @@ public static ModelBase Create(string ggufPath, BackendType backend, int tpDegre
                 "mistral3" => new Mistral3Model(ggufPath, backend, tpDegree, tpGroup),
                 "muse-glimmer" or "muse_glimmer" => new MuseGlimmerModel(ggufPath, backend, tpDegree, tpGroup, draftModelPath),
                 "deepseek4" => new DeepSeek4Model(ggufPath, backend, tpDegree, tpGroup, draftModelPath),
+                // Qwen3.8-Flash-Next: hyper-connections, PLE n-gram embeddings, Qwen
+                // Sparse Attention and Gated DeltaNet over a 512-expert MoE.
+                "qwen4exp" => new Qwen4ExpModel(ggufPath, backend, tpDegree, tpGroup, layerSplit),
                 // GLM-5.x with DeepSeek Sparse Attention (MLA + lightning indexer + sigmoid MoE).
                 "glm-dsa" or "glm_dsa" => new GlmDsaModel(ggufPath, backend, tpDegree, tpGroup),
+                // GLM-5.3-Flash: hybrid KDA linear attention + nope-only MLA with a
+                // pooled DSA indexer, Sinkhorn hyper-connections, 288-expert MoE.
+                "glm5next" => new GlmDsaModel(ggufPath, backend, tpDegree, tpGroup),
                 _ => throw new NotSupportedException($"Unsupported architecture: {arch}"),
             };
+
+            model.WarnIfTensorParallelShardedNothing(arch);
+            return model;
+        }
+
+        /// <summary>
+        /// Architectures that accept a <c>tpDegree</c> parameter but implement no
+        /// weight sharding, so tensor parallelism would load the whole model on
+        /// rank 0 and leave the other GPUs holding nothing but a CUDA context and
+        /// NCCL buffers.
+        ///
+        /// This list exists because TP is opt-in PER MODEL CLASS (each TP-capable
+        /// ctor shards behind <c>if (IsTensorParallel)</c>), while the flag is
+        /// accepted globally. An architecture that never wrote that code got a
+        /// real 2-GPU context, a real NCCL comm, and the banner "Tensor
+        /// parallelism (GGML Cuda): 2 GPUs" - and then ran entirely on GPU 0, with
+        /// nothing anywhere saying so. Silence plus a banner asserting the
+        /// opposite is the worst possible outcome; be explicit instead.
+        ///
+        /// Keep in sync with the ctors: an architecture belongs here exactly when
+        /// nothing under its Models/ directory references <c>IsTensorParallel</c>
+        /// or <c>_tpGroup</c>. (DeepSeek V4 is deliberately absent - it drives
+        /// multiple GPUs through its own executor, sized by TS_DSV4_NGPU, not
+        /// through the shared TP group.)
+        /// </summary>
+        internal static readonly Dictionary<string, string> ArchitecturesWithoutTensorParallel =
+            new(StringComparer.OrdinalIgnoreCase)
+            {
+                ["qwen4exp"] =
+                    "qwen4exp (Qwen3.8-Flash-Next) has no tensor-parallel path: none of its weights are " +
+                    "sharded, and its decode is one persisted single-device GGML graph per token whose " +
+                    "GDN/PLE recurrent state lives in device buffers owned by a single backend.",
+            };
+
+        /// <summary>
+        /// Architectures that use several GPUs by LAYER SPLIT instead - each GPU
+        /// owns a contiguous run of whole layers, nothing is sharded and no
+        /// collective is issued. This is what llama.cpp does by default
+        /// (<c>--split-mode layer</c>); for these architectures it is also the ONLY
+        /// multi-GPU mode llama.cpp offers, since <c>-sm row</c> refuses to load them.
+        ///
+        /// <c>--tp N</c> is honoured as "use N GPUs" for these, because that is what
+        /// an operator asking for N GPUs means; the startup line says which mode
+        /// actually ran so nobody has to infer it from nvidia-smi.
+        /// </summary>
+        internal static readonly HashSet<string> ArchitecturesWithLayerSplit =
+            new(StringComparer.OrdinalIgnoreCase) { "qwen4exp" };
+
+        /// <summary>
+        /// Decide whether the requested tensor-parallel degree can actually be
+        /// honoured for <paramref name="arch"/>. A single-node request degrades to
+        /// one GPU with a loud explanation (so an existing <c>--tp N</c> script
+        /// keeps working, just honestly); a DISTRIBUTED group throws, because one
+        /// node quietly dropping to a single rank desynchronises the collective.
+        /// </summary>
+        internal static int ResolveTensorParallelSupport(string arch, BackendType backend, int tpDegree,
+            ref ITensorParallelGroup tpGroup, out int layerSplitDegree)
+        {
+            layerSplitDegree = 1;
+            bool wantsTp = tpDegree > 1 || tpGroup != null;
+            if (!wantsTp)
+                return tpDegree;
+            if (!ArchitecturesWithoutTensorParallel.TryGetValue(arch ?? string.Empty, out string why))
+                return tpDegree;
+
+            if (tpGroup != null)
+            {
+                throw new NotSupportedException(
+                    why + " A distributed tensor-parallel group cannot be downgraded on one node without " +
+                    "desynchronising the others, so this run is refused. Start the node without --tp-node-id/--tp-peers.");
+            }
+
+            // No sharding, but the architecture can still spread its LAYERS across
+            // the GPUs. That is what an operator asking for N GPUs wants, and it is
+            // the same mode llama.cpp uses for these models, so honour --tp N as a
+            // layer split rather than throwing the second GPU away.
+            bool splitCapable = ArchitecturesWithLayerSplit.Contains(arch ?? string.Empty)
+                && (backend == BackendType.GgmlCuda || backend == BackendType.GgmlVulkan);
+            if (splitCapable)
+            {
+                layerSplitDegree = tpDegree;
+                Console.WriteLine(
+                    $"  Multi-GPU: {tpDegree} GPUs by LAYER SPLIT (each GPU holds a contiguous run of whole " +
+                    "layers), not tensor parallelism - this architecture shards no weights. Same mode " +
+                    "llama.cpp uses for it. This raises capacity; it is not expected to raise decode speed.");
+                return 1;
+            }
+
+            Console.Error.WriteLine(
+                $"WARNING: --tp {tpDegree} ignored. {why} Running on ONE GPU; the extra GPUs would have been " +
+                "given a CUDA context and NCCL buffers and then left idle. To choose WHICH GPU, set " +
+                "CUDA_VISIBLE_DEVICES (e.g. CUDA_VISIBLE_DEVICES=1).");
+            return 1;
+        }
+
+        /// <summary>
+        /// Backstop for the next architecture that lands without TP: tensor
+        /// parallelism was requested and the group is live, yet the model sharded
+        /// no weights at all, so every rank but 0 is idle. Costs one dictionary
+        /// count and only ever runs on a TP load.
+        /// </summary>
+        private void WarnIfTensorParallelShardedNothing(string arch)
+        {
+            if (!IsTensorParallel)
+                return;
+            if (_tpQuantWeights.Count > 0 || _tpWeights.Count > 0)
+                return;
+            Console.Error.WriteLine(
+                $"WARNING: tensor parallelism is active ({_tpGroup.Degree} ranks) but architecture '{arch}' " +
+                "sharded 0 weights - the whole model is resident on rank 0 and the other GPUs are idle. " +
+                "This architecture has no tensor-parallel implementation; run without --tp.");
         }
     }
 }
diff --git a/TensorSharp.Models/ModelMultimodalInjector.cs b/TensorSharp.Models/ModelMultimodalInjector.cs
index 388d14a9..0819a51f 100644
--- a/TensorSharp.Models/ModelMultimodalInjector.cs
+++ b/TensorSharp.Models/ModelMultimodalInjector.cs
@@ -149,6 +149,12 @@ public void LoadProjectors(string mmProjPath)
                 case Qwen35Model q35:
                     q35.LoadVisionEncoder(mmProjPath);
                     break;
+                case Qwen4ExpModel q4e:
+                    q4e.LoadVisionEncoder(mmProjPath);
+                    break;
+                case GlmDsaModel glm:
+                    glm.LoadVisionEncoder(mmProjPath);
+                    break;
                 case Mistral3Model m3:
                     m3.LoadVisionEncoder(mmProjPath);
                     break;
@@ -180,6 +186,10 @@ public List<int> ProcessPromptTokens(List<ChatMessage> history, List<int> inputT
                 return ProcessGemma3History(g3, history, inputTokens);
             if (_model is Qwen35Model q35)
                 return ProcessQwen35History(q35, history, inputTokens);
+            if (_model is Qwen4ExpModel q4e)
+                return ProcessQwenVLHistory(q4e.VisionEncoder, history, inputTokens);
+            if (_model is GlmDsaModel glm)
+                return ProcessGlmNextHistory(glm, history, inputTokens);
             if (_model is Mistral3Model m3)
                 return ProcessMistral3History(m3, history, inputTokens);
             if (_model is NemotronModel nem)
@@ -254,6 +264,11 @@ public bool QueuePromptEmbeddingsForSlice(int promptStartToken, int tokenCount,
                 q35.SetMRoPEPositions(mropeSlice);
                 queued = true;
             }
+            else if (mropeSlice != null && _model is Qwen4ExpModel q4m)
+            {
+                q4m.SetMRoPEPositions(mropeSlice);
+                queued = true;
+            }
             return queued;
         }
 
@@ -496,8 +511,16 @@ private List<int> ProcessGemma3History(Gemma3Model model, List<ChatMessage> hist
         }
 
         private List<int> ProcessQwen35History(Qwen35Model model, List<ChatMessage> history, List<int> inputTokens)
+            => ProcessQwenVLHistory(model.VisionEncoder, history, inputTokens);
+
+        /// <summary>
+        /// Shared Qwen-VL-family prompt processing: Qwen3.5-VL and Qwen3.8-Flash-Next
+        /// use the same qwen3vl_merger tower, image-pad expansion and (T,H,W) IMRoPE
+        /// position assignment.
+        /// </summary>
+        private List<int> ProcessQwenVLHistory(Qwen35VisionEncoder encoder, List<ChatMessage> history, List<int> inputTokens)
         {
-            if (model.VisionEncoder == null)
+            if (encoder == null)
                 return inputTokens;
 
             var imagePaths = GetImagePathsInPromptOrder(history);
@@ -508,12 +531,12 @@ private List<int> ProcessQwen35History(Qwen35Model model, List<ChatMessage> hist
             if (imagePadId < 0)
                 return inputTokens;
 
-            var processor = new Qwen35ImageProcessor(model.VisionEncoder.PatchSize, model.VisionEncoder.SpatialMergeSize);
+            var processor = new Qwen35ImageProcessor(encoder.PatchSize, encoder.SpatialMergeSize);
             var cachedEmbeddings = new CachedEmbedding[imagePaths.Count];
             var tokenCounts = new int[imagePaths.Count];
             for (int i = 0; i < imagePaths.Count; i++)
             {
-                cachedEmbeddings[i] = GetOrCreateQwen35VisionEmbedding(model, processor, imagePaths[i]);
+                cachedEmbeddings[i] = GetOrCreateQwenVLVisionEmbedding(encoder, processor, imagePaths[i]);
                 tokenCounts[i] = cachedEmbeddings[i].TokenCount;
             }
 
@@ -815,21 +838,82 @@ private CachedEmbedding GetOrCreateGemma3VisionEmbedding(
             });
         }
 
-        private CachedEmbedding GetOrCreateQwen35VisionEmbedding(
-            Qwen35Model model,
+        private CachedEmbedding GetOrCreateQwenVLVisionEmbedding(
+            Qwen35VisionEncoder encoder,
             Qwen35ImageProcessor processor,
             string imagePath)
         {
             return GetOrCreateCachedEmbedding(_visionCache, imagePath, fullPath =>
             {
                 var (pixels, resizedHeight, resizedWidth) = processor.ProcessImage(fullPath);
-                Tensor embeddings = model.VisionEncoder.Encode(pixels, resizedHeight, resizedWidth);
+                Tensor embeddings = encoder.Encode(pixels, resizedHeight, resizedWidth);
                 int mergedH = resizedHeight / processor.PatchSize / processor.MergeSize;
                 int mergedW = resizedWidth / processor.PatchSize / processor.MergeSize;
                 return CreateCachedEmbedding(fullPath, embeddings, mergedH, mergedW);
             });
         }
 
+        /// <summary>
+        /// GLM-5.3-Flash (glm5next) prompt processing: expand each <c>&lt;|image|&gt;</c>
+        /// placeholder to the image's merged-patch token count and record the
+        /// embedding spans. The text tower is NoPE, so unlike the Qwen-VL family
+        /// no MRoPE position table is built - image tokens occupy ordinary
+        /// sequential positions.
+        /// </summary>
+        private List<int> ProcessGlmNextHistory(GlmDsaModel model, List<ChatMessage> history, List<int> inputTokens)
+        {
+            var encoder = model.VisionEncoder;
+            if (encoder == null)
+                return inputTokens;
+
+            var imagePaths = GetImagePathsInPromptOrder(history);
+            if (imagePaths.Count == 0)
+                return inputTokens;
+
+            int imageId = _model.Tokenizer.LookupToken("<|image|>");
+            if (imageId < 0)
+                return inputTokens;
+
+            var processor = new GlmNextImageProcessor(encoder.PatchSize, encoder.SpatialMergeSize);
+            var cachedEmbeddings = new CachedEmbedding[imagePaths.Count];
+            var tokenCounts = new int[imagePaths.Count];
+            for (int i = 0; i < imagePaths.Count; i++)
+            {
+                cachedEmbeddings[i] = GetOrCreateGlmNextVisionEmbedding(encoder, processor, imagePaths[i]);
+                tokenCounts[i] = cachedEmbeddings[i].TokenCount;
+            }
+
+            inputTokens = ChatTemplate.ExpandImageTokens(inputTokens, imageId, tokenCounts);
+
+            int searchFrom = 0;
+            for (int i = 0; i < imagePaths.Count; i++)
+            {
+                int start = FindTokenPosition(inputTokens, imageId, searchFrom);
+                if (start < 0)
+                    break;
+                _preparedVisionEmbeddings.Add(new PreparedEmbeddingSpan(
+                    cachedEmbeddings[i], start, start, start + tokenCounts[i]));
+                searchFrom = start + tokenCounts[i];
+            }
+
+            return inputTokens;
+        }
+
+        private CachedEmbedding GetOrCreateGlmNextVisionEmbedding(
+            GlmNextVisionEncoder encoder,
+            GlmNextImageProcessor processor,
+            string imagePath)
+        {
+            return GetOrCreateCachedEmbedding(_visionCache, imagePath, fullPath =>
+            {
+                var (pixels, canvasH, canvasW) = processor.ProcessImage(fullPath);
+                Tensor embeddings = encoder.Encode(pixels, canvasH, canvasW);
+                int mergedH = canvasH / processor.PatchSize / processor.MergeSize;
+                int mergedW = canvasW / processor.PatchSize / processor.MergeSize;
+                return CreateCachedEmbedding(fullPath, embeddings, mergedH, mergedW);
+            });
+        }
+
         private CachedEmbedding GetOrCreateMistral3VisionEmbedding(
             Mistral3Model model,
             Mistral3ImageProcessor processor,
@@ -914,6 +998,26 @@ private bool QueuePreparedVisionEmbeddings(List<PreparedEmbeddingSpan> bucket, i
                         queued = true;
                     }
                     break;
+                case Qwen4ExpModel q4pv:
+                    foreach (var span in bucket)
+                    {
+                        if (span.EndPosition <= reusablePrefixTokenCount)
+                            continue;
+
+                        q4pv.SetVisionEmbeddings(CloneTensor(span.CacheEntry.Embeddings), span.InsertPosition - reusablePrefixTokenCount);
+                        queued = true;
+                    }
+                    break;
+                case GlmDsaModel glmv:
+                    foreach (var span in bucket)
+                    {
+                        if (span.EndPosition <= reusablePrefixTokenCount)
+                            continue;
+
+                        glmv.SetVisionEmbeddings(CloneTensor(span.CacheEntry.Embeddings), span.InsertPosition - reusablePrefixTokenCount);
+                        queued = true;
+                    }
+                    break;
                 case Mistral3Model m3:
                     foreach (var span in bucket)
                     {
@@ -1009,6 +1113,28 @@ private bool QueuePreparedVisionEmbeddingsForSlice(List<PreparedEmbeddingSpan> b
                         queued = true;
                     }
                     break;
+                case Qwen4ExpModel q4v:
+                    foreach (var span in bucket)
+                    {
+                        if (!TryCloneOverlappingEmbeddingRows(span, promptStartToken, promptEndToken,
+                                out Tensor embeddings, out int insertPosition))
+                            continue;
+
+                        q4v.SetVisionEmbeddings(embeddings, insertPosition);
+                        queued = true;
+                    }
+                    break;
+                case GlmDsaModel glmsv:
+                    foreach (var span in bucket)
+                    {
+                        if (!TryCloneOverlappingEmbeddingRows(span, promptStartToken, promptEndToken,
+                                out Tensor embeddings, out int insertPosition))
+                            continue;
+
+                        glmsv.SetVisionEmbeddings(embeddings, insertPosition);
+                        queued = true;
+                    }
+                    break;
                 case Mistral3Model m3:
                     foreach (var span in bucket)
                     {
diff --git a/TensorSharp.Models/Models/Gemma4/Gemma4Model.PerSeqCache.cs b/TensorSharp.Models/Models/Gemma4/Gemma4Model.PerSeqCache.cs
index 9c57d96a..0cafe3ab 100644
--- a/TensorSharp.Models/Models/Gemma4/Gemma4Model.PerSeqCache.cs
+++ b/TensorSharp.Models/Models/Gemma4/Gemma4Model.PerSeqCache.cs
@@ -88,6 +88,25 @@ private sealed class Gemma4KvCacheHolder
         public bool HasFusedSequenceCache(string requestId)
             => requestId != null && _fusedHolders != null && _fusedHolders.ContainsKey(requestId);
 
+        // Continuous-batching cache-handoff trace, off unless TS_CB_DEBUG=1.
+        //
+        // Worth keeping: this state machine has four ways in and out of a fused
+        // episode and its failures are silent - the model keeps decoding, just
+        // against the wrong cache. Reading the handoff order directly is what
+        // identified the un-zeroed replacement primary cache that made the first
+        // single-stream request after any concurrent burst emit <pad> forever.
+        private static readonly bool _cbDebug =
+            string.Equals(Environment.GetEnvironmentVariable("TS_CB_DEBUG"), "1", StringComparison.Ordinal);
+        private void CbTrace(string what)
+        {
+            if (!_cbDebug) return;
+            Console.Error.WriteLine(
+                $"[cb] {what} activeKey={_activeFusedKey ?? "<primary>"} seqLen={_cacheSeqLen} " +
+                $"cap={_kvCacheGlobalCapacity} holders={(_fusedHolders?.Count ?? 0)} " +
+                $"retained={(_retainedFusedHolders?.Count ?? 0)} primarySaved={(_primaryHolder != null)} " +
+                $"k0hash={(_kvCacheK != null && _kvCacheK.Length > 0 && _kvCacheK[0] != null ? _kvCacheK[0].GetHashCode() : 0)}");
+        }
+
         private Gemma4KvCacheHolder SnapshotActiveCache() => new Gemma4KvCacheHolder
         {
             K = _kvCacheK,
@@ -113,19 +132,12 @@ private void LoadCacheHolder(Gemma4KvCacheHolder h)
 
         private Gemma4KvCacheHolder CreateFreshHolder()
         {
+            // AllocateKvCacheArrays zero-fills: the token-batched fused-decode
+            // kernel reads a FIXED 256-padded attention window over each holder's
+            // cache, and positions beyond the written length are masked (-inf)
+            // but must still be finite or the softmax is poisoned.
             AllocateKvCacheArrays(_initialGlobalCacheLength,
                 out var k, out var v, out var sizes, out _);
-            // The token-batched fused-decode kernel reads a FIXED 256-padded
-            // attention window over each holder's cache; positions beyond the
-            // written length are masked (-inf) but must still be finite, so zero
-            // the freshly-allocated caches (AllocateKvCacheArrays skips zeroing on
-            // GgmlCuda/Mlx). Garbage (NaN/Inf) there otherwise poisons the softmax.
-            var zeroed = new HashSet<Tensor>();
-            for (int l = 0; l < Config.NumLayers; l++)
-            {
-                if (k[l] != null && zeroed.Add(k[l])) Ops.Fill(k[l], 0f);
-                if (v[l] != null && zeroed.Add(v[l])) Ops.Fill(v[l], 0f);
-            }
             return new Gemma4KvCacheHolder
             {
                 K = k,
@@ -151,6 +163,7 @@ public bool BindSequenceCache(string requestId)
 
             if (string.Equals(_activeFusedKey, requestId, StringComparison.Ordinal))
                 return false; // already active
+            CbTrace($"BindSequenceCache({requestId}) ENTER");
 
             // Save whatever cache is currently checked out so its (possibly
             // grown) tensors aren't lost when we repoint the active fields.
@@ -172,6 +185,7 @@ public bool BindSequenceCache(string requestId)
             }
             LoadCacheHolder(holder);
             _activeFusedKey = requestId;
+            CbTrace($"BindSequenceCache({requestId}) fresh={fresh}");
             return fresh;
         }
 
@@ -189,6 +203,7 @@ public void AdoptPrimaryCacheToFused(string requestId)
             // Only meaningful when the primary cache is the one currently active
             // (i.e. the N==1 owner ran most recently). If a fused holder is
             // already checked out there is nothing to adopt.
+            CbTrace($"AdoptPrimaryCacheToFused({requestId}) ENTER");
             if (_activeFusedKey != null)
                 return;
             if (_fusedHolders.ContainsKey(requestId))
@@ -202,6 +217,9 @@ public void AdoptPrimaryCacheToFused(string requestId)
 
             // Give the primary a fresh empty allocation so a future N==1 step for
             // a never-fused request doesn't reset the adopted holder's tensors.
+            // AllocateKvCacheArrays zero-fills it - this cache is handed straight
+            // to the next single-stream request by RestorePrimaryCache, whose
+            // fused decode reads the 256-padded window past the written length.
             AllocateKvCacheArrays(_initialGlobalCacheLength,
                 out var k, out var v, out var sizes, out _);
             _primaryHolder = new Gemma4KvCacheHolder
@@ -213,6 +231,7 @@ public void AdoptPrimaryCacheToFused(string requestId)
                 SeqLen = 0,
                 HostDirty = false,
             };
+            CbTrace($"AdoptPrimaryCacheToFused({requestId}) DONE freshPrimary");
         }
 
         /// <summary>Reinstate the primary cache as the model's active cache.
@@ -222,6 +241,7 @@ public void AdoptPrimaryCacheToFused(string requestId)
         /// No-op when the primary cache is already active.</summary>
         public void RestorePrimaryCache()
         {
+            CbTrace("RestorePrimaryCache ENTER");
             if (_activeFusedKey == null)
                 return;
             // Save the checked-out fused holder, then swap the primary back in.
@@ -232,6 +252,7 @@ public void RestorePrimaryCache()
                 LoadCacheHolder(_primaryHolder);
                 _primaryHolder = null;
             }
+            CbTrace("RestorePrimaryCache DONE");
         }
 
         /// <summary>Release a finished/aborted request's per-request cache. The
@@ -244,6 +265,7 @@ public void OnSequenceReleased(string requestId)
                 return;
             if (!_fusedHolders.TryGetValue(requestId, out var holder))
                 return;
+            CbTrace($"OnSequenceReleased({requestId})");
 
             if (string.Equals(_activeFusedKey, requestId, StringComparison.Ordinal))
             {
@@ -282,6 +304,7 @@ public bool RetainSequenceCache(string requestId)
                 return false;
             if (!_fusedHolders.TryGetValue(requestId, out var holder))
                 return false;
+            CbTrace($"RetainSequenceCache({requestId})");
 
             if (string.Equals(_activeFusedKey, requestId, StringComparison.Ordinal))
             {
diff --git a/TensorSharp.Models/Models/Gemma4/Gemma4Model.TensorParallel.cs b/TensorSharp.Models/Models/Gemma4/Gemma4Model.TensorParallel.cs
index 1a69c188..19fc7745 100644
--- a/TensorSharp.Models/Models/Gemma4/Gemma4Model.TensorParallel.cs
+++ b/TensorSharp.Models/Models/Gemma4/Gemma4Model.TensorParallel.cs
@@ -477,8 +477,9 @@ private void InitGemma4TpKVCache(int initialSeqLen, int maxSeqLen)
                     var alloc = _tpGroup.GetAllocator(r);
                     _tpKvCacheK[l][r] = new Tensor(alloc, kvDtype, kvHeadsPerGpu, cacheLen, headDim);
                     _tpKvCacheV[l][r] = new Tensor(alloc, kvDtype, kvHeadsPerGpu, cacheLen, headDim);
-                    InitializeCacheTensor(_tpKvCacheK[l][r]);
-                    InitializeCacheTensor(_tpKvCacheV[l][r]);
+                    // Same finite-padding requirement as the single-GPU cache.
+                    InitGemma4CacheTensor(_tpKvCacheK[l][r]);
+                    InitGemma4CacheTensor(_tpKvCacheV[l][r]);
                 }
             }
 
@@ -516,8 +517,8 @@ private void EnsureGemma4TpCacheCapacity(int requiredSeqLen)
                     var alloc = _tpGroup.GetAllocator(r);
                     var newK = new Tensor(alloc, kvDtype, kvHeadsPerGpu, newCapacity, headDim);
                     var newV = new Tensor(alloc, kvDtype, kvHeadsPerGpu, newCapacity, headDim);
-                    InitializeCacheTensor(newK);
-                    InitializeCacheTensor(newV);
+                    InitGemma4CacheTensor(newK);
+                    InitGemma4CacheTensor(newV);
 
                     if (_cacheSeqLen > 0)
                     {
diff --git a/TensorSharp.Models/Models/Gemma4/Gemma4Model.cs b/TensorSharp.Models/Models/Gemma4/Gemma4Model.cs
index 24f4780d..50a2f472 100644
--- a/TensorSharp.Models/Models/Gemma4/Gemma4Model.cs
+++ b/TensorSharp.Models/Models/Gemma4/Gemma4Model.cs
@@ -763,8 +763,9 @@ private void AllocateKvCacheArrays(
                 cacheSize[l] = cacheLen;
                 cacheK[l] = new Tensor(_allocator, kvDtype, kvHeads, cacheLen, hd);
                 cacheV[l] = new Tensor(_allocator, kvDtype, kvHeads, cacheLen, hd);
-                InitializeCacheTensor(cacheK[l]);
-                InitializeCacheTensor(cacheV[l]);
+                // Zeroing happens once for the whole set below (ZeroKvCacheArrays),
+                // which also covers the backends ModelBase.InitializeCacheTensor
+                // skips - so no per-tensor init here.
                 // Q8_0 has fractional bytes/elem (1.0625) - go through ByteLengthFor
                 // so block-quantized layouts are accounted for correctly.
                 long perLayerElems = (long)kvHeads * cacheLen * hd;
@@ -777,6 +778,54 @@ private void AllocateKvCacheArrays(
                 cacheV[kv.Key] = cacheV[kv.Value];
                 cacheSize[kv.Key] = cacheSize[kv.Value];
             }
+
+            // Every freshly-allocated cache set MUST start finite. The fused
+            // decode kernels read a FIXED 256-padded attention window over the
+            // cache; rows past the written length are masked (-inf) but are
+            // still multiplied/added, so uninitialised VRAM (NaN/Inf) there
+            // poisons the softmax and every logit becomes NaN - argmax then
+            // returns token 0 (<pad>) forever.
+            // InitializeCacheTensor above skips the fill on GgmlCuda/Mlx, so do
+            // it here, in the ONE place every cache set is born. It used to live
+            // in CreateFreshHolder only, which is why the replacement primary
+            // cache minted by AdoptPrimaryCacheToFused came up uninitialised and
+            // the first single-stream request after any concurrent episode
+            // decoded nothing but <pad>.
+            ZeroKvCacheArrays(cacheK, cacheV);
+        }
+
+        /// <summary>Zero every distinct tensor in a K/V cache array pair.
+        /// Donor-aliased layers share tensors, so fill each object once.</summary>
+        private void ZeroKvCacheArrays(Tensor[] cacheK, Tensor[] cacheV)
+        {
+            if (cacheK == null || cacheV == null)
+                return;
+            var zeroed = new HashSet<Tensor>();
+            for (int l = 0; l < cacheK.Length; l++)
+            {
+                if (cacheK[l] != null && zeroed.Add(cacheK[l])) InitGemma4CacheTensor(cacheK[l]);
+                if (cacheV[l] != null && zeroed.Add(cacheV[l])) InitGemma4CacheTensor(cacheV[l]);
+            }
+        }
+
+        /// <summary>
+        /// Zero one freshly-allocated Gemma 4 KV cache tensor.
+        ///
+        /// UNCONDITIONAL, unlike ModelBase.InitializeCacheTensor, which skips the
+        /// fill on GgmlCuda/Mlx. Gemma 4's fused decode graph reads a FIXED
+        /// 256-padded attention window: rows past the written length are masked
+        /// with -inf, which cancels a FINITE score but turns a non-finite one into
+        /// NaN - and one NaN takes the whole softmax row, then every logit, then
+        /// argmax returns token 0 (&lt;pad&gt;) forever. Pool blocks are recycled
+        /// without clearing, so on those backends the padding was live activation
+        /// garbage. Every cache allocation in this model must come through here:
+        /// the grow path and both tensor-parallel paths allocate their own tensors
+        /// and would otherwise reintroduce the same defect on a long conversation.
+        /// </summary>
+        private void InitGemma4CacheTensor(Tensor tensor)
+        {
+            if (tensor == null) return;
+            Ops.Fill(tensor, 0f);
         }
 
         // Grow the global-attention layers' KV cache to fit requiredSeqLen
@@ -819,8 +868,11 @@ private void EnsureCacheCapacity(int requiredSeqLen)
                 int hd = HeadDimForLayer(l);
                 var newK = new Tensor(_allocator, kvDtype, kvHeads, newCapacity, hd);
                 var newV = new Tensor(_allocator, kvDtype, kvHeads, newCapacity, hd);
-                InitializeCacheTensor(newK);
-                InitializeCacheTensor(newV);
+                // Zeroed, not just allocated: only rows [0, _cacheSeqLen) are copied
+                // in below, and the flash window reads past them. See
+                // InitGemma4CacheTensor.
+                InitGemma4CacheTensor(newK);
+                InitGemma4CacheTensor(newV);
 
                 if (_cacheSeqLen > 0)
                 {
diff --git a/TensorSharp.Models/Models/GlmDsa/GlmDsaModel.Native.cs b/TensorSharp.Models/Models/GlmDsa/GlmDsaModel.Native.cs
index fe7f44cc..e260266f 100644
--- a/TensorSharp.Models/Models/GlmDsa/GlmDsaModel.Native.cs
+++ b/TensorSharp.Models/Models/GlmDsa/GlmDsaModel.Native.cs
@@ -197,6 +197,7 @@ public override void WarmUpKernels()
 
         public override void Dispose()
         {
+            VisionEncoder?.Dispose();
             lock (_nativeSync)
             {
                 if (_native != IntPtr.Zero)
diff --git a/TensorSharp.Models/Models/GlmDsa/GlmDsaModel.Vision.cs b/TensorSharp.Models/Models/GlmDsa/GlmDsaModel.Vision.cs
new file mode 100644
index 00000000..77651334
--- /dev/null
+++ b/TensorSharp.Models/Models/GlmDsa/GlmDsaModel.Vision.cs
@@ -0,0 +1,81 @@
+// Copyright (c) Zhongkai Fu. All rights reserved.
+// https://github.com/zhongkaifu/TensorSharp
+//
+// This file is part of TensorSharp.
+//
+// TensorSharp is licensed under the BSD-3-Clause license found in the LICENSE file in the root directory of this source tree.
+//
+// TensorSharp is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the BSD-3-Clause License for more details.
+//
+// GLM-5.3-Flash (glm5next) vision support. The managed GlmNextVisionEncoder
+// produces projected [nTokens, n_embd] embeddings; they reach the native
+// executor as queued OVERRIDE rows that replace the token embeddings of the
+// <|image|> placeholder positions in the next prompt forward (the text tower is
+// NoPE, so image tokens simply occupy sequential cache positions - no MRoPE
+// bookkeeping like the Qwen-VL family needs).
+using System;
+using TensorSharp;
+using TensorSharp.GGML;
+
+namespace TensorSharp.Models
+{
+    public partial class GlmDsaModel
+    {
+        public GlmNextVisionEncoder VisionEncoder { get; private set; }
+
+        public void LoadVisionEncoder(string mmProjPath)
+        {
+            if (Config.Architecture != "glm5next")
+            {
+                Console.WriteLine($"Warning: {Config.Architecture} has no vision tower; ignoring mmproj {mmProjPath}.");
+                return;
+            }
+            VisionEncoder = new GlmNextVisionEncoder(mmProjPath, _allocator);
+        }
+
+        /// <summary>
+        /// Queue projected vision embeddings to replace the token embeddings of
+        /// the placeholder span starting at <paramref name="startPosition"/>
+        /// (an index into the token array of the NEXT Forward call). Takes
+        /// ownership of <paramref name="visionEmbeddings"/>.
+        /// </summary>
+        public void SetVisionEmbeddings(Tensor visionEmbeddings, int startPosition)
+        {
+            if (visionEmbeddings == null)
+                return;
+            try
+            {
+                if (startPosition < 0)
+                    return;
+                if (!UsesNativeExecutor)
+                {
+                    Console.WriteLine("Warning: glm5next vision requires the native executor; dropping image embeddings.");
+                    return;
+                }
+
+                int rows = (int)visionEmbeddings.Sizes[0];
+                int dim = (int)visionEmbeddings.Sizes[1];
+                if (dim != Config.HiddenSize)
+                {
+                    Console.WriteLine($"Warning: vision embedding dim {dim} != hidden {Config.HiddenSize}; dropping.");
+                    return;
+                }
+
+                Tensor src = visionEmbeddings.IsContiguous() ? visionEmbeddings : Ops.NewContiguous(visionEmbeddings);
+                float[] data = src.GetElementsAsFloat(rows * dim);
+                if (!ReferenceEquals(src, visionEmbeddings))
+                    src.Dispose();
+
+                lock (_nativeSync)
+                {
+                    GgmlGlmNative.QueueVisionRows(_native, data, rows, startPosition);
+                }
+            }
+            finally
+            {
+                visionEmbeddings.Dispose();
+            }
+        }
+    }
+}
diff --git a/TensorSharp.Models/Models/GlmDsa/GlmDsaModel.cs b/TensorSharp.Models/Models/GlmDsa/GlmDsaModel.cs
index 3517e9e6..4d226beb 100644
--- a/TensorSharp.Models/Models/GlmDsa/GlmDsaModel.cs
+++ b/TensorSharp.Models/Models/GlmDsa/GlmDsaModel.cs
@@ -186,6 +186,11 @@ public GlmDsaModel(string ggufPath, BackendType backend, int tpDegree = 1, ITens
                 return;
             }
 
+            if (arch == "glm5next")
+                throw new NotSupportedException(
+                    "glm5next (GLM-5.3-Flash) runs through the native GGML executor only; " +
+                    "use a GGML backend (there is no managed per-op fallback for the KDA/mHC layers yet).");
+
             LoadWeights();
             BuildLayerNames();
             CacheMoeWeightHandles();
@@ -209,6 +214,14 @@ public GlmDsaModel(string ggufPath, BackendType backend, int tpDegree = 1, ITens
             AllocateScratch();
         }
 
+        /// <summary>
+        /// glm5next's KDA recurrence cannot be rewound to an earlier position, so a
+        /// cached prefix is only reusable when the new prompt EXTENDS it exactly
+        /// (same contract as the Qwen 3.x GDN models). glm-dsa proper has no
+        /// recurrent state and keeps the base behaviour.
+        /// </summary>
+        public override bool SupportsKVCacheTruncation => Config.Architecture != "glm5next";
+
         private int CountIndexerFull()
         {
             int n = 0;
@@ -304,6 +317,22 @@ private bool[] ResolveIndexerTypes(string arch)
         {
             var full = new bool[_numTrunkLayers];
 
+            if (arch == "glm5next")
+            {
+                // GLM-5.3-Flash: attention.head_count_kv is a per-layer array,
+                // 0 on KDA (linear-attention) layers and 1 on MLA+DSA layers.
+                // Every MLA layer carries a full pooled indexer; KDA layers have
+                // none and never share a selection.
+                var kvh = _gguf.GetInt32Array($"{arch}.attention.head_count_kv")
+                          ?? ToInt32(_gguf.GetUint32Array($"{arch}.attention.head_count_kv"));
+                if (kvh != null)
+                {
+                    for (int i = 0; i < full.Length && i < kvh.Length; i++)
+                        full[i] = kvh[i] != 0;
+                    return full;
+                }
+            }
+
             int trainCtx = (int)_gguf.GetUint32($"{arch}.context_length", 0);
             bool pre52 = trainCtx > 0 && trainCtx < 1048576;
 
diff --git a/TensorSharp.Models/Models/GlmDsa/GlmNextImageProcessor.cs b/TensorSharp.Models/Models/GlmDsa/GlmNextImageProcessor.cs
new file mode 100644
index 00000000..f22c2523
--- /dev/null
+++ b/TensorSharp.Models/Models/GlmDsa/GlmNextImageProcessor.cs
@@ -0,0 +1,257 @@
+// Copyright (c) Zhongkai Fu. All rights reserved.
+// https://github.com/zhongkaifu/TensorSharp
+//
+// This file is part of TensorSharp.
+//
+// TensorSharp is licensed under the BSD-3-Clause license found in the LICENSE file in the root directory of this source tree.
+//
+// TensorSharp is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the BSD-3-Clause License for more details.
+//
+// GLM-5.3-Flash (glm5next) image preprocessing, mirroring llama.cpp's
+// mtmd_image_preprocessor_glm5next (itself a port of the HF Glm5NextProcessor):
+//
+//   1. smart_resize: CEIL-align both edges to factor = patch * merge (28). An
+//      image below the minimum token budget is upscaled by sqrt(min/area) and
+//      re-aligned; one above the maximum is shrunk by BINARY SEARCH over the
+//      content height, because aligning both edges is not monotone in the
+//      Qwen-style sqrt scale and that scale leaves budget unspent.
+//   2. The CONTENT keeps its aspect ratio inside the canvas: scale =
+//      min(canvasH/h, canvasW/w), never upscaling an image that already spends
+//      the minimum budget. It is composited at the TOP-LEFT (not centred) and
+//      the remainder is black padding.
+//   3. Bicubic (PyTorch-equivalent) resize, CLIP mean/std normalization,
+//      channel-first output.
+using System;
+
+namespace TensorSharp.Models
+{
+    public class GlmNextImageProcessor
+    {
+        public int PatchSize { get; }
+        public int MergeSize { get; }
+        public int Factor { get; }
+        /// <summary>Minimum pixel budget (16 output tokens' worth).</summary>
+        public long MinPixels { get; }
+        /// <summary>Maximum pixel budget (8000 output tokens' worth).</summary>
+        public long MaxPixels { get; }
+
+        private static readonly float[] Mean = { 0.48145467f, 0.4578275f, 0.40821072f };
+        private static readonly float[] Std = { 0.26862955f, 0.2613026f, 0.2757771f };
+
+        public GlmNextImageProcessor(int patchSize = 14, int mergeSize = 2,
+            int minTokens = 16, int maxTokens = 8000)
+        {
+            PatchSize = patchSize;
+            MergeSize = mergeSize;
+            Factor = patchSize * mergeSize;
+            MinPixels = (long)minTokens * Factor * Factor;
+            MaxPixels = (long)maxTokens * Factor * Factor;
+        }
+
+        public static (int width, int height) ReadImageDimensions(string path)
+            => Gemma3ImageProcessor.ReadImageDimensions(path);
+
+        private int Align(long v) => (int)((v + Factor - 1) / Factor * Factor);
+
+        /// <summary>Canvas (aligned) size for an input of the given size.</summary>
+        public (int width, int height) SmartResize(int width, int height)
+        {
+            if (width <= 0 || height <= 0)
+                return (0, 0);
+
+            int alignedH = Align(height);
+            int alignedW = Align(width);
+
+            // upscale an image that is too small to spend the minimum token budget
+            if ((long)alignedH * alignedW < MinPixels)
+            {
+                double scale = Math.Sqrt((double)MinPixels / ((double)height * width));
+                alignedH = Align(Math.Max(1L, (long)Math.Ceiling(height * scale)));
+                alignedW = Align(Math.Max(1L, (long)Math.Ceiling(width * scale)));
+            }
+
+            if ((long)alignedH * alignedW > MaxPixels)
+            {
+                // binary search the tallest content height whose aligned canvas
+                // still fits the budget
+                int low = 1, high = height;
+                alignedH = Factor;
+                alignedW = Factor;
+                while (low <= high)
+                {
+                    int contentH = (low + high) / 2;
+                    int contentW = Math.Max(1, (int)Math.Floor((double)width * contentH / height));
+                    int candH = Align(contentH);
+                    int candW = Align(contentW);
+                    if ((long)candH * candW <= MaxPixels)
+                    {
+                        alignedH = candH;
+                        alignedW = candW;
+                        low = contentH + 1;
+                    }
+                    else
+                    {
+                        high = contentH - 1;
+                    }
+                }
+            }
+
+            return (alignedW, alignedH);
+        }
+
+        /// <summary>Content size (aspect-preserving) inside the canvas.</summary>
+        public (int width, int height) ContentSize(int width, int height, int canvasW, int canvasH)
+        {
+            if (canvasW == 0 || canvasH == 0)
+                return (0, 0);
+            double scale = Math.Min((double)canvasH / height, (double)canvasW / width);
+            if ((long)height * width >= MinPixels)
+                scale = Math.Min(1.0, scale);
+            return (Math.Max(1, Math.Min(canvasW, (int)Math.Floor(width * scale))),
+                    Math.Max(1, Math.Min(canvasH, (int)Math.Floor(height * scale))));
+        }
+
+        public int ComputeImageTokenCount(int origWidth, int origHeight)
+        {
+            var (cw, ch) = SmartResize(origWidth, origHeight);
+            return (cw / PatchSize / MergeSize) * (ch / PatchSize / MergeSize);
+        }
+
+        public int ComputeImageTokenCount(string imagePath)
+        {
+            var (w, h) = ReadImageDimensions(imagePath);
+            return ComputeImageTokenCount(w, h);
+        }
+
+        /// <summary>
+        /// Full pipeline: decode, smart-resize, bicubic content resize, top-left
+        /// composite on a black canvas, CLIP-normalize. Returns channel-first
+        /// [3, canvasH, canvasW] floats plus the canvas geometry.
+        /// </summary>
+        public (float[] pixels, int canvasH, int canvasW) ProcessImage(string imagePath)
+        {
+            byte[] fileBytes = System.IO.File.ReadAllBytes(imagePath);
+            byte[] rgba = Gemma3ImageProcessor.DecodeImageToRGBA(fileBytes, out int origW, out int origH);
+
+            var (canvasW, canvasH) = SmartResize(origW, origH);
+            if (canvasW == 0 || canvasH == 0)
+                throw new ArgumentException($"Image {imagePath} is empty ({origW}x{origH}).");
+            var (contentW, contentH) = ContentSize(origW, origH, canvasW, canvasH);
+
+            float[] content = ResizeBicubicCHW(rgba, origW, origH, contentW, contentH);
+
+            // Composite at the top-left of a black canvas, then normalize. The pad
+            // color is 0 (llama.cpp's default for this projector), so the padded
+            // area normalizes to -mean/std per channel.
+            var pixels = new float[3L * canvasH * canvasW];
+            long plane = (long)canvasH * canvasW;
+            long contentPlane = (long)contentH * contentW;
+            for (int c = 0; c < 3; c++)
+            {
+                float mean = Mean[c], std = Std[c];
+                float pad = (0f - mean) / std;
+                long cBase = c * plane;
+                long sBase = c * contentPlane;
+                for (int y = 0; y < canvasH; y++)
+                {
+                    long row = cBase + (long)y * canvasW;
+                    if (y < contentH)
+                    {
+                        long srow = sBase + (long)y * contentW;
+                        for (int x = 0; x < contentW; x++)
+                            pixels[row + x] = (content[srow + x] - mean) / std;
+                        for (int x = contentW; x < canvasW; x++)
+                            pixels[row + x] = pad;
+                    }
+                    else
+                    {
+                        for (int x = 0; x < canvasW; x++)
+                            pixels[row + x] = pad;
+                    }
+                }
+            }
+
+            return (pixels, canvasH, canvasW);
+        }
+
+        // ---- PyTorch-equivalent bicubic (align_corners=False, no antialias) ----
+
+        private static float ClampUnit(double v) => (float)Math.Max(0.0, Math.Min(1.0, v));
+
+        private static int ClampIndex(int v, int lo, int hi) => v < lo ? lo : (v > hi ? hi : v);
+
+        private static void TorchBicubicWeights(double t, Span<double> w)
+        {
+            // a = -0.75, the PyTorch/OpenCV constant.
+            const double a = -0.75;
+            double t2 = t * t, t3 = t2 * t;
+            w[0] = a * (t3 - 2 * t2 + t);
+            w[1] = (a + 2) * t3 - (a + 3) * t2 + 1;
+            w[2] = -(a + 2) * t3 + (2 * a + 3) * t2 - a * t;
+            w[3] = -a * (t3 - t2);
+        }
+
+        private static float[] ResizeBicubicCHW(byte[] rgba, int srcW, int srcH, int dstW, int dstH)
+        {
+            long srcPlane = (long)srcW * srcH;
+            var src = new float[3 * srcPlane];
+            System.Threading.Tasks.Parallel.For(0, srcH, y =>
+            {
+                for (int x = 0; x < srcW; x++)
+                {
+                    int idx = (y * srcW + x) * 4;
+                    long d = (long)y * srcW + x;
+                    src[d] = rgba[idx] / 255.0f;
+                    src[srcPlane + d] = rgba[idx + 1] / 255.0f;
+                    src[2 * srcPlane + d] = rgba[idx + 2] / 255.0f;
+                }
+            });
+
+            if (dstW == srcW && dstH == srcH)
+                return src;
+
+            long dstPlane = (long)dstW * dstH;
+            var dst = new float[3 * dstPlane];
+            double scaleX = (double)srcW / dstW;
+            double scaleY = (double)srcH / dstH;
+
+            System.Threading.Tasks.Parallel.For(0, dstH, oy =>
+            {
+                double srcY = scaleY * (oy + 0.5) - 0.5;
+                int yBase = (int)Math.Floor(srcY);
+                double yFrac = ClampUnit(srcY - yBase);
+                Span<double> wy = stackalloc double[4];
+                TorchBicubicWeights(yFrac, wy);
+
+                Span<double> wx = stackalloc double[4];
+                for (int ox = 0; ox < dstW; ox++)
+                {
+                    double srcX = scaleX * (ox + 0.5) - 0.5;
+                    int xBase = (int)Math.Floor(srcX);
+                    double xFrac = ClampUnit(srcX - xBase);
+                    TorchBicubicWeights(xFrac, wx);
+
+                    for (int c = 0; c < 3; c++)
+                    {
+                        double sum = 0;
+                        long channelBase = c * srcPlane;
+                        for (int ky = 0; ky < 4; ky++)
+                        {
+                            int iy = ClampIndex(yBase - 1 + ky, 0, srcH - 1);
+                            long rowBase = channelBase + (long)iy * srcW;
+                            for (int kx = 0; kx < 4; kx++)
+                            {
+                                int ix = ClampIndex(xBase - 1 + kx, 0, srcW - 1);
+                                sum += src[rowBase + ix] * wy[ky] * wx[kx];
+                            }
+                        }
+                        dst[c * dstPlane + (long)oy * dstW + ox] = (float)sum;
+                    }
+                }
+            });
+
+            return dst;
+        }
+    }
+}
diff --git a/TensorSharp.Models/Models/GlmDsa/GlmNextVisionEncoder.cs b/TensorSharp.Models/Models/GlmDsa/GlmNextVisionEncoder.cs
new file mode 100644
index 00000000..fcacf8be
--- /dev/null
+++ b/TensorSharp.Models/Models/GlmDsa/GlmNextVisionEncoder.cs
@@ -0,0 +1,735 @@
+// Copyright (c) Zhongkai Fu. All rights reserved.
+// https://github.com/zhongkaifu/TensorSharp
+//
+// This file is part of TensorSharp.
+//
+// TensorSharp is licensed under the BSD-3-Clause license found in the LICENSE file in the root directory of this source tree.
+//
+// TensorSharp is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the BSD-3-Clause License for more details.
+//
+// GLM-5.3-Flash (glm5next) vision tower: the GLM-OCR ViT plus the GLM4V-style
+// merger, mirroring llama.cpp's clip_graph_glm5next / clip_graph_glm4v.
+//
+// Structure per block (RMS norms, no biases on the norms):
+//   x += attn_out( SDPA( rope(q_norm(q)), rope(k_norm(k)), v ) )   [fused qkv +bias]
+//   x += ffn_down( clampsilu(ffn_gate(x')) * clamp(ffn_up(x')) )   [all +bias]
+// with per-head RMS q/k norms (weight [head_dim]), NeoX 2D vision RoPE in the
+// same blocked Y|X layout as Qwen3-VL, and the SwiGLU clamp (gate <= L,
+// up in [-L, L], L = clip.vision.swiglu_limit) that also applies to the merger.
+//
+// No learned position embeddings and no post-conv norm (GLM-OCR); the dual
+// patch-embed convs collapse into one by adding their kernels.
+//
+// Merger: RMS post_ln -> 2x2 conv patch merger (run as a linear over the
+// block-ordered 2x2 groups with a re-laid-out weight) -> FC (no bias) ->
+// LayerNorm -> erf-GELU -> SwiGLU-clamp FFN -> [nTokens, 4096].
+using System;
+using System.Collections.Generic;
+using System.Diagnostics;
+using System.Numerics;
+using System.Threading.Tasks;
+using TensorSharp;
+using TensorSharp.Runtime;
+
+namespace TensorSharp.Models
+{
+    public class GlmNextVisionEncoder : IDisposable
+    {
+        private readonly Dictionary<string, Tensor> _weights = new();
+        private readonly Dictionary<string, Tensor> _transposedWeights = new();
+        private readonly Dictionary<long, RopeCache> _ropeCache = new();
+        private readonly Dictionary<long, int[]> _blockOrderCache = new();
+        private readonly IAllocator _allocator;
+
+        private readonly int _patchSize;
+        private readonly int _hiddenSize;
+        private readonly int _intermediateSize;
+        private readonly int _numHeads;
+        private readonly int _blockCount;
+        private readonly float _eps;
+        private readonly int _projectionDim;
+        private readonly int _spatialMergeSize;
+        private readonly float _ropeTheta;
+        private readonly float _swigluLimit;
+        private readonly string[] _blockPrefixes;
+
+        private sealed class RopeCache
+        {
+            public required float[] CosTable { get; init; }
+            public required float[] SinTable { get; init; }
+        }
+
+        public int ProjectionDim => _projectionDim;
+        public int PatchSize => _patchSize;
+        public int SpatialMergeSize => _spatialMergeSize;
+
+        public GlmNextVisionEncoder(string mmProjPath, IAllocator allocator)
+        {
+            _allocator = allocator;
+            var gguf = new GgufFile(mmProjPath);
+
+            string projector = gguf.GetString("clip.projector_type") ?? "";
+            if (projector != "glm5next")
+                Console.WriteLine($"Warning: mmproj projector_type is '{projector}', expected 'glm5next'.");
+
+            _patchSize = (int)gguf.GetUint32("clip.vision.patch_size", 14);
+            _hiddenSize = (int)gguf.GetUint32("clip.vision.embedding_length", 1024);
+            _intermediateSize = (int)gguf.GetUint32("clip.vision.feed_forward_length", 4096);
+            _numHeads = (int)gguf.GetUint32("clip.vision.attention.head_count", 16);
+            _blockCount = (int)gguf.GetUint32("clip.vision.block_count", 24);
+            _eps = gguf.GetFloat32("clip.vision.attention.layer_norm_epsilon", 1e-5f);
+            _projectionDim = (int)gguf.GetUint32("clip.vision.projection_dim", 4096);
+            _spatialMergeSize = (int)gguf.GetUint32("clip.vision.spatial_merge_size", 2);
+            _ropeTheta = gguf.GetFloat32("clip.vision.rope.freq_base", 10000f);
+            _swigluLimit = gguf.GetFloat32("clip.vision.swiglu_limit", 10f);
+
+            Console.WriteLine($"GLM-5.3 vision encoder: patchSize={_patchSize}, hidden={_hiddenSize}, " +
+                $"intermediate={_intermediateSize}, heads={_numHeads}, blocks={_blockCount}, " +
+                $"projDim={_projectionDim}, mergeSize={_spatialMergeSize}, ropeTheta={_ropeTheta}, " +
+                $"swigluLimit={_swigluLimit}, eps={_eps}");
+
+            _blockPrefixes = new string[_blockCount];
+            for (int i = 0; i < _blockCount; i++)
+                _blockPrefixes[i] = $"v.blk.{i}";
+
+            LoadWeights(gguf);
+            CombinePatchEmbedWeights();
+            BuildMergerLinearWeight();
+            gguf.Dispose();
+        }
+
+        private void LoadWeights(GgufFile gguf)
+        {
+            Console.Write("Loading GLM-5.3 vision weights...");
+            int count = 0;
+            foreach (var kv in gguf.Tensors)
+            {
+                var info = kv.Value;
+                byte[] raw = gguf.ReadTensorData(info);
+
+                long numElements = info.NumElements;
+                float[] f32 = new float[numElements];
+
+                if (info.Type == GgmlTensorType.F32)
+                    Buffer.BlockCopy(raw, 0, f32, 0, raw.Length);
+                else
+                    NativeDequant.DequantizeToFloat32((int)info.Type, raw, 0, f32, 0, numElements);
+
+                long[] tsShape = new long[info.Shape.Length];
+                for (int i = 0; i < info.Shape.Length; i++)
+                    tsShape[i] = (long)info.Shape[info.Shape.Length - 1 - i];
+
+                var tensor = new Tensor(_allocator, DType.Float32, tsShape);
+                tensor.SetElementsAsFloat(f32);
+                _weights[info.Name] = tensor;
+                count++;
+            }
+            Console.WriteLine($" done ({count} tensors)");
+        }
+
+        /// <summary>The dual patch-embed convs read the SAME input, so their sum
+        /// collapses into a single conv with added kernels.</summary>
+        private void CombinePatchEmbedWeights()
+        {
+            if (!_weights.ContainsKey("v.patch_embd.weight") ||
+                !_weights.ContainsKey("v.patch_embd.weight.1"))
+                return;
+
+            var w0 = _weights["v.patch_embd.weight"];
+            var w1 = _weights["v.patch_embd.weight.1"];
+            var combined = new Tensor(_allocator, DType.Float32, w0.Sizes);
+            Ops.Add(combined, w0, w1);
+            _weights["v.patch_embd.combined"] = combined;
+        }
+
+        /// <summary>
+        /// Re-lay the [2, 2, hidden, proj] patch-merger conv kernel as a linear
+        /// weight over the block-ordered 2x2 groups. A merged row is
+        /// [patch0 | patch1 | patch2 | patch3] with patches in (ky, kx) row-major
+        /// order and each patch's channels contiguous; the conv kernel is
+        /// (kx fastest, then ky, then channel) per output, so
+        ///   W2[o][(ky*2+kx)*hidden + c] = W[o][c*4 + ky*2 + kx].
+        /// </summary>
+        private unsafe void BuildMergerLinearWeight()
+        {
+            if (!_weights.TryGetValue("mm.patch_merger.weight", out var conv))
+                return;
+
+            int merge = _spatialMergeSize;
+            int cells = merge * merge;
+            int hidden = _hiddenSize;
+            int proj = (int)conv.Sizes[0];
+            int inDim = cells * hidden;
+
+            var linear = new Tensor(_allocator, DType.Float32, proj, inDim);
+            float* src = TensorComputePrimitives.GetFloatPointer(conv);
+            float* dst = TensorComputePrimitives.GetFloatPointer(linear);
+            long srcL = (long)src, dstL = (long)dst;
+            Parallel.For(0, proj, o =>
+            {
+                float* s = (float*)srcL + (long)o * inDim;
+                float* d = (float*)dstL + (long)o * inDim;
+                for (int c = 0; c < hidden; c++)
+                    for (int p = 0; p < cells; p++)
+                        d[p * hidden + c] = s[(long)c * cells + p];
+            });
+
+            _weights["mm.patch_merger.linear"] = linear;
+        }
+
+        /// <summary>
+        /// Encode a preprocessed image (channel-first [3, H, W] floats) into
+        /// projected embeddings [numMergedTokens, projectionDim].
+        /// </summary>
+        public unsafe Tensor Encode(float[] pixelValues, int resizedH, int resizedW)
+        {
+            long encodeStart = Stopwatch.GetTimestamp();
+            int gridH = resizedH / _patchSize;
+            int gridW = resizedW / _patchSize;
+            int numPatches = gridH * gridW;
+            int headDim = _hiddenSize / _numHeads;
+            int halfDim = headDim / 2;
+
+            // 1. Patch embedding straight into spatial-merge block order.
+            var hidden = PatchEmbed(pixelValues, resizedH, resizedW, gridH, gridW);
+
+            // 2. RoPE tables in the same block order (no learned position embd).
+            RopeCache rope = GetOrCreateRopeCache(gridH, gridW, numPatches, halfDim);
+
+            // 3. Encoder blocks. Fast path: all 24 blocks as ONE device-resident
+            // GGML graph (weights cached on the device across encodes); the
+            // managed per-block loop stays as the reference and fallback.
+            bool fused = s_fusedEncoderEnabled && TryWholeEncoderFused(hidden, numPatches, headDim, halfDim,
+                rope.CosTable, rope.SinTable);
+            if (!fused)
+            {
+                for (int i = 0; i < _blockCount; i++)
+                {
+                    Console.Write($"\r  GLM vision block {i + 1}/{_blockCount}...");
+                    EncoderBlock(hidden, i, numPatches, headDim, halfDim, rope.CosTable, rope.SinTable);
+                }
+                Console.WriteLine(" done");
+            }
+
+            // 4. RMS post_ln.
+            var postNormed = RmsNormOp(hidden, "v.post_ln.weight");
+            hidden.Dispose();
+
+            // 5. Merger: 2x2 conv as a linear over the block-ordered groups.
+            int mergedTokens = numPatches / (_spatialMergeSize * _spatialMergeSize);
+            int mergedDim = _hiddenSize * _spatialMergeSize * _spatialMergeSize;
+
+            using var mergedView = postNormed.View(mergedTokens, mergedDim);
+            var merged = Ops.NewContiguous(mergedView);
+            postNormed.Dispose();
+
+            var pooled = LinearForwardWithBias(merged, "mm.patch_merger.linear", "mm.patch_merger.bias");
+            merged.Dispose();
+
+            // 6. FC projector: fc (no bias) -> LayerNorm -> erf-GELU.
+            var fc = LinearForwardWithBias(pooled, "mm.model.fc.weight", null);
+            pooled.Dispose();
+            var fcn = Ops.LayerNorm(null, fc, _weights["mm.post_norm.weight"], _weights["mm.post_norm.bias"], 1e-5f);
+            fc.Dispose();
+            GeluErf(fcn);
+
+            // 7. SwiGLU-clamp FFN projector.
+            var gate = LinearForwardWithBias(fcn, "mm.gate.weight", null);
+            var up = LinearForwardWithBias(fcn, "mm.up.weight", null);
+            fcn.Dispose();
+            ClampSiluMul(gate, up);
+            up.Dispose();
+            var projected = LinearForwardWithBias(gate, "mm.down.weight", null);
+            gate.Dispose();
+
+            double totalMs = (Stopwatch.GetTimestamp() - encodeStart) * 1000.0 / Stopwatch.Frequency;
+            Console.WriteLine($"  GLM vision encode: {totalMs:F0} ms, {numPatches} patches -> {mergedTokens} tokens");
+            return projected;
+        }
+
+        // ---- patch embedding ------------------------------------------------
+
+        private unsafe Tensor PatchEmbed(float[] pixelValues, int imgH, int imgW, int gridH, int gridW)
+        {
+            int numPatches = gridH * gridW;
+            int C = 3;
+            int P = _patchSize;
+            int patchStride = C * P * P;
+
+            string wName = _weights.ContainsKey("v.patch_embd.combined")
+                ? "v.patch_embd.combined" : "v.patch_embd.weight";
+            var convWeight = _weights[wName];
+            Tensor weightT = GetOrCreatePatchEmbedTransposed(convWeight, wName, patchStride);
+
+            int[] blockOrder = GetOrCreateBlockOrder(gridH, gridW);
+            var im2col = new Tensor(_allocator, DType.Float32, numPatches, patchStride);
+            float* im2colPtr = TensorComputePrimitives.GetFloatPointer(im2col);
+
+            fixed (float* pixSrc = pixelValues)
+            fixed (int* orderSrc = blockOrder)
+            {
+                long pixSrcL = (long)pixSrc;
+                long orderSrcL = (long)orderSrc;
+                long im2colL = (long)im2colPtr;
+                Parallel.For(0, gridH, brow =>
+                {
+                    float* pix = (float*)pixSrcL;
+                    int* order = (int*)orderSrcL;
+                    float* dst = (float*)im2colL;
+                    for (int col = 0; col < gridW; col++)
+                    {
+                        int destIdx = brow * gridW + col;
+                        int rasterIdx = order[destIdx];
+                        int py = rasterIdx / gridW;
+                        int px = rasterIdx - py * gridW;
+                        float* outRow = dst + (long)destIdx * patchStride;
+                        int yBase = py * P;
+                        int xBase = px * P;
+                        for (int c = 0; c < 3; c++)
+                        {
+                            long imgChannelOffset = (long)c * imgH * imgW;
+                            long outChannelOffset = (long)c * P * P;
+                            for (int ky = 0; ky < P; ky++)
+                            {
+                                long srcOffset = imgChannelOffset + (long)(yBase + ky) * imgW + xBase;
+                                Buffer.MemoryCopy(pix + srcOffset, outRow + outChannelOffset + (long)ky * P,
+                                    P * sizeof(float), P * sizeof(float));
+                            }
+                        }
+                    }
+                });
+            }
+
+            var result = new Tensor(_allocator, DType.Float32, numPatches, _hiddenSize);
+            Ops.Addmm(result, 0, result, 1.0f, im2col, weightT);
+            im2col.Dispose();
+
+            if (_weights.TryGetValue("v.patch_embd.bias", out var bias))
+                Ops.Add(result, result, bias);
+
+            return result;
+        }
+
+        private unsafe Tensor GetOrCreatePatchEmbedTransposed(Tensor convWeight, string weightName, int patchStride)
+        {
+            string key = weightName + ".2d.T";
+            if (_transposedWeights.TryGetValue(key, out var cached))
+                return cached;
+
+            int outDim = (int)convWeight.Sizes[0];
+            using var flat = convWeight.View(outDim, patchStride);
+            using var t = flat.Transpose();
+            var result = Ops.NewContiguous(t);
+            _transposedWeights[key] = result;
+            return result;
+        }
+
+        private int[] GetOrCreateBlockOrder(int gridH, int gridW)
+        {
+            long key = ((long)gridH << 32) | (uint)gridW;
+            if (_blockOrderCache.TryGetValue(key, out var cached))
+                return cached;
+
+            int merge = _spatialMergeSize;
+            var order = new int[gridH * gridW];
+            int idx = 0;
+            for (int bh = 0; bh < gridH; bh += merge)
+                for (int bw = 0; bw < gridW; bw += merge)
+                    for (int mh = 0; mh < merge; mh++)
+                        for (int mw = 0; mw < merge; mw++)
+                            order[idx++] = (bh + mh) * gridW + (bw + mw);
+
+            _blockOrderCache[key] = order;
+            return order;
+        }
+
+        // TS_GLM_VENC_FUSED=0 forces the managed per-block path (A/B + kill switch).
+        private static readonly bool s_fusedEncoderEnabled =
+            Environment.GetEnvironmentVariable("TS_GLM_VENC_FUSED") != "0";
+        private bool _fusedEncoderUnavailable;
+
+        /// <summary>Run all encoder blocks as one native GGML graph. Returns false
+        /// (and never retries) when the native path is unavailable.</summary>
+        private bool TryWholeEncoderFused(Tensor hidden, int numPatches, int headDim, int halfDim,
+            float[] cosTable, float[] sinTable)
+        {
+            if (_fusedEncoderUnavailable || _allocator is not TensorSharp.GGML.GgmlAllocator)
+                return false;
+
+            var ln1W = new Tensor[_blockCount];
+            var qkvW = new Tensor[_blockCount];
+            var qkvB = new Tensor[_blockCount];
+            var qnW = new Tensor[_blockCount];
+            var knW = new Tensor[_blockCount];
+            var outW = new Tensor[_blockCount];
+            var outB = new Tensor[_blockCount];
+            var ln2W = new Tensor[_blockCount];
+            var gateW = new Tensor[_blockCount];
+            var gateB = new Tensor[_blockCount];
+            var upW = new Tensor[_blockCount];
+            var upB = new Tensor[_blockCount];
+            var downW = new Tensor[_blockCount];
+            var downB = new Tensor[_blockCount];
+
+            for (int i = 0; i < _blockCount; i++)
+            {
+                string p = _blockPrefixes[i];
+                if (!_weights.TryGetValue($"{p}.ln1.weight", out ln1W[i]) ||
+                    !_weights.TryGetValue($"{p}.attn_qkv.weight", out qkvW[i]) ||
+                    !_weights.TryGetValue($"{p}.attn_qkv.bias", out qkvB[i]) ||
+                    !_weights.TryGetValue($"{p}.attn_q_norm.weight", out qnW[i]) ||
+                    !_weights.TryGetValue($"{p}.attn_k_norm.weight", out knW[i]) ||
+                    !_weights.TryGetValue($"{p}.attn_out.weight", out outW[i]) ||
+                    !_weights.TryGetValue($"{p}.attn_out.bias", out outB[i]) ||
+                    !_weights.TryGetValue($"{p}.ln2.weight", out ln2W[i]) ||
+                    !_weights.TryGetValue($"{p}.ffn_gate.weight", out gateW[i]) ||
+                    !_weights.TryGetValue($"{p}.ffn_gate.bias", out gateB[i]) ||
+                    !_weights.TryGetValue($"{p}.ffn_up.weight", out upW[i]) ||
+                    !_weights.TryGetValue($"{p}.ffn_up.bias", out upB[i]) ||
+                    !_weights.TryGetValue($"{p}.ffn_down.weight", out downW[i]) ||
+                    !_weights.TryGetValue($"{p}.ffn_down.bias", out downB[i]))
+                {
+                    _fusedEncoderUnavailable = true;
+                    return false;
+                }
+            }
+
+            try
+            {
+                bool ok = TensorSharp.GGML.GgmlBasicOps.GlmVisionEncoder(hidden, _eps,
+                    1f / MathF.Sqrt(headDim), _swigluLimit,
+                    numPatches, _numHeads, headDim, halfDim, cosTable, sinTable,
+                    ln1W, qkvW, qkvB, qnW, knW, outW, outB, ln2W,
+                    gateW, gateB, upW, upB, downW, downB);
+                if (!ok)
+                    _fusedEncoderUnavailable = true;
+                return ok;
+            }
+            catch (Exception ex)
+            {
+                Console.WriteLine($"GLM fused vision encoder unavailable ({ex.Message}); using the per-block path.");
+                _fusedEncoderUnavailable = true;
+                return false;
+            }
+        }
+
+        // ---- encoder block --------------------------------------------------
+
+        private unsafe void EncoderBlock(Tensor hidden, int blockIdx, int numPatches,
+            int headDim, int halfDim, float[] cosTable, float[] sinTable)
+        {
+            string prefix = _blockPrefixes[blockIdx];
+
+            // attention
+            using (var ln1 = RmsNormOp(hidden, $"{prefix}.ln1.weight"))
+            using (var attnOut = SelfAttention(ln1, prefix, numPatches, headDim, halfDim, cosTable, sinTable))
+            {
+                Ops.Add(hidden, hidden, attnOut);
+            }
+
+            // SwiGLU-clamp MLP
+            using var ln2 = RmsNormOp(hidden, $"{prefix}.ln2.weight");
+            using var gate = LinearForwardWithBias(ln2, $"{prefix}.ffn_gate.weight", $"{prefix}.ffn_gate.bias");
+            using var up = LinearForwardWithBias(ln2, $"{prefix}.ffn_up.weight", $"{prefix}.ffn_up.bias");
+            ClampSiluMul(gate, up);
+            using var down = LinearForwardWithBias(gate, $"{prefix}.ffn_down.weight", $"{prefix}.ffn_down.bias");
+            Ops.Add(hidden, hidden, down);
+        }
+
+        private unsafe Tensor SelfAttention(Tensor input, string prefix, int numPatches,
+            int headDim, int halfDim, float[] cosTable, float[] sinTable)
+        {
+            using var qkv = LinearForwardWithBias(input, $"{prefix}.attn_qkv.weight", $"{prefix}.attn_qkv.bias");
+
+            int hiddenSize = _hiddenSize;
+            int tripleHidden = 3 * hiddenSize;
+            var q = new Tensor(_allocator, DType.Float32, numPatches, hiddenSize);
+            var k = new Tensor(_allocator, DType.Float32, numPatches, hiddenSize);
+            var v = new Tensor(_allocator, DType.Float32, numPatches, hiddenSize);
+
+            float* qkvPtr = TensorComputePrimitives.GetFloatPointer(qkv);
+            float* qPtr = TensorComputePrimitives.GetFloatPointer(q);
+            float* kPtr = TensorComputePrimitives.GetFloatPointer(k);
+            float* vPtr = TensorComputePrimitives.GetFloatPointer(v);
+            long rowBytes = (long)hiddenSize * sizeof(float);
+            long qkvL = (long)qkvPtr, qL = (long)qPtr, kL = (long)kPtr, vL = (long)vPtr;
+
+            Parallel.For(0, numPatches, p =>
+            {
+                float* src = (float*)qkvL + (long)p * tripleHidden;
+                Buffer.MemoryCopy(src, (float*)qL + (long)p * hiddenSize, rowBytes, rowBytes);
+                Buffer.MemoryCopy(src + hiddenSize, (float*)kL + (long)p * hiddenSize, rowBytes, rowBytes);
+                Buffer.MemoryCopy(src + 2 * hiddenSize, (float*)vL + (long)p * hiddenSize, rowBytes, rowBytes);
+            });
+
+            // per-head RMS q/k norms (weight [headDim], one weight shared by
+            // every head), BEFORE the rope - transformers applies them to the
+            // projected heads.
+            ApplyHeadRmsNorm(q, numPatches, headDim, $"{prefix}.attn_q_norm.weight");
+            ApplyHeadRmsNorm(k, numPatches, headDim, $"{prefix}.attn_k_norm.weight");
+
+            ApplyVisionRoPE(q, numPatches, headDim, halfDim, cosTable, sinTable);
+            ApplyVisionRoPE(k, numPatches, headDim, halfDim, cosTable, sinTable);
+
+            float scale = 1f / MathF.Sqrt(headDim);
+
+            using var qR = q.View(numPatches, _numHeads, headDim);
+            using var kR = k.View(numPatches, _numHeads, headDim);
+            using var vR = v.View(numPatches, _numHeads, headDim);
+            using var qT0 = qR.Transpose(0, 1);
+            using var kT0 = kR.Transpose(0, 1);
+            using var vT0 = vR.Transpose(0, 1);
+            using var qHeads = Ops.NewContiguous(qT0);
+            using var kHeads = Ops.NewContiguous(kT0);
+            using var vHeads = Ops.NewContiguous(vT0);
+            q.Dispose();
+            k.Dispose();
+            v.Dispose();
+
+            using var kT = kHeads.Transpose(1, 2);
+
+            var scores = new Tensor(_allocator, DType.Float32, _numHeads, numPatches, numPatches);
+            Ops.AddmmBatch(scores, 0, scores, scale, qHeads, kT);
+            Ops.Softmax(scores, scores);
+
+            var attnOutput = new Tensor(_allocator, DType.Float32, _numHeads, numPatches, headDim);
+            Ops.AddmmBatch(attnOutput, 0, attnOutput, 1.0f, scores, vHeads);
+            scores.Dispose();
+
+            using var transposed = attnOutput.Transpose(0, 1);
+            using var contiguous = Ops.NewContiguous(transposed);
+            using var flat = contiguous.View(numPatches, _hiddenSize);
+            attnOutput.Dispose();
+
+            return LinearForwardWithBias(flat, $"{prefix}.attn_out.weight", $"{prefix}.attn_out.bias");
+        }
+
+        /// <summary>RMS-normalize each head's slice of every row against a shared
+        /// [headDim] weight, in place.</summary>
+        private unsafe void ApplyHeadRmsNorm(Tensor data, int numPatches, int headDim, string weightName)
+        {
+            if (!_weights.TryGetValue(weightName, out var w))
+                return;
+            float* ptr = TensorComputePrimitives.GetFloatPointer(data);
+            float* wp = TensorComputePrimitives.GetFloatPointer(w);
+            int numHeads = _numHeads;
+            float eps = _eps;
+            long ptrL = (long)ptr, wpL = (long)wp;
+
+            Parallel.For(0, numPatches, p =>
+            {
+                float* dp = (float*)ptrL + (long)p * numHeads * headDim;
+                float* wl = (float*)wpL;
+                for (int h = 0; h < numHeads; h++)
+                {
+                    float* head = dp + (long)h * headDim;
+                    double ss = 0;
+                    for (int d = 0; d < headDim; d++)
+                        ss += (double)head[d] * head[d];
+                    float inv = 1.0f / MathF.Sqrt((float)(ss / headDim) + eps);
+                    for (int d = 0; d < headDim; d++)
+                        head[d] = head[d] * inv * wl[d];
+                }
+            });
+        }
+
+        private unsafe void ApplyVisionRoPE(Tensor data, int numPatches, int headDim, int halfDim,
+            float[] cosTable, float[] sinTable)
+        {
+            float* ptr = TensorComputePrimitives.GetFloatPointer(data);
+            int numHeads = _numHeads;
+            int vLen = Vector<float>.Count;
+
+            fixed (float* cosPtr = cosTable, sinPtr = sinTable)
+            {
+                long ptrL = (long)ptr, cosPtrL = (long)cosPtr, sinPtrL = (long)sinPtr;
+                Parallel.For(0, numPatches, p =>
+                {
+                    float* dataPtr = (float*)ptrL;
+                    float* cTbl = (float*)cosPtrL;
+                    float* sTbl = (float*)sinPtrL;
+                    int cosBase = p * halfDim;
+
+                    for (int h = 0; h < numHeads; h++)
+                    {
+                        float* head = dataPtr + ((long)p * numHeads + h) * headDim;
+                        float* head1 = head + halfDim;
+                        float* cosRow = cTbl + cosBase;
+                        float* sinRow = sTbl + cosBase;
+
+                        int d = 0;
+                        for (; d <= halfDim - vLen; d += vLen)
+                        {
+                            var x0 = TensorComputePrimitives.LoadVector(head + d);
+                            var x1 = TensorComputePrimitives.LoadVector(head1 + d);
+                            var cv = TensorComputePrimitives.LoadVector(cosRow + d);
+                            var sv = TensorComputePrimitives.LoadVector(sinRow + d);
+                            TensorComputePrimitives.StoreVector(head + d, x0 * cv - x1 * sv);
+                            TensorComputePrimitives.StoreVector(head1 + d, x0 * sv + x1 * cv);
+                        }
+                        for (; d < halfDim; d++)
+                        {
+                            float x0 = head[d];
+                            float x1 = head1[d];
+                            head[d] = x0 * cosRow[d] - x1 * sinRow[d];
+                            head1[d] = x0 * sinRow[d] + x1 * cosRow[d];
+                        }
+                    }
+                });
+            }
+        }
+
+        /// <summary>gate = silu(min(gate, L)) * clamp(up, -L, L), in place into gate.</summary>
+        private unsafe void ClampSiluMul(Tensor gate, Tensor up)
+        {
+            long total = gate.ElementCount();
+            int rows = (int)gate.Sizes[0];
+            long cols = total / rows;
+            float* gp = TensorComputePrimitives.GetFloatPointer(gate);
+            float* upp = TensorComputePrimitives.GetFloatPointer(up);
+            float limit = _swigluLimit;
+            long gL = (long)gp, uL = (long)upp;
+
+            Parallel.For(0, rows, r =>
+            {
+                float* g = (float*)gL + r * cols;
+                float* u = (float*)uL + r * cols;
+                for (long i = 0; i < cols; i++)
+                {
+                    float gv = g[i];
+                    if (limit > 0f && gv > limit) gv = limit;
+                    float uv = u[i];
+                    if (limit > 0f)
+                    {
+                        if (uv > limit) uv = limit;
+                        else if (uv < -limit) uv = -limit;
+                    }
+                    float silu = gv / (1.0f + MathF.Exp(-gv));
+                    g[i] = silu * uv;
+                }
+            });
+        }
+
+        private static float Erf(float x)
+        {
+            // Abramowitz & Stegun 7.1.26 in double precision: max abs error 1.5e-7.
+            double t = 1.0 / (1.0 + 0.3275911 * Math.Abs(x));
+            double y = 1.0 - (((((1.061405429 * t - 1.453152027) * t) + 1.421413741) * t - 0.284496736) * t
+                              + 0.254829592) * t * Math.Exp(-(double)x * x);
+            return (float)(x >= 0 ? y : -y);
+        }
+
+        private unsafe void GeluErf(Tensor t)
+        {
+            long total = t.ElementCount();
+            int rows = (int)t.Sizes[0];
+            long cols = total / rows;
+            float* ptr = TensorComputePrimitives.GetFloatPointer(t);
+            long ptrL = (long)ptr;
+            float invSqrt2 = 1.0f / MathF.Sqrt(2.0f);
+
+            Parallel.For(0, rows, r =>
+            {
+                float* row = (float*)ptrL + r * cols;
+                for (long i = 0; i < cols; i++)
+                    row[i] = 0.5f * row[i] * (1.0f + Erf(row[i] * invSqrt2));
+            });
+        }
+
+        private Tensor RmsNormOp(Tensor input, string weightName)
+        {
+            return Ops.RMSNorm(null, input, _weights[weightName], null, _eps);
+        }
+
+        private Tensor LinearForwardWithBias(Tensor input, string weightName, string biasName)
+        {
+            Tensor weightT = GetOrCreateTransposedWeight(weightName);
+            int seqLen = (int)input.Sizes[0];
+            int outDim = (int)weightT.Sizes[1];
+
+            var result = new Tensor(_allocator, DType.Float32, seqLen, outDim);
+            Tensor contiguousInput = input.IsContiguous() ? null : Ops.NewContiguous(input);
+            Tensor src = contiguousInput ?? input;
+            Ops.Addmm(result, 0, result, 1.0f, src, weightT);
+            contiguousInput?.Dispose();
+
+            if (biasName != null && _weights.TryGetValue(biasName, out var bias))
+                Ops.Add(result, result, bias);
+
+            return result;
+        }
+
+        private Tensor GetOrCreateTransposedWeight(string weightName)
+        {
+            if (_transposedWeights.TryGetValue(weightName, out var transposed))
+                return transposed;
+
+            Tensor weight = _weights[weightName];
+            using var weightViewT = weight.Transpose();
+            transposed = Ops.NewContiguous(weightViewT);
+            _transposedWeights[weightName] = transposed;
+            return transposed;
+        }
+
+        private RopeCache GetOrCreateRopeCache(int gridH, int gridW, int numPatches, int halfDim)
+        {
+            long key = ((long)gridH << 32) | (uint)gridW;
+            if (_ropeCache.TryGetValue(key, out var cache))
+                return cache;
+
+            int[] gridY = new int[numPatches];
+            int[] gridX = new int[numPatches];
+            int idx = 0;
+            for (int bh = 0; bh < gridH; bh += _spatialMergeSize)
+                for (int bw = 0; bw < gridW; bw += _spatialMergeSize)
+                    for (int mh = 0; mh < _spatialMergeSize; mh++)
+                        for (int mw = 0; mw < _spatialMergeSize; mw++)
+                        {
+                            gridY[idx] = bh + mh;
+                            gridX[idx] = bw + mw;
+                            idx++;
+                        }
+
+            // Blocked frequency layout, exactly as Qwen3-VL / ggml VISION rope:
+            // the first halfDim/2 rotary pairs encode the ROW position, the next
+            // halfDim/2 the COLUMN position, each over theta^(2j/halfDim) bands.
+            int numBands = halfDim / 2;
+            float[] cosTable = new float[numPatches * halfDim];
+            float[] sinTable = new float[numPatches * halfDim];
+            float[] invFreqs = new float[numBands];
+            for (int j = 0; j < numBands; j++)
+                invFreqs[j] = 1f / MathF.Pow(_ropeTheta, (2f * j) / halfDim);
+
+            for (int p = 0; p < numPatches; p++)
+            {
+                int baseIdx = p * halfDim;
+                for (int j = 0; j < numBands; j++)
+                {
+                    float angleY = gridY[p] * invFreqs[j];
+                    float angleX = gridX[p] * invFreqs[j];
+                    cosTable[baseIdx + j] = MathF.Cos(angleY);
+                    sinTable[baseIdx + j] = MathF.Sin(angleY);
+                    cosTable[baseIdx + numBands + j] = MathF.Cos(angleX);
+                    sinTable[baseIdx + numBands + j] = MathF.Sin(angleX);
+                }
+            }
+
+            cache = new RopeCache { CosTable = cosTable, SinTable = sinTable };
+            _ropeCache[key] = cache;
+            return cache;
+        }
+
+        public void Dispose()
+        {
+            foreach (var w in _transposedWeights.Values)
+                w.Dispose();
+            _transposedWeights.Clear();
+            foreach (var w in _weights.Values)
+                w.Dispose();
+            _weights.Clear();
+            _ropeCache.Clear();
+            _blockOrderCache.Clear();
+        }
+    }
+}
diff --git a/TensorSharp.Models/Models/MuseGlimmer/MuseGlimmerModel.DFlash.cs b/TensorSharp.Models/Models/MuseGlimmer/MuseGlimmerModel.DFlash.cs
index c77c3c23..da0a41c3 100644
--- a/TensorSharp.Models/Models/MuseGlimmer/MuseGlimmerModel.DFlash.cs
+++ b/TensorSharp.Models/Models/MuseGlimmer/MuseGlimmerModel.DFlash.cs
@@ -1,4 +1,4 @@
-// Copyright (c) Zhongkai Fu. All rights reserved.
+// Copyright (c) Zhongkai Fu. All rights reserved.
 // https://github.com/zhongkaifu/TensorSharp
 //
 // This file is part of TensorSharp.
@@ -9,42 +9,19 @@
 // MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the BSD-3-Clause License for more details.
 //
 // ---------------------------------------------------------------------------
-// DFlash speculative drafting for Muse-Glimmer.
+// Muse-Glimmer's half of DFlash speculative drafting.
 //
-// DFlash (llama.cpp src/models/dflash.cpp) is a BLOCK drafter: one forward pass
-// proposes the whole speculative window, so it plugs into the shared
-// draft/verify/rollback core through ISpeculativeModel.DraftBlock instead
-// of the per-token DraftStep, and it reports a WIDE hidden row through
-// SpecFeatureSize -- the concatenated per-layer input residuals of the target
-// layers its encoder consumes (5 x 6656 = 33280 for Muse-Glimmer 30B).
+// The drafter itself - loading, the KV ring, the encoder, the KV injection and
+// the block draft, in both the per-op and the fused flavour - is shared and
+// lives in TensorSharp.Models/Speculative/ModelBase.DFlash*.cs. What is left
+// here is the part that genuinely belongs to this target model:
 //
-// Three passes, transcribed from llama_model_dflash::graph:
+//   * the trunk forward that captures the per-layer residuals the drafter's
+//     encoder consumes (SpecForward / SpecForwardCore), and
+//   * the ISpeculativeModel wiring that points the shared draft/verify loop at
+//     the shared drafter.
 //
-//   PASS A -- encoder (graph<true>)
-//       feat = concat(target input residual of layers dflash.target_layers)
-//       g    = rmsnorm(fc @ feat, enc.output_norm, eps)          [1 row per position]
-//
-//   PASS B -- KV injection (graph<false>, the ubatch.embd branch)
-//       K = rope_neox(headnorm(attn_k @ g, attn_k_norm), target position)
-//       V = attn_v @ g                                          (no norm, no rope)
-//       ring[pos % ringRows] <- K, V                            per draft layer
-//       No Q, no attention, no FFN, no output.
-//
-//   PASS C -- block draft (graph<false>, the token branch)
-//       ids  = [anchor, MASK x (block_size-1)]  at positions p .. p+B-1
-//       inpL = target token_embd[ids]                           (no embedding scale)
-//       5 x { pre-norm, QK head-norm, NeoX rope, attention over
-//             [ring window | this block's own B keys] -- NON-CAUSAL inside the
-//             block, SWA-masked against the ring -- , o_proj, residual,
-//             ffn-norm, SwiGLU, residual }
-//       logits = target output.weight @ rmsnorm(inpL, output_norm)
-//   The drafter's logits get NEITHER the target's logit_scale NOR its tanh
-//   softcap: llama.cpp's dflash graph ends at build_lora_mm(output, cur). argmax
-//   is invariant to both, but the softmax CONFIDENCE is not, so the per-position
-//   acceptance probabilities handed to the executor are the softmax of the RAW
-//   drafter logits.
-//
-// Rollback is free: the trunk verify writes correct KV for every row it
+// Rollback is free here: the trunk verify writes correct KV for every row it
 // processes into the linear (non-circular) cache and Muse-Glimmer has no
 // recurrent state, so partial acceptance only rewinds the position counter
 // (SpecVerifyPersistsAcceptedKv).
@@ -63,85 +40,6 @@ namespace TensorSharp.Models
 {
     public partial class MuseGlimmerModel : ModelBase, ISpeculativeModel
     {
-        // Per-layer weight-name slots (index into _dflashLayerNames[il]).
-        private const int DfAttnNorm = 0;
-        private const int DfAttnQ = 1;
-        private const int DfAttnK = 2;
-        private const int DfAttnV = 3;
-        private const int DfAttnQNorm = 4;
-        private const int DfAttnKNorm = 5;
-        private const int DfAttnOutput = 6;
-        private const int DfFfnNorm = 7;
-        private const int DfFfnGate = 8;
-        private const int DfFfnUp = 9;
-        private const int DfFfnDown = 10;
-
-        /// <summary>
-        /// Default prompt-prefill chunk the speculative executor should use, and
-        /// the reason it is not the model's own 2048.
-        ///
-        /// This value drives the TRUNK forward, not only the drafter, so it decides
-        /// how many full 52-layer Muse-Glimmer forwards a prompt costs. It used to
-        /// be 128, chosen purely to bound the host-side capture buffer (one row is
-        /// <see cref="DFlashConfig.FeatureSize"/> = 33280 floats = 130 KB), on the
-        /// assumption that "the trunk's per-chunk fixed overhead is small next to a
-        /// 128-row forward". Measured against llama.cpp it is not: the extra cost
-        /// per chunk is a FLAT ~60 ms from 2K to 128K of context (a 52-layer graph
-        /// rebuild plus a DFlash host round trip), against ~13 ms of useful work in
-        /// a 128-row chunk. At a 124K prompt that is 980 trunk forwards instead of
-        /// 61 - 58 s added to a 112 s prefill, which was the whole of the
-        /// 0.69x-versus-llama.cpp DFlash prefill ratio.
-        ///
-        /// 1024 keeps one 130 MB host buffer (PrefillStep now shifts the pairing in
-        /// place instead of keeping a second one) and removes 87% of the extra
-        /// chunks. It also stays well inside both hard limits: the drafter's ring is
-        /// <see cref="DFlashConfig.RingRows"/> = 2080 rows, and the trunk's SWA ring
-        /// refuses a forward wider than rows - n_swa = 2304.
-        /// Override with TS_DFLASH_PREFILL_CHUNK.
-        /// </summary>
-        private const int DFlashPrefillChunkDefault = 1024;
-
-        private int _dflashPrefillChunk;
-
-        private int ResolveDFlashPrefillChunk()
-        {
-            int chunk = DFlashPrefillChunkDefault;
-            string raw = Environment.GetEnvironmentVariable("TS_DFLASH_PREFILL_CHUNK");
-            if (!string.IsNullOrWhiteSpace(raw) && int.TryParse(raw, out int parsed) && parsed > 0)
-                chunk = parsed;
-
-            // Never exceed what either ring can absorb in one forward: the drafter
-            // would alias two live positions onto one ring slot, and the trunk's
-            // ForwardCore throws outright.
-            int draftCap = _dflash != null ? _dflash.RingRows - _dflash.BlockSize - 1 : chunk;
-            int trunkCap = _kvSwaRows > 0 ? _kvSwaRows - _slidingWindow : chunk;
-            chunk = Math.Min(chunk, Math.Max(1, draftCap));
-            chunk = Math.Min(chunk, Math.Max(1, trunkCap));
-            return Math.Max(1, chunk);
-        }
-
-        private DFlashConfig _dflash;
-        private bool _hasDFlash;
-
-        /// <summary>target layer index -> its column block in a feature row, or -1.</summary>
-        private int[] _dflashCaptureSlot;
-
-        /// <summary>Per draft layer, the 11 weight names of that block.</summary>
-        private string[][] _dflashLayerNames;
-
-        /// <summary>The drafter's own KV ring, [numKVHeads, ringRows, headDim] per
-        /// draft layer, indexed by (absolute position % ringRows).</summary>
-        private Tensor[] _dflashRingK;
-        private Tensor[] _dflashRingV;
-        private int _dflashRingRows;
-
-        /// <summary>True when a usable DFlash drafter is attached to this model.</summary>
-        public bool HasDFlash => _hasDFlash;
-
-        // ====================================================================
-        // construction / loading
-        // ====================================================================
-
         /// <summary>Path of the DFlash drafter GGUF, if one was configured
         /// (<c>--draft-model</c> / <c>TS_MUSE_GLIMMER_DFLASH</c>).</summary>
         internal static string ResolveDFlashPath(string explicitPath)
@@ -152,209 +50,14 @@ internal static string ResolveDFlashPath(string explicitPath)
             return string.IsNullOrWhiteSpace(path) ? null : path;
         }
 
-        /// <summary>
-        /// Loads the DFlash drafter GGUF and attaches it to this target model. Its
-        /// tensors are merged into the shared weight dictionaries under the
-        /// "dflash." prefix so the existing matmul/norm machinery serves them (the
-        /// same trick Gemma4Model.LoadMtpDraftTensors uses with "mtp."); the drafter
-        /// borrows the TARGET's token_embd.weight and output.weight, which the file
-        /// does not carry.
-        /// </summary>
-        public void LoadDFlashDraftWeights(string ggufPath)
-        {
-            if (string.IsNullOrEmpty(ggufPath) || !System.IO.File.Exists(ggufPath))
-                throw new System.IO.FileNotFoundException("Muse-Glimmer DFlash drafter GGUF not found.", ggufPath);
-
-            using var draft = new GgufFile(ggufPath);
-            var cfg = DFlashConfig.FromGguf(draft);
-
-            if (cfg.HiddenSize != Config.HiddenSize)
-            {
-                throw new InvalidOperationException(
-                    $"DFlash embedding_length {cfg.HiddenSize} != target hidden size {Config.HiddenSize}.");
-            }
-            foreach (int lid in cfg.TargetLayerIds)
-            {
-                if (lid < 0 || lid >= Config.NumLayers)
-                {
-                    throw new InvalidOperationException(
-                        $"DFlash target layer {lid} is outside the target's {Config.NumLayers} layers.");
-                }
-            }
-            if (cfg.BlockSize > cfg.RingRows)
-                throw new InvalidOperationException($"DFlash block_size {cfg.BlockSize} exceeds the ring ({cfg.RingRows} rows).");
-
-            LoadDFlashDraftTensors(draft);
-
-            _dflash = cfg;
-            _dflashLayerNames = new string[cfg.NumLayers][];
-            for (int il = 0; il < cfg.NumLayers; il++)
-            {
-                string p = $"{DFlashConfig.WeightPrefix}blk.{il}.";
-                _dflashLayerNames[il] = new[]
-                {
-                    p + "attn_norm.weight",     // 0
-                    p + "attn_q.weight",        // 1
-                    p + "attn_k.weight",        // 2
-                    p + "attn_v.weight",        // 3
-                    p + "attn_q_norm.weight",   // 4
-                    p + "attn_k_norm.weight",   // 5
-                    p + "attn_output.weight",   // 6
-                    p + "ffn_norm.weight",      // 7
-                    p + "ffn_gate.weight",      // 8
-                    p + "ffn_up.weight",        // 9
-                    p + "ffn_down.weight",      // 10
-                };
-            }
-
-            if (!VerifyDFlashTensors(out string missing))
-            {
-                Console.WriteLine($"  DFlash drafter GGUF loaded but '{missing}' is missing; DFlash drafting disabled.");
-                _dflash = null;
-                _dflashLayerNames = null;
-                return;
-            }
-
-            _dflashCaptureSlot = new int[Config.NumLayers];
-            for (int l = 0; l < Config.NumLayers; l++)
-                _dflashCaptureSlot[l] = -1;
-            for (int i = 0; i < cfg.TargetLayerIds.Length; i++)
-                _dflashCaptureSlot[cfg.TargetLayerIds[i]] = i;
-
-            _dflashRingRows = cfg.RingRows;
-            _dflashRingK = new Tensor[cfg.NumLayers];
-            _dflashRingV = new Tensor[cfg.NumLayers];
-            for (int il = 0; il < cfg.NumLayers; il++)
-            {
-                _dflashRingK[il] = new Tensor(_allocator, DType.Float32, cfg.NumKVHeads, _dflashRingRows, cfg.HeadDim);
-                _dflashRingV[il] = new Tensor(_allocator, DType.Float32, cfg.NumKVHeads, _dflashRingRows, cfg.HeadDim);
-                // Unconditional zero fill (not InitializeCacheTensor, which skips
-                // GgmlCuda): the ring is only ever read over positions that have
-                // been written, but a finite ring keeps a mis-sized window from
-                // silently producing NaNs instead of failing loudly.
-                Ops.Fill(_dflashRingK[il], 0f);
-                Ops.Fill(_dflashRingV[il], 0f);
-            }
-
-            _hasDFlash = true;
-
-            long ringBytes = 2L * cfg.NumLayers * cfg.NumKVHeads * _dflashRingRows * cfg.HeadDim * sizeof(float);
-            Console.WriteLine($"  DFlash drafter ready: {cfg}");
-            Console.WriteLine($"  DFlash KV ring: {_dflashRingRows} rows x {cfg.NumLayers} layers ({ringBytes / (1024 * 1024)} MB F32)");
-        }
-
-        /// <summary>
-        /// Merges every tensor of the drafter GGUF into the shared weight
-        /// dictionaries under the "dflash." prefix. Byte-for-byte the same shape as
-        /// Gemma4Model.LoadMtpDraftTensors (which uses "mtp."), minus the
-        /// converter-spelling normalization DFlash does not need: the drafter's
-        /// tensor names are already the final ones.
-        /// </summary>
-        private unsafe void LoadDFlashDraftTensors(GgufFile draft)
-        {
-            foreach (var kv in draft.Tensors)
-            {
-                var info = kv.Value;
-                string name = DFlashConfig.WeightPrefix + info.Name;
-                long byteCount = draft.GetTensorByteCount(info);
-
-                if (IsQuantizedLinearWeight(info))
-                {
-                    if (IsGgmlBackend)
-                        EnsureQuantBackendAvailable();
-                    IntPtr ptr = QuantizedWeight.AllocateBuffer(byteCount);
-                    draft.ReadTensorDataToNative(info, ptr, byteCount);
-                    _quantWeights[name] = new QuantizedWeight(ptr, byteCount, (int)info.Type, (long)info.Shape[0], (long)info.Shape[1]);
-                }
-                else
-                {
-                    long numElements = info.NumElements;
-                    long[] tsShape = new long[info.Shape.Length];
-                    for (int i = 0; i < info.Shape.Length; i++)
-                        tsShape[i] = (long)info.Shape[info.Shape.Length - 1 - i];
-
-                    var tensor = new Tensor(_allocator, DType.Float32, tsShape);
-                    IntPtr destPtr = TensorComputePrimitives.GetStoragePointer(tensor);
-                    if (info.Type == GgmlTensorType.F32)
-                    {
-                        draft.ReadTensorDataToFloat32Native(info, destPtr, numElements);
-                    }
-                    else
-                    {
-                        IntPtr tempPtr = QuantizedWeight.AllocateBuffer(byteCount);
-                        try
-                        {
-                            draft.ReadTensorDataToNative(info, tempPtr, byteCount);
-                            NativeDequant.DequantizeToFloat32Native((int)info.Type, tempPtr, destPtr, numElements);
-                        }
-                        finally
-                        {
-                            QuantizedWeight.FreeBuffer(tempPtr);
-                        }
-                    }
-                    _weights[name] = tensor;
-                }
-            }
-        }
-
-        /// <summary>True when <paramref name="name"/> resolves to something
-        /// <see cref="ModelBase.LinearForward"/> can multiply by.</summary>
-        private bool HasDFlashLinear(string name)
-            => _quantWeights.ContainsKey(name) || _weights.ContainsKey(name);
-
-        private bool VerifyDFlashTensors(out string missing)
-        {
-            string[] globals =
-            {
-                DFlashConfig.WeightPrefix + "fc.weight",
-                DFlashConfig.WeightPrefix + "enc.output_norm.weight",
-                DFlashConfig.WeightPrefix + "output_norm.weight",
-            };
-            foreach (string g in globals)
-            {
-                bool ok = g.EndsWith("norm.weight", StringComparison.Ordinal)
-                    ? _weights.ContainsKey(g)
-                    : HasDFlashLinear(g);
-                if (!ok) { missing = g; return false; }
-            }
-
-            for (int il = 0; il < _dflash.NumLayers; il++)
-            {
-                string[] n = _dflashLayerNames[il];
-                foreach (int slot in new[] { DfAttnNorm, DfAttnQNorm, DfAttnKNorm, DfFfnNorm })
-                {
-                    if (!_weights.ContainsKey(n[slot])) { missing = n[slot]; return false; }
-                }
-                foreach (int slot in new[] { DfAttnQ, DfAttnK, DfAttnV, DfAttnOutput, DfFfnGate, DfFfnUp, DfFfnDown })
-                {
-                    if (!HasDFlashLinear(n[slot])) { missing = n[slot]; return false; }
-                }
-            }
-
-            // The drafter has no LM head of its own: it borrows the target's.
-            if (!HasDFlashLinear(TargetOutputWeightName)) { missing = TargetOutputWeightName; return false; }
-            if (!HasDFlashLinear("token_embd.weight")) { missing = "token_embd.weight"; return false; }
-
-            missing = null;
-            return true;
-        }
-
-        /// <summary>The target's LM head, which the drafter borrows.</summary>
-        private string TargetOutputWeightName => _hasTiedOutput ? "token_embd.weight" : "output.weight";
+        /// <summary>The trunk's own ceiling on one speculative prefill chunk: its
+        /// ForwardCore throws outright on a batch wider than the SWA ring can
+        /// hold.</summary>
+        private protected override int DFlashTrunkPrefillChunkCap
+            => _kvSwaRows > 0 ? _kvSwaRows - _slidingWindow : int.MaxValue;
 
-        /// <summary>Called from <see cref="Dispose"/> in MuseGlimmerModel.cs. The
-        /// drafter's weights live in the shared dictionaries and are released by
-        /// <see cref="ModelBase.Dispose"/>; only the rings are ours.</summary>
-        private void DisposeDFlash()
-        {
-            if (_dflashRingK != null)
-                foreach (var t in _dflashRingK) t?.Dispose();
-            if (_dflashRingV != null)
-                foreach (var t in _dflashRingV) t?.Dispose();
-            _dflashRingK = null;
-            _dflashRingV = null;
-            _hasDFlash = false;
-        }
+        /// <summary>Called from <see cref="Dispose"/> in MuseGlimmerModel.cs.</summary>
+        private void DisposeDFlashDrafter() => DisposeDFlash();
 
         // ====================================================================
         // ISpeculativeModel
@@ -364,7 +67,7 @@ private void DisposeDFlash()
 
         /// <summary>DFlash proposes a whole block per pass, so it is served by
         /// <see cref="BlockDraftSpeculator"/>.</summary>
-        public DraftHeadKind DraftHeadKind => _hasDFlash ? DraftHeadKind.Block : DraftHeadKind.None;
+        public DraftHeadKind DraftHeadKind => HasDFlash ? DraftHeadKind.Block : DraftHeadKind.None;
 
         /// <summary>Speculation is profitable exactly when a drafter is loaded: the
         /// verify batch is a single ordinary multi-token Muse-Glimmer forward (the
@@ -377,30 +80,15 @@ private void DisposeDFlash()
         public bool SpeculationProfitable => HasDFlash;
 
         /// <summary>The concatenated target-layer input residuals the encoder
-        /// consumes: TargetLayerIds.Length * hidden (5 * 6656 = 33280), NOT the
-        /// model's hidden size.</summary>
+        /// consumes (5 * 6656 = 33280), NOT the model's hidden size.</summary>
         public int SpecFeatureSize => _dflash != null ? _dflash.FeatureSize : Config.HiddenSize;
 
-        /// <summary>Number of DRAFTS a block produces, i.e. block_size - 1 (15):
-        /// row 0 of the block is the anchor's own prediction and plain DFlash
-        /// discards it (only DSpark consumes it).</summary>
-        public int DraftBlockSize => _hasDFlash ? _dflash.MaxDraftTokens : 0;
+        /// <summary>Number of DRAFTS a block produces, i.e. block_size - 1: row 0 of
+        /// the block is the anchor's own prediction and DFlash discards it.</summary>
+        public int DraftBlockSize => HasDFlash ? _dflash.MaxDraftTokens : 0;
 
-        /// <summary>See <see cref="DFlashPrefillChunkDefault"/>.</summary>
-        public int SpecPrefillChunkSize
-        {
-            get
-            {
-                // Resolved lazily and only once the drafter is loaded: the caps
-                // below come off the drafter's ring and the trunk's SWA ring, so
-                // answering before either exists would cache a wrong value.
-                if (_dflash == null)
-                    return 0;
-                if (_dflashPrefillChunk <= 0)
-                    _dflashPrefillChunk = ResolveDFlashPrefillChunk();
-                return _dflashPrefillChunk;
-            }
-        }
+        /// <summary>See <c>ModelBase.DFlashPrefillChunkSize</c>.</summary>
+        public int SpecPrefillChunkSize => DFlashPrefillChunkSize;
 
         /// <summary>The drafter is fed from the host: every prefill chunk hands its
         /// per-row features back so <see cref="DraftCatchUp"/> can fill the ring.</summary>
@@ -411,6 +99,51 @@ public int SpecPrefillChunkSize
         /// state, so a partial acceptance only needs a position rewind.</summary>
         public bool SpecVerifyPersistsAcceptedKv => true;
 
+        public int DraftBlock(int lastToken, float[] hPrev, int position, int[] draftOut, float[] confOut)
+            => DFlashPropose(lastToken, hPrev, position, draftOut, confOut);
+
+        public void DraftCatchUp(int[] tokens, float[] hRows, int startPos)
+            => DFlashCommit(tokens, hRows, startPos);
+
+        /// <summary>DFlash drafts whole blocks; the per-token entry point is never
+        /// used.</summary>
+        public void DraftStep(int token, float[] hPrev, int pos, float[] logitsOut, float[] hOut)
+            => throw new NotSupportedException("Muse-Glimmer DFlash drafts whole blocks; use DraftBlock.");
+
+        /// <summary>Pre-grows the trunk KV cache to cover the whole speculative
+        /// window. The drafter's ring is fixed-size and needs nothing.</summary>
+        public void SpecEnsureCapacity(int requiredSeqLen)
+        {
+            if (!HasDFlash)
+                return;
+            EnsureCacheCapacity(requiredSeqLen);
+        }
+
+        /// <summary>No recurrent (GDN/SSM) state in Muse-Glimmer -- drafting and
+        /// verifying are stateless given the KV cache.</summary>
+        public void SpecSnapshotRecurrentState() { }
+
+        /// <summary>See <see cref="SpecSnapshotRecurrentState"/>.</summary>
+        public void SpecRestoreRecurrentState() { }
+
+        /// <summary>
+        /// Rewinds the trunk KV position counter after rejected speculative tokens.
+        /// Rows past <paramref name="length"/> are overwritten by later writes and
+        /// the causal/SWA mask never reads past the live position, so no data moves.
+        /// The drafter's ring is untouched: it only ever holds committed positions,
+        /// and the next catch-up rewrites the ones a rejected tail would have needed.
+        /// </summary>
+        public void SpecRewindCache(int length)
+        {
+            if (length < 0 || length > _cacheSeqLen)
+            {
+                throw new ArgumentOutOfRangeException(nameof(length),
+                    $"Rewind length {length} outside [0, {_cacheSeqLen}].");
+            }
+            _cacheSeqLen = length;
+            _cachedSWAMaskStartPos = -1;
+        }
+
         /// <summary>
         /// Trunk forward that additionally captures, per row, the concatenated
         /// INPUT residuals of the target layers in dflash.target_layers (row stride
@@ -420,7 +153,8 @@ public int SpecPrefillChunkSize
         /// </summary>
         public void SpecForward(int[] tokens, float[] hAllOut, float[] logitsOut, bool allLogitsRows)
         {
-            RequireDFlash();
+            if (!HasDFlash)
+                throw new InvalidOperationException("No DFlash drafter is loaded for this Muse-Glimmer model.");
             if (tokens == null || tokens.Length == 0)
                 throw new ArgumentException("Token batch must not be empty.", nameof(tokens));
             // No lock here: like Gemma4Model.SpecForward, the caller owns
@@ -498,7 +232,7 @@ private unsafe void SpecForwardCore(int[] tokens, float[] hAllOut, float[] logit
             Tensor normed = RMSNormOp(hidden, "output_norm.weight");
             hidden.Dispose();
 
-            string outputWeight = TargetOutputWeightName;
+            string outputWeight = DFlashTargetOutputWeightName;
             if (allLogitsRows)
             {
                 t0 = Stopwatch.GetTimestamp();
@@ -612,525 +346,5 @@ private bool TryFusedSpecForward(Tensor hidden, int seqLen, int startPos,
         }
 
         private float[] _specCaptureScratch;
-
-        /// <summary>Copies one target layer's input residual into its column block
-        /// of the caller's feature rows.</summary>
-        private unsafe void DFlashCaptureFeature(Tensor hidden, int slot, int seqLen, float[] hAllOut, bool lastRowOnly)
-        {
-            int hs = Config.HiddenSize;
-            int feat = _dflash.FeatureSize;
-            long rowBytes = (long)hs * sizeof(float);
-            float* src = GetFloatPtr(hidden);
-            fixed (float* dst0 = hAllOut)
-            {
-                if (lastRowOnly)
-                {
-                    Buffer.MemoryCopy(src + (long)(seqLen - 1) * hs, dst0 + (long)slot * hs, rowBytes, rowBytes);
-                    return;
-                }
-                for (int r = 0; r < seqLen; r++)
-                    Buffer.MemoryCopy(src + (long)r * hs, dst0 + (long)r * feat + (long)slot * hs, rowBytes, rowBytes);
-            }
-        }
-
-        /// <summary>
-        /// Replays committed trunk positions through the drafter so its KV ring
-        /// tracks the real context. Row k of <paramref name="hRows"/> is the feature
-        /// row of the token PRECEDING tokens[k] -- i.e. of absolute position
-        /// <paramref name="startPos"/> + k - 1, which is exactly the position whose
-        /// drafter key it writes (hence the -1, as in DeepSeek4Model.DraftCatchUp).
-        /// </summary>
-        public void DraftCatchUp(int[] tokens, float[] hRows, int startPos)
-        {
-            RequireDFlash();
-            if (tokens == null || tokens.Length == 0 || hRows == null)
-                return;
-            DFlashCatchUp(hRows, tokens.Length, startPos - 1);
-        }
-
-        /// <summary>Encodes <paramref name="rows"/> feature rows whose first row is
-        /// at absolute position <paramref name="firstPos"/> and writes the resulting
-        /// keys/values into the ring. Rows before position 0 (the zeroed "hidden
-        /// state of the token before the prompt") are skipped, and rows older than
-        /// the ring modulus are dropped -- dropping them is what keeps the ring
-        /// writes collision-free.</summary>
-        private void DFlashCatchUp(float[] hRows, int rows, int firstPos)
-        {
-            if (rows <= 0)
-                return;
-            int skip = firstPos < 0 ? Math.Min(-firstPos, rows) : 0;
-            rows -= skip;
-            firstPos += skip;
-            if (rows <= 0)
-                return;
-
-            int keep = Math.Min(rows, _dflashRingRows);
-            int drop = rows - keep;
-
-            DFlashEncodeAndInject(hRows, skip + drop, keep, firstPos + drop);
-        }
-
-        /// <summary>Encode + ring injection, fused into one GGML graph when the backend
-        /// supports it, otherwise the per-op pair.</summary>
-        private void DFlashEncodeAndInject(float[] hRows, int rowOffset, int n, int startPos)
-        {
-            if (TryFusedDFlashInject(hRows, rowOffset, n, startPos))
-                return;
-
-            EnsureDFlashRingHostSynchronized();
-            using Tensor g = DFlashEncode(hRows, rowOffset, n);
-            DFlashInjectKv(g, n, startPos);
-        }
-
-        /// <summary>
-        /// Drafts one block. <paramref name="hPrev"/> holds the target features of
-        /// the last FORWARDED position (<paramref name="position"/> - 1) and
-        /// <paramref name="lastToken"/> the token at <paramref name="position"/>
-        /// (drawn but not yet forwarded). Returns the number of tokens written to
-        /// <paramref name="draftOut"/>; <paramref name="confOut"/> receives their
-        /// top-1 softmax probabilities.
-        /// </summary>
-        public int DraftBlock(int lastToken, float[] hPrev, int position, int[] draftOut, float[] confOut)
-        {
-            RequireDFlash();
-            if (position <= 0 || draftOut == null || draftOut.Length == 0)
-                return 0;
-
-            // The drafter's own key for the last committed position, exactly like
-            // the reference's ring[start_pos % win] = kv(main_x).
-            DFlashEncodeAndInject(hPrev, 0, 1, position - 1);
-
-            int b = Math.Min(_dflash.BlockSize, draftOut.Length + 1);
-            return DFlashDraftBlockCore(lastToken, position, b, draftOut, confOut);
-        }
-
-        /// <summary>DFlash drafts whole blocks; the per-token entry point is never
-        /// used.</summary>
-        public void DraftStep(int token, float[] hPrev, int pos, float[] logitsOut, float[] hOut)
-            => throw new NotSupportedException("Muse-Glimmer DFlash drafts whole blocks; use DraftBlock.");
-
-        /// <summary>Pre-grows the trunk KV cache to cover the whole speculative
-        /// window. The drafter's ring is fixed-size and needs nothing.</summary>
-        public void SpecEnsureCapacity(int requiredSeqLen)
-        {
-            if (!_hasDFlash)
-                return;
-            EnsureCacheCapacity(requiredSeqLen);
-        }
-
-        /// <summary>No recurrent (GDN/SSM) state in Muse-Glimmer -- drafting and
-        /// verifying are stateless given the KV cache.</summary>
-        public void SpecSnapshotRecurrentState() { }
-
-        /// <summary>See <see cref="SpecSnapshotRecurrentState"/>.</summary>
-        public void SpecRestoreRecurrentState() { }
-
-        /// <summary>
-        /// Rewinds the trunk KV position counter after rejected speculative tokens.
-        /// Rows past <paramref name="length"/> are overwritten by later writes and
-        /// the causal/SWA mask never reads past the live position, so no data moves.
-        /// The drafter's ring is untouched: it only ever holds committed positions,
-        /// and the next catch-up rewrites the ones a rejected tail would have needed.
-        /// </summary>
-        public void SpecRewindCache(int length)
-        {
-            if (length < 0 || length > _cacheSeqLen)
-            {
-                throw new ArgumentOutOfRangeException(nameof(length),
-                    $"Rewind length {length} outside [0, {_cacheSeqLen}].");
-            }
-            _cacheSeqLen = length;
-            _cachedSWAMaskStartPos = -1;
-        }
-
-        private void RequireDFlash()
-        {
-            if (!_hasDFlash)
-                throw new InvalidOperationException("No DFlash drafter is loaded for this Muse-Glimmer model.");
-        }
-
-        // ====================================================================
-        // PASS A -- encoder
-        // ====================================================================
-
-        /// <summary>
-        /// g = rmsnorm(fc @ feat, enc.output_norm). <paramref name="rowOffset"/> is
-        /// the first feature row of <paramref name="hRows"/> to consume and
-        /// <paramref name="n"/> how many. Returns [n, hidden].
-        /// </summary>
-        private unsafe Tensor DFlashEncode(float[] hRows, int rowOffset, int n)
-        {
-            int feat = _dflash.FeatureSize;
-            long need = (long)(rowOffset + n) * feat;
-            if (hRows == null || hRows.LongLength < need)
-            {
-                throw new ArgumentException(
-                    $"DFlash encoder needs {need} feature floats but got {hRows?.LongLength ?? 0}.", nameof(hRows));
-            }
-
-            var featTensor = new Tensor(_allocator, DType.Float32, n, feat);
-            long bytes = (long)n * feat * sizeof(float);
-            float* dst = GetFloatPtr(featTensor);
-            fixed (float* src = &hRows[(long)rowOffset * feat])
-                Buffer.MemoryCopy(src, dst, bytes, bytes);
-            InvalidateTensorDeviceCache(featTensor);
-
-            Tensor g = LinearForward(featTensor, DFlashConfig.WeightPrefix + "fc.weight");
-            featTensor.Dispose();
-
-            Ops.RMSNorm(g, g, _weights[DFlashConfig.WeightPrefix + "enc.output_norm.weight"], null, _dflash.Eps);
-            return g;
-        }
-
-        // ====================================================================
-        // PASS B -- KV injection
-        // ====================================================================
-
-        /// <summary>
-        /// Writes the drafter's per-position keys/values for <paramref name="n"/>
-        /// encoder rows starting at absolute position <paramref name="startPos"/>.
-        /// K is head-normed and NeoX-RoPE'd at the TARGET position; V gets neither.
-        /// </summary>
-        private void DFlashInjectKv(Tensor g, int n, int startPos)
-        {
-            var cfg = _dflash;
-            int kvHeads = cfg.NumKVHeads, hd = cfg.HeadDim;
-
-            int[] positions = new int[n];
-            for (int i = 0; i < n; i++)
-                positions[i] = startPos + i;
-            _dflashRingFilled = Math.Max(_dflashRingFilled, startPos + n);
-
-            for (int il = 0; il < cfg.NumLayers; il++)
-            {
-                string[] names = _dflashLayerNames[il];
-
-                Tensor k = LinearForward(g, names[DfAttnK]);
-                Tensor v = LinearForward(g, names[DfAttnV]);
-
-                k = DFlashHeadNorm(k, _weights[names[DfAttnKNorm]], kvHeads, n, hd);
-                k = DFlashRoPE(k, kvHeads, n, hd, positions);
-
-                using (Tensor kHeads = ReshapeToHeads(k, kvHeads, n, hd))
-                    DFlashRingWrite(_dflashRingK[il], kHeads, startPos, n);
-                using (Tensor vHeads = ReshapeToHeads(v, kvHeads, n, hd))
-                    DFlashRingWrite(_dflashRingV[il], vHeads, startPos, n);
-
-                k.Dispose();
-                v.Dispose();
-            }
-        }
-
-        /// <summary>Scatters a head-first [kvHeads, n, headDim] tensor into the ring
-        /// at (startPos + i) % ringRows, splitting the write at the wrap.</summary>
-        private void DFlashRingWrite(Tensor ring, Tensor headFirst, int startPos, int n)
-        {
-            int rows = _dflashRingRows;
-            int keep = Math.Min(n, rows);
-            int first = n - keep;                 // older rows would be overwritten anyway
-            int done = 0;
-            while (done < keep)
-            {
-                int slot = (startPos + first + done) % rows;
-                int len = Math.Min(keep - done, rows - slot);
-                using var src = headFirst.Narrow(1, first + done, len);
-                CopyToCache(ring, src, slot, len);
-                done += len;
-            }
-        }
-
-        // ====================================================================
-        // PASS C -- block draft
-        // ====================================================================
-
-        /// <summary>
-        /// Runs [anchor, MASK x (b-1)] at positions p..p+b-1 through the drafter and
-        /// fills <paramref name="draftOut"/> / <paramref name="confOut"/> from rows
-        /// 1..b-1. Returns b-1.
-        /// </summary>
-        private unsafe int DFlashDraftBlockCore(int anchorToken, int position, int b, int[] draftOut, float[] confOut)
-        {
-            int fused = TryFusedDFlashDraftBlock(anchorToken, position, b, draftOut, confOut);
-            if (fused >= 0)
-                return fused;
-
-            EnsureDFlashRingHostSynchronized();
-            var cfg = _dflash;
-            int heads = cfg.NumHeads, hd = cfg.HeadDim, kvHeads = cfg.NumKVHeads;
-            float eps = cfg.Eps;
-
-            int[] ids = new int[b];
-            int[] positions = new int[b];
-            for (int i = 0; i < b; i++)
-            {
-                ids[i] = i == 0 ? anchorToken : cfg.MaskTokenId;
-                positions[i] = position + i;
-            }
-
-            // llama.cpp's dflash graph feeds build_inp_embd straight in: no
-            // embedding scale, and (unlike the Muse-Glimmer trunk) no weightless
-            // RMSNorm over the embeddings.
-            Tensor inpL = Embedding(ids);
-
-            for (int il = 0; il < cfg.NumLayers; il++)
-            {
-                string[] names = _dflashLayerNames[il];
-
-                Tensor h = RMSNormWithEps(inpL, names[DfAttnNorm], eps);
-                Tensor q = LinearForward(h, names[DfAttnQ]);
-                Tensor k = LinearForward(h, names[DfAttnK]);
-                Tensor v = LinearForward(h, names[DfAttnV]);
-                h.Dispose();
-
-                q = DFlashHeadNorm(q, _weights[names[DfAttnQNorm]], heads, b, hd);
-                k = DFlashHeadNorm(k, _weights[names[DfAttnKNorm]], kvHeads, b, hd);
-                q = DFlashRoPE(q, heads, b, hd, positions);
-                k = DFlashRoPE(k, kvHeads, b, hd, positions);
-                Ops.Mul(q, q, 1f / MathF.Sqrt(hd));
-
-                Tensor attn = DFlashBlockAttention(il, q, k, v, position, b);
-                q.Dispose();
-                k.Dispose();
-                v.Dispose();
-
-                Tensor attnOut = LinearForward(attn, names[DfAttnOutput]);
-                attn.Dispose();
-
-                Ops.Add(attnOut, attnOut, inpL);          // ffn_inp = attn + inpL
-                inpL.Dispose();
-
-                using (Tensor ffnIn = RMSNormWithEps(attnOut, names[DfFfnNorm], eps))
-                using (Tensor ffnOut = DFlashSwiGLU(ffnIn, names[DfFfnGate], names[DfFfnUp], names[DfFfnDown]))
-                {
-                    Ops.Add(attnOut, attnOut, ffnOut);    // inpL = ffn + ffn_inp
-                }
-                inpL = attnOut;
-            }
-
-            Tensor cur = RMSNormWithEps(inpL, DFlashConfig.WeightPrefix + "output_norm.weight", eps);
-            inpL.Dispose();
-
-            // The TARGET's LM head, with NEITHER logit_scale NOR the tanh softcap:
-            // llama.cpp's dflash graph ends at build_lora_mm(output, cur).
-            Tensor logits = LinearForward(cur, TargetOutputWeightName);
-            cur.Dispose();
-
-            // Softmax on the backend, then a max scan per row: argmax is invariant
-            // under softmax, and the winning probability IS the confidence the
-            // executor multiplies cumulatively (a zero there drafts nothing).
-            Ops.Softmax(logits, logits);
-
-            int vocab = Config.VocabSize;
-            int n = b - 1;
-            float* lp = GetFloatPtr(logits);
-            for (int i = 0; i < n; i++)
-            {
-                // Row 0 is the anchor's own prediction; plain DFlash discards it.
-                float* row = lp + (long)(i + 1) * vocab;
-                int best = ArgmaxRow(row, vocab, out float prob);
-                draftOut[i] = best;
-                if (confOut != null && i < confOut.Length)
-                    confOut[i] = prob;
-            }
-            logits.Dispose();
-            return n;
-        }
-
-        private static unsafe int ArgmaxRow(float* row, int n, out float best)
-        {
-            int bestIdx = 0;
-            float bestVal = row[0];
-            for (int i = 1; i < n; i++)
-            {
-                float v = row[i];
-                if (v > bestVal)
-                {
-                    bestVal = v;
-                    bestIdx = i;
-                }
-            }
-            best = bestVal;
-            return bestIdx;
-        }
-
-        /// <summary>
-        /// One draft layer's attention: the b block queries attend
-        /// [ring window | this block's own b keys], NON-CAUSALLY inside the block
-        /// (llama_set_causal_attn(ctx_dft, false)) and sliding-window masked against
-        /// the ring (a cached key at p0 is masked from a query at p1 when
-        /// p1 - p0 &gt;= n_swa).
-        /// </summary>
-        private unsafe Tensor DFlashBlockAttention(int il, Tensor q, Tensor k, Tensor v, int position, int b)
-        {
-            var cfg = _dflash;
-            int kvHeads = cfg.NumKVHeads, hd = cfg.HeadDim, heads = cfg.NumHeads;
-            int groupSize = heads / kvHeads;
-            int rings = _dflashRingRows;
-
-            // The FIRST query (position p) sees cached keys down to p - (n_swa - 1);
-            // later queries in the block see a strict subset, masked below.
-            int winStart = Math.Max(0, position - (cfg.SlidingWindow - 1));
-            int w = position - winStart;
-            int total = w + b;
-
-            var gk = new Tensor(_allocator, DType.Float32, kvHeads, total, hd);
-            var gv = new Tensor(_allocator, DType.Float32, kvHeads, total, hd);
-
-            int done = 0;
-            while (done < w)
-            {
-                int slot = (winStart + done) % rings;
-                int len = Math.Min(w - done, rings - slot);
-                using (var srcK = _dflashRingK[il].Narrow(1, slot, len))
-                using (var dstK = gk.Narrow(1, done, len))
-                    Ops.Copy(dstK, srcK);
-                using (var srcV = _dflashRingV[il].Narrow(1, slot, len))
-                using (var dstV = gv.Narrow(1, done, len))
-                    Ops.Copy(dstV, srcV);
-                done += len;
-            }
-
-            using (Tensor kHeads = ReshapeToHeads(k, kvHeads, b, hd))
-            using (var dstBlockK = gk.Narrow(1, w, b))
-                Ops.Copy(dstBlockK, kHeads);
-            using (Tensor vHeads = ReshapeToHeads(v, kvHeads, b, hd))
-            using (var dstBlockV = gv.Narrow(1, w, b))
-                Ops.Copy(dstBlockV, vHeads);
-
-            // GQA without materializing the expanded K/V: a contiguous head-first
-            // [heads, b, hd] query tensor reinterprets exactly as
-            // [kvHeads, groupSize*b, hd] (heads 4g..4g+3 are adjacent blocks of
-            // b*hd), so one batched GEMM per kv head serves its whole query group.
-            // ExpandKVHeads would instead allocate heads*total*hd floats per layer
-            // per draft (33 MB here) purely to repeat rows.
-            Tensor qHeads = ReshapeToHeads(q, heads, b, hd);
-            Tensor scores;
-            using (Tensor qGrouped = qHeads.View(kvHeads, (long)groupSize * b, hd))
-            using (Tensor kT = gk.Transpose(1, 2))
-            {
-                scores = new Tensor(_allocator, DType.Float32, kvHeads, (long)groupSize * b, total);
-                Ops.AddmmBatch(scores, 0, scores, 1f, qGrouped, kT);
-            }
-            qHeads.Dispose();
-            gk.Dispose();
-
-            DFlashApplyWindowMask(scores, b, groupSize, kvHeads, w, total, position, winStart);
-            Ops.Softmax(scores, scores);
-
-            var attnGrouped = new Tensor(_allocator, DType.Float32, kvHeads, (long)groupSize * b, hd);
-            Ops.AddmmBatch(attnGrouped, 0, attnGrouped, 1f, scores, gv);
-            scores.Dispose();
-            gv.Dispose();
-
-            Tensor attn;
-            using (Tensor attnHeads = attnGrouped.View(heads, b, hd))
-                attn = ReshapeFromHeads(attnHeads, heads, b, hd);
-            attnGrouped.Dispose();
-            return attn;
-        }
-
-        /// <summary>
-        /// Masks the cached (ring) columns a query cannot see. Query row j of kv
-        /// group g belongs to block slot s = j % b (see the grouping comment in
-        /// <see cref="DFlashBlockAttention"/>), i.e. absolute position
-        /// <paramref name="position"/> + s; cached column c holds position
-        /// <paramref name="winStart"/> + c and is masked when
-        /// (position + s) - (winStart + c) &gt;= n_swa. The block's own b columns
-        /// are never masked -- attention inside the block is non-causal.
-        /// </summary>
-        private unsafe void DFlashApplyWindowMask(Tensor scores, int b, int groupSize, int kvHeads,
-            int w, int total, int position, int winStart)
-        {
-            if (w <= 0)
-                return;
-
-            int swa = _dflash.SlidingWindow;
-            Span<int> widths = stackalloc int[b];
-            bool any = false;
-            for (int s = 0; s < b; s++)
-            {
-                int width = position + s - swa - winStart + 1;
-                if (width < 0) width = 0;
-                if (width > w) width = w;
-                widths[s] = width;
-                any |= width > 0;
-            }
-            if (!any)
-                return;
-
-            float* sp = GetFloatPtr(scores);
-            int rowsPerGroup = groupSize * b;
-            for (int g = 0; g < kvHeads; g++)
-            {
-                float* groupScores = sp + (long)g * rowsPerGroup * total;
-                for (int j = 0; j < rowsPerGroup; j++)
-                {
-                    int width = widths[j % b];
-                    if (width > 0)
-                        new Span<float>(groupScores + (long)j * total, width).Fill(float.NegativeInfinity);
-                }
-            }
-            InvalidateTensorDeviceCache(scores);
-        }
-
-        // ====================================================================
-        // small shared pieces
-        // ====================================================================
-
-        /// <summary>Per-head RMSNorm over a [rows, numHeads*headDim] tensor, with
-        /// the drafter's own epsilon. Consumes <paramref name="data"/>. Same shape
-        /// as MuseGlimmerModel.ApplyBatchRMSNorm, but takes the weight tensor and
-        /// eps directly (the trunk helper hardcodes Config.Eps).</summary>
-        private Tensor DFlashHeadNorm(Tensor data, Tensor alpha, int numHeads, int rows, int headDim)
-        {
-            using var reshaped = data.View((long)rows * numHeads, headDim);
-            Tensor normed = Ops.RMSNorm(null, reshaped, alpha, null, _dflash.Eps);
-            data.Dispose();
-            Tensor flat = normed.View(rows, (long)numHeads * headDim);
-            normed.Dispose();
-            return flat;
-        }
-
-        /// <summary>
-        /// NeoX-flavour RoPE (split halves) over a [rows, numHeads*headDim] tensor
-        /// with an explicit per-row position. llama.cpp maps LLM_ARCH_DFLASH to
-        /// LLAMA_ROPE_TYPE_NEOX, so this is mode 2 -- NOT the interleaved-pair
-        /// (mode 0) RoPE MuseGlimmerModel.ApplyRoPEPrefill uses for the trunk.
-        /// Consumes <paramref name="data"/>.
-        /// </summary>
-        private Tensor DFlashRoPE(Tensor data, int numHeads, int rows, int headDim, int[] positions)
-        {
-            int totalRows = rows * numHeads;
-            int[] rowPositions = new int[totalRows];
-            for (int s = 0; s < rows; s++)
-                for (int h = 0; h < numHeads; h++)
-                    rowPositions[s * numHeads + h] = positions[s];
-            using var posTensor = CreateIntTensorOn(data.Storage.Allocator, rowPositions, totalRows);
-
-            using var reshaped = data.View(1, rows, numHeads, headDim);
-            Tensor result = Ops.RoPEEx(
-                null, reshaped, posTensor, headDim, DFlashConfig.RopeTypeNeoX, 0,
-                _dflash.RopeBase, 1.0f,
-                0.0f, 1.0f, 0.0f, 0.0f);
-
-            data.Dispose();
-            Tensor flat = result.View(rows, (long)numHeads * headDim);
-            result.Dispose();
-            return flat;
-        }
-
-        /// <summary>silu(gate(x)) * up(x) -> down. The drafter ships gate and up as
-        /// SEPARATE tensors (ModelBase.FuseGateUpWeights only fuses the target's
-        /// "blk.{l}." names), so this cannot use ModelBase.FFN.</summary>
-        private Tensor DFlashSwiGLU(Tensor input, string gateName, string upName, string downName)
-        {
-            Tensor gate = LinearForward(input, gateName);
-            Tensor up = LinearForward(input, upName);
-            Ops.SiLUMul(gate, gate, up);
-            up.Dispose();
-            Tensor down = LinearForward(gate, downName);
-            gate.Dispose();
-            return down;
-        }
     }
 }
diff --git a/TensorSharp.Models/Models/MuseGlimmer/MuseGlimmerModel.cs b/TensorSharp.Models/Models/MuseGlimmer/MuseGlimmerModel.cs
index 718e6676..3f52f304 100644
--- a/TensorSharp.Models/Models/MuseGlimmer/MuseGlimmerModel.cs
+++ b/TensorSharp.Models/Models/MuseGlimmer/MuseGlimmerModel.cs
@@ -1483,7 +1483,7 @@ public override void Dispose()
                 embeddings?.Dispose();
             _pendingVisionEmbeddingsList.Clear();
             _onesForEmbNorm?.Dispose();
-            DisposeDFlash();
+            DisposeDFlashDrafter();
             DisposeMuseGlimmerTpState();
             if (_kvCacheK != null)
                 foreach (var t in _kvCacheK) t?.Dispose();
diff --git a/TensorSharp.Models/Models/Qwen35/Qwen35Model.BatchedFusedDecode.cs b/TensorSharp.Models/Models/Qwen35/Qwen35Model.BatchedFusedDecode.cs
index 5e83787f..6fcd072e 100644
--- a/TensorSharp.Models/Models/Qwen35/Qwen35Model.BatchedFusedDecode.cs
+++ b/TensorSharp.Models/Models/Qwen35/Qwen35Model.BatchedFusedDecode.cs
@@ -134,7 +134,7 @@ private unsafe Tensor TryRunBatchedFusedDecode(
                             && _layerStackedGate[l] != null && _layerStackedUp[l] != null && _layerStackedDown[l] != null
                             && HasW(_ffnGateShexpQW[l], _ffnGateShexpF32[l]) && HasW(_ffnUpShexpQW[l], _ffnUpShexpF32[l])
                             && HasW(_ffnDownShexpQW[l], _ffnDownShexpF32[l]) && _ffnGateInpShexpVec[l] != null)
-                        : (HasW(_ffnGateUpQW[l], _ffnGateUpF32[l]) && HasW(_ffnDownQW[l], _ffnDownF32[l]));
+                        : (HasDenseFfnWeights(l) && HasW(_ffnDownQW[l], _ffnDownF32[l]));
                     bool ok = _attnNormW[l] != null && _postAttnNormW[l] != null && ffnOk;
                     if (ok && !_isRecurrent[l])
                         ok = (HasW(_attnQkvQW[l], _attnQkvF32[l])
@@ -226,11 +226,9 @@ private unsafe Tensor TryRunBatchedFusedDecode(
                 a.IsMoe = isMoe ? 1 : 0;
                 if (!isMoe)
                 {
-                    var gu = ResolveW(_ffnGateUpQW[l], _ffnGateUpF32[l]);
                     var dn = ResolveW(_ffnDownQW[l], _ffnDownF32[l]);
-                    a.GuW = gu.Item1; a.GuType = gu.Item2; a.GuNe0 = gu.Item3; a.GuNe1 = gu.Item4; a.GuBytes = gu.Item5;
                     a.DownW = dn.Item1; a.DownType = dn.Item2; a.DownNe0 = dn.Item3; a.DownNe1 = dn.Item4; a.DownBytes = dn.Item5;
-                    a.FfDense = (int)(gu.Item4 / 2);
+                    FillDenseFfnArgs(ref a, l);
                 }
                 else
                 {
diff --git a/TensorSharp.Models/Models/Qwen35/Qwen35Model.DFlash.cs b/TensorSharp.Models/Models/Qwen35/Qwen35Model.DFlash.cs
new file mode 100644
index 00000000..b247e1e9
--- /dev/null
+++ b/TensorSharp.Models/Models/Qwen35/Qwen35Model.DFlash.cs
@@ -0,0 +1,277 @@
+// Copyright (c) Zhongkai Fu. All rights reserved.
+// https://github.com/zhongkaifu/TensorSharp
+//
+// This file is part of TensorSharp.
+//
+// TensorSharp is licensed under the BSD-3-Clause license found in the LICENSE file in the root directory of this source tree.
+//
+// TensorSharp is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the BSD-3-Clause License for more details.
+//
+// ---------------------------------------------------------------------------
+// Qwen 3.5 / 3.8 as a DFlash (and DFlash2) speculation TARGET.
+//
+// The drafter itself is shared - see ModelBase.DFlash*.cs. What is here is the
+// half that only this trunk can provide: the per-layer residuals the drafter's
+// encoder was trained on, tapped out of a forward that otherwise runs exactly
+// like the non-speculative one.
+//
+// The GGUF names those layers in dflash.target_layers (already +1-shifted by the
+// converter, so id L means "the residual ENTERING 0-based layer L" == "the output
+// of layer L-1"), and the drafter's fc projects the concatenation of them.
+//
+// A Qwen 3.8 checkpoint can carry BOTH drafters: a NextN/MTP block inside the
+// trunk GGUF and an external DFlash file. When a DFlash drafter is attached it
+// wins, because it is the one the operator explicitly asked for with
+// --draft-model, and because the two cannot be mixed - they consume different
+// hidden rows (SpecFeatureSize) and drive different speculators.
+// ---------------------------------------------------------------------------
+using System;
+using System.Diagnostics;
+using TensorSharp;
+using TensorSharp.GGML;
+using TensorSharp.Runtime;
+using TensorSharp.Runtime.Speculative;
+
+namespace TensorSharp.Models
+{
+    public partial class Qwen35Model
+    {
+        /// <summary>
+        /// Path of a DFlash drafter GGUF, if one was configured
+        /// (<c>--draft-model</c> / <c>TS_QWEN35_DFLASH</c>) AND it really is one.
+        /// The same flag also names an MTP-only file for other architectures, so
+        /// the architecture string decides rather than the extension.
+        /// </summary>
+        internal static string ResolveQwen35DFlashPath(string explicitPath)
+        {
+            string path = !string.IsNullOrWhiteSpace(explicitPath)
+                ? explicitPath
+                : Environment.GetEnvironmentVariable("TS_QWEN35_DFLASH");
+            if (string.IsNullOrWhiteSpace(path) || !System.IO.File.Exists(path))
+                return null;
+            try
+            {
+                using var probe = new GgufFile(path);
+                string arch = probe.GetString("general.architecture") ?? string.Empty;
+                return string.Equals(arch, DFlashConfig.ArchName, StringComparison.Ordinal) ? path : null;
+            }
+            catch
+            {
+                // A file we cannot open is not a drafter; the loader below will say
+                // so if the operator really meant it.
+                return null;
+            }
+        }
+
+        /// <summary>
+        /// Attach a DFlash drafter to this trunk. Called from the constructor after
+        /// the trunk weights and caches exist, because the drafter's tensors are
+        /// merged into the same dictionaries and its ring is allocated from the same
+        /// allocator.
+        /// </summary>
+        private void TryLoadQwen35DFlash(string draftModelPath)
+        {
+            string path = ResolveQwen35DFlashPath(draftModelPath);
+            if (path == null)
+                return;
+
+            if (IsTensorParallel)
+            {
+                // The drafter borrows the trunk's LM head and token embedding, both
+                // of which are sharded under TP, and its capture path runs the
+                // single-device fused verify. Refuse rather than draft from a shard.
+                Console.WriteLine("  DFlash speculative decoding is not supported under tensor parallelism; ignoring the drafter.");
+                return;
+            }
+
+            try
+            {
+                LoadDFlashDraftWeights(path);
+            }
+            catch (Exception ex)
+            {
+                Console.WriteLine($"  DFlash drafter '{System.IO.Path.GetFileName(path)}' could not be attached: {ex.Message}");
+                return;
+            }
+
+            if (HasDFlash && _numNextnLayers > 0)
+                Console.WriteLine("  DFlash drafter attached; the checkpoint's NextN/MTP block will not be used.");
+        }
+
+        /// <summary>Number of DRAFTS a block produces (block_size - 1: row 0 is the
+        /// anchor's own prediction, which DFlash discards).</summary>
+        public int DraftBlockSize => HasDFlash ? _dflash.MaxDraftTokens : 0;
+
+        /// <summary>One whole-block draft. See <c>ModelBase.DFlashPropose</c>.</summary>
+        public int DraftBlock(int lastToken, float[] hPrev, int position, int[] draftOut, float[] confOut)
+        {
+            EnterSpecSession();
+            return DFlashPropose(lastToken, hPrev, position, draftOut, confOut);
+        }
+
+        // ====================================================================
+        // the trunk's half: capturing dflash.target_layers residuals
+        // ====================================================================
+
+        private float[] _dflashCaptureScratch;
+
+        /// <summary>
+        /// SpecForward for a DFlash-drafted trunk: the ordinary verify forward plus
+        /// one [hidden, n] block per entry of dflash.target_layers, transposed into
+        /// the ROW-major FeatureSize-wide layout the drafter's encoder consumes.
+        ///
+        /// Returns false when the fused kernel declines the shape, in which case the
+        /// caller runs the op-by-op loop (which taps the same residuals by hand).
+        /// </summary>
+        private bool TryDFlashSpecForwardFused(Tensor hidden, int startPos, int seqLen,
+            float[] hAllOut, float[] logitsOut, bool allLogitsRows, bool captureAll, bool captureLast)
+        {
+            if (!_fusedVerifyEnabled)
+                return false;
+
+            int nCap = _dflash.TargetLayerIds.Length;
+            int hs = Config.HiddenSize;
+            int feat = _dflash.FeatureSize;
+            bool wantCapture = captureAll || captureLast;
+
+            if (wantCapture)
+            {
+                long need = (long)nCap * hs * seqLen;
+                if (_dflashCaptureScratch == null || _dflashCaptureScratch.LongLength < need)
+                    _dflashCaptureScratch = new float[need];
+            }
+
+            int vocab = Config.VocabSize;
+            float[] logitTarget;
+            int logitRows;
+            if (allLogitsRows)
+            {
+                if (logitsOut != null && logitsOut.LongLength >= (long)seqLen * vocab)
+                {
+                    logitTarget = logitsOut;
+                }
+                else
+                {
+                    if (_fvLogitsAllBuf == null || _fvLogitsAllBuf.LongLength < (long)seqLen * vocab)
+                        _fvLogitsAllBuf = new float[(long)seqLen * vocab];
+                    logitTarget = _fvLogitsAllBuf;
+                }
+                logitRows = -1;
+            }
+            else
+            {
+                // Last-row-only callers (prompt prefill chunks, rollback re-forwards)
+                // ask the kernel for ONE logit row: a whole-prompt lm_head is
+                // vocab*N floats of device and host memory for rows nobody reads.
+                if (_fvLogitsLastBuf == null || _fvLogitsLastBuf.Length < vocab)
+                    _fvLogitsLastBuf = new float[vocab];
+                logitTarget = _fvLogitsLastBuf;
+                logitRows = 1;
+            }
+
+            if (!TryFullModelVerify(hidden, startPos, seqLen, normedOut: null, logitsOut: logitTarget,
+                    nLogitRows: logitRows, rowOffset: 0,
+                    captureData: wantCapture ? _dflashCaptureScratch : null,
+                    captureLayers: wantCapture ? _dflash.TargetLayerIds : null))
+            {
+                return false;
+            }
+
+            if (wantCapture)
+            {
+                for (int c = 0; c < nCap; c++)
+                {
+                    long blockBase = (long)c * hs * seqLen;
+                    if (captureLast)
+                    {
+                        Array.Copy(_dflashCaptureScratch, blockBase + (long)(seqLen - 1) * hs,
+                            hAllOut, (long)c * hs, hs);
+                        continue;
+                    }
+                    for (int r = 0; r < seqLen; r++)
+                    {
+                        Array.Copy(_dflashCaptureScratch, blockBase + (long)r * hs,
+                            hAllOut, (long)r * feat + (long)c * hs, hs);
+                    }
+                }
+            }
+
+            if (!allLogitsRows && logitsOut != null)
+                Array.Copy(logitTarget, logitsOut, vocab);
+            return true;
+        }
+
+        /// <summary>
+        /// Op-by-op SpecForward with the DFlash residual taps, for the shapes and
+        /// backends the fused kernel declines. Same layer loop as
+        /// <see cref="SpecForward"/>'s fallback, with the capture hook in front of
+        /// each block.
+        /// </summary>
+        private unsafe void DFlashSpecForwardPerOp(Tensor hidden, int startPos, int seqLen,
+            float[] hAllOut, float[] logitsOut, bool allLogitsRows, bool captureAll, bool captureLast)
+        {
+            EnsureKvCacheHostSynchronized();
+            EnsureFusedDecodeStateHostSynchronized();
+            for (int layer = 0; layer < Config.NumLayers; layer++)
+            {
+                int slot = _dflashCaptureSlot[layer];
+                if (slot >= 0 && (captureAll || captureLast))
+                    DFlashCaptureFeature(hidden, slot, seqLen, hAllOut, captureLast);
+
+                long tl = Stopwatch.GetTimestamp();
+                if (_isRecurrent[layer])
+                {
+                    hidden = RecurrentBlock(hidden, layer, seqLen, startPos);
+                    SpecRecurrentLayerTicks += Stopwatch.GetTimestamp() - tl;
+                }
+                else
+                {
+                    hidden = AttentionBlock(hidden, layer, seqLen, startPos);
+                    SpecAttnLayerTicks += Stopwatch.GetTimestamp() - tl;
+                }
+                TryEvaluateMlxLayerBoundary(hidden, layer, seqLen);
+            }
+
+            Tensor normed = RMSNormOpCached(hidden, _finalNormW);
+            hidden.Dispose();
+
+            long t2 = Stopwatch.GetTimestamp();
+            if (allLogitsRows)
+            {
+                Tensor logitsT = LinearForwardCached(normed, _lmHeadQW, _lmHeadF32);
+                normed.Dispose();
+                fixed (float* dst = logitsOut)
+                {
+                    float* src = GetFloatPtr(logitsT);
+                    Buffer.MemoryCopy(src, dst, (long)logitsOut.Length * 4, (long)seqLen * Config.VocabSize * 4);
+                }
+                logitsT.Dispose();
+            }
+            else
+            {
+                Tensor lastRow;
+                if (seqLen > 1)
+                {
+                    using var narrowed = normed.Narrow(0, seqLen - 1, 1);
+                    lastRow = Ops.NewContiguous(narrowed);
+                    normed.Dispose();
+                }
+                else
+                {
+                    lastRow = normed;
+                }
+                Tensor logitsT = LinearForwardCached(lastRow, _lmHeadQW, _lmHeadF32);
+                lastRow.Dispose();
+                fixed (float* dst = logitsOut)
+                {
+                    float* src = GetFloatPtr(logitsT);
+                    Buffer.MemoryCopy(src, dst, (long)logitsOut.Length * 4, (long)Config.VocabSize * 4);
+                }
+                logitsT.Dispose();
+            }
+            _lmHeadTicks += Stopwatch.GetTimestamp() - t2;
+            SpecLmHeadTicks += Stopwatch.GetTimestamp() - t2;
+        }
+    }
+}
diff --git a/TensorSharp.Models/Models/Qwen35/Qwen35Model.GatedDeltaNet.cs b/TensorSharp.Models/Models/Qwen35/Qwen35Model.GatedDeltaNet.cs
index 386b8236..9a75cf4d 100644
--- a/TensorSharp.Models/Models/Qwen35/Qwen35Model.GatedDeltaNet.cs
+++ b/TensorSharp.Models/Models/Qwen35/Qwen35Model.GatedDeltaNet.cs
@@ -1,4 +1,4 @@
-// Copyright (c) Zhongkai Fu. All rights reserved.
+// Copyright (c) Zhongkai Fu. All rights reserved.
 // https://github.com/zhongkaifu/TensorSharp
 //
 // This file is part of TensorSharp.
@@ -842,6 +842,12 @@ private unsafe bool TryFusedRecLayerPrefill(Tensor hidden, int layer, int seqLen
 
         private Tensor RecurrentBlock(Tensor hidden, int layer, int seqLen, int startPos)
         {
+            // The op-by-op recurrent path reads the HOST state mirrors, so a
+            // speculative session that has been keeping the authoritative state in
+            // the verify kernel's device slices has to hand it back first. A no-op
+            // after the first layer of the first such forward.
+            DrainDeviceRecurrentState();
+
             bool isMoeLayer = _isMoeLayer != null && _isMoeLayer[layer];
 
             // ---- Path A: Fused dense FFN (non-MoE layers) ----
@@ -1164,6 +1170,31 @@ internal void EnterSpecSession()
             InvalidateFullDecodeState(hardBindings: true);
         }
 
+        /// <summary>
+        /// Point a layer descriptor's dense FFN at either the fused gate_up tensor
+        /// or, for a mixed-quant layer that could not be fused, at the original
+        /// gate and up pair. Exactly one of the two is populated, and the native
+        /// graphs branch on GuW being null.
+        /// </summary>
+        private void FillDenseFfnArgs(ref Qwen35LayerDecodeArgs a, int l)
+        {
+            if (_ffnGateUpQW[l] != null || _ffnGateUpF32[l] != null)
+            {
+                var gu = ResolveW(_ffnGateUpQW[l], _ffnGateUpF32[l]);
+                a.GuW = gu.ptr; a.GuType = gu.type; a.GuNe0 = gu.ne0; a.GuNe1 = gu.ne1; a.GuBytes = gu.bytes;
+                a.FfDense = (int)(gu.ne1 / 2);
+                a.FfnGateW = IntPtr.Zero; a.FfnUpW = IntPtr.Zero;
+                return;
+            }
+
+            var g = ResolveW(_ffnGateSplitQW[l], _ffnGateSplitF32[l]);
+            var u = ResolveW(_ffnUpSplitQW[l], _ffnUpSplitF32[l]);
+            a.GuW = IntPtr.Zero; a.GuType = 0; a.GuNe0 = 0; a.GuNe1 = 0; a.GuBytes = 0;
+            a.FfnGateW = g.ptr; a.FfnGateType = g.type; a.FfnGateNe0 = g.ne0; a.FfnGateNe1 = g.ne1; a.FfnGateBytes = g.bytes;
+            a.FfnUpW = u.ptr; a.FfnUpType = u.type; a.FfnUpNe0 = u.ne0; a.FfnUpNe1 = u.ne1; a.FfnUpBytes = u.bytes;
+            a.FfDense = (int)g.ne1;
+        }
+
         // Resolve a linear-projection weight to (ptr, ggml-type, ne0, ne1, bytes)
         // from EITHER its quantized form or its F32 form (small projections such as
         // ssm_beta / ssm_alpha are stored F32). F32 weights are [out, in] tensors,
@@ -1310,7 +1341,7 @@ private unsafe bool TryFullModelDecodeCore(
                             && _layerStackedGate[l] != null && _layerStackedUp[l] != null && _layerStackedDown[l] != null
                             && HasW(_ffnGateShexpQW[l], _ffnGateShexpF32[l]) && HasW(_ffnUpShexpQW[l], _ffnUpShexpF32[l])
                             && HasW(_ffnDownShexpQW[l], _ffnDownShexpF32[l]) && _ffnGateInpShexpVec[l] != null)
-                        : (HasW(_ffnGateUpQW[l], _ffnGateUpF32[l]) && HasW(_ffnDownQW[l], _ffnDownF32[l]));
+                        : (HasDenseFfnWeights(l) && HasW(_ffnDownQW[l], _ffnDownF32[l]));
                     bool ok = _attnNormW[l] != null && _postAttnNormW[l] != null && ffnOk;
                     if (ok && !_isRecurrent[l])
                         ok = (HasW(_attnQkvQW[l], _attnQkvF32[l])
@@ -1430,11 +1461,9 @@ private unsafe bool TryFullModelDecodeCore(
                     a.IsMoe = isMoe ? 1 : 0;
                     if (!isMoe)
                     {
-                        var gu = ResolveW(_ffnGateUpQW[l], _ffnGateUpF32[l]);
                         var dn = ResolveW(_ffnDownQW[l], _ffnDownF32[l]);
-                        a.GuW = gu.ptr; a.GuType = gu.type; a.GuNe0 = gu.ne0; a.GuNe1 = gu.ne1; a.GuBytes = gu.bytes;
                         a.DownW = dn.ptr; a.DownType = dn.type; a.DownNe0 = dn.ne0; a.DownNe1 = dn.ne1; a.DownBytes = dn.bytes;
-                        a.FfDense = (int)(gu.ne1 / 2);
+                        FillDenseFfnArgs(ref a, l);
                     }
                     else
                     {
@@ -1656,7 +1685,64 @@ internal static bool ShouldUseVerifyResidentState(
                 !(nLogitRows > 0 && nLogitRows < seqLen);
         }
 
-        internal unsafe bool TryFullModelVerify(Tensor hidden, int startPos, int seqLen, float[] normedOut, float[] logitsOut, int nLogitRows = -1, int rowOffset = 0)
+        /// <param name="captureData">Optional DFlash residual taps: receives
+        /// <paramref name="captureLayers"/>.Length consecutive [hidden, seqLen]
+        /// blocks, block c holding the residual ENTERING layer captureLayers[c].
+        /// Null (the normal case) costs nothing.</param>
+        // ---- per-token recurrent-state snapshots (see SpecOnVerifyAccepted) ----
+
+        /// <summary>Recurrent layers, in the order the native descriptor array lists
+        /// them - the order TSGgml_Qwen35FetchStateSnapshot returns slots in.</summary>
+        private int[] _fvRecurrentLayers;
+        private IntPtr[] _fvSnapConvPtrs;
+        private IntPtr[] _fvSnapDeltaPtrs;
+
+        /// <summary>Rows of the verify whose snapshots are live on the device, or 0
+        /// when the last verify downloaded its post-window state the old way.</summary>
+        private int _fvSnapshotRows;
+
+        /// <summary>True when the accepted prefix's recurrent state was recovered
+        /// from a snapshot, so the executor can keep the verify's KV writes and skip
+        /// the kept-prefix re-forward.</summary>
+        private bool _fvAcceptedPrefixCommitted;
+
+        /// <summary>
+        /// The LIVE recurrent state lives on the device and the host mirror is stale.
+        /// Set when a snapshot is committed device-side, cleared the moment anything
+        /// drains it back. While it holds, a verify skips both halves of the ~300 MB
+        /// per-step state round trip - which is what made speculative decoding on
+        /// this trunk cost more than the plain decode it was meant to beat.
+        /// </summary>
+        private bool _fvDeviceStateCurrent;
+
+        /// <summary>Pull the live device state back into the host mirrors and clear
+        /// <see cref="_fvDeviceStateCurrent"/>. Every path that reads the mirrors -
+        /// the op-by-op recurrent block, a prefill chunk, a state snapshot - goes
+        /// through here first.</summary>
+        internal unsafe void DrainDeviceRecurrentState()
+        {
+            if (!_fvDeviceStateCurrent)
+                return;
+            _fvDeviceStateCurrent = false;
+            if (!PrepareRecurrentStatePointers())
+                return;
+            if (!GgmlBasicOps.Qwen35DrainDeviceState(_fvSnapConvPtrs, _fvSnapDeltaPtrs, _fvRecurrentLayers.Length))
+                return;
+            UnpackRecurrentStateFromNative();
+        }
+
+        private static readonly bool _fvSnapshotsEnabled =
+            !string.Equals(Environment.GetEnvironmentVariable("TS_Q35_VERIFY_SNAPSHOTS"), "0", StringComparison.Ordinal);
+
+        /// Leave the post-window recurrent state on the device and commit it there,
+        /// instead of downloading 151 MB and uploading it again next call. Separable
+        /// from the snapshots themselves because it is the part that also applies to
+        /// the single-row plain steps a speculative session interleaves with verifies.
+        private static readonly bool _fvDeferStateEnabled =
+            !string.Equals(Environment.GetEnvironmentVariable("TS_Q35_VERIFY_DEFER_STATE"), "0", StringComparison.Ordinal);
+
+        internal unsafe bool TryFullModelVerify(Tensor hidden, int startPos, int seqLen, float[] normedOut, float[] logitsOut, int nLogitRows = -1, int rowOffset = 0,
+            float[] captureData = null, int[] captureLayers = null)
         {
             // Run one whole-model prefill/verify graph on every GGML GPU backend.
             // CUDA and Vulkan use per-head set_rows KV writes. Metal uses contiguous
@@ -1727,7 +1813,7 @@ internal unsafe bool TryFullModelVerify(Tensor hidden, int startPos, int seqLen,
                             && _layerStackedGate[l] != null && _layerStackedUp[l] != null && _layerStackedDown[l] != null
                             && HasW(_ffnGateShexpQW[l], _ffnGateShexpF32[l]) && HasW(_ffnUpShexpQW[l], _ffnUpShexpF32[l])
                             && HasW(_ffnDownShexpQW[l], _ffnDownShexpF32[l]) && _ffnGateInpShexpVec[l] != null)
-                        : (HasW(_ffnGateUpQW[l], _ffnGateUpF32[l]) && HasW(_ffnDownQW[l], _ffnDownF32[l]));
+                        : (HasDenseFfnWeights(l) && HasW(_ffnDownQW[l], _ffnDownF32[l]));
                     bool ok = _attnNormW[l] != null && _postAttnNormW[l] != null && ffnOk;
                     if (ok && !_isRecurrent[l])
                         ok = (HasW(_attnQkvQW[l], _attnQkvF32[l])
@@ -1791,11 +1877,9 @@ internal unsafe bool TryFullModelVerify(Tensor hidden, int startPos, int seqLen,
                 a.IsMoe = isMoe ? 1 : 0;
                 if (!isMoe)
                 {
-                    var gu = ResolveW(_ffnGateUpQW[l], _ffnGateUpF32[l]);
                     var dn = ResolveW(_ffnDownQW[l], _ffnDownF32[l]);
-                    a.GuW = gu.ptr; a.GuType = gu.type; a.GuNe0 = gu.ne0; a.GuNe1 = gu.ne1; a.GuBytes = gu.bytes;
                     a.DownW = dn.ptr; a.DownType = dn.type; a.DownNe0 = dn.ne0; a.DownNe1 = dn.ne1; a.DownBytes = dn.bytes;
-                    a.FfDense = (int)(gu.ne1 / 2);
+                    FillDenseFfnArgs(ref a, l);
                 }
                 else
                 {
@@ -1920,9 +2004,36 @@ internal unsafe bool TryFullModelVerify(Tensor hidden, int startPos, int seqLen,
 
             var lmh = ResolveW(_lmHeadQW, _lmHeadF32);
             IntPtr finalNormPtr = (IntPtr)GetFloatPtr(_finalNormW);
+            int capCount = (captureData != null && captureLayers != null) ? captureLayers.Length : 0;
+
+            // Ask the kernel to keep one recurrent-state snapshot per row of an
+            // all-rows verify (the speculative one). Without them a partial accept
+            // has to restore a pre-verify copy of the state and re-forward the
+            // accepted prefix through the whole trunk; with them the state it would
+            // have recomputed is already on the device and costs one slot fetch.
+            // Only the all-rows verify: a prefill chunk is never rolled back, and
+            // asking for N snapshots there would size the graph for nothing.
+            int requestedSnapshots = (_fvSnapshotsEnabled && !residentThisCall && nLogitRows <= 0 && seqLen > 1)
+                ? seqLen : 1;
+            // Defer the state download on every call the kernel will persist - the
+            // all-rows verify AND the single-row plain steps a speculative session
+            // interleaves with it. Those plain steps used to break the device-state
+            // chain: each one downloaded 151 MB and forced the NEXT verify to upload
+            // it again, which on an MTP run (46 plain steps of 125) was most of the
+            // gap to the captured decode. A prefill chunk (0 < nLogitRows < seqLen)
+            // is not persisted and keeps the host path.
+            bool deferState = _fvSnapshotsEnabled && _fvDeferStateEnabled && !residentThisCall
+                && (nLogitRows <= 0 || seqLen == 1);
+            int snapshotsUsed = 1;
+            // The live state is already correct on the device exactly when the last
+            // step committed a snapshot into it and nothing has drained it since.
+            bool deviceStateCurrent = _fvDeviceStateCurrent;
+            _fvSnapshotRows = 0;
+            _fvAcceptedPrefixCommitted = false;
             bool ok2;
             fixed (float* lp = logitsOut)
             fixed (float* np = normedOut)
+            fixed (float* cp = captureData)
             {
                 ok2 = GgmlBasicOps.Qwen35ModelVerify(
                     _fvLayers, n,
@@ -1938,7 +2049,14 @@ internal unsafe bool TryFullModelVerify(Tensor hidden, int startPos, int seqLen,
                     (IntPtr)lp, Config.VocabSize,
                     lmh.ptr, lmh.type, lmh.ne0, lmh.ne1, lmh.bytes,
                     finalNormPtr, normedOut != null ? (IntPtr)np : IntPtr.Zero, nLogitRows,
-                    mropePos, mropeSecs);
+                    mropePos, mropeSecs, tpDegree: 1, tpPlanOut: null,
+                    captureData: capCount > 0 ? (IntPtr)cp : IntPtr.Zero,
+                    captureLayers: capCount > 0 ? captureLayers : null,
+                    captureCount: capCount,
+                    stateSnapshots: requestedSnapshots,
+                    stateSnapshotsUsed: (IntPtr)(&snapshotsUsed),
+                    deviceStateCurrent: deviceStateCurrent,
+                    deferStateDownload: deferState);
             }
             if (!ok2)
             {
@@ -1946,6 +2064,42 @@ internal unsafe bool TryFullModelVerify(Tensor hidden, int startPos, int seqLen,
                 return false;
             }
 
+            if (snapshotsUsed > 1)
+            {
+                // Whatever the device held on entry, the graph has now consumed it;
+                // the authoritative state is the snapshot set until one is committed.
+                _fvDeviceStateCurrent = false;
+                // The state is on the device in snapshotsUsed slots and stays there
+                // until SpecOnVerifyAccepted knows which one the accepted prefix
+                // wants. Nothing below (which drains the post-window state into the
+                // host mirror) applies, and nothing may read that mirror in between.
+                _fvSnapshotRows = seqLen;
+                _fvStateResident = false;
+                _kvCacheHostDirty = true;
+                return true;
+            }
+
+            if (snapshotsUsed == 0)
+            {
+                // Deferred with no snapshots: a single-row step, whose post-window
+                // state is simply the *_state_out slices. Nothing decides anything
+                // about it later, so commit it now - one device-to-device copy
+                // instead of 151 MB down here and 151 MB back up next call.
+                _fvDeviceStateCurrent = false;
+                _fvSnapshotRows = 0;
+                _fvStateResident = false;
+                _kvCacheHostDirty = true;
+                if (!CommitRecurrentStateSnapshot(-1))
+                {
+                    throw new InvalidOperationException(
+                        "Qwen3.5 verify deferred its recurrent state but it could not be committed; "
+                        + "the host mirror is stale. Set TS_Q35_VERIFY_SNAPSHOTS=0 to fall back.");
+                }
+                return true;
+            }
+
+            _fvDeviceStateCurrent = false;
+
             // Write the post-window GDN state back to the C# (host) representation so the
             // snapshot / rollback / any op-by-op fallback see the current state.
             //   host mode: the native already downloaded conv_state_out -> _fvConvOut and
@@ -1991,6 +2145,114 @@ internal unsafe bool TryFullModelVerify(Tensor hidden, int startPos, int seqLen,
             return true;
         }
 
+        /// <summary>
+        /// Pull one per-row recurrent-state snapshot out of the verify that just ran
+        /// into the host mirrors the rest of the model reads: the GDN delta state
+        /// tensors and the conv ring.
+        ///
+        /// The conv ring is transposed on the way in for the same reason the ordinary
+        /// post-verify drain transposes it - the kernel stores a conv window
+        /// time-major ([convDim, conv_dim]) and the op-by-op path indexes it
+        /// channel-major.
+        /// </summary>
+        private unsafe bool PrepareRecurrentStatePointers()
+        {
+            if (!IsGgmlBackend || _fvConvOut == IntPtr.Zero || _convState == null || _fvGdnSlot == null)
+                return false;
+
+            int n = Config.NumLayers;
+            if (_fvRecurrentLayers == null)
+            {
+                int count = 0;
+                for (int l = 0; l < n; l++)
+                    if (_isRecurrent[l]) count++;
+                _fvRecurrentLayers = new int[count];
+                int w = 0;
+                for (int l = 0; l < n; l++)
+                    if (_isRecurrent[l]) _fvRecurrentLayers[w++] = l;
+                _fvSnapConvPtrs = new IntPtr[_fvRecurrentLayers.Length];
+                _fvSnapDeltaPtrs = new IntPtr[_fvRecurrentLayers.Length];
+            }
+
+            int qkvDim = _headKDim * _numKHeads * 2 + _headVDim * _numVHeads;
+            int convBlock = (_convKernel - 1) * qkvDim;
+            float* convOutBase = (float*)_fvConvOut;
+            for (int i = 0; i < _fvRecurrentLayers.Length; i++)
+            {
+                int l = _fvRecurrentLayers[i];
+                if (_deltaStateTensor[l] == null)
+                    return false;
+                _fvSnapConvPtrs[i] = (IntPtr)(convOutBase + (long)_fvGdnSlot[l] * convBlock);
+                _fvSnapDeltaPtrs[i] = (IntPtr)GetFloatPtr(_deltaStateTensor[l]);
+            }
+            return true;
+        }
+
+        /// <summary>The conv window comes back time-major ([convDim, conv_dim], the
+        /// kernel's layout) and the op-by-op path indexes it channel-major, so the
+        /// ring is transposed on the way in - the same transpose the ordinary
+        /// post-verify drain does.</summary>
+        private unsafe void UnpackRecurrentStateFromNative()
+        {
+            int qkvDim = _headKDim * _numKHeads * 2 + _headVDim * _numVHeads;
+            int convDim = _convKernel - 1;
+            int convBlock = convDim * qkvDim;
+            float* convOutBase = (float*)_fvConvOut;
+
+            for (int i = 0; i < _fvRecurrentLayers.Length; i++)
+            {
+                int l = _fvRecurrentLayers[i];
+                float* convSrc = convOutBase + (long)_fvGdnSlot[l] * convBlock;
+                float[] ring = _convState[l];
+                for (int t = 0; t < convDim; t++)
+                {
+                    int dstBase = t * qkvDim;
+                    for (int ch = 0; ch < qkvDim; ch++)
+                        ring[dstBase + ch] = convSrc[ch * convDim + t];
+                }
+                _convStateWriteIdx[l] = 0;
+                InvalidateTensorDeviceCache(_deltaStateTensor[l]);
+                if (_backend != BackendType.GgmlMetal)
+                    GgmlBasicOps.InvalidateHostBuffer((IntPtr)GetFloatPtr(_deltaStateTensor[l]));
+            }
+            _gdnStateHostDirty = false;
+        }
+
+        /// <summary>
+        /// Settle the accepted prefix's recurrent state. Preferred form: commit the
+        /// snapshot into the LIVE state on the device and leave it there, so neither
+        /// this step nor the next pays the ~300 MB round trip. Falls back to pulling
+        /// the slot into the host mirrors when the device commit is unavailable.
+        /// </summary>
+        /// <param name="slot">Tokens back from the end of the verified batch, or -1
+        /// for the post-window state (what a single-row step commits).</param>
+        private unsafe bool CommitRecurrentStateSnapshot(int slot)
+        {
+            if (!PrepareRecurrentStatePointers())
+                return false;
+
+            if (GgmlBasicOps.Qwen35CommitStateSnapshot(slot, _fvRecurrentLayers.Length))
+            {
+                _fvDeviceStateCurrent = true;
+                _gdnStateHostDirty = true;   // the host mirror is now behind
+                return true;
+            }
+
+            // The device commit is unavailable; fall back to pulling the slot into the
+            // host mirrors. There is no host fallback for slot -1 (the caller only
+            // asks for it when the kernel said it deferred, which implies a live
+            // entry), so report failure rather than read a slot that is not there.
+            if (slot < 0
+                || !GgmlBasicOps.Qwen35FetchStateSnapshot(slot, _fvSnapConvPtrs, _fvSnapDeltaPtrs,
+                        _fvRecurrentLayers.Length))
+            {
+                return false;
+            }
+            UnpackRecurrentStateFromNative();
+            _fvDeviceStateCurrent = false;
+            return true;
+        }
+
         // Reusable single-layer descriptor for the fused MTP draft block.
         private Qwen35LayerDecodeArgs[] _mtpDraftLayer;
 
@@ -2097,13 +2359,11 @@ private unsafe bool TryFillMtpDraftLayerArgs(int l, ref Qwen35LayerDecodeArgs a)
 
             if (!isMoe)
             {
-                if (!HasW(_ffnGateUpQW[l], _ffnGateUpF32[l]) || !HasW(_ffnDownQW[l], _ffnDownF32[l]))
+                if (!HasDenseFfnWeights(l) || !HasW(_ffnDownQW[l], _ffnDownF32[l]))
                     return false;
-                var gu = ResolveW(_ffnGateUpQW[l], _ffnGateUpF32[l]);
                 var dn = ResolveW(_ffnDownQW[l], _ffnDownF32[l]);
-                a.GuW = gu.ptr; a.GuType = gu.type; a.GuNe0 = gu.ne0; a.GuNe1 = gu.ne1; a.GuBytes = gu.bytes;
                 a.DownW = dn.ptr; a.DownType = dn.type; a.DownNe0 = dn.ne0; a.DownNe1 = dn.ne1; a.DownBytes = dn.bytes;
-                a.FfDense = (int)(gu.ne1 / 2);
+                FillDenseFfnArgs(ref a, l);
             }
             else
             {
diff --git a/TensorSharp.Models/Models/Qwen35/Qwen35Model.Speculative.cs b/TensorSharp.Models/Models/Qwen35/Qwen35Model.Speculative.cs
index cbb10adc..a3343f24 100644
--- a/TensorSharp.Models/Models/Qwen35/Qwen35Model.Speculative.cs
+++ b/TensorSharp.Models/Models/Qwen35/Qwen35Model.Speculative.cs
@@ -58,11 +58,66 @@ public partial class Qwen35Model : IBatchedSpeculativeModel
         /// <summary>
         /// True when the loaded GGUF contains a usable NextN/MTP draft block.
         /// </summary>
-        public bool HasDraftHead { get; private set; }
+        private bool HasMtpDraftHead { get; set; }
+
+        /// <summary>
+        /// True when SOME learned drafter is attached: the trunk's own NextN/MTP
+        /// block, or an external DFlash/DFlash2 file. They are alternatives, not
+        /// layers - see <see cref="DraftHeadKind"/>.
+        /// </summary>
+        public bool HasDraftHead => HasMtpDraftHead || HasDFlash;
 
         /// <summary>Qwen 3.6's NextN block drafts one token per pass, so it is
-        /// served by <see cref="DraftHeadSpeculator"/>.</summary>
-        public DraftHeadKind DraftHeadKind => HasDraftHead ? DraftHeadKind.PerToken : DraftHeadKind.None;
+        /// served by <see cref="DraftHeadSpeculator"/>; a DFlash drafter proposes a
+        /// whole block per pass and is served by
+        /// <see cref="BlockDraftSpeculator"/>. An attached DFlash file wins, because
+        /// the operator named it explicitly and the two consume different hidden
+        /// rows.</summary>
+        public DraftHeadKind DraftHeadKind => HasDFlash
+            ? DraftHeadKind.Block
+            : (HasMtpDraftHead ? DraftHeadKind.PerToken : DraftHeadKind.None);
+
+        /// <summary>The drafter's hidden row: the trunk's own hidden size for MTP,
+        /// and the concatenated dflash.target_layers residuals for DFlash.</summary>
+        public int SpecFeatureSize => HasDFlash ? _dflash.FeatureSize : Config.HiddenSize;
+
+        /// <summary>DFlash prefills the drafter's ring from the trunk's per-row
+        /// features, so it wants whole micro-batches; MTP has no preference.</summary>
+        public int SpecPrefillChunkSize => HasDFlash ? DFlashPrefillChunkSize : 0;
+
+        /// <summary>
+        /// Three, not the shared default of eight, whenever this checkpoint has
+        /// GatedDeltaNet layers - which every Qwen 3.5/3.6/3.8 hybrid does.
+        ///
+        /// A wide window is priced by the trunk here, not by the drafter: a verify
+        /// over N rows runs the GDN chunked scan rather than the single-token
+        /// recurrent update, and a partial rejection has to restore the recurrent
+        /// state (there is no per-row checkpoint to rewind to) and re-advance over
+        /// the accepted prefix - a second whole-trunk forward, plus the state moving
+        /// across PCIe twice. Both costs grow with the window while acceptance per
+        /// position falls, so the marginal drafted token stops paying long before it
+        /// would on a dense-attention trunk.
+        ///
+        /// Measured on Qwen3.8-27B-UD-IQ3_XXS (RTX 3080 Laptop, greedy, 256 tokens,
+        /// DFlash2 drafter): window 8 -> 9.4 tok/s, 4 -> 13.7, 3 -> 15.5, against
+        /// 18.3 plain. The trunk's own NextN/MTP head behaves the same way, so this
+        /// is not a property of either drafter.
+        ///
+        /// An operator who passes --spec-draft still gets exactly that number.
+        /// </summary>
+        public int SpecPreferredDraftWindow => HasAnyRecurrentLayer ? 3 : 0;
+
+        private bool HasAnyRecurrentLayer
+        {
+            get
+            {
+                if (_isRecurrent == null)
+                    return false;
+                foreach (bool r in _isRecurrent)
+                    if (r) return true;
+                return false;
+            }
+        }
 
         /// <summary>Trunk layer count (excludes NextN/MTP blocks).</summary>
         public int NumTrunkLayers => Config.NumLayers;
@@ -106,16 +161,16 @@ private void CacheMtpWeights()
             // draft from the wrong weight; the trunk itself is unaffected.
             bool borrowsSplitHead = _tpLmHeadKey != null && _mtpHeadQW == null && _mtpHeadF32 == null;
 
-            HasDraftHead = _numNextnLayers == 1 && hasProj && _mtpEnormW != null && _mtpHnormW != null
+            HasMtpDraftHead = _numNextnLayers == 1 && hasProj && _mtpEnormW != null && _mtpHnormW != null
                 && hasAttn && _attnNormW[_mtpLayerIdx] != null && _postAttnNormW[_mtpLayerIdx] != null
                 && !borrowsSplitHead;
 
             if (borrowsSplitHead)
                 Console.WriteLine("  NextN/MTP block has no own head and the LM head is column-parallel under TP; " +
                     "MTP drafting disabled.");
-            else if (_numNextnLayers > 0 && !HasDraftHead)
+            else if (_numNextnLayers > 0 && !HasMtpDraftHead)
                 Console.WriteLine("  NextN/MTP block present but incomplete; MTP drafting disabled.");
-            else if (HasDraftHead)
+            else if (HasMtpDraftHead)
                 Console.WriteLine($"  NextN/MTP draft head ready (layer {_mtpLayerIdx}, " +
                     $"moe={( _isMoeLayer != null && _isMoeLayer[_mtpLayerIdx] ? "yes" : "no")}, " +
                     $"ownHead={(_mtpHeadQW != null || _mtpHeadF32 != null ? "yes" : "no")})");
@@ -208,7 +263,9 @@ private unsafe Tensor MtpProjectInput(int[] tokens, float[] hRows)
         /// </summary>
         public unsafe void DraftStep(int token, float[] hPrev, int pos, float[] logitsOut, float[] hOut)
         {
-            if (!HasDraftHead)
+            if (HasDFlash)
+                throw new NotSupportedException("A DFlash drafter proposes whole blocks; use DraftBlock.");
+            if (!HasMtpDraftHead)
                 throw new InvalidOperationException("Model has no NextN/MTP draft block.");
             EnterSpecSession();
             EnsureCacheCapacity(pos + 1);
@@ -263,7 +320,15 @@ public unsafe void DraftStep(int token, float[] hPrev, int pos, float[] logitsOu
 
         public void DraftCatchUp(int[] tokens, float[] hRows, int startPos)
         {
-            if (!HasDraftHead)
+            if (HasDFlash)
+            {
+                // The DFlash ring holds committed positions only, and the executor
+                // hands back exactly the rows it committed.
+                EnterSpecSession();
+                DFlashCommit(tokens, hRows, startPos);
+                return;
+            }
+            if (!HasMtpDraftHead)
                 throw new InvalidOperationException("Model has no NextN/MTP draft block.");
             EnterSpecSession();
             EnsureCacheCapacity(startPos + tokens.Length);
@@ -285,6 +350,60 @@ public void DraftCatchUp(int[] tokens, float[] hRows, int startPos)
             x.Dispose();
         }
 
+        // Fold the MTP catch-up into the first draft step (llama.cpp's draft-mtp
+        // runs its block over n_accepted + 1 rows). TS_MTP_FOLD_CATCHUP=0 goes back
+        // to a catch-up pass plus a separate first DraftStep.
+        private static readonly bool _mtpFoldCatchUpEnabled =
+            !string.Equals(Environment.GetEnvironmentVariable("TS_MTP_FOLD_CATCHUP"), "0", StringComparison.Ordinal);
+
+        /// <summary>Only the NextN/MTP head folds. A DFlash drafter proposes whole
+        /// blocks and its commit is a ring write costing ~1 ms, so there is nothing
+        /// worth folding there.</summary>
+        public bool SupportsFusedCatchUpStep
+            => _mtpFoldCatchUpEnabled && HasMtpDraftHead && !HasDFlash;
+
+        private float[] _mtpFoldNormed;
+
+        /// <inheritdoc />
+        public unsafe void DraftCatchUpAndStep(int[] tokens, float[] hRows, int startPos,
+            float[] logitsOut, float[] hOut)
+        {
+            if (!HasMtpDraftHead || HasDFlash)
+                throw new InvalidOperationException("Model has no NextN/MTP draft block to fold.");
+            int n = tokens.Length;
+            int H = Config.HiddenSize;
+            EnterSpecSession();
+            EnsureCacheCapacity(startPos + n);
+
+            // ONE block pass over every row. The kernel folds the LM head over the
+            // last n_logits rows (here 1), so logitsOut is already the last row's;
+            // the normed hidden comes back for all n rows and the last is the one
+            // that chains the next draft step.
+            Tensor x = MtpProjectInput(tokens, hRows);
+            if (_mtpFoldNormed == null || _mtpFoldNormed.Length < (long)H * n)
+                _mtpFoldNormed = new float[(long)H * n];
+            if (TryFusedMtpBlock(x, startPos, n, _mtpFoldNormed, logitsOut, nLogitRows: 1))
+            {
+                x.Dispose();
+                Array.Copy(_mtpFoldNormed, (long)(n - 1) * H, hOut, 0, H);
+                return;
+            }
+            x.Dispose();
+
+            // The fused block declined this shape. Fall back to the two-call form
+            // rather than the op-by-op fold, so this path stays the one that is
+            // already covered by DraftCatchUp/DraftStep.
+            if (n > 1)
+            {
+                var replay = new int[n - 1];
+                Array.Copy(tokens, replay, n - 1);
+                DraftCatchUp(replay, hRows, startPos);
+            }
+            var hLast = new float[H];
+            Array.Copy(hRows, (long)(n - 1) * H, hLast, 0, H);
+            DraftStep(tokens[n - 1], hLast, startPos + n - 1, logitsOut, hOut);
+        }
+
         /// <summary>
         /// Trunk forward for speculative decoding. Identical math to Forward()
         /// but additionally captures the post-final-norm hidden state of every
@@ -317,6 +436,41 @@ public unsafe void SpecForward(int[] tokens, float[] hAllOut, float[] logitsOut,
             Tensor hidden = Embedding(tokens);
             _embTicks += Stopwatch.GetTimestamp() - t0;
 
+            if (HasDFlash)
+            {
+                // Buffer-size contract (ISpeculativeTarget.SpecForward): a hidden
+                // buffer too small for one row per token means the caller only wants
+                // the LAST row, written to row 0.
+                int feat = _dflash.FeatureSize;
+                bool captureAll = false, captureLast = false;
+                if (hAllOut != null && hAllOut.LongLength > 0)
+                {
+                    captureAll = hAllOut.LongLength >= (long)seqLen * feat;
+                    captureLast = !captureAll;
+                    if (captureLast && hAllOut.LongLength < feat)
+                    {
+                        throw new ArgumentException(
+                            $"DFlash hidden capture buffer holds {hAllOut.LongLength} floats; one feature row needs {feat}.",
+                            nameof(hAllOut));
+                    }
+                }
+
+                if (!TryDFlashSpecForwardFused(hidden, startPos, seqLen, hAllOut, logitsOut,
+                        allLogitsRows, captureAll, captureLast))
+                {
+                    DFlashSpecForwardPerOp(hidden, startPos, seqLen, hAllOut, logitsOut,
+                        allLogitsRows, captureAll, captureLast);
+                }
+                else
+                {
+                    hidden.Dispose();
+                }
+                _cacheSeqLen += seqLen;
+                _forwardCount++;
+                _forwardSw.Stop();
+                return;
+            }
+
             // Fast path: run the whole trunk over the N tokens as ONE fused GGML
             // graph (TSGgml_Qwen35ModelVerify) instead of the op-by-op layer loop.
             // Writes hAllOut (post-norm hidden) + logitsOut directly; advances KV +
@@ -456,13 +610,84 @@ private bool TryFusedVerifyTrunk(Tensor hidden, int startPos, int seqLen,
         /// </summary>
         public void SpecEnsureCapacity(int requiredSeqLen) => EnsureCacheCapacity(requiredSeqLen);
 
+        /// <summary>
+        /// True when the accepted prefix of the last verify is already committed:
+        /// its attention KV was written at the right positions by the verify itself,
+        /// and its recurrent state came back out of a per-row snapshot in
+        /// <see cref="SpecOnVerifyAccepted"/>. The executor then only has to rewind
+        /// the position, instead of restoring a pre-verify state copy and
+        /// re-forwarding the accepted prefix through all 64 layers.
+        ///
+        /// Necessarily per-step rather than a constant: whether the verify kept
+        /// snapshots depends on the shape it ran at (only the persisted, all-rows,
+        /// host-state verify does), and getting it wrong in either direction decodes
+        /// from the wrong recurrent state.
+        /// </summary>
+        public bool SpecVerifyPersistsAcceptedKv => _fvAcceptedPrefixCommitted;
+
+        /// <summary>
+        /// Settle the recurrent state for the accepted prefix. When the verify left
+        /// per-row snapshots on the device, the state after row <paramref name="acceptedRows"/>
+        /// is slot (rows - 1 - accepted) - counting back from the end of the batch -
+        /// and one fetch replaces the whole restore-and-re-forward.
+        ///
+        /// Called on EVERY speculative step, full acceptance included: with snapshots
+        /// on, even a fully-accepted verify has not written its post-window state to
+        /// the host mirror yet, and slot 0 is that state.
+        /// </summary>
+        public void SpecOnVerifyAccepted(int acceptedRows, int verifyRows)
+        {
+            _fvAcceptedPrefixCommitted = false;
+            if (_fvSnapshotRows <= 0)
+                return;                       // the old path already drained the state
+
+            int rows = _fvSnapshotRows;
+            _fvSnapshotRows = 0;
+            int slot = rows - 1 - acceptedRows;
+            if (slot < 0 || slot >= rows)
+            {
+                // Cannot happen for a well-formed accept count, and silently taking
+                // the wrong slot would decode from the wrong state.
+                throw new InvalidOperationException(
+                    $"Recurrent-state snapshot slot {slot} outside [0, {rows}) (accepted {acceptedRows} of {verifyRows}).");
+            }
+
+            if (!CommitRecurrentStateSnapshot(slot))
+            {
+                // The state is still on the device and the host mirror is stale, so
+                // the executor MUST take the restore-and-re-forward path - which is
+                // exactly what leaving _fvAcceptedPrefixCommitted false selects, and
+                // which is correct because the pre-verify snapshot is untouched.
+                return;
+            }
+            _fvAcceptedPrefixCommitted = true;
+        }
+
         /// <summary>
         /// Snapshot the GDN recurrent state of every trunk layer. Taken right
         /// before a speculative verify batch so a partial rejection can roll the
         /// recurrent state back (attention KV needs only a position rewind).
         /// </summary>
+        /// <summary>
+        /// Set when <see cref="SpecSnapshotRecurrentState"/> skipped its host copy
+        /// because the pre-verify state was already sitting in the verify kernel's
+        /// live device slices - which a verify only ever READS, so those slices ARE
+        /// the snapshot until a snapshot commit overwrites them.
+        /// </summary>
+        private bool _fvSnapshotIsDeviceLive;
+
         public void SpecSnapshotRecurrentState()
         {
+            // Nothing to copy: the state the verify is about to run from lives in the
+            // shared device slices, the verify writes its results elsewhere (the
+            // *_state_out slices and the snapshot slots), and the only thing that ever
+            // overwrites the live slices is a snapshot commit - which happens after
+            // the rollback decision. So the slices remain a perfectly good "snapshot"
+            // for as long as one is needed, at no cost.
+            _fvSnapshotIsDeviceLive = _fvDeviceStateCurrent;
+            if (_fvSnapshotIsDeviceLive)
+                return;
+
             // Direct-CUDA fast path: snapshot the GDN state device-to-device
             // (async cuMemcpyDtoD on the stream) instead of draining it to host
             // bytes. The host path does an EnsureHostReadable DtoH per recurrent
@@ -492,6 +717,17 @@ public void SpecSnapshotRecurrentState()
         /// <summary>Restore the GDN recurrent state captured by <see cref="SpecSnapshotRecurrentState"/>.</summary>
         public void SpecRestoreRecurrentState()
         {
+            if (_fvSnapshotIsDeviceLive)
+            {
+                // The pre-verify state is in the live device slices; the host mirrors
+                // are whatever the verify left there. Bring the slices back so the
+                // kept-prefix re-forward and any op-by-op path see the right state.
+                _fvSnapshotIsDeviceLive = false;
+                _fvDeviceStateCurrent = true;      // the slices are authoritative
+                DrainDeviceRecurrentState();
+                return;
+            }
+
             if (_backend == BackendType.Cuda)
             {
                 MtpRestoreRecurrentStateCudaDevice();
@@ -623,7 +859,10 @@ public void SpecRewindCache(int length)
         /// verify (<see cref="TryFullModelVerify"/>) is enabled we route spec to the
         /// LINEAR trunk instead (SpecForward), whose KV/GDN state the fused verify
         /// reads/writes; the batched paged trunk uses a different (paged) store.</summary>
-        public bool SupportsBatchedSpecTrunk => HasDraftHead && IsGgmlBackend && IsBatchedPathEnabled() && !_fusedVerifyEnabled;
+        // A DFlash drafter keeps ONE ring for ONE sequence and its catch-up is driven
+        // from the linear trunk's per-row features, so it cannot ride the paged
+        // multi-sequence trunk; those requests take the linear speculative path.
+        public bool SupportsBatchedSpecTrunk => HasMtpDraftHead && !HasDFlash && IsGgmlBackend && IsBatchedPathEnabled() && !_fusedVerifyEnabled;
 
         public void SpecForwardBatched(SequenceState seq, int[] tokens, int startPos,
             float[] hAllOut, float[] logitsOut, bool allLogitsRows)
diff --git a/TensorSharp.Models/Models/Qwen35/Qwen35Model.cs b/TensorSharp.Models/Models/Qwen35/Qwen35Model.cs
index 7ada3442..df06624b 100644
--- a/TensorSharp.Models/Models/Qwen35/Qwen35Model.cs
+++ b/TensorSharp.Models/Models/Qwen35/Qwen35Model.cs
@@ -1,4 +1,4 @@
-// Copyright (c) Zhongkai Fu. All rights reserved.
+// Copyright (c) Zhongkai Fu. All rights reserved.
 // https://github.com/zhongkaifu/TensorSharp
 //
 // This file is part of TensorSharp.
@@ -284,6 +284,18 @@ private static int ResolveMlxEvalEveryNLayers()
 
         // Pre-resolved weight references for non-MoE FFN paths.
         private QuantizedWeight[] _ffnGateUpQW;
+        // Mixed-quant "UD"/dynamic GGUFs can store ffn_gate and ffn_up in different
+        // GGML types (IQ2_XS vs IQ2_S, IQ1_S vs IQ2_XXS, ...). One fused tensor
+        // cannot represent that, and both of those types need an importance matrix
+        // to requantize, so ModelBase.FuseGateUpWeights leaves such a layer alone.
+        // These hold the unfused pair for exactly those layers; the FFN then runs
+        // two matmuls instead of one, with the weights untouched. Half the layers
+        // of an unsloth Qwen3.8 UD quant land here, so this is the normal case for
+        // that family, not a corner.
+        private QuantizedWeight[] _ffnGateSplitQW;
+        private Tensor[] _ffnGateSplitF32;
+        private QuantizedWeight[] _ffnUpSplitQW;
+        private Tensor[] _ffnUpSplitF32;
         private Tensor[] _ffnGateUpF32;
         private QuantizedWeight[] _ffnDownQW;
         private Tensor[] _ffnDownF32;
@@ -356,7 +368,11 @@ private void WarnFusedFfnDeclinedOnce(string reason)
         private long _mlxEvalBoundaryTicks;
         private long _mlxCacheEvalTicks;
 
-        public Qwen35Model(string ggufPath, BackendType backend, int tpDegree = 1, ITensorParallelGroup tpGroup = null)
+        /// <param name="draftModelPath">Optional DFlash / DFlash2 drafter GGUF
+        /// (general.architecture = "dflash"). When present it replaces the trunk's
+        /// own NextN/MTP block as the drafter.</param>
+        public Qwen35Model(string ggufPath, BackendType backend, int tpDegree = 1, ITensorParallelGroup tpGroup = null,
+            string draftModelPath = null)
             : base(ggufPath, backend, tpDegree, tpGroup)
         {
             _useMetalGdnInplaceState = ShouldUseMetalGdnInplaceState(
@@ -489,6 +505,11 @@ public Qwen35Model(string ggufPath, BackendType backend, int tpDegree = 1, ITens
             InitGDNBuffers();
             CacheRecurrentWeights();
             CacheMtpWeights();
+
+            // --draft-model / TS_QWEN35_DFLASH. Last, because the drafter's tensors
+            // are merged into the trunk's weight dictionaries and its KV ring comes
+            // off the same allocator.
+            TryLoadQwen35DFlash(draftModelPath);
         }
 
         private unsafe void FuseAttentionProjectionWeights()
@@ -963,6 +984,10 @@ private unsafe void CacheRecurrentWeights()
 
             _ffnGateUpQW = new QuantizedWeight[n];
             _ffnGateUpF32 = new Tensor[n];
+            _ffnGateSplitQW = new QuantizedWeight[n];
+            _ffnGateSplitF32 = new Tensor[n];
+            _ffnUpSplitQW = new QuantizedWeight[n];
+            _ffnUpSplitF32 = new Tensor[n];
             _ffnDownQW = new QuantizedWeight[n];
             _ffnDownF32 = new Tensor[n];
 
@@ -981,6 +1006,15 @@ private unsafe void CacheRecurrentWeights()
                 _weights.TryGetValue(_postAttnNormKey[l], out _postAttnNormW[l]);
                 _quantWeights.TryGetValue(_ffnGateUpKey[l], out _ffnGateUpQW[l]);
                 _weights.TryGetValue(_ffnGateUpKey[l], out _ffnGateUpF32[l]);
+                if (_ffnGateUpQW[l] == null && _ffnGateUpF32[l] == null)
+                {
+                    // Unfused mixed-quant layer: keep the pair as it was quantized.
+                    string p = $"blk.{l}.";
+                    _quantWeights.TryGetValue(p + "ffn_gate.weight", out _ffnGateSplitQW[l]);
+                    _weights.TryGetValue(p + "ffn_gate.weight", out _ffnGateSplitF32[l]);
+                    _quantWeights.TryGetValue(p + "ffn_up.weight", out _ffnUpSplitQW[l]);
+                    _weights.TryGetValue(p + "ffn_up.weight", out _ffnUpSplitF32[l]);
+                }
                 _quantWeights.TryGetValue(_ffnDownKey[l], out _ffnDownQW[l]);
                 _weights.TryGetValue(_ffnDownKey[l], out _ffnDownF32[l]);
 
@@ -1246,6 +1280,14 @@ protected override void ResetKVCacheCore()
 
         public override bool SupportsKVCacheTruncation => false;
 
+        /// <summary>Qwen 3.5/3.8 implement the split FFN: BuildLayerKeys populates
+        /// _ffnGateSplitQW/_ffnUpSplitQW when the fused tensor is absent,
+        /// FFNCachedSplitGateUp runs the managed path, and all three fused native
+        /// graphs branch on <c>gu_w == nullptr</c> to emit two matmuls. Every
+        /// unsloth "UD" quant of this family needs it - about half their layers
+        /// store ffn_gate and ffn_up in different IQ types.</summary>
+        protected override bool SupportsSplitGateUpFfn => true;
+
         // Per-block snapshot for Qwen 3.5 (mix of attention layers and GDN
         // recurrent layers). Each block bundles:
         //   * For every attention layer L: K bytes for [start,start+B), V bytes
@@ -1469,6 +1511,9 @@ private static bool CopyAttentionIn(Tensor cacheTensor, int destToken, int token
 
         private bool CopyGdnStateOut(int layer, Span<byte> destination, out int written)
         {
+            // Reads the host mirrors; see RecurrentBlock.
+            DrainDeviceRecurrentState();
+
             written = 0;
             SyncCudaGdnConvStateToHost(layer);
             float[] conv = _convState[layer];
@@ -1498,6 +1543,9 @@ private bool CopyGdnStateOut(int layer, Span<byte> destination, out int written)
 
         private bool CopyGdnStateIn(int layer, ReadOnlySpan<byte> source, out int read)
         {
+            // Writes the host mirrors, so whatever the device holds is superseded.
+            DrainDeviceRecurrentState();
+
             read = 0;
             float[] conv = _convState[layer];
             int convBytes = conv.Length * sizeof(float);
@@ -3389,6 +3437,8 @@ private unsafe void DeinterleaveQGate(Tensor qFull, int seqLen, int numHeads, in
         private Tensor FFNCached(Tensor input, int layer, int seqLen)
         {
             int intermSize = Config.IntermediateSize;
+            if (_ffnGateUpQW[layer] == null && _ffnGateUpF32[layer] == null)
+                return FFNCachedSplitGateUp(input, layer, seqLen);
             Tensor gateUp = LinearForwardCached(input, _ffnGateUpQW[layer], _ffnGateUpF32[layer]);
             int halfDim = intermSize > 0 ? intermSize : (int)(gateUp.Sizes[1] / 2);
 
@@ -3478,6 +3528,15 @@ private Tensor FFNCachedFused(Tensor residual, Tensor postNormW, int layer, int
             // Fused norm + gate_up projection. Decode reuses a pre-allocated [1, 2*intermSize]
             // buffer to avoid one tensor allocation per layer per token.
             Tensor gateUp = null;
+            if (_ffnGateUpQW[layer] == null && _ffnGateUpF32[layer] == null)
+            {
+                // Unfused mixed-quant gate/up: norm once, then two matmuls.
+                Tensor splitNormed = RMSNormOpCached(residual, postNormW);
+                Tensor splitGate = FFNCachedSplitGateUp(splitNormed, layer, seqLen);
+                splitNormed.Dispose();
+                return splitGate;
+            }
+
             bool ownsGateUp = true;
             if (postNormW != null && _ffnGateUpQW[layer] != null && IsGgmlBackend)
             {
@@ -3549,6 +3608,29 @@ private Tensor FFNCachedFused(Tensor residual, Tensor postNormW, int layer, int
             return down;
         }
 
+        /// <summary>
+        /// silu(gate(x)) * up(x) -> down for a layer whose gate and up could not be
+        /// fused (different GGML types, neither requantizable without an importance
+        /// matrix). Same arithmetic as the fused path, one extra matmul.
+        /// </summary>
+        private Tensor FFNCachedSplitGateUp(Tensor input, int layer, int seqLen)
+        {
+            Tensor gate = LinearForwardCached(input, _ffnGateSplitQW[layer], _ffnGateSplitF32[layer]);
+            Tensor up = LinearForwardCached(input, _ffnUpSplitQW[layer], _ffnUpSplitF32[layer]);
+            Ops.SiLUMul(gate, gate, up);
+            up.Dispose();
+            Tensor down = LinearForwardCached(gate, _ffnDownQW[layer], _ffnDownF32[layer]);
+            gate.Dispose();
+            return down;
+        }
+
+        /// <summary>True when layer <paramref name="l"/> has a usable dense FFN,
+        /// fused or split.</summary>
+        private bool HasDenseFfnWeights(int l)
+            => (_ffnGateUpQW[l] != null || _ffnGateUpF32[l] != null)
+               || ((_ffnGateSplitQW[l] != null || _ffnGateSplitF32[l] != null)
+                   && (_ffnUpSplitQW[l] != null || _ffnUpSplitF32[l] != null));
+
         /// <summary>
         /// RMSNorm with a pre-resolved alpha tensor, avoiding the dictionary lookup that
         /// <see cref="ModelBase.RMSNormOp"/> performs per call. The arithmetic is identical.
@@ -5994,6 +6076,8 @@ public override void Dispose()
             // Free the on-device MoE decode pointer tables (device u64 buffers).
             FreeQwenCudaMoETables();
 
+            DisposeDFlash();
+
             VisionEncoder?.Dispose();
             foreach (var (visionEmbeddings, _) in _visionEmbeddingsList)
                 visionEmbeddings?.Dispose();
diff --git a/TensorSharp.Models/Models/Qwen4Exp/Qwen4ExpModel.Forward.cs b/TensorSharp.Models/Models/Qwen4Exp/Qwen4ExpModel.Forward.cs
new file mode 100644
index 00000000..1d725ded
--- /dev/null
+++ b/TensorSharp.Models/Models/Qwen4Exp/Qwen4ExpModel.Forward.cs
@@ -0,0 +1,1737 @@
+// Copyright (c) Zhongkai Fu. All rights reserved.
+// https://github.com/zhongkaifu/TensorSharp
+//
+// This file is part of TensorSharp.
+//
+// TensorSharp is licensed under the BSD-3-Clause license found in the LICENSE file in the root directory of this source tree.
+//
+// TensorSharp is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the BSD-3-Clause License for more details.
+using System;
+using System.Diagnostics;
+using TensorSharp.Core;
+using TensorSharp.GGML;
+
+namespace TensorSharp.Models
+{
+    public partial class Qwen4ExpModel
+    {
+        // KV for the full-attention layers only; the GDN layers carry recurrent state
+        // instead. Indexed by absolute layer, null on a recurrent one.
+        private Tensor[] _kCache;
+        private Tensor[] _vCache;
+
+        // Indexer keys, cached RAW: the pooling that produces a block key happens
+        // before the norm and the rotation, so a cached post-rotation key could not be
+        // re-pooled at a different block boundary.
+        private Tensor[] _idxKCache;
+
+        // GDN state, host side. conv is a [convKernel-1, convDim] ring per layer and
+        // delta is [numVHeads, headKDim, headVDim].
+        private float[][] _gdnConvState;
+
+        // PLE depthwise conv history: (kernel-1)*ngram rows of hcDim.
+        private float[] _pleConvState;
+
+        private int _convDim;
+
+        // Rows currently allocated in the attention caches; grows on demand up to
+        // _maxContextLength.
+        private int _kvCacheCapacity;
+        private int _initialKvCacheCapacity;
+
+        // Per-phase tick counters. Cheap (one timestamp per phase per layer) and the
+        // only way to tell a slow recurrence from a slow MoE without a profiler.
+        // Reported by TS_Q4E_PROFILE=1.
+        internal long Q4ePleTicks, Q4eHcTicks, Q4eGdnTicks, Q4eAttnTicks, Q4eMoeTicks, Q4eHeadTicks, Q4eSpanTicks;
+        internal long Q4eGdnProjTicks, Q4eGdnPrepTicks, Q4eGdnKernelTicks, Q4eGdnOutTicks;
+        internal long Q4eAttnProjTicks, Q4eAttnNormTicks, Q4eAttnRopeTicks, Q4eAttnCoreTicks, Q4eAttnOutTicks;
+        private static readonly bool _q4eProfile =
+            string.Equals(Environment.GetEnvironmentVariable("TS_Q4E_PROFILE"), "1", StringComparison.Ordinal);
+
+        private void ReportQ4eProfile(int tokens)
+        {
+            if (!_q4eProfile) return;
+            double f = 1000.0 / Stopwatch.Frequency;
+            Console.WriteLine(
+                $"[q4e-profile] tokens={tokens} ple={Q4ePleTicks * f:F0}ms hc={Q4eHcTicks * f:F0}ms " +
+                $"gdn={Q4eGdnTicks * f:F0}ms attn={Q4eAttnTicks * f:F0}ms moe={Q4eMoeTicks * f:F0}ms " +
+                $"span={Q4eSpanTicks * f:F0}ms " +
+                $"head={Q4eHeadTicks * f:F0}ms");
+            Console.WriteLine(
+                $"[q4e-profile]   gdn: proj={Q4eGdnProjTicks * f:F0} prep={Q4eGdnPrepTicks * f:F0} " +
+                $"kernel={Q4eGdnKernelTicks * f:F0} out={Q4eGdnOutTicks * f:F0}");
+            Console.WriteLine(
+                $"[q4e-profile]   attn: proj={Q4eAttnProjTicks * f:F0} norm={Q4eAttnNormTicks * f:F0} " +
+                $"rope={Q4eAttnRopeTicks * f:F0} core={Q4eAttnCoreTicks * f:F0} out={Q4eAttnOutTicks * f:F0}");
+        }
+
+        private void InitCaches(int initialSeqLen, int maxSeqLen)
+        {
+            _maxContextLength = maxSeqLen;
+            _kvCacheCapacity = initialSeqLen;
+            _initialKvCacheCapacity = initialSeqLen;
+            ApplyModelAlignedKvCacheDefault(_quantWeights);
+            DType kvDtype = _kvCacheDtype.ToDType();
+
+            int nLayer = Config.NumLayers;
+            _kCache = new Tensor[nLayer];
+            _vCache = new Tensor[nLayer];
+            _idxKCache = new Tensor[nLayer];
+            _gdnConvState = new float[nLayer][];
+
+            int keyDim = _headKDim * _numKHeads;
+            int valueDim = _headVDim * _numVHeads;
+            _convDim = keyDim * 2 + valueDim;
+
+            for (int l = 0; l < nLayer; l++)
+            {
+                if (_isRecurrent[l])
+                {
+                    _gdnConvState[l] = new float[(long)(_convKernel - 1) * _convDim];
+                    continue;
+                }
+
+                _kCache[l] = new Tensor(_allocator, kvDtype, Config.NumKVHeads, initialSeqLen, Config.HeadDim);
+                _vCache[l] = new Tensor(_allocator, kvDtype, Config.NumKVHeads, initialSeqLen, Config.HeadDim);
+                InitializeCacheTensor(_kCache[l]);
+                InitializeCacheTensor(_vCache[l]);
+
+                if (UsesQsa(l))
+                {
+                    _idxKCache[l] = new Tensor(_allocator, DType.Float32, 1, initialSeqLen, _indexerHeadDim);
+                    InitializeCacheTensor(_idxKCache[l]);
+                }
+            }
+
+            if (_pleHeads > 0)
+                // PINNED: the span kernel seeds its device conv state from this address.
+                _pleConvState = GC.AllocateArray<float>(
+                    checked((int)((long)(_pleConvKernel - 1) * _pleNgram * _hcDim)), pinned: true);
+
+            _cacheSeqLen = 0;
+        }
+
+        private void EnsureCacheCapacity(int requiredSeqLen)
+        {
+            if (requiredSeqLen <= _kvCacheCapacity)
+                return;
+            if (requiredSeqLen > _maxContextLength)
+            {
+                throw new InvalidOperationException(
+                    $"Sequence length {requiredSeqLen} exceeds the configured context length {_maxContextLength}.");
+            }
+
+            int newCapacity = Math.Min(_maxContextLength, Math.Max(requiredSeqLen, _kvCacheCapacity * 2));
+            // The fused path writes the KV cache on the DEVICE; the host mirror
+            // is stale until synced. Growing from the stale mirror would carry
+            // garbage forward for every past position.
+            EnsureKvCacheHostSynchronized();
+            DType kvDtype = _kvCacheDtype.ToDType();
+            for (int l = 0; l < Config.NumLayers; l++)
+            {
+                if (_isRecurrent[l]) continue;
+                GrowCache(ref _kCache[l], Config.NumKVHeads, newCapacity, Config.HeadDim, kvDtype);
+                GrowCache(ref _vCache[l], Config.NumKVHeads, newCapacity, Config.HeadDim, kvDtype);
+                if (_idxKCache[l] != null)
+                    GrowCache(ref _idxKCache[l], 1, newCapacity, _indexerHeadDim, DType.Float32);
+            }
+            _kvCacheCapacity = newCapacity;
+            // The KV tensors just moved: the pinned attention descriptors hold the
+            // old storage pointers and byte counts, and every graph keyed on them -
+            // per-layer and span alike - is stale. Dropping the array forces a refill
+            // with the new pointers and, through the graph keys, a rebuild.
+            _attnArgs = null;
+        }
+
+        private void GrowCache(ref Tensor cache, int heads, int newLen, int dim, DType dtype)
+        {
+            var grown = new Tensor(_allocator, dtype, heads, newLen, dim);
+            // Zero UNCONDITIONALLY. InitializeCacheTensor skips the fill on
+            // GgmlCuda/Mlx, and the native span kernel's own memset is guarded by
+            // `position == 0`, which a mid-sequence grow never satisfies - so the
+            // rows past _cacheSeqLen would keep whatever the memory pool handed
+            // back. The flash window is padded to a 256 stride past n_kv and those
+            // rows ARE read: a masked column contributes nothing only if its K row
+            // is finite, and an Inf there plus the -inf mask is NaN, which takes
+            // the whole softmax row. Fires on any conversation past the initial
+            // 8192-token capacity - e.g. a --max-tokens 20000 answer.
+            Ops.Fill(grown, 0f);
+            if (_cacheSeqLen > 0)
+            {
+                using var src = cache.Narrow(1, 0, _cacheSeqLen);
+                using var dst = grown.Narrow(1, 0, _cacheSeqLen);
+                Ops.Copy(dst, src);
+            }
+            // Evict the old tensor's device-resident copy before its host
+            // pointer is freed: a recycled pinned address must never re-find
+            // the smaller stale device slab.
+            InvalidateTensorDeviceCache(cache);
+            cache.Dispose();
+            cache = grown;
+        }
+
+        /// <summary>Download the device-resident KV copies back into the host
+        /// mirrors. The fused kernels write KV on the device only; anything that
+        /// reads or copies the host arrays afterwards must sync first.</summary>
+        private void EnsureKvCacheHostSynchronized()
+        {
+            if (!_kvCacheHostStale || !IsGgmlBackend || _kCache == null)
+                return;
+            for (int l = 0; l < Config.NumLayers; l++)
+            {
+                if (_kCache[l] != null) SyncTensorHostCache(_kCache[l]);
+                if (_vCache[l] != null) SyncTensorHostCache(_vCache[l]);
+                if (_idxKCache[l] != null) SyncTensorHostCache(_idxKCache[l]);
+            }
+            _kvCacheHostStale = false;
+        }
+
+        private unsafe void InvalidateActiveSeqState()
+        {
+            if (_gdnConvStateT != null)
+                foreach (var t in _gdnConvStateT)
+                    if (t != null) GgmlBasicOps.Qwen4ExpInvalidateSeqState((IntPtr)GetFloatPtr(t));
+            if (_pleConvState != null && _pleConvState.Length > 0)
+                GgmlBasicOps.Qwen4ExpInvalidateSeqState(
+                    System.Runtime.InteropServices.Marshal.UnsafeAddrOfPinnedArrayElement(_pleConvState, 0));
+        }
+
+        protected override void ResetKVCacheCore()
+        {
+            _cacheSeqLen = 0;
+            for (int l = 0; l < Config.NumLayers; l++)
+            {
+                if (_gdnConvState[l] != null) Array.Clear(_gdnConvState[l]);
+                // The delta-net state lives in a device tensor now (the fused kernel
+                // updates it in place), so clearing the old host array is not enough.
+                if (_gdnStateT != null && _gdnStateT[l] != null)
+                {
+                    Ops.Fill(_gdnStateT[l], 0f);
+                    InvalidateTensorDeviceCache(_gdnStateT[l]);
+                }
+                // The fused recurrent kernel keeps the conv history in its own
+                // [d_conv-1, conv_dim] tensor rather than the host ring.
+                if (_gdnConvStateT != null && _gdnConvStateT[l] != null)
+                {
+                    Ops.Fill(_gdnConvStateT[l], 0f);
+                    InvalidateTensorDeviceCache(_gdnConvStateT[l]);
+                }
+                if (_gdnConvWriteIdx != null) _gdnConvWriteIdx[l] = 0;
+            }
+            if (_pleConvState != null) Array.Clear(_pleConvState);
+            ResetPleHistory();
+            _mropeCacheGap = 0;
+
+            // The fused kernels keep the conv and recurrent state in their OWN device
+            // buffers inside a persisted graph, so zeroing the host copies above does
+            // not reach them. Dropping the graphs makes the next call rebuild and
+            // re-upload from the (now zeroed) host state. Without this the kernel
+            // warmup's state leaked into the first real generation, which read as
+            // fluent-looking noise while every per-layer check passed.
+            if (IsGgmlBackend && (_ffnArgs != null || _gdnArgs != null))
+            {
+                // Re-arm the seed upload for THIS conversation's recurrent-state
+                // entries (their host copies were just zeroed above); other
+                // sequences' entries stay device-authoritative. The graph reset
+                // below then forces the rebuild that performs the re-seed.
+                InvalidateActiveSeqState();
+                GgmlBasicOps.Qwen4ExpResetFfnCache();
+            }
+        }
+
+        protected override float[] ForwardCore(int[] tokens)
+        {
+            try
+            {
+                return ForwardCoreInner(tokens);
+            }
+            catch
+            {
+                // A thrown forward must not leak this request's pending
+                // multimodal state into the NEXT sequence the engine runs (the
+                // per-sequence executor catches per-request errors and moves on).
+                _pendingMRoPEPositions = null;
+                foreach (var (emb, _) in _visionEmbeddingsList) emb?.Dispose();
+                _visionEmbeddingsList.Clear();
+                throw;
+            }
+        }
+
+        private float[] ForwardCoreInner(int[] tokens)
+        {
+            _forwardSw.Start();
+            int seqLen = tokens.Length;
+            int startPos = _cacheSeqLen;
+            EnsureCacheCapacity(startPos + seqLen);
+            WarnIfQsaBudgetExceeded(startPos + seqLen);
+
+            long tEmb = Stopwatch.GetTimestamp();
+            Tensor embd = Embedding(tokens);           // [T, n_embd]
+            InjectVisionEmbeddings(embd, seqLen);
+            _embTicks += Stopwatch.GetTimestamp() - tEmb;
+            PhaseLog(seqLen, "embedding", tEmb);
+
+            // The wide residual starts as hc identical copies of the embedding.
+            long tBr = Stopwatch.GetTimestamp();
+            Tensor res = BroadcastToStreams(embd, seqLen);
+            embd.Dispose();
+            PhaseLog(seqLen, "broadcast", tBr);
+
+            // Hand the residual to the device once; the fused halves chain through it
+            // and only the layers still running op-by-op pull it back.
+            _resOnDevice = false;
+
+            // The whole token as (almost) one graph: every PLE-free run of layers -
+            // both halves of each - chained into a single persisted GGML graph, so a
+            // decode token is 2 graph launches instead of 96 and the residual crosses
+            // the bus twice instead of 192 times. Anything the span declines falls
+            // back to the per-layer loop below.
+            long tSpan = Stopwatch.GetTimestamp();
+            bool spanDone = TryFusedTokenSpans(res, tokens, seqLen, startPos);
+            if (spanDone) Q4eSpanTicks += Stopwatch.GetTimestamp() - tSpan;
+            if (!spanDone && LayerSplitDegree > 1)
+            {
+                // The op-by-op fallback allocates from _allocator (rank 0) and reads
+                // the HOST recurrent-state arrays. Under a layer split most layers'
+                // weights, KV device copies and GDN/PLE state live on another GPU, so
+                // running it would silently compute against the wrong device's state -
+                // the same class of failure as the mid-sequence QSA fallback that used
+                // to collapse generation. Refuse instead.
+                throw new NotSupportedException(
+                    "qwen4exp: the token-span path declined while a layer split is active. "
+                    + "The per-layer fallback is single-GPU only, so it cannot run here. "
+                    + "Re-run without --tp to use the fallback.");
+            }
+            if (!spanDone && _pendingMRoPEPositions != null)
+            {
+                // The fallback paths rotate with scalar positions only; running an
+                // image prompt through them would be silently wrong.
+                throw new NotSupportedException(
+                    "qwen4exp image prompts need the token-span path "
+                    + "(TS_Q4E_TOKEN_GRAPH=0 and image inputs cannot be combined).");
+            }
+
+            if (!spanDone && _ffnArgs != null && !_fusedFfnUnsupported)
+                ResidualToDevice(res, seqLen);
+
+            for (int il = spanDone ? Config.NumLayers : 0; il < Config.NumLayers; il++)
+            {
+                long t0 = Stopwatch.GetTimestamp();
+                if (_isPle[il])
+                {
+                    ResidualToHost(res, seqLen);
+                    PleLayer(res, tokens, seqLen, startPos, il);
+                    ResidualToDevice(res, seqLen);
+                }
+                long t1 = Stopwatch.GetTimestamp(); Q4ePleTicks += t1 - t0;
+
+                // The recurrent half - mixer, projections, causal conv, the delta-net
+                // recurrence and the scatter - is one graph when the kernel takes the
+                // shape. That is ~9 GGML submissions collapsed into 1 on 36 of the 48
+                // layers, on a path that is dispatch bound.
+                // Verify once for a single-token step and once for a multi-token one:
+                // the conv history and the multi-token recurrence only exercise the
+                // second, and a kernel can be right for one and wrong for the other.
+                if (_isRecurrent[il] && _gdnVerify
+                    && ((seqLen == 1 && !_gdnVerified1) || (seqLen > 1 && !_gdnVerifiedN)))
+                {
+                    VerifyGdnBlock(res, il, seqLen);
+                    if (seqLen == 1) _gdnVerified1 = true; else _gdnVerifiedN = true;
+                }
+
+                if (_isRecurrent[il] && TryFusedGdnBlock(res, il, seqLen))
+                {
+                    long tg = Stopwatch.GetTimestamp(); Q4eGdnTicks += tg - t1;
+                    if (TryFusedFfnBlock(res, il, seqLen))
+                    {
+                        Q4eMoeTicks += Stopwatch.GetTimestamp() - tg;
+                        continue;
+                    }
+                    // FFN fell back; run the op-by-op FFN half below with a fresh mixer.
+                    ResidualToHost(res, seqLen);
+                    Tensor injF;
+                    Tensor curF = HcMix(res, seqLen,
+                        $"blk.{il}.hc_ffn_norm.weight", $"blk.{il}.hc_ffn_down.weight",
+                        $"blk.{il}.hc_ffn_up.weight", $"blk.{il}.hc_ffn_inject.weight", out injF);
+                    Tensor ffnF = MoeFfn(curF, il, seqLen);
+                    curF.Dispose();
+                    HcCombine(res, ffnF, injF, seqLen);
+                    ffnF.Dispose();
+                    injF.Dispose();
+                    ResidualToDevice(res, seqLen);
+                    continue;
+                }
+
+                // The full-attention half is one graph too when the kernel takes the
+                // shape: mixer, query|gate, norms, rotary, KV append, gated attention
+                // and the scatter.
+                if (!_isRecurrent[il] && TryFusedAttnBlock(res, il, seqLen, startPos))
+                {
+                    Q4eAttnTicks += Stopwatch.GetTimestamp() - t1;
+                    if (TryFusedFfnBlock(res, il, seqLen))
+                    {
+                        Q4eMoeTicks += Stopwatch.GetTimestamp() - t1;
+                        continue;
+                    }
+                    ResidualToHost(res, seqLen);
+                    Tensor injA;
+                    Tensor curA = HcMix(res, seqLen,
+                        $"blk.{il}.hc_ffn_norm.weight", $"blk.{il}.hc_ffn_down.weight",
+                        $"blk.{il}.hc_ffn_up.weight", $"blk.{il}.hc_ffn_inject.weight", out injA);
+                    Tensor ffnA = MoeFfn(curA, il, seqLen);
+                    curA.Dispose();
+                    HcCombine(res, ffnA, injA, seqLen);
+                    ffnA.Dispose();
+                    injA.Dispose();
+                    ResidualToDevice(res, seqLen);
+                    continue;
+                }
+
+                ResidualToHost(res, seqLen);
+                Tensor inject;
+                Tensor cur = HcMix(res, seqLen,
+                    $"blk.{il}.hc_attn_norm.weight", $"blk.{il}.hc_attn_down.weight",
+                    $"blk.{il}.hc_attn_up.weight", $"blk.{il}.hc_attn_inject.weight", out inject);
+                long t2 = Stopwatch.GetTimestamp(); Q4eHcTicks += t2 - t1;
+
+                Tensor blockOut = _isRecurrent[il]
+                    ? GdnLayer(cur, il, seqLen)
+                    : AttentionLayer(cur, il, seqLen, startPos);
+                cur.Dispose();
+                long t3 = Stopwatch.GetTimestamp();
+                if (_isRecurrent[il]) Q4eGdnTicks += t3 - t2; else Q4eAttnTicks += t3 - t2;
+
+                HcCombine(res, blockOut, inject, seqLen);
+                blockOut.Dispose();
+                inject.Dispose();
+                _ = t3;
+                if (_ffnArgs != null && !_fusedFfnUnsupported)
+                    ResidualToDevice(res, seqLen);
+
+                // The mixer, the experts and the scatter are one graph when the
+                // fused kernel takes the shape - 8 GGML submissions collapsed into
+                // 1, 48 times a token, on a path that is dispatch bound.
+                if (TryFusedFfnBlock(res, il, seqLen))
+                {
+                    Q4eMoeTicks += Stopwatch.GetTimestamp() - t3;
+                    continue;
+                }
+
+                ResidualToHost(res, seqLen);
+                cur = HcMix(res, seqLen,
+                    $"blk.{il}.hc_ffn_norm.weight", $"blk.{il}.hc_ffn_down.weight",
+                    $"blk.{il}.hc_ffn_up.weight", $"blk.{il}.hc_ffn_inject.weight", out inject);
+                long t4 = Stopwatch.GetTimestamp(); Q4eHcTicks += t4 - t3;
+
+                Tensor ffnOut = MoeFfn(cur, il, seqLen);
+                cur.Dispose();
+                long t5 = Stopwatch.GetTimestamp(); Q4eMoeTicks += t5 - t4;
+
+                HcCombine(res, ffnOut, inject, seqLen);
+                ffnOut.Dispose();
+                inject.Dispose();
+                Q4eHcTicks += Stopwatch.GetTimestamp() - t5;
+            }
+
+            ResidualToHost(res, seqLen);
+
+            if (_pendingMRoPEPositions != null)
+                UpdateMropeGap(startPos, seqLen);
+            _pendingMRoPEPositions = null;
+
+            if (_spanLogitsValid)
+            {
+                // The last span already produced the logits; the residual is dead.
+                res.Dispose();
+                _logitsBuffer = _spanLogits;
+                _cacheSeqLen += seqLen;
+                _forwardCount++;
+                _forwardSw.Stop();
+                ReportQ4eProfile(seqLen);
+                return _logitsBuffer;
+            }
+
+            // The final mixer IS the output norm - qwen4exp ships no separate one.
+            Tensor normed = HcMix(res, seqLen,
+                "output_hc_norm.weight", "output_hc_down.weight", "output_hc_up.weight",
+                null, out _);
+            res.Dispose();
+
+            Tensor lastHidden;
+            if (seqLen > 1)
+            {
+                using var narrowed = normed.Narrow(0, seqLen - 1, 1);
+                lastHidden = Ops.NewContiguous(narrowed);
+            }
+            else
+            {
+                lastHidden = normed.CopyRef();
+            }
+            normed.Dispose();
+
+            long tHead = Stopwatch.GetTimestamp();
+            Tensor logits = LinearForward(lastHidden, "output.weight")
+                            ?? LinearForward(lastHidden, "token_embd.weight");
+            _lmHeadTicks += Stopwatch.GetTimestamp() - tHead;
+            Q4eHeadTicks += Stopwatch.GetTimestamp() - tHead;
+            lastHidden.Dispose();
+
+            _logitsBuffer = TensorToFloatArray(logits);
+            logits.Dispose();
+
+            _cacheSeqLen += seqLen;
+            _forwardCount++;
+            _forwardSw.Stop();
+            ReportQ4eProfile(seqLen);
+            return _logitsBuffer;
+        }
+
+        /// <summary>[T, n_embd] -> [T, hc*n_embd], every stream a copy.</summary>
+        private unsafe Tensor BroadcastToStreams(Tensor x, int seqLen)
+        {
+            int n = Config.HiddenSize;
+            var res = new Tensor(_allocator, DType.Float32, seqLen, _hcDim);
+            long srcA = (long)GetFloatPtr(x);
+            long dstA = (long)GetFloatPtr(res);
+            int hc = _hc, hcDim = _hcDim;
+            void CopyToken(int t)
+            {
+                float* s = (float*)srcA + (long)t * n;
+                float* d = (float*)dstA + (long)t * hcDim;
+                for (int c = 0; c < hc; c++)
+                    Buffer.MemoryCopy(s, d + (long)c * n, n * 4L, n * 4L);
+            }
+            if (seqLen >= 16) System.Threading.Tasks.Parallel.For(0, seqLen, CopyToken);
+            else for (int t = 0; t < seqLen; t++) CopyToken(t);
+            InvalidateTensorDeviceCache(res);
+            return res;
+        }
+
+        /// <summary>
+        /// Read the wide residual: grouped RMS norm over each stream, a low-rank
+        /// sigmoid gate, then collapse the streams by their mean. Also produces the
+        /// per-stream scatter weights the matching <see cref="HcCombine"/> needs.
+        /// </summary>
+        private unsafe Tensor HcMix(Tensor res, int seqLen,
+            string normName, string downName, string upName, string injectName, out Tensor inject)
+        {
+            int n = Config.HiddenSize;
+
+            // xn = groupRmsNorm(res) * w_norm. The converter folded the gammas to
+            // (1 + w), so this is a plain multiply rather than an affine.
+            var xn = new Tensor(_allocator, DType.Float32, seqLen, _hcDim);
+            {
+                float* src = GetFloatPtr(res);
+                float* dst = GetFloatPtr(xn);
+                float* w = GetFloatPtr(_weights[normName]);
+                float eps = Config.Eps;
+                for (int t = 0; t < seqLen; t++)
+                {
+                    float* r = src + (long)t * _hcDim;
+                    float* o = dst + (long)t * _hcDim;
+                    for (int c = 0; c < _hc; c++)
+                    {
+                        float* rc = r + (long)c * n;
+                        float* oc = o + (long)c * n;
+                        double ss = 0;
+                        for (int i = 0; i < n; i++) ss += (double)rc[i] * rc[i];
+                        float inv = (float)(1.0 / Math.Sqrt(ss / n + eps));
+                        int b = c * n;
+                        for (int i = 0; i < n; i++) oc[i] = rc[i] * inv * w[b + i];
+                    }
+                }
+                InvalidateTensorDeviceCache(xn);
+            }
+
+            // gate = sigmoid(W_up @ silu(W_down @ xn / hc))
+            //
+            // The scale+SiLU on [T, 320] and the sigmoid on [T, 10240] are three more
+            // GGML dispatches for a few thousand elements. At 96 mixers per token that
+            // was 288 dispatches of pure launch overhead, on a path that is already
+            // dispatch-bound, so they run on the host instead.
+            Tensor lo = LinearForward(xn, downName);                 // [T, hcLowRank]
+            if (seqLen > HostElementwiseMaxRows)
+            {
+                Ops.Mul(lo, lo, 1.0f / _hc);
+                Ops.SiLU(lo, lo);
+            }
+            else
+            {
+                float* p = GetFloatPtr(lo);
+                long count = (long)seqLen * _hcLowRank;
+                float inv = 1.0f / _hc;
+                for (long i = 0; i < count; i++)
+                {
+                    float x = p[i] * inv;
+                    p[i] = x / (1.0f + MathF.Exp(-x));               // silu
+                }
+                InvalidateTensorDeviceCache(lo);
+            }
+            Tensor gate = LinearForward(lo, upName);                 // [T, hcDim]
+            lo.Dispose();
+            if (seqLen > HostElementwiseMaxRows)
+            {
+                Ops.Sigmoid(gate, gate);
+            }
+            else
+            {
+                float* p = GetFloatPtr(gate);
+                long count = (long)seqLen * _hcDim;
+                for (long i = 0; i < count; i++)
+                    p[i] = 1.0f / (1.0f + MathF.Exp(-p[i]));
+                InvalidateTensorDeviceCache(gate);
+            }
+
+            // mixed = mean over the streams of (xn * gate)
+            var mixed = new Tensor(_allocator, DType.Float32, seqLen, n);
+            {
+                float* xp = GetFloatPtr(xn);
+                float* gp = GetFloatPtr(gate);
+                float* mp = GetFloatPtr(mixed);
+                float invHc = 1.0f / _hc;
+                for (int t = 0; t < seqLen; t++)
+                {
+                    float* x = xp + (long)t * _hcDim;
+                    float* g = gp + (long)t * _hcDim;
+                    float* m = mp + (long)t * n;
+                    for (int i = 0; i < n; i++) m[i] = x[i] * g[i];
+                    for (int c = 1; c < _hc; c++)
+                    {
+                        int b = c * n;
+                        for (int i = 0; i < n; i++) m[i] += x[b + i] * g[b + i];
+                    }
+                    for (int i = 0; i < n; i++) m[i] *= invHc;
+                }
+                InvalidateTensorDeviceCache(mixed);
+            }
+            gate.Dispose();
+
+            inject = injectName != null ? LinearForward(xn, injectName) : null;  // [T, hc]
+            xn.Dispose();
+            return mixed;
+        }
+
+        /// <summary>
+        /// Write a block's output back into every residual stream, each scaled by its
+        /// own gate. 2*sigmoid centres the weights on 1, so an untrained injection
+        /// reproduces a plain residual add.
+        /// </summary>
+        private unsafe void HcCombine(Tensor res, Tensor blockOut, Tensor inject, int seqLen)
+        {
+            int n = Config.HiddenSize;
+            float* rp = GetFloatPtr(res);
+            float* bp = GetFloatPtr(blockOut);
+            float* ip = GetFloatPtr(inject);
+            float invHc = 1.0f / _hc;
+            for (int t = 0; t < seqLen; t++)
+            {
+                float* r = rp + (long)t * _hcDim;
+                float* b = bp + (long)t * n;
+                float* inj = ip + (long)t * _hc;
+                for (int c = 0; c < _hc; c++)
+                {
+                    float w = 2.0f / (1.0f + MathF.Exp(-inj[c] * invHc));
+                    float* rc = r + (long)c * n;
+                    for (int i = 0; i < n; i++) rc[i] += b[i] * w;
+                }
+            }
+            InvalidateTensorDeviceCache(res);
+        }
+
+        /// <summary>
+        /// 512-expert MoE: softmax over all experts, top-k, renormalise, plus a shared
+        /// expert behind its own scalar sigmoid gate.
+        /// </summary>
+        private unsafe Tensor MoeFfn(Tensor input, int il, int seqLen)
+        {
+            int n = Config.HiddenSize;
+            if (_fusedGateUpExperts)
+            {
+                throw new NotSupportedException(
+                    "This qwen4exp GGUF stacks gate and up into blk.N.ffn_gate_up_exps; "
+                    + "only the separate ffn_gate_exps / ffn_up_exps layout is wired up so far.");
+            }
+            var gateStack = _stackedExpertWeights[$"blk.{il}.ffn_gate_exps.weight"];
+            var upStack = _stackedExpertWeights[$"blk.{il}.ffn_up_exps.weight"];
+            var downStack = _stackedExpertWeights[$"blk.{il}.ffn_down_exps.weight"];
+
+            Tensor routerLogits = LinearForward(input, $"blk.{il}.ffn_gate_inp.weight"); // [T, nExperts]
+
+            // Host top-k routing for every token at once: softmax over all 512 experts,
+            // keep the best n_expert_used, renormalise (llama.cpp's norm_w).
+            int K = _numExpertsUsed;
+            if (_moeSelExperts == null || _moeSelExperts.Length < (long)seqLen * K)
+            {
+                _moeSelExperts = new int[(long)seqLen * K];
+                _moeRouteWts = new float[(long)seqLen * K];
+            }
+            var probs = _moeProbs ??= new float[_numExperts];
+
+            float* rl = GetFloatPtr(routerLogits);
+            for (int t = 0; t < seqLen; t++)
+            {
+                float* row = rl + (long)t * _numExperts;
+                float max = float.NegativeInfinity;
+                for (int e = 0; e < _numExperts; e++) if (row[e] > max) max = row[e];
+                float sum = 0f;
+                for (int e = 0; e < _numExperts; e++) { probs[e] = MathF.Exp(row[e] - max); sum += probs[e]; }
+                float invSum = 1.0f / sum;
+                for (int e = 0; e < _numExperts; e++) probs[e] *= invSum;
+
+                float wSum = 0f;
+                for (int k = 0; k < K; k++)
+                {
+                    int best = -1; float bestV = float.NegativeInfinity;
+                    for (int e = 0; e < _numExperts; e++)
+                        if (probs[e] > bestV) { bestV = probs[e]; best = e; }
+                    probs[best] = float.NegativeInfinity;
+                    _moeSelExperts[t * K + k] = best;
+                    _moeRouteWts[t * K + k] = bestV;
+                    wSum += bestV;
+                }
+                float invW = 1.0f / wSum;
+                for (int k = 0; k < K; k++) _moeRouteWts[t * K + k] *= invW;
+            }
+            routerLogits.Dispose();
+
+            var result = new Tensor(_allocator, DType.Float32, seqLen, n);
+            int nFf = (int)gateStack.PerExpertNe1;
+
+            if (IsGgmlBackend)
+            {
+                // ONE ggml_mul_mat_id over the whole batch. The per-token call this
+                // replaced issued seqLen * n_layers native graph builds - 49k of them
+                // for a 1024-token prefill - and was the dominant prefill cost once the
+                // recurrence moved to the GPU.
+                GgmlBasicOps.MoEFFNPrefill(
+                    input, result, seqLen, n, nFf,
+                    gateStack.NumExperts, K, _moeSelExperts, _moeRouteWts,
+                    gateStack.Data, gateStack.GgmlType, gateStack.PerExpertNe0, gateStack.PerExpertNe1, gateStack.TotalRawBytes,
+                    upStack.Data, upStack.GgmlType, upStack.PerExpertNe0, upStack.PerExpertNe1, upStack.TotalRawBytes,
+                    downStack.Data, downStack.GgmlType, downStack.PerExpertNe0, downStack.PerExpertNe1, downStack.TotalRawBytes,
+                    gateBias: null, upBias: null, downBias: null,
+                    activation: GgmlBasicOps.MoEActivation.SwiGLUSplit);
+            }
+            else
+            {
+                // Portable path for the direct-CUDA and pure-C# backends, which have no
+                // stacked-expert kernel: one expert at a time through the ordinary
+                // quantized linear, accumulated by route weight.
+                var ids = new int[K];
+                var wts = new float[K];
+                for (int t = 0; t < seqLen; t++)
+                {
+                    for (int k = 0; k < K; k++)
+                    {
+                        ids[k] = _moeSelExperts[t * K + k];
+                        wts[k] = _moeRouteWts[t * K + k];
+                    }
+                    using var tokenIn = input.Narrow(0, t, 1);
+                    using var tokenOut = result.Narrow(0, t, 1);
+                    MoeExpertsPortable(tokenOut, tokenIn, il, ids, wts);
+                }
+            }
+            InvalidateTensorDeviceCache(result);
+
+            // Shared expert, batched over all tokens, behind its own sigmoid gate.
+            Tensor sg = LinearForward(input, $"blk.{il}.ffn_gate_shexp.weight");
+            Tensor su = LinearForward(input, $"blk.{il}.ffn_up_shexp.weight");
+            if (seqLen > HostElementwiseMaxRows)
+            {
+                Ops.SiLUMul(sg, sg, su);
+            }
+            else
+            {
+                // silu(gate) * up on [T, 640] - host, for the same reason as the mixer.
+                float* gp = GetFloatPtr(sg);
+                float* up = GetFloatPtr(su);
+                long count = (long)seqLen * sg.Sizes[1];
+                for (long i = 0; i < count; i++)
+                    gp[i] = (gp[i] / (1.0f + MathF.Exp(-gp[i]))) * up[i];
+                InvalidateTensorDeviceCache(sg);
+            }
+            su.Dispose();
+            Tensor sd = LinearForward(sg, $"blk.{il}.ffn_down_shexp.weight");
+            sg.Dispose();
+
+            // The gate is ONE scalar per token and its weight is a bare [n_embd] vector
+            // rather than a matrix - LinearForward would read that as a 2560-wide
+            // projection - so the dot product is explicit.
+            Tensor gateVec = _weights[$"blk.{il}.ffn_gate_inp_shexp.weight"];
+            {
+                float* gv = GetFloatPtr(gateVec);
+                float* xp = GetFloatPtr(input);
+                float* sp = GetFloatPtr(sd);
+                float* rp = GetFloatPtr(result);
+                for (int t = 0; t < seqLen; t++)
+                {
+                    float* x = xp + (long)t * n;
+                    double dot = 0;
+                    for (int i = 0; i < n; i++) dot += (double)x[i] * gv[i];
+                    float g = 1.0f / (1.0f + MathF.Exp(-(float)dot));
+                    float* sv = sp + (long)t * n;
+                    float* r = rp + (long)t * n;
+                    for (int i = 0; i < n; i++) r[i] += sv[i] * g;
+                }
+                InvalidateTensorDeviceCache(result);
+            }
+            sd.Dispose();
+            return result;
+        }
+
+        /// <summary>
+        /// Above this many rows the small elementwise steps go back to GGML.
+        ///
+        /// A decode step is dispatch-bound - ~850 GGML launches per token, each costing
+        /// far more than the arithmetic - so running a 320- or 10240-element sigmoid on
+        /// the host removes three launches per mixer, 288 per token. A 1024-row prefill
+        /// is the opposite: the same code becomes 10.5 M host exp() calls per mixer and
+        /// cost 2.7 s of prefill when it was applied unconditionally.
+        /// </summary>
+        private const int HostElementwiseMaxRows = 8;
+
+        // TS_Q4E_GDN_VERIFY=1 runs the fused recurrent half and the op-by-op one on
+        // the SAME input and state, and reports how far apart they are. Diagnostic
+        // only: it runs once, on the first recurrent layer of the first forward,
+        // where the state is still zero so both paths start from the same place.
+        private static readonly bool _gdnVerify =
+            string.Equals(Environment.GetEnvironmentVariable("TS_Q4E_GDN_VERIFY"), "1", StringComparison.Ordinal);
+        private bool _gdnVerified1, _gdnVerifiedN;
+
+        private unsafe void VerifyGdnBlock(Tensor res, int il, int seqLen)
+        {
+            long n = (long)seqLen * _hcDim;
+            var input = new float[n];
+            var fused = new float[n];
+            float* rp = GetFloatPtr(res);
+            for (long i = 0; i < n; i++) input[i] = rp[i];
+
+            void ZeroState()
+            {
+                EnsureGdnScratch();
+                if (_gdnStateT?[il] != null) { Ops.Fill(_gdnStateT[il], 0f); InvalidateTensorDeviceCache(_gdnStateT[il]); }
+                if (_gdnConvStateT?[il] != null) { Ops.Fill(_gdnConvStateT[il], 0f); InvalidateTensorDeviceCache(_gdnConvStateT[il]); }
+                if (_gdnConvState?[il] != null) Array.Clear(_gdnConvState[il]);
+                if (_gdnConvWriteIdx != null) _gdnConvWriteIdx[il] = 0;
+            }
+
+            // TWO consecutive calls: the second one is the only thing that exercises
+            // the state written back by the first, which a single call cannot check.
+            const int Iters = 2;
+
+            // ---- fused ----
+            ZeroState();
+            for (int it = 0; it < Iters; it++)
+            {
+                for (long i = 0; i < n; i++) rp[i] = input[i];
+                InvalidateTensorDeviceCache(res);
+                if (!TryFusedGdnBlock(res, il, seqLen))
+                { Console.WriteLine("[q4e-verify] fused GDN declined the shape."); return; }
+            }
+            for (long i = 0; i < n; i++) fused[i] = rp[i];
+
+            // ---- op-by-op, from the same input and state ----
+            ZeroState();
+            for (int it = 0; it < Iters; it++)
+            {
+                for (long i = 0; i < n; i++) rp[i] = input[i];
+                InvalidateTensorDeviceCache(res);
+
+                Tensor inj;
+                Tensor cur = HcMix(res, seqLen,
+                    $"blk.{il}.hc_attn_norm.weight", $"blk.{il}.hc_attn_down.weight",
+                    $"blk.{il}.hc_attn_up.weight", $"blk.{il}.hc_attn_inject.weight", out inj);
+                Tensor blockOut = GdnLayer(cur, il, seqLen);
+                cur.Dispose();
+                HcCombine(res, blockOut, inj, seqLen);
+                blockOut.Dispose();
+                inj.Dispose();
+            }
+
+            // Compare the block's CONTRIBUTION (res_after - res_before), not the
+            // residual: the residual is dominated by the pass-through, so a large
+            // error in what the block adds hides inside it.
+            double maxAbs = 0, refMax = 0, contribMax = 0, contribErr = 0;
+            long worst = -1;
+            for (long i = 0; i < n; i++)
+            {
+                double refC = rp[i] - input[i];
+                double fusC = fused[i] - input[i];
+                double d = Math.Abs(fusC - refC);
+                if (d > contribErr) { contribErr = d; worst = i; }
+                contribMax = Math.Max(contribMax, Math.Abs(refC));
+                maxAbs = Math.Max(maxAbs, Math.Abs(fused[i] - rp[i]));
+                refMax = Math.Max(refMax, Math.Abs(rp[i]));
+            }
+            Console.WriteLine($"[q4e-verify] layer {il} seqLen {seqLen}: " +
+                $"residual maxAbs={maxAbs:E3} rel={(refMax > 0 ? maxAbs / refMax : 0):E3} | " +
+                $"CONTRIB maxAbs={contribErr:E3} rel={(contribMax > 0 ? contribErr / contribMax : 0):E3} " +
+                $"contribMax={contribMax:E3}" +
+                (worst >= 0 ? $" | fusedC={fused[worst] - input[worst]:E3} refC={rp[worst] - input[worst]:E3}" : ""));
+
+            // Leave the state as the op-by-op path left it: that one is the reference.
+            ZeroState();
+            for (long i = 0; i < n; i++) rp[i] = input[i];
+            InvalidateTensorDeviceCache(res);
+        }
+
+        // TS_Q4E_FUSED_ATTN=0 falls back to the op-by-op attention half.
+        private static readonly bool _fusedAttnEnabled =
+            !string.Equals(Environment.GetEnvironmentVariable("TS_Q4E_FUSED_ATTN"), "0", StringComparison.Ordinal);
+        private bool _fusedAttnUnsupported;
+        private Qwen4ExpAttnArgs[] _attnArgs;
+        private ushort[] _attnMask;
+
+        // Must mirror kQwen4ExpKvStride and q4e_flash_attn_ok in ggml_ops_qwen4exp.cpp.
+        private const int KvStride = 256;
+        private static readonly bool FlashAttnEnabled =
+            !string.Equals(Environment.GetEnvironmentVariable("TS_Q4E_FLASH_ATTN"), "0", StringComparison.Ordinal);
+
+        private static bool UseFlashAttn(DType kvType, int headDim)
+        {
+            if (!FlashAttnEnabled || kvType != DType.Float16) return false;
+            return headDim is 64 or 80 or 96 or 112 or 128 or 256;
+        }
+
+        private unsafe bool TryFusedAttnBlock(Tensor res, int il, int seqLen, int startPos)
+        {
+            if (!_fusedAttnEnabled || _fusedAttnUnsupported || !IsGgmlBackend)
+                return false;
+            // No QSA-budget guard here either - it computes the same dense
+            // attention the fallback would, and declining mid-sequence loses the
+            // device-resident recurrent state. See TryFusedTokenSpans.
+            int totalLen = startPos + seqLen;
+
+            try
+            {
+                if (!EnsureAttnArgs()) return false;
+                BuildAttnMask(totalLen, seqLen, startPos, _kCache[il].ElementType);
+
+                bool ok;
+                fixed (ushort* maskPtr = _attnMask)
+                {
+                    ok = GgmlBasicOps.Qwen4ExpAttnBlock(ref _attnArgs[il],
+                        (IntPtr)GetFloatPtr(res), (IntPtr)maskPtr,
+                        Config.HiddenSize, _hc, _hcLowRank, seqLen,
+                        Config.HeadDim, Config.NumHeads, Config.NumKVHeads,
+                        _kvCacheCapacity, totalLen, startPos,
+                        _ropeDimCount, Config.RopeBase, 1.0f / Config.RopeScale, _attnScale,
+                        Config.Eps, cacheSlot: il, resResident: _resOnDevice);
+                }
+                if (!ok) { _fusedAttnUnsupported = true; return false; }
+                if (!_resOnDevice) InvalidateTensorDeviceCache(res);
+                _kvCacheHostStale = true;
+                return true;
+            }
+            catch (Exception)
+            {
+                _fusedAttnUnsupported = true;
+                return false;
+            }
+        }
+
+        // The fused attention kernel writes the KV cache on the device; the host
+        // mirror is behind until something syncs it.
+        private bool _kvCacheHostStale;
+
+        private unsafe bool TryFillAttnArgs(int il, ref Qwen4ExpAttnArgs a)
+        {
+            if (!TryResolveQuant($"blk.{il}.hc_attn_down.weight", out IntPtr hd, out int hdT, out long hdB)
+                || !TryResolveQuant($"blk.{il}.hc_attn_up.weight", out IntPtr hu, out int huT, out long huB)
+                || !TryResolveQuant($"blk.{il}.hc_attn_inject.weight", out IntPtr hj, out int hjT, out long hjB)
+                || !TryResolveQuant($"blk.{il}.attn_q.weight", out IntPtr wq, out int wqT, out long wqB)
+                || !TryResolveQuant($"blk.{il}.attn_k.weight", out IntPtr wk, out int wkT, out long wkB)
+                || !TryResolveQuant($"blk.{il}.attn_v.weight", out IntPtr wv, out int wvT, out long wvB)
+                || !TryResolveQuant($"blk.{il}.attn_output.weight", out IntPtr wo, out int woT, out long woB))
+            {
+                return false;
+            }
+            if (_kCache[il] == null || _vCache[il] == null)
+                return false;
+
+            a.HcNorm = (IntPtr)GetFloatPtr(_weights[$"blk.{il}.hc_attn_norm.weight"]);
+            a.HcDown = hd; a.HcDownType = hdT; a.HcDownBytes = hdB;
+            a.HcUp = hu; a.HcUpType = huT; a.HcUpBytes = huB;
+            a.HcInject = hj; a.HcInjectType = hjT; a.HcInjectBytes = hjB;
+            a.Wq = wq; a.WqType = wqT; a.WqBytes = wqB;
+            a.Wk = wk; a.WkType = wkT; a.WkBytes = wkB;
+            a.Wv = wv; a.WvType = wvT; a.WvBytes = wvB;
+            a.Wo = wo; a.WoType = woT; a.WoBytes = woB;
+            a.QNorm = (IntPtr)GetFloatPtr(_weights[$"blk.{il}.attn_q_norm.weight"]);
+            a.KNorm = (IntPtr)GetFloatPtr(_weights[$"blk.{il}.attn_k_norm.weight"]);
+            a.KCache = GetStorageBasePtrOf(_kCache[il]);
+            a.VCache = GetStorageBasePtrOf(_vCache[il]);
+            a.KvType = FusedGraphKvTypeId(_kCache[il].ElementType);
+            a.KvBytes = _kCache[il].ElementCount() * KvElementSize(_kCache[il].ElementType);
+            return a.KCache != IntPtr.Zero && a.VCache != IntPtr.Zero && a.KvType >= 0;
+        }
+
+        private static int FusedGraphKvTypeId(DType t) => t switch
+        {
+            DType.Float32 => 0,     // GGML_TYPE_F32
+            DType.Float16 => 1,     // GGML_TYPE_F16
+            _ => -1,
+        };
+
+        private static long KvElementSize(DType t) => t == DType.Float32 ? 4 : 2;
+
+        /// <summary>The KV cache's backing device pointer, which is what the fused
+        /// kernel binds so both paths write the same copy.</summary>
+        private static IntPtr GetStorageBasePtrOf(Tensor t)
+            => TensorComputePrimitives.GetStorageBasePointer(t);
+
+        // ------------------------------------------------------------------
+        // The whole token as (almost) one graph. TS_Q4E_TOKEN_GRAPH=0 falls back to
+        // the per-layer fused kernels; those in turn fall back op-by-op.
+        // ------------------------------------------------------------------
+        // ON by default: a decode token is two graph launches. The long hunt that
+        // once kept this off ended at a single root cause - gallocr frees an
+        // uploaded leaf weight after its last consumer unless it carries the OUTPUT
+        // flag, so the first compute overwrote the small weights (dt/a/ssm-norm,
+        // the attention q/k norms) and every replay read decayed garbage. With the
+        // flag in place every configuration that used to degenerate verifies clean.
+        private static readonly bool _tokenGraphEnabled =
+            !string.Equals(Environment.GetEnvironmentVariable("TS_Q4E_TOKEN_GRAPH"), "0", StringComparison.Ordinal);
+        // TS_Q4E_DRIVER_TRACE=1 prints the residual L2 after every span and
+        // attention call the driver makes. Diagnosis only.
+        private static readonly bool _driverTrace =
+            string.Equals(Environment.GetEnvironmentVariable("TS_Q4E_DRIVER_TRACE"), "1", StringComparison.Ordinal);
+
+        // TS_Q4E_PHASE=1 prints host-side wall times for a prefill forward.
+        private static readonly bool _phaseLog =
+            string.Equals(Environment.GetEnvironmentVariable("TS_Q4E_PHASE"), "1", StringComparison.Ordinal);
+
+        private static void PhaseLog(int seqLen, string what, long t0)
+        {
+            if (!_phaseLog || seqLen <= 1) return;
+            double ms = (Stopwatch.GetTimestamp() - t0) * 1000.0 / Stopwatch.Frequency;
+            Console.Error.WriteLine($"[q4e-cs] T={seqLen} {what}={ms:F1}ms");
+        }
+
+        private unsafe void DriverTrace(Tensor res, int seqLen, string what)
+        {
+            if (!_driverTrace) return;
+            float* rp = GetFloatPtr(res);
+            long n = (long)seqLen * _hcDim;
+            double n2 = 0;
+            for (long i = 0; i < n; i++) n2 += (double)rp[i] * rp[i];
+            Console.Error.WriteLine($"[q4e-drv] {what} l2={Math.Sqrt(n2):E9}");
+        }
+
+        // TS_Q4E_SPAN_ATTN=0 cuts the spans at attention layers and runs those
+        // halves through the per-layer kernel - the hybrid that served while
+        // attention-in-span was misdiagnosed as a numeric amplifier. The actual
+        // culprit was the q/k norm weights (1KB gallocr leafs) being freed and
+        // overwritten; with them protected, attention chains into the span.
+        private static readonly bool _spanAttnEnabled =
+            !string.Equals(Environment.GetEnvironmentVariable("TS_Q4E_SPAN_ATTN"), "0", StringComparison.Ordinal);
+        private bool _tokenGraphUnsupported;
+        private byte[] _layerKinds;
+        // The final mixer + LM head riding the last span. _spanLogits holds the
+        // downloaded [vocab] row when the head was fused this forward.
+        // The PLE block inside the span: only the hash and the table gather stay on
+        // the host; the gathered rows ride in as a graph input.
+        private Qwen4ExpPleArgs[] _pleArgs;
+        private bool _pleArgsFailed;
+        private float[] _pleConvWT;
+        private float[] _pleEmbBuf;
+        private int _pleLayerIndex = -1;
+
+        private unsafe bool EnsurePleArgs()
+        {
+            if (_pleArgs != null) return true;
+            if (_pleArgsFailed) return false;
+            int il = -1;
+            for (int l = 0; l < Config.NumLayers; l++) if (_isPle[l]) { il = l; break; }
+            if (il < 0) { _pleArgsFailed = true; return false; }
+            if (!TryResolveQuant($"blk.{il}.ple_key.weight", out IntPtr kw, out int kwT, out long kwB)
+                || !TryResolveQuant($"blk.{il}.ple_value.weight", out IntPtr vw, out int vwT, out long vwB)
+                || !_weights.ContainsKey($"blk.{il}.ple_norm_key.weight")
+                || !_weights.ContainsKey($"blk.{il}.ple_norm_query.weight")
+                || !_weights.ContainsKey($"blk.{il}.ple_norm_conv.weight")
+                || !_weights.TryGetValue($"blk.{il}.ple_conv1d.weight", out Tensor convW))
+            {
+                _pleArgsFailed = true;
+                return false;
+            }
+
+            int kern = _pleConvKernel;
+            // ple_conv1d is [channels, kern] with the taps fastest; the graph wants a
+            // contiguous per-channel column per tap, so transpose once. PINNED: the
+            // kernel binds this address for the graph's lifetime.
+            _pleConvWT = GC.AllocateArray<float>(_hcDim * kern, pinned: true);
+            float* wp = GetFloatPtr(convW);
+            for (int c = 0; c < _hcDim; c++)
+                for (int kk = 0; kk < kern; kk++)
+                    _pleConvWT[(long)kk * _hcDim + c] = wp[(long)c * kern + kk];
+
+            var args = GC.AllocateArray<Qwen4ExpPleArgs>(1, pinned: true);
+            args[0].KeyW = kw; args[0].KeyType = kwT; args[0].KeyBytes = kwB;
+            args[0].ValueW = vw; args[0].ValueType = vwT; args[0].ValueBytes = vwB;
+            args[0].NormKey = (IntPtr)GetFloatPtr(_weights[$"blk.{il}.ple_norm_key.weight"]);
+            args[0].NormQuery = (IntPtr)GetFloatPtr(_weights[$"blk.{il}.ple_norm_query.weight"]);
+            args[0].NormConv = (IntPtr)GetFloatPtr(_weights[$"blk.{il}.ple_norm_conv.weight"]);
+            fixed (float* ct = _pleConvWT) args[0].Conv1dT = (IntPtr)ct;   // pinned array
+            fixed (float* cs = _pleConvState) args[0].ConvState = (IntPtr)cs; // pinned array
+            args[0].Kern = kern;
+            args[0].Dil = _pleNgram;
+            _pleLayerIndex = il;
+            _pleArgs = args;
+            return true;
+        }
+
+        private Qwen4ExpHeadArgs[] _headArgs;
+        private bool _headArgsFailed;
+        private float[] _spanLogits;
+        private bool _spanLogitsValid;
+
+        private unsafe bool EnsureHeadArgs()
+        {
+            if (_headArgs != null) return true;
+            if (_headArgsFailed) return false;
+            if (!TryResolveQuant("output_hc_down.weight", out IntPtr hd, out int hdT, out long hdB)
+                || !TryResolveQuant("output_hc_up.weight", out IntPtr hu, out int huT, out long huB)
+                || !_weights.ContainsKey("output_hc_norm.weight")
+                || (!TryResolveQuant("output.weight", out IntPtr wh, out int whT, out long whB)
+                    && !TryResolveQuant("token_embd.weight", out wh, out whT, out whB)))
+            {
+                _headArgsFailed = true;
+                return false;
+            }
+            // PINNED: the kernel keys the span graph on the descriptor address.
+            var args = GC.AllocateArray<Qwen4ExpHeadArgs>(1, pinned: true);
+            args[0].HcNorm = (IntPtr)GetFloatPtr(_weights["output_hc_norm.weight"]);
+            args[0].HcDown = hd; args[0].HcDownType = hdT; args[0].HcDownBytes = hdB;
+            args[0].HcUp = hu; args[0].HcUpType = huT; args[0].HcUpBytes = huB;
+            args[0].Head = wh; args[0].HeadType = whT; args[0].HeadBytes = whB;
+            args[0].Vocab = Config.VocabSize;
+            _spanLogits = GC.AllocateArray<float>(Config.VocabSize, pinned: true);
+            _headArgs = args;
+            return true;
+        }
+
+        private unsafe bool TryFusedTokenSpans(Tensor res, int[] tokens, int seqLen, int startPos)
+        {
+            if (!_tokenGraphEnabled || _tokenGraphUnsupported || !IsGgmlBackend
+                || _fusedGateUpExperts
+                || !_fusedFfnEnabled || _fusedFfnUnsupported
+                || !_fusedGdnEnabled || _fusedGdnUnsupported
+                || !_fusedAttnEnabled || _fusedAttnUnsupported
+                || _gdnMaxLayers >= 0 || _gdnVerify)
+            {
+                return false;
+            }
+
+
+            // NOTE: there used to be a "past the QSA budget, decline the whole
+            // token" guard here. It was worse than useless.
+            //
+            // QSA is not implemented (see AttentionLayer's doc comment): both this
+            // path and the per-layer fallback compute DENSE attention, above the
+            // budget and below it. So the guard changed no arithmetic - it existed
+            // only to route the token to the code that printed the one-shot
+            // warning, which now happens in ForwardCoreInner instead.
+            //
+            // What it DID do was switch code paths in the middle of a live
+            // sequence, and this architecture cannot survive that: the span kernel
+            // keeps the GDN conv/ssm state and the PLE n-gram conv history in
+            // DEVICE buffers written in place, while the per-layer fallback reads
+            // and writes the HOST arrays those buffers were seeded from at startup.
+            // Falling back mid-sequence therefore silently restarts every recurrent
+            // stream from its startup contents. Symptom: generation is fine for
+            // ~1940 tokens and then collapses into one repeated token forever, the
+            // moment the context crosses indexer_top_k + compress_ratio - 1.
+            int totalLen = startPos + seqLen;
+
+            try
+            {
+                if (!EnsureFfnArgs() || !EnsureGdnArgs() || !EnsureAttnArgs())
+                {
+                    _tokenGraphUnsupported = true;
+                    return false;
+                }
+                if (_layerKinds == null)
+                {
+                    // PINNED: its address rides in the span's graph key.
+                    _layerKinds = GC.AllocateArray<byte>(Config.NumLayers, pinned: true);
+                    for (int l = 0; l < Config.NumLayers; l++)
+                        _layerKinds[l] = _isRecurrent[l] ? (byte)1 : (byte)0;
+                }
+
+                int firstAttn = Array.IndexOf(_layerKinds, (byte)0);
+                long tMask = Stopwatch.GetTimestamp();
+                BuildAttnMask(totalLen, seqLen, startPos,
+                        firstAttn >= 0 ? _kCache[firstAttn].ElementType : DType.Float16);
+                PhaseLog(seqLen, "mask", tMask);
+
+                bool ranAnything = false;
+                _spanLogitsValid = false;
+                bool fuseHead = EnsureHeadArgs();
+                bool fusePle = EnsurePleArgs();
+
+                // Both of the remaining per-layer cuts run OUTSIDE a span, and only a
+                // span carries the device: the per-layer attention kernel and the host
+                // PleLayer would execute on whatever rank the thread happens to hold
+                // (always 0) while their layer's weights, KV device copy and recurrent
+                // state live on another GPU. Neither is reachable in a default run -
+                // the attention cut needs TS_Q4E_SPAN_ATTN=0 and the PLE cut needs the
+                // PLE descriptors to fail - so refuse the combination rather than add
+                // an untested placement path.
+                if (LayerSplitDegree > 1 && (!_spanAttnEnabled || !fusePle))
+                {
+                    _tokenGraphUnsupported = true;
+                    throw new NotSupportedException(
+                        "qwen4exp: a layer split needs every layer inside a token span, but "
+                        + (!_spanAttnEnabled
+                            ? "TS_Q4E_SPAN_ATTN=0 cuts attention out of the span"
+                            : "the PLE block could not be built into the span")
+                        + ". These per-layer paths are single-GPU only. Re-run without --tp, "
+                        + "or without TS_Q4E_SPAN_ATTN=0.");
+                }
+
+                // Image prompts carry a (T,H,W) IMRoPE position table; the span
+                // kernel takes it as-is and text forwards pass nothing.
+                bool useMrope = _pendingMRoPEPositions != null
+                    && _pendingMRoPEPositions.Length >= 3 * seqLen
+                    && EnsureMropeSections();
+                if (useMrope)
+                {
+                    if (_mropePosPinned == null || _mropePosPinned.Length < 3 * seqLen)
+                        _mropePosPinned = GC.AllocateArray<int>(3 * seqLen, pinned: true);
+                    Array.Copy(_pendingMRoPEPositions, _mropePosPinned, 3 * seqLen);
+                }
+                if (fusePle)
+                {
+                    // The hash mutates the n-gram history, so it runs exactly once and
+                    // in call order, same as the host path did.
+                    long tG = Stopwatch.GetTimestamp();
+                    int[] pleRows = ComputePleRows(tokens, startPos);
+                    long need = (long)seqLen * Config.HiddenSize;
+                    if (_pleEmbBuf == null || _pleEmbBuf.Length < need)
+                        _pleEmbBuf = GC.AllocateArray<float>(checked((int)need), pinned: true);
+                    fixed (float* eb = _pleEmbBuf)
+                        GatherPleRowsRaw(eb, pleRows, seqLen);
+                    PhaseLog(seqLen, "ple.gather", tG);
+                }
+                fixed (Qwen4ExpFfnArgs* fp = _ffnArgs)
+                fixed (Qwen4ExpGdnArgs* gp = _gdnArgs)
+                fixed (Qwen4ExpAttnArgs* ap = _attnArgs)
+                fixed (byte* kp = _layerKinds)
+                fixed (ushort* mp = _attnMask)
+                {
+                    int begin = 0, spanIdx = 0;
+                    bool beginFfnOnly = false;
+                    for (int il = 0; il <= Config.NumLayers; il++)
+                    {
+                        // The host-side PLE layer cuts the token into spans - unless
+                        // the PLE block rides inside the span, which is the default.
+                        // So does every attention layer unless TS_Q4E_SPAN_ATTN=1.
+                        bool attnCut = il < Config.NumLayers && !_isRecurrent[il] && !_spanAttnEnabled;
+                        // LAYER SPLIT: a span runs entirely on one GPU, so the token
+                        // is also cut wherever the owning device changes. The residual
+                        // crosses the seam through the host buffer it is already
+                        // passed in - the same hand-off a PLE cut uses - so a seam
+                        // costs one 40 KB round trip per decode token and nothing else.
+                        bool deviceCut = il < Config.NumLayers && il > begin
+                            && DeviceForLayer(il) != DeviceForLayer(begin);
+                        bool cut = il == Config.NumLayers || (_isPle[il] && !fusePle) || attnCut || deviceCut;
+                        if (!cut) continue;
+                        if (il > begin)
+                        {
+                            // The last span carries the final mixer + LM head and
+                            // hands back logits instead of the residual.
+                            bool last = il == Config.NumLayers && fuseHead;
+                            IntPtr headPtr = IntPtr.Zero, logitsPtr = IntPtr.Zero;
+                            if (last)
+                            {
+                                fixed (Qwen4ExpHeadArgs* hp = _headArgs)
+                                fixed (float* lp = _spanLogits)
+                                { headPtr = (IntPtr)hp; logitsPtr = (IntPtr)lp; }
+                            }
+                            IntPtr mropePtr = IntPtr.Zero, sectPtr = IntPtr.Zero;
+                            if (useMrope)
+                            {
+                                fixed (int* mp2 = _mropePosPinned)
+                                fixed (int* sp2 = _mropeSectionsPinned)
+                                { mropePtr = (IntPtr)mp2; sectPtr = (IntPtr)sp2; }
+                            }
+                            IntPtr plePtr = IntPtr.Zero, pleEmbPtr = IntPtr.Zero;
+                            int pleLayerArg = -1;
+                            if (fusePle && _pleLayerIndex >= begin && _pleLayerIndex < il)
+                            {
+                                fixed (Qwen4ExpPleArgs* pp = _pleArgs)
+                                fixed (float* eb = _pleEmbBuf)
+                                { plePtr = (IntPtr)pp; pleEmbPtr = (IntPtr)eb; }
+                                pleLayerArg = _pleLayerIndex;
+                            }
+                            long tSpan = Stopwatch.GetTimestamp();
+                            bool ok = GgmlBasicOps.Qwen4ExpTokenSpan(
+                                (IntPtr)fp, (IntPtr)gp, (IntPtr)ap, (IntPtr)kp,
+                                begin, il,
+                                (IntPtr)GetFloatPtr(res), (IntPtr)mp,
+                                Config.HiddenSize, _hc, _hcLowRank, seqLen,
+                                _headKDim, _headVDim, _numKHeads, _numVHeads, _convKernel,
+                                Config.HeadDim, Config.NumHeads, Config.NumKVHeads,
+                                _kvCacheCapacity, totalLen, startPos,
+                                _ropeDimCount, Config.RopeBase, 1.0f / Config.RopeScale, _attnScale,
+                                _numExperts, _numExpertsUsed, _expertFf, _sharedFf,
+                                Config.Eps, cacheSlot: spanIdx + _seqSlotBase, firstFfnOnly: beginFfnOnly,
+                                head: headPtr, logitsOut: logitsPtr,
+                                ple: plePtr, pleLayer: pleLayerArg, pleEmb: pleEmbPtr,
+                                mropePos: mropePtr, mropeSections: sectPtr,
+                                ropePosition: useMrope ? -1 : startPos - _mropeCacheGap,
+                                device: DeviceForLayer(begin));
+                            if (ok && last) _spanLogitsValid = true;
+                            if (!ok)
+                            {
+                                _tokenGraphUnsupported = true;
+                                if (ranAnything)
+                                {
+                                    // Layers [0, begin) already advanced the KV cache
+                                    // and the recurrent state; re-running them would
+                                    // apply the token twice. Fail loudly instead of
+                                    // quietly double-stepping the model; the next
+                                    // forward takes the per-layer fallback.
+                                    throw new InvalidOperationException(
+                                        "qwen4exp token span failed mid-token; the per-layer " +
+                                        "fallback takes over on the next forward.");
+                                }
+                                return false;
+                            }
+                            ranAnything = true;
+                            PhaseLog(seqLen, $"span[{begin},{il})", tSpan);
+                            DriverTrace(res, seqLen, $"span{spanIdx} [{begin},{il}) T={seqLen} pos={startPos}");
+                            spanIdx++;
+                            InvalidateTensorDeviceCache(res);
+                        }
+                        // `&& !fusePle` MUST match the cut condition above. Without it,
+                        // ANY other reason to cut at a PLE layer - a device boundary, or
+                        // attnCut - ran the HOST PleLayer even though the PLE block is
+                        // also built into the spans, so PLE executed twice and its
+                        // n-gram conv history advanced twice for one token.
+                        if (il < Config.NumLayers && _isPle[il] && !fusePle)
+                        {
+                            long tPle = Stopwatch.GetTimestamp();
+                            PleLayer(res, tokens, seqLen, startPos, il);
+                            PhaseLog(seqLen, "ple", tPle);
+                            begin = il;
+                            beginFfnOnly = false;
+                        }
+                        else if (attnCut)
+                        {
+                            // The attention HALF through the proven per-layer kernel;
+                            // the FFN half of this layer rides at the head of the
+                            // next span instead of costing its own launch.
+                            if (!TryFusedAttnBlock(res, il, seqLen, startPos))
+                            {
+                                _tokenGraphUnsupported = true;
+                                if (ranAnything)
+                                {
+                                    throw new InvalidOperationException(
+                                        "qwen4exp token span: the per-layer attention fallback " +
+                                        "failed mid-token; the next forward runs per-layer.");
+                                }
+                                return false;
+                            }
+                            ranAnything = true;
+                            DriverTrace(res, seqLen, $"attn{il} T={seqLen} pos={startPos}");
+                            begin = il;
+                            beginFfnOnly = true;
+                        }
+                        else if (deviceCut)
+                        {
+                            // Pure layer-split seam: no host work here at all. The span
+                            // just ended wrote the residual back to its host buffer and
+                            // the InvalidateTensorDeviceCache above dropped the device
+                            // copy, so the next span re-uploads it onto ITS gpu.
+                            begin = il;
+                            beginFfnOnly = false;
+                        }
+                    }
+                }
+                _kvCacheHostStale = true;
+                return true;
+            }
+            catch (InvalidOperationException)
+            {
+                throw;
+            }
+            catch (Exception)
+            {
+                _tokenGraphUnsupported = true;
+                return false;
+            }
+        }
+
+        private bool EnsureAttnArgs()
+        {
+            if (_attnArgs != null) return true;
+            // PINNED: the kernel keys its cached graph on the descriptor address.
+            var args = GC.AllocateArray<Qwen4ExpAttnArgs>(Config.NumLayers, pinned: true);
+            for (int l = 0; l < Config.NumLayers; l++)
+            {
+                if (_isRecurrent[l]) continue;
+                if (!TryFillAttnArgs(l, ref args[l])) { _fusedAttnUnsupported = true; return false; }
+            }
+            _attnArgs = args;
+            return true;
+        }
+
+        private bool EnsureGdnArgs()
+        {
+            if (_gdnArgs != null) return true;
+            // PINNED for the same reason as the FFN descriptors: the kernel keys its
+            // cached graph on the descriptor address.
+            var args = GC.AllocateArray<Qwen4ExpGdnArgs>(Config.NumLayers, pinned: true);
+            // A per-sequence holder swap may have installed pre-seeded state
+            // tensors; their HOST addresses key the native device-state entries,
+            // so they must be reused, never re-allocated.
+            _gdnConvStateT ??= new Tensor[Config.NumLayers];
+            EnsureGdnScratch();
+            for (int l = 0; l < Config.NumLayers; l++)
+            {
+                if (!_isRecurrent[l]) continue;
+                // The kernel wants the conv history as [d_conv-1, conv_dim] rather
+                // than the host ring the op-by-op path walks.
+                if (_gdnConvStateT[l] == null)
+                {
+                    _gdnConvStateT[l] = new Tensor(_allocator, DType.Float32, _convKernel - 1, _convDim);
+                    Ops.Fill(_gdnConvStateT[l], 0f);
+                }
+                if (!TryFillGdnArgs(l, ref args[l])) { _fusedGdnUnsupported = true; return false; }
+            }
+            _gdnArgs = args;
+            return true;
+        }
+
+        private bool EnsureFfnArgs()
+        {
+            if (_ffnArgs != null) return true;
+            // PINNED: the kernel keys its cached graph on the descriptor's address, so
+            // a moving array would look like a different layer every token and rebuild
+            // the graph each time - exactly the cost the cache exists to remove.
+            var args = GC.AllocateArray<Qwen4ExpFfnArgs>(Config.NumLayers, pinned: true);
+            for (int l = 0; l < Config.NumLayers; l++)
+                if (!TryFillFfnArgs(l, ref args[l])) { _fusedFfnUnsupported = true; return false; }
+            _ffnArgs = args;
+            return true;
+        }
+
+        /// <summary>Build the causal F16 mask at the width the kernel will read
+        /// ([n_kv_pad, T]) and return that padded width. Must agree with the kernel
+        /// on the padding - same predicate, same stride - because it reads exactly
+        /// n_kv_pad*T entries.</summary>
+        private unsafe int BuildAttnMask(int totalLen, int seqLen, int startPos, DType kvType)
+        {
+            int padded = totalLen;
+            if (UseFlashAttn(kvType, Config.HeadDim))
+            {
+                padded = ((totalLen + KvStride - 1) / KvStride) * KvStride;
+                if (padded > _kvCacheCapacity) padded = _kvCacheCapacity;
+            }
+            long need = (long)padded * seqLen;
+            if (_attnMask == null || _attnMask.Length < need)
+                _attnMask = new ushort[need];
+            const ushort NegInfF16 = 0xFC00;
+            // ONE fixed block around the whole fill. Pinning per row ends the pin at
+            // that statement and leaves the writes going through a pointer into an
+            // array the GC is free to move.
+            fixed (ushort* m = _attnMask)
+            {
+                for (int t = 0; t < seqLen; t++)
+                {
+                    int limit = startPos + t;
+                    ushort* row = m + (long)t * padded;
+                    for (int j = 0; j < padded; j++) row[j] = j <= limit ? (ushort)0 : NegInfF16;
+                }
+            }
+            return padded;
+        }
+
+        // TS_Q4E_FUSED_GDN=0 falls back to the op-by-op recurrent half.
+        // Keeping the residual in the kernels' device buffer across a layer, so the
+        // fused halves chain without a host round trip each call.
+        //
+        // OFF by default: it measures ~18% on decode (73.3 vs 62.0 t/s) but produces
+        // wrong output, and that does not buy a correctness risk.
+        //
+        // The earlier note here blamed the hand-off with layers still running op-by-op.
+        // That is wrong. Bisected:
+        //
+        //   fused GDN + op-by-op FFN, resident   -> correct
+        //   op-by-op GDN + fused FFN, resident   -> correct
+        //   fused GDN + fused FFN,    resident   -> garbage
+        //
+        // So each kernel's residency is right on its own; it is the two persisted
+        // graphs alternating that breaks. Forcing the residual down to host memory and
+        // back between the two halves does NOT fix it, which rules out both the shared
+        // device buffer hand-off and the residual values themselves - by then the data
+        // has made a full round trip through the host and is provably correct. Also
+        // ruled out: the ggml-cuda graph-uid stamp (TS_Q4E_GRAPH_UID=0 is byte-identical
+        // to =1 here). Something the two kernels share besides the residual is being
+        // disturbed; the shared g_q4e_res_buf binding and ggml-cuda's single captured
+        // graph slot are the two candidates left.
+        //
+        // TS_Q4E_RES_RESIDENT=1 re-enables it for debugging.
+        private static readonly bool _resResidentEnabled =
+            string.Equals(Environment.GetEnvironmentVariable("TS_Q4E_RES_RESIDENT"), "1", StringComparison.Ordinal);
+        private bool _resOnDevice;
+
+        /// <summary>Bring the residual back to the host so op-by-op code can read it.</summary>
+        private unsafe void ResidualToHost(Tensor res, int seqLen)
+        {
+            if (!_resOnDevice) return;
+            long bytes = (long)seqLen * _hcDim * sizeof(float);
+            if (GgmlBasicOps.Qwen4ExpResDownload((IntPtr)GetFloatPtr(res), bytes))
+                InvalidateTensorDeviceCache(res);
+            _resOnDevice = false;
+        }
+
+        /// <summary>Hand the residual to the device so the fused kernels can chain.</summary>
+        private unsafe bool ResidualToDevice(Tensor res, int seqLen)
+        {
+            if (_resOnDevice) return true;
+            // Never under a layer split: g_q4e_res is per-device, so a residual left
+            // resident on one GPU is invisible to the next span on another. The
+            // fallback that calls this is already refused under a split; this is the
+            // second lock on the same door.
+            if (!_resResidentEnabled || !IsGgmlBackend || LayerSplitDegree > 1) return false;
+            long bytes = (long)seqLen * _hcDim * sizeof(float);
+            _resOnDevice = GgmlBasicOps.Qwen4ExpResUpload((IntPtr)GetFloatPtr(res), bytes);
+            return _resOnDevice;
+        }
+
+        private static readonly bool _fusedGdnEnabled =
+            !string.Equals(Environment.GetEnvironmentVariable("TS_Q4E_FUSED_GDN"), "0", StringComparison.Ordinal);
+        private bool _fusedGdnUnsupported;
+        private static readonly int _gdnMaxLayers =
+            int.TryParse(Environment.GetEnvironmentVariable("TS_Q4E_GDN_MAX_LAYERS"), out int gm) ? gm : -1;
+        private Qwen4ExpGdnArgs[] _gdnArgs;
+        private Tensor[] _gdnConvStateT;
+
+        private unsafe bool TryFusedGdnBlock(Tensor res, int il, int seqLen)
+        {
+            if (!_fusedGdnEnabled || _fusedGdnUnsupported || !IsGgmlBackend)
+                return false;
+            // Bisect handle: TS_Q4E_GDN_MAX_LAYERS=N fuses only layers below N and
+            // leaves the rest op-by-op, which separates a per-layer error from one
+            // that only appears once it compounds across 36 of them.
+            if (_gdnMaxLayers >= 0 && il >= _gdnMaxLayers)
+                return false;
+
+            try
+            {
+                if (!EnsureGdnArgs()) return false;
+                if (_gdnConvStateT[il] == null) return false;
+
+                bool ok = GgmlBasicOps.Qwen4ExpGdnBlock(ref _gdnArgs[il],
+                    (IntPtr)GetFloatPtr(res),
+                    Config.HiddenSize, _hc, _hcLowRank, seqLen,
+                    _headKDim, _headVDim, _numKHeads, _numVHeads, _convKernel,
+                    Config.Eps, cacheSlot: il, resResident: _resOnDevice);
+                if (!ok) { _fusedGdnUnsupported = true; return false; }
+                if (!_resOnDevice) InvalidateTensorDeviceCache(res);
+                return true;
+            }
+            catch (Exception)
+            {
+                _fusedGdnUnsupported = true;
+                return false;
+            }
+        }
+
+        private unsafe bool TryFillGdnArgs(int il, ref Qwen4ExpGdnArgs a)
+        {
+            if (!TryResolveQuant($"blk.{il}.hc_attn_down.weight", out IntPtr hd, out int hdT, out long hdB)
+                || !TryResolveQuant($"blk.{il}.hc_attn_up.weight", out IntPtr hu, out int huT, out long huB)
+                || !TryResolveQuant($"blk.{il}.hc_attn_inject.weight", out IntPtr hj, out int hjT, out long hjB)
+                || !TryResolveQuant($"blk.{il}.attn_qkv.weight", out IntPtr qv, out int qvT, out long qvB)
+                || !TryResolveQuant($"blk.{il}.attn_gate.weight", out IntPtr gt, out int gtT, out long gtB)
+                || !TryResolveQuant($"blk.{il}.ssm_beta.weight", out IntPtr bt, out int btT, out long btB)
+                || !TryResolveQuant($"blk.{il}.ssm_alpha.weight", out IntPtr al, out int alT, out long alB)
+                || !TryResolveQuant($"blk.{il}.ssm_out.weight", out IntPtr op, out int opT, out long opB))
+            {
+                return false;
+            }
+            // The conv kernel has to be F32: the graph multiplies it against F32
+            // activations without a cast.
+            if (!_weights.TryGetValue($"blk.{il}.ssm_conv1d.weight", out Tensor convW))
+                return false;
+
+            a.HcNorm = (IntPtr)GetFloatPtr(_weights[$"blk.{il}.hc_attn_norm.weight"]);
+            a.HcDown = hd; a.HcDownType = hdT; a.HcDownBytes = hdB;
+            a.HcUp = hu; a.HcUpType = huT; a.HcUpBytes = huB;
+            a.HcInject = hj; a.HcInjectType = hjT; a.HcInjectBytes = hjB;
+            a.Qkv = qv; a.QkvType = qvT; a.QkvBytes = qvB;
+            a.Gate = gt; a.GateType = gtT; a.GateBytes = gtB;
+            a.Beta = bt; a.BetaType = btT; a.BetaBytes = btB;
+            a.Alpha = al; a.AlphaType = alT; a.AlphaBytes = alB;
+            a.OutProj = op; a.OutProjType = opT; a.OutProjBytes = opB;
+            a.Conv1d = (IntPtr)GetFloatPtr(convW);
+            a.SsmDt = (IntPtr)GetFloatPtr(_weights[$"blk.{il}.ssm_dt.bias"]);
+            a.SsmA = (IntPtr)GetFloatPtr(_weights[$"blk.{il}.ssm_a"]);
+            a.SsmNorm = (IntPtr)GetFloatPtr(_weights[$"blk.{il}.ssm_norm.weight"]);
+            a.ConvState = (IntPtr)GetFloatPtr(_gdnConvStateT[il]);
+            a.SsmState = (IntPtr)GetFloatPtr(_gdnStateT[il]);
+            return true;
+        }
+
+        // TS_Q4E_FUSED_FFN=0 falls back to the op-by-op mixer + MoE + scatter,
+        // which is also the automatic fallback for any shape the kernel declines.
+        private static readonly bool _fusedFfnEnabled =
+            !string.Equals(Environment.GetEnvironmentVariable("TS_Q4E_FUSED_FFN"), "0", StringComparison.Ordinal);
+        private bool _fusedFfnUnsupported;
+        private Qwen4ExpFfnArgs[] _ffnArgs;
+
+        private unsafe bool TryFusedFfnBlock(Tensor res, int il, int seqLen)
+        {
+            if (!_fusedFfnEnabled || _fusedFfnUnsupported || !IsGgmlBackend || _fusedGateUpExperts)
+                return false;
+
+            try
+            {
+                if (!EnsureFfnArgs()) return false;
+
+                bool ok = GgmlBasicOps.Qwen4ExpFfnBlock(ref _ffnArgs[il],
+                    (IntPtr)GetFloatPtr(res),
+                    Config.HiddenSize, _hc, _hcLowRank, seqLen,
+                    _numExperts, _numExpertsUsed, _expertFf, _sharedFf, Config.Eps,
+                    cacheSlot: il, resResident: _resOnDevice);
+                if (!ok)
+                {
+                    _fusedFfnUnsupported = true;
+                    return false;
+                }
+                if (!_resOnDevice) InvalidateTensorDeviceCache(res);
+                return true;
+            }
+            catch (Exception)
+            {
+                _fusedFfnUnsupported = true;
+                return false;
+            }
+        }
+
+        private unsafe bool TryFillFfnArgs(int il, ref Qwen4ExpFfnArgs a)
+        {
+            if (!_stackedExpertWeights.TryGetValue($"blk.{il}.ffn_gate_exps.weight", out var g)
+                || !_stackedExpertWeights.TryGetValue($"blk.{il}.ffn_up_exps.weight", out var u)
+                || !_stackedExpertWeights.TryGetValue($"blk.{il}.ffn_down_exps.weight", out var d))
+            {
+                return false;
+            }
+
+            if (!TryResolveQuant($"blk.{il}.hc_ffn_down.weight", out IntPtr hd, out int hdT, out long hdB)
+                || !TryResolveQuant($"blk.{il}.hc_ffn_up.weight", out IntPtr hu, out int huT, out long huB)
+                || !TryResolveQuant($"blk.{il}.hc_ffn_inject.weight", out IntPtr hj, out int hjT, out long hjB)
+                || !TryResolveQuant($"blk.{il}.ffn_gate_inp.weight", out IntPtr rt, out int rtT, out long rtB)
+                || !TryResolveQuant($"blk.{il}.ffn_gate_shexp.weight", out IntPtr sg, out int sgT, out long sgB)
+                || !TryResolveQuant($"blk.{il}.ffn_up_shexp.weight", out IntPtr su, out int suT, out long suB)
+                || !TryResolveQuant($"blk.{il}.ffn_down_shexp.weight", out IntPtr sd, out int sdT, out long sdB))
+            {
+                return false;
+            }
+
+            a.HcNorm = (IntPtr)GetFloatPtr(_weights[$"blk.{il}.hc_ffn_norm.weight"]);
+            a.HcDown = hd; a.HcDownType = hdT; a.HcDownBytes = hdB;
+            a.HcUp = hu; a.HcUpType = huT; a.HcUpBytes = huB;
+            a.HcInject = hj; a.HcInjectType = hjT; a.HcInjectBytes = hjB;
+            a.Router = rt; a.RouterType = rtT; a.RouterBytes = rtB;
+            a.GateExps = g.Data; a.GateExpsType = g.GgmlType; a.GateExpsBytes = g.TotalRawBytes;
+            a.UpExps = u.Data; a.UpExpsType = u.GgmlType; a.UpExpsBytes = u.TotalRawBytes;
+            a.DownExps = d.Data; a.DownExpsType = d.GgmlType; a.DownExpsBytes = d.TotalRawBytes;
+            a.ShGateInp = (IntPtr)GetFloatPtr(_weights[$"blk.{il}.ffn_gate_inp_shexp.weight"]);
+            a.ShGate = sg; a.ShGateType = sgT; a.ShGateBytes = sgB;
+            a.ShUp = su; a.ShUpType = suT; a.ShUpBytes = suB;
+            a.ShDown = sd; a.ShDownType = sdT; a.ShDownBytes = sdB;
+            return true;
+        }
+
+        /// <summary>Resolve a weight to (device cache key, ggml type, raw bytes),
+        /// from either its quantized or its F32 form.</summary>
+        private unsafe bool TryResolveQuant(string name, out IntPtr ptr, out int type, out long bytes)
+        {
+            if (_quantWeights.TryGetValue(name, out var qw))
+            {
+                ptr = qw.CacheKey; type = qw.GgmlType; bytes = qw.RawBytes;
+                return true;
+            }
+            if (_weights.TryGetValue(name, out var w))
+            {
+                ptr = (IntPtr)GetFloatPtr(w);
+                type = 0;                                   // GGML_TYPE_F32
+                bytes = w.ElementCount() * sizeof(float);
+                return true;
+            }
+            ptr = IntPtr.Zero; type = 0; bytes = 0;
+            return false;
+        }
+
+        private int[] _moeSelExperts;
+        private float[] _moeRouteWts;
+        private float[] _moeProbs;
+
+        // Per-expert non-owning views over the stacked tensors, built on first use.
+        // Only the backends without a stacked-expert kernel need them.
+        private QuantizedWeight[][] _expertGateView, _expertUpView, _expertDownView;
+        private Tensor _moeScratchGate, _moeScratchUp, _moeScratchDown;
+
+        private void MoeExpertsPortable(Tensor outRow, Tensor inRow, int il, int[] expertIds, float[] routeW)
+        {
+            EnsureExpertViews(il);
+            int n = Config.HiddenSize;
+            int ff = (int)_expertGateView[il][expertIds[0]].Ne1;
+
+            _moeScratchGate ??= new Tensor(_allocator, DType.Float32, 1, ff);
+            _moeScratchUp ??= new Tensor(_allocator, DType.Float32, 1, ff);
+            _moeScratchDown ??= new Tensor(_allocator, DType.Float32, 1, n);
+
+            Ops.Fill(outRow, 0f);
+            for (int k = 0; k < expertIds.Length; k++)
+            {
+                int e = expertIds[k];
+                AddmmQuantManaged(_moeScratchGate, inRow, _expertGateView[il][e]);
+                AddmmQuantManaged(_moeScratchUp, inRow, _expertUpView[il][e]);
+                Ops.SiLUMul(_moeScratchGate, _moeScratchGate, _moeScratchUp);
+                AddmmQuantManaged(_moeScratchDown, _moeScratchGate, _expertDownView[il][e]);
+                Ops.AddMulV(outRow, outRow, _moeScratchDown, routeW[k]);
+            }
+        }
+
+        private void EnsureExpertViews(int il)
+        {
+            _expertGateView ??= new QuantizedWeight[Config.NumLayers][];
+            _expertUpView ??= new QuantizedWeight[Config.NumLayers][];
+            _expertDownView ??= new QuantizedWeight[Config.NumLayers][];
+            if (_expertGateView[il] != null) return;
+
+            var g = _stackedExpertWeights[$"blk.{il}.ffn_gate_exps.weight"];
+            var u = _stackedExpertWeights[$"blk.{il}.ffn_up_exps.weight"];
+            var d = _stackedExpertWeights[$"blk.{il}.ffn_down_exps.weight"];
+            _expertGateView[il] = new QuantizedWeight[_numExperts];
+            _expertUpView[il] = new QuantizedWeight[_numExperts];
+            _expertDownView[il] = new QuantizedWeight[_numExperts];
+            for (int e = 0; e < _numExperts; e++)
+            {
+                _expertGateView[il][e] = QuantizedWeight.CreateExpertView(g, e);
+                _expertUpView[il][e] = QuantizedWeight.CreateExpertView(u, e);
+                _expertDownView[il][e] = QuantizedWeight.CreateExpertView(d, e);
+            }
+        }
+
+        public override void Dispose()
+        {
+            DisposeAllFusedHolders();
+            if (IsGgmlBackend)
+                GgmlBasicOps.Qwen4ExpReleaseAllSeqState();
+            if (_kCache != null)
+                foreach (var t in _kCache) t?.Dispose();
+            if (_vCache != null)
+                foreach (var t in _vCache) t?.Dispose();
+            if (_idxKCache != null)
+                foreach (var t in _idxKCache) t?.Dispose();
+            base.Dispose();
+        }
+    }
+}
diff --git a/TensorSharp.Models/Models/Qwen4Exp/Qwen4ExpModel.Layers.cs b/TensorSharp.Models/Models/Qwen4Exp/Qwen4ExpModel.Layers.cs
new file mode 100644
index 00000000..aed2c2e3
--- /dev/null
+++ b/TensorSharp.Models/Models/Qwen4Exp/Qwen4ExpModel.Layers.cs
@@ -0,0 +1,573 @@
+// Copyright (c) Zhongkai Fu. All rights reserved.
+// https://github.com/zhongkaifu/TensorSharp
+//
+// This file is part of TensorSharp.
+//
+// TensorSharp is licensed under the BSD-3-Clause license found in the LICENSE file in the root directory of this source tree.
+//
+// TensorSharp is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the BSD-3-Clause License for more details.
+using System;
+using System.Diagnostics;
+using TensorSharp.Core;
+using TensorSharp.GGML;
+
+namespace TensorSharp.Models
+{
+    public partial class Qwen4ExpModel
+    {
+        // Transposed conv kernels, [tap][channel], built once per layer: the GGUF
+        // stores [kernel, channels] and the step wants a contiguous row per tap.
+        private float[][] _gdnConvWT;
+        private int[] _gdnConvWriteIdx;
+
+        // Scratch, sized on first use.
+        private float[] _gdnConvOut, _gdnQ, _gdnK, _gdnV, _gdnQx, _gdnKx, _gdnDelta, _gdnCore;
+
+        private bool _qsaBudgetWarned;
+
+        /// <summary>
+        /// Gated DeltaNet, the same recurrence Qwen 3.5 runs, with two differences
+        /// that matter: the input arrives already normed (the hyper-connection mixer
+        /// did it) and the output gate is a SIGMOID rather than a SiLU.
+        ///
+        /// Deliberately a self-contained per-token loop rather than a call into
+        /// Qwen35Model's chunked path: that one is welded to its own state arrays and
+        /// weight caches, and correctness here is worth more than sharing the fused
+        /// kernel, which this can adopt later.
+        /// </summary>
+        private unsafe Tensor GdnLayer(Tensor cur, int il, int seqLen)
+        {
+            int valueDim = _headVDim * _numVHeads;
+
+            EnsureGdnScratch();
+            EnsureGdnConvWeights(il);
+
+            long tp0 = Stopwatch.GetTimestamp();
+            Tensor qkv = LinearForward(cur, $"blk.{il}.attn_qkv.weight");     // [T, convDim]
+            Tensor z = LinearForward(cur, $"blk.{il}.attn_gate.weight");      // [T, valueDim]
+            Tensor betaT = LinearForward(cur, $"blk.{il}.ssm_beta.weight");   // [T, numVHeads]
+            Tensor alphaT = LinearForward(cur, $"blk.{il}.ssm_alpha.weight"); // [T, numVHeads]
+            Q4eGdnProjTicks += Stopwatch.GetTimestamp() - tp0;
+
+            var gated = new Tensor(_allocator, DType.Float32, seqLen, valueDim);
+
+            // The conv, the q/k norm+tiling and the alpha/beta gate arithmetic stay on
+            // the host - the fused kernel wants them pre-computed - but the recurrence
+            // itself, which was 41% of a decode token as a C# loop, goes to the GPU.
+            EnsureChunkedStaging(seqLen);
+            long tp1 = Stopwatch.GetTimestamp();
+            PrepareGdnInputs(qkv, z, betaT, alphaT, il, seqLen);
+            Q4eGdnPrepTicks += Stopwatch.GetTimestamp() - tp1;
+
+            using Tensor qv = _gdnQBuf.Narrow(0, 0, seqLen);
+            using Tensor kv = _gdnKBuf.Narrow(0, 0, seqLen);
+            using Tensor vv = _gdnVBuf.Narrow(0, 0, seqLen);
+            using Tensor zv = _gdnZBuf.Narrow(0, 0, seqLen);
+            using Tensor av = _gdnAlphaBuf.Narrow(0, 0, seqLen);
+            using Tensor bv = _gdnBetaBuf.Narrow(0, 0, seqLen);
+
+            // The kernel reads [T, H, D]; `gated` is the same memory laid out [T, H*D].
+            using Tensor gated3 = gated.View(seqLen, _numVHeads, _headVDim);
+            long tp2 = Stopwatch.GetTimestamp();
+            GgmlBasicOps.GatedDeltaNetChunked(
+                qv, kv, vv, zv, av, bv, _gdnStateT[il], gated3,
+                (IntPtr)GetFloatPtr(_weights[$"blk.{il}.ssm_dt.bias"]),
+                (IntPtr)GetFloatPtr(_weights[$"blk.{il}.ssm_a"]),
+                (IntPtr)GetFloatPtr(_weights[$"blk.{il}.ssm_norm.weight"]),
+                // Fixed 64. Sizing the chunk to the batch looked appealing - a 1-token
+                // decode is padded into a 64-wide chunk - but it measured within noise
+                // and an arbitrary chunk (seqLen 58) aborts the kernel, which assumes
+                // the tuned width.
+                GdnChunkSize, Config.Eps,
+                // qwen4exp gates the delta-net output with a SIGMOID where Qwen 3.5
+                // uses SiLU. Same kernel, one flag.
+                gateMode: 1);
+            Q4eGdnKernelTicks += Stopwatch.GetTimestamp() - tp2;
+
+            qkv.Dispose(); z.Dispose(); betaT.Dispose(); alphaT.Dispose();
+            InvalidateTensorDeviceCache(gated);
+
+            long tp3 = Stopwatch.GetTimestamp();
+            Tensor outProj = LinearForward(gated, $"blk.{il}.ssm_out.weight");
+            Q4eGdnOutTicks += Stopwatch.GetTimestamp() - tp3;
+            gated.Dispose();
+            return outProj;
+        }
+
+        private const int GdnChunkSize = 64;
+
+        // Staging for the fused kernel: [T, H, D] q/k/v/z and [T, H] alpha/beta, grown
+        // to the largest sequence seen so far and sub-viewed per call.
+        private Tensor _gdnQBuf, _gdnKBuf, _gdnVBuf, _gdnZBuf, _gdnAlphaBuf, _gdnBetaBuf;
+        private Tensor[] _gdnStateT;
+        private int _gdnStagingRows;
+
+        private void EnsureChunkedStaging(int seqLen)
+        {
+            if (_gdnStagingRows >= seqLen && _gdnQBuf != null) return;
+            _gdnQBuf?.Dispose(); _gdnKBuf?.Dispose(); _gdnVBuf?.Dispose();
+            _gdnZBuf?.Dispose(); _gdnAlphaBuf?.Dispose(); _gdnBetaBuf?.Dispose();
+            int rows = Math.Max(seqLen, 1);
+            _gdnQBuf = new Tensor(_allocator, DType.Float32, rows, _numVHeads, _headKDim);
+            _gdnKBuf = new Tensor(_allocator, DType.Float32, rows, _numVHeads, _headKDim);
+            _gdnVBuf = new Tensor(_allocator, DType.Float32, rows, _numVHeads, _headVDim);
+            _gdnZBuf = new Tensor(_allocator, DType.Float32, rows, _numVHeads, _headVDim);
+            _gdnAlphaBuf = new Tensor(_allocator, DType.Float32, rows, _numVHeads);
+            _gdnBetaBuf = new Tensor(_allocator, DType.Float32, rows, _numVHeads);
+            _gdnStagingRows = rows;
+        }
+
+        /// <summary>
+        /// Host-side front of the recurrence: the causal depthwise conv over the ring
+        /// history, the q/k L2 norm and head tiling, and the alpha/beta gate arithmetic
+        /// the fused kernel expects pre-computed.
+        /// </summary>
+        private unsafe void PrepareGdnInputs(Tensor qkv, Tensor z, Tensor betaT, Tensor alphaT,
+            int il, int seqLen)
+        {
+            int keyDim = _headKDim * _numKHeads;
+            int valueDim = _headVDim * _numVHeads;
+            int convDim = _convKernel - 1;
+            float eps = Config.Eps;
+            float qScale = 1.0f / MathF.Sqrt(_headVDim);
+
+            float* qkvP = GetFloatPtr(qkv);
+            float* zP = GetFloatPtr(z);
+            float* betaP = GetFloatPtr(betaT);
+            float* alphaP = GetFloatPtr(alphaT);
+            float* dtBias = GetFloatPtr(_weights[$"blk.{il}.ssm_dt.bias"]);
+            float* aLog = GetFloatPtr(_weights[$"blk.{il}.ssm_a"]);
+
+            float* qOut = GetFloatPtr(_gdnQBuf);
+            float* kOut = GetFloatPtr(_gdnKBuf);
+            float* vOut = GetFloatPtr(_gdnVBuf);
+            float* zOut = GetFloatPtr(_gdnZBuf);
+            float* aOut = GetFloatPtr(_gdnAlphaBuf);
+            float* bOut = GetFloatPtr(_gdnBetaBuf);
+
+            float[] convState = _gdnConvState[il];
+            float[] convWT = _gdnConvWT[il];
+
+            fixed (float* convOut = _gdnConvOut, qBuf = _gdnQ, kBuf = _gdnK, vBuf = _gdnV,
+                          convStatePtr = convState, convWPtr = convWT)
+            {
+                for (int t = 0; t < seqLen; t++)
+                {
+                    float* xin = qkvP + (long)t * _convDim;
+                    int writeIdx = _gdnConvWriteIdx[il];
+
+                    for (int c = 0; c < _convDim; c++) convOut[c] = 0f;
+                    for (int ki = 0; ki < convDim; ki++)
+                    {
+                        int slot = (writeIdx + ki) % convDim;
+                        float* sp = convStatePtr + (long)slot * _convDim;
+                        float* wp = convWPtr + (long)ki * _convDim;
+                        for (int c = 0; c < _convDim; c++) convOut[c] += sp[c] * wp[c];
+                    }
+                    {
+                        float* wp = convWPtr + (long)convDim * _convDim;
+                        for (int c = 0; c < _convDim; c++) convOut[c] += xin[c] * wp[c];
+                    }
+                    for (int c = 0; c < _convDim; c++)
+                        convOut[c] = convOut[c] / (1.0f + MathF.Exp(-convOut[c]));
+
+                    if (convDim > 0)
+                    {
+                        Buffer.MemoryCopy(xin, convStatePtr + (long)writeIdx * _convDim,
+                            _convDim * 4L, _convDim * 4L);
+                        _gdnConvWriteIdx[il] = (writeIdx + 1) % convDim;
+                    }
+
+                    Buffer.MemoryCopy(convOut, qBuf, keyDim * 4L, keyDim * 4L);
+                    Buffer.MemoryCopy(convOut + keyDim, kBuf, keyDim * 4L, keyDim * 4L);
+                    Buffer.MemoryCopy(convOut + 2 * keyDim, vBuf, valueDim * 4L, valueDim * 4L);
+
+                    // L2 norm per K head, then TILE the K heads across the V heads:
+                    // ggml_repeat tiles rather than interleaves, so head h reads
+                    // h % num_k_heads. Getting this backwards is silent nonsense.
+                    L2NormHeads(qBuf, _numKHeads, _headKDim, eps);
+                    L2NormHeads(kBuf, _numKHeads, _headKDim, eps);
+
+                    float* qRow = qOut + (long)t * _numVHeads * _headKDim;
+                    float* kRow = kOut + (long)t * _numVHeads * _headKDim;
+                    for (int h = 0; h < _numVHeads; h++)
+                    {
+                        int src = (h % _numKHeads) * _headKDim;
+                        float* qd = qRow + (long)h * _headKDim;
+                        Buffer.MemoryCopy(qBuf + src, qd, _headKDim * 4L, _headKDim * 4L);
+                        Buffer.MemoryCopy(kBuf + src, kRow + (long)h * _headKDim,
+                            _headKDim * 4L, _headKDim * 4L);
+                        for (int i = 0; i < _headKDim; i++) qd[i] *= qScale;
+                    }
+
+                    Buffer.MemoryCopy(vBuf, vOut + (long)t * valueDim, valueDim * 4L, valueDim * 4L);
+                    Buffer.MemoryCopy(zP + (long)t * valueDim, zOut + (long)t * valueDim,
+                        valueDim * 4L, valueDim * 4L);
+
+                    float* aRow = aOut + (long)t * _numVHeads;
+                    float* bRow = bOut + (long)t * _numVHeads;
+                    float* alphaRow = alphaP + (long)t * _numVHeads;
+                    float* betaRow = betaP + (long)t * _numVHeads;
+                    for (int h = 0; h < _numVHeads; h++)
+                    {
+                        // ssm_a ships pre-negated, so this decays.
+                        aRow[h] = Softplus(alphaRow[h] + dtBias[h]) * aLog[h];
+                        bRow[h] = 1.0f / (1.0f + MathF.Exp(-betaRow[h]));
+                    }
+                }
+            }
+
+            InvalidateTensorDeviceCache(_gdnQBuf);
+            InvalidateTensorDeviceCache(_gdnKBuf);
+            InvalidateTensorDeviceCache(_gdnVBuf);
+            InvalidateTensorDeviceCache(_gdnZBuf);
+            InvalidateTensorDeviceCache(_gdnAlphaBuf);
+            InvalidateTensorDeviceCache(_gdnBetaBuf);
+        }
+
+        private static float Softplus(float x)
+            => x > 20f ? x : MathF.Log(1.0f + MathF.Exp(x));
+
+        private static unsafe void L2NormHeads(float* x, int heads, int dim, float eps)
+        {
+            for (int h = 0; h < heads; h++)
+            {
+                float* p = x + (long)h * dim;
+                double ss = 0;
+                for (int i = 0; i < dim; i++) ss += (double)p[i] * p[i];
+                float inv = (float)(1.0 / Math.Sqrt(ss + eps));
+                for (int i = 0; i < dim; i++) p[i] *= inv;
+            }
+        }
+
+        private void EnsureGdnScratch()
+        {
+            if (_gdnConvOut != null) return;
+            // The fused kernel updates the recurrent state in place, so it lives in a
+            // Tensor rather than the float[] the per-token loop used. A holder
+            // swap may have installed pre-seeded tensors whose addresses key the
+            // native state entries - reuse them.
+            _gdnStateT ??= new Tensor[Config.NumLayers];
+            for (int l = 0; l < Config.NumLayers; l++)
+            {
+                if (!_isRecurrent[l]) continue;
+                if (_gdnStateT[l] != null) continue;
+                _gdnStateT[l] = new Tensor(_allocator, DType.Float32, _numVHeads, _headVDim, _headKDim);
+                Ops.Fill(_gdnStateT[l], 0f);
+            }
+            int keyDim = _headKDim * _numKHeads;
+            int valueDim = _headVDim * _numVHeads;
+            _gdnConvOut = new float[_convDim];
+            _gdnQ = new float[keyDim];
+            _gdnK = new float[keyDim];
+            _gdnV = new float[valueDim];
+            _gdnQx = new float[(long)_numVHeads * _headKDim];
+            _gdnKx = new float[(long)_numVHeads * _headKDim];
+            _gdnDelta = new float[_headVDim];
+            _gdnCore = new float[_headVDim];
+            _gdnConvWriteIdx = new int[Config.NumLayers];
+        }
+
+        private unsafe void EnsureGdnConvWeights(int il)
+        {
+            _gdnConvWT ??= new float[Config.NumLayers][];
+            if (_gdnConvWT[il] != null) return;
+
+            Tensor w = _weights[$"blk.{il}.ssm_conv1d.weight"];
+            var wt = new float[(long)_convKernel * _convDim];
+            float* src = GetFloatPtr(w);
+            // GGUF ships [kernel, channels]; the step wants one contiguous row per tap.
+            for (int c = 0; c < _convDim; c++)
+                for (int k = 0; k < _convKernel; k++)
+                    wt[(long)k * _convDim + c] = src[(long)c * _convKernel + k];
+            _gdnConvWT[il] = wt;
+        }
+
+        /// <summary>
+        /// Full attention: a joint Q|gate projection (per head, interleaved), Q/K RMS
+        /// norm, partial IMRoPE, then a sigmoid gate on the attention output.
+        ///
+        /// QSA is not applied yet. It does not have to be below its budget: the indexer
+        /// keeps <c>indexer_top_k + compress_ratio - 1</c> cells, so at or under that
+        /// many cached tokens the selection is every cell and the result is exactly
+        /// dense. Past it this warns once rather than silently drifting.
+        /// </summary>
+        private unsafe Tensor AttentionLayer(Tensor cur, int il, int seqLen, int startPos)
+        {
+            int nHead = Config.NumHeads;
+            int nKvHead = Config.NumKVHeads;
+            int headDim = Config.HeadDim;
+            int totalLen = startPos + seqLen;
+
+            long ta0 = Stopwatch.GetTimestamp();
+            Tensor qgFull = LinearForward(cur, $"blk.{il}.attn_q.weight");  // [T, nHead*2*headDim]
+            var q = new Tensor(_allocator, DType.Float32, seqLen, nHead * headDim);
+            var gate = new Tensor(_allocator, DType.Float32, seqLen, nHead * headDim);
+            {
+                float* src = GetFloatPtr(qgFull);
+                float* qp = GetFloatPtr(q);
+                float* gp = GetFloatPtr(gate);
+                int stride = 2 * headDim;
+                for (int t = 0; t < seqLen; t++)
+                {
+                    float* s = src + (long)t * nHead * stride;
+                    float* qd = qp + (long)t * nHead * headDim;
+                    float* gd = gp + (long)t * nHead * headDim;
+                    for (int h = 0; h < nHead; h++)
+                    {
+                        Buffer.MemoryCopy(s + (long)h * stride, qd + (long)h * headDim, headDim * 4L, headDim * 4L);
+                        Buffer.MemoryCopy(s + (long)h * stride + headDim, gd + (long)h * headDim, headDim * 4L, headDim * 4L);
+                    }
+                }
+                InvalidateTensorDeviceCache(q);
+                InvalidateTensorDeviceCache(gate);
+            }
+            qgFull.Dispose();
+
+            Tensor k = LinearForward(cur, $"blk.{il}.attn_k.weight");
+            Tensor v = LinearForward(cur, $"blk.{il}.attn_v.weight");
+            long ta1 = Stopwatch.GetTimestamp(); Q4eAttnProjTicks += ta1 - ta0;
+
+            RmsNormHeads(q, $"blk.{il}.attn_q_norm.weight", nHead, headDim, seqLen);
+            RmsNormHeads(k, $"blk.{il}.attn_k_norm.weight", nKvHead, headDim, seqLen);
+            long ta2 = Stopwatch.GetTimestamp(); Q4eAttnNormTicks += ta2 - ta1;
+
+            q = ApplyRope(q, nHead, headDim, seqLen, startPos);
+            k = ApplyRope(k, nKvHead, headDim, seqLen, startPos);
+            long ta3 = Stopwatch.GetTimestamp(); Q4eAttnRopeTicks += ta3 - ta2;
+
+            Tensor attn = RunAttention(q, k, v, il, seqLen, startPos, totalLen);
+            long ta4 = Stopwatch.GetTimestamp(); Q4eAttnCoreTicks += ta4 - ta3;
+            _attnTicks += ta4 - ta3;
+
+            // Gate the attention output before the output projection. Two dispatches
+            // for 6144 elements; on a dispatch-bound decode step that is pure launch
+            // overhead, so small batches do it on the host (see HostElementwiseMaxRows).
+            if (seqLen > HostElementwiseMaxRows)
+            {
+                Ops.Sigmoid(gate, gate);
+                Ops.Mul(attn, attn, gate);
+            }
+            else
+            {
+                float* gp = GetFloatPtr(gate);
+                float* ap = GetFloatPtr(attn);
+                long count = (long)seqLen * nHead * headDim;
+                for (long i = 0; i < count; i++)
+                    ap[i] *= 1.0f / (1.0f + MathF.Exp(-gp[i]));
+                InvalidateTensorDeviceCache(attn);
+            }
+            gate.Dispose();
+
+            Tensor outProj = LinearForward(attn, $"blk.{il}.attn_output.weight");
+            attn.Dispose();
+            Q4eAttnOutTicks += Stopwatch.GetTimestamp() - ta4;
+            return outProj;
+        }
+
+        private Tensor RunAttention(Tensor q, Tensor k, Tensor v, int il,
+            int seqLen, int startPos, int totalLen)
+        {
+            int nHead = Config.NumHeads;
+            int nKvHead = Config.NumKVHeads;
+            int headDim = Config.HeadDim;
+
+            if (seqLen == 1)
+            {
+                var res = new Tensor(_allocator, DType.Float32, 1, nHead * headDim);
+
+                // Prefer the device kernel: it appends K/V and attends without ever
+                // pulling the cache to the host. AttentionDecodePureCS walks the cache
+                // as a host buffer, which on this model (head_dim 256, 12 attention
+                // layers) was 2.75 ms per layer - 33 ms of a 106 ms decode token, the
+                // single largest cost after the recurrence.
+                if (IsGgmlBackend && TryFlashDecode(q, k, v, res, il, startPos))
+                {
+                    q.Dispose(); k.Dispose(); v.Dispose();
+                    return res;
+                }
+
+                CopyToCacheDecode(_kCache[il], k, _vCache[il], v, nKvHead, headDim, startPos);
+                k.Dispose(); v.Dispose();
+                AttentionDecodePureCS(q, _kCache[il], _vCache[il], res,
+                    nHead, nKvHead, headDim, totalLen, _attnScale);
+                q.Dispose();
+                return res;
+            }
+
+            Tensor qH = ReshapeToHeads(q, nHead, seqLen, headDim); q.Dispose();
+            Tensor kH = ReshapeToHeads(k, nKvHead, seqLen, headDim); k.Dispose();
+            Tensor vH = ReshapeToHeads(v, nKvHead, seqLen, headDim); v.Dispose();
+
+            CopyToCache(_kCache[il], kH, startPos, seqLen);
+            CopyToCache(_vCache[il], vH, startPos, seqLen);
+            kH.Dispose(); vH.Dispose();
+
+            int group = nHead / nKvHead;
+            Tensor kExp = ExpandKVHeads(_kCache[il], group, totalLen);
+            Tensor vExp = ExpandKVHeads(_vCache[il], group, totalLen);
+
+            using var kT = kExp.Transpose(1, 2);
+            var scores = new Tensor(_allocator, DType.Float32, nHead, seqLen, totalLen);
+            Ops.AddmmBatch(scores, 0, scores, _attnScale, qH, kT);
+            qH.Dispose(); kExp.Dispose();
+
+            if (IsGgmlBackend)
+            {
+                GgmlBasicOps.AttentionSoftmaxWithSinks(scores, sinks: null,
+                    numHeads: nHead, seqLen: seqLen, kvLen: totalLen,
+                    maskStartPos: startPos, slidingWindow: 0, scale: 1.0f);
+            }
+            else
+            {
+                Ops.AddCausalMask(scores, seqLen, startPos, float.NegativeInfinity);
+                Ops.Softmax(scores, scores);
+            }
+
+            var attnOut = new Tensor(_allocator, DType.Float32, nHead, seqLen, headDim);
+            Ops.AddmmBatch(attnOut, 0, attnOut, 1.0f, scores, vExp);
+            scores.Dispose(); vExp.Dispose();
+
+            Tensor flat = ReshapeFromHeads(attnOut, nHead, seqLen, headDim);
+            attnOut.Dispose();
+            return flat;
+        }
+
+        /// <summary>Device-side single-token attention: appends K/V to the cache and
+        /// attends over [0, position]. False on any shape or dtype the kernel declines,
+        /// which falls back to the host walk.</summary>
+        private bool TryFlashDecode(Tensor q, Tensor k, Tensor v, Tensor output, int il, int position)
+        {
+            Tensor kCache = _kCache[il], vCache = _vCache[il];
+            if (kCache == null || vCache == null || kCache.ElementType != vCache.ElementType)
+                return false;
+            if (kCache.ElementType != DType.Float32 && kCache.ElementType != DType.Float16)
+                return false;
+
+            try
+            {
+                GgmlBasicOps.FlashAttnDecode(q, k, v, kCache, vCache, output,
+                    Config.NumHeads, Config.NumKVHeads, Config.HeadDim,
+                    _kvCacheCapacity, position, _attnScale);
+                InvalidateTensorDeviceCache(output);
+                return true;
+            }
+            catch (Exception)
+            {
+                return false;
+            }
+        }
+
+        private unsafe void RmsNormHeads(Tensor data, string weightName, int heads, int dim, int seqLen)
+        {
+            float* p = GetFloatPtr(data);
+            float* w = GetFloatPtr(_weights[weightName]);
+            float eps = Config.Eps;
+            for (int t = 0; t < seqLen; t++)
+            {
+                float* row = p + (long)t * heads * dim;
+                for (int h = 0; h < heads; h++)
+                {
+                    float* x = row + (long)h * dim;
+                    double ss = 0;
+                    for (int i = 0; i < dim; i++) ss += (double)x[i] * x[i];
+                    float inv = (float)(1.0 / Math.Sqrt(ss / dim + eps));
+                    for (int i = 0; i < dim; i++) x[i] = x[i] * inv * w[i];
+                }
+            }
+            InvalidateTensorDeviceCache(data);
+        }
+
+        /// <summary>
+        /// Say once, per model, that the context has passed the size at which the
+        /// (unimplemented) sparse-attention indexer would start selecting rather
+        /// than keeping every cell. Every path runs dense attention either way, so
+        /// this is a note about reference fidelity, not a behaviour switch.
+        ///
+        /// It lives on the common forward entry point deliberately. It used to be
+        /// printed from the op-by-op AttentionLayer, which the fused paths reached
+        /// only by DECLINING the whole token - so printing the warning cost a
+        /// mid-sequence path switch that reset the GDN and PLE recurrent state and
+        /// wrecked the generation. A diagnostic must never be the reason a code
+        /// path is taken.
+        /// </summary>
+        private void WarnIfQsaBudgetExceeded(int totalLen)
+        {
+            if (_qsaBudgetWarned || _compressRatios == null) return;
+            for (int il = 0; il < Config.NumLayers; il++)
+            {
+                if (!UsesQsa(il)) continue;
+                int budget = _indexerTopK + _compressRatios[il] - 1;
+                if (totalLen <= budget) continue;
+                _qsaBudgetWarned = true;
+                Console.WriteLine(
+                    $"[qwen4exp] context {totalLen} exceeds the QSA budget " +
+                    $"({_indexerTopK} + {_compressRatios[il]} - 1); running dense attention. " +
+                    "Output stays close but is no longer bit-exact against the reference.");
+                return;
+            }
+        }
+
+        /// <summary>
+        /// Partial rotary over the first <c>rope.dimension_count</c> of each head.
+        /// The file asks for IMRoPE, which interleaves the t/h/w position components
+        /// across pairs; for text every component is the same position, so it reduces
+        /// exactly to NEOX RoPE. Vision will need the real thing.
+        /// </summary>
+        private unsafe Tensor ApplyRope(Tensor data, int heads, int headDim, int seqLen, int startPos)
+        {
+            // Partial NEOX rotary over the first _ropeDimCount dims. For a decode step
+            // this is 6144 (Q) or 512 (K) elements - far cheaper on the host than a
+            // GGML dispatch, and it is 24 of them per token across the 12 attention
+            // layers.
+            if (seqLen <= HostElementwiseMaxRows)
+            {
+                int half = _ropeDimCount / 2;
+                float* p = GetFloatPtr(data);
+                // Same frequency scale the GGML path (below) and both fused kernels
+                // pass. Omitting it here made this fast path rotate on a different
+                // convention from every other path the moment a GGUF declares
+                // rope.scaling.factor != 1, so K rows written before and after a
+                // fallback would disagree. No-op for factor == 1 (today's qwen4exp
+                // GGUFs), which is why it went unnoticed.
+                float freqScale = 1.0f / Config.RopeScale;
+                for (int t = 0; t < seqLen; t++)
+                {
+                    int pos = startPos + t;
+                    float* row = p + (long)t * heads * headDim;
+                    for (int h = 0; h < heads; h++)
+                    {
+                        float* x = row + (long)h * headDim;
+                        for (int i = 0; i < half; i++)
+                        {
+                            float theta = freqScale * pos / MathF.Pow(Config.RopeBase, (2.0f * i) / _ropeDimCount);
+                            float cos = MathF.Cos(theta), sin = MathF.Sin(theta);
+                            float a = x[i], b = x[i + half];
+                            x[i] = a * cos - b * sin;
+                            x[i + half] = a * sin + b * cos;
+                        }
+                    }
+                }
+                InvalidateTensorDeviceCache(data);
+                return data;
+            }
+
+            int rows = seqLen * heads;
+            var positions = new int[rows];
+            for (int s = 0; s < seqLen; s++)
+                for (int h = 0; h < heads; h++)
+                    positions[s * heads + h] = startPos + s;
+
+            using var posTensor = CreateIntTensorOn(data.Storage.Allocator, positions, rows);
+            using var reshaped = data.View(1, seqLen, heads, headDim);
+            Tensor rotated = Ops.RoPEEx(null, reshaped, posTensor, _ropeDimCount, 2, 0,
+                Config.RopeBase, 1.0f / Config.RopeScale, 0.0f, 1.0f, 0.0f, 0.0f);
+            data.Dispose();
+
+            Tensor flat = rotated.View(seqLen, heads * headDim);
+            rotated.Dispose();
+            return flat;
+        }
+    }
+}
diff --git a/TensorSharp.Models/Models/Qwen4Exp/Qwen4ExpModel.PerSeqCache.cs b/TensorSharp.Models/Models/Qwen4Exp/Qwen4ExpModel.PerSeqCache.cs
new file mode 100644
index 00000000..9621a187
--- /dev/null
+++ b/TensorSharp.Models/Models/Qwen4Exp/Qwen4ExpModel.PerSeqCache.cs
@@ -0,0 +1,363 @@
+// Copyright (c) Zhongkai Fu. All rights reserved.
+// https://github.com/zhongkaifu/TensorSharp
+//
+// This file is part of TensorSharp.
+//
+// TensorSharp is licensed under the BSD-3-Clause license found in the LICENSE file in the root directory of this source tree.
+//
+// Per-request KV + recurrent-state holders for the per-sequence fused path
+// (the Qwen3.8-Flash-Next analogue of Qwen35Model.PerSeqCache).
+//
+// qwen4exp keeps three kinds of per-conversation state: the attention KV (and
+// QSA indexer key) caches, the GatedDeltaNet conv + delta-net state, and the
+// PLE n-gram/conv history. The fused span kernel holds the GDN and PLE state
+// in DEVICE buffers that are updated in place inside persisted graphs, keyed
+// on the HOST seed pointers the descriptors carry (see the seq-state map in
+// ggml_ops_qwen4exp.cpp). Giving each in-flight request its own set of host
+// arrays/tensors therefore gives it its own device state and its own cached
+// graphs (the span keys its graph on the descriptor-array addresses), and
+// switching requests is a cheap reference swap - no state download/upload, no
+// graph invalidation.
+//
+// Each sequence then decodes through the proven single-graph fused Forward;
+// concurrency is served by the engine's per-sequence round-robin. The N==1
+// path is untouched: it keeps the model's primary cache, reinstated by
+// RestorePrimaryCache() after any multi-sequence episode.
+using System;
+using System.Collections.Generic;
+using System.Runtime.InteropServices;
+using TensorSharp.GGML;
+using TensorSharp.Runtime;
+using TensorSharp.Runtime.Scheduling;
+
+namespace TensorSharp.Models
+{
+    public partial class Qwen4ExpModel : IBatchedPagedModel
+    {
+        private sealed class Qwen4ExpKvCacheHolder
+        {
+            // Attention KV + QSA indexer keys (null on recurrent layers).
+            public Tensor[] K;
+            public Tensor[] V;
+            public Tensor[] IdxK;
+            public int KvCapacity;
+            public int CacheSeqLen;
+            public bool KvHostStale;
+            // GDN state: host ring (op-by-op path) + the fused path's seed
+            // tensors, whose HOST pointers key the native device-state entries.
+            public float[][] GdnConvState;
+            public int[] GdnConvWriteIdx;
+            public Tensor[] GdnConvStateT;
+            public Tensor[] GdnStateT;
+            // PLE conv history (pinned; seed source + native state key) and the
+            // n-gram token window.
+            public float[] PleConvState;
+            public List<int> PleHistory;
+            public int PleNextPos;
+            public int MropeCacheGap;
+            // Pinned descriptor arrays. Their addresses are the native graph
+            // signature, so per-holder arrays select per-holder graphs.
+            public Qwen4ExpAttnArgs[] AttnArgs;
+            public Qwen4ExpGdnArgs[] GdnArgs;
+            public Qwen4ExpPleArgs[] PleArgs;
+            // Span graph-cache slot base for this holder (multiples of 8).
+            public int SlotBase;
+        }
+
+        private Dictionary<string, Qwen4ExpKvCacheHolder> _fusedHolders;
+        private string _activeFusedKey;
+        private Qwen4ExpKvCacheHolder _primaryHolder;
+        // Span graph-slot base of the ACTIVE holder (0 = primary).
+        private int _seqSlotBase;
+        // Span slot bases in use (multiples of 16 up to the native slot count).
+        private readonly HashSet<int> _usedSlotBases = new() { 0 };
+
+        private int ClaimSlotBase()
+        {
+            for (int b = 16; b <= 112; b += 16)
+            {
+                if (_usedSlotBases.Add(b))
+                    return b;
+            }
+            // More concurrent holders than bases: share the last base. Graph
+            // signatures still keep correctness; alternation costs rebuilds.
+            return 112;
+        }
+
+        private void ReleaseSlotBase(int b)
+        {
+            if (b != 0) _usedSlotBases.Remove(b);
+        }
+
+        /// <summary>The batched paged forward has no qwen4exp implementation (the
+        /// GDN/PLE state has no paged layout); the planner routes concurrency
+        /// through the per-sequence fused path below.</summary>
+        public bool BatchedForwardAvailable => false;
+
+        public IReadOnlyList<float[]> ForwardBatch(BatchedForwardContext ctx)
+            => throw new NotSupportedException(
+                "qwen4exp serves concurrency through per-sequence state holders, not ForwardBatch.");
+
+        /// <summary>Per-sequence holders need the GGML fused span path: it is
+        /// where the per-holder graph/state keying lives. The managed op-by-op
+        /// path shares scratch that is not per-sequence.</summary>
+        public bool SupportsPerSequenceFusedForward =>
+            IsGgmlBackend && _tokenGraphEnabled && !_tokenGraphUnsupported;
+
+        public bool HasFusedSequenceCache(string requestId)
+            => requestId != null && _fusedHolders != null && _fusedHolders.ContainsKey(requestId);
+
+        private Qwen4ExpKvCacheHolder SnapshotActiveCache() => new Qwen4ExpKvCacheHolder
+        {
+            K = _kCache,
+            V = _vCache,
+            IdxK = _idxKCache,
+            KvCapacity = _kvCacheCapacity,
+            CacheSeqLen = _cacheSeqLen,
+            KvHostStale = _kvCacheHostStale,
+            GdnConvState = _gdnConvState,
+            GdnConvWriteIdx = _gdnConvWriteIdx,
+            GdnConvStateT = _gdnConvStateT,
+            GdnStateT = _gdnStateT,
+            PleConvState = _pleConvState,
+            PleHistory = _pleHistory,
+            PleNextPos = _pleNextPos,
+            MropeCacheGap = _mropeCacheGap,
+            AttnArgs = _attnArgs,
+            GdnArgs = _gdnArgs,
+            PleArgs = _pleArgs,
+            SlotBase = _seqSlotBase,
+        };
+
+        private void LoadCacheHolder(Qwen4ExpKvCacheHolder h)
+        {
+            _kCache = h.K;
+            _vCache = h.V;
+            _idxKCache = h.IdxK;
+            _kvCacheCapacity = h.KvCapacity;
+            _cacheSeqLen = h.CacheSeqLen;
+            _kvCacheHostStale = h.KvHostStale;
+            _gdnConvState = h.GdnConvState;
+            _gdnConvWriteIdx = h.GdnConvWriteIdx;
+            _gdnConvStateT = h.GdnConvStateT;
+            _gdnStateT = h.GdnStateT;
+            _pleConvState = h.PleConvState;
+            _pleHistory = h.PleHistory;
+            _pleNextPos = h.PleNextPos;
+            _mropeCacheGap = h.MropeCacheGap;
+            // Null args are lazily rebuilt by Ensure*Args from THIS holder's
+            // tensors, which is what keys the native graphs per holder.
+            _attnArgs = h.AttnArgs;
+            _gdnArgs = h.GdnArgs;
+            _pleArgs = h.PleArgs;
+            _seqSlotBase = h.SlotBase;
+        }
+
+        private Qwen4ExpKvCacheHolder CreateFreshHolder()
+        {
+            int nLayer = Config.NumLayers;
+            DType kvDtype = _kvCacheDtype.ToDType();
+            int cap = _initialKvCacheCapacity > 0 ? _initialKvCacheCapacity : _kvCacheCapacity;
+
+            var k = new Tensor[nLayer];
+            var v = new Tensor[nLayer];
+            var idx = new Tensor[nLayer];
+            var convState = new float[nLayer][];
+            var convWriteIdx = new int[nLayer];
+            var convStateT = new Tensor[nLayer];
+            var stateT = new Tensor[nLayer];
+
+            for (int l = 0; l < nLayer; l++)
+            {
+                if (_isRecurrent[l])
+                {
+                    convState[l] = new float[(long)(_convKernel - 1) * _convDim];
+                    convStateT[l] = new Tensor(_allocator, DType.Float32, _convKernel - 1, _convDim);
+                    Ops.Fill(convStateT[l], 0f);
+                    stateT[l] = new Tensor(_allocator, DType.Float32, _numVHeads, _headVDim, _headKDim);
+                    Ops.Fill(stateT[l], 0f);
+                    continue;
+                }
+                k[l] = new Tensor(_allocator, kvDtype, Config.NumKVHeads, cap, Config.HeadDim);
+                v[l] = new Tensor(_allocator, kvDtype, Config.NumKVHeads, cap, Config.HeadDim);
+                InitializeCacheTensor(k[l]);
+                InitializeCacheTensor(v[l]);
+                if (UsesQsa(l))
+                {
+                    idx[l] = new Tensor(_allocator, DType.Float32, 1, cap, _indexerHeadDim);
+                    InitializeCacheTensor(idx[l]);
+                }
+            }
+
+            float[] pleConv = null;
+            if (_pleHeads > 0)
+                pleConv = GC.AllocateArray<float>(
+                    checked((int)((long)(_pleConvKernel - 1) * _pleNgram * _hcDim)), pinned: true);
+
+            int slotBase = ClaimSlotBase();
+
+            return new Qwen4ExpKvCacheHolder
+            {
+                K = k,
+                V = v,
+                IdxK = idx,
+                KvCapacity = cap,
+                CacheSeqLen = 0,
+                KvHostStale = false,
+                GdnConvState = convState,
+                GdnConvWriteIdx = convWriteIdx,
+                GdnConvStateT = convStateT,
+                GdnStateT = stateT,
+                PleConvState = pleConv,
+                PleHistory = new List<int>(),
+                PleNextPos = 0,
+                MropeCacheGap = 0,
+                AttnArgs = null,
+                GdnArgs = null,
+                PleArgs = null,
+                SlotBase = slotBase,
+            };
+        }
+
+        public bool BindSequenceCache(string requestId)
+        {
+            if (string.IsNullOrEmpty(requestId))
+                throw new ArgumentException("RequestId required", nameof(requestId));
+            _fusedHolders ??= new Dictionary<string, Qwen4ExpKvCacheHolder>(StringComparer.Ordinal);
+
+            if (string.Equals(_activeFusedKey, requestId, StringComparison.Ordinal))
+                return false;
+
+            if (_activeFusedKey == null)
+                _primaryHolder = SnapshotActiveCache();
+            else
+                _fusedHolders[_activeFusedKey] = SnapshotActiveCache();
+
+            bool fresh;
+            if (_fusedHolders.TryGetValue(requestId, out var holder))
+            {
+                fresh = false;
+            }
+            else
+            {
+                holder = CreateFreshHolder();
+                _fusedHolders[requestId] = holder;
+                fresh = true;
+            }
+            LoadCacheHolder(holder);
+            _activeFusedKey = requestId;
+            return fresh;
+        }
+
+        public void AdoptPrimaryCacheToFused(string requestId)
+        {
+            if (string.IsNullOrEmpty(requestId)) return;
+            _fusedHolders ??= new Dictionary<string, Qwen4ExpKvCacheHolder>(StringComparer.Ordinal);
+            if (_activeFusedKey != null) return;
+            if (_fusedHolders.ContainsKey(requestId)) return;
+
+            var holder = SnapshotActiveCache();
+            _fusedHolders[requestId] = holder;
+            _activeFusedKey = requestId;
+
+            _primaryHolder = CreateFreshHolder();
+            // The adopted primary keeps slot base 0; the fresh primary takes a
+            // fused-range base so the two never share span graph slots.
+        }
+
+        public void RestorePrimaryCache()
+        {
+            if (_activeFusedKey == null)
+                return;
+            _fusedHolders[_activeFusedKey] = SnapshotActiveCache();
+            _activeFusedKey = null;
+            if (_primaryHolder != null)
+            {
+                LoadCacheHolder(_primaryHolder);
+                _primaryHolder = null;
+            }
+        }
+
+        public void OnSequenceReleased(string requestId)
+        {
+            if (_fusedHolders == null || string.IsNullOrEmpty(requestId))
+                return;
+            if (!_fusedHolders.TryGetValue(requestId, out var holder))
+                return;
+
+            if (string.Equals(_activeFusedKey, requestId, StringComparison.Ordinal))
+            {
+                _activeFusedKey = null;
+                if (_primaryHolder != null)
+                {
+                    LoadCacheHolder(_primaryHolder);
+                    _primaryHolder = null;
+                }
+            }
+
+            _fusedHolders.Remove(requestId);
+            DisposeHolder(holder);
+            ReleaseSlotBase(holder.SlotBase);
+        }
+
+        private unsafe IntPtr[] HolderStateKeys(Qwen4ExpKvCacheHolder holder)
+        {
+            var keys = new List<IntPtr>();
+            if (holder.GdnConvStateT != null)
+                foreach (var t in holder.GdnConvStateT)
+                    if (t != null) keys.Add((IntPtr)GetFloatPtr(t));
+            if (holder.PleConvState != null && holder.PleConvState.Length > 0)
+                keys.Add(Marshal.UnsafeAddrOfPinnedArrayElement(holder.PleConvState, 0));
+            return keys.ToArray();
+        }
+
+        private void DisposeHolder(Qwen4ExpKvCacheHolder holder)
+        {
+            if (holder == null) return;
+
+            // Free the native device-state entries FIRST (that also drops every
+            // cached graph, so nothing baked can reference the buffers below);
+            // then the tensors.
+            if (IsGgmlBackend)
+                GgmlBasicOps.Qwen4ExpReleaseSeqState(HolderStateKeys(holder));
+
+            void DisposeSet(Tensor[] set)
+            {
+                if (set == null) return;
+                foreach (var t in set)
+                {
+                    if (t == null) continue;
+                    InvalidateTensorDeviceCache(t);
+                    t.Dispose();
+                }
+            }
+            DisposeSet(holder.K);
+            DisposeSet(holder.V);
+            DisposeSet(holder.IdxK);
+            DisposeSet(holder.GdnConvStateT);
+            DisposeSet(holder.GdnStateT);
+        }
+
+        private void DisposeAllFusedHolders()
+        {
+            if (_fusedHolders != null)
+            {
+                foreach (var kv in _fusedHolders)
+                {
+                    if (string.Equals(kv.Key, _activeFusedKey, StringComparison.Ordinal))
+                        continue;
+                    DisposeHolder(kv.Value);
+                }
+                _fusedHolders.Clear();
+                _fusedHolders = null;
+            }
+            if (_primaryHolder != null)
+            {
+                if (_activeFusedKey != null)
+                    DisposeHolder(_primaryHolder);
+                _primaryHolder = null;
+            }
+            _activeFusedKey = null;
+        }
+    }
+}
diff --git a/TensorSharp.Models/Models/Qwen4Exp/Qwen4ExpModel.Ple.cs b/TensorSharp.Models/Models/Qwen4Exp/Qwen4ExpModel.Ple.cs
new file mode 100644
index 00000000..053a436a
--- /dev/null
+++ b/TensorSharp.Models/Models/Qwen4Exp/Qwen4ExpModel.Ple.cs
@@ -0,0 +1,411 @@
+// Copyright (c) Zhongkai Fu. All rights reserved.
+// https://github.com/zhongkaifu/TensorSharp
+//
+// This file is part of TensorSharp.
+//
+// TensorSharp is licensed under the BSD-3-Clause license found in the LICENSE file in the root directory of this source tree.
+//
+// TensorSharp is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the BSD-3-Clause License for more details.
+using System;
+using System.Collections.Generic;
+using System.Diagnostics;
+using System.Threading.Tasks;
+using TensorSharp.Core;
+
+namespace TensorSharp.Models
+{
+    public partial class Qwen4ExpModel
+    {
+        // Tokens preceding the current batch, for the n-gram window. A fresh sequence
+        // (or one whose positions are not contiguous with the last batch) reads EOS,
+        // which is what the reference's zero-padded start gives.
+        // Not readonly: the per-sequence holder swap repoints it (PerSeqCache).
+        private List<int> _pleHistory = new();
+        private int _pleNextPos;
+
+        private void ResetPleHistory()
+        {
+            _pleHistory.Clear();
+            _pleNextPos = 0;
+        }
+
+        /// <summary>
+        /// The n-gram hash. For each token and each n in 2..ngram:
+        ///   <c>mixed = ctx[0]*m[0] ^ ctx[1]*m[1] ^ ... ^ ctx[n-1]*m[n-1]</c>
+        ///   <c>row[h] = mixed % vocab[h] + offset[h]</c>
+        /// Predecessors reset at an EOS: an EOS anywhere in the window hides everything
+        /// at or before it. The token's OWN id does not cut its context - the reference
+        /// takes the last EOS strictly before the position - so a boundary only hides
+        /// tokens from the positions that follow it.
+        ///
+        /// Multipliers reach ~2^45, so this runs in ulong on the host; there is no
+        /// 64-bit integer multiply or xor to do it with on the device.
+        /// </summary>
+        private int[] ComputePleRows(int[] tokens, int startPos)
+        {
+            int n = tokens.Length;
+            int nGram = _pleNgram;
+            int eos = _pleEosTokenId;
+            var rows = new int[(long)_pleHeads * n];
+
+            // Snapshot the incoming history first: reading and updating in one pass
+            // would let an early token in this batch pick up a later one as context.
+            if (_pleNextPos != startPos)
+            {
+                _pleHistory.Clear();
+            }
+            var hist = new List<int>(_pleHistory);
+            while (hist.Count < nGram - 1) hist.Insert(0, eos);
+
+            var ctx = new long[nGram];
+            for (int i = 0; i < n; i++)
+            {
+                int pos = startPos + i;
+                ctx[0] = tokens[i];
+                bool cut = false;
+                for (int s = 1; s < nGram; s++)
+                {
+                    long tok;
+                    if (cut)
+                    {
+                        tok = eos;
+                    }
+                    else
+                    {
+                        int j = i - s;
+                        if (j >= 0)
+                        {
+                            tok = tokens[j];
+                        }
+                        else
+                        {
+                            int back = s - i;                     // positions before this batch
+                            int idx = hist.Count - back;
+                            tok = (back > 0 && idx >= 0 && idx < hist.Count && pos - s >= 0)
+                                ? hist[idx] : eos;
+                        }
+                    }
+                    ctx[s] = tok;
+                    if (tok == eos) cut = true;
+                }
+
+                for (int g = 2; g <= nGram; g++)
+                {
+                    ulong mixed = (ulong)ctx[0] * _pleMultipliers[0];
+                    for (int j = 1; j < g; j++)
+                        mixed ^= (ulong)ctx[j] * _pleMultipliers[j];
+
+                    int baseHead = (g - 2) * _pleHeadsPerNgram;
+                    for (int hh = 0; hh < _pleHeadsPerNgram; hh++)
+                    {
+                        int h = baseHead + hh;
+                        rows[(long)i * _pleHeads + h] =
+                            (int)(mixed % _pleHeadVocabSizes[h] + _pleHeadOffsets[h]);
+                    }
+                }
+            }
+
+            foreach (int t in tokens) _pleHistory.Add(t);
+            if (_pleHistory.Count > nGram - 1)
+                _pleHistory.RemoveRange(0, _pleHistory.Count - (nGram - 1));
+            _pleNextPos = startPos + n;
+
+            return rows;
+        }
+
+        /// <summary>
+        /// The PLE block, run on the layers named by <c>ple.layers</c> (one, in the
+        /// shipped file). It gathers <c>ple_n_heads</c> rows of the ~320 M row n-gram
+        /// table per token, scores them against the residual, and adds both the gated
+        /// value and a dilated depthwise convolution of it back into every stream.
+        ///
+        /// Updates <paramref name="res"/> in place.
+        /// </summary>
+        // The per-token stages parallelize above this batch size; below it the
+        // Parallel.For overhead costs more than it saves (decode is T=1).
+        private const int PleParallelThreshold = 16;
+
+        private unsafe void PleLayer(Tensor res, int[] tokens, int seqLen, int startPos, int il)
+        {
+            int n = Config.HiddenSize;
+            long tStage = Stopwatch.GetTimestamp();
+            int[] rows = ComputePleRows(tokens, startPos);
+            PhaseLog(seqLen, "ple.rows", tStage);
+
+            // Gather: heads laid out slowest, so a token's rows concatenate into one
+            // hidden-wide vector.
+            tStage = Stopwatch.GetTimestamp();
+            var emb = new Tensor(_allocator, DType.Float32, seqLen, n);
+            GatherPleRows(emb, rows, seqLen);
+            PhaseLog(seqLen, "ple.gather", tStage);
+
+            tStage = Stopwatch.GetTimestamp();
+            Tensor key = LinearForward(emb, $"blk.{il}.ple_key.weight");     // [T, hcDim]
+            Tensor value = LinearForward(emb, $"blk.{il}.ple_value.weight"); // [T, n_embd]
+            emb.Dispose();
+            PhaseLog(seqLen, "ple.proj", tStage);
+
+            tStage = Stopwatch.GetTimestamp();
+            GroupedNormInPlace(key, $"blk.{il}.ple_norm_key.weight", seqLen);
+            var query = new Tensor(_allocator, DType.Float32, seqLen, _hcDim);
+            Ops.Copy(query, res);
+            GroupedNormInPlace(query, $"blk.{il}.ple_norm_query.weight", seqLen);
+            PhaseLog(seqLen, "ple.norms", tStage);
+            tStage = Stopwatch.GetTimestamp();
+
+            // Per-stream dot product, then a signed square root before the sigmoid.
+            // Token rows are independent, so a prefill spreads them across cores.
+            var gated = new Tensor(_allocator, DType.Float32, seqLen, _hcDim);
+            {
+                long kA = (long)GetFloatPtr(key);
+                long qA = (long)GetFloatPtr(query);
+                long vA = (long)GetFloatPtr(value);
+                long gA = (long)GetFloatPtr(gated);
+                float invSqrt = 1.0f / MathF.Sqrt(n);
+                int hc = _hc, hcDim = _hcDim;
+                void GateToken(int t)
+                {
+                    float* kr = (float*)kA + (long)t * hcDim;
+                    float* qr = (float*)qA + (long)t * hcDim;
+                    float* vr = (float*)vA + (long)t * n;
+                    float* gr = (float*)gA + (long)t * hcDim;
+                    for (int c = 0; c < hc; c++)
+                    {
+                        float* kc = kr + (long)c * n;
+                        float* qc = qr + (long)c * n;
+                        double dot = 0;
+                        for (int i = 0; i < n; i++) dot += (double)kc[i] * qc[i];
+                        float s = (float)dot * invSqrt;
+                        float mag = MathF.Sqrt(Math.Clamp(MathF.Abs(s), 1e-6f, float.PositiveInfinity));
+                        float signed = MathF.Sign(s) * mag;
+                        float g = 1.0f / (1.0f + MathF.Exp(-signed));
+                        float* gc = gr + (long)c * n;
+                        for (int i = 0; i < n; i++) gc[i] = vr[i] * g;
+                    }
+                }
+                if (seqLen >= PleParallelThreshold) Parallel.For(0, seqLen, GateToken);
+                else for (int t = 0; t < seqLen; t++) GateToken(t);
+                InvalidateTensorDeviceCache(gated);
+            }
+            key.Dispose(); query.Dispose(); value.Dispose();
+            PhaseLog(seqLen, "ple.gate", tStage);
+            tStage = Stopwatch.GetTimestamp();
+
+            // Dilated causal depthwise conv over the gated value, as a sum of shifted
+            // per-channel-scaled copies. History from earlier batches is prepended so a
+            // chunked prefill sees what a single-shot one would.
+            var normed = new Tensor(_allocator, DType.Float32, seqLen, _hcDim);
+            Ops.Copy(normed, gated);
+            GroupedNormInPlace(normed, $"blk.{il}.ple_norm_conv.weight", seqLen);
+
+            var conv = new Tensor(_allocator, DType.Float32, seqLen, _hcDim);
+            {
+                int kern = _pleConvKernel;
+                int dil = _pleNgram;
+                int hist = (kern - 1) * dil;
+                int total = hist + seqLen;
+
+                // One contiguous [hist + T, hcDim] buffer: history rows first, this
+                // batch after, so a chunked prefill sees what a single-shot one would.
+                var padded = new float[(long)total * _hcDim];
+                Array.Copy(_pleConvState, padded, (long)hist * _hcDim);
+                float* np = GetFloatPtr(normed);
+                fixed (float* pad = padded)
+                {
+                    for (int t = 0; t < seqLen; t++)
+                    {
+                        Buffer.MemoryCopy(np + (long)t * _hcDim,
+                            pad + (long)(hist + t) * _hcDim, _hcDim * 4L, _hcDim * 4L);
+                    }
+
+                    long cA = (long)GetFloatPtr(conv);
+                    long wA = (long)GetFloatPtr(_weights[$"blk.{il}.ple_conv1d.weight"]);
+                    long pA = (long)pad;
+                    int hcDim = _hcDim;
+                    // `padded` is fully written above and only read here, so the
+                    // output rows are independent.
+                    void ConvToken(int t)
+                    {
+                        float* o = (float*)cA + (long)t * hcDim;
+                        for (int c = 0; c < hcDim; c++) o[c] = 0f;
+                        for (int kk = 0; kk < kern; kk++)
+                        {
+                            // tap kk reads (kern-1-kk) dilated positions back
+                            int r = hist + t - (kern - 1 - kk) * dil;
+                            if (r < 0) continue;
+                            float* x = (float*)pA + (long)r * hcDim;
+                            float* wp = (float*)wA;
+                            // ple_conv1d is [kernel, channels]: column kk is one
+                            // weight per channel
+                            for (int c = 0; c < hcDim; c++)
+                                o[c] += x[c] * wp[(long)c * kern + kk];
+                        }
+                        for (int c = 0; c < hcDim; c++)
+                            o[c] = o[c] / (1.0f + MathF.Exp(-o[c]));
+                    }
+                    if (seqLen >= PleParallelThreshold) Parallel.For(0, seqLen, ConvToken);
+                    else for (int t = 0; t < seqLen; t++) ConvToken(t);
+                }
+                // Keep the last `hist` rows for the next batch.
+                Array.Copy(padded, (long)(total - hist) * _hcDim, _pleConvState, 0L,
+                    (long)hist * _hcDim);
+                InvalidateTensorDeviceCache(conv);
+            }
+            normed.Dispose();
+            PhaseLog(seqLen, "ple.conv", tStage);
+            tStage = Stopwatch.GetTimestamp();
+
+            // res += gated + conv
+            {
+                long rA = (long)GetFloatPtr(res);
+                long gA = (long)GetFloatPtr(gated);
+                long cA = (long)GetFloatPtr(conv);
+                int hcDim = _hcDim;
+                void AddToken(int t)
+                {
+                    float* rp = (float*)rA + (long)t * hcDim;
+                    float* gp = (float*)gA + (long)t * hcDim;
+                    float* cp = (float*)cA + (long)t * hcDim;
+                    for (int i = 0; i < hcDim; i++) rp[i] += gp[i] + cp[i];
+                }
+                if (seqLen >= PleParallelThreshold) Parallel.For(0, seqLen, AddToken);
+                else for (int t = 0; t < seqLen; t++) AddToken(t);
+                InvalidateTensorDeviceCache(res);
+            }
+            gated.Dispose();
+            conv.Dispose();
+            PhaseLog(seqLen, "ple.add", tStage);
+        }
+
+        /// <summary>
+        /// Gather this batch's n-gram rows out of the PLE table.
+        ///
+        /// <paramref name="dest"/> is [T, hidden], which is the same memory as
+        /// [T * ple_heads, ple_head_dim] - the heads concatenate into one hidden-wide
+        /// row per token, which is what the reference's flatten over the head axis
+        /// gives. The table is ~320 M rows and is read 16 rows per token, so it is
+        /// dequantized row by row from host memory rather than made device-resident.
+        /// </summary>
+        private unsafe void GatherPleRows(Tensor dest, int[] rows, int seqLen)
+        {
+            const string name = "per_layer_token_embd.weight";
+            using Tensor flat = dest.View((long)seqLen * _pleHeads, _pleHeadDim);
+
+            if (_quantWeights.TryGetValue(name, out var qw))
+            {
+                if (!qw.HasHostData)
+                {
+                    throw new InvalidOperationException(
+                        "qwen4exp needs a host copy of the PLE n-gram table: it is gathered "
+                        + $"{_pleHeads} scattered rows per token, which no device get_rows path serves.");
+                }
+                // Row dequants are independent native calls scattered over a ~24 GB
+                // table; a prefill reads tens of thousands of them, so spread the
+                // page-touching and the dequant across cores.
+                long qRowBytes = NativeDequant.RowSize(qw.GgmlType, qw.Ne0);
+                int dim = (int)qw.Ne0;
+                long baseA = (long)qw.Data;
+                long dstA = (long)GetFloatPtr(flat);
+                var ggmlType = qw.GgmlType;
+                void DequantRow(int i)
+                {
+                    NativeDequant.DequantizeToFloat32Native(
+                        ggmlType,
+                        (IntPtr)((byte*)baseA + (long)rows[i] * qRowBytes),
+                        (IntPtr)((float*)dstA + (long)i * dim),
+                        dim);
+                }
+                if (rows.Length >= PleParallelThreshold * _pleHeads)
+                    Parallel.For(0, rows.Length, DequantRow);
+                else
+                    for (int i = 0; i < rows.Length; i++) DequantRow(i);
+                InvalidateTensorDeviceCache(dest);
+                return;
+            }
+
+            Tensor table = _weights[name];
+            float* src = GetFloatPtr(table);
+            float* dst = GetFloatPtr(flat);
+            long rowBytes = _pleHeadDim * sizeof(float);
+            for (int i = 0; i < rows.Length; i++)
+            {
+                Buffer.MemoryCopy(src + (long)rows[i] * _pleHeadDim,
+                    dst + (long)i * _pleHeadDim, rowBytes, rowBytes);
+            }
+            InvalidateTensorDeviceCache(dest);
+        }
+
+        /// <summary>
+        /// Gather the n-gram rows into a raw [T * hidden] float buffer - the span
+        /// uploads it as a graph input.
+        /// </summary>
+        private unsafe void GatherPleRowsRaw(float* dst, int[] rows, int seqLen)
+        {
+            const string name = "per_layer_token_embd.weight";
+            if (_quantWeights.TryGetValue(name, out var qw))
+            {
+                if (!qw.HasHostData)
+                    throw new InvalidOperationException("qwen4exp needs a host copy of the PLE n-gram table.");
+                long qRowBytes = NativeDequant.RowSize(qw.GgmlType, qw.Ne0);
+                int dim = (int)qw.Ne0;
+                long baseA = (long)qw.Data;
+                long dstA = (long)dst;
+                var ggmlType = qw.GgmlType;
+                void DequantRow(int i)
+                {
+                    NativeDequant.DequantizeToFloat32Native(
+                        ggmlType,
+                        (IntPtr)((byte*)baseA + (long)rows[i] * qRowBytes),
+                        (IntPtr)((float*)dstA + (long)i * dim),
+                        dim);
+                }
+                if (rows.Length >= PleParallelThreshold * _pleHeads)
+                    Parallel.For(0, rows.Length, DequantRow);
+                else
+                    for (int i = 0; i < rows.Length; i++) DequantRow(i);
+                return;
+            }
+
+            Tensor table = _weights[name];
+            float* src = GetFloatPtr(table);
+            long rowBytes = _pleHeadDim * sizeof(float);
+            for (int i = 0; i < rows.Length; i++)
+            {
+                Buffer.MemoryCopy(src + (long)rows[i] * _pleHeadDim,
+                    dst + (long)i * _pleHeadDim, rowBytes, rowBytes);
+            }
+        }
+
+        /// <summary>
+        /// RMS norm over each residual stream separately, then an affine weight that
+        /// spans the whole hc*n_embd row - the same shape the hyper-connection mixer
+        /// uses. In place.
+        /// </summary>
+        private unsafe void GroupedNormInPlace(Tensor x, string weightName, int seqLen)
+        {
+            int n = Config.HiddenSize;
+            long pA = (long)GetFloatPtr(x);
+            long wA = (long)GetFloatPtr(_weights[weightName]);
+            float eps = Config.Eps;
+            int hc = _hc, hcDim = _hcDim;
+            void NormToken(int t)
+            {
+                float* row = (float*)pA + (long)t * hcDim;
+                float* w = (float*)wA;
+                for (int c = 0; c < hc; c++)
+                {
+                    float* xc = row + (long)c * n;
+                    double ss = 0;
+                    for (int i = 0; i < n; i++) ss += (double)xc[i] * xc[i];
+                    float inv = (float)(1.0 / Math.Sqrt(ss / n + eps));
+                    int b = c * n;
+                    for (int i = 0; i < n; i++) xc[i] = xc[i] * inv * w[b + i];
+                }
+            }
+            if (seqLen >= PleParallelThreshold) Parallel.For(0, seqLen, NormToken);
+            else for (int t = 0; t < seqLen; t++) NormToken(t);
+            InvalidateTensorDeviceCache(x);
+        }
+    }
+}
diff --git a/TensorSharp.Models/Models/Qwen4Exp/Qwen4ExpModel.Vision.cs b/TensorSharp.Models/Models/Qwen4Exp/Qwen4ExpModel.Vision.cs
new file mode 100644
index 00000000..6950d533
--- /dev/null
+++ b/TensorSharp.Models/Models/Qwen4Exp/Qwen4ExpModel.Vision.cs
@@ -0,0 +1,133 @@
+// Copyright (c) Zhongkai Fu. All rights reserved.
+// https://github.com/zhongkaifu/TensorSharp
+//
+// This file is part of TensorSharp.
+//
+// TensorSharp is licensed under the BSD-3-Clause license found in the LICENSE file in the root directory of this source tree.
+//
+// TensorSharp is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the BSD-3-Clause License for more details.
+using System;
+using System.Collections.Generic;
+using TensorSharp.Core;
+
+namespace TensorSharp.Models
+{
+    public partial class Qwen4ExpModel
+    {
+        // Qwen3.8-Flash-Next ships the same qwen3vl_merger vision tower as
+        // Qwen3.5-VL - identical tensor names, spatial merge, projector MLP and
+        // (in this checkpoint) no active deepstack layers - so the proven Qwen3.5
+        // encoder runs it as is.
+        public Qwen35VisionEncoder VisionEncoder { get; private set; }
+
+        // The GDN recurrence, the PLE conv history and the n-gram history cannot be
+        // rewound to an earlier position, so a cached prefix is only reusable when
+        // the new prompt EXTENDS it exactly - same contract as Qwen3.5. The base
+        // default of true would let a reuse plan truncate mid-conversation and
+        // silently continue from unrewindable state.
+        public override bool SupportsKVCacheTruncation => false;
+
+        public void LoadVisionEncoder(string mmProjPath)
+        {
+            VisionEncoder = new Qwen35VisionEncoder(mmProjPath, _allocator);
+            VisionEncoder.SetHostModel(this);
+        }
+
+        private readonly List<(Tensor Embeddings, int StartPosition)> _visionEmbeddingsList = new();
+
+        public void SetVisionEmbeddings(Tensor visionEmbeddings, int startPosition)
+        {
+            _visionEmbeddingsList.Add((visionEmbeddings, startPosition));
+        }
+
+        /// <summary>
+        /// Replace the text embeddings of the image-pad placeholder tokens with the
+        /// projected vision embeddings. The placeholder token IDS stay in the token
+        /// array, which is what the PLE hash wants: the reference hashes input_ids,
+        /// where image positions hold the placeholder (ple.image_token_id), and this
+        /// checkpoint's placeholder IS that token.
+        /// </summary>
+        private unsafe void InjectVisionEmbeddings(Tensor textEmbeddings, int seqLen)
+        {
+            if (_visionEmbeddingsList.Count == 0)
+                return;
+
+            float* textPtr = GetFloatPtr(textEmbeddings);
+            int dim = Config.HiddenSize;
+            foreach (var (visionEmbeddings, startPos) in _visionEmbeddingsList)
+            {
+                if (visionEmbeddings == null || startPos < 0)
+                    continue;
+
+                int numVisionTokens = (int)visionEmbeddings.Sizes[0];
+                int projDim = (int)visionEmbeddings.Sizes[1];
+
+                if (projDim != dim || startPos + numVisionTokens > seqLen)
+                {
+                    Console.WriteLine($"Warning: qwen4exp vision span [{startPos}, +{numVisionTokens}) x {projDim} " +
+                                      $"does not fit [T={seqLen}, dim={dim}]; skipping injection.");
+                    visionEmbeddings.Dispose();
+                    continue;
+                }
+
+                float* visPtr = GetFloatPtr(visionEmbeddings);
+                long bytes = (long)numVisionTokens * dim * sizeof(float);
+                Buffer.MemoryCopy(visPtr, textPtr + (long)startPos * dim, bytes, bytes);
+                visionEmbeddings.Dispose();
+            }
+
+            _visionEmbeddingsList.Clear();
+            InvalidateTensorDeviceCache(textEmbeddings);
+        }
+
+        // The per-token (T,H,W) IMRoPE position table for the upcoming forward,
+        // flat [3 * seqLen], pushed by the multimodal injector just before each
+        // prefill slice that overlaps an image. Null for text-only forwards, and
+        // for decode steps - which continue on scalar cache positions, exactly as
+        // llama.cpp's mtmd path does.
+        private int[] _pendingMRoPEPositions;
+        private int[] _mropeSections;
+        private int[] _mropePosPinned;      // pinned copy the span kernel reads
+        private int[] _mropeSectionsPinned;
+
+        public void SetMRoPEPositions(int[] flatThw)
+        {
+            _pendingMRoPEPositions = flatThw;
+        }
+
+        // How far the rotary position stream lags the KV cache index. An HxW image
+        // occupies H*W cache rows but only max(H,W) positions (IMRoPE compaction),
+        // so after every image the text positions fall behind the cache. llama.cpp's
+        // mtmd advances n_past by the compacted span the same way. Text-only
+        // forwards and decode subtract this gap from their scalar positions so the
+        // stream stays continuous across turns.
+        private int _mropeCacheGap;
+
+        private void UpdateMropeGap(int startPos, int seqLen)
+        {
+            if (_pendingMRoPEPositions == null || _pendingMRoPEPositions.Length < 3 * seqLen || seqLen <= 0)
+                return;
+            // The last prompt token is text, so its T component is the scalar
+            // position stream; the next token continues at that + 1.
+            int lastT = _pendingMRoPEPositions[3 * (seqLen - 1)];
+            _mropeCacheGap = (startPos + seqLen - 1) - lastT;
+            if (_mropeCacheGap < 0) _mropeCacheGap = 0;
+        }
+
+        private bool EnsureMropeSections()
+        {
+            if (_mropeSectionsPinned != null) return true;
+            if (_mropeSections == null)
+            {
+                _mropeSections = _gguf.GetInt32Array($"{Config.Architecture}.rope.dimension_sections");
+                if (_mropeSections == null || _mropeSections.Length < 4)
+                    return false;
+            }
+            var pinned = GC.AllocateArray<int>(4, pinned: true);
+            for (int i = 0; i < 4; i++) pinned[i] = _mropeSections[i];
+            _mropeSectionsPinned = pinned;
+            return true;
+        }
+    }
+}
diff --git a/TensorSharp.Models/Models/Qwen4Exp/Qwen4ExpModel.cs b/TensorSharp.Models/Models/Qwen4Exp/Qwen4ExpModel.cs
new file mode 100644
index 00000000..481f69b1
--- /dev/null
+++ b/TensorSharp.Models/Models/Qwen4Exp/Qwen4ExpModel.cs
@@ -0,0 +1,634 @@
+// Copyright (c) Zhongkai Fu. All rights reserved.
+// https://github.com/zhongkaifu/TensorSharp
+//
+// This file is part of TensorSharp.
+//
+// TensorSharp is licensed under the BSD-3-Clause license found in the LICENSE file in the root directory of this source tree.
+//
+// TensorSharp is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the BSD-3-Clause License for more details.
+using System;
+using System.Collections.Generic;
+using System.Diagnostics;
+using TensorSharp.Core;
+using TensorSharp.Runtime;
+
+namespace TensorSharp.Models
+{
+    /// <summary>
+    /// Qwen3.8-Flash-Next - GGUF <c>general.architecture = "qwen4exp"</c>, a preview
+    /// of the Qwen4 architecture. 125 B total / 6 B active.
+    ///
+    /// Four things make it different from Qwen 3.5/3.6/3.8, and all four are load
+    /// bearing rather than cosmetic:
+    ///
+    /// 1. <b>Hyper-connections.</b> The residual stream is <c>hc</c> (4) times wide.
+    ///    Every block reads it through a learned gated mixer that collapses the four
+    ///    streams to one, and writes back through a learned per-stream scatter. There
+    ///    is no separate output norm - the final mixer IS the output norm. Same idea
+    ///    as DeepSeek V4's, except the gate is produced by a low-rank pair
+    ///    (hc_dim -> 320 -> hc_dim) rather than a full-rank matrix.
+    ///
+    /// 2. <b>PLE n-gram embeddings.</b> One layer (layer 1 in the shipped file) adds a
+    ///    lookup into a ~320 M row table addressed by a hash of the token's bigram and
+    ///    trigram context. The table is 51 B parameters on its own - most of the file -
+    ///    and is read one row per head per token, so it belongs in host memory.
+    ///
+    /// 3. <b>Qwen Sparse Attention.</b> The full-attention layers (every 4th) score
+    ///    blocks of <c>compress_ratio</c> KV cells with a small indexer, then attend
+    ///    densely to the best <c>indexer_top_k</c> of them. Below the budget the result
+    ///    is exactly dense.
+    ///
+    /// 4. <b>Gated DeltaNet everywhere else</b>, which is the same recurrence Qwen 3.5
+    ///    uses, so it reuses <see cref="GgmlBasicOps.GatedDeltaNetChunked"/>.
+    ///
+    /// The MoE is conventional apart from its size: 512 experts, top 10, plus a gated
+    /// shared expert.
+    ///
+    /// This is the correctness-first managed path. Text only: the vision tower and
+    /// MTP block are not wired up yet, and IMRoPE degenerates to NEOX RoPE when every
+    /// position component is equal, which is the case for text.
+    /// </summary>
+    public partial class Qwen4ExpModel : ModelBase
+    {
+        public const string ArchitectureId = "qwen4exp";
+
+        // ---- hyper-connections ----
+        private int _hc;             // residual stream multiplicity (4)
+        private int _hcDim;          // _hc * hidden
+        private int _hcLowRank;      // gate bottleneck (320)
+
+        // ---- layer typing ----
+        private bool[] _isRecurrent; // GDN rather than full attention
+        private bool[] _isPle;
+        private int[] _compressRatios;
+
+        // ---- QSA indexer ----
+        private int _indexerHeads;
+        private int _indexerHeadDim;
+        private int _indexerTopK;
+
+        // ---- PLE ----
+        private int _pleNgram;
+        private int _pleHeadsPerNgram;
+        private int _pleConvKernel;
+        private int _pleHeads;       // (ngram - 1) * heads_per_ngram
+        private int _pleHeadDim;
+        private int _pleEosTokenId;
+        private int _pleImageTokenId;
+        private ulong[] _pleMultipliers;
+        private ulong[] _pleHeadOffsets;
+        private ulong[] _pleHeadVocabSizes;
+
+        // ---- GDN ----
+        private int _convKernel;
+        private int _headKDim, _headVDim;
+        private int _numKHeads, _numVHeads;
+
+        // ---- MoE ----
+        private int _numExperts, _numExpertsUsed;
+        private int _expertFf, _sharedFf;
+
+        // ---- RoPE ----
+        private int _ropeDimCount;
+        private int[] _ropeSections;
+
+        private float _attnScale;
+
+        public Qwen4ExpModel(string ggufPath, BackendType backend, int tpDegree = 1,
+            ITensorParallelGroup tpGroup = null, int layerSplitDegree = 1)
+            : base(ggufPath, backend, tpDegree, tpGroup, layerSplitDegree)
+        {
+            Config = new ModelConfig { Architecture = ArchitectureId };
+            ParseBaseConfig();
+            ParseQwen4ExpConfig();
+            ParseTokenizer();
+
+            Console.WriteLine($"Model: {ArchitectureId}, Layers={Config.NumLayers}, Hidden={Config.HiddenSize}, " +
+                $"Heads={Config.NumHeads}, KVHeads={Config.NumKVHeads}, HeadDim={Config.HeadDim}, Vocab={Config.VocabSize}");
+            Console.WriteLine($"  hyper-connections: count={_hc} lowRank={_hcLowRank} (residual is {_hcDim} wide)");
+            Console.WriteLine($"  experts={_numExperts} used={_numExpertsUsed} ff={_expertFf} sharedFf={_sharedFf}");
+            Console.WriteLine($"  GDN layers={CountTrue(_isRecurrent)}/{Config.NumLayers}, " +
+                $"QSA indexer heads={_indexerHeads} dim={_indexerHeadDim} topK={_indexerTopK}");
+            if (_pleHeads > 0)
+                Console.WriteLine($"  PLE: layers=[{string.Join(",", PleLayerList())}] ngram={_pleNgram} " +
+                    $"heads={_pleHeads} headDim={_pleHeadDim}");
+
+            LoadWeights();
+            VerifyQwen4ExpTensors();
+            // The layer -> GPU map has to exist BEFORE the preload: that is what
+            // decides which device each weight is uploaded to, and the preload frees
+            // the host copy immediately afterwards so there is no second chance.
+            BuildLayerDeviceMap();
+            PrepareCudaQuantizedWeightsForInference();
+
+            int maxContextLength = ResolveConfiguredContextLength();
+            int initialCacheLength = ResolveInitialCacheAllocationLength(maxContextLength);
+            InitCaches(initialCacheLength, maxContextLength);
+        }
+
+        // ---- layer split ------------------------------------------------------
+        //
+        // Which GPU owns each layer. All zeros on a single-GPU run, which is every
+        // run that does not pass --tp N.
+        private int[] _layerDevice;
+
+        /// <summary>Assign each layer to a GPU as a contiguous run, in pipeline
+        /// order, balancing the bytes that actually become device-resident.
+        ///
+        /// Device 0 is charged the token embedding up front and the LAST device the
+        /// final mixer + LM head, so each takes correspondingly fewer layers than an
+        /// equal share.
+        ///
+        /// NOT modelled: the vision tower. Qwen35VisionEncoder owns its weights in
+        /// its own dictionary, loaded from the mmproj GGUF AFTER this constructor has
+        /// run, and it executes on rank 0 - so a multimodal run puts roughly another
+        /// gigabyte on device 0 that this balance cannot see. TS_Q4E_LAYER_SPLIT
+        /// exists for exactly that case.
+        ///
+        /// See <see cref="PackLayersOntoDevices"/> for why the runs are contiguous.
+        /// </summary>
+        private void BuildLayerDeviceMap()
+        {
+            int n = Config.NumLayers;
+            _layerDevice = new int[n];
+            if (LayerSplitDegree <= 1)
+                return;
+
+            long[] layerBytes = new long[n];
+            long sharedBytes = 0;   // rides on device 0 (embedding, PLE gather source, vision)
+            long headBytes = 0;     // rides on the LAST device (final mixer + LM head)
+            foreach (var kv in _quantWeights)
+            {
+                // Only weights that actually take VRAM count. per_layer_token_embd is
+                // the one that matters: ~24 GB of PLE table that is served by a host
+                // gather and never uploaded, so counting it would push nearly every
+                // layer onto the second GPU to "balance" bytes that are not there.
+                // The stacked experts ARE counted even though they are vetoed from the
+                // eager preload - the span binds them lazily and they are the bulk of
+                // each layer.
+                if (!ShouldPreloadCudaQuantWeightToDevice(kv.Key)
+                    && !_stackedExpertMemberNames.Contains(kv.Key))
+                    continue;
+                // Charge the head group to the device that will actually hold it
+                // (PreloadRankForWeight sends it to the last one), not to device 0.
+                if (IsHeadSpanWeight(kv.Key)) { headBytes += kv.Value.RawBytes; continue; }
+                AccumulateWeightBytes(kv.Key, kv.Value.RawBytes, layerBytes, ref sharedBytes);
+            }
+            foreach (var kv in _weights)
+            {
+                if (IsHeadSpanWeight(kv.Key)) { headBytes += kv.Value.Storage.ByteLength; continue; }
+                AccumulateWeightBytes(kv.Key, kv.Value.Storage.ByteLength, layerBytes, ref sharedBytes);
+            }
+
+            _layerDevice = ParseLayerSplitOverride(Environment.GetEnvironmentVariable("TS_Q4E_LAYER_SPLIT"),
+                                  n, LayerSplitDegree)
+                ?? PackLayersOntoDevices(layerBytes, sharedBytes, headBytes, LayerSplitDegree);
+
+            var counts = new int[LayerSplitDegree];
+            var bytes = new long[LayerSplitDegree];
+            for (int l = 0; l < n; l++) { counts[_layerDevice[l]]++; bytes[_layerDevice[l]] += layerBytes[l]; }
+            bytes[0] += sharedBytes;
+            bytes[LayerSplitDegree - 1] += headBytes;
+            var parts = new System.Collections.Generic.List<string>(LayerSplitDegree);
+            for (int d = 0; d < LayerSplitDegree; d++)
+                parts.Add($"gpu{d}={counts[d]} layers/{bytes[d] / (1024 * 1024)} MB");
+            Console.WriteLine($"  Layer split across {LayerSplitDegree} GPUs: {string.Join(", ", parts)}");
+        }
+
+        /// <summary>
+        /// Assign each layer to a device as a CONTIGUOUS, MONOTONIC run, balancing
+        /// device-resident bytes. Device 0 starts already holding
+        /// <paramref name="sharedBytes"/> (the token embedding) and the LAST device
+        /// starts holding <paramref name="headBytes"/> (the final hyper-connection
+        /// mixer and the LM head, which ride the last span), so each takes
+        /// correspondingly fewer layers than an equal share.
+        ///
+        /// Contiguous and monotonic is a correctness property, not tidiness: each
+        /// device boundary inside a token is a span cut and a residual hand-off, so
+        /// an interleaved assignment would add a seam per interleave, and a
+        /// non-monotonic one would need the residual to travel backwards.
+        /// </summary>
+        internal static int[] PackLayersOntoDevices(long[] layerBytes, long sharedBytes,
+            long headBytes, int deviceCount)
+        {
+            int n = layerBytes.Length;
+            var map = new int[n];
+            if (deviceCount <= 1 || n == 0)
+                return map;
+
+            long total = sharedBytes + headBytes;
+            foreach (long b in layerBytes) total += b;
+            long perDevice = total / deviceCount;
+
+            int dev = 0;
+            long used = sharedBytes;
+            for (int l = 0; l < n; l++)
+            {
+                // Never advance on an empty device: with fewer layers than devices
+                // (or a huge sharedBytes) that would leave a device with no layers
+                // and still produce a seam.
+                bool anyOnThisDevice = l > 0 && map[l - 1] == dev;
+                int layersLeft = n - l;
+                int devicesLeft = deviceCount - dev;
+                if (dev + 1 < deviceCount
+                    && (anyOnThisDevice || dev == 0)
+                    && used + layerBytes[l] > perDevice
+                    && layersLeft > devicesLeft - 1)
+                {
+                    dev++;
+                    // The last device also holds the final mixer + LM head.
+                    used = dev == deviceCount - 1 ? headBytes : 0;
+                }
+                map[l] = dev;
+                used += layerBytes[l];
+            }
+            return map;
+        }
+
+        /// <summary>
+        /// Explicit layer counts per GPU from <c>TS_Q4E_LAYER_SPLIT</c>, e.g.
+        /// <c>TS_Q4E_LAYER_SPLIT=20,28</c> for "20 layers on GPU 0, 28 on GPU 1".
+        ///
+        /// The automatic balance prices weights, and cannot see everything that ends
+        /// up on a device: the vision tower lands on device 0 after the map is built,
+        /// and per-request KV grows on the layer's own device. When a run is close
+        /// enough to the VRAM ceiling for that to matter, this is the override -
+        /// llama.cpp's <c>--tensor-split</c> serves the same purpose.
+        ///
+        /// Returns null (use the automatic balance) when unset; throws on a value
+        /// that cannot be honoured, because silently ignoring an explicit placement
+        /// request is how a run OOMs with the operator believing they fixed it.
+        /// </summary>
+        internal static int[] ParseLayerSplitOverride(string spec, int numLayers, int deviceCount)
+        {
+            if (string.IsNullOrWhiteSpace(spec) || deviceCount <= 1 || numLayers <= 0)
+                return null;
+
+            string[] parts = spec.Split(new[] { ',', ';', ' ' }, StringSplitOptions.RemoveEmptyEntries);
+            if (parts.Length != deviceCount)
+            {
+                throw new ArgumentException(
+                    $"TS_Q4E_LAYER_SPLIT lists {parts.Length} device(s) but {deviceCount} GPU(s) were requested.");
+            }
+
+            var counts = new int[deviceCount];
+            long sum = 0;
+            for (int d = 0; d < deviceCount; d++)
+            {
+                if (!int.TryParse(parts[d].Trim(), out counts[d]) || counts[d] < 1)
+                {
+                    throw new ArgumentException(
+                        $"TS_Q4E_LAYER_SPLIT entry '{parts[d]}' is not a positive layer count. "
+                        + "Every GPU in the split must run at least one layer.");
+                }
+                sum += counts[d];
+            }
+            if (sum != numLayers)
+            {
+                throw new ArgumentException(
+                    $"TS_Q4E_LAYER_SPLIT assigns {sum} layers but the model has {numLayers}.");
+            }
+
+            var map = new int[numLayers];
+            int l = 0;
+            for (int d = 0; d < deviceCount; d++)
+                for (int i = 0; i < counts[d]; i++)
+                    map[l++] = d;
+            Console.WriteLine($"  Layer split: honouring TS_Q4E_LAYER_SPLIT={spec}");
+            return map;
+        }
+
+        /// <summary>Attribute one tensor's bytes to its layer, or to the shared
+        /// (non-layer) pool that rides on device 0.</summary>
+        private static void AccumulateWeightBytes(string name, long bytes, long[] layerBytes, ref long sharedBytes)
+        {
+            if (name != null && name.StartsWith("blk.", StringComparison.Ordinal))
+            {
+                int dot = name.IndexOf('.', 4);
+                if (dot > 4 && int.TryParse(name.AsSpan(4, dot - 4), out int il)
+                    && il >= 0 && il < layerBytes.Length)
+                {
+                    layerBytes[il] += bytes;
+                    return;
+                }
+            }
+            sharedBytes += bytes;
+        }
+
+        /// <inheritdoc/>
+        protected override int PreloadRankForWeight(string weightName)
+        {
+            if (_layerDevice == null || weightName == null) return 0;
+            if (weightName.StartsWith("blk.", StringComparison.Ordinal))
+            {
+                int dot = weightName.IndexOf('.', 4);
+                if (dot <= 4 || !int.TryParse(weightName.AsSpan(4, dot - 4), out int il)) return 0;
+                if (il < 0 || il >= _layerDevice.Length) return 0;
+                return _layerDevice[il];
+            }
+
+            // The final hyper-connection mixer and the LM head are built into the
+            // LAST span (EnsureHeadArgs -> output_hc_down/up, output_hc_norm, and
+            // output.weight or the tied token_embd), so they must be resident on the
+            // last layer's GPU. Getting this wrong is not a slow path: the preload
+            // frees the host copy right after uploading, so a head bound on the wrong
+            // rank reads freed memory and the process dies in kernel warmup.
+            if (IsHeadSpanWeight(weightName))
+                return _layerDevice[_layerDevice.Length - 1];
+            return 0;
+        }
+
+        /// <summary>Weights the head span binds, and therefore weights that live on
+        /// the last device under a layer split.</summary>
+        private bool IsHeadSpanWeight(string weightName)
+        {
+            if (weightName.StartsWith("output", StringComparison.Ordinal))
+                return true;
+            // Tied head: with no output.weight the head matmuls against token_embd.
+            return string.Equals(weightName, "token_embd.weight", StringComparison.Ordinal)
+                && !_quantWeights.ContainsKey("output.weight")
+                && !_weights.ContainsKey("output.weight");
+        }
+
+        /// <summary>
+        /// Keep the host copy of every non-layer weight while a layer split is
+        /// active. They are small next to the model, and it turns "this weight was
+        /// uploaded to the wrong GPU" from a segfault on freed memory into a
+        /// re-upload - worth the RAM as a second lock on the placement above.
+        /// </summary>
+        protected override bool ShouldRetainCudaHostQuantWeight(string weightName)
+        {
+            if (LayerSplitDegree > 1 && weightName != null
+                && !weightName.StartsWith("blk.", StringComparison.Ordinal))
+                return true;
+            return base.ShouldRetainCudaHostQuantWeight(weightName);
+        }
+
+        /// <summary>GPU that runs layer <paramref name="il"/>. 0 without a split.</summary>
+        internal int DeviceForLayer(int il)
+            => _layerDevice != null && il >= 0 && il < _layerDevice.Length ? _layerDevice[il] : 0;
+
+        private static int CountTrue(bool[] a)
+        {
+            int n = 0;
+            foreach (bool b in a) if (b) n++;
+            return n;
+        }
+
+        private List<int> PleLayerList()
+        {
+            var l = new List<int>();
+            for (int i = 0; i < _isPle.Length; i++) if (_isPle[i]) l.Add(i);
+            return l;
+        }
+
+        private void ParseQwen4ExpConfig()
+        {
+            const string a = ArchitectureId;
+            int nLayer = Config.NumLayers;
+
+            Config.NumKVHeads = (int)_gguf.GetUint32($"{a}.attention.head_count_kv");
+
+            // Hyper-connections. Both keys are required: without the count there is no
+            // residual shape and without the low rank there is no gate, and guessing
+            // either produces a model that loads and emits noise.
+            _hc = (int)_gguf.GetUint32($"{a}.hyper_connection.count");
+            _hcLowRank = (int)_gguf.GetUint32($"{a}.hyper_connection.low_rank");
+            if (_hc <= 0 || _hcLowRank <= 0)
+            {
+                throw new NotSupportedException(
+                    $"qwen4exp needs {a}.hyper_connection.count and .low_rank; got {_hc} and {_hcLowRank}.");
+            }
+            _hcDim = _hc * Config.HiddenSize;
+
+            // MoE
+            _numExperts = (int)_gguf.GetUint32($"{a}.expert_count");
+            _numExpertsUsed = (int)_gguf.GetUint32($"{a}.expert_used_count");
+            _expertFf = (int)_gguf.GetUint32($"{a}.expert_feed_forward_length");
+            _sharedFf = (int)_gguf.GetUint32($"{a}.expert_shared_feed_forward_length", (uint)_expertFf);
+
+            // Gated DeltaNet, named with the SSM keys it shares with Qwen 3.5.
+            _convKernel = (int)_gguf.GetUint32($"{a}.ssm.conv_kernel");
+            _headKDim = (int)_gguf.GetUint32($"{a}.ssm.state_size");
+            _headVDim = _headKDim;
+            _numKHeads = (int)_gguf.GetUint32($"{a}.ssm.group_count");
+            _numVHeads = (int)_gguf.GetUint32($"{a}.ssm.time_step_rank");
+
+            // RoPE. dimension_count is the PARTIAL rotary width (64 of a 256-wide head).
+            _ropeDimCount = (int)_gguf.GetUint32($"{a}.rope.dimension_count", (uint)Config.HeadDim);
+            _ropeSections = _gguf.GetInt32Array($"{a}.rope.dimension_sections") ?? new[] { 0, 0, 0, 0 };
+
+            _attnScale = _gguf.GetFloat32($"{a}.attention.scale", 0f);
+            if (_attnScale == 0f)
+                _attnScale = 1.0f / MathF.Sqrt(Config.HeadDim);
+
+            // Indexer (QSA). Absent keys mean a dense model, which still runs.
+            _indexerHeads = (int)_gguf.GetUint32($"{a}.attention.indexer.head_count");
+            _indexerHeadDim = (int)_gguf.GetUint32($"{a}.attention.indexer.key_length");
+            _indexerTopK = (int)_gguf.GetUint32($"{a}.attention.indexer.top_k");
+
+            _compressRatios = new int[nLayer];
+            int[] ratios = _gguf.GetInt32Array($"{a}.attention.compress_ratios");
+            if (ratios != null)
+            {
+                for (int i = 0; i < nLayer && i < ratios.Length; i++)
+                    _compressRatios[i] = ratios[i];
+            }
+
+            // Layer typing: linear (GDN) everywhere except every full_attention_interval-th.
+            _isRecurrent = new bool[nLayer];
+            uint[] recr = _gguf.GetUint32Array($"{a}.attention.recurrent_layers");
+            if (recr != null && recr.Length >= nLayer)
+            {
+                for (int i = 0; i < nLayer; i++) _isRecurrent[i] = recr[i] != 0;
+            }
+            else
+            {
+                int interval = (int)_gguf.GetUint32($"{a}.full_attention_interval", 4);
+                if (interval <= 0) interval = 4;
+                for (int i = 0; i < nLayer; i++) _isRecurrent[i] = ((i + 1) % interval) != 0;
+            }
+
+            // PLE. The whole group is optional; when the layer list is absent every
+            // field stays zero and the model is a plain hyper-connection stack.
+            _isPle = new bool[nLayer];
+            int[] pleLayers = _gguf.GetInt32Array($"{a}.ple.layers");
+            if (pleLayers != null && pleLayers.Length > 0)
+            {
+                foreach (int il in pleLayers)
+                {
+                    if (il < 0 || il >= nLayer)
+                        throw new NotSupportedException($"qwen4exp {a}.ple.layers names layer {il}, outside 0..{nLayer - 1}.");
+                    _isPle[il] = true;
+                }
+
+                _pleNgram = (int)_gguf.GetUint32($"{a}.ple.ngram_size");
+                _pleHeadsPerNgram = (int)_gguf.GetUint32($"{a}.ple.heads_per_ngram");
+                _pleConvKernel = (int)_gguf.GetUint32($"{a}.ple.conv_kernel");
+                _pleEosTokenId = (int)_gguf.GetUint32($"{a}.ple.eos_token_id");
+                _pleImageTokenId = (int)_gguf.GetUint32($"{a}.ple.image_token_id");
+                _pleHeadDim = (int)_gguf.GetUint32($"{a}.embedding_length_per_layer_input");
+
+                _pleHeads = (_pleNgram - 1) * _pleHeadsPerNgram;
+                _pleMultipliers = _gguf.GetUint64Array($"{a}.ple.layer_multipliers");
+                _pleHeadOffsets = _gguf.GetUint64Array($"{a}.ple.head_offsets");
+                _pleHeadVocabSizes = _gguf.GetUint64Array($"{a}.ple.head_vocab_sizes");
+
+                if (_pleNgram < 2 || _pleHeads <= 0 || _pleHeadDim <= 0
+                    || _pleMultipliers == null || _pleMultipliers.Length < _pleNgram
+                    || _pleHeadOffsets == null || _pleHeadOffsets.Length < _pleHeads
+                    || _pleHeadVocabSizes == null || _pleHeadVocabSizes.Length < _pleHeads)
+                {
+                    throw new NotSupportedException(
+                        "qwen4exp declares PLE layers but its n-gram hash constants are missing or short: "
+                        + $"ngram={_pleNgram} heads={_pleHeads} headDim={_pleHeadDim} "
+                        + $"multipliers={_pleMultipliers?.Length ?? 0} offsets={_pleHeadOffsets?.Length ?? 0} "
+                        + $"vocabSizes={_pleHeadVocabSizes?.Length ?? 0}.");
+                }
+
+                // The gather concatenates the heads into one hidden-sized row, so the
+                // two have to agree or every PLE projection is fed a wrong-width input.
+                if (_pleHeads * _pleHeadDim != Config.HiddenSize)
+                {
+                    throw new NotSupportedException(
+                        $"qwen4exp PLE heads*headDim ({_pleHeads}*{_pleHeadDim}) must equal the hidden size "
+                        + $"({Config.HiddenSize}).");
+                }
+            }
+        }
+
+        // Names are checked up front rather than on first use: a missing hyper-connection
+        // tensor otherwise surfaces as a null dereference 40 layers into the first
+        // forward, long after the useful context is gone.
+        private void VerifyQwen4ExpTensors()
+        {
+            var missing = new List<string>();
+            // Stacked expert tensors land in _stackedExpertWeights rather than the
+            // plain weight maps, so all three have to be consulted.
+            void Need(string n)
+            {
+                if (!_weights.ContainsKey(n) && !_quantWeights.ContainsKey(n)
+                    && !_stackedExpertWeights.ContainsKey(n))
+                {
+                    missing.Add(n);
+                }
+            }
+
+            Need("token_embd.weight");
+            Need("output_hc_norm.weight");
+            Need("output_hc_down.weight");
+            Need("output_hc_up.weight");
+            if (_pleHeads > 0) Need("per_layer_token_embd.weight");
+
+            for (int il = 0; il < Config.NumLayers; il++)
+            {
+                Need($"blk.{il}.hc_attn_norm.weight");
+                Need($"blk.{il}.hc_attn_down.weight");
+                Need($"blk.{il}.hc_attn_up.weight");
+                Need($"blk.{il}.hc_attn_inject.weight");
+                Need($"blk.{il}.hc_ffn_norm.weight");
+                Need($"blk.{il}.hc_ffn_down.weight");
+                Need($"blk.{il}.hc_ffn_up.weight");
+                Need($"blk.{il}.hc_ffn_inject.weight");
+
+                if (_isRecurrent[il])
+                {
+                    Need($"blk.{il}.attn_qkv.weight");
+                    Need($"blk.{il}.attn_gate.weight");
+                    Need($"blk.{il}.ssm_conv1d.weight");
+                    Need($"blk.{il}.ssm_norm.weight");
+                    Need($"blk.{il}.ssm_out.weight");
+                }
+                else
+                {
+                    Need($"blk.{il}.attn_q.weight");
+                    Need($"blk.{il}.attn_k.weight");
+                    Need($"blk.{il}.attn_v.weight");
+                    Need($"blk.{il}.attn_output.weight");
+                    Need($"blk.{il}.attn_q_norm.weight");
+                    Need($"blk.{il}.attn_k_norm.weight");
+                    if (UsesQsa(il))
+                    {
+                        Need($"blk.{il}.indexer.q_proj.weight");
+                        Need($"blk.{il}.indexer.k_proj.weight");
+                        Need($"blk.{il}.indexer.q_norm.weight");
+                        Need($"blk.{il}.indexer.k_norm.weight");
+                    }
+                }
+
+                if (_isPle[il])
+                {
+                    Need($"blk.{il}.ple_key.weight");
+                    Need($"blk.{il}.ple_value.weight");
+                    Need($"blk.{il}.ple_norm_key.weight");
+                    Need($"blk.{il}.ple_norm_query.weight");
+                    Need($"blk.{il}.ple_norm_conv.weight");
+                    Need($"blk.{il}.ple_conv1d.weight");
+                }
+
+                Need($"blk.{il}.ffn_gate_inp.weight");
+                Need($"blk.{il}.ffn_down_exps.weight");
+                // llama.cpp accepts either a fused ffn_gate_up_exps or the separate
+                // pair, so which one a given quant ships is not knowable up front.
+                bool hasSplit = _stackedExpertWeights.ContainsKey($"blk.{il}.ffn_gate_exps.weight")
+                                && _stackedExpertWeights.ContainsKey($"blk.{il}.ffn_up_exps.weight");
+                bool hasFused = _stackedExpertWeights.ContainsKey($"blk.{il}.ffn_gate_up_exps.weight");
+                if (!hasSplit && !hasFused)
+                    missing.Add($"blk.{il}.ffn_gate_exps.weight (or ffn_gate_up_exps.weight)");
+                else if (!hasSplit && il == 0)
+                    _fusedGateUpExperts = true;
+                Need($"blk.{il}.ffn_gate_inp_shexp.weight");
+                Need($"blk.{il}.ffn_gate_shexp.weight");
+                Need($"blk.{il}.ffn_up_shexp.weight");
+                Need($"blk.{il}.ffn_down_shexp.weight");
+            }
+
+            if (missing.Count > 0)
+            {
+                string head = string.Join(", ", missing.GetRange(0, Math.Min(8, missing.Count)));
+                throw new NotSupportedException(
+                    $"qwen4exp GGUF is missing {missing.Count} expected tensor(s): {head}"
+                    + (missing.Count > 8 ? ", ..." : "") + ".");
+            }
+        }
+
+        /// <summary>
+        /// Keep the PLE n-gram table OUT of VRAM.
+        ///
+        /// It is the single largest tensor in the file - ~320 M rows, tens of GB, most
+        /// of the checkpoint - and the forward never multiplies by it: every use is a
+        /// gather of <c>ple_n_heads</c> scattered rows per token, which
+        /// <see cref="GatherPleRows"/> serves by dequantizing from the retained host
+        /// copy. Preloading it spent the VRAM that the rest of the model and the KV
+        /// cache need, and on a 96 GB card that was the difference between running and
+        /// failing to allocate 0.4 MiB.
+        /// </summary>
+        protected override bool ShouldPreloadCudaQuantWeightToDevice(string weightName)
+        {
+            if (string.Equals(weightName, "per_layer_token_embd.weight", StringComparison.Ordinal))
+                return false;
+
+            // The routed experts reach the device ONCE, as the stacked tensor the
+            // batched mul_mat_id kernel binds. ModelBase also exposes each expert as
+            // its own 2D view, and preloading those put a second full copy of all 512
+            // experts per layer in VRAM - 47 GB of it - which is what made the batched
+            // MoE fail to allocate its graph. The host views stay mapped for the
+            // portable per-expert path on backends without the stacked kernel.
+            if (_stackedExpertMemberNames.Contains(weightName))
+                return false;
+
+            return base.ShouldPreloadCudaQuantWeightToDevice(weightName);
+        }
+
+        /// <summary>A full-attention layer runs QSA when it has both an indexer and a
+        /// block size; either missing means plain dense attention.</summary>
+        /// <summary>The GGUF stacks gate and up into one expert tensor rather than
+        /// shipping them separately.</summary>
+        private bool _fusedGateUpExperts;
+
+        private bool UsesQsa(int il)
+            => !_isRecurrent[il] && _indexerHeads > 0 && _indexerHeadDim > 0
+               && _compressRatios != null && il < _compressRatios.Length && _compressRatios[il] > 0;
+    }
+}
diff --git a/TensorSharp.Models/Models/MuseGlimmer/DFlashConfig.cs b/TensorSharp.Models/Speculative/DFlashConfig.cs
similarity index 53%
rename from TensorSharp.Models/Models/MuseGlimmer/DFlashConfig.cs
rename to TensorSharp.Models/Speculative/DFlashConfig.cs
index 3e458723..3a5470be 100644
--- a/TensorSharp.Models/Models/MuseGlimmer/DFlashConfig.cs
+++ b/TensorSharp.Models/Speculative/DFlashConfig.cs
@@ -15,9 +15,31 @@ namespace TensorSharp.Models
     /// <summary>
     /// Hyper-parameters of a DFlash speculative drafter GGUF
     /// (general.architecture = "dflash"), the block drafter that ships alongside a
-    /// Muse-Glimmer target. Parsed from the drafter file only; every tensor the
-    /// drafter needs beyond its own blocks (token_embd, output) is borrowed from
-    /// the target model.
+    /// target model (Muse-Glimmer, Qwen 3.8, ...). Parsed from the drafter file
+    /// only; every tensor the drafter needs beyond its own blocks (token_embd,
+    /// output) is borrowed from the target model.
+    ///
+    /// TWO GENERATIONS share this architecture id, and which one a file is comes
+    /// from the keys it carries, not from its name:
+    ///
+    ///   DFlash  - plain block diffusion. Each block position's token is the
+    ///             argmax of the target LM head over that position's draft hidden
+    ///             state, chosen INDEPENDENTLY of its neighbours.
+    ///   DFlash2 - the same backbone plus two additions
+    ///             (<see cref="HasConv"/> / <see cref="HasSelector"/>):
+    ///               * a grouped dynamic depthwise K-tap convolution wrapped
+    ///                 around every attention and every FFN sublayer
+    ///                 (dflash.conv_kernel_size / dflash.conv_group_size), whose
+    ///                 taps are produced per token by a projection of that
+    ///                 sublayer's own input and masked at the block boundary, and
+    ///               * a CANDIDATE SELECTOR (dflash.selector_rank /
+    ///                 dflash.selector_top_k): instead of an independent argmax,
+    ///                 the top-K candidates of adjacent positions are scored
+    ///                 pairwise through two low-rank [vocab, r] codebooks and the
+    ///                 block is read off as a walk through that lattice, so a
+    ///                 position's token is conditioned on the one before it.
+    ///             Both are no-ops when their keys are absent, which is what lets
+    ///             one code path serve both generations.
     ///
     /// DFlash runs three passes (llama.cpp src/models/dflash.cpp):
     ///   A. ENCODE  - the target's per-layer INPUT residuals at
@@ -98,6 +120,68 @@ public sealed class DFlashConfig
         /// past the anchor.</summary>
         public int MaskTokenId { get; private set; } = -1;
 
+        /// <summary>dflash.conv_kernel_size: taps of the grouped dynamic
+        /// convolution (2 for the shipped DFlash2 drafters). 0 = no convolution,
+        /// i.e. a first-generation DFlash file.</summary>
+        public int ConvKernelSize { get; private set; }
+
+        /// <summary>dflash.conv_group_size: channels sharing one dynamic tap
+        /// coefficient (16). The STATIC part of the kernel is per channel
+        /// (blk.N.attn_conv_base); only the per-token delta is per group.</summary>
+        public int ConvGroupSize { get; private set; }
+
+        /// <summary>dflash.selector_rank: width of the two [vocab, r] transition
+        /// codebooks. 0 = no selector (plain DFlash).</summary>
+        public int SelectorRank { get; private set; }
+
+        /// <summary>dflash.selector_top_k: candidates kept per block position
+        /// before the lattice walk (16).</summary>
+        public int SelectorTopK { get; private set; }
+
+        /// <summary>Channel groups of the dynamic convolution
+        /// (<see cref="HiddenSize"/> / <see cref="ConvGroupSize"/>).</summary>
+        public int ConvNumGroups => ConvGroupSize > 0 ? HiddenSize / ConvGroupSize : 0;
+
+        /// <summary>Columns of one conv_proj output row: both sides (the sublayer's
+        /// input and its output) x taps x groups.</summary>
+        public int ConvProjOutSize => 2 * ConvKernelSize * ConvNumGroups;
+
+        /// <summary>True when this drafter wraps its sublayers in the DFlash2
+        /// grouped dynamic convolution.</summary>
+        public bool HasConv => ConvKernelSize > 0 && ConvGroupSize > 0;
+
+        /// <summary>True when this drafter picks its block through the DFlash2
+        /// candidate-selector lattice instead of a per-position argmax.</summary>
+        public bool HasSelector => SelectorRank > 0 && SelectorTopK > 0;
+
+        /// <summary>
+        /// dflash.logit_scale: the multiplier the TARGET applies to its LM-head
+        /// output. Only the DFlash2 selector needs it, and it needs it badly: the
+        /// lattice ADDS the unary logit to a transition score, so an unscaled unary
+        /// term is simply the wrong size and swamps the transition it is supposed to
+        /// compete with. (Plain DFlash takes an argmax, which is invariant under a
+        /// positive scale, which is why llama.cpp's DFlash graph can ignore it and
+        /// why this key only appears on a DFlash2 file whose target has one -
+        /// Muse-Glimmer's 0.196, against Qwen 3.8's absent = 1.0.)
+        /// </summary>
+        public float LogitScale { get; private set; } = 1f;
+
+        /// <summary>dflash.final_logit_softcapping: the target's tanh softcap, applied
+        /// to the selector's unary term after <see cref="LogitScale"/>. 0 = none.
+        /// Same reasoning as <see cref="LogitScale"/>: monotonic, so it cannot change
+        /// which candidates the top-k picks, but it very much changes how they weigh
+        /// against the transition scores.</summary>
+        public float FinalLogitSoftcap { get; private set; }
+
+        /// <summary>True when the selector's unary term needs the target's logit
+        /// transform applied before it enters the lattice.</summary>
+        public bool HasUnaryLogitTransform => LogitScale != 1f || FinalLogitSoftcap > 0f;
+
+        /// <summary>True for a second-generation drafter (either extension
+        /// present). Descriptive only - every code path keys on
+        /// <see cref="HasConv"/> / <see cref="HasSelector"/> individually.</summary>
+        public bool IsDFlash2 => HasConv || HasSelector;
+
         /// <summary>Width of one encoder input row = TargetLayerIds.Length * HiddenSize
         /// (33280 for the Muse-Glimmer drafter). This is what the model reports as
         /// ISpeculativeModel.SpecFeatureSize.</summary>
@@ -154,12 +238,51 @@ public static DFlashConfig FromGguf(GgufFile gguf)
                 // A missing key yields uint.MaxValue, which casts to -1 and is
                 // rejected by ValidateSelfConsistent below.
                 MaskTokenId = (int)gguf.GetUint32("tokenizer.ggml.mask_token_id", uint.MaxValue),
+                // DFlash2 extensions. Absent in a first-generation file, and a zero
+                // there means the same thing as absent: the feature is off.
+                ConvKernelSize = (int)gguf.GetUint32($"{ArchName}.conv_kernel_size", 0),
+                ConvGroupSize = (int)gguf.GetUint32($"{ArchName}.conv_group_size", 0),
+                SelectorRank = (int)gguf.GetUint32($"{ArchName}.selector_rank", 0),
+                SelectorTopK = (int)gguf.GetUint32($"{ArchName}.selector_top_k", 0),
+                LogitScale = gguf.GetFloat32($"{ArchName}.logit_scale", 1f),
+                FinalLogitSoftcap = gguf.GetFloat32($"{ArchName}.final_logit_softcapping", 0f),
             };
 
+            cfg.ApplyDiagnosticOverrides();
             cfg.ValidateSelfConsistent();
             return cfg;
         }
 
+        /// <summary>
+        /// TS_DFLASH_SELECTOR=0 / TS_DFLASH_CONV=0 turn off a DFlash2 extension and
+        /// run the checkpoint as a first-generation DFlash drafter.
+        ///
+        /// DIAGNOSTIC ONLY. Neither is a supported way to run the model: the weights
+        /// were trained WITH both, so switching one off changes what the drafter
+        /// predicts. What they are for is attribution - how much of the acceptance
+        /// rate comes from the selector, how much from the convolution, and what each
+        /// costs per draft step - which is otherwise unanswerable without a second
+        /// checkpoint.
+        /// </summary>
+        private void ApplyDiagnosticOverrides()
+        {
+            if (IsDisabled("TS_DFLASH_SELECTOR") && HasSelector)
+            {
+                Console.WriteLine("  DFlash: TS_DFLASH_SELECTOR=0 - drafting with per-position argmax instead of the candidate lattice (diagnostic).");
+                SelectorRank = 0;
+                SelectorTopK = 0;
+            }
+            if (IsDisabled("TS_DFLASH_CONV") && HasConv)
+            {
+                Console.WriteLine("  DFlash: TS_DFLASH_CONV=0 - drafting without the grouped dynamic convolution (diagnostic).");
+                ConvKernelSize = 0;
+                ConvGroupSize = 0;
+            }
+        }
+
+        private static bool IsDisabled(string envVar)
+            => string.Equals(Environment.GetEnvironmentVariable(envVar), "0", StringComparison.Ordinal);
+
         private void ValidateSelfConsistent()
         {
             if (NumLayers <= 0)
@@ -183,6 +306,38 @@ private void ValidateSelfConsistent()
                 throw new InvalidOperationException($"{ArchName}.attention.sliding_window {SlidingWindow} must be positive.");
             if (MaskTokenId < 0)
                 throw new InvalidOperationException("tokenizer.ggml.mask_token_id is missing from the DFlash GGUF; the block draft has no mask id to fill its slots with.");
+            // Both DFlash2 extensions are all-or-nothing: half a convolution, or a
+            // rank with no top-k, describes a file we cannot execute, and quietly
+            // ignoring the half that IS present would draft from a different model
+            // than the one that was trained.
+            if ((ConvKernelSize > 0) != (ConvGroupSize > 0))
+            {
+                throw new InvalidOperationException(
+                    "DFlash grouped convolution needs conv_kernel_size and conv_group_size together "
+                    + $"(got {ConvKernelSize} / {ConvGroupSize}).");
+            }
+            if (HasConv)
+            {
+                if (HiddenSize % ConvGroupSize != 0)
+                {
+                    throw new InvalidOperationException(
+                        $"{ArchName}.conv_group_size {ConvGroupSize} must divide embedding_length {HiddenSize}.");
+                }
+                if (ConvKernelSize > BlockSize)
+                {
+                    // Tap t is masked out for the first t positions of a block, so a
+                    // kernel wider than the block silently reduces to a narrower one -
+                    // a shape the trained weights were never meant to run in.
+                    throw new NotSupportedException(
+                        $"{ArchName}.conv_kernel_size {ConvKernelSize} exceeds block_size {BlockSize}.");
+                }
+            }
+            if ((SelectorRank > 0) != (SelectorTopK > 0))
+            {
+                throw new InvalidOperationException(
+                    "DFlash selector needs selector_rank and selector_top_k together "
+                    + $"(got {SelectorRank} / {SelectorTopK}).");
+            }
             if (SwaPattern.Length != NumLayers)
             {
                 throw new InvalidOperationException(
@@ -202,8 +357,13 @@ private void ValidateSelfConsistent()
         }
 
         public override string ToString()
-            => $"dflash(layers={NumLayers}, hidden={HiddenSize}, ffn={IntermediateSize}, heads={NumHeads}/{NumKVHeads}x{HeadDim}, " +
+            => $"{(IsDFlash2 ? "dflash2" : "dflash")}(layers={NumLayers}, hidden={HiddenSize}, " +
+               $"ffn={IntermediateSize}, heads={NumHeads}/{NumKVHeads}x{HeadDim}, " +
                $"block={BlockSize}, drafts={MaxDraftTokens}, swa={SlidingWindow}, ring={RingRows}, " +
-               $"targets=[{string.Join(",", TargetLayerIds)}], feature={FeatureSize}, mask={MaskTokenId})";
+               $"targets=[{string.Join(",", TargetLayerIds)}], feature={FeatureSize}, mask={MaskTokenId}" +
+               (HasConv ? $", conv={ConvKernelSize}x{ConvGroupSize}({ConvNumGroups}g)" : string.Empty) +
+               (HasSelector ? $", selector=r{SelectorRank}/k{SelectorTopK}" : string.Empty) +
+               (LogitScale != 1f ? $", logit_scale={LogitScale:G6}" : string.Empty) +
+               (FinalLogitSoftcap > 0f ? $", softcap={FinalLogitSoftcap:G6}" : string.Empty) + ")";
     }
 }
diff --git a/TensorSharp.Models/Models/MuseGlimmer/MuseGlimmerModel.DFlash.Fused.cs b/TensorSharp.Models/Speculative/ModelBase.DFlash.Fused.cs
similarity index 66%
rename from TensorSharp.Models/Models/MuseGlimmer/MuseGlimmerModel.DFlash.Fused.cs
rename to TensorSharp.Models/Speculative/ModelBase.DFlash.Fused.cs
index 163d3e8e..fb277c8f 100644
--- a/TensorSharp.Models/Models/MuseGlimmer/MuseGlimmerModel.DFlash.Fused.cs
+++ b/TensorSharp.Models/Speculative/ModelBase.DFlash.Fused.cs
@@ -29,7 +29,7 @@ namespace TensorSharp.Models
     /// plus a 3.2 M-element scan -- and that readback is a large share of its per-step
     /// cost. Two 16-element tensors carry everything the executor needs.
     /// </summary>
-    public partial class MuseGlimmerModel
+    public abstract partial class ModelBase
     {
         private sealed class DFlashArrays
         {
@@ -61,6 +61,23 @@ private sealed class DFlashArrays
             public long LmHeadNe0, LmHeadNe1, LmHeadBytes;
 
             public int RingRows;
+
+            /// <summary>DFlash2 grouped convolution: per layer, the static base
+            /// kernel [hidden, taps, 2] and the projection that produces both taps
+            /// from the sublayer input. Null on a first-generation drafter.</summary>
+            public IntPtr[] AttnConvBase, FfnConvBase;
+            public IntPtr[] AttnConvProj, FfnConvProj;
+            public int[] AttnConvProjType, FfnConvProjType;
+            public long[] AttnConvProjNe0, AttnConvProjNe1, AttnConvProjBytes;
+            public long[] FfnConvProjNe0, FfnConvProjNe1, FfnConvProjBytes;
+
+            /// <summary>DFlash2 candidate selector: the hidden projection and the
+            /// two [vocab, rank] transition codebooks. Zero when absent.</summary>
+            public IntPtr SelHidden, SelPred, SelSucc;
+            public int SelHiddenType, SelPredType, SelSuccType;
+            public long SelHiddenNe0, SelHiddenNe1, SelHiddenBytes;
+            public long SelPredNe0, SelPredNe1, SelPredBytes;
+            public long SelSuccNe0, SelSuccNe1, SelSuccBytes;
         }
 
         private DFlashArrays _dflashArrays;
@@ -120,6 +137,11 @@ private void EnsureDFlashRingHostSynchronized()
             (_backend == BackendType.GgmlCuda || _backend == BackendType.GgmlVulkan ||
              _backend == BackendType.GgmlMetal);
 
+        /// <summary>A quantized weight the fused kernels can bind by cache key.
+        /// (ModelBase has no such helper; the trunk kernels each roll their own.)</summary>
+        private bool TryDFlashQuant(string name, out QuantizedWeight qw)
+            => _quantWeights.TryGetValue(name, out qw) && qw != null && qw.CacheKey != IntPtr.Zero;
+
         private unsafe void BuildDFlashArrays()
         {
             _dflashFusedProbed = true;
@@ -146,14 +168,22 @@ private unsafe void BuildDFlashArrays()
                 RingK = new IntPtr[n], RingV = new IntPtr[n],
                 RingRows = _dflashRingRows,
             };
+            if (cfg.HasConv)
+            {
+                a.AttnConvBase = new IntPtr[n]; a.FfnConvBase = new IntPtr[n];
+                a.AttnConvProj = new IntPtr[n]; a.FfnConvProj = new IntPtr[n];
+                a.AttnConvProjType = new int[n]; a.FfnConvProjType = new int[n];
+                a.AttnConvProjNe0 = new long[n]; a.AttnConvProjNe1 = new long[n]; a.AttnConvProjBytes = new long[n];
+                a.FfnConvProjNe0 = new long[n]; a.FfnConvProjNe1 = new long[n]; a.FfnConvProjBytes = new long[n];
+            }
 
             for (int l = 0; l < n; l++)
             {
                 string[] wn = _dflashLayerNames[l];
-                if (!TryQuant(wn[DfAttnQ], out var q) || !TryQuant(wn[DfAttnK], out var k)
-                    || !TryQuant(wn[DfAttnV], out var v) || !TryQuant(wn[DfAttnOutput], out var o)
-                    || !TryQuant(wn[DfFfnGate], out var gate) || !TryQuant(wn[DfFfnUp], out var up)
-                    || !TryQuant(wn[DfFfnDown], out var down))
+                if (!TryDFlashQuant(wn[DfAttnQ], out var q) || !TryDFlashQuant(wn[DfAttnK], out var k)
+                    || !TryDFlashQuant(wn[DfAttnV], out var v) || !TryDFlashQuant(wn[DfAttnOutput], out var o)
+                    || !TryDFlashQuant(wn[DfFfnGate], out var gate) || !TryDFlashQuant(wn[DfFfnUp], out var up)
+                    || !TryDFlashQuant(wn[DfFfnDown], out var down))
                 {
                     Console.WriteLine($"  DFlash fused drafter disabled: layer {l} has a non-quantized projection.");
                     return;
@@ -180,10 +210,29 @@ private unsafe void BuildDFlashArrays()
 
                 a.RingK[l] = TensorComputePrimitives.GetStoragePointer(_dflashRingK[l]);
                 a.RingV[l] = TensorComputePrimitives.GetStoragePointer(_dflashRingV[l]);
+
+                if (!cfg.HasConv)
+                    continue;
+                if (!TryDFlashQuant(wn[DfAttnConvProj], out var acp) || !TryDFlashQuant(wn[DfFfnConvProj], out var fcp))
+                {
+                    Console.WriteLine($"  DFlash fused drafter disabled: layer {l} has a non-quantized conv projection.");
+                    return;
+                }
+                if (!_weights.TryGetValue(wn[DfAttnConvBase], out var acb) || !_weights.TryGetValue(wn[DfFfnConvBase], out var fcb))
+                {
+                    Console.WriteLine($"  DFlash fused drafter disabled: layer {l} is missing a conv base kernel.");
+                    return;
+                }
+                a.AttnConvBase[l] = (IntPtr)GetFloatPtr(acb);
+                a.FfnConvBase[l] = (IntPtr)GetFloatPtr(fcb);
+                a.AttnConvProj[l] = acp.CacheKey; a.AttnConvProjType[l] = acp.GgmlType;
+                a.AttnConvProjNe0[l] = acp.Ne0; a.AttnConvProjNe1[l] = acp.Ne1; a.AttnConvProjBytes[l] = acp.RawBytes;
+                a.FfnConvProj[l] = fcp.CacheKey; a.FfnConvProjType[l] = fcp.GgmlType;
+                a.FfnConvProjNe0[l] = fcp.Ne0; a.FfnConvProjNe1[l] = fcp.Ne1; a.FfnConvProjBytes[l] = fcp.RawBytes;
             }
 
             string fcName = DFlashConfig.WeightPrefix + "fc.weight";
-            if (!TryQuant(fcName, out var fc))
+            if (!TryDFlashQuant(fcName, out var fc))
             {
                 Console.WriteLine("  DFlash fused drafter disabled: the encoder projection is not quantized.");
                 return;
@@ -201,7 +250,8 @@ private unsafe void BuildDFlashArrays()
 
             // Both borrowed from the target, exactly as llama.cpp's dflash graph does
             // through cparams.ctx_other -- the drafter owns neither.
-            if (!TryQuant("token_embd.weight", out var tok) || !TryQuant(TargetOutputWeightName, out var head))
+            if (!TryDFlashQuant("token_embd.weight", out var tok)
+                || !TryDFlashQuant(DFlashTargetOutputWeightName, out var head))
             {
                 Console.WriteLine("  DFlash fused drafter disabled: the target embedding/LM head is not quantized.");
                 return;
@@ -209,8 +259,27 @@ private unsafe void BuildDFlashArrays()
             a.TokEmbd = tok.CacheKey; a.TokEmbdType = tok.GgmlType; a.TokEmbdNe0 = tok.Ne0; a.TokEmbdNe1 = tok.Ne1; a.TokEmbdBytes = tok.RawBytes;
             a.LmHead = head.CacheKey; a.LmHeadType = head.GgmlType; a.LmHeadNe0 = head.Ne0; a.LmHeadNe1 = head.Ne1; a.LmHeadBytes = head.RawBytes;
 
+            if (cfg.HasSelector)
+            {
+                if (!TryDFlashQuant(DFlashConfig.WeightPrefix + "selector_hidden.weight", out var selH)
+                    || !TryDFlashQuant(DFlashConfig.WeightPrefix + "selector_predecessor.weight", out var selP)
+                    || !TryDFlashQuant(DFlashConfig.WeightPrefix + "selector_successor.weight", out var selS))
+                {
+                    Console.WriteLine("  DFlash fused drafter disabled: a selector table is not quantized.");
+                    return;
+                }
+                a.SelHidden = selH.CacheKey; a.SelHiddenType = selH.GgmlType;
+                a.SelHiddenNe0 = selH.Ne0; a.SelHiddenNe1 = selH.Ne1; a.SelHiddenBytes = selH.RawBytes;
+                a.SelPred = selP.CacheKey; a.SelPredType = selP.GgmlType;
+                a.SelPredNe0 = selP.Ne0; a.SelPredNe1 = selP.Ne1; a.SelPredBytes = selP.RawBytes;
+                a.SelSucc = selS.CacheKey; a.SelSuccType = selS.GgmlType;
+                a.SelSuccNe0 = selS.Ne0; a.SelSuccNe1 = selS.Ne1; a.SelSuccBytes = selS.RawBytes;
+            }
+
             _dflashArrays = a;
-            Console.WriteLine($"  DFlash fused drafter armed ({n} draft layers, ring {a.RingRows} rows, on-device top-1).");
+            Console.WriteLine($"  DFlash fused drafter armed ({n} draft layers, ring {a.RingRows} rows, "
+                + (cfg.HasSelector ? "on-device lattice" : "on-device top-1")
+                + (cfg.HasConv ? $", conv {cfg.ConvKernelSize}x{cfg.ConvGroupSize}" : string.Empty) + ").");
         }
 
         private DFlashArrays GetDFlashArrays()
@@ -302,7 +371,8 @@ private bool TryFusedDFlashInject(float[] hRows, int rowOffset, int n, int start
             if (!ok)
                 return false;
 
-            _dflashRingFilled = Math.Max(_dflashRingFilled, startPos + n);
+            // See DFlashInjectKv: the frontier must be able to move backwards.
+            _dflashRingFilled = startPos + n;
             _dflashRingHostStale = true;
             return true;
         }
@@ -332,6 +402,21 @@ private int TryFusedDFlashDraftBlock(int anchorToken, int position, int b, int[]
                 positions[i] = position + i;
             }
 
+            // The selector's lattice comes back instead of an argmax: k*k floats per
+            // transition plus one k-wide row for the anchor's own position, which is
+            // ~7 KB against the 12.9 MB a [vocab, b] readback would cost. The walk
+            // itself is gamma steps over k candidates and belongs on the host.
+            int gamma = b - 1;
+            int k = cfg.SelectorTopK;
+            if (cfg.HasSelector)
+            {
+                long need = (long)k + (long)k * k * Math.Max(0, gamma - 1);
+                if (_dflashSelScores == null || _dflashSelScores.LongLength < need)
+                    _dflashSelScores = new float[need];
+                if (_dflashSelCand == null || _dflashSelCand.Length < gamma * k)
+                    _dflashSelCand = new int[gamma * k];
+            }
+
             bool ok = GgmlBasicOps.DFlashDraftBlock(
                 ids, b, positions,
                 cfg.NumLayers, cfg.HiddenSize, cfg.HeadDim, cfg.NumHeads, cfg.NumKVHeads, _dflashRingRows,
@@ -351,10 +436,22 @@ private int TryFusedDFlashDraftBlock(int anchorToken, int position, int b, int[]
                 a.OutNorm,
                 a.TokEmbd, a.TokEmbdType, a.TokEmbdNe0, a.TokEmbdNe1, a.TokEmbdBytes,
                 a.LmHead, a.LmHeadType, a.LmHeadNe0, a.LmHeadNe1, a.LmHeadBytes,
-                Config.VocabSize, _dflashDraftIds, _dflashDraftConf);
+                Config.VocabSize, _dflashDraftIds, _dflashDraftConf,
+                cfg.HasConv ? cfg.ConvKernelSize : 0, cfg.ConvGroupSize, cfg.ConvNumGroups,
+                a.AttnConvBase, a.AttnConvProj, a.AttnConvProjType, a.AttnConvProjNe0, a.AttnConvProjNe1, a.AttnConvProjBytes,
+                a.FfnConvBase, a.FfnConvProj, a.FfnConvProjType, a.FfnConvProjNe0, a.FfnConvProjNe1, a.FfnConvProjBytes,
+                cfg.HasSelector ? cfg.SelectorRank : 0, cfg.HasSelector ? cfg.SelectorTopK : 0,
+                cfg.LogitScale, cfg.FinalLogitSoftcap,
+                a.SelHidden, a.SelHiddenType, a.SelHiddenNe0, a.SelHiddenNe1, a.SelHiddenBytes,
+                a.SelPred, a.SelPredType, a.SelPredNe0, a.SelPredNe1, a.SelPredBytes,
+                a.SelSucc, a.SelSuccType, a.SelSuccNe0, a.SelSuccNe1, a.SelSuccBytes,
+                cfg.HasSelector ? _dflashSelScores : null, cfg.HasSelector ? _dflashSelCand : null);
             if (!ok)
                 return -1;
 
+            if (cfg.HasSelector)
+                return DFlashWalkLattice(gamma, k, _dflashSelScores, _dflashSelCand, draftOut, confOut);
+
             // Row 0 is the anchor's own prediction; plain DFlash discards it.
             int drafted = b - 1;
             for (int i = 0; i < drafted; i++)
@@ -365,5 +462,40 @@ private int TryFusedDFlashDraftBlock(int anchorToken, int position, int b, int[]
             }
             return drafted;
         }
+
+        private float[] _dflashSelScores;
+        private int[] _dflashSelCand;
+
+        /// <summary>
+        /// The greedy walk through the transition lattice the kernel produced.
+        /// <paramref name="scores"/> holds the anchor row first (k floats: position
+        /// 0's scores against the verified anchor) and then one [k(pred), k(cand)]
+        /// matrix per following position, candidate-fastest. Each step takes the
+        /// argmax over candidates of the row selected by the previous step's choice,
+        /// which is exactly what the reference implementation's temperature-0 path
+        /// does.
+        /// </summary>
+        internal static int DFlashWalkLattice(int gamma, int k, float[] scores, int[] cand, int[] draftOut, float[] confOut)
+        {
+            float[] row = new float[k];
+
+            int chosen = 0;
+            for (int e = 0; e < gamma; e++)
+            {
+                long baseIdx = e == 0
+                    ? 0
+                    : (long)k + ((long)(e - 1) * k + chosen) * k;
+                Array.Copy(scores, baseIdx, row, 0, k);
+
+                chosen = 0;
+                for (int c = 1; c < k; c++)
+                    if (row[c] > row[chosen]) chosen = c;
+
+                draftOut[e] = cand[e * k + chosen];
+                if (confOut != null && e < confOut.Length)
+                    confOut[e] = DFlashSoftmaxAt(row, chosen);
+            }
+            return gamma;
+        }
     }
 }
diff --git a/TensorSharp.Models/Speculative/ModelBase.DFlash.cs b/TensorSharp.Models/Speculative/ModelBase.DFlash.cs
new file mode 100644
index 00000000..94d2572b
--- /dev/null
+++ b/TensorSharp.Models/Speculative/ModelBase.DFlash.cs
@@ -0,0 +1,1382 @@
+// Copyright (c) Zhongkai Fu. All rights reserved.
+// https://github.com/zhongkaifu/TensorSharp
+//
+// This file is part of TensorSharp.
+//
+// TensorSharp is licensed under the BSD-3-Clause license found in the LICENSE file in the root directory of this source tree.
+//
+// TensorSharp is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the BSD-3-Clause License for more details.
+//
+// ---------------------------------------------------------------------------
+// The DFlash / DFlash2 block drafter, for ANY target model.
+//
+// DFlash is a BLOCK drafter: one forward pass proposes the whole speculative
+// window, so it plugs into the shared draft/verify/rollback core through
+// IDraftHead.DraftBlock instead of the per-token DraftStep, and it consumes a
+// WIDE hidden row - the concatenated per-layer input residuals of the target
+// layers its encoder was trained on (dflash.target_layers).
+//
+// Nothing here is model-specific. A target model gets DFlash by doing three
+// things and nothing else:
+//
+//   1. call LoadDFlashDraftWeights(path) once the trunk weights are loaded,
+//   2. capture the residual entering each dflash.target_layers layer during
+//      its SpecForward (DFlashCaptureFeature does the row packing), and
+//   3. forward the IDraftHead members to DFlashPropose / DFlashCommit.
+//
+// Three passes, transcribed from llama.cpp src/models/dflash.cpp and, for the
+// DFlash2 extensions, from sglang python/sglang/srt/models/dflash.py:
+//
+//   PASS A -- encoder
+//       feat = concat(target input residual of layers dflash.target_layers)
+//       g    = rmsnorm(fc @ feat, enc.output_norm, eps)      [1 row per position]
+//
+//   PASS B -- KV injection
+//       K = rope_neox(headnorm(attn_k @ g, attn_k_norm), target position)
+//       V = attn_v @ g                                       (no norm, no rope)
+//       ring[pos % ringRows] <- K, V                         per draft layer
+//       No Q, no attention, no FFN -- and, in DFlash2, no convolution either:
+//       the encoder output is the trained KV source for context positions.
+//
+//   PASS C -- block draft
+//       ids  = [anchor, MASK x (block_size-1)] at positions p .. p+B-1
+//       inpL = target token_embd[ids]                        (no embedding scale)
+//       per draft layer:
+//           h = rmsnorm(inpL, attn_norm)
+//           [DFlash2] h, kOut = conv.prepare(h)
+//           attention over [ring window | this block's own B keys] --
+//               NON-CAUSAL inside the block, SWA-masked against the ring
+//           [DFlash2] attn = conv.finish(attn, kOut)
+//           inpL += attn ; f = rmsnorm(inpL, ffn_norm)
+//           [DFlash2] f, kOut = conv.prepare(f)
+//           f = SwiGLU(f) ; [DFlash2] f = conv.finish(f, kOut)
+//           inpL += f
+//       cur = rmsnorm(inpL, output_norm)
+//       DFlash : logits = target_lm_head @ cur; each row's argmax is its draft
+//       DFlash2: the candidate selector picks the block as a walk (below)
+//
+// The drafter's logits get NEITHER the target's logit_scale NOR its tanh
+// softcap: llama.cpp's dflash graph ends at build_lora_mm(output, cur). argmax
+// is invariant to both, but the softmax CONFIDENCE is not, so the per-position
+// acceptance probabilities handed to the executor are the softmax of the RAW
+// drafter logits.
+//
+// ---------------------------------------------------------------------------
+// DFlash2, in one paragraph each.
+//
+// GROUPED DYNAMIC CONVOLUTION. Every attention and every FFN sublayer is
+// wrapped: one projection of the sublayer's INPUT produces both the kernel that
+// convolves that input and the kernel that convolves the sublayer's output.
+// The kernel is a K-tap depthwise filter whose static part is per CHANNEL
+// (blk.N.*_conv_base, initialised to the identity) and whose per-token delta is
+// per GROUP of conv_group_size channels, so tap t of channel c at block
+// position r is (base[t][c] + delta[r][t][c / group_size]) and multiplies
+// x[r-t][c] - masked to zero for r < t, i.e. the filter never reaches across a
+// block boundary. It is the piece that gives a block-diffusion draft a local
+// left-to-right signal without a second forward pass.
+//
+// CANDIDATE SELECTOR. Plain DFlash takes each block position's argmax over the
+// target vocabulary INDEPENDENTLY, which is exactly the weakness of block
+// diffusion: position i+1 is chosen without knowing what position i chose. The
+// selector keeps the top selector_top_k candidates per position and scores every
+// (predecessor, candidate) pair through two low-rank [vocab, r] codebooks:
+//
+//     score[e][p][c] = unary[e][c]
+//                    + < A[pred[e][p]] * (P h_e) , B[cand[e][c]] >
+//
+// where A / B are selector_predecessor / selector_successor, P is
+// selector_hidden, pred[0] is the verified anchor token and pred[e] is
+// cand[e-1]. The block is then read off as a greedy walk through that lattice,
+// which is one extra small matmul per position and no extra draft forward.
+// ---------------------------------------------------------------------------
+using System;
+using System.Collections.Generic;
+using TensorSharp;
+using TensorSharp.GGML;
+using TensorSharp.Runtime;
+
+namespace TensorSharp.Models
+{
+    public abstract partial class ModelBase
+    {
+        // Per-layer weight-name slots (index into _dflashLayerNames[il]).
+        private protected const int DfAttnNorm = 0;
+        private protected const int DfAttnQ = 1;
+        private protected const int DfAttnK = 2;
+        private protected const int DfAttnV = 3;
+        private protected const int DfAttnQNorm = 4;
+        private protected const int DfAttnKNorm = 5;
+        private protected const int DfAttnOutput = 6;
+        private protected const int DfFfnNorm = 7;
+        private protected const int DfFfnGate = 8;
+        private protected const int DfFfnUp = 9;
+        private protected const int DfFfnDown = 10;
+        // DFlash2 only; absent (null) in a first-generation drafter.
+        private protected const int DfAttnConvBase = 11;
+        private protected const int DfAttnConvProj = 12;
+        private protected const int DfFfnConvBase = 13;
+        private protected const int DfFfnConvProj = 14;
+        private protected const int DfLayerNameCount = 15;
+
+        /// <summary>
+        /// Default prompt-prefill chunk the speculative executor should use, and
+        /// the reason it is not the model's own.
+        ///
+        /// This value drives the TRUNK forward, not only the drafter, so it decides
+        /// how many full target forwards a prompt costs. It used to be 128, chosen
+        /// purely to bound the host-side capture buffer (one Muse-Glimmer feature
+        /// row is 33280 floats = 130 KB), on the assumption that "the trunk's
+        /// per-chunk fixed overhead is small next to a 128-row forward". Measured
+        /// against llama.cpp it is not: the extra cost per chunk is a FLAT ~60 ms
+        /// from 2K to 128K of context (a whole-trunk graph rebuild plus a DFlash
+        /// host round trip), against ~13 ms of useful work in a 128-row chunk. At a
+        /// 124K prompt that is 980 trunk forwards instead of 61 - 58 s added to a
+        /// 112 s prefill, which was the whole of the 0.69x-versus-llama.cpp DFlash
+        /// prefill ratio.
+        ///
+        /// 1024 keeps one 130 MB host buffer (PrefillStep shifts the pairing in
+        /// place instead of keeping a second one) and removes 87% of the extra
+        /// chunks. Override with TS_DFLASH_PREFILL_CHUNK.
+        /// </summary>
+        private const int DFlashPrefillChunkDefault = 1024;
+
+        private int _dflashPrefillChunk;
+
+        /// <summary>
+        /// Trunk-imposed ceiling on one speculative prefill chunk. A model whose
+        /// own forward refuses a batch wider than some window (Muse-Glimmer's SWA
+        /// ring) narrows the chunk here; the default is "no trunk limit".
+        /// </summary>
+        private protected virtual int DFlashTrunkPrefillChunkCap => int.MaxValue;
+
+        private int ResolveDFlashPrefillChunk()
+        {
+            int chunk = DFlashPrefillChunkDefault;
+            string raw = Environment.GetEnvironmentVariable("TS_DFLASH_PREFILL_CHUNK");
+            if (!string.IsNullOrWhiteSpace(raw) && int.TryParse(raw, out int parsed) && parsed > 0)
+                chunk = parsed;
+
+            // Never exceed what either ring can absorb in one forward: the drafter
+            // would alias two live positions onto one ring slot, and a trunk with a
+            // window limit throws outright.
+            int draftCap = _dflash != null ? _dflash.RingRows - _dflash.BlockSize - 1 : chunk;
+            chunk = Math.Min(chunk, Math.Max(1, draftCap));
+            chunk = Math.Min(chunk, Math.Max(1, DFlashTrunkPrefillChunkCap));
+            return Math.Max(1, chunk);
+        }
+
+        /// <summary>
+        /// The prefill chunk a DFlash-drafted model reports through
+        /// ISpeculativeTarget.SpecPrefillChunkSize. Resolved lazily and once: the
+        /// caps come off the drafter's ring and the trunk's own window, so
+        /// answering before the drafter exists would cache a wrong value.
+        /// </summary>
+        private protected int DFlashPrefillChunkSize
+        {
+            get
+            {
+                if (_dflash == null)
+                    return 0;
+                if (_dflashPrefillChunk <= 0)
+                    _dflashPrefillChunk = ResolveDFlashPrefillChunk();
+                return _dflashPrefillChunk;
+            }
+        }
+
+        private protected DFlashConfig _dflash;
+        private bool _hasDFlash;
+
+        /// <summary>target layer index -> its column block in a feature row, or -1.</summary>
+        private protected int[] _dflashCaptureSlot;
+
+        /// <summary>Per draft layer, the weight names of that block.</summary>
+        private string[][] _dflashLayerNames;
+
+        /// <summary>The drafter's own KV ring, [numKVHeads, ringRows, headDim] per
+        /// draft layer, indexed by (absolute position % ringRows).</summary>
+        private Tensor[] _dflashRingK;
+        private Tensor[] _dflashRingV;
+        private int _dflashRingRows;
+
+        /// <summary>True when a usable DFlash drafter is attached to this model.</summary>
+        public bool HasDFlash => _hasDFlash;
+
+        /// <summary>The attached drafter's hyper-parameters, or null.</summary>
+        public DFlashConfig DFlashSettings => _dflash;
+
+        // ====================================================================
+        // construction / loading
+        // ====================================================================
+
+        /// <summary>
+        /// Loads the DFlash drafter GGUF and attaches it to this target model. Its
+        /// tensors are merged into the shared weight dictionaries under the
+        /// "dflash." prefix so the existing matmul/norm machinery serves them (the
+        /// same trick Gemma4Model.LoadMtpDraftTensors uses with "mtp."); the drafter
+        /// borrows the TARGET's token_embd.weight and output.weight, which the file
+        /// does not carry.
+        /// </summary>
+        public void LoadDFlashDraftWeights(string ggufPath)
+        {
+            if (string.IsNullOrEmpty(ggufPath) || !System.IO.File.Exists(ggufPath))
+                throw new System.IO.FileNotFoundException("DFlash drafter GGUF not found.", ggufPath);
+
+            using var draft = new GgufFile(ggufPath);
+            var cfg = DFlashConfig.FromGguf(draft);
+
+            if (cfg.HiddenSize != Config.HiddenSize)
+            {
+                throw new InvalidOperationException(
+                    $"DFlash embedding_length {cfg.HiddenSize} != target hidden size {Config.HiddenSize}.");
+            }
+            foreach (int lid in cfg.TargetLayerIds)
+            {
+                if (lid < 0 || lid >= Config.NumLayers)
+                {
+                    throw new InvalidOperationException(
+                        $"DFlash target layer {lid} is outside the target's {Config.NumLayers} layers.");
+                }
+            }
+            if (cfg.BlockSize > cfg.RingRows)
+                throw new InvalidOperationException($"DFlash block_size {cfg.BlockSize} exceeds the ring ({cfg.RingRows} rows).");
+
+            LoadDFlashDraftTensors(draft);
+
+            _dflash = cfg;
+            _dflashLayerNames = new string[cfg.NumLayers][];
+            for (int il = 0; il < cfg.NumLayers; il++)
+            {
+                string p = $"{DFlashConfig.WeightPrefix}blk.{il}.";
+                var names = new string[DfLayerNameCount];
+                names[DfAttnNorm] = p + "attn_norm.weight";
+                names[DfAttnQ] = p + "attn_q.weight";
+                names[DfAttnK] = p + "attn_k.weight";
+                names[DfAttnV] = p + "attn_v.weight";
+                names[DfAttnQNorm] = p + "attn_q_norm.weight";
+                names[DfAttnKNorm] = p + "attn_k_norm.weight";
+                names[DfAttnOutput] = p + "attn_output.weight";
+                names[DfFfnNorm] = p + "ffn_norm.weight";
+                names[DfFfnGate] = p + "ffn_gate.weight";
+                names[DfFfnUp] = p + "ffn_up.weight";
+                names[DfFfnDown] = p + "ffn_down.weight";
+                if (cfg.HasConv)
+                {
+                    names[DfAttnConvBase] = p + "attn_conv_base";
+                    names[DfAttnConvProj] = p + "attn_conv_proj.weight";
+                    names[DfFfnConvBase] = p + "ffn_conv_base";
+                    names[DfFfnConvProj] = p + "ffn_conv_proj.weight";
+                }
+                _dflashLayerNames[il] = names;
+            }
+
+            if (!VerifyDFlashTensors(out string missing))
+            {
+                Console.WriteLine($"  DFlash drafter GGUF loaded but '{missing}' is missing; DFlash drafting disabled.");
+                _dflash = null;
+                _dflashLayerNames = null;
+                return;
+            }
+
+            _dflashCaptureSlot = new int[Config.NumLayers];
+            for (int l = 0; l < Config.NumLayers; l++)
+                _dflashCaptureSlot[l] = -1;
+            for (int i = 0; i < cfg.TargetLayerIds.Length; i++)
+                _dflashCaptureSlot[cfg.TargetLayerIds[i]] = i;
+
+            _dflashRingRows = cfg.RingRows;
+            _dflashRingK = new Tensor[cfg.NumLayers];
+            _dflashRingV = new Tensor[cfg.NumLayers];
+            for (int il = 0; il < cfg.NumLayers; il++)
+            {
+                _dflashRingK[il] = new Tensor(_allocator, DType.Float32, cfg.NumKVHeads, _dflashRingRows, cfg.HeadDim);
+                _dflashRingV[il] = new Tensor(_allocator, DType.Float32, cfg.NumKVHeads, _dflashRingRows, cfg.HeadDim);
+                // Unconditional zero fill (not InitializeCacheTensor, which skips
+                // GgmlCuda): the ring is only ever read over positions that have
+                // been written, but a finite ring keeps a mis-sized window from
+                // silently producing NaNs instead of failing loudly.
+                Ops.Fill(_dflashRingK[il], 0f);
+                Ops.Fill(_dflashRingV[il], 0f);
+            }
+
+            _hasDFlash = true;
+
+            long ringBytes = 2L * cfg.NumLayers * cfg.NumKVHeads * _dflashRingRows * cfg.HeadDim * sizeof(float);
+            Console.WriteLine($"  DFlash drafter ready: {cfg}");
+            Console.WriteLine($"  DFlash KV ring: {_dflashRingRows} rows x {cfg.NumLayers} layers ({ringBytes / (1024 * 1024)} MB F32)");
+        }
+
+        /// <summary>
+        /// Merges every tensor of the drafter GGUF into the shared weight
+        /// dictionaries under the "dflash." prefix. Byte-for-byte the same shape as
+        /// Gemma4Model.LoadMtpDraftTensors (which uses "mtp."), minus the
+        /// converter-spelling normalization DFlash does not need: the drafter's
+        /// tensor names are already the final ones.
+        /// </summary>
+        private unsafe void LoadDFlashDraftTensors(GgufFile draft)
+        {
+            foreach (var kv in draft.Tensors)
+            {
+                var info = kv.Value;
+                string name = DFlashConfig.WeightPrefix + info.Name;
+                long byteCount = draft.GetTensorByteCount(info);
+
+                if (IsQuantizedLinearWeight(info))
+                {
+                    if (IsGgmlBackend)
+                        EnsureQuantBackendAvailable();
+                    IntPtr ptr = QuantizedWeight.AllocateBuffer(byteCount);
+                    draft.ReadTensorDataToNative(info, ptr, byteCount);
+                    _quantWeights[name] = new QuantizedWeight(ptr, byteCount, (int)info.Type, (long)info.Shape[0], (long)info.Shape[1]);
+                }
+                else
+                {
+                    long numElements = info.NumElements;
+                    long[] tsShape = new long[info.Shape.Length];
+                    for (int i = 0; i < info.Shape.Length; i++)
+                        tsShape[i] = (long)info.Shape[info.Shape.Length - 1 - i];
+
+                    var tensor = new Tensor(_allocator, DType.Float32, tsShape);
+                    IntPtr destPtr = TensorComputePrimitives.GetStoragePointer(tensor);
+                    if (info.Type == GgmlTensorType.F32)
+                    {
+                        draft.ReadTensorDataToFloat32Native(info, destPtr, numElements);
+                    }
+                    else
+                    {
+                        IntPtr tempPtr = QuantizedWeight.AllocateBuffer(byteCount);
+                        try
+                        {
+                            draft.ReadTensorDataToNative(info, tempPtr, byteCount);
+                            NativeDequant.DequantizeToFloat32Native((int)info.Type, tempPtr, destPtr, numElements);
+                        }
+                        finally
+                        {
+                            QuantizedWeight.FreeBuffer(tempPtr);
+                        }
+                    }
+                    _weights[name] = tensor;
+                }
+            }
+        }
+
+        /// <summary>True when <paramref name="name"/> resolves to something
+        /// <see cref="LinearForward"/> can multiply by.</summary>
+        private bool HasDFlashLinear(string name)
+            => _quantWeights.ContainsKey(name) || _weights.ContainsKey(name);
+
+        private bool VerifyDFlashTensors(out string missing)
+        {
+            string[] globals =
+            {
+                DFlashConfig.WeightPrefix + "fc.weight",
+                DFlashConfig.WeightPrefix + "enc.output_norm.weight",
+                DFlashConfig.WeightPrefix + "output_norm.weight",
+            };
+            foreach (string g in globals)
+            {
+                bool ok = g.EndsWith("norm.weight", StringComparison.Ordinal)
+                    ? _weights.ContainsKey(g)
+                    : HasDFlashLinear(g);
+                if (!ok) { missing = g; return false; }
+            }
+
+            for (int il = 0; il < _dflash.NumLayers; il++)
+            {
+                string[] n = _dflashLayerNames[il];
+                foreach (int slot in new[] { DfAttnNorm, DfAttnQNorm, DfAttnKNorm, DfFfnNorm })
+                {
+                    if (!_weights.ContainsKey(n[slot])) { missing = n[slot]; return false; }
+                }
+                foreach (int slot in new[] { DfAttnQ, DfAttnK, DfAttnV, DfAttnOutput, DfFfnGate, DfFfnUp, DfFfnDown })
+                {
+                    if (!HasDFlashLinear(n[slot])) { missing = n[slot]; return false; }
+                }
+                if (_dflash.HasConv)
+                {
+                    // The conv base kernels are small F32 tensors, the projections
+                    // ordinary linears - both are required together, because a
+                    // half-built convolution silently changes the model.
+                    foreach (int slot in new[] { DfAttnConvBase, DfFfnConvBase })
+                    {
+                        if (!_weights.ContainsKey(n[slot])) { missing = n[slot]; return false; }
+                        long need = 2L * _dflash.ConvKernelSize * _dflash.HiddenSize;
+                        if (_weights[n[slot]].ElementCount() != need)
+                        {
+                            missing = $"{n[slot]} (expected {need} elements for "
+                                    + $"2 x taps {_dflash.ConvKernelSize} x hidden {_dflash.HiddenSize})";
+                            return false;
+                        }
+                    }
+                    foreach (int slot in new[] { DfAttnConvProj, DfFfnConvProj })
+                    {
+                        if (!HasDFlashLinear(n[slot])) { missing = n[slot]; return false; }
+                    }
+                }
+            }
+
+            if (_dflash.HasSelector)
+            {
+                foreach (string s in new[]
+                {
+                    DFlashConfig.WeightPrefix + "selector_hidden.weight",
+                    DFlashConfig.WeightPrefix + "selector_predecessor.weight",
+                    DFlashConfig.WeightPrefix + "selector_successor.weight",
+                })
+                {
+                    if (!HasDFlashLinear(s)) { missing = s; return false; }
+                }
+            }
+
+            // The drafter has no LM head of its own: it borrows the target's.
+            if (!HasDFlashLinear(DFlashTargetOutputWeightName)) { missing = DFlashTargetOutputWeightName; return false; }
+            if (!HasDFlashLinear("token_embd.weight")) { missing = "token_embd.weight"; return false; }
+
+            missing = null;
+            return true;
+        }
+
+        /// <summary>The target's LM head, which the drafter borrows. A model that
+        /// keeps its head somewhere other than "output.weight"/"token_embd.weight"
+        /// overrides this.</summary>
+        private protected virtual string DFlashTargetOutputWeightName
+            => HasDFlashLinear("output.weight") ? "output.weight" : "token_embd.weight";
+
+        /// <summary>Called from the owning model's Dispose. The drafter's weights
+        /// live in the shared dictionaries and are released by
+        /// <see cref="Dispose"/>; only the rings are ours.</summary>
+        private protected void DisposeDFlash()
+        {
+            if (_dflashRingK != null)
+                foreach (var t in _dflashRingK) t?.Dispose();
+            if (_dflashRingV != null)
+                foreach (var t in _dflashRingV) t?.Dispose();
+            _dflashRingK = null;
+            _dflashRingV = null;
+            _hasDFlash = false;
+        }
+
+        // ====================================================================
+        // IDraftHead surface (models forward to these)
+        // ====================================================================
+
+        /// <summary>Copies one target layer's input residual into its column block
+        /// of the caller's feature rows. <paramref name="slot"/> comes from
+        /// <see cref="_dflashCaptureSlot"/>.</summary>
+        private protected unsafe void DFlashCaptureFeature(Tensor hidden, int slot, int seqLen, float[] hAllOut, bool lastRowOnly)
+        {
+            int hs = Config.HiddenSize;
+            int feat = _dflash.FeatureSize;
+            long rowBytes = (long)hs * sizeof(float);
+            float* src = GetFloatPtr(hidden);
+            fixed (float* dst0 = hAllOut)
+            {
+                if (lastRowOnly)
+                {
+                    Buffer.MemoryCopy(src + (long)(seqLen - 1) * hs, dst0 + (long)slot * hs, rowBytes, rowBytes);
+                    return;
+                }
+                for (int r = 0; r < seqLen; r++)
+                    Buffer.MemoryCopy(src + (long)r * hs, dst0 + (long)r * feat + (long)slot * hs, rowBytes, rowBytes);
+            }
+        }
+
+        /// <summary>
+        /// Replays committed trunk positions through the drafter so its KV ring
+        /// tracks the real context. Row k of <paramref name="hRows"/> is the feature
+        /// row of the token PRECEDING tokens[k] -- i.e. of absolute position
+        /// <paramref name="startPos"/> + k - 1, which is exactly the position whose
+        /// drafter key it writes (hence the -1, as in DeepSeek4Model.DraftCatchUp).
+        /// </summary>
+        private protected void DFlashCommit(int[] tokens, float[] hRows, int startPos)
+        {
+            RequireDFlash();
+            if (tokens == null || tokens.Length == 0 || hRows == null)
+                return;
+            DFlashCatchUp(hRows, tokens.Length, startPos - 1);
+        }
+
+        /// <summary>Encodes <paramref name="rows"/> feature rows whose first row is
+        /// at absolute position <paramref name="firstPos"/> and writes the resulting
+        /// keys/values into the ring. Rows before position 0 (the zeroed "hidden
+        /// state of the token before the prompt") are skipped, and rows older than
+        /// the ring modulus are dropped -- dropping them is what keeps the ring
+        /// writes collision-free.</summary>
+        private void DFlashCatchUp(float[] hRows, int rows, int firstPos)
+        {
+            if (rows <= 0)
+                return;
+            int skip = firstPos < 0 ? Math.Min(-firstPos, rows) : 0;
+            rows -= skip;
+            firstPos += skip;
+            if (rows <= 0)
+                return;
+
+            int keep = Math.Min(rows, _dflashRingRows);
+            int drop = rows - keep;
+
+            DFlashEncodeAndInject(hRows, skip + drop, keep, firstPos + drop);
+        }
+
+        /// <summary>Encode + ring injection, fused into one GGML graph when the backend
+        /// supports it, otherwise the per-op pair.</summary>
+        private void DFlashEncodeAndInject(float[] hRows, int rowOffset, int n, int startPos)
+        {
+            if (TryFusedDFlashInject(hRows, rowOffset, n, startPos))
+                return;
+
+            EnsureDFlashRingHostSynchronized();
+            using Tensor g = DFlashEncode(hRows, rowOffset, n);
+            DFlashInjectKv(g, n, startPos);
+        }
+
+        /// <summary>
+        /// Drafts one block. <paramref name="hPrev"/> holds the target features of
+        /// the last FORWARDED position (<paramref name="position"/> - 1) and
+        /// <paramref name="lastToken"/> the token at <paramref name="position"/>
+        /// (drawn but not yet forwarded). Returns the number of tokens written to
+        /// <paramref name="draftOut"/>; <paramref name="confOut"/> receives their
+        /// per-position acceptance estimates.
+        /// </summary>
+        private protected int DFlashPropose(int lastToken, float[] hPrev, int position, int[] draftOut, float[] confOut)
+        {
+            RequireDFlash();
+            if (position <= 0 || draftOut == null || draftOut.Length == 0)
+                return 0;
+
+            // The drafter's own key for the last committed position, exactly like
+            // the reference's ring[start_pos % win] = kv(main_x).
+            DFlashEncodeAndInject(hPrev, 0, 1, position - 1);
+
+            int b = Math.Min(_dflash.BlockSize, draftOut.Length + 1);
+            return DFlashDraftBlockCore(lastToken, position, b, draftOut, confOut);
+        }
+
+        private void RequireDFlash()
+        {
+            if (!_hasDFlash)
+                throw new InvalidOperationException("No DFlash drafter is loaded for this model.");
+        }
+
+        // ====================================================================
+        // PASS A -- encoder
+        // ====================================================================
+
+        /// <summary>
+        /// g = rmsnorm(fc @ feat, enc.output_norm). <paramref name="rowOffset"/> is
+        /// the first feature row of <paramref name="hRows"/> to consume and
+        /// <paramref name="n"/> how many. Returns [n, hidden].
+        /// </summary>
+        private unsafe Tensor DFlashEncode(float[] hRows, int rowOffset, int n)
+        {
+            int feat = _dflash.FeatureSize;
+            long need = (long)(rowOffset + n) * feat;
+            if (hRows == null || hRows.LongLength < need)
+            {
+                throw new ArgumentException(
+                    $"DFlash encoder needs {need} feature floats but got {hRows?.LongLength ?? 0}.", nameof(hRows));
+            }
+
+            var featTensor = new Tensor(_allocator, DType.Float32, n, feat);
+            long bytes = (long)n * feat * sizeof(float);
+            float* dst = GetFloatPtr(featTensor);
+            fixed (float* src = &hRows[(long)rowOffset * feat])
+                Buffer.MemoryCopy(src, dst, bytes, bytes);
+            InvalidateTensorDeviceCache(featTensor);
+
+            Tensor g = LinearForward(featTensor, DFlashConfig.WeightPrefix + "fc.weight");
+            featTensor.Dispose();
+
+            Ops.RMSNorm(g, g, _weights[DFlashConfig.WeightPrefix + "enc.output_norm.weight"], null, _dflash.Eps);
+            return g;
+        }
+
+        // ====================================================================
+        // PASS B -- KV injection
+        // ====================================================================
+
+        /// <summary>
+        /// Writes the drafter's per-position keys/values for <paramref name="n"/>
+        /// encoder rows starting at absolute position <paramref name="startPos"/>.
+        /// K is head-normed and NeoX-RoPE'd at the TARGET position; V gets neither.
+        /// The DFlash2 convolution does NOT run here: context positions enter the
+        /// draft KV straight off the encoder, which is how the drafter was trained
+        /// (sglang's DFlashAttention.kv_proj_only takes the projected target hidden
+        /// with neither the input layernorm nor the conv applied).
+        /// </summary>
+        private void DFlashInjectKv(Tensor g, int n, int startPos)
+        {
+            var cfg = _dflash;
+            int kvHeads = cfg.NumKVHeads, hd = cfg.HeadDim;
+
+            int[] positions = new int[n];
+            for (int i = 0; i < n; i++)
+                positions[i] = startPos + i;
+            // ASSIGNMENT, not max: this is the drafter's write FRONTIER, and it
+            // has to be able to move backwards. A new sequence (a chat turn after
+            // a cache reset, a fresh request) starts injecting at position 0 again,
+            // and a frontier still parked at the previous sequence's length would
+            // label every live ring slot with a position from that sequence - which
+            // the draft mask then reads as "in the future" and drops, leaving the
+            // drafter with no context at all. Injection is always contiguous and
+            // forward WITHIN a sequence, so the frontier is exactly startPos + n.
+            _dflashRingFilled = startPos + n;
+
+            for (int il = 0; il < cfg.NumLayers; il++)
+            {
+                string[] names = _dflashLayerNames[il];
+
+                Tensor k = LinearForward(g, names[DfAttnK]);
+                Tensor v = LinearForward(g, names[DfAttnV]);
+
+                k = DFlashHeadNorm(k, _weights[names[DfAttnKNorm]], kvHeads, n, hd);
+                k = DFlashRoPE(k, kvHeads, n, hd, positions);
+
+                using (Tensor kHeads = ReshapeToHeads(k, kvHeads, n, hd))
+                    DFlashRingWrite(_dflashRingK[il], kHeads, startPos, n);
+                using (Tensor vHeads = ReshapeToHeads(v, kvHeads, n, hd))
+                    DFlashRingWrite(_dflashRingV[il], vHeads, startPos, n);
+
+                k.Dispose();
+                v.Dispose();
+            }
+        }
+
+        /// <summary>Scatters a head-first [kvHeads, n, headDim] tensor into the ring
+        /// at (startPos + i) % ringRows, splitting the write at the wrap.</summary>
+        private void DFlashRingWrite(Tensor ring, Tensor headFirst, int startPos, int n)
+        {
+            int rows = _dflashRingRows;
+            int keep = Math.Min(n, rows);
+            int first = n - keep;                 // older rows would be overwritten anyway
+            int done = 0;
+            while (done < keep)
+            {
+                int slot = (startPos + first + done) % rows;
+                int len = Math.Min(keep - done, rows - slot);
+                using var src = headFirst.Narrow(1, first + done, len);
+                CopyToCache(ring, src, slot, len);
+                done += len;
+            }
+        }
+
+        // ====================================================================
+        // PASS C -- block draft
+        // ====================================================================
+
+        /// <summary>
+        /// Runs [anchor, MASK x (b-1)] at positions p..p+b-1 through the drafter and
+        /// fills <paramref name="draftOut"/> / <paramref name="confOut"/> from rows
+        /// 1..b-1. Returns b-1.
+        /// </summary>
+        private unsafe int DFlashDraftBlockCore(int anchorToken, int position, int b, int[] draftOut, float[] confOut)
+        {
+            int fused = TryFusedDFlashDraftBlock(anchorToken, position, b, draftOut, confOut);
+            if (fused >= 0)
+                return fused;
+
+            EnsureDFlashRingHostSynchronized();
+            var cfg = _dflash;
+            int heads = cfg.NumHeads, hd = cfg.HeadDim, kvHeads = cfg.NumKVHeads;
+            float eps = cfg.Eps;
+
+            int[] ids = new int[b];
+            int[] positions = new int[b];
+            for (int i = 0; i < b; i++)
+            {
+                ids[i] = i == 0 ? anchorToken : cfg.MaskTokenId;
+                positions[i] = position + i;
+            }
+
+            // llama.cpp's dflash graph feeds build_inp_embd straight in: no
+            // embedding scale, and (unlike some trunks) no weightless RMSNorm over
+            // the embeddings.
+            Tensor inpL = Embedding(ids);
+
+            for (int il = 0; il < cfg.NumLayers; il++)
+            {
+                string[] names = _dflashLayerNames[il];
+
+                Tensor h = DFlashRmsNorm(inpL, names[DfAttnNorm], eps);
+
+                // DFlash2: one projection of the sublayer input yields both this
+                // sublayer's input filter and the filter its OUTPUT is convolved
+                // with, so the coefficients are computed once and held across the
+                // attention.
+                float[] attnFinishDelta = null;
+                if (cfg.HasConv)
+                    h = DFlashConvPrepare(h, b, names[DfAttnConvProj], names[DfAttnConvBase], out attnFinishDelta);
+
+                Tensor q = LinearForward(h, names[DfAttnQ]);
+                Tensor k = LinearForward(h, names[DfAttnK]);
+                Tensor v = LinearForward(h, names[DfAttnV]);
+                h.Dispose();
+
+                q = DFlashHeadNorm(q, _weights[names[DfAttnQNorm]], heads, b, hd);
+                k = DFlashHeadNorm(k, _weights[names[DfAttnKNorm]], kvHeads, b, hd);
+                q = DFlashRoPE(q, heads, b, hd, positions);
+                k = DFlashRoPE(k, kvHeads, b, hd, positions);
+                Ops.Mul(q, q, 1f / MathF.Sqrt(hd));
+
+                Tensor attn = DFlashBlockAttention(il, q, k, v, position, b);
+                q.Dispose();
+                k.Dispose();
+                v.Dispose();
+
+                Tensor attnOut = LinearForward(attn, names[DfAttnOutput]);
+                attn.Dispose();
+
+                if (attnFinishDelta != null)
+                    attnOut = DFlashConvFinish(attnOut, b, attnFinishDelta, names[DfAttnConvBase]);
+
+                Ops.Add(attnOut, attnOut, inpL);          // ffn_inp = attn + inpL
+                inpL.Dispose();
+
+                Tensor ffnIn = DFlashRmsNorm(attnOut, names[DfFfnNorm], eps);
+                float[] ffnFinishDelta = null;
+                if (cfg.HasConv)
+                    ffnIn = DFlashConvPrepare(ffnIn, b, names[DfFfnConvProj], names[DfFfnConvBase], out ffnFinishDelta);
+
+                Tensor ffnOut = DFlashSwiGLU(ffnIn, names[DfFfnGate], names[DfFfnUp], names[DfFfnDown]);
+                ffnIn.Dispose();
+                if (ffnFinishDelta != null)
+                    ffnOut = DFlashConvFinish(ffnOut, b, ffnFinishDelta, names[DfFfnConvBase]);
+
+                Ops.Add(attnOut, attnOut, ffnOut);        // inpL = ffn + ffn_inp
+                ffnOut.Dispose();
+                inpL = attnOut;
+            }
+
+            Tensor cur = DFlashRmsNorm(inpL, DFlashConfig.WeightPrefix + "output_norm.weight", eps);
+            inpL.Dispose();
+
+            if (cfg.HasSelector)
+            {
+                int produced = DFlashSelectorBlock(cur, b, anchorToken, draftOut, confOut);
+                cur.Dispose();
+                return produced;
+            }
+
+            // The TARGET's LM head, with NEITHER logit_scale NOR the tanh softcap:
+            // llama.cpp's dflash graph ends at build_lora_mm(output, cur).
+            Tensor logits = LinearForward(cur, DFlashTargetOutputWeightName);
+            cur.Dispose();
+
+            // Softmax on the backend, then a max scan per row: argmax is invariant
+            // under softmax, and the winning probability IS the confidence the
+            // executor multiplies cumulatively (a zero there drafts nothing).
+            Ops.Softmax(logits, logits);
+
+            int vocab = Config.VocabSize;
+            int n = b - 1;
+            float* lp = GetFloatPtr(logits);
+            for (int i = 0; i < n; i++)
+            {
+                // Row 0 is the anchor's own prediction; plain DFlash discards it.
+                float* row = lp + (long)(i + 1) * vocab;
+                int best = DFlashArgmaxRow(row, vocab, out float prob);
+                draftOut[i] = best;
+                if (confOut != null && i < confOut.Length)
+                    confOut[i] = prob;
+            }
+            logits.Dispose();
+            return n;
+        }
+
+        private static unsafe int DFlashArgmaxRow(float* row, int n, out float best)
+        {
+            int bestIdx = 0;
+            float bestVal = row[0];
+            for (int i = 1; i < n; i++)
+            {
+                float v = row[i];
+                if (v > bestVal)
+                {
+                    bestVal = v;
+                    bestIdx = i;
+                }
+            }
+            best = bestVal;
+            return bestIdx;
+        }
+
+        /// <summary>
+        /// One draft layer's attention: the b block queries attend
+        /// [ring window | this block's own b keys], NON-CAUSALLY inside the block
+        /// (llama_set_causal_attn(ctx_dft, false)) and sliding-window masked against
+        /// the ring (a cached key at p0 is masked from a query at p1 when
+        /// p1 - p0 &gt;= n_swa).
+        /// </summary>
+        private unsafe Tensor DFlashBlockAttention(int il, Tensor q, Tensor k, Tensor v, int position, int b)
+        {
+            var cfg = _dflash;
+            int kvHeads = cfg.NumKVHeads, hd = cfg.HeadDim, heads = cfg.NumHeads;
+            int groupSize = heads / kvHeads;
+            int rings = _dflashRingRows;
+
+            // The FIRST query (position p) sees cached keys down to p - (n_swa - 1);
+            // later queries in the block see a strict subset, masked below.
+            int winStart = Math.Max(0, position - (cfg.SlidingWindow - 1));
+            int w = position - winStart;
+            int total = w + b;
+
+            var gk = new Tensor(_allocator, DType.Float32, kvHeads, total, hd);
+            var gv = new Tensor(_allocator, DType.Float32, kvHeads, total, hd);
+
+            int done = 0;
+            while (done < w)
+            {
+                int slot = (winStart + done) % rings;
+                int len = Math.Min(w - done, rings - slot);
+                using (var srcK = _dflashRingK[il].Narrow(1, slot, len))
+                using (var dstK = gk.Narrow(1, done, len))
+                    Ops.Copy(dstK, srcK);
+                using (var srcV = _dflashRingV[il].Narrow(1, slot, len))
+                using (var dstV = gv.Narrow(1, done, len))
+                    Ops.Copy(dstV, srcV);
+                done += len;
+            }
+
+            using (Tensor kHeads = ReshapeToHeads(k, kvHeads, b, hd))
+            using (var dstBlockK = gk.Narrow(1, w, b))
+                Ops.Copy(dstBlockK, kHeads);
+            using (Tensor vHeads = ReshapeToHeads(v, kvHeads, b, hd))
+            using (var dstBlockV = gv.Narrow(1, w, b))
+                Ops.Copy(dstBlockV, vHeads);
+
+            // GQA without materializing the expanded K/V: a contiguous head-first
+            // [heads, b, hd] query tensor reinterprets exactly as
+            // [kvHeads, groupSize*b, hd] (heads g*gs..g*gs+gs-1 are adjacent blocks
+            // of b*hd), so one batched GEMM per kv head serves its whole query
+            // group. ExpandKVHeads would instead allocate heads*total*hd floats per
+            // layer per draft purely to repeat rows.
+            Tensor qHeads = ReshapeToHeads(q, heads, b, hd);
+            Tensor scores;
+            using (Tensor qGrouped = qHeads.View(kvHeads, (long)groupSize * b, hd))
+            using (Tensor kT = gk.Transpose(1, 2))
+            {
+                scores = new Tensor(_allocator, DType.Float32, kvHeads, (long)groupSize * b, total);
+                Ops.AddmmBatch(scores, 0, scores, 1f, qGrouped, kT);
+            }
+            qHeads.Dispose();
+            gk.Dispose();
+
+            DFlashApplyWindowMask(scores, b, groupSize, kvHeads, w, total, position, winStart);
+            Ops.Softmax(scores, scores);
+
+            var attnGrouped = new Tensor(_allocator, DType.Float32, kvHeads, (long)groupSize * b, hd);
+            Ops.AddmmBatch(attnGrouped, 0, attnGrouped, 1f, scores, gv);
+            scores.Dispose();
+            gv.Dispose();
+
+            Tensor attn;
+            using (Tensor attnHeads = attnGrouped.View(heads, b, hd))
+                attn = ReshapeFromHeads(attnHeads, heads, b, hd);
+            attnGrouped.Dispose();
+            return attn;
+        }
+
+        /// <summary>
+        /// Masks the cached (ring) columns a query cannot see. Query row j of kv
+        /// group g belongs to block slot s = j % b (see the grouping comment in
+        /// <see cref="DFlashBlockAttention"/>), i.e. absolute position
+        /// <paramref name="position"/> + s; cached column c holds position
+        /// <paramref name="winStart"/> + c and is masked when
+        /// (position + s) - (winStart + c) &gt;= n_swa. The block's own b columns
+        /// are never masked -- attention inside the block is non-causal.
+        /// </summary>
+        private unsafe void DFlashApplyWindowMask(Tensor scores, int b, int groupSize, int kvHeads,
+            int w, int total, int position, int winStart)
+        {
+            if (w <= 0)
+                return;
+
+            int swa = _dflash.SlidingWindow;
+            Span<int> widths = stackalloc int[b];
+            bool any = false;
+            for (int s = 0; s < b; s++)
+            {
+                int width = position + s - swa - winStart + 1;
+                if (width < 0) width = 0;
+                if (width > w) width = w;
+                widths[s] = width;
+                any |= width > 0;
+            }
+            if (!any)
+                return;
+
+            float* sp = GetFloatPtr(scores);
+            int rowsPerGroup = groupSize * b;
+            for (int g = 0; g < kvHeads; g++)
+            {
+                float* groupScores = sp + (long)g * rowsPerGroup * total;
+                for (int j = 0; j < rowsPerGroup; j++)
+                {
+                    int width = widths[j % b];
+                    if (width > 0)
+                        new Span<float>(groupScores + (long)j * total, width).Fill(float.NegativeInfinity);
+                }
+            }
+            InvalidateTensorDeviceCache(scores);
+        }
+
+        // ====================================================================
+        // DFlash2 -- grouped dynamic convolution
+        // ====================================================================
+
+        /// <summary>
+        /// Convolves a sublayer's input and returns the coefficients its OUTPUT is
+        /// convolved with. One projection produces both halves, which is why they
+        /// cannot be separate calls: the output filter is keyed on the INPUT, not
+        /// on whatever the sublayer produced.
+        ///
+        /// Consumes <paramref name="x"/> and returns a new [b, hidden] tensor.
+        /// </summary>
+        private unsafe Tensor DFlashConvPrepare(Tensor x, int b, string projName, string baseName,
+            out float[] finishDelta)
+        {
+            var cfg = _dflash;
+            int taps = cfg.ConvKernelSize, groups = cfg.ConvNumGroups;
+            int half = taps * groups;
+
+            using Tensor coef = LinearForward(x, projName);   // [b, 2 * taps * groups]
+            float* cp = GetFloatPtr(coef);
+            int stride = cfg.ConvProjOutSize;
+
+            // The output half is copied out because the sublayer between prepare
+            // and finish reuses (and may free) every device buffer in flight.
+            finishDelta = new float[(long)b * half];
+            for (int r = 0; r < b; r++)
+                for (int i = 0; i < half; i++)
+                    finishDelta[(long)r * half + i] = cp[(long)r * stride + half + i];
+
+            Tensor result = DFlashConvApply(x, b, cp, stride, /*deltaOffset=*/0, baseName, /*side=*/0);
+            x.Dispose();
+            return result;
+        }
+
+        /// <summary>Applies the output-side filter produced by
+        /// <see cref="DFlashConvPrepare"/>. Consumes <paramref name="y"/>.</summary>
+        private unsafe Tensor DFlashConvFinish(Tensor y, int b, float[] finishDelta, string baseName)
+        {
+            int half = _dflash.ConvKernelSize * _dflash.ConvNumGroups;
+            fixed (float* dp = finishDelta)
+            {
+                Tensor result = DFlashConvApply(y, b, dp, half, /*deltaOffset=*/0, baseName, /*side=*/1);
+                y.Dispose();
+                return result;
+            }
+        }
+
+        /// <summary>
+        /// out[r][c] = sum over taps t of (base[side][t][c] + delta[r][t][c / group])
+        ///             * x[r-t][c], with the t-th tap zeroed for the first t rows of
+        /// the block (the filter never reaches across a block boundary).
+        ///
+        /// Runs on the host: this is the per-op fallback, b is one block (8-16 rows)
+        /// and the whole thing is b*hidden*taps multiply-adds - two orders of
+        /// magnitude below one of the projections around it. The fused kernel does
+        /// it in the graph.
+        /// </summary>
+        private unsafe Tensor DFlashConvApply(Tensor x, int b, float* delta, int deltaStride, int deltaOffset,
+            string baseName, int side)
+        {
+            var cfg = _dflash;
+            var outT = new Tensor(_allocator, DType.Float32, b, cfg.HiddenSize);
+            DFlashGroupedConvolve(
+                GetFloatPtr(x), GetFloatPtr(outT), GetFloatPtr(_weights[baseName]), delta,
+                b, cfg.HiddenSize, cfg.ConvKernelSize, cfg.ConvGroupSize, cfg.ConvNumGroups,
+                deltaStride, deltaOffset, side);
+            InvalidateTensorDeviceCache(outT);
+            return outT;
+        }
+
+        /// <summary>
+        /// The convolution arithmetic itself, on raw rows, so the fused graph has
+        /// something independent to be checked against:
+        ///
+        ///   out[r][c] = sum over taps t of
+        ///               (base[side][t][c] + delta[r][t][c / groupSize]) * x[r-t][c]
+        ///
+        /// with tap t contributing nothing for r &lt; t. baseKernel is the
+        /// checkpoint's [side, tap, hidden] block; delta is row-major with
+        /// deltaStride floats per row, its (tap, group) pairs starting at
+        /// deltaOffset.
+        /// </summary>
+        internal static unsafe void DFlashGroupedConvolve(
+            float* x, float* dst0, float* baseKernel, float* delta,
+            int rows, int hidden, int taps, int groupSize, int groups,
+            int deltaStride, int deltaOffset, int side)
+        {
+            for (int r = 0; r < rows; r++)
+            {
+                float* dst = dst0 + (long)r * hidden;
+                for (int tap = 0; tap < taps; tap++)
+                {
+                    if (tap > r)
+                    {
+                        // Masked tap: the filter never reaches across the block
+                        // boundary. Tap 0 is never masked, so row 0 is still written.
+                        continue;
+                    }
+                    float* src = x + (long)(r - tap) * hidden;
+                    float* bt = baseKernel + ((long)side * taps + tap) * hidden;
+                    float* dt = delta + (long)r * deltaStride + deltaOffset + (long)tap * groups;
+                    if (tap == 0)
+                    {
+                        for (int g = 0; g < groups; g++)
+                        {
+                            float d = dt[g];
+                            int c0 = g * groupSize;
+                            for (int c = c0; c < c0 + groupSize; c++)
+                                dst[c] = (bt[c] + d) * src[c];
+                        }
+                    }
+                    else
+                    {
+                        for (int g = 0; g < groups; g++)
+                        {
+                            float d = dt[g];
+                            int c0 = g * groupSize;
+                            for (int c = c0; c < c0 + groupSize; c++)
+                                dst[c] += (bt[c] + d) * src[c];
+                        }
+                    }
+                }
+            }
+        }
+
+        /// <summary>Managed-array entry point for the convolution, for tests and for
+        /// callers that do not already hold pinned rows.</summary>
+        internal static unsafe float[] DFlashGroupedConvolve(
+            float[] x, float[] baseKernel, float[] delta,
+            int rows, int hidden, int taps, int groupSize,
+            int deltaStride, int deltaOffset, int side)
+        {
+            var result = new float[(long)rows * hidden];
+            fixed (float* xp = x)
+            fixed (float* bp = baseKernel)
+            fixed (float* dp = delta)
+            fixed (float* op = result)
+            {
+                DFlashGroupedConvolve(xp, op, bp, dp, rows, hidden, taps, groupSize,
+                    hidden / groupSize, deltaStride, deltaOffset, side);
+            }
+            return result;
+        }
+
+        // ====================================================================
+        // DFlash2 -- candidate selector
+        // ====================================================================
+
+        /// <summary>
+        /// Turns the block's post-norm hidden states into a token per position by
+        /// walking the transition lattice instead of taking b-1 independent argmaxes.
+        ///
+        /// Row 0 of <paramref name="cur"/> is the anchor's own hidden state and is
+        /// not a proposal; the gamma = b-1 rows after it are, and each contributes
+        /// selector_top_k candidates. Returns gamma.
+        /// </summary>
+        private unsafe int DFlashSelectorBlock(Tensor cur, int b, int anchorToken, int[] draftOut, float[] confOut)
+        {
+            var cfg = _dflash;
+            int gamma = b - 1;
+            int k = cfg.SelectorTopK;
+            int rank = cfg.SelectorRank;
+            int vocab = Config.VocabSize;
+
+            Tensor predRows;
+            using (var view = cur.Narrow(0, 1, gamma))
+                predRows = Ops.NewContiguous(view);
+
+            // Unary term: the ordinary DFlash head, kept only at its top-k.
+            int[] candIds = new int[gamma * k];
+            float[] unary = new float[gamma * k];
+            using (Tensor logits = LinearForward(predRows, DFlashTargetOutputWeightName))
+            {
+                float* lp = GetFloatPtr(logits);
+                for (int e = 0; e < gamma; e++)
+                    DFlashTopK(lp + (long)e * vocab, vocab, k, candIds, unary, e * k);
+            }
+            // The target's own logit transform, applied AFTER the top-k because both
+            // halves of it are monotonic (so the candidate set is unchanged) and doing
+            // it here costs gamma*k operations instead of gamma*vocab.
+            DFlashTransformUnary(unary);
+
+            // P h, once per position.
+            float[] projected = new float[gamma * rank];
+            using (Tensor ph = LinearForward(predRows, DFlashConfig.WeightPrefix + "selector_hidden.weight"))
+            {
+                float* pp = GetFloatPtr(ph);
+                for (int i = 0; i < gamma * rank; i++)
+                    projected[i] = pp[i];
+            }
+            predRows.Dispose();
+
+            // B[cand] for every candidate, and A[pred] for every predecessor: the
+            // anchor for position 0, and position e-1's candidates for position e.
+            float[] succ = DFlashGatherCodebook(DFlashConfig.WeightPrefix + "selector_successor.weight",
+                candIds, gamma * k, rank);
+
+            int[] predIds = new int[gamma * k];
+            for (int p = 0; p < k; p++)
+                predIds[p] = anchorToken;
+            Array.Copy(candIds, 0, predIds, k, (gamma - 1) * k);
+            float[] pred = DFlashGatherCodebook(DFlashConfig.WeightPrefix + "selector_predecessor.weight",
+                predIds, gamma * k, rank);
+
+            // Greedy walk. Position 0's predecessor row is the anchor's, replicated
+            // over p, so only p = 0 is scored there.
+            float[] row = new float[k];
+            float[] m = new float[rank];
+            int chosen = 0;
+            for (int e = 0; e < gamma; e++)
+            {
+                int pRow = e == 0 ? 0 : chosen;
+                long predBase = ((long)e * k + pRow) * rank;
+                long projBase = (long)e * rank;
+                for (int r = 0; r < rank; r++)
+                    m[r] = pred[predBase + r] * projected[projBase + r];
+
+                for (int c = 0; c < k; c++)
+                {
+                    long keyBase = ((long)e * k + c) * rank;
+                    float dot = 0f;
+                    for (int r = 0; r < rank; r++)
+                        dot += m[r] * succ[keyBase + r];
+                    row[c] = unary[(long)e * k + c] + dot;
+                }
+
+                chosen = 0;
+                for (int c = 1; c < k; c++)
+                    if (row[c] > row[chosen]) chosen = c;
+
+                if (DFlashSelectorDebug && _dflashSelectorDebugBlocks < 3)
+                {
+                    // Attribution for the lattice: if the transition term never moves
+                    // the choice off the unary argmax, the selector is doing nothing
+                    // and its cost is pure loss.
+                    int unaryBest = 0;
+                    float loMin = float.MaxValue, loMax = float.MinValue;
+                    for (int c = 0; c < k; c++)
+                    {
+                        float u = unary[(long)e * k + c];
+                        if (u > unary[(long)e * k + unaryBest]) unaryBest = c;
+                        float d = row[c] - u;
+                        if (d < loMin) loMin = d;
+                        if (d > loMax) loMax = d;
+                    }
+                    Console.WriteLine(
+                        $"  [dflash-sel] slot {e}: unary[{unaryBest}]={unary[(long)e * k + unaryBest]:F3} " +
+                        $"range=[{MinOf(unary, e * k, k):F3},{MaxOf(unary, e * k, k):F3}] " +
+                        $"transition=[{loMin:F3},{loMax:F3}] chose={chosen}{(chosen == unaryBest ? " (= unary argmax)" : " (MOVED)")}");
+                }
+
+                draftOut[e] = candIds[(long)e * k + chosen];
+                if (confOut != null && e < confOut.Length)
+                    confOut[e] = DFlashSoftmaxAt(row, chosen);
+            }
+
+            if (DFlashSelectorDebug)
+                _dflashSelectorDebugBlocks++;
+            return gamma;
+        }
+
+        /// <summary>TS_DFLASH_SELECTOR_DEBUG=1 prints the first few blocks' lattice
+        /// attribution: the unary spread, the transition spread, and whether the walk
+        /// left the unary argmax. Managed (per-op) drafter only.</summary>
+        private static readonly bool DFlashSelectorDebug =
+            string.Equals(Environment.GetEnvironmentVariable("TS_DFLASH_SELECTOR_DEBUG"), "1", StringComparison.Ordinal);
+
+        private int _dflashSelectorDebugBlocks;
+
+        private static float MinOf(float[] a, long off, int n)
+        {
+            float v = a[off];
+            for (int i = 1; i < n; i++) if (a[off + i] < v) v = a[off + i];
+            return v;
+        }
+
+        private static float MaxOf(float[] a, long off, int n)
+        {
+            float v = a[off];
+            for (int i = 1; i < n; i++) if (a[off + i] > v) v = a[off + i];
+            return v;
+        }
+
+        /// <summary>
+        /// scale, then tanh-softcap: the target's LM-head transform, which the
+        /// selector's unary term has to carry because the lattice ADDS it to a
+        /// transition score rather than taking an argmax over it. A no-op on a
+        /// checkpoint whose target has neither (Qwen 3.8).
+        /// </summary>
+        private void DFlashTransformUnary(float[] unary)
+        {
+            var cfg = _dflash;
+            if (!cfg.HasUnaryLogitTransform)
+                return;
+            float scale = cfg.LogitScale;
+            float cap = cfg.FinalLogitSoftcap;
+            for (int i = 0; i < unary.Length; i++)
+            {
+                float v = unary[i] * scale;
+                unary[i] = cap > 0f ? MathF.Tanh(v / cap) * cap : v;
+            }
+        }
+
+        /// <summary>Top <paramref name="k"/> of one logits row, unsorted, written at
+        /// <paramref name="outOffset"/> of the id/value arrays. A k-element
+        /// insertion scan: k is 16 and the row is the vocabulary, so anything that
+        /// touches each element once is the right shape.</summary>
+        internal static unsafe void DFlashTopK(float* rowPtr, int n, int k, int[] idsOut, float[] valsOut, int outOffset)
+        {
+            // Seed with the first k entries and track the weakest of them.
+            int worst = 0;
+            for (int i = 0; i < k; i++)
+            {
+                idsOut[outOffset + i] = i;
+                valsOut[outOffset + i] = rowPtr[i];
+                if (rowPtr[i] < valsOut[outOffset + worst]) worst = i;
+            }
+            float cutoff = valsOut[outOffset + worst];
+            for (int i = k; i < n; i++)
+            {
+                float v = rowPtr[i];
+                if (v <= cutoff)
+                    continue;
+                idsOut[outOffset + worst] = i;
+                valsOut[outOffset + worst] = v;
+                worst = 0;
+                for (int j = 1; j < k; j++)
+                    if (valsOut[outOffset + j] < valsOut[outOffset + worst]) worst = j;
+                cutoff = valsOut[outOffset + worst];
+            }
+        }
+
+        /// <summary>Managed-array entry point for the top-k selection.</summary>
+        internal static unsafe void DFlashTopK(float[] row, int k, int[] idsOut, float[] valsOut, int outOffset)
+        {
+            fixed (float* rp = row)
+                DFlashTopK(rp, row.Length, k, idsOut, valsOut, outOffset);
+        }
+
+        /// <summary>softmax(row)[index], computed with the usual max shift.</summary>
+        internal static float DFlashSoftmaxAt(float[] row, int index)
+        {
+            float max = row[0];
+            for (int i = 1; i < row.Length; i++)
+                if (row[i] > max) max = row[i];
+            double sum = 0;
+            for (int i = 0; i < row.Length; i++)
+                sum += Math.Exp(row[i] - max);
+            return sum > 0 ? (float)(Math.Exp(row[index] - max) / sum) : 0f;
+        }
+
+        /// <summary>Gathers <paramref name="count"/> rows of a [vocab, rank]
+        /// selector codebook into a flat host array. The codebooks are ordinary
+        /// quantized tensors, so this is the same row-gather the token embedding
+        /// uses.</summary>
+        private unsafe float[] DFlashGatherCodebook(string weightName, int[] ids, int count, int rank)
+        {
+            var result = new float[(long)count * rank];
+            int[] rows = ids;
+            if (ids.Length != count)
+            {
+                rows = new int[count];
+                Array.Copy(ids, rows, count);
+            }
+
+            if (_quantWeights.TryGetValue(weightName, out var qw))
+            {
+                using var gathered = new Tensor(_allocator, DType.Float32, count, rank);
+                PopulateQuantizedRows(gathered, qw, rows);
+                float* gp = GetFloatPtr(gathered);
+                for (long i = 0; i < (long)count * rank; i++)
+                    result[i] = gp[i];
+                return result;
+            }
+
+            Tensor w = _weights[weightName];
+            float* wp = GetFloatPtr(w);
+            for (int i = 0; i < count; i++)
+            {
+                long src = (long)rows[i] * rank;
+                for (int r = 0; r < rank; r++)
+                    result[(long)i * rank + r] = wp[src + r];
+            }
+            return result;
+        }
+
+        // ====================================================================
+        // small shared pieces
+        // ====================================================================
+
+        /// <summary>RMSNorm against a named drafter weight with the drafter's own
+        /// epsilon. The trunk helpers hardcode Config.Eps, which is not the
+        /// drafter's.</summary>
+        private Tensor DFlashRmsNorm(Tensor input, string weightName, float eps)
+        {
+            var alpha = _weights[weightName];
+            int rows = (int)input.Sizes[0];
+            int dim = (int)(input.ElementCount() / rows);
+            Tensor input2d = input.Sizes.Length != 2 ? input.View(rows, dim) : null;
+            Tensor src = input2d ?? input;
+            Tensor result = Ops.RMSNorm(null, src, alpha, null, eps);
+            input2d?.Dispose();
+            return result;
+        }
+
+        /// <summary>Per-head RMSNorm over a [rows, numHeads*headDim] tensor, with
+        /// the drafter's own epsilon. Consumes <paramref name="data"/>.</summary>
+        private Tensor DFlashHeadNorm(Tensor data, Tensor alpha, int numHeads, int rows, int headDim)
+        {
+            using var reshaped = data.View((long)rows * numHeads, headDim);
+            Tensor normed = Ops.RMSNorm(null, reshaped, alpha, null, _dflash.Eps);
+            data.Dispose();
+            Tensor flat = normed.View(rows, (long)numHeads * headDim);
+            normed.Dispose();
+            return flat;
+        }
+
+        /// <summary>
+        /// NeoX-flavour RoPE (split halves) over a [rows, numHeads*headDim] tensor
+        /// with an explicit per-row position. llama.cpp maps LLM_ARCH_DFLASH to
+        /// LLAMA_ROPE_TYPE_NEOX, so this is mode 2 -- NOT the interleaved-pair
+        /// (mode 0) RoPE some trunks use. Consumes <paramref name="data"/>.
+        /// </summary>
+        private Tensor DFlashRoPE(Tensor data, int numHeads, int rows, int headDim, int[] positions)
+        {
+            int totalRows = rows * numHeads;
+            int[] rowPositions = new int[totalRows];
+            for (int s = 0; s < rows; s++)
+                for (int h = 0; h < numHeads; h++)
+                    rowPositions[s * numHeads + h] = positions[s];
+            using var posTensor = CreateIntTensorOn(data.Storage.Allocator, rowPositions, totalRows);
+
+            using var reshaped = data.View(1, rows, numHeads, headDim);
+            Tensor result = Ops.RoPEEx(
+                null, reshaped, posTensor, headDim, DFlashConfig.RopeTypeNeoX, 0,
+                _dflash.RopeBase, 1.0f,
+                0.0f, 1.0f, 0.0f, 0.0f);
+
+            data.Dispose();
+            Tensor flat = result.View(rows, (long)numHeads * headDim);
+            result.Dispose();
+            return flat;
+        }
+
+        /// <summary>silu(gate(x)) * up(x) -> down. The drafter ships gate and up as
+        /// SEPARATE tensors (FuseGateUpWeights only fuses the target's "blk.{l}."
+        /// names), so this cannot use ModelBase.FFN.</summary>
+        private Tensor DFlashSwiGLU(Tensor input, string gateName, string upName, string downName)
+        {
+            Tensor gate = LinearForward(input, gateName);
+            Tensor up = LinearForward(input, upName);
+            Ops.SiLUMul(gate, gate, up);
+            up.Dispose();
+            Tensor down = LinearForward(gate, downName);
+            gate.Dispose();
+            return down;
+        }
+    }
+}
diff --git a/TensorSharp.Models/SpeculativeDraftHeadLoader.cs b/TensorSharp.Models/SpeculativeDraftHeadLoader.cs
index 00a14bba..c45860ea 100644
--- a/TensorSharp.Models/SpeculativeDraftHeadLoader.cs
+++ b/TensorSharp.Models/SpeculativeDraftHeadLoader.cs
@@ -55,6 +55,43 @@ public static bool TryAttachConfiguredDraftHead(ModelBase model, out string erro
             if (draftPath == null)
                 return true;
 
+            if (!File.Exists(draftPath))
+            {
+                error = $"Draft-head model file not found: {draftPath}";
+                return false;
+            }
+
+            // A DFlash / DFlash2 drafter is architecture-agnostic on this side: any
+            // target that can tap the residuals its encoder reads can host one, and
+            // the file says which it is. --draft-model may already have attached it
+            // during construction, in which case there is nothing to do.
+            if (IsDFlashDrafter(draftPath))
+            {
+                if (model == null)
+                {
+                    error = "No model is loaded to attach a DFlash drafter to.";
+                    return false;
+                }
+                if (model.HasDFlash)
+                    return true;
+                try
+                {
+                    model.LoadDFlashDraftWeights(draftPath);
+                }
+                catch (Exception ex)
+                {
+                    error = $"Failed to load DFlash drafter '{Path.GetFileName(draftPath)}': {ex.Message}";
+                    return false;
+                }
+                if (!model.HasDFlash)
+                {
+                    error = $"DFlash drafter '{Path.GetFileName(draftPath)}' loaded but is incomplete "
+                            + "(required draft tensors missing).";
+                    return false;
+                }
+                return true;
+            }
+
             if (model is not Gemma4Model gemma4)
             {
                 // A draft GGUF was named but this architecture does not consume a
@@ -66,12 +103,6 @@ public static bool TryAttachConfiguredDraftHead(ModelBase model, out string erro
                 return false;
             }
 
-            if (!File.Exists(draftPath))
-            {
-                error = $"Draft-head model file not found: {draftPath}";
-                return false;
-            }
-
             try
             {
                 gemma4.LoadMtpDraftWeights(draftPath);
@@ -90,5 +121,22 @@ public static bool TryAttachConfiguredDraftHead(ModelBase model, out string erro
             }
             return true;
         }
+
+        /// <summary>True when the file at <paramref name="path"/> declares itself a
+        /// DFlash drafter. Read from the GGUF rather than inferred from the name:
+        /// the same flag also names MTP-only assistant files.</summary>
+        private static bool IsDFlashDrafter(string path)
+        {
+            try
+            {
+                using var probe = new GgufFile(path);
+                return string.Equals(probe.GetString("general.architecture"),
+                    DFlashConfig.ArchName, StringComparison.Ordinal);
+            }
+            catch
+            {
+                return false;
+            }
+        }
     }
 }
diff --git a/TensorSharp.Runtime/ChatTemplate.cs b/TensorSharp.Runtime/ChatTemplate.cs
index c36e9b94..28882634 100644
--- a/TensorSharp.Runtime/ChatTemplate.cs
+++ b/TensorSharp.Runtime/ChatTemplate.cs
@@ -1,4 +1,4 @@
-// Copyright (c) Zhongkai Fu. All rights reserved.
+// Copyright (c) Zhongkai Fu. All rights reserved.
 // https://github.com/zhongkaifu/TensorSharp
 //
 // This file is part of TensorSharp.
@@ -670,12 +670,18 @@ private static string RenderHardcoded(List<ChatMessage> messages,
             if (IsGlmDsa(architecture))
                 return RenderGlmDsa(messages, addGenerationPrompt, enableThinking, tools);
 
+            if (IsGlm5Next(architecture))
+                return RenderGlm5Next(messages, addGenerationPrompt, enableThinking, tools);
+
             return RenderQwen3(messages, addGenerationPrompt, tools, enableThinking);
         }
 
         internal static bool IsGlmDsa(string? architecture)
             => architecture == "glm-dsa" || architecture == "glm_dsa";
 
+        internal static bool IsGlm5Next(string? architecture)
+            => architecture == "glm5next";
+
         private const string GlmToolsHeader =
             "<|system|>\n# Tools\n\nYou may call one or more functions to assist with the user query.\n\n" +
             "You are provided with function signatures within <tools></tools> XML tags:\n<tools>\n";
@@ -768,6 +774,90 @@ public static string RenderGlmDsa(List<ChatMessage> messages, bool addGeneration
             return sb.ToString();
         }
 
+        /// <summary>
+        /// GLM-5.3-Flash (glm5next) chat format, mirroring the template shipped in
+        /// the GGUF. Differences from GLM-5.2:
+        /// <list type="bullet">
+        /// <item>The reasoning-effort system line is ALWAYS emitted (the template
+        /// defaults <c>reasoning_effort</c> to <c>max</c>); this family has no
+        /// thinking-off prompt shape, so <paramref name="enableThinking"/> only
+        /// decides whether the generation prompt's <c>&lt;think&gt;</c> block is
+        /// left open or closed immediately.</item>
+        /// <item><c>clear_thinking</c> defaults to FALSE: historical assistant
+        /// turns KEEP their reasoning when the message still carries it.</item>
+        /// <item>No newline after the <c>&lt;|assistant|&gt;</c> tag.</item>
+        /// </list>
+        /// </summary>
+        public static string RenderGlm5Next(List<ChatMessage> messages, bool addGenerationPrompt = true,
+            bool enableThinking = true, List<ToolFunction>? tools = null)
+        {
+            var sb = new StringBuilder();
+            sb.Append("[gMASK]<sop>");
+            sb.Append("<|system|>Reasoning Effort: Max");
+
+            if (tools != null && tools.Count > 0)
+            {
+                sb.Append(GlmToolsHeader);
+                foreach (var tool in tools)
+                    sb.Append(ToolFunctionToJson(tool)).Append('\n');
+                sb.Append(GlmToolsFooter);
+            }
+
+            bool prevWasTool = false;
+            foreach (var m in messages)
+            {
+                switch (m.Role)
+                {
+                    case "system":
+                        sb.Append("<|system|>").Append(m.Content ?? "");
+                        prevWasTool = false;
+                        break;
+                    case "user":
+                    case "developer":
+                        sb.Append("<|user|>").Append(m.Content ?? "");
+                        prevWasTool = false;
+                        break;
+                    case "tool":
+                        // One <|observation|> opens a RUN of tool results.
+                        if (!prevWasTool)
+                            sb.Append("<|observation|>");
+                        sb.Append("<tool_response>").Append(m.Content ?? "").Append("</tool_response>");
+                        prevWasTool = true;
+                        break;
+                    case "assistant":
+                    {
+                        sb.Append("<|assistant|>");
+                        string content = m.Content ?? string.Empty;
+                        int open = content.IndexOf("<think>", StringComparison.Ordinal);
+                        int close = content.IndexOf("</think>", StringComparison.Ordinal);
+                        if (close >= 0)
+                        {
+                            // clear_thinking defaults false: past reasoning stays.
+                            string reasoning = content.Substring(
+                                open >= 0 ? open + "<think>".Length : 0,
+                                (close) - (open >= 0 ? open + "<think>".Length : 0));
+                            sb.Append("<think>").Append(reasoning).Append("</think>");
+                            content = content.Substring(close + "</think>".Length);
+                        }
+                        else
+                        {
+                            sb.Append("<think></think>");
+                        }
+                        content = content.Trim();
+                        if (content.Length > 0)
+                            sb.Append(content);
+                        prevWasTool = false;
+                        break;
+                    }
+                }
+            }
+
+            if (addGenerationPrompt)
+                sb.Append("<|assistant|>").Append(enableThinking ? "<think>" : "<think></think>");
+
+            return sb.ToString();
+        }
+
         /// <summary>
         /// DeepSeek V4 chat format (mirrors models/templates/deepseek-ai-DeepSeek-V4.jinja):
         /// leading system prompt(s) concatenated after BOS, then
@@ -1072,7 +1162,7 @@ internal static List<ChatMessage> InjectMultimodalTokens(List<ChatMessage> messa
                     if (msg.ImagePaths != null)
                         foreach (var _ in msg.ImagePaths) sb.Append("<start_of_image>");
                 }
-                else if (architecture is "qwen35" or "qwen35moe" or "qwen3next" or "qwen3vl" or "qwen3vlmoe")
+                else if (architecture is "qwen35" or "qwen35moe" or "qwen3next" or "qwen3vl" or "qwen3vlmoe" or "qwen4exp")
                 {
                     if (msg.ImagePaths != null)
                         foreach (var _ in msg.ImagePaths)
@@ -1084,6 +1174,15 @@ internal static List<ChatMessage> InjectMultimodalTokens(List<ChatMessage> messa
                         foreach (var _ in msg.ImagePaths)
                             sb.Append("[IMG]");
                 }
+                else if (architecture == "glm5next")
+                {
+                    // GLM-5.3-Flash: the template's emit_image() macro. The host
+                    // later expands the single <|image|> into N placeholder
+                    // tokens matching the merged patch count.
+                    if (msg.ImagePaths != null)
+                        foreach (var _ in msg.ImagePaths)
+                            sb.Append("<|begin_of_image|><|image|><|end_of_image|>");
+                }
                 else if (architecture is "muse-glimmer" or "muse_glimmer")
                 {
                     // The GGUF Jinja template renders an image content part as a
diff --git a/TensorSharp.Runtime/GgufReader.cs b/TensorSharp.Runtime/GgufReader.cs
index 99767d7d..243fc126 100644
--- a/TensorSharp.Runtime/GgufReader.cs
+++ b/TensorSharp.Runtime/GgufReader.cs
@@ -1,4 +1,4 @@
-// Copyright (c) Zhongkai Fu. All rights reserved.
+// Copyright (c) Zhongkai Fu. All rights reserved.
 // https://github.com/zhongkaifu/TensorSharp
 //
 // This file is part of TensorSharp.
@@ -63,6 +63,11 @@ public partial class GgufFile : IDisposable
         public Dictionary<string, GgufTensorInfo> Tensors { get; } = new();
         public long DataOffset { get; private set; }
 
+        /// <summary>Unaligned end of the KV + tensor table; a shard with no
+        /// tensor data of its own may legitimately end here, before the
+        /// alignment padding that <see cref="DataOffset"/> assumes.</summary>
+        private long _tableEnd;
+
         private FileStream _stream;
         private string _path;
         private MemoryMappedFile? _mappedFile;
@@ -388,6 +393,7 @@ private void Parse()
             int alignment = 32;
             if (Metadata.TryGetValue("general.alignment", out var a))
                 alignment = Convert.ToInt32(a);
+            _tableEnd = pos;
             DataOffset = pos + (alignment - pos % alignment) % alignment;
         }
 
@@ -400,7 +406,12 @@ private void Parse()
         /// </summary>
         public long GetRequiredLength(out string? lastTensorName)
         {
-            long required = DataOffset;
+            // Split GGUFs often front-load a metadata-only first shard: every
+            // tensor in its table lives in a sibling file, so the file ends
+            // right after the table and the alignment padding DataOffset
+            // assumes never exists. Only demand bytes past the table when a
+            // tensor actually claims them.
+            long required = _tableEnd;
             lastTensorName = null;
             foreach (var t in Tensors.Values)
             {
@@ -526,6 +537,31 @@ public bool GetBool(string key, bool defaultValue = false)
             return null;
         }
 
+        /// <summary>
+        /// A UINT64 metadata array. Used by the qwen4exp PLE n-gram hash, whose
+        /// multipliers and per-head vocabulary sizes are 64-bit by construction -
+        /// the hash multiplies token ids by ~2^44 constants and takes the result
+        /// modulo a ~20 M row count, so nothing narrower carries it.
+        /// </summary>
+        public ulong[]? GetUint64Array(string key)
+        {
+            if (!Metadata.TryGetValue(key, out var v)) return null;
+            if (v is ulong[] ua) return ua;
+            if (v is uint[] u32)
+            {
+                var result = new ulong[u32.Length];
+                for (int i = 0; i < u32.Length; i++) result[i] = u32[i];
+                return result;
+            }
+            if (v is long[] i64)
+            {
+                var result = new ulong[i64.Length];
+                for (int i = 0; i < i64.Length; i++) result[i] = (ulong)i64[i];
+                return result;
+            }
+            return null;
+        }
+
         public uint[]? GetUint32Array(string key)
         {
             if (!Metadata.TryGetValue(key, out var v)) return null;
diff --git a/TensorSharp.Runtime/KVCache.cs b/TensorSharp.Runtime/KVCache.cs
index a2740179..066c27d9 100644
--- a/TensorSharp.Runtime/KVCache.cs
+++ b/TensorSharp.Runtime/KVCache.cs
@@ -196,6 +196,21 @@ public ReusePlan PlanReuse(IReadOnlyList<int> inputTokens, bool supportsTruncati
 
             int common = CommonPrefixLength(inputTokens);
 
+            if (Environment.GetEnvironmentVariable("TS_KV_DEBUG") == "1")
+            {
+                Console.Error.WriteLine($"[kv-debug] cached={_tokens.Count} input={inputTokens.Count} common={common}");
+                if (common < _tokens.Count && common < inputTokens.Count)
+                {
+                    int lo = Math.Max(0, common - 3);
+                    var cs = new System.Text.StringBuilder();
+                    var ns = new System.Text.StringBuilder();
+                    for (int i = lo; i < Math.Min(common + 4, _tokens.Count); i++) cs.Append(_tokens[i]).Append(' ');
+                    for (int i = lo; i < Math.Min(common + 4, inputTokens.Count); i++) ns.Append(inputTokens[i]).Append(' ');
+                    Console.Error.WriteLine($"[kv-debug] cache@{lo}: {cs}");
+                    Console.Error.WriteLine($"[kv-debug] input@{lo}: {ns}");
+                }
+            }
+
             // For non-truncatable models (recurrent state): only reuse if the cache is a
             // prefix of the new input.
             if (!supportsTruncation && common < _tokens.Count)
diff --git a/TensorSharp.Runtime/KVCachePromptRenderer.cs b/TensorSharp.Runtime/KVCachePromptRenderer.cs
index 5c2369b6..2e20f228 100644
--- a/TensorSharp.Runtime/KVCachePromptRenderer.cs
+++ b/TensorSharp.Runtime/KVCachePromptRenderer.cs
@@ -1,4 +1,4 @@
-// Copyright (c) Zhongkai Fu. All rights reserved.
+// Copyright (c) Zhongkai Fu. All rights reserved.
 // https://github.com/zhongkaifu/TensorSharp
 //
 // This file is part of TensorSharp.
@@ -106,6 +106,22 @@ internal static string GetAssistantGenerationSuffix(string architecture, bool en
             if (IsQwen35FamilyArch(architecture) && enableThinking)
                 return "<think>\n";
 
+            // qwen4exp appends `<think>` to the generation prompt UNCONDITIONALLY -
+            // the model always reasons - so the cache always holds it, whatever the
+            // thinking flag says. (The trailing newline is trimmed away again at
+            // interior boundaries when the renderer's own TrimEnd behaviour calls
+            // for it, mirroring the generation prompt.)
+            if (architecture == "qwen4exp")
+                return "<think>\n";
+
+            // GLM-5.3-Flash (glm5next) likewise ALWAYS opens a <think> block in the
+            // generation prompt (its template has no thinking-off shape), with no
+            // newline after it. Re-rendered history goes through the template's
+            // empty-<think></think> branch, which the strip above removes; this
+            // restores the half the cache actually holds.
+            if (architecture == "glm5next")
+                return "<think>";
+
             // Qwen 3.5 family with thinking DISABLED is rendered through the hardcoded
             // RenderQwen35 path, which DOES emit `<think>\n\n</think>\n\n` for past
             // assistant messages already - so no injection is needed.
@@ -189,7 +205,8 @@ private static bool IsQwen35FamilyArch(string architecture)
                 || architecture == "qwen35moe"
                 || architecture == "qwen3next"
                 || architecture == "qwen3vl"
-                || architecture == "qwen3vlmoe";
+                || architecture == "qwen3vlmoe"
+                || architecture == "qwen4exp";
         }
 
         /// <summary>
@@ -290,7 +307,7 @@ public List<int> RenderToTokens(
             // at the first assistant boundary, so the re-rendered prefix diverges there
             // and every multi-turn request re-prefills the whole conversation. Drop it
             // before injecting the suffix that reproduces what the cache actually saw.
-            if (enableThinking)
+            if (enableThinking || architecture == "qwen4exp" || architecture == "glm5next")
                 text = StripEmptyThinkBlockBeforePlaceholders(text);
 
             string suffix = GetAssistantGenerationSuffix(architecture, enableThinking);
diff --git a/TensorSharp.Runtime/OutputParser.cs b/TensorSharp.Runtime/OutputParser.cs
index b6918851..aad5f066 100644
--- a/TensorSharp.Runtime/OutputParser.cs
+++ b/TensorSharp.Runtime/OutputParser.cs
@@ -2068,7 +2068,7 @@ public static IOutputParser Create(string architecture)
                 "gptoss" or "gpt-oss" => new HarmonyOutputParser(),
                 "muse-glimmer" => new MuseGlimmerOutputParser(),
                 "deepseek4" => new DeepSeek4OutputParser(),
-                "glm-dsa" or "glm_dsa" => new GlmDsaOutputParser(),
+                "glm-dsa" or "glm_dsa" or "glm5next" => new GlmDsaOutputParser(),
                 "nemotron_h" or "nemotron_h_moe" => new Qwen3OutputParser(),
                 _ => new PassthroughOutputParser()
             };
@@ -2100,7 +2100,7 @@ public static bool IsAlwaysRequired(string architecture)
             // arrives on the "to=self" channel, so an unparsed stream shows the
             // raw tags and the whole chain of thought as if it were the answer.
             return architecture is "gptoss" or "gpt-oss" or "gemma4" or "deepseek4" or "muse-glimmer"
-                                or "glm-dsa" or "glm_dsa";
+                                or "glm-dsa" or "glm_dsa" or "glm5next";
         }
     }
 }
diff --git a/TensorSharp.Runtime/Scheduling/BatchExecutor.cs b/TensorSharp.Runtime/Scheduling/BatchExecutor.cs
index e8e6e7f6..03ca5f1f 100644
--- a/TensorSharp.Runtime/Scheduling/BatchExecutor.cs
+++ b/TensorSharp.Runtime/Scheduling/BatchExecutor.cs
@@ -1,4 +1,4 @@
-// Copyright (c) Zhongkai Fu. All rights reserved.
+// Copyright (c) Zhongkai Fu. All rights reserved.
 // https://github.com/zhongkaifu/TensorSharp
 //
 // This file is part of TensorSharp.
@@ -641,6 +641,12 @@ private List<SequenceStepResult> ExecuteStepBatched(IBatchedPagedModel batched,
             return results;
         }
 
+        // Continuous-batching routing trace, off unless TS_CB_DEBUG=1. Prints
+        // which path each step took, the scheduled work, and the current owner -
+        // the context a per-sequence cache bug is impossible to read without.
+        private static readonly bool _cbDebug =
+            string.Equals(Environment.GetEnvironmentVariable("TS_CB_DEBUG"), "1", StringComparison.Ordinal);
+
         /// <summary>Run every scheduled sequence through the model's fused
         /// single-graph <see cref="IModelArchitecture.Forward"/> with its own
         /// per-request KV cache (bound via
@@ -662,6 +668,14 @@ private List<SequenceStepResult> ExecuteStepPerSequenceFused(
             int n = output.ScheduledWork.Count;
             var results = new List<SequenceStepResult>(n);
             if (n == 0) return results;
+            if (_cbDebug)
+            {
+                var ids = new List<string>(n);
+                foreach (var w in output.ScheduledWork)
+                    ids.Add($"{w.Sequence.RequestId}:{(w.IsPrefill ? "P" : "D")}@{w.Sequence.NumComputedTokens}");
+                Console.Error.WriteLine($"[cb] FUSED step n={n} owner={_currentOwner?.RequestId ?? "<none>"}" +
+                    $" ownerStatus={(_currentOwner != null ? _currentOwner.Status.ToString() : "-")} work=[{string.Join(",", ids)}]");
+            }
 
             // Transition from the single-stream (N==1) path: if a prior owner's
             // K/V is still live in the model's primary cache, hand it to that
@@ -837,6 +851,14 @@ private List<SequenceStepResult> ExecuteStepPerSequence(SchedulerOutput output)
             var results = new List<SequenceStepResult>(1);
             if (output.ScheduledWork.Count == 0)
                 return results;
+            if (_cbDebug)
+            {
+                var ids = new List<string>();
+                foreach (var w in output.ScheduledWork)
+                    ids.Add($"{w.Sequence.RequestId}:{(w.IsPrefill ? "P" : "D")}@{w.Sequence.NumComputedTokens}");
+                Console.Error.WriteLine($"[cb] SOLO step n={output.ScheduledWork.Count} owner={_currentOwner?.RequestId ?? "<none>"}" +
+                    $" work=[{string.Join(",", ids)}]");
+            }
 
             // If a per-sequence-fused episode preceded this single-stream step,
             // the model's active KV cache may be a per-request holder. Reinstate
@@ -844,7 +866,11 @@ private List<SequenceStepResult> ExecuteStepPerSequence(SchedulerOutput output)
             // we never clobber a (possibly still-running) concurrent request's
             // cache. No-op when the primary cache is already active or the model
             // doesn't use per-request caches.
-            if (_model is IBatchedPagedModel pf && pf.SupportsPerSequenceFusedForward)
+            // Reinstated regardless of the CURRENT capability value: the
+            // capability can latch off after holders already exist (a fused-path
+            // failure), and skipping the restore would leave a per-request
+            // holder checked out for the universal path to trample.
+            if (_model is IBatchedPagedModel pf)
                 pf.RestorePrimaryCache();
 
             // The byte-level KV-state extract/inject in EnsureOwnership does
diff --git a/TensorSharp.Runtime/Speculative/DraftHeadSpeculator.cs b/TensorSharp.Runtime/Speculative/DraftHeadSpeculator.cs
index a221c1b6..e1def36f 100644
--- a/TensorSharp.Runtime/Speculative/DraftHeadSpeculator.cs
+++ b/TensorSharp.Runtime/Speculative/DraftHeadSpeculator.cs
@@ -48,6 +48,21 @@ public sealed class DraftHeadSpeculator : ISpeculator
         private readonly float[] _hA;
         private readonly float[] _hB;
 
+        // Catch-up folding (llama.cpp's draft-mtp, which runs its block over
+        // n_accepted + 1 rows instead of a catch-up pass plus a first draft step).
+        // Commit stashes the verified run instead of replaying it; the next Propose
+        // appends the token it starts from and does both in ONE head call. Worth a
+        // whole head call per speculative step, which on Qwen 3.8 is 6.4 ms of 100.
+        private readonly bool _fold;
+        private int[] _pendTokens;
+        private float[] _pendH;
+        private int _pendCount;
+        private int _pendStart;
+        private bool _hasPend;
+        // The folded call's inputs: the stashed run plus one row.
+        private int[] _foldTokens;
+        private float[] _foldH;
+
         public DraftHeadSpeculator(IDraftHead head, int vocabSize, int featureSize, int maxDraftTokens)
         {
             _head = head ?? throw new ArgumentNullException(nameof(head));
@@ -56,6 +71,7 @@ public DraftHeadSpeculator(IDraftHead head, int vocabSize, int featureSize, int
             _vocab = vocabSize;
             _featureSize = featureSize;
             MaxDraftTokens = maxDraftTokens;
+            _fold = head.SupportsFusedCatchUpStep;
             _logits = new float[vocabSize];
             _hA = new float[featureSize];
             _hB = new float[featureSize];
@@ -80,8 +96,51 @@ public int Propose(in DraftContext ctx, List<int> draftOut)
             float[] hIn = ctx.CarryHidden;
             float[] hOut = _hA;
             int tokIn = ctx.LastToken;
+            int first = 0;
+
+            // A stashed catch-up whose rows run right up to this step's position
+            // folds into draft step 0: one head call replays the verified run AND
+            // produces the first draft. Any other position means some path put a
+            // step in between, so replay it on its own and draft normally.
+            if (_hasPend)
+            {
+                if (_pendStart + _pendCount != ctx.Position)
+                {
+                    FlushPending();
+                }
+                else
+                {
+                    int n = _pendCount + 1;
+                    // EXACTLY n: the head takes its row count from tokens.Length, so a
+                    // buffer left long by an earlier, longer step would replay stale
+                    // trailing tokens as if they were verified rows.
+                    if (_foldTokens == null || _foldTokens.Length != n)
+                        _foldTokens = new int[n];
+                    if (_foldH == null || _foldH.Length < (long)n * _featureSize)
+                        _foldH = new float[(long)n * _featureSize];
+                    Array.Copy(_pendTokens, _foldTokens, _pendCount);
+                    _foldTokens[_pendCount] = ctx.LastToken;
+                    Array.Copy(_pendH, _foldH, (long)_pendCount * _featureSize);
+                    Array.Copy(ctx.CarryHidden, 0, _foldH, (long)_pendCount * _featureSize, _featureSize);
+                    _hasPend = false;
+
+                    _head.DraftCatchUpAndStep(_foldTokens, _foldH, _pendStart, _logits, hOut);
+                    if (ctx.MaxTokens < 1)
+                        return 0;
+                    ctx.AdjustLogits?.Invoke(_logits, draftOut);
+                    int d0 = ArgmaxWithTopKConfidence(_logits, _vocab, out float p0);
+                    if (p0 < MinDraftProb)
+                        return 0;
+                    draftOut.Add(d0);
+                    tokIn = d0;
+                    float[] nx = ReferenceEquals(hOut, _hA) ? _hB : _hA;
+                    hIn = hOut;
+                    hOut = nx;
+                    first = 1;
+                }
+            }
 
-            for (int i = 0; i < ctx.MaxTokens; i++)
+            for (int i = first; i < ctx.MaxTokens; i++)
             {
                 _head.DraftStep(tokIn, hIn, ctx.Position + i, _logits, hOut);
                 // Penalty-aligned drafting: argmax the SAME distribution
@@ -105,11 +164,42 @@ public int Propose(in DraftContext ctx, List<int> draftOut)
         }
 
         public void Commit(int[] tokens, float[] hRows, int startPos)
-            => _head.DraftCatchUp(tokens, hRows, startPos);
+        {
+            if (!_fold || hRows == null)
+            {
+                _head.DraftCatchUp(tokens, hRows, startPos);
+                return;
+            }
+            // Two commits with no Propose between them (a governor-declined step
+            // straight after a speculative one) must not lose the first.
+            FlushPending();
+            int n = tokens.Length;
+            if (_pendTokens == null || _pendTokens.Length < n)
+                _pendTokens = new int[n];
+            if (_pendH == null || _pendH.Length < (long)n * _featureSize)
+                _pendH = new float[(long)n * _featureSize];
+            Array.Copy(tokens, _pendTokens, n);
+            Array.Copy(hRows, _pendH, (long)n * _featureSize);
+            _pendCount = n;
+            _pendStart = startPos;
+            _hasPend = true;
+        }
+
+        /// <summary>Replay a stashed catch-up on its own, for when it cannot be
+        /// folded into the next draft (or there is no next draft).</summary>
+        private void FlushPending()
+        {
+            if (!_hasPend)
+                return;
+            _hasPend = false;
+            var toks = new int[_pendCount];
+            Array.Copy(_pendTokens, toks, _pendCount);
+            _head.DraftCatchUp(toks, _pendH, _pendStart);
+        }
 
-        public void Reset() { }
+        public void Reset() => FlushPending();
 
-        public void Dispose() { }
+        public void Dispose() => _hasPend = false;
 
         /// <summary>
         /// Argmax plus the top-1 probability computed over the top-10 logits
diff --git a/TensorSharp.Runtime/Speculative/ISpecTrunk.cs b/TensorSharp.Runtime/Speculative/ISpecTrunk.cs
index 7339c187..40be3695 100644
--- a/TensorSharp.Runtime/Speculative/ISpecTrunk.cs
+++ b/TensorSharp.Runtime/Speculative/ISpecTrunk.cs
@@ -34,6 +34,17 @@ public interface ISpecTrunk
         /// <summary>Snapshot recurrent state before a verify batch.</summary>
         void SnapshotRecurrentState();
 
+        /// <summary>
+        /// The accept count for the verify that just ran, handed to the trunk
+        /// BEFORE any rollback decision and on EVERY step - full acceptance
+        /// included. A trunk that defers part of its post-verify bookkeeping until
+        /// it knows how much was accepted settles it here; Qwen 3.5/3.8 uses it to
+        /// pick the recurrent-state snapshot for the accepted prefix, which is what
+        /// lets <see cref="TryCommitVerifiedPrefix"/> then succeed on a recurrent
+        /// model at all. Default: nothing to do.
+        /// </summary>
+        void OnVerifyAccepted(int acceptedRows, int verifyRows) { }
+
         /// <summary>Roll the trunk back to <paramref name="position"/>
         /// committed tokens: restore the recurrent snapshot and rewind any
         /// attention-KV bookkeeping.</summary>
@@ -65,6 +76,9 @@ public void Forward(int[] tokens, float[] hAllOut, float[] logitsOut, bool allLo
 
         public void SnapshotRecurrentState() => _model.SpecSnapshotRecurrentState();
 
+        public void OnVerifyAccepted(int acceptedRows, int verifyRows)
+            => _model.SpecOnVerifyAccepted(acceptedRows, verifyRows);
+
         public void Rollback(int position)
         {
             _model.SpecRestoreRecurrentState();
diff --git a/TensorSharp.Runtime/Speculative/SpeculationCostGovernor.cs b/TensorSharp.Runtime/Speculative/SpeculationCostGovernor.cs
index 57c77fa1..1d8d7c40 100644
--- a/TensorSharp.Runtime/Speculative/SpeculationCostGovernor.cs
+++ b/TensorSharp.Runtime/Speculative/SpeculationCostGovernor.cs
@@ -108,9 +108,15 @@ public sealed class SpeculationCostGovernor
         /// <summary>
         /// Measure speculation against plain decoding at runtime and skip drafting
         /// while it is measurably slower. On by default; set false to force the
-        /// drafter on for A/B measurement.
+        /// drafter on for A/B measurement (TS_SPEC_ADAPTIVE=0 does the same from
+        /// the environment, which is how the plain-baseline cost itself is
+        /// measured - a round's baseline steps are plain decodes, and they are not
+        /// free).
         /// </summary>
-        public bool Enabled { get; set; } = true;
+        public bool Enabled { get; set; } = DefaultEnabled;
+
+        private static readonly bool DefaultEnabled =
+            !string.Equals(Environment.GetEnvironmentVariable("TS_SPEC_ADAPTIVE"), "0", StringComparison.Ordinal);
 
         /// <summary>True while a losing verdict is actively suppressing drafting
         /// (as opposed to a round merely taking its plain baseline). Only these
diff --git a/TensorSharp.Runtime/Speculative/SpeculationOptions.cs b/TensorSharp.Runtime/Speculative/SpeculationOptions.cs
index 6a6dce79..d3a94ce1 100644
--- a/TensorSharp.Runtime/Speculative/SpeculationOptions.cs
+++ b/TensorSharp.Runtime/Speculative/SpeculationOptions.cs
@@ -38,6 +38,15 @@ public sealed record SpeculationOptions
         /// <c>TS_SPEC_DRAFT</c> / <c>TS_MTP_DRAFT</c>.</summary>
         public int MaxDraftTokens { get; init; } = DefaultMaxDraftTokens;
 
+        /// <summary>
+        /// True when the operator actually asked for <see cref="MaxDraftTokens"/>,
+        /// rather than inheriting the default. A model whose trunk makes a wide
+        /// window expensive can narrow the DEFAULT
+        /// (<see cref="ISpeculativeTarget.SpecPreferredDraftWindow"/>); it must not
+        /// silently override a number the operator typed.
+        /// </summary>
+        public bool MaxDraftTokensExplicit { get; init; }
+
         /// <summary>
         /// Confidence gate, or null to let the algorithm pick its own
         /// (<see cref="ISpeculator.DefaultMinDraftProb"/>). The gates threshold
@@ -87,6 +96,10 @@ public static SpeculationOptions FromEnvironment()
                 SpeculatorName = ReadString(SpeculationEnvVars.Type, null) ?? SpeculatorRegistry.Auto,
                 MaxDraftTokens = ReadPositiveInt(SpeculationEnvVars.Draft, SpeculationEnvVars.LegacyDraft,
                     DefaultMaxDraftTokens),
+                // The flags layer writes these only when the operator passed one.
+                MaxDraftTokensExplicit =
+                    !string.IsNullOrWhiteSpace(Environment.GetEnvironmentVariable(SpeculationEnvVars.Draft))
+                    || !string.IsNullOrWhiteSpace(Environment.GetEnvironmentVariable(SpeculationEnvVars.LegacyDraft)),
                 MinDraftProb = ReadFloatOrNull(SpeculationEnvVars.PMin, SpeculationEnvVars.LegacyPMin),
             };
         }
diff --git a/TensorSharp.Runtime/Speculative/SpeculativeCliFlags.cs b/TensorSharp.Runtime/Speculative/SpeculativeCliFlags.cs
index 164e7743..839aa850 100644
--- a/TensorSharp.Runtime/Speculative/SpeculativeCliFlags.cs
+++ b/TensorSharp.Runtime/Speculative/SpeculativeCliFlags.cs
@@ -53,6 +53,35 @@ public static class SpeculativeCliFlags
         /// <summary>Speculation algorithm (<c>--spec-type</c>).</summary>
         public const string TypeEnvVar = SpeculationEnvVars.Type;
 
+        /// <summary>
+        /// Every valueless switch <see cref="Apply"/> consumes, current spelling
+        /// and historical alias alike.
+        ///
+        /// This exists because the flags are applied in a pass SEPARATE from the
+        /// host's own argument parse, and that pass does not REMOVE what it
+        /// consumes: TensorSharp.Server then walks the same argv and throws
+        /// "Unknown option" for anything it does not recognise. Two hand-written
+        /// lists of the same flag names is a drift bug waiting to happen, and it
+        /// happened - the server knew only the legacy <c>--mtp-*</c> spellings, so
+        /// every documented <c>--spec*</c> flag made it refuse to start. Hosts MUST
+        /// consume these tables rather than re-typing the names.
+        /// </summary>
+        public static readonly string[] SwitchFlags =
+        {
+            "--spec", "--no-spec", "--mtp-spec", "--no-mtp-spec",
+        };
+
+        /// <summary>Every <c>--flag VALUE</c> option <see cref="Apply"/> consumes.
+        /// Longer names come first so a prefix match can never swallow a longer
+        /// flag's value. See <see cref="SwitchFlags"/> for why this table exists.</summary>
+        public static readonly string[] ValueFlags =
+        {
+            "--spec-draft-model", "--mtp-draft-model",
+            "--spec-draft", "--mtp-draft",
+            "--spec-type", "--mtp-type",
+            "--spec-pmin", "--mtp-pmin",
+        };
+
         /// <summary>Largest accepted draft window; see
         /// <see cref="SpeculationOptions.MaxAllowedDraftTokens"/>.</summary>
         public const int MaxDraftTokens = SpeculationOptions.MaxAllowedDraftTokens;
diff --git a/TensorSharp.Runtime/Speculative/SpeculativeExecution.cs b/TensorSharp.Runtime/Speculative/SpeculativeExecution.cs
index 4e66a9ad..8d386dd1 100644
--- a/TensorSharp.Runtime/Speculative/SpeculativeExecution.cs
+++ b/TensorSharp.Runtime/Speculative/SpeculativeExecution.cs
@@ -336,6 +336,12 @@ public SpeculativeStepOutcome DecodeStep(
             Stats.TokensDrafted += k;
             Stats.TokensAccepted += m;
 
+            // Tell the trunk the accept count BEFORE deciding how to roll back: a
+            // trunk that deferred post-verify state until this was known settles it
+            // now, and that is what can turn the partial-acceptance branch below from
+            // a whole second forward into a position rewind.
+            _trunk.OnVerifyAccepted(m, k);
+
             // The tokens this step commits to the trunk: the verify batch's
             // accepted prefix plus the token it started from.
             int[] keep = new int[m + 1];
diff --git a/TensorSharp.Runtime/Speculative/SpeculativeModelContracts.cs b/TensorSharp.Runtime/Speculative/SpeculativeModelContracts.cs
index 635af100..f03772f7 100644
--- a/TensorSharp.Runtime/Speculative/SpeculativeModelContracts.cs
+++ b/TensorSharp.Runtime/Speculative/SpeculativeModelContracts.cs
@@ -90,6 +90,23 @@ public interface ISpeculativeTarget : IModelArchitecture
         /// per-expert weights once per micro-batch want whole micro-batches.</summary>
         int SpecPrefillChunkSize => 0;
 
+        /// <summary>
+        /// Draft window this trunk would rather have by DEFAULT, or 0 for "no
+        /// preference". It narrows the default only; an operator who passed
+        /// <c>--spec-draft</c> gets exactly what they asked for.
+        ///
+        /// This exists because the cost of a wide window is a property of the
+        /// TRUNK, not of the drafter. On a model with recurrent state a verify
+        /// over N rows runs the chunked recurrent scan instead of the single-token
+        /// update, AND a partial rejection has to restore that state and re-advance
+        /// over the accepted prefix - so the marginal token of window costs far
+        /// more than it does on a dense-attention trunk, where the verify's own KV
+        /// writes are reusable and the rollback is a position rewind. Measured on
+        /// Qwen3.8-27B: the default window 8 gave 9.4 tok/s against 15.5 at 3,
+        /// with the same drafter and the same acceptance per position.
+        /// </summary>
+        int SpecPreferredDraftWindow => 0;
+
         /// <summary>
         /// True when a verify batch has already written reusable attention KV for
         /// EVERY token it processed (so the accepted prefix's KV is correct in the
@@ -121,6 +138,14 @@ public interface ISpeculativeTarget : IModelArchitecture
         /// <summary>Snapshot the recurrent (GDN/SSM) state before a verify batch.</summary>
         void SpecSnapshotRecurrentState();
 
+        /// <summary>
+        /// How much of the verify that just ran was accepted, delivered on every
+        /// speculative step (full acceptance included) before any rollback happens.
+        /// A trunk that left post-verify state on the device until the accept count
+        /// was known settles it here. Default: nothing to do.
+        /// </summary>
+        void SpecOnVerifyAccepted(int acceptedRows, int verifyRows) { }
+
         /// <summary>Restore the recurrent state captured by <see cref="SpecSnapshotRecurrentState"/>.</summary>
         void SpecRestoreRecurrentState();
 
@@ -249,6 +274,38 @@ void DraftStep(int token, float[] hPrev, int pos, float[] logitsOut, float[] hOu
         /// process()). Row k of <paramref name="hRows"/> is the hidden state of
         /// the token PRECEDING tokens[k].</summary>
         void DraftCatchUp(int[] tokens, float[] hRows, int startPos);
+
+        /// <summary>
+        /// True when this head can replay the verified tokens AND take the first
+        /// draft step in ONE pass, via <see cref="DraftCatchUpAndStep"/>.
+        ///
+        /// This is what llama.cpp's draft-mtp does: it runs its block over
+        /// <c>n_accepted + 1</c> rows rather than a catch-up pass followed by a
+        /// separate first draft. On a head whose per-call cost is mostly fixed
+        /// (a whole extra graph, its own launch and readback), that is one call
+        /// saved per speculative step - measured at 6.4 ms of a 100 ms step on
+        /// Qwen 3.8, i.e. the entire remaining gap to llama.cpp.
+        /// </summary>
+        bool SupportsFusedCatchUpStep => false;
+
+        /// <summary>
+        /// Replay verified trunk tokens and take the first draft step in one
+        /// pass. <paramref name="tokens"/> is the verified run followed by the
+        /// token the next draft starts from, so the last entry is NOT a replay;
+        /// row k of <paramref name="hRows"/> is the hidden state of the token
+        /// preceding tokens[k], as in <see cref="DraftCatchUp"/>. Fills
+        /// <paramref name="logitsOut"/> and <paramref name="hOut"/> from the LAST
+        /// row - byte-identical to what
+        /// <c>DraftCatchUp(tokens[..^1], ...)</c> followed by
+        /// <c>DraftStep(tokens[^1], ...)</c> would produce, because the block is
+        /// causal over its own KV and the last row therefore sees exactly the
+        /// replayed rows either way.
+        /// Only called when <see cref="SupportsFusedCatchUpStep"/> is true.
+        /// </summary>
+        void DraftCatchUpAndStep(int[] tokens, float[] hRows, int startPos,
+            float[] logitsOut, float[] hOut)
+            => throw new NotSupportedException(
+                "This draft head cannot fuse its catch-up with the first draft step.");
     }
 
     /// <summary>Convenience alias for the common case: a model that is its own
diff --git a/TensorSharp.Runtime/Speculative/SpeculatorRegistry.cs b/TensorSharp.Runtime/Speculative/SpeculatorRegistry.cs
index 63cc96d4..385d5fa1 100644
--- a/TensorSharp.Runtime/Speculative/SpeculatorRegistry.cs
+++ b/TensorSharp.Runtime/Speculative/SpeculatorRegistry.cs
@@ -182,7 +182,7 @@ private static ISpeculator CreateDraftHead(ISpeculativeTarget target, Speculatio
             if (target is not IDraftHead head || head.DraftHeadKind != DraftHeadKind.PerToken)
                 return null;
             return new DraftHeadSpeculator(head, target.Config.VocabSize, target.SpecFeatureSize,
-                Math.Max(1, options.MaxDraftTokens));
+                ResolveDraftWindow(target, options));
         }
 
         private static ISpeculator CreateBlock(ISpeculativeTarget target, SpeculationOptions options)
@@ -192,7 +192,22 @@ private static ISpeculator CreateBlock(ISpeculativeTarget target, SpeculationOpt
             int block = head.DraftBlockSize;
             if (block < 1)
                 return null;
-            return new BlockDraftSpeculator(head, block, Math.Max(1, options.MaxDraftTokens));
+            return new BlockDraftSpeculator(head, block, ResolveDraftWindow(target, options));
+        }
+
+        /// <summary>
+        /// The window to draft with: what the operator asked for, or - when they
+        /// asked for nothing - narrowed to what the trunk prefers. See
+        /// <see cref="ISpeculativeTarget.SpecPreferredDraftWindow"/> for why the
+        /// preference belongs to the target model and not to the algorithm.
+        /// </summary>
+        private static int ResolveDraftWindow(ISpeculativeTarget target, SpeculationOptions options)
+        {
+            int window = Math.Max(1, options.MaxDraftTokens);
+            int preferred = target.SpecPreferredDraftWindow;
+            if (!options.MaxDraftTokensExplicit && preferred > 0)
+                window = Math.Min(window, preferred);
+            return window;
         }
     }
 }
diff --git a/TensorSharp.Server/Hosting/ServerOptionsBuilder.cs b/TensorSharp.Server/Hosting/ServerOptionsBuilder.cs
index 07a9979a..e8c14854 100644
--- a/TensorSharp.Server/Hosting/ServerOptionsBuilder.cs
+++ b/TensorSharp.Server/Hosting/ServerOptionsBuilder.cs
@@ -534,18 +534,27 @@ public static bool ApplySpeculativeCliFlags(string[] args)
 
             for (int i = 0; i < args.Length; i++)
             {
-                // Path to a BLOCK drafter GGUF that has to be resident before
-                // the model's layer split runs (DeepSeek V4's DSpark). Unlike
+                // Path to a BLOCK drafter GGUF that has to be resident before the
+                // model's layer split runs: DeepSeek V4's DSpark, and the DFlash /
+                // DFlash2 drafters for Muse-Glimmer and Qwen 3.8. Unlike
                 // --spec-draft-model this one is handed to the model factory, so
-                // it is carried as the same env var the CLI's --draft-model
-                // reads and picked up again by a runtime model switch. It stays
+                // it is carried as the same env vars the CLI's --draft-model reads
+                // and picked up again by a runtime model switch. It stays
                 // server-local because the CLI passes its own --draft-model
                 // straight to ModelBase.Create instead.
+                //
+                // All THREE are set, because each architecture reads its own and
+                // only the loaded model reads any of them. Setting just the DSpark
+                // one - which is what this did - meant --draft-model was silently
+                // ignored on the server for every DFlash target, the flag's most
+                // common use.
                 if (SpeculativeCliFlags.TryReadOption(args, ref i, "--draft-model", out string dsparkOpt))
                 {
                     if (string.IsNullOrWhiteSpace(dsparkOpt) || !File.Exists(dsparkOpt))
                         throw new ArgumentException($"--draft-model file not found: '{dsparkOpt}'.");
                     Environment.SetEnvironmentVariable("TS_DSV4_DSPARK", dsparkOpt);
+                    Environment.SetEnvironmentVariable("TS_QWEN35_DFLASH", dsparkOpt);
+                    Environment.SetEnvironmentVariable("TS_MUSE_GLIMMER_DFLASH", dsparkOpt);
                     changed = true;
                 }
             }
@@ -1183,18 +1192,23 @@ private static void ParseArgs(
                 {
                     continue;
                 }
-                // MTP speculative-decoding flags are consumed by
-                // ApplySpeculativeCliFlags(args) in a separate earlier pass.
-                // Recognise + skip them here so they don't trip the
-                // unknown-arg trap below.
-                if (string.Equals(args[i], "--mtp-spec", StringComparison.OrdinalIgnoreCase) ||
-                    string.Equals(args[i], "--no-mtp-spec", StringComparison.OrdinalIgnoreCase))
+                // Speculative-decoding flags are consumed by
+                // ApplySpeculativeCliFlags(args) in a separate earlier pass, which
+                // READS argv without removing anything. Recognise + skip them here
+                // so they don't trip the unknown-arg trap below.
+                //
+                // Driven off SpeculativeCliFlags' own tables rather than a second
+                // hand-written list: the two lists used to be maintained
+                // separately, so when the flags were renamed --mtp-* -> --spec*
+                // the applier learned the new spellings and this trap did not.
+                // Every documented --spec* flag then made the server refuse to
+                // start with "Unknown option '--spec-draft'". A copy of a list is
+                // a drift bug; consume the source of truth.
+                if (MatchesAny(args[i], SpeculativeCliFlags.SwitchFlags))
                 {
                     continue;
                 }
-                if (TryReadOption(args, ref i, "--mtp-draft", out _)
-                    || TryReadOption(args, ref i, "--mtp-pmin", out _)
-                    || TryReadOption(args, ref i, "--mtp-draft-model", out _)
+                if (TryReadAnyOption(args, ref i, SpeculativeCliFlags.ValueFlags)
                     || TryReadOption(args, ref i, "--draft-model", out _))
                 {
                     continue;
@@ -1274,8 +1288,12 @@ private static string SuggestFlagCorrection(string typo)
                 "--paged-kv-ssd-dir", "--paged-kv-ssd-mb", "--paged-kv-quant-bits",
                 "--continuous-batching", "--no-continuous-batching",
                 "--paged-batching", "--no-paged-batching", "--prefill-chunk-size",
-                "--mtp-spec", "--no-mtp-spec", "--mtp-draft", "--mtp-pmin", "--mtp-draft-model",
+                // Speculative flags come from SpeculativeCliFlags' tables below
+                // (appended after this literal) so a new spelling is suggestible
+                // the moment it is accepted.
                 "--draft-model",
+                "--redis-url", "--paged-kv-redis-url", "--paged-kv-redis-ttl",
+                "--n-cpu-moe", "--cpu-moe", "--cpu-moe-threads",
                 "--qwen-image-vae", "--qwen-image-vl", "--qwen-image-mmproj", "--qwen-image-lora",
                 "--video-vae", "--video-text-encoder", "--video-te", "--video-dit2", "--audio-vae",
                 "--video-width", "--video-height", "--video-steps", "--video-mode",
@@ -1289,6 +1307,16 @@ private static string SuggestFlagCorrection(string typo)
             string best = null;
             int bestDist = int.MaxValue;
             foreach (var flag in knownFlags)
+            {
+                int d0 = LevenshteinDistance(typo, flag);
+                if (d0 < bestDist) { bestDist = d0; best = flag; }
+            }
+            foreach (var flag in SpeculativeCliFlags.SwitchFlags)
+            {
+                int d1 = LevenshteinDistance(typo, flag);
+                if (d1 < bestDist) { bestDist = d1; best = flag; }
+            }
+            foreach (var flag in SpeculativeCliFlags.ValueFlags)
             {
                 int d = LevenshteinDistance(typo, flag);
                 if (d < bestDist) { bestDist = d; best = flag; }
@@ -1298,6 +1326,32 @@ private static string SuggestFlagCorrection(string typo)
             return bestDist <= 2 ? best : null;
         }
 
+        /// <summary>True when <paramref name="arg"/> is exactly one of
+        /// <paramref name="flags"/> (case-insensitive). Used to consume the
+        /// valueless switches an earlier applier pass already handled.</summary>
+        private static bool MatchesAny(string arg, string[] flags)
+        {
+            foreach (var flag in flags)
+            {
+                if (string.Equals(arg, flag, StringComparison.OrdinalIgnoreCase))
+                    return true;
+            }
+            return false;
+        }
+
+        /// <summary>Consume the first of <paramref name="flags"/> that matches at
+        /// <paramref name="index"/>, in the order given (longest names first, so
+        /// <c>--spec-draft</c> can never eat <c>--spec-draft-model</c>'s value).</summary>
+        private static bool TryReadAnyOption(string[] args, ref int index, string[] flags)
+        {
+            foreach (var flag in flags)
+            {
+                if (TryReadOption(args, ref index, flag, out _))
+                    return true;
+            }
+            return false;
+        }
+
         private static int LevenshteinDistance(string a, string b)
         {
             if (string.IsNullOrEmpty(a)) return b?.Length ?? 0;
diff --git a/TensorSharp.Server/Hosting/ServerUsage.cs b/TensorSharp.Server/Hosting/ServerUsage.cs
index b688ad42..02d81adb 100644
--- a/TensorSharp.Server/Hosting/ServerUsage.cs
+++ b/TensorSharp.Server/Hosting/ServerUsage.cs
@@ -9,6 +9,7 @@
 // MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the BSD-3-Clause License for more details.
 
 using System;
+using System.Collections.Generic;
 using System.IO;
 
 namespace TensorSharp.Server.Hosting
@@ -135,6 +136,7 @@ private static readonly (string Section, OptionHelp[] Options)[] Sections =
                     "Split the model across N GPUs on this machine (tensor parallelism): each GPU holds 1/N of every " +
                     "weight and the shards cooperate on every token. Use it when a model does not fit on one GPU. " +
                     "Range: 1 to the number of local GPUs. Applies to the cuda, ggml_cuda, and ggml_vulkan backends. " +
+                    "Multi-GPU is implemented PER ARCHITECTURE, not per backend, and in two forms. Architectures that shard weights run true tensor parallelism. qwen4exp (Qwen3.8-Flash-Next) shards nothing, so --tp N runs it as a LAYER SPLIT instead - each GPU holds a contiguous run of whole layers, which is the same and only multi-GPU mode llama.cpp offers for it. That is a CAPACITY feature: it lets a model, context or resident-weight set that one GPU cannot hold fit across several, and is not expected to raise tok/s. The startup line says which mode actually ran. An architecture that supports neither says so on stderr and runs on one GPU rather than silently leaving the others idle. " +
                     "Default: 1 — no splitting (TENSORSHARP_TP_DEGREE env var overrides).",
                     "--model Qwen3.5-35B-A3B-Q4_K_M.gguf --backend ggml_cuda --tp 2"),
                 new OptionHelp("--tp-node-id <N>",
@@ -293,10 +295,12 @@ private static readonly (string Section, OptionHelp[] Options)[] Sections =
                     "--spec-draft-model gemma-4-E4B-it-assistant.Q8_0.gguf"),
                 new OptionHelp("--draft-model <path>",
                     "Block drafter GGUF for architectures whose drafter must be resident before the layer " +
-                    "split (DeepSeek V4's DSpark). Naming the file IS the request, so it needs no --spec; " +
-                    "engages for solo sequences on the cuda and ggml_cuda backends. Default: none; " +
-                    "env TS_DSV4_DSPARK.",
-                    "--draft-model DSpark-drafter-Q2K-Q8-0731.gguf"),
+                    "split: DeepSeek V4's DSpark, and the DFlash / DFlash2 drafters for Muse-Glimmer and " +
+                    "Qwen 3.8. The file's general.architecture decides which it is, not its name. Naming the " +
+                    "file IS the request, so it needs no --spec; engages for solo sequences on the cuda and " +
+                    "ggml_cuda backends. Default: none; env TS_DSV4_DSPARK / TS_QWEN35_DFLASH / " +
+                    "TS_MUSE_GLIMMER_DFLASH.",
+                    "--draft-model Qwen3.8-27B-DFlash2-Q4_K_M.gguf"),
             }),
             ("Qwen-Image-Edit companion models (qwen_image DiT GGUFs)", new[]
             {
@@ -423,6 +427,43 @@ private static readonly (string Section, OptionHelp[] Options)[] Sections =
             }),
         };
 
+        /// <summary>
+        /// Every flag token named on the usage page, placeholders stripped.
+        ///
+        /// Exists so a test can assert the page and the parser agree. They drifted
+        /// twice: <c>--wan-vae</c>/<c>--wan-te</c> and later every <c>--spec*</c>
+        /// spelling were documented here while ServerOptionsBuilder.ParseArgs
+        /// rejected them as unknown options, so the server refused to start on a
+        /// flag its own <c>--help</c> advertised.
+        ///
+        /// Placeholders are removed BEFORE splitting on '|', because a value
+        /// placeholder can itself contain one (<c>--mmproj &lt;path|none&gt;</c>,
+        /// <c>--sampling-precedence &lt;config|request&gt;</c>).
+        /// </summary>
+        internal static IEnumerable<string> DocumentedFlags()
+        {
+            foreach (var (_, options) in Sections)
+            {
+                foreach (var opt in options)
+                {
+                    string flag = opt.Flag;
+                    int lt;
+                    while ((lt = flag.IndexOf('<')) >= 0)
+                    {
+                        int gt = flag.IndexOf('>', lt);
+                        if (gt < 0) { flag = flag.Substring(0, lt); break; }
+                        flag = flag.Remove(lt, gt - lt + 1);
+                    }
+                    foreach (string part in flag.Split('|'))
+                    {
+                        string token = part.Trim();
+                        if (token.StartsWith("--", StringComparison.Ordinal))
+                            yield return token;
+                    }
+                }
+            }
+        }
+
         public static void PrintUsage(TextWriter writer)
         {
             writer.WriteLine("Usage: TensorSharp.Server [options]");
diff --git a/USAGE.md b/USAGE.md
index 9f4ca3ed..48da8317 100644
--- a/USAGE.md
+++ b/USAGE.md
@@ -277,7 +277,7 @@ quietly. Measured on gemma-4-26B-A4B (`--cpu-moe`, peak VRAM): `ggml_cuda`
 | `--spec-pmin <f>`<br>*(alias `--mtp-pmin`)* | Draft-confidence gate in `(0, 1]`; drafting stops at the first token below it. What the number MEANS is the algorithm's business, so each brings its own default: `0.75` for a per-token head (top-1 probability over its top-10 logits), `0.35` for a block drafter (the CUMULATIVE prefix probability, so the same number is far stricter), `0` for n-gram (where it scales the required match length instead). Env: `TS_SPEC_PMIN` (or `TS_MTP_PMIN`). |
 | `--spec-draft-model <path>`<br>*(alias `--mtp-draft-model`)* | Draft-head GGUF for architectures whose speculator weights ship as their own file (Gemma 4's `gemma4-assistant`). Loaded onto the target at startup so `--spec` can engage. The draft's hidden size must match the target (pair the 12B target with its 12B draft, not the 26B-A4B one). Qwen 3.6 and GLM 5.2 embed their NextN block in the trunk GGUF and need no such flag. Env: `TS_SPEC_DRAFT_MODEL`. |
 | `--draft-model <path>` | Speculative-decoding drafter GGUF for architectures whose drafter ships as its own file — DeepSeek V4's DSpark support module (see [DeepSeek V4](docs/models/deepseek4.md#dspark-speculative-decoding)) and Muse-Glimmer's DFlash block drafter (see [Muse-Glimmer](docs/models/muse-glimmer.md#3-dflash-speculative-decoding); env `TS_MUSE_GLIMMER_DFLASH`). It drafts a whole block per step and the trunk verifies it in one batched forward. Every emitted token is still drawn from a trunk row — with argmax under a greedy config, with the run's own sampler otherwise — so the output stream is unchanged either way. Engages on every single-sequence path (`--input`, `--multi-turn-jsonl`, `--interactive`) with `--backend cuda` or `--backend ggml_cuda`. Env: `TS_DSV4_DSPARK`. |
-| `--spec-draft-n-max <N>` | Older spelling of `--spec-draft`, kept for block drafters. Cap on tokens drafted per speculative block; a block drafter additionally clamps it to its trained block size. Range 1-64; default: that block size — 5 for DSpark, 15 for Muse-Glimmer's DFlash. |
+| `--spec-draft-n-max <N>` | Older spelling of `--spec-draft`, kept for block drafters. Cap on tokens drafted per speculative block; a block drafter additionally clamps it to its trained block size. Range 1-64; default: that block size — 5 for DSpark, 15 for Muse-Glimmer's DFlash, 7 for Qwen 3.8's DFlash2. On a **recurrent** trunk (Qwen 3.5/3.8's GatedDeltaNet layers) a narrow window is worth far more than a wide one: it bounds both the verify width and the rollback re-forward, and `--spec-draft 3` was 1.6x faster than the default on Qwen3.8-27B. |
 | `--spec-draft-conf-min <p>` | Older spelling of `--spec-pmin`, kept for block drafters, where the gate is the cumulative acceptance probability — the product of the confidence head's per-position estimates. Lower drafts further and rolls back more; higher falls back to plain decode more often. Default: `0.35` for a block drafter, `0.75` for a per-token head. |
 | `--temperature <f>` | Sampling temperature (0 = greedy) |
 | `--top-k <N>` | Top-K filtering (0 = disabled) |
@@ -331,7 +331,7 @@ quietly. Measured on gemma-4-26B-A4B (`--cpu-moe`, peak VRAM): `ggml_cuda`
 | `--ref-audio <file>` | Reference audio clip. Repeatable; referred to as `<Audio 1>`, `<Audio 2>`, … Resampled to the audio VAE's 32 kHz stereo and truncated to the generated clip's duration. |
 | `--no-audio` | Skip audio decoding on models that generate an audio track jointly with the video (MiniMax-H3), saving the audio VAE's time and memory. Ignored by video-only models. |
 | _(renamed flags)_ | `--wan-vae`, `--wan-te` and `--wan-dit2` became `--video-vae`, `--video-text-encoder` and `--video-dit2` when video generation stopped being Wan-only. The old spellings are still accepted everywhere — on the CLI, on the server, and as config-file keys — so existing configs keep working unchanged. |
-| `--tp <N>` | Tensor parallelism degree — split the model across N GPUs in a single process (default: `1`). Requires `--backend cuda`, `ggml_cuda`, or `ggml_vulkan`. See [Tensor Parallelism & Distributed Inference](#tensor-parallelism--distributed-inference). |
+| `--tp <N>` | Multi-GPU degree — how many GPUs to spread the model over in a single process (default: `1`). Which of the two multi-GPU modes you get is the architecture's business, not yours: **tensor parallelism** (the weights split *inside* every layer) where it is implemented, and a **layer split** (whole layers per GPU — capacity, not speed) on Qwen 3.8 Flash Next (`qwen4exp`) and DeepSeek V4. An architecture that supports neither says so on stderr and runs on one GPU. Requires `--backend cuda`, `ggml_cuda`, or `ggml_vulkan`. See [Tensor Parallelism & Distributed Inference](#tensor-parallelism--distributed-inference). |
 | `--tp-node-id <N>` | This node's 0-based ID for multi-node (distributed) tensor parallelism. Requires `--tp-peers`. |
 | `--tp-peers <list>` | Comma-separated `host:port` list of all nodes in the distributed TP cluster (e.g. `192.168.1.10:9500,192.168.1.11:9500`). Requires `--tp-node-id`. |
 | `--test` | Run built-in tokenizer + Qwen3 chat-template + ollama-comparison tests |
@@ -488,7 +488,7 @@ Running `TensorSharp.Server` with no arguments prints the full parameter referen
 | `--model <path>` | GGUF file to host (required for inference; when other options are passed without it, the server starts but `/api/models/load` will report no hosted model) |
 | `--mmproj <path>` | Multimodal projector GGUF (resolved relative to the model directory when only a filename is given; pass `none` to disable). Requires `--model`. |
 | `--backend <type>` | Default compute backend: `cpu`, `cuda`, `mlx`, `ggml_cpu`, `ggml_metal`, `ggml_cuda`, or `ggml_vulkan` |
-| `--tp <N>` | Tensor parallelism degree — split the hosted model across N local GPUs (default: `1`). Requires `--backend cuda`, `ggml_cuda`, or `ggml_vulkan`. Env: `TENSORSHARP_TP_DEGREE`. See [Tensor Parallelism & Distributed Inference](#tensor-parallelism--distributed-inference). |
+| `--tp <N>` | Multi-GPU degree — how many local GPUs to spread the hosted model over (default: `1`). Tensor parallelism where the architecture implements it; a layer split (whole layers per GPU — capacity, not speed) on Qwen 3.8 Flash Next (`qwen4exp`) and DeepSeek V4. Requires `--backend cuda`, `ggml_cuda`, or `ggml_vulkan`. Env: `TENSORSHARP_TP_DEGREE`. See [Tensor Parallelism & Distributed Inference](#tensor-parallelism--distributed-inference). |
 | `--tp-node-id <N>` | This node's 0-based ID for multi-node (distributed) tensor parallelism. The server can only be node `0` (the driver that serves HTTP); start worker nodes with `TensorSharp.Cli`. Requires `--tp-peers`. Env: `TENSORSHARP_TP_NODE_ID`. |
 | `--tp-peers <list>` | Comma-separated `host:port` list of all nodes in the distributed TP cluster, ordered by node ID (e.g. `192.168.1.10:9500,192.168.1.11:9500`). Requires `--tp-node-id`. Env: `TENSORSHARP_TP_PEERS`. |
 | `--gpu-device <N>` | Vulkan device index for the `ggml_vulkan` backend on multi-GPU hosts (e.g. an integrated Intel GPU next to a discrete NVIDIA one). Defaults to device 0; use `--list-gpus` to see the indices. Also settable via the `TS_GGML_VULKAN_DEVICE` env var. |
@@ -571,8 +571,9 @@ did unconditionally.
 | `TENSORSHARP_LOG_LEVEL` | Minimum log level for both console and file loggers: `Trace`, `Debug`, `Information`, `Warning`, `Error`, `Critical` (default: `Information`). Also honored by `TensorSharp.Cli`. |
 | `TENSORSHARP_LOG_DIR` | Directory the JSON-line file logger writes to (default: `<binDir>/logs`). Also honored by `TensorSharp.Cli`. |
 | `TENSORSHARP_LOG_FILE` | Set to `0` to disable the file logger and keep only the console output (default: enabled). Also honored by `TensorSharp.Cli`. |
-| `TENSORSHARP_TP_DEGREE` | Tensor parallelism degree — number of local GPUs to split the model across (default: `1`). Fallback in `ModelBase.Create` when no `--tp` flag is passed; both `TensorSharp.Cli` and `TensorSharp.Server` expose it as `--tp <N>`. Requires `--backend cuda`, `ggml_cuda`, or `ggml_vulkan`. |
+| `TENSORSHARP_TP_DEGREE` | Multi-GPU degree — number of local GPUs to spread the model over (default: `1`). Fallback in `ModelBase.Create` when no `--tp` flag is passed; both `TensorSharp.Cli` and `TensorSharp.Server` expose it as `--tp <N>`. Requires `--backend cuda`, `ggml_cuda`, or `ggml_vulkan`. On the architectures that run a layer split instead of tensor parallelism (`qwen4exp`, DeepSeek V4) it is a device count, not a shard count. |
 | `TENSORSHARP_TP_DEVICES` | GPU ordinals the TP ranks map to, comma-separated (e.g. `0,2`; default `0..tp-1`). Used by TP on the GGML backends. |
+| `TS_Q4E_LAYER_SPLIT` | Explicit per-GPU layer counts for the Qwen 3.8 Flash Next (`qwen4exp`) multi-GPU layer split, comma-separated (e.g. `20,28`), replacing the automatic VRAM balance. Throws rather than silently ignoring a value it cannot honour. |
 | `TENSORSHARP_TP_NODE_ID` | This node's 0-based ID for multi-node distributed tensor parallelism. Must be set together with `TENSORSHARP_TP_PEERS`. |
 | `TENSORSHARP_TP_PEERS` | Comma-separated `host:port` list of all nodes in the distributed TP cluster (e.g. `192.168.1.10:9500,192.168.1.11:9500`). Must be set together with `TENSORSHARP_TP_NODE_ID`. |
 | `TENSORSHARP_TP_CONNECT_TIMEOUT_SECONDS` | How long each node keeps retrying outbound connections to its peers before giving up (default: `120`). Raise it when nodes are started far apart by hand or by a slow orchestrator. |
@@ -1318,26 +1319,50 @@ and so on. The cut points are not a naive `n_layer/N`: the loader measures each
 device's free VRAM and bin-packs the layers to balance the largest
 fraction-of-budget used, so an uneven set of cards still fills up evenly. There
 are no collectives and no per-layer split, so it costs nothing on a slow
-interconnect, but only one GPU is busy at a time. This applies to
-exactly **two architectures — DeepSeek V4 Flash (`deepseek4`) and GLM 5.x
-(`glm-dsa`)** — because they are the two that run through their own whole-model
-executors, and it is what they do **by default, with no flag at all**: they spread
-across every visible GPU because neither fits on one card. `TS_DSV4_NGPU` and
-`TS_GLM_NGPU` cap how many devices they use.
+interconnect, but only one GPU is busy at a time. It applies to the architectures
+that run through their own whole-model executors: **DeepSeek V4 Flash
+(`deepseek4`)**, **GLM 5.x (`glm-dsa` / `glm5next`, GLM-5.2 and GLM-5.3-Flash alike)** and
+**Qwen 3.8 Flash Next (`qwen4exp`)**. On the first two it is what they do **by
+default, with no flag at all**: they spread across every visible GPU because
+neither fits on one card, and `TS_DSV4_NGPU` / `TS_GLM_NGPU` cap how many devices
+they use. On `qwen4exp` it is opt-in — one GPU unless you pass `--tp N`.
 
 On **every other architecture**, running without `--tp` uses a **single GPU**.
 There is no automatic layer split on the generic per-op or fused-graph paths — a
 model that does not fit one card fails at load rather than being spread silently.
 (The refusal that names the exact `--n-cpu-moe N` you would need comes from the
-DeepSeek V4 and GLM 5.x whole-model loaders.)
+DeepSeek V4 and GLM 5.x whole-model loaders.) Pass `--tp N` to an architecture
+that supports neither tensor parallelism nor a layer split and it now says so on
+stderr and runs on one GPU, instead of silently leaving the other cards idle.
 
 So on a 3-GPU box, `--backend ggml_cuda` alone gives you all three GPUs on GLM 5.x
-and DeepSeek V4 (layer split) and one GPU on Gemma 4; adding `--tp 3` switches the
-GLM 5.x to tensor parallelism, caps DeepSeek V4's layer split at three devices
+and DeepSeek V4 (layer split) and one GPU on Gemma 4 and Qwen 3.8 Flash Next;
+adding `--tp 3` switches the
+GLM 5.x to tensor parallelism (GLM-5.3-Flash refuses `--tp` and keeps its
+layer split), caps DeepSeek V4's layer split at three devices
 (there `--tp N` is only a device count — the same thing `TS_DSV4_NGPU` sets), and
-gives Gemma 4 all three GPUs. On GLM 5.x that
-switch is a downgrade in speed and buys only capacity — see the **What to
-expect** measurements below.
+gives Gemma 4 all three GPUs and Qwen 3.8 Flash Next a three-way layer split. On
+GLM 5.x that switch is a downgrade in speed and buys only capacity — see the
+**What to expect** measurements below.
+
+**Qwen 3.8 Flash Next (`qwen4exp`) is the new one.** `--tp N` there runs a layer
+split: each GPU holds a contiguous run of whole layers. It is not tensor
+parallelism — `qwen4exp` shards no weights — and it is the same (and only)
+multi-GPU mode llama.cpp offers this architecture, whose `-sm row` refuses to load
+it. Treat it as capacity. Measured on 2× A100-80GB with
+Qwen3.8-Flash-Next-UD-Q2_K_XL (73.4 GiB), the 1-GPU and 2-GPU runs produce
+**byte-identical greedy output** (same SHA-256); VRAM goes from one card holding
+everything to 24.2 GB + 26.2 GB, roughly half the model each; and throughput is
+unchanged either way — prefill ~1520-1550 t/s, decode ~56 t/s. For reference,
+llama.cpp on the same box measures pp1536 1094 / tg128 61.2 on one GPU and
+1200 / 61.5 on two with `-sm layer`, so it too gains ~10% prefill and nothing on
+decode. Startup prints which mode ran and the per-GPU layer/byte split.
+
+`TS_Q4E_LAYER_SPLIT=20,28` overrides the automatic balance with explicit layer
+counts per GPU (llama.cpp's `--tensor-split` in spirit). It throws rather than
+silently ignoring a value it cannot honour. It is worth reaching for because the
+automatic balance prices weights and cannot see the vision tower, which loads
+later and lands on GPU 0.
 
 ### Local tensor parallelism (single process, multiple GPUs)
 
@@ -1407,9 +1432,10 @@ must be reachable between all nodes.
 | Gemma 3 | ✅ | Separate Q/K/V, GELU, sliding window |
 | Gemma 4 | ✅ | Dense TP + MoE. On GGML the fused whole-model MoE trunk splits *inside* each expert (gate/up column-parallel, down row-parallel) so global expert ids keep working; `TS_GEMMA4_TP_FUSED_MOE=0` falls back to the whole-expert per-op path. Per-expert slicing on direct CUDA |
 | Qwen 3.5 / 3.6 family | ✅ | GatedDeltaNet SSM with per-rank V-head ownership; expert-parallel MoE on GGML (whole experts per rank, Megatron-split shared expert), expert slicing on direct CUDA. Runs on both `cuda` and `ggml_cuda` / `ggml_vulkan` — the GGML path uses the packed per-rank GDN kernel (`TSGgml_Qwen35GdnLayerTP`) with device-resident recurrent state |
+| Qwen 3.8 Flash Next | layer split | Not tensor parallelism: `--tp N` gives each GPU a contiguous run of whole layers, which is also the only multi-GPU mode llama.cpp offers `qwen4exp` (`-sm row` refuses to load it). Capacity, not speed — 2× A100-80GB on Qwen3.8-Flash-Next-UD-Q2_K_XL (73.4 GiB): greedy output byte-identical to the 1-GPU run (same SHA-256), VRAM 24.2 + 26.2 GB instead of one card holding everything, prefill ~1520-1550 t/s and decode ~56 t/s either way. `TS_Q4E_LAYER_SPLIT=20,28` sets the per-GPU layer counts by hand |
 | GPT OSS | ✅ | Attention sinks, YaRN. Runs on `cuda` and the GGML backends; the GGML path is expert-parallel (whole experts per rank, one batched `ggml_mul_mat_id` dispatch per projection per layer) and falls back to per-expert slicing only when the expert count does not divide the TP degree |
 | Nemotron-H | ✅ | Mamba2 replicated on rank 0, MoE expert slicing. Still walks experts per token per rank on GGML (no expert parallelism yet) |
-| GLM 5.x | ✅ | MLA heads column-parallel (`attn_q_b` / `attn_k_b` / `attn_v_b`) with row-parallel `attn_output`; the 256 routed experts are split **row-wise inside every expert** (column-parallel gate/up, row-parallel down) rather than by expert id, because `ggml_mul_mat_id` needs a token's selected expert ids to stay distinct. Router, norms, the DSA indexer, the shared expert and the 3 dense layers are replicated; two all-reduces per layer. `TS_GLM_TP_SHARD` picks the halves (1 heads, 2 experts, 3 both), `TS_GLM_TP_OVERSUBSCRIBE=1` packs several ranks on one GPU for testing. GGML backends only |
+| GLM 5.x | ✅ | MLA heads column-parallel (`attn_q_b` / `attn_k_b` / `attn_v_b`) with row-parallel `attn_output`; the 256 routed experts are split **row-wise inside every expert** (column-parallel gate/up, row-parallel down) rather than by expert id, because `ggml_mul_mat_id` needs a token's selected expert ids to stay distinct. Router, norms, the DSA indexer, the shared expert and the 3 dense layers are replicated; two all-reduces per layer. `TS_GLM_TP_SHARD` picks the halves (1 heads, 2 experts, 3 both), `TS_GLM_TP_OVERSUBSCRIBE=1` packs several ranks on one GPU for testing. GGML backends only. GLM-5.3-Flash (`glm5next`) is the exception: it refuses `--tp` and uses its default layer split instead |
 | DiffusionGemma | — | Not applicable (diffusion model) |
 | Qwen-Image-Edit | — | Not applicable (image generation) |
 
@@ -1481,6 +1507,7 @@ for the combined numbers.
 | `TS_GGML_TP_PARALLEL=0` | Drive ranks sequentially instead of concurrently (diagnostic) |
 | `TS_GGML_TP_FUSED_MATMUL=1` | Submit both ranks' linears from one thread (off by default; it allocates a device buffer per rank per call, measured 2.3× slower on Qwen 3.5 35B) |
 | `TS_GGML_TP_DEVICE_AR_THRESHOLD` | Element count above which AllReduce uses the device collective (default 262144) |
+| `TS_Q4E_LAYER_SPLIT=20,28` | Qwen 3.8 Flash Next only: explicit layer counts per GPU for its layer split, instead of the automatic VRAM balance. Throws rather than silently ignoring a value it cannot honour — useful because the automatic balance prices weights and cannot see the vision tower, which loads later and lands on GPU 0 |
 | `TS_GGML_F32_RESIDENT=0` | Bind F32 linear weights per call instead of keeping them device-resident (diagnostic) |
 | `TS_GEMMA4_TP_FUSED_MOE=0` | Gemma 4 only: fall back from the fused whole-model MoE trunk (Megatron split inside each expert) to the whole-expert per-op path. The fused path materializes ~10.5 GB of expert slices at load on the 26B (~36 s) in exchange for a ~10× decode. Layers offloaded by `--n-cpu-moe` are skipped by that materialization — they never run on the accelerator — so `--cpu-moe` also removes the load-time cost |
 | `GGML_CUDA_AR_BF16_THRESHOLD` | Payload size above which ggml-cuda's collective converts F32 to BF16 before reducing. TensorSharp raises ggml's default (1 byte — i.e. always) to 1 MB so decode-sized collectives reduce exactly; `0` disables the conversion entirely |
@@ -1569,8 +1596,13 @@ Quick reference for which environment variables (and matching CLI flags) gate ea
 | Max tokens drafted per step | `8` | `TS_SPEC_DRAFT` (legacy `TS_MTP_DRAFT`) | `--spec-draft N` |
 | Draft-confidence gate | per algorithm (`0.75` / `0.35` / `0`) | `TS_SPEC_PMIN` (legacy `TS_MTP_PMIN`) | `--spec-pmin X` |
 | Gemma 4 separate draft GGUF (`gemma4-assistant`) | none | `TS_SPEC_DRAFT_MODEL` (legacy `TS_MTP_DRAFT_MODEL`) | `--spec-draft-model <path>` |
-| Muse-Glimmer DFlash drafter GGUF | none | `TS_MUSE_GLIMMER_DFLASH` | `--draft-model <path>` |
-| Muse-Glimmer fused DFlash graphs (ggml) | ON | `TS_DFLASH_FUSED=0` falls back to the per-op drafter | — |
+| Muse-Glimmer DFlash / DFlash2 drafter GGUF | none | `TS_MUSE_GLIMMER_DFLASH` | `--draft-model <path>` |
+| Qwen 3.5 / 3.8 DFlash2 drafter GGUF | none | `TS_QWEN35_DFLASH` | `--draft-model <path>` |
+| Fused DFlash graphs (ggml) | ON | `TS_DFLASH_FUSED=0` falls back to the per-op drafter | — |
+| DFlash speculative prefill chunk | `1024` (capped by the drafter ring and the trunk's window) | `TS_DFLASH_PREFILL_CHUNK` | — |
+| DFlash2 candidate selector | ON when the checkpoint has one | `TS_DFLASH_SELECTOR=0` drafts by per-position argmax instead (diagnostic only) | — |
+| DFlash2 grouped convolution | ON when the checkpoint has one | `TS_DFLASH_CONV=0` drops it (diagnostic only) | — |
+| Qwen 3.5/3.8 recurrent-state snapshots | ON | `TS_Q35_VERIFY_SNAPSHOTS=0` reverts to restoring a pre-verify state copy and re-forwarding the accepted prefix (slower; see [qwen35.md §12.5](docs/models/qwen35.md)) | — |
 | Gemma 4 fused verify / draft kernels (ggml) | ON | `TS_GMTP_NO_FUSED=1` falls back to per-op | — |
 | Gemma 4 dense fast rollback on partial accept | ON | `TS_GMTP_NO_FAST_ROLLBACK=1` restores kept-prefix rollback | — |
 | Gemma 4 verify trunk path | linear (solo) | `TS_GMTP_BATCHED_TRUNK=1` runs the batched paged trunk | — |
@@ -1612,6 +1644,7 @@ The full list, including the debug and A/B knobs, is in the
 |---|---|---|---|
 | Local tensor parallelism (multi-GPU, single process) | OFF (`1` GPU) | **`TENSORSHARP_TP_DEGREE=N`** | `--tp N` (CLI and server) |
 | GPU ordinals used by the TP ranks (GGML backends) | `0..tp-1` | `TENSORSHARP_TP_DEVICES=0,2` | — |
+| Explicit per-GPU layer counts for the `qwen4exp` layer split (`--tp N`) | automatic VRAM balance | `TS_Q4E_LAYER_SPLIT=20,28` | — |
 | Distributed TP node ID (multi-node) | unset (disabled) | **`TENSORSHARP_TP_NODE_ID=N`** | `--tp-node-id N` (CLI and server; the server must be node `0`) |
 | Distributed TP peer endpoints | unset (disabled) | **`TENSORSHARP_TP_PEERS=host1:port1,host2:port2`** | `--tp-peers host1:port1,host2:port2` (CLI and server) |
 | Peer connect retry window (multi-node) | `120` s | `TENSORSHARP_TP_CONNECT_TIMEOUT_SECONDS=N` | — |
diff --git a/USAGE_zh-cn.md b/USAGE_zh-cn.md
index 2b41ff22..ceb4488f 100644
--- a/USAGE_zh-cn.md
+++ b/USAGE_zh-cn.md
@@ -245,7 +245,7 @@ dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model <model.gguf> --backend cu
 | `--cpu-moe` / `-cmoe` | `--n-cpu-moe all` 的简写。默认：关闭（环境变量 `TS_CPU_MOE`）。 |
 | `--cpu-moe-threads <N>` | 主机侧专家矩阵乘的工作线程数。默认：在核数多于 8 的主机上取本进程实际可用 CPU 并行度（`hardware_concurrency`，再受调度亲和性掩码与 cgroup CPU 配额约束）的**一半**，低于 8 时取「全部减一」。另一半并非浪费——加速器提交线程，以及 `TensorSharp.Server` 里的 Kestrel 与调度器，同样需要可被调度，而 .NET 自己的线程池是按机器 CPU 数而不是 cgroup 配额来定的。把它设到接近配额是悬崖而不是缓坡：在 95 CPU 配额下，托管的 26B MoE 在 64 线程时实测 20.7 tok/s，71 线程时只剩 8.2。独占机器上可以调高（环境变量 `TS_CPU_MOE_THREADS`）。 |
 | `--kv-cache-dtype <type>` | KV 缓存精度：`f32`（默认）、`f16`、`q8_0` 或 `q4_0`。量化 / 半精度 KV 缓存以微小数值漂移换取内存节省；`q4_0`（约 0.56 字节/元素，约为 f32 的 1/7）是最激进的档位，面向 KV 缓存占主导内存的超长（128K–256K）上下文。块量化缓存（`q8_0`/`q4_0`）需要原生 GGML flash 路径。 |
-| `--tp <N>` | 张量并行度 —— 在单个进程内把模型切分到 N 张 GPU 上（默认：`1`）。需要 `--backend cuda`、`ggml_cuda` 或 `ggml_vulkan`。详见[张量并行与分布式推理](#张量并行与分布式推理)。 |
+| `--tp <N>` | 多卡度 —— 单个进程内把模型摊到几张 GPU 上（默认：`1`）。到底走哪一种多卡模式由架构决定，而不是由你决定：实现了张量并行的走**张量并行**（在层*内部*切权重），Qwen 3.8 Flash Next（`qwen4exp`）与 DeepSeek V4 走**按层切分**（整层落在单卡 —— 买的是容量，不是速度）。两者都不支持的架构会在 stderr 上明确说明并只用一张卡。需要 `--backend cuda`、`ggml_cuda` 或 `ggml_vulkan`。详见[张量并行与分布式推理](#张量并行与分布式推理)。 |
 | `--tp-node-id <N>` | 多节点分布式张量并行中本节点的 0 起始编号。必须与 `--tp-peers` 一起使用。 |
 | `--tp-peers <list>` | 集群中所有节点的 `host:port` 列表（逗号分隔，例如 `192.168.1.10:9500,192.168.1.11:9500`）。所有节点必须使用完全相同的列表。必须与 `--tp-node-id` 一起使用。 |
 | `--interactive` / `-i` | 进入交互式 REPL 聊天会话（逐轮输入/输出），支持 KV 缓存复用、斜杠命令、运行时热切换 模型/后端/投影器、文件附件（图像、音频、视频、文本）以及实时调整采样参数。完整命令列表见下文「**交互式 REPL 命令**」一节 |
@@ -463,7 +463,7 @@ dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --config config/server-basi
 | `--model <path>` | 需要托管的 GGUF 文件（推理时必填；如传入了其他参数但未指定该项，服务仍可启动，但 `/api/models/load` 会报告未加载模型） |
 | `--mmproj <path>` | 多模态投影器 GGUF（仅给文件名时按模型目录解析；传 `none` 可显式禁用）。需要先指定 `--model`。 |
 | `--backend <type>` | 默认计算后端：`cpu`、`cuda`、`mlx`、`ggml_cpu`、`ggml_metal`、`ggml_cuda` 或 `ggml_vulkan` |
-| `--tp <N>` | 张量并行度 —— 把托管的模型切分到本机 N 张 GPU 上（默认：`1`）。需要 `--backend cuda`、`ggml_cuda` 或 `ggml_vulkan`。环境变量：`TENSORSHARP_TP_DEGREE`。详见[张量并行与分布式推理](#张量并行与分布式推理)。 |
+| `--tp <N>` | 多卡度 —— 把托管的模型摊到本机几张 GPU 上（默认：`1`）。架构实现了张量并行就走张量并行；Qwen 3.8 Flash Next（`qwen4exp`）与 DeepSeek V4 走按层切分（整层落在单卡 —— 买的是容量，不是速度）。需要 `--backend cuda`、`ggml_cuda` 或 `ggml_vulkan`。环境变量：`TENSORSHARP_TP_DEGREE`。详见[张量并行与分布式推理](#张量并行与分布式推理)。 |
 | `--tp-node-id <N>` | 多节点（分布式）张量并行中本节点的 0 起始编号。服务端只能是节点 `0`（对外提供 HTTP 的 driver）；其余节点请用 `TensorSharp.Cli` 启动。必须与 `--tp-peers` 一起使用。环境变量：`TENSORSHARP_TP_NODE_ID`。 |
 | `--tp-peers <list>` | 分布式 TP 集群中所有节点的 `host:port` 列表（逗号分隔，按节点 ID 排序，例如 `192.168.1.10:9500,192.168.1.11:9500`）。必须与 `--tp-node-id` 一起使用。环境变量：`TENSORSHARP_TP_PEERS`。 |
 | `--gpu-device <N>` | `ggml_vulkan` 后端使用的 Vulkan 设备索引，用于多 GPU 主机（例如同时装有 Intel 集成显卡和 NVIDIA 独立显卡的机器）。默认使用设备 0；可用 `--list-gpus` 查看索引。也可通过环境变量 `TS_GGML_VULKAN_DEVICE` 设置。 |
@@ -544,8 +544,9 @@ dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --config config/server-basi
 | `TENSORSHARP_LOG_LEVEL` | 控制台与文件日志的最低输出级别：`Trace`、`Debug`、`Information`、`Warning`、`Error`、`Critical`（默认：`Information`）。`TensorSharp.Cli` 同样识别该变量。 |
 | `TENSORSHARP_LOG_DIR` | JSON-line 文件日志的写入目录（默认：`<binDir>/logs`）。`TensorSharp.Cli` 同样识别该变量。 |
 | `TENSORSHARP_LOG_FILE` | 设为 `0` 可关闭文件日志，仅保留控制台输出（默认：开启）。`TensorSharp.Cli` 同样识别该变量。 |
-| `TENSORSHARP_TP_DEGREE` | 张量并行度 —— 把模型切分到本机多少张 GPU 上（默认：`1`）。当未传 `--tp` 参数时作为 `ModelBase.Create` 的兜底来源；`TensorSharp.Cli` 与 `TensorSharp.Server` 都提供了 `--tp <N>` 参数。需要 `--backend cuda`、`ggml_cuda` 或 `ggml_vulkan`。 |
+| `TENSORSHARP_TP_DEGREE` | 多卡度 —— 把模型摊到本机多少张 GPU 上（默认：`1`）。当未传 `--tp` 参数时作为 `ModelBase.Create` 的兜底来源；`TensorSharp.Cli` 与 `TensorSharp.Server` 都提供了 `--tp <N>` 参数。需要 `--backend cuda`、`ggml_cuda` 或 `ggml_vulkan`。在改走按层切分而非张量并行的架构（`qwen4exp`、DeepSeek V4）上，它只是个设备数，而不是切分度。 |
 | `TENSORSHARP_TP_DEVICES` | 各 rank 使用的 GPU 序号（逗号分隔，例如 `0,2`；默认 `0..tp-1`）。用于 GGML 后端上的 TP。 |
+| `TS_Q4E_LAYER_SPLIT` | Qwen 3.8 Flash Next（`qwen4exp`）多卡按层切分时每张 GPU 分到的层数（逗号分隔，例如 `20,28`），用于取代自动的显存均衡。给出无法满足的值时会直接抛错，而不是静默忽略。 |
 | `TENSORSHARP_TP_NODE_ID` | 多节点分布式张量并行中本节点的 0 起始编号。必须与 `TENSORSHARP_TP_PEERS` 一起设置。 |
 | `TENSORSHARP_TP_PEERS` | 分布式 TP 集群中所有节点的 `host:port` 列表（逗号分隔，例如 `192.168.1.10:9500,192.168.1.11:9500`）。必须与 `TENSORSHARP_TP_NODE_ID` 一起设置。 |
 | `TENSORSHARP_TP_CONNECT_TIMEOUT_SECONDS` | 各节点向 peer 发起连接的重试窗口（默认：`120` 秒）。若节点由人工或较慢的编排系统间隔较久启动，可调大该值。 |
@@ -1181,21 +1182,39 @@ TensorSharp 支持**张量并行（TP）**——按 Megatron-LM 列/行并行范
 **按层切分**则是把*整层*放到不同设备上并顺序执行——设备 0 算第 0..k 层，把隐状态
 交给设备 1，依此类推。切点并不是简单的 `n_layer/N`：加载器会测量每张卡的空闲显存，
 再做装箱以均衡「占用预算的最大比例」，因此配置不一致的一组卡也能被均匀填满。没有
-集合通信、层内也不切分，因此在慢速互连上不花额外代价，但同一时刻只有一张卡在忙。这只适用于**两个架构——DeepSeek V4 Flash（`deepseek4`）
-与 GLM 5.x（`glm-dsa`）**，因为只有它们走各自的整模型执行器；而且这是它们**不加任何
-参数时的默认行为**：它们会摊到所有可见 GPU 上，因为两者都装不进单卡。`TS_DSV4_NGPU`
-与 `TS_GLM_NGPU` 用来限制使用几张卡。
+集合通信、层内也不切分，因此在慢速互连上不花额外代价，但同一时刻只有一张卡在忙。它适用于所有
+走自己整模型执行器的架构——**DeepSeek V4 Flash（`deepseek4`）**、**GLM 5.x（`glm-dsa` / `glm5next`，
+GLM-5.2 与 GLM-5.3-Flash 都算）**与 **Qwen 3.8 Flash Next（`qwen4exp`）**。前两者**不加任何
+参数时的默认行为**就是它：它们会摊到所有可见 GPU 上，因为两者都装不进单卡，`TS_DSV4_NGPU`
+与 `TS_GLM_NGPU` 用来限制使用几张卡。`qwen4exp` 上则是按需开启——默认单卡，写了 `--tp N` 才切分。
 
 在**其他所有架构**上，不加 `--tp` 就是**只用一张 GPU**。通用的逐算子路径与融合图
 路径都没有自动按层切分——装不下单卡的模型会在加载时失败，而不会被悄悄摊开。（那条
 会告诉你正好需要多少 `--n-cpu-moe N` 的拒绝信息，来自 DeepSeek V4 与 GLM 5.x 的
-整模型加载器。）
+整模型加载器。）对既不支持张量并行、也没有按层切分的架构传 `--tp N`，现在会在 stderr
+上明确说明并回到单卡运行，而不是让其余的卡白白闲着。
 
 所以在一台 3 卡机器上：只写 `--backend ggml_cuda`，GLM 5.x 与 DeepSeek V4 会用满三张卡
-（按层切分），Gemma 4 只用一张；再加上 `--tp 3`，GLM 5.x 切换成张量并行，DeepSeek V4 只是把按层切分限制在这 3 张卡上
-（那里的 `--tp N` 仅仅是个设备数，与 `TS_DSV4_NGPU` 等价），Gemma 4 则用上三张卡。
+（按层切分），Gemma 4 与 Qwen 3.8 Flash Next 只用一张；再加上 `--tp 3`，GLM 5.x 切换成张量并行（GLM-5.3-Flash 会拒绝
+`--tp`，继续用按层切分），DeepSeek V4 只是把按层切分限制在这 3 张卡上
+（那里的 `--tp N` 仅仅是个设备数，与 `TS_DSV4_NGPU` 等价），Gemma 4 则用上三张卡，
+Qwen 3.8 Flash Next 得到三路按层切分。
 在 GLM 5.x 上这个切换只会更慢，换来的仅仅是容量——见下文的**预期效果**实测数据。
 
+**Qwen 3.8 Flash Next（`qwen4exp`）是新加入的那个。**它的 `--tp N` 走的是按层切分：每张
+GPU 拿到一段连续的整层。这不是张量并行——`qwen4exp` 不切分任何权重——而且这也是
+llama.cpp 对这个架构唯一提供的多卡模式（它的 `-sm row` 会直接拒绝加载）。所以请把它
+当作容量特性。在 2× A100-80GB、Qwen3.8-Flash-Next-UD-Q2_K_XL（73.4 GiB）上实测：单卡与
+双卡的贪心输出**逐字节一致**（SHA-256 相同）；显存从一张卡扛下全部变成 24.2 GB + 26.2 GB，
+大致各占一半；吞吐则没有变化——prefill 约 1520-1550 t/s、decode 约 56 t/s，两种跑法都一样。
+作为参照，同一台机器上的 llama.cpp 单卡为 pp1536 1094 / tg128 61.2，双卡 `-sm layer` 为
+1200 / 61.5——它同样是 prefill 约 +10%、decode 不变。启动时会打印实际走的是哪种模式，
+以及每张卡分到的层数与字节数。
+
+`TS_Q4E_LAYER_SPLIT=20,28` 可以用显式的每卡层数覆盖自动均衡（精神上等同于 llama.cpp 的
+`--tensor-split`）。给出无法满足的值时它会直接抛错，而不是静默忽略。这个开关值得一用，
+是因为自动均衡只按权重定价，看不到视觉塔——视觉塔是后加载的，会落在 GPU 0 上。
+
 ### 本地张量并行（单进程，多 GPU）
 
 在一个进程内把模型切分到 N 张 GPU 上（Direct CUDA，或 GGML CUDA / Vulkan 后端）。
@@ -1261,9 +1280,10 @@ dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model <model.gguf> --backend cu
 | Gemma 3 | ✅ | 分离 Q/K/V，GELU，滑动窗口 |
 | Gemma 4 | ✅ | 稠密 TP + MoE。GGML 上融合的整模 MoE 主干在**每个专家内部**切分（gate/up 列并行、down 行并行），从而保留全局专家 id；`TS_GEMMA4_TP_FUSED_MOE=0` 可回退到逐算子的整专家路径。Direct CUDA 上为逐专家切分 |
 | Qwen 3.5 / 3.6 family | ✅ | GatedDeltaNet SSM 按 rank 划分 V-head 归属；GGML 上为专家并行 MoE（每个 rank 持有整个专家，shared expert 仍按 Megatron 切分）与列并行 LM head，Direct CUDA 上为专家切分。`cuda` 与 `ggml_cuda` / `ggml_vulkan` 均可运行——GGML 路径使用打包的按 rank GDN 内核（`TSGgml_Qwen35GdnLayerTP`）并把循环状态常驻设备 |
+| Qwen 3.8 Flash Next | 按层切分 | 不是张量并行：`--tp N` 让每张 GPU 拿到一段连续的整层，这也是 llama.cpp 对 `qwen4exp` 唯一提供的多卡模式（它的 `-sm row` 会直接拒绝加载）。买的是容量而不是速度——2× A100-80GB、Qwen3.8-Flash-Next-UD-Q2_K_XL（73.4 GiB）实测：贪心输出与单卡逐字节一致（SHA-256 相同），显存从一张卡扛下全部变成 24.2 + 26.2 GB，prefill 约 1520-1550 t/s、decode 约 56 t/s 两种跑法一致。`TS_Q4E_LAYER_SPLIT=20,28` 可手动指定每卡层数 |
 | GPT OSS | ✅ | MoE 专家切分，attention sink，YaRN。`cuda` 与 GGML 后端均可运行；GGML 路径目前仍按 token 逐个遍历专家（尚未使用专家并行） |
 | Nemotron-H | ✅ | Mamba2 在 rank 0 上复制计算，MoE 专家切分。GGML 上的限制与 GPT OSS 相同 |
-| GLM 5.x | ✅ | MLA 的头按列并行（`attn_q_b` / `attn_k_b` / `attn_v_b`）、`attn_output` 按行并行；256 个路由专家是**在每个专家内部按行切**（gate/up 列并行、down 行并行）而不是按专家 id 切，因为 `ggml_mul_mat_id` 要求一个 token 选中的专家 id 互不相同。路由器、norm、DSA 索引器、共享专家与前 3 个稠密层都是复制的；每层两次 all-reduce。`TS_GLM_TP_SHARD` 选择切哪一半（1 头、2 专家、3 两者都切），`TS_GLM_TP_OVERSUBSCRIBE=1` 允许多个 rank 挤在同一张卡上做正确性测试。仅限 GGML 后端 |
+| GLM 5.x | ✅ | MLA 的头按列并行（`attn_q_b` / `attn_k_b` / `attn_v_b`）、`attn_output` 按行并行；256 个路由专家是**在每个专家内部按行切**（gate/up 列并行、down 行并行）而不是按专家 id 切，因为 `ggml_mul_mat_id` 要求一个 token 选中的专家 id 互不相同。路由器、norm、DSA 索引器、共享专家与前 3 个稠密层都是复制的；每层两次 all-reduce。`TS_GLM_TP_SHARD` 选择切哪一半（1 头、2 专家、3 两者都切），`TS_GLM_TP_OVERSUBSCRIBE=1` 允许多个 rank 挤在同一张卡上做正确性测试。仅限 GGML 后端。GLM-5.3-Flash（`glm5next`）是例外：它会拒绝 `--tp`，改用其默认的按层切分 |
 | DiffusionGemma | — | 不适用（扩散模型） |
 | Qwen-Image-Edit | — | 不适用（图像生成） |
 
@@ -1322,6 +1342,7 @@ tg64 17.6，而按层切分是 915.9 / 43.9——78 层里每一层都要对 `[6
 | `TS_GGML_TP_PARALLEL=0` | 顺序而非并发地驱动各 rank（诊断用） |
 | `TS_GGML_TP_FUSED_MATMUL=1` | 由单个线程提交两个 rank 的线性层（默认关闭；它每次调用都要为每个 rank 分配设备缓冲，在 Qwen 3.5 35B 上实测慢 2.3×） |
 | `TS_GGML_TP_DEVICE_AR_THRESHOLD` | 超过该元素数量时 AllReduce 走设备集合通信（默认 262144） |
+| `TS_Q4E_LAYER_SPLIT=20,28` | 仅 Qwen 3.8 Flash Next：用显式的每卡层数取代按层切分的自动显存均衡。给出无法满足的值时直接抛错而不是静默忽略——这个开关有用，是因为自动均衡只按权重定价，看不到后加载、会落在 GPU 0 上的视觉塔 |
 | `TS_GGML_F32_RESIDENT=0` | 每次调用重新绑定 F32 线性层权重，而不是常驻设备（诊断用） |
 | `TS_GEMMA4_TP_FUSED_MOE=0` | 仅 Gemma 4：从融合的整模 MoE 主干（专家内部 Megatron 切分）回退到逐算子的整专家路径。融合路径在 26B 上加载时会物化约 10.5 GB 的专家分片（约 36 秒），换来约 10× 的 decode |
 | `GGML_CUDA_AR_BF16_THRESHOLD` | ggml-cuda 集合通信在多大载荷以上把 F32 转成 BF16 再归约。TensorSharp 把 ggml 的默认值（1 字节，即总是转换）提高到 1 MB，使 decode 规模的集合通信精确归约；设为 `0` 则完全禁用转换 |
@@ -1449,6 +1470,7 @@ dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model <model.gguf> --back
 |---|---|---|---|
 | 本地张量并行（单进程，多 GPU） | 关闭（`1` 张 GPU） | **`TENSORSHARP_TP_DEGREE=N`** | `--tp N`（CLI 与服务端） |
 | TP 各 rank 使用的 GPU 序号（GGML 后端） | `0..tp-1` | `TENSORSHARP_TP_DEVICES=0,2` | — |
+| `qwen4exp` 按层切分（`--tp N`）每张卡的显式层数 | 自动显存均衡 | `TS_Q4E_LAYER_SPLIT=20,28` | — |
 | 分布式 TP 节点编号（多节点） | 未设置（关闭） | **`TENSORSHARP_TP_NODE_ID=N`** | `--tp-node-id N`（CLI 与服务端；服务端必须是节点 `0`） |
 | 分布式 TP peer 端点 | 未设置（关闭） | **`TENSORSHARP_TP_PEERS=host1:port1,host2:port2`** | `--tp-peers host1:port1,host2:port2`（CLI 与服务端） |
 | peer 连接重试窗口（多节点） | `120` 秒 | `TENSORSHARP_TP_CONNECT_TIMEOUT_SECONDS=N` | — |
diff --git a/benchmarks/ParityHarness/ParityHarness.csproj b/benchmarks/ParityHarness/ParityHarness.csproj
new file mode 100644
index 00000000..898e72e0
--- /dev/null
+++ b/benchmarks/ParityHarness/ParityHarness.csproj
@@ -0,0 +1,14 @@
+<Project Sdk="Microsoft.NET.Sdk">
+  <PropertyGroup>
+    <OutputType>Exe</OutputType>
+    <TargetFramework>net10.0</TargetFramework>
+    <Nullable>disable</Nullable>
+    <AllowUnsafeBlocks>true</AllowUnsafeBlocks>
+    <ImplicitUsings>disable</ImplicitUsings>
+    <TensorSharpSkipGgmlNative>true</TensorSharpSkipGgmlNative>
+    <TensorSharpSkipMlxNative>true</TensorSharpSkipMlxNative>
+  </PropertyGroup>
+  <ItemGroup>
+    <ProjectReference Include="../../TensorSharp.Models/TensorSharp.Models.csproj" />
+  </ItemGroup>
+</Project>
diff --git a/benchmarks/ParityHarness/Program.cs b/benchmarks/ParityHarness/Program.cs
new file mode 100644
index 00000000..bd695341
--- /dev/null
+++ b/benchmarks/ParityHarness/Program.cs
@@ -0,0 +1,322 @@
+// Copyright (c) Zhongkai Fu. All rights reserved.
+// https://github.com/zhongkaifu/TensorSharp
+//
+// This file is part of TensorSharp.
+//
+// TensorSharp is licensed under the BSD-3-Clause license found in the LICENSE file in the root directory of this source tree.
+//
+// Cross-engine parity + throughput harness. Feeds RAW token ids (so the model
+// is isolated from tokenizer/template differences) and compares against
+// llama.cpp greedy goldens, measures llama-bench-shaped throughput, and checks
+// that batched (continuous-batching) decode reproduces serial decode.
+//
+// Modes:
+//   parity <model.gguf> --ref <golden.json> [backend] [max_new]
+//   parity <model.gguf> --bench <backend> <pp1,pp2,...> <tg> [reps]
+//   parity <model.gguf> --batched <backend> <steps> <promptA> <promptB> [...]
+//       prompts are comma-separated token ids; each runs on its own sequence
+//       slot serially first, then all together through the fused batched
+//       decode; the two must agree token for token.
+//   parity <model.gguf> <tok0,tok1,...> [n_predict] [backend]   raw greedy
+using System;
+using System.Collections.Generic;
+using System.Diagnostics;
+using System.Globalization;
+using System.IO;
+using System.Linq;
+using System.Text.Json;
+using TensorSharp;
+using TensorSharp.Models;
+using TensorSharp.Runtime;
+using TensorSharp.Runtime.Scheduling;
+
+public static class Program
+{
+    private sealed class RefRecord
+    {
+        public string prompt { get; set; }
+        public int[] prompt_tokens { get; set; }
+        public int[] generated_tokens { get; set; }
+    }
+
+    private static BackendType ResolveBackend(string s) => s switch
+    {
+        "ggmlcuda" or "ggml_cuda" => BackendType.GgmlCuda,
+        "ggmlcpu" or "ggml_cpu" => BackendType.GgmlCpu,
+        "ggmlvulkan" or "ggml_vulkan" => BackendType.GgmlVulkan,
+        "cuda" => BackendType.Cuda,
+        _ => BackendType.Cpu,
+    };
+
+    private static int ResolveTp()
+    {
+        string raw = Environment.GetEnvironmentVariable("TENSORSHARP_TP_DEGREE");
+        return int.TryParse(raw, out int v) && v > 1 ? v : 1;
+    }
+
+    private static int ArgMax(float[] v)
+    {
+        int best = 0;
+        for (int i = 1; i < v.Length; i++)
+            if (v[i] > v[best]) best = i;
+        return best;
+    }
+
+    public static int Main(string[] args)
+    {
+        if (args.Length < 2)
+        {
+            Console.Error.WriteLine("usage: parity <model.gguf> --ref|--bench|--batched|<tokens> ...");
+            return 1;
+        }
+
+        string modelPath = args[0];
+        if (args[1] == "--ref") return RunReference(modelPath, args);
+        if (args[1] == "--bench") return RunBench(modelPath, args);
+        if (args[1] == "--batched") return RunBatched(modelPath, args);
+        return RunRaw(modelPath, args);
+    }
+
+    private static int RunRaw(string modelPath, string[] args)
+    {
+        int[] tokens = args[1].Split(',', StringSplitOptions.RemoveEmptyEntries)
+                              .Select(t => int.Parse(t.Trim(), CultureInfo.InvariantCulture)).ToArray();
+        int nPredict = args.Length > 2 ? int.Parse(args[2]) : 0;
+        BackendType backend = ResolveBackend(args.Length > 3 ? args[3] : "ggmlcuda");
+
+        using var model = ModelBase.Create(modelPath, backend, ResolveTp());
+        float[] logits = model.Forward(tokens);
+        Console.WriteLine($"n_vocab {logits.Length}");
+        Console.WriteLine("logits " + string.Join(' ', logits.Select(v => v.ToString("F6", CultureInfo.InvariantCulture))));
+        if (nPredict > 0)
+        {
+            var gen = new List<int>();
+            int tok = ArgMax(logits);
+            gen.Add(tok);
+            for (int i = 1; i < nPredict; i++)
+            {
+                logits = model.Forward(new[] { tok });
+                tok = ArgMax(logits);
+                gen.Add(tok);
+            }
+            Console.WriteLine("generated " + string.Join(' ', gen));
+        }
+        return 0;
+    }
+
+    private static int RunReference(string modelPath, string[] args)
+    {
+        string refPath = args[2];
+        BackendType backend = ResolveBackend(args.Length > 3 ? args[3] : "ggmlcuda");
+        int maxNew = args.Length > 4 ? int.Parse(args[4]) : int.MaxValue;
+
+        var records = JsonSerializer.Deserialize<List<RefRecord>>(File.ReadAllText(refPath));
+        var sw = Stopwatch.StartNew();
+        using var model = ModelBase.Create(modelPath, backend, ResolveTp());
+        Console.WriteLine($"[parity] model loaded in {sw.Elapsed.TotalSeconds:F1}s, backend={backend}, arch={model.Config.Architecture}");
+
+        int total = 0, matched = 0;
+        foreach (var rec in records)
+        {
+            if (rec.prompt_tokens == null || rec.prompt_tokens.Length == 0) continue;
+            int want = Math.Min(rec.generated_tokens.Length, maxNew);
+
+            model.ResetKVCache();
+            var swPrefill = Stopwatch.StartNew();
+            float[] logits = model.ForwardRefill(rec.prompt_tokens);
+            swPrefill.Stop();
+
+            var produced = new int[want];
+            var swDecode = Stopwatch.StartNew();
+            for (int i = 0; i < want; i++)
+            {
+                produced[i] = ArgMax(logits);
+                if (i + 1 < want)
+                    logits = model.Forward(new[] { produced[i] });
+            }
+            swDecode.Stop();
+
+            int firstDiff = -1;
+            for (int i = 0; i < want; i++)
+                if (produced[i] != rec.generated_tokens[i]) { firstDiff = i; break; }
+
+            total++;
+            string label = rec.prompt != null && rec.prompt.Length > 44 ? rec.prompt.Substring(0, 44) + "..." : rec.prompt;
+            double pp = rec.prompt_tokens.Length / Math.Max(1e-9, swPrefill.Elapsed.TotalSeconds);
+            double tg = Math.Max(0, want - 1) / Math.Max(1e-9, swDecode.Elapsed.TotalSeconds);
+            if (firstDiff < 0)
+            {
+                matched++;
+                Console.WriteLine($"[MATCH ] {label}  ({want} tokens)  prefill {pp:F1} tok/s  decode {tg:F2} tok/s");
+            }
+            else
+            {
+                Console.WriteLine($"[DIFF  ] {label}  diverges at {firstDiff}/{want}  prefill {pp:F1} tok/s  decode {tg:F2} tok/s");
+                Console.WriteLine($"          ref: {string.Join(' ', rec.generated_tokens.Take(want))}");
+                Console.WriteLine($"          ts : {string.Join(' ', produced)}");
+            }
+        }
+        Console.WriteLine($"[parity] {matched}/{total} prompts reproduce llama.cpp token-for-token");
+        return matched == total ? 0 : 2;
+    }
+
+    /// <summary>llama-bench-shaped throughput: synthetic prompt of P tokens
+    /// (prefill t/s), then TG greedy decode steps (decode t/s), best of reps.</summary>
+    private static int RunBench(string modelPath, string[] args)
+    {
+        BackendType backend = ResolveBackend(args[2]);
+        int[] ppLens = args[3].Split(',', StringSplitOptions.RemoveEmptyEntries).Select(int.Parse).ToArray();
+        int tg = args.Length > 4 ? int.Parse(args[4]) : 64;
+        int reps = args.Length > 5 ? int.Parse(args[5]) : 2;
+
+        var sw = Stopwatch.StartNew();
+        using var model = ModelBase.Create(modelPath, backend, ResolveTp());
+        Console.WriteLine($"[bench] loaded in {sw.Elapsed.TotalSeconds:F1}s, backend={backend}, arch={model.Config.Architecture}");
+
+        var rng = new Random(42);
+        int vocab = Math.Max(1000, model.Config.VocabSize - 1000);
+
+        foreach (int pp in ppLens)
+        {
+            double best = 0;
+            var prompt = new int[pp];
+            for (int i = 0; i < pp; i++) prompt[i] = 1000 + rng.Next(vocab - 1000);
+            for (int r = 0; r < reps; r++)
+            {
+                model.ResetKVCache();
+                var t = Stopwatch.StartNew();
+                model.ForwardRefill(prompt);
+                t.Stop();
+                best = Math.Max(best, pp / t.Elapsed.TotalSeconds);
+            }
+            Console.WriteLine($"[bench] pp{pp,-8} {best,10:F2} tok/s");
+        }
+
+        {
+            double best = 0;
+            var prompt = new int[32];
+            for (int i = 0; i < 32; i++) prompt[i] = 1000 + rng.Next(vocab - 1000);
+            for (int r = 0; r < reps; r++)
+            {
+                model.ResetKVCache();
+                float[] logits = model.ForwardRefill(prompt);
+                int tok = ArgMax(logits);
+                var t = Stopwatch.StartNew();
+                for (int i = 0; i < tg; i++)
+                {
+                    logits = model.Forward(new[] { tok });
+                    tok = ArgMax(logits);
+                }
+                t.Stop();
+                best = Math.Max(best, tg / t.Elapsed.TotalSeconds);
+            }
+            Console.WriteLine($"[bench] tg{tg,-8} {best,10:F2} tok/s");
+        }
+        return 0;
+    }
+
+    /// <summary>Continuous-batching equivalence: each prompt decodes serially on
+    /// its own sequence slot, then all together through the fused batched-decode
+    /// step. Batching changes when the weights are read, not what the model
+    /// computes, so the streams must agree token for token.</summary>
+    private static int RunBatched(string modelPath, string[] args)
+    {
+        BackendType backend = ResolveBackend(args.Length > 2 ? args[2] : "ggmlcuda");
+        int steps = args.Length > 3 ? int.Parse(args[3]) : 8;
+        var prompts = new List<int[]>();
+        for (int i = 4; i < args.Length; i++)
+            prompts.Add(args[i].Split(',', StringSplitOptions.RemoveEmptyEntries)
+                               .Select(t => int.Parse(t.Trim(), CultureInfo.InvariantCulture)).ToArray());
+        if (prompts.Count < 2) { Console.Error.WriteLine("need at least two prompts"); return 1; }
+
+        using var model = ModelBase.Create(modelPath, backend, ResolveTp());
+        var seq = model as IBatchedPagedModel;
+        if (seq == null || !seq.SupportsPerSequenceFusedForward)
+        {
+            Console.Error.WriteLine("[batched] model has no per-sequence slots");
+            return 1;
+        }
+
+        int n = prompts.Count;
+        var ids = new string[n];
+        for (int i = 0; i < n; i++) ids[i] = "req" + i;
+
+        // --- serial: each sequence decoded on its own slot ---
+        var serial = new List<List<int>>();
+        for (int i = 0; i < n; i++)
+        {
+            seq.BindSequenceCache(ids[i]);
+            float[] lg = model.Forward(prompts[i]);
+            var outs = new List<int>();
+            int tok = ArgMax(lg);
+            outs.Add(tok);
+            for (int s = 1; s < steps; s++)
+            {
+                lg = model.Forward(new[] { tok });
+                tok = ArgMax(lg);
+                outs.Add(tok);
+            }
+            serial.Add(outs);
+            seq.OnSequenceReleased(ids[i]);
+        }
+
+        // --- batched: same sequences, one fused step per token ---
+        var lastTok = new int[n];
+        var pos = new int[n];
+        var batched = new List<List<int>>();
+        for (int i = 0; i < n; i++)
+        {
+            seq.BindSequenceCache(ids[i]);
+            float[] lg = model.Forward(prompts[i]);
+            lastTok[i] = ArgMax(lg);
+            pos[i] = prompts[i].Length;
+            batched.Add(new List<int> { lastTok[i] });
+        }
+
+        var outLogits = new float[n][];
+        int fusedSteps = 0, fallbackSteps = 0;
+        for (int s = 1; s < steps; s++)
+        {
+            if (seq.TryForwardBatchedFusedDecode(ids, lastTok, pos, outLogits))
+            {
+                fusedSteps++;
+                for (int i = 0; i < n; i++)
+                {
+                    lastTok[i] = ArgMax(outLogits[i]);
+                    pos[i]++;
+                    batched[i].Add(lastTok[i]);
+                }
+            }
+            else
+            {
+                // round-robin fallback, as the engine would
+                fallbackSteps++;
+                for (int i = 0; i < n; i++)
+                {
+                    seq.BindSequenceCache(ids[i]);
+                    float[] lg = model.Forward(new[] { lastTok[i] });
+                    lastTok[i] = ArgMax(lg);
+                    pos[i]++;
+                    batched[i].Add(lastTok[i]);
+                }
+            }
+        }
+        for (int i = 0; i < n; i++) seq.OnSequenceReleased(ids[i]);
+
+        bool allMatch = true;
+        for (int i = 0; i < n; i++)
+        {
+            bool same = serial[i].SequenceEqual(batched[i]);
+            allMatch &= same;
+            Console.WriteLine($"[batched] seq{i}: {(same ? "MATCH" : "DIFF")}");
+            if (!same)
+            {
+                Console.WriteLine($"          serial : {string.Join(' ', serial[i])}");
+                Console.WriteLine($"          batched: {string.Join(' ', batched[i])}");
+            }
+        }
+        Console.WriteLine($"[batched] fused steps={fusedSteps} fallback steps={fallbackSteps}");
+        Console.WriteLine(allMatch ? "[batched] CONCURRENT_MATCH" : "[batched] CONCURRENT_DIFFERS");
+        return allMatch ? 0 : 2;
+    }
+}
diff --git a/docs/env_var_feature_matrix.md b/docs/env_var_feature_matrix.md
index b2221b07..cda0d199 100644
--- a/docs/env_var_feature_matrix.md
+++ b/docs/env_var_feature_matrix.md
@@ -180,8 +180,20 @@ A/B switch, plus long-context sizing knobs. None are registered in
 | `TS_MUSE_GLIMMER_VENC_F32` | Muse-Glimmer vision tower | Dequantize the tower to F32 (~7.4 GB) instead of feeding the GGUF quantization to `AddmmQuant` | OFF | not registered | no |
 | `TS_MUSE_GLIMMER_VENC_FUSED` | Muse-Glimmer vision tower on CUDA | Fused vision-block / flash-attention path | ON | not registered | no |
 | `TS_MUSE_GLIMMER_DFLASH` | Muse-Glimmer | DFlash drafter GGUF path (same as the CLI's `--draft-model`) | none | not registered | no |
-| `TS_DFLASH_FUSED` | Muse-Glimmer DFlash | Fused `TSGgml_DFlashInject` / `TSGgml_DFlashDraftBlock` graphs vs the per-op drafter | ON | not registered | no |
-| `TS_DFLASH_PERSIST` | Muse-Glimmer DFlash | Replay the persistent draft graphs instead of rebuilding every step | ON | not registered | no |
+| `TS_QWEN35_DFLASH` | Qwen 3.5 / 3.8 | DFlash / DFlash2 drafter GGUF path (same as the CLI's `--draft-model`) | none | not registered | no |
+| `TS_DFLASH_FUSED` | any DFlash drafter | Fused `TSGgml_DFlashInject` / `TSGgml_DFlashDraftBlock` graphs vs the per-op drafter | ON | not registered | no |
+| `TS_DFLASH_PERSIST` | any DFlash drafter | Replay the persistent draft graphs instead of rebuilding every step | ON | not registered | no |
+| `TS_DFLASH_PREFILL_CHUNK` | any DFlash drafter | Tokens per speculative prefill forward (drives the TRUNK, not only the drafter) | `1024`, capped by the drafter ring and the trunk's own window | not registered | no |
+| `TS_DFLASH_SELECTOR` | DFlash2 drafter | `0` drafts by per-position argmax instead of the candidate lattice (attribution only - the weights were trained with it) | ON | not registered | no |
+| `TS_DFLASH_CONV` | DFlash2 drafter | `0` drops the grouped dynamic convolution (attribution only, as above) | ON | not registered | no |
+| `TS_DFLASH_SELECTOR_DEBUG` | DFlash2 drafter (per-op path) | `1` prints the first blocks' lattice attribution: unary spread, transition spread, and whether the walk left the unary argmax | OFF | not registered | no |
+| `TS_Q35_VERIFY_SNAPSHOTS` | Qwen 3.5 / 3.8 speculative verify | `0` reverts to restoring a pre-verify recurrent-state copy and re-forwarding the accepted prefix instead of keeping one snapshot per row | ON | not registered | no |
+| `TS_Q35_VERIFY_DEFER_STATE` | Qwen 3.5 / 3.8 speculative verify | `0` downloads the post-window recurrent state after every persisted call instead of leaving it on the device for a slot commit; separable from the snapshots because it also covers the single-row steps a speculative session interleaves with verifies | ON | not registered | no |
+| `TS_Q35_VERIFY_STRIDED_VIEWS` | Qwen 3.5 / 3.8 speculative verify | `0` disables the contiguous strided KV views on CUDA and Metal, falling back to per-head `set_rows` writes | ON | not registered | no |
+| `TS_Q35_MTP_DRAFT_PERSIST` | Qwen 3.5 / 3.8 MTP draft graph | `1` lets the single-layer MTP draft graph use the persist/replay cache. Default off: the graph used to deadlock on CUDA-graph capture replay, and the knob exists to re-test that on a current ggml. Worth ~1% | OFF | not registered | no |
+| `TS_MTP_FOLD_CATCHUP` | Qwen 3.x NextN/MTP speculation | `0` runs the draft-head catch-up and the first draft step as two calls instead of folding them into one pass over `n_accepted + 1` rows (llama.cpp's draft-mtp shape). Worth ~4-5% | ON | not registered | no |
+| `TS_SPEC_ADAPTIVE` | Speculative decoding (all drafters) | `0` disables the cost governor, so drafting is never measured against a plain baseline and never parked. For A/B measurement: a governor round's baseline steps are plain decodes and they are not free | ON | not registered | no |
+| `TS_GGML_LOG_DEBUG` | GGML backends | `1` passes ggml's DEBUG log channel through instead of dropping it. Carries the CUDA backend's "CUDA graph warmup complete"/"reset" lines, which are the only way to see whether a graph is actually being CUDA-graph-captured | OFF | not registered | no |
 
 ## Out-of-Matrix GLM 5.x (`glm-dsa`) Knobs
 
@@ -203,6 +215,8 @@ in the TP table below.
 | `TS_GLM_FUSED_LID` | GLM 5.x | `0` builds the DSA lightning indexer out of primitives instead of the fused `ggml_lightning_indexer` op | `1` (fused) | `0`, `1` | no |
 | `TS_GLM_TOPK` | GLM 5.x | `0` attends densely past the indexer top-k — an A/B for the sparse selection itself, not a production setting | `1` (sparse) | `0`, `1` | no |
 | `TS_GLM_OP_OFFLOAD` | GLM 5.x on GGML | Scheduler op-offload; turned off automatically once any layer's experts are host-resident | auto | `0`, `1` | no |
+| `TS_GLM_HC_NATIVE` | GLM 5.3-Flash | `0` decomposes the Sinkhorn hyper-connection pre/post ops into batched mul_mats instead of the fused `ggml_dsv4_hc_*` kernels (A/B; auto-decomposed where the backend has no kernel) | probed | `0`, `1` | no |
+| `TS_GLM_VENC_FUSED` | GLM 5.3-Flash vision | `0` runs the GLM-OCR ViT block-by-block through managed ops instead of the one-graph native encoder (`TSGgml_GlmVisionEncoderF32`) | `1` (fused) | `0`, `1` | no |
 | `TS_GLM_VRAM_RESERVE_MB` | GLM 5.x on GGML | Per-device headroom the layer split leaves for compute buffers before it starts placing layers | `3072` | — | no |
 | `TS_GLM_GRAPH_CACHE` | GLM 5.x on GGML | How many built+allocated graphs are kept, so a repeated shape replays instead of rebuilding | `8` | — | no |
 | `TS_GLM_NODES_PER_LAYER` | GLM 5.x on GGML | Graph node budget per layer per rank | `256` | — | no |
@@ -242,6 +256,7 @@ Vulkan backends (`ggml_cuda`, `ggml_vulkan`). `TENSORSHARP_TP_DEGREE`,
 | `TS_GEMMA4_TP_FUSED_MOE` | Gemma 4 MoE under TP on GGML | `0` falls back from the fused whole-model MoE trunk (Megatron split inside each expert) to the whole-expert per-op path | on (fused trunk) | not registered | no |
 | `TS_GLM_TP_SHARD` | GLM 5.x under TP on GGML | Which halves of the split are applied: `1` heads, `2` routed experts, `3` both. The experts are split row-wise inside every expert rather than by expert id, because `ggml_mul_mat_id` needs a token's selected expert ids to stay distinct | `3` (both) | `1`, `2`, `3` | no |
 | `TS_GLM_TP_OVERSUBSCRIBE` | GLM 5.x under TP on GGML | `1` packs several ranks onto one GPU so the split can be checked for correctness on a single-GPU machine | `0` (one rank per GPU) | `0`, `1` | no |
+| `TS_Q4E_LAYER_SPLIT` | Qwen 3.8 Flash Next (`qwen4exp`) multi-GPU layer split under `--tp N` | Explicit layer counts per GPU, comma-separated (e.g. `20,28`), instead of the automatic VRAM balance; throws rather than silently ignoring a value it cannot honour. `--tp N` on this architecture is a layer split, not tensor parallelism — `qwen4exp` shards no weights | automatic (layers bin-packed to each device's free VRAM) | not registered | no |
 | `GGML_CUDA_ALLREDUCE` | local TP, `ggml_cuda` | `nccl` / `internal` / `none` — passed through to ggml's collective selection; setting it explicitly also skips the pre-flight probe | auto (NCCL when the build finds it and it passes the probe) | not registered | no |
 | `TS_GGML_TP_CUDA_GRAPHS` | local TP, `ggml_cuda` | `0` turns CUDA graph capture off for multi-GPU runs. Capture is ON by default under TP because a tensor-parallel token is dozens of small per-rank submissions that replay far more cheaply than they re-issue (4×A40: Qwen3.5-9B tp4 88 → 128.5 tok/s, Qwen3.5-35B-A3B tp2 71.3 → 104.1). It was historically disabled over a capture-poisoning hazard that no longer applies — ggml captures with `cudaStreamCaptureModeRelaxed`. The opt-out is translated into a native `GGML_CUDA_DISABLE_GRAPHS` before the first backend call, because ggml latches that value on first use | capture enabled | not registered | no |
 | `TS_GGML_TP_AR_PROBE` | local TP, `ggml_cuda` | `0` skips both pre-flight probes; `force` re-probes, ignoring the cached verdicts (`~/.cache/tensorsharp/tp-collective-probe`). Before model load the group checks that peer copies between advertised device pairs actually deliver bytes, and that one small NCCL AllReduce completes end to end — some cloud hosts advertise P2P that never arrives, and NCCL's first collective then spins every GPU forever. A failed peer check keeps NCCL but takes peer transport away from it (`NCCL_P2P_DISABLE=1`), which is what preserves a device collective past 2 GPUs | probes on, verdicts cached per driver/NCCL/GPU set | not registered | no |
diff --git a/docs/env_var_feature_matrix_zh-cn.md b/docs/env_var_feature_matrix_zh-cn.md
index 087bda95..1e1f89e9 100644
--- a/docs/env_var_feature_matrix_zh-cn.md
+++ b/docs/env_var_feature_matrix_zh-cn.md
@@ -206,6 +206,7 @@ Muse-Glimmer 的融合整模型内核与它的 DFlash 块级草稿模型各有
 | `TS_GEMMA4_TP_FUSED_MOE` | GGML 上 TP 下的 Gemma 4 MoE | `0` 表示从融合的整模 MoE 主干（专家内部 Megatron 切分）回退到逐算子的整专家路径 | 开启（融合主干） | 未注册 | 否 |
 | `TS_GLM_TP_SHARD` | GGML 上 TP 下的 GLM 5.x | 切分哪一半：`1` 注意力头，`2` 路由专家，`3` 两者都切。路由专家是在每个专家内部按行切分，而不是按专家 id 分配，因为 `ggml_mul_mat_id` 要求同一 token 选中的专家 id 互不相同 | `3`（两者） | `1`, `2`, `3` | 否 |
 | `TS_GLM_TP_OVERSUBSCRIBE` | GGML 上 TP 下的 GLM 5.x | `1` 允许多个 rank 共享一张 GPU，用于在单卡机器上验证切分的正确性 | `0`（一 rank 一卡） | `0`, `1` | 否 |
+| `TS_Q4E_LAYER_SPLIT` | `--tp N` 下按层切分的 Qwen 3.8 Flash Next（`qwen4exp`） | 直接指定每张 GPU 分到的层数（逗号分隔，例如 `20,28`），取代自动的显存均衡；给出无法满足的值时会直接抛错，而不是静默忽略。这个架构上的 `--tp N` 是按层切分而非张量并行——`qwen4exp` 不切分任何权重 | 自动（按各设备空闲显存装箱） | 未注册 | 否 |
 | `GGML_CUDA_ALLREDUCE` | 本地 TP，`ggml_cuda` | `nccl` / `internal` / `none` —— 直接透传给 ggml 的集合通信选择；显式设置同时会跳过启动前探测 | 自动（构建时能找到 NCCL 且通过探测就用 NCCL） | 未注册 | 否 |
 | `TS_GGML_TP_AR_PROBE` | 本地 TP，`ggml_cuda` | `0` 跳过 NCCL 启动前探测；`force` 忽略缓存的判定（`~/.cache/tensorsharp/tp-collective-probe`）重新探测。探测在模型加载前端到端跑一次小型 AllReduce —— 一些云主机声称支持 P2P 但数据永远送不到，NCCL 的第一次集合通信会让两块 GPU 永远空转 | 探测开启，判定按 驱动/NCCL/GPU 组合缓存 | 未注册 | 否 |
 | `TS_GGML_TP_AR_PROBE_MS` | 本地 TP，`ggml_cuda` | 探测 AllReduce 的完成期限；超时即判定集合通信不可用并改走钉页主机内存的 `internal` 管线；`0` 关闭探测 | `10000` 毫秒 | 未注册 | 否 |
diff --git a/docs/model_cards.md b/docs/model_cards.md
index 0e22058b..f7071e07 100644
--- a/docs/model_cards.md
+++ b/docs/model_cards.md
@@ -34,7 +34,7 @@ input.
 | GPT OSS | `gptoss`, `gpt-oss` | MXFP4 MoE text model with attention sinks and Harmony thinking/tools | [models/gptoss.md](models/gptoss.md) | [models/gptoss_zh-cn.md](models/gptoss_zh-cn.md) |
 | Nemotron-H | `nemotron_h`, `nemotron_h_moe` | Hybrid Mamba2 SSM + attention + (MoE) FFN text model; the Omni checkpoints add image input | [models/nemotron.md](models/nemotron.md) | [models/nemotron_zh-cn.md](models/nemotron_zh-cn.md) |
 | Mistral 3 | `mistral3` | Dense text + image chat with YaRN-corrected RoPE and the Pixtral vision encoder | [models/mistral3.md](models/mistral3.md) | [models/mistral3_zh-cn.md](models/mistral3_zh-cn.md) |
-| Muse-Glimmer | `muse-glimmer`, `muse_glimmer` | Interleaved-SWA text + image chat with thinking and ATEM tools; DFlash block speculative decoding via a separate `--draft-model` GGUF | [models/muse-glimmer.md](models/muse-glimmer.md) | [models/muse-glimmer_zh-cn.md](models/muse-glimmer_zh-cn.md) |
+| Muse-Glimmer | `muse-glimmer`, `muse_glimmer` | Interleaved-SWA text + image chat with thinking and ATEM tools; DFlash / DFlash2 block speculative decoding via a separate `--draft-model` GGUF | [models/muse-glimmer.md](models/muse-glimmer.md) | [models/muse-glimmer_zh-cn.md](models/muse-glimmer_zh-cn.md) |
 | Qwen-Image-Edit | `qwen_image`, `qwen-image` | **Image editing** — prompt + input image → edited image, through a 60-block MMDiT diffusion loop; a Lightning LoRA cuts 60 DiT forwards to 4–8 | [models/qwenimage.md](models/qwenimage.md) | [models/qwenimage_zh-cn.md](models/qwenimage_zh-cn.md) |
 | MiniMax-H3 | `minimax-h3`, `minimax_h3` | **Joint audio-video generation** — prompt (+ optional keyframes or references) → video **and native 32 kHz stereo audio generated together in one packed latent**, by a single diffusion transformer. Text-to-video, image-to-video, first/last frame and reference-to-video, all CFG-free at 4-8 steps | [models/minimax-h3.md](models/minimax-h3.md) | [models/minimax-h3_zh-cn.md](models/minimax-h3_zh-cn.md) |
 | Wan video | `wan`, `wan2.1`, `wan2.2` | **Video generation, video only** — prompt (+ optional first frame) → H.264 MP4, Wan 2.1 T2V and Wan 2.2 TI2V-5B / A14B; a step-distilled checkpoint turns the 100-DiT-pass recipe into 4 | [models/wan.md](models/wan.md) | [models/wan_zh-cn.md](models/wan_zh-cn.md) |
diff --git a/docs/models/README.md b/docs/models/README.md
index 6cacc10b..b7223587 100644
--- a/docs/models/README.md
+++ b/docs/models/README.md
@@ -62,7 +62,8 @@ a complete multimodal inference engine—use Zhongkai Fu's
 | Architecture | Card | Verified download (HF) | Source class | GGUF keys | Modalities | Reasoning | Tools | Batched / paged forward | Notable acceleration |
 |---|---|---|---|---|---|---|---|---|---|
 | DeepSeek V4 Flash | [deepseek4.md](deepseek4.md) | [unsloth/DeepSeek-V4-Flash-0731-GGUF](https://huggingface.co/unsloth/DeepSeek-V4-Flash-0731-GGUF) (multi-shard per quant directory; point `--model` at the `-00001-of-` shard). DSpark drafters: [MODEL_DOWNLOADS.md](../../MODEL_DOWNLOADS.md#dspark-drafters) | `DeepSeek4Model` (+ `DeepSeek4CudaExecutor`, `DeepSeek4CpuExecutor`) | `deepseek4` | Text | Yes | Yes (DSML markup) | Native per-sequence slots (`DeepSeek4Model.PerSeqCache.cs`) rather than `IBatchedPagedModel` — servable with continuous batching through the same engine | Three whole-model executors (direct CUDA, native ggml, pure C#), automatic layer split across every visible GPU, on-device compressed KV state (SWA ring + CSA/HCA + lightning indexer), shape-signature graph cache replaying a captured CUDA graph, fused decode index-gather over `[ring \| top-512]` K, and DSpark block speculative decoding (1.3–1.4× decode) |
-| GLM 5.x | [glm.md](glm.md) | [unsloth/GLM-5.2-GGUF](https://huggingface.co/unsloth/GLM-5.2-GGUF) (multi-shard per quant directory; point `--model` at the `-00001-of-` shard) | `GlmDsaModel` (+ the native `ggml_ops_glm_dsa.cpp` whole-model executor) | `glm-dsa` | Text | Yes | Yes (XML tool calls) | Native per-sequence slots (`TSGgml_GlmSlotAlloc`) rather than `IBatchedPagedModel` — servable with continuous batching through the same engine | Native whole-model ggml executor and a pure-C# per-op reference, automatic layer split across every visible GPU **or** Megatron tensor parallelism (`--tp N`: column/row-parallel heads, every routed expert split row-wise), `--cpu-moe` host-resident experts served straight from the GGUF mapping, MLA weight absorption with a 576-wide cache row, DSA lightning indexer with a selection reused across 57 of 78 layers, and a shape-keyed graph cache replaying a captured CUDA graph |
+| Qwen 3.8 Flash Next | [qwen38-flash-next.md](qwen38-flash-next.md) | [unsloth/Qwen3.8-Flash-Next-GGUF](https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF) (multi-shard; point `--model` at the `-00001-of-` shard) | `Qwen4ExpModel` (whole-token fused graph on GGML) | `qwen4exp` | Text + image | Yes | Yes | Per-sequence state holders (`SupportsPerSequenceFusedForward`): per-request KV + GDN + PLE state, round-robin fused decode | One captured graph per token incl. in-graph PLE and fused LM head; IMRoPE vision; KV reuse across turns (extend-only); multi-GPU **layer split** across every visible GPU (`--tp N`: contiguous whole layers per GPU — a capacity feature, not tensor parallelism, since `qwen4exp` shards no weights; byte-identical output, `TS_Q4E_LAYER_SPLIT` overrides the balance) |
+| GLM 5.x | [glm.md](glm.md) | [unsloth/GLM-5.2-GGUF](https://huggingface.co/unsloth/GLM-5.2-GGUF), [unsloth/GLM-5.3-Flash-GGUF](https://huggingface.co/unsloth/GLM-5.3-Flash-GGUF) (multi-shard per quant directory; point `--model` at the `-00001-of-` shard) | `GlmDsaModel` (+ the native `ggml_ops_glm_dsa.cpp` whole-model executor) | `glm-dsa`, `glm5next` | Text (5.2); text + image (5.3-Flash via `mmproj`) | Yes | Yes (XML tool calls) | Native per-sequence slots (`TSGgml_GlmSlotAlloc`) rather than `IBatchedPagedModel` — servable with continuous batching through the same engine | Native whole-model ggml executor and a pure-C# per-op reference, automatic layer split across every visible GPU **or** Megatron tensor parallelism (`--tp N`: column/row-parallel heads, every routed expert split row-wise), `--cpu-moe` host-resident experts served straight from the GGUF mapping, MLA weight absorption with a 576-wide cache row, DSA lightning indexer with a selection reused across 57 of 78 layers, and a shape-keyed graph cache replaying a captured CUDA graph |
 | Gemma 3 | [gemma3.md](gemma3.md) | [ggml-org/gemma-3-4b-it-GGUF](https://huggingface.co/ggml-org/gemma-3-4b-it-GGUF) | `Gemma3Model` | `gemma3` | Text, image | No | No | No (legacy per-seq) | Alternating SWA / global attention, GeGLU FFN, QK-norm, V-norm |
 | Gemma 4 | [gemma4.md](gemma4.md) | E4B Q8_0 is the verified native-GGML family/path tier; [ggml-org/gemma-4-E4B-it-GGUF](https://huggingface.co/ggml-org/gemma-4-E4B-it-GGUF) is the recommended public artifact | `Gemma4Model` | `gemma4` (`gemma4-assistant` / `gemma4_assistant` load only as the MTP draft) | Text, image, video, audio | Yes | Yes | **Default** (toggle off with `TS_GEMMA4_BATCHED=0`) | Single-graph fused decode (all layers in one GGML dispatch), fused whole-model prefill/verify with in-kernel PLE + shared-KV handling, chunked prefill, circular SWA cache, and MoE variants. Batched path matches legacy logits within FP noise (`Gemma4BatchedForwardTests`); reaches ~1.5× legacy at batch=8 and ~1.6× at 4×800-token prompts. |
 | DiffusionGemma | [diffusiongemma.md](diffusiongemma.md) | [unsloth/diffusiongemma-26B-A4B-it-GGUF](https://huggingface.co/unsloth/diffusiongemma-26B-A4B-it-GGUF) | `DiffusionGemmaModel` + `DiffusionGemmaSampler` | `diffusion-gemma`, `diffusion_gemma` | Text | No | No | Separate Web UI `DiffusionBatchScheduler`; not an autoregressive `IBatchedPagedModel` path | EntropyBound block denoising over `[prompt \| canvas]`, prompt-KV caching on GPU backends, self-conditioning, fused GGML whole-model diffusion decode and fused lm-head tail |
diff --git a/docs/models/README_zh-cn.md b/docs/models/README_zh-cn.md
index 80fe17e9..05e16eb1 100644
--- a/docs/models/README_zh-cn.md
+++ b/docs/models/README_zh-cn.md
@@ -39,7 +39,8 @@ Zhongkai Fu 的 [《From Tensors to Tokens》书籍指南](../BOOK_zh-cn.md)，
 | 架构 | 卡片 | 已验证下载（HF） | 模型类 | GGUF keys | 模态 | 思维链 | 工具调用 | 批处理 / 分页前向 | 主要加速路径 |
 |---|---|---|---|---|---|---|---|---|---|
 | DeepSeek V4 Flash | [deepseek4_zh-cn.md](deepseek4_zh-cn.md) | [unsloth/DeepSeek-V4-Flash-0731-GGUF](https://huggingface.co/unsloth/DeepSeek-V4-Flash-0731-GGUF)（每个量化档一个子目录、均为多分片；`--model` 指向 `-00001-of-` 那一片）。DSpark 草稿器见 [MODEL_DOWNLOADS_zh-cn.md](../../MODEL_DOWNLOADS_zh-cn.md) | `DeepSeek4Model`（+ `DeepSeek4CudaExecutor`、`DeepSeek4CpuExecutor`） | `deepseek4` | 文本 | 是 | 是（DSML 标记） | 使用原生 per-sequence slot（`DeepSeek4Model.PerSeqCache.cs`）而非 `IBatchedPagedModel`——仍可通过同一引擎以连续批处理对外服务 | 三套整模型执行器（Direct CUDA、原生 ggml、纯 C#）、按层自动切分到所有可见 GPU、设备端压缩 KV 状态（SWA 环 + CSA/HCA + lightning indexer）、按形状签名的计算图缓存以重放已捕获的 CUDA 图、对 `[ring \| top-512]` K 的融合 decode index-gather，以及 DSpark 块级投机解码（decode 提速 1.3–1.4×） |
-| GLM 5.x | [glm_zh-cn.md](glm_zh-cn.md) | [unsloth/GLM-5.2-GGUF](https://huggingface.co/unsloth/GLM-5.2-GGUF)（每个量化档一个子目录、均为多分片；`--model` 指向 `-00001-of-` 那一片） | `GlmDsaModel`（+ 原生整模型执行器 `ggml_ops_glm_dsa.cpp`） | `glm-dsa` | 文本 | 是 | 是（XML 工具调用） | 使用原生 per-sequence slot（`TSGgml_GlmSlotAlloc`）而非 `IBatchedPagedModel`——仍可通过同一引擎以连续批处理对外服务 | 原生 ggml 整模型执行器加一套纯 C# 逐算子参考实现、按层自动切分到所有可见 GPU **或** Megatron 张量并行（`--tp N`：head 按列/行并行，每个路由专家按行切开）、`--cpu-moe` 让主机端专家直接由 GGUF 映射提供、带权重吸收的 MLA（每 token 一行 576 宽缓存）、DSA lightning indexer（选择结果被 78 层中的 57 层复用），以及按形状索引、重放已捕获 CUDA 图的图缓存 |
+| Qwen 3.8 Flash Next | [qwen38-flash-next_zh-cn.md](qwen38-flash-next_zh-cn.md) | [unsloth/Qwen3.8-Flash-Next-GGUF](https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF)（多分片；`--model` 指向 `-00001-of-` 那一片） | `Qwen4ExpModel`（GGML 上的整 token 融合图） | `qwen4exp` | 文本 + 图像 | 是 | 是 | 逐序列状态持有者（`SupportsPerSequenceFusedForward`）：每个请求各自的 KV + GDN + PLE 状态，轮询式融合解码 | 每个 token 一张已捕获的图（含图内 PLE 与融合 LM head）、IMRoPE 视觉、跨轮次 KV 复用（只能扩展）、跨所有可见 GPU 的多 GPU **按层切分**（`--tp N`：每张 GPU 持有一段连续的完整层——这是容量特性而非张量并行，`qwen4exp` 不切分任何权重；输出逐字节一致，`TS_Q4E_LAYER_SPLIT` 可覆盖自动均衡） |
+| GLM 5.x | [glm_zh-cn.md](glm_zh-cn.md) | [unsloth/GLM-5.2-GGUF](https://huggingface.co/unsloth/GLM-5.2-GGUF)、[unsloth/GLM-5.3-Flash-GGUF](https://huggingface.co/unsloth/GLM-5.3-Flash-GGUF)（每个量化档一个子目录、均为多分片；`--model` 指向 `-00001-of-` 那一片） | `GlmDsaModel`（+ 原生整模型执行器 `ggml_ops_glm_dsa.cpp`） | `glm-dsa`、`glm5next` | 文本（5.2）；文本 + 图像（5.3-Flash，经 `mmproj`） | 是 | 是（XML 工具调用） | 使用原生 per-sequence slot（`TSGgml_GlmSlotAlloc`）而非 `IBatchedPagedModel`——仍可通过同一引擎以连续批处理对外服务 | 原生 ggml 整模型执行器加一套纯 C# 逐算子参考实现、按层自动切分到所有可见 GPU **或** Megatron 张量并行（`--tp N`：head 按列/行并行，每个路由专家按行切开）、`--cpu-moe` 让主机端专家直接由 GGUF 映射提供、带权重吸收的 MLA（每 token 一行 576 宽缓存）、DSA lightning indexer（选择结果被 78 层中的 57 层复用），以及按形状索引、重放已捕获 CUDA 图的图缓存 |
 | Gemma 3 | [gemma3_zh-cn.md](gemma3_zh-cn.md) | [ggml-org/gemma-3-4b-it-GGUF](https://huggingface.co/ggml-org/gemma-3-4b-it-GGUF) | `Gemma3Model` | `gemma3` | 文本、图像 | 否 | 否 | 否（仅旧单序列路径） | SWA / 全局注意力交替、GeGLU FFN、QK-norm、V-norm |
 | Gemma 4 | [gemma4_zh-cn.md](gemma4_zh-cn.md) | E4B Q8_0 是已验证的原生 GGML 家族 / 路径层级；[ggml-org/gemma-4-E4B-it-GGUF](https://huggingface.co/ggml-org/gemma-4-E4B-it-GGUF) 是推荐的公开文件来源 | `Gemma4Model` | `gemma4`（`gemma4-assistant` / `gemma4_assistant` 仅作为 MTP 草稿加载） | 文本、图像、视频、音频 | 是 | 是 | **默认启用**（可用 `TS_GEMMA4_BATCHED=0` 关闭） | 整模型融合 decode（一次 GGML 调度）、带内核内 PLE + 共享 KV 处理的融合整模型 prefill/verify、分块 prefill、SWA 环形缓存与 MoE 变体。批处理路径与旧路径 logits 在 FP 噪声内一致（`Gemma4BatchedForwardTests`）；batch=8 短 prompt 达 ~1.5×，4×800-token prompt 达 ~1.6×。 |
 | DiffusionGemma | [diffusiongemma_zh-cn.md](diffusiongemma_zh-cn.md) | [unsloth/diffusiongemma-26B-A4B-it-GGUF](https://huggingface.co/unsloth/diffusiongemma-26B-A4B-it-GGUF) | `DiffusionGemmaModel` + `DiffusionGemmaSampler` | `diffusion-gemma`、`diffusion_gemma` | 文本 | 否 | 否 | 独立的 Web UI `DiffusionBatchScheduler`；不是自回归 `IBatchedPagedModel` 路径 | `[prompt \| canvas]` 上的 EntropyBound 分块去噪、GPU prompt-KV 缓存、self-conditioning、融合 GGML 整模型 diffusion decode 与融合 lm-head tail |
diff --git a/docs/models/glm.md b/docs/models/glm.md
index 20185772..41c7cd0e 100644
--- a/docs/models/glm.md
+++ b/docs/models/glm.md
@@ -1,4 +1,4 @@
-# GLM-5.x (`glm-dsa`)
+# GLM-5.x (`glm-dsa`, `glm5next`)
 
 [← back to model index](README.md)
 
@@ -6,7 +6,8 @@ GLM-5.2 is a 744B-parameter MoE (256 routed experts, top-8, plus one shared
 expert) built on **DeepSeek Sparse Attention**: Multi-head Latent Attention with
 weight absorption, and a "lightning indexer" that decides which cached tokens
 each query may attend to. Advertised context: 1M tokens. The GGUF architecture
-id is `glm-dsa`.
+id is `glm-dsa`. **GLM-5.3-Flash** (`glm5next`) runs through the same executor -
+see [its section below](#glm-53-flash-glm5next).
 
 ## The block
 
@@ -457,3 +458,87 @@ dropped from the prompt, matching the template's `clear_thinking` default. Tool
 `<tool_call>NAME<arg_key>k</arg_key><arg_value>v</arg_value>...</tool_call>`,
 one XML element per argument (values that were rendered with `tojson` are parsed
 back into numbers / arrays / objects).
+
+## GLM-5.3-Flash (`glm5next`)
+
+GLM-5.3-Flash is the hybrid successor: 320B parameters, 288 routed experts
+(top-8, one shared, ×2.5 routed scale), 46 blocks = 45 trunk + 1 NextN. The
+GGUF architecture id is `glm5next`, and it loads through the **same native
+executor** (`ggml_ops_glm_dsa.cpp`) and the same `GlmDsaModel` — the MLA
+attention, the MoE and the graph plumbing are shared with GLM-5.2, with four
+architectural changes layered on top:
+
+| Piece | Shape (GLM-5.3-Flash) | Notes |
+|---|---|---|
+| KDA linear attention | 34 of 45 trunk layers; 64 heads × 128 | `attention.head_count_kv` is a per-layer array: 0 = KDA, 1 = MLA. Short conv (kernel 4, persistent per-sequence tail), l2-normed q/k, per-CHANNEL decay gate bounded below multiplicatively (`kda.gate_lower_bound` −5), fused gated-delta-net recurrence with in-graph state commit |
+| MLA + DSA layers | 11 of 45 (layers 3, 7, …, 43), **NoPE** | `rope.dimension_count` 0: no rope anywhere in the text tower, the 512-wide latent IS the cache row, softmax scale 1/√256 |
+| Pooled indexer | every MLA layer, 4-cell pools, top-k 2048 | key + compressor gate cached as `[key\|gate]` per cell; a softmax over the gates (plus a per-slot position embedding) compresses each pool; **top-k over POOLS then expand to members**, with the query's own trailing pool always attended. Dense below `top_k + kpool − 1` = 2051 cached tokens |
+| Sinkhorn hyper-connections | every layer, ×4 streams | the DeepSeek-V4 mHC recipe (fused `ggml_dsv4_hc_pre/comb/post`, 20 Sinkhorn iterations), embedding replicated ×4, head = UNWEIGHTED stream mean |
+| SwiGLU clamp | all FFNs, limit 10 | `up ∈ [−L, L]`, `gate ∈ (−∞, L]`, before the activation — dense layers, shared expert and routed experts alike |
+| Vision | `mmproj-BF16.gguf` (GLM-OCR ViT) | see below |
+
+The KDA recurrent state (conv tail + delta-net state, ~150 MB per sequence)
+cannot be rewound, so a cached prefix is only reused when the new prompt
+extends it exactly — the same contract as the Qwen 3.x GDN family — and
+`Reset` wipes the state along with the position counter.
+
+### What runs today
+
+- **Layer split across every visible GPU** (the default): ~99 GiB of
+  UD-Q2_K_XL loads across 2×96 GB in ~17 s warm.
+- **`--cpu-moe` / `--n-cpu-moe N`** host-resident experts: works (measured
+  ~35–40 t/s decode with the first 10 layers' experts on the host).
+- **Serving**: per-sequence native slots, concurrent requests decode
+  round-robin (the fused one-graph-per-step batched decode declines glm5next
+  for now and the engine falls back automatically).
+- **Vision**: `--image` / multi-image / multi-turn image sessions through the
+  managed `GlmNextVisionEncoder` (the GLM-OCR ViT: RMS norms, fused QKV,
+  per-head q/k RMS norms, 2D vision RoPE, SwiGLU-clamp MLP, 2×2 conv merger).
+  All 24 blocks run as one device-resident GGML graph
+  (`TSGgml_GlmVisionEncoderF32`); the projected embeddings override the
+  `<|image|>` placeholder rows inside the native executor
+  (`TSGgml_GlmQueueVisionRows`) — the text tower is NoPE, so image tokens
+  need no MRoPE bookkeeping.
+- **Not yet**: `--tp` tensor parallelism (cleanly refused; use the layer
+  split) and NextN/MTP speculation (llama.cpp asserts its glm5next MTP graph
+  unimplemented too; `--mtp-spec` prints a notice and serves standard decode).
+
+### Measured
+
+2× RTX PRO 6000 Blackwell (96 GB), GLM-5.3-Flash-UD-Q2_K_XL (101 GiB), layer
+split, flash attention on, both engines at `n_ubatch` 2048, back to back in one
+session (llama.cpp build 2e0e57f / PR #27754 via `llama-bench`; TensorSharp via the parity harness `--bench`):
+
+| test | llama.cpp | TensorSharp |
+|---|---:|---:|
+| pp2048 | **2070 t/s** | 2014 t/s |
+| pp16384 | 1690 t/s | **1692 t/s** |
+| pp32768 | **1483 t/s** | 1446 t/s |
+| tg64 | 36.6 t/s | **73.5 t/s** |
+
+Decode runs at **2.0× llama.cpp**; prefill is within a few percent either way
+(the same MoE tile-padding economics as GLM-5.2 apply, so `TS_GLM_UBATCH=2048`
+is the setting to keep for long prompts). Greedy replay of llama.cpp goldens
+reproduces the 2741-token long-context record — the pooled sparse-selection
+path — token for token; short records flip on Q2-quant near-ties (llama.cpp's
+own top-2 margin at a flip point is ~0.13 logits with the same candidate set).
+
+### Chat format
+
+GLM-5.3's template always reasons: the `<|system|>Reasoning Effort: Max` line
+is unconditional, the generation prompt always opens `<think>`, and past
+turns keep their reasoning (`clear_thinking` defaults to false). Tool calls
+use the same XML element form as GLM-5.2. Images render as
+`<|begin_of_image|><|image|><|end_of_image|>`, and the host expands
+`<|image|>` to the merged-patch token count.
+
+### Continuous batching (glm5next)
+
+glm5next serves concurrent requests through the same native per-sequence slots
+as GLM-5.2, **plus a fused batched decode**: one graph decodes one token for
+each of 2-16 sequences per step, with per-token KDA recurrence against each
+slot's own persistent state, per-token pooled-indexer scoring and per-token
+attention, while the projections, hyper-connections, router, experts and the
+LM head run once over the batch. Verified by a serial-vs-batched equality
+harness (`benchmarks/ParityHarness --batched`): 3 concurrent sequences, every
+step fused, token-for-token equal to serial decode.
diff --git a/docs/models/glm_zh-cn.md b/docs/models/glm_zh-cn.md
index 5f17b2a1..0edbcd88 100644
--- a/docs/models/glm_zh-cn.md
+++ b/docs/models/glm_zh-cn.md
@@ -1,11 +1,12 @@
-# GLM-5.x（`glm-dsa`）
+# GLM-5.x（`glm-dsa`、`glm5next`）
 
 [← 返回模型索引](README_zh-cn.md) | [English](glm.md)
 
 GLM-5.2 是一个 744B 参数的 MoE 模型（256 个路由专家，top-8，外加 1 个共享专家），
 构建在 **DeepSeek 稀疏注意力**之上：带权重吸收的 Multi-head Latent Attention，
 再加一个 "lightning indexer"，由它决定每个 query 可以看见哪些已缓存的 token。
-官方宣称上下文：1M token。GGUF 架构 id 是 `glm-dsa`。
+官方宣称上下文：1M token。GGUF 架构 id 是 `glm-dsa`。**GLM-5.3-Flash**（`glm5next`）
+复用同一个执行器——见[下文专节](#glm-53-flashglm5next)。
 
 ## 这一层长什么样
 
@@ -377,3 +378,71 @@ GGUF 宣称 1,048,576 token，但这并不意味着缓存放得下：78 层里
 作答。历史轮次的思考内容始终不会带进提示，与模板 `clear_thinking` 的默认行为一致。工具调用回来的形式是
 `<tool_call>NAME<arg_key>k</arg_key><arg_value>v</arg_value>...</tool_call>`，
 每个参数一个 XML 元素（用 `tojson` 渲染的值会被解析回数字 / 数组 / 对象）。
+
+
+## GLM-5.3-Flash（`glm5next`）
+
+GLM-5.3-Flash 是混合架构的后继者：320B 参数、288 个路由专家（top-8、1 个共享、
+路由权重 ×2.5），46 个 block = 45 层主干 + 1 个 NextN。GGUF 架构 id 是
+`glm5next`，通过**同一个原生执行器**（`ggml_ops_glm_dsa.cpp`）和同一个
+`GlmDsaModel` 加载——MLA 注意力、MoE 与图机制全部与 GLM-5.2 共享，在其上叠加
+四处架构差异：
+
+| 部件 | 形状（GLM-5.3-Flash） | 说明 |
+|---|---|---|
+| KDA 线性注意力 | 45 层主干中的 34 层；64 头 × 128 | `attention.head_count_kv` 是逐层数组：0 = KDA，1 = MLA。短卷积（核 4，逐序列持久尾部）、l2 归一的 q/k、乘法下界（−5）的逐通道衰减门、fused gated-delta-net 递归、图内状态提交 |
+| MLA + DSA 层 | 45 层中的 11 层（第 3、7、…、43 层），**NoPE** | `rope.dimension_count` 为 0：整个文本塔没有 rope，512 宽 latent 即缓存行，softmax 缩放 1/√256 |
+| 池化 indexer | 每个 MLA 层，4 格一池，top-k 2048 | 每格缓存 key + 压缩门（`[key|gate]`）；对门做 softmax（加逐槽位位置嵌入）压缩每池；**对"池"取 top-k 再展开成员**，query 自己的尾池始终可见。缓存低于 `top_k + kpool − 1` = 2051 个 token 时等价于稠密 |
+| Sinkhorn 超连接 | 每一层，×4 流 | DeepSeek-V4 的 mHC 配方（fused `ggml_dsv4_hc_pre/comb/post`，20 次 Sinkhorn 迭代），嵌入复制 ×4，头部是**无权重的流均值** |
+| SwiGLU 截断 | 所有 FFN，上限 10 | 激活前 `up ∈ [−L, L]`、`gate ∈ (−∞, L]`——稠密层、共享专家、路由专家一视同仁 |
+| 视觉 | `mmproj-BF16.gguf`（GLM-OCR ViT） | 见下文 |
+
+KDA 递归状态（卷积尾部 + delta-net 状态，每序列约 150 MB）无法回退，所以只有当
+新 prompt **恰好扩展**缓存前缀时才复用——与 Qwen 3.x GDN 家族相同的契约；`Reset`
+会连同位置计数一起清空该状态。
+
+### 目前能跑什么
+
+- **跨所有可见 GPU 的层切分**（默认）：UD-Q2_K_XL 约 99 GiB，2×96 GB 上热缓存
+  约 17 秒装载。
+- **`--cpu-moe` / `--n-cpu-moe N`** 专家驻留主机内存：可用（前 10 层专家在主机时
+  实测解码约 35–40 t/s）。
+- **服务化**：原生逐序列 slot；并发请求轮询式解码（fused 批量解码对 glm5next
+  暂时拒绝，引擎自动回退）。
+- **视觉**：`--image` / 多图 / 多轮图像会话，经 `GlmNextVisionEncoder`
+  （GLM-OCR ViT：RMS 归一、fused QKV、逐头 q/k RMS 归一、2D 视觉 RoPE、
+  SwiGLU-截断 MLP、2×2 卷积 merger）。24 个 block 作为一张设备驻留 GGML 图执行
+  （`TSGgml_GlmVisionEncoderF32`）；投影后的嵌入在原生执行器内覆盖
+  `<|image|>` 占位行（`TSGgml_GlmQueueVisionRows`）——文本塔是 NoPE，
+  完全不需要 MRoPE 记账。
+- **暂未支持**：`--tp` 张量并行（干净地拒绝；用层切分）与 NextN/MTP 投机
+  （llama.cpp 同样 assert 其 glm5next MTP 图未实现；`--mtp-spec` 打印提示后按
+  标准解码服务）。
+
+### 实测
+
+2× RTX PRO 6000 Blackwell（96 GB），GLM-5.3-Flash-UD-Q2_K_XL（101 GiB），层切分，
+flash attention 开启，两个引擎都用 `n_ubatch` 2048，同一会话背靠背测
+（llama.cpp build 2e0e57f / PR #27754 用 `llama-bench`；TensorSharp 用 parity
+harness 的 `--bench`）：
+
+| 测试 | llama.cpp | TensorSharp |
+|---|---:|---:|
+| pp2048 | **2070 t/s** | 2014 t/s |
+| pp16384 | 1690 t/s | **1692 t/s** |
+| pp32768 | **1483 t/s** | 1446 t/s |
+| tg64 | 36.6 t/s | **73.5 t/s** |
+
+解码达到 **llama.cpp 的 2.0 倍**；prefill 双方相差几个百分点以内（GLM-5.2 的
+MoE tile padding 经济学同样适用，长 prompt 建议保持 `TS_GLM_UBATCH=2048`）。
+对 llama.cpp golden 的贪心重放中，2741 token 的长上下文记录——正是池化稀疏
+选择路径——**逐 token 一致**；短记录会在 Q2 量化的近平手处翻转（翻转点上
+llama.cpp 自己的 top-2 边距也只有约 0.13 logit，候选集完全相同）。
+
+### 对话格式
+
+GLM-5.3 的模板始终思考：`<|system|>Reasoning Effort: Max` 无条件出现，生成提示
+总是以 `<think>` 开启，历史轮次保留思考内容（`clear_thinking` 默认 false）。
+工具调用与 GLM-5.2 相同的 XML 元素形式。图像渲染为
+`<|begin_of_image|><|image|><|end_of_image|>`，宿主把 `<|image|>` 展开为合并
+patch 的 token 数。
diff --git a/docs/models/muse-glimmer.md b/docs/models/muse-glimmer.md
index a8210504..45edc00d 100644
--- a/docs/models/muse-glimmer.md
+++ b/docs/models/muse-glimmer.md
@@ -6,7 +6,7 @@
 |---|---|
 | GGUF architecture key | `muse-glimmer` |
 | Source class | [`MuseGlimmerModel`](../../TensorSharp.Models/Models/MuseGlimmer/MuseGlimmerModel.cs) (legacy per-seq) |
-| Speculative drafter | [`MuseGlimmerModel.DFlash.cs`](../../TensorSharp.Models/Models/MuseGlimmer/MuseGlimmerModel.DFlash.cs) + [`DFlashConfig`](../../TensorSharp.Models/Models/MuseGlimmer/DFlashConfig.cs) |
+| Speculative drafter | [`MuseGlimmerModel.DFlash.cs`](../../TensorSharp.Models/Models/MuseGlimmer/MuseGlimmerModel.DFlash.cs) + [`DFlashConfig`](../../TensorSharp.Models/Speculative/DFlashConfig.cs) |
 | Vision encoder | [`MuseGlimmerVisionEncoder`](../../TensorSharp.Models/Models/MuseGlimmer/MuseGlimmerVisionEncoder.cs) |
 | Image processor | [`MuseGlimmerImageProcessor`](../../TensorSharp.Models/Models/MuseGlimmer/MuseGlimmerImageProcessor.cs) |
 | Example models | Muse-Glimmer-30B |
diff --git a/docs/models/muse-glimmer_zh-cn.md b/docs/models/muse-glimmer_zh-cn.md
index 156eeebd..981093f9 100644
--- a/docs/models/muse-glimmer_zh-cn.md
+++ b/docs/models/muse-glimmer_zh-cn.md
@@ -8,7 +8,7 @@
 |---|---|
 | GGUF 架构标识 | `muse-glimmer` |
 | 源码类 | [`MuseGlimmerModel`](../../TensorSharp.Models/Models/MuseGlimmer/MuseGlimmerModel.cs)（传统单序列） |
-| 投机草稿模型 | [`MuseGlimmerModel.DFlash.cs`](../../TensorSharp.Models/Models/MuseGlimmer/MuseGlimmerModel.DFlash.cs) + [`DFlashConfig`](../../TensorSharp.Models/Models/MuseGlimmer/DFlashConfig.cs) |
+| 投机草稿模型 | [`MuseGlimmerModel.DFlash.cs`](../../TensorSharp.Models/Models/MuseGlimmer/MuseGlimmerModel.DFlash.cs) + [`DFlashConfig`](../../TensorSharp.Models/Speculative/DFlashConfig.cs) |
 | 视觉编码器 | [`MuseGlimmerVisionEncoder`](../../TensorSharp.Models/Models/MuseGlimmer/MuseGlimmerVisionEncoder.cs) |
 | 图像预处理 | [`MuseGlimmerImageProcessor`](../../TensorSharp.Models/Models/MuseGlimmer/MuseGlimmerImageProcessor.cs) |
 | 示例模型 | Muse-Glimmer-30B |
diff --git a/docs/models/qwen35.md b/docs/models/qwen35.md
index fce2ed78..b171c147 100644
--- a/docs/models/qwen35.md
+++ b/docs/models/qwen35.md
@@ -14,7 +14,7 @@
 | Thinking mode | Yes (`<think> ... </think>`) |
 | Tool calling | Yes (`<tool_call>{...}</tool_call>`) |
 | Batched / paged forward | **Default ON** — set `TS_QWEN35_BATCHED=0` (or `--no-continuous-batching`) to force the legacy per-sequence KV-swap path for A/B comparison. Includes a per-slot GatedDeltaNet recurrent-state pool and optional native batched GDN kernel (`TS_QWEN35_BATCHED_GDN_NATIVE=1`). See §11. |
-| MTP speculative decoding | Qwen 3.6 — NextN draft block embedded in the trunk GGUF (no separate file; MTP-retaining GGUFs only, see [Downloads](#downloads)); engage with `--spec` on **either host** — `TensorSharp.Cli` and `TensorSharp.Server` share [`SpeculativeCliFlags`](../../TensorSharp.Runtime/Speculative/SpeculativeCliFlags.cs), and `--mtp-spec` is an accepted alias. GDN recurrent-state snapshot/rollback on partial accept. Engages for solo (non-concurrent) sequences whenever the GGUF retains the NextN block. See §12. |
+| MTP speculative decoding | Qwen 3.6 — NextN draft block embedded in the trunk GGUF (no separate file; MTP-retaining GGUFs only, see [Downloads](#downloads)); engage with `--spec` on **either host** — `TensorSharp.Cli` and `TensorSharp.Server` share [`SpeculativeCliFlags`](../../TensorSharp.Runtime/Speculative/SpeculativeCliFlags.cs), and `--mtp-spec` is an accepted alias. GDN recurrent-state snapshot/rollback on partial accept. Engages for solo (non-concurrent) sequences whenever the GGUF retains the NextN block. Qwen 3.8 additionally accepts a **DFlash2** block drafter as a separate `--draft-model` GGUF (§12.4). See §12. |
 | Output parser | `Qwen35OutputParser` (inherits `Qwen3OutputParser`) |
 
 ## Downloads
@@ -829,6 +829,121 @@ list and the other three algorithms — including `--spec-type ngram`, which nee
 no trained weights at all and therefore also runs on the Qwen 3.5 checkpoints that
 ship no draft block.
 
+### 12.4 DFlash2 block drafting (Qwen 3.8)
+
+Qwen 3.8 has a second, unrelated drafter available: **DFlash2**, a 5-layer
+block-diffusion model that ships as its own GGUF
+([z-lab/Qwen3.8-27B-DFlash2-GGUF](https://huggingface.co/z-lab/Qwen3.8-27B-DFlash2-GGUF),
+`general.architecture = dflash`). Where the NextN block drafts one token per
+forward, DFlash2 proposes a whole block of 7 in one, reading the trunk's own
+residuals rather than only its last hidden state. Attach it with `--draft-model`;
+naming the file IS the request, no `--spec` needed. The drafter and the NextN
+block are alternatives, not layers - when a DFlash file is attached it wins, and
+the loader says so.
+
+The algorithm, the drafter's KV ring, its fused graphs and the candidate-selector
+lattice are all shared code
+([`ModelBase.DFlash.cs`](../../TensorSharp.Models/Speculative/ModelBase.DFlash.cs),
+[`speculative_decoding.md`](../speculative_decoding.md#dflash-and-dflash2)). What
+belongs to this model is only the residual tap: `SpecForward` additionally writes
+the residual ENTERING each layer named in `dflash.target_layers` (`[6, 20, 34, 48,
+62]` for the shipped drafter), packed into one 25600-wide row per token. It is
+done inside the fused whole-model verify kernel - one `ggml_cpy` per tapped layer
+into a `[hidden, N]` output block - so speculation does not force the op-by-op
+layer loop
+([`Qwen35Model.DFlash.cs`](../../TensorSharp.Models/Models/Qwen35/Qwen35Model.DFlash.cs)).
+
+**What it is worth depends almost entirely on the workload.** Measured on one
+RTX 3080 Laptop with Qwen3.8-27B-UD-IQ3_XXS, greedy, best of two runs: on
+free-form prose 18.3 tok/s plain against 20.9 with DFlash2 (1.14x) and 19.1 with
+the trunk's own NextN block; on a highly predictable prompt ("list the first 20
+primes") 19.5 plain against 31.7 (1.63x), or 34.8 at `--spec-draft 7`. The
+default window is 3 (§12.5) and the wider window is only worth taking when
+acceptance is high - on prose it costs 40%.
+
+Quality is unaffected: on the factual prompt the DFlash2 continuation was
+byte-identical to the plain one, and to the pre-snapshot rollback path.
+
+### 12.5 Recurrent-state snapshots (why speculation stopped losing)
+
+Both drafters used to be a NET LOSS on this trunk - 15.5 tok/s for DFlash2 and
+15.7 for MTP against 18.3 plain - and the reason was not the drafters. It was
+what a REJECTION cost. The GDN recurrent state of §12.2 cannot be truncated like
+a KV cache, so a partially-rejected verify restored a pre-verify copy of it and
+re-forwarded the accepted prefix through all 64 layers: a second whole-model
+forward per rejection. The state also crossed PCIe twice per step - 151 MB up
+into the verify graph, 151 MB back down after it - whether anything was rejected
+or not.
+
+Three changes in the fused verify kernel and its caller removed all of it:
+
+1. `ggml_gated_delta_net` already accepts a snapshot count K and emits the last
+   K per-token states; the conv state after row *m* is a window of the
+   `conv_input` tensor the graph already builds. The verify now keeps one
+   snapshot per row, so the state a rollback wants is never recomputed - it is
+   slot `N-1-accepted`.
+2. `TSGgml_Qwen35CommitStateSnapshot` writes that slot into the LIVE state
+   entirely on the device. Every cached verify graph binds `*_state_in` from one
+   shared device buffer (`g_q35v_state_buf`), so the write is visible to the next
+   verify whatever shape it runs at - and that verify then skips its state
+   upload, as this step skipped its download.
+3. A verify only READS the live slices (it writes `*_state_out` and the snapshot
+   slots), so those slices ARE the pre-verify state until a commit overwrites
+   them - and a commit only happens after the rollback decision.
+   `SpecSnapshotRecurrentState` therefore copies nothing.
+
+4. The single-row steps a speculative session falls back to (when the drafter
+   declines) defer their download too. Such a step's post-window state is just
+   the `*_state_out` slices and nothing decides anything about it later, so the
+   caller commits slot -1 at once. Before this, each one broke the device-state
+   chain - 151 MB down, and the next verify uploaded it again - which on an MTP
+   run was 46 steps out of 125.
+
+Effect on 256 prose tokens with DFlash2: `rollbackMs` 3604 -> 0, `snapshotMs`
+919 -> 69. Deferring the one-row steps is worth a further 5-20% on top,
+paired-run (DFlash2 factual 22.0 -> 27.1, MTP prose 19.1 -> 21.4), and it is also
+the more exact path: committing on the device is a raw copy of the tensor the
+graph produced, where the host round trip went through an unpack-and-repack, so
+on the factual prompt it reproduces plain decoding byte for byte while the host
+path drifted in the last few tokens.
+
+The cost is VRAM - the GDN op's output grows by one ~150 MB state per slot across
+the 48 recurrent layers - which is why the default window is 3 and not 8.
+`TS_Q35_VERIFY_SNAPSHOTS=0` restores the old restore-and-re-forward path and
+`TS_Q35_VERIFY_DEFER_STATE=0` keeps the snapshots but restores the download, so
+the two halves can be measured apart; either is also the automatic fallback for
+any shape the kernel will not persist.
+
+### 12.6 Folding the MTP catch-up into the first draft step
+
+llama.cpp's `draft-mtp` runs the MTP block once over `n_accepted + 1` rows: one
+pass that both replays the verified tokens through the draft head and takes the
+first draft step. TensorSharp ran those as two calls, and because a draft call's
+cost is mostly fixed - its own graph, launch and readback, ~6 ms of which only
+about 1 ms is arithmetic - that extra call was the largest single difference
+between the two engines on this model.
+
+`Qwen35Model.DraftCatchUpAndStep` does both in one `TryFusedMtpBlock` call over
+all rows. The kernel already folds the LM head over the LAST `n_logits` rows, so
+asking for one logit row returns exactly the row the draft needs; the normed
+hidden comes back for every row and the last one chains the next step. The fold
+is an identity rather than an approximation: the block is causal over its own
+KV, so its last row sees exactly the replayed rows either way, and the output is
+byte-identical with acceptance unchanged.
+
+`DraftHeadSpeculator` stashes the commit and folds it into the next `Propose`,
+flushing it as an ordinary catch-up whenever the stashed rows do not run right up
+to the next step's position. Measured on Qwen3.8-27B-UD-IQ3_XXS: `catchUpMs`
+191 -> 0, +4.0% at 256 tokens and +5.3% on prose. `TS_MTP_FOLD_CATCHUP=0`
+restores the two-call shape.
+
+The remaining per-step difference is `MtpProjectInput`, the C# front end that
+builds the block's input (embedding, `enorm`, `hnorm`, concat, `eh_proj`). Over a
+256-token run it costs 462 ms against the fused kernel's 804 ms across 208
+calls - 2.2 ms of every 6.1 ms draft call - spent on about six separate device op
+launches, each of which synchronises because the lazy-sync path is Metal-only.
+Folding it into the fused MTP graph is the next step.
+
 ## 13. Output parser and chat template
 
 - `Qwen35OutputParser` inherits `Qwen3OutputParser`, so the wire format is
diff --git a/docs/models/qwen38-flash-next.md b/docs/models/qwen38-flash-next.md
new file mode 100644
index 00000000..2a2c7bf0
--- /dev/null
+++ b/docs/models/qwen38-flash-next.md
@@ -0,0 +1,61 @@
+# Qwen 3.8 Flash Next (`qwen4exp`)
+
+[← back to model index](README.md)
+
+Qwen3.8-Flash-Next is a hybrid MoE: GatedDeltaNet recurrent layers interleaved
+with full-attention layers (some behind Qwen Sparse Attention's indexer), a
+PLE n-gram embedding block, ×4 hyper-connection streams and a 512-expert MoE.
+The GGUF architecture id is `qwen4exp`. Weights:
+[unsloth/Qwen3.8-Flash-Next-GGUF](https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF)
+(multi-shard per quant directory; point `--model` at the `-00001-of-` shard;
+`mmproj-BF16.gguf` beside the model enables image input).
+
+## How TensorSharp runs it
+
+On the GGML backends the whole token runs as (almost) one graph — embedding,
+PLE (in-graph), all 48 layers, the final mixer and the LM head — with a
+shape-keyed cache of captured graphs (`TS_Q4E_TOKEN_GRAPH=0` falls back to
+per-layer fused kernels, which in turn fall back op-by-op). Vision rides the
+Qwen3.5-VL tower with (T,H,W) IMRoPE positions; multi-image and multi-turn
+image sessions are supported, with KV reuse across turns (the GDN recurrence
+cannot rewind, so a cached prefix is reused only when the new prompt extends
+it exactly).
+
+## Continuous batching
+
+Concurrent requests are served through **per-sequence state holders**: each
+in-flight request owns its attention KV + QSA indexer caches, its GDN conv +
+delta-net state, its PLE conv history and n-gram window, and its pinned kernel
+descriptors. The native kernel keys its device-resident recurrent state by the
+holder's host seed pointers and its cached graphs by the descriptor addresses,
+so switching requests is a reference swap — no state download/upload, no graph
+rebuild — and each sequence decodes through its own captured single-graph
+fused decode. The engine round-robins sequences per step
+(`SupportsPerSequenceFusedForward`); a fused N-way batched decode is a future
+optimization.
+
+## Multi-GPU
+
+`--tp N` on `qwen4exp` runs a **layer split**: each GPU holds a contiguous run
+of whole layers. It is not tensor parallelism — `qwen4exp` shards no weights —
+and it is the same (and only) multi-GPU mode llama.cpp offers this architecture
+(`-sm row` refuses to load it). It is a capacity feature, not a speed feature:
+it is how you fit the model when one card cannot hold it.
+
+Measured on 2× A100-80GB, Qwen3.8-Flash-Next-UD-Q2_K_XL (73.4 GiB):
+
+- greedy output is **byte-identical** between the 1-GPU and the 2-GPU run
+  (same SHA-256).
+- VRAM 24.2 GB + 26.2 GB — roughly half the model on each card instead of all
+  of it on one.
+- throughput unchanged: prefill ~1520–1550 t/s and decode ~56 t/s either way.
+  For reference, llama.cpp on the same box: 1 GPU pp1536 1094 / tg128 61.2;
+  2 GPUs `-sm layer` 1200 / 61.5 — so llama.cpp also gains ~10% prefill and
+  ~0 decode from the second card.
+
+Startup prints which mode ran and the per-GPU layer/byte split.
+`TS_Q4E_LAYER_SPLIT=20,28` overrides the automatic balance with explicit layer
+counts per GPU (llama.cpp's `--tensor-split` in spirit) and throws rather than
+silently ignoring a value it cannot honour — useful because the automatic
+balance prices weights and cannot see the vision tower, which loads later and
+lands on GPU 0.
diff --git a/docs/models/qwen38-flash-next_zh-cn.md b/docs/models/qwen38-flash-next_zh-cn.md
new file mode 100644
index 00000000..854972ba
--- /dev/null
+++ b/docs/models/qwen38-flash-next_zh-cn.md
@@ -0,0 +1,51 @@
+# Qwen 3.8 Flash Next（`qwen4exp`）
+
+[← 返回模型索引](README_zh-cn.md) | [English](qwen38-flash-next.md)
+
+Qwen3.8-Flash-Next 是一个混合型 MoE：GatedDeltaNet 递归层与全注意力层交错
+（其中一部分全注意力层挂在 Qwen Sparse Attention 的 indexer 后面），再加上一个
+PLE n-gram 嵌入块、×4 hyper-connection 流以及 512 专家的 MoE。GGUF 架构 id 是
+`qwen4exp`。权重：
+[unsloth/Qwen3.8-Flash-Next-GGUF](https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF)
+（每个量化档一个子目录、均为多分片；`--model` 指向 `-00001-of-` 那一片；把
+`mmproj-BF16.gguf` 放在模型旁边即可启用图像输入）。
+
+## TensorSharp 如何运行它
+
+在 GGML 后端上，整个 token（几乎）只跑一张图——嵌入、PLE（在图内）、全部 48 层、
+最后的 mixer 以及 LM head——并配一个按形状索引的已捕获图缓存
+（`TS_Q4E_TOKEN_GRAPH=0` 回退到逐层融合 kernel，后者再逐算子回退）。视觉沿用
+Qwen3.5-VL 塔，位置用 (T,H,W) IMRoPE；支持多图与多轮图像会话，并在轮次之间复用
+KV（GDN 递归无法回退，因此只有当新 prompt **恰好扩展**已缓存前缀时才复用）。
+
+## 连续批处理
+
+并发请求通过**逐序列状态持有者**（per-sequence state holders）来服务：每个在飞请求
+各自拥有自己的注意力 KV 与 QSA indexer 缓存、GDN 卷积与 delta-net 状态、PLE 卷积
+历史与 n-gram 窗口，以及固定下来的 kernel 描述符。原生 kernel 用持有者的 host 种子
+指针作为设备驻留递归状态的键，用描述符地址作为已缓存图的键，所以切换请求只是一次
+引用交换——不需要状态下载 / 上传，也不需要重建图——每个序列都在自己那张已捕获的
+单图融合 decode 上解码。引擎按步在各序列间轮询（`SupportsPerSequenceFusedForward`）；
+融合的 N 路批量 decode 属于后续优化。
+
+## 多 GPU
+
+`qwen4exp` 上的 `--tp N` 跑的是**按层切分**：每张 GPU 持有一段连续的完整层。它不是
+张量并行——`qwen4exp` 不切分任何权重——而且这也正是 llama.cpp 为该架构提供的
+（唯一）多 GPU 模式（`-sm row` 直接拒绝加载）。它是**容量**特性，不是速度特性：
+单卡装不下时靠它把模型装下。
+
+实测：2× A100-80GB，Qwen3.8-Flash-Next-UD-Q2_K_XL（73.4 GiB）：
+
+- 1 卡与 2 卡运行的贪心输出**逐字节一致**（SHA-256 相同）。
+- 显存 24.2 GB + 26.2 GB——大约每张卡各放半个模型，而不是一张卡放下全部。
+- 吞吐不变：两种情况下 prefill 都在 ~1520–1550 t/s，decode 都在 ~56 t/s。
+  作为参照，同一台机器上的 llama.cpp：1 张 GPU pp1536 1094 / tg128 61.2；
+  2 张 GPU `-sm layer` 1200 / 61.5——也就是说 llama.cpp 从第二张卡上同样只拿到
+  约 10% 的 prefill 提升、decode 基本为 0。
+
+启动时会打印实际走的是哪种模式，以及每张 GPU 分到的层数 / 字节数。
+`TS_Q4E_LAYER_SPLIT=20,28` 可以用显式的每卡层数覆盖自动均衡（精神上等同于
+llama.cpp 的 `--tensor-split`），并且在无法满足给定值时直接抛异常，而不是悄悄忽略
+——这很有用，因为自动均衡只按权重计价，看不见视觉塔，而视觉塔加载得更晚、会落在
+GPU 0 上。
diff --git a/docs/speculative_decoding.md b/docs/speculative_decoding.md
index 7f77a6b2..e8f4a9e2 100644
--- a/docs/speculative_decoding.md
+++ b/docs/speculative_decoding.md
@@ -83,7 +83,7 @@ Shipped implementations:
 | Name | Class | Weights | Notes |
 | --- | --- | --- | --- |
 | `draft-head` | `DraftHeadSpeculator` | required | One token per pass, chaining its own hidden output: NextN/MTP (Qwen 3.6, GLM 5.2, Gemma 4's separate assistant GGUF). EAGLE-shaped heads fit here unchanged. |
-| `block` | `BlockDraftSpeculator` | required | A whole block per pass with a confidence head: DeepSeek V4 DSpark, Muse-Glimmer DFlash. |
+| `block` | `BlockDraftSpeculator` | required | A whole block per pass with a confidence head: DeepSeek V4 DSpark, DFlash and DFlash2 (Muse-Glimmer, Qwen 3.8). |
 | `ngram` | `NGramSpeculator` | **none** | Suffix matching over the sequence's own tokens (prompt-lookup decoding). Works on every model. |
 | `auto` | — | — | Default: use whatever drafter the checkpoint carries. |
 
@@ -204,6 +204,288 @@ probability over its top-10 logits), `0.35` for a block drafter (the CUMULATIVE
 prefix probability, so the same number is far stricter), `0` for n-gram (where it
 scales the required match length instead).
 
+## DFlash and DFlash2
+
+A **DFlash** drafter is a small block-diffusion model that ships as its own GGUF
+(`general.architecture = dflash`) and is bound to one target. It reads the
+target's own residuals rather than only its tokens, and it proposes the whole
+speculative window in ONE forward pass instead of one token at a time:
+
+```
+PASS A  encoder      feat = concat(target residual entering dflash.target_layers)
+                     g    = rmsnorm(fc @ feat, enc.output_norm)
+PASS B  KV inject    K = rope_neox(headnorm(attn_k @ g)) ; V = attn_v @ g
+                     ring[pos % ringRows] <- K, V        (no Q, no attention, no FFN)
+PASS C  block draft  ids = [anchor, MASK x (block_size-1)]
+                     -> draft blocks -> the TARGET's LM head -> block_size-1 drafts
+```
+
+The drafter owns a small sliding-window KV ring of its own, sized from
+`dflash.attention.sliding_window`; the target's KV cache is untouched. Everything
+the drafter needs beyond its own blocks - the token embedding and the LM head -
+is borrowed from the target, so the file is ~1-3 GB against a 27-30B trunk.
+
+**DFlash2** is the same backbone with two additions, both keyed off the GGUF, so
+one code path serves both generations:
+
+* **A grouped dynamic depthwise convolution** around every attention and every
+  FFN sublayer (`dflash.conv_kernel_size`, `dflash.conv_group_size`). One
+  projection of the sublayer's INPUT produces both the filter applied to that
+  input and the filter applied to the sublayer's OUTPUT. Tap *t* of channel *c*
+  at block position *r* is `base[t][c] + delta[r][t][c / group_size]` - static
+  per channel, dynamic per group - multiplying `x[r-t][c]`, and masked to zero
+  for `r < t` so the filter never reaches across a block boundary. It is what
+  gives a block-diffusion draft a local left-to-right signal without a second
+  forward pass.
+
+* **A candidate selector** (`dflash.selector_rank`, `dflash.selector_top_k`).
+  Plain DFlash takes each block position's argmax over the vocabulary
+  INDEPENDENTLY - exactly the weakness of block diffusion, since position *i+1*
+  is chosen without knowing what *i* chose. The selector keeps the top-K
+  candidates per position and scores every (predecessor, candidate) pair through
+  two low-rank `[vocab, r]` codebooks:
+
+  ```
+  score[e][p][c] = unary[e][c] + < A[pred[e][p]] * (P h_e) , B[cand[e][c]] >
+  ```
+
+  `A`/`B` are `selector_predecessor`/`selector_successor`, `P` is
+  `selector_hidden`, `pred[0]` is the verified anchor token and `pred[e]` is
+  `cand[e-1]`. The block is then read off as a greedy walk through that lattice:
+  one small matmul per position, no extra draft forward.
+
+  `unary` is the target LM head's logit for that candidate **after the target's
+  own logit transform** (`dflash.logit_scale`, `dflash.final_logit_softcapping`),
+  which is why those keys exist on a DFlash2 file at all. Plain DFlash takes an
+  argmax and is invariant to both; the lattice ADDS the unary term to a
+  transition score, so an untransformed unary is simply the wrong size and
+  swamps the transition it is meant to compete with. Skipping it on the
+  Muse-Glimmer drafter (scale 0.196, softcap 20) cost more than half the
+  acceptance rate.
+
+Both extensions are no-ops when their keys are absent, so a first-generation
+DFlash file runs through the same code unchanged. `TS_DFLASH_SELECTOR=0` and
+`TS_DFLASH_CONV=0` switch one off for attribution; neither is a supported way to
+run a model, since the weights were trained with both.
+
+### Where it runs
+
+Both passes are one fused GGML graph each (`ggml_ops_dflash.cpp`,
+`TSGgml_DFlashInject` / `TSGgml_DFlashDraftBlock`) on CUDA, Vulkan and Metal,
+with a persistent graph that ggml-cuda can capture and replay; the per-op
+managed drafter is the fallback and the reference the fused path is checked
+against. `TS_DFLASH_FUSED=0` forces it.
+
+The selector's lattice comes back to the host as `k + k*k*(gamma-1)` floats
+(~7 KB) rather than the `[vocab, block]` block a naive readback would move
+(12.9 MB), and the walk itself - inherently sequential, tiny - runs on the host.
+
+### Attaching one
+
+`--draft-model <path>` (or `--spec-draft-model`, or `TS_QWEN35_DFLASH` /
+`TS_MUSE_GLIMMER_DFLASH`). The file's `general.architecture` decides what it is,
+not its name. A target that already carries a NextN/MTP block (Qwen 3.8 does)
+uses the DFlash drafter instead when one is attached: they consume different
+hidden rows and drive different speculators, and the operator named the file
+explicitly.
+
+### What the target has to provide
+
+Only the residual tap. A DFlash target implements `SpecForward` so that, per
+row, it also writes the concatenated residuals ENTERING each layer in
+`dflash.target_layers` - `SpecFeatureSize` wide instead of one hidden. Both
+shipped targets do it inside their fused whole-model kernel (a `ggml_cpy` per
+tapped layer), so speculation does not force the op-by-op loop.
+
+### What to expect
+
+Measured on one RTX 3080 Laptop (16 GB), greedy, best of two runs. Two prompts,
+because acceptance - and therefore everything - depends entirely on how
+predictable the continuation is: a free-form "explain how a GPU does a matmul"
+(prose) and a "list the first 20 primes" (factual).
+
+| target | drafter | prose tok/s | factual tok/s |
+| --- | --- | ---: | ---: |
+| Muse-Glimmer 30B IQ2_XXS | none | 18.7 | - |
+| Muse-Glimmer 30B IQ2_XXS | DFlash (1.6 GB) | 25.4 (1.36x) | - |
+| Muse-Glimmer 30B IQ2_XXS | DFlash2 Q4_K_M (1.6 GB) | 23.0 (1.23x) | - |
+| Muse-Glimmer 30B IQ2_XXS | DFlash2 Q8_0 (3.0 GB) | 14.1 (0.75x) | - |
+| Qwen 3.8 27B IQ3_XXS | none | 17.9 | 17.1 |
+| Qwen 3.8 27B IQ3_XXS | NextN/MTP | 20.4 (1.14x) | 30.1 (1.76x) |
+| Qwen 3.8 27B IQ3_XXS | DFlash2 Q4_K_M | 15.3 (0.85x) | 25.8 (1.51x) |
+| Qwen 3.8 27B IQ3_XXS | DFlash2 Q4_K_M, `--spec-draft 7` | 9.8 (0.55x) | 23.5 (1.37x) |
+
+The four Qwen rows are one uninterrupted sweep, so they are comparable to each
+other; the Muse-Glimmer rows are from a separate one and are not comparable to
+them in absolute terms.
+
+Treat the absolute numbers as indicative, not exact. On this laptop card a plain
+decode - which does identical work per token whatever the prompt - measured 18.7
+and 16.5 tok/s in two back-to-back runs of the same binary. Anything under about
+15% apart on a single run is noise here; the comparisons below that matter were
+all made as paired runs, alternating the configurations inside one batch.
+
+That caveat is not theoretical: an earlier revision of this page reported DFlash2
+on the prose prompt at 20.9 tok/s (1.14x), and it does not reproduce. Repeated
+paired runs put it at 0.85-0.96x - break-even at best on free-form prose - while
+the plain baseline measured beside them barely moved. The factual rows and the
+MTP rows did reproduce. Believe the ratios, re-measure before believing a
+single-run figure, and do not compare a number here against one taken on another
+day.
+
+### Against llama.cpp
+
+llama.cpp b10630 on the same files and card: Muse-Glimmer plain 19.7 / DFlash
+22.0. It cannot load a DFlash2 drafter at all - it rejects the file with "wrong
+number of tensors; expected 81, got 58", the 23 convolution and selector tensors
+it has no code for - so on DFlash2 there is nothing to compare against.
+
+On MTP there is, and getting it right took two corrections. A first pass ran the
+two engines on the same prompt without noticing that llama.cpp turns thinking
+mode ON by default for this checkpoint and TensorSharp does not, so they were
+answering with different continuations; since acceptance is a property of the
+continuation, that measured the text rather than the engine. The numbers below
+are a true like-for-like: same prompt, thinking disabled on both
+(`chat_template_kwargs: {"enable_thinking": false}`), greedy, 256 tokens, draft
+window 3.
+
+| | tokens/accept call | acceptance | tok/s | ms/step |
+| --- | ---: | ---: | ---: | ---: |
+| llama.cpp `draft-mtp` | 3.63 | 0.885 | 39.4 | 92.1 |
+| TensorSharp `--spec` | 3.67 | 0.932 | 33.9 | 97.4 |
+
+**Drafting is at parity or better** - TensorSharp gets slightly more tokens per
+verify call than llama.cpp does. The gap is entirely per-step cost, and
+TensorSharp's own phase counters locate it: a verify is 77 ms (llama.cpp's works
+out to about the same), and the draft calls are 20 ms against roughly 13.
+
+One caution about llama.cpp as a yardstick: its eval time reproduces to within
+0.04% run to run on this card (2886.84 ms and 2885.74 ms on two identical
+requests), where TensorSharp swings by several percent. The variance is
+TensorSharp's, not the machine's.
+
+#### What was eliminated
+
+llama.cpp runs its MTP block ONCE over `n_accepted + 1` rows, folding the
+catch-up over the accepted tokens and the first draft step into a single call.
+TensorSharp ran a catch-up and then a separate first `DraftStep`, and on a head
+whose per-call cost is mostly fixed that extra call was the largest single
+difference. It now folds too (`SupportsFusedCatchUpStep` /
+`DraftCatchUpAndStep`, `TS_MTP_FOLD_CATCHUP=0` to revert): `catchUpMs` 191 -> 0,
+worth +4.0% at 256 tokens and +5.3% on prose, with byte-identical output and
+unchanged acceptance.
+
+#### What is left, measured
+
+Three things were checked and are NOT the problem, which is worth recording
+because each looks like an obvious suspect:
+
+- **CUDA-graph capture of the verify.** `TS_GGML_LOG_DEBUG=1` surfaces ggml's
+  "CUDA graph warmup complete"/"reset" lines. Capture does churn (the persist
+  cache evicts across draft shapes), but raising
+  `TS_Q35_VERIFY_CACHE_BUDGET_MB` from 1536 to 3072 halves the resets and
+  changes throughput not at all.
+- **The MTP draft graph not persisting.** `TS_Q35_MTP_DRAFT_PERSIST=1` moves
+  `draftMs` by less than the run-to-run noise.
+- **The confidence gate.** llama.cpp does not gate at all; dropping `--spec-pmin`
+  to 0.05 is a wash on both prompts, because the steps it declines genuinely
+  would have drafted badly.
+
+What IS left is the per-call overhead of the MTP block. Instrumenting the two
+halves over a 256-token run: the C# input projection (`MtpProjectInput` -
+embedding, two RMS norms, a concat and `eh_proj`) costs 462 ms against the fused
+block kernel's 804 ms, over 208 calls. That is 2.2 ms of every 6.1 ms draft
+call, and **6.4% of the whole run**, spent on about six separate device op
+launches. Caching its scratch tensors changes nothing (the allocator already
+pools), so the cost is the launches themselves: on CUDA every op synchronises,
+because the lazy-sync path (`TS_GGML_ASYNC_COMPUTE`) is Metal-only - it relies
+on Metal's zero-copy host mapping. Folding the projection into the fused MTP
+graph, so the whole draft step is one graph, is the next concrete step.
+
+Three things in that table are worth reading carefully.
+
+**The drafter's SIZE is a first-order performance variable on a card with no
+headroom.** The same DFlash2 drafter at Q8_0 (3.0 GB) instead of Q4_K_M (1.6 GB)
+turns a 1.23x win into a 0.75x loss - not because it drafts worse (its
+acceptance is identical) but because the extra 1.4 GB pushes the trunk into
+WDDM paging and the trunk's own verify slows from 78 ms to 128 ms. Match the
+drafter quant to the headroom, not to the best available fidelity.
+
+**The window is a workload choice, and the default is the conservative one.**
+Qwen 3.8 defaults to 3 (see `SpecPreferredDraftWindow`). Widening it to 7 costs
+9% on the factual prompt and 36% on prose, because a wider window buys verify
+rows that get rejected AND makes the recurrent-state snapshots below
+proportionally larger - ~150 MB per slot here, which on a card with no headroom
+is its own second penalty. `--spec-draft N` overrides it. (An earlier revision
+claimed a window of 7 WON by 10% on the factual prompt; that was measured before
+the state stopped round-tripping, when a wider window amortised a fixed per-step
+transfer that no longer exists.)
+
+**What the drafter proposes is only half the story on a recurrent trunk** - the
+other half is what a REJECTION costs, which is the next section.
+
+## Rejection on a recurrent trunk
+
+Qwen 3.5/3.6/3.8 are hybrids: 48 of Qwen 3.8's 64 layers are GatedDeltaNet, and
+GDN carries a recurrent state that a KV cache's "drop the rejected tail" does not
+apply to. Rolling a partially-rejected verify back used to mean restoring a
+pre-verify copy of that state and re-forwarding the accepted prefix through the
+entire trunk - a second whole-model forward - because the state after row *m*
+simply did not exist anywhere. On top of that the state (151 MB for this model)
+crossed PCIe twice per step: uploaded into the verify graph, downloaded again
+after it. Speculation therefore cost MORE than the plain decode it was meant to
+beat: 15.5 tok/s against 18.3 for DFlash2, 15.7 against 18.3 for MTP.
+
+Three changes, all in the fused verify kernel and its Qwen 3.5 caller, removed
+that (`ggml_ops_qwen35_verify.cpp`, `Qwen35Model.GatedDeltaNet.cs`):
+
+1. **The verify keeps one recurrent-state snapshot per row.**
+   `ggml_gated_delta_net` already takes a snapshot count and emits the last K
+   per-token states; the conv state after row *m* is a window of a tensor the
+   graph already builds. The state a rollback wants is therefore never
+   recomputed - it is slot `N-1-accepted`.
+
+2. **A snapshot is committed into the live state on the DEVICE.** Every cached
+   verify graph binds its `*_state_in` from one shared device buffer, so writing
+   a slot into it is visible to the next verify whatever shape it runs at. The
+   state stops round-tripping: the next verify skips its upload, this one skips
+   its download.
+
+3. **The pre-verify snapshot becomes free.** A verify only READS the live
+   slices - it writes its results to `*_state_out` and the snapshot slots - so
+   the slices ARE the pre-verify state until a commit overwrites them, and a
+   commit only happens after the rollback decision. `SpecSnapshotRecurrentState`
+   copies nothing.
+
+4. **The single-row steps stop round-tripping too.** A speculative session is
+   not all verifies: when the drafter declines to propose, the step falls
+   through to an ordinary one-row forward, and those ran the old download.
+   Each one broke the device-state chain - 151 MB down, and the *next* verify
+   had to upload it again - which on an MTP run (46 such steps out of 125) was
+   most of what was left. A one-row step's post-window state is simply the
+   `*_state_out` slices and nothing decides anything about it later, so the
+   kernel now defers it as well and the caller commits slot -1 immediately:
+   one device-to-device copy instead of 302 MB across PCIe.
+
+The measured effect on Qwen3.8-27B, DFlash2, 256 prose tokens: `rollbackMs`
+3604 -> 0, `snapshotMs` 919 -> 69, and 15.5 -> 20.9 tok/s. On the factual prompt
+24.3 -> 31.7. Deferring the one-row steps (4) is worth a further 5-20% on top,
+paired-run: DFlash2 factual 22.0 -> 27.1, MTP prose 19.1 -> 21.4.
+
+Output is unchanged - in fact it is *more* exactly unchanged than before.
+Committing on the device is a raw copy of the tensor the graph produced, where
+the host round trip went through the state's unpack-and-repack; on the factual
+prompt the device path reproduces plain decoding byte for byte while the host
+path drifted in the last few tokens.
+
+`TS_Q35_VERIFY_SNAPSHOTS=0` restores the old path entirely, and
+`TS_Q35_VERIFY_DEFER_STATE=0` keeps the snapshots but restores the download, so
+the two halves can be measured apart. Either is also what a shape the kernel
+will not persist falls back to, automatically. The cost of the snapshots is
+VRAM: the GDN op's output grows by one state per slot, ~150 MB per slot for this
+model across all 48 recurrent layers, which is the other reason the default
+window is 3 rather than 8.
+
 ## Where n-gram pays
 
 `--spec-type ngram` needs no trained weights, so it works on every checkpoint,
diff --git a/website/backends.html b/website/backends.html
index 26fa6165..0768a06e 100644
--- a/website/backends.html
+++ b/website/backends.html
@@ -55,7 +55,7 @@ <h3 id="ggml">GGML Metal, GGML CUDA &amp; GGML Vulkan</h3>
     <li><strong>GGML Vulkan</strong> (<code>ggml_vulkan</code>, Windows/Linux + AMD/Intel/NVIDIA) — vendor-neutral GPU path for any GPU with a Vulkan 1.3 driver, using cooperative-matrix shaders (KHR coopmat / NV coopmat2) where the driver supports them. Weights are device-resident like GGML CUDA and the same fused whole-model decode/prefill graphs are used. On multi-GPU hosts (e.g. an integrated Intel GPU next to a discrete NVIDIA one), pick the device with <code>--gpu-device N</code> (or the <code>TS_GGML_VULKAN_DEVICE</code> env var) and list the visible devices with <code>--list-gpus</code>.</li>
   </ul>
   <p>All three run native quantized matmul (Q4_K_M, Q8_0, …) without dequantizing to FP32, plus a native paged-attention kernel that drives <code>ggml_flash_attn_ext</code>.</p>
-  <p><strong>Multi-GPU.</strong> GGML CUDA and GGML Vulkan also support <a href="distributed.html">tensor parallelism</a>: <code>--tp N</code> gives each rank its own ggml backend, weight shards, and KV cache on its own GPU, driven concurrently by a rank worker pool, with cross-GPU AllReduce through ggml's collective (NCCL when the build finds it) or a host reduction for small payloads. Fused per-rank block graphs make <code>--tp 2</code> decode faster than a single GPU on Gemma 4, and let models larger than one card's VRAM run entirely on GPUs.</p>
+  <p><strong>Multi-GPU.</strong> GGML CUDA and GGML Vulkan also support <a href="distributed.html">tensor parallelism</a>: <code>--tp N</code> gives each rank its own ggml backend, weight shards, and KV cache on its own GPU, driven concurrently by a rank worker pool, with cross-GPU AllReduce through ggml's collective (NCCL when the build finds it) or a host reduction for small payloads. Fused per-rank block graphs make <code>--tp 2</code> decode faster than a single GPU on Gemma 4, and let models larger than one card's VRAM run entirely on GPUs. They also carry the other multi-GPU mode, the <strong>layer split</strong> — whole layers per GPU, no collectives, nothing sharded: automatic on DeepSeek V4 Flash and GLM 5.x, and what <code>--tp N</code> runs on Qwen 3.8 Flash Next, where it buys capacity rather than speed. → <a href="distributed.html#vs-layer-split">TP vs. the layer split</a></p>
   <div class="callout tip"><div class="ic">⚡</div><p><strong>Verified E4B fast lane:</strong> repository benchmarks exercise the Gemma 4 E4B Q8_0 family on these native GGML GPU backends. E4B's PLE and shared-KV layout stays on the fused whole-model prefill/verify and single-graph decode paths, with automatic fused N=1 server routing. Start with the <a href="getting-started.html#e4b-fast-lane">Gemma 4 E4B guide</a>.</p></div>
 
   <h3 id="mlx">MLX Metal</h3>
@@ -63,7 +63,7 @@ <h3 id="mlx">MLX Metal</h3>
 
   <h3 id="cuda">Direct CUDA</h3>
   <p><code>--backend cuda</code> is a pure-C# path using the CUDA Driver API, cuBLAS GEMM, and PTX kernels for common float32 ops (fill, unary/binary/ternary, activations, RMSNorm, softmax, RoPE/RoPEEx, SDPA, GQA prefill/decode, causal mask, gather/concat) plus native quantized matmul/get-rows for supported quant types. Unsupported ops route through CPU fallbacks while preserving tensor semantics. It is also the pure-C# backend where <a href="advanced.html#mtp">MTP speculative decoding</a> is profitable.</p>
-  <p>It supports <strong>tensor parallelism</strong>: <code>--tp N</code> shards one model across N CUDA GPUs, and <code>--tp-node-id</code> / <code>--tp-peers</code> extend the group across machines. The GGML CUDA and GGML Vulkan backends shard too (see above); MLX and the CPU backends are single-device — on a multi-GPU host they pick one device (<code>--gpu-device</code> for Vulkan without <code>--tp</code>) rather than splitting the model. → <a href="distributed.html">Multi-GPU &amp; Multi-Node</a></p>
+  <p>It supports <strong>tensor parallelism</strong>: <code>--tp N</code> shards one model across N CUDA GPUs, and <code>--tp-node-id</code> / <code>--tp-peers</code> extend the group across machines. The GGML CUDA and GGML Vulkan backends shard too (see above), though on the whole-model-executor architectures <code>--tp</code> means something else — a layer split on Qwen 3.8 Flash Next, a device cap on DeepSeek V4 Flash, and a clean refusal on GLM-5.3-Flash. MLX and the CPU backends are single-device — on a multi-GPU host they pick one device (<code>--gpu-device</code> for Vulkan without <code>--tp</code>) rather than splitting the model. → <a href="distributed.html">Multi-GPU &amp; Multi-Node</a></p>
 
   <h3 id="cpu">CPU backends</h3>
   <ul>
@@ -85,6 +85,7 @@ <h3 id="deepseek4">DeepSeek V4 &amp; GLM 5.x: dedicated whole-model executors</h
     <li><strong><code>--backend ggml_cuda</code> / <code>ggml_vulkan</code> / <code>ggml_cpu</code> / <code>ggml_metal</code></strong> — the native ggml executor loads the 6-shard split GGUF itself, layer-splits 226 GiB of weights across every visible GPU, owns the MLA and lightning-indexer caches on-device, and submits one graph per micro-batch through a shape-keyed graph cache so steady-state decode replays an allocated (and on CUDA, captured) graph.</li>
     <li><strong><code>--backend cpu</code> (100% managed, no native dependencies) and <code>--backend cuda</code></strong> — the per-op path in <code>TensorSharp.Models/Models/GlmDsa</code>, which is also the reference the native executor is checked against; <code>TS_GLM_NATIVE=0</code> selects it on a GGML backend for an A/B. MLX does not run this family.</li>
   </ul>
+  <p><strong>GLM-5.3-Flash</strong> (<code>glm5next</code>) loads through that same native executor and the same <code>GlmDsaModel</code>, and layer-splits the same way; it refuses <code>--tp</code> rather than sharding, and NextN/MTP speculation is not implemented for it yet.</p>
   <p>Because 1M tokens of MLA cache is ~93 GiB, the advertised context is treated as a ceiling: after the weights land the loader measures the VRAM actually free and sizes the context to fit, logging its pick (342,272 tokens on a 3-GPU layer split). <code>MAX_CONTEXT</code> turns a specific length into a hard requirement instead. See <a href="models-text.html#glm">GLM 5.x</a>.</p>
 
   <div class="callout note"><div class="ic">🔎</div><p>The server reports which backends are actually available on the host in <code>GET /api/models</code> (<code>supportedBackends</code>). If a CUDA or MLX backend is missing, the host did not detect a usable driver/runtime at startup. If <code>ggml_vulkan</code> is missing, the native bridge was not built with Vulkan enabled or no Vulkan 1.3 device/driver was found.</p></div>
diff --git a/website/backends_zh-cn.html b/website/backends_zh-cn.html
index 3a2b04fe..273f6fd9 100644
--- a/website/backends_zh-cn.html
+++ b/website/backends_zh-cn.html
@@ -55,7 +55,7 @@ <h3 id="ggml">GGML Metal、GGML CUDA 与 GGML Vulkan</h3>
     <li><strong>GGML Vulkan</strong>（<code>ggml_vulkan</code>，Windows/Linux + AMD/Intel/NVIDIA）—— 与厂商无关的 GPU 路径，支持任何带 Vulkan 1.3 驱动的 GPU，驱动支持时使用 cooperative-matrix（KHR coopmat / NV coopmat2）着色器。权重与 GGML CUDA 一样常驻显存，并复用同样的融合整模型 decode/prefill 图。在多 GPU 主机上（例如同时装有 Intel 集成显卡和 NVIDIA 独立显卡），用 <code>--gpu-device N</code>（或环境变量 <code>TS_GGML_VULKAN_DEVICE</code>）选择设备，用 <code>--list-gpus</code> 列出可见设备。</li>
   </ul>
   <p>三者都在不反量化为 FP32 的情况下运行原生量化 matmul（Q4_K_M、Q8_0 ……），并配有驱动 <code>ggml_flash_attn_ext</code> 的原生分页注意力内核。</p>
-  <p><strong>多 GPU。</strong>GGML CUDA 与 GGML Vulkan 同样支持<a href="distributed_zh-cn.html">张量并行</a>：<code>--tp N</code> 让每个 rank 在自己的 GPU 上拥有独立的 ggml 后端、权重分片与 KV 缓存，由 rank 工作线程池并发驱动，跨 GPU AllReduce 走 ggml 的集合通信（构建时能找到 NCCL 就用 NCCL），小载荷则在主机内存中归约。融合的按 rank block 计算图让 Gemma 4 上 <code>--tp 2</code> 的 decode 快于单卡，也让超出单卡显存的模型能够完整跑在 GPU 上。</p>
+  <p><strong>多 GPU。</strong>GGML CUDA 与 GGML Vulkan 同样支持<a href="distributed_zh-cn.html">张量并行</a>：<code>--tp N</code> 让每个 rank 在自己的 GPU 上拥有独立的 ggml 后端、权重分片与 KV 缓存，由 rank 工作线程池并发驱动，跨 GPU AllReduce 走 ggml 的集合通信（构建时能找到 NCCL 就用 NCCL），小载荷则在主机内存中归约。融合的按 rank block 计算图让 Gemma 4 上 <code>--tp 2</code> 的 decode 快于单卡，也让超出单卡显存的模型能够完整跑在 GPU 上。它们同样承载另一种多 GPU 模式 —— <strong>按层切分</strong>：每张 GPU 拿整层、没有集合通信、不切分任何权重，DeepSeek V4 Flash 与 GLM 5.x 自动如此，Qwen 3.8 Flash Next 上的 <code>--tp N</code> 跑的也是它，换来的是容量而非速度。→ <a href="distributed_zh-cn.html#vs-layer-split">TP 与按层切分</a></p>
   <div class="callout tip"><div class="ic">⚡</div><p><strong>已验证的 E4B 快路径：</strong>仓库基准已在这些 GGML 原生 GPU 后端上验证 Gemma 4 E4B Q8_0 家族。E4B 的 PLE 与共享 KV 布局会进入融合整模型 prefill/verify 和单图 decode 路径，服务端 N=1 时也会自动选择融合路径。请从 <a href="getting-started_zh-cn.html#e4b-fast-lane">Gemma 4 E4B 指南</a>开始。</p></div>
 
   <h3 id="mlx">MLX Metal</h3>
@@ -63,7 +63,7 @@ <h3 id="mlx">MLX Metal</h3>
 
   <h3 id="cuda">直接 CUDA</h3>
   <p><code>--backend cuda</code> 是一条纯 C# 路径，使用 CUDA Driver API、cuBLAS GEMM，以及常见 float32 运算的 PTX 内核（fill、一元/二元/三元、激活、RMSNorm、softmax、RoPE/RoPEEx、SDPA、GQA prefill/decode、因果掩码、gather/concat），外加受支持量化类型的原生量化 matmul/get-rows。未支持的运算在保留张量语义的前提下走 CPU 回退。它也是<a href="advanced_zh-cn.html#mtp">MTP 推测解码</a>能获益的纯 C# 后端。</p>
-  <p>它支持<strong>张量并行</strong>：<code>--tp N</code> 把一个模型切分到 N 张 CUDA GPU 上，<code>--tp-node-id</code> / <code>--tp-peers</code> 则把并行组扩展到多台机器。GGML CUDA 与 GGML Vulkan 后端同样支持切分（见上文）；MLX 与 CPU 后端为单设备 —— 在多 GPU 主机上它们只会选择其中一张卡（不带 <code>--tp</code> 时 Vulkan 用 <code>--gpu-device</code>），而不会切分模型。→ <a href="distributed_zh-cn.html">多 GPU 与多节点</a></p>
+  <p>它支持<strong>张量并行</strong>：<code>--tp N</code> 把一个模型切分到 N 张 CUDA GPU 上，<code>--tp-node-id</code> / <code>--tp-peers</code> 则把并行组扩展到多台机器。GGML CUDA 与 GGML Vulkan 后端同样支持切分（见上文）；不过在走整模型执行器的架构上 <code>--tp</code> 是另一层含义 —— 在 Qwen 3.8 Flash Next 上是按层切分，在 DeepSeek V4 Flash 上只是设备数上限，在 GLM-5.3-Flash 上则是干脆拒绝。MLX 与 CPU 后端为单设备 —— 在多 GPU 主机上它们只会选择其中一张卡（不带 <code>--tp</code> 时 Vulkan 用 <code>--gpu-device</code>），而不会切分模型。→ <a href="distributed_zh-cn.html">多 GPU 与多节点</a></p>
 
   <h3 id="cpu">CPU 后端</h3>
   <ul>
@@ -85,6 +85,7 @@ <h3 id="deepseek4">DeepSeek V4 与 GLM 5.x：专属整模型执行器</h3>
     <li><strong><code>--backend ggml_cuda</code> / <code>ggml_vulkan</code> / <code>ggml_cpu</code> / <code>ggml_metal</code></strong> —— 原生 ggml 执行器：自行加载 6 分片的 split GGUF，把 226 GiB 权重按层切分到所有可见 GPU，在设备上持有 MLA 与 lightning indexer 的缓存，并通过按形状键控的图缓存把每个微批作为一张计算图提交，使稳态 decode 重放同一张已分配（在 CUDA 上还已捕获）的图。</li>
     <li><strong><code>--backend cpu</code>（100% 托管、零原生依赖）与 <code>--backend cuda</code></strong> —— <code>TensorSharp.Models/Models/GlmDsa</code> 里的逐算子路径，同时也是原生执行器对拍所依据的参考实现；在 GGML 后端上设 <code>TS_GLM_NATIVE=0</code> 即可切过去做 A/B。MLX 不支持这一系列。</li>
   </ul>
+  <p><strong>GLM-5.3-Flash</strong>（<code>glm5next</code>）走的是同一个原生执行器与同一个 <code>GlmDsaModel</code>，按层切分方式也相同；它会拒绝 <code>--tp</code> 而不是去切分权重，NextN/MTP 推测解码目前也尚未为它实现。</p>
   <p>由于 1M token 的 MLA 缓存约 93 GiB，自报的上下文被当作上限处理：权重落盘后，加载器会测量实际空闲的显存并按能装下的大小定上下文，同时打印它的选择（3 卡按层切分为 342,272 token）。设 <code>MAX_CONTEXT</code> 则把某个长度变成硬性要求。参见 <a href="models-text_zh-cn.html#glm">GLM 5.x</a>。</p>
 
   <div class="callout note"><div class="ic">🔎</div><p>服务器会在 <code>GET /api/models</code>（<code>supportedBackends</code>）中报告主机上实际可用的后端。如果缺少 CUDA 或 MLX 后端，说明主机在启动时未检测到可用的驱动 / 运行时。如果缺少 <code>ggml_vulkan</code>，说明原生桥接库未启用 Vulkan 构建，或未找到支持 Vulkan 1.3 的设备/驱动。</p></div>
diff --git a/website/cli.html b/website/cli.html
index 455c9ea8..71473c38 100644
--- a/website/cli.html
+++ b/website/cli.html
@@ -259,7 +259,7 @@ <h4>Runtime</h4>
       <tr><td><code>--tools &lt;path&gt;</code></td><td>JSON file with tool / function definitions.</td></tr>
       <tr><td><code>--spec</code> · <code>--spec-type &lt;name&gt;</code> · <code>--draft-model &lt;path&gt;</code></td><td>Speculative decoding: a drafter proposes the next few tokens and the trunk verifies them in one batched forward. Off by default, and it has to be on the command line before the model loads — see <a href="#spec">Speculative decoding</a> below for the full set.</td></tr>
       <tr><td><code>--dump-prompt</code></td><td>Render the prompt + tokenization and exit (no generation).</td></tr>
-      <tr><td><code>--tp &lt;N&gt;</code></td><td>Tensor parallelism degree — split the model across N GPUs in one process (default: 1). Requires <code>--backend cuda</code>, <code>ggml_cuda</code>, or <code>ggml_vulkan</code>; <code>TENSORSHARP_TP_DEVICES</code> picks which GPUs. → <a href="distributed.html">Multi-GPU &amp; Multi-Node</a></td></tr>
+      <tr><td><code>--tp &lt;N&gt;</code></td><td>Multi-GPU degree — run the model on N GPUs in one process (default: 1). Requires <code>--backend cuda</code>, <code>ggml_cuda</code>, or <code>ggml_vulkan</code>; <code>TENSORSHARP_TP_DEVICES</code> picks which GPUs. On most architectures this is <strong>tensor parallelism</strong>: every layer is sharded across the ranks. On the whole-model executors it means something else — on <strong>Qwen 3.8 Flash Next</strong> it is a <strong>layer split</strong> (a contiguous run of whole layers per GPU, nothing sharded; capacity, not speed — greedy output and throughput are unchanged, the weights just stop having to fit one card, and <code>TS_Q4E_LAYER_SPLIT=20,28</code> overrides the balance), on <strong>DeepSeek V4 Flash</strong> it only caps how many GPUs its automatic layer split uses, and <strong>GLM-5.3-Flash</strong> refuses it and keeps that split. An architecture that supports neither mode prints a notice on stderr and runs on one GPU. → <a href="distributed.html#vs-layer-split">TP vs. the layer split</a></td></tr>
       <tr><td><code>--tp-node-id &lt;N&gt;</code></td><td>This node's 0-based ID for multi-node distributed tensor parallelism. Use together with <code>--tp-peers</code>.</td></tr>
       <tr><td><code>--tp-peers &lt;list&gt;</code></td><td>Comma-separated <code>host:port</code> list of every node in the cluster (e.g. <code>192.168.1.10:9500,192.168.1.11:9500</code>). Identical on all nodes; the port is not a default and must be reachable between them.</td></tr>
       <tr><td><code>--config &lt;path&gt;</code></td><td>Read options from a JSON <a href="#config">config file</a> (command-line options override it). Supports <code>${variables}</code> and auto-downloading models. Repeatable.</td></tr>
diff --git a/website/cli_zh-cn.html b/website/cli_zh-cn.html
index 487b964c..48bfc926 100644
--- a/website/cli_zh-cn.html
+++ b/website/cli_zh-cn.html
@@ -258,7 +258,7 @@ <h4>运行时</h4>
       <tr><td><code>--tools &lt;path&gt;</code></td><td>含工具 / 函数定义的 JSON 文件。</td></tr>
       <tr><td><code>--spec</code> · <code>--spec-type &lt;name&gt;</code> · <code>--draft-model &lt;path&gt;</code></td><td>投机解码：由草稿器提议接下来的若干 token，主干用一次批量前向完成验证。默认关闭，且必须在模型加载之前出现在命令行上 —— 完整参数见下方的<a href="#spec">投机解码</a>。</td></tr>
       <tr><td><code>--dump-prompt</code></td><td>渲染提示词 + 分词后退出（不生成）。</td></tr>
-      <tr><td><code>--tp &lt;N&gt;</code></td><td>张量并行度 —— 在单个进程内把模型切分到 N 张 GPU 上（默认：1）。需要 <code>--backend cuda</code>、<code>ggml_cuda</code> 或 <code>ggml_vulkan</code>；用 <code>TENSORSHARP_TP_DEVICES</code> 指定使用哪几张卡。→ <a href="distributed_zh-cn.html">多 GPU 与多节点</a></td></tr>
+      <tr><td><code>--tp &lt;N&gt;</code></td><td>多 GPU 度 —— 在单个进程内把模型跑在 N 张 GPU 上（默认：1）。需要 <code>--backend cuda</code>、<code>ggml_cuda</code> 或 <code>ggml_vulkan</code>；用 <code>TENSORSHARP_TP_DEVICES</code> 指定使用哪几张卡。在多数架构上它是<strong>张量并行</strong>：每一层都被切分到各 rank 上。在走整模型执行器的架构上它另有含义 —— 在 <strong>Qwen 3.8 Flash Next</strong> 上是<strong>按层切分</strong>（每张 GPU 拿一段连续的完整层，不切分任何权重；买的是容量而不是速度 —— 贪心输出与吞吐都不变，只是权重不必再挤进一张卡，<code>TS_Q4E_LAYER_SPLIT=20,28</code> 可覆盖均衡结果），在 <strong>DeepSeek V4 Flash</strong> 上只是限制其自动按层切分使用几张卡，而 <strong>GLM-5.3-Flash</strong> 会拒绝它并继续用那套切分。两种模式都不支持的架构会在 stderr 上打印提示并只用一张 GPU。→ <a href="distributed_zh-cn.html#vs-layer-split">TP 与按层切分</a></td></tr>
       <tr><td><code>--tp-node-id &lt;N&gt;</code></td><td>多节点分布式张量并行中本节点的 0 起始编号。需与 <code>--tp-peers</code> 一起使用。</td></tr>
       <tr><td><code>--tp-peers &lt;list&gt;</code></td><td>集群中所有节点的 <code>host:port</code> 列表，逗号分隔（例如 <code>192.168.1.10:9500,192.168.1.11:9500</code>）。所有节点必须完全一致；端口没有默认值，且需在节点之间可达。</td></tr>
       <tr><td><code>--config &lt;path&gt;</code></td><td>从 JSON <a href="#config">配置文件</a>读取参数（命令行参数会覆盖它）。支持 <code>${变量}</code> 与模型自动下载。可重复。</td></tr>
diff --git a/website/distributed.html b/website/distributed.html
index 11ee2923..05f69e9c 100644
--- a/website/distributed.html
+++ b/website/distributed.html
@@ -16,9 +16,9 @@
 <main class="content" id="content">
   <div class="breadcrumb"><a href="index.html">Home</a> › Multi-GPU &amp; Multi-Node</div>
   <h1>Multi-GPU &amp; Multi-Node</h1>
-  <p class="lede">One model, many GPUs — and, when one machine is not enough, many machines. TensorSharp implements <strong>tensor parallelism</strong> in the Megatron-LM column/row-parallel pattern, and extends it across hosts with a peer-to-peer TCP mesh.</p>
+  <p class="lede">One model, many GPUs — and, when one machine is not enough, many machines. TensorSharp implements <strong>tensor parallelism</strong> in the Megatron-LM column/row-parallel pattern, and extends it across hosts with a peer-to-peer TCP mesh. On the architectures that shard no weights, the same <code>--tp N</code> flag runs a <strong>layer split</strong> instead — a capacity feature, not a speed one.</p>
 
-  <div class="callout note"><div class="ic">🧭</div><p><strong>In one line:</strong> add <code>--tp N</code> to run on N local GPUs; add <code>--tp-node-id</code> and <code>--tp-peers</code> on top of that to span machines. Both <code>TensorSharp.Cli</code> and <code>TensorSharp.Server</code> take the flags, on the direct <a href="backends.html#cuda"><code>cuda</code></a> backend and on the <a href="backends.html#ggml">GGML CUDA / Vulkan</a> backends.</p></div>
+  <div class="callout note"><div class="ic">🧭</div><p><strong>In one line:</strong> add <code>--tp N</code> to run on N local GPUs — sharding every layer where the architecture supports it, splitting the model by whole layers where it does not; add <code>--tp-node-id</code> and <code>--tp-peers</code> on top of that to span machines. Both <code>TensorSharp.Cli</code> and <code>TensorSharp.Server</code> take the flags, on the direct <a href="backends.html#cuda"><code>cuda</code></a> backend and on the <a href="backends.html#ggml">GGML CUDA / Vulkan</a> backends.</p></div>
 
   <h2 id="why">When do you need it?</h2>
   <ul>
@@ -31,12 +31,12 @@ <h2 id="why">When do you need it?</h2>
   <h2 id="vs-layer-split">TP vs. the layer split — what a multi-GPU box actually does</h2>
   <p>There are two different ways a model can occupy more than one GPU, and only one of them is <code>--tp</code>.</p>
   <ul>
-    <li><strong>Tensor parallelism (<code>--tp N</code>)</strong> puts <em>every</em> layer on <em>every</em> rank and splits the weights <em>within</em> each layer, so a decode step reads <code>1/N</code> of the bytes per device and the ranks all-reduce at each layer boundary. Every architecture in the table below supports it, and it is <strong>opt-in</strong> — nothing splits a tensor unless you ask.</li>
+    <li><strong>Tensor parallelism (<code>--tp N</code>)</strong> puts <em>every</em> layer on <em>every</em> rank and splits the weights <em>within</em> each layer, so a decode step reads <code>1/N</code> of the bytes per device and the ranks all-reduce at each layer boundary. It is <strong>opt-in</strong> — nothing splits a tensor unless you ask — and most, though no longer all, of the architectures in the table below implement it.</li>
     <li><strong>The layer split</strong> puts <em>whole layers</em> on different devices and runs them in sequence: device 0 evaluates layers 0..k, hands the hidden state to device 1, and so on. The cut points are not a naive <code>n_layer/N</code> — the loader measures each device's free VRAM and bin-packs the layers to balance the largest fraction-of-budget used, so an uneven set of cards still fills evenly. There are no collectives and no per-layer split, so it costs nothing on a slow interconnect, but only one GPU is busy at a time.</li>
   </ul>
-  <p>The layer split applies to exactly <strong>two architectures — DeepSeek V4 Flash (<code>deepseek4</code>) and GLM 5.x (<code>glm-dsa</code>)</strong>, the two that run through their own whole-model executors, and it is what they do <strong>by default, with no flag at all</strong>: they spread across every visible GPU because neither fits on one card. <code>TS_DSV4_NGPU</code> and <code>TS_GLM_NGPU</code> cap how many devices they use.</p>
-  <p>On <strong>every other architecture</strong>, running without <code>--tp</code> uses a <strong>single GPU</strong>. There is no automatic layer split on the generic per-op or fused-graph paths — a model that does not fit one card fails at load rather than being spread silently (the refusal that names the exact <code>--n-cpu-moe N</code> comes from the DeepSeek V4 and GLM 5.x whole-model loaders).</p>
-  <p>So on a 3-GPU box, <code>--backend ggml_cuda</code> alone gives you all three GPUs on GLM 5.x and DeepSeek V4 (layer split) and one GPU on Gemma 4; adding <code>--tp 3</code> switches GLM 5.x to tensor parallelism, caps DeepSeek V4's layer split at three devices (there <code>--tp N</code> is only a device count — the same thing <code>TS_DSV4_NGPU</code> sets), and gives Gemma 4 all three GPUs. On GLM 5.x that switch is a downgrade in speed and buys only capacity — see <a href="#performance">Measured results</a>.</p>
+  <p>The layer split applies to <strong>three architectures</strong>, all of which run through their own whole-model executors. <strong>DeepSeek V4 Flash (<code>deepseek4</code>) and GLM 5.x (<code>glm-dsa</code>, <code>glm5next</code>)</strong> do it <strong>by default, with no flag at all</strong>: they spread across every visible GPU because neither fits on one card. <code>TS_DSV4_NGPU</code> and <code>TS_GLM_NGPU</code> cap how many devices they use. <strong>Qwen 3.8 Flash Next (<code>qwen4exp</code>)</strong> is the one that asks: it stays on a single GPU until you pass <code>--tp N</code>, and what that runs is a layer split, not tensor parallelism — <code>qwen4exp</code> shards no weights, and this is the same (and only) multi-GPU mode llama.cpp offers the architecture, whose <code>-sm row</code> refuses to load it. GLM-5.3-Flash goes the other way: it refuses <code>--tp</code> cleanly and keeps its default layer split.</p>
+  <p>On <strong>every other architecture</strong>, running without <code>--tp</code> uses a <strong>single GPU</strong>. There is no automatic layer split on the generic per-op or fused-graph paths — a model that does not fit one card fails at load rather than being spread silently (the refusal that names the exact <code>--n-cpu-moe N</code> comes from the DeepSeek V4 and GLM 5.x whole-model loaders). An architecture that supports <em>neither</em> tensor parallelism nor a layer split now says so on stderr and runs on one GPU, instead of accepting <code>--tp</code> and quietly leaving the other cards idle.</p>
+  <p>So on a 3-GPU box, <code>--backend ggml_cuda</code> alone gives you all three GPUs on GLM 5.x and DeepSeek V4 (layer split) and one GPU on Gemma 4 and Qwen 3.8 Flash Next; adding <code>--tp 3</code> switches GLM-5.2 to tensor parallelism, caps DeepSeek V4's layer split at three devices (there <code>--tp N</code> is only a device count — the same thing <code>TS_DSV4_NGPU</code> sets), gives Gemma 4 all three GPUs by sharding inside its layers, and gives Qwen 3.8 Flash Next all three by splitting whole layers across them. On GLM-5.2 that switch is a downgrade in speed and buys only capacity; on Qwen 3.8 Flash Next the layer split is neither faster nor slower, and buys only capacity too — see <a href="#performance">Measured results</a>.</p>
 
   <h2 id="how">How tensor parallelism works</h2>
   <p>Each transformer block is rewritten into a pair of complementary shardings, so exactly one collective is needed per block half:</p>
@@ -96,7 +96,7 @@ <h3 id="multinode">Distributed TP — several machines</h3>
   <p>Nodes are usually started by hand a few seconds or minutes apart, so a node that comes up first keeps retrying its outbound connections for up to 120 seconds instead of failing on the first refused connection. Once the mesh is complete each node prints <code>[TcpCommunicator] Rank r/N connected to all peers.</code></p>
 
   <h2 id="architectures">Supported architectures</h2>
-  <p>Every autoregressive architecture in TensorSharp runs under TP; heterogeneous layers get their own sharding strategy.</p>
+  <p>Nearly every autoregressive architecture in TensorSharp runs under TP; heterogeneous layers get their own sharding strategy. The ones whose <code>--tp</code> means something else are marked as such.</p>
   <div class="table-scroll">
   <table>
     <thead><tr><th>Architecture</th><th>TP</th><th>Strategy / notes</th></tr></thead>
@@ -106,11 +106,12 @@ <h2 id="architectures">Supported architectures</h2>
       <tr><td>Gemma 3</td><td class="yes">✅</td><td>Separate Q/K/V, GELU, sliding-window attention.</td></tr>
       <tr><td>Gemma 4</td><td class="yes">✅</td><td>Dense and MoE, per-layer head dims; multimodal embeddings are injected into the TP path so vision/audio prompts survive. On GGML the fused whole-model MoE trunk splits <em>inside</em> each expert (gate/up column-parallel, down row-parallel) so global expert ids keep working — <code>TS_GEMMA4_TP_FUSED_MOE=0</code> falls back to the whole-expert per-op path. Direct CUDA uses per-expert slicing.</td></tr>
       <tr><td>Qwen 3.5 / 3.6 family</td><td class="yes">✅</td><td>Block-cyclic V-head ownership for the GatedDeltaNet recurrent layers — each rank keeps its own delta/conv state, device-resident, and needs no cross-rank traffic for the recurrent path. On GGML the whole GDN block runs as one packed per-rank kernel, MoE uses expert parallelism (whole experts per rank, Megatron-split shared expert), and the LM head is column-parallel with no collective at all. Direct CUDA uses expert slicing.</td></tr>
+      <tr><td>Qwen 3.8 Flash Next</td><td class="yes">✅ (layer split)</td><td>Different mechanism: <code>qwen4exp</code> shards no weights. <code>--tp N</code> gives each GPU a contiguous run of whole layers — the whole-token graph is cut at the device boundaries and the hidden state handed across — which is the only multi-GPU mode llama.cpp offers this architecture too (<code>-sm row</code> refuses to load it). Greedy output stays byte-identical to the single-GPU run and throughput is unchanged, so it is a capacity feature; see <a href="#perf-layer-split">The layer split, measured</a>. <code>TS_Q4E_LAYER_SPLIT=20,28</code> overrides the automatic balance with explicit per-GPU layer counts, and throws rather than ignoring a value it cannot honour.</td></tr>
       <tr><td>GPT OSS</td><td class="yes">✅</td><td>MoE expert slicing, attention sinks, YaRN. Runs on the GGML backends too, though the GGML MoE path still walks experts per token per rank rather than using expert parallelism.</td></tr>
       <tr><td>Nemotron-H</td><td class="yes">✅</td><td>MoE expert slicing; Mamba2 SSM layers are computed on rank 0 and broadcast. Same GGML MoE caveat as GPT OSS.</td></tr>
       <tr><td>Muse-Glimmer</td><td class="yes">✅</td><td>Dense, but with three shapes that need care: the fused <code>[gate|up]</code> is split per segment (a contiguous split silently hands one rank all of <em>gate</em>), the per-head QK RMSNorms are 1-D <code>[head_dim]</code> vectors and stay <strong>replicated</strong> (the Q norm also carries the folded <code>qk_scale_factor</code>), and the attention output gate is column-parallel by head and applied inside the per-rank region. Both AllReduces land on the raw matmul output, <em>before</em> the 1e-8 post-norms — reducing after a non-linear norm produces fluent but wrong output. <strong>2 KV heads cap the degree at <code>--tp 2</code>.</strong></td></tr>
       <tr><td>DeepSeek V4 Flash</td><td class="yes">✅ (layer split)</td><td>Different mechanism: DSV4's whole-model executors split the model <em>by layer</em> across GPUs rather than sharding every weight, so there is no per-layer AllReduce. <code>--tp N</code> simply caps how many GPUs the split uses (same as <code>TS_DSV4_NGPU</code>); with no flag it uses every visible device. The split balances the largest per-device load, counting the fixed residents — embedding table on the first device, output head and the whole DSpark drafter on the last — where they actually land.</td></tr>
-      <tr><td>GLM 5.x</td><td class="yes">✅</td><td>MLA heads column-parallel (<code>attn_q_b</code> / <code>attn_k_b</code> / <code>attn_v_b</code>) with row-parallel <code>attn_output</code>; the 256 routed experts are Megatron-split <em>inside every expert</em> (gate/up column-parallel, down row-parallel) rather than partitioned by expert id, because <code>ggml_mul_mat_id</code> needs a token's selected expert ids to stay distinct. Router, norms, the lightning indexer, the shared expert and the 3 dense layers are replicated — two all-reduces per layer. GGML backends only; <code>TS_GLM_TP_SHARD</code> picks the halves (1 heads, 2 experts, 3 both) and <code>TS_GLM_TP_OVERSUBSCRIBE=1</code> packs several ranks onto one GPU for testing. Without <code>--tp</code>, GLM layer-splits across every visible GPU like DeepSeek V4.</td></tr>
+      <tr><td>GLM 5.x</td><td class="yes">✅</td><td>MLA heads column-parallel (<code>attn_q_b</code> / <code>attn_k_b</code> / <code>attn_v_b</code>) with row-parallel <code>attn_output</code>; the 256 routed experts are Megatron-split <em>inside every expert</em> (gate/up column-parallel, down row-parallel) rather than partitioned by expert id, because <code>ggml_mul_mat_id</code> needs a token's selected expert ids to stay distinct. Router, norms, the lightning indexer, the shared expert and the 3 dense layers are replicated — two all-reduces per layer. GGML backends only; <code>TS_GLM_TP_SHARD</code> picks the halves (1 heads, 2 experts, 3 both) and <code>TS_GLM_TP_OVERSUBSCRIBE=1</code> packs several ranks onto one GPU for testing. Without <code>--tp</code>, GLM layer-splits across every visible GPU like DeepSeek V4. That sharding is GLM-5.2 only: <strong>GLM-5.3-Flash (<code>glm5next</code>) refuses <code>--tp</code> cleanly</strong> — its KDA linear-attention layers, pooled indexer and Sinkhorn hyper-connections have no TP strategy yet — and runs the layer split instead.</td></tr>
       <tr><td>DiffusionGemma</td><td class="no">—</td><td>Not applicable (text-diffusion sampler, not autoregressive decode).</td></tr>
       <tr><td>Qwen-Image-Edit</td><td class="no">—</td><td>Not applicable (MMDiT image generation).</td></tr>
     </tbody>
@@ -146,6 +147,20 @@ <h2 id="performance">Measured results</h2>
 
   <p>That is not universal. On 3× RTX PRO 6000 (PCIe, no NVLink), GLM-5.2 UD-IQ2_XXS at prefill 2048 / decode 64 measures <strong>915.9 / 43.9</strong> tok/s on the plain layer split against <strong>505.6 / 17.6</strong> under <code>--tp 3</code>: each of the 78 layers needs two all-reduces of a <code>[6144, n_tokens]</code> hidden state, and on a PCIe host that costs more than the split saves. TP also holds a full-length cache on every rank, dropping the fitted context from 342,272 to 91,136 tokens, and it changes the reduction order — against the recorded llama.cpp goldens a 2-bit MoE reproduces 3 of 6 prompts under <code>--tp 3</code> where the layer split reproduces 5 of 6. There, TP is a capacity feature rather than a latency one. How much TP buys depends on the interconnect and on how much of a layer can be split at all.</p>
   <p>Decode — the memory-bound half TP is meant to help — reaches 1.39× a single GPU on Gemma 4 E4B and 1.06× on Qwen 3.5-9B, with both Gemma 4 models producing output <strong>byte-identical</strong> to their single-GPU runs. Prefill is compute-bound and pays the collectives, so it lands at or below the single-GPU figure for models that fit on one card. Qwen 3.5-35B-A3B does not fit a 16 GB card at all — TP is the only way to run it, and memory splits 9.4 + 8.0 GB across the pair.</p>
+  <h3 id="perf-layer-split">The layer split, measured</h3>
+  <p>A layer split shards no weights, so it should cost nothing and gain nothing — and that is what it measures. 2× A100-80GB, Qwen3.8-Flash-Next-UD-Q2_K_XL (73.4 GiB), <code>--tp 2</code> against one GPU:</p>
+  <div class="table-scroll">
+  <table>
+    <thead><tr><th>Measure</th><th>1 GPU</th><th><code>--tp 2</code> (layer split)</th></tr></thead>
+    <tbody>
+      <tr><td>Greedy output</td><td colspan="2"><strong>Byte-identical</strong> — same SHA-256</td></tr>
+      <tr><td>VRAM</td><td>whole model on one card</td><td><strong>24.2 + 26.2 GB</strong></td></tr>
+      <tr><td>Prefill</td><td>~1520–1550 t/s</td><td>~1520–1550 t/s</td></tr>
+      <tr><td>Decode</td><td>~56 t/s</td><td>~56 t/s</td></tr>
+    </tbody>
+  </table>
+  </div>
+  <p>llama.cpp on the same box behaves the same way: <code>-sm layer</code> takes pp1536 / tg128 from 1094 / 61.2 on one GPU to 1200 / 61.5 on two — about 10% prefill, nothing on decode — and <code>-sm row</code> refuses to load this architecture at all. So the reason to pass <code>--tp N</code> here is that the weights, the caches and the context do not fit on one card, not throughput. Startup prints which mode ran and the per-GPU layer/byte split. <code>TS_Q4E_LAYER_SPLIT=20,28</code> gives explicit layer counts per GPU (llama.cpp's <code>--tensor-split</code> in spirit) and throws rather than silently ignoring a value it cannot honour — useful, because the automatic balance prices weights and cannot see the vision tower, which loads later and lands on GPU 0.</p>
   <h3 id="perf-graphs">CUDA graph capture under TP</h3>
   <p>A tensor-parallel token is dozens of small per-rank submissions, and replaying them is worth about <strong>45% of decode throughput</strong> — so graph capture stays <em>on</em> under TP (disable with <code>TS_GGML_TP_CUDA_GRAPHS=0</code>). On 4×A40:</p>
   <div class="table-scroll">
@@ -206,7 +221,7 @@ <h2 id="troubleshooting">Troubleshooting</h2>
     <thead><tr><th>Symptom</th><th>Likely cause &amp; fix</th></tr></thead>
     <tbody>
       <tr><td>Startup fails: requested TP degree exceeds device count</td><td>The process sees fewer CUDA devices than <code>--tp</code> asks for. Check <code>CUDA_VISIBLE_DEVICES</code> and the driver.</td></tr>
-      <tr><td>Model loads on one GPU despite <code>--tp 2</code></td><td>The backend is <code>mlx</code>, <code>cpu</code>, or <code>ggml_cpu</code>/<code>ggml_metal</code>. TP applies to <code>cuda</code>, <code>ggml_cuda</code>, and <code>ggml_vulkan</code>.</td></tr>
+      <tr><td>Model loads on one GPU despite <code>--tp 2</code></td><td>The backend is <code>mlx</code>, <code>cpu</code>, or <code>ggml_cpu</code>/<code>ggml_metal</code>. TP applies to <code>cuda</code>, <code>ggml_cuda</code>, and <code>ggml_vulkan</code>. If the backend is right, check stderr: an architecture that supports neither tensor parallelism nor a layer split prints a notice and runs on one GPU rather than failing.</td></tr>
       <tr><td>A dimension is not divisible by the TP degree</td><td>Pick a degree that divides <code>numHeads</code>, <code>numKVHeads</code>, and <code>intermediateSize</code> — usually a power of two.</td></tr>
       <tr><td>A node hangs waiting for peers</td><td>The peer list, order, or port does not match on all nodes, or a firewall blocks the port. Raise <code>TENSORSHARP_TP_CONNECT_TIMEOUT_SECONDS</code> if the nodes simply start far apart.</td></tr>
       <tr><td>Garbled output only with multiple GPUs</td><td>Suspect the P2P DMA path. Re-run with <code>TENSORSHARP_TP_HOST_ALLREDUCE=1</code>, then <code>TENSORSHARP_TP_DISABLE_P2P=1</code>; if the output becomes correct, the topology's peer DMA is at fault.</td></tr>
diff --git a/website/distributed_zh-cn.html b/website/distributed_zh-cn.html
index 9e50aabc..117a3c76 100644
--- a/website/distributed_zh-cn.html
+++ b/website/distributed_zh-cn.html
@@ -16,9 +16,9 @@
 <main class="content" id="content">
   <div class="breadcrumb"><a href="index_zh-cn.html">首页</a> › 多 GPU 与多节点</div>
   <h1>多 GPU 与多节点</h1>
-  <p class="lede">一个模型，多张 GPU——当一台机器不够用时，还可以是多台机器。TensorSharp 按 Megatron-LM 列/行并行范式实现了<strong>张量并行</strong>，并通过点对点 TCP 网格把它扩展到多台主机。</p>
+  <p class="lede">一个模型，多张 GPU——当一台机器不够用时，还可以是多台机器。TensorSharp 按 Megatron-LM 列/行并行范式实现了<strong>张量并行</strong>，并通过点对点 TCP 网格把它扩展到多台主机。对于不切分权重的架构，同一个 <code>--tp N</code> 参数运行的是<strong>按层切分</strong> —— 那是容量特性，而不是速度特性。</p>
 
-  <div class="callout note"><div class="ic">🧭</div><p><strong>一句话上手：</strong>加上 <code>--tp N</code> 即可在本机 N 张 GPU 上运行；在此基础上再加 <code>--tp-node-id</code> 与 <code>--tp-peers</code> 就能跨机器扩展。<code>TensorSharp.Cli</code> 与 <code>TensorSharp.Server</code> 都支持这些参数，可运行在 Direct <a href="backends_zh-cn.html#cuda"><code>cuda</code></a> 后端以及 <a href="backends_zh-cn.html#ggml">GGML CUDA / Vulkan</a> 后端上。</p></div>
+  <div class="callout note"><div class="ic">🧭</div><p><strong>一句话上手：</strong>加上 <code>--tp N</code> 即可在本机 N 张 GPU 上运行 —— 架构支持时切分每一层，不支持时则按整层把模型摊开；在此基础上再加 <code>--tp-node-id</code> 与 <code>--tp-peers</code> 就能跨机器扩展。<code>TensorSharp.Cli</code> 与 <code>TensorSharp.Server</code> 都支持这些参数，可运行在 Direct <a href="backends_zh-cn.html#cuda"><code>cuda</code></a> 后端以及 <a href="backends_zh-cn.html#ggml">GGML CUDA / Vulkan</a> 后端上。</p></div>
 
   <h2 id="why">什么时候需要它？</h2>
   <ul>
@@ -31,12 +31,12 @@ <h2 id="why">什么时候需要它？</h2>
   <h2 id="vs-layer-split">TP 与按层切分 —— 多卡机器上到底发生了什么</h2>
   <p>模型占用多张 GPU 有两种完全不同的方式，其中只有一种是 <code>--tp</code>。</p>
   <ul>
-    <li><strong>张量并行（<code>--tp N</code>）</strong>把<em>每一层</em>都放到<em>每个 rank</em> 上，切分的是层<em>内部</em>的权重，于是一次 decode 每张卡只读 <code>1/N</code> 的字节，各 rank 在每个层边界做 all-reduce。下表里的所有架构都支持它，而且它是<strong>按需开启</strong>的 —— 不主动要求，就不会有任何张量被切开。</li>
+    <li><strong>张量并行（<code>--tp N</code>）</strong>把<em>每一层</em>都放到<em>每个 rank</em> 上，切分的是层<em>内部</em>的权重，于是一次 decode 每张卡只读 <code>1/N</code> 的字节，各 rank 在每个层边界做 all-reduce。它是<strong>按需开启</strong>的 —— 不主动要求，就不会有任何张量被切开 —— 下表里的多数（但已不是全部）架构实现了它。</li>
     <li><strong>按层切分</strong>则是把<em>整层</em>放到不同设备上顺序执行：设备 0 算第 0..k 层，把隐状态交给设备 1，依此类推。切点并不是简单的 <code>n_layer/N</code> —— 加载器会测量每张卡的空闲显存，再做装箱以均衡「占用预算的最大比例」，因此配置不一致的一组卡也能被均匀填满。没有集合通信、层内也不切分，因此在慢速互连上不花额外代价，但同一时刻只有一张卡在忙。</li>
   </ul>
-  <p>按层切分只适用于<strong>两个架构 —— DeepSeek V4 Flash（<code>deepseek4</code>）与 GLM 5.x（<code>glm-dsa</code>）</strong>，也就是走各自整模型执行器的那两个；而且这是它们<strong>不加任何参数时的默认行为</strong>：它们会摊到所有可见 GPU 上，因为两者都装不进单卡。<code>TS_DSV4_NGPU</code> 与 <code>TS_GLM_NGPU</code> 用来限制使用几张卡。</p>
-  <p>在<strong>其他所有架构</strong>上，不加 <code>--tp</code> 就是<strong>只用一张 GPU</strong>。通用的逐算子路径与融合图路径都没有自动按层切分 —— 装不下单卡的模型会在加载时失败，而不会被悄悄摊开（那条带具体 <code>--n-cpu-moe N</code> 建议的拒绝信息，只有 DeepSeek V4 与 GLM 5.x 的整模型加载器会打印）。</p>
-  <p>所以在一台 3 卡机器上：只写 <code>--backend ggml_cuda</code>，GLM 5.x 与 DeepSeek V4 会用满三张卡（按层切分），Gemma 4 只用一张；再加上 <code>--tp 3</code>，GLM 5.x 切换成张量并行，DeepSeek V4 只是把按层切分限制在这 3 张卡上（那里的 <code>--tp N</code> 仅仅是个设备数，等同 <code>TS_DSV4_NGPU</code>），Gemma 4 则用上三张卡。在 GLM 5.x 上这个切换只会更慢，换来的仅仅是容量 —— 见<a href="#performance">实测结果</a>。</p>
+  <p>按层切分适用于<strong>三个架构</strong>，它们都走各自的整模型执行器。<strong>DeepSeek V4 Flash（<code>deepseek4</code>）与 GLM 5.x（<code>glm-dsa</code>、<code>glm5next</code>）</strong><strong>不加任何参数时就默认如此</strong>：它们会摊到所有可见 GPU 上，因为两者都装不进单卡。<code>TS_DSV4_NGPU</code> 与 <code>TS_GLM_NGPU</code> 用来限制使用几张卡。<strong>Qwen 3.8 Flash Next（<code>qwen4exp</code>）</strong>则是需要你开口的那一个：不传 <code>--tp N</code> 时它只用一张 GPU，而传了之后跑的是按层切分而非张量并行 —— <code>qwen4exp</code> 不切分任何权重，这也是 llama.cpp 为该架构提供的唯一多 GPU 模式（它的 <code>-sm row</code> 拒绝加载这个模型）。GLM-5.3-Flash 则相反：它会干脆拒绝 <code>--tp</code>，继续用默认的按层切分。</p>
+  <p>在<strong>其他所有架构</strong>上，不加 <code>--tp</code> 就是<strong>只用一张 GPU</strong>。通用的逐算子路径与融合图路径都没有自动按层切分 —— 装不下单卡的模型会在加载时失败，而不会被悄悄摊开（那条带具体 <code>--n-cpu-moe N</code> 建议的拒绝信息，只有 DeepSeek V4 与 GLM 5.x 的整模型加载器会打印）。<em>两种模式都不支持</em>的架构现在会在 stderr 上明说并只用一张 GPU，而不是接受 <code>--tp</code> 之后默默让其余卡闲置。</p>
+  <p>所以在一台 3 卡机器上：只写 <code>--backend ggml_cuda</code>，GLM 5.x 与 DeepSeek V4 会用满三张卡（按层切分），Gemma 4 与 Qwen 3.8 Flash Next 只用一张；再加上 <code>--tp 3</code>，GLM-5.2 切换成张量并行，DeepSeek V4 只是把按层切分限制在这 3 张卡上（那里的 <code>--tp N</code> 仅仅是个设备数，等同 <code>TS_DSV4_NGPU</code>），Gemma 4 通过层内切分用上三张卡，Qwen 3.8 Flash Next 则通过把整层摊开用上三张卡。在 GLM-5.2 上这个切换只会更慢，换来的仅仅是容量；在 Qwen 3.8 Flash Next 上按层切分不快也不慢，换来的同样只是容量 —— 见<a href="#performance">实测结果</a>。</p>
 
   <h2 id="how">张量并行的工作原理</h2>
   <p>每个 transformer block 被改写为一对互补的切分方式，因此每半个 block 只需要一次集合通信：</p>
@@ -96,7 +96,7 @@ <h3 id="multinode">分布式 TP —— 多台机器</h3>
   <p>节点通常由人工间隔数秒到数分钟启动，因此先起来的节点会持续重试对外连接（最长 120 秒），而不是在第一次连接被拒绝时就失败。网格建立完成后，每个节点会打印 <code>[TcpCommunicator] Rank r/N connected to all peers.</code>。</p>
 
   <h2 id="architectures">支持的架构</h2>
-  <p>TensorSharp 中的每一种自回归架构都可以在 TP 下运行；异构层各有自己的切分策略。</p>
+  <p>TensorSharp 中几乎每一种自回归架构都可以在 TP 下运行；异构层各有自己的切分策略。<code>--tp</code> 含义不同的那几个会单独标注。</p>
   <div class="table-scroll">
   <table>
     <thead><tr><th>架构</th><th>TP</th><th>策略 / 说明</th></tr></thead>
@@ -106,10 +106,11 @@ <h2 id="architectures">支持的架构</h2>
       <tr><td>Gemma 3</td><td class="yes">✅</td><td>分离 Q/K/V、GELU、滑动窗口注意力。</td></tr>
       <tr><td>Gemma 4</td><td class="yes">✅</td><td>稠密与 MoE、逐层 head 维度；多模态嵌入会注入 TP 路径，因此视觉 / 音频提示不会丢失。GGML 上融合的整模 MoE 主干在<em>每个专家内部</em>切分（gate/up 列并行、down 行并行），从而保留全局专家 id —— <code>TS_GEMMA4_TP_FUSED_MOE=0</code> 可回退到逐算子的整专家路径。Direct CUDA 使用逐专家切分。</td></tr>
       <tr><td>Qwen 3.5 / 3.6 family</td><td class="yes">✅</td><td>GatedDeltaNet 循环层采用块循环 V-head 归属 —— 每个 rank 维护自己的、常驻设备的 delta/conv 状态，循环路径无需跨 rank 通信。GGML 上整个 GDN block 作为一个打包的按 rank 内核运行，MoE 采用专家并行（每个 rank 持有整个专家，shared expert 按 Megatron 切分），LM head 为列并行且完全不需要集合通信。Direct CUDA 使用专家切分。</td></tr>
+      <tr><td>Qwen 3.8 Flash Next</td><td class="yes">✅（按层切分）</td><td>机制不同：<code>qwen4exp</code> 不切分任何权重。<code>--tp N</code> 给每张 GPU 一段连续的完整层 —— 整 token 的计算图在设备边界处切开，隐状态在卡之间传递 —— 这也是 llama.cpp 为该架构提供的唯一多 GPU 模式（<code>-sm row</code> 拒绝加载它）。贪心输出与单卡运行逐字节一致、吞吐也没有变化，因此它是容量特性；见<a href="#perf-layer-split">按层切分的实测</a>。<code>TS_Q4E_LAYER_SPLIT=20,28</code> 可用明确的每卡层数覆盖自动均衡，遇到无法满足的取值会抛错而不是忽略。</td></tr>
       <tr><td>GPT OSS</td><td class="yes">✅</td><td>MoE 专家切分、attention sink、YaRN。GGML 后端同样可运行，但 GGML 上的 MoE 路径目前仍按 token 逐个遍历专家，尚未使用专家并行。</td></tr>
       <tr><td>Nemotron-H</td><td class="yes">✅</td><td>MoE 专家切分；Mamba2 SSM 层在 rank 0 上计算后广播。GGML 上的 MoE 限制与 GPT OSS 相同。</td></tr>
       <tr><td>DeepSeek V4 Flash</td><td class="yes">✅（按层切分）</td><td>机制不同：DSV4 的整模型执行器是把模型<em>按层</em>切分到各 GPU，而不是切分每个权重，因此没有逐层 AllReduce。<code>--tp N</code> 只是限制切分使用几张 GPU（等同 <code>TS_DSV4_NGPU</code>）；不传参数时使用全部可见设备。切分以单卡最大负载为优化目标，并按各设备固定常驻项的实际落点计入——embedding 表在第一张卡，输出头与整个 DSpark 草稿器在最后一张卡。</td></tr>
-      <tr><td>GLM 5.x</td><td class="yes">✅</td><td>MLA 的头列并行（<code>attn_q_b</code> / <code>attn_k_b</code> / <code>attn_v_b</code>）配行并行 <code>attn_output</code>；256 个路由专家不是按专家 id 划分，而是在<em>每个专家内部</em>按 Megatron 切分（gate/up 列并行、down 行并行），因为 <code>ggml_mul_mat_id</code> 要求同一 token 选中的专家 id 互不相同。router、各处 norm、lightning indexer、共享专家与 3 个稠密层均为复制——每层两次 all-reduce。仅限 GGML 后端；<code>TS_GLM_TP_SHARD</code> 选择切哪一半（1 头、2 专家、3 两者都切），<code>TS_GLM_TP_OVERSUBSCRIBE=1</code> 可把多个 rank 挤在一张卡上做测试。不加 <code>--tp</code> 时，GLM 像 DeepSeek V4 一样按层切分到所有可见 GPU。</td></tr>
+      <tr><td>GLM 5.x</td><td class="yes">✅</td><td>MLA 的头列并行（<code>attn_q_b</code> / <code>attn_k_b</code> / <code>attn_v_b</code>）配行并行 <code>attn_output</code>；256 个路由专家不是按专家 id 划分，而是在<em>每个专家内部</em>按 Megatron 切分（gate/up 列并行、down 行并行），因为 <code>ggml_mul_mat_id</code> 要求同一 token 选中的专家 id 互不相同。router、各处 norm、lightning indexer、共享专家与 3 个稠密层均为复制——每层两次 all-reduce。仅限 GGML 后端；<code>TS_GLM_TP_SHARD</code> 选择切哪一半（1 头、2 专家、3 两者都切），<code>TS_GLM_TP_OVERSUBSCRIBE=1</code> 可把多个 rank 挤在一张卡上做测试。不加 <code>--tp</code> 时，GLM 像 DeepSeek V4 一样按层切分到所有可见 GPU。上述切分仅限 GLM-5.2：<strong>GLM-5.3-Flash（<code>glm5next</code>）会干脆拒绝 <code>--tp</code></strong> —— 它的 KDA 线性注意力层、池化 indexer 与 Sinkhorn 超连接尚无 TP 策略 —— 转而使用按层切分。</td></tr>
       <tr><td>DiffusionGemma</td><td class="no">—</td><td>不适用（文本扩散采样器，非自回归 decode）。</td></tr>
       <tr><td>Qwen-Image-Edit</td><td class="no">—</td><td>不适用（MMDiT 图像生成）。</td></tr>
     </tbody>
@@ -140,6 +141,20 @@ <h2 id="performance">实测结果</h2>
   </table>
   </div>
   <p>Decode —— TP 本该受益的访存瓶颈部分 —— 在 Gemma 4 E4B 上达到单卡的 1.39×，在 Qwen 3.5-9B 上为 1.06×，且两个 Gemma 4 模型的输出与单卡运行<strong>逐字节一致</strong>。Prefill 受计算约束且要承担集合通信开销，因此在单卡装得下的模型上持平或略低于单卡。Qwen 3.5-35B-A3B 在 16 GB 卡上根本装不下 —— 只有 TP 能跑，两张卡上的显存占用为 9.4 + 8.0 GB。</p>
+  <h3 id="perf-layer-split">按层切分的实测</h3>
+  <p>按层切分不切分任何权重，因此它既不该有开销，也不该有收益 —— 实测正是如此。2× A100-80GB，Qwen3.8-Flash-Next-UD-Q2_K_XL（73.4 GiB），<code>--tp 2</code> 对比单卡：</p>
+  <div class="table-scroll">
+  <table>
+    <thead><tr><th>指标</th><th>1 GPU</th><th><code>--tp 2</code>（按层切分）</th></tr></thead>
+    <tbody>
+      <tr><td>贪心输出</td><td colspan="2"><strong>逐字节一致</strong> —— SHA-256 相同</td></tr>
+      <tr><td>显存</td><td>整个模型压在一张卡上</td><td><strong>24.2 + 26.2 GB</strong></td></tr>
+      <tr><td>Prefill</td><td>约 1520–1550 t/s</td><td>约 1520–1550 t/s</td></tr>
+      <tr><td>Decode</td><td>约 56 t/s</td><td>约 56 t/s</td></tr>
+    </tbody>
+  </table>
+  </div>
+  <p>llama.cpp 在同一台机器上表现相同：<code>-sm layer</code> 把 pp1536 / tg128 从单卡的 1094 / 61.2 提到双卡的 1200 / 61.5 —— prefill 约 10%，decode 几乎为零 —— 而 <code>-sm row</code> 干脆拒绝加载这个架构。所以这里传 <code>--tp N</code> 的理由是权重、缓存与上下文装不进一张卡，而不是吞吐。启动时会打印实际使用的模式与每张 GPU 的层数 / 字节划分。<code>TS_Q4E_LAYER_SPLIT=20,28</code> 可为每张 GPU 指定明确的层数（精神上等同 llama.cpp 的 <code>--tensor-split</code>），遇到无法满足的取值会抛错而不是默默忽略 —— 这很有用，因为自动均衡只按权重计价，看不到稍后才加载、并落在 GPU 0 上的视觉塔。</p>
 
   <p>但这并不普适。在 3× RTX PRO 6000（PCIe，无 NVLink）上，GLM-5.2 UD-IQ2_XXS 以 prefill 2048 / decode 64 测得：按层切分 <strong>915.9 / 43.9</strong> tok/s，而 <code>--tp 3</code> 只有 <strong>505.6 / 17.6</strong>——78 层里每层都要对 <code>[6144, n_tokens]</code> 的隐状态做两次 all-reduce，在 PCIe 主机上这笔开销超过了拆分省下的算力。TP 还要求每个 rank 各自持有一份全长缓存，能装下的上下文因此从 342,272 掉到 91,136 token；它也改变了归约顺序——对着录制的 llama.cpp 金标准，2-bit MoE 在 <code>--tp 3</code> 下复现 3/6 条提示，而按层切分复现 5/6。那里的 TP 是容量特性而非延迟特性。TP 能带来多少，取决于互连带宽以及一层里究竟有多少能拆。</p>
 
@@ -202,7 +217,7 @@ <h2 id="troubleshooting">故障排查</h2>
     <thead><tr><th>现象</th><th>可能原因与处理</th></tr></thead>
     <tbody>
       <tr><td>启动失败：请求的 TP 度超过设备数</td><td>进程看到的 CUDA 设备少于 <code>--tp</code> 所要求的数量。检查 <code>CUDA_VISIBLE_DEVICES</code> 与驱动。</td></tr>
-      <tr><td>指定了 <code>--tp 2</code>，模型却只加载到一张 GPU</td><td>后端是 <code>mlx</code>、<code>cpu</code> 或 <code>ggml_cpu</code>/<code>ggml_metal</code>。TP 适用于 <code>cuda</code>、<code>ggml_cuda</code> 与 <code>ggml_vulkan</code>。</td></tr>
+      <tr><td>指定了 <code>--tp 2</code>，模型却只加载到一张 GPU</td><td>后端是 <code>mlx</code>、<code>cpu</code> 或 <code>ggml_cpu</code>/<code>ggml_metal</code>。TP 适用于 <code>cuda</code>、<code>ggml_cuda</code> 与 <code>ggml_vulkan</code>。若后端没问题，请看 stderr：既不支持张量并行也不支持按层切分的架构会打印一条提示并只用一张 GPU，而不是直接失败。</td></tr>
       <tr><td>某个维度无法被 TP 度整除</td><td>选择能整除 <code>numHeads</code>、<code>numKVHeads</code> 与 <code>intermediateSize</code> 的并行度，通常取 2 的幂。</td></tr>
       <tr><td>节点一直等待 peer</td><td>各节点的 peer 列表、顺序或端口不一致，或者防火墙拦截了该端口。若只是节点启动间隔较久，调大 <code>TENSORSHARP_TP_CONNECT_TIMEOUT_SECONDS</code>。</td></tr>
       <tr><td>只有多 GPU 时输出乱码</td><td>怀疑 P2P DMA 路径。先用 <code>TENSORSHARP_TP_HOST_ALLREDUCE=1</code> 重跑，再试 <code>TENSORSHARP_TP_DISABLE_P2P=1</code>；如果输出恢复正常，问题出在该拓扑的 peer DMA 上。</td></tr>
diff --git a/website/features.html b/website/features.html
index 5ea7139b..328fc8e1 100644
--- a/website/features.html
+++ b/website/features.html
@@ -20,7 +20,7 @@ <h1>Features</h1>
 
   <h2 id="catalog">Highlights</h2>
   <div class="card-grid">
-    <div class="card"><div class="ic">🧠</div><h3>Multi-architecture</h3><p>DeepSeek V4 Flash, GLM 5.2, Gemma 4 / 3, Qwen 3 / 3.5 / 3.6, GPT OSS, Nemotron-H, Mistral 3, Muse-Glimmer, DiffusionGemma, Qwen-Image-Edit, MiniMax-H3 audio+video, Wan video.</p></div>
+    <div class="card"><div class="ic">🧠</div><h3>Multi-architecture</h3><p>DeepSeek V4 Flash, GLM 5.2 &amp; GLM-5.3-Flash, Gemma 4 / 3, Qwen 3 / 3.5 / 3.6 / 3.8 Flash Next, GPT OSS, Nemotron-H, Mistral 3, Muse-Glimmer, DiffusionGemma, Qwen-Image-Edit, MiniMax-H3 audio+video, Wan video.</p></div>
     <div class="card"><div class="ic">🖼️</div><h3>Multimodal</h3><p>Image, video, and audio inputs (Gemma 4); image input for several others.</p></div>
     <div class="card"><div class="ic">📄</div><h3>PDF documents</h3><p>Upload PDFs in the Web UI or pass <code>--pdf</code> on the CLI — text PDFs are inlined, scanned pages go to vision models.</p></div>
     <div class="card"><div class="ic">🎨</div><h3>Image editing</h3><p>Qwen-Image-Edit turns a prompt + input image into an edited image (MMDiT diffusion).</p></div>
@@ -30,20 +30,22 @@ <h2 id="catalog">Highlights</h2>
     <div class="card"><div class="ic">📦</div><h3>Native quantized compute</h3><p>Q4_K_M, Q8_0, MXFP4, IQ2_XXS and more run in matmul without dequantizing to FP32.</p></div>
     <div class="card"><div class="ic">🔀</div><h3>Continuous batching</h3><p>vLLM-style paged KV cache with cross-request prefix sharing.</p></div>
     <a class="card" href="advanced.html#dspark"><div class="ic">⏩</div><h3>Speculative decoding</h3><p>MTP / NextN draft heads, DeepSeek V4's DSpark and Muse-Glimmer's DFlash block drafters accelerate solo decode.</p></a>
-    <a class="card" href="distributed.html"><div class="ic">🔗</div><h3>Multi-GPU &amp; multi-node</h3><p>Tensor parallelism splits one model across GPUs — CUDA and GGML alike — and across machines over a TCP mesh.</p></a>
+    <a class="card" href="distributed.html"><div class="ic">🔗</div><h3>Multi-GPU &amp; multi-node</h3><p>Tensor parallelism shards every layer across GPUs — CUDA and GGML alike — and across machines over a TCP mesh; the whole-model executors take a layer split instead.</p></a>
     <div class="card"><div class="ic">🔌</div><h3>Ollama &amp; OpenAI APIs</h3><p>Drop-in endpoints for existing tooling, plus a browser chat UI.</p></div>
   </div>
 
   <h2 id="models-features">Models &amp; modalities</h2>
   <ul>
-    <li><strong>Multi-architecture support</strong> — DeepSeek V4 Flash, GLM 5.2, Gemma 4, Gemma 3, DiffusionGemma, Qwen 3, Qwen 3.5/3.6-family, GPT OSS, Nemotron-H, Mistral 3, Muse-Glimmer, Qwen-Image-Edit, MiniMax-H3 audio+video, Wan 2.1 / 2.2 video. → <a href="models.html#table">Supported models</a></li>
+    <li><strong>Multi-architecture support</strong> — DeepSeek V4 Flash, GLM 5.2, GLM-5.3-Flash, Gemma 4, Gemma 3, DiffusionGemma, Qwen 3, Qwen 3.5/3.6-family, Qwen 3.8 Flash Next, GPT OSS, Nemotron-H, Mistral 3, Muse-Glimmer, Qwen-Image-Edit, MiniMax-H3 audio+video, Wan 2.1 / 2.2 video. → <a href="models.html#table">Supported models</a></li>
     <li><strong>DeepSeek V4 Flash (284B MoE)</strong> — a compressed-sparse-attention, 1M-context architecture with three dedicated whole-model executors: a direct-CUDA engine (<code>--backend cuda</code>), the native ggml executor (<code>ggml_cuda</code> / <code>ggml_vulkan</code>), and a 100% pure-C# CPU executor (<code>--backend cpu</code>, no native dependencies). Weights layer-split automatically across every visible GPU, and the server hosts it with native per-sequence slots and continuous batching. → <a href="models-text.html#deepseek4">DeepSeek V4</a></li>
     <li><strong>GLM 5.2 (744B-A40B MoE)</strong> — Multi-head Latent Attention with weight absorption plus DeepSeek Sparse Attention's lightning indexer, 256 routed experts at top-8 with one shared expert, and a 1M advertised context, loaded straight from a 6-shard split GGUF. Two implementations: a native whole-model ggml executor (<code>ggml_cuda</code> / <code>ggml_vulkan</code> / <code>ggml_cpu</code> / <code>ggml_metal</code>) that layer-splits 226 GiB across every visible GPU, and a managed per-op path used by <code>--backend cpu</code> (100% pure C#, no native dependencies) and <code>--backend cuda</code>. → <a href="models-text.html#glm">GLM 5.x</a></li>
-    <li><strong>Multimodal inference</strong> — image, video, and audio inputs for Gemma 4; images for Gemma 3, Qwen 3.5-family, Mistral 3, Nemotron-H Omni, and Muse-Glimmer. → <a href="models-text.html#multimodal">Multimodal</a></li>
+    <li><strong>GLM-5.3-Flash (320B MoE, text + image)</strong> — the hybrid successor loads through the <em>same</em> native executor and the same <code>GlmDsaModel</code> as GLM-5.2, under the GGUF arch id <code>glm5next</code>, with four architectural changes layered on: KDA linear attention on 34 of the 45 trunk layers, NoPE MLA + DSA on the other 11, a <em>pooled</em> indexer (4-cell pools, top-k 2048 over pools, then expanded to their members), Sinkhorn hyper-connections over ×4 streams, and a SwiGLU clamp at 10; 288 routed experts at top-8 with one shared and a ×2.5 routed scale, 46 blocks = 45 trunk + 1 NextN. It layer-splits across every visible GPU by default, takes <code>--cpu-moe</code> / <code>--n-cpu-moe</code>, and serves through per-sequence native slots. <code>--tp</code> tensor parallelism and NextN/MTP speculation are <em>refused cleanly</em> rather than half-supported — use the layer split. Vision comes from <code>mmproj-BF16.gguf</code> (the GLM-OCR ViT): <code>--image</code>, multi-image, and multi-turn image sessions. Measured on 2× RTX PRO 6000 Blackwell (96 GB) against llama.cpp, UD-Q2_K_XL (101 GiB), both engines layer-split at <code>n_ubatch</code> 2048, back to back: <strong>tg64 73.5 vs 36.6 t/s — decode at 2.0× llama.cpp</strong>, with prefill within a few percent either way (pp2048 2014 vs 2070, pp16384 1692 vs 1690, pp32768 1446 vs 1483). → <a href="models-text.html#glm">GLM 5.x</a></li>
+    <li><strong>Qwen 3.8 Flash Next (hybrid MoE, text + image)</strong> — GatedDeltaNet recurrent layers on 36 of its 48 layers, interleaved with full-attention layers (some behind Qwen Sparse Attention's indexer), a PLE n-gram embedding block on layer 1, ×4 hyper-connection streams and a 512-expert MoE at top-10 (hidden 2560, 24 query / 2 KV heads, head_dim 256, vocab 248320). The GGUF arch id is <code>qwen4exp</code>. On the GGML backends the whole token runs as (almost) <em>one</em> graph — embedding, in-graph PLE, all 48 layers, the final mixer and the LM head — replayed from a shape-keyed cache of captured graphs. Vision rides the Qwen3.5-VL tower with (T,H,W) IMRoPE, so multi-image and multi-turn image sessions work with KV reuse across turns (extend-only: the GDN recurrence cannot rewind). Thinking, tool calling, the server, and continuous batching through per-sequence state holders are all supported, and <code>--tp N</code> runs a multi-GPU <a href="distributed.html#vs-layer-split">layer split</a>. → <a href="models-text.html#qwen4exp">Qwen 3.8 Flash Next</a></li>
+    <li><strong>Multimodal inference</strong> — image, video, and audio inputs for Gemma 4; images for Gemma 3, Qwen 3.5-family, Qwen 3.8 Flash Next, GLM-5.3-Flash, Mistral 3, Nemotron-H Omni, and Muse-Glimmer. → <a href="models-text.html#multimodal">Multimodal</a></li>
     <li><strong>PDF document input</strong> — born-digital PDFs have their complete text layer extracted and inlined into the prompt; scanned PDFs fall back to page images for vision-capable models. Available as a Web UI upload and via the CLI's one-shot <code>--pdf</code> flag; cap the pages read with <code>TS_PDF_MAX_PAGES</code> (default: all). → <a href="server.html#webui">Web UI</a></li>
     <li><strong>Mixture of Experts (MoE)</strong> — Gemma 4 MoE (e.g. 26B-A4B), GPT OSS MoE (gpt-oss-20b), Qwen 3.5/3.6 MoE (35B-A3B), and Nemotron-H MoE FFN layers, with a fused batched GPU MoE dispatch.</li>
     <li><strong>Hybrid SSM-Transformer</strong> — Nemotron-H mixes Mamba2 SSM layers, attention layers, and MoE FFN in one model.</li>
-    <li><strong>Hybrid Attention-Recurrent</strong> — Qwen 3.5/3.6-family mix full-attention layers with GatedDeltaNet recurrent layers.</li>
+    <li><strong>Hybrid Attention-Recurrent</strong> — Qwen 3.5/3.6-family mix full-attention layers with GatedDeltaNet recurrent layers; Qwen 3.8 Flash Next does the same on 36 of 48 layers and adds a PLE n-gram block, ×4 hyper-connection streams and a 512-expert MoE, while GLM-5.3-Flash pairs KDA linear attention (34 of 45 trunk layers) with NoPE MLA and a pooled sparse indexer on the rest.</li>
     <li><strong>Video generation with audio (MiniMax-H3)</strong> — one 50-block diffusion transformer denoises video <em>and</em> a native 32 kHz stereo soundtrack as a single packed latent, so the audio is model output rather than something added afterwards. Four conditioning modes (<code>--video-mode t2v</code> / <code>i2v</code> / <code>fl2v</code> / <code>ref</code>): a prompt alone, a photo animated as the first frame (<code>--image</code>), a first-and-last keyframe pair (<code>--end-image</code>), or up to nine identity references — stills, clips and soundtracks — for a brand-new scene (<code>--ref-image</code> / <code>--ref-video</code> / <code>--ref-audio</code>). The two denoisers are separate checkpoints, not settings: keyframes need the FL2VA file, references the Ref2VA one. It is CFG-distilled, so <code>--cfg 1.0</code> is required and 4–8 steps is the fast operating point against a 20-step default. Drives from the CLI, <code>/api/video-generate</code>, <code>/v1/videos/generations</code>, or the Web UI; the MP4 arrives with a sidecar <code>.wav</code>. → <a href="models-video.html#minimax-h3">MiniMax-H3</a></li>
     <li><strong>Video generation, video-only (Wan 2.1 / 2.2)</strong> — Wan 2.1 (text → video) and Wan 2.2 TI2V-5B / A14B (text → video and image → video, where the uploaded image becomes the first frame) render an H.264 MP4 from the CLI, <code>/v1/videos/generations</code>, or the Web UI. Point <code>--model</code> at a <em>step-distilled</em> checkpoint (Turbo / Lightning / FastWan) and the pipeline auto-detects it, dropping the denoise loop from 100 DiT passes to 4. → <a href="models-video.html#video">Video generation</a></li>
     <li><strong>Text-diffusion generation</strong> — DiffusionGemma uses an iterative EntropyBound denoising sampler instead of autoregressive decode. → <a href="advanced.html#diffusion">DiffusionGemma</a></li>
@@ -66,7 +68,7 @@ <h2 id="performance">Performance &amp; scale</h2>
     <li><strong>GPU-accelerated</strong> — GGML Metal (macOS), GGML CUDA (Windows/Linux + NVIDIA), GGML Vulkan (Windows/Linux + AMD/Intel/NVIDIA), a direct CUDA/cuBLAS backend, and an MLX backend for Apple Silicon — all with CPU fallbacks. → <a href="backends.html">Backends</a></li>
     <li><strong>Continuous batching &amp; paged KV cache</strong> — block-paged KV pool with block-hash prefix sharing, an iteration-level scheduler that admits/preempts sequences mid-batch, optional SSD-backed tier, and a native fused paged-attention kernel. → <a href="advanced.html#continuous-batching">Deep dive</a></li>
     <li><strong>Batched / parallel inference</strong> — N sequences packed into a single forward pass with paged K/V scatter (Mistral 3, Gemma 4, GPT OSS, Qwen 3 / 3.5 / 3.6, Nemotron-H).</li>
-    <li><strong>Tensor parallelism &amp; distributed inference</strong> — split one model across N GPUs with <code>--tp N</code> on the CLI or the server (Megatron-LM column/row-parallel, replicated norms/embeddings/LM head), on the direct <code>cuda</code> backend and on GGML CUDA / Vulkan, and extend the group across machines with a peer-to-peer TCP mesh (<code>--tp-node-id</code> / <code>--tp-peers</code>). Hierarchical AllReduce keeps only <code>1/tp_local</code> of each collective on the network; MoE expert slicing and expert parallelism, GatedDeltaNet per-rank V-head ownership, and Mamba2 replication cover the heterogeneous layers. Fused per-rank block graphs put <code>--tp 2</code> decode above a single GPU on Gemma 4 (51.7 vs 37.3 tok/s) and run models too large for one card. TP is not the only way a model reaches several GPUs, though: DeepSeek V4 and GLM 5.x <strong>layer-split across every visible GPU by default, with no flag</strong>, and <code>--tp</code> switches GLM 5.x from that to Megatron sharding within each layer while on DeepSeek V4 it only caps how many GPUs the layer split uses, while every other architecture uses a single GPU unless <code>--tp</code> is passed (see <a href="distributed.html#vs-layer-split">TP vs. the layer split</a>). On a large MoE with MLA over PCIe the trade can invert — GLM 5.2's two all-reduces per layer across 78 layers put <code>--tp 3</code> at pp2048 505.6 / tg64 17.6 against 915.9 / 43.9 for the default layer split on 3× RTX PRO 6000 with no NVLink, so TP there buys capacity rather than latency. → <a href="distributed.html">Multi-GPU &amp; Multi-Node</a></li>
+    <li><strong>Tensor parallelism &amp; distributed inference</strong> — split one model across N GPUs with <code>--tp N</code> on the CLI or the server (Megatron-LM column/row-parallel, replicated norms/embeddings/LM head), on the direct <code>cuda</code> backend and on GGML CUDA / Vulkan, and extend the group across machines with a peer-to-peer TCP mesh (<code>--tp-node-id</code> / <code>--tp-peers</code>). Hierarchical AllReduce keeps only <code>1/tp_local</code> of each collective on the network; MoE expert slicing and expert parallelism, GatedDeltaNet per-rank V-head ownership, and Mamba2 replication cover the heterogeneous layers. Fused per-rank block graphs put <code>--tp 2</code> decode above a single GPU on Gemma 4 (51.7 vs 37.3 tok/s) and run models too large for one card. TP is not the only way a model reaches several GPUs, though: DeepSeek V4 and GLM 5.x <strong>layer-split across every visible GPU by default, with no flag</strong>, and <code>--tp</code> switches GLM-5.2 from that to Megatron sharding within each layer (GLM-5.3-Flash refuses <code>--tp</code> and keeps the split) while on DeepSeek V4 it only caps how many GPUs the layer split uses. On <strong>Qwen 3.8 Flash Next</strong>, <code>--tp N</code> is a layer split too — each GPU holds a contiguous run of whole layers, no weight is sharded, and it is the same (and only) multi-GPU mode llama.cpp offers that architecture. It is a capacity feature, not a speed one: on 2× A100-80GB with Qwen3.8-Flash-Next-UD-Q2_K_XL (73.4 GiB) the 2-GPU greedy output is byte-identical to the 1-GPU run and throughput is unchanged (prefill ~1520–1550 t/s, decode ~56 t/s either way) — what it buys is weights landing as 24.2 + 26.2 GB instead of all on one card. Startup prints which mode ran and the per-GPU layer/byte split, and <code>TS_Q4E_LAYER_SPLIT=20,28</code> overrides the automatic balance. Every other architecture uses a single GPU unless <code>--tp</code> is passed, and one that supports neither mode now says so on stderr rather than silently leaving the other GPUs idle (see <a href="distributed.html#vs-layer-split">TP vs. the layer split</a>). On a large MoE with MLA over PCIe the trade can invert — GLM 5.2's two all-reduces per layer across 78 layers put <code>--tp 3</code> at pp2048 505.6 / tg64 17.6 against 915.9 / 43.9 for the default layer split on 3× RTX PRO 6000 with no NVLink, so TP there buys capacity rather than latency. → <a href="distributed.html">Multi-GPU &amp; Multi-Node</a></li>
     <li><strong>MoE CPU offload</strong> — <code>--cpu-moe</code> / <code>--n-cpu-moe N</code> keeps the routed experts of the first N layers in system RAM, served straight from the GGUF mapping with no private copy, and composes with <code>--tp</code>. It is how a checkpoint that does not fit runs at all rather than a speed knob: GLM 5.2 at <code>--n-cpu-moe 30</code> measures pp2048 94.7 / tg64 16.4 against 915.9 / 43.9 fully resident, and frees enough VRAM to raise the sized context from 342,272 to 646,400 tokens. → <a href="advanced.html#memory">Memory</a></li>
     <li><strong>MTP / NextN speculative decoding</strong> — multi-token-prediction draft heads accelerate solo decode; lossless because the request's own sampler drives both draft and verify. → <a href="advanced.html#mtp">Speculative decoding</a></li>
     <li><strong>DSpark block speculative decoding</strong> — DeepSeek V4's drafter proposes a whole <em>block</em> of tokens per step (a Markov head conditions each block position on the one before it, a confidence head gates how far to draft) and the trunk verifies the block in one batched forward. Loaded as a separate GGUF with <code>--draft-model</code>; measured 1.3–1.4× decode on 4×A40, up to 2.0× on multi-turn chat, with greedy output byte-identical to the baseline. → <a href="advanced.html#dspark">DSpark</a></li>
diff --git a/website/features_zh-cn.html b/website/features_zh-cn.html
index 269e6bf0..e2a1d79a 100644
--- a/website/features_zh-cn.html
+++ b/website/features_zh-cn.html
@@ -20,7 +20,7 @@ <h1>功能特性</h1>
 
   <h2 id="catalog">亮点</h2>
   <div class="card-grid">
-    <div class="card"><div class="ic">🧠</div><h3>多架构</h3><p>DeepSeek V4 Flash、GLM 5.2、Gemma 4 / 3、Qwen 3 / 3.5 / 3.6、GPT OSS、Nemotron-H、Mistral 3、Muse-Glimmer、DiffusionGemma、Qwen-Image-Edit、MiniMax-H3 音视频、Wan 视频。</p></div>
+    <div class="card"><div class="ic">🧠</div><h3>多架构</h3><p>DeepSeek V4 Flash、GLM 5.2 与 GLM-5.3-Flash、Gemma 4 / 3、Qwen 3 / 3.5 / 3.6 / 3.8 Flash Next、GPT OSS、Nemotron-H、Mistral 3、Muse-Glimmer、DiffusionGemma、Qwen-Image-Edit、MiniMax-H3 音视频、Wan 视频。</p></div>
     <div class="card"><div class="ic">🖼️</div><h3>多模态</h3><p>图像、视频与音频输入（Gemma 4）；多个其他模型支持图像输入。</p></div>
     <div class="card"><div class="ic">📄</div><h3>PDF 文档</h3><p>在 Web UI 上传 PDF，或在 CLI 传 <code>--pdf</code> —— 文本型 PDF 直接内联，扫描页则交给视觉模型。</p></div>
     <div class="card"><div class="ic">🎨</div><h3>图像编辑</h3><p>Qwen-Image-Edit 将提示词 + 输入图像转为编辑后的图像（MMDiT 扩散）。</p></div>
@@ -30,20 +30,22 @@ <h2 id="catalog">亮点</h2>
     <div class="card"><div class="ic">📦</div><h3>原生量化计算</h3><p>Q4_K_M、Q8_0、MXFP4、IQ2_XXS 等在 matmul 中直接运算，无需反量化到 FP32。</p></div>
     <div class="card"><div class="ic">🔀</div><h3>连续批处理</h3><p>vLLM 式分页 KV 缓存，跨请求前缀共享。</p></div>
     <a class="card" href="advanced_zh-cn.html#dspark"><div class="ic">⏩</div><h3>推测解码</h3><p>MTP / NextN 草稿头，以及 DeepSeek V4 的 DSpark 与 Muse-Glimmer 的 DFlash 块级草稿器加速单序列解码。</p></a>
-    <a class="card" href="distributed_zh-cn.html"><div class="ic">🔗</div><h3>多 GPU 与多节点</h3><p>张量并行把一个模型切分到多张 GPU 上（CUDA 与 GGML 皆可），并通过 TCP 网格跨机器扩展。</p></a>
+    <a class="card" href="distributed_zh-cn.html"><div class="ic">🔗</div><h3>多 GPU 与多节点</h3><p>张量并行把每一层的权重切分到多张 GPU 上（CUDA 与 GGML 皆可），并通过 TCP 网格跨机器扩展；整模型执行器走的则是分层切分。</p></a>
     <div class="card"><div class="ic">🔌</div><h3>Ollama 与 OpenAI API</h3><p>面向现有工具的即插即用端点，外加浏览器聊天 UI。</p></div>
   </div>
 
   <h2 id="models-features">模型与模态</h2>
   <ul>
-    <li><strong>多架构支持</strong> —— DeepSeek V4 Flash、GLM 5.2、Gemma 4、Gemma 3、DiffusionGemma、Qwen 3、Qwen 3.5/3.6-family、GPT OSS、Nemotron-H、Mistral 3、Muse-Glimmer、Qwen-Image-Edit、MiniMax-H3 音视频、Wan 2.1 / 2.2 视频。→ <a href="models_zh-cn.html#table">支持的模型</a></li>
+    <li><strong>多架构支持</strong> —— DeepSeek V4 Flash、GLM 5.2、GLM-5.3-Flash、Gemma 4、Gemma 3、DiffusionGemma、Qwen 3、Qwen 3.5/3.6-family、Qwen 3.8 Flash Next、GPT OSS、Nemotron-H、Mistral 3、Muse-Glimmer、Qwen-Image-Edit、MiniMax-H3 音视频、Wan 2.1 / 2.2 视频。→ <a href="models_zh-cn.html#table">支持的模型</a></li>
     <li><strong>DeepSeek V4 Flash（284B MoE）</strong> —— 一套压缩稀疏注意力、1M 上下文的架构，配有三套专属的整模型执行器：Direct CUDA 引擎（<code>--backend cuda</code>）、原生 ggml 执行器（<code>ggml_cuda</code> / <code>ggml_vulkan</code>），以及 100% 纯 C# 的 CPU 执行器（<code>--backend cpu</code>，零原生依赖）。权重自动按层切分到所有可见 GPU，服务端以原生 per-sequence slot 与连续批处理托管它。→ <a href="models-text_zh-cn.html#deepseek4">DeepSeek V4</a></li>
     <li><strong>GLM 5.2（744B-A40B MoE）</strong> —— 带权重吸收的多头潜在注意力（MLA）加上 DeepSeek 稀疏注意力的 lightning indexer，256 个路由专家 top-8 加一个共享专家，自报 1M 上下文，直接从 6 分片的 split GGUF 加载。两套实现：一个原生整模型 ggml 执行器（<code>ggml_cuda</code> / <code>ggml_vulkan</code> / <code>ggml_cpu</code> / <code>ggml_metal</code>，把 226 GiB 按层切分到所有可见 GPU），以及 <code>--backend cpu</code>（100% 纯 C#、零原生依赖）与 <code>--backend cuda</code> 使用的托管逐算子路径。→ <a href="models-text_zh-cn.html#glm">GLM 5.x</a></li>
-    <li><strong>多模态推理</strong> —— Gemma 4 支持图像、视频与音频输入；Gemma 3、Qwen 3.5-family、Mistral 3、Nemotron-H Omni 与 Muse-Glimmer 支持图像。→ <a href="models-text_zh-cn.html#multimodal">多模态</a></li>
+    <li><strong>GLM-5.3-Flash（320B MoE，文本 + 图像）</strong> —— 这个混合后继型号走的是与 GLM-5.2 <em>完全相同</em>的原生执行器和同一个 <code>GlmDsaModel</code>，GGUF 架构 id 为 <code>glm5next</code>，在其之上叠了四处架构改动：45 个主干层中有 34 层用 KDA 线性注意力，另外 11 层用 NoPE MLA + DSA，一个<em>池化</em> indexer（4 格一池，先在池上取 top-k 2048，再展开到池内成员），×4 流的 Sinkhorn 超连接，以及上限为 10 的 SwiGLU 截断；288 个路由专家 top-8 加一个共享专家、×2.5 路由缩放，46 个 block = 45 主干 + 1 NextN。默认按层切分到所有可见 GPU，支持 <code>--cpu-moe</code> / <code>--n-cpu-moe</code>，并通过原生 per-sequence slot 提供服务。<code>--tp</code> 张量并行与 NextN/MTP 推测解码是<em>干脆拒绝</em>而非半支持 —— 请使用分层切分。视觉能力来自 <code>mmproj-BF16.gguf</code>（GLM-OCR ViT）：支持 <code>--image</code>、多图与多轮图像会话。在 2× RTX PRO 6000 Blackwell（96 GB）上与 llama.cpp 对比实测，UD-Q2_K_XL（101 GiB），两个引擎都用分层切分、<code>n_ubatch</code> 均为 2048、背靠背运行：<strong>tg64 为 73.5 对 36.6 t/s —— decode 达到 llama.cpp 的 2.0×</strong>，prefill 两边相差均在几个百分点以内（pp2048 2014 对 2070，pp16384 1692 对 1690，pp32768 1446 对 1483）。→ <a href="models-text_zh-cn.html#glm">GLM 5.x</a></li>
+    <li><strong>Qwen 3.8 Flash Next（混合 MoE，文本 + 图像）</strong> —— 48 层里有 36 层是 GatedDeltaNet 循环层，与全注意力层交错（其中一些位于 Qwen 稀疏注意力的 indexer 之后），第 1 层还有一个 PLE n-gram 嵌入块，外加 ×4 超连接流与 512 专家 top-10 的 MoE（hidden 2560，24 个 query / 2 个 KV 头，head_dim 256，词表 248320）。GGUF 架构 id 为 <code>qwen4exp</code>。在 GGML 后端上，整个 token（几乎）作为<em>一张</em>计算图运行 —— 嵌入、图内 PLE、全部 48 层、最后的 mixer 与 LM head —— 并从按形状索引的已捕获图缓存中重放。视觉走 Qwen3.5-VL 塔并使用 (T,H,W) IMRoPE，因此多图与多轮图像会话都可用，并能跨轮复用 KV（只能向后延长：GDN 的循环状态无法回退）。思考、工具调用、服务端，以及基于 per-sequence 状态持有器的连续批处理均已支持，<code>--tp N</code> 则运行多 GPU <a href="distributed_zh-cn.html#vs-layer-split">分层切分</a>。→ <a href="models-text_zh-cn.html#qwen4exp">Qwen 3.8 Flash Next</a></li>
+    <li><strong>多模态推理</strong> —— Gemma 4 支持图像、视频与音频输入；Gemma 3、Qwen 3.5-family、Qwen 3.8 Flash Next、GLM-5.3-Flash、Mistral 3、Nemotron-H Omni 与 Muse-Glimmer 支持图像。→ <a href="models-text_zh-cn.html#multimodal">多模态</a></li>
     <li><strong>PDF 文档输入</strong> —— 原生数字（born-digital）PDF 会完整提取文本层并内联进提示词；扫描版 PDF 则回退为页面图像，交给具备视觉能力的模型。可通过 Web UI 上传使用，也可用 CLI 一次性生成的 <code>--pdf</code> 参数；用 <code>TS_PDF_MAX_PAGES</code> 限制读取的页数（默认：全部）。→ <a href="server_zh-cn.html#webui">Web UI</a></li>
     <li><strong>专家混合（MoE）</strong> —— Gemma 4 MoE（如 26B-A4B）、GPT OSS MoE（gpt-oss-20b）、Qwen 3.5/3.6 MoE（35B-A3B）、Nemotron-H MoE FFN 层，以及 GLM 5.2（744B-A40B：256 个路由专家 top-8 加 1 个共享专家，sigmoid 门控路由带一个只影响选择的 bias 与 x2.5 的路由缩放），配以融合的批量 GPU MoE 调度。</li>
     <li><strong>混合 SSM-Transformer</strong> —— Nemotron-H 在一个模型中混合 Mamba2 SSM 层、注意力层与 MoE FFN。</li>
-    <li><strong>混合注意力-循环</strong> —— Qwen 3.5/3.6-family 将全注意力层与 GatedDeltaNet 循环层混合。</li>
+    <li><strong>混合注意力-循环</strong> —— Qwen 3.5/3.6-family 将全注意力层与 GatedDeltaNet 循环层混合；Qwen 3.8 Flash Next 在 48 层中的 36 层上同样如此，并加上 PLE n-gram 块、×4 超连接流与 512 专家 MoE；GLM-5.3-Flash 则在 45 个主干层中用 34 层 KDA 线性注意力、其余用 NoPE MLA 加池化稀疏 indexer。</li>
     <li><strong>音视频联合生成（MiniMax-H3）</strong> —— 一个 50 块的扩散 Transformer 把视频<em>与</em>原生 32 kHz 立体声音轨当作同一份打包潜变量去噪，音轨因此是模型输出本身，而不是事后配上去的。四种条件模式（<code>--video-mode t2v</code> / <code>i2v</code> / <code>fl2v</code> / <code>ref</code>）：只给提示词、把一张照片当首帧动起来（<code>--image</code>）、给定首尾两张关键帧（<code>--end-image</code>），或用最多九个身份参考——静图、片段与音轨——去生成全新场景（<code>--ref-image</code> / <code>--ref-video</code> / <code>--ref-audio</code>）。两个去噪器是彼此独立的检查点而非开关：关键帧要 FL2VA 那个文件，参考要 Ref2VA 那个。它经过 CFG 蒸馏，因此必须用 <code>--cfg 1.0</code>，默认 20 步、4–8 步是快车道。可从 CLI、<code>/api/video-generate</code>、<code>/v1/videos/generations</code> 或 Web UI 驱动；MP4 会附带一个旁挂的 <code>.wav</code>。→ <a href="models-video_zh-cn.html#minimax-h3">MiniMax-H3</a></li>
     <li><strong>视频生成，仅画面（Wan 2.1 / 2.2）</strong> —— Wan 2.1（文本 → 视频）与 Wan 2.2 TI2V-5B / A14B（文本 → 视频，以及图像 → 视频，上传的图像作为首帧）可从 CLI、<code>/v1/videos/generations</code> 或 Web UI 输出 H.264 MP4。把 <code>--model</code> 指向<em>步数蒸馏</em>权重（Turbo / Lightning / FastWan），管线会自动识别，把去噪循环从 100 次 DiT 前向降到 4 次。→ <a href="models-video_zh-cn.html#video">视频生成</a></li>
     <li><strong>文本扩散生成</strong> —— DiffusionGemma 使用迭代式 EntropyBound 去噪采样器，而非自回归解码。→ <a href="advanced_zh-cn.html#diffusion">DiffusionGemma</a></li>
@@ -66,7 +68,7 @@ <h2 id="performance">性能与扩展</h2>
     <li><strong>GPU 加速</strong> —— GGML Metal（macOS）、GGML CUDA（Windows/Linux + NVIDIA）、GGML Vulkan（Windows/Linux + AMD/Intel/NVIDIA）、直接 CUDA/cuBLAS 后端，以及面向 Apple Silicon 的 MLX 后端 —— 均带 CPU 回退。→ <a href="backends_zh-cn.html">后端</a></li>
     <li><strong>连续批处理与分页 KV 缓存</strong> —— 块分页 KV 池、块哈希前缀共享、可在批中接纳 / 抢占序列的迭代级调度器、可选 SSD 后备层，以及原生融合分页注意力内核。→ <a href="advanced_zh-cn.html#continuous-batching">深入了解</a></li>
     <li><strong>批量 / 并行推理</strong> —— 将 N 个序列打包进一次前向，配以分页 K/V 散写（Mistral 3、Gemma 4、GPT OSS、Qwen 3 / 3.5 / 3.6、Nemotron-H）。</li>
-    <li><strong>张量并行与分布式推理</strong> —— 用 <code>--tp N</code>（CLI 与服务端均支持）把一个模型切分到 N 张 GPU 上（Megatron-LM 列/行并行，归一化层 / 词嵌入 / LM head 复制），Direct <code>cuda</code> 后端与 GGML CUDA / Vulkan 后端都可运行，并通过点对点 TCP 网格（<code>--tp-node-id</code> / <code>--tp-peers</code>）把并行组扩展到多台机器。分层 AllReduce 让每次集合通信只有 <code>1/tp_local</code> 的数据上网；MoE 专家切分与专家并行、GatedDeltaNet 按 rank 的 V-head 归属与 Mamba2 复制覆盖了各类异构层。融合的按 rank block 计算图让 Gemma 4 上 <code>--tp 2</code> 的 decode 超过单卡（51.7 对 37.3 tok/s），也让单卡装不下的模型得以运行。不过 TP 并不是模型用上多张 GPU 的唯一途径：DeepSeek V4 与 GLM 5.x <strong>不加任何开关就会按层切分到所有可见 GPU</strong>，<code>--tp</code> 在 GLM 5.x 上会把这种切分换成层内部的 Megatron 切分，在 DeepSeek V4 上则只是限制按层切分使用几张卡（等同 <code>TS_DSV4_NGPU</code>）；其他所有架构在不加 <code>--tp</code> 时只用一张 GPU（见<a href="distributed_zh-cn.html#vs-layer-split">TP 与按层切分</a>）。→ <a href="distributed_zh-cn.html">多 GPU 与多节点</a></li>
+    <li><strong>张量并行与分布式推理</strong> —— 用 <code>--tp N</code>（CLI 与服务端均支持）把一个模型切分到 N 张 GPU 上（Megatron-LM 列/行并行，归一化层 / 词嵌入 / LM head 复制），Direct <code>cuda</code> 后端与 GGML CUDA / Vulkan 后端都可运行，并通过点对点 TCP 网格（<code>--tp-node-id</code> / <code>--tp-peers</code>）把并行组扩展到多台机器。分层 AllReduce 让每次集合通信只有 <code>1/tp_local</code> 的数据上网；MoE 专家切分与专家并行、GatedDeltaNet 按 rank 的 V-head 归属与 Mamba2 复制覆盖了各类异构层。融合的按 rank block 计算图让 Gemma 4 上 <code>--tp 2</code> 的 decode 超过单卡（51.7 对 37.3 tok/s），也让单卡装不下的模型得以运行。不过 TP 并不是模型用上多张 GPU 的唯一途径：DeepSeek V4 与 GLM 5.x <strong>不加任何开关就会按层切分到所有可见 GPU</strong>，<code>--tp</code> 在 GLM-5.2 上会把这种切分换成层内部的 Megatron 切分（GLM-5.3-Flash 则拒绝 <code>--tp</code>，继续用分层切分），在 DeepSeek V4 上则只是限制按层切分使用几张卡（等同 <code>TS_DSV4_NGPU</code>）。在 <strong>Qwen 3.8 Flash Next</strong> 上，<code>--tp N</code> 同样是分层切分 —— 每张 GPU 拿一段连续的完整层，不切分任何权重，这也是 llama.cpp 为这个架构提供的唯一多 GPU 模式。它是容量特性而非速度特性：在 2× A100-80GB 上用 Qwen3.8-Flash-Next-UD-Q2_K_XL（73.4 GiB）实测，双卡贪心输出与单卡逐字节相同、吞吐也没有变化（prefill 约 1520–1550 t/s，decode 约 56 t/s），换来的只是权重从全部压在一张卡上变成 24.2 + 26.2 GB 分放。启动时会打印实际使用的模式与每张 GPU 的层数 / 字节划分，<code>TS_Q4E_LAYER_SPLIT=20,28</code> 可覆盖自动均衡。其他所有架构在不加 <code>--tp</code> 时只用一张 GPU；两种模式都不支持的架构现在会在 stderr 上明说，而不是默默让其余 GPU 闲置（见<a href="distributed_zh-cn.html#vs-layer-split">TP 与按层切分</a>）。→ <a href="distributed_zh-cn.html">多 GPU 与多节点</a></li>
     <li><strong>MoE CPU 卸载</strong> —— <code>--cpu-moe</code> / <code>--n-cpu-moe N</code> 把前 N 层的路由专家留在系统内存里，直接从 GGUF 映射取用、不做私有拷贝，并且能与 <code>--tp</code> 组合。它是让装不下的权重能跑起来的手段，而不是提速开关：GLM 5.2 在 <code>--n-cpu-moe 30</code> 下实测 pp2048 94.7 / tg64 16.4，全部常驻时为 915.9 / 43.9，但腾出的显存把能定下的上下文从 342,272 抬到 646,400 token。→ <a href="advanced_zh-cn.html#memory">内存</a></li>
     <li><strong>MTP / NextN 推测解码</strong> —— 多 token 预测草稿头加速单序列解码；因请求自身的采样器同时驱动草稿与验证，故无损。→ <a href="advanced_zh-cn.html#mtp">推测解码</a></li>
     <li><strong>DSpark 块级投机解码</strong> —— DeepSeek V4 的草稿器每步提议一整<em>块</em> token（Markov 头让块内每个位置以前一个 token 为条件，置信度头决定起草多远），主干用一次批量前向验证整块。以独立 GGUF 通过 <code>--draft-model</code> 加载；4×A40 实测 decode 提速 1.3–1.4×，多轮对话最高 2.0×，贪心输出与基线逐字节一致。→ <a href="advanced_zh-cn.html#dspark">DSpark</a></li>
diff --git a/website/index.html b/website/index.html
index a514ae05..c9a83c33 100644
--- a/website/index.html
+++ b/website/index.html
@@ -68,7 +68,7 @@ <h2 id="explore">Explore the wiki</h2>
     <a class="card" href="http-api.html"><div class="ic">🔌</div><h3>HTTP API</h3><p>Call it from curl, Python, or any Ollama/OpenAI client.</p></a>
     <a class="card" href="code-api.html"><div class="ic">🧩</div><h3>C# Library</h3><p>Embed the engine directly in your .NET application.</p></a>
     <a class="card" href="api-reference.html"><div class="ic">📚</div><h3>API Reference</h3><p>Searchable tables of flags, env vars, endpoints, and types.</p></a>
-    <a class="card" href="models.html"><div class="ic">🧠</div><h3>Models</h3><p>All fourteen architectures, downloads, multimodal, reasoning — and which checkpoint to pick when you want it fast.</p></a>
+    <a class="card" href="models.html"><div class="ic">🧠</div><h3>Models</h3><p>All fifteen architectures, downloads, multimodal, reasoning — and which checkpoint to pick when you want it fast.</p></a>
     <a class="card" href="distributed.html"><div class="ic">🔗</div><h3>Multi-GPU &amp; Multi-Node</h3><p>Tensor parallelism across GPUs — CUDA and GGML — and across machines.</p></a>
     <a class="card" href="glossary.html"><div class="ic">📖</div><h3>Glossary &amp; FAQ</h3><p>New to LLMs? Plain-language definitions and common questions.</p></a>
   </div>
@@ -110,11 +110,11 @@ <h2 id="why">Why TensorSharp?</h2>
     <div class="card"><div class="ic">💸</div><h3>No per-token bill</h3><p>Run as much as your hardware allows — predictable cost, no metered API.</p></div>
     <div class="card"><div class="ic">🔁</div><h3>Drop-in compatible</h3><p>Speaks the Ollama and OpenAI wire formats, so existing tools and SDKs just work.</p></div>
     <div class="card"><div class="ic">🖥️</div><h3>Runs anywhere</h3><p>NVIDIA (CUDA), AMD / Intel / NVIDIA (Vulkan), Apple Silicon (Metal/MLX), or pure CPU — with automatic fallbacks.</p></div>
-    <a class="card" href="models.html#table"><div class="ic">🧠</div><h3>Modern model support</h3><p>Eleven text families — DeepSeek V4 Flash, GLM 5.2, Gemma 3 / 4, DiffusionGemma, Qwen 3, Qwen 3.5 / 3.6, GPT-OSS, Nemotron-H, Mistral 3, Muse-Glimmer — plus vision, audio, PDF documents, reasoning &amp; tools. Three more generate media instead of text: images (Qwen-Image-Edit), video with a native stereo soundtrack (MiniMax-H3), and video alone (Wan 2.1 / 2.2).</p></a>
+    <a class="card" href="models.html#table"><div class="ic">🧠</div><h3>Modern model support</h3><p>Twelve text families — DeepSeek V4 Flash, GLM 5.2 &amp; GLM-5.3-Flash, Gemma 3 / 4, DiffusionGemma, Qwen 3, Qwen 3.5 / 3.6, Qwen 3.8 Flash Next, GPT-OSS, Nemotron-H, Mistral 3, Muse-Glimmer — plus vision, audio, PDF documents, reasoning &amp; tools. Three more generate media instead of text: images (Qwen-Image-Edit), video with a native stereo soundtrack (MiniMax-H3), and video alone (Wan 2.1 / 2.2).</p></a>
     <a class="card" href="models-video.html#minimax-h3"><div class="ic">🎬</div><h3>Images and video out, too</h3><p>MiniMax-H3 generates video <strong>and native 32 kHz stereo audio together in one packed latent</strong>; Qwen-Image-Edit rewrites an image from a prompt; Wan 2.1 / 2.2 generate H.264 video alone — same engine, same GGUF plumbing. H3 ships CFG-distilled, so 4–8 steps at <code>--cfg 1.0</code> is the operating point; pick a <strong>step-distilled Wan checkpoint</strong> and the same 5-second 720p clip takes 17 min instead of 3½ hours.</p></a>
     <div class="card"><div class="ic">⚙️</div><h3>Built in .NET</h3><p>A native C# engine you can embed in your apps, not just a black-box binary.</p></div>
-    <a class="card" href="distributed.html"><div class="ic">🔗</div><h3>Scales past one GPU</h3><p>Tensor parallelism splits a model across several GPUs with <code>--tp N</code> — on the direct CUDA backend and on GGML CUDA / Vulkan — and across machines over a peer-to-peer TCP mesh when one host is not enough.</p></a>
-    <a class="card" href="benchmarks.html#head-to-head"><div class="ic">🏁</div><h3>Benchmarked vs llama.cpp</h3><p>On identical GGUF files and the same GPU it trades wins with the C++ engine — in the current CUDA + Vulkan comparison run (reproducible via <code>benchmarks/engine_comparison</code>): Gemma 4 E4B and the 2-bit Qwen 3.6 35B-A3B MoE prefill 1.28× faster on CUDA with first tokens 1.27× sooner, multi-turn prompts prefill faster on every model (up to 1.49×), and Gemma 4 12B decodes 1.21× faster on Vulkan.</p></a>
+    <a class="card" href="distributed.html"><div class="ic">🔗</div><h3>Scales past one GPU</h3><p>Tensor parallelism splits every layer's weights across several GPUs with <code>--tp N</code> — on the direct CUDA backend and on GGML CUDA / Vulkan — and across machines over a peer-to-peer TCP mesh when one host is not enough. The whole-model executors use a <strong>layer split</strong> instead — whole layers per GPU, no collectives: DeepSeek V4 Flash and GLM 5.x with no flag at all, Qwen 3.8 Flash Next under <code>--tp N</code>. That one is capacity, not speed.</p></a>
+    <a class="card" href="benchmarks.html#head-to-head"><div class="ic">🏁</div><h3>Benchmarked vs llama.cpp</h3><p>GLM-5.3-Flash decodes at <strong>2.0× llama.cpp</strong> — 73.5 vs 36.6 tok/s at tg64 on 2× RTX PRO 6000 Blackwell, UD-Q2_K_XL, both engines layer-split back to back — with prefill within a few percent either way. On identical GGUF files and the same GPU it trades wins with the C++ engine — in the current CUDA + Vulkan comparison run (reproducible via <code>benchmarks/engine_comparison</code>): Gemma 4 E4B and the 2-bit Qwen 3.6 35B-A3B MoE prefill 1.28× faster on CUDA with first tokens 1.27× sooner, multi-turn prompts prefill faster on every model (up to 1.49×), and Gemma 4 12B decodes 1.21× faster on Vulkan.</p></a>
   </div>
 
   <h2 id="audience">Who is this for?</h2>
diff --git a/website/index_zh-cn.html b/website/index_zh-cn.html
index 52863200..cbc4ecf9 100644
--- a/website/index_zh-cn.html
+++ b/website/index_zh-cn.html
@@ -68,7 +68,7 @@ <h2 id="explore">浏览维基</h2>
     <a class="card" href="http-api_zh-cn.html"><div class="ic">🔌</div><h3>HTTP API</h3><p>用 curl、Python 或任意 Ollama / OpenAI 客户端调用它。</p></a>
     <a class="card" href="code-api_zh-cn.html"><div class="ic">🧩</div><h3>C# 库</h3><p>将引擎直接嵌入你的 .NET 应用。</p></a>
     <a class="card" href="api-reference_zh-cn.html"><div class="ic">📚</div><h3>API 参考</h3><p>可搜索的参数、环境变量、端点与类型表。</p></a>
-    <a class="card" href="models_zh-cn.html"><div class="ic">🧠</div><h3>模型</h3><p>全部十四种架构、下载方式、多模态与推理能力 —— 以及想要更快时该选哪个权重。</p></a>
+    <a class="card" href="models_zh-cn.html"><div class="ic">🧠</div><h3>模型</h3><p>全部十五种架构、下载方式、多模态与推理能力 —— 以及想要更快时该选哪个权重。</p></a>
     <a class="card" href="distributed_zh-cn.html"><div class="ic">🔗</div><h3>多 GPU 与多节点</h3><p>跨 GPU（CUDA 与 GGML）、跨机器的张量并行。</p></a>
     <a class="card" href="glossary_zh-cn.html"><div class="ic">📖</div><h3>术语表与 FAQ</h3><p>初识大模型？这里有通俗的定义和常见问题。</p></a>
   </div>
@@ -110,11 +110,11 @@ <h2 id="why">为什么选择 TensorSharp？</h2>
     <div class="card"><div class="ic">💸</div><h3>没有按 token 账单</h3><p>硬件允许多少就跑多少 —— 成本可预测，没有计量 API。</p></div>
     <div class="card"><div class="ic">🔁</div><h3>即插即用兼容</h3><p>支持 Ollama 与 OpenAI 协议，现有工具与 SDK 直接可用。</p></div>
     <div class="card"><div class="ic">🖥️</div><h3>随处可运行</h3><p>NVIDIA (CUDA)、AMD / Intel / NVIDIA (Vulkan)、Apple Silicon (Metal/MLX) 或纯 CPU —— 均带自动回退。</p></div>
-    <a class="card" href="models_zh-cn.html#table"><div class="ic">🧠</div><h3>现代模型支持</h3><p>十一个文本家族 —— DeepSeek V4 Flash、GLM 5.2、Gemma 3 / 4、DiffusionGemma、Qwen 3、Qwen 3.5 / 3.6、GPT-OSS、Nemotron-H、Mistral 3、Muse-Glimmer —— 外加视觉、音频、PDF 文档、推理与工具调用。另有三个家族产出媒体而非文本：图像（Qwen-Image-Edit）、带原生立体声音轨的视频（MiniMax-H3），以及只有画面的视频（Wan 2.1 / 2.2）。</p></a>
+    <a class="card" href="models_zh-cn.html#table"><div class="ic">🧠</div><h3>现代模型支持</h3><p>十二个文本家族 —— DeepSeek V4 Flash、GLM 5.2 与 GLM-5.3-Flash、Gemma 3 / 4、DiffusionGemma、Qwen 3、Qwen 3.5 / 3.6、Qwen 3.8 Flash Next、GPT-OSS、Nemotron-H、Mistral 3、Muse-Glimmer —— 外加视觉、音频、PDF 文档、推理与工具调用。另有三个家族产出媒体而非文本：图像（Qwen-Image-Edit）、带原生立体声音轨的视频（MiniMax-H3），以及只有画面的视频（Wan 2.1 / 2.2）。</p></a>
     <a class="card" href="models-video_zh-cn.html#minimax-h3"><div class="ic">🎬</div><h3>还能产出图像与视频</h3><p>MiniMax-H3 在<strong>一份打包潜变量里同时生成视频与原生 32 kHz 立体声音频</strong>；Qwen-Image-Edit 按提示词改写图像；Wan 2.1 / 2.2 只生成 H.264 画面——同一个引擎、同一套 GGUF 管线。H3 出厂即经 CFG 蒸馏，工作点是 <code>--cfg 1.0</code> 下的 4–8 步；改用<strong>步数蒸馏（step-distilled）的 Wan 权重</strong>，同一段 5 秒 720p 视频只需 17 分钟，而不是 3 个半小时。</p></a>
     <div class="card"><div class="ic">⚙️</div><h3>用 .NET 构建</h3><p>原生 C# 引擎，可嵌入你的应用，而不只是一个黑盒二进制。</p></div>
-    <a class="card" href="distributed_zh-cn.html"><div class="ic">🔗</div><h3>不止一张 GPU</h3><p>张量并行用 <code>--tp N</code> 把模型切分到多张 GPU 上（Direct CUDA 与 GGML CUDA / Vulkan 均可）；一台机器不够时，还可通过点对点 TCP 网格跨机器扩展。</p></a>
-    <a class="card" href="benchmarks_zh-cn.html#head-to-head"><div class="ic">🏁</div><h3>对比 llama.cpp 的基准</h3><p>在相同 GGUF 文件与相同 GPU 上与 C++ 引擎互有胜负 —— 在当前的 CUDA + Vulkan 对比运行中（可通过 <code>benchmarks/engine_comparison</code> 复现）：Gemma 4 E4B 与 2-bit 量化的 Qwen 3.6 35B-A3B MoE 在 CUDA 上 prefill 快 1.28×、首 token 早 1.27×，多轮提示的 prefill 在每个模型上都更快（最高 1.49×），Gemma 4 12B 在 Vulkan 上 decode 快 1.21×。</p></a>
+    <a class="card" href="distributed_zh-cn.html"><div class="ic">🔗</div><h3>不止一张 GPU</h3><p>张量并行用 <code>--tp N</code> 把每层的权重切分到多张 GPU 上（Direct CUDA 与 GGML CUDA / Vulkan 均可）；一台机器不够时，还可通过点对点 TCP 网格跨机器扩展。整模型执行器用的则是<strong>分层切分（layer split）</strong> —— 每张 GPU 拿一段完整的层、没有集合通信：DeepSeek V4 Flash 与 GLM 5.x 不需任何参数，Qwen 3.8 Flash Next 则在 <code>--tp N</code> 下走这条路。后者买的是容量，不是速度。</p></a>
+    <a class="card" href="benchmarks_zh-cn.html#head-to-head"><div class="ic">🏁</div><h3>对比 llama.cpp 的基准</h3><p>GLM-5.3-Flash 的 decode 达到 <strong>llama.cpp 的 2.0×</strong> —— 在 2× RTX PRO 6000 Blackwell 上以 UD-Q2_K_XL 分层切分背靠背实测，tg64 为 73.5 vs 36.6 tok/s，prefill 两边相差均在几个百分点以内。在相同 GGUF 文件与相同 GPU 上与 C++ 引擎互有胜负 —— 在当前的 CUDA + Vulkan 对比运行中（可通过 <code>benchmarks/engine_comparison</code> 复现）：Gemma 4 E4B 与 2-bit 量化的 Qwen 3.6 35B-A3B MoE 在 CUDA 上 prefill 快 1.28×、首 token 早 1.27×，多轮提示的 prefill 在每个模型上都更快（最高 1.49×），Gemma 4 12B 在 Vulkan 上 decode 快 1.21×。</p></a>
   </div>
 
   <h2 id="audience">这是为谁准备的？</h2>
diff --git a/website/models-downloads.html b/website/models-downloads.html
index 30719fa2..084dfb43 100644
--- a/website/models-downloads.html
+++ b/website/models-downloads.html
@@ -26,6 +26,7 @@ <h2 id="downloads">Model downloads (GGUF)</h2>
       <tr><td>DeepSeek V4</td><td>DeepSeek-V4-Flash-0731 (284B MoE)</td><td><a href="https://huggingface.co/unsloth/DeepSeek-V4-Flash-0731-GGUF" target="_blank" rel="noopener">unsloth/DeepSeek-V4-Flash-0731-GGUF</a></td><td>One subdirectory per quant (<code>UD-Q8_K_XL/</code>, <code>UD-IQ4_XS/</code>, <code>UD-IQ1_S/</code>, …), each a multi-shard set — point <code>--model</code> at the <code>-00001-of-</code> shard. Text only; weights layer-split across every visible GPU</td></tr>
       <tr><td>DeepSeek V4</td><td>DSpark speculative drafter (optional)</td><td><a href="https://huggingface.co/bleysg/DeepSeek-V4-Flash-DSpark-drafter-GGUF" target="_blank" rel="noopener">bleysg/DeepSeek-V4-Flash-DSpark-drafter-GGUF</a></td><td><code>DSpark-drafter-Q2K-Q8-0731.gguf</code> (7.0 GB) for the 0731 release, loaded with <code>--draft-model</code> for ~1.3–1.4× decode. Two other publishers' builds (5.6 GB / 10.9 GB) also load as-is — see <code>MODEL_DOWNLOADS.md</code>. Drafters for other architectures are a different design and are <strong>not</strong> supported</td></tr>
       <tr><td>GLM 5.x</td><td>GLM-5.2 (744B-A40B MoE)</td><td><a href="https://huggingface.co/unsloth/GLM-5.2-GGUF" target="_blank" rel="noopener">unsloth/GLM-5.2-GGUF</a></td><td>One subdirectory per quant (<code>UD-Q4_K_XL/</code>, <code>UD-IQ2_XXS/</code>, …), each a multi-shard set — point <code>--model</code> at the <code>-00001-of-</code> shard and <code>GgufFile</code> reads the rest. Text only; no projector or drafter</td></tr>
+      <tr><td>GLM 5.x</td><td>GLM-5.3-Flash (320B MoE, <code>glm5next</code>)</td><td><a href="https://huggingface.co/unsloth/GLM-5.3-Flash-GGUF" target="_blank" rel="noopener">unsloth/GLM-5.3-Flash-GGUF</a></td><td>One subdirectory per quant, each a multi-shard set — point <code>--model</code> at the <code>-00001-of-</code> shard. mmproj: <code>mmproj-BF16.gguf</code> (same repo, the GLM-OCR ViT) for image input. Loads through the same native executor as GLM-5.2; <code>--tp</code> is refused, so the multi-GPU mode is the layer split across every visible GPU</td></tr>
       <tr><td>Gemma 4</td><td>gemma-4-E4B-it</td><td><a href="https://huggingface.co/ggml-org/gemma-4-E4B-it-GGUF" target="_blank" rel="noopener">ggml-org/gemma-4-E4B-it-GGUF</a></td><td>Recommended public artifact for the verified E4B Q8_0 native-GGML tier: <code>gemma-4-E4B-it-Q8_0.gguf</code>. Modalities: <code>mmproj-gemma-4-E4B-it-Q8_0.gguf</code> (same repo). MTP draft: <a href="https://huggingface.co/AtomicChat/gemma-4-E4B-it-assistant-GGUF" target="_blank" rel="noopener">AtomicChat/gemma-4-E4B-it-assistant-GGUF</a></td></tr>
       <tr><td>Gemma 4</td><td>gemma-4-12B-it (QAT)</td><td><a href="https://huggingface.co/unsloth/gemma-4-12B-it-qat-GGUF" target="_blank" rel="noopener">unsloth/gemma-4-12B-it-qat-GGUF</a></td><td>mmproj: <code>mmproj-BF16.gguf</code>; MTP draft: <code>mtp-gemma-4-12B-it.gguf</code> (both in the same repo)</td></tr>
       <tr><td>Gemma 4</td><td>gemma-4-31B-it</td><td><a href="https://huggingface.co/ggml-org/gemma-4-31B-it-GGUF" target="_blank" rel="noopener">ggml-org/gemma-4-31B-it-GGUF</a></td><td>mmproj: <code>mmproj-gemma-4-31B-it-Q8_0.gguf</code> (same repo)</td></tr>
@@ -37,6 +38,7 @@ <h2 id="downloads">Model downloads (GGUF)</h2>
       <tr><td>Qwen 3.5 / 3.6</td><td>Qwen3.5-9B</td><td><a href="https://huggingface.co/unsloth/Qwen3.5-9B-GGUF" target="_blank" rel="noopener">unsloth/Qwen3.5-9B-GGUF</a></td><td>mmproj: <code>mmproj-F16.gguf</code> (same repo)</td></tr>
       <tr><td>Qwen 3.5 / 3.6</td><td>Qwen3.5-35B-A3B (MoE)</td><td><a href="https://huggingface.co/ggml-org/Qwen3.5-35B-A3B-GGUF" target="_blank" rel="noopener">ggml-org/Qwen3.5-35B-A3B-GGUF</a></td><td>mmproj: <code>mmproj-Qwen3.5-35B-A3B-Q8_0.gguf</code> (same repo)</td></tr>
       <tr><td>Qwen 3.5 / 3.6</td><td>Qwen3.6-35B-A3B (MoE, NextN MTP)</td><td><a href="https://huggingface.co/unsloth/Qwen3.6-35B-A3B-MTP-GGUF" target="_blank" rel="noopener">unsloth/Qwen3.6-35B-A3B-MTP-GGUF</a></td><td>mmproj: <code>mmproj-F16.gguf</code> (same repo). These GGUFs retain the embedded NextN block for <code>--mtp-spec</code>; the base-repo GGUFs (<a href="https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF" target="_blank" rel="noopener">unsloth/Qwen3.6-35B-A3B-GGUF</a>) strip it and silently fall back to standard decode</td></tr>
+      <tr><td>Qwen 3.8 Flash Next</td><td>Qwen3.8-Flash-Next (hybrid MoE, <code>qwen4exp</code>)</td><td><a href="https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF" target="_blank" rel="noopener">unsloth/Qwen3.8-Flash-Next-GGUF</a></td><td>One subdirectory per quant, each a multi-shard set — point <code>--model</code> at the <code>-00001-of-</code> shard. mmproj: <code>mmproj-BF16.gguf</code> (same repo) for image input. <code>--tp N</code> here is a layer split across N GPUs, not tensor parallelism</td></tr>
       <tr><td>GPT OSS</td><td>gpt-oss-20b (MoE)</td><td><a href="https://huggingface.co/ggml-org/gpt-oss-20b-GGUF" target="_blank" rel="noopener">ggml-org/gpt-oss-20b-GGUF</a></td><td>— (text only)</td></tr>
       <tr><td>Nemotron-H</td><td>Nemotron-H-8B-Reasoning-128K</td><td><a href="https://huggingface.co/bartowski/nvidia_Nemotron-H-8B-Reasoning-128K-GGUF" target="_blank" rel="noopener">bartowski/nvidia_Nemotron-H-8B-…</a></td><td>— (text only)</td></tr>
       <tr><td>Nemotron-H</td><td>Nemotron-H-47B-Reasoning-128K</td><td><a href="https://huggingface.co/bartowski/nvidia_Nemotron-H-47B-Reasoning-128K-GGUF" target="_blank" rel="noopener">bartowski/nvidia_Nemotron-H-47B-…</a></td><td>— (text only)</td></tr>
diff --git a/website/models-downloads_zh-cn.html b/website/models-downloads_zh-cn.html
index 015ce462..216df4a8 100644
--- a/website/models-downloads_zh-cn.html
+++ b/website/models-downloads_zh-cn.html
@@ -26,6 +26,7 @@ <h2 id="downloads">模型下载（GGUF）</h2>
       <tr><td>DeepSeek V4</td><td>DeepSeek-V4-Flash-0731（284B MoE）</td><td><a href="https://huggingface.co/unsloth/DeepSeek-V4-Flash-0731-GGUF" target="_blank" rel="noopener">unsloth/DeepSeek-V4-Flash-0731-GGUF</a>；每个量化档一个子目录（<code>UD-Q8_K_XL/</code>、<code>UD-IQ4_XS/</code>、<code>UD-IQ1_S/</code> …），均为多分片——<code>--model</code> 指向 <code>-00001-of-</code> 那一片。仅文本；权重按层切分到所有可见 GPU</td></tr>
       <tr><td>DeepSeek V4</td><td>DSpark 投机草稿器（可选）</td><td><a href="https://huggingface.co/bleysg/DeepSeek-V4-Flash-DSpark-drafter-GGUF" target="_blank" rel="noopener">bleysg/DeepSeek-V4-Flash-DSpark-drafter-GGUF</a> 的 <code>DSpark-drafter-Q2K-Q8-0731.gguf</code>（7.0 GB，对应 0731 版本），用 <code>--draft-model</code> 加载可获得约 1.3–1.4× 的 decode 提速。另有两家发布的构建（5.6 GB / 10.9 GB）也可直接加载，见 <code>MODEL_DOWNLOADS_zh-cn.md</code>。其他架构的草稿器属于不同设计，<strong>暂不支持</strong></td></tr>
       <tr><td>GLM 5.x</td><td>GLM-5.2（744B-A40B MoE）</td><td><a href="https://huggingface.co/unsloth/GLM-5.2-GGUF" target="_blank" rel="noopener">unsloth/GLM-5.2-GGUF</a>；每个量化档一个子目录（<code>UD-Q4_K_XL/</code>、<code>UD-IQ2_XXS/</code> 等），每档都是多分片集合——<code>--model</code> 指向 <code>-00001-of-</code> 那一片，其余由 <code>GgufFile</code> 自行读取。纯文本；无投影器、无草稿器</td></tr>
+      <tr><td>GLM 5.x</td><td>GLM-5.3-Flash（320B MoE，<code>glm5next</code>）</td><td><a href="https://huggingface.co/unsloth/GLM-5.3-Flash-GGUF" target="_blank" rel="noopener">unsloth/GLM-5.3-Flash-GGUF</a>；每个量化档一个子目录，每档都是多分片集合——<code>--model</code> 指向 <code>-00001-of-</code> 那一片。图像输入需要同仓库的 <code>mmproj-BF16.gguf</code>（GLM-OCR ViT）。与 GLM-5.2 走同一个原生执行器；<code>--tp</code> 会被拒绝，多卡模式是分到所有可见 GPU 的按层切分</td></tr>
       <tr><td>Gemma 4</td><td>gemma-4-E4B-it</td><td><a href="https://huggingface.co/ggml-org/gemma-4-E4B-it-GGUF" target="_blank" rel="noopener">ggml-org/gemma-4-E4B-it-GGUF</a>；已验证 E4B Q8_0 原生 GGML 规格推荐使用公开文件 <code>gemma-4-E4B-it-Q8_0.gguf</code>；多模态投影器为同仓库的 <code>mmproj-gemma-4-E4B-it-Q8_0.gguf</code></td></tr>
       <tr><td>Gemma 4</td><td>gemma-4-12B-it（QAT）</td><td><a href="https://huggingface.co/unsloth/gemma-4-12B-it-qat-GGUF" target="_blank" rel="noopener">unsloth/gemma-4-12B-it-qat-GGUF</a>；同仓库 <code>mmproj-BF16.gguf</code> 与 <code>mtp-gemma-4-12B-it.gguf</code></td></tr>
       <tr><td>Gemma 4</td><td>gemma-4-31B-it</td><td><a href="https://huggingface.co/ggml-org/gemma-4-31B-it-GGUF" target="_blank" rel="noopener">ggml-org/gemma-4-31B-it-GGUF</a></td></tr>
@@ -37,6 +38,7 @@ <h2 id="downloads">模型下载（GGUF）</h2>
       <tr><td>Qwen 3.5 / 3.6</td><td>Qwen3.5-9B</td><td><a href="https://huggingface.co/unsloth/Qwen3.5-9B-GGUF" target="_blank" rel="noopener">unsloth/Qwen3.5-9B-GGUF</a></td></tr>
       <tr><td>Qwen 3.5 / 3.6</td><td>Qwen3.5-35B-A3B (MoE)</td><td><a href="https://huggingface.co/ggml-org/Qwen3.5-35B-A3B-GGUF" target="_blank" rel="noopener">ggml-org/Qwen3.5-35B-A3B-GGUF</a></td></tr>
       <tr><td>Qwen 3.6</td><td>Qwen3.6-35B-A3B NextN MTP</td><td><a href="https://huggingface.co/unsloth/Qwen3.6-35B-A3B-MTP-GGUF" target="_blank" rel="noopener">unsloth/Qwen3.6-35B-A3B-MTP-GGUF</a>；<code>UD-Q4_K_M</code> 约 21.11 GiB。基础仓库 GGUF 会剥离 NextN 块</td></tr>
+      <tr><td>Qwen 3.8 Flash Next</td><td>Qwen3.8-Flash-Next（混合 MoE，<code>qwen4exp</code>）</td><td><a href="https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF" target="_blank" rel="noopener">unsloth/Qwen3.8-Flash-Next-GGUF</a>；每个量化档一个子目录，每档都是多分片集合——<code>--model</code> 指向 <code>-00001-of-</code> 那一片。图像输入需要同仓库的 <code>mmproj-BF16.gguf</code>。这里的 <code>--tp N</code> 是把层切分到 N 张 GPU，而不是张量并行</td></tr>
       <tr><td>GPT OSS</td><td>gpt-oss-20b (MoE)</td><td><a href="https://huggingface.co/ggml-org/gpt-oss-20b-GGUF" target="_blank" rel="noopener">ggml-org/gpt-oss-20b-GGUF</a></td></tr>
       <tr><td>Nemotron-H</td><td>Nemotron-H-8B-Reasoning-128K</td><td><a href="https://huggingface.co/bartowski/nvidia_Nemotron-H-8B-Reasoning-128K-GGUF" target="_blank" rel="noopener">bartowski/nvidia_Nemotron-H-8B-…</a></td></tr>
       <tr><td>Nemotron-H</td><td>Nemotron-H-47B-Reasoning-128K</td><td><a href="https://huggingface.co/bartowski/nvidia_Nemotron-H-47B-Reasoning-128K-GGUF" target="_blank" rel="noopener">bartowski/nvidia_Nemotron-H-47B-…</a></td></tr>
diff --git a/website/models-text.html b/website/models-text.html
index fc5dc285..aa755acc 100644
--- a/website/models-text.html
+++ b/website/models-text.html
@@ -4,7 +4,7 @@
   <meta charset="utf-8">
   <meta name="viewport" content="width=device-width, initial-scale=1">
   <title>Text &amp; LLM Models — TensorSharp Wiki</title>
-  <meta name="description" content="Download-and-run recipes for all eleven text architectures, plus multimodal input, thinking mode and tool calling.">
+  <meta name="description" content="Download-and-run recipes for all twelve text architectures, plus multimodal input, thinking mode and tool calling.">
   <link rel="icon" href="assets/logo.svg" type="image/svg+xml">
   <link rel="stylesheet" href="assets/style.css">
   <script>try{var t=localStorage.getItem('ts-theme')||(matchMedia('(prefers-color-scheme: dark)').matches?'dark':'light');document.documentElement.classList.toggle('dark',t==='dark');}catch(e){}</script>
@@ -84,6 +84,36 @@ <h3 id="glm">GLM 5.x (744B-A40B MoE, text, thinking, tools)</h3>
   <p>The advertised 1M-token context is a <em>ceiling</em>, not a promise: 1M tokens of MLA cache is about 93 GiB, so once the weights land the loader asks the devices how much VRAM is actually free, sizes the context to what fits alongside one full prefill graph, and logs what it picked. On 3× RTX PRO 6000 that is 342,272 tokens on the plain layer split, 646,400 with <code>--n-cpu-moe 30</code>, and 91,136 under <code>--tp 3</code> (every rank holds a full-length cache). Set <code>MAX_CONTEXT</code> to make a specific length a hard requirement instead — it is honoured if it fits and refused with the numbers if it does not.</p>
   <p><code>--tp N</code> works here too, but on PCIe-attached cards it is a capacity feature rather than a speed one: the two all-reduces every one of the 78 layers needs cost more bus time than the split saves, so <code>--tp 3</code> measures pp2048 505.6 / tg64 17.6 tok/s against 915.9 / 43.9 on the layer split. Concurrency is served by native per-sequence slots (each request owns its MLA and indexer caches), and <code>TS_BATCHED_FUSED_DECODE=1</code> opts into a batched fused decode worth 1.81× aggregate at 4 concurrent requests. See <a href="distributed.html#architectures">Tensor parallelism</a> and <a href="advanced.html#continuous-batching">Continuous batching</a>.</p>
 
+  <h3 id="glm53">GLM-5.3-Flash (320B MoE, text + image, thinking, tools)</h3>
+  <p>The hybrid successor loads through the <em>same</em> native whole-model ggml executor and the same model class as GLM-5.2 — its GGUF architecture id is <code>glm5next</code> — with four architectural changes layered on top. 320B parameters, 288 routed experts at top-8 plus one shared, 46 blocks = 45 trunk + 1 NextN. <strong>KDA linear attention</strong> runs on 34 of the 45 trunk layers; the other 11 are <strong>NoPE MLA + DSA</strong> (no rope anywhere in the text tower, the 512-wide latent <em>is</em> the cache row); the indexer is <strong>pooled</strong> — 4-cell pools, top-k 2048 taken over pools and then expanded to their members; hyper-connections are Sinkhorn over ×4 streams; and every SwiGLU is clamped at limit 10. The KDA recurrent state cannot be rewound, so a cached prefix is reused only when the new prompt extends it exactly.</p>
+  <pre data-lang="bash"><code># ~101 GiB for the UD-Q2_K_XL tier — pick a different quant directory for more or less
+hf download unsloth/GLM-5.3-Flash-GGUF --include "UD-Q2_K_XL/*" --local-dir models
+hf download unsloth/GLM-5.3-Flash-GGUF mmproj-BF16.gguf --local-dir models
+
+# Layer split across every visible GPU — point --model at the FIRST shard
+# (N is however many shards the quant directory you downloaded holds)
+dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll \
+    --model models/UD-Q2_K_XL/GLM-5.3-Flash-UD-Q2_K_XL-00001-of-0000N.gguf \
+    --backend ggml_cuda --input prompt.txt --max-tokens 200
+
+# Image understanding — the GLM-OCR ViT ships as mmproj-BF16.gguf
+dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll \
+    --model models/UD-Q2_K_XL/GLM-5.3-Flash-UD-Q2_K_XL-00001-of-0000N.gguf \
+    --mmproj models/mmproj-BF16.gguf --image photo.png --input question.txt \
+    --max-tokens 300 --backend ggml_cuda
+
+# Not enough VRAM? Keep the routed experts of the first N layers in system RAM
+dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll \
+    --model models/UD-Q2_K_XL/GLM-5.3-Flash-UD-Q2_K_XL-00001-of-0000N.gguf \
+    --backend ggml_cuda --n-cpu-moe 10 --input prompt.txt --max-tokens 200
+
+# Served over HTTP
+dotnet TensorSharp.Server/bin/TensorSharp.Server.dll \
+    --model models/UD-Q2_K_XL/GLM-5.3-Flash-UD-Q2_K_XL-00001-of-0000N.gguf \
+    --mmproj models/mmproj-BF16.gguf --backend ggml_cuda</code></pre>
+  <p>What runs today: the layer split across every visible GPU (the default), <code>--cpu-moe</code> / <code>--n-cpu-moe N</code> host-resident experts, per-sequence native slots for serving, and vision — <code>--image</code>, multi-image prompts and multi-turn image sessions, through the GLM-OCR ViT. <strong>Not yet</strong>: <code>--tp</code> tensor parallelism, which is cleanly refused (use the layer split), and NextN/MTP speculation.</p>
+  <p>Measured on 2× RTX PRO 6000 Blackwell (96 GB) with GLM-5.3-Flash-UD-Q2_K_XL (101 GiB), layer split, both engines at <code>n_ubatch</code> 2048, back to back: pp2048 <strong>2070 t/s</strong> for llama.cpp against 2014 for TensorSharp, pp16384 1690 against <strong>1692</strong>, pp32768 <strong>1483</strong> against 1446, and tg64 36.6 against <strong>73.5</strong>. Prefill is within a few percent either way; decode runs at <strong>2.0× llama.cpp</strong>.</p>
+
   <h3 id="gemma3">Gemma 3 (text + image)</h3>
   <pre data-lang="bash"><code>hf download ggml-org/gemma-3-4b-it-GGUF gemma-3-4b-it-Q4_K_M.gguf --local-dir models
 hf download ggml-org/gemma-3-4b-it-GGUF mmproj-model-f16.gguf --local-dir models
@@ -109,6 +139,30 @@ <h3 id="qwen35">Qwen 3.5 / 3.6 (text + image, thinking, tools, NextN MTP on 3.6)
 
 dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model models/Qwen3.6-35B-A3B-UD-Q4_K_M.gguf --backend ggml_cuda --mtp-spec</code></pre>
 
+  <h3 id="qwen4exp">Qwen 3.8 Flash Next (hybrid MoE, text + image, thinking, tools)</h3>
+  <p>A hybrid mixture-of-experts model whose GGUF architecture id is <code>qwen4exp</code>: GatedDeltaNet recurrent layers interleaved with full-attention layers (some behind Qwen Sparse Attention's indexer), a PLE n-gram embedding block, ×4 hyper-connection streams and a 512-expert MoE. 48 layers, hidden size 2560, 24 query / 2 KV heads at head_dim 256, a 248,320-token vocabulary, 10 of the 512 experts used per token, GDN on 36 of the 48 layers and PLE on layer 1. On the GGML backends the whole token runs as (almost) <strong>one graph</strong> — embedding, in-graph PLE, all 48 layers, the final mixer and the LM head — replayed from a shape-keyed cache of captured graphs. Vision rides the Qwen3.5-VL tower with (T,H,W) IMRoPE positions, so multi-image prompts and multi-turn image sessions both work, with KV reused across turns as long as the new prompt extends the cached prefix exactly (the GDN recurrence cannot rewind). Concurrent requests are served through per-sequence state holders — each request owns its attention KV and indexer caches, its GDN and PLE state — so switching between them is a reference swap. Thinking and tool calling both work.</p>
+  <pre data-lang="bash"><code># ~73.4 GiB for the UD-Q2_K_XL tier — pick a different quant directory for more or less
+hf download unsloth/Qwen3.8-Flash-Next-GGUF --include "UD-Q2_K_XL/*" --local-dir models
+hf download unsloth/Qwen3.8-Flash-Next-GGUF mmproj-BF16.gguf --local-dir models
+
+# Text — point --model at the FIRST shard
+# (N is however many shards the quant directory you downloaded holds)
+dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll \
+    --model models/UD-Q2_K_XL/Qwen3.8-Flash-Next-UD-Q2_K_XL-00001-of-0000N.gguf \
+    --backend ggml_cuda --input prompt.txt --max-tokens 300
+
+# Image understanding
+dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll \
+    --model models/UD-Q2_K_XL/Qwen3.8-Flash-Next-UD-Q2_K_XL-00001-of-0000N.gguf \
+    --mmproj models/mmproj-BF16.gguf --image photo.png --input question.txt \
+    --max-tokens 300 --backend ggml_cuda
+
+# Two GPUs: --tp N runs a LAYER SPLIT here, not tensor parallelism
+dotnet TensorSharp.Server/bin/TensorSharp.Server.dll \
+    --model models/UD-Q2_K_XL/Qwen3.8-Flash-Next-UD-Q2_K_XL-00001-of-0000N.gguf \
+    --mmproj models/mmproj-BF16.gguf --backend ggml_cuda --tp 2</code></pre>
+  <p><code>--tp N</code> on <code>qwen4exp</code> is a <strong>layer split</strong>: each GPU holds a contiguous run of whole layers. Nothing is sharded, so it is not tensor parallelism — and it is the same (and only) multi-GPU mode llama.cpp offers this architecture, whose <code>-sm row</code> refuses to load it. Treat it as a capacity feature rather than a speed one. On 2× A100-80GB with Qwen3.8-Flash-Next-UD-Q2_K_XL (73.4 GiB) the greedy output is <strong>byte-identical</strong> between the 1-GPU and 2-GPU runs (same SHA-256), VRAM lands at 24.2 GB + 26.2 GB instead of the whole model on one card, and throughput is unchanged: prefill ~1520–1550 t/s and decode ~56 t/s either way. llama.cpp on the same box measures pp1536 1094 / tg128 61.2 on one GPU and 1200 / 61.5 on two with <code>-sm layer</code>, so it too gains about 10% prefill and nothing on decode. Startup prints which mode ran and the per-GPU layer/byte split; <code>TS_Q4E_LAYER_SPLIT=20,28</code> overrides the balance with explicit layer counts per GPU (llama.cpp's <code>--tensor-split</code> in spirit) and throws rather than silently ignoring a value it cannot honour — useful, because the automatic balance prices weights and cannot see the vision tower, which loads later and lands on GPU 0.</p>
+
   <h3 id="gptoss">GPT OSS (text, thinking always on, tools)</h3>
   <pre data-lang="bash"><code>hf download ggml-org/gpt-oss-20b-GGUF gpt-oss-20b-MXFP4.gguf --local-dir models
 
@@ -162,6 +216,8 @@ <h2 id="multimodal">Multimodal support</h2>
       <tr><td>Gemma 4</td><td>Image · Video · Audio</td><td>Images PNG/JPEG/HEIC; Video MP4 (1 fps via OpenCV); Audio WAV 16 kHz mono / MP3 / OGG. E4B projector: <code>mmproj-gemma-4-E4B-it-Q8_0.gguf</code>.</td></tr>
       <tr><td>Gemma 3</td><td>Image</td><td>PNG / JPEG / HEIC. Non-gated 4B projector: <code>mmproj-model-f16.gguf</code>.</td></tr>
       <tr><td>Qwen 3.5 / 3.6</td><td>Image</td><td>Dynamic-resolution vision encoder. The 9B / 3.6 repositories use <code>mmproj-F16.gguf</code>.</td></tr>
+      <tr><td>Qwen 3.8 Flash Next</td><td>Image</td><td>The Qwen3.5-VL tower with (T,H,W) IMRoPE positions; multi-image prompts and multi-turn image sessions, with KV reuse across turns when the new prompt extends the cached prefix exactly. Projector: <code>mmproj-BF16.gguf</code> (same repo).</td></tr>
+      <tr><td>GLM 5.x (5.3-Flash)</td><td>Image</td><td>The GLM-OCR ViT — all 24 blocks run as one device-resident graph, and the projected rows override the <code>&lt;|image|&gt;</code> placeholders inside the native executor. Multi-image and multi-turn image sessions. Projector: <code>mmproj-BF16.gguf</code> (same repo). GLM-5.2 is text only.</td></tr>
       <tr><td>Mistral 3</td><td>Image</td><td>Pixtral vision encoder. Projector: <code>mmproj-mistralai_Mistral-Small-3.1-24B-Instruct-2503-f16.gguf</code>.</td></tr>
       <tr><td>Muse-Glimmer</td><td>Image</td><td>50-layer sparse-window ViT with 2D RoPE and a 2×2 pixel shuffle; the image is stretched (no padding, no tiling) to a grid chosen the same way llama.cpp chooses it. Projector: <code>mmproj-Muse-Glimmer-30B-Q8_0.gguf</code>.</td></tr>
       <tr><td>Nemotron-H (Omni)</td><td>Image</td><td>RADIO / v2_vl ViT encoder. Pass the matching <code>--mmproj</code>; image tokens expand at <code>&lt;image&gt;</code> placeholders. Audio is preprocessed only — real audio inference needs a Parakeet audio mmproj that the GGUF distribution does not ship.</td></tr>
@@ -171,13 +227,14 @@ <h2 id="multimodal">Multimodal support</h2>
   <p>Send images/audio/video via the <a href="cli.html#examples">CLI</a> (<code>--image</code>, <code>--video</code>, <code>--audio</code>), the <a href="server.html#webui">Web UI</a> uploads, or the <a href="http-api.html">HTTP API</a> (base64 <code>images</code> array for Ollama, <code>image_url</code> data URI for OpenAI). <strong>PDF documents</strong> are supported too — born-digital PDFs have their complete text layer extracted and inlined into the prompt; scanned PDFs fall back to page images for vision-capable models — via the CLI's <code>--pdf</code> flag (one-shot mode) or the Web UI upload (<code>TS_PDF_MAX_PAGES</code> caps the page count; default: all pages).</p>
 
   <h2 id="thinking">Thinking / reasoning mode</h2>
-  <p>Thinking-capable models (Qwen 3, Qwen 3.5/3.6, Gemma 4, GPT OSS, Nemotron-H, DeepSeek V4, GLM 5.x, Muse-Glimmer) produce structured chain-of-thought before the final answer. The thinking content is separated from the visible response so the client can show or hide it.</p>
+  <p>Thinking-capable models (Qwen 3, Qwen 3.5/3.6, Qwen 3.8 Flash Next, Gemma 4, GPT OSS, Nemotron-H, DeepSeek V4, GLM 5.x, Muse-Glimmer) produce structured chain-of-thought before the final answer. The thinking content is separated from the visible response so the client can show or hide it.</p>
   <ul>
     <li><strong>Qwen 3 / Qwen 3.5/3.6 / Nemotron-H</strong> — <code>&lt;think&gt;…&lt;/think&gt;</code> tags.</li>
+    <li><strong>Qwen 3.8 Flash Next</strong> — the same <code>&lt;think&gt;…&lt;/think&gt;</code> tags, but always on: the generation prompt opens the block unconditionally, so there is no non-thinking mode to switch to.</li>
     <li><strong>Gemma 4</strong> — <code>&lt;|channel&gt;thought …&lt;channel|&gt;</code> tags.</li>
     <li><strong>GPT OSS</strong> — Harmony format: <code>&lt;|channel|&gt;analysis</code> for thinking, <code>&lt;|channel|&gt;final</code> for the answer.</li>
     <li><strong>DeepSeek V4</strong> — <code>&lt;think&gt;…&lt;/think&gt;</code> tags; the chat template closes the block immediately unless thinking is requested, so reasoning is opt-in.</li>
-    <li><strong>GLM 5.x</strong> — <code>&lt;think&gt;…&lt;/think&gt;</code> tags, also opt-in: <code>--think</code> adds a <code>Reasoning Effort: Max</code> system line and leaves the block open for the model to close, and without it the prompt emits an empty <code>&lt;think&gt;&lt;/think&gt;</code>. Earlier turns' reasoning is always dropped from the prompt.</li>
+    <li><strong>GLM 5.x</strong> — <code>&lt;think&gt;…&lt;/think&gt;</code> tags, also opt-in: <code>--think</code> adds a <code>Reasoning Effort: Max</code> system line and leaves the block open for the model to close, and without it the prompt emits an empty <code>&lt;think&gt;&lt;/think&gt;</code>. Earlier turns' reasoning is always dropped from the prompt. <strong>GLM-5.3-Flash always reasons</strong>: its <code>Reasoning Effort: Max</code> system line is unconditional, the generation prompt always opens <code>&lt;think&gt;</code>, and past turns keep their reasoning.</li>
     <li><strong>Muse-Glimmer</strong> — an <code>assistant to=self</code> reasoning channel emitted by the chat template.</li>
   </ul>
   <p>Enable it via <code>--think</code> (CLI), <code>"think": true</code> (Ollama API / Web UI), or the thinking toggle in the browser. Responses expose the reasoning separately — e.g. <code>message.thinking</code> in the Ollama chat response.</p>
diff --git a/website/models-text_zh-cn.html b/website/models-text_zh-cn.html
index 57e2eacd..ef47779d 100644
--- a/website/models-text_zh-cn.html
+++ b/website/models-text_zh-cn.html
@@ -4,7 +4,7 @@
   <meta charset="utf-8">
   <meta name="viewport" content="width=device-width, initial-scale=1">
   <title>文本与 LLM 模型 — TensorSharp 维基</title>
-  <meta name="description" content="十一种文本架构的下载与运行示例，以及多模态输入、思考模式与工具调用。">
+  <meta name="description" content="十二种文本架构的下载与运行示例，以及多模态输入、思考模式与工具调用。">
   <link rel="icon" href="assets/logo.svg" type="image/svg+xml">
   <link rel="stylesheet" href="assets/style.css">
   <script>try{var t=localStorage.getItem('ts-theme')||(matchMedia('(prefers-color-scheme: dark)').matches?'dark':'light');document.documentElement.classList.toggle('dark',t==='dark');}catch(e){}</script>
@@ -84,6 +84,36 @@ <h3 id="glm">GLM 5.x（744B-A40B MoE，文本、思考、工具）</h3>
   <p>自报的 1M 上下文是<em>上限</em>而非承诺：1M token 的 MLA 缓存约 93 GiB，所以权重落盘之后加载器会去问各设备实际还剩多少显存，按「缓存加一整个 prefill 计算图」能装下的大小定上下文，并把选中的值打印出来。在 3× RTX PRO 6000 上，按层切分是 342,272 token，<code>--n-cpu-moe 30</code> 是 646,400，<code>--tp 3</code> 是 91,136（每个 rank 都要各自持有一份全长缓存）。想把某个长度变成硬性要求，就设 <code>MAX_CONTEXT</code>——放得下就照办，放不下会带着数字直接报错。</p>
   <p><code>--tp N</code> 这里也能用，但在 PCIe 互连的卡上它是容量特性而非速度特性：78 层里每层都要两次 all-reduce，占用的总线时间比拆分省下的算力还多，所以 <code>--tp 3</code> 实测 pp2048 505.6 / tg64 17.6 tok/s，而按层切分是 915.9 / 43.9。并发靠原生的按序列槽位承载（每个请求拥有自己的 MLA 与索引器缓存），<code>TS_BATCHED_FUSED_DECODE=1</code> 可以开启批处理融合解码，4 路并发下总吞吐 1.81 倍。参见<a href="distributed_zh-cn.html#architectures">张量并行</a>与<a href="advanced_zh-cn.html#continuous-batching">连续批处理</a>。</p>
 
+  <h3 id="glm53">GLM-5.3-Flash（320B MoE，文本 + 图像、思考、工具）</h3>
+  <p>这个混合架构的后继型号走的是与 GLM-5.2 <em>完全相同</em>的原生整模型 ggml 执行器和同一个模型类——GGUF 架构 id 是 <code>glm5next</code>——只是在其上叠了四处架构改动。320B 参数，288 个路由专家 top-8 外加一个共享专家，46 个块 = 45 个主干块 + 1 个 NextN 块。45 个主干层里有 34 层是 <strong>KDA 线性注意力</strong>；另外 11 层是 <strong>NoPE MLA + DSA</strong>（文本塔里任何地方都没有 rope，512 宽的潜变量本身<em>就是</em>缓存行）；索引器是<strong>池化</strong>的——4 格一池，top-k 2048 在池上取，再展开到池内成员；超连接是 ×4 流的 Sinkhorn 版本；所有 SwiGLU 都按上限 10 做钳位。KDA 的递归状态无法回退，因此只有当新提示恰好是缓存前缀的延长时才会复用该前缀。</p>
+  <pre data-lang="bash"><code># UD-Q2_K_XL 这一档约 101 GiB——想要更大或更小可换一个量化目录
+hf download unsloth/GLM-5.3-Flash-GGUF --include "UD-Q2_K_XL/*" --local-dir models
+hf download unsloth/GLM-5.3-Flash-GGUF mmproj-BF16.gguf --local-dir models
+
+# 按层切分到所有可见 GPU —— --model 指向第一个分片
+# （N 取决于你下载的那个量化目录里有多少个分片）
+dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll \
+    --model models/UD-Q2_K_XL/GLM-5.3-Flash-UD-Q2_K_XL-00001-of-0000N.gguf \
+    --backend ggml_cuda --input prompt.txt --max-tokens 200
+
+# 图像理解 —— GLM-OCR ViT 以 mmproj-BF16.gguf 形式发布
+dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll \
+    --model models/UD-Q2_K_XL/GLM-5.3-Flash-UD-Q2_K_XL-00001-of-0000N.gguf \
+    --mmproj models/mmproj-BF16.gguf --image photo.png --input question.txt \
+    --max-tokens 300 --backend ggml_cuda
+
+# 显存不够？把前 N 层的路由专家留在系统内存里
+dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll \
+    --model models/UD-Q2_K_XL/GLM-5.3-Flash-UD-Q2_K_XL-00001-of-0000N.gguf \
+    --backend ggml_cuda --n-cpu-moe 10 --input prompt.txt --max-tokens 200
+
+# 以 HTTP 服务方式运行
+dotnet TensorSharp.Server/bin/TensorSharp.Server.dll \
+    --model models/UD-Q2_K_XL/GLM-5.3-Flash-UD-Q2_K_XL-00001-of-0000N.gguf \
+    --mmproj models/mmproj-BF16.gguf --backend ggml_cuda</code></pre>
+  <p>目前能跑的：默认的按层切分（分到所有可见 GPU）、<code>--cpu-moe</code> / <code>--n-cpu-moe N</code> 把专家放在主机内存、以服务形式运行时的原生按序列槽位，以及视觉——<code>--image</code>、多图提示与多轮图像会话，都走 GLM-OCR ViT。<strong>暂不支持</strong>：<code>--tp</code> 张量并行（会被明确拒绝，请改用按层切分）与 NextN/MTP 推测解码。</p>
+  <p>实测：2× RTX PRO 6000 Blackwell（96 GB）、GLM-5.3-Flash-UD-Q2_K_XL（101 GiB）、按层切分、两个引擎都用 <code>n_ubatch</code> 2048、背靠背运行：pp2048 llama.cpp <strong>2070 t/s</strong> 对 TensorSharp 2014，pp16384 1690 对 <strong>1692</strong>，pp32768 <strong>1483</strong> 对 1446，tg64 36.6 对 <strong>73.5</strong>。prefill 双方相差不过几个百分点；decode 是 llama.cpp 的 <strong>2.0×</strong>。</p>
+
   <h3 id="gemma3">Gemma 3（文本 + 图像）</h3>
   <pre data-lang="bash"><code>hf download ggml-org/gemma-3-4b-it-GGUF gemma-3-4b-it-Q4_K_M.gguf --local-dir models
 hf download ggml-org/gemma-3-4b-it-GGUF mmproj-model-f16.gguf --local-dir models
@@ -109,6 +139,30 @@ <h3 id="qwen35">Qwen 3.5 / 3.6（文本 + 图像、思维链、工具；3.6 支
 
 dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model models/Qwen3.6-35B-A3B-UD-Q4_K_M.gguf --backend ggml_cuda --mtp-spec</code></pre>
 
+  <h3 id="qwen4exp">Qwen 3.8 Flash Next（混合 MoE，文本 + 图像、思考、工具）</h3>
+  <p>一个混合专家模型，GGUF 架构 id 为 <code>qwen4exp</code>：GatedDeltaNet 递归层与全注意力层交错（其中一部分还挂在 Qwen 稀疏注意力的索引器之后），一个 PLE n-gram 嵌入块，×4 的超连接流，以及 512 专家的 MoE。48 层，隐藏维度 2560，24 个 query 头 / 2 个 KV 头、head_dim 256，词表 248,320，每个 token 从 512 个专家里用 10 个，48 层中有 36 层带 GDN，PLE 在第 1 层。在各 ggml 后端上，整个 token（几乎）作为<strong>一张图</strong>跑完——嵌入、图内 PLE、全部 48 层、最后的混合器与 LM 头——并从按形状索引的已捕获图缓存中回放。视觉走 Qwen3.5-VL 塔，使用 (T,H,W) IMRoPE 位置，因此多图提示与多轮图像会话都可用；只要新提示恰好是缓存前缀的延长，跨轮的 KV 就会被复用（GDN 递归无法回退）。并发请求由按序列的状态持有者承载——每个请求拥有自己的注意力 KV 与索引器缓存、自己的 GDN 与 PLE 状态——因此请求之间的切换只是换一个引用。思考与工具调用都可用。</p>
+  <pre data-lang="bash"><code># UD-Q2_K_XL 这一档约 73.4 GiB——想要更大或更小可换一个量化目录
+hf download unsloth/Qwen3.8-Flash-Next-GGUF --include "UD-Q2_K_XL/*" --local-dir models
+hf download unsloth/Qwen3.8-Flash-Next-GGUF mmproj-BF16.gguf --local-dir models
+
+# 纯文本 —— --model 指向第一个分片
+# （N 取决于你下载的那个量化目录里有多少个分片）
+dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll \
+    --model models/UD-Q2_K_XL/Qwen3.8-Flash-Next-UD-Q2_K_XL-00001-of-0000N.gguf \
+    --backend ggml_cuda --input prompt.txt --max-tokens 300
+
+# 图像理解
+dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll \
+    --model models/UD-Q2_K_XL/Qwen3.8-Flash-Next-UD-Q2_K_XL-00001-of-0000N.gguf \
+    --mmproj models/mmproj-BF16.gguf --image photo.png --input question.txt \
+    --max-tokens 300 --backend ggml_cuda
+
+# 两张 GPU：这里的 --tp N 走的是按层切分，不是张量并行
+dotnet TensorSharp.Server/bin/TensorSharp.Server.dll \
+    --model models/UD-Q2_K_XL/Qwen3.8-Flash-Next-UD-Q2_K_XL-00001-of-0000N.gguf \
+    --mmproj models/mmproj-BF16.gguf --backend ggml_cuda --tp 2</code></pre>
+  <p><code>qwen4exp</code> 上的 <code>--tp N</code> 是<strong>按层切分</strong>：每张 GPU 持有一段连续的完整层。它不切分任何权重，所以并不是张量并行——而且这也是 llama.cpp 为该架构提供的同一种（且唯一的）多卡模式，它的 <code>-sm row</code> 会直接拒绝加载这个模型。请把它当作容量特性而非速度特性。在 2× A100-80GB、Qwen3.8-Flash-Next-UD-Q2_K_XL（73.4 GiB）上：单卡与双卡的贪心输出<strong>逐字节相同</strong>（SHA-256 一致）；显存是 24.2 GB + 26.2 GB，而不是整个模型压在一张卡上；吞吐没有变化——两种跑法都是 prefill 约 1520–1550 t/s、decode 约 56 t/s。作为对照，llama.cpp 在同一台机器上单卡 pp1536 1094 / tg128 61.2，双卡 <code>-sm layer</code> 1200 / 61.5，同样是 prefill 涨约 10%、decode 不涨。启动时会打印实际走的是哪种模式，以及每张卡分到的层数与字节数；<code>TS_Q4E_LAYER_SPLIT=20,28</code> 可以用显式的每卡层数覆盖自动均衡（相当于 llama.cpp 的 <code>--tensor-split</code>），遇到无法满足的值会直接抛错而不是悄悄忽略——这很有用，因为自动均衡只按权重定价，看不到稍后才加载、并且会落在 GPU 0 上的视觉塔。</p>
+
   <h3 id="gptoss">GPT OSS（文本、始终思考、工具）</h3>
   <pre data-lang="bash"><code>hf download ggml-org/gpt-oss-20b-GGUF gpt-oss-20b-MXFP4.gguf --local-dir models
 
@@ -162,6 +216,8 @@ <h2 id="multimodal">多模态支持</h2>
       <tr><td>Gemma 4</td><td>图像 · 视频 · 音频</td><td>图像 PNG/JPEG/HEIC；视频 MP4（经 OpenCV 以 1 fps 采样）；音频 WAV 16 kHz 单声道 / MP3 / OGG。E4B 投影器：<code>mmproj-gemma-4-E4B-it-Q8_0.gguf</code>。</td></tr>
       <tr><td>Gemma 3</td><td>图像</td><td>PNG / JPEG / HEIC。非 gated 4B 投影器：<code>mmproj-model-f16.gguf</code>。</td></tr>
       <tr><td>Qwen 3.5 / 3.6</td><td>图像</td><td>动态分辨率视觉编码器；9B / 3.6 仓库使用 <code>mmproj-F16.gguf</code>。</td></tr>
+      <tr><td>Qwen 3.8 Flash Next</td><td>图像</td><td>Qwen3.5-VL 视觉塔，使用 (T,H,W) IMRoPE 位置；支持多图提示与多轮图像会话，新提示恰好延长缓存前缀时跨轮复用 KV。投影器：<code>mmproj-BF16.gguf</code>（同仓库）。</td></tr>
+      <tr><td>GLM 5.x（5.3-Flash）</td><td>图像</td><td>GLM-OCR ViT——24 个块作为一张常驻设备的图跑完，投影后的行在原生执行器内部覆盖 <code>&lt;|image|&gt;</code> 占位。支持多图与多轮图像会话。投影器：<code>mmproj-BF16.gguf</code>（同仓库）。GLM-5.2 仅文本。</td></tr>
       <tr><td>Mistral 3</td><td>图像</td><td>Pixtral 视觉编码器。投影器：<code>mmproj-mistralai_Mistral-Small-3.1-24B-Instruct-2503-f16.gguf</code>。</td></tr>
       <tr><td>Muse-Glimmer</td><td>图像</td><td>50 层稀疏窗口 ViT，采用 2D RoPE 与 2×2 像素混洗；图像按 llama.cpp 相同的方式选定网格后直接拉伸（不填充、不切块）。投影器：<code>mmproj-Muse-Glimmer-30B-Q8_0.gguf</code>。</td></tr>
       <tr><td>Nemotron-H (Omni)</td><td>图像</td><td>RADIO / v2_vl ViT 编码器。传入匹配的 <code>--mmproj</code>；图像 token 在 <code>&lt;image&gt;</code> 占位处展开。当前 GGUF 分发未附真实音频推理需要的 Parakeet mmproj。</td></tr>
@@ -171,13 +227,14 @@ <h2 id="multimodal">多模态支持</h2>
   <p>通过 <a href="cli_zh-cn.html#examples">CLI</a>（<code>--image</code>、<code>--video</code>、<code>--audio</code>、<code>--pdf</code>）、<a href="server_zh-cn.html#webui">Web UI</a> 上传，或 <a href="http-api_zh-cn.html">HTTP API</a>（Ollama 用 base64 <code>images</code> 数组，OpenAI 用 <code>image_url</code> data URI）发送文件。数字版 PDF 会提取文本；扫描版 PDF 会转换为页面图像并需要视觉模型。</p>
 
   <h2 id="thinking">思考 / 推理模式</h2>
-  <p>具备思考能力的模型（Qwen 3、Qwen 3.5/3.6、Gemma 4、GPT OSS、Nemotron-H、DeepSeek V4、GLM 5.x、Muse-Glimmer）会在最终答案前产生结构化的思维链。思考内容与可见回复分离，便于客户端显示或隐藏。</p>
+  <p>具备思考能力的模型（Qwen 3、Qwen 3.5/3.6、Qwen 3.8 Flash Next、Gemma 4、GPT OSS、Nemotron-H、DeepSeek V4、GLM 5.x、Muse-Glimmer）会在最终答案前产生结构化的思维链。思考内容与可见回复分离，便于客户端显示或隐藏。</p>
   <ul>
     <li><strong>Qwen 3 / Qwen 3.5/3.6 / Nemotron-H</strong> —— <code>&lt;think&gt;…&lt;/think&gt;</code> 标签。</li>
+    <li><strong>Qwen 3.8 Flash Next</strong> —— 同样是 <code>&lt;think&gt;…&lt;/think&gt;</code> 标签，但始终开启：生成提示会无条件打开该块，没有非思考模式可切。</li>
     <li><strong>Gemma 4</strong> —— <code>&lt;|channel&gt;thought …&lt;channel|&gt;</code> 标签。</li>
     <li><strong>GPT OSS</strong> —— Harmony 格式：<code>&lt;|channel|&gt;analysis</code> 用于思考，<code>&lt;|channel|&gt;final</code> 用于答案。</li>
     <li><strong>DeepSeek V4</strong> —— <code>&lt;think&gt;…&lt;/think&gt;</code> 标签；不显式开启思考时聊天模板会立即闭合该块，因此推理是按需启用的。</li>
-    <li><strong>GLM 5.x</strong> —— 同样是 <code>&lt;think&gt;…&lt;/think&gt;</code> 标签，也是按需开启：加 <code>--think</code> 会补上 <code>Reasoning Effort: Max</code> 系统行并留下一个未闭合的块由模型自己收尾，不加时提示里写的是空的 <code>&lt;think&gt;&lt;/think&gt;</code>。历史轮次的思考内容始终不会带进提示。</li>
+    <li><strong>GLM 5.x</strong> —— 同样是 <code>&lt;think&gt;…&lt;/think&gt;</code> 标签，也是按需开启：加 <code>--think</code> 会补上 <code>Reasoning Effort: Max</code> 系统行并留下一个未闭合的块由模型自己收尾，不加时提示里写的是空的 <code>&lt;think&gt;&lt;/think&gt;</code>。历史轮次的思考内容始终不会带进提示。<strong>GLM-5.3-Flash 则始终思考</strong>：它的 <code>Reasoning Effort: Max</code> 系统行是无条件的，生成提示总是打开 <code>&lt;think&gt;</code>，历史轮次的思考内容也会保留。</li>
     <li><strong>Muse-Glimmer</strong> —— 聊天模板输出的 <code>assistant to=self</code> 推理通道。</li>
   </ul>
   <p>通过 <code>--think</code>（CLI）、<code>"think": true</code>（Ollama API / Web UI）或浏览器中的思考开关启用。响应会单独暴露推理 —— 例如 Ollama 聊天响应中的 <code>message.thinking</code>。</p>
diff --git a/website/models.html b/website/models.html
index b97d9da1..ed3dd9de 100644
--- a/website/models.html
+++ b/website/models.html
@@ -23,7 +23,7 @@ <h2 id="browse">Browse the model reference</h2>
   <p>The full reference is split across four pages. Start with downloads if you know which model you want, or with a category page if you are still choosing.</p>
   <div class="card-grid">
     <a class="card" href="models-downloads.html"><div class="ic">⬇️</div><h3>Model Downloads</h3><p>Every GGUF, with the projector, VAE, encoder and drafter files each family needs.</p></a>
-    <a class="card" href="models-text.html"><div class="ic">🧠</div><h3>Text &amp; LLM Models</h3><p>Eleven architectures, their run recipes, plus multimodal input, thinking and tool calling.</p></a>
+    <a class="card" href="models-text.html"><div class="ic">🧠</div><h3>Text &amp; LLM Models</h3><p>Twelve architectures, their run recipes, plus multimodal input, thinking and tool calling.</p></a>
     <a class="card" href="models-image.html"><div class="ic">🖼️</div><h3>Image Generation</h3><p>Qwen-Image-Edit, the Lightning LoRA, and the DiT step cache.</p></a>
     <a class="card" href="models-video.html"><div class="ic">🎬</div><h3>Video Generation</h3><p>MiniMax-H3 with native stereo audio, and Wan 2.1 / 2.2 with the distilled checkpoint that turns hours into minutes.</p></a>
   </div>
@@ -34,11 +34,12 @@ <h2 id="table">Supported architectures</h2>
     <thead><tr><th>Architecture</th><th>GGUF arch keys</th><th>Example models</th><th>Multimodal</th><th>Thinking</th><th>Tools</th><th>MTP spec</th></tr></thead>
     <tbody>
       <tr><td><strong>DeepSeek V4 Flash</strong></td><td><code>deepseek4</code></td><td>DeepSeek-V4-Flash (284B MoE, 256 experts, compressed sparse attention, 1M context)</td><td>Text only</td><td class="yes">Yes</td><td class="yes">Yes (DSML)</td><td class="yes">Yes (DSpark block drafter, separate GGUF)</td></tr>
-      <tr><td><strong>GLM 5.x</strong></td><td><code>glm-dsa</code></td><td>GLM-5.2 (744B-A40B MoE, 256 experts, MLA + DeepSeek Sparse Attention, 1M context)</td><td>Text only</td><td class="yes">Yes</td><td class="yes">Yes (XML tool calls)</td><td class="no">—</td></tr>
+      <tr><td><strong>GLM 5.x</strong></td><td><code>glm-dsa</code>, <code>glm5next</code></td><td>GLM-5.2 (744B-A40B MoE, 256 experts, MLA + DeepSeek Sparse Attention, 1M context); GLM-5.3-Flash (320B, 288 routed experts, KDA linear attention on 34 of 45 trunk layers + NoPE MLA/DSA with a pooled indexer on the other 11)</td><td>Image on 5.3-Flash (5.2 is text only)</td><td class="yes">Yes</td><td class="yes">Yes (XML tool calls)</td><td class="no">—</td></tr>
       <tr><td><strong>Gemma 4</strong></td><td><code>gemma4</code></td><td>gemma-4-E4B, 12B, 31B, 26B-A4B (MoE)</td><td>Image, Video, Audio</td><td class="yes">Yes</td><td class="yes">Yes</td><td class="yes">Yes (separate draft)</td></tr>
       <tr><td><strong>Gemma 3</strong></td><td><code>gemma3</code></td><td>gemma-3-4b</td><td>Image</td><td class="no">No</td><td class="no">No</td><td class="no">—</td></tr>
       <tr><td><strong>Qwen 3</strong></td><td><code>qwen3</code>, <code>qwen2</code>, <code>qwen2vl</code>, <code>qwen2_vl</code></td><td>Qwen3-4B. Qwen2 / Qwen2.5-VL GGUFs load through the same class as text-only chat</td><td>Text only</td><td class="yes">Yes</td><td class="yes">Yes</td><td class="no">—</td></tr>
       <tr><td><strong>Qwen 3.5 / 3.6</strong></td><td><code>qwen35</code>, <code>qwen35moe</code>, <code>qwen3next</code></td><td>Qwen3.5-9B, Qwen3.5/3.6-35B-A3B (MoE)</td><td>Image</td><td class="yes">Yes</td><td class="yes">Yes</td><td class="yes">Yes on 3.6 (embedded NextN — only in GGUFs that retain the NextN block, e.g. the <code>-MTP-</code> repos)</td></tr>
+      <tr><td><strong>Qwen 3.8 Flash Next</strong></td><td><code>qwen4exp</code></td><td>Qwen3.8-Flash-Next (hybrid MoE: GatedDeltaNet recurrent layers interleaved with full attention, some behind Qwen Sparse Attention's indexer, a PLE n-gram embedding block, ×4 hyper-connection streams, 512 experts / 10 used)</td><td>Image</td><td class="yes">Yes</td><td class="yes">Yes</td><td class="no">—</td></tr>
       <tr><td><strong>GPT OSS</strong></td><td><code>gptoss</code>, <code>gpt-oss</code></td><td>gpt-oss-20b (MoE)</td><td>Text only</td><td class="yes">Yes (always)</td><td class="yes">Yes</td><td class="no">—</td></tr>
       <tr><td><strong>Nemotron-H</strong></td><td><code>nemotron_h</code>, <code>nemotron_h_moe</code></td><td>Nemotron-H-8B, 47B, Nemotron 3 Nano Omni</td><td>Image (Omni)</td><td class="yes">Yes</td><td class="yes">Yes</td><td class="no">—</td></tr>
       <tr><td><strong>Mistral 3</strong></td><td><code>mistral3</code></td><td>Mistral-Small-3.1-24B-Instruct</td><td>Image</td><td class="no">No</td><td class="no">No</td><td class="no">—</td></tr>
@@ -72,7 +73,7 @@ <h2 id="fast-lane">Which one is fastest — the lever that matters per family</h
     </tbody>
   </table>
   </div>
-  <p class="page-foot-note">MiniMax-H3, Wan, Qwen-Image-Edit and DiffusionGemma are constructed without a tensor-parallel degree, so <code>--tp</code> does not apply to them. Of the video families, Wan is also the one that refuses a backend outright: <code>mlx</code> is not supported.</p>
+  <p class="page-foot-note">MiniMax-H3, Wan, Qwen-Image-Edit and DiffusionGemma are constructed without a tensor-parallel degree, so <code>--tp</code> does not apply to them. Of the video families, Wan is also the one that refuses a backend outright: <code>mlx</code> is not supported. On Qwen 3.8 Flash Next <code>--tp N</code> means something else again — it runs a <strong>layer split</strong>, each GPU holding a contiguous run of whole layers with nothing sharded, which is a capacity feature rather than a speed one: on 2× A100-80GB with UD-Q2_K_XL (73.4 GiB) the greedy output is byte-identical to the single-GPU run, VRAM lands at 24.2 + 26.2 GB, and prefill (~1520–1550 t/s) and decode (~56 t/s) are unchanged. GLM-5.3-Flash refuses <code>--tp</code> cleanly and uses its layer split instead. Architectures that support neither tensor parallelism nor a layer split now say so on stderr and run on one GPU instead of silently leaving the others idle.</p>
 
   <div class="content-footer">
     <a href="backends.html">← Compute Backends</a>
diff --git a/website/models_zh-cn.html b/website/models_zh-cn.html
index 30f43e00..8f4d5789 100644
--- a/website/models_zh-cn.html
+++ b/website/models_zh-cn.html
@@ -23,7 +23,7 @@ <h2 id="browse">浏览模型参考</h2>
   <p>完整参考分为四个页面。如果你已经确定要用哪个模型，从"模型下载"开始；如果还在挑选，从对应的分类页面开始。</p>
   <div class="card-grid">
     <a class="card" href="models-downloads_zh-cn.html"><div class="ic">⬇️</div><h3>模型下载</h3><p>所有 GGUF，以及各家族所需的投影器、VAE、编码器与草稿模型。</p></a>
-    <a class="card" href="models-text_zh-cn.html"><div class="ic">🧠</div><h3>文本与 LLM 模型</h3><p>十一种架构及其运行示例，加上多模态输入、思考模式与工具调用。</p></a>
+    <a class="card" href="models-text_zh-cn.html"><div class="ic">🧠</div><h3>文本与 LLM 模型</h3><p>十二种架构及其运行示例，加上多模态输入、思考模式与工具调用。</p></a>
     <a class="card" href="models-image_zh-cn.html"><div class="ic">🖼️</div><h3>图像生成</h3><p>Qwen-Image-Edit、Lightning LoRA，以及 DiT 步缓存。</p></a>
     <a class="card" href="models-video_zh-cn.html"><div class="ic">🎬</div><h3>视频生成</h3><p>MiniMax-H3 带原生立体声音频，以及 Wan 2.1 / 2.2 与那个把数小时变成几分钟的蒸馏检查点。</p></a>
   </div>
@@ -34,11 +34,12 @@ <h2 id="table">支持的架构</h2>
     <thead><tr><th>架构</th><th>GGUF 架构键</th><th>示例模型</th><th>多模态</th><th>思考</th><th>工具</th><th>MTP 推测</th></tr></thead>
     <tbody>
       <tr><td><strong>DeepSeek V4 Flash</strong></td><td><code>deepseek4</code></td><td>DeepSeek-V4-Flash（284B MoE，256 专家，压缩稀疏注意力，1M 上下文）</td><td>仅文本</td><td class="yes">是</td><td class="yes">是（DSML）</td><td class="yes">是（DSpark 块级草稿器，独立 GGUF）</td></tr>
-      <tr><td><strong>GLM 5.x</strong></td><td><code>glm-dsa</code></td><td>GLM-5.2（744B-A40B MoE，256 专家，MLA + DeepSeek 稀疏注意力，1M 上下文）</td><td>仅文本</td><td class="yes">是</td><td class="yes">是（XML 工具调用）</td><td class="no">—</td></tr>
+      <tr><td><strong>GLM 5.x</strong></td><td><code>glm-dsa</code>、<code>glm5next</code></td><td>GLM-5.2（744B-A40B MoE，256 专家，MLA + DeepSeek 稀疏注意力，1M 上下文）；GLM-5.3-Flash（320B，288 个路由专家，45 个主干层里 34 层 KDA 线性注意力，另外 11 层是 NoPE MLA/DSA 配池化索引器）</td><td>5.3-Flash 支持图像（5.2 仅文本）</td><td class="yes">是</td><td class="yes">是（XML 工具调用）</td><td class="no">—</td></tr>
       <tr><td><strong>Gemma 4</strong></td><td><code>gemma4</code></td><td>gemma-4-E4B、12B、31B、26B-A4B (MoE)</td><td>图像、视频、音频</td><td class="yes">是</td><td class="yes">是</td><td class="yes">是（独立草稿）</td></tr>
       <tr><td><strong>Gemma 3</strong></td><td><code>gemma3</code></td><td>gemma-3-4b</td><td>图像</td><td class="no">否</td><td class="no">否</td><td class="no">—</td></tr>
       <tr><td><strong>Qwen 3</strong></td><td><code>qwen3</code>、<code>qwen2</code>、<code>qwen2vl</code>、<code>qwen2_vl</code></td><td>Qwen3-4B。Qwen2 / Qwen2.5-VL 的 GGUF 也由同一实现加载，按纯文本对话使用</td><td>仅文本</td><td class="yes">是</td><td class="yes">是</td><td class="no">—</td></tr>
       <tr><td><strong>Qwen 3.5 / 3.6</strong></td><td><code>qwen35</code>、<code>qwen35moe</code>、<code>qwen3next</code></td><td>Qwen3.5-9B、Qwen3.5/3.6-35B-A3B (MoE)</td><td>图像</td><td class="yes">是</td><td class="yes">是</td><td class="yes">3.6 支持（内嵌 NextN——仅限保留 NextN 块的 GGUF，例如 <code>-MTP-</code> 仓库）</td></tr>
+      <tr><td><strong>Qwen 3.8 Flash Next</strong></td><td><code>qwen4exp</code></td><td>Qwen3.8-Flash-Next（混合 MoE：GatedDeltaNet 递归层与全注意力层交错，其中一部分挂在 Qwen 稀疏注意力的索引器之后；PLE n-gram 嵌入块；×4 超连接流；512 专家、每 token 用 10 个）</td><td>图像</td><td class="yes">是</td><td class="yes">是</td><td class="no">—</td></tr>
       <tr><td><strong>GPT OSS</strong></td><td><code>gptoss</code>、<code>gpt-oss</code></td><td>gpt-oss-20b (MoE)</td><td>仅文本</td><td class="yes">是（始终）</td><td class="yes">是</td><td class="no">—</td></tr>
       <tr><td><strong>Nemotron-H</strong></td><td><code>nemotron_h</code>、<code>nemotron_h_moe</code></td><td>Nemotron-H-8B、47B、Nemotron 3 Nano Omni</td><td>图像（Omni）</td><td class="yes">是</td><td class="yes">是</td><td class="no">—</td></tr>
       <tr><td><strong>Mistral 3</strong></td><td><code>mistral3</code></td><td>Mistral-Small-3.1-24B-Instruct</td><td>图像</td><td class="no">否</td><td class="no">否</td><td class="no">—</td></tr>
@@ -72,7 +73,7 @@ <h2 id="fast-lane">哪个更快——每个家族最关键的那一项设置</h2
     </tbody>
   </table>
   </div>
-  <p class="page-foot-note">MiniMax-H3、Wan、Qwen-Image-Edit 与 DiffusionGemma 构造时不接受张量并行度，因此 <code>--tp</code> 对它们无效。在视频家族里，Wan 还是唯一会直接拒绝某个后端的：不支持 <code>mlx</code>。</p>
+  <p class="page-foot-note">MiniMax-H3、Wan、Qwen-Image-Edit 与 DiffusionGemma 构造时不接受张量并行度，因此 <code>--tp</code> 对它们无效。在视频家族里，Wan 还是唯一会直接拒绝某个后端的：不支持 <code>mlx</code>。在 Qwen 3.8 Flash Next 上，<code>--tp N</code> 的含义又不一样——它跑的是<strong>按层切分</strong>，每张 GPU 持有一段连续的完整层、不切分任何权重，因此是容量特性而非速度特性：2× A100-80GB、UD-Q2_K_XL（73.4 GiB）下，贪心输出与单卡逐字节相同，显存变成 24.2 + 26.2 GB，prefill（约 1520–1550 t/s）与 decode（约 56 t/s）都没有变化。GLM-5.3-Flash 会明确拒绝 <code>--tp</code>，改用它的按层切分。既不支持张量并行、也不支持按层切分的架构，现在会在 stderr 上直说，并只用一张 GPU 运行，而不是让其余的卡白白闲着。</p>
 
   <div class="content-footer">
     <a href="backends_zh-cn.html">← 计算后端</a>
diff --git a/website/overview.html b/website/overview.html
index 3296bf4b..8c183a69 100644
--- a/website/overview.html
+++ b/website/overview.html
@@ -87,14 +87,14 @@ <h2 id="status">Current status &amp; capabilities</h2>
   <table>
     <thead><tr><th>Area</th><th>Status</th></tr></thead>
     <tbody>
-      <tr><td>Model families</td><td><strong>Fourteen</strong>, dispatched by <code>ModelBase.Create()</code> from the GGUF's <code>general.architecture</code>: DeepSeek V4 Flash (<code>deepseek4</code>), GLM 5.x (<code>glm-dsa</code>), Gemma 3 (<code>gemma3</code>), Gemma 4 (<code>gemma4</code>), DiffusionGemma (<code>diffusion-gemma</code>, <code>diffusion_gemma</code>), Qwen 3 (<code>qwen3</code>, and Qwen2 / Qwen2.5-VL checkpoints as text-only chat via <code>qwen2</code>, <code>qwen2vl</code>, <code>qwen2_vl</code>), Qwen 3.5/3.6-family (<code>qwen35</code>, <code>qwen35moe</code>, <code>qwen3next</code>), GPT OSS (<code>gptoss</code>, <code>gpt-oss</code>), Nemotron-H incl. Nemotron 3 Nano Omni (<code>nemotron_h</code>, <code>nemotron_h_moe</code>), Mistral 3 (<code>mistral3</code>), Muse-Glimmer (<code>muse-glimmer</code>, <code>muse_glimmer</code>), Qwen-Image-Edit image editing (<code>qwen_image</code>, <code>qwen-image</code>), MiniMax-H3 joint audio-video generation (<code>minimax-h3</code>, <code>minimax_h3</code>), and Wan 2.1 / 2.2 video generation (<code>wan</code>, <code>wan2.1</code>, <code>wan2.2</code>). MiniMax-H3 is the exception to the metadata rule: its published GGUFs carry no metadata at all, so <code>ModelBase.Create()</code> recognises them by their tensor table instead. → <a href="models.html#table">Supported architectures</a></td></tr>
+      <tr><td>Model families</td><td><strong>Fifteen</strong>, dispatched by <code>ModelBase.Create()</code> from the GGUF's <code>general.architecture</code>: DeepSeek V4 Flash (<code>deepseek4</code>), GLM 5.x (<code>glm-dsa</code>, <code>glm5next</code>), Gemma 3 (<code>gemma3</code>), Gemma 4 (<code>gemma4</code>), DiffusionGemma (<code>diffusion-gemma</code>, <code>diffusion_gemma</code>), Qwen 3 (<code>qwen3</code>, and Qwen2 / Qwen2.5-VL checkpoints as text-only chat via <code>qwen2</code>, <code>qwen2vl</code>, <code>qwen2_vl</code>), Qwen 3.5/3.6-family (<code>qwen35</code>, <code>qwen35moe</code>, <code>qwen3next</code>), Qwen 3.8 Flash Next (<code>qwen4exp</code>), GPT OSS (<code>gptoss</code>, <code>gpt-oss</code>), Nemotron-H incl. Nemotron 3 Nano Omni (<code>nemotron_h</code>, <code>nemotron_h_moe</code>), Mistral 3 (<code>mistral3</code>), Muse-Glimmer (<code>muse-glimmer</code>, <code>muse_glimmer</code>), Qwen-Image-Edit image editing (<code>qwen_image</code>, <code>qwen-image</code>), MiniMax-H3 joint audio-video generation (<code>minimax-h3</code>, <code>minimax_h3</code>), and Wan 2.1 / 2.2 video generation (<code>wan</code>, <code>wan2.1</code>, <code>wan2.2</code>). MiniMax-H3 is the exception to the metadata rule: its published GGUFs carry no metadata at all, so <code>ModelBase.Create()</code> recognises them by their tensor table instead. → <a href="models.html#table">Supported architectures</a></td></tr>
       <tr><td>Inference hosts</td><td>CLI, interactive REPL, ASP.NET Core web UI, Ollama-style API, OpenAI Chat Completions-style API.</td></tr>
-      <tr><td>Backends</td><td>Pure C# CPU, direct CUDA/cuBLAS (<code>cuda</code>), MLX Metal (<code>mlx</code>), GGML CPU, GGML Metal, GGML CUDA, GGML Vulkan. DeepSeek V4 additionally has three whole-model executors of its own — direct CUDA, native ggml, and a pure-C# CPU one — each layer-splitting the weights across every visible GPU. GLM 5.x has two: a native ggml executor on <code>ggml_cuda</code> / <code>ggml_vulkan</code> / <code>ggml_cpu</code> / <code>ggml_metal</code> (also layer-splitting across every GPU) and the managed per-op path on <code>cpu</code> and <code>cuda</code>; MLX does not run it.</td></tr>
+      <tr><td>Backends</td><td>Pure C# CPU, direct CUDA/cuBLAS (<code>cuda</code>), MLX Metal (<code>mlx</code>), GGML CPU, GGML Metal, GGML CUDA, GGML Vulkan. DeepSeek V4 additionally has three whole-model executors of its own — direct CUDA, native ggml, and a pure-C# CPU one — each layer-splitting the weights across every visible GPU. GLM 5.x — GLM-5.2 and GLM-5.3-Flash alike — has two: a native ggml executor on <code>ggml_cuda</code> / <code>ggml_vulkan</code> / <code>ggml_cpu</code> / <code>ggml_metal</code> (also layer-splitting across every GPU) and the managed per-op path on <code>cpu</code> and <code>cuda</code>; MLX does not run it.</td></tr>
       <tr><td>Speed levers</td><td>The choice that matters most is usually the <em>checkpoint</em>, not the flag. <strong>MiniMax-H3</strong> is the one family where it is not: it ships CFG-distilled, so there is no faster file to fetch — <code>--cfg 1.0</code> at 4–8 steps is the operating point against its 20-step default, and <code>--width</code> / <code>--height</code> is the dominant quality-versus-cost lever (640×384 is the documented starting point). A <strong>step-distilled Wan</strong> DiT (<code>Turbo</code> / <code>Lightning</code> / <code>lightx2v</code> / <code>FastWan</code> / <code>…-4steps-…</code> in the file name) is detected at load and runs 4 guidance-free DiT passes instead of the base recipe's 100: the same 1088×832×121-frame image-to-video request measured ≈3 h 30 m on the base Wan2.2-TI2V-5B and 17 m 30 s on the Turbo one (M5 Pro, <code>ggml_metal</code>). Qwen-Image-Edit's Lightning LoRA (<code>--qwen-image-lora</code>) likewise replaces 30 steps at CFG 2.5 with 4–8 at CFG 1.0. For text models the levers are the backend (<code>ggml_cuda</code> on NVIDIA, <code>ggml_metal</code> on Apple Silicon, <code>ggml_cpu</code> rather than the pure-managed <code>cpu</code>), whole-model fused decode graphs (GPT OSS 24 → 154 tok/s on an A40), speculative decoding, <code>--tp N</code>, and <code>--n-cpu-moe</code> for a model that would otherwise not fit. → <a href="models.html">Models</a> · <a href="benchmarks.html">Benchmarks</a></td></tr>
-      <tr><td>Multimodal</td><td>Gemma 4 image/video/audio; Gemma 3, Qwen 3.5-family, Mistral 3, Nemotron-H Omni, and Muse-Glimmer image input. PDF documents via Web UI upload and CLI <code>--pdf</code> (text extracted and inlined; scanned pages rendered as images for vision models). Media <em>out</em>: MiniMax-H3 (H.264 MP4 <strong>plus a native 32 kHz stereo <code>.wav</code></strong>, denoised together in one packed latent — text → video, image → video, first/last frame, reference → video), Qwen-Image-Edit (image), and Wan 2.1 / 2.2 (H.264 MP4 video only, text → video and image → video). → <a href="models-video.html#minimax-h3">Video generation</a></td></tr>
-      <tr><td>Continuous batching</td><td>vLLM-style paged KV cache, block-hash prefix sharing, iteration-level scheduler (on by default; opt-out via <code>--no-continuous-batching</code>). DeepSeek V4 and GLM 5.x serve through their own native per-sequence slots on the same engine — a compressed MLA cache row per token has no paged layout to page — and GLM adds an opt-in batched fused decode (<code>TS_BATCHED_FUSED_DECODE=1</code>, 1.81× aggregate at four concurrent requests).</td></tr>
+      <tr><td>Multimodal</td><td>Gemma 4 image/video/audio; Gemma 3, Qwen 3.5-family, Qwen 3.8 Flash Next, GLM-5.3-Flash, Mistral 3, Nemotron-H Omni, and Muse-Glimmer image input. PDF documents via Web UI upload and CLI <code>--pdf</code> (text extracted and inlined; scanned pages rendered as images for vision models). Media <em>out</em>: MiniMax-H3 (H.264 MP4 <strong>plus a native 32 kHz stereo <code>.wav</code></strong>, denoised together in one packed latent — text → video, image → video, first/last frame, reference → video), Qwen-Image-Edit (image), and Wan 2.1 / 2.2 (H.264 MP4 video only, text → video and image → video). → <a href="models-video.html#minimax-h3">Video generation</a></td></tr>
+      <tr><td>Continuous batching</td><td>vLLM-style paged KV cache, block-hash prefix sharing, iteration-level scheduler (on by default; opt-out via <code>--no-continuous-batching</code>). DeepSeek V4 and GLM 5.x serve through their own native per-sequence slots on the same engine — a compressed MLA cache row per token has no paged layout to page — and GLM adds an opt-in batched fused decode (<code>TS_BATCHED_FUSED_DECODE=1</code>, 1.81× aggregate at four concurrent requests). Qwen 3.8 Flash Next uses per-sequence state holders: each in-flight request owns its KV, GatedDeltaNet, PLE and indexer state, so switching between them is a reference swap rather than a state download.</td></tr>
       <tr><td>Speculative decoding</td><td>MTP / NextN draft heads for Qwen 3.6 (embedded) and Gemma 4 (separate draft GGUF); DSpark block drafting for DeepSeek V4 and DFlash block drafting for Muse-Glimmer (separate drafter GGUF via <code>--draft-model</code> on the <code>cuda</code> / <code>ggml_cuda</code> backends). Verification is greedy against the target, so the emitted stream is the plain-greedy stream. Off by default — opt in with the server's <code>--mtp-spec</code> flag (env <code>TS_MTP_SPEC</code>), or by passing <code>--draft-model</code> on the CLI. → <a href="advanced.html#dspark">DSpark</a></td></tr>
-      <tr><td>Tensor parallelism</td><td>Megatron-LM column/row-parallel TP on the direct <code>cuda</code> backend and on GGML CUDA / Vulkan (<code>--tp N</code> / <code>TENSORSHARP_TP_DEGREE</code>, on both the CLI and the server), plus multi-node distributed TP over a peer-to-peer TCP mesh (<code>--tp-node-id</code> / <code>--tp-peers</code>), for every autoregressive architecture. Fused per-rank graphs and MoE expert parallelism make <code>--tp 2</code> decode faster than one GPU on Gemma 4. Optional Redis-backed shared KV cache and Responses API store. → <a href="distributed.html">Multi-GPU &amp; Multi-Node</a></td></tr>
+      <tr><td>Multi-GPU</td><td>Two different mechanisms share the <code>--tp</code> flag. <strong>Tensor parallelism</strong> shards the weights <em>inside</em> every layer — Megatron-LM column/row-parallel, replicated norms/embeddings/LM head — on the direct <code>cuda</code> backend and on GGML CUDA / Vulkan (<code>--tp N</code> / <code>TENSORSHARP_TP_DEGREE</code>, on both the CLI and the server), plus multi-node distributed TP over a peer-to-peer TCP mesh (<code>--tp-node-id</code> / <code>--tp-peers</code>). Fused per-rank graphs and MoE expert parallelism make <code>--tp 2</code> decode faster than one GPU on Gemma 4. A <strong>layer split</strong> gives each GPU a contiguous run of whole layers and shards nothing, so there are no collectives: DeepSeek V4 Flash and GLM 5.x (GLM-5.3-Flash included — it refuses <code>--tp</code> cleanly) do it by default with no flag, and <code>--tp N</code> on Qwen 3.8 Flash Next runs one too. That makes it a <strong>capacity feature, not a speed one</strong>: on 2× A100-80GB with Qwen3.8-Flash-Next-UD-Q2_K_XL (73.4 GiB) the 2-GPU run is byte-identical to the 1-GPU run and no faster (prefill ~1520–1550 t/s, decode ~56 t/s either way) — what changes is that the weights land as 24.2 + 26.2 GB rather than all on one card. <code>TS_Q4E_LAYER_SPLIT=20,28</code> overrides the automatic balance; architectures that support neither mode say so on stderr and run on one GPU. Optional Redis-backed shared KV cache and Responses API store. → <a href="distributed.html">Multi-GPU &amp; Multi-Node</a></td></tr>
       <tr><td>Observability</td><td>Structured per-turn logs, queue status, and KV-cache reuse metrics across Web UI, Ollama, and OpenAI response shapes.</td></tr>
     </tbody>
   </table>
diff --git a/website/overview_zh-cn.html b/website/overview_zh-cn.html
index 38e608d1..f23219b9 100644
--- a/website/overview_zh-cn.html
+++ b/website/overview_zh-cn.html
@@ -87,14 +87,14 @@ <h2 id="status">当前状态与能力</h2>
   <table>
     <thead><tr><th>领域</th><th>状态</th></tr></thead>
     <tbody>
-      <tr><td>模型家族</td><td>共 <strong>14 个</strong>，由 <code>ModelBase.Create()</code> 依据 GGUF 的 <code>general.architecture</code> 分发：DeepSeek V4 Flash（<code>deepseek4</code>）、GLM 5.x（<code>glm-dsa</code>）、Gemma 3（<code>gemma3</code>）、Gemma 4（<code>gemma4</code>）、DiffusionGemma（<code>diffusion-gemma</code>、<code>diffusion_gemma</code>）、Qwen 3（<code>qwen3</code>；Qwen2 / Qwen2.5-VL 权重亦可经 <code>qwen2</code>、<code>qwen2vl</code>、<code>qwen2_vl</code> 以纯文本对话方式加载）、Qwen 3.5/3.6-family（<code>qwen35</code>、<code>qwen35moe</code>、<code>qwen3next</code>）、GPT OSS（<code>gptoss</code>、<code>gpt-oss</code>）、Nemotron-H 含 Nemotron 3 Nano Omni（<code>nemotron_h</code>、<code>nemotron_h_moe</code>）、Mistral 3（<code>mistral3</code>）、Muse-Glimmer（<code>muse-glimmer</code>、<code>muse_glimmer</code>）、Qwen-Image-Edit 图像编辑（<code>qwen_image</code>、<code>qwen-image</code>）、MiniMax-H3 音视频联合生成（<code>minimax-h3</code>、<code>minimax_h3</code>），以及 Wan 2.1 / 2.2 视频生成（<code>wan</code>、<code>wan2.1</code>、<code>wan2.2</code>）。MiniMax-H3 是这条元数据规则的例外：它公开的 GGUF 完全没有元数据，因此 <code>ModelBase.Create()</code> 改用张量表来识别它。→ <a href="models_zh-cn.html#table">支持的架构</a></td></tr>
+      <tr><td>模型家族</td><td>共 <strong>15 个</strong>，由 <code>ModelBase.Create()</code> 依据 GGUF 的 <code>general.architecture</code> 分发：DeepSeek V4 Flash（<code>deepseek4</code>）、GLM 5.x（<code>glm-dsa</code>、<code>glm5next</code>）、Gemma 3（<code>gemma3</code>）、Gemma 4（<code>gemma4</code>）、DiffusionGemma（<code>diffusion-gemma</code>、<code>diffusion_gemma</code>）、Qwen 3（<code>qwen3</code>；Qwen2 / Qwen2.5-VL 权重亦可经 <code>qwen2</code>、<code>qwen2vl</code>、<code>qwen2_vl</code> 以纯文本对话方式加载）、Qwen 3.5/3.6-family（<code>qwen35</code>、<code>qwen35moe</code>、<code>qwen3next</code>）、Qwen 3.8 Flash Next（<code>qwen4exp</code>）、GPT OSS（<code>gptoss</code>、<code>gpt-oss</code>）、Nemotron-H 含 Nemotron 3 Nano Omni（<code>nemotron_h</code>、<code>nemotron_h_moe</code>）、Mistral 3（<code>mistral3</code>）、Muse-Glimmer（<code>muse-glimmer</code>、<code>muse_glimmer</code>）、Qwen-Image-Edit 图像编辑（<code>qwen_image</code>、<code>qwen-image</code>）、MiniMax-H3 音视频联合生成（<code>minimax-h3</code>、<code>minimax_h3</code>），以及 Wan 2.1 / 2.2 视频生成（<code>wan</code>、<code>wan2.1</code>、<code>wan2.2</code>）。MiniMax-H3 是这条元数据规则的例外：它公开的 GGUF 完全没有元数据，因此 <code>ModelBase.Create()</code> 改用张量表来识别它。→ <a href="models_zh-cn.html#table">支持的架构</a></td></tr>
       <tr><td>推理宿主</td><td>CLI、交互式 REPL、ASP.NET Core Web UI、Ollama 式 API、OpenAI Chat Completions 式 API。</td></tr>
-      <tr><td>后端</td><td>纯 C# CPU、直接 CUDA/cuBLAS（<code>cuda</code>）、MLX Metal（<code>mlx</code>）、GGML CPU、GGML Metal、GGML CUDA、GGML Vulkan。DeepSeek V4 另有三套专属的整模型执行器——Direct CUDA、原生 ggml 与纯 C#——都会把权重按层切分到所有可见 GPU。GLM 5.x 另有两套：<code>ggml_cuda</code> / <code>ggml_vulkan</code> / <code>ggml_cpu</code> / <code>ggml_metal</code> 上的原生 ggml 执行器（同样按层切分到所有卡），以及 <code>cpu</code> 与 <code>cuda</code> 上的托管逐算子路径；MLX 不支持它。</td></tr>
+      <tr><td>后端</td><td>纯 C# CPU、直接 CUDA/cuBLAS（<code>cuda</code>）、MLX Metal（<code>mlx</code>）、GGML CPU、GGML Metal、GGML CUDA、GGML Vulkan。DeepSeek V4 另有三套专属的整模型执行器——Direct CUDA、原生 ggml 与纯 C#——都会把权重按层切分到所有可见 GPU。GLM 5.x（GLM-5.2 与 GLM-5.3-Flash 相同）另有两套：<code>ggml_cuda</code> / <code>ggml_vulkan</code> / <code>ggml_cpu</code> / <code>ggml_metal</code> 上的原生 ggml 执行器（同样按层切分到所有卡），以及 <code>cpu</code> 与 <code>cuda</code> 上的托管逐算子路径；MLX 不支持它。</td></tr>
       <tr><td>提速手段</td><td>真正决定快慢的往往是<em>选哪个权重</em>，而不是加哪个参数。<strong>MiniMax-H3</strong> 是唯一的例外：它出厂即经 CFG 蒸馏，没有更快的文件可换——<code>--cfg 1.0</code> 下的 4–8 步就是工作点（默认 20 步），而 <code>--width</code> / <code>--height</code> 才是质量与成本之间的主导杠杆（文档给出的起点是 640×384）。<strong>步数蒸馏的 Wan</strong> DiT（文件名含 <code>Turbo</code> / <code>Lightning</code> / <code>lightx2v</code> / <code>FastWan</code> / <code>…-4steps-…</code>）在加载时被自动识别，只跑 4 次无引导 DiT 前向，而非基础配方的 100 次：同一个 1088×832×121 帧的图生视频请求，基础 Wan2.2-TI2V-5B 实测约 3 小时 30 分，Turbo 权重为 17 分 30 秒（M5 Pro、<code>ggml_metal</code>）。Qwen-Image-Edit 的 Lightning LoRA（<code>--qwen-image-lora</code>）同理，把 30 步 CFG 2.5 换成 4–8 步 CFG 1.0。文本模型这边的手段是：选对后端（NVIDIA 用 <code>ggml_cuda</code>，Apple Silicon 用 <code>ggml_metal</code>，CPU 推理用 <code>ggml_cpu</code> 而非纯托管的 <code>cpu</code>）、整模型融合 decode 计算图（GPT OSS 在 A40 上 24 → 154 tok/s）、推测解码、<code>--tp N</code>，以及让放不下的模型跑起来的 <code>--n-cpu-moe</code>。→ <a href="models_zh-cn.html">模型</a> · <a href="benchmarks_zh-cn.html">基准测试</a></td></tr>
-      <tr><td>多模态</td><td>Gemma 4 图像/视频/音频；Gemma 3、Qwen 3.5-family、Mistral 3、Nemotron-H Omni 与 Muse-Glimmer 图像输入。PDF 文档可经 Web UI 上传或 CLI <code>--pdf</code> 传入（提取文本并内联；扫描页渲染为图像交给视觉模型）。媒体<em>输出</em>：MiniMax-H3（H.264 MP4 <strong>外加一份原生 32 kHz 立体声 <code>.wav</code></strong>，在同一份打包潜变量里一起去噪——文本 → 视频、图像 → 视频、首尾帧、参考 → 视频）、Qwen-Image-Edit（图像），以及 Wan 2.1 / 2.2（只有画面的 H.264 MP4，文本 → 视频与图像 → 视频）。→ <a href="models-video_zh-cn.html#minimax-h3">视频生成</a></td></tr>
-      <tr><td>连续批处理</td><td>vLLM 式分页 KV 缓存、块哈希前缀共享、迭代级调度器（默认开启；可用 <code>--no-continuous-batching</code> 关闭）。DeepSeek V4 与 GLM 5.x 在同一引擎上通过各自原生的 per-sequence slot 提供服务——压缩后的 MLA 每 token 只有一行缓存，没有可分页的布局——GLM 还提供可选的批处理融合解码（<code>TS_BATCHED_FUSED_DECODE=1</code>，4 路并发下总吞吐 1.81 倍）。</td></tr>
+      <tr><td>多模态</td><td>Gemma 4 图像/视频/音频；Gemma 3、Qwen 3.5-family、Qwen 3.8 Flash Next、GLM-5.3-Flash、Mistral 3、Nemotron-H Omni 与 Muse-Glimmer 图像输入。PDF 文档可经 Web UI 上传或 CLI <code>--pdf</code> 传入（提取文本并内联；扫描页渲染为图像交给视觉模型）。媒体<em>输出</em>：MiniMax-H3（H.264 MP4 <strong>外加一份原生 32 kHz 立体声 <code>.wav</code></strong>，在同一份打包潜变量里一起去噪——文本 → 视频、图像 → 视频、首尾帧、参考 → 视频）、Qwen-Image-Edit（图像），以及 Wan 2.1 / 2.2（只有画面的 H.264 MP4，文本 → 视频与图像 → 视频）。→ <a href="models-video_zh-cn.html#minimax-h3">视频生成</a></td></tr>
+      <tr><td>连续批处理</td><td>vLLM 式分页 KV 缓存、块哈希前缀共享、迭代级调度器（默认开启；可用 <code>--no-continuous-batching</code> 关闭）。DeepSeek V4 与 GLM 5.x 在同一引擎上通过各自原生的 per-sequence slot 提供服务——压缩后的 MLA 每 token 只有一行缓存，没有可分页的布局——GLM 还提供可选的批处理融合解码（<code>TS_BATCHED_FUSED_DECODE=1</code>，4 路并发下总吞吐 1.81 倍）。Qwen 3.8 Flash Next 则使用 per-sequence 状态持有器：每个在飞请求拥有自己的 KV、GatedDeltaNet、PLE 与 indexer 状态，因此在请求之间切换只是换一个引用，而不需要回传状态。</td></tr>
       <tr><td>推测解码</td><td>Qwen 3.6（内嵌）与 Gemma 4（独立草稿 GGUF）的 MTP / NextN 草稿头；DeepSeek V4 的 DSpark 与 Muse-Glimmer 的 DFlash 块级起草（在 <code>cuda</code> / <code>ggml_cuda</code> 后端上通过 <code>--draft-model</code> 加载独立草稿 GGUF）。验证以贪心方式对齐主干，因此输出与普通贪心 decode 一致。默认关闭——服务端用 <code>--mtp-spec</code>（环境变量 <code>TS_MTP_SPEC</code>）启用，CLI 直接传 <code>--draft-model</code> 即可。→ <a href="advanced_zh-cn.html#dspark">DSpark</a></td></tr>
-      <tr><td>张量并行</td><td>Direct <code>cuda</code> 后端与 GGML CUDA / Vulkan 后端上的 Megatron-LM 列/行并行 TP（<code>--tp N</code> / <code>TENSORSHARP_TP_DEGREE</code>，CLI 与服务端均支持），以及基于点对点 TCP 网格的多节点分布式 TP（<code>--tp-node-id</code> / <code>--tp-peers</code>），覆盖全部自回归架构。融合的按 rank 计算图与 MoE 专家并行让 Gemma 4 上 <code>--tp 2</code> 的 decode 快于单卡。可选 Redis 支撑的共享 KV 缓存与 Responses API 存储。→ <a href="distributed_zh-cn.html">多 GPU 与多节点</a></td></tr>
+      <tr><td>多 GPU</td><td><code>--tp</code> 这一个参数实际对应两种不同的机制。<strong>张量并行</strong>在<em>每一层内部</em>切分权重 —— Megatron-LM 列/行并行，norm、词嵌入与 LM head 复制 —— 可用于 Direct <code>cuda</code> 后端与 GGML CUDA / Vulkan 后端（<code>--tp N</code> / <code>TENSORSHARP_TP_DEGREE</code>，CLI 与服务端均支持），并可通过点对点 TCP 网格扩展为多节点分布式 TP（<code>--tp-node-id</code> / <code>--tp-peers</code>）。融合的按 rank 计算图与 MoE 专家并行让 Gemma 4 上 <code>--tp 2</code> 的 decode 快于单卡。<strong>分层切分（layer split）</strong>则是每张 GPU 拿一段连续的完整层、不切分任何权重，因此没有集合通信：DeepSeek V4 Flash 与 GLM 5.x（含 GLM-5.3-Flash —— 它会干脆地拒绝 <code>--tp</code>）不需任何参数即默认如此，Qwen 3.8 Flash Next 在 <code>--tp N</code> 下走的也是这条路。因此它是<strong>容量特性而非速度特性</strong>：在 2× A100-80GB 上用 Qwen3.8-Flash-Next-UD-Q2_K_XL（73.4 GiB）实测，双卡输出与单卡逐字节相同，速度也一样（prefill 约 1520–1550 t/s，decode 约 56 t/s），变的只是权重从全部压在一张卡上变成 24.2 + 26.2 GB 分放。<code>TS_Q4E_LAYER_SPLIT=20,28</code> 可覆盖自动均衡；两种模式都不支持的架构会在 stderr 上明说，并仅使用一张 GPU。可选 Redis 支撑的共享 KV 缓存与 Responses API 存储。→ <a href="distributed_zh-cn.html">多 GPU 与多节点</a></td></tr>
       <tr><td>可观测性</td><td>结构化的逐轮日志、队列状态，以及跨 Web UI、Ollama 与 OpenAI 响应形态的 KV 缓存复用指标。</td></tr>
     </tbody>
   </table>