Skip to content

fix(cuda): support GLM IQ2_XXS routed down projection - #598

Closed
hy-sde wants to merge 1 commit into
antirez:mainfrom
hy-sde:fix/cuda-glm-iq2-xxs
Closed

fix(cuda): support GLM IQ2_XXS routed down projection#598
hy-sde wants to merge 1 commit into
antirez:mainfrom
hy-sde:fix/cuda-glm-iq2-xxs

Conversation

@hy-sde

@hy-sde hy-sde commented Jul 24, 2026

Copy link
Copy Markdown

Summary

Fix GLM-5.2 all-IQ2_XXS routed-MoE execution on CUDA. This resolves the cuda GLM prefill failed reproduction reported in #595 for the antirez/GLM-5.2-GGUF IQ2_XXS model with --ssd-streaming on DGX Spark / GB10.

This PR is based directly on current main and is independent of #592.

Root cause

The generic CUDA routed-MoE dispatcher accepted:

  • Q4_K gate/up with Q4_K down
  • IQ2_XXS gate/up with Q2_K down

Most routed layers in GLM-5.2-UD-IQ2_XXS_RoutedIQ2XXS_blk78Q2K.gguf use IQ2_XXS for gate, up, and down. The dispatcher rejected that combination before launching the routed projection, causing GLM prefill to fail.

Changes

  • Add an IQ2_XXS down-projection CUDA kernel using the existing IQ2_XXS-to-f32 dot-product implementation.
  • Dispatch IQ2_XXS gate/up/down layers through the existing f32 gate/up/SwiGLU path, the new IQ2_XXS down path, and the existing expert sum kernel.
  • Preserve the existing Q4_K and IQ2_XXS-with-Q2_K paths unchanged.
  • Add test-cuda-iq2-routed, a deterministic model-independent numerical test comparing every routed-MoE stage against the scalar CPU IQ2_XXS oracle.

Reproduction

Hardware: NVIDIA DGX Spark, GB10 (sm_121), CUDA backend.

./ds4 \
  --cuda \
  -m gguf/GLM-5.2-UD-IQ2_XXS_RoutedIQ2XXS_blk78Q2K.gguf \
  --ssd-streaming \
  --ctx 8192 \
  --temp 0 \
  --tokens 4 \
  -p 'Reply with exactly: OK'

Before this change:

ds4: prompt processing failed: cuda GLM prefill failed

After this change:

processing 23 input tokens: 23/23 (100.0%)
I need to reply
ds4: GLM prefill: 0.54 t/s, generation: 0.80 t/s

The process exited successfully after four non-EOS generation steps.

Numerical verification

make test-cuda-iq2-routed CUDA_ARCH=
gate: count=1024 worst_abs=3.33786e-06
up: count=1024 worst_abs=3.33786e-06
mid: count=1024 worst_abs=4.29153e-06
down: count=1024 worst_abs=8.39233e-05
out: count=512 worst_abs=8.39233e-05
test_cuda_iq2_routed PASS tokens=2 experts=2/3 dim=256

The test covers selected-expert routing, route weights, IQ2_XXS gate/up projections, SwiGLU, IQ2_XXS down projection, and expert summation.

Build verification

make ds4 ds4-server CUDA_ARCH=

Both CUDA binaries built successfully.

Fixes #595

@hy-sde hy-sde closed this Jul 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

GLM-5.2 (glm-dsa) prefill fails on CUDA / DGX Spark GB10 (sm_121): "cuda GLM prefill failed" with --ssd-streaming

1 participant