GLM-5.2 (glm-dsa) prefill fails on CUDA / DGX Spark GB10 (sm_121): "cuda GLM prefill failed" with --ssd-streaming

[alphagold]

Summary

On a DGX Spark (GB10, sm_121, CUDA 13.0, built with make cuda-spark), GLM-5.2 IQ2_XXS single-node with --ssd-streaming loads and allocates the GLM compact-DSA graph, but then fails at prefill:

ds4: prompt processing failed: cuda GLM prefill failed

There is no OOM (~116 GiB free at the time). DeepSeek-V4-Flash is the CUDA path this engine primarily targets and builds fine; this looks GLM-specific on the CUDA backend.

Environment

• Hardware: NVIDIA DGX Spark / GB10 (sm_121), 128 GiB unified memory
• Build: make cuda-spark, nvcc CUDA 13.0, Ubuntu 24.04, driver 580.x
• Model: antirez/GLM-5.2-GGUF → GLM-5.2-UD-IQ2_XXS_RoutedIQ2XXS_blk78Q2K.gguf
  (arch glm-dsa, block_count 79, expert_count 256)

Repro — single node, SSD streaming (fails)

./ds4 -m GLM-5.2-UD-IQ2_XXS_RoutedIQ2XXS_blk78Q2K.gguf --ssd-streaming -c 8192 -p "..."

Log:

ds4: CUDA backend initialized on NVIDIA GB10 (sm_121) dev=0
ds4: SSD streaming mixed-precision model: 1/76 routed layers off the slab size class
     will bypass the expert cache and read experts via mapped model views
ds4: GLM graph using compact DSA KV only; expanded full-attention KV cache is skipped
ds4: GLM graph allocating compact DSA cache: rows=8192 logical_ctx=8192 kv_layers=78 indexer_layers=21 f32 1.45 GiB
ds4: GLM compact indexed prefill chunk=4096 score_rows=4096 score_scratch=128.00 MiB
ds4: prompt processing failed: cuda GLM prefill failed

Data point — 2-node pipeline, resident weights, CUDA (reaches prefill)

Same GGUF, split across two GB10 nodes with resident weights (no --ssd-streaming):

# worker (node B)
./ds4 -m <gguf> --role worker --layers 40:output --coordinator <ip> 9911 -c 4096
# coordinator (node A)
./ds4 -m <gguf> --role coordinator --layers 0:39 --listen <ip> 9911 -c 4096

This does load both slices (95.99 + 97.34 GiB), initialize the CUDA backend on both, allocate the GLM-DSA graph and reach prefill. It is then only blocked by memory (196 GiB model / 2×121 GiB nodes; the prefill working set spikes and leaves ~4 GiB headroom, so it gets externally killed to avoid a wedge). So the resident CUDA GLM-DSA prefill path appears functional — the failure above seems specific to the --ssd-streaming path.

Questions

1. Is GLM --ssd-streaming prefill supported on the CUDA backend, or Metal/ROCm only? The surrounding code and env vars (DS4_METAL_*, metal_graph_seed_streaming_expert_cache_from_hotlist) look Metal/ROCm-oriented.
2. Is there a flag to get GLM-5.2 running single-node streamed on GB10, or is resident (multi-node) the only CUDA route for a model larger than one node's memory?
3. Cross-node --tensor-parallel returns tensor parallelism requires the Metal backend. Is CUDA cross-node TP planned, or is pipeline (--role/--layers) the intended CUDA multi-node path?

Thanks for the project — happy to run more diagnostics on GB10 if useful.