run cli: ./ds4 -m glm52_ds4/models--antirez--GLM-5.2-UD-IQ2_XXS_RoutedIQ2XXS_blk78Q2K.gguf --glm-mtp-timing --temp 0 --ssd-streaming
use make strix-halo build, the free report:
total used free shared buff/cache available
Mem: 31956712 4282560 16017176 77616 12208512 27674152
Swap: 33548404 29060 33519344
gpu mem:15.936G/96.00G
Environment:
Hardware: AMD Radeon Graphics (sm_115), 128 GiB unified memory
Build: make strix-halo, Ubuntu 26.04, rocm is already the newest version (7.2.4.70204-93~24.04).
Model: antirez/GLM-5.2-GGUF → GLM-5.2-UD-IQ2_XXS_RoutedIQ2XXS_blk78Q2K.gguf
ds4: Linux rocm backend set oom_score_adj=1000
ds4: ROCm backend initialized on AMD Radeon Graphics (sm_115)
ds4: SSD streaming auto cache budget
ds4: rocm recommends 96.00 GiB working set
ds4: using 80% total for model + cached experts: 76.80 GiB
ds4: non-routed weights: 22.55 GiB
ds4: routed expert size: 9.28 MiB
ds4: expert budget before prefill/full-layer reserve: 3469 (31.44 GiB)
ds4: GLM ROCm cache capped to 31.44 GiB by the memory guard for ctx=32768
ds4: GLM SSD streaming full resident layers: 0 (auto selected 0 layers)
ds4: rocm SSD streaming total expert budget 31.44 GiB = 5.91 GiB prefill headroom + 25.53 GiB dynamic cache (2817 experts, 9.28 MiB each)
ds4: SSD streaming mixed-precision model: 1/76 routed layers off the slab size class will bypass the expert cache and read experts via mapped model views
ds4: SSD streaming initial rocm model map restricted to token embedding (1 spans, 0.94 GiB tensor span)
ds4: ROCm preparing model tensor mappings: 0.00 GiB
ds4: ROCm startup model preparation covered 0.94 GiB of tensor spans in 0.000s
ds4: rocm backend initialized for graph diagnostics
ds4: memory: KV 5.81 GiB (raw 0.00 + compressed 5.81) + buffers 4.19 GiB + resident model 0.94 GiB + expert cache 25.53 GiB + prefill expert reserve 5.91 GiB = 42.39 GiB planned
ds4: memory detail: ctx=32768 prefill_cap=4096 raw_kv_rows=0 compressed_kv_rows=32768 backend=rocm
ds4: memory: KV 5.81 GiB (raw 0.00 + compressed 5.81) + buffers 4.19 GiB = 10.01 GiB context
ds4: memory detail: ctx=32768 prefill_cap=4096 raw_kv_rows=0 compressed_kv_rows=32768 backend=rocm
ds4: GLM Metal session ctx=32768 (model max=1048576); full-attention prefill/work cap=8192; compact indexed decode is used beyond the cap
ds4: GLM graph using compact DSA KV only; expanded full-attention KV cache is skipped
ds4: GLM graph allocating compact DSA cache: rows=32768 logical_ctx=32768 kv_layers=78 indexer_layers=21 f32 5.81 GiB
ds4: GLM compact indexed prefill chunk=4096 score_rows=2048 score_scratch=256.00 MiB
Commands:
/help Show this help.
/think Use normal thinking mode.
/think-max Use Think Max only when context is at least 393216 tokens.
/nothink Disable thinking mode.
/ctx N Set context size for following prompts.
/power N Set GPU duty cycle percentage, 1..100.
/read FILE Read a prompt from FILE and run it.
/quit, /exit Leave the prompt.
Ctrl+C Stop generation and return to the prompt.
ds4> write a python socket example
processing 23 input tokens: 0/23 (0.0%)ds4: prompt processing failed: rocm GLM prefill failed
ds4>
run cli: ./ds4 -m glm52_ds4/models--antirez--GLM-5.2-UD-IQ2_XXS_RoutedIQ2XXS_blk78Q2K.gguf --glm-mtp-timing --temp 0 --ssd-streaming
use make strix-halo build, the free report:
total used free shared buff/cache available
Mem: 31956712 4282560 16017176 77616 12208512 27674152
Swap: 33548404 29060 33519344
gpu mem:15.936G/96.00G
Environment:
Hardware: AMD Radeon Graphics (sm_115), 128 GiB unified memory
Build: make strix-halo, Ubuntu 26.04, rocm is already the newest version (7.2.4.70204-93~24.04).
Model: antirez/GLM-5.2-GGUF → GLM-5.2-UD-IQ2_XXS_RoutedIQ2XXS_blk78Q2K.gguf
ds4: Linux rocm backend set oom_score_adj=1000
ds4: ROCm backend initialized on AMD Radeon Graphics (sm_115)
ds4: SSD streaming auto cache budget
ds4: rocm recommends 96.00 GiB working set
ds4: using 80% total for model + cached experts: 76.80 GiB
ds4: non-routed weights: 22.55 GiB
ds4: routed expert size: 9.28 MiB
ds4: expert budget before prefill/full-layer reserve: 3469 (31.44 GiB)
ds4: GLM ROCm cache capped to 31.44 GiB by the memory guard for ctx=32768
ds4: GLM SSD streaming full resident layers: 0 (auto selected 0 layers)
ds4: rocm SSD streaming total expert budget 31.44 GiB = 5.91 GiB prefill headroom + 25.53 GiB dynamic cache (2817 experts, 9.28 MiB each)
ds4: SSD streaming mixed-precision model: 1/76 routed layers off the slab size class will bypass the expert cache and read experts via mapped model views
ds4: SSD streaming initial rocm model map restricted to token embedding (1 spans, 0.94 GiB tensor span)
ds4: ROCm preparing model tensor mappings: 0.00 GiB
ds4: ROCm startup model preparation covered 0.94 GiB of tensor spans in 0.000s
ds4: rocm backend initialized for graph diagnostics
ds4: memory: KV 5.81 GiB (raw 0.00 + compressed 5.81) + buffers 4.19 GiB + resident model 0.94 GiB + expert cache 25.53 GiB + prefill expert reserve 5.91 GiB = 42.39 GiB planned
ds4: memory detail: ctx=32768 prefill_cap=4096 raw_kv_rows=0 compressed_kv_rows=32768 backend=rocm
ds4: memory: KV 5.81 GiB (raw 0.00 + compressed 5.81) + buffers 4.19 GiB = 10.01 GiB context
ds4: memory detail: ctx=32768 prefill_cap=4096 raw_kv_rows=0 compressed_kv_rows=32768 backend=rocm
ds4: GLM Metal session ctx=32768 (model max=1048576); full-attention prefill/work cap=8192; compact indexed decode is used beyond the cap
ds4: GLM graph using compact DSA KV only; expanded full-attention KV cache is skipped
ds4: GLM graph allocating compact DSA cache: rows=32768 logical_ctx=32768 kv_layers=78 indexer_layers=21 f32 5.81 GiB
ds4: GLM compact indexed prefill chunk=4096 score_rows=2048 score_scratch=256.00 MiB
Commands:
/help Show this help.
/think Use normal thinking mode.
/think-max Use Think Max only when context is at least 393216 tokens.
/nothink Disable thinking mode.
/ctx N Set context size for following prompts.
/power N Set GPU duty cycle percentage, 1..100.
/read FILE Read a prompt from FILE and run it.
/quit, /exit Leave the prompt.
Ctrl+C Stop generation and return to the prompt.
ds4> write a python socket example
processing 23 input tokens: 0/23 (0.0%)ds4: prompt processing failed: rocm GLM prefill failed
ds4>