CUDA OOM regression on DGX Spark (GB10)

[xXRaidohXx]

Hi,

I've found what appears to be a regression in the CUDA backend on a NVIDIA DGX Spark (GB10).

I spent some time trying to bisect the problem, but unfortunately many of the intermediate commits no longer compile for me, so I couldn't identify the exact commit that introduced the regression.

Working commit

Commit:

519c4d8

Command:

./ds4-server
--cuda
-m ./ds4flash.gguf
--host 0.0.0.0
--port 5000
--ctx 500000
--kv-disk-dir ../cache/ds4-kv
--kv-disk-space-mb 32768

The server starts correctly and is fully usable.

Startup log:

ds4: Linux cuda backend set oom_score_adj=1000
ds4: CUDA backend initialized on NVIDIA GB10 (sm_121)
ds4: CUDA host registration skipped: operation not supported
ds4: CUDA preparing model tensor mappings
ds4: CUDA loading model tensors into device cache
ds4: CUDA loading model tensors 16.02 GiB cached
ds4: CUDA prepared model tensor mappings 16.37 GiB
ds4: CUDA loading model tensors 32.06 GiB cached
ds4: CUDA prepared model tensor mappings 32.07 GiB
ds4: CUDA loading model tensors 48.02 GiB cached
ds4: CUDA prepared model tensor mappings 48.43 GiB
ds4: CUDA loading model tensors 64.01 GiB cached
ds4: CUDA prepared model tensor mappings 64.70 GiB
ds4: CUDA loading model tensors 80.01 GiB cached
ds4: CUDA prepared model tensor mappings 80.45 GiB
ds4: CUDA startup model preparation covered 90.88 GiB of tensor spans in 21.776s
ds4: cuda backend initialized for graph diagnostics
ds4-server: context buffers 10835.27 MiB (ctx=500000, backend=cuda, prefill_chunk=4096, raw_kv_rows=4352, compressed_kv_rows=125002)
ds4: CUDA using managed KV cache for ctx=500000 (kv cache 6.76 GiB, context buffers 10.58 GiB); this may degrade performance but is needed for very large contexts
ds4-server: KV disk cache ../cache/ds4-kv (...)
ds4-server: listening on http://0.0.0.0:5000
Current HEAD

Running the same model with the current HEAD consistently results in an out-of-memory condition during startup.

I also tried reducing the context size to 300000, but the result is the same: the server fails with an OOM before becoming usable.

The relevant part of the log is:

ds4: CUDA backend initialized on NVIDIA GB10 (sm_121) dev=0
ds4: CUDA (no-copy) host registration skipped: operation not supported
ds4: CUDA preparing model tensor mappings
ds4: CUDA loading model tensors into device cache
...
ds4: CUDA startup model preparation covered 90.88 GiB of tensor spans
ds4: cuda backend initialized for graph diagnostics
ds4: memory: KV 4.20 GiB (raw 0.36 + compressed 3.84) + buffers 2.29 GiB + resident model 90.88 GiB = 97.38 GiB planned
ds4: memory detail: ctx=300000 prefill_cap=4096 raw_kv_rows=4352 compressed_kv_rows=75002 backend=cuda
ds4-server: context buffers 6648.24 MiB (...)

After that, the process runs out of memory.

Additional observations

I found one interesting behavior while testing:

Reducing --prefill-chunk to a much smaller value allows the server to start.
However, as soon as the first prompt is processed, it immediately runs out of memory.

I don't know whether this is related or just another symptom, but I thought it was worth mentioning.

Notes
--ssd-streaming avoids the startup OOM, but performance drops to roughly 1 token/sec, so it isn't a practical workaround.
I intentionally avoided speculating about the root cause because I don't have enough information to draw conclusions from the code changes alone.

If it helps, I'm happy to test experimental patches, add temporary logging, or run any additional experiments on the DGX Spark hardware.

[wiltaylor]

Getting this also.

[wiltaylor]

Working on a pr for this https://github.com/wiltaylor/ds4/tree/fix/glm-gb10-cuda-streaming