DSpark never produces a draft on ROCm / gfx1151: 1197 cycles, proposed=0, prop_hidden=0.000

[citroale90]

Summary

With --dspark enabled on a Strix Halo (gfx1151) ROCm build, the DSpark
speculative path is entered on every decode cycle but never emits a single
proposed token. DS4_DSPARK_STATS=1 shows the proposal pipeline running
prop_setup and prop_chain, then stopping: every stage from prop_hidden
onward stays at exactly 0.000 ms. draft_len_hist=none.

Net effect is a small loss, never a gain. The adaptive scheduler correctly
suppresses most attempts, which is why the cost is normally invisible.

Environment

CPU / GPU	AMD Ryzen AI Max+ 395, Radeon 8060S Graphics (gfx1151), 128 GB unified
Host OS	Arch Linux
Container	archlinux:latest, Docker
ROCm	7.14.0, TheRock tarball therock-dist-linux-gfx1151-7.14.0.tar.gz extracted to /opt/rocm
ds4 commit	0a7ad776b9068348e6cb09df8cafa9cadd285298
Build	make strix-halo
Model	DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix.gguf
Support model	DeepSeek-V4-Flash-DSpark-support.gguf

Startup reports the support model as healthy and capture as active:

ds4: DSpark support model detected: /models/DeepSeek-V4-Flash-DSpark-support.gguf
     (stages=3 block=5 markov_rank=256 tensors=81 missing=0 invalid=0 metadata_errors=0)
ds4: ROCm backend initialized on AMD Radeon 8060S Graphics (sm_115)
ds4: DSpark target-hidden capture enabled: layers=40,41,42

Reproduction

Server:

ds4-server \
  -m DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix.gguf \
  --mtp DeepSeek-V4-Flash-DSpark-support.gguf \
  --dspark \
  --host 0.0.0.0 --ctx 200000

with DS4_DSPARK_STATS=1 in the environment.

Three identical greedy requests, minimal payload, no client-side sampler
parameters:

for i in 1 2 3; do
  curl -s http://localhost:8000/v1/chat/completions \
    -H 'Content-Type: application/json' \
    -d '{"model":"deepseek-v4-flash",
         "messages":[{"role":"user","content":"Write a Python LinkedList class with push, pop, reverse, __len__ and __iter__. Include docstrings and type hints."}],
         "temperature":0,
         "max_tokens":400}' >/dev/null
done

Then stop the server so the session is destroyed and stats are flushed.

Observed

Run A — default scheduler

DSpark stats cycles=360 first_tokens=360 proposed=0 accepted_draft=0
accept_rate=0.00% avg_accept=0.000 full=0 partial=0 miss_first=0 no_draft=360
no_room=0 invalid=0 scheduler_skips=275 tail_skips=8 verifier_unavailable=0
errors=0 time_ms propose=162.318 prop_stage0=0.002 prop_setup=25.770
prop_cache=2.956 prop_chain=133.404 prop_hidden=0.000 prop_conf0=0.000
prop_logits=0.000 prop_markov=0.000 prop_confidence=0.000 snapshot=0.000
verify=0.000 target=85230.997 saved=0.000 net_saved=-162.420
draft_len_hist=none accepted_len_hist=0:360

Run B — DS4_DSPARK_SCHEDULER=0

DSpark stats cycles=1197 first_tokens=1197 proposed=0 accepted_draft=0
accept_rate=0.00% avg_accept=0.000 full=0 partial=0 miss_first=0 no_draft=1197
no_room=0 invalid=0 scheduler_skips=0 tail_skips=0 verifier_unavailable=0
errors=0 time_ms propose=2207.187 prop_stage0=0.028 prop_setup=360.438
prop_cache=12.962 prop_chain=1832.794 prop_hidden=0.000 prop_conf0=0.000
prop_logits=0.000 prop_markov=0.000 prop_confidence=0.000 snapshot=0.000
verify=0.000 spec_total=0.100 target=73028.190 saved=0.000 net_saved=-2207.287
draft_len_hist=none accepted_len_hist=0:1197

With the scheduler disabled every cycle reaches the proposal path
(scheduler_skips=0, tail_skips=0) and every cycle still ends in
no_draft.

Decode throughput, unchanged across every configuration tried:

configuration	decode
--dspark, default scheduler	16.2–16.3 t/s
--dspark, DS4_DSPARK_SCHEDULER=0	15.85–16.10 t/s
no --dspark	16.2–16.3 t/s

The ~3% drop in run B is the forced proposal overhead: 2207 ms against
73028 ms of target decode.

Where it stops

prop_setup and prop_chain accumulate real time (360 ms and 1833 ms in
run B). Everything downstream is exactly zero:

prop_hidden=0.000  prop_conf0=0.000  prop_logits=0.000
prop_markov=0.000  prop_confidence=0.000  snapshot=0.000  verify=0.000

So the chain stage runs to completion and the hidden-state consumption never
starts, even though capture is reported as enabled for layers 40, 41, 42.
errors=0, invalid=0 and verifier_unavailable=0 — nothing is failing
loudly, the draft simply never forms.

Ruled out

• Sampled decoding. Requests use temperature: 0 with no other sampler
  parameters, issued by curl directly against the server.
• Scheduler suppression. DS4_DSPARK_SCHEDULER=0, scheduler_skips=0.
• Confidence threshold. proposed=0, so no suffix is ever pruned; the
  threshold is never reached.
• Support model integrity. missing=0 invalid=0 metadata_errors=0.
• Capacity. no_room=0.

Not ruled out

The build carries a community patch that changes the ROCm decode kernels
(routing n_tok == 1 matmuls through the prequantized dp4a path, and
disabling the batch hc-norm fusion which otherwise produces corrupted output
on this GPU). It does not touch strict_dspark or the proposal pipeline, but
it does modify rocm/ds4_rocm_attention_launch.cuh, so it cannot be excluded
as a contributor without a clean-tree comparison. Happy to rerun on an
unpatched build with DS4_DSPARK_STATS=1 if that is useful — note that an
unpatched build needs DS4_METAL_DISABLE_BATCH_HC_NORM_FUSION=1 to produce
coherent output at all on gfx1151.

Expected

Some non-zero proposed count, and prop_hidden accumulating time.

Impact

Low severity on its own: DSpark is documented as experimental and opt-in, the
scheduler keeps the wasted work near 0.2% in normal operation, and correctness
is unaffected since the target model stays authoritative.

One caveat for anyone benchmarking this. Registering the support model map
sets g_q8_f16_disabled_for_multi_model, which disables the expanded Q8→F16
cache — so toggling --dspark silently toggles that cache as well. Measured
GTT on this machine, at rest after startup:

configuration	GTT used	decode
--mtp + --dspark	99.98 GiB	16.31 t/s
neither	106.96 GiB	16.38 t/s

Enabling a feature that adds a 5.58 GiB resident model reduces total GTT by
7 GiB, because the cache it suppresses is worth about 12.6 GiB here. An A/B on
--dspark alone therefore moves two variables at once; setting
DS4_CUDA_NO_Q8_F16_CACHE=1 on both sides isolates the speculation path.

Secondary observation, unrelated to the bug itself: on this UMA system that
cache costs 12.6 GiB for ~0.4% of decode throughput, which is inside
run-to-run noise. Worth revisiting as a default on unified-memory parts, where
it competes directly with KV and context tensors instead of living in separate
VRAM.

Happy to provide DS4_DSPARK_PROBE output, run with
--dspark-confidence 0, or test any patch.