Qwen3.8-27B: field notes from two more stacks (GB10 + gfx1151) - MTP speculative decoding at up to 3.5x, the quant that ships the draft head, and two small template notes

[Defilan]

Summary

Field notes on the Qwen3.8-27B guide from two stacks it does not cover: a DGX Spark GB10 on CUDA and a Strix Halo gfx1151 on llama.cpp Vulkan. I used the guide as a test plan rather than as background reading, so most of this is confirmation, and the parts that are not are additions rather than disagreements.

Everything in the guide that I probed held up. The xhigh default reproduced, your Trap 2 render behaviour reproduced exactly, and the low-beats-xhigh finding reproduced with a mechanism I can show. I went in expecting to find at least one thing to push back on and did not.

Four things, and the first is the one worth your time:

1. The MTP head is worth up to 3.5x, and it is not in the quant you tested on. Your tested_on line is the unsloth Q4_K_M, and that repo has no mtp-*.gguf in it. The ggml-org repo does. I think that is why the guide has no MTP section: there was nothing to find.
2. A gfx1151 row for the throughput table, plus two Vulkan-specific cautions that do not apply to your CUDA and Metal numbers.
3. reasoning_effort: high raises a 500 on the stock template rather than degrading quietly.
4. A behavioural addendum to Trap 2, and a second route to the turn-6 signature that an operator with a conservative max_tokens will hit instead of yours.

Environment

• GPUs: NVIDIA GB10 (DGX Spark, sm_121, 128GB unified) and AMD Radeon 8060S (Strix Halo / gfx1151, RADV, 128GB unified)
• Serving: llama.cpp via my operator, stock upstream, no fork
• Models: ggml-org/Qwen3.8-27B-GGUF Q4_K_M and unsloth/Qwen3.8-27B-GGUF Q4_K_M, both pulled 2026-08-14, byte sizes below
• Flags: --jinja --flash-attn on --cache-type-k/v f16 --ctx-size 16384 --parallel 1 -fit off, stock template, temperature 0, max_tokens 512 for throughput. Strix additionally needs --no-mmap.
• Method: n=5 per cell, three prompts (code generation, conceptual explanation, debugging). GPU residency verified before every measurement rather than assumed, for a reason I will come back to.

One caveat up front on the throughput numbers. My first cross-platform comparison varied context, KV type, temperature and template all at once and produced a table that looked clean and was not comparable. Everything below is from a re-run where the only flag differing between the two machines is --no-mmap. I mention it because the first version of these numbers reversed one of the conclusions.

---

1. MTP speculative decoding, and the quant that ships it

Qwen3.8 ships an MTP head as a separate GGUF rather than carrying it in the target model, so drafting is much cheaper than self-speculation and the speedup is large.

The file is mtp-Qwen3.8-27B-Q4_0.gguf, 1.6 GiB, and it exists only in ggml-org/Qwen3.8-27B-GGUF. I searched the unsloth repo including under BF16/ and there is no mtp-*.gguf there.

GB10, ctx 16384, f16 KV, temperature 0, n=5 per cell:

prompt	no spec	nDraftMax 3	nDraftMax 5	nDraftMax 7
code generation	11.02	29.40	35.89	37.70
conceptual explanation	10.98	22.14	21.00	19.91
debugging	10.98	20.98	19.34	18.08

The optimum depends on the workload, which I did not expect. Code generation keeps improving as you draft further ahead. Prose peaks at 3 and then declines, and by 7 it has given back a third of the gain. The mechanism is draft acceptance falling as nDraftMax rises: 0.469 at 3, 0.308 at 5, 0.229 at 7 on the debugging prompt.

Worth stating plainly because it nearly caught me: my first pass measured only a code prompt, concluded "5 is the winner at 3.13x", and that conclusion is wrong on two of the three workloads. If you add MTP guidance I would suggest framing it as measure-it-on-your-workload, with 3 as the safe general default and higher values only for code-heavy use.

This interacts with your quant-versus-engine section, which I think is the right analysis and this only extends. The two Q4_K_M files are not the same file:

repo	bytes	size	MTP head
unsloth/Qwen3.8-27B-GGUF	17,106,775,008	15.93 GiB	no
ggml-org/Qwen3.8-27B-GGUF	18,973,870,432	17.67 GiB	yes

Same node, same flags, swapping only the model: unsloth 12.02 tok/s, ggml-org 10.98. A 10.9% smaller file gives 9.5% more throughput, which is what your "batch-1 decode is memory-bandwidth bound" line predicts, so treat this as a datapoint for that conclusion rather than a new one.

The trade is lopsided though. Best code throughput available from each choice on the same GB10:

	tok/s
unsloth Q4_K_M, no MTP possible	12.07
ggml-org Q4_K_M + MTP at nDraftMax 7	37.70

Taking unsloth for a 9.5% faster baseline forfeits 3.1x. For anyone optimising single-stream speed the recommendation inverts: take the larger, slower file, because it is the only one that lets you turn the big lever.

For calibration against your 11.3: on the unsloth file I get 12.02 on my GB10. Different build and box, and I would not read anything into the difference.

2. A gfx1151 row, and two Vulkan cautions

Strix Halo, same methodology: 11.26 tok/s baseline, marginally ahead of the GB10's 10.98 on the same file. With MTP it reaches 33.03 at nDraftMax 5, and unlike CUDA it gets worse at 7 rather than better.

Two things specific to the Vulkan path that your CUDA and Metal numbers would not surface, and that I would want to know before quoting a speculated Vulkan figure:

Speculation makes output non-deterministic at temperature 0. On Strix, identical requests at nDraftMax 5 and 7 return different generation lengths run to run. On CUDA every arm was deterministic across all five samples. Same model, same draft head, same flags.

Speculation is unstable there. Worst-case spread across five identical samples:

arm	GB10	Strix
no spec	0.68%	0.14%
nDraftMax 3	1.49%	9.27%
nDraftMax 5	3.52%	31.11%
nDraftMax 7	3.64%	26.83%

Its unspeculated baseline is the single most stable measurement I took, so this is speculation introducing the variance rather than the platform being noisy. A single-sample speculated Vulkan number could be off by a third.

3. reasoning_effort: high is a hard 500, not a quiet degrade

Your comparison table notes froggeric adds a working high rung, which implies stock has none. On stock it does not fall back or no-op, it raises:

HTTP 500
Jinja Exception: Unexpected reasoning effort high.
Supported types are xhigh (default), medium, and low.

Small thing, but "the community template adds a high rung" reads like the stock behaviour is a silent downgrade, and anyone wiring high into a client gets a request-time failure instead. The error text is also a tidy independent confirmation of the xhigh default.

4. Trap 2 and the turn-6 trap: one addendum each, both agreeing with you

Trap 2 reproduced exactly. Your /apply-template numbers were 297 / 226 / 297 / 60 chars for none / low / xhigh / medium. My prompt-token counts are 71 / 59 / 71 / 29. Same shape, same conclusion: medium renders no effort line.

The addendum is behavioural. With no instruction rendered, the model does not fall back to something neutral. It reasons more than an explicit low. Reasoning characters at max_tokens 1024, temperature 0:

prompt	low	medium	xhigh
code	347	374	644
explain	1031	1633	1329
debug	1429	2974	4297

So medium is inert as an instruction and not inert in effect. Someone setting it expecting a midpoint gets unguided behaviour, which sat above low on all three prompts and above xhigh on one. If Trap 2 is worth a second line, that might be it: the danger is not that nothing happens, it is that nothing steers.

On the turn-6 failure, your diagnosis is right and I want to be clear I am not disputing it. Your completion counts fall monotonically (2900, 2805, 2714, 2645, 1645) without ever reaching your 32,768 budget, which is exactly the ctx minus prompt signature, and the arithmetic you show lands on it.

I initially thought I had refuted it, and I had not. I was running max_tokens 2048 against the same ctx 16384, and I get empty content with finish_reason: length at turn 3, three times out of three, with prompt_tokens at 3851 of 16384. That is 23% of context, so mine is output-budget exhaustion, a different wall reached first because my budget was small.

Two different causes, one identical signature, and your mitigation (a bigger window) fixes yours and does nothing for mine. They are only distinguishable by checking completion_tokens against both max_tokens and ctx minus prompt. Might be worth a sentence in that section, since an operator running a conservative output budget will hit the other one and reasonably conclude your fix did not work.

---

What I would most want your read on

The MTP-versus-quant tradeoff, because it cuts against the obvious reading of your serving section. Your quant analysis says take the 4-bit GGUF and do not overthink the engine, which is right. But between two files both called Q4_K_M, the slower one is the better choice by 3x, purely because of a 1.6 GiB side file that only one publisher ships. That feels like the kind of thing the guide exists to say, and I would rather you write it in your own voice than have me guess at the framing.

Happy to package the measurement harness as a scripts/mtp-probes/ contribution in the shape of the thinking and spine probes, standard library only, if that is useful. It does the arm sweep, asserts the running process actually carries the speculation config before each measurement, and refuses to record a number if the model is not resident on the GPU. Both guards exist because I got bitten by their absence.

There is one more thing I left out because it is a container problem rather than a model one, and I did not want to pad a model guide with it. Briefly: I lost a measurement to a silent CPU fallback, where the pod held the GPU, the device node was present, and llama.cpp still served from CPU at half speed with only a single startup warning to show for it. The number it produced looked entirely plausible. If you think a line about verifying GPU residency belongs in the serving notes for container users, say the word and I will write it up properly. If not, it is happily out of scope.

Raw JSON for every cell is available if you want it.

[TheTom]

This is a genuinely great writeup, thank you. You used the guide as a test plan instead of background reading and it shows, this is exactly the kind of field note that makes the card better. Folded it into the guide in #25.

On the one you most wanted my read on, the MTP versus quant tradeoff: you are right and I am writing it in the guide's voice. The finding that between two files both called Q4_K_M the slower one wins by 3x, purely because of a 1.6 GiB side file only ggml-org ships, is exactly the kind of thing the card exists to say. I inverted the single-stream recommendation accordingly: take the larger ggml-org file and run MTP, and kept the plain-decode logic for the batched and no-spec case.

The part that made me grin is your workload-dependent optimum, because I spent this week reproducing it from the other side. On the same GB10 I ran the vLLM and SGLang speculative paths (MTP, EAGLE, and DSpark) and got the identical shape: 45 to 53 tok/s on structured code and math, cratering to about 16 on free prose, and acceptance length rather than acceptance rate is what tracks throughput. Two engines, same conclusion, so nDraftMax 3 as the safe default with higher only for code is now in the guide as measure-it-on-your-workload. Your acceptance-falls-with-depth numbers are the cleanest statement of the mechanism I have seen.

Everything else landed too: the reasoning_effort high 500 (nice catch, and a tidy confirmation of the xhigh default), the medium-reasons-more-than-low behavioral addendum to Trap 2 which is a sharper framing than my silent-no-op line, the gfx1151 row, and the two Vulkan cautions. The non-determinism and 31 percent spread on RADV speculation are important, I would not have caught those on CUDA and Metal. And the second cause of the turn-6 signature is a real gap in my writeup, output-budget exhaustion reaching the same wall first, so I added the check-completion-tokens-against-both-bounds distinction.

Yes please to both offers. The scripts/mtp-probes harness in the shape of the spine and thinking probes would be very welcome, especially the two guards you mention (assert the process actually carries the speculation config, and refuse to record if the model is not GPU-resident), since a silent CPU fallback that produces a plausible number is exactly the trap I want the harness to prevent. Send that as a PR whenever it is ready. And yes, please write up the GPU-residency verification note for container users, that belongs in the serving section and you clearly have the scar tissue for it.

Raw JSON, if you are offering, I will take it and archive it alongside the run data. Thanks again, this was a pleasure to read.

[TheTom]

Reopening to track the two follow-ups you offered and I took you up on: the scripts/mtp-probes harness PR and the GPU-residency verification note. The field-notes fold-in itself landed in #25.