Feature/support dflash2 - #175
Merged
Merged
Conversation
DFlash2 is DFlash plus two things, both keyed off the GGUF so one code path
serves either generation:
* a grouped dynamic depthwise convolution around every attention and every
FFN sublayer - one projection of the sublayer INPUT yields both the filter
on that input and the filter on its output, masked at the block boundary;
* a candidate selector - instead of taking each block position's argmax over
the vocabulary independently (block diffusion's weakness), the top-K
candidates of adjacent positions are scored pairwise through two low-rank
[vocab, r] codebooks and the block is read off as a walk through that
lattice.
The unary term of that lattice carries the TARGET's logit transform
(dflash.logit_scale, dflash.final_logit_softcapping). Plain DFlash takes an
argmax and is invariant to both; the lattice ADDS the unary to a transition
score, so skipping it makes the unary the wrong size - it cost more than half
the acceptance rate on the Muse-Glimmer drafter.
The drafter is now architecture-agnostic: loading, the KV ring, the three
passes and the fused graphs moved from MuseGlimmerModel to ModelBase, and a
target model gets DFlash by tapping the per-layer residuals the drafter's
encoder reads. Qwen 3.5/3.8 does that inside its fused whole-model verify
kernel, so speculation does not force the op-by-op loop.
Speculation on a Qwen 3.x hybrid used to be a NET LOSS - 15.5 tok/s for
DFlash2 and 15.7 for MTP against 18.3 plain - and the reason was rejection
cost, not drafting. The GDN recurrent state cannot be truncated like a KV
cache, so a partial rejection restored a pre-verify copy of it and re-forwarded
the accepted prefix through all 64 layers, and the state (151 MB) crossed PCIe
twice per step regardless. The verify now keeps one recurrent-state snapshot
per row (ggml_gated_delta_net already emits them; the conv state after row m is
a window of a tensor the graph builds anyway), commits the accepted prefix's
slot into the live state entirely on the device, and skips both halves of the
round trip. rollbackMs 3604 -> 0, snapshotMs 919 -> 69.
Also fixes Qwen 3.8 loading at all: half the layers of an unsloth UD quant
store ffn_gate and ffn_up in different GGML types that no imatrix-free
requantization can fuse, and the dense FFN dereferenced the missing fused
tensor. Those layers now run two matmuls, weights untouched, on both the
managed and the fused-graph paths.
Measured on one RTX 3080 Laptop (16 GB), greedy, best of two:
Qwen3.8-27B-UD-IQ3_XXS prose 18.3 plain -> 20.9 DFlash2 (19.1 MTP)
Qwen3.8-27B-UD-IQ3_XXS factual 19.5 plain -> 31.7 DFlash2 (32.6 MTP)
Muse-Glimmer-30B IQ2_XXS 18.7 plain -> 23.0 DFlash2, 25.4 DFlash
Greedy output is byte-identical to plain decoding and to the old
restore-and-re-forward path. llama.cpp b10630 cannot load a DFlash2 drafter at
all ("wrong number of tensors; expected 81, got 58").
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A speculative session is not all verifies. When the drafter declines to propose, the step falls through to an ordinary one-row forward, and those still ran the old state download: 151 MB down, after which the next verify had to upload it again. On an MTP run that was 46 steps out of 125, and it was most of what remained of the gap to the captured decode. Such a step's post-window state is simply the *_state_out slices, and nothing decides anything about it later, so the kernel now defers it the way it already defers a snapshotting verify and the caller commits slot -1 immediately - one device-to-device copy instead of 302 MB across PCIe. TSGgml_Qwen35CommitStateSnapshot takes slot -1 for that; the cache entry records whether it deferred rather than inferring it from the snapshot count, since "deferred with no snapshots" is now a real state and is reported to the caller as 0. Worth 5-20% in paired runs (DFlash2 factual 22.0 -> 27.1 tok/s, MTP prose 19.1 -> 21.4), and it is also the more exact path: the device commit is a raw copy of the tensor the graph produced, where the host round trip went through an unpack-and-repack. On the factual prompt DFlash2 now reproduces plain decoding byte for byte, which it did not before. TS_Q35_VERIFY_DEFER_STATE=0 keeps the snapshots but restores the download, so the two halves can be measured apart. Also here: - Strided KV views on CUDA, not just Metal, behind TS_Q35_VERIFY_STRIDED_VIEWS. Byte-identical, ~1%. - TS_Q35_MTP_DRAFT_PERSIST re-arms the persist/replay cache for the single-layer MTP draft graph. Default off - that graph used to deadlock on CUDA-graph capture replay and the knob is there to re-test it. - Route the verify replay through graph_compute_profiled so it shows up under TS_GGML_NODE_PROFILE like every other graph. Docs: corrected the llama.cpp MTP comparison, which was confounded - llama.cpp defaults to thinking mode on this checkpoint and TensorSharp does not, so the two engines were answering with different continuations and the comparison measured the text rather than the engine. Matched, drafting is at parity (3.55 vs 3.5 tokens per accept call, 0.856 vs 0.838 acceptance) and the whole residual is per-step cost. Also re-measured the Qwen 3.8 table as one uninterrupted sweep and corrected two figures that did not reproduce. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
llama.cpp's draft-mtp runs its MTP block over n_accepted + 1 rows: one pass that both replays the verified tokens through the draft head and takes the first draft step. TensorSharp ran those as two calls - DraftCatchUp, then a separate first DraftStep - and on a head whose per-call cost is mostly fixed (its own graph, launch and readback) that extra call was measurably the largest remaining difference between the two engines. Measured on Qwen3.8-27B-UD-IQ3_XXS, 120 tokens, paired runs: catchUpMs 191 -> 0 against draftMs 534 -> 608, so 34.35 -> 35.45 tok/s on the factual prompt (+3.2%) and 22.8 -> 24.0 on prose (+5.3%). Output is byte-identical to both the two-call path and plain decoding, and acceptance is unchanged (1.000 factual, 0.908 prose) - the fold is an identity, because the block is causal over its own KV so its last row sees exactly the replayed rows either way. New optional capability on IDraftHead (SupportsFusedCatchUpStep + DraftCatchUpAndStep, both defaulted), implemented only by Qwen 3.x's NextN head. DraftHeadSpeculator stashes the commit and folds it into the next Propose, flushing instead whenever the stashed rows do not run up to the next step's position - so any path that puts a step in between still gets a plain replay. A DFlash drafter does not fold: its commit is a ring write costing ~1 ms, and it is left on exactly the path it was on. TS_MTP_FOLD_CATCHUP=0 restores the two-call shape. Also: TS_GGML_LOG_DEBUG=1 passes ggml's DEBUG channel through instead of dropping it. That channel carries the CUDA backend's "CUDA graph warmup complete"/"reset" lines, which are the only way to see whether a graph is actually being CUDA-graph-captured - not otherwise observable, and the thing that finally showed the verify graph was not the problem here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…bers
TS_SPEC_ADAPTIVE=0 disables the speculation cost governor. Its own doc
comment already called for this ("set false to force the drafter on for A/B
measurement") but nothing wired it up, and it is needed to measure what the
governor itself costs: a round's plain baseline steps are ordinary plain
decodes, and on a short generation they are a visible fraction of the run
(10 of 77 steps at 256 tokens, though only 2 of those turn out to be the
governor's - the rest are the confidence gate).
Docs: replaced the llama.cpp comparison with a true like-for-like one. The
previous numbers had thinking mode on for llama.cpp and off for TensorSharp,
so the engines were answering with different continuations and the
comparison measured the text. Matched (same prompt, thinking disabled on
both, greedy, 256 tokens): llama.cpp 39.4 tok/s at 3.63 tokens per accept
call, TensorSharp 33.9 at 3.67. Drafting is at parity or better; the gap is
per-step cost.
Also recorded what was checked and ruled OUT, since each is an obvious
suspect and none of them is the problem: CUDA-graph capture churn on the
verify (raising TS_Q35_VERIFY_CACHE_BUDGET_MB halves the resets and changes
nothing), the MTP draft graph not persisting, and the confidence gate. What
is left is measured: MtpProjectInput, the C# front end of the draft call,
is 462 ms against the fused kernel's 804 ms over 208 calls - 6.4% of the
run - spent on six device op launches that each synchronise because the
lazy-sync path is Metal-only.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
--draft-model set only TS_DSV4_DSPARK, which is DeepSeek V4's DSpark. Qwen 3.5/3.8 read TS_QWEN35_DFLASH and Muse-Glimmer reads TS_MUSE_GLIMMER_DFLASH, so on the server the flag was silently ignored for every DFlash / DFlash2 target - the flag's most common use, and silent because a missing drafter just decodes without one. Only the loaded model reads any of these, and each resolver validates the file's general.architecture before accepting it, so setting all three is safe. The CLI was never affected: it passes --draft-model straight to ModelBase.Create rather than through an env var. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Qwen3.8-Flash-Next is the first Qwen4-architecture checkpoint: 125 B total,
6 B active, and four subsystems Qwen 3.x does not have. All four are load
bearing, and none of them is a variation on something already in the tree:
* Hyper-connections. The residual stream is 4x wide. Every block reads it
through a learned gated mixer that collapses the four streams and writes
back through a learned per-stream scatter. There is no output norm - the
final mixer IS the output norm.
* PLE n-gram embeddings. One layer adds a lookup into a ~320 M row table
addressed by a host-side hash of the token's bigram and trigram context,
then a gated value and a dilated depthwise convolution.
* Qwen Sparse Attention. At or below indexer_top_k + compress_ratio - 1
cached tokens the selection is every cell, so dense attention is exact;
past it this warns rather than drifting silently. The indexer itself is
not implemented yet.
* Gated DeltaNet on 36 of 48 layers, identical to Qwen 3.5's except that
the output gate is a SIGMOID rather than a SiLU.
Verified against llama.cpp's PR #27742 on the same UD-Q2_K_XL weights: the
same prompt produces the same reasoning and the same answer.
Performance, RTX PRO 6000 Blackwell, from the first working version:
prefill 8.4 -> 89.9 tok/s (10.7x; 1024-token warmup 95.0s -> 18.5s)
decode 6.0 -> 17.0 tok/s (2.8x)
Four changes, each picked from a profiler rather than a guess:
1. The delta-net recurrence was a C# per-token loop - 41% of a decode token
and ~70% of prefill. It now runs on the existing chunked GDN kernel. That
needed one native change: a gate_mode parameter so the same kernel can
close with a sigmoid instead of a SiLU. Existing callers pass 0 and are
byte-identical.
2. The expert FFN was one native call PER TOKEN - 49k graph builds for a
1024-token prefill. It is one ggml_mul_mat_id over the whole batch now.
3. The routed experts were device-resident TWICE, once as the stacked tensor
the batched kernel binds and once as 512 per-expert views. Skipping the
views took device-resident weights from 74,489 MB to 3,073 MB, which is
what let the batched kernel allocate at all. The ~24 GB PLE table is
skipped for the same reason: every use is a host-side gather.
4. A decode step is dispatch-bound, so the tiny elementwise steps (the
mixer's sigmoid, the attention gate, partial RoPE) run on the host below
HostElementwiseMaxRows rows and go back to GGML above it. Applying that
unconditionally cost 2.7 s of prefill, which is why it is gated.
Also: GgufReader gains GetUint64Array (the PLE hash constants are 64-bit),
and QuantizedWeight.CreateExpertView gives a non-owning per-expert view over
a stacked tensor so backends without the stacked kernel can still route.
No regression: Qwen3.8-27B plain/MTP/DFlash2 on both prompts are byte-
identical to their pre-change outputs with throughput within noise, and all
1675 tests pass.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A qwen4exp decode step was dispatch bound, not arithmetic bound. The op-by-op
path issued ~850 GGML submissions per token, each its own context, allocation
and device synchronise, and the GPU idled between them - ggml_cpu and
ggml_cuda measured within 4% of each other, which is that bottleneck's
signature rather than a slow kernel. llama.cpp builds ONE graph for the whole
model; this takes the same road half a layer at a time.
New native kernel ggml_ops_qwen4exp.cpp with two entries:
Qwen4ExpFfnBlock - hyper-connection mixer -> 512-expert MoE through
ggml_mul_mat_id -> gated shared expert -> scatter back into the 4-wide
residual. 8 submissions per layer become 1.
Qwen4ExpGdnBlock - mixer -> qkv/gate/beta/alpha -> causal depthwise conv via
ggml_ssm_conv -> ggml_gated_delta_net -> gated norm -> scatter. Another 9
become 1, on 36 of the 48 layers.
Both persist their graph per layer and replay it, so a token is an upload, a
replay and a download rather than a rebuilt topology 48 times over.
Measured on the RTX PRO 6000, against llama.cpp PR #27742 on the same weights:
before after llama.cpp
prefill 8.4 197.9 116.4 (1.70x FASTER)
decode 6.0 47.9 84.1 (57%)
That is 23.6x on prefill and 8.0x on decode from the first working version.
Output still matches llama.cpp's reasoning and answer, and a 1549-token prompt
summarises correctly at 154 tok/s prefill.
Three bugs in the recurrent kernel are worth recording, because each produced
fluent-looking output rather than an error:
* ggml_ssm_conv asserts a 3-D input; a 2-D one silently mis-shapes.
* The recurrent state must not be written back into the tensor the same
graph reads. One call matched the reference to 0.3%, but two consecutive
calls diverged 100% because the second still started from the initial
state. It now reads *_in and writes a separate *_out.
* The state must live in its OWN device buffer, not the graph's gallocr
plan. A shape change (prefill -> decode) rebuilds the graph, and rebuilding
lost the state - the model then ignored its prompt while every per-layer
check passed. Carrying it across the rebuild by copying did not hold
either; giving it a buffer the rebuild cannot touch did.
Diagnostics kept behind env flags: TS_Q4E_PROFILE per-phase timing,
TS_Q4E_GDN_VERIFY to run both paths on the same input and state and report the
difference (compare the block's CONTRIBUTION, not the residual - the residual
is dominated by pass-through and hides it), TS_Q4E_GDN_MAX_LAYERS to bisect by
layer, and TS_Q4E_FUSED_FFN / TS_Q4E_FUSED_GDN kill switches.
TS_Q4E_RES_RESIDENT keeps the residual on the device across a layer so the
fused halves chain without a host round trip. OFF: it measures ~8% on decode
but currently produces wrong output at the boundary with the layers still
running op-by-op, and 8% does not buy a correctness risk.
Qwen3.8-27B plain/MTP/DFlash2 remain byte-identical and all 1675 tests pass.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The third and last per-layer kernel: hyper-connection mixer -> joint
query|gate projection -> Q/K RMS norm -> partial rotary -> KV append ->
masked attention -> sigmoid gate -> output projection -> scatter, as ONE
graph. That is the ~8 remaining submissions on each of the 12 full-attention
layers collapsed into 1, and it completes the set - every layer of the model
now runs as at most two graphs instead of ~18.
prefill (58 tok) 197.9 -> 389.0 tok/s
prefill (1549 tok) 861.7 tok/s
decode 47.9 -> 57.1 tok/s
Against llama.cpp PR #27742, SAME 1549-token prompt on the same weights:
prefill 861.7 vs 1353.1 (64%)
decode 50.7 vs 66.4 (76%)
Measure prefill on a long prompt, not a short one. On a 58-token prompt this
reads 389 against llama.cpp's 116, but at that size llama.cpp's fixed
per-request cost dominates and the two engines do not even tokenize the prompt
to the same length, so the ratio says nothing about the kernels. The long
prompt is the honest comparison and llama.cpp is still ahead on both axes.
The causal mask is built host-side and passed in as an F16 input, which keeps
the policy where it is cheap to express and the graph shape stable. The cache
entry keys on n_kv and position as well as shape, because an attention graph's
topology follows the context length rather than staying fixed like the other
two.
QSA is still declined rather than approximated: past
indexer_top_k + compress_ratio - 1 cached tokens the fused path returns false
and the op-by-op path takes over, which warns. At or below the budget the
selection is every cell, so dense attention is exact.
TS_Q4E_FUSED_ATTN=0 falls back, as with the other two.
TS_Q4E_RES_RESIDENT (keeping the residual on the device across a layer) is
still OFF. With every half fused it is now worth ~10% - 57.1 -> 62.6 - but it
still produces wrong output, so it stays behind the flag. Chaining all 48
layers into a single graph removes the intermediate transfers by construction
and is the better answer than debugging this one.
Qwen3.8-27B remains byte-identical and all 1675 tests pass.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three small changes to what a replay costs, and - more usefully - a measured
answer to where a decode token actually goes.
* Stamp every persisted graph with a stable non-zero ggml_cgraph::uid.
ggml_new_graph leaves it at 0, which ggml-cuda reads as "unknown", so on
every replay it re-walked all ~30 nodes comparing a copy of each tensor
struct and its sources to decide whether the captured CUDA graph was still
valid. With an id it recognises the graph and skips the walk. +1.4%.
* Route the MoE through ggml_argsort_top_k rather than ggml_top_k, which is
the exact node shape llama.cpp's build_moe_ffn emits. ggml-cuda's topk_moe
fusion matches on the node sequence rather than on intent. +1.9%.
* Drop the ggml_backend_synchronize immediately before each
ggml_backend_tensor_get. On CUDA the get already drains the stream, so it
was a second full drain 96 times a token. No measurable change, but it was
redundant either way.
What the measurements say about the rest, since the obvious structural story
turned out to be wrong:
* CUDA graph capture is ALREADY engaging - 85 warmups complete and 1 reset
over a decode run - so the per-layer graphs are being launched as captured
graphs, not as loose kernels.
* Keeping the residual on the device removes 96 host round trips a token and
is worth ~10%, not the majority.
* llama.cpp spends 0.25 ms per layer where this spends 0.36 - 1.45x, not the
order of magnitude a dispatch-bound path would show.
So the bulk of a decode token is real per-layer GPU work at batch 1, and the
remaining structural lever - chaining all 48 layers into one graph - is worth
roughly the per-graph_compute overhead, on the order of 10-15%, rather than
the 45% that would close the gap to llama.cpp. Recording that here so the next
attempt starts from the measurement instead of the assumption.
decode 47.9 -> 59.0 tok/s (llama.cpp 84.1, 70%)
prefill 382.8 tok/s at 58 tokens, ~861 at 1549 (llama.cpp 1353 there)
Qwen3.8-27B MTP remains byte-identical and all 1675 tests pass.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The mask fill pinned per row:
ushort* row = null;
fixed (ushort* m = _attnMask) row = m + (long)t * totalLen;
for (int j = 0; j < totalLen; j++) row[j] = ...;
The fixed statement ends at the assignment, so every write went through a
pointer into an array the GC was free to move by then. It survived because the
mask is small and the window between pinning and writing is short, but it was
undefined behaviour on every decode token. One fixed block around the whole
fill.
The rest of this is a negative result worth recording, because the idea is
sound and someone will try it again.
Decode profiling said the attention half was the one place still rebuilding its
graph every token: the cache keys on n_kv and position, both of which move each
step, so all 12 attention graphs were torn down and rebuilt - new context, new
bindings, a fresh gallocr plan - 12 times a token, and a graph that is rebuilt
is never a graph ggml-cuda has captured. llama.cpp avoids this by padding the
KV window to a stride so the topology only changes every N tokens, with the pad
masked off.
Implemented: window padded to a stride, position and the KV write row turned
into I64/I32 graph INPUTS, ggml_cpy-into-a-view replaced by ggml_set_rows, and
the KV write moved ahead of res_out in node order (nothing in res_out's tree
depends on the write - k_full is a plain view and ggml does not treat view
aliasing as an edge - so node order was the only thing sequencing the write
against the read, and this token has to be able to attend to itself).
It is worth a lot and it is wrong:
stride 1 (exact window) 57.4 t/s clean answer
stride 64 67.3 t/s rambles, eventually right
stride 256 65.3 t/s incoherent
Quality degrades with pad width, which should be impossible: pad columns are
masked -inf and their K/V rows are zero, so they contribute nothing. Ruled out,
each by measurement rather than argument:
* K/V content - zeroed the host buffer through the same pointer the kernel
binds, then zeroed the device tensors directly at position 0. No change.
* The mask pin above. Fixed first, no change.
* The graph uid stamp telling ggml-cuda to skip its staleness check. A/B with
it off is byte-identical to on.
* Reading past the cache. Capacity equals the allocation here, and the pad is
clamped to it.
* Prefill. All three paths agree token-for-token at the start; divergence
only appears as decode proceeds.
Reverted to the exact window rather than shipping 14% for degraded output. The
kernel is back at its committed state; only the pinning fix stays.
Two things a next attempt should look at first, both untested: whether
mul_mat's reduction over a padded ne0 reorders the summation enough to flip MoE
routing under greedy decoding, and TryFillAttnArgs caching KCache/VCache
pointers and KvBytes for the process lifetime - GrowCache reallocates those
tensors and nothing invalidates the cached args, which is latent today because
an exact window never reads past the old length.
Qwen3.8-27B MTP byte-identical, all 1675 tests pass.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Read llama.cpp's qwen4exp (llm_graph_context::build_attn_mha) and the answer to
last session's puzzle was sitting in it: llama.cpp runs ggml_flash_attn_ext
here, not soft_max_ext plus two mul_mats.
That single substitution removes, per attention layer per token:
* the [n_kv, n_tokens, n_head] scores tensor, never materialised;
* ggml_cont(ggml_permute(v_full)), which copied the ENTIRE V window - at 1542
tokens that is 1542*256*2 elements a layer, every token;
* the trailing cont(permute) on the result, because flash attention already
returns [head_dim, n_head, n_tokens], the layout the gate and the output
projection want.
It also explains why padding the KV window degraded output last session. The
padding is not a free-standing trick - it exists to serve flash attention:
ggml-cuda only selects its GQA-optimised kernel when K->ne[1] % FATTN_KQ_STRIDE
== 0, which is why llama.cpp's get_n_kv rounds to max(n_pad, 256). Paired with
soft_max the padding buys nothing and pays for every padded column; paired with
flash attention the masked blocks are skipped and the output is correct. So the
window is padded only when flash attention is actually in use, and with the
topology now stable the graph persists across a whole 256-token block instead of
being rebuilt 12 times a token.
Alongside: the KV write is ggml_set_rows with the row as an I64 INPUT rather
than a baked view offset (position and the write row change value without
moving the shape), and it is expanded ahead of res_out - nothing in res_out's
tree depends on it, since k_full is a plain view and ggml does not treat view
aliasing as an edge, so node order is the only thing sequencing the write
against the read, and this token has to attend to itself.
decode, 58-token context 59.5 -> 62.2 t/s
decode, 1542-token context 66.4 t/s
prefill, 1542-token context 95 -> 855 t/s
That last row is a regression this repairs, not a win invented here: 43a31db
fused the attention half onto soft_max_ext and I benchmarked it only on a
58-token prompt, so I did not see that materialising a [1542, 1542, 24] score
tensor per layer had made long-context prefill ~9x slower. It is back to where
it was before that commit.
llama.cpp on the same box, same weights: pp1536 2896.98, tg128 92.39 t/s.
TS_Q4E_FLASH_ATTN=0 restores the soft_max path (and with it the exact window);
it is also taken automatically for non-F16 KV or an unsupported head size.
ggml_cpu still runs. Qwen3.8-27B MTP byte-identical, all 1675 tests pass.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Retested keeping the residual in the kernels' device buffer across a layer, now
that the attention half is fused too. It is worth far more than the 8% the old
note claimed - 73.3 vs 62.0 t/s on decode, ~18% - and it still produces wrong
output, so it stays off. What changed is that the note explaining why is no
longer wrong.
It blamed the hand-off with layers still running op-by-op. Bisected:
fused GDN + op-by-op FFN, resident -> correct
op-by-op GDN + fused FFN, resident -> correct
fused GDN + fused FFN, resident -> garbage
Each kernel's residency is right on its own. It is the two persisted graphs
alternating that breaks. Forcing the residual down to host memory and back
between the two halves does not fix it either, which rules out the shared
device-buffer hand-off and the residual values themselves - by that point the
data has made a full round trip through the host and is provably correct. The
ggml-cuda graph-uid stamp is also ruled out: TS_Q4E_GRAPH_UID=0 is byte-identical
to =1 with residency on.
That leaves something the two kernels share besides the residual. The two
candidates are the shared g_q4e_res_buf binding - roughly 96 tensors across the
persisted graphs all bound to the same base address, so a q4e_res_ensure realloc
would strand every earlier binding - and ggml-cuda's single captured-graph slot
per context, which two graphs alternating every layer will thrash.
Also adds TS_Q4E_GRAPH_UID as a kill switch, which is how the uid was ruled out.
Verified after the change, both correct:
58-token context prefill 384.8 decode 58.8 t/s
1542-token context prefill 824.4 decode 66.1 t/s
All 1675 tests pass.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…real fixes
The goal was llama.cpp's shape: one GGML graph per token instead of 96. This
lands the machinery - and turns it OFF by default, because one unresolved
defect survived a very long bisection. What ships enabled is everything the
work proved safe.
Structure:
* The three per-layer kernels now build their nodes through shared builders
(q4e_nodes_ffn / q4e_nodes_gdn / q4e_nodes_attn), so a graph half has one
source of truth. The per-layer entries reproduce their old behavior
exactly - verified byte-stable at 58 and 1542-token contexts.
* TSGgml_Qwen4ExpTokenSpan chains layers [begin, end) - both halves each -
into ONE persisted graph: residual in, masks/positions/KV-rows as inputs,
GDN state carried through per-layer state buffers shared with the
per-layer path, attention KV via set_rows expanded ahead of the reads.
A token is 2 launches (PLE is the only host interruption), or ~26 in the
hybrid where attention halves run per-layer.
* TS_Q4E_TOKEN_GRAPH=1 opts in. Measured on the VM (RTX PRO 6000, UD-Q2_K_XL):
decode 62 -> 78-84 t/s, prefill 385 -> 500+ (llama.cpp: 92.4 / 2897).
Fixes that apply to the per-layer path too, found by the span's failures:
* A weight uploaded into a gallocr-owned leaf MUST carry the INPUT flag:
gallocr reuses an unflagged leaf's memory after its last consumer, so the
first compute overwrites the weight in place and every REPLAY of a
persisted graph reads garbage. A rebuilt-per-token graph works exactly
once, which is precisely why this class of bug hides.
* The device K/V copies are zeroed at position 0: the flash window is padded
past the written rows, the host buffer the caches upload from has no zero
guarantee, and a non-finite value in a masked-off column NaNs the whole
softmax row.
* Replays re-resolve their cache-bound weights and rebuild if any device
copy moved - the graph-uid stamp tells ggml-cuda to skip the staleness
walk that might otherwise have noticed.
* The GDN state copies are preceded by a backend synchronize: tensor_copy
goes through a per-thread stream, ggml-cuda's compute stream is
non-blocking, and nothing else orders the copy behind the graph.
The open defect, instrumented to the node and recorded for the next attempt:
a REPLAYED span computes ggml_gated_delta_net slightly wrong - inputs match a
fresh build of the identical graph bit-for-bit (residual, conv state, ssm
state, projections all traced at %.9e), gdn_out does not, the error scales
with context length, and generations decay into repetition. Rebuilding every
token is correct at every length; per-layer graphs - the same builder's nodes
- replay correctly at every length. Measured and ruled out: CUDA graph
capture, ggml-cuda fusion, resident-cache movement, uploaded-leaf reuse, the
KV pad and its zeroing, stream races around the state copies, in-place state
writes, shared vs private input tensors, prefill-vs-decode interaction. The
residue is specific to replaying a multi-layer graph.
Also parked, documented in-code: attention inside a span amplifies the
FA-vs-exact kernel-rounding drift into degenerate text (per-layer attention
with the same pad and kernels does not), so the hybrid keeps attention halves
per-layer; and in-graph in-place state writes remain wrong under conditions
where every ordering and capture explanation fails.
Defaults verified unregressed: short 388/62 t/s, long 834/65 t/s, ggml_cpu
8.1 t/s, all correct; Qwen3.8-27B MTP byte-identical; 1675 tests pass.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The blocker that kept the one-graph-per-token path opt-in is root-caused, and
it was one line. ggml_gallocr_free_node exempts ONLY output-flagged tensors
("graph outputs are never freed"); the INPUT flag governs early allocation and
the free path ignores it. So every small weight uploaded into a gallocr leaf -
the GDN dt/a biases, the ssm norm, the attention q/k norms, all under the
resident cache's 4096-byte threshold - was freed after its last consumer and
its memory handed to later intermediates. The FIRST compute of a graph read
the weight correctly, then overwrote it in place; every REPLAY read whatever
activations landed there, an error scaling with their magnitude - which is
what made it look context-dependent. Uploaded leafs now carry the OUTPUT flag
too.
That one mechanism was behind every "impossible" result of this effort:
* the replayed span computing gated_delta_net wrong with bit-identical inputs;
* attention-in-span "amplifying" FA rounding into degenerate text (q/k norm
weights corrupted) while per-layer attention stayed clean by plan luck;
* softmax-over-a-zeroed-pad failing bit-exactness it provably has;
* the historical "in-place state writes do not take effect", which this file
confidently documented as measurably wrong - the write-back was innocent.
With the flag in place, every previously-degenerate configuration verifies
clean, so the defaults flip all the way: token spans ON, attention chained
into the span, GDN state written in place inside the graph (no host copies,
no extra synchronize). The attention layers share one mask/position/write-row
tensor per span again - the private copies were bisection scaffolding - and
the rebind insurance samples every 32nd replay instead of walking every
binding every token.
On top of that, the last span now carries the FINAL hyper-connection mixer
(which IS the output norm) and the LM head, computed for the LAST token only,
returning logits directly: the managed tail used to mix every prefill
position and throw all but one away, then download the residual, then launch
the head separately. A decode token is now two graph launches and a logits
download.
Benchmarks (RTX PRO 6000 Blackwell, UD-Q2_K_XL, same box and session as
llama-bench b035e...):
decode, 1542-token context 96.4-99.8 t/s (mean 98.4)
decode, 1024 / 512 92.3 / 96.4
decode, trivial context 84.8-87.1
prefill, 1542 1366-1443 t/s
llama.cpp: tg128 92.88 +/- 1.37 (empty context); decode at 1536 context
~91.0 (from pp1536+tg128 = 859.62 combined); pp1536 2903.
Decode is AHEAD of llama.cpp at every context of 512+, ~8% at long
context. Short-context decode and prefill remain behind - recorded as the
next targets.
ggml_cpu rides the same span path: 8.1 -> 13.7 t/s. Session start for this
model was 6.0 t/s decode; this lands at ~16x that.
All 1675 tests pass; Qwen3.8-27B MTP byte-identical; 512/1024/1542-token
contexts and both prompts verify coherent, correct output.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…e 103.6
Phase-timing the 1542-token prefill answered where the 2x gap to llama.cpp
lived, and it was not the GPU: the two span computes totalled ~404 ms against
llama.cpp's ~531 ms for the whole thing. The other ~650 ms was host-side, and
609 ms of it was ONE thing - the PLE layer, a chain of six single-threaded
per-token loops (gather, projections, grouped norms, gating dots, the dilated
conv, the residual add) walking 1542 x 10240 floats each.
Two steps:
* Parallelise every per-token PLE loop (and the residual broadcast, and the
row dequants scattered over the 24 GB n-gram table). 609 -> 215 ms,
prefill 1429 -> 2304 t/s. The loops are token-independent; only the
n-gram hash keeps order, and it was 0.1 ms.
* Then make most of that work disappear entirely: the PLE block now builds
INTO the span graph - key/value projections, grouped norms, the
signed-sqrt sigmoid gate (|s| as s*sgn(s) to stay on ops every backend
has), the dilated depthwise conv as kern shifted views against a
tap-major transposed weight, and the residual add. The conv history is
persistent device state written in place, ordered behind its reads like
the GDN state. Only the hash and the table gather stay on the host - the
~320M-row table is gathered 16 scattered rows per token, which no device
get_rows serves - and the gathered rows ride in as one graph input
(11 ms at 1542 tokens).
With the PLE cut gone, a decode token is ONE graph launch end to end
(embedding to logits, PLE included), which lifted decode as well - and short
prompts most of all, since the fixed cost fell hardest there.
Same box, same session, 3 runs each (RTX PRO 6000 Blackwell, UD-Q2_K_XL):
prefill 1542 tokens 3361-3386 t/s llama.cpp pp1536: 2903.56 +/- 12.68
decode @1542 ctx 103.1-104.0 t/s llama.cpp tg128: 92.84 +/- 1.36
decode @512 / @1024 104.8 / 97.8
short prompt 769.6 prefill, 92.0 decode
ggml_cpu 13.7 t/s (PLE-in-graph runs there too)
TensorSharp is now AHEAD of llama.cpp on this model on both prefill (+16%)
and decode (+12%). The session started at 6.0 t/s decode and 8.4 t/s prefill.
All 1675 tests pass; Qwen3.8-27B MTP byte-identical; 512/1024/1542-token
contexts and the short prompts verify coherent, correct output.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Qwen3.8-Flash-Next's mmproj is the same qwen3vl_merger family as Qwen3.5-VL:
identical tensor names, fused qkv, learned 48x48 position grid, 2x2 spatial
merge, the mm.0/mm.2 projector - and in this checkpoint no active deepstack
layers. So the proven Qwen35VisionEncoder runs it unchanged (its loader
already dequantizes BF16), and the multimodal injector's Qwen3.5 pipeline -
image-pad expansion, embedding cache, the vLLM-style (T,H,W) position table -
generalizes onto the encoder rather than the model type.
What qwen4exp itself needed:
* IMRoPE in the token span. A graph whose pos tensor carries the four
sections rotates q/k with ggml_rope_multi in IMROPE mode - the op
llama.cpp's qwen4exp runs - using the GGUF's rope sections [11,11,10,0].
Text graphs keep the exact NEOX path they always had: IMRoPE with every
component equal IS NEOX, so text stays byte-stable and the two shapes
just key different graphs. KV rows, the causal mask and the QSA budget
all stay in cache coordinates, untouched by the compacted positions.
* Embedding splice before the residual broadcast; the placeholder token
IDS stay in the token array, which is exactly what the PLE hash wants -
the reference hashes input_ids where image positions hold the
placeholder, and this checkpoint's ple.image_token_id IS <|image_pad|>
(248056). PLE needed zero changes.
* Decode continues on scalar cache positions, as llama.cpp's mtmd does.
* Image prompts refuse the non-span fallbacks loudly - those rotate with
scalar positions and would be silently wrong.
* CLI: the one-shot --image path routes through the injector (which also
owns the position table), --mmproj works as before, and a *mmproj*.gguf
beside the model auto-loads. The chat template pre-pass inserts
<|vision_start|><|image_pad|><|vision_end|> for qwen4exp.
Verified against llama-mtmd-cli on the same weights and images, temperature 0:
on a synthetic scene both engines answer "a red circle upper-left, a blue
square lower-right" (TensorSharp's answer included the positions unprompted);
on a real illustration both describe the same subject, attire and floating
cubes. With an image in context: prefill 1324-1571 t/s, decode 103-106 t/s -
the same speed class as text. Vision encode is 682 ms for a 448x448 image,
mostly first-touch weight upload; embeddings are cached per file.
A coverage guard test caught the one path I missed on the first pass - the
reusable-prefix embedding queue had no Qwen4ExpModel case - which is exactly
what that test exists for.
No regressions: text long 3384 prefill / 104.7 decode byte-comparable,
short 737/90.6, ggml_cpu 13.7, Qwen3.8-27B MTP byte-identical, all 1675
tests pass.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Multi-image was mostly plumbing: the injector's Qwen-VL pipeline already
handled several images per prompt, but the one-shot CLI kept only the LAST
--image flag. It now passes every image in order, and the (T,H,W) IMRoPE
table spans them all.
Multi-turn took three real fixes:
* The multi-turn jsonl driver accepts an "images" array per turn, runs the
multimodal injector over the rendered conversation, clamps the KV reuse
boundary so it cannot split an image span, and queues the vision
embeddings and IMRoPE slice for exactly the token range being forwarded.
* qwen4exp now declares SupportsKVCacheTruncation = false - the GDN
recurrence, the PLE conv history and the n-gram history cannot rewind, so
a cached prefix is only reusable when the new prompt extends it exactly.
The inherited default of true was a latent wrong-state bug for ANY
truncating reuse plan, text or vision.
* Multi-turn prefix reuse never engaged for qwen4exp at all: its template
appends `<think>` to the generation prompt UNCONDITIONALLY (the model
always reasons), and the KV-cache prompt renderer's empty-think-block
strip and `<think>` suffix injection were gated on the thinking flag. The
re-render diverged from the cache four tokens into the first assistant
turn, so every turn re-prefilled the whole conversation. Both mechanisms
now apply for qwen4exp regardless of the flag.
And one correctness refinement over the first vision cut: IMRoPE compaction
gives an HxW image max(H,W) rotary positions but H*W cache rows, so after an
image the position stream falls behind the cache index. Decode and later text
turns previously roped at raw cache positions - a growing drift per image
that llama.cpp's mtmd avoids by advancing n_past by the compacted span. The
model now tracks that gap and text/decode forwards rope at cache position
minus gap, keeping the stream continuous across turns while KV rows, masks
and the QSA budget stay in cache coordinates.
Verified on the VM (temperature 0):
* Two-image one-shot: both images described correctly with positions
(red circle / blue square; green triangle / orange circle). Prefill
2015 t/s with two images in context, decode 103.6.
* Three-turn jsonl session: turn 2 recalls the circle color from turn 1's
image with no image attached ("Red"); turn 3 attaches a second image and
correctly identifies the shared shape across both. KV reuse: turn 2
keeps 340/365 tokens (prefill 44 ms, was 391), turn 3 keeps 413/641 -
forwarding just the new image and question.
* Interactive stdin session with /image per turn: correct counts, correct
which-is-closest-to-bottom answer, second encode 209 ms warm.
* Text-only multi-turn now reuses too (86% saved) - the renderer fix
benefits every qwen4exp conversation, not just vision.
No regressions: text long 3363 prefill / 104.6 decode with identical output,
ggml_cpu 14.0, Qwen3.8-27B MTP byte-identical, all 1675 tests pass (one
run showed the known flaky-CUDA-test singleton; two consecutive clean runs).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f1d7f399db
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Comment on lines
+2071
to
2073
| "glm-dsa" or "glm_dsa" or "glm5next" => new GlmDsaOutputParser(), | ||
| "nemotron_h" or "nemotron_h_moe" => new Qwen3OutputParser(), | ||
| _ => new PassthroughOutputParser() |
There was a problem hiding this comment.
Register qwen4exp with a Qwen thinking/tool parser and mark it as always requiring parsing. The new architecture unconditionally opens a <think> block, but this switch currently falls through to PassthroughOutputParser; moreover, when clients disable thinking, the server adapters do not instantiate any parser because IsAlwaysRequired also omits it. Consequently Qwen 3.8 responses can expose reasoning and raw </think>/tool-call markup as user-visible content.
Useful? React with 👍 / 👎.
| /// per-step state round trip - which is what made speculative decoding on | ||
| /// this trunk cost more than the plain decode it was meant to beat. | ||
| /// </summary> | ||
| private bool _fvDeviceStateCurrent; |
There was a problem hiding this comment.
Clear _fvDeviceStateCurrent whenever InvalidateVerifyCache() discards the native verify graph. After a speculative step commits recurrent state on-device, this latch is true; ResetKVCacheCore and cache-capacity growth then reset the native graph without clearing it. The next newly built verify receives deviceStateCurrent=true, so the native kernel skips uploading the freshly reset or preserved host GDN state and starts from uninitialized graph buffers, corrupting the next request or the continuation after a cache growth.
Useful? React with 👍 / 👎.
Engine comparison — TensorSharp vs llama.cpp (PR smoke)No report artifact was produced — the benchmark failed before generating results (see the workflow logs). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.