ggml-cuda : fall back to cuBLAS when no MMQ tile size fits in shared memory - #26141
Open
KakaruHayate wants to merge 1 commit into
Open
ggml-cuda : fall back to cuBLAS when no MMQ tile size fits in shared memory#26141KakaruHayate wants to merge 1 commit into
KakaruHayate wants to merge 1 commit into
Conversation
github-actions
Bot
added
ggml
|
Hi @KakaruHayate, thanks for your contribution! Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:
Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below. |
KakaruHayate
changed the title
KakaruHayate
changed the title
…emory
ggml_cuda_should_use_mmq() decides MMQ eligibility purely from the
quantization type and never considers the device's shared memory budget.
mul_mat_q_switch_J() then skips every tile size J whose shared memory
requirement exceeds smpbo -- and when all of them are skipped, J_best stays 0,
falls through to the default case and hits GGML_ABORT("fatal error").
So on devices with a small shared memory budget, the eligibility check admits
a code path that is guaranteed to abort.
Reproduced on a Moore Threads MTT S70 (arch mp_21, 28 KiB shared memory per
block) with an RWKV-7 0.1B Q8_0 model:
$ llama-bench -m rwkv7-g1d-0.1b-Q8_0.gguf -p 128 -n 0
J_best=0
ggml/src/ggml-cuda/template-instances/../mmq.cuh:1521: fatal error
(core dumped)
Only prefill (batch > 1) is affected; token generation is fine. FP16 and
Q4_K_M work on the same device -- their MMQ configs do fit.
Fix: check in ggml_cuda_should_use_mmq() whether any J fits into smpbo, and
return false if none does, so the caller falls back to cuBLAS instead of
aborting.
After the fix on the same device:
Q8_0 pp128 1470.7 t/s, tg8 55.3 t/s (was: abort)
FP16 unchanged
Q4_K_M unchanged
This matches a -DGGML_CUDA_FORCE_CUBLAS=ON build (pp128 1464.2 t/s), which
confirms the fallback path is the one being taken.
This is not MUSA-specific -- any device with a small enough shared memory
budget can hit it.
Clarify fallback behavior for MMQ in comments
Updated comment to clarify fallback behavior for MMQ.
KakaruHayate
force-pushed
the
fix-mmq-smem-fallback
branch
from
July 27, 2026 05:02
d8776de to
1eee668
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
ggml_cuda_should_use_mmq() selects MMQ purely by quantization type
without checking the device's shared memory budget. When all tile
sizes J are skipped by mul_mat_q_switch_J() because they exceed
smpbo, J_best stays 0 and the code hits GGML_ABORT.
Fix this by verifying that at least one J fits in smpbo before
declaring MMQ as usable, falling back to cuBLAS otherwise.
Requirements