Skip to content

hexagon: enable quantized matmul for multi-sequence inputs#26113

Open
w1049 wants to merge 1 commit into
ggml-org:masterfrom
w1049:hexagon/multi-sequence-matmul
Open

hexagon: enable quantized matmul for multi-sequence inputs#26113
w1049 wants to merge 1 commit into
ggml-org:masterfrom
w1049:hexagon/multi-sequence-matmul

Conversation

@w1049

@w1049 w1049 commented Jul 25, 2026

Copy link
Copy Markdown
Contributor

Overview

This PR allows the Hexagon backend to handle a safe subset of quantized MUL_MAT cases with broadcasted src1 dimensions by flattening contiguous rows into a single 2D matrix. The existing HVX 2D matmul kernel already handled this layout, but the backend previously rejected these shapes, causing operations such as Qwen3.5's multi-sequence ssm_out projection to fall back to the CPU. It also updates the HMX path to support the flattened layout.

Fixes #26094

Additional information

Performance

Performance results for Qwen3.5-2B-Q4_0 using llama-batched-bench:

HTP NPL Overall master (t/s) Overall this-pr (t/s) Change
v73 1 83.55 ± 0.30 83.21 ± 0.28 −0.4%
v73 2 32.37 ± 0.13 52.29 ± 0.24 +61.5%
v73 4 46.08 ± 0.36 74.72 ± 2.40 +62.2%
v73 8 60.21 ± 0.50 97.42 ± 1.66 +61.8%
v75 1 87.99 ± 0.75 87.87 ± 0.44 −0.1%
v75 2 27.42 ± 0.20 48.27 ± 0.43 +76.0%
v75 4 38.41 ± 0.25 65.79 ± 0.60 +71.3%
v75 8 54.16 ± 0.41 86.49 ± 1.45 +59.7%
PP512 details
HTP NPL master (t/s) this-pr (t/s) Change
v73 1 460.49 ± 2.92 461.72 ± 1.63 +0.3%
v73 2 112.76 ± 0.52 133.29 ± 0.60 +18.2%
v73 4 129.77 ± 3.60 147.86 ± 7.88 +13.9%
v73 8 131.82 ± 0.57 152.66 ± 4.69 +15.8%
v75 1 423.51 ± 1.52 425.10 ± 2.33 +0.4%
v75 2 114.92 ± 0.60 137.19 ± 0.38 +19.4%
v75 4 112.91 ± 0.67 141.68 ± 5.36 +25.5%
v75 8 112.82 ± 2.13 139.03 ± 3.93 +23.2%
TG64 details
HTP NPL master (t/s) this-pr (t/s) Change
v73 1 11.07 ± 0.03 11.01 ± 0.05 −0.5%
v73 2 4.83 ± 0.02 8.92 ± 0.06 +84.7%
v73 4 7.48 ± 0.05 15.08 ± 0.23 +101.6%
v73 8 11.26 ± 0.19 25.02 ± 0.03 +122.2%
v75 1 11.99 ± 0.13 11.96 ± 0.07 −0.3%
v75 2 3.87 ± 0.03 7.80 ± 0.10 +101.6%
v75 4 6.12 ± 0.04 12.46 ± 0.15 +103.6%
v75 8 10.50 ± 0.01 21.50 ± 0.06 +104.8%

Graph splits

By keeping more operators on the HTP, the number of graph splits is reduced:
(but still a large number)

Details
NPL PP512 master PP512 this-pr TG64 master TG64 this-pr
1 3 3 75 75
2 51 15 123 87
4 51 15 123 87
8 51 15 123 87

Requirements

  • I have read and agree with the contributing guidelines
  • AI usage disclosure: YES, Codex for code investigation and testing

@w1049
w1049 requested a review from a team as a code owner July 25, 2026 11:11
@github-actions github-actions Bot added ggml changes relating to the ggml tensor library for machine learning Hexagon labels Jul 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ggml changes relating to the ggml tensor library for machine learning Hexagon

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Eval bug: quantized Qwen3.5 ssm_out falls back to CPU for batch>1 on Hexagon

1 participant