ggml-cpu: fuse element-wise epilogues (ADD, MUL, SILU) into ggml_compute_forward_mul_mat

[codspeed-hq]

What

The CPU graph executor already fuses RMS_NORM + MUL (ggml_cpu_try_fuse_ops, with a TODO inviting more variants). This extends that framework to the matmul: when a MUL_MAT node is immediately followed by a node that only consumes its result element-wise, the operation is applied straight out of the accumulator tile in ggml_compute_forward_mul_mat_one_chunk() instead of being run as a separate graph node.

In one transformer layer this fires 4 times:

Fused pair	Where
MUL_MAT + ADD (×2)	the attention and FFN residual connections
MUL_MAT + SILU	the SwiGLU gate projection
MUL_MAT + MUL	the gate × up product

Each fusion removes:

• a full read + write pass over the matmul output, and
• a graph node, and with it one global thread barrier — decode graphs spend a large share of their time synchronising, not computing.

Guards

Fusion is refused unless the epilogue operands are F32, identically shaped (no broadcast), row-contiguous and non-transposed, and it is skipped for the Hadamard hint. ggml_can_fuse() additionally guarantees the matmul result has exactly one use and is neither a view nor a graph output.

tinyBLAS is preserved. The GGML_LLAMAFILE kernels write dst themselves, so the epilogue cannot be folded into the accumulator tile there. Rather than skipping tinyBLAS (which measured ~2× slower prompt processing for F32/F16 in a GGML_LLAMAFILE=ON build — the default for full llama.cpp builds), those paths run the epilogue as a second pass over the result. That keeps tinyBLAS and still saves the graph node.

Correctness

The epilogue reuses the same ggml_vec_add_f32 / ggml_vec_mul_f32 / ggml_vec_silu_f32 kernels on the same values. Verified by running chained residual + SwiGLU graphs with GGML_CPU_DISABLE_FUSION=1 and =0 and comparing the raw output bytes: 252 tensors over F32 / F16 / Q8_0 / Q4_0 / Q4_K weights, 8 shapes (including non-block-aligned ones), 1 and 4 threads, with and without GGML_LLAMAFILE.

• ADD and MUL epilogues are bit-identical.
• SILU epilogues are bit-identical when the accumulator tiles align with the vector width, and otherwise differ by at most 2 ulp (max observed relative difference 2.7e-07). This is because ggml_vec_silu_f32() uses a vectorized exp approximation for full vector lanes and the scalar ggml_silu_f32() for the tail, so a row split into tiles hits the scalar tail at different offsets. Same kernel, same values, only a different vector/tail boundary.

Repository tests run: test-rope, test-quantize-fns, test-opt, test-barrier, test-alloc, test-col2im-1d — all pass. test-backend-ops skips the CPU backend when it is the only device. (test-export-graph-ops fails on the base commit too: it requires a model file.)

Measured (CodSpeed, CPU simulation)

Macro suite — impact +11.7%, no regressions:

Benchmark	Base	Head	Change
decode_deep[q4_k]	591.3 ms	525.7 ms	+12.5%
decode_layer[q8_0]	168.1 ms	151.6 ms	+10.9%
decode_layer[q4_k]	155.1 ms	145.1 ms	+6.4%
decode_layer[q4_0]	151.8 ms	144.4 ms	+4.9%
decode_deep[f32]	1.04 s	986.8 ms	+5.1%
decode_layer[f32]	263.4 ms	252.1 ms	+4.3%
prompt_layer[q8_0]	1.02 s	992.6 ms	+2.7%
prompt_layer[q4_k]	801.8 ms	786.4 ms	+1.9%
prompt_layer[q4_0]	1.08 s	1.06 s	+1.9%
lm_head[*], prompt_layer[f32]	—	—	unchanged

Micro suite (15 benchmarks): unchanged (impact -0.03%), no regressions.

Caveat

A large share of the decode-side gain is the 4 removed global barriers per layer. Under CPU simulation threads are serialised, which makes barriers disproportionately expensive, so the walltime macro run on the CI macro runner should be expected to show a smaller improvement than 12% — the part that transfers is the removed memory passes plus 4 fewer synchronisation points per layer. The change is architecture-independent, so it applies to the aarch64 walltime runner as well.

[codspeed-hq]

Merging this PR will not alter performance

✅ 28 untouched benchmarks

---

_{Comparing codspeed-optim-fuse-element-wise-epilogues-add-mul-silu-into-ggml-1785427000468 (6542414) with master (46819c9)}