ggml-cpu: run the token-generation matmul without a global barrier

[codspeed-hq]

What

During token generation every matmul has a single src1 row (one token). For a quantized src0 that row still has to be converted to vec_dot_type (Q8_0/Q8_1/Q8_K/F16), and today that conversion is split across the threads and published with a global ggml_barrier() - one full rendezvous per matmul, and a transformer layer contains 9 of them. The same barrier also publishes thread 0's reset of the shared chunk counter.

Converting one row is a tiny fraction of one matmul chunk (2048 floats vs. millions of weight elements), so this change makes every thread convert the whole row into its own slot of the work buffer instead:

• ggml_graph_plan() sizes the MUL_MAT scratch as n_tasks * row_size when ggml_nrows(src1) == 1 (a few KB more),
• each thread writes and then reads only its own copy, so nothing has to be published,
• the chunking switches to one chunk per thread (nchunk0 = nth). With a single src1 row all chunks carry the same work, so there is nothing for the dynamic chunk stealing to rebalance - and because no thread ever reads the shared counter, the counter reset (the barrier's second job) is not needed either.

The barrier is therefore skipped entirely on this path. Every other shape (prompt processing, lm_head, F32 weights, mul_mat_id) keeps the existing code path unchanged, and the tinyBLAS/GGML_LLAMAFILE and KleidiAI paths are unaffected - they set up and publish their own chunk counter.

Correctness

The arithmetic is untouched: same kernels, same operand order; only who converts src1 and how the row chunks are handed out changes.

Verified locally against a build of the unmodified tree with a harness that dumps the raw result buffers and compares them byte-for-byte: 450 cases byte-identical - F32/F16/Q8_0/Q4_0/Q4_K x 9 shapes (including n = 1/2/3/5/16 columns and batched/broadcast 3-D shapes) x 1/2/5/8/13 threads, each shape run both as a single matmul and as a chain of 3 matmuls executed 3 times with changing inputs (so a skipped or double-processed chunk, or a stale chunk counter leaking into the next matmul, would show up).

test-backend-ops -o MUL_MAT, test-barrier, test-quantize-fns and test-rope also pass.

Measured (CodSpeed, CPU simulation, 8 threads)

Macro suite, base vs. head (overall impact +18%):

Benchmark	Base	Head	Change
decode_deep[q4_k]	584.7 ms	476.0 ms	+22.8%
decode_layer[q8_0]	162.7 ms	139.1 ms	+16.9%
decode_layer[q4_0]	154.6 ms	134.0 ms	+15.3%
decode_layer[q4_k]	150.4 ms	137.3 ms	+8.7%
decode_layer[f32] (control, unchanged path)	266.3 ms	265.2 ms	unchanged
prompt_layer[*], lm_head[*]			unchanged

The single-threaded micro suite (15 benchmarks, deterministic) is unchanged across the board - no regression on the prompt/matmul kernels.

Caveat

Most of the gain is removed synchronization. Under CPU simulation the threads are serialized by Valgrind, which makes the OpenMP runtime spin far longer than on real hardware, so the walltime numbers from the macro runner will be smaller than the ~20% above. What transfers to the macro runner is 9 fewer rendezvous per decoded token, minus the cost of converting one row nth times instead of once (a few thousand instructions per matmul).

[codspeed-hq]

Merging this PR will not alter performance

✅ 28 untouched benchmarks

---

_{Comparing codspeed-optim-run-the-token-generation-matmul-without-a-global-b-1785433054815 (9cc9f20) with master (46819c9)}