bench: decode-regime GEMV bandwidth vs the read ceiling, with a frozen A/B baseline

[terrizoaguimor]

bench_idot measures the idot kernels at their warm-cache compute ceiling. Decode lives in the
opposite regime: expert weights exceed every cache, each token streams them from RAM at
~1 MAC/byte, and the number that decides any kernel work is how close the real row loops get to
the machine's read bandwidth. This bench measures that — the real quant.h row loops, their own
internal OMP, over a working set that defeats the LLC, with a parallel-read ceiling in the same
process so the ratio cancels host noise.

What it measured on the way in

Two AVX2-only cloud hosts (no VNNI), 2/4/8 threads, medians of 5, ratios within one pass:

	share of the same-thread read ceiling
idot int8 (dot_i8i8)	~73%, every width
idot int4 (dot_i4i8, the GLM fmt-4 decode kernel)	~47%, every width

So on the most common consumer ISA, int4's halved bytes buy 1.28–1.46× tokens instead of ~2×
— the unpack path eats the rest. Three follow-ups, all in the bench:

• Multi-accumulator restructure of the plain-AVX2 branch (mirroring what the AVX-VNNI and
  NEON siblings already carry): 1.00× cold, exactly nothing. The VNNI branch's own comment
  names its win — breaking a ~5-cycle vpdpbusd chain — and the plain-AVX2 chain is a 1-cycle
  add the OoO already hides. Discarded; recorded here so nobody re-walks it.
• Deinterleaved-x candidate: split x even/odd once per call (amortised over every row of
  the projection), so lo/hi nibbles are used straight after and/srli — no unpacks, no lane
  insert. Warm ALU ceiling 1.25–1.54×; bit-exact against the live kernel on sampled rows
  plus I=1..200 ragged tails, checked inline every run.
• Cold, on shared vCPUs: only 1.02–1.09×. And there is the cap on what cloud hardware can
  say: the read ceiling scaled with thread count (30→60→110 GB/s at 2/4/8 threads) — a guest
  never sees the physical RAM roof, so whether the candidate pays at production thread counts
  is a question only physical AVX2-only silicon can answer (Zen 1–3, pre-Ice-Lake Intel —
  the Zen3 data: SMT collapse (2.3× at 16 threads), ±15% run variance, PIN_GB starves LRU — HIP/gfx1100 + NVMe #718 5950X is exactly the right machine).

What this PR adds

The bench only — no production code changes. It ships a frozen verbatim copy of today's
plain-AVX2 dot_i4i8 (any future kernel change A/Bs against it in one process) and the
deinterleaved candidate, both gated for bit-exactness inline. Build on demand, not in
TEST_BINS, per the bench_idot / bench_topp convention:

make tests/bench_gemv_stream ARCH=native && OMP_NUM_THREADS=N ./tests/bench_gemv_stream

If a physical-silicon run shows the candidate holding its warm-side win in the cold regime, I
will send the quant.h change as its own PR — matmul_i4_idot plus the three PIPE call sites
in colibri.c, with test_idot extended to cover the deinterleaved pair. If it collapses to
the cloud numbers, the frozen baseline still leaves the next person a one-command A/B and this
table as the reason nothing was changed.

🤖 Generated with Claude Code