Rebuild the exact INT8 MATMUL fast path on the raw MMUL with zero-point correction (#151)

[aravishankar-mp]

Why

First optimization pass for #151, following its investigation order: profile on metal, then keep
more native INT8 work on the matrix unit.

Step 1 evidence (Peano remarks / disassembly). The shipped kernel's K-step loop body was ~200
sequential scalar instructions — one lda.s8 per element with per-element address arithmetic and
lane-by-lane vpush.hi.16 widening — feeding exactly one vmac. The matrix unit was <1% of
kernel time. Measured on the reference 1022:17f0 NPU: 340 µs submit-to-complete p50 against an
~86 µs per-submission overhead floor (the 1,024-element FP8 case), i.e. ~250 µs of almost purely
scalar kernel time.

The fix (investigation-order step 2). The exact expansion
C = A·B − zb·rowsum(A) − za·colsum(B) + K·za·zb lets the MMUL run on the caller's raw INT8
bytes — no widening, no packing. Shapes on the native 8×8×8 grid (M % 16, K % 8 with K ≥ 16, N % 16; the vectorized kernel unrolls 2×2 micro-tiles, and a single-tile K degenerates the DMA
transform into a zero step size) now take a design that is structurally the BF16 _build_matmul
(the fork's single-core design, per the OpenVINO-precedent invariants in #151):

• DMA-side micro-tile layout transforms deliver A, B, and C in MMUL order at zero core cost;
• the fork's vectorized i8/i32 mm kernel and its zero sibling compute the raw product
  (subsumed, not reimplemented);
• one small correction kernel derives both sums via MMUL against a constant ones tile and applies
  the correction in place. Every term is provably inside INT32 for the admitted envelope (K ≤ 512);
  the proof is in the kernel's docstring.

Off-grid shapes keep the scalar exact kernel (including the shared 2×3×2 corpus case); the
INT16-widening template is retired. Admission, the binding plan, and the guest ABI are unchanged —
the helper reports the identical [4096, 2048] / [8192] plan for the benchmark shape.

Result, on metal

Same NPU (1022:17f0 rev 0x20), same v2026.08 toolchain, same benchmark protocol (20 warmups, 200
measured submissions, exact-oracle validation of every output after timing):

	Aug 26 baseline	This change
submit-to-complete p50	334.065 µs	72.089 µs	4.7×
p95	343.723 µs	99.427 µs	3.5×
effective GOPS	0.785	3.636	4.6×

The p50 now sits at the measured per-submission overhead floor, so the remaining latency is
submission-path cost, not kernel cost — #151's overlap (ring depth) and striping steps stay open,
and are where the next order of magnitude lives.

A toolchain miscompile, found and worked around

The new non-ignored oracle test (below) caught a real code-generation bug in the pinned Peano
release: a default-constructed aie::mmul carries a zero-on-first-mac flag, and the
software-pipelined loop epilogue mis-rotates that flag's config register at trip count exactly 2 —
the final mac re-zeroes the accumulator, collapsing the sum to the last tile. Diagnosed from the
disassembly (vmac … r29 where mov r29, r28 shares a bundle with mova r28, #0x308, reading the
stale #0x309). The 64×64×32 benchmark could never see it (K/S = 8); a 16×16×16 shape hits it
directly. The correction kernel now constructs its accumulators explicitly zeroed
(aie::zeros<acc32, …>(), the fork's own kernels' pattern), which makes every mac's config
uniform and is correct at every trip count. The erratum is documented at the workaround site.
Note the shipped BF16/mm.cc designs are structurally immune (they accumulate via C loaded from
memory, zeroed by zero.cc).

Coverage

tosa_int8_matmul_tiled_path_matches_the_exact_oracle_on_the_npu — non-ignored, hardware-gated,
fail-loud under VIRTIO_ACCEL_XDNA_REQUIRE_HARDWARE=1 — checks the tiled path bit-exactly against
the shared dot_i8_i32 oracle across three shapes (16×16×16, 32×24×16, 64×64×32) with zero points
at both extremes ([-128,127], [127,-128], [0,-1]) and full-range inputs including −128. This
closes the review finding that the fast path was oracle-checked only by an #[ignore]d benchmark.

Compatibility

• No wire effect. This changes no accepted or emitted protocol bytes.

Admission is unchanged; which compiled template a shape takes changes (INT16-widening path
retired; shapes like 64×68×32 that took it now take the scalar exact kernel — both exact, and the
retired path's throughput was within noise of scalar). Artifact cache keys rotate with the helper
source, so no stale kernel can be served.

Checklist

• Does not alter payload lengths, ownership, reset, error, timeout, or feature-negotiation
  behavior.
• layout.json, vectors.json, scenarios.json, requirements.json, and performance budgets
  are still authoritative inputs, not regenerated by accident.
• Public Rust API changes: none.
• No dependency, Cargo feature, or target moves platform behavior into a portable crate. No
  dependency changes.
• No unsafe code was added or modified.
• Deferred optional features remain unadvertised and documented as out of scope.

Verification

All on this branch, on the reference NPU host:

cargo fmt --all -- --check
git diff --check
python3 ci/check-release-policy.py
cargo clippy -p virtio-accel-xdna --all-targets --all-features --no-deps -- -D warnings   # portable
VIRTIO_ACCEL_HRX_LIB_DIR=<dir> cargo clippy -p virtio-accel-xdna --all-targets --all-features --no-deps -- -D warnings   # va_xdna
cargo test --workspace --all-targets --all-features    # 308 passed, 0 failed
RUSTDOCFLAGS=-D warnings cargo doc -p virtio-accel-xdna --all-features --no-deps
python3 ci/check-performance-budgets.py --check

On metal with VIRTIO_ACCEL_XDNA_REQUIRE_HARDWARE=1: the full hardware suite (26 passed, 2
intentionally ignored benchmarks) run three times from a cold artifact cache (36–40 s each,
recompiling every tier through the new helper) plus warm; the shared conformance suite; and both
ignored benchmarks. One early suite run failed a single test immediately after a helper edit and
never reproduced across the three cold reruns whose logs I kept — I could not identify the test
from the discarded output, so flagging it here rather than pretending it didn't happen; the three
captured cold runs are clean.

🤖 Generated with Claude Code

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.