Fuse the FP8 promotion into MATMUL so BF16 never reaches DDR

[aravishankar-mp]

Why

The FP8 storage tier could only promote to a BF16 block output. That makes it a poor deal for the
case it exists to serve. Expanding FP8 operands to feed a multiply meant materializing both BF16
tensors in DDR, so the peak working set went up relative to simply storing BF16, and the
consumer paid extra submissions to read them back. A storage tier that costs more memory at the
point of use is not doing its job.

This admits the same explicit promotion feeding a MATMUL as a single program. The graph is unchanged
in meaning — two TOSA CAST operators still state the widening, so promotion stays explicit per
CONTEXT.md — but the backend performs it per L1 tile on the compute core. The caller binds two FP8
operands and one FP32 result; no BF16 tensor is ever allocated or transferred.

M=K=N=256	Fused	Unfused (CAST, CAST, MATMUL)
Peak DDR working set	384 KiB	640 KiB
DDR traffic	384 KiB	896 KiB
Submissions	1	3

At the tier's measured ~80 µs fixed per-submission cost, dropping two submissions is worth more than
the bytes for small shapes.

Correctness

Fusing is a placement choice, not a numerical one, and the argument is bit-exact by construction:
FP8 → BF16 is exact for every encoding, the widening uses the same decoders as the validated
CAST tier, and the multiply is the identical bf16→f32 kernels.mm the BF16 MATMUL tier already
uses. So the fused result must equal CAST-then-MATMUL bit for bit.

fused_fp8_matmul_is_bit_identical_to_cast_then_matmul checks precisely that on the NPU by running
both paths and comparing the FP32 bytes — not against a host oracle that could drift from either.

The generated MLIR confirms the intended dataflow inside a single core:

func.call @widen_a_fp8e4m3(%4, %A_bf16_scratch) : (memref<2048xui8>, memref<2048xbf16>)
func.call @widen_b_fp8e4m3(%6, %B_bf16_scratch) : (memref<2048xui8>, memref<2048xbf16>)
func.call @..._matmul_bf16_f32(%A_bf16_scratch, %B_bf16_scratch, %2)

All three symbols are defined text in the linked core ELF, so nothing is dead-stripped. The L2→L1
layout transform is expressed in elements and so is dtype-agnostic; widening afterwards is
elementwise and preserves the micro-tile ordering mm expects.

Admission

This is the first admitted tier with graph-interior values, so the dataflow is pinned tightly. Each
MATMUL operand must be produced by its own CAST; each CAST must consume a distinct block input; both
operands must carry the same FP8 encoding (the kernel instantiates one decoder); and a promoted value
that also escapes as a block output is rejected, because the fused kernel never writes BF16 to DDR.
Anything looser would let a kernel that binds two FP8 inputs stand in for a graph it does not
implement.

The advertised FP8 capability gains MATMUL, BF16 as INTERMEDIATE, and FP32 as OUTPUT — the
descriptor must cover the graph's interior as well as its boundary. fp8_capability_is_storage_ conversion_only asserted the old CAST-only surface; it is rewritten as fp8_target_never_produces_ fp8, which pins the property that actually still holds (FP8 is only ever consumed) rather than
being deleted.

Refactor note

The FP8 decoders are now shared between a new sized emitter and the standalone CAST tier's fixed
1,024-element entry points. That tier's kernel source is unchanged byte for byte (asserted
against HEAD during development), so its compiler cache key and existing on-metal evidence still
hold. This also removes the hard-coded 1024 the tier's emitter previously carried twice.

Compatibility

• No wire effect. This changes no accepted or emitted protocol bytes.

Purely additive to admission: graphs that were rejected are now admitted. No previously admitted
graph changes behavior. The advertised FP8 capability widens.

Checklist

• Does not alter payload lengths, ownership, reset, error, timeout, or feature-negotiation
  behavior — or does, and says so above.
• layout.json, vectors.json, scenarios.json, requirements.json, and performance budgets
  are still authoritative inputs, not regenerated by accident.
• Public Rust API changes: CompilerSpec::Fp8Matmul is new, and XDNA_TOSA_FP8_CAPABILITY
  advertises MATMUL plus the BF16 INTERMEDIATE and FP32 OUTPUT roles.
• No dependency, Cargo feature, or target moves platform behavior into a portable crate. No
  dependency changes at all.
• No unsafe code was added or modified.
• Deferred optional features remain unadvertised and documented as out of scope.

Verification

316 passed, 0 failed, 23 ignored across the workspace.

cargo fmt --all -- --check
git diff --check
python3 ci/check-release-policy.py
cargo clippy -p virtio-accel-xdna --all-targets --all-features --no-deps -- -D warnings
cargo test --workspace --all-targets --all-features
RUSTDOCFLAGS=-D warnings cargo doc -p virtio-accel-xdna --all-features --no-deps
python3 ci/check-performance-budgets.py --check

The offline compile path was exercised for real against the pinned v2026.08 toolchain on this host
(no NPU needed): fused_fp8_matmul_compiles_to_a_wellformed_artifact compiles the fused graph and
asserts the container binds [2048, 2048] -> [4096] for a 32x64x32 multiply — FP8 operands at one
byte per element, and no BF16 slot at all. Shapes 32x64x32 through 256x256x256 and both FP8
encodings were compiled successfully during development.

native.rs/hardware.rs are behind cfg(va_xdna), which CI never enables, so they were also
type-checked and linted with the cfg forced on via the build script's bare-lib-dir escape hatch, and
that path was confirmed non-vacuous by planting a deliberate error in the new test and watching the
check fail.

Not run: the on-metal suite. This host has no NPU or HRX, so
fused_fp8_matmul_is_bit_identical_to_cast_then_matmul is type-checked but unexecuted. It is the
tier's acceptance criterion and must pass on the reference 1022:17f0 part before this merges.
Until then the numerical claim rests on the by-construction argument above plus the verified
generated code, which is evidence but not "on metal".

Follow-ups (not in scope)

The fused kernel inherits the BF16 tier's single-worker structure and its scalar C-tile zeroing and
B re-streaming, so it does not itself close the throughput gap tracked in #149/#151 — it removes a
DDR round trip and two submissions, which is a different axis. A fused FP8 path for the INT8 tier,
and mixed-encoding operands (two decoders), are both deliberately unimplemented.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.