cuda: fuse MoE expert weighting and reduction (+3% to +9% prefill, MoE models only)

[coder543]

⚠️ This PR does not meet the contributor requirements, this is draft only unless someone else takes ownership. ⚠️

If someone wants to take ownership of this PR and drive it to completion, or just borrow ideas from it, they are welcome to. This PR is one of two that came out of a multi-hour GPT-5.6-Sol optimization session, as I wanted to experiment with it and see if it could actually find any low hanging fruit in the CUDA kernels. I was obviously involved in guiding the agent, and ensured that it did proper testing against a variety of models, but I have not had time to review every line of change.

I figured it was better to make this available in case anyone else cares enough to drive it forward, rather than throwing away the experiments entirely, but I am not requesting any reviews on this.

Overview

It makes MoE models 3% to 9% faster on my DGX Spark for the MoE models that were tested.

Requirements

• I have read and agree with the contributing guidelines: Yes, which is why this is not being submitted for review.
• AI usage disclosure: Yes, all of it.

Everything below this point is written by Codex/GPT-5.6-Sol:

---

MoE output currently applies router weights with one or two MUL nodes, creates one view per selected expert, and reduces those views through an ADD chain. Recognize that subgraph and replace it with one CUDA kernel that applies the factors and accumulates experts in the original order.

Write directly to the graph output when allocations do not overlap. When the allocator reuses an input range for the output, use a pooled temporary and one device copy so the fusion remains race-free.

DGX Spark / GB10 pp8192 benchmark:

llama-bench -p 8192 -n 0 -b 16384 -ub 2048 -r 7 -fa 1 -lm none

Values are median prompt tokens/s. Each row compares upstream 0d47ea7 with this standalone change:

north-mini-code NVFP4 3152.7 -> 3435.0 (+9.0%)
qwen3.6-35b-a3b NVFP4 2863.9 -> 3070.3 (+7.2%)
laguna-xs Q4_K 2856.1 -> 3016.7 (+5.6%)
qwen3.6-35b-a3b BF16 1601.2 -> 1674.1 (+4.6%)
qwen3.6-27b BF16 dense 938.9 -> 935.5 (-0.4%, noise)
gemma-4-26b-a4b Q4_K 3097.9 -> 3287.4 (+6.1%)
gpt-oss-20b MXFP4 4587.9 -> 4852.3 (+5.8%)
gpt-oss-120b MXFP4 2403.6 -> 2491.9 (+3.7%)

Every MoE format tested improves. Dense Qwen does not match the fusion and is performance-neutral.

Deterministic output validation in the separately tested two-patch series showed that applying this reduction to the scheduler-only control was byte-identical for all eight models above. Laguna and Gemma used their Jinja chat templates.

The standalone change builds successfully and passes all 73 NVFP4 MUL_MAT_ID backend tests.

Overview

Additional information

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure:

[am17an]

Isn't this the same as #25952?