[Bug]: RuntimeError: Rowwise scaling is not currently supported on your device

[voidhaze]

Running Win 11 with a 4070 Super, was trying to run Qwen3.6-35B-A3B NVFP4; when the model is loading / starting it gets to about 60% and then crashes out with the error:

"RuntimeError: Rowwise scaling is not currently supported on your device"

Logs attached

crash_log.txt

[Buccolieri96]

Same

[Thomaswang0822]

Same here, Qwen3.8-27B-NVFP4 on 5090

[oro872gioioso]

Same with 5060Ti 16GB. Still no solution?

[Usigned]

same, Qwen3.6-35B-A3B-NVFP4 on 4090D

[qualitymaterial]

Same on a 5070, and I think this is the cause — it isn't NVFP4- or model-specific.

Rowwise _scaled_mm is a Hopper-only CUTLASS kernel, restricted to sm_90 in pytorch/pytorch#145728. But fp8_pertensor_linear() dispatches to it on e4m3_native() (cuda_capability_geq(8, 9)) — that's the fp8 tensor core capability, not the rowwise capability. So every consumer card fails, sm_89 and sm_120 alike, which matches the 4070S/4090D/5060Ti/5090 reports above. Probed on my 5070 (torch 2.11.0+cu130): tensorwise OK, rowwise raises.

Which weights hit it is decided in Fp8PerTensorLinear.load_state_dict:

self._uniform_scale = bool((scale == scale[0]).all().item())

A real per-tensor weight is uniform and takes the safe tensorwise path. A fused projection is piecewise-constant, so it's never uniform and always takes rowwise — hence GDN in_proj_qkvz (qwen3_5_moe/gdn.py:165). Only that and glm_moe_dsa/attention.py can trigger it. It fires during CUDA graph capture, so it kills the scheduler process instead of failing a request — the daemon stays up and the app reports it as the engine failing to start.

Fix: the W8A16 triton fallback already in that file handles this correctly, so it just needs the fused case routed to it when rowwise is unavailable —

-    elif input_scale is not None and e4m3_native():
+    elif input_scale is not None and e4m3_native() and (
+        uniform_scale or _rowwise_scaled_mm_supported()
+    ):

where _rowwise_scaled_mm_supported() is an lru_cache'd probe that tries a 16x32 rowwise _scaled_mm once and returns False on RuntimeError. A static cuda_capability_eq(9, 0) would do too, but the probe keeps working if PyTorch widens support. Happy to post the full patch or open a PR.

Caveat: I verified this at unit level only — probe returns False on sm_120, the fused-scale Fp8PerTensorLinear forward now returns finite output at M=1 and M=4, rel_err ~3.3e-3 vs a bf16 dequant reference. I have not confirmed a full model boots, since the checkpoint was removed from my models dir after the crash. Someone still holding Qwen/Qwen3.6-35B-A3B-FP8 would need to confirm it clears Start capturing CUDA graphs. No throughput numbers either.