[BUG] qwen3_5 dense: mixed-precision NVFP4 crashes in ct_bf16_fuse (Float8 × BFloat16 promotion)

[zhaochang22]

[BUG] qwen3_5 dense checkpoints: mixed-precision NVFP4 (FP8 attention + NVFP4 MLP) crashes in ct_bf16_fuse — Float8_e4m3fn × BFloat16 promotion

Environment

• FreeToken 0.1.2 (PyPI wheel freetoken-0.1.2-cp312, kernel-cache 0.1.2+cu130)
• Python 3.12.13 (uv-managed venv), Ubuntu 22.04, driver 580.119.02
• GPU: 1× NVIDIA RTX 4090 (also reproduced logic on a 2×4090 box; TP=1)
• Checkpoint: unsloth/Qwen3.8-27B-NVFP4 (llm-compressor export, quantization_config.format == "mixed-precision")

Summary

Serving a Qwen3.8-27B dense checkpoint (qwen3_5 family) quantized with the standard NVFP4 recipe — attention/GDN projections FP8, rest NVFP4 — crashes during weight loading. Both ft serve and ft checkpoint fail identically, which suggests the shared compressed-tensors walker lacks a dequant branch for Float8-quantized linears when they land in a bf16 fuse group.

Reproduction

ft serve --model unsloth/Qwen3.8-27B-NVFP4 --model-source modelscope
# same failure with:
ft serve --model unsloth/Qwen3.8-27B-NVFP4 --model-source modelscope --nvfp4-backend flashinfer
ft checkpoint --model <snapshot-dir> --out <out-dir>

Error

RuntimeError: Promotion for Float8 Types is not supported, attempted to promote Float8_e4m3fn and BFloat16

Key frames (identical for serve and checkpoint):

freetoken/engine/engine.py(459)        _load_weight_state_dict
freetoken/models/weight.py(251)        load_weight
freetoken/models/qwen3_5_moe/weight.py(648)  _iter_weights_compressed_tensors
freetoken/models/qwen3_5_moe/weight.py(589)  _emit_bf16_weight
freetoken/models/loader.py(251)        ct_bf16_fuse  →  torch.cat([...], dim=0)

Analysis

• The checkpoint's config.json reports quantization_config.format == "mixed-precision" with 303 ignored modules — the usual llm-compressor NVFP4 recipe (FP8 for attention/GDN linears, NVFP4 for the rest).
• The NVFP4 (weight_packed 4-bit) parts dequantize fine — loading progresses ~5s deep before failing.
• The crash happens when a fuse group for GDN in_proj is concatenated: one member tensor is still raw Float8_e4m3fn (an FP8-quantized linear that the compressed-tensors walker passed through unconverted), and torch.cat refuses to promote Float8 against BFloat16.
• --nvfp4-backend triton|flashinfer makes no difference (failure is upstream of kernel backends).

Expected behavior

Either (a) Float8-quantized linears in compressed-tensors checkpoints are dequantized to the compute dtype before fusion/emit, or (b) a clear early error naming the unsupported mixed format.

Notes

• A single-recipe export (Qwen/Qwen3.8-27B-FP8, pure FP8) presumably takes the mainline FP8 path and is unaffected — not yet tested.
• Side note: docs/models.md does not list Qwen3.8 yet, though the Desktop catalog does; might be worth a docs refresh alongside the fix.
• Happy to test a patch.

[zhaochang22]

Cross-referencing two related reports found while scanning the tracker:

• Loading nvidia/Qwen3.6-35B-A3B-NVFP4 fails: "Incomplete fp8 fusions: ['model.layers.37.linear_attn.in_proj_qkvz']" #206 (fixed 2026-08-26): the same GDN fuse group (linear_attn.in_proj_qkvz) failing on FP8 content, but for nvidia/Qwen3.6-35B-A3B-NVFP4. That fix appears not to cover the dense qwen3_5 path with an unsloth mixed-precision export (quantization_config.format == "mixed-precision": FP8 attention/GDN linears + NVFP4 rest), which is exactly where this issue crashes — ct_bf16_fuse → torch.cat hitting Float8_e4m3fn × BFloat16 promotion.
• Qwen3.8-27B-NVFP4 fails to load — RuntimeError: CUDA driver error: device not ready #139: the same checkpoint (unsloth/Qwen3.8-27B-NVFP4) failing with a different symptom (CUDA driver error: device not ready); possibly the same loader gap surfacing differently on that setup.

Happy to test a fix for the dense mixed-precision path.