Two blocking bugs in Windows Desktop on consumer Blackwell (RTX 50): FP8 row-wise gate crash + Mistral tokenizer empty Chinese output

[xiaoma-quant]

FreeToken Windows Desktop 在消费级 Blackwell（RTX 50 系）上的两个阻断性 Bug

场景：Windows 11 + RTX 5070 Ti（Blackwell sm_120）+ FreeToken Desktop 0.1.2
对照：同一模型在 WSL2 内（同一 FreeToken 0.1.2 / 同一 torch 2.11.0+cu130）完全正常，说明问题出在 Windows Desktop 原生运行时的 CUDA / tokenizers 路径。

---

Bug 1 — 消费级 Blackwell 上 Rowwise scaling is not currently supported 崩溃

环境

• OS: Windows 11
• GPU: NVIDIA RTX 5070 Ti（Blackwell，compute capability sm_120）
• FreeToken: 0.1.2（Windows Desktop，引擎装在 C:\Users\<user>\AppData\Local\FreeToken）
• 捆绑 torch: 2.11.0+cu130，torch.cuda.get_arch_list() == ['sm_75','sm_80','sm_86','sm_90','sm_100','sm_120']（含 sm_120，硬件 FP8 张量核可用）
• 驱动: R595.79（≥ R570，满足要求）
• 模型: Qwen3.6-35B-A3B-NVFP4（quant_method: modelopt，原始 NVFP4 目录）

复现步骤

1. 导入本地 NVFP4 模型到模型库（拖入或指向目录）
2. 点「转换」→ 生成 FTW（NVFP4→FTW 转换本身成功）
3. 点「启动」→ serve 崩溃
4. 也可绕过 FTW 直接复现：ft serve --model <原始NVFP4目录> 同样崩溃

报错

RuntimeError: Rowwise scaling is not currently supported on your device

调用栈（节选）：

freetoken/models/qwen3_5_moe/gdn.py:165  qkvz = self.in_proj_qkvz.forward(hidden_states)
freetoken/kernel/triton/fp8_pertensor_linear.py:323  return fp8_pertensor_linear(...)
freetoken/kernel/triton/fp8_pertensor_linear.py:273  out = _scaled_mm(...)
freetoken/kernel/triton/fp8_pertensor_linear.py:244  return torch._scaled_mm(   # row-wise 路径
torch._scaled_mm  ->  RuntimeError: Rowwise scaling is not currently supported on your device

根因（读源码定位）

• gdn.py 注意力层 if self._fp8: 强制把融合投影 in_proj_qkvz 走 fp8_pertensor_linear
• 融合投影（qkvz 拼在一起）的 weight_scale 是 piecewise-constant → Fp8PerTensorLinear._uniform_scale = False
• _scaled_mm wrapper：uniform_scale=True → tensor-wise（RTX 5070 Ti 支持）；uniform_scale=False → row-wise（scale 形状 [1,-1]）→ RTX 5070 Ti 不支持 → 崩
• 消费级 Blackwell（RTX 50 系）的 cuBLASLt 在 Windows 原生运行时下关掉了 row-wise FP8 scaled_mm（数据中心卡 sm_90 如 H100 支持）
• 对照：同一 GPU、同一代码，在 WSL2 内 serve 同一模型正常

已验证的规避方案（非满速）

设置 FREETOKEN_FORCE_E4M3_EMU=1 → 跳过 torch._scaled_mm，改走 triton W8A16 GEMM。

• ✅ 模型能加载并 serve
• ❌ 代价：慢 1.5–2.2×
• ❌ 仍触发 Bug 2（中文输出空）

建议修复

1. 当 row-wise torch._scaled_mm 在当前设备不可用时，fallback 到已有的 triton W8A16 _gemm/_gemv 路径（这些路径本来就在代码里，只是 e4m3_native() 为 True 时被跳过），而非抛 RuntimeError
2. 或在加载期用 try/except torch._scaled_mm 探测设备是否支持 row-wise，不支持则自动走 emulated 路径
3. 或在消费级 Blackwell 上把融合投影的缩放改为 tensor-wise，复用已支持的 tensor-wise scaled_mm

---

Bug 2 — Mistral 系 tokenizer 未设 fix_mistral_regex → 中文 decode 成空串

环境

• 模型 Qwen3.6-35B-A3B-NVFP4（使用 Mistral 系 tokenizer）
• 捆绑 transformers: 5.16.1

复现步骤

1. 在 emu 模式下 serve 起来（绕过 Bug 1）
2. 英文提问 → 正常返回
3. 中文提问 → 返回 finish_reason=length、completion_tokens=59 但 content 为空字符串

加载期警告

The tokenizer you are loading ... with an incorrect regex pattern ...
You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.

根因

• Qwen3.6 的 tokenizer 是 Mistral 系，正则 pattern 错误 → 中文（CJK）tokenization 错乱
• transformers 5.16.1 提供 fix_mistral_regex=True 修正，但 FreeToken 在 from_pretrained 加载 tokenizer 时未透传该参数（源码仅从 kwargs 读取，无环境变量映射）
• 结果：中文 token 被错误切分，decode 出空/不可见字符

建议修复

FreeToken 在 from_pretrained 加载 tokenizer 时，对 Mistral 系（及派生）tokenizer 自动透传 fix_mistral_regex=True（或检测未显式指定时默认开启）。

---

总结

两个 Bug 均阻断消费级 Blackwell（RTX 50 系）Windows Desktop 用户：Bug 1 使模型无法加载（FP8 row-wise gate），Bug 2 使中文输出为空（tokenizer regex）。同一模型 + 同一 FreeToken 版本在 WSL2 下两处都正常 → 根因在 Windows Desktop 原生 CUDA / tokenizers 运行时差异。

复现最小信息

• FreeToken Desktop 0.1.2 / Windows 11 / RTX 5070 Ti (sm_120) / torch 2.11.0+cu130
• 模型：任意 NVFP4（modelopt）MoE，注意力层用 GDN（如 Qwen3.6-35B）
• 步骤：导入 → 转换 → 启动（或直接 ft serve --model <nvfp4 dir>）

---

English version (for flashml.ai / GitHub)

Two blocking bugs in FreeToken Windows Desktop on consumer Blackwell (RTX 50 series)

Setup: Windows 11 + RTX 5070 Ti (Blackwell sm_120) + FreeToken Desktop 0.1.2
Control: the same model under WSL2 (same FreeToken 0.1.2 / same torch 2.11.0+cu130) works perfectly — so the issue is in the Windows Desktop native CUDA / tokenizers runtime path.

Bug 1 — Rowwise scaling is not currently supported crash on consumer Blackwell

Repro

1. Import a local NVFP4 model into the library
2. Click Convert → FTW is produced (NVFP4→FTW conversion itself succeeds)
3. Click Start → serve crashes
4. Reproducible without FTW too: ft serve --model <raw NVFP4 dir> crashes identically

Error

RuntimeError: Rowwise scaling is not currently supported on your device

Stack (abridged):

freetoken/models/qwen3_5_moe/gdn.py:165  qkvz = self.in_proj_qkvz.forward(hidden_states)
freetoken/kernel/triton/fp8_pertensor_linear.py:323  return fp8_pertensor_linear(...)
freetoken/kernel/triton/fp8_pertensor_linear.py:273  out = _scaled_mm(...)
freetoken/kernel/triton/fp8_pertensor_linear.py:244  return torch._scaled_mm(   # row-wise path
torch._scaled_mm  ->  RuntimeError: Rowwise scaling is not currently supported on your device

Root cause (from source)

• gdn.py attention layer if self._fp8: forces fused projection in_proj_qkvz through fp8_pertensor_linear
• Fused projection (qkvz concatenated) has piecewise-constant weight_scale → Fp8PerTensorLinear._uniform_scale = False
• _scaled_mm wrapper: uniform_scale=True → tensor-wise (RTX 5070 Ti supports); uniform_scale=False → row-wise (scale shape [1,-1]) → RTX 5070 Ti does not support → crash
• Consumer Blackwell (RTX 50) cuBLASLt disables row-wise FP8 scaled_mm in the Windows native runtime (datacenter sm_90 e.g. H100 supports it)
• Control: same GPU + same code under WSL2 serves the same model fine

Workaround (not full speed)

Set FREETOKEN_FORCE_E4M3_EMU=1 → skips torch._scaled_mm, uses triton W8A16 GEMM.

• ✅ model loads & serves
• ❌ cost: 1.5–2.2× slower
• ❌ still hits Bug 2 (empty Chinese output)

Suggested fixes

1. When row-wise torch._scaled_mm is unavailable on the device, fallback to the existing triton W8A16 _gemm/_gemv path (already in code, only skipped when e4m3_native() is True) instead of raising RuntimeError
2. Or probe device support with try/except torch._scaled_mm at load time and auto-route to emulated path
3. Or change fused projection scaling to tensor-wise on consumer Blackwell, reusing the supported tensor-wise scaled_mm

Bug 2 — Mistral tokenizer missing fix_mistral_regex → empty Chinese decode

Repro

1. Serve in emu mode (bypassing Bug 1)
2. English prompt → normal response
3. Chinese prompt → finish_reason=length, completion_tokens=59 but content is empty string

Load-time warning:

The tokenizer you are loading ... with an incorrect regex pattern ...
You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.

Root cause

• Qwen3.6 uses a Mistral-family tokenizer with a broken regex pattern → CJK tokenization corrupted
• transformers 5.16.1 provides fix_mistral_regex=True, but FreeToken does not pass it through when loading the tokenizer via from_pretrained (read from kwargs only, no env-var mapping)
• result: Chinese tokens mis-split, decode yields empty/invisible chars

Suggested fix

FreeToken should auto-pass fix_mistral_regex=True when loading Mistral-family (and derived) tokenizers via from_pretrained (or default it on when not explicitly specified).

Summary

Both bugs block consumer Blackwell (RTX 50) Windows Desktop users: Bug 1 prevents model load (FP8 row-wise gate), Bug 2 empties Chinese output (tokenizer regex). Same model + same FreeToken version works fine under WSL2 → root cause is in the Windows Desktop native CUDA / tokenizers runtime difference.

Minimal repro

• FreeToken Desktop 0.1.2 / Windows 11 / RTX 5070 Ti (sm_120) / torch 2.11.0+cu130
• Model: any NVFP4 (modelopt) MoE with GDN attention (e.g. Qwen3.6-35B)
• Steps: import → convert → start (or ft serve --model <nvfp4 dir>)