Summary
ft serve --model Qwen/Qwen3.8-27B-FP8 consistently dies partway through Loading fp8 weights with RuntimeError: CUDA driver error: device not ready, raised from freetoken/models/qwen3_5_moe/weight.py (iter_weights → _iter_weights_fp8, lines ~195/~777 on current main).
The config phase succeeds — the arch is recognized and resolves cache_type='hybrid_radix', attention_backend='fi' — so this looks like a weight-loader issue specific to this checkpoint's path, not missing arch support.
Reproduction
ft serve --model Qwen/Qwen3.8-27B-FP8 --port 1919
Fails at a varying point in the fp8 shard loop (observed at 31/66 and 47%±). Reproduced twice back-to-back:
- once with
--gpu 1
- once with
CUDA_VISIBLE_DEVICES=1 and no --gpu
Tail of the log:
Loading fp8 weights: 47%|████▋ | 31/66 [00:22<00:25, 1.35it/s]
RuntimeError: CUDA driver error: device not ready
File ".../freetoken/models/qwen3_5_moe/weight.py", line 195, in iter_weights
File ".../freetoken/models/qwen3_5_moe/weight.py", line 777, in _iter_weights_fp8
Works fine in the exact same environment (control)
Qwen/Qwen3.6-35B-A3B-FP8 — loads, serves, correct outputopenai/gpt-oss-120b — loads, serves, correct output (CPU-MoE hybrid path)
So the install, GPU, and driver stack are healthy; the failure tracks the Qwen3.8-27B-FP8 loading path.
Environment
- FreeToken: git main @
9ef3651 (installed via uv pip install "freetoken[accel] @ git+..."); also fails on 0.1.2
- WSL2 Ubuntu 24.04 on Windows 11, NVIDIA driver 616.56 (Windows host), CUDA toolkit 13.0 in-distro
- GPU: RTX 5070 Ti 16 GB (sm_120); second GPU (RTX 5060 Ti) present but masked out in both repro runs
- Python 3.12 (distro venv)
Happy to run diagnostics or test patches — this box reproduces it reliably.
Summary
ft serve --model Qwen/Qwen3.8-27B-FP8consistently dies partway throughLoading fp8 weightswithRuntimeError: CUDA driver error: device not ready, raised fromfreetoken/models/qwen3_5_moe/weight.py(iter_weights→_iter_weights_fp8, lines ~195/~777 on current main).The config phase succeeds — the arch is recognized and resolves
cache_type='hybrid_radix',attention_backend='fi'— so this looks like a weight-loader issue specific to this checkpoint's path, not missing arch support.Reproduction
Fails at a varying point in the fp8 shard loop (observed at 31/66 and 47%±). Reproduced twice back-to-back:
--gpu 1CUDA_VISIBLE_DEVICES=1and no--gpuTail of the log:
Works fine in the exact same environment (control)
Qwen/Qwen3.6-35B-A3B-FP8— loads, serves, correct outputopenai/gpt-oss-120b— loads, serves, correct output (CPU-MoE hybrid path)So the install, GPU, and driver stack are healthy; the failure tracks the Qwen3.8-27B-FP8 loading path.
Environment
9ef3651(installed viauv pip install "freetoken[accel] @ git+..."); also fails on 0.1.2Happy to run diagnostics or test patches — this box reproduces it reliably.