Prefill hangs on RTX 4060 Ti (SM89) for prompts beyond warmup length

[2stacks]

Summary

On an RTX 4060 Ti 16GB (Ada, SM89), a ~180-token prompt sent to a freshly started server hangs indefinitely. An ~11-token prompt works and decodes at 50–66 tok/s.

While hung: no Prefill batch line, no access-log line, connection stays open. GPU sits flat at 100% utilization / ~35W of 165W. Scheduler thread spins at ~100% CPU. SIGINT/SIGTERM are ignored; kill -9 required.

Is SM89 desktop Ada a tested configuration? The paper benchmarks an RTX 4060 laptop, RTX 5090, and RTX PRO 6000.

Environment

• FreeToken 0.1.2+g0ab982f10 (Desktop installer engine wheel)
• RTX 4060 Ti 16GB (SM89), driver 590.48.01, Ubuntu 24.04
• CUDA 13.0 toolkit installed, nvcc on PATH and confirmed in use
• torch==2.11.0+cu130, flashinfer-python==0.6.17, sglang-kernel==0.4.5+cu130, freetoken-kernel-cache==0.1.2+cu130.g0ab982f10
• Model: nvidia/Qwen3.6-35B-A3B-NVFP4 (FTW)

Reproduction

Start a fresh server:

ft serve --model ~/.freetoken/models/Qwen3.6-35B-A3B-NVFP4 \
         --served-model-name qwen3.6-35b-a3b

Send a ~180-token prompt as the first request:

python3 -c 'import json;p="The quick brown fox jumps over the lazy dog. "*20;print(json.dumps({"model":"qwen3.6-35b-a3b","messages":[{"role":"user","content":p}],"max_tokens":32}))' > /tmp/p.json
 
curl http://127.0.0.1:1919/v1/chat/completions -H 'Content-Type: application/json' -d @/tmp/p.json

Never returns. Restart required.

For contrast, an ~11-token prompt ("hi") as the first request completes normally.

Startup logs Prefill warmup complete for lengths [80, 128].

Request history affects the outcome

The same 20-rep request that hangs from a cold server succeeds if smaller prompts are served first. Sending 10 → 15 → 20 succeeds, and continuing 25 → 30 → 35 → 40 also succeeds. Sending 10 then 20 hangs.

That escalation is not reliable, though, and the step size does not cleanly predict failure: a run of 9 → 10 → 12 → 15 → 20 → 27 → 35 → 44 all landed, then 44 → 54 hung — a smaller relative jump than several earlier steps that worked. So this may be non-deterministic rather than a size threshold. I have not run enough repetitions to say.

Either way there is no practical warm-up path to agent-sized prompts: Hermes sends ~23,000 tokens on its first message and never returns.

Notes

Full logs, py-spy dumps, nvidia-smi/nvtop output available on request. Happy to run repeated trials to establish whether this is non-deterministic, apply patches, or collect further traces.

[CygnusBill]

I spent a day chasing what I'm confident is this same bug on very different hardware (RTX PRO 6000 96GB) and got it to a root cause with a deterministic reproduction and the failing traceback: an unhandled CUDA OOM in the token-proportional prefill workspace (fused_nvfp4.py:336) kills the engine process, and the frontend then hangs every request silently. The trigger is the gap between the memory plan (--moe-cache-auto budgets 90% of boot-time free VRAM) and real free VRAM at request time — on a desktop 16GB card the OS/compositor eats into that continuously, which is why your threshold sits barely past the warmup lengths.

It also explains your escalation observation: stepwise prompt growth lets the torch allocator's block cache grow incrementally (reuse), while a direct jump demands one novel large block from a sliver that can't provide it.

Full report, reproduction recipe, suggested fixes, and the complete debug trail: #110. (A second, distinct silent-hang mechanism — prompts exceeding the KV pool waiting forever in admission — is at #111; worth checking which of the two you're hitting: if a small request succeeds right after your hung one times out, it's the admission case; if everything hangs until you kill -9, it's the OOM case.)

[z19822783667-design]

FreeToken 0.1.2 A highly suspected CUDA is a highly suspected CUDA synchronous/sorting defect when running Qwen3.6-35B-A3B-NVFP4, MoE offload, and longer prefills on the Ada SM89 GPU. The problem manifests as a prefill infinitely suspended and has a distinct timing sensitivity. Synchronizing the main compute stream of each Transformer layer stably eliminates the problem, stating that the error is related to cross-layer asynchronous execution.

[endenis]

I may be experiencing the same issue or a related one.
I am using an RTX 4070 (SM89).

@2stacks As a diagnostic, could you please try running with CUDA_LAUNCH_BLOCKING=1 ft serve ... and see if it changes the behavior?

[atlicgu]

Reproduction on RTX 4060 Ti 16GB (SM89) — Desktop v0.2.0-beta.15

Environment:

• GPU: NVIDIA GeForce RTX 4060 Ti 16GB (SM89), driver 595.84, CUDA 13.2
• OS: Ubuntu (Linux x86_64)
• FreeToken engine: 0.1.2 (bundled in Desktop v0.2.0-beta.15)
• Model: nvidia/Qwen3.6-35B-A3B-NVFP4 (FTW)
• Default config: --moe-backend offload --memory-ratio 0.85

Symptoms match the issue exactly:

• Server starts fine, warmup completes for lengths [80, 128]
• Short prompts (< 128 tokens) decode normally at ~50 tok/s
• Prompts > ~180 tokens hang indefinitely: GPU sits at 100% utilization, no Prefill batch log, scheduler thread spins at 100% CPU
• SIGINT/SIGTERM ignored; only kill -9 works

Threshold found by testing:

Prompt tokens	Result
11	✅ OK
60	✅ OK
110	✅ OK
~210	❌ Hang

Additional context:

• Tried --disable-moe-prefill-overlap, --max-prefill-length 4096, --attention-backend triton — none resolved the hang
• Lowering --memory-ratio to 0.7 changes KV/expert cache split but doesn't prevent the hang
• The same model + config works on RTX 4060 Laptop (SM89) per the paper benchmarks — this appears to be specific to the Ti variant or driver combination
• Desktop GUI built-in chat works (short prompts), but coding agents (e.g. DeepSeek Harness) send long system prompts on first request, triggering the hang immediately

Happy to provide py-spy dumps, full logs, or run additional repro trials if helpful.

[jason-fxz]

Fixed in main by #243 (row-wise FP8 _scaled_mm on sm_89 + torch < 2.12 stalls on a side stream; fused projections now run per-part tensor-wise GEMMs). Ships in the next engine release — please reopen if it still hangs after updating.