Request hang (silent, zero-log) with hybrid_radix cache + moe-backend offload + nvfp4 triton backend — reproduced on 2 independent models, not explained by #110

[hardwork9047]

Summary

A ~2,400-token chat completion request hangs forever (no error, no log
line — not even "Prefill batch") with cache_type='hybrid_radix' +
--moe-backend offload + nvfp4_backend='triton', on plenty of free
VRAM. The engine process pins one CPU thread at ~190% while GPU shows
100% utilization at only ~90W (an idle-spin signature, not active
compute), and the frontend never recovers — subsequent requests, even
trivial ones, also hang forever and the process must be kill -9'd.

I initially suspected this was the OOM-crash-then-zombie mechanism from
#110, so I tested mitigating it with --memory-ratio (0.9 → 0.8 → 0.5,
raising free-after-init VRAM from 1.82 GiB up to 10.08 GiB — roughly half
the GPU's total free memory). The hang reproduced identically at every
memory-ratio setting, which rules out #110's OOM-driven mechanism as the
cause here (10 GiB free is far beyond any plausible workspace-allocation
OOM for a ~2,400-token prompt). I then reproduced the exact same hang on a
second, independent model with the same architecture family
(hybrid attention + mamba/linear-state layers, hybrid_radix cache),
which points at something specific to this cache-type/backend combination
rather than either model's weights.

Environment

• FreeToken 0.1.2 (PyPI install), torch 2.11.0+cu130
• NVIDIA driver 595.71.05, 2× RTX 4090 24GB (sm_89, no NVLink), only
  CUDA_VISIBLE_DEVICES=0 used
• Ubuntu 22.04, kernel 6.8.0-124-generic
• Models tested (both hang identically):
  • nvidia/Qwen3.6-35B-A3B-NVFP4
  • ornith-ai/Ornith-1.5-35B-A3B-NVFP4
• Launch flags (both models):

  ft serve --model-path <local snapshot> \
    --max-running-requests 1 \
    --max-seq-len-override 32768 \
    --moe-backend offload \
    --moe-cache-size 1024 \
    --host 0.0.0.0 --port 1920
  

• Resolved config (both models, identical): moe_backend='offload',
  attention_backend='fi', cache_type='hybrid_radix',
  nvfp4_backend='triton' (auto-selected — marlin fails at startup with
  "donor symbols unavailable" and flashinfer requires sm_120+; triton
  is the only usable NVFP4 backend on this sm_89 card)

Deterministic reproduction

1. Start clean, e.g.:

   ft serve --model-path <Qwen3.6-35B-A3B-NVFP4 or Ornith-1.5-35B-A3B-NVFP4 snapshot> \
     --max-running-requests 1 --max-seq-len-override 32768 \
     --moe-backend offload --moe-cache-size 1024 --port 1920
   

   Startup completes normally, e.g.:

   Allocating 712629 tokens for KV cache, K + V = 13.59 GiB
   Free memory after initialization: 1.82 GiB
   API server is ready to serve on 0.0.0.0:1920
   

2. Send a short prompt (~18 tokens) — completes normally, full
   "Prefill batch" / "Decode batch" logging, 200 OK.
3. Send a ~2,400-token Japanese-text prompt (well under the 8,192
   max_extend_tokens chunk boundary, so this is a single-chunk prefill):

   curl --max-time 60 http://localhost:1920/v1/chat/completions \
     -H "Content-Type: application/json" \
     -d '{"model": "<served-model-name>", "messages": [{"role":"user","content":"<~2400 tokens of Japanese filler text>"}], "max_tokens": 200}'
   

   → 60s client timeout, 0 bytes received. serve.log gets zero new
   lines — not even the "Prefill batch, #new-seq: 1, ..." line that
   normally appears the instant a request starts processing.
4. nvidia-smi during the hang: 100% GPU utilization, but only
   ~85-91W power draw (idle-spin, not compute — a real prefill/decode on
   this model draws much higher). The surviving engine-worker
   multiprocessing.spawn_main process is pinned at ~150-190% CPU.
5. The frontend never recovers: a subsequent trivial request (10-token
   prompt) also hangs indefinitely and must be killed.

I did not find a clean token-count threshold — in earlier bisection on
qwen36-35b-a3b, 1,453/1,622/2,055-token requests all succeeded (2-3s
each) but ~2,400 tokens hung, and removing a single character from the
2,400-token prompt still hung (so it's not an exact-length coincidence
either). The trigger seems to depend on some state beyond raw token count,
not yet isolated.

What I ruled out

• Not Unhandled CUDA OOM in prefill expert workspace kills the engine; server keeps accepting requests that never return (root cause behind #20-style headless state; likely also #72) #110's OOM-then-zombie mechanism: --memory-ratio 0.5 raised
  free-after-init VRAM to 10.08 GiB (KV cache shrunk 712K→279,510 tokens)
  and the hang still reproduced identically. A genuinely larger request
  (~264K target tokens) sent against the same server did process and log
  normally (multiple "Prefill batch" lines, increasing token usage %,
  ~3,000-3,400 tok/s input throughput) before I killed it — so the engine
  is not simply incapable of large prefills; something about this
  specific mid-size request class triggers total silence instead.
• Not --disable-moe-prefill-overlap — tested, hang identical.
• Not multi-chunk prefill — the ~2,400-token repro is a single chunk
  (well under max_extend_tokens=8192).
• Not --nvfp4-backend choice — only triton runs at all on this GPU
  (marlin/flashinfer both fail at startup, unrelated to this bug).
• Not model-specific — reproduces identically on two independent
  models sharing only the hybrid_radix/mamba-slot architecture family
  and moe_backend=offload + nvfp4_backend=triton combination.

What I haven't tried yet

• py-spy dump on the hung spawn_main PID (didn't have it installed at
  reproduction time; will attach next repro)
• --moe-prefill-hit-d2d, or substantially reducing --max-extend-tokens
• A non-hybrid (pure-transformer, non-mamba, non-hybrid_radix) MoE model
  of similar size, to test whether hybrid_radix cache handling
  specifically is implicated

Happy to gather more diagnostics (py-spy, nsys, etc.) if useful — this is
consistently reproducible on my setup.

[jason-fxz]

Thanks for the thorough investigation.

Fixed in main by #243: on sm_89 + torch < 2.12, PyTorch's row-wise FP8 _scaled_mm (used by the fused q/k/v and GDN projections both models share) races the side stream and spins forever — that's the 100 % util at ~90 W and the pinned thread. Fused projections now run per-part tensor-wise GEMMs instead. Ships in the next engine release — please reopen if it still hangs after updating.