Requests longer than the KV pool are queued forever with no error — and --moe-cache-auto leaves only ~8k tokens of KV on a 96GB GPU

[CygnusBill]

Summary

Two related problems, one report:

1. A request whose prompt needs more KV pages than num_pages is never scheduled and never rejected: no prefill batch, no log line, no HTTP error — the client waits until its own timeout, and the engine (which is perfectly healthy) simply serves the next request after the abort.
2. With --moe-cache-auto on a 96GB RTX PRO 6000 serving MiniMax-M2.5-NVFP4, the joint budget solve yields moe_cache_size=10114 num_pages=8209 (page_size=1) — an effective context of ~8.2k tokens for a model advertising far more, with nothing in the logs flagging it. The pool lands just above the kv_reserve_tokens=8192 floor because the solve gives nearly all remaining budget to MoE cache slots.

Together: any agent-style workload (system prompt + tools + history routinely >8k tokens) silently hangs at defaults on high-end hardware.

Environment

FreeToken v0.1.2 @ f0abe58, torch 2.11.0+cu130, CUDA 13.0.1, driver 595.91.07, RTX PRO 6000 Blackwell 96GB, EPYC 128c/503GB, nvidia/MiniMax-M2.5-NVFP4.

Reproduction

1. ft serve --model nvidia/MiniMax-M2.5-NVFP4 --moe-cache-auto → log: resolved moe_cache_size=10114 num_pages=8209 (prefill_overlap=True).
2. Send prompts up to ~7.1k tokens → fine (single prefill batch, ~1.5k tok/s ingest).
3. Send a prompt of ~8.8k tokens (> 8209 pages) → the request never returns. Server log shows no prefill batch for it at all; GPU idle; no error anywhere.
4. Abort the client; send any normal request → served immediately (the engine was healthy the whole time — this cleanly discriminates it from the engine-death issue at Unhandled CUDA OOM in prefill expert workspace kills the engine; server keeps accepting requests that never return (root cause behind #20-style headless state; likely also #72) #110).

Verified allocator-independent (identical under PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True) and MoE-overlap-independent (identical with --disable-moe-prefill-overlap). The pass/hang boundary tracks token count against num_pages exactly: a prompt tokenizing to 8,011 passes; 8,760 hangs. Full experiment trail in the first comment.

Suggested fixes

1. Reject over-capacity requests at admission with a clear 4xx ("prompt requires N pages, pool has M") instead of infinite queueing.
2. Surface the effective KV context at startup — num_pages=8209 next to a long-context model deserves a WARN.
3. Reconsider the auto-solve's split (or expose a --context-tokens knob): reserving only the floor for KV means the flagship long-context models can't hold an agent conversation at defaults; users would likely trade some expert-cache hit rate for context if the trade were visible.

Related observation (mentioning rather than filing separately — say the word and I'll split it out)

Any batch classified as prefill streams the entire expert bank set over PCIe (~4.5s / ~137GB for this model), including a 1-token extend on a full radix-cache hit — so multi-turn conversations pay a full-weights toll per turn even when KV is fully cached, and each chunk of a chunked prefill pays it again (verified: 2 chunks = 9.0s, 3 chunks = 13.5s). A token-count threshold below which prefill uses the decode path's on-demand expert loading would remove it.

Offer

Same rig and offer as #110: deterministic repros, full logs, and hardware available for validating patches.

[CygnusBill]

How this was found and pinned down — the short version of a longer investigation (full trail on #110, which shares the campaign):

Discovery

Extended prompt-size sweeps (5k → 6.5k → 8k "words") on two configurations differing only in --disable-moe-prefill-overlap both sailed through ~7.1k tokens at increasing ingest rates (1,204 → 1,532 tok/s) and then both timed out (>300s, later >600s) on the 8k-word cell. Overlap-independence immediately ruled out the MoE movement machinery.

The false lead and the tell

A replication that inserted a few seconds' gap before the big prompt passed — which briefly looked like an arrival-timing race. The token audit killed it: the gapped variant's prompt happened to tokenize to 8,011 tokens while the sweep's generator produced ~8,760 for the "same" word count. 8,011 < 8,209 (num_pages) < 8,760. The boundary was capacity, not timing.

Confirmation by prediction

• Re-ran the failing cell under PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True: identical infinite wait (admission logic doesn't touch the allocator) — predicted in advance.
• Server-log fingerprint: every smaller request logs its Prefill batch line; the over-capacity request logs nothing at all — it is never scheduled, not slow.
• Post-abort discriminator: immediately after the client gives up, the next request (a full re-prefill of ~7.1k cached tokens) completes in one expert-stream floor (~4.5s). Engine healthy throughout — the opposite signature of the engine-death issue.

Numbers for this hardware (96GB RTX PRO 6000, MiniMax-M2.5-NVFP4, defaults)

Prompt tokens	Result
1,146 → 7,113 (six steps)	All pass; ingest rises 205 → 1,532 tok/s (flat ~4.5s wall = full expert-stream floor dominating)
8,011	Passes (single batch, 834 tok/s)
~8,760	Never scheduled; client timeout; engine unaffected

--moe-cache-auto resolved moe_cache_size=10114 num_pages=8209 (prefill_overlap=True) — the whole failure boundary in one startup line that currently draws no attention to itself.

[pedro-moser]

Opened #198 for the explicit-capacity side of this issue: --num-pages / resolved --num-tokens now participates in the joint MoE/KV auto-budget before allocation. This is complementary to #118's scheduler admission fix and does not close the remaining default-capacity tradeoff.