SWA pool exhaustion under concurrency kills the scheduler worker (RuntimeError in alloc_swa), and the advertised pool floor is not concurrency-aware

[salekseev]

Summary

Under concurrency alloc_swa finds the window pool empty and raises, which kills the scheduler
worker and takes the server down. It is not recoverable, and in-flight requests are not told —
they hang until their own timeout.

RuntimeError: SWA pool exhausted: need 1669, have 1306
  File "freetoken/kvcache/hybrid_swa_pool.py", line 178, in alloc_swa
  File "freetoken/scheduler/cache.py", line 274, in allocate_paged
  File "freetoken/scheduler/scheduler.py", line 777, in _prepare_batch
Backend supervisor: backend worker freetoken-TP0-scheduler exited
Backend worker is gone and cannot be restarted; stopping the API server

Reached after ~9 minutes / 275 requests / 182 tool calls of steady load: 4 concurrent multi-turn
tool-calling conversations, each with a ~7.7k-token system prompt, on
--max-running-requests 4 with --swa-full-tokens-ratio 0.07 (a 9176-token window pool).

Root cause is #204 — this is one of its two presentations

Instrumentation traced the underlying fault to a duplicate-page double-free in the full KV
free list: free_slots persistently holds 2125 pages twice, so _allocate hands one page to
two owners, and the second owner's alloc_swa overwrites the first's mapping. Full evidence in
#204.

The window pool size only decides which symptom you see:

window pool	outcome
9176 tokens (--swa-full-tokens-ratio 0.07)	runs out first → SWA pool exhausted (this issue), ~9 min
26231 tokens (default 0.2)	survives long enough for the accounting mismatch to be detected → check_integrity assert, ~15 min

Suggest closing this into #204, keeping the two points below if they are worth tracking
separately.

Two robustness points independent of the root cause

1. Running out is fatal rather than throttling. Everything else in the scheduler treats
capacity as a scheduling constraint — a request waits for KV pages. Window slots are the one
pool whose exhaustion is an exception, so a transient shortfall of 363 slots ends the process.
Deferring the batch, or refusing the request with a 503, would keep the server up.

2. The advertised floor is not concurrency-aware. /v1/cache/status reported the window
pool as resizable down to 8777 tokens, and --swa-full-tokens-ratio 0.07 resolved to 9176 —
both accepted at startup without complaint. But 4 concurrent sequences each need a live trailing
window plus the retain gap, and prefill of a long prompt needs its own slots on top, so that
configuration cannot actually be served. A floor accounting for
max_running_requests * (window + _SWA_RETAIN_GAP) plus the prefill chunk would have rejected
it at startup instead of crashing 9 minutes into production traffic.

Why a small pool is tempting, for context: on this model the window pool costs ~105 KiB/token
against the full pool's ~10, so at 131072 tokens the default ratio spends 2.7 GiB on it, and
that memory otherwise goes to the MoE expert cache — worth +19% decode (141.6 → 168.6 tok/s at
1200 tokens). Single-stream benchmarks, needle recall and a prefill sweep all passed at 0.07;
only sustained concurrency exposed it.

Environment

• FreeToken 0.1.2, RTX 4080 SUPER (sm_89, 16 GB), driver 610.57.04 / CUDA 13.3
• unsloth/gemma-4-26B-A4B-it-qat-GGUF (Q4_0), window 1024, 25 SWA / 5 full layers
• --kv-reserve-tokens 131072 --kv-cache-dtype q8_0 --max-running-requests 4
• Tree also carries fix(kernel): launch GGUF MoE GEMV token axis on grid.x, not grid.z #185, feat(kvcache): 8-bit KV cache (q8_0 / fp8_e4m3) behind --kv-cache-dtype #103, feat(server): expose swa_full_tokens_ratio as a startup CLI flag #109 and local gemma4 fixes for gemma4 GGUF: embedding quant type hardcoded to Q6_K, so a Q4_0-embedding GGUF fails to load with a bare AssertionError #188 / gemma-4 GGUF: <|tool_response> is not a stop token, so every tool call burns max_tokens and returns finish_reason=length #201; none touch page or
  slot accounting.

[salekseev]

Duplicate of #204 — same root cause, established by instrumentation: a duplicate-page double-free in the full KV free list (free_slots persistently holding 2125 pages twice). The window-pool size only decides which symptom appears first: at 9176 tokens the pool runs out (SWA pool exhausted, ~9 min), at the 26231-token default it survives until check_integrity detects the mismatch (~15 min).

The two robustness points from this report — exhaustion being fatal rather than throttling, and the advertised pool floor not being concurrency-aware — have been folded into #204 so nothing is lost. Continuing there.