Batched server prefills can starve active decoders between quanta

[krunkosaurus]

Summary

With --batched-session 3, a long prefill can repeatedly reacquire the model between its nominal 128-token mixed-prefill quanta. Active decoders may consequently make no progress for the full duration of the prefill (60–70 seconds in the traces below).

This appears to be a scheduler fairness race rather than a Metal throughput or correctness problem: once all three sessions are decoding, aggregate throughput remains around 24–25 tok/s.

Environment

• ds4 main at 0a7ad77 (Fix batched server session recovery race)
• Apple M5 Max, 128 GiB RAM
• Metal backend with Metal 4 tensor API enabled
• DeepSeek V4 Flash, quant 2
• OpenCode using the OpenAI-compatible chat/tool API

Launch command:

./ds4-server \
  --ctx 200000 \
  --prefill-chunk 2048 \
  --batched-session 3 \
  --kv-disk-dir /tmp/ds4-kv \
  --kv-disk-space-mb 24576

Startup confirms the intended scheduler configuration:

ds4-server: batched mode enabled resident_sessions=3 prefill_quantum=2048 mixed_prefill_quantum=128 decode_coalesce_us=2000

Reproduction / evidence

One resident session was actively decoding while another began a resumed 5,361-token prefill. The decoder produced token 600 at 20:37:08, then made no progress until the entire prefill finished at 20:38:19:

20:37:08 chat ctx=7469..7519:50 gen=600 THINKING decoding ...
20:37:08 chat ctx=164081..169442:5361 TOOLS prompt start
20:37:10 chat ... prefill chunk 128/5361
...
20:38:17 chat ... prefill chunk 5248/5361
20:38:19 chat ... prefill chunk 5361/5361
20:38:19 chat ... prompt done 70.785s
20:38:19 chat ctx=7519..7569:50 gen=650 THINKING decoding ...

The same pattern repeated immediately afterward with a 4,664-token prefill, causing another roughly 60-second decode pause.

Expected behavior

When any generation is active, completing a bounded prefill quantum should hand the next model turn to a pending/active decoder before another prefill quantum can start. The bounded quantum should place a finite upper bound on decode latency.

Root cause

server_prefill_leave() clears model_busy and broadcasts model_cv, but there is no explicit handoff to an active generator. The prefill worker immediately loops back into server_prefill_enter() and can reacquire model_mu before a generation worker reaches server_eval_token() and increments decode_pending.

The admission condition prioritizes an already-pending decode, but it does not reserve the next turn for an active generator that has not enqueued its next token yet. Repeated wins by the prefill thread turn the 128-token quantum into an effectively unbounded decode stall.

Proposed fix

Use an explicit prefill-to-decode handoff:

1. On leaving a prefill quantum while active_generations > 0, set a scheduler handoff flag.
2. Block further prefill admission while that flag is set.
3. Clear it after the next decode batch completes, or when the active generation count reaches zero/shutdown begins.

I have a small local patch implementing this and a threaded regression test that verifies the prefill worker remains blocked until a decode completion releases the handoff.

Validation

After applying the handoff locally:

• Server unit tests pass.
• The Metal server builds without warnings.
• A real OpenCode request plus two identical concurrent streamed requests occupied all three resident slots.
• The duplicate requests returned identical content, reasoning, finish reason, and token counts.
• During three-way decode, each stream progressed at approximately 7.5–9 tok/s, consistent with the expected 24–25 tok/s aggregate.
• No long prefill-induced decoder pause was observed.

I can submit the patch as a PR if this scheduling approach looks appropriate.