Host state-slot pool leaks under sustained load; prefix reuse dies silently after saturation

[igorls]

Summary

After ~4.5 h of continuous production traffic the host state-slot pool saturates at
--host-state-slots and never releases. Once saturated, every subsequent request
falls to reuse=root with cache=0 — prefix reuse is completely dead for the
lifetime of the process. Only a restart recovers it.

Observed on feat/upstream-integration @ a5e9b7be + #13 fix (2b75ed5f), native
Windows build, RTX PRO 6000 Blackwell, Qwen3.8-27B NVFP4.

Evidence

Engine started 07:24. occupancy.host_state_slots (from the throughput event's
context_cache block) over the soak:

time   device_state(/16)   host_state(/24)   kv_pages(/13312)
07:20        1 -  4              0 -  0          130 -   616
07:40        4 - 16              0 - 11          285 -  1132
07:50       16 - 16             11 - 24         1088 -  3729
08:00       16 - 16             24 - 24         3604 -  4159   <== saturated
08:10       16 - 16             24 - 24         3943 -  3943
...
15:40       16 - 16             24 - 24         3950 -  4256   (7.5 h later)

host_state_slots reaches 24/24 and never drops below it again. Earlier in the
soak (04:00-07:20, a prior process) the pool breathed normally: 0 -> 8 -> 0.

Last request with any prefix cache hit: 08:09:53. Every one of the following
74 requests was reuse=root, cache=0, across 7.5 h and multiple distinct
clients — 563,042 tokens of avoidable prefill.

KV pages sit at ~4,000 of 13,312 throughout, so this is not KV capacity pressure.
It is specifically the host state-slot pool.

Reproduction (deterministic, once saturated)

Send the same byte-identical request body three times, one second apart:

before restart:  root                   3903 prompt  3903 recomputed     0 hit  0.67s
                 root                   3903 prompt  3903 recomputed     0 hit  0.64s
                 root                   3903 prompt  3903 recomputed     0 hit  0.71s

A request cannot fail to match its own byte-identical predecessor. After
restarting the engine with no other change, the identical sequence:

after restart:   root                   3903 prompt  3903 recomputed     0 hit  0.67s
                 private_turn_closure   3903 prompt     5 recomputed  3898 hit  0.13s
                 private_turn_closure   3903 prompt     5 recomputed  3898 hit  0.14s

Impact

• Silent. No error, no warning, no health-check failure. The engine stays "healthy"
  and serves correct output the whole time — it just recomputes every prompt from
  scratch.
• Cost scales with prompt length. On agentic traffic (10-15k token prompts) it is
  the dominant TTFT term: measured TTFT p50 went from 581 ms (07:00, reuse working)
  to ~900-2000 ms after saturation.

Notes

pressure.private_owners_evicted, pressure.search_budget_exhaustions and
pressure.checkpoints_dropped all increment across the same window, which suggests
owners are being evicted from the logical store while their host state slots are
never returned to the pool.

Two things worth fixing independently:

1. The leak itself.
2. Detection — the supervisor should surface "prefix reuse collapsed" as an
   operator insight (host_state_slots pinned at max AND N consecutive root
   selections). This class of failure is invisible to health checks.