Skip to content

Server: chat/completions live tool continuation via tool_call_id#611

Open
starforge-labs wants to merge 1 commit into
antirez:mainfrom
starforge-labs:chat-live-continuation
Open

Server: chat/completions live tool continuation via tool_call_id#611
starforge-labs wants to merge 1 commit into
antirez:mainfrom
starforge-labs:chat-live-continuation

Conversation

@starforge-labs

@starforge-labs starforge-labs commented Jul 26, 2026

Copy link
Copy Markdown

Fixes #609 - On the Anthropic endpoint, tool rounds continue the live KV slot cheaply
(anthropic live continuation match=tool-output-ids). On the OpenAI
chat/completions endpoint the equivalent never fires: clients replay the
assistant tool-call turn re-serialised (thinking stripped to an empty
block, tool call re-encoded from JSON), so exact token/text prefix
matching fails at the previous generation's tail even though common
reaches the entire previous prompt. The live slot is discarded and
prefill restarts from an older disk checkpoint — O(context) per tool
round. Before (Open WebUI, native function calling):

live kv cache miss live=15139 prompt=20460 common=14903 reason=token-mismatch
kv cache stored tokens=15139 trimmed=0 reason=evict key=token-text size=221.61 MiB
chat ctx=5779..14903:9124 TOOLS prompt done 31.253s

The trailing role:"tool" message's tool_call_id is the same precise
continuation handle as Anthropic's tool_use_id, so this PR reuses that
fast path for API_OPENAI chat: collect trailing tool ids at parse time
(chat_prepare_live_continuation, mirroring the Anthropic prepare), and
when the ids and live token frontier match, keep the sampled KV and
append only EOS + tool results + next assistant prefix.

After, same client, same conversation shape, ~24k context:

chat live continuation match=tool-output-ids ids=1 cached=19597 prompt=22207
chat ctx=19597..22207:2610 TOOLS prompt done 11.234s
chat live continuation match=tool-output-ids ids=1 cached=23507 prompt=23552
chat ctx=23507..23552:45 TOOLS prompt done 0.798s
chat live continuation match=tool-output-ids ids=1 cached=24151 prompt=24164
chat ctx=24151..24164:13 TOOLS prompt done 0.230s

Tool rounds drop from 45-90s to 0.2-11s, and the per-round ~300 MiB
evict/store KV disk cycle disappears. A genuine mid-conversation prompt
change still evicts and re-prefills correctly (verified by editing
history mid-chat).

Design notes:

  • State is stored in the existing anthropic_live_* request/slot
    fields; the matcher, remember, and clear sites are protocol-gated. A
    rename to something protocol-neutral would touch many more lines —
    happy to do that if preferred.
  • The frontier is only remembered for chat when
    should_canonicalize_tool_checkpoint() is false, since
    canonicalization rewrites the live session and would strand the
    remembered frontier.
  • Every gate failure returns 0 and falls through to the existing
    matching chain, so behaviour is unchanged whenever the fast path
    doesn't apply.
  • Adds test_chat_live_tail_renders_tool_results_only covering the
    role:"tool" tail rendering and the protocol gate; full embedded suite
    passes.

Testing

  • make test: full suite passes — all tracks including --server,
    --tool-call-quality, --long-context, and --logprob-vectors
    on Apple M5 Max 128GB, Metal backend, macOS 26.5.2, model
    DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2.gguf (q2).
  • make cpu: builds clean (verified on x86-64 Linux).
  • Live verification: same model fully resident on Metal, Open WebUI 0.10.x
    native function calling — eleven consecutive chat live continuation hits
    across chained tool rounds at 6k–24k context (0.2–11s per round vs 45–90s
    before), zero evict/store KV disk writes between rounds, and correct
    evict + re-prefill on a genuine mid-conversation prompt edit.

OpenAI chat clients replay the assistant tool-call turn re-serialised
(thinking stripped, tool call re-encoded), so exact token/text prefix
matching fails at the previous generation's tail and the live slot is
discarded, forcing a re-prefill from an older checkpoint on every tool
round. The trailing role:"tool" tool_call_id is the same precise
continuation handle as Anthropic's tool_use_id, so reuse that fast
path: collect trailing tool ids at parse time, and when they match the
remembered live frontier, keep the sampled KV and append only
EOS + tool results + assistant prefix.

The frontier is only remembered when the sampled state survives
(skipped when canonicalize_tool_checkpoint will rewrite the session),
and every matcher failure falls through to the existing behaviour.

Measured on M5 Max 128GB / Open WebUI native function calling:
tool rounds at 24k context drop from 45-90s full re-prefill to
0.2-11s delta prefill, and the per-round ~300 MiB evict/store KV
disk write cycle disappears.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

OpenAI /v1/chat/completions tool rounds never hit live continuation (match=tool-output-ids); live slot discarded despite full-prompt common

1 participant