Server: chat/completions live tool continuation via tool_call_id#611
Open
starforge-labs wants to merge 1 commit into
Open
Server: chat/completions live tool continuation via tool_call_id#611starforge-labs wants to merge 1 commit into
starforge-labs wants to merge 1 commit into
Conversation
OpenAI chat clients replay the assistant tool-call turn re-serialised (thinking stripped, tool call re-encoded), so exact token/text prefix matching fails at the previous generation's tail and the live slot is discarded, forcing a re-prefill from an older checkpoint on every tool round. The trailing role:"tool" tool_call_id is the same precise continuation handle as Anthropic's tool_use_id, so reuse that fast path: collect trailing tool ids at parse time, and when they match the remembered live frontier, keep the sampled KV and append only EOS + tool results + assistant prefix. The frontier is only remembered when the sampled state survives (skipped when canonicalize_tool_checkpoint will rewrite the session), and every matcher failure falls through to the existing behaviour. Measured on M5 Max 128GB / Open WebUI native function calling: tool rounds at 24k context drop from 45-90s full re-prefill to 0.2-11s delta prefill, and the per-round ~300 MiB evict/store KV disk write cycle disappears.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #609 - On the Anthropic endpoint, tool rounds continue the live KV slot cheaply
(
anthropic live continuation match=tool-output-ids). On the OpenAIchat/completions endpoint the equivalent never fires: clients replay the
assistant tool-call turn re-serialised (thinking stripped to an empty
block, tool call re-encoded from JSON), so exact token/text prefix
matching fails at the previous generation's tail even though
commonreaches the entire previous prompt. The live slot is discarded and
prefill restarts from an older disk checkpoint — O(context) per tool
round. Before (Open WebUI, native function calling):
The trailing
role:"tool"message'stool_call_idis the same precisecontinuation handle as Anthropic's
tool_use_id, so this PR reuses thatfast path for API_OPENAI chat: collect trailing tool ids at parse time
(
chat_prepare_live_continuation, mirroring the Anthropic prepare), andwhen the ids and live token frontier match, keep the sampled KV and
append only EOS + tool results + next assistant prefix.
After, same client, same conversation shape, ~24k context:
Tool rounds drop from 45-90s to 0.2-11s, and the per-round ~300 MiB
evict/store KV disk cycle disappears. A genuine mid-conversation prompt
change still evicts and re-prefills correctly (verified by editing
history mid-chat).
Design notes:
anthropic_live_*request/slotfields; the matcher, remember, and clear sites are protocol-gated. A
rename to something protocol-neutral would touch many more lines —
happy to do that if preferred.
should_canonicalize_tool_checkpoint()is false, sincecanonicalization rewrites the live session and would strand the
remembered frontier.
matching chain, so behaviour is unchanged whenever the fast path
doesn't apply.
test_chat_live_tail_renders_tool_results_onlycovering therole:"tool" tail rendering and the protocol gate; full embedded suite
passes.
Testing
make test: full suite passes — all tracks including--server,--tool-call-quality,--long-context, and--logprob-vectors—on Apple M5 Max 128GB, Metal backend, macOS 26.5.2, model
DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2.gguf(q2).make cpu: builds clean (verified on x86-64 Linux).native function calling — eleven consecutive
chat live continuationhitsacross chained tool rounds at 6k–24k context (0.2–11s per round vs 45–90s
before), zero evict/store KV disk writes between rounds, and correct
evict + re-prefill on a genuine mid-conversation prompt edit.