Summary
On the gemma-4 GGUF path the tool-call opening marker <|tool_call reaches the client in
message.content, and sometimes in message.reasoning_content, alongside a correctly parsed
tool_calls array. Non-streaming requests.
Impact is on anything that stores or displays assistant content: transcripts, evaluation
harnesses, and multi-turn clients that echo prior content back into the next prompt, where the
stray marker becomes part of the model's own context.
It only happens under concurrency
Two soaks of the same workload (multi-turn tool-calling conversations, ~7.7k-token system
prompt, tools offered, tool results fed back), same build, same server flags, differing only in
how many conversations run at once:
|
tool calls |
responses with <|tool_call in content/reasoning_content |
| 4 concurrent sessions, 16 min |
182 |
31 |
| 1 session, sequential, 14 min |
553 |
0 |
Three times as many tool calls, sequentially, leaked nothing; the concurrent run leaked on
roughly 1 in 6. A later 7-minute concurrent run reproduced it at 35 occurrences.
The sequential run was otherwise clean — 748 requests, 553 tool calls, zero transport errors,
zero malformed tool calls, zero empty completions, engine healthy at the end. Tool-call parsing
and multi-turn tool results are in good shape; it is specifically concurrent decoding that
surfaces this.
Source-level narrowing
Gemma4Detector (server/function_call_parser.py):
toolcall_opener = "<|tool_call>"
self.bot_token = "<|tool_call>"
self.eot_token = "<tool_call|>"
call_regex = re.compile(r"<\|tool_call>\s*call:([A-Za-z_][\w.:-]*)\{(.*?)\}<tool_call\|>", re.DOTALL)detect_and_parse is correct for a well-formed call — it slices strictly before the first
opener, so the marker cannot leak:
idx = text.find(self.bot_token)
normal_text = text[:idx].strip() if idx != -1 else text
Note the else branch: when find misses, the entire text becomes normal_text. So any
output containing a partial or malformed opener — anything that is not the exact
<|tool_call> byte sequence, e.g. a split marker or the <|tool_call prefix followed by
something unexpected — is surfaced verbatim as content. That matches the observed strings,
which began with <|tool_call in every case.
For comparison, finish_streaming in the base class strips only the closing marker from
residual text:
if self.eot_token and self.eot_token in residual:
residual = residual.replace(self.eot_token, "")There is no equivalent for bot_token on any surfaced-content path.
gemma-4's vocab uses paired markers; the relevant ids are 46-51:
id 46 '<|tool>' id 47 '<tool|>'
id 48 '<|tool_call>' id 49 '<tool_call|>'
id 50 '<|tool_response>' id 51 '<tool_response|>'
Suggested mitigation
Strip both markers wherever content is surfaced, so engine internals cannot reach clients
regardless of which path emits them:
- if self.eot_token and self.eot_token in residual:
- residual = residual.replace(self.eot_token, "")
+ for _marker in (self.eot_token, self.bot_token):
+ if _marker and _marker in residual:
+ residual = residual.replace(_marker, "")
plus the same scrub in detect_and_parse's idx == -1 branch.
This is a mitigation, not a root-cause fix — it does not explain why the opener is malformed in
the first place. A stricter version would filter by token id rather than string, since ids
46-51 and 105/106 are all markers no client should ever see, and a string filter cannot catch a
marker that was split during detokenisation.
Open question
Why concurrency changes the rate: 31 of 182 concurrent calls versus 0 of 553 sequential, same
build and flags. If the opener is only ever malformed when several sequences decode in one
batch, that points at detokenisation or per-sequence buffering rather than at the parser.
Environment
Distinct from #201, where <|tool_response> was never a stop id so generation ran to
max_tokens. Here generation terminates correctly and a valid tool_calls array is produced —
only the surfaced text is contaminated.
Summary
On the gemma-4 GGUF path the tool-call opening marker
<|tool_callreaches the client inmessage.content, and sometimes inmessage.reasoning_content, alongside a correctly parsedtool_callsarray. Non-streaming requests.Impact is on anything that stores or displays assistant content: transcripts, evaluation
harnesses, and multi-turn clients that echo prior content back into the next prompt, where the
stray marker becomes part of the model's own context.
It only happens under concurrency
Two soaks of the same workload (multi-turn tool-calling conversations, ~7.7k-token system
prompt, tools offered, tool results fed back), same build, same server flags, differing only in
how many conversations run at once:
<|tool_callin content/reasoning_contentThree times as many tool calls, sequentially, leaked nothing; the concurrent run leaked on
roughly 1 in 6. A later 7-minute concurrent run reproduced it at 35 occurrences.
The sequential run was otherwise clean — 748 requests, 553 tool calls, zero transport errors,
zero malformed tool calls, zero empty completions, engine healthy at the end. Tool-call parsing
and multi-turn tool results are in good shape; it is specifically concurrent decoding that
surfaces this.
Source-level narrowing
Gemma4Detector(server/function_call_parser.py):detect_and_parseis correct for a well-formed call — it slices strictly before the firstopener, so the marker cannot leak:
Note the
elsebranch: whenfindmisses, the entire text becomesnormal_text. So anyoutput containing a partial or malformed opener — anything that is not the exact
<|tool_call>byte sequence, e.g. a split marker or the<|tool_callprefix followed bysomething unexpected — is surfaced verbatim as content. That matches the observed strings,
which began with
<|tool_callin every case.For comparison,
finish_streamingin the base class strips only the closing marker fromresidual text:
There is no equivalent for
bot_tokenon any surfaced-content path.gemma-4's vocab uses paired markers; the relevant ids are 46-51:
Suggested mitigation
Strip both markers wherever content is surfaced, so engine internals cannot reach clients
regardless of which path emits them:
plus the same scrub in
detect_and_parse'sidx == -1branch.This is a mitigation, not a root-cause fix — it does not explain why the opener is malformed in
the first place. A stricter version would filter by token id rather than string, since ids
46-51 and 105/106 are all markers no client should ever see, and a string filter cannot catch a
marker that was split during detokenisation.
Open question
Why concurrency changes the rate: 31 of 182 concurrent calls versus 0 of 553 sequential, same
build and flags. If the opener is only ever malformed when several sequences decode in one
batch, that points at detokenisation or per-sequence buffering rather than at the parser.
Environment
freetoken_kernel_cache-0.1.2+cu130)unsloth/gemma-4-26B-A4B-it-qat-GGUF(Q4_0),--tool-call-parserauto-resolved togemma4--kv-cache-dtype q8_0 --kv-reserve-tokens 131072 --max-running-requests 4tool-call parser.
Distinct from #201, where
<|tool_response>was never a stop id so generation ran tomax_tokens. Here generation terminates correctly and a validtool_callsarray is produced —only the surfaced text is contaminated.