fix(serve): recover drifted Qwen tool calls - #10
Conversation
|
Have you ever seen this with qwen3.8-27b? |
|
I've seen it happen a few times a day. Very long thinking and/or writing long files seems to increase the likelyhood of it. |
…ciliation (issue #5) Final parser verified equivalent to upstream #10's grammar grafted onto the fork's streaming filter; adversarial-review findings from the first attempt resolved by faithful re-port (tolerant mode relaxes exactly two checks, <tool_call>-only trigger). allowed_types byte-faithful to upstream Neroued#57. Clean-volume independent test verification green. Claude-Session: https://claude.ai/code/session_01Wv1ehCcaeL86hBzw74iqgr # Conflicts: # RUNBOOK.md
Near-miss tag recovery was removed in the faithful #10 re-port; the header comment (upstream's own wording) describes what tolerant mode actually does. Claude-Session: https://claude.ai/code/session_01Wv1ehCcaeL86hBzw74iqgr
|
I confirm this is happening to me too. I converted a QUASAR qwen3.8-27b NVFP4 model from vllm to ninfer and I am experiencing this tool broken command, and I was wondering where the issue was coming from. |
|
Can confirm it is happening occasionally on 3.8-27b NVFP4 model. |
Adapt upstream Neroued/ninfer PR Neroued#10 (7aa9200) to this fork's diverged serving stack: - add --tolerant-tool-calls to recover complete Qwen calls with trailing wrapper tags, suffix text, a missing outer tool-call close tag, or a call emitted inside the reasoning channel before the closing think tag - buffer streaming reasoning for tolerant tool requests so malformed call text is not emitted as thinking before the final parser classifies it - request log schema v10 -> v11: server_start records carry server.tolerant_tool_calls; the bench corpus runner constant is aligned with the C++ version (it had lagged at v9 since 3b60754) - update serve docs, usage text, and the affected parser/option/log/schema tests
|
Confirmed this still reproduces and this fix still resolves it as of today (2026-08-27), including on Hand-merged this PR's actual fix onto a more current pin ( (Separately: filed #101 for an unrelated tool-call parsing bug we hit even with this fix applied — a strictly-typed parameter's JSON value containing an un-doubled backslash, e.g. a literal grep/regex \* or \|, rejects the whole call outright since your tolerant recovery here only engages once a call has already been parsed.) |
Hand-port of the parser portion of upstream PR Neroued#10 (Neroued/ninfer, 'fix(serve): recover drifted Qwen tool calls' by Sciguymjm). The PR itself targets an older base (request-log schema v4->v5 vs v9 here) and predates the Responses API, preserve_thinking, and ToolCallStreamFilter that this tree carries, so cherry-picking it either failed to build or silently dropped those features. Only the parser semantics + the opt-in flag are ported; the PR's request-log schema bump, docs, and bench changes are not. Motivation: Qwen drifts from the tool-call syntax during long agentic sessions. The strict parser then returns the whole thing as plain text, so the client renders raw <tool_call> XML instead of executing the call. That also forces the client to rewrite history, which breaks the prompt prefix and costs a full KV re-prefill on the next turn (~5s at 37k ctx, ~20s at 100k) - observed in a Claude Code session against this server. --tolerant-tool-calls (default off, strict behaviour unchanged) recovers a complete function call despite: duplicate/garbage closing tags, a missing </tool_call>, or trailing explanatory prose. A call is only promoted when its parameters parse; genuinely unparseable output still falls back to text. Tests: tests/test_tolerant_tool_calls.cpp (10 assertions covering both modes, the real drift captured from a live session, and the must-still-fall- back cases). Existing ninfer_tool_call_parser_test still passes unchanged.
5 participants
Summary
Qwen3.6 can occasionally drift from its XML-like tool-call format during long agentic sessions. NInfer's strict parser then returns the complete response as text, so clients such as OpenCode display a naked
<tool_call>instead of receiving a structured tool call.This adds an opt-in
--tolerant-tool-callsserving mode that recovers complete function calls despite common Qwen wrapper/suffix drift, missing outer</tool_call>, or a call emitted before</think>.Design
Verification
ninfer_tool_call_parser_test: passed.ninfer_serve_options_test: passed.ninfer_request_log_test: passed.PYTHONPATH=. python3 -m pytest -q tests/test_serve_corpus.py: 2 passed.No automated live-model HTTP integration fixture is included.