Skip to content

fix(frontend): keep literal vision tokens out of media binding - #88

Open
geoffwatts wants to merge 1 commit into
Neroued:masterfrom
geoffwatts:fix/anthropic-tool-result-media
Open

fix(frontend): keep literal vision tokens out of media binding#88
geoffwatts wants to merge 1 commit into
Neroued:masterfrom
geoffwatts:fix/anthropic-tool-result-media

Conversation

@geoffwatts

Copy link
Copy Markdown

A valid Anthropic Messages request containing parallel tool_use history and
image-bearing tool_result blocks returned:

  400 chat media order does not match rendered placeholders

The captured request returned tool results in completion order rather than the
preceding invocation order. It also contained literal Qwen Vision control-token
spellings inside system text that quoted a chat template.

Root cause

The Anthropic schema path already preserved both tool_result order and nested
content order. Media collection followed that same order.

The mismatch occurred later in the Qwen frontend. Media binding searched the
complete rendered prompt for raw <|image_pad|> and <|video_pad|> strings.
Literal occurrences in system text, reasoning, tool definitions, or tool
arguments were therefore mistaken for structured media placeholders. This
shifted binding away from the actual image blocks and produced the ordering
error.

Design

The compiled chat renderer now neutralizes exact Qwen Vision control-token
spellings in textual input by inserting U+2060 WORD JOINER after <|.

This keeps the quoted text readable while preventing it from becoming an exact
added token or media-binding marker. The rule is applied to:

  • text content;
  • explicit reasoning content;
  • serialized tool definitions; and
  • rendered tool-call arguments.

Structured image and video parts continue to emit the original raw control
tokens and remain the sole source of media placeholders.

Tool results are not sorted or remapped to the original tool_use order.
Message, block, and nested-content traversal remain in client-supplied order.

Affected behavior

  • Image-bearing tool_result blocks may arrive in a different order from the
    preceding parallel tool_use blocks.
  • Mixed text/image results and multiple images within one result retain their
    nested order.
  • Literal Vision control tokens remain ordinary text.
  • Direct user images and text-only tool results are unchanged.

The serving documentation now states these ordering and literal-token
semantics.

Verification

Clean V100-port checkout, Tesla V100-SXM2-32GB, GCC 13.3, CUDA 12.8.61:

cmake -S . -B build-cuda128 \
  -DCMAKE_BUILD_TYPE=Release \
  -DBUILD_TESTING=ON \
  -DCMAKE_CUDA_ARCHITECTURES=70 \
  -DCMAKE_CUDA_COMPILER=/usr/local/cuda-12.8/bin/nvcc

cmake --build build-cuda128 -j \
  --target ninfer_anthropic_schema_test ninfer_qwen3_6_frontend_test

ctest --test-dir build-cuda128 \
  -R '^ninfer_anthropic_schema_test$' \
  --output-on-failure

Results:

- both focused test executables built successfully;
- ninfer_anthropic_schema_test: passed;
- git diff --check origin/master...HEAD: passed;
- captured 58,922-token Anthropic request, with max_tokens reduced to 1:
  HTTP 200, one output token, no media-order error;

- ordinary direct-user image through /v1/messages/count_tokens:
  HTTP 200, 415 input tokens.

The complete ninfer_qwen3_6_frontend_test executable could not run on this
host because an existing test reads tokenizer resources from the maintainer’s
hard-coded /home/neroued/... path, which is not present. The executable
compiled successfully, and the affected frontend preparation path was exercised
by the live captured request.

The exact upstream sm_120a configuration was not built because this host does
not have the required CUDA 13.1/RTX 5090 environment. No performance claim is
made.

## Related work

#71 adds multimodal tool outputs to the OpenAI Responses schema. This change is
independent: it fixes shared Qwen frontend media binding and the reproduced
Anthropic Messages failure.

## AI-assisted implementation disclosure

The implementation and tests were generated and reviewed in OpenAI Codex using
GPT-5. The session did not expose a more specific model revision.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5eb3176a23

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/targets/qwen3_6/impl/frontend/chat_template.cpp Outdated
Neutralize exact Qwen Vision control-token spellings in textual chat inputs so only structured image and video parts create media placeholders.

Preserve Anthropic tool_result and nested content order, and cover reversed parallel results with mixed and repeated media.
@geoffwatts
geoffwatts force-pushed the fix/anthropic-tool-result-media branch from 5eb3176 to 1c4bfa5 Compare August 24, 2026 13:38
@geoffwatts

Copy link
Copy Markdown
Author

The renderer regression now splits all four Vision control tokens across text-part boundaries while structured media is present.

Verification:

  • split-token renderer regression passed;
  • ninfer_anthropic_schema_test passed;
  • both focused test targets built successfully.

Thanks for catching this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant