fix(openai): preserve tool-result images on the Responses (Codex) path - #488
Merged
Conversation
A tool that returns an image (screenshot, computer-use, an MCP vision tool) puts an `image` block in its `tool_result`. On the Responses path grob built the `function_call_output` with `content.to_string()`, and `ToolResultContent`'s Display renders an image block as the literal text `[Image]`. The model received no pixels and hallucinated — verified live through grob against gpt-5.5: before: tool image → "[Image]" → model answered "White" after: tool image → input_image part → model answered "Magenta" The Codex backend accepts `function_call_output.output` as an array of `input_text`/`input_image` parts (probed directly); text-only results stay a bare string so the common path and prompt cache are unchanged. `output` widened from `String` to `serde_json::Value` to carry either shape. Unknown tool-result blocks are now folded into text instead of being dropped. Same class as the message-content vision fix; this closes the tool-output half. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Destynova2
enabled auto-merge (squash)
July 28, 2026 21:00
This was referenced Jul 28, 2026
Merged
Destynova2
added a commit
that referenced
this pull request
Jul 28, 2026
## 🤖 New release * `grob`: 0.36.84 -> 0.36.85 <details><summary><i><b>Changelog</b></i></summary><p> <blockquote> ## [0.36.85](v0.36.84...v0.36.85) - 2026-07-28 ### Fixed - *(openai)* preserve tool-result images on the Responses (Codex) path ([#488](#488)) ### Other - *(docs)* exclude flaky istio.io from lychee link check ([#483](#483)) </blockquote> </p></details> --- This PR was generated with [release-plz](https://github.com/release-plz/release-plz/).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The fidelity loss (targeted spot-check from the competitive analysis)
The analysis names "tool arguments / images lost in translation" as the deadliest
gateway defect class —
HTTP 200 + valid stream + visible text, but the agentturn is semantically corrupt. I spot-checked grob's Responses/Codex path and
found one:
A tool that returns an image (screenshot, computer-use, an MCP vision tool) puts
an
imageblock in itstool_result. grob built thefunction_call_outputwithcontent.to_string(), andToolResultContent'sDisplayrenders an image blockas the literal text
[Image]. The model gets no pixels — and hallucinates.Verified live, end to end, through grob (gpt-5.5 over Codex OAuth)
Two-turn agent loop: user asks the model to call
get_pixel, the tool returns asolid-magenta PNG in its
tool_result:The Codex backend accepts
function_call_output.outputas an array ofinput_text/input_imageparts — I probed the backend directly before writingthe fix (both pure-image and mixed text+image accepted). Isolated the test daemon
under its own
GROB_HOMEso the live check never touched the running daemon.Change
push_blocks_as_itemsnow serializes a tool result viaresponses_tool_output:a bare string for text-only results (wire shape unchanged, prompt cache
undisturbed), an array of parts when an image is present — text and image in
original order.
OpenAIResponsesItem::FunctionCallOutput.outputwidenedString → serde_json::Valueto carry either shape (only construction site is this transform).
Tests
tool_result_text_stays_a_bare_string— text path is byte-for-byte unchanged.tool_result_image_survives_as_input_image— image →input_imagedata URI, no[Image].cargo test --lib providers::openai::transform— 38 passed.Same class as the message-content vision fix (#476); this closes the tool-output
half. Feeds the loss-matrix work in #484.
🤖 Generated with Claude Code