Skip to content

fix(openai): preserve tool-result images on the Responses (Codex) path - #488

Merged
Destynova2 merged 1 commit into
mainfrom
fix/codex-tool-result-images
Jul 28, 2026
Merged

fix(openai): preserve tool-result images on the Responses (Codex) path#488
Destynova2 merged 1 commit into
mainfrom
fix/codex-tool-result-images

Conversation

@Destynova2

Copy link
Copy Markdown
Contributor

The fidelity loss (targeted spot-check from the competitive analysis)

The analysis names "tool arguments / images lost in translation" as the deadliest
gateway defect class — HTTP 200 + valid stream + visible text, but the agent
turn is semantically corrupt. I spot-checked grob's Responses/Codex path and
found one:

A tool that returns an image (screenshot, computer-use, an MCP vision tool) puts
an image block in its tool_result. grob built the function_call_output with
content.to_string(), and ToolResultContent's Display renders an image block
as the literal text [Image]. The model gets no pixels — and hallucinates.

Verified live, end to end, through grob (gpt-5.5 over Codex OAuth)

Two-turn agent loop: user asks the model to call get_pixel, the tool returns a
solid-magenta PNG in its tool_result:

before (0.36.84, tool image → "[Image]"):   model answered "White"   ← hallucinated
after  (this PR, image → input_image part):  model answered "Magenta" ← saw it

The Codex backend accepts function_call_output.output as an array of
input_text/input_image parts — I probed the backend directly before writing
the fix (both pure-image and mixed text+image accepted). Isolated the test daemon
under its own GROB_HOME so the live check never touched the running daemon.

Change

  • push_blocks_as_items now serializes a tool result via responses_tool_output:
    a bare string for text-only results (wire shape unchanged, prompt cache
    undisturbed), an array of parts when an image is present — text and image in
    original order.
  • OpenAIResponsesItem::FunctionCallOutput.output widened String → serde_json::Value
    to carry either shape (only construction site is this transform).
  • Unknown tool-result blocks are folded into text instead of silently dropped.

Tests

  • tool_result_text_stays_a_bare_string — text path is byte-for-byte unchanged.
  • tool_result_image_survives_as_input_image — image → input_image data URI, no [Image].
  • cargo test --lib providers::openai::transform — 38 passed.

Same class as the message-content vision fix (#476); this closes the tool-output
half. Feeds the loss-matrix work in #484.

🤖 Generated with Claude Code

A tool that returns an image (screenshot, computer-use, an MCP vision tool)
puts an `image` block in its `tool_result`. On the Responses path grob built the
`function_call_output` with `content.to_string()`, and `ToolResultContent`'s
Display renders an image block as the literal text `[Image]`. The model received
no pixels and hallucinated — verified live through grob against gpt-5.5:

  before: tool image → "[Image]"        → model answered "White"
  after:  tool image → input_image part → model answered "Magenta"

The Codex backend accepts `function_call_output.output` as an array of
`input_text`/`input_image` parts (probed directly); text-only results stay a
bare string so the common path and prompt cache are unchanged. `output` widened
from `String` to `serde_json::Value` to carry either shape. Unknown tool-result
blocks are now folded into text instead of being dropped.

Same class as the message-content vision fix; this closes the tool-output half.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@Destynova2
Destynova2 enabled auto-merge (squash) July 28, 2026 21:00
@Destynova2
Destynova2 merged commit c401176 into main Jul 28, 2026
38 checks passed
@Destynova2
Destynova2 deleted the fix/codex-tool-result-images branch July 28, 2026 21:09
Destynova2 added a commit that referenced this pull request Jul 28, 2026
## 🤖 New release

* `grob`: 0.36.84 -> 0.36.85

<details><summary><i><b>Changelog</b></i></summary><p>

<blockquote>

##
[0.36.85](v0.36.84...v0.36.85)
- 2026-07-28

### Fixed

- *(openai)* preserve tool-result images on the Responses (Codex) path
([#488](#488))

### Other

- *(docs)* exclude flaky istio.io from lychee link check
([#483](#483))
</blockquote>


</p></details>

---
This PR was generated with
[release-plz](https://github.com/release-plz/release-plz/).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant