feat(openai): forward image blocks on the Responses (Codex) path - #476
Merged
Conversation
The Responses transform's `push_blocks_as_items` handled text, tool_use and
tool_result blocks and dropped everything else via a `_ => {}` arm — including
`image` blocks. Its own doc comment claimed "text and image blocks collapse
into message items", but images never did. So a vision request through the
Codex/OAuth path (`/codex/responses`) reached the model with the image
silently removed: it saw only the text, and being the Codex agent it went
agentic ("let me look at the current folder…") instead of describing the
picture.
The Chat Completions path already mapped images to `image_url` parts; this
brings the Responses path to parity. `image` blocks now become `input_image`
content parts (base64 → `data:` URI, URL sources pass through), interleaved
with `input_text` in the same message item so a "describe this" + image turn
stays one message.
Verified live against gpt-5.5 over Codex OAuth with a solid-magenta PNG:
before (0.36.79, image dropped): model answered "Unknown"
after: model answered "Magenta"
and the outbound `/codex/responses` body carries
`{"type":"input_image","image_url":"data:image/png;base64,…"}`.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Destynova2
enabled auto-merge
July 26, 2026 07:17
Destynova2
added a commit
that referenced
this pull request
Jul 28, 2026
#488) ## The fidelity loss (targeted spot-check from the competitive analysis) The analysis names "tool arguments / images lost in translation" as the deadliest gateway defect class — `HTTP 200 + valid stream + visible text`, but the agent turn is semantically corrupt. I spot-checked grob's Responses/Codex path and found one: A tool that returns an image (screenshot, computer-use, an MCP vision tool) puts an `image` block in its `tool_result`. grob built the `function_call_output` with `content.to_string()`, and `ToolResultContent`'s `Display` renders an image block as the **literal text `[Image]`**. The model gets no pixels — and hallucinates. ## Verified live, end to end, through grob (gpt-5.5 over Codex OAuth) Two-turn agent loop: user asks the model to call `get_pixel`, the tool returns a solid-magenta PNG in its `tool_result`: ``` before (0.36.84, tool image → "[Image]"): model answered "White" ← hallucinated after (this PR, image → input_image part): model answered "Magenta" ← saw it ``` The Codex backend accepts `function_call_output.output` as an array of `input_text`/`input_image` parts — I probed the backend directly before writing the fix (both pure-image and mixed text+image accepted). Isolated the test daemon under its own `GROB_HOME` so the live check never touched the running daemon. ## Change - `push_blocks_as_items` now serializes a tool result via `responses_tool_output`: a **bare string** for text-only results (wire shape unchanged, prompt cache undisturbed), an **array of parts** when an image is present — text and image in original order. - `OpenAIResponsesItem::FunctionCallOutput.output` widened `String → serde_json::Value` to carry either shape (only construction site is this transform). - Unknown tool-result blocks are folded into text instead of silently dropped. ## Tests - `tool_result_text_stays_a_bare_string` — text path is byte-for-byte unchanged. - `tool_result_image_survives_as_input_image` — image → `input_image` data URI, no `[Image]`. - `cargo test --lib providers::openai::transform` — 38 passed. Same class as the message-content vision fix (#476); this closes the tool-output half. Feeds the loss-matrix work in #484. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bug behind "grob has no vision support"
grob does translate
imagecontent blocks — on the Chat Completions path(
image_url), Gemini (inline_data) and Anthropic (native). But the Responsestransform used by the Codex / ChatGPT-OAuth path silently dropped them.
push_blocks_as_itemsmatchedText,ToolUse,ToolResultand sent everythingelse to a
_ => {}arm. Its own doc comment said "text and image blocks collapseinto message items" — but
Imagewas never in the match. So a vision requestthrough
/codex/responsesreached the model with the image removed: it saw onlythe text, and being the Codex code agent it responded agentically
("let me look at the current folder…
{cmd: bash -lc 'rg --files'}") instead ofdescribing the picture. That's the exact symptom reported.
Fix
Bring the Responses path to parity with Chat Completions.
imageblocks becomeinput_imageparts — base64 →data:URI, URL sources pass through — interleavedwith
input_textin the same message item, so a "describe this" + image turnstays one message.
The Responses API takes
image_urlas a bare string (unlike Chat Completions,where it's an object) — handled.
Verified live (gpt-5.5 over Codex OAuth, a solid-magenta 16×16 PNG)
Outbound
/codex/responsesbody now carries:{ "type": "input_image", "image_url": "data:image/png;base64,iVBORw…" }Unit tests: base64 image →
input_imagedata URI interleaved after the textpart; URL source passes through.
cargo test --lib providers::openai::transform— 36 passed. Full local prek gate — clean.
Not in this PR (the reporter's other two points)
gpt-5.5not routable / "not configured": grob normalizes inbound modelnames dots→dashes (
gpt-5.5→gpt-5-5) for lookup but matches[[models]]names verbatim, so a dotted config name never matches. Workaround: name the
model entry
gpt-5-5. A real ergonomic fix (normalize config names too) is aseparate change.
choice, not a grob limitation — point a model/tier at a vision-first backend
(Claude, or Pixtral via an OpenAI-compatible endpoint) for clean image
description. With this PR, gpt-5.5 over Codex also describes images correctly.
🤖 Generated with Claude Code