Skip to content

feat(openai): forward image blocks on the Responses (Codex) path - #476

Merged
Destynova2 merged 1 commit into
mainfrom
feat/codex-responses-vision
Jul 26, 2026
Merged

feat(openai): forward image blocks on the Responses (Codex) path#476
Destynova2 merged 1 commit into
mainfrom
feat/codex-responses-vision

Conversation

@Destynova2

Copy link
Copy Markdown
Contributor

The bug behind "grob has no vision support"

grob does translate image content blocks — on the Chat Completions path
(image_url), Gemini (inline_data) and Anthropic (native). But the Responses
transform
used by the Codex / ChatGPT-OAuth path silently dropped them.

push_blocks_as_items matched Text, ToolUse, ToolResult and sent everything
else to a _ => {} arm. Its own doc comment said "text and image blocks collapse
into message items"
— but Image was never in the match. So a vision request
through /codex/responses reached the model with the image removed: it saw only
the text, and being the Codex code agent it responded agentically
("let me look at the current folder… {cmd: bash -lc 'rg --files'}") instead of
describing the picture. That's the exact symptom reported.

Fix

Bring the Responses path to parity with Chat Completions. image blocks become
input_image parts — base64 → data: URI, URL sources pass through — interleaved
with input_text in the same message item, so a "describe this" + image turn
stays one message.

The Responses API takes image_url as a bare string (unlike Chat Completions,
where it's an object) — handled.

Verified live (gpt-5.5 over Codex OAuth, a solid-magenta 16×16 PNG)

before (0.36.79, image dropped):  model answered "Unknown"
after:                            model answered "Magenta"

Outbound /codex/responses body now carries:

{ "type": "input_image", "image_url": "data:image/png;base64,iVBORw…" }

Unit tests: base64 image → input_image data URI interleaved after the text
part; URL source passes through. cargo test --lib providers::openai::transform
— 36 passed. Full local prek gate — clean.

Not in this PR (the reporter's other two points)

  • gpt-5.5 not routable / "not configured": grob normalizes inbound model
    names dots→dashes (gpt-5.5gpt-5-5) for lookup but matches [[models]]
    names verbatim, so a dotted config name never matches. Workaround: name the
    model entry gpt-5-5. A real ergonomic fix (normalize config names too) is a
    separate change.
  • The dev config's routable models are code-oriented (Codex): that's a config
    choice, not a grob limitation — point a model/tier at a vision-first backend
    (Claude, or Pixtral via an OpenAI-compatible endpoint) for clean image
    description. With this PR, gpt-5.5 over Codex also describes images correctly.

🤖 Generated with Claude Code

The Responses transform's `push_blocks_as_items` handled text, tool_use and
tool_result blocks and dropped everything else via a `_ => {}` arm — including
`image` blocks. Its own doc comment claimed "text and image blocks collapse
into message items", but images never did. So a vision request through the
Codex/OAuth path (`/codex/responses`) reached the model with the image
silently removed: it saw only the text, and being the Codex agent it went
agentic ("let me look at the current folder…") instead of describing the
picture.

The Chat Completions path already mapped images to `image_url` parts; this
brings the Responses path to parity. `image` blocks now become `input_image`
content parts (base64 → `data:` URI, URL sources pass through), interleaved
with `input_text` in the same message item so a "describe this" + image turn
stays one message.

Verified live against gpt-5.5 over Codex OAuth with a solid-magenta PNG:

  before (0.36.79, image dropped):  model answered "Unknown"
  after:                            model answered "Magenta"

and the outbound `/codex/responses` body carries
`{"type":"input_image","image_url":"data:image/png;base64,…"}`.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@Destynova2
Destynova2 enabled auto-merge July 26, 2026 07:17
@Destynova2
Destynova2 merged commit 7e0182a into main Jul 26, 2026
41 checks passed
@Destynova2
Destynova2 deleted the feat/codex-responses-vision branch July 26, 2026 07:23
Destynova2 added a commit that referenced this pull request Jul 28, 2026
#488)

## The fidelity loss (targeted spot-check from the competitive analysis)

The analysis names "tool arguments / images lost in translation" as the
deadliest
gateway defect class — `HTTP 200 + valid stream + visible text`, but the
agent
turn is semantically corrupt. I spot-checked grob's Responses/Codex path
and
found one:

A tool that returns an image (screenshot, computer-use, an MCP vision
tool) puts
an `image` block in its `tool_result`. grob built the
`function_call_output` with
`content.to_string()`, and `ToolResultContent`'s `Display` renders an
image block
as the **literal text `[Image]`**. The model gets no pixels — and
hallucinates.

## Verified live, end to end, through grob (gpt-5.5 over Codex OAuth)

Two-turn agent loop: user asks the model to call `get_pixel`, the tool
returns a
solid-magenta PNG in its `tool_result`:

```
before (0.36.84, tool image → "[Image]"):   model answered "White"   ← hallucinated
after  (this PR, image → input_image part):  model answered "Magenta" ← saw it
```

The Codex backend accepts `function_call_output.output` as an array of
`input_text`/`input_image` parts — I probed the backend directly before
writing
the fix (both pure-image and mixed text+image accepted). Isolated the
test daemon
under its own `GROB_HOME` so the live check never touched the running
daemon.

## Change

- `push_blocks_as_items` now serializes a tool result via
`responses_tool_output`:
a **bare string** for text-only results (wire shape unchanged, prompt
cache
undisturbed), an **array of parts** when an image is present — text and
image in
  original order.
- `OpenAIResponsesItem::FunctionCallOutput.output` widened `String →
serde_json::Value`
  to carry either shape (only construction site is this transform).
- Unknown tool-result blocks are folded into text instead of silently
dropped.

## Tests

- `tool_result_text_stays_a_bare_string` — text path is byte-for-byte
unchanged.
- `tool_result_image_survives_as_input_image` — image → `input_image`
data URI, no `[Image]`.
- `cargo test --lib providers::openai::transform` — 38 passed.

Same class as the message-content vision fix (#476); this closes the
tool-output
half. Feeds the loss-matrix work in #484.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant