[codex] Support raw image refs for multimodal rendering

[eligotts]

Design update — inline/offload image storage

This PR now supports both raw image transport modes used by prime-rl:

• offload: existing behavior, raw image bytes are written to run-scoped image assets and refs carry a file-backed image id.
• inline: data-image URIs remain inline and raw refs carry the inline source instead of requiring raw_image_id.

This repo adds inline-capable mmraw:v3 refs while preserving mmraw:v2 parsing, keeps Qwen image hashes aligned to the raw decoded bytes, and emits raw descriptor items with either raw_image_id or raw_uri.

Validation after latest push: uv run pytest tests/test_client.py -q passed (14 passed).

Design update — dropped the None/cache-only image path

This PR and its companions (prime-rl #2836 / verifiers #1746 / renderers #89) no longer use the "send None for already-cached images" mechanism. Every image carries its raw descriptor ref at every slot (current and prior turns); /inference/v1/generate rematerializes each ref from disk every request.

Why: the None path coupled correctness to deployment (LRU cache present, single replica / DP-affinity, no eviction) and surfaced a miss as a hard vLLM EngineDeadError (qwen3-vl mrope dereferences a None image_grid_thw) that the retry net couldn't catch across the engine→API IPC. Dropping it is deployment-agnostic (a miss is impossible) and non-hacky. vLLM's mm_hash encoder cache still skips the expensive GPU re-encode for free — we only forgo the cheap IPC/CPU-reprocess dedup.

Validated: color-codeword (Qwen3-VL-4B) under DP=2, no affinity / no cache reliance: 0 crashes, 0 data=None, multi-turn accumulation correct, reward ~0.84. Also confirmed under TP.

This repo: every image emits a raw descriptor ref at every slot. _descriptor_only_mm_data no longer strips the pointer (pixel_values were never present in v1, so the strip was both stale and the root cause of the descriptor-only/rebuild churn). Removed the materialize_all_image_refs flag and the now-orphaned materialize_image_refs / materialize_kimi_image_refs.

---

Original description

Summary

• adds generic mmraw:v2 raw multimodal refs in renderers.mm_store, parsed as RawMMRef objects with family, fingerprint, modality, hash, asset id, and adapter-owned payload
• emits strict prime_raw_mm_item envelopes instead of processed image payloads for Qwen-VL and Kimi K2.5 image rendering
• keeps adapter-specific layout details in renderer-owned payloads (image_grid_thw for Qwen, grid_thws/media token metadata for Kimi)
• supports materializing all raw image refs for retry paths after vLLM multimodal cache misses
• keeps run-scoped image asset refs file-backed so downstream Prime-RL trainer materializes images with its own processor

Companion PRs

• Prime-RL: Support v1 raw multimodal image offload prime-rl#2836
• Verifiers: [codex] Support raw image offload in v1 train client verifiers#1746

Notes

• Draft/WIP: stacked with the Verifiers and Prime-RL raw image offload PRs.
• Verifiers is expected to offload image content to file://.../assets/images/... refs before rendering.
• This intentionally treats raw image refs as the supported path, not processed multimodal feature sidecars.

Validation

• uvx ruff@0.15.18 check . passed.
• uvx ruff@0.15.18 format --check . passed.
• uvx 'ty<0.0.22' check . exited 0; remaining diagnostics are warning-level advisories under the repo config.
• PYTHONPATH=/home/ubuntu/renderers uv run --no-project --active pytest -q tests/test_client.py passed: 14 passed.
• End-to-end hosted-style smoke through Prime-RL with /home/ubuntu/renderers, /home/ubuntu/verifiers, and /home/ubuntu/prime-rl-v1-raw-mm-offload completed inference, env rollouts, train batch creation, trainer step 0, and decoded strict trainer-bound raw image refs.

[!NOTE]

Add raw image ref support as default multimodal output for Qwen and Kimi renderers

• Renderers now emit raw image references with layout metadata (grid shape, token count, file URI) by default instead of processed pixel tensors; set multimodal_output='processed' in renderer config to retain the old behavior.
• Adds renderers/mm_store.py with utilities to construct, offload, and parse compact raw multimodal reference strings used by the client and renderers.
• The generate client now forwards raw multimodal descriptor refs to vLLM instead of encoding processor outputs; _build_mm_features is removed in favor of a modality-agnostic _build_vllm_mm_features.
• Processor loading is deferred and lazy across all renderers; the processor constructor kwarg is removed from Qwen3VLRenderer, Qwen35Renderer, and KimiK25Renderer.
• image_cache_max is removed from all renderer configs; processed-path cache size is now a fixed internal constant.
• Risk: any code passing processor= to renderer constructors or relying on pixel_values/image_grid_thw in multi_modal_data by default will break without opting into multimodal_output='processed'.

^{Macroscope summarized e64cc58.}

Update: review hardening (e3c12e9)

• Fixed render_completion_update mutating the caller's previous_multi_modal_data in place (shallow dict copy + setdefault().extend()); the merge now lives in a shared merge_multi_modal_data helper in base.py that copies inner lists, and the bridge test asserts the previous sidecar is unmutated.
• Qwen resize math is imported from transformers (torch-free PIL-backend module, with a fallback for older layouts) instead of maintaining a port.
• Raw layout describes resolve and read each image asset once; mm_store uses full sha256 content-addressed filenames and raises on undecodable base64.
• Added test_raw_layout_math_matches_image_processor: parity against the real Qwen3-VL-4B and Kimi-K2.5 (pinned revision) processors at rounding-boundary dimensions. Full suite: 2182 passed.

Update: merged main (e64cc58)

Reconciled with renderers main (12 commits: Hy3/PrimeQwen3/LagunaXS21 configs, kept-tokens parsing, SFT stop-token opt-in, a qwen thinking-wrapper fix, and bdb96b0 — main's own independent fix for the same bridge-mutation bug this PR's merge_multi_modal_data() already covers).

• kimi_k25.py/qwen35.py/qwen3_vl.py: kept this branch's shared merge_multi_modal_data() helper over main's three duplicated inline fixes for the identical bug — same behavior, one implementation instead of three.
• test_multimodal.py: re-injected main's _skip_for_disabled_thinking_deviation skip guard into this branch's parallel renderer_cases/processor_cases structure at all 4 call sites; took main's tuple-based prior_mm/prior_counts assertion over this branch's redundant duplicate check.
• test_client.py, __init__.py, configs.py: non-overlapping additions on both sides, no semantic conflict.
• client.py: no textual conflict (main's fix landed near, not on, this branch's rewritten _build_vllm_mm_features) — manually verified the merged generate() is consistent (prompt_attr/mm_data naming intact).

Full suite: 2578 passed, 153 skipped, 1 xfailed.

---

Note

High Risk
Large multimodal contract change across inference client, bridge merging, and three VL renderers; constructor API removal and mandatory offloaded image assets can break downstream integrations.

Overview
Multimodal inference path switches from embedding processed pixel_values in the sidecar to JSON-safe raw image descriptors (prime_raw_mm_item) with layout metadata and file:// URIs. Qwen-VL, Qwen3.5, and Kimi K2.5 compute placeholder counts via baked layout math (or transformers smart_resize for Qwen) without running the image processor in the default mode.

Adds multimodal_output on renderer config (raw default, processed for SFT/training), renderers/mm_store (offload, fingerprints, mmraw refs), and merge_multi_modal_data so bridge turns concatenate prior media without mutating the caller’s sidecar. generate() now builds vLLM features from raw refs only—removing renderer-specific torch/vLLM encoding—and expects offloaded images at every slot.

Breaking / API: VL renderer constructors no longer take processor=; processors lazy-load only for multimodal_output="processed". image_cache_max is removed from VL configs. New optional vision extra for Pillow/torch when using processed mode. Tests require file:// image fixtures for raw render parity.

^{Reviewed by Cursor Bugbot for commit a49e0fc. Bugbot is set up for automated code reviews on this repo. Configure here.}

[macroscopeapp]

Approvability

Verdict: Needs human review

This PR introduces a new multimodal rendering mode that changes default runtime behavior, adds a new module with 243 lines, and refactors multiple renderers. There is also an unresolved review comment identifying a potential missing dependency bug in the default code path.

^{You can customize Macroscope's approvability policy. Learn more.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit e64cc58. Configure here.}

[cursor]

Raw mode needs undeclared Pillow

Medium Severity

Default multimodal_output='raw' still calls _image_dimensions, which imports Pillow, but Pillow is only declared under the optional vision extra with torch/torchvision. A bare renderers install therefore fails on the default multimodal path before any processed tensors are involved.

Additional Locations (1)

• renderers/qwen3_vl.py#L210-L217

 

^{Reviewed by Cursor Bugbot for commit e64cc58. Configure here.}