Skip to content

Inline raw image refs for multimodal rendering#110

Draft
eligotts wants to merge 22 commits into
mainfrom
feat/inline-raw-mm
Draft

Inline raw image refs for multimodal rendering#110
eligotts wants to merge 22 commits into
mainfrom
feat/inline-raw-mm

Conversation

@eligotts

@eligotts eligotts commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Summary

Intermediate multimodal storage mode — the inline bundle. Keeps this repo's raw multimodal contract (renderer computes token layout via pure geometry math, no image processor at render time; refs + descriptors instead of processed tensors) but images travel inline as data:image/...;base64 URLs instead of offloaded file:// run assets.

  • mm_store: the mmraw ref envelope carries raw_image_data (the inline data URL) instead of raw_image_uri. offload_image_to_run_assets, run_image_dir, and VF_RENDERER_IMAGE_OFFLOAD_DIR are gone. New decode_data_image_url() is the single supported source decode.
  • Qwen3-VL / Qwen3.5 / Kimi-K2.5 raw layout describes decode the inline source once for dimensions + sha256 hash; descriptors embed the source.
  • client.generate() serializes the inline source into every image slot's ref — every request to /inference/v1/generate carries the full inline payload for every image in the prompt (current and prior turns), and the prime-rl serving layer dedups processor work by content hash.

Relationship to the offload bundle

This PR is cut from the offload branch (#89) and is the storage-layer intermediate: same processor-ownership architecture, no shared image directory. The offload bundle (renderers #89 / verifiers #1746 / prime-rl #2836) layers content-addressed file:// offload on top of this contract.

Companion inline PRs: verifiers feat/v1-inline-raw-images, prime-rl feat/v1-inline-mm.

Validation

  • Full suite: 2153 passed, 121 skipped, 1 xfailed (processor-parity tests included, run against the real Qwen3-VL-4B and pinned Kimi-K2.5 processors).

Note

Send raw image refs to vLLM by default for multimodal rendering

  • Introduces a multimodal_output config field (default "raw") on BaseRendererConfig; renderers for Qwen 3.5, Qwen3-VL, and Kimi K2.5 now emit raw image reference items instead of processed pixel tensors unless multimodal_output="processed" is set.
  • Adds a new renderers/mm_store.py module with primitives for building, encoding, and decoding raw multimodal item references consumed by vLLM.
  • Layout metadata (grid dimensions, token counts) is now computed deterministically from a baked spec via describe_qwen_image_layout / describe_kimi_image_layout without requiring an HF processor, enabling token counting without loading model weights.
  • Processor loading is now lazy across all renderers; constructors no longer accept an injected processor, and the image_cache_max config field is removed from all renderer configs.
  • renderers/client.py serializes raw mm refs into vLLM feature payloads via _build_vllm_mm_features, replacing the prior processed-tensor dispatch path.
  • Behavioral Change: image parts in multi_modal_data now carry raw reference descriptors by default; callers expecting processed pixel tensors must set multimodal_output="processed".

Macroscope summarized d54e7fc.

eligotts and others added 22 commits June 18, 2026 07:03
Drop the cache-only None path. Every image (current and prior turns) carries its raw descriptor ref; _descriptor_only_mm_data no longer strips the pointer, so refs carry forward without a rebuild. Removes the now-orphaned materialize_image_refs / materialize_kimi_image_refs and the materialize_all_image_refs flag.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…tale comments

- Drop the render-time processor constructor arg from Qwen3VL/Qwen35/Kimi renderers: geometry is computed deterministically from config; no renderer runs the HF image processor at render. Remove Kimi dead _get_processor/_process_image/self._processor/_image_cache.

- mm_store: remove all backcompat aliases (MMRAW_PREFIX, MM_RAW_PAYLOAD_KEY/VALUE, mmraw_ref, split_mmraw_ref, image_asset_dir) -- no consumers.

- client.py: fix stale generate() docstring + comment that referenced the removed None/cache path.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
It only sized Kimi per-renderer image cache, which was deleted with the render-time processor path. No consumers.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…s-renderers

# Conflicts:
#	renderers/configs.py
#	renderers/qwen3_vl.py
- Fix bridge merge mutating the caller's previous sidecar in place:
  shared merge_multi_modal_data helper copies inner lists, replacing the
  three per-renderer merge blocks; bridge test asserts no mutation.
- Import Qwen's smart_resize from transformers (torch-free PIL-backend
  module) instead of maintaining a port.
- Resolve and read each raw image asset once per layout describe.
- mm_store: full sha256 content-addressed filenames; raise on
  undecodable base64 instead of silently passing the data URL through.
- Drop the dead features/mm_data tuple plumbing in client.generate.
- Add layout-math parity test against the real Qwen3-VL and Kimi-K2.5
  image processors at rounding-boundary dimensions.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…s-renderers

Weaves main's independent bridge-mutation fix and thinking-deviation
skip guard into this branch's raw/processed offload restructuring:

- kimi_k25.py/qwen35.py/qwen3_vl.py: kept our shared
  merge_multi_modal_data() helper over main's three duplicated inline
  fixes for the same bug (identical behavior, single implementation).
- test_multimodal.py: re-injected main's
  _skip_for_disabled_thinking_deviation guard into this branch's
  parallel renderer_cases/processor_cases structure at all 4 call
  sites; took main's tuple-based prior_mm/prior_counts assertion over
  our redundant duplicate.
- test_client.py: kept both additions (non-overlapping).

Full suite: 2578 passed, 153 skipped, 1 xfailed.
Intermediate storage mode: the image processor stays out of the render
path (raw geometry math + descriptors), but images travel inline as
data:image/...;base64 URLs instead of offloaded file:// run assets.

- mm_store: the mmraw ref envelope carries raw_image_data (the inline
  data URL) instead of raw_image_uri; offload_image_to_run_assets,
  run_image_dir, and VF_RENDERER_IMAGE_OFFLOAD_DIR are gone. New
  decode_data_image_url() is the one supported source decode.
- qwen3_vl/qwen35/kimi_k25: raw layout describes decode the inline
  source once for dimensions + sha256 hash; descriptors embed it.
- client: _build_vllm_mm_features serializes the inline source into
  every image slot's ref.

Suite: 2153 passed, 121 skipped, 1 xfailed (parity deselection as usual).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants