Inline raw image refs for multimodal rendering#110
Draft
eligotts wants to merge 22 commits into
Draft
Conversation
Drop the cache-only None path. Every image (current and prior turns) carries its raw descriptor ref; _descriptor_only_mm_data no longer strips the pointer, so refs carry forward without a rebuild. Removes the now-orphaned materialize_image_refs / materialize_kimi_image_refs and the materialize_all_image_refs flag. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…tale comments - Drop the render-time processor constructor arg from Qwen3VL/Qwen35/Kimi renderers: geometry is computed deterministically from config; no renderer runs the HF image processor at render. Remove Kimi dead _get_processor/_process_image/self._processor/_image_cache. - mm_store: remove all backcompat aliases (MMRAW_PREFIX, MM_RAW_PAYLOAD_KEY/VALUE, mmraw_ref, split_mmraw_ref, image_asset_dir) -- no consumers. - client.py: fix stale generate() docstring + comment that referenced the removed None/cache path. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
It only sized Kimi per-renderer image cache, which was deleted with the render-time processor path. No consumers. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…s-renderers # Conflicts: # renderers/configs.py # renderers/qwen3_vl.py
- Fix bridge merge mutating the caller's previous sidecar in place: shared merge_multi_modal_data helper copies inner lists, replacing the three per-renderer merge blocks; bridge test asserts no mutation. - Import Qwen's smart_resize from transformers (torch-free PIL-backend module) instead of maintaining a port. - Resolve and read each raw image asset once per layout describe. - mm_store: full sha256 content-addressed filenames; raise on undecodable base64 instead of silently passing the data URL through. - Drop the dead features/mm_data tuple plumbing in client.generate. - Add layout-math parity test against the real Qwen3-VL and Kimi-K2.5 image processors at rounding-boundary dimensions. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…s-renderers Weaves main's independent bridge-mutation fix and thinking-deviation skip guard into this branch's raw/processed offload restructuring: - kimi_k25.py/qwen35.py/qwen3_vl.py: kept our shared merge_multi_modal_data() helper over main's three duplicated inline fixes for the same bug (identical behavior, single implementation). - test_multimodal.py: re-injected main's _skip_for_disabled_thinking_deviation guard into this branch's parallel renderer_cases/processor_cases structure at all 4 call sites; took main's tuple-based prior_mm/prior_counts assertion over our redundant duplicate. - test_client.py: kept both additions (non-overlapping). Full suite: 2578 passed, 153 skipped, 1 xfailed.
Intermediate storage mode: the image processor stays out of the render path (raw geometry math + descriptors), but images travel inline as data:image/...;base64 URLs instead of offloaded file:// run assets. - mm_store: the mmraw ref envelope carries raw_image_data (the inline data URL) instead of raw_image_uri; offload_image_to_run_assets, run_image_dir, and VF_RENDERER_IMAGE_OFFLOAD_DIR are gone. New decode_data_image_url() is the one supported source decode. - qwen3_vl/qwen35/kimi_k25: raw layout describes decode the inline source once for dimensions + sha256 hash; descriptors embed it. - client: _build_vllm_mm_features serializes the inline source into every image slot's ref. Suite: 2153 passed, 121 skipped, 1 xfailed (parity deselection as usual).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Intermediate multimodal storage mode — the inline bundle. Keeps this repo's raw multimodal contract (renderer computes token layout via pure geometry math, no image processor at render time; refs + descriptors instead of processed tensors) but images travel inline as
data:image/...;base64URLs instead of offloadedfile://run assets.mm_store: themmrawref envelope carriesraw_image_data(the inline data URL) instead ofraw_image_uri.offload_image_to_run_assets,run_image_dir, andVF_RENDERER_IMAGE_OFFLOAD_DIRare gone. Newdecode_data_image_url()is the single supported source decode.client.generate()serializes the inline source into every image slot's ref — every request to/inference/v1/generatecarries the full inline payload for every image in the prompt (current and prior turns), and the prime-rl serving layer dedups processor work by content hash.Relationship to the offload bundle
This PR is cut from the offload branch (#89) and is the storage-layer intermediate: same processor-ownership architecture, no shared image directory. The offload bundle (renderers #89 / verifiers #1746 / prime-rl #2836) layers content-addressed
file://offload on top of this contract.Companion inline PRs: verifiers
feat/v1-inline-raw-images, prime-rlfeat/v1-inline-mm.Validation
2153 passed, 121 skipped, 1 xfailed(processor-parity tests included, run against the real Qwen3-VL-4B and pinned Kimi-K2.5 processors).Note
Send raw image refs to vLLM by default for multimodal rendering
multimodal_outputconfig field (default"raw") onBaseRendererConfig; renderers for Qwen 3.5, Qwen3-VL, and Kimi K2.5 now emit raw image reference items instead of processed pixel tensors unlessmultimodal_output="processed"is set.renderers/mm_store.pymodule with primitives for building, encoding, and decoding raw multimodal item references consumed by vLLM.describe_qwen_image_layout/describe_kimi_image_layoutwithout requiring an HF processor, enabling token counting without loading model weights.image_cache_maxconfig field is removed from all renderer configs.renderers/client.pyserializes raw mm refs into vLLM feature payloads via_build_vllm_mm_features, replacing the prior processed-tensor dispatch path.multi_modal_datanow carry raw reference descriptors by default; callers expecting processed pixel tensors must setmultimodal_output="processed".Macroscope summarized d54e7fc.