Inline raw multimodal images in the v1 train client

[eligotts]

Summary

Intermediate multimodal storage mode — the inline bundle. Keeps the v1 raw-descriptor training contract (sidecars carry renderer descriptors + hashes + placeholders; processed tensors are rejected) but base64 data images stay inline in message content end to end — no ingress offload, no shared image directory.

• Removed verifiers/utils/multimodal.py (the ingress offload walker), the Client.prepare_request_body hook, its TrainClient override, and the interception server's request-preparation step: requests flow to the model boundary unmodified.
• graph.py sidecar validation requires the inline raw_image_data field on raw descriptors; processed-tensor payloads (pixel_values etc.) are still rejected.
• The v0 renderer_client passes messages through unmodified.
• Rollout records and traces carry the inline base64 images (deliberate for this bundle — expect larger results.jsonl on multimodal runs).

Relationship to the offload bundle

Cut from the offload branch (#1746); same processor-ownership architecture (renderer geometry math, vLLM-front + trainer materialization in prime-rl), minus the storage layer. The offload bundle (renderers #89 / verifiers #1746 / prime-rl #2836) adds content-addressed file:// offload at ingress on top of this contract.

Companion inline PRs: renderers #110, prime-rl feat/v1-inline-mm.

Validation

• Full suite with the inline renderers checkout (minus PRIME_API_KEY-gated e2e/envs): all passing.

Note

Inline raw multimodal images in the v1 train client bridging path

• Removes the multimodal-content prohibition in TrainClient.generate so bridging is now allowed for prompts with images; multimodal renderers receive concatenated previous-turn multimodal sidecars via previous_multi_modal_data during bridge_to_next_turn.
• Adds validation in graph.py that multimodal sidecars are raw image descriptor dicts (containing raw_image_data), rejecting processed payloads with pixel_values, image_embeds, or image_features at (de)serialization time.
• Extends multimodal attribution in _attribute_mm to track per-modality placeholder ranges alongside items and hashes, and exposes these via PendingTurn.previous_multi_modal_data and Branch.multi_modal_data.
• Propagates multi_modal_data through the legacy token conversion path in legacy.py and includes live trajectory in legacy rollout outputs.
• Risk: serialization of any previously accepted processed multimodal payloads will now raise TypeError/ValueError.

📊 Macroscope summarized 6e09559. 6 files reviewed, 0 issues evaluated, 0 issues filtered, 0 comments posted

🗂️ Filtered Issues

No issues evaluated.

[macroscopeapp]

🟡 Medium

verifiers/verifiers/v1/graph.py

Line 633 in 6e09559

MessageNode.model_construct(

The assistant node created in _commit_turn omits finish_reason and usage, so every committed response records finish_reason=None and usage=None even though Response carries those values. Truncation detection and provider usage/cost dashboards cannot work because the data is silently dropped. The MessageNode.model_construct(...) call for the assistant node needs to pass finish_reason=response.finish_reason and usage=response.usage.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file @verifiers/v1/graph.py around line 633:

The assistant node created in `_commit_turn` omits `finish_reason` and `usage`, so every committed response records `finish_reason=None` and `usage=None` even though `Response` carries those values. Truncation detection and provider usage/cost dashboards cannot work because the data is silently dropped. The `MessageNode.model_construct(...)` call for the assistant node needs to pass `finish_reason=response.finish_reason` and `usage=response.usage`.