Skip to content

[Feature] Support image-text multimodal input for training #1

Description

@slepshep106

Motivation

Training in AngelSpec is text-only today, so vision-language targets (Qwen3.5,
Qwen3-VL, Hy3-VL) can't be used. We'd like to train DFlash/DSpark drafters for
VLM targets on image-text multimodal input.

Some of the plumbing already exists (media extraction in the dataset, an
image_placeholder in templates, enable_multimodal in the inference config,
multimodal_inputs passed to vLLM). The gap is on the draft-training side,
where block-parallel drafters have no mRoPE support.

Proposed Solution

Support image-text records end to end: accept images in the training data, carry
the target's mRoPE positions through to the draft training forward, and make the
block-parallel draft RoPE mRoPE-aware.

The drafter core shouldn't need to change — KV injection, anchor sampling, and
the losses are all modality-agnostic — and since DFlash, DSpark, and DFly share
a backbone, one implementation should cover all three.

Happy to help implement or test.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions