UEmbed: Unified Sparse and Dense Multimodal Embeddings

Official repository for UEmbed: Unified Sparse and Dense Multimodal Embeddings.

UEmbed is a decoder-only multimodal embedding model that produces both dense embeddings and SPLADE-style sparse lexical embeddings from a single causal forward pass. It supports text, image, video, and mixed-modal inputs for retrieval, multimodal search, and visual-document retrieval.

---

Table of Contents

• Overview
• Features
• Model Overview
• Installation
• Download Models
• Usage
• Input Format
• Citation

---

Overview

UEmbed addresses learned sparse retrieval in decoder-only multimodal models. Traditional learned sparse retrievers are usually built on bidirectional encoder architectures and are mostly text-only. UEmbed instead keeps the causal decoder backbone and appends multiple special tokens to the input, allowing the model to produce a sparse vocabulary vector and a dense semantic vector in the same model.

The core design is:

• Dense representation: last-pool the EOS hidden state immediately before the appended special tokens.
• Sparse representation: append N=16 learnable special tokens, assign each token to a disjoint vocabulary subset, project each token hidden state with its own sparse head, and concatenate the outputs into one sparse vector.
• Unified training: optimize dense InfoNCE, sparse InfoNCE, and FLOPS regularization together.
• Multimodal support: use a Qwen3.5 multimodal backbone to encode text, image, video, and mixed-modal inputs.

---

Features

• Unified dense and sparse retrieval: one checkpoint can return normalized dense vectors and sparse lexical vectors.
• Multimodal inputs: text, images, videos, and mixed inputs can be represented in the same retrieval space.
• Sparse interpretability: sparse activations correspond to vocabulary terms and can be used with inverted indexes.
• Causal-model serving compatibility: the sparse design avoids converting the backbone into a bidirectional encoder.
• Public-data training recipe: the paper trains on E5, M3/MLDR, and MMEB data, with hard negatives mined for multimodal data.

---

Model Overview

Model Family

Model	Backbone	Parameters	Outputs	Modalities
UEmbed-2B	Qwen3.5	2B	Dense + Sparse	Text, image, video
UEmbed-4B	Qwen3.5	4B	Dense + Sparse	Text, image, video
UEmbed-9B	Qwen3.5	9B	Dense + Sparse	Text, image, video

Architecture

Component	Design
Backbone	Decoder-only Qwen3.5 multimodal model
Dense pooling	Hidden state of the EOS token before sparse special tokens
Sparse tokens	N=16 appended special tokens
Sparse heads	One subset-specific linear head per special token
Sparse vocabulary	Compressed from 248,320 tokenizer entries to 184,016 canonical entries in the paper
Sparse activation	log(1 + ReLU(logits))
Training objective	Dense InfoNCE + sparse InfoNCE + query/document FLOPS regularization

Training Data Summary

The paper reports training on 3.94M public samples from:

• E5 training data for broad text retrieval coverage.
• M3 training data, using the MLDR subset.
• MMEB training sets for multimodal query-document pairs.

For multimodal data, hard negatives are mined with Qwen3-VL-Embedding-8B as the teacher retriever.

---

Installation

Requires a recent transformers build with Qwen3.5/Qwen3-VL support.

pip install "transformers>=5.4.0" torch qwen-vl-utils tokenizers huggingface-hub pillow numpy

For faster inference on supported GPUs, install FlashAttention separately and pass attn_implementation="flash_attention_2" when loading the model.

---

Download Models

Download the complete model repository because sparse inference requires both sparse_info.json and sparse_weights.pt in the local model directory.

Model	Model ID	Local Directory
UEmbed-2B	Alibaba-NLP/UEmbed-2B	./models/UEmbed-2B
UEmbed-4B	Alibaba-NLP/UEmbed-4B	./models/UEmbed-4B
UEmbed-9B	Alibaba-NLP/UEmbed-9B	./models/UEmbed-9B

Download with Hugging Face Hub:

pip install huggingface-hub
huggingface-cli download Alibaba-NLP/UEmbed-2B --local-dir ./models/UEmbed-2B

---

Usage

Quick Start

Set pooling="last.normal" for dense embeddings or pooling="splade.last" for sparse embeddings. Instantiate only the mode needed by your retrieval pipeline.

import torch
from src.models.qwen35_embedding import Qwen35Embedder

model = Qwen35Embedder(
    model_name_or_path="./models/Qwen3-VL-Embedding-2B",
    # flash_attention_2 for better acceleration and memory saving
    torch_dtype=torch.bfloat16, 
    attn_implementation="flash_attention_2"
)

inputs = [{
    "text": "A woman playing with her dog on a beach at sunset.",
    "instruction": "Retrieve images or text relevant to the user's query.",
}, {
    "text": "A woman shares a joyful moment with her golden retriever on a sun-drenched beach at sunset, as the dog offers its paw in a heartwarming display of companionship and trust."
}, {
    "image": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg"
}, {
    "text": "A woman shares a joyful moment with her golden retriever on a sun-drenched beach at sunset, as the dog offers its paw in a heartwarming display of companionship and trust.", 
    "image": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg"
}]

embeddings = model.process(inputs)
print(embeddings @ embeddings.T)

Multimodal Inputs

Each input dictionary can contain text, image, video, or a mixture of modalities:

inputs = [
    {
        "text": "A photo of Beijing.",
        "image": "./assets/beijing.jpg",
        "instruction": "Represent the user's input.",
    },
    {
        "text": "A short city video.",
        "video": "./assets/city.mp4",
        "fps": 1.0,
        "max_frames": 8,
    },
]

embeddings = model.process(inputs)

Experiments

To run the experiments on your own data, just move the src/models/qwen35_embedding.py to Qwen3-VL-Embedding/src/models.

---

Input Format

Qwen35Embedder.process accepts a list of dictionaries. Each dictionary supports the following fields:

Field	Type	Description
text	str or list[str]	Text content.
image	path, URL, PIL.Image, or list	One or more images.
video	path, URL, frame list, or list	One or more videos.
instruction	str	Optional task-specific instruction.
fps	float	Optional frame sampling rate for video files.
max_frames	int	Optional maximum number of sampled video frames.

Examples:

{"text": "A text input"}
{"image": "./local_image.jpg"}
{"image": "https://example.com/image.jpg", "text": "An optional caption"}
{"video": "./local_video.mp4", "fps": 1.0, "max_frames": 8}

Use task-specific instructions for retrieval tasks, for example:

{"text": "...", "instruction": "Retrieve passages relevant to the user's query."}

Citation

If you use UEmbed, please cite the paper:

@misc{uembed2026,
      title={UEmbed: Unified Sparse and Dense Multimodal Embeddings}, 
      author={Tingyu Song and Mingxin Li and Yanzhao Zhang and Dingkun Long and Pengjun Xie and Zhijie Nie and Yilun Zhao and Shu Wu},
      year={2026},
      eprint={2608.02583},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2608.02583}, 
}

The citation entry will be updated when the final paper metadata is available.

Acknowledgements

Thanks for the Qwen3-VL-Embedding repo for the evaluation framework.