On the Convergence of Attention Positions in SigLIP Across Different Images: A Contrast with CLIP

[HAOobserver]

In your paper, the figure 11 "Visualization of Redundancy in the SigLIP Model" shows the visualization of the attention maps of SigLIP. Why do the attentions of SigLIP roughly focus on the same positions for different images, while this problem does not occur in CLIP?

[HAOobserver]

like this

[Yangsenqiao]

To be honest, I’m also curious about this 🤔. Although the high-attention regions aren’t exactly the same, they do look very similar. My guess is that this might be related to the pretraining loss function used in SigLIP—but I’m not entirely sure. If anyone has insights, feel free to share! I'm really interested in how these foundation models behave 🔍

[xjtupanda]

To be honest, I’m also curious about this 🤔. Although the high-attention regions aren’t exactly the same, they do look very similar. My guess is that this might be related to the pretraining loss function used in SigLIP—but I’m not entirely sure. If anyone has insights, feel free to share! I'm really interested in how these foundation models behave 🔍

That's a behavior/feature of SigLIP ViT. Interested readers may refer to the Appendix of our paper for an empirical explanation. We also compare two variants of attention map extraction, and find that using the original attention pooling head of SigLIP could be better than pooling on the attention weights inside ViT.