VQA testing

[zza622]

Hello, during the VQA testing, I used the second-stage pre-training weights and conducted one round of training on the LLAVA OneVision dataset for evaluation. I found that some metrics differed significantly from the results reported in the paper (such as OCRBench, DocVQA, etc.). Could you please provide detailed information about your training configuration, or make the corresponding evaluation weights publicly available?

[speedinghzl]

Hi @zza622

Thanks for your interest!
To help us locate the issue faster, could you share a few more details?

Training config — your exact fine-tuning settings.
Specific gaps — which benchmarks differ and the exact numbers (yours vs. paper).
Eval toolkit — which framework/version you used (lmms-eval, VLMEvalKit, custom script), since these strongly affect OCR/Doc benchmarks.

[zza622]

LLM backbone: Qwen2.5-1.5B-Instruct
Vision encoder: GenLIP, path: genlip-l-navit
Trainable modules: mm_projector + language model
Frozen modules: vision tower
Projector: 2-layer MLP, mlp2x_gelu
Vision feature layer: last layer, mm_vision_select_layer = -1
Image preprocessing: resize shortest edge to 224, center crop 224x224
AnyRes: disabled
Image aspect ratio: square
Precision: bf16
Attention: flash_attention_2
DeepSpeed: ZeRO-3, scripts/zero3.json
Epochs: 1
LR: 1e-5
Weight decay: 0
Warmup ratio: 0.03
LR scheduler: cosine
Gradient checkpointing: enabled
TF32: enabled
Dataloader workers: 8
Drop last: true
per_device_train_batch_size = 2
gradient_accumulation_steps = 1
num_nodes = 4
gpus_per_node = 8
effective global batch = 64
model_max_length = 16384
PROMPT_VERSION = qwen_1_5
The image shows the metrics I am currently testing. Some of the differences are too large, and I want to know what the problem is.

[zza622]

Framework: lmms-eval / LMMs-Eval
Local version in pyproject.toml: 0.7.1
This is my assessment instruction
cd /lpai/volumes/base-ov-ali-sh-mix/zhouziang/2d_encoder/GenLIP/src/lmms-eval
export EVAL_MODEL=/lpai/volumes/base-ov-ali-sh-mix/zhouziang/result/genlip/llava_genlip_qwen25_1p5b
export EVAL_OUT=/lpai/volumes/base-ov-ali-sh-mix/zhouziang/result/genlip/eval_logs/seedbench_2_plus_full
CUDA_VISIBLE_DEVICES=3 accelerate launch --num_processes=1 -m lmms_eval
--model llava_onevision
--model_args pretrained=$EVAL_MODEL,conv_template=qwen_1_5,model_name=llava_qwen,device_map=auto,attn_implementation=flash_attention_2,torch_dtype=bfloat16
--tasks seedbench_2_plus
--batch_size 1
--log_samples
--log_samples_suffix seedbench_2_plus_full
--output_path $EVAL_OUT

[YanFangCS]

Hi @zza622,

Thanks for your interest!

The gap may be related to several implementation details in the VQA fine-tuning / evaluation pipeline. In particular, please check the following two points:

1. Input image resolution and normalization. In our implementation, images are resized to 384 × 384 rather than 224 × 224, and we use normalization with mean=[0.5, 0.5, 0.5] and std=[0.5, 0.5, 0.5]. Using 224px inputs can noticeably affect the results, especially on OCR- and document-oriented benchmarks such as OCRBench and DocVQA. You can also try to use image processor of google/siglip2-so400m-patch16-384 for convenience.
2. Layer-normalized image features. We use the image features after the final LayerNorm of the vision encoder. Using pre-LN features may also lead to different downstream results.

Could you please check whether these two settings are consistent with your setup? If your current reproduction uses 224 × 224 inputs or pre-LN features, this may also lead to inferior performance.

Regarding the detailed VQA training configuration and evaluation weights: we are further checking and cleaning up the corresponding fine-tuning/evaluation recipe these days. If possible, we will provide more detailed configuration information or release the evaluation checkpoint later. In the meantime, if you can share your training command, image preprocessing settings, and evaluation config, we can help further diagnose the discrepancy.