Panda-MultiRef-21 is a multi-reference annotation dataset containing a set of relatively difficult problems from Panda-CVL. The annotations are in English, and the same set of 21 problems is provided with four independent panda.json annotations.
All four annotations for each problem start from the same initial rollout. The dataset is intended to study the distribution of token-level corrections and the consistency and variation among annotators in first-error localization and correction direction. Compared with a single reference answer, multiple annotations provide a more complete picture of the range of reasonable corrections. One of the four annotation sets was converted from post editing data.
The rollout policy was Qwen3.5-35B-A3B Instruct, with temperature=0.7 and top_p=0.8.
Panda-CVL is a Chinese vision-language dataset annotated with onPanda. Although Panda-MultiRef-21 was added later, it was incorporated into Panda-CVL and is therefore a subset of Panda-CVL. It consists of relatively difficult problems selected from Panda-CVL, annotated in English, and follows the same panda.json format and annotation scheme while preserving multiple annotations for each problem to support analysis of annotation distributions and consistency.
To remain consistent with Panda-CVL, Panda-MultiRef-21 is also split into train and test. However, because the dataset contains only 21 problems, if the model has not been trained on the data in train, we recommend using all 21 problems together as a benchmark.
Drag any .panda.json file into the onPanda Web App to view it.
The archive password is not-train-corpora. In Bash, run:
unzip -P 'not-train-corpora' 'Panda-MultiRef-21_password(not-train-corpora).zip'- GitHub repository: Panda-MultiRef-21
- Project homepage: onPanda
- Paper: onPanda: Efficient Annotation of On-Policy Alignment Data for LLMs and Agents via Token-Level Correction
This dataset is in the public domain and is released under the CC0 (Public Domain) license.