MCS-Bench

Introduction

With the rapid development of Multimodal Large Language Models (MLLMs), their potential in Chinese Classical Studies (CCS), a field which plays a vital role in preserving and promoting China’s rich cultural heritage, remains largely unexplored due to the absence of specialized benchmarks. To bridge this gap, we propose MCS-Bench, the first-of-its-kind multimodal benchmark specifically designed for CCS across multiple subdomains. MCS-Bench spans seven core subdomains (Ancient Chinese Text, Calligraphy, Painting, Oracle Bone Script, Seal, Cultural Relic, and Illustration), with a total of 45 meticulously designed tasks. Through extensive evaluation of 37 representative MLLMs, we observe that even the top-performing model (InternVL2.5-78B) achieves an average score below 50, indicating substantial room for improvement. Our analysis reveals significant performance variations across different tasks and identifies critical challenges in areas such as Optical Character Recognition (OCR) and cultural context interpretation. MCS-Bench not only establishes a standardized baseline for CCS-focused MLLM research but also provides valuable insights for advancing cultural heritage preservation and innovation in the Artificial General Intelligence (AGI) era.

Figure 1: Overview of MCS-Bench, covering 7 subdomains and 45 fine-grained tasks.

Data Examples

Figure 2 showcases examples from seven subdomains in MCS-Bench.

Figure 2: Data examples from MCS-Bench.

Data Acquisition Method

The MCS-Bench dataset is available for non-commercial research purposes only. To request access, please follow these steps:

Step 1: Download and complete the agreement:

application form

Have the document signed and stamped by your institution. Please also prepare 1–2 recent publications (within the last 6 years) as evidence that you or your team conduct research in handwriting verification, handwriting analysis and recognition, document image processing, or related fields.

Step 2: Submit your application online:

🔗 SCUT DLVC Lab Dataset Access Portal → Apply for MCS-Bench

Upload a signed document through the portal and fill out the "Recent Publications" block. Your application will be reviewed manually and you will be notified by email once a decision has been made (typically within 1–5 business days).

Step 3: Download the dataset:

After approval, you will receive the download link and decompression password via email.

⚠️ All users must comply with the use conditions at all times; failure to do so will result in revocation of access.

Inference

Calculate metrics based on the output of LLMs:

python calculate_metrics.py

Evaluation Metrics

Table 1: Results of 37 MLLMs on MCS-Bench. “MCQ” represents “Multiple-Choice Questions”; “QA” represents “open-ended Q&A”. Each score is the average of metrics across all corresponding tasks. Bold represents the best results, while underline represents the second best.

Table 2: Results of 37 MLLMs and 3 OCR models across six OCR tasks. Different colors represent different subdomains, with the color-domain mapping provided in Figure 1. In addition to CR, we also provide the F1-Score metric. “Average” represents the average metric of 37 MLLMs. Bold represents the best results, while underline represents the second best.

License

The work is licensed under a MIT License.

The dataset is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

Important Notice:

The original data of this dataset are collected from publicly accessible sources such as the Internet, and the copyright remains with the original content providers. The curated and annotated dataset reported in this case is intended for non-commercial use only and is currently licensed exclusively to universities and research institutions. If you wish to apply for access to this dataset, please complete the required application form in accordance with the instructions provided on the dataset website. The signature section of the application must be signed by a full-time staff member of a university or research institute. Where possible, please affix an official institutional seal (a seal from a secondary-level unit is acceptable) to facilitate the review and approval process.