SCAFFOLD: A Large-Scale Structured Dataset of Computer Science Research Figures with Diagram QA and Chain-of-Thought Reasoning Traces is a large-scale multimodal reasoning dataset designed for training and evaluating Vision-Language Models (VLMs) on scientific figure understanding and visual reasoning.
The dataset is constructed from figures extracted from publicly available arXiv research papers and contains 157,387 question-answer pairs covering diverse scientific reasoning tasks. Each question is paired with a scientific figure and is accompanied by metadata such as reasoning type, difficulty, confidence, and an optional chain-of-thought reasoning trace.
SCAFFOLD is intended for research in Scientific Visual Question Answering (VQA), Multimodal Reasoning, Scientific Document Understanding, and Vision-Language Instruction Tuning.
Note: This dataset is released for research purposes only. Any unwanted use can lead to the violation of the intended terms of use and may result in legal action or revocation of access rights.
Existing multimodal reasoning datasets primarily focus on synthetic charts, general-purpose images, or document understanding tasks. Comparatively few datasets target real scientific figures extracted directly from research publications while simultaneously providing rich reasoning annotations suitable for instruction tuning.
SCAFFOLD bridges this gap by providing a large-scale collection of real scientific figures paired with diverse reasoning questions and chain-of-thought annotations. The dataset covers a broad spectrum of scientific visual content, including charts, plots, algorithms, architecture diagrams, flowcharts, conceptual illustrations, and experimental visualizations.
- 29,887 real scientific figures extracted from publicly available arXiv papers
- 157,387 multimodal question-answer pairs
- 3,058 research papers
- Figure-level reasoning annotations
- Multiple visual reasoning categories
- Difficulty and confidence annotations
- Optional chain-of-thought reasoning traces
- Instruction-tuning ready chat format
- Paper-level metadata for every source document
| Metric | Count |
|---|---|
| Research Papers | 3,058 |
| Scientific Figures | 29,887 |
| Total QA Pairs | 157,387 |
| Training Instances | 120,225 |
| Validation Instances | 36,797 |
SCAFFOLD supports research in:
- Vision-Language Instruction Tuning
- Scientific Visual Question Answering (VQA)
- Scientific Figure Understanding
- Scientific Diagram Understanding
- Chart and Plot Reasoning
- Multimodal Chain-of-Thought Reasoning
- Scientific Document Understanding
- Vision-Language Benchmarking
The dataset contains diverse reasoning tasks including:
- Boolean Reasoning
- Label Reading
- Counting
- Component Identification
- Comparative Reasoning
- Relational Reasoning
- Structural Reasoning
- Causal Reasoning
- Object Identification
SCAFFOLD/
│
├── README.md
├── LICENSE
├── records.jsonl
├── train.jsonl
├── validation.jsonl
├── pdf_metadata.jsonl
└── images.zip
| File | Description |
|---|---|
| images.zip | Archive containing all extracted scientific figures organized by paper ID. |
| records.jsonl | Complete dataset with every QA instance, metadata, and reasoning traces. |
| train.jsonl | Training split formatted for multimodal instruction tuning. |
| validation.jsonl | Validation split using the same format as the training data. |
| pdf_metadata.jsonl | Per-paper metadata including extraction statistics and QA distributions. |
Images are grouped by their source paper.
Example:
images/
└── 1112.1768/
├── 1112.1768_p08_fig00.png
├── 1112.1768_p09_fig01.png
└── ...
Each filename follows the convention
{paper_id}_p{page}_fig{figure_number}.png
Example:
1112.1768_p08_fig00.png
which corresponds to the first extracted figure on page 8 of paper 1112.1768.
Each record contains metadata describing a single visual reasoning instance.
| Field | Description |
|---|---|
figure_id |
Unique identifier of the QA instance |
paper_id |
Source arXiv paper identifier |
figure_number |
Figure number within the paper |
caption |
Figure caption (if available) |
ref_sentence |
Reference sentence extracted from the paper |
question |
Question associated with the figure |
answer |
Ground-truth answer |
question_type |
Visual reasoning category |
difficulty |
Difficulty level (easy, medium, hard) |
confidence |
Annotation confidence |
has_cot |
Indicates whether a reasoning trace is included |
cot_trace |
Chain-of-thought reasoning trace |
image_path |
Relative path to the corresponding figure |
Contains document-level metadata including:
- Number of extracted figures
- Number of generated QA pairs
- Question-type distribution
- Difficulty distribution
- Confidence distribution
- Figure extraction statistics
Training and validation sets are stored in multimodal chat format suitable for instruction tuning.
{
"messages": [
{
"role": "user",
"content": [
{
"type": "image",
"image": "images/1112.1768/1112.1768_p08_fig00.png"
},
{
"type": "text",
"text": "What algorithm is presented in the figure?"
}
]
},
{
"role": "assistant",
"content": [
{
"type": "text",
"text": ""
}
]
}
]
}SCARLET was generated through a fully automated data generation pipeline consisting of:
- Selection of publicly available arXiv papers licensed for redistribution.
- Figure extraction from research paper PDFs.
- Figure organization and metadata generation.
- Prompt construction using figure metadata.
- Automatic generation of question-answer pairs.
- Generation of chain-of-thought reasoning traces.
- Metadata aggregation and dataset splitting.
The complete data generation pipeline, preprocessing scripts, and utilities used to construct SCARLET are provided in the Code section of this dataset.
Additionally, scripts for downloading the original arXiv PDFs corresponding to the released dataset are also available in the Code section.
SCAFFOLD is suitable for:
- Vision-Language Model pretraining
- Multimodal instruction tuning
- Scientific Visual Question Answering
- Scientific figure understanding
- Scientific chart understanding
- Scientific diagram reasoning
- Multimodal reasoning research
- Chain-of-thought supervised fine-tuning
- Benchmarking multimodal reasoning systems
SCAFFOLD is automatically generated and should be used with the following considerations:
- Question-answer pairs are generated automatically and may occasionally contain inaccuracies.
- Chain-of-thought traces are machine-generated and should not be considered human-authored explanations.
- Some figures contain dense scientific notation or small text that may be difficult for current VLMs.
- Figure captions and reference sentences are unavailable for a subset of figures due to limitations in PDF extraction.
- The dataset is derived from arXiv publications and may not fully represent scientific literature from other publishers or disciplines.
Users should be aware of several potential sources of bias:
- The dataset is dominated by English-language scientific publications.
- Paper selection is limited to publicly available arXiv papers.
- Certain research domains (e.g., computer science, machine learning, and mathematics) are more heavily represented than others.
- Question-answer annotations reflect the capabilities and biases of the language model used during automatic generation.
If you use SCAFFOLD in your research, please cite:
@dataset{scaffold2026,
title={SCAFFOLD: A Large-Scale Structured Dataset of Computer Science Research Figures with Diagram QA and Chain-of-Thought Reasoning Traces},
author={Ranjit Raut, Aarav Subedi, Sagun Rai, and Sudan Jha},
year={2026},
publisher={}
}The annotations, metadata, dataset organization, and generated reasoning traces are released under the (CC BY NC 4.0)** license.
Scientific figures are extracted from publicly available arXiv papers. Copyright for each figure remains with its respective authors and is governed by the license of the corresponding source paper.
Each dataset instance contains the source paper_id, enabling users to trace every figure back to its original publication.
SCAFFOLD was created by automatically extracting scientific figures from publicly available arXiv research papers and generating multimodal reasoning annotations using large language models.
The authors gratefully acknowledge the arXiv repository and the original authors of the source publications for making their research openly accessible.