Fine-tuning Problem

[lky-violet]

Dear author,

Thank you for open-sourcing such an excellent work with DDCoT. I have a question regarding the training process.

In Figure 5 of the paper, it is indicated that the Rationales are encoded and input into the model for fine-tuning. I understand Rationales to represent the reasoning process involving the breakdown into sub-questions and corresponding answers. However, when examining the fine-tuning code, I noticed that in main.py, the training dataset being loaded is the description of questions from ScienceQA, similar to MMCoT. Therefore, I would like to clarify what the input Rationales specifically consist of during the fine-tuning process of DDCoT.

Thank you very much for your time, and I look forward to your response!

[zszh-gezheng]

Thank you very much for your recognition and interest in our work with DDCoT.

To clarify, during the fine-tuning process, we use the Rationales generated after applying the Integrate step for training. The detailed process for generating these Rationales is described in Section 3.2 of our paper. The loading process is implemented in the ScienceQADatasetImg class within the utils_data.py file. In the code, we use new_rationale_train.json and new_rationale_test.json as placeholders, which should be replaced with the files containing the Rationales generated following the described process.

Please feel free to reach out if you have further questions!