How can i do continue pretraining?

[hwaking]

can you share the continue pretraining scripts and sample data?

[GoneZ5]

We apologize for the delayed response. To address your issue, please follow these steps:

1. First, use the run_offline_tokenization.sh script to tokenize your data.
2. Then, make the following modifications in the full_train_config.json file:
   • "data_paths": "[/workspace/data/data1,/workspace/data/data2]"
   • "data_weights": "[1.,1.]"
   • "tokenize_mode": "sst"
3. In the ds_multinode_launch.sh file, modify the entry script from pefts/mft_accelerate.py to mpt/mpt_accelerate.py.

Let us know if you need any further clarification or have additional questions.