Question about scaling of data

[Perkins729]

Hello,

I would like to ask whether a 6M parameter model would be too small for 4 million data samples, based on the details provided in the paper. Could you share any insights on the relative relationship between model size and data quantity?

Thank you!

[WhoKnowsssss]

Hi there,

Thanks for your question. We find that the model size scales more with the number of skills (i.e. diversity of the data), less with the data samples. For example, for the same 4 million data samples, if the data include 50 skills instead of 5, you would probably need a much larger model. If most data is repetitive (as in our case), a relatively small model is sufficient to capture the diversity within the dataset.