Questions about training performance with multiple workers (v0.3.20)

[i7878]

We found that the number of workers significantly affects the training results. Topaz job were run through the CryoSPARC wrapper job.
Topaz v0.3.20:

For comparison, Topaz v0.2.5

Inputs were based on a subset of data from https://doi.org/10.6019/EMPIAR-10025
A user posted separate observations on the CryoSPARC discussion forum.

[DarnellGranberry]

Hi, thanks so much for bringing this to our attention. I'll see what I can do and hopefully find a solution tomorrow or next week.

[DarnellGranberry]

Hi @i7878 , sorry for the delay, but we've found a solution. When we recently fixed a bug that set num_workers=1 when using GPU, we unearthed a bug where each worker is seeded the same due to lack of a worker_init_fn and the model received the same batch num_workers-times in a row. In v0.2.5, we used a different dataset architecture with a dedicated Sampler object that took care of that. I will push a new version with that soon.

[DarnellGranberry]

Additionally, there seems to be another, separate issue in that CryoSPARC discussion.

@i7878 and @bsobol, would you be able to tell us more about the num_distribute parameter you're passing through cryosparc? That doesn't look like any topaz command/argument, I haven't been able to find any documentation about it, and the only place I've been able to find it otherwise has been in an error message in this other cryosparc forum post.

Is this similar to SLURM's --nodes=X argument to sbatch? Is it possible that there's a cryosparc error where they intended to allocate num_distribute*num_workers for both CPUs and workers, but accidentally left off the num_distribute factor when passing the product to topaz?

Even if that were the case, the 1x8 worker scenario being slower in 0.3.20 than 0.2.5 (19min vs 34min ) may indicate another issue. @bsobol, did your tests output any lines related to preloading data (as in this section of code)?

[bsobol]

Regarding num_distribute, I've described my observations/doubts in the first post here https://discuss.cryosparc.com/t/topaz-integration-inefficiencies/24895. I believe it's a historical feature of the wrapper to introduce multiprocessing during the preprocessing stage in Topaz (maybe CryoSPARC devs can say more), which is now redundant and misleading, and can lead to inefficient use of resources. However, I don't think that it is related to the core of the issue.

Is this similar to SLURM's --nodes=X argument to sbatch?

IIRC from my reverse engineering of the wrapper:
num_workers sets --num-workers flag in Topaz
num_distribute is an internal parameter of the wrapper
Together, they control the number of CPU cores (num_workers*num_distribute), and everything always stays on a single node.

@bsobol, did your tests output any lines related to preloading data

Estimated dataset memory footprint: 0.02 GB. Available system memory: 899.19 GB. Setting preload to True.