Feat fsdp diloco - #29

Open

samsja wants to merge 12 commits into

mainfrom

feat-fsdp-diloco

Commits on Sep 10, 2024

wip add pure torch fsdp
samsja
committed
75ede2a
add something somewhat working
samsja
committed
ee89d33

Commits on Sep 11, 2024

add wandb and real data
samsja
committed
c217dd4
fix data
samsja
committed
94bc0ae
do all reduce on cpu
samsja
committed
5b701d7
do all reduce on cpu
samsja
committed
04dd585
fix opt step on cpu
samsja
committed
e926e6c
fix opt step on cpu
samsja
committed
9f680f2
fix batch size stuff
samsja
committed
8576f1c

Commits on Sep 24, 2024

add helper script
samsja
committed
71a9f89
fix it
samsja
committed
88b5250
fix it
samsja
committed
5957cae