Feature request: Hugging Face mirror or alternative distribution for large The Well datasets

[Yangxingxing-dzw]

Hi, thank you for releasing The Well.

I am downloading turbulent_radiative_layer_3D using the documented
the-well-download workflow. The dataset is served from the current
SDSC/Flatiron static URLs rather than Hugging Face.

From a cloud instance in Asia, the direct download speed is around 10-15 KB/s.
For this dataset, the official train + test payload is about 639 GiB, making
the download practically infeasible despite resumable transfers.

Would it be possible to provide one of the following?

1. An official Hugging Face dataset mirror for large datasets;
2. A region-friendly mirror or object-storage endpoint;
3. Checksums and a manifest for verified third-party mirrors;
4. A documented multi-connection downloader for the current source.

A Hugging Face mirror would be especially helpful because it supports robust
resume and high-throughput transfer tools in many cloud environments.

Thank you.

[rubenohana]

Hello,

Regarding large datasets, HF storage can't be supported since they are too large for a free tier if I remember correctly (we were in contact with the Hugging Face data storage team at the time).
For now, the only way large datasets will be served is through the Flatiron storage. Maybe @gkrawezik @payelmuk150 could provide a more detailed answer?

Cheers,

Ruben