[Bug]: Windows: os.path.join breaks hf:// streaming paths in WellDataset (HFValidationError)

[Andrew-DCA]

Summary

On Windows, WellDataset builds its data path with os.path.join, which inserts backslashes. When well_base_path is a remote URI (hf://...), the backslashes end up inside the Hugging Face repo path and HfFileSystem rejects it with HFValidationError, so Hugging Face streaming is unusable on Windows via the documented constructor.

PR #81 fixed the same class of problem for normalization_path, but the data_path joins in WellDataset.__init__ still use os.path.join (the_well/data/datasets.py, lines 231, 237-240 on current master).

Environment

• Windows 11, Python 3.14.3
• the_well 1.2.0 (from PyPI; code identical on current master)
• huggingface_hub 1.23.0, fsspec 2024.10.0, torch 2.13.0+cpu

Reproduction

from the_well.data import WellDataset

ds = WellDataset(
    well_base_path="hf://datasets/polymathic-ai/",
    well_dataset_name="turbulent_radiative_layer_2D",
    well_split_name="train",
    n_steps_input=4,
    n_steps_output=1,
)

Result

huggingface_hub.errors.HfUriError: Invalid HF URI 'hf://datasets/polymathic-ai/turbulent_radiative_layer_2D\data\train/*.h5'.
Repo id must use alphanumeric chars, '-', '_' or '.'. The name cannot start or end with '-' or '.' and the maximum length is 96:
'polymathic-ai/turbulent_radiative_layer_2D\data\train'

(raised from self.fs.glob(self.data_path + "/*.h5") in WellDataset.__init__; on the installed 1.2.0 the underlying error surfaces as HFValidationError via fsspec)

Note the mixed separators in the URI: os.path.join("hf://datasets/polymathic-ai/", "turbulent_radiative_layer_2D", "data", "train") returns 'hf://datasets/polymathic-ai/turbulent_radiative_layer_2D\\data\\train' on Windows.

Suggested fix

URI-style paths should never go through os.path.join. One option, mirroring the approach of #81:

if "://" in str(well_base_path):
    self.data_path = f"{str(well_base_path).rstrip('/')}/{well_dataset_name}/data/{well_split_name}"
else:
    self.data_path = os.path.join(well_base_path, well_dataset_name, "data", well_split_name)

(same treatment for trunk_path, the path+well_split_name branch at line 231, and the normalization_path join at lines 242-244). Alternatively posixpath.join for the remote branch. fsspec itself treats remote paths as POSIX regardless of platform, so forward slashes are always safe there.

Workaround

Passing the fully joined URI directly avoids all the joins:

ds = WellDataset(
    path="hf://datasets/polymathic-ai/turbulent_radiative_layer_2D/data/train",
    well_split_name=None,
    use_normalization=False,
    n_steps_input=4,
    n_steps_output=1,
)

Verified working on the same Windows machine (streams and batches fine, with and without use_normalization).

Happy to send a PR if the maintainers agree on the preferred approach.