diff --git a/docs/hub/storage-buckets-access.md b/docs/hub/storage-buckets-access.md index 1ce890aae..2d93f2163 100644 --- a/docs/hub/storage-buckets-access.md +++ b/docs/hub/storage-buckets-access.md @@ -12,6 +12,8 @@ Beyond the [CLI and Python SDK](./storage-buckets#managing-files), there are sev | **CLI sync** | Batch transfers, backups | [Sync docs](./storage-buckets#syncing-directories) | | **S3 API** | Existing S3 tooling (AWS CLI, boto3, s5cmd) | [S3-Compatible API](./storage-buckets-s3) | +For tools that use buckets as a backend (SkyPilot, Inspect, …), see [Integrations](./storage-buckets-integrations). + ## Mount as a Local Filesystem [hf-mount](https://github.com/huggingface/hf-mount) lets you mount buckets (and repos) as local filesystems via NFS (recommended) or FUSE. Files are fetched lazily — only the bytes your code reads hit the network. diff --git a/docs/hub/storage-buckets-integrations.md b/docs/hub/storage-buckets-integrations.md index 4a26c8a49..24fa653c7 100644 --- a/docs/hub/storage-buckets-integrations.md +++ b/docs/hub/storage-buckets-integrations.md @@ -79,6 +79,48 @@ inspect view See [Inspect's eval logs guide](https://inspect.aisi.org.uk/eval-logs.html#sec-hugging-face-storage-buckets) for details. +## SkyPilot + +[SkyPilot](https://docs.skypilot.co/) runs AI workloads across 20+ clouds, Kubernetes, and on-prem, and can use Hugging Face storage as a backend — so one bucket is readable from every cloud with no per-cloud copies. Set `store: hf` on a `file_mounts` entry to mount a bucket read-write or a repo read-only: + +```yaml +# qwen-sft.yaml — launch anywhere: sky launch qwen-sft.yaml --infra aws|gcp|... +resources: + accelerators: H100:1 + +file_mounts: + /base-model: + source: hf://Qwen/Qwen2.5-3B # model repo, read-only + store: hf + mode: MOUNT + /data: + source: hf://datasets/username/my-data@v1.0 # dataset repo, pinned to a tag, read-only + store: hf + mode: MOUNT + /checkpoints: + source: hf://buckets/username/qwen-sft # bucket, read-write — checkpoints sync back + store: hf + mode: MOUNT + +run: | + python train.py --model /base-model --output_dir /checkpoints +``` + +Authenticate once - `hf auth login` (or `export HF_TOKEN=`) is all SkyPilot needs. It forwards your local Hugging Face token to every cloud, so the bucket and repo mounts authenticate automatically: + +```bash +pip install "skypilot[huggingface]" +hf auth login # or: export HF_TOKEN= +sky launch qwen-sft.yaml +``` + +If your own `run` code pulls gated repos, add `--secret HF_TOKEN` to the launch command to also expose the token as an env var. + +> [!TIP] +> `MOUNT` and `MOUNT_CACHED` behave identically for `hf` and use the [hf-mount](https://github.com/huggingface/hf-mount) FUSE backend, which needs a base image with glibc ≥ 2.34 and `/dev/fuse`. Bare-VM clouds provide both. SkyPilot's default Kubernetes image ships older glibc, so set a newer `image_id` (e.g. `docker:mirror.gcr.io/ubuntu:22.04`). See the [SkyPilot storage docs](https://docs.skypilot.co/en/latest/reference/storage.html) for the current environment requirements. + +See the [SkyPilot + Hugging Face storage blog post](https://huggingface.co/blog/skypilot-hf-storage) for benchmarks and a full walkthrough. + ## Filesystem operations For direct file operations, `huggingface_hub` exposes a pre-instantiated [filesystem object](/docs/huggingface_hub/guides/hf_file_system), `hffs`: