Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/hub/storage-buckets-access.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ Beyond the [CLI and Python SDK](./storage-buckets#managing-files), there are sev
|--------|----------|---------|
| **hf-mount** | Mount as local filesystem — any tool works | [See below](#mount-as-a-local-filesystem) |
| **Volume mounts** | HF Jobs & Spaces (same idea, managed for you) | [See below](#volume-mounts-in-jobs-and-spaces) |
| **SkyPilot** | Cloud training/serving jobs across 20+ clouds | [Bucket Integrations](./storage-buckets-integrations#skypilot) |

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

more natural to move this at the end of the list imo

| **hf:// paths** (fsspec) | Python data tools (pandas, DuckDB) | [See below](#python-data-tools) |
| **CLI sync** | Batch transfers, backups | [Sync docs](./storage-buckets#syncing-directories) |
| **S3 API** | Existing S3 tooling (AWS CLI, boto3, s5cmd) | [S3-Compatible API](./storage-buckets-s3) |
Expand Down
42 changes: 42 additions & 0 deletions docs/hub/storage-buckets-integrations.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,48 @@ inspect view

See [Inspect's eval logs guide](https://inspect.aisi.org.uk/eval-logs.html#sec-hugging-face-storage-buckets) for details.

## SkyPilot

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Might make sense to move this to its own page, but cc @davanstrien wdyt


[SkyPilot](https://docs.skypilot.co/) runs AI workloads across 20+ clouds, Kubernetes, and on-prem, and can use Hugging Face storage as a backend — so one bucket is readable from every cloud with no per-cloud copies. Set `store: hf` on a `file_mounts` entry to mount a bucket read-write or a repo read-only:

```yaml
# qwen-sft.yaml — launch anywhere: sky launch qwen-sft.yaml --infra aws|gcp|...
resources:
accelerators: H100:1

file_mounts:
/base-model:
source: hf://Qwen/Qwen2.5-3B # model repo, read-only
store: hf
mode: MOUNT
/data:
source: hf://datasets/username/my-data@v1.0 # dataset repo, pinned to a tag, read-only
store: hf
mode: MOUNT
/checkpoints:
source: hf://buckets/username/qwen-sft # bucket, read-write — checkpoints sync back
store: hf
mode: MOUNT

run: |
python train.py --model /base-model --output_dir /checkpoints
```

Authenticate once - `hf auth login` (or `export HF_TOKEN=<your-token>`) is all SkyPilot needs. It forwards your local Hugging Face token to every cloud, so the bucket and repo mounts authenticate automatically:

```bash
pip install "skypilot[huggingface]"
hf auth login # or: export HF_TOKEN=<your-token>
sky launch qwen-sft.yaml
```

If your own `run` code pulls gated repos, add `--secret HF_TOKEN` to the launch command to also expose the token as an env var.

> [!TIP]
> `MOUNT` and `MOUNT_CACHED` behave identically for `hf` and use the [hf-mount](https://github.com/huggingface/hf-mount) FUSE backend, which needs a base image with glibc ≥ 2.34 and `/dev/fuse`. Bare-VM clouds provide both. SkyPilot's default Kubernetes image ships older glibc, so set a newer `image_id` (e.g. `docker:mirror.gcr.io/ubuntu:22.04`). See the [SkyPilot storage docs](https://docs.skypilot.co/en/latest/reference/storage.html) for the current environment requirements.

See the [SkyPilot + Hugging Face storage blog post](https://huggingface.co/blog/skypilot-hf-storage) for benchmarks and a full walkthrough.

## Filesystem operations

For direct file operations, `huggingface_hub` exposes a pre-instantiated [filesystem object](/docs/huggingface_hub/guides/hf_file_system), `hffs`:
Expand Down