Add Dask Gateway support to lc run (JupyterHub deployments)

docs/api/dask_cluster.md

@@ -1,24 +1,49 @@
 # lightcone.engine.dask_cluster
 
 Cluster lifecycle for `lc run`. One context manager (`cluster_for_run`),
-three branches, no service to manage.
+four branches, no service to manage.
 
 Source: `src/lightcone/engine/dask_cluster.py`.
 
-## `cluster_for_run(*, verbose=False) → Iterator[str]`
-
-Yields a Dask scheduler address valid for the duration of `lc run`.
-Three branches in priority order:
-
-1. **`DASK_SCHEDULER_ADDRESS` already set** → yield as-is. We don't
-   own the cluster, so we don't tear it down.
-2. **`SLURM_JOB_ID` set** → start an in-process scheduler bound to the
+## `cluster_for_run(*, verbose=False, local_directory=None, expected_worker_image=None) → Iterator[dict[str, str]]`
+
+Yields the **env overlay** the child snakemake process needs to reach
+the cluster — the parent and the executor plugin are separate
+processes, so connection info travels via environment variables.
+Four branches in priority order:
+
+1. **`DASK_SCHEDULER_ADDRESS` already set** → yield
+   `{"DASK_SCHEDULER_ADDRESS": addr}` as-is. We don't own the cluster,
+   so we don't tear it down.
+2. **Dask Gateway detected** (`LIGHTCONE_GATEWAY_CLUSTER` or
+   `DASK_GATEWAY__ADDRESS` set, e.g. a JupyterHub pod) → **attach** to
+   the user's running Gateway cluster; yield
+   `{"LIGHTCONE_GATEWAY_CLUSTER": name}`. Attach-only by design: the
+   user creates clusters from JupyterLab (that's where the
+   image/cores/memory options widget and the dashboard live); the
+   Gateway API is user-scoped, so exactly one running cluster attaches
+   unambiguously, and zero or several raises with the fix spelled out
+   (`LIGHTCONE_GATEWAY_CLUSTER` disambiguates). Gateway scheduler
+   addresses use a `gateway://` comm scheme a bare `Client` cannot
+   dial, so the child rejoins **by name** through the authenticated
+   Gateway API. The cluster, and its scaling, are left untouched on
+   exit — same convention as branch 1. Startup fails fast if live
+   workers don't advertise the resource contract below (zero workers
+   is fine — adaptive clusters scale on demand), and warns when the
+   cluster's actual worker image (read from the scheduler pod's
+   `LIGHTCONE_WORKER_IMAGE`) differs from *expected_worker_image*.
+   Requires the optional dependency:
+   `pip install lightcone-cli[gateway]`.
+3. **`SLURM_JOB_ID` set** → start an in-process scheduler bound to the
    driver's SLURM hostname (`SLURMD_NODENAME` or `gethostname()`),
    then `srun` one `dask worker` per node across the allocation.
-3. **Neither** → `LocalCluster()` sized to the local machine.
+4. **None of the above** → `LocalCluster()` sized to the local machine.
 
-The scheduler is always in-process, so its lifetime equals the run's
-lifetime: no orphaned schedulers if the driver crashes.
+Outside the Gateway branch the scheduler is always in-process, so its
+lifetime equals the run's lifetime: no orphaned schedulers if the
+driver crashes. On the Gateway branch there is nothing to orphan
+either — lc never creates the cluster, and idle clusters are reaped
+by the deployment's `idle_timeout`.
 
 ## Resource keys
 
@@ -73,6 +98,8 @@ and keeps everything in one process tree.
 
 ## Tests
 
-`tests/test_dask_cluster.py` covers the three branches and the
+`tests/test_dask_cluster.py` covers all four branches and the
 resource-advertising contract. The SLURM branch is tested with mocked
-`subprocess.Popen` plus a stubbed `Client.wait_for_workers`.
+`subprocess.Popen` plus a stubbed `Client.wait_for_workers`; the
+Gateway branch (discovery, attach-only lifecycle, contract and image
+verification) against a fake `dask_gateway` module.

docs/contributing/backends.md

@@ -6,9 +6,15 @@ of these:
 
 ## Adding a container runtime
 
-The supported runtimes are `docker`, `podman`, and `podman-hpc` (plus the
-`none` no-op). They are listed in
-`src/lightcone/engine/container.py::RUNTIMES`. To add a new one:
+The supported OCI runtimes are `docker`, `podman`, and `podman-hpc`,
+listed in `src/lightcone/engine/container.py::RUNTIMES`. Two non-OCI
+values sit outside that tuple: `none` (no isolation, recipes run on the
+host) and `kubernetes` (the Dask Gateway worker pod *is* the container;
+recipes run unwrapped inside it and images resolve against the
+deployment registry — see `REGISTRY_ENV` / `resolve_worker_image`).
+`kubernetes` is never auto-detected: a site declares it
+(`site_registry.py`, `container_runtime: "kubernetes"`) or the user
+pins it in config. To add a new OCI runtime:
 
 1. Append the binary name to `RUNTIMES` (detection priority is the tuple
    order).
@@ -23,8 +29,9 @@ The supported runtimes are `docker`, `podman`, and `podman-hpc` (plus the
 
 ## Adding a Dask cluster shape
 
-Today the cluster manager has three branches: existing scheduler, SLURM
-allocation, local. To add a fourth (for example, a custom GPU farm):
+Today the cluster manager has four branches: existing scheduler, Dask
+Gateway (JupyterHub), SLURM allocation, local. To add another (for
+example, a custom GPU farm):
 
 1. Add a branch to `cluster_for_run()` in
    `src/lightcone/engine/dask_cluster.py`.