From 6568ee8a2058b5d9bf71e224780d524ba635a193 Mon Sep 17 00:00:00 2001 From: spara-earthscope Date: Wed, 8 Jul 2026 15:40:16 -0600 Subject: [PATCH] added guides --- advanced_topics/env_mgmt.md | 13 +- .../environments/binder_for_images.md | 182 ++++++++++++++ .../environments/building_custom_images.md | 233 ++++++++++++++++++ .../environments/managing_environments.md | 159 ++++++++++++ myst.yml | 3 + 5 files changed, 584 insertions(+), 6 deletions(-) create mode 100644 advanced_topics/environments/binder_for_images.md create mode 100644 advanced_topics/environments/building_custom_images.md create mode 100644 advanced_topics/environments/managing_environments.md diff --git a/advanced_topics/env_mgmt.md b/advanced_topics/env_mgmt.md index 88d3b27..94391a3 100644 --- a/advanced_topics/env_mgmt.md +++ b/advanced_topics/env_mgmt.md @@ -7,6 +7,8 @@ When you open GeoLab and select an environment when starting your server, you la * [Bring Your Own Image:](#bring-your-own-image) Many other organizations (e.g, NASA, NOAA) maintain their own JupyterHub compute environment images. Many of these will run in GeoLab. ## Ephemeral Installation +For detailed instructions for installing packages in GeoLab see the [guide](./environments/managing_environments.md) + ### Installing python packages Use `pip` or `conda` to install the package yourself at the beginning of your session by typing the below in a GeoLab Terminal. ``` @@ -57,15 +59,14 @@ Some software can be installed via the terminal using `apt-get`. Please be advis Most GeoLab images have `gcc`, `g++`, and `gfortran` compilers installed. You will need to move your makefiles into your home directory and compile them within GeoLab. ## Create a Custom Image -:::{attention} Section Under Development! -::: +You can build a custom own image using our existing image as a template. This strategy is an advanced approach for software with complex installations, and only recommended for users with some prior knowledge of building containers. -EarthScope is working to build more comprehensive documentation on custom image creation. In the mean time, please see: +Follow the instructions for building a custom image in this [guide](./environments/building_custom_images.md). -- [2i2c guide to binderhub](https://docs.2i2c.org/user/environment/dynamic-imagebuilding/#user-environment-building) - This strategy is an option if you want to use an environment file that extends the base GeoLab image with additional python packages, but don't want to re-install it every time (i.e., with an environment file). You can specify your environment file once to generate a stable container, and load that at launch time. Note, there's a 15-20 minute overhead to build the image in binderhub the first time and any time you change the environment file, so you'll need to consider whether this will save you time in the long run. +## Binder for Images +[2i2c guide to binderhub](https://docs.2i2c.org/user/environment/dynamic-imagebuilding/#user-environment-building) - This strategy is an option if you want to use an environment file that extends the base GeoLab image with additional python packages, but don't want to build and host an image. You can specify your environment file once to generate a stable container, and load that at launch time. Note, there's a 15-20 minute overhead to build the image in GeoLab the first time and any time you change the environment file, you will need to consider whether this will save you time in the long run. -- If you need to make extensive software changes, you can build your own image using our existing image as a template. Instructions are located in the `README.md` file in the [GeoLab GitHub repository.]({{ geolab_github }}) You will need to clone the repository, modify the build files to suit your needs, build your container and upload it to a public container registry like DockerHub or AWS ECR, and then select it as 'Other' from the dropdown when you [launch your GeoLab server](../getting_started/server_launch.md). This strategy is an advanced approach for software with complex installations, and only recommended for users with some prior knowledge of building docker containers. +Follow the instructions to deploy a custom image from a GitHub repository using binder in this [guide](./environments/binder_for_images.md). ## Bring Your Own Image Many other organizations (e.g., NASA, NOAA, Pangeo, The Rocker Project, etc) maintain JupyterHub images with their software installed. Many of these will run in GeoLab (but we make no guarantees that images built for other platforms will work here!) We recommend using these images interdisciplinary research, where you may need software installed in another image, but want to run it on data stored in the NSF NGF archive. At [Server Launch](../getting_started/server_launch.md#select-an-environment), select the 'Other' option from the Environment dropdown and enter the URL to the public image. - \ No newline at end of file diff --git a/advanced_topics/environments/binder_for_images.md b/advanced_topics/environments/binder_for_images.md new file mode 100644 index 0000000..8e7ee39 --- /dev/null +++ b/advanced_topics/environments/binder_for_images.md @@ -0,0 +1,182 @@ +# GeoLab Binder + +There are two ways to launch a custom environment in GeoLab. The first is to build a Docker image, push it to a registry, and paste the image URL. The second, called **GeoLab Binder**, is to point GeoLab at a GitHub repository and let it build the environment for you. This guide covers how the second approach works and when to use it. + +--- + +## What Is GeoLab Binder? + +GeoLab Binder lets you launch a session from a GitHub repository instead of a prebuilt image. You put configuration files in your repo that describe what packages you need, and GeoLab builds the environment automatically when you start a session. + +Behind the scenes, GeoLab reads the Dockerfile and configuration files in your repository and builds the container image for you. You write the Dockerfile (using the same template as a custom image), but you never run Docker locally or push to a registry, as GeoLab handles the build. + +Both approaches run inside GeoLab, so you have the same access to EarthScope data services either way. + +```text +GitHub repo with config files → GeoLab builds image → Session launches in GeoLab +``` + +--- + +## GeoLab Binder vs. Custom Image + +Both approaches use GeoLab's environment selector, but they use different options. The difference is what you select and what you paste. + +| | **GeoLab Binder** | **Custom Image** | +| --- | --- | --- | +| Menu option | **Build your own image** | **Other** | +| What you paste | A GitHub repo URL | A container registry URL (`ghcr.io/...`) | +| Who builds the image | GeoLab, automatically from your repo | You, locally with Docker | +| What you need | Dockerfile + config files in a GitHub repo | Docker Desktop, local build, registry push | +| EarthScope data access | Yes | Yes | +| How updates work | Push a commit, and the next launch picks up the changes | Rebuild and push a new image tag | +| Best for | Getting started quickly, iterating, sharing a repo | Stable, versioned environments; full Docker control | + +**Use GeoLab Binder when** you want to share a GitHub repository as a runnable environment without running Docker locally or managing a container registry. + +**Use a custom image when** you need a fixed, versioned environment that won't change between sessions, or when you need more control than config files allow. See [Building a Custom GeoLab Image](./building-custom-images.md). + +--- + +## Before You Start + +You need: + +- A **GitHub account** and a **GitHub repository** for your project. +- A GeoLab account at [GeoLab](https://geolab.earthscope.cloud). + +No local Docker builds or container registry required. + +--- + +## Step 1: Set Up Your GitHub Repository + +Create a new repository on GitHub (or use an existing one). Add your notebooks and at least one configuration file: + +```text +my-project/ +├── Dockerfile ← required (copy from the geolab-base template) +├── environment.yml ← your packages (conda) +├── requirements.txt ← PyPI-only packages (optional) +├── apt.txt ← system software (optional, rarely needed) +├── postBuild ← setup commands to run after install (optional) +└── my_notebook.ipynb ← your notebooks +``` + +Copy the `Dockerfile` from the `geolab-base` template at [EarthScope/GeoLab](https://github.com/EarthScope/GeoLab/tree/main/geolab-base) and leave it unchanged. All your customization goes in the other files. GeoLab Binder reads everything in the repo and builds the image when you launch. + +--- + +## Step 2: Write Your Configuration Files + +### `environment.yml`: Your Main Package List + +```yaml +name: my-binder-env +channels: + - conda-forge +dependencies: + - python=3.11 + - numpy + - matplotlib + - obspy +``` + +### `requirements.txt`: PyPI-only Packages (If Needed) + +```text +some-pypi-package==1.2.3 +``` + +### `apt.txt`: System Software (If Needed) + +```text +build-essential +``` + +### `postBuild`: One-time Setup Commands (Optional) + +If you need to run something after packages install, such as downloading a data file or enabling a Jupyter extension, create a `postBuild` file: + +```bash +#!/bin/bash +set -e +jupyter labextension install my-extension +``` + +Before committing, mark it executable: + +```bash +chmod +x postBuild +git add postBuild +git commit -m "add postBuild script" +``` + +These files use the exact same format as the custom image template. If you've worked through [Building a Custom GeoLab Image](./building-custom-images.md), you can reuse them directly. + +Once your files are ready, commit, and push them to GitHub: + +```bash +git add environment.yml requirements.txt apt.txt +git commit -m "add environment configuration" +git push +``` + +If this is a new repository, you may need to set the upstream branch on your first push: + +```bash +git push -u origin main +``` + +--- + +## Step 3: Launch from Your Repo in GeoLab + +1. Go to [earthscope.org/data/geolab](https://www.earthscope.org/data/geolab/) and click **Launch GeoLab**. +2. Enter your username and password, then click **Continue**. +3. In the environment selector, choose **Build your own image**. +4. In the **Repository** field, paste the git clone URL for your repository: + + ```text + https://github.com/your-github-username/my-project.git + ``` + +5. Click **Build Image**. GeoLab will build the environment from your repository. This can take several minutes the first time. +6. When you receive a notification that the build is complete, click **Start**. + +> **If the build fails:** Check the build log for error messages. Common causes are a misspelled package name, an unavailable package version, or a syntax error in `environment.yml`. Fix the file, push the commit, and try again. + +--- + +## Step 4: Share Your Environment + +To share your environment with someone else, give them the git clone URL for your repository. They follow the same steps: go to GeoLab, choose **Build your own image**, paste the URL, and launch. + +Because the environment is defined by files in the repo, anyone who uses the same URL gets the same packages. If you update `environment.yml` and push, the next person to launch picks up the new version automatically. + +--- + +## Updating Your Environment + +Edit your configuration files, commit, and push. The next GeoLab launch from that repo URL will rebuild with the updated packages. + +```bash +# edit environment.yml, then: +git add environment.yml +git commit -m "add pandas to environment" +git push +``` + +> **Note:** A new commit triggers a fresh build on the next launch. If you're changing packages frequently, expect slower first launches while GeoLab rebuilds the image. + +--- + +## Quick Reference + +| What you want to do | How | +| --- | --- | +| Define your packages | `environment.yml` (conda), `requirements.txt` (PyPI) | +| Launch from a repo | GeoLab → Start Server → Build your own image → paste git clone URL | +| Share your environment | Share the git clone URL (`https://github.com/username/repo.git`) | +| Update your environment | Edit config files, commit, and push | +| Use a fixed versioned environment instead | See [Building a Custom GeoLab Image](./building-custom-images.md) | diff --git a/advanced_topics/environments/building_custom_images.md b/advanced_topics/environments/building_custom_images.md new file mode 100644 index 0000000..8d4db79 --- /dev/null +++ b/advanced_topics/environments/building_custom_images.md @@ -0,0 +1,233 @@ +# Building a Custom GeoLab Image + +A GeoLab image is a complete, prepackaged computing environment, including Python, JupyterLab, and all the scientific packages your project needs, bundled together so anyone can run them. When you start a GeoLab session, it launches from an image. This guide walks you through building your own. + +--- + +## How It Works + +Think of an **image** as a recipe and a **container** as a meal made from that recipe. The recipe doesn't change; you can make the same meal over and over. GeoLab does the same thing: it takes your image and launches a fresh session from it every time. + +You define the image by editing a few plaintext files that list what you want installed. A tool called Docker reads those files and builds the image. You then publish it so GeoLab can access it. + +```text +Config files → Docker builds → Image → GeoLab runs it +``` + +--- + +## Before You Start + +You need two things installed on **your computer**: + +1. **Docker Desktop**, download it at [docker.com](https://www.docker.com/products/docker-desktop/). Install it on your computer, open it and leave it running in the background. +2. **Git**, to download the template. + +You also need a **GitHub** account at [github.com](https://github.com), where you'll publish your image using the GitHub Container Registry (`ghcr.io`). If you already use GitHub for code, you can use the same account. + +Before you can push images, you need a **Personal Access Token (PAT)** with package permissions: + +1. Go to **GitHub → Settings → Developer settings → Personal access tokens → Tokens (classic)**. +2. Click **Generate new token (classic)**. +3. Give it a name (e.g. `geolab-image`), set an expiration, and check the **`write:packages`** scope. +4. Click **Generate token** and copy it, as you won't be able to see it again. + +Save your token somewhere safe (a password manager works well). You'll use it to log in to the registry in Step 4. + +Verify Docker is working by opening a terminal and running: + +```bash +docker --version +``` + +If it prints a version number, you're good to go. + +--- + +## Step 1: Get the Template + +EarthScope provides a starter template. Download it and set up your working folder: + +```bash +git clone --depth 1 https://github.com/EarthScope/GeoLab.git +cp -R GeoLab/geolab-base my-geolab-image +cd my-geolab-image +``` + +Your `my-geolab-image` folder contains these files: + +```text +my-geolab-image/ +├── Dockerfile ← do not edit this +├── environment.yml ← add your conda packages here +├── requirements.txt ← add PyPI-only packages here +├── apt.txt ← add system software here (rarely needed) +├── start ← do not edit this +├── test_packages.py ← smoke test for installed packages +└── test_notebook.ipynb ← interactive version of the smoke test +``` + +> **The only files you need to edit are `environment.yml`, `requirements.txt`, and `apt.txt`.** Everything else is set up for you. + +--- + +## Step 2: Add Your Packages + +### `environment.yml`: Your Main Package List + +This is where you add Python packages. Open the file and add packages under the `dependencies` section: + +```yaml +channels: + - conda-forge + - nodefaults +dependencies: + - python=3.12 + # --- Geophysics --- + - obspy + - pygmt + # --- Geospatial --- + - cartopy + - geopandas + # add your packages below: + - my-package-name +``` + +Use conda packages whenever you can, as conda checks that everything works together before installing. + +### `requirements.txt`: Packages Only on PyPI + +Some packages aren't available through conda and must be installed from PyPI. Add them here, one per line: + +```text +earthscope-sdk==1.4.1 +my-pypi-package +``` + +### `apt.txt`: System Software (Rarely Needed) + +Most scientific packages go in `environment.yml`. Only use `apt.txt` for low-level system tools that can't be installed any other way: + +```text +build-essential +git +``` + +--- + +## Step 3: Build and Test Locally + +Before publishing, build the image on your own computer and make sure everything works. + +**Build the image:** + +```bash +docker build -f Dockerfile --tag my-geolab-image:0.1.0 . +``` + +This reads your config files and assembles the image. It can take several minutes the first time. + +**Run it:** + +```bash +docker run --rm -p 8888:8888 my-geolab-image:0.1.0 +``` + +Look in the output for a line like: + +```text +http://127.0.0.1:8888/lab?token=... +``` + +Copy that URL into a browser. You'll see a JupyterLab session running from your image. + +**Test that your packages installed correctly:** + +Open a terminal inside JupyterLab (File → New → Terminal) and run: + +```bash +pytest test_packages.py -v +``` + +Each package gets a pass or fail. If something fails, it usually means a package name is misspelled or a version is unavailable, so go back to `environment.yml` and fix it, then rebuild. + +--- + +## Step 4: Publish Your Image + +Once the local test passes, rebuild the image for GeoLab's platform and push it to the GitHub Container Registry. + +**Rebuild for GeoLab's platform:** + +```bash +docker build --no-cache -f Dockerfile \ + --platform linux/amd64 \ + --tag ghcr.io/your-github-username/my-geolab-image:0.1.0 . +``` + +Replace `your-github-username` with your GitHub username. + +> **Why `--platform linux/amd64`?** GeoLab runs on Linux. If you're on a Mac with Apple Silicon, your local machine uses a different architecture. This flag ensures the image works on GeoLab regardless of what you built it on. + +**Log in to the GitHub Container Registry and push:** + +```bash +echo YOUR_GITHUB_TOKEN | docker login ghcr.io -u your-github-username --password-stdin +docker push ghcr.io/your-github-username/my-geolab-image:0.1.0 +``` + +Replace `YOUR_GITHUB_TOKEN` with the token you created in the prerequisites and `your-github-username` with your GitHub username. + +**Make the image public:** + +By default, images published to the GitHub Container Registry are private. GeoLab needs to be able to pull it, so you must make it public: + +1. Go to your GitHub profile and click **Packages**. +2. Click on `my-geolab-image`. +3. Click **Package settings** (bottom of the right sidebar). +4. Scroll to the **Danger Zone** and click **Change visibility → Public**. + +--- + +## Step 5: Launch It in GeoLab + +1. Go to [earthscope.org/data/geolab](https://www.earthscope.org/data/geolab/) and click **Launch GeoLab**. +2. Enter your username and password, then click **Continue**. +3. If a **Stop My Server** button appears, click it first. +4. Click **Start My Server**. +5. Under **Environment**, choose **Other**. +6. In the **Custom image** field, enter your image name, e.g. `ghcr.io/your-github-username/my-geolab-image:0.1.0`. +7. Click **Start**. + +GeoLab will pull your image and launch a session from it. The first launch takes a minute while it downloads; after that it's cached and starts quickly. + +--- + +## Making Changes Later + +Edit your config files, then rebuild and push with a new version number: + +```bash +docker build --no-cache -f Dockerfile \ + --platform linux/amd64 \ + --tag ghcr.io/your-github-username/my-geolab-image:0.1.1 . + +docker push ghcr.io/your-github-username/my-geolab-image:0.1.1 +``` + +> **Always use a new version number** (`0.1.1`, `0.1.2`, etc.) when you rebuild. If you reuse the same tag, GeoLab may load the old cached version instead of your new one. + +--- + +## Quick Reference + +| What you want to do | Where to do it | +| --- | --- | +| Add a Python package | `environment.yml` under `dependencies` | +| Add a PyPI-only package | `requirements.txt` | +| Add a system tool | `apt.txt` | +| Build locally for testing | `docker build --tag my-geolab-image:0.1.0 .` | +| Run locally | `docker run --rm -p 8888:8888 my-geolab-image:0.1.0` | +| Test packages | `pytest test_packages.py -v` in the container terminal | +| Build for GeoLab | `docker build --no-cache --platform linux/amd64 --tag ghcr.io/username/image:version .` | +| Publish | `docker push ghcr.io/username/my-geolab-image:0.1.0` | diff --git a/advanced_topics/environments/managing_environments.md b/advanced_topics/environments/managing_environments.md new file mode 100644 index 0000000..d70f33f --- /dev/null +++ b/advanced_topics/environments/managing_environments.md @@ -0,0 +1,159 @@ +# Creating Your Own Python Environment in GeoLab + +Python projects rely on add-on packages, which are collections of code that do useful things like process data, make maps, or analyze signals. Different projects often need different packages, or even different versions of the same package. An **environment** is a way to keep all of that organized in one tidy, self-contained workspace. + +This guide shows you how to create your own environment in GeoLab. + +--- + +## What Is a Package Manager? + +GeoLab uses a tool called **conda** to install and manage packages. Think of conda like an app store for Python packages: you tell it what you want, it figures out what else is needed to make it work, and installs everything together. + +> **Heads up:** GeoLab resets when you sign out, so any packages you installed during a session won't be there next time. The solution is to define your environment in a file (explained below) so you can recreate it anytime. + +--- + +## See What's Already Installed + +Open the **terminal** in GeoLab and try these commands to get your bearings. + +**See all available environments:** + +```bash +conda env list +``` + +The one with a `*` is the currently active environment: + +```text +# conda environments: +# +base * /srv/conda +notebook /srv/conda/envs/notebook +custom_environment /home/jovyan/.conda/envs/custom_environment +``` + +**See all packages in the current environment:** + +```bash +conda list +``` + +This prints a long list. Each row shows a package name, its version, and where it came from: + +```text +# Name Version Build Channel +cartopy 0.24.1 py312h78ddc71_0 conda-forge +numpy 2.2.4 py312h7e3fe57_0 conda-forge +obspy 1.4.1 py312h7b8d3f4_0 conda-forge +xarray 2025.3.0 pyhd8ed1ab_0 conda-forge +… +``` + +--- + +## Installing a Package Without Rebuilding + +If you just need to add one package quickly, you can install it directly from inside a notebook cell using a special `%` command. This installs into the notebook's active environment: + +```python +%conda install pandas +``` + +Or, for packages only available on PyPI (a different package source): + +```python +%pip install some-package +``` + +> **Warning:** Installing with `%conda` inside a notebook can use a lot of memory. If your notebook becomes unresponsive or the kernel crashes, use the `environment.yml` approach instead, which is more reliable for larger installs. + +--- + +## Create Your Own Environment + +The best way to create a custom environment is to write a short file that lists everything you need. Conda reads the file and builds the environment from it. This also means you can recreate the exact same environment later, or share it with a teammate. + +### Step 1: Write an `environment.yml` File + +Create a new file called `environment.yml` and paste in something like this: + +```yml +name: my_environment +channels: + - conda-forge +dependencies: + - python=3.11 + - ipykernel + - numpy + - matplotlib +``` + +Here's what each part means: + +- **name**: What you want to call your environment. +- **channels**: Where to download packages from (`conda-forge` is a large, reliable source). +- **dependencies**: The packages you want installed. +- `ipykernel` is required so your environment can be used as a notebook kernel, so always include it. + +Replace `numpy` and `matplotlib` with whatever packages your project actually needs. + +### Step 2: Build the Environment + +In the **terminal**, run: + +```bash +conda env create -f environment.yml +``` + +Conda will figure out which versions of everything are compatible and download them. This can take a few minutes, which is normal. + +> **If it fails:** Read the error message. Conda usually names the package that's causing the problem. Try removing it from the file or changing its version. + +### Step 3: Activate the Environment + +```bash +conda activate my_environment +``` + +Activating an environment switches your terminal into that workspace, so any Python commands you run use that environment's packages. Your terminal prompt will update to show the environment name, confirming it worked. + +### Step 4: Register It as a Notebook Kernel + +A new environment isn't automatically available in JupyterLab. Run this command **in the terminal** to register it: + +```bash +python -m ipykernel install --user --name my_environment --display-name "Python (my_environment)" +``` + +### Step 5: Switch to Your Environment in a Notebook + +1. Open a notebook and go to **Kernel > Change Kernel…** + + ![Kernel menu showing the Change Kernel option](./images/select_kernel.png) + +2. Select **Python (my_environment)**. + + ![Kernel selector dialog with custom environment listed](./images/select_custom.png) + +3. Check the upper-right corner of the notebook to confirm the kernel changed. + + ![Notebook header showing the active custom environment kernel](./images/custom_env.png) + +> **Note:** You have to do this for each notebook separately. There's no way to set it as the default for everything. + +--- + +## Quick Reference + +| What you want to do | Command | +| --- | --- | +| See all environments | `conda env list` | +| See installed packages | `conda list` | +| Activate an environment | `conda activate ` | +| Leave an environment | `conda deactivate` | +| Build from a file | `conda env create -f environment.yml` | +| Register as a kernel | `python -m ipykernel install --user --name --display-name "