diff --git a/docs/_config.yml b/docs/_config.yml index da4e259e..1cb8dc48 100644 --- a/docs/_config.yml +++ b/docs/_config.yml @@ -36,6 +36,9 @@ sphinx: local_extensions: azure_functions_simple: . config: + # Generate slug anchors for h1-h3 so in-page "[text](#heading)" links (the + # Contents lists) resolve as cross-references instead of warning. + myst_heading_anchors: 3 autodoc_default_options: members: true undoc-members: true diff --git a/docs/_static/usage/publishing/edit-metadata.png b/docs/_static/usage/publishing/edit-metadata.png new file mode 100644 index 00000000..3190ae69 Binary files /dev/null and b/docs/_static/usage/publishing/edit-metadata.png differ diff --git a/docs/_static/usage/publishing/explorer-damage-layer.png b/docs/_static/usage/publishing/explorer-damage-layer.png new file mode 100644 index 00000000..87d1280c Binary files /dev/null and b/docs/_static/usage/publishing/explorer-damage-layer.png differ diff --git a/docs/_static/usage/publishing/publish-dialog.png b/docs/_static/usage/publishing/publish-dialog.png new file mode 100644 index 00000000..7c342b29 Binary files /dev/null and b/docs/_static/usage/publishing/publish-dialog.png differ diff --git a/docs/_static/usage/publishing/publish-menu-item.png b/docs/_static/usage/publishing/publish-menu-item.png new file mode 100644 index 00000000..48007075 Binary files /dev/null and b/docs/_static/usage/publishing/publish-menu-item.png differ diff --git a/docs/_static/usage/publishing/publish-target-dropdown.png b/docs/_static/usage/publishing/publish-target-dropdown.png new file mode 100644 index 00000000..55220c9c Binary files /dev/null and b/docs/_static/usage/publishing/publish-target-dropdown.png differ diff --git a/docs/_static/usage/publishing/published-dataset-detail.png b/docs/_static/usage/publishing/published-dataset-detail.png new file mode 100644 index 00000000..1d134172 Binary files /dev/null and b/docs/_static/usage/publishing/published-dataset-detail.png differ diff --git a/docs/_static/usage/publishing/published-datasets-list.png b/docs/_static/usage/publishing/published-datasets-list.png new file mode 100644 index 00000000..00b0c2d0 Binary files /dev/null and b/docs/_static/usage/publishing/published-datasets-list.png differ diff --git a/docs/_toc.yml b/docs/_toc.yml index b4d283d0..d97e89ae 100644 --- a/docs/_toc.yml +++ b/docs/_toc.yml @@ -7,6 +7,7 @@ parts: - file: usage/keyboard-shortcuts - file: usage/rapid-building-assessment - file: usage/damage-mapping + - file: usage/data-publishing - file: usage/building-blocks sections: - file: usage/projects diff --git a/docs/configuration.md b/docs/configuration.md index 97067a27..4a2f9a21 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -19,6 +19,7 @@ This guide documents each configuration mode. For the end-to-end workflow, see - [Email sender domain](#email-sender-domain) - [Front Door](#front-door) - [Development mode](#development-mode) +- [Data publishing](#data-publishing) - [First-admin bootstrap](#first-admin-bootstrap) - [Cleaning up an environment](#cleaning-up-an-environment) @@ -296,6 +297,64 @@ The Docker Compose UI image pre-fills `administrators` in the SWA emulator's mock-login form. The source `staticwebapp.config.json` used for production is unchanged. +## Data publishing + +Enables the **Published Datasets** feature and the **Publish** action on model results. The +**Local** target (an immutable copy in the app's storage) is on by default; the **Planetary +Computer** target is off until you configure it. For how the feature is used, see +{doc}`Publishing datasets `. + +### Feature flags + +| Variable | Default | Purpose | +|---|---|---| +| `HASTE_PUBLISHING_ENABLED` | `true` | Master switch for the Published Datasets section and the Publish action. | +| `HASTE_PC_PROVIDER_ENABLED` | `false` | Register/expose the Planetary Computer publishing target. | +| `HASTE_PUBLISH_EXPLORER_RENDER_ENABLED` | `true` | Render a damage-classification COG and register the Explorer render/mosaic/tile config on Planetary Computer publish. | + +### Planetary Computer target + +Required to publish to a Microsoft Planetary Computer Pro GeoCatalog. The GeoCatalog is +**external** to this template — you provision and own it (see the +[out-of-app setup](usage/data-publishing.md#enabling-and-configuring-publishing)). + +| Variable | Default | Purpose | +|---|---|---| +| `HASTE_PC_GEOCATALOG_URL` | — | GeoCatalog base URL (no trailing slash). Required for the PC target. | +| `HASTE_PC_EXPLORER_URL` | — | Explorer base URL used to build published-dataset links. | +| `HASTE_PC_INGESTION_SOURCE` | — | GeoCatalog ingestion-source name. Only for **private** publish containers; public containers need none. | +| `HASTE_PC_COLLECTION_PREFIX` | `haste-` | Prefix for STAC collection ids (one collection per project/event). | +| `HASTE_PC_PUBLISHING_LICENSE` | `CC-BY-4.0` | STAC license id applied to published collections/items. | +| `HASTE_PC_GEOCATALOG_INGEST_PRINCIPAL_ID` | — | Object id of the GeoCatalog managed identity to grant **Storage Blob Data Reader** on HASTE storage (asset ingestion). Empty = skip the role grant. | + +### Publish storage and attribution + +HASTE copies published assets into a network-reachable container the GeoCatalog ingests from, +and records the operating organization as the STAC `processor` provider. + +| Variable | Default | Purpose | +|---|---|---| +| `HASTE_PUBLISH_STORAGE_ACCOUNT_URL` | — | Storage account URL the GeoCatalog ingests published assets from. | +| `HASTE_PUBLISH_BLOB_CONTAINER` | — | Blob container (on the publish storage account) HASTE copies published PC assets into. | + +```{admonition} The publish-store settings are an all-or-nothing pair +:class: warning +`HASTE_PUBLISH_STORAGE_ACCOUNT_URL` and `HASTE_PUBLISH_BLOB_CONTAINER` are used **only when +both are set**. If either is empty, HASTE silently falls back to referencing assets **in +place** from the primary artifact store — which a firewalled GeoCatalog cannot ingest, so +publishing appears configured but fails at ingestion. Set both, or neither. +``` +| `HASTE_PUBLISHING_ORGANIZATION_NAME` | — | Organization operating this deployment, recorded as the STAC `processor` provider. Empty = omit. | +| `HASTE_PUBLISHING_ORGANIZATION_URL` | — | URL companion to `HASTE_PUBLISHING_ORGANIZATION_NAME`. | + +```{admonition} Organization attribution and PC URLs are treated as secrets in CI +:class: note +On the GitHub Actions deploy path the organization values and the GeoCatalog/publish-storage +URLs are read from environment **secrets** (masked in logs), while the non-sensitive feature +flags and collection prefix are environment **variables**. The Bicep/azd path reads them all +as `HASTE_*` settings. +``` + ## First-admin bootstrap Production uses `DEVELOPMENT_MODE=false`, so users are managed explicitly and are diff --git a/docs/deployment.md b/docs/deployment.md index b4488723..b3191732 100644 --- a/docs/deployment.md +++ b/docs/deployment.md @@ -75,6 +75,23 @@ with per-job user-delegation SAS for tenant data isolation and capacity-aware routing. See [Shared multi-tenant GPU pools](configuration.md#shared-multi-tenant-gpu-pools) in the configuration guide. +### Data publishing (optional) + +The **Local** publishing target works out of the box. Publishing to **Planetary Computer** +depends on an **external** Microsoft Planetary Computer Pro GeoCatalog that you provision and +own — it is not created by `azd up`. Before enabling the PC target, provision the GeoCatalog, +give HASTE a publish storage container the GeoCatalog can ingest from, and set up the three +role grants: + +- the HASTE Function App identity needs a **GeoCatalog data-plane RBAC role** on the + GeoCatalog resource (to call the STAC/ingestion APIs — this authorizes publishing itself); +- the HASTE Function App identity needs **Storage Blob Data Contributor** on the publish store; +- the GeoCatalog identity needs **Storage Blob Data Reader** on that store. + +The settings are documented under [Data publishing](configuration.md#data-publishing) in the +configuration guide, and the end-to-end setup under +[Publishing datasets](usage/data-publishing.md#enabling-and-configuring-publishing). + ### Security For production deployments, follow the [Secure Configuration Guidance](security-configuration.md) — it covers identity and authentication setup, secrets management with managed identity and Key Vault, CORS and HTTP security headers, container hardening, logging and monitoring, and known limitations with operational mitigations. The guide also includes a pre-production checklist. diff --git a/docs/usage/damage-mapping.md b/docs/usage/damage-mapping.md index 7cc4c13d..788d010d 100644 --- a/docs/usage/damage-mapping.md +++ b/docs/usage/damage-mapping.md @@ -152,6 +152,12 @@ reference. Each download shows its file size. +### Publish the dataset + +If your administrator has enabled publishing, the Results menu also offers **Publish +dataset…** — archive the result as a downloadable copy or share it to a Planetary Computer +STAC catalog. See {doc}`Publishing datasets `. + ### Reports If you've validated buildings for this layer, the Results menu also offers a **Validation diff --git a/docs/usage/data-publishing.md b/docs/usage/data-publishing.md new file mode 100644 index 00000000..0284d6a5 --- /dev/null +++ b/docs/usage/data-publishing.md @@ -0,0 +1,262 @@ + + +# Publishing Datasets + +**Goal: share and archive a finished damage assessment.** Once a model result is +processed, you can *publish* it — either as an immutable copy in the app's own storage +(for download), or to a **Planetary Computer** STAC catalog where others can discover, +reference, and visualize it. + +Publishing is available from either workflow — {doc}`Rapid Building Assessment +` or {doc}`Damage Mapping ` — wherever you have a +processed result. + +```{admonition} The workflow at a glance +:class: tip + +Open a processed model's **Results** menu → **Publish dataset…** → pick a **target** and +fill in the details → track and manage it under **Published Datasets**. +``` + +```{admonition} Is publishing available? +:class: note + +Publishing is an operator-enabled feature. If you don't see **Publish dataset…** in the +Results menu or a **Published Datasets** section, an administrator hasn't enabled it for your +deployment — see {ref}`Enabling and configuring publishing `. +``` + +## Contents + +- [Before you start](#before-you-start) +- [Choose a target](#choose-a-target) +- [Publish a dataset](#publish-a-dataset) +- [Manage published datasets](#manage-published-datasets) +- [Source-imagery attribution](#source-imagery-attribution) +- [Viewing in the Planetary Computer Explorer](#viewing-in-the-planetary-computer-explorer) +- [Enabling and configuring publishing](#enabling-and-configuring-publishing) + +## Before you start + +You need a **processed model result** to publish: + +- a {doc}`Rapid Building Assessment ` (per-building damaged/intact), + or +- a {doc}`Damage Mapping ` model whose training and inference have finished. + +Publishing captures the outputs that already exist for that result — the predicted-damage +geopackage, building footprints, the valid-area mask, and (for local publishing) the +processed image. You don't prepare anything special beforehand. + +## Choose a target + +When you publish, you pick where the dataset goes. Which targets appear depends on what your +administrator has configured. + +| | **Local (in-app storage)** | **Planetary Computer** | +|---|---|---| +| **What it is** | An immutable copy kept in the app's storage. | A STAC **collection + item** in an external Microsoft Planetary Computer Pro GeoCatalog. | +| **Who can see it** | Anyone with access to your HASTE deployment. | Anyone with access to the GeoCatalog — discoverable via its STAC API and map Explorer. | +| **Best for** | Keeping a durable, downloadable snapshot of a result. | Sharing results in an interoperable catalog and visualizing them on a map. | +| **Outputs** | Downloadable assets (geopackage, footprints, processed image, valid-area mask). | STAC item with the same vector assets, plus a rendered damage layer in the Explorer. | + +You pick the target in the **Publish dataset** dialog; a target your administrator hasn't +configured is shown with the reason it's unavailable. + +![The target publishing location dropdown showing Local and Planetary Computer](../_static/usage/publishing/publish-target-dropdown.png) + +## Publish a dataset + +1. Open the model's **Results** menu and choose **Publish dataset…**. + + ![The Publish dataset… item in a model's Results menu](../_static/usage/publishing/publish-menu-item.png) + + ```{admonition} Why is it disabled? + :class: note + **Publish dataset…** is enabled only once inference is **Processed** and a predicted-damage + geopackage exists. + ``` + +2. In the **Publish dataset** dialog, complete the details: + + - **Dataset name** — prefilled as *` – `*; edit as you like. + - **Description** — prefilled from the assessment summary. + - **Interactive viewer URL** *(optional)* — an `https` link to an external interactive view + of the result. On Planetary Computer it becomes the item's preview link. + - **Source imagery citation** *(optional)* — free text, or a URL that becomes a provenance + link. If your image layer was built from the Open Data Catalog, the source scenes are + listed and linked automatically just above this field — you don't enter them by hand. See + {ref}`Source-imagery attribution `. + +3. Under **Assets to publish**, select which outputs to include — the predicted-damage + geopackage, building footprints, valid-area mask, and (for Local) the processed image. + + ```{admonition} Planetary Computer needs the valid-area mask + :class: warning + For the **Planetary Computer** target the **valid-area mask** must stay selected — it + bounds the published extent, and a publish without it is rejected. Local publishing has no + such requirement. + ``` + +4. Choose the **Target publishing location**, then select **Publish**. + + ![The Publish dataset dialog with details, assets, and target](../_static/usage/publishing/publish-dialog.png) + +Publishing runs in the background. The confirmation dialog offers **View** to jump straight to +**Published Datasets**, where the new entry appears and updates as it progresses. + +## Manage published datasets + +The **Published Datasets** section lists **all datasets published in your HASTE deployment** — +not just your own — each with its publisher, **target**, **status**, and a per-row menu. Use +the search box and the **target**/**status** filters to narrow the list. Anyone with access +can browse and inspect these datasets; only a dataset's owner or an administrator can manage +it (see [Actions](#actions)). + +![The Published Datasets list with a row's action menu open](../_static/usage/publishing/published-datasets-list.png) + +Select a row to open its detail view — the assessment summary, source imagery, project/layer, +model, publish time, assessment counts, and the published assets with their sizes and links. + +![A published dataset's detail view](../_static/usage/publishing/published-dataset-detail.png) + +### Statuses + +| Status | Meaning | +|---|---| +| **Pending / In progress** | The publish (or unpublish) operation is running. In-progress rows can't be edited. | +| **Published** | Live. For Planetary Computer, the STAC item and collection are available in the GeoCatalog. | +| **Failed** | The publish didn't complete. Use **Retry**, or **Force remove** if it can't recover. | +| **Unpublishing** | A removal is in progress. | +| **Unpublish failed** | Removal didn't complete — **Retry** it, or **Force remove**. | + +### Actions + +Anyone with access to HASTE can **View details** for any dataset in the list — the full +metadata, assets, and links. + +A dataset's **owner** (its publisher) and **administrators** can additionally: + +- **Edit metadata** — change the name, description, interactive viewer URL, and source-imagery + citation. Editing is allowed only in a settled state (**Published**, **Failed**, or + **Unpublish failed**); for a published Planetary Computer dataset the edit is pushed to the + live STAC item. + + ![The edit-metadata form](../_static/usage/publishing/edit-metadata.png) +- **Retry** — re-run a **Failed** or **Unpublish failed** operation. +- **Unpublish** — remove the published copies. For Planetary Computer this deletes the STAC + item (and the collection once its last dataset is removed). +- **Force remove** — a last-resort escape hatch for a row stuck in **Failed** / + **Unpublish failed**: it makes a best-effort cleanup and then drops the tracking record so + the row leaves the list. + + ```{admonition} Force remove can leave orphaned resources + :class: warning + If cleanup can't complete, force remove still removes the row — so resources already created + in Planetary Computer may remain and must be deleted manually in the catalog. Use it only + when **Retry** can't recover the dataset. + ``` + +(source-imagery-attribution)= +## Source-imagery attribution + +If an image layer was assembled from the **Open Data Catalog**, HASTE records where the +imagery came from and carries that provenance onto every dataset published from it — so +published results credit the imagery correctly and link back to the exact source scenes. + +On a Planetary Computer dataset this becomes standard STAC attribution and provenance: + +- **Providers** — the imagery vendor is recorded as the `licensor`; the organization operating + your deployment (see `HASTE_PUBLISHING_ORGANIZATION_NAME`) as the `producer` and `processor`. +- **`derived_from` links** — each source scene the output was derived from. +- **Citation** — your optional free-text/URL citation is shown on the dataset and item. + +```{admonition} Only registered open-data imagery is attributed +:class: note +Attribution is applied only for imagery from **registered open-data programs**. This is a +licensing safeguard for the **Planetary Computer** target: publishing there never +redistributes the source imagery to the external catalog — only your derived +damage-assessment outputs are sent. (**Local** publishing can include the processed image, +but it stays inside your own HASTE storage and isn't shared to an external catalog.) +``` + +(viewing-in-the-planetary-computer-explorer)= +## Viewing in the Planetary Computer Explorer + +For Planetary Computer datasets, HASTE also produces a **damage-classification layer** you can +view on the map in the GeoCatalog **Explorer** — damaged buildings in red over undamaged +buildings in grey. + +![The damage-classification layer in the Planetary Computer Explorer](../_static/usage/publishing/explorer-damage-layer.png) + +This layer is HASTE's own derived output (a classification raster), **not** the source +imagery, so it carries the same license and attribution as the vector outputs and never +exposes licensed imagery. Along with it, HASTE registers the render, mosaic, and tile +configuration the Explorer needs, so the collection becomes selectable there. + +To view it: open the collection in the GeoCatalog and choose **Launch in Explorer** (or pick +the collection from the Explorer's dataset list). See the Planetary Computer +[Explorer guide](https://learn.microsoft.com/azure/planetary-computer/use-explorer). + +```{admonition} Requires a render configuration +:class: note +The Explorer can only display a collection that has a render configuration. HASTE creates one +automatically when `HASTE_PUBLISH_EXPLORER_RENDER_ENABLED` is on (the default). The vector +assets (geopackage / GeoJSON) remain downloadable but are not drawn on the map — the Explorer +renders raster layers. +``` + +(enabling-and-configuring-publishing)= +## Enabling and configuring publishing + +Publishing has two parts to set up: **in-app** configuration (environment settings on your +HASTE deployment) and, for the Planetary Computer target, **out-of-app** resources you +provision in Azure. + +### In-app configuration + +Publishing is controlled by `HASTE_*` environment settings applied at deploy time. The Local +target is on by default; the Planetary Computer target is off until you configure it. The full +matrix — feature flags, the GeoCatalog URLs, the publish storage target, and organization +attribution — is documented in the +{doc}`Configuration guide → Data publishing `. + +### Out-of-app: Planetary Computer + +The GeoCatalog is **external** to the HASTE template — you provision and own it. At a high +level: + +1. **Provision a Planetary Computer Pro GeoCatalog** and note its API and Explorer URLs. See + [Deploy a GeoCatalog resource](https://learn.microsoft.com/azure/planetary-computer/deploy-geocatalog-resource). + +2. **Grant the HASTE Function App identity a GeoCatalog data-plane RBAC role** on the + GeoCatalog resource, so it can call the STAC and ingestion APIs. This is the grant that + authorizes publishing itself — without it, HASTE can reach storage but cannot create + catalog resources. Verify the exact role against your GeoCatalog. + +3. **Give HASTE a publish storage container** the GeoCatalog can ingest from, and grant the + **GeoCatalog's managed identity** read access to it (Storage Blob Data Reader). HASTE copies + published assets into this container and points STAC hrefs at it. See + [Manage ingestion sources](https://learn.microsoft.com/azure/planetary-computer/ingestion-source). + +4. **Grant the HASTE Function App identity** write access (Storage Blob Data Contributor) to the + publish storage account so it can stage assets. + +5. **For a private container**, create an ingestion source in the GeoCatalog and set + `HASTE_PC_INGESTION_SOURCE`. Public containers need none. + +Once configured, HASTE handles the rest per publish: it creates the STAC +[collection](https://learn.microsoft.com/azure/planetary-computer/create-stac-collection) and +item, uploads assets, and — when Explorer rendering is enabled — registers the +[render configuration](https://learn.microsoft.com/azure/planetary-computer/render-configuration). + +```{admonition} Managed identity and RBAC +:class: tip +Publishing to Planetary Computer relies on Azure managed identities and role assignments +rather than keys. If a publish fails with an authorization error, check that the three role +grants above are in place — the GeoCatalog data-plane role on the HASTE identity, plus the two +storage roles. See {doc}`Secure configuration `. +``` diff --git a/docs/usage/overview.md b/docs/usage/overview.md index ec4e9cf3..2e8fd807 100644 --- a/docs/usage/overview.md +++ b/docs/usage/overview.md @@ -39,3 +39,6 @@ Those shared building blocks have their own pages: {doc}`Projects `, {doc}`Image layers `, and the {doc}`Model catalog `. See {doc}`Keyboard shortcuts ` for the controls available across map and labeling views. + +When you have a result from either workflow, you can {doc}`publish it ` — +archive it as a downloadable copy or share it to a Planetary Computer STAC catalog. diff --git a/docs/usage/rapid-building-assessment.md b/docs/usage/rapid-building-assessment.md index f4cbcf74..461065d5 100644 --- a/docs/usage/rapid-building-assessment.md +++ b/docs/usage/rapid-building-assessment.md @@ -148,6 +148,12 @@ From the embedding row's **Reports** menu: ![The Assessment Report](../_static/usage/interactive/assessment-report.png) +## Publish the dataset + +If your administrator has enabled publishing, you can also **Publish dataset…** from the +embedding row — archive the assessment as a downloadable copy or share it to a Planetary +Computer STAC catalog. See {doc}`Publishing datasets `. + ## Tips - Label a **diverse** set of buildings (varied roofs, colors, damage severity) rather than