From 30c58ff0515da553e467677e4aa33c79720fb78b Mon Sep 17 00:00:00 2001 From: prbatero <42007693+prbatero@users.noreply.github.com> Date: Sat, 29 Aug 2026 08:03:58 -0400 Subject: [PATCH 1/9] docs(spec): add Planetary Computer Explorer visualization design Stacked feature (-pc-explorer): make PC published collections explorable by rendering a licensing-safe damage classification COG (our derived output, no source pixels) plus the render-option/mosaic/tile-settings configuration the Explorer requires. --- .../data-publishing/explorer-visualization.md | 165 ++++++++++++++++++ 1 file changed, 165 insertions(+) create mode 100644 spec/features/data-publishing/explorer-visualization.md diff --git a/spec/features/data-publishing/explorer-visualization.md b/spec/features/data-publishing/explorer-visualization.md new file mode 100644 index 00000000..1c5dfeb2 --- /dev/null +++ b/spec/features/data-publishing/explorer-visualization.md @@ -0,0 +1,165 @@ +# Design: Planetary Computer Explorer Visualization + +Stacked on top of the data-publishing feature (`-pc`). Delivered as its own +PR (`-pc-explorer`) whose base is `prbatero/feat/data-publishing-pc`. + +## Goal + +Make Planetary Computer (PC) published collections **visible and explorable in +the GeoCatalog Explorer**. Today a published collection is created and listed +under the Collections tab, but it has no "Launch in Explorer" action and does +not appear in the Explorer's dataset picker. + +## Why they don't show today + +Per the GeoCatalog docs, a collection is explorable only when it has, in +addition to `item_assets` + ingested items, a **visualization configuration**: + +- a **render configuration** (`…/configurations/render-options`) — mandatory; + without it "Launch in Explorer" is disabled, +- a **mosaic** (`…/configurations/mosaics`), +- **tile settings** (`…/configurations/tile-settings`). + +We create none of these. Two facts compound the problem: + +1. **We create no render/mosaic/tile config** — so nothing is explorable. +2. The Explorer renderer is **TiTiler raster-only** (every render `type` is + `raster-tile`; `options` is a TiTiler query string over raster bands). Our + PC item assets are **vector-only** (`damage`/`footprints` GeoPackage, `aoi` + GeoJSON), so even with a render config there is nothing raster to draw. + +## Core principle: visualize our output, never redistribute source pixels + +The source imagery (Planet/Vantor/etc.) is licensed; republishing the processed +COG into a PC collection is a redistribution governed by that license and is +**out of scope**. Instead we render **our own derived output**: a small +single-band classification COG (damaged / undamaged), rasterized from the damage +GeoPackage we already publish. It contains **no source pixels**, so it carries +exactly the same license/attribution the vector output already carries and adds +no new licensing exposure. See [source-imagery-provenance](source-imagery-provenance.md) +for the attribution/provenance model this reuses. + +## What we add + +| Piece | Mechanism | Notes | +|---|---|---| +| Damage classification COG | `rasterio.features.rasterize` over footprints | our output; single-band `uint8`; PC target only | +| Item asset `damage_class` | STAC item asset, role `data` | the renderable raster; + collection `item_assets` entry | +| Render option | `POST …/configurations/render-options` | colormap `1→red, 0→grey, 255→transparent`, matching the thumbnail | +| Mosaic | `POST …/configurations/mosaics` | `most-recent`, `cql: []` (show all items) | +| Tile settings | `PUT …/configurations/tile-settings` | `minZoom` at building scale | + +## Rasterization + +Reuse `publishing/tile.py`'s `detect_damage_mask(buildings)` (the same +damaged/undamaged detection the collection thumbnail uses) so the raster and the +thumbnail agree. + +- **Input:** the damage GeoPackage (already fetched for the thumbnail) + the AOI + valid-mask, both already in hand during publish. +- **Grid:** a projected CRS (AOI's UTM / the layer CRS) for correct pixel sizes; + extent clipped to the AOI bounds. +- **Encoding (single band, `uint8`):** `1 = damaged`, `0 = undamaged`, + `255 = nodata` (outside footprints). Undamaged footprints are burned as `0` + so they read as grey; everything outside a footprint is transparent. +- **Resolution:** `PUBLISH_DAMAGE_RASTER_METERS` (default `0.5` m), with a hard + cap on output dimensions (`PUBLISH_DAMAGE_RASTER_MAX_PIXELS`, default e.g. + 8192 per side) — if the AOI at target resolution would exceed the cap, coarsen + the pixel size to fit. Log when coarsened (no silent truncation). +- **Output:** a valid COG (tiled + overviews) so TiTiler serves it efficiently. + +## Render configuration + +```jsonc +// render-options (one entry) +{ + "id": "damage", + "name": "Damage classification", + "description": "Predicted building damage (red) over undamaged (grey).", + "type": "raster-tile", + "options": "assets=damage_class&nodata=255&colormap=", + "minZoom": 13 +} +// mosaic +{ "id": "most-recent", "name": "Most recent available", "cql": [] } +// tile-settings +{ "minZoom": 13, "maxItemsPerTile": 35 } +``` + +The colormap is the URL-encoded JSON discrete map (`0 → grey RGBA`, +`1 → red RGBA`); `nodata=255` keeps outside-footprint pixels transparent. + +## Flow (PC publish) + +1. Collection ensured/updated + item created (existing flow, unchanged). +2. Rasterize damage → COG (temp dir); upload to the publish store + `published//damage_class.tif`. +3. Item gains a `damage_class` asset (role `data`, COG media type); the + collection `item_assets` gains a matching `damage_class` entry. +4. After the item is confirmed, register render-option + mosaic + tile-settings + for the collection (idempotent; once per collection). +5. Unpublish: `finalize_unpublish` already deletes `published//`, so + the classification COG is cleaned up with the rest of the staging prefix. + +Staging + cleanup reuse the `-pc` publish store and `finalize_unpublish` hook. + +## Idempotency (re-publish / second dataset) + +- The COG is per-dataset (`published//`), regenerated per publish. +- Collection-level config (render/mosaic/tile) is per-collection: create once, + tolerate "already exists" on re-publish (GET-or-409 → skip, or PUT where the + API is replace-semantics). Never duplicate render options across datasets. + +## Configuration (env) + +| Setting | Default | Notes | +|---|---|---| +| `PUBLISH_EXPLORER_RENDER_ENABLED` | `true` | master toggle for this feature | +| `PUBLISH_DAMAGE_RASTER_METERS` | `0.5` | target pixel size (m) | +| `PUBLISH_DAMAGE_RASTER_MAX_PIXELS` | `8192` | per-side cap; coarsen to fit | +| `PUBLISH_DAMAGE_RASTER_MIN_ZOOM` | `13` | render/tile `minZoom` | + +## Scope + +- **PC target only.** Local publishing is unaffected (downloads only). +- **Vendor imagery is never redistributed.** Only the derived classification COG + is published. +- Vector assets (`damage`, `footprints`, `aoi`) remain as downloadable item + assets; they are not rendered (the Explorer can't render vector). + +## Non-goals (v1) + +- Damage **severity** classes (data carries a binary damaged flag today) — the + encoding leaves room (`uint8`) to extend later. +- Rendering the source imagery backdrop (licensing; see principle above). +- Local-target Explorer parity (Local has no Explorer). + +## Open decisions + +| # | Decision | Choice | +|---|---|---| +| 1 | What to render | Binary damaged/undamaged classification COG (our output) | +| 2 | Colormap | `1→red`, `0→grey`, `255→transparent` (matches thumbnail) | +| 3 | Resolution | `0.5` m default, capped per-side, coarsen to fit | +| 4 | Where generated | PC provider at publish time (reuses publish store + cleanup) | +| 5 | Config lifecycle | Per-collection, idempotent; COG per-dataset | + +## Testing sketch + +- Rasterization: damaged → `1`, undamaged → `0`, outside → `255`; extent matches + AOI; coarsening triggers past the pixel cap; output is a readable COG. +- Transport: render-option / mosaic / tile-settings POST/PUT bodies + idempotent + re-publish (existing config not duplicated). +- Provider: publish wires the `damage_class` asset + registers config; second + dataset to the same collection doesn't duplicate render options; unpublish + removes the staging COG. + +## Execution plan + +| Phase | Task | Files | Status | +|---|---|---|---| +| 1 | Rasterize damage → classification COG (reuse `detect_damage_mask`) | `publishing/raster.py` (new), `publishing/tile.py` | not-started | +| 2 | Publish COG as `damage_class` item asset (+ `item_assets`) | `publishing/planetary_computer_provider.py`, `publishing/stac.py` | not-started | +| 3 | Transport: render-options / mosaics / tile-settings methods | `publishing/planetary_computer_transport.py` | not-started | +| 4 | Wire config registration into publish flow (idempotent) | `publishing/planetary_computer_provider.py` | not-started | +| 5 | Config knobs + tests + this spec | `config.py`, `tests/core/publishing/` | not-started | From 8d46355468e9342ffbce89cb782084084aecaf57 Mon Sep 17 00:00:00 2001 From: prbatero <42007693+prbatero@users.noreply.github.com> Date: Sat, 29 Aug 2026 08:07:12 -0400 Subject: [PATCH 2/9] feat(pc-explorer): rasterize damage output into a classification COG Phase 1. New publishing/raster.py rasterizes the published damage GeoPackage into a single-band uint8 COG (1=damaged, 0=undamaged, 255=nodata), reusing tile.py's detect_damage_mask so it agrees with the collection thumbnail. Projects to a metric UTM grid clipped to the AOI, coarsens resolution to stay within a per-side pixel cap, and writes a valid COG (GTiff-in-memory -> GDAL COG create-copy with overviews). This is HASTE's derived output, so it carries no source imagery pixels. --- .../src/hastegeo/core/publishing/raster.py | 172 ++++++++++++++++++ hastelib/tests/core/publishing/test_raster.py | 115 ++++++++++++ 2 files changed, 287 insertions(+) create mode 100644 hastelib/src/hastegeo/core/publishing/raster.py create mode 100644 hastelib/tests/core/publishing/test_raster.py diff --git a/hastelib/src/hastegeo/core/publishing/raster.py b/hastelib/src/hastegeo/core/publishing/raster.py new file mode 100644 index 00000000..3d946d15 --- /dev/null +++ b/hastelib/src/hastegeo/core/publishing/raster.py @@ -0,0 +1,172 @@ +"""Rasterize a damage GeoPackage into a single-band classification COG. + +This is HASTE's own derived output (predicted building damage), not the source +imagery — so it can be published and rendered in the Planetary Computer Explorer +without redistributing licensed source pixels. See +``spec/features/data-publishing/explorer-visualization.md``. + +The encoding is a single ``uint8`` band: ``1`` = damaged, ``0`` = undamaged, +``255`` = nodata (outside every footprint). The damaged/undamaged split reuses +``tile.py``'s ``detect_damage_mask`` so the raster agrees with the collection +thumbnail. Heavy geo deps (rasterio/numpy) are imported lazily inside the render +call to keep them off the module import path. +""" + +from __future__ import annotations + +import math +from dataclasses import dataclass +from typing import Any, Optional + +from .tile import detect_damage_mask + +# Single-band uint8 encoding. +UNDAMAGED_VALUE = 0 +DAMAGED_VALUE = 1 +DAMAGE_CLASS_NODATA = 255 + +# Discrete colormap the Explorer render option applies (value -> RGBA). Mirrors +# tile.py's palette: undamaged grey (#7f8fa6), damaged red (#ff4d4d). +DAMAGE_CLASS_COLORMAP = { + UNDAMAGED_VALUE: (127, 143, 166, 255), + DAMAGED_VALUE: (255, 77, 77, 255), +} + + +@dataclass(frozen=True) +class DamageRasterResult: + """Outcome of a successful rasterization (the COG is written to ``path``).""" + + path: str + width: int + height: int + resolution_m: float + total_buildings: int + damaged_buildings: int + coarsened: bool + + +def rasterize_damage_cog( + buildings: Any, + aoi: Any, + out_path: str, + *, + target_meters: float = 0.5, + max_pixels_per_side: int = 8192, + logger: Any = None, +) -> Optional[DamageRasterResult]: + """Rasterize ``buildings`` (classified by damage) into a COG at ``out_path``. + + Returns ``None`` when there is nothing to render (no AOI extent or no + building geometries); otherwise writes a Cloud-Optimized GeoTIFF and returns + its dimensions/counts. The grid is a metric UTM projection clipped to the + AOI bounds; the pixel size starts at ``target_meters`` and is coarsened if + the AOI would otherwise exceed ``max_pixels_per_side`` on a side. + """ + import numpy as np + import rasterio + import rasterio.shutil + from rasterio.features import rasterize + from rasterio.transform import from_origin + + if aoi is None or len(aoi) == 0: + return None + + # Project both layers to a metric CRS so pixel sizes are in real meters. + metric_crs = aoi.estimate_utm_crs() + aoi_m = aoi.to_crs(metric_crs) + minx, miny, maxx, maxy = (float(v) for v in aoi_m.total_bounds) + if not (math.isfinite(minx) and math.isfinite(maxx)) or maxx <= minx or maxy <= miny: + return None + + buildings_m = ( + buildings.to_crs(metric_crs) + if buildings is not None and len(buildings) + else None + ) + if buildings_m is None or len(buildings_m) == 0: + return None + + # Pick a resolution that keeps the raster within the per-side pixel cap. + span = max(maxx - minx, maxy - miny) + resolution = float(target_meters) + coarsened = False + if span / resolution > max_pixels_per_side: + resolution = span / max_pixels_per_side + coarsened = True + if logger is not None: + logger.info( + "Damage raster coarsened to %.3f m to fit %d px/side cap", + resolution, + max_pixels_per_side, + ) + width = max(1, int(math.ceil((maxx - minx) / resolution))) + height = max(1, int(math.ceil((maxy - miny) / resolution))) + transform = from_origin(minx, maxy, resolution, resolution) + + # Damaged wins over undamaged on overlap: burn undamaged first, damaged last. + damaged_mask = detect_damage_mask(buildings_m) + geoms = list(buildings_m.geometry.values) + if damaged_mask is None: + damaged_flags = [False] * len(geoms) + else: + damaged_flags = [bool(v) for v in damaged_mask] + damaged_count = sum(damaged_flags) + + shapes = [ + (geom, DAMAGED_VALUE if flag else UNDAMAGED_VALUE) + for flag, geom in sorted( + zip(damaged_flags, geoms), key=lambda pair: pair[0] + ) + if geom is not None and not geom.is_empty + ] + if not shapes: + return None + + raster = rasterize( + shapes, + out_shape=(height, width), + transform=transform, + fill=DAMAGE_CLASS_NODATA, + default_value=UNDAMAGED_VALUE, + dtype="uint8", + all_touched=False, + ).astype(np.uint8) + + profile = { + "driver": "GTiff", + "height": height, + "width": width, + "count": 1, + "dtype": "uint8", + "crs": metric_crs, + "transform": transform, + "nodata": DAMAGE_CLASS_NODATA, + "tiled": True, + "blockxsize": 512, + "blockysize": 512, + "compress": "deflate", + } + # Write a tiled GeoTIFF in memory, then copy to a valid COG on disk (the GDAL + # COG driver is create-copy only, so it can't be opened for direct write). + with rasterio.MemoryFile() as memfile: + with memfile.open(**profile) as tmp: + tmp.write(raster, 1) + tmp.write_colormap(1, DAMAGE_CLASS_COLORMAP) + rasterio.shutil.copy( + memfile.name, + out_path, + driver="COG", + compress="DEFLATE", + overview_resampling="nearest", + ) + + return DamageRasterResult( + path=out_path, + width=width, + height=height, + resolution_m=resolution, + total_buildings=len(geoms), + damaged_buildings=damaged_count, + coarsened=coarsened, + ) diff --git a/hastelib/tests/core/publishing/test_raster.py b/hastelib/tests/core/publishing/test_raster.py new file mode 100644 index 00000000..488a6070 --- /dev/null +++ b/hastelib/tests/core/publishing/test_raster.py @@ -0,0 +1,115 @@ +import os +import tempfile +import unittest + +import geopandas as gpd +import numpy as np +import rasterio +from shapely.geometry import box + +from hastegeo.core.publishing.raster import ( + DAMAGED_VALUE, + DAMAGE_CLASS_NODATA, + UNDAMAGED_VALUE, + rasterize_damage_cog, +) + +# A ~small AOI near the equator (degrees); two buildings inside it. +AOI = box(-67.10, 10.40, -67.099, 10.401) +DAMAGED_BLDG = box(-67.0999, 10.4001, -67.0997, 10.4003) +INTACT_BLDG = box(-67.0995, 10.4005, -67.0993, 10.4007) + + +def _aoi_gdf(): + return gpd.GeoDataFrame(geometry=[AOI], crs="EPSG:4326") + + +def _buildings_gdf(): + return gpd.GeoDataFrame( + {"predicted_damage": [1, 0]}, + geometry=[DAMAGED_BLDG, INTACT_BLDG], + crs="EPSG:4326", + ) + + +class TestRasterizeDamageCog(unittest.TestCase): + def test_writes_readable_cog_with_expected_classes(self) -> None: + with tempfile.TemporaryDirectory() as tmp: + out = os.path.join(tmp, "damage_class.tif") + result = rasterize_damage_cog( + _buildings_gdf(), _aoi_gdf(), out, target_meters=1.0 + ) + self.assertIsNotNone(result) + self.assertEqual(result.total_buildings, 2) + self.assertEqual(result.damaged_buildings, 1) + self.assertFalse(result.coarsened) + self.assertTrue(os.path.exists(out)) + + with rasterio.open(out) as ds: + self.assertEqual(ds.count, 1) + self.assertEqual(ds.dtypes[0], "uint8") + self.assertEqual(ds.nodata, DAMAGE_CLASS_NODATA) + # COG-style: internally tiled (overviews only appear once the + # raster is larger than the overview threshold, so not asserted + # for this tiny fixture). + self.assertTrue(ds.profile.get("tiled", False)) + band = ds.read(1) + + values = set(np.unique(band).tolist()) + self.assertIn(DAMAGED_VALUE, values) + self.assertIn(UNDAMAGED_VALUE, values) + self.assertIn(DAMAGE_CLASS_NODATA, values) + # Most of the AOI is empty -> nodata dominates. + self.assertGreater((band == DAMAGE_CLASS_NODATA).sum(), (band != DAMAGE_CLASS_NODATA).sum()) + + def test_larger_raster_builds_cog_overviews(self) -> None: + # A ~1 km AOI at 1 m exceeds the overview threshold, so the COG driver + # builds internal overviews (what makes it efficient for TiTiler). + big_aoi = gpd.GeoDataFrame( + geometry=[box(-67.10, 10.40, -67.090, 10.410)], crs="EPSG:4326" + ) + with tempfile.TemporaryDirectory() as tmp: + out = os.path.join(tmp, "damage_class.tif") + result = rasterize_damage_cog( + _buildings_gdf(), big_aoi, out, target_meters=1.0 + ) + self.assertIsNotNone(result) + self.assertGreater(min(result.width, result.height), 512) + with rasterio.open(out) as ds: + self.assertTrue(ds.overviews(1)) + + def test_coarsens_past_pixel_cap(self) -> None: + with tempfile.TemporaryDirectory() as tmp: + out = os.path.join(tmp, "damage_class.tif") + result = rasterize_damage_cog( + _buildings_gdf(), + _aoi_gdf(), + out, + target_meters=0.01, # would be huge; force coarsening + max_pixels_per_side=64, + ) + self.assertIsNotNone(result) + self.assertTrue(result.coarsened) + self.assertLessEqual(max(result.width, result.height), 64) + + def test_returns_none_without_buildings(self) -> None: + empty = _buildings_gdf().iloc[0:0] + with tempfile.TemporaryDirectory() as tmp: + out = os.path.join(tmp, "damage_class.tif") + self.assertIsNone( + rasterize_damage_cog(empty, _aoi_gdf(), out) + ) + self.assertFalse(os.path.exists(out)) + + def test_returns_none_without_aoi(self) -> None: + with tempfile.TemporaryDirectory() as tmp: + out = os.path.join(tmp, "damage_class.tif") + self.assertIsNone( + rasterize_damage_cog( + _buildings_gdf(), _aoi_gdf().iloc[0:0], out + ) + ) + + +if __name__ == "__main__": + unittest.main() From 63af794f5514a5a00c478b1519b8600d85709435 Mon Sep 17 00:00:00 2001 From: prbatero <42007693+prbatero@users.noreply.github.com> Date: Sat, 29 Aug 2026 08:12:29 -0400 Subject: [PATCH 3/9] feat(pc-explorer): publish damage classification COG as an item asset Phase 2. At PC publish time the provider rasterizes the damage output to a COG (idempotent per dataset), stages it under published//, and injects a 'damage_class' asset (role data) into the item plus a matching item_assets entry on the collection -- the renderable raster the Explorer render config will point at. Shares the buildings/AOI loader with the thumbnail and the finalize_unpublish cleanup (now also runs when Explorer rendering is on, since a COG is written even without a dedicated store). Best-effort: rasterization failure never fails the publish. --- .../publishing/planetary_computer_provider.py | 155 +++++++++++++++--- .../src/hastegeo/core/publishing/raster.py | 7 + .../test_planetary_computer_provider.py | 84 +++++++++- 3 files changed, 222 insertions(+), 24 deletions(-) diff --git a/hastelib/src/hastegeo/core/publishing/planetary_computer_provider.py b/hastelib/src/hastegeo/core/publishing/planetary_computer_provider.py index 0a68c8cd..47043770 100644 --- a/hastelib/src/hastegeo/core/publishing/planetary_computer_provider.py +++ b/hastelib/src/hastegeo/core/publishing/planetary_computer_provider.py @@ -117,6 +117,19 @@ def __init__( ).strip(), } self.max_verify_attempts = int(settings.get("pc_verify_attempts") or 5) + # Explorer visualization (damage classification COG + render config). + self._explorer_render_enabled = bool( + settings.get("publish_explorer_render_enabled", True) + ) + self._damage_raster_meters = float( + settings.get("publish_damage_raster_meters") or 0.5 + ) + self._damage_raster_max_pixels = int( + settings.get("publish_damage_raster_max_pixels") or 8192 + ) + self._damage_raster_min_zoom = int( + settings.get("publish_damage_raster_min_zoom") or 13 + ) self.artifact_storage = artifact_storage or UnifiedArtifactStorage( storage_type=self.config.artifact_storage_type, **self.config.artifact_storage_config, @@ -1161,29 +1174,11 @@ def _upload_collection_tile( # attach it as the collection thumbnail via the Collection Asset API. # Best-effort: never fail the publish over a tile. try: - import geopandas as gpd - from .tile import render_collection_tile - mask = source.get(ArtifactKind.VALID_MASK) - if mask is None: - return - valid_mask = self.json_reader(mask) - aoi_gdf = gpd.GeoDataFrame.from_features( - valid_mask.get("features") or [], crs="EPSG:4326" - ) - if aoi_gdf.empty: + buildings_gdf, aoi_gdf = self._load_buildings_aoi(source) + if aoi_gdf is None: return - buildings_artifact = source.get(ArtifactKind.GPKG) or source.get( - ArtifactKind.FOOTPRINTS - ) - if buildings_artifact is not None: - with self._materialized_artifact( - buildings_artifact - ) as local_path: - buildings_gdf = gpd.read_file(local_path) - else: - buildings_gdf = aoi_gdf.iloc[0:0] png = render_collection_tile( buildings_gdf, aoi_gdf, @@ -1379,7 +1374,9 @@ def _build_documents( organization=self.organization, ) self.stac_validator(objects) - return serialize_stac_objects(objects) + documents = serialize_stac_objects(objects) + self._attach_damage_class_asset(dataset, source, documents) + return documents @staticmethod def _valid_mask_crs(valid_mask: Mapping[str, Any]) -> str: @@ -1543,6 +1540,118 @@ def _stage_to_publish( namespace=["published", str(dataset_id)], ) + def _load_buildings_aoi(self, source: ArtifactBundle): + """Load ``(buildings, aoi)`` GeoDataFrames from the published bundle. + + ``aoi`` is ``None`` when there is no valid-area mask (or it is empty); + ``buildings`` is an empty frame when no building artifact is present. + Shared by the collection thumbnail and the damage classification COG. + """ + import geopandas as gpd + + mask = source.get(ArtifactKind.VALID_MASK) + if mask is None: + return None, None + valid_mask = self.json_reader(mask) + aoi_gdf = gpd.GeoDataFrame.from_features( + valid_mask.get("features") or [], crs="EPSG:4326" + ) + if aoi_gdf.empty: + return None, None + buildings_artifact = source.get(ArtifactKind.GPKG) or source.get( + ArtifactKind.FOOTPRINTS + ) + if buildings_artifact is not None: + with self._materialized_artifact(buildings_artifact) as local_path: + buildings_gdf = gpd.read_file(local_path) + else: + buildings_gdf = aoi_gdf.iloc[0:0] + return buildings_gdf, aoi_gdf + + def _stage_damage_class_asset( + self, dataset: PublishedDataset, source: ArtifactBundle + ) -> Optional[dict]: + """Rasterize the damage output to a COG, stage it, return its STAC asset. + + Returns the item asset dict (with a publish-store href), or ``None`` + when the feature is disabled or there is nothing to rasterize. + Best-effort: a failure never fails the publish (the collection is still + created, just without Explorer visualization). + """ + if not self._explorer_render_enabled: + return None + from .raster import ( + DAMAGE_CLASS_ASSET_TITLE, + DAMAGE_CLASS_MEDIA_TYPE, + rasterize_damage_cog, + ) + + dest_name = "damage_class.tif" + destination = f"published/{dataset.datasetId}/{dest_name}" + try: + if not self.publish_storage.artifact_exists(destination): + buildings_gdf, aoi_gdf = self._load_buildings_aoi(source) + if aoi_gdf is None: + return None + with tempfile.TemporaryDirectory() as staging_dir: + cog_path = str(Path(staging_dir, dest_name)) + result = rasterize_damage_cog( + buildings_gdf, + aoi_gdf, + cog_path, + target_meters=self._damage_raster_meters, + max_pixels_per_side=self._damage_raster_max_pixels, + logger=self.logger, + ) + if result is None: + return None + self.publish_storage.store_artifact( + artifact_name=dest_name, + src_path=cog_path, + namespace=["published", str(dataset.datasetId)], + ) + return { + "href": self._publish_href(destination), + "type": DAMAGE_CLASS_MEDIA_TYPE, + "title": DAMAGE_CLASS_ASSET_TITLE, + "roles": ["data"], + } + except Exception as error: + self.logger.warning( + "Skipping Planetary Computer damage classification COG: %s", + type(error).__name__, + ) + return None + + def _attach_damage_class_asset( + self, + dataset: PublishedDataset, + source: ArtifactBundle, + documents, + ) -> None: + """Inject the ``damage_class`` COG asset into the item + collection. + + Added post-serialization so the raster asset (our derived output) is + the renderable asset the Explorer render configuration points at. + """ + from .raster import ( + DAMAGE_CLASS_ASSET_KEY, + DAMAGE_CLASS_ASSET_TITLE, + DAMAGE_CLASS_MEDIA_TYPE, + ) + + asset = self._stage_damage_class_asset(dataset, source) + if asset is None: + return + documents.item.setdefault("assets", {})[DAMAGE_CLASS_ASSET_KEY] = asset + documents.collection.setdefault("item_assets", {})[ + DAMAGE_CLASS_ASSET_KEY + ] = { + "type": DAMAGE_CLASS_MEDIA_TYPE, + "title": DAMAGE_CLASS_ASSET_TITLE, + "roles": ["data"], + } + def finalize_unpublish(self, dataset: PublishedDataset) -> None: """Remove staging copies once an unpublish has fully completed. @@ -1551,7 +1660,9 @@ def finalize_unpublish(self, dataset: PublishedDataset) -> None: item/collection, so those staging blobs would otherwise accumulate indefinitely. Best-effort: a failure here must not fail the unpublish. """ - if not self._stages_to_publish: + # A damage classification COG is written under the same prefix even + # without a dedicated publish store, so clean up when either applies. + if not (self._stages_to_publish or self._explorer_render_enabled): return prefix = f"published/{dataset.datasetId}/" try: diff --git a/hastelib/src/hastegeo/core/publishing/raster.py b/hastelib/src/hastegeo/core/publishing/raster.py index 3d946d15..fc868aa2 100644 --- a/hastelib/src/hastegeo/core/publishing/raster.py +++ b/hastelib/src/hastegeo/core/publishing/raster.py @@ -25,6 +25,13 @@ DAMAGED_VALUE = 1 DAMAGE_CLASS_NODATA = 255 +# STAC asset identity for the classification COG (the renderable raster). +DAMAGE_CLASS_ASSET_KEY = "damage_class" +DAMAGE_CLASS_ASSET_TITLE = "Damage classification" +DAMAGE_CLASS_MEDIA_TYPE = ( + "image/tiff; application=geotiff; profile=cloud-optimized" +) + # Discrete colormap the Explorer render option applies (value -> RGBA). Mirrors # tile.py's palette: undamaged grey (#7f8fa6), damaged red (#ff4d4d). DAMAGE_CLASS_COLORMAP = { diff --git a/hastelib/tests/core/publishing/test_planetary_computer_provider.py b/hastelib/tests/core/publishing/test_planetary_computer_provider.py index 16d3d936..cdabd939 100644 --- a/hastelib/tests/core/publishing/test_planetary_computer_provider.py +++ b/hastelib/tests/core/publishing/test_planetary_computer_provider.py @@ -844,8 +844,29 @@ def test_finalize_unpublish_deletes_staging_prefix(self) -> None: [f"published/{self.dataset.datasetId}/"], ) - def test_finalize_unpublish_noop_without_dedicated_store(self) -> None: - # No publish_storage -> staging never happened; nothing to delete. + def test_finalize_unpublish_cleans_cog_without_dedicated_store( + self, + ) -> None: + # Even without a dedicated publish store, a damage classification COG + # is written under published//, so finalize must clean it up when + # Explorer rendering is enabled (the default). + provider = PlanetaryComputerPublishingProvider( + config=self.config, + artifact_storage=self.storage, + sdk_adapter=self.sdk, + json_reader=lambda artifact: self.valid_mask, + projection_resolver=lambda artifact: "EPSG:4326", + asset_reachability_checker=lambda href: None, + ) + provider.finalize_unpublish(self.dataset) + self.assertEqual( + self.storage.deleted_prefixes, + [f"published/{self.dataset.datasetId}/"], + ) + + def test_finalize_unpublish_noop_when_nothing_staged(self) -> None: + # No dedicated store and Explorer rendering disabled -> nothing was + # written under the prefix, so finalize is a no-op. provider = PlanetaryComputerPublishingProvider( config=self.config, artifact_storage=self.storage, @@ -854,9 +875,68 @@ def test_finalize_unpublish_noop_without_dedicated_store(self) -> None: projection_resolver=lambda artifact: "EPSG:4326", asset_reachability_checker=lambda href: None, ) + provider._explorer_render_enabled = False provider.finalize_unpublish(self.dataset) self.assertEqual(self.storage.deleted_prefixes, []) + def test_damage_class_asset_injected_into_documents(self) -> None: + publish = FakeArtifactStorage( + "https://publishsa.blob.core.windows.net/publish" + ) + provider = PlanetaryComputerPublishingProvider( + config=self.config, + artifact_storage=self.storage, + publish_storage=publish, + sdk_adapter=self.sdk, + json_reader=lambda artifact: self.valid_mask, + projection_resolver=lambda artifact: "EPSG:4326", + asset_reachability_checker=lambda href: None, + ) + # Stub the (heavy) rasterization; assert only the injection wiring here. + provider._stage_damage_class_asset = lambda dataset, source: { + "href": ( + "https://publishsa.blob.core.windows.net/publish/" + f"published/{dataset.datasetId}/damage_class.tif" + ), + "type": "image/tiff; application=geotiff; profile=cloud-optimized", + "title": "Damage classification", + "roles": ["data"], + } + documents = provider._build_documents( + self.dataset, + self.bundle, + provider._projection_codes(self.dataset, self.bundle), + None, + ) + self.assertIn("damage_class", documents.item["assets"]) + self.assertEqual( + documents.item["assets"]["damage_class"]["roles"], ["data"] + ) + self.assertIn("damage_class", documents.collection["item_assets"]) + + def test_damage_class_asset_absent_when_disabled(self) -> None: + provider = PlanetaryComputerPublishingProvider( + config=self.config, + artifact_storage=self.storage, + sdk_adapter=self.sdk, + json_reader=lambda artifact: self.valid_mask, + projection_resolver=lambda artifact: "EPSG:4326", + asset_reachability_checker=lambda href: None, + ) + provider._explorer_render_enabled = False + documents = provider._build_documents( + self.dataset, + self.bundle, + provider._projection_codes(self.dataset, self.bundle), + None, + ) + self.assertNotIn( + "damage_class", documents.item.get("assets", {}) + ) + self.assertNotIn( + "damage_class", documents.collection.get("item_assets", {}) + ) + def test_ingestion_source_validates_against_publish_container( self, ) -> None: From 42d299c2af698faa364db444cb8ac07e7bc175d6 Mon Sep 17 00:00:00 2001 From: prbatero <42007693+prbatero@users.noreply.github.com> Date: Sat, 29 Aug 2026 08:13:52 -0400 Subject: [PATCH 4/9] feat(pc-explorer): transport methods for render/mosaic/tile config Phase 3. Add get/create render-options, get/create mosaics, and replace tile-settings on the GeoCatalog REST adapter (the .../configurations/* endpoints the Explorer requires). GETs unwrap list/dict-wrapped bodies and treat 404 as empty so the provider can ensure config idempotently. --- .../planetary_computer_transport.py | 65 +++++++++++++++++++ .../test_planetary_computer_transport.py | 53 +++++++++++++++ 2 files changed, 118 insertions(+) diff --git a/hastelib/src/hastegeo/core/publishing/planetary_computer_transport.py b/hastelib/src/hastegeo/core/publishing/planetary_computer_transport.py index 2b830707..4a1a9b91 100644 --- a/hastelib/src/hastegeo/core/publishing/planetary_computer_transport.py +++ b/hastelib/src/hastegeo/core/publishing/planetary_computer_transport.py @@ -245,6 +245,71 @@ def upload_collection_asset( expected=(200, 201), ) + # ------------------------------------------------ visualization config + + @staticmethod + def _config_list(payload: Any) -> list: + """Normalize a configurations GET body to a list of dicts.""" + if isinstance(payload, list): + return payload + if isinstance(payload, Mapping): + for key in ("renderOptions", "render_options", "mosaics", "value"): + value = payload.get(key) + if isinstance(value, list): + return value + return [] + + def get_render_options(self, collection_id: str) -> list: + response = self.client.request( + "GET", + f"/stac/collections/{collection_id}/configurations/render-options", + expected=(200, 404), + ) + if response.status_code == 404: + return [] + return self._config_list(response.json()) + + def create_render_option( + self, collection_id: str, body: Mapping[str, Any] + ) -> None: + self.client.request( + "POST", + f"/stac/collections/{collection_id}/configurations/render-options", + json=dict(body), + expected=(200, 201), + ) + + def get_mosaics(self, collection_id: str) -> list: + response = self.client.request( + "GET", + f"/stac/collections/{collection_id}/configurations/mosaics", + expected=(200, 404), + ) + if response.status_code == 404: + return [] + return self._config_list(response.json()) + + def create_mosaic( + self, collection_id: str, body: Mapping[str, Any] + ) -> None: + self.client.request( + "POST", + f"/stac/collections/{collection_id}/configurations/mosaics", + json=dict(body), + expected=(200, 201), + ) + + def replace_tile_settings( + self, collection_id: str, body: Mapping[str, Any] + ) -> None: + # PUT is replace-semantics, so this is naturally idempotent. + self.client.request( + "PUT", + f"/stac/collections/{collection_id}/configurations/tile-settings", + json=dict(body), + expected=(200, 201, 202, 204), + ) + # ------------------------------------------------------------------ items def get_item( diff --git a/hastelib/tests/core/publishing/test_planetary_computer_transport.py b/hastelib/tests/core/publishing/test_planetary_computer_transport.py index 60e841ac..d48415ce 100644 --- a/hastelib/tests/core/publishing/test_planetary_computer_transport.py +++ b/hastelib/tests/core/publishing/test_planetary_computer_transport.py @@ -205,6 +205,59 @@ def handler(method, url): self.assertNotIn("assets", put["json"]) self.assertEqual(put["json"]["description"], "d") + def test_create_render_option_posts_to_configurations(self): + client = FakeClient(lambda m, u: FakeResponse(201, payload={})) + rest = PlanetaryComputerRestAdapter(ENDPOINT, client=client) + rest.create_render_option("haste-c", {"id": "damage", "type": "raster-tile"}) + call = client.calls[-1] + self.assertEqual(call["method"], "POST") + self.assertEqual( + call["url"], + "/stac/collections/haste-c/configurations/render-options", + ) + self.assertEqual(call["json"]["id"], "damage") + + def test_get_render_options_unwraps_and_handles_404(self): + # Bare list. + got = adapter( + lambda m, u: FakeResponse(200, payload=[{"id": "damage"}]) + ).get_render_options("c") + self.assertEqual(got, [{"id": "damage"}]) + # Dict-wrapped. + got = adapter( + lambda m, u: FakeResponse( + 200, payload={"renderOptions": [{"id": "x"}]} + ) + ).get_render_options("c") + self.assertEqual(got, [{"id": "x"}]) + # 404 -> empty. + self.assertEqual( + adapter(lambda m, u: FakeResponse(404)).get_render_options("c"), [] + ) + + def test_create_mosaic_posts_to_configurations(self): + client = FakeClient(lambda m, u: FakeResponse(201, payload={})) + rest = PlanetaryComputerRestAdapter(ENDPOINT, client=client) + rest.create_mosaic("haste-c", {"id": "most-recent", "cql": []}) + call = client.calls[-1] + self.assertEqual(call["method"], "POST") + self.assertEqual( + call["url"], "/stac/collections/haste-c/configurations/mosaics" + ) + self.assertEqual(call["json"]["id"], "most-recent") + + def test_replace_tile_settings_puts(self): + client = FakeClient(lambda m, u: FakeResponse(200, payload={})) + rest = PlanetaryComputerRestAdapter(ENDPOINT, client=client) + rest.replace_tile_settings("haste-c", {"minZoom": 13}) + call = client.calls[-1] + self.assertEqual(call["method"], "PUT") + self.assertEqual( + call["url"], + "/stac/collections/haste-c/configurations/tile-settings", + ) + self.assertEqual(call["json"]["minZoom"], 13) + def test_start_collection_async_202_pins_operation_url(self): step = adapter( lambda m, u: FakeResponse(202, {"operation-location": OP_URL}) From 139d090d9a386d2834c6cc2ea3844382c5e35ebd Mon Sep 17 00:00:00 2001 From: prbatero <42007693+prbatero@users.noreply.github.com> Date: Sat, 29 Aug 2026 08:16:22 -0400 Subject: [PATCH 5/9] feat(pc-explorer): register render/mosaic/tile config on publish Phase 4. After the item is published (alongside the thumbnail), register the Explorer visualization config idempotently: a 'damage' raster-tile render option (colormap 1=red/0=grey, nodata transparent, pointing at the damage_class asset), a 'most-recent' mosaic, and tile-settings. Gated on a staged damage COG so the render option always references a real asset; create-if-absent so a second dataset to the same collection doesn't duplicate. Best-effort: never fails the publish. --- .../publishing/planetary_computer_provider.py | 87 +++++++++++++++++++ .../test_planetary_computer_provider.py | 84 ++++++++++++++++++ 2 files changed, 171 insertions(+) diff --git a/hastelib/src/hastegeo/core/publishing/planetary_computer_provider.py b/hastelib/src/hastegeo/core/publishing/planetary_computer_provider.py index 47043770..f77d091a 100644 --- a/hastelib/src/hastegeo/core/publishing/planetary_computer_provider.py +++ b/hastelib/src/hastegeo/core/publishing/planetary_computer_provider.py @@ -1153,6 +1153,7 @@ def _completed_publish( ) collection_id, _ = self._ids(dataset) self._upload_collection_tile(dataset, source, collection_id) + self._ensure_explorer_config(dataset, collection_id) metadata = self._stable_metadata(dataset) metadata["assetsCopiedToManagedStorage"] = True return PublishResult( @@ -1652,6 +1653,92 @@ def _attach_damage_class_asset( "roles": ["data"], } + def _damage_render_option(self) -> dict: + import json + import urllib.parse + + from .raster import ( + DAMAGE_CLASS_ASSET_KEY, + DAMAGE_CLASS_COLORMAP, + DAMAGE_CLASS_NODATA, + ) + + colormap = json.dumps( + {str(value): list(rgba) for value, rgba in DAMAGE_CLASS_COLORMAP.items()} + ) + options = urllib.parse.urlencode( + { + "assets": DAMAGE_CLASS_ASSET_KEY, + "nodata": DAMAGE_CLASS_NODATA, + "colormap": colormap, + } + ) + return { + "id": "damage", + "name": "Damage classification", + "description": ( + "Predicted building damage (red) over undamaged buildings " + "(grey)." + ), + "type": "raster-tile", + "options": options, + "minZoom": self._damage_raster_min_zoom, + } + + def _ensure_explorer_config( + self, dataset: PublishedDataset, collection_id: str + ) -> None: + """Register the render/mosaic/tile config the Explorer requires. + + Idempotent (create-if-absent; tile-settings is a PUT). Runs only when a + damage classification COG was actually staged for this dataset, so the + render option always points at a real renderable asset. Best-effort: a + failure never fails the publish, it just leaves the collection + non-explorable until the next publish. + """ + if not self._explorer_render_enabled: + return + destination = f"published/{dataset.datasetId}/damage_class.tif" + try: + if not self.publish_storage.artifact_exists(destination): + return + render_ids = { + option.get("id") + for option in self.sdk.get_render_options(collection_id) + if isinstance(option, Mapping) + } + if "damage" not in render_ids: + self.sdk.create_render_option( + collection_id, self._damage_render_option() + ) + mosaic_ids = { + mosaic.get("id") + for mosaic in self.sdk.get_mosaics(collection_id) + if isinstance(mosaic, Mapping) + } + if "most-recent" not in mosaic_ids: + self.sdk.create_mosaic( + collection_id, + { + "id": "most-recent", + "name": "Most recent available", + "description": "Show the most recent available data", + "cql": [], + }, + ) + self.sdk.replace_tile_settings( + collection_id, + { + "minZoom": self._damage_raster_min_zoom, + "maxItemsPerTile": 35, + }, + ) + except Exception as error: + self.logger.warning( + "Skipping Planetary Computer Explorer configuration: %s", + type(error).__name__, + ) + def finalize_unpublish(self, dataset: PublishedDataset) -> None: """Remove staging copies once an unpublish has fully completed. diff --git a/hastelib/tests/core/publishing/test_planetary_computer_provider.py b/hastelib/tests/core/publishing/test_planetary_computer_provider.py index cdabd939..1037cc78 100644 --- a/hastelib/tests/core/publishing/test_planetary_computer_provider.py +++ b/hastelib/tests/core/publishing/test_planetary_computer_provider.py @@ -108,6 +108,28 @@ def __init__(self) -> None: self.pending_collection_delete = None self.collection_asset_uploads = [] self.item_updates = [] + self.render_options = [] + self.mosaics = [] + self.tile_settings = None + self.render_option_creates = 0 + self.mosaic_creates = 0 + + def get_render_options(self, collection_id): + return list(self.render_options) + + def create_render_option(self, collection_id, body): + self.render_option_creates += 1 + self.render_options.append(dict(body)) + + def get_mosaics(self, collection_id): + return list(self.mosaics) + + def create_mosaic(self, collection_id, body): + self.mosaic_creates += 1 + self.mosaics.append(dict(body)) + + def replace_tile_settings(self, collection_id, body): + self.tile_settings = dict(body) def get_ingestion_source(self, source_id): self.ingestion_source_calls += 1 @@ -937,6 +959,68 @@ def test_damage_class_asset_absent_when_disabled(self) -> None: "damage_class", documents.collection.get("item_assets", {}) ) + def _explorer_provider(self): + publish = FakeArtifactStorage( + "https://publishsa.blob.core.windows.net/publish" + ) + # A staged damage COG gates config registration. + publish.blobs.add(f"published/{self.dataset.datasetId}/damage_class.tif") + provider = PlanetaryComputerPublishingProvider( + config=self.config, + artifact_storage=self.storage, + publish_storage=publish, + sdk_adapter=self.sdk, + json_reader=lambda artifact: self.valid_mask, + projection_resolver=lambda artifact: "EPSG:4326", + asset_reachability_checker=lambda href: None, + ) + return provider + + def test_ensure_explorer_config_registers_render_mosaic_tile(self) -> None: + provider = self._explorer_provider() + provider._ensure_explorer_config(self.dataset, "haste-c") + self.assertEqual([o["id"] for o in self.sdk.render_options], ["damage"]) + self.assertEqual([m["id"] for m in self.sdk.mosaics], ["most-recent"]) + self.assertIsNotNone(self.sdk.tile_settings) + self.assertEqual(self.sdk.tile_settings["minZoom"], 13) + # The render option points at the damage_class raster asset. + self.assertIn("assets=damage_class", self.sdk.render_options[0]["options"]) + + def test_ensure_explorer_config_is_idempotent(self) -> None: + provider = self._explorer_provider() + self.sdk.render_options = [{"id": "damage"}] + self.sdk.mosaics = [{"id": "most-recent"}] + provider._ensure_explorer_config(self.dataset, "haste-c") + # No duplicate render option / mosaic; tile-settings still PUT. + self.assertEqual(self.sdk.render_option_creates, 0) + self.assertEqual(self.sdk.mosaic_creates, 0) + self.assertIsNotNone(self.sdk.tile_settings) + + def test_ensure_explorer_config_skips_without_cog(self) -> None: + # No staged COG -> nothing renderable -> register nothing. + publish = FakeArtifactStorage( + "https://publishsa.blob.core.windows.net/publish" + ) + provider = PlanetaryComputerPublishingProvider( + config=self.config, + artifact_storage=self.storage, + publish_storage=publish, + sdk_adapter=self.sdk, + json_reader=lambda artifact: self.valid_mask, + projection_resolver=lambda artifact: "EPSG:4326", + asset_reachability_checker=lambda href: None, + ) + provider._ensure_explorer_config(self.dataset, "haste-c") + self.assertEqual(self.sdk.render_option_creates, 0) + self.assertIsNone(self.sdk.tile_settings) + + def test_ensure_explorer_config_noop_when_disabled(self) -> None: + provider = self._explorer_provider() + provider._explorer_render_enabled = False + provider._ensure_explorer_config(self.dataset, "haste-c") + self.assertEqual(self.sdk.render_option_creates, 0) + self.assertIsNone(self.sdk.tile_settings) + def test_ingestion_source_validates_against_publish_container( self, ) -> None: From 2e6ae14ef011ae75efc5fce4161f0deae7541503 Mon Sep 17 00:00:00 2001 From: prbatero <42007693+prbatero@users.noreply.github.com> Date: Sat, 29 Aug 2026 08:18:17 -0400 Subject: [PATCH 6/9] feat(pc-explorer): config knobs for damage raster + render toggle Phase 5. Wire PUBLISH_EXPLORER_RENDER_ENABLED (default on), PUBLISH_DAMAGE_RASTER_METERS (0.5), PUBLISH_DAMAGE_RASTER_MAX_PIXELS (8192), and PUBLISH_DAMAGE_RASTER_MIN_ZOOM (13) into the publishing config, and mark the spec execution plan complete. Env-drift check green. --- hastelib/src/hastegeo/core/config.py | 15 +++++++++++++++ .../data-publishing/explorer-visualization.md | 10 +++++----- 2 files changed, 20 insertions(+), 5 deletions(-) diff --git a/hastelib/src/hastegeo/core/config.py b/hastelib/src/hastegeo/core/config.py index fc4472f0..bf8d5dbe 100644 --- a/hastelib/src/hastegeo/core/config.py +++ b/hastelib/src/hastegeo/core/config.py @@ -368,6 +368,21 @@ def get_publishing_config(): "pc_verify_attempts": _get_bounded_int_env( "PC_VERIFY_ATTEMPTS", 20, 1, 60 ), + # Explorer visualization: render a damage-classification COG (our + # derived output, not source imagery) plus the render/mosaic/tile + # config the GeoCatalog Explorer requires. + "publish_explorer_render_enabled": _get_bool_env( + "PUBLISH_EXPLORER_RENDER_ENABLED", True + ), + "publish_damage_raster_meters": os.getenv( + "PUBLISH_DAMAGE_RASTER_METERS", "0.5" + ), + "publish_damage_raster_max_pixels": _get_bounded_int_env( + "PUBLISH_DAMAGE_RASTER_MAX_PIXELS", 8192, 256, 20000 + ), + "publish_damage_raster_min_zoom": _get_bounded_int_env( + "PUBLISH_DAMAGE_RASTER_MIN_ZOOM", 13, 0, 24 + ), "lease_connection_string": os.getenv("AzureWebJobsStorage"), "lease_account_url": os.getenv("BLOB_ACCOUNT_URL"), "lease_container": os.getenv( diff --git a/spec/features/data-publishing/explorer-visualization.md b/spec/features/data-publishing/explorer-visualization.md index 1c5dfeb2..bb8c6e1d 100644 --- a/spec/features/data-publishing/explorer-visualization.md +++ b/spec/features/data-publishing/explorer-visualization.md @@ -158,8 +158,8 @@ Staging + cleanup reuse the `-pc` publish store and `finalize_unpublish` hook. | Phase | Task | Files | Status | |---|---|---|---| -| 1 | Rasterize damage → classification COG (reuse `detect_damage_mask`) | `publishing/raster.py` (new), `publishing/tile.py` | not-started | -| 2 | Publish COG as `damage_class` item asset (+ `item_assets`) | `publishing/planetary_computer_provider.py`, `publishing/stac.py` | not-started | -| 3 | Transport: render-options / mosaics / tile-settings methods | `publishing/planetary_computer_transport.py` | not-started | -| 4 | Wire config registration into publish flow (idempotent) | `publishing/planetary_computer_provider.py` | not-started | -| 5 | Config knobs + tests + this spec | `config.py`, `tests/core/publishing/` | not-started | +| 1 | Rasterize damage → classification COG (reuse `detect_damage_mask`) | `publishing/raster.py` (new), `publishing/tile.py` | done | +| 2 | Publish COG as `damage_class` item asset (+ `item_assets`) | `publishing/planetary_computer_provider.py` | done | +| 3 | Transport: render-options / mosaics / tile-settings methods | `publishing/planetary_computer_transport.py` | done | +| 4 | Wire config registration into publish flow (idempotent) | `publishing/planetary_computer_provider.py` | done | +| 5 | Config knobs + tests + this spec | `config.py`, `tests/core/publishing/` | done | From 082dab761904a52c32d9b9f600500b4b4eecf3bf Mon Sep 17 00:00:00 2001 From: prbatero <42007693+prbatero@users.noreply.github.com> Date: Sat, 29 Aug 2026 12:00:34 -0400 Subject: [PATCH 7/9] feat(pc-explorer): thread PUBLISH_EXPLORER_RENDER_ENABLED through deploy Expose the Explorer-render toggle as a first-class deploy input across both paths (functions.bicep + main.bicep/param, deploy_apps.sh, and the deploy-apps workflow), mirroring PC_PROVIDER_ENABLED. Defaults on. The advanced raster knobs stay code-defaults (rarely tuned). Env-drift check green; bicep builds. --- .github/scripts/deploy_apps.sh | 3 +++ .github/workflows/deploy-apps.yml | 1 + infra/main.bicep | 4 ++++ infra/main.bicepparam | 1 + infra/modules/functions.bicep | 4 ++++ 5 files changed, 13 insertions(+) diff --git a/.github/scripts/deploy_apps.sh b/.github/scripts/deploy_apps.sh index 8789954f..04ec7d9f 100644 --- a/.github/scripts/deploy_apps.sh +++ b/.github/scripts/deploy_apps.sh @@ -72,6 +72,8 @@ PUBLISHING_ORGANIZATION_NAME="${PUBLISHING_ORGANIZATION_NAME:-}" PUBLISHING_ORGANIZATION_URL="${PUBLISHING_ORGANIZATION_URL:-}" PUBLISH_STORAGE_ACCOUNT_URL="${PUBLISH_STORAGE_ACCOUNT_URL:-}" PUBLISH_BLOB_CONTAINER="${PUBLISH_BLOB_CONTAINER:-}" +# Explorer visualization (damage classification COG + render config); default on. +PUBLISH_EXPLORER_RENDER_ENABLED="${PUBLISH_EXPLORER_RENDER_ENABLED:-true}" MAPS_ACCOUNT="${RESOURCE_PREFIX}haste${RANDOM_SUFFIX}maps" API_MANAGEMENT="${RESOURCE_PREFIX}-haste-${RANDOM_SUFFIX}-apim" FIXED_TAGS="project=haste created_by=deploy_apps" @@ -167,6 +169,7 @@ deploy_function() { "PUBLISHING_ORGANIZATION_URL=${PUBLISHING_ORGANIZATION_URL}" \ "PUBLISH_STORAGE_ACCOUNT_URL=${PUBLISH_STORAGE_ACCOUNT_URL}" \ "PUBLISH_BLOB_CONTAINER=${PUBLISH_BLOB_CONTAINER}" \ + "PUBLISH_EXPLORER_RENDER_ENABLED=${PUBLISH_EXPLORER_RENDER_ENABLED}" \ "AzureFunctionsWebHost__hostId=${HOST_ID}" \ --output none fi diff --git a/.github/workflows/deploy-apps.yml b/.github/workflows/deploy-apps.yml index 67e96d76..063e94f9 100644 --- a/.github/workflows/deploy-apps.yml +++ b/.github/workflows/deploy-apps.yml @@ -118,6 +118,7 @@ jobs: # Each defaults to off/unset in deploy_apps.sh when absent. PUBLISHING_ENABLED: ${{ vars.PUBLISHING_ENABLED }} PC_PROVIDER_ENABLED: ${{ vars.PC_PROVIDER_ENABLED }} + PUBLISH_EXPLORER_RENDER_ENABLED: ${{ vars.PUBLISH_EXPLORER_RENDER_ENABLED }} PC_COLLECTION_PREFIX: ${{ vars.PC_COLLECTION_PREFIX }} PUBLISHING_ORGANIZATION_NAME: ${{ secrets.PUBLISHING_ORGANIZATION_NAME }} PUBLISHING_ORGANIZATION_URL: ${{ secrets.PUBLISHING_ORGANIZATION_URL }} diff --git a/infra/main.bicep b/infra/main.bicep index d4c401d6..4decc070 100644 --- a/infra/main.bicep +++ b/infra/main.bicep @@ -126,6 +126,9 @@ param publishingEnabled bool = true @description('Register/expose the Planetary Computer publishing provider.') param pcProviderEnabled bool = false +@description('Render a damage classification COG + Explorer visualization config on PC publish.') +param publishExplorerRenderEnabled bool = true + @description('MPC Pro GeoCatalog base URL (no trailing slash). Operator-provisioned.') param pcGeocatalogUrl string = '' @@ -331,6 +334,7 @@ module functions 'modules/functions.bicep' = { managePools: managePools publishingEnabled: publishingEnabled pcProviderEnabled: pcProviderEnabled + publishExplorerRenderEnabled: publishExplorerRenderEnabled pcGeocatalogUrl: pcGeocatalogUrl pcExplorerUrl: pcExplorerUrl pcIngestionSource: pcIngestionSource diff --git a/infra/main.bicepparam b/infra/main.bicepparam index 7d8de00e..9e684f4c 100644 --- a/infra/main.bicepparam +++ b/infra/main.bicepparam @@ -52,6 +52,7 @@ param publishingEnabled = bool(readEnvironmentVariable('HASTE_PUBLISHING_ENABLED // Planetary Computer publishing target. Provisioned/owned by the operator; the // GeoCatalog is external to this template. Default off / unset. param pcProviderEnabled = bool(readEnvironmentVariable('HASTE_PC_PROVIDER_ENABLED', 'false')) +param publishExplorerRenderEnabled = bool(readEnvironmentVariable('HASTE_PUBLISH_EXPLORER_RENDER_ENABLED', 'true')) param pcGeocatalogUrl = readEnvironmentVariable('HASTE_PC_GEOCATALOG_URL', '') param pcExplorerUrl = readEnvironmentVariable('HASTE_PC_EXPLORER_URL', '') param pcIngestionSource = readEnvironmentVariable('HASTE_PC_INGESTION_SOURCE', '') diff --git a/infra/modules/functions.bicep b/infra/modules/functions.bicep index 0ade4d95..958eb1c7 100644 --- a/infra/modules/functions.bicep +++ b/infra/modules/functions.bicep @@ -97,6 +97,9 @@ param publishingEnabled bool = true @description('Register/expose the Planetary Computer publishing provider.') param pcProviderEnabled bool = false +@description('Render a damage classification COG + Explorer visualization config on PC publish.') +param publishExplorerRenderEnabled bool = true + @description('MPC Pro GeoCatalog base URL (no trailing slash). Operator-provisioned.') param pcGeocatalogUrl string = '' @@ -205,6 +208,7 @@ var appConfigSettings = [ { name: 'PUBLISHING_ORGANIZATION_URL', value: publishingOrganizationUrl } { name: 'PUBLISH_STORAGE_ACCOUNT_URL', value: publishStorageAccountUrl } { name: 'PUBLISH_BLOB_CONTAINER', value: publishBlobContainer } + { name: 'PUBLISH_EXPLORER_RENDER_ENABLED', value: publishExplorerRenderEnabled ? 'true' : 'false' } ] module apiApp 'functionApp.bicep' = { From d70a8097197e9ceaa40d02005f12615bf13b60a4 Mon Sep 17 00:00:00 2001 From: prbatero <42007693+prbatero@users.noreply.github.com> Date: Mon, 31 Aug 2026 10:00:48 -0400 Subject: [PATCH 8/9] fix(pc-explorer): address Copilot review on #175 - raster/provider: only produce the damage_class COG when a predicted-damage geopackage is present. A footprints-only publish has no damage column, so rasterizing it would publish a misleading all-undamaged layer -- skip both the rasterize and the reuse-of-staged-COG paths. - finalize_unpublish: always delete the dataset-specific published// prefix instead of gating on the current feature flags. If rendering was enabled at publish and disabled before unpublish, the staged COG would otherwise leak. - transport: accept HTTP 409 as idempotent success when creating render options and mosaics, so a concurrent publish racing the get-then-create doesn't abort before mosaic/tile-settings are registered. - config: add _get_bounded_float_env and bound PUBLISH_DAMAGE_RASTER_METERS to a positive finite float, so an operator typo fails fast with a clear message rather than crashing provider construction or raster generation. - tests: assert the COG's projected CRS, resolution, and AOI-derived bounds; cover the footprints-only skip, the always-clean unpublish, and 409 tolerance. - spec: add a Contents list to explorer-visualization.md. --- hastelib/src/hastegeo/core/config.py | 19 +++++++++-- .../publishing/planetary_computer_provider.py | 21 +++++++----- .../planetary_computer_transport.py | 9 +++-- .../src/hastegeo/core/publishing/raster.py | 12 ++++--- .../test_planetary_computer_provider.py | 15 +++++--- .../test_planetary_computer_transport.py | 7 ++++ hastelib/tests/core/publishing/test_raster.py | 34 +++++++++++++++++++ .../data-publishing/explorer-visualization.md | 17 ++++++++++ 8 files changed, 113 insertions(+), 21 deletions(-) diff --git a/hastelib/src/hastegeo/core/config.py b/hastelib/src/hastegeo/core/config.py index bf8d5dbe..9675d1ba 100644 --- a/hastelib/src/hastegeo/core/config.py +++ b/hastelib/src/hastegeo/core/config.py @@ -1,6 +1,7 @@ # Copyright (c) Microsoft Corporation. All rights reserved. # Licensed under the MIT License. import logging +import math import os import re import tempfile @@ -30,6 +31,20 @@ def _get_bounded_int_env(name, default, minimum, maximum=None): return value +def _get_bounded_float_env(name, default, minimum, maximum=None): + raw = os.getenv(name, str(default)) + try: + value = float(raw) + except (TypeError, ValueError): + raise ValueError(f"{name} must be a number") + if not math.isfinite(value): + raise ValueError(f"{name} must be a finite number") + if value < minimum or (maximum is not None and value > maximum): + upper = f" and {maximum}" if maximum is not None else "" + raise ValueError(f"{name} must be between {minimum}{upper}") + return value + + def _strip_scheme(value): """Reduce a registry URL to the bare login server. @@ -374,8 +389,8 @@ def get_publishing_config(): "publish_explorer_render_enabled": _get_bool_env( "PUBLISH_EXPLORER_RENDER_ENABLED", True ), - "publish_damage_raster_meters": os.getenv( - "PUBLISH_DAMAGE_RASTER_METERS", "0.5" + "publish_damage_raster_meters": _get_bounded_float_env( + "PUBLISH_DAMAGE_RASTER_METERS", 0.5, 0.01, 100.0 ), "publish_damage_raster_max_pixels": _get_bounded_int_env( "PUBLISH_DAMAGE_RASTER_MAX_PIXELS", 8192, 256, 20000 diff --git a/hastelib/src/hastegeo/core/publishing/planetary_computer_provider.py b/hastelib/src/hastegeo/core/publishing/planetary_computer_provider.py index f77d091a..3848e6a6 100644 --- a/hastelib/src/hastegeo/core/publishing/planetary_computer_provider.py +++ b/hastelib/src/hastegeo/core/publishing/planetary_computer_provider.py @@ -122,7 +122,7 @@ def __init__( settings.get("publish_explorer_render_enabled", True) ) self._damage_raster_meters = float( - settings.get("publish_damage_raster_meters") or 0.5 + settings.get("publish_damage_raster_meters", 0.5) ) self._damage_raster_max_pixels = int( settings.get("publish_damage_raster_max_pixels") or 8192 @@ -1581,6 +1581,11 @@ def _stage_damage_class_asset( """ if not self._explorer_render_enabled: return None + # The classification is derived from the predicted-damage geopackage; + # without it (e.g. a footprints-only publish) there is nothing to + # render, and reusing a previously staged COG would be misleading. + if source.get(ArtifactKind.GPKG) is None: + return None from .raster import ( DAMAGE_CLASS_ASSET_TITLE, DAMAGE_CLASS_MEDIA_TYPE, @@ -1742,15 +1747,15 @@ def _ensure_explorer_config( def finalize_unpublish(self, dataset: PublishedDataset) -> None: """Remove staging copies once an unpublish has fully completed. - With a dedicated publish container, published assets are copied under + Published assets and the damage-classification COG are staged under ``published//``. GeoCatalog cleanup only removes the STAC - item/collection, so those staging blobs would otherwise accumulate - indefinitely. Best-effort: a failure here must not fail the unpublish. + item/collection, so those blobs would otherwise accumulate. Always + attempt to delete this dataset-specific prefix -- the current feature + flags are not reliable evidence of what was staged when the dataset was + published (rendering may have since been toggled, or a dedicated store + added/removed), and the delete is a no-op when nothing is there. + Best-effort: a failure here must not fail the unpublish. """ - # A damage classification COG is written under the same prefix even - # without a dedicated publish store, so clean up when either applies. - if not (self._stages_to_publish or self._explorer_render_enabled): - return prefix = f"published/{dataset.datasetId}/" try: self.publish_storage.delete_prefix(prefix) diff --git a/hastelib/src/hastegeo/core/publishing/planetary_computer_transport.py b/hastelib/src/hastegeo/core/publishing/planetary_computer_transport.py index 4a1a9b91..2c0639b7 100644 --- a/hastelib/src/hastegeo/core/publishing/planetary_computer_transport.py +++ b/hastelib/src/hastegeo/core/publishing/planetary_computer_transport.py @@ -272,11 +272,14 @@ def get_render_options(self, collection_id: str) -> list: def create_render_option( self, collection_id: str, body: Mapping[str, Any] ) -> None: + # 409 = the render option already exists. Concurrent publishes to the + # same collection can both pass the get-then-create check and race here; + # an existing config is the idempotent success we want, not a failure. self.client.request( "POST", f"/stac/collections/{collection_id}/configurations/render-options", json=dict(body), - expected=(200, 201), + expected=(200, 201, 409), ) def get_mosaics(self, collection_id: str) -> list: @@ -292,11 +295,13 @@ def get_mosaics(self, collection_id: str) -> list: def create_mosaic( self, collection_id: str, body: Mapping[str, Any] ) -> None: + # 409 = the mosaic already exists; accept it as idempotent success so a + # concurrent publish's race doesn't abort the rest of the config. self.client.request( "POST", f"/stac/collections/{collection_id}/configurations/mosaics", json=dict(body), - expected=(200, 201), + expected=(200, 201, 409), ) def replace_tile_settings( diff --git a/hastelib/src/hastegeo/core/publishing/raster.py b/hastelib/src/hastegeo/core/publishing/raster.py index fc868aa2..68862fcb 100644 --- a/hastelib/src/hastegeo/core/publishing/raster.py +++ b/hastelib/src/hastegeo/core/publishing/raster.py @@ -111,13 +111,15 @@ def rasterize_damage_cog( height = max(1, int(math.ceil((maxy - miny) / resolution))) transform = from_origin(minx, maxy, resolution, resolution) - # Damaged wins over undamaged on overlap: burn undamaged first, damaged last. + # Require a recognizable damage classification. Footprints (or any layer + # without a damage column) yield no mask; rendering them would publish a + # misleading all-undamaged layer, so skip instead. damaged_mask = detect_damage_mask(buildings_m) - geoms = list(buildings_m.geometry.values) if damaged_mask is None: - damaged_flags = [False] * len(geoms) - else: - damaged_flags = [bool(v) for v in damaged_mask] + return None + geoms = list(buildings_m.geometry.values) + # Damaged wins over undamaged on overlap: burn undamaged first, damaged last. + damaged_flags = [bool(v) for v in damaged_mask] damaged_count = sum(damaged_flags) shapes = [ diff --git a/hastelib/tests/core/publishing/test_planetary_computer_provider.py b/hastelib/tests/core/publishing/test_planetary_computer_provider.py index 1037cc78..6e21e979 100644 --- a/hastelib/tests/core/publishing/test_planetary_computer_provider.py +++ b/hastelib/tests/core/publishing/test_planetary_computer_provider.py @@ -886,9 +886,13 @@ def test_finalize_unpublish_cleans_cog_without_dedicated_store( [f"published/{self.dataset.datasetId}/"], ) - def test_finalize_unpublish_noop_when_nothing_staged(self) -> None: - # No dedicated store and Explorer rendering disabled -> nothing was - # written under the prefix, so finalize is a no-op. + def test_finalize_unpublish_cleans_prefix_even_when_render_disabled( + self, + ) -> None: + # The dataset may have been published while rendering was enabled and + # the flag flipped since, so finalize must always attempt to delete the + # dataset-specific prefix regardless of the current flags (the delete is + # a harmless no-op when nothing is there). provider = PlanetaryComputerPublishingProvider( config=self.config, artifact_storage=self.storage, @@ -899,7 +903,10 @@ def test_finalize_unpublish_noop_when_nothing_staged(self) -> None: ) provider._explorer_render_enabled = False provider.finalize_unpublish(self.dataset) - self.assertEqual(self.storage.deleted_prefixes, []) + self.assertEqual( + self.storage.deleted_prefixes, + [f"published/{self.dataset.datasetId}/"], + ) def test_damage_class_asset_injected_into_documents(self) -> None: publish = FakeArtifactStorage( diff --git a/hastelib/tests/core/publishing/test_planetary_computer_transport.py b/hastelib/tests/core/publishing/test_planetary_computer_transport.py index d48415ce..d864b85b 100644 --- a/hastelib/tests/core/publishing/test_planetary_computer_transport.py +++ b/hastelib/tests/core/publishing/test_planetary_computer_transport.py @@ -246,6 +246,13 @@ def test_create_mosaic_posts_to_configurations(self): ) self.assertEqual(call["json"]["id"], "most-recent") + def test_config_creates_tolerate_409_already_exists(self): + # A concurrent publish can race the get-then-create and receive 409; + # that already-exists outcome is idempotent success, not an error. + rest = adapter(lambda m, u: FakeResponse(409)) + rest.create_render_option("haste-c", {"id": "damage"}) + rest.create_mosaic("haste-c", {"id": "most-recent"}) + def test_replace_tile_settings_puts(self): client = FakeClient(lambda m, u: FakeResponse(200, payload={})) rest = PlanetaryComputerRestAdapter(ENDPOINT, client=client) diff --git a/hastelib/tests/core/publishing/test_raster.py b/hastelib/tests/core/publishing/test_raster.py index 488a6070..d3dcd6c8 100644 --- a/hastelib/tests/core/publishing/test_raster.py +++ b/hastelib/tests/core/publishing/test_raster.py @@ -53,6 +53,25 @@ def test_writes_readable_cog_with_expected_classes(self) -> None: # raster is larger than the overview threshold, so not asserted # for this tiny fixture). self.assertTrue(ds.profile.get("tiled", False)) + + # Grid is a metric UTM projection derived from the AOI... + self.assertTrue(ds.crs.is_projected) + self.assertEqual( + ds.crs, _aoi_gdf().estimate_utm_crs() + ) + # ...at the requested resolution... + self.assertAlmostEqual(ds.res[0], 1.0, places=6) + self.assertAlmostEqual(ds.res[1], 1.0, places=6) + self.assertEqual((ds.width, ds.height), (result.width, result.height)) + # ...clipped to the AOI bounds (top-left exact; bottom-right + # within one pixel of ceil padding). + aoi_m = _aoi_gdf().to_crs(ds.crs) + minx, miny, maxx, maxy = aoi_m.total_bounds + self.assertAlmostEqual(ds.bounds.left, minx, places=3) + self.assertAlmostEqual(ds.bounds.top, maxy, places=3) + self.assertLess(abs(ds.bounds.right - maxx), ds.res[0] + 1e-6) + self.assertLess(abs(ds.bounds.bottom - miny), ds.res[1] + 1e-6) + band = ds.read(1) values = set(np.unique(band).tolist()) @@ -101,6 +120,21 @@ def test_returns_none_without_buildings(self) -> None: ) self.assertFalse(os.path.exists(out)) + def test_returns_none_without_damage_column(self) -> None: + # Footprints (no recognized damage column) must not produce an + # all-undamaged raster -- skip so nothing misleading is published. + footprints = gpd.GeoDataFrame( + {"height_m": [10, 12]}, + geometry=[DAMAGED_BLDG, INTACT_BLDG], + crs="EPSG:4326", + ) + with tempfile.TemporaryDirectory() as tmp: + out = os.path.join(tmp, "damage_class.tif") + self.assertIsNone( + rasterize_damage_cog(footprints, _aoi_gdf(), out) + ) + self.assertFalse(os.path.exists(out)) + def test_returns_none_without_aoi(self) -> None: with tempfile.TemporaryDirectory() as tmp: out = os.path.join(tmp, "damage_class.tif") diff --git a/spec/features/data-publishing/explorer-visualization.md b/spec/features/data-publishing/explorer-visualization.md index bb8c6e1d..457fd75c 100644 --- a/spec/features/data-publishing/explorer-visualization.md +++ b/spec/features/data-publishing/explorer-visualization.md @@ -3,6 +3,23 @@ Stacked on top of the data-publishing feature (`-pc`). Delivered as its own PR (`-pc-explorer`) whose base is `prbatero/feat/data-publishing-pc`. +## Contents + +- [Goal](#goal) +- [Why they don't show today](#why-they-dont-show-today) +- [Core principle](#core-principle-visualize-our-output-never-redistribute-source-pixels) +- [What we add](#what-we-add) +- [Rasterization](#rasterization) +- [Render configuration](#render-configuration) +- [Flow (PC publish)](#flow-pc-publish) +- [Idempotency](#idempotency-re-publish--second-dataset) +- [Configuration (env)](#configuration-env) +- [Scope](#scope) +- [Non-goals (v1)](#non-goals-v1) +- [Open decisions](#open-decisions) +- [Testing sketch](#testing-sketch) +- [Execution plan](#execution-plan) + ## Goal Make Planetary Computer (PC) published collections **visible and explorable in From fbab1f260f8fa15583027f8b004b36dbd560b8d1 Mon Sep 17 00:00:00 2001 From: prbatero <42007693+prbatero@users.noreply.github.com> Date: Tue, 1 Sep 2026 07:16:38 -0400 Subject: [PATCH 9/9] refactor(pc-explorer): address reviewer follow-ups on #175 - raster: replace the sorted(zip(...)) with an explicit two-pass partition (undamaged then damaged). O(n) instead of O(n log n), no intermediate sorted copy, and avoids relying on a sort key to keep non-orderable shapely geometries from being compared. damaged_count now counts the geometries actually burned (skips empty ones). (mgmachado) - provider: soften the _stage_damage_class_asset docstring so the best-effort guarantee isn't read as unconditional -- an OOM-kill terminates the worker rather than raising, so it can still fail the publish; the per-side pixel cap keeps that path unlikely. (mgmachado) - spec: add a Resource use section documenting the raster memory bound (~400 MB at the 20000-px ceiling vs a 4096 MB instance) and that vector-side memory is uncapped and scales with building count. (mgmachado) - deploy_apps.sh: drop the lone inline comment on the publishing env block to match the surrounding params. (calebrob6) --- .github/scripts/deploy_apps.sh | 1 - .../publishing/planetary_computer_provider.py | 7 ++++-- .../src/hastegeo/core/publishing/raster.py | 23 +++++++++++-------- .../data-publishing/explorer-visualization.md | 15 ++++++++++++ 4 files changed, 34 insertions(+), 12 deletions(-) diff --git a/.github/scripts/deploy_apps.sh b/.github/scripts/deploy_apps.sh index 04ec7d9f..3635acb1 100644 --- a/.github/scripts/deploy_apps.sh +++ b/.github/scripts/deploy_apps.sh @@ -72,7 +72,6 @@ PUBLISHING_ORGANIZATION_NAME="${PUBLISHING_ORGANIZATION_NAME:-}" PUBLISHING_ORGANIZATION_URL="${PUBLISHING_ORGANIZATION_URL:-}" PUBLISH_STORAGE_ACCOUNT_URL="${PUBLISH_STORAGE_ACCOUNT_URL:-}" PUBLISH_BLOB_CONTAINER="${PUBLISH_BLOB_CONTAINER:-}" -# Explorer visualization (damage classification COG + render config); default on. PUBLISH_EXPLORER_RENDER_ENABLED="${PUBLISH_EXPLORER_RENDER_ENABLED:-true}" MAPS_ACCOUNT="${RESOURCE_PREFIX}haste${RANDOM_SUFFIX}maps" API_MANAGEMENT="${RESOURCE_PREFIX}-haste-${RANDOM_SUFFIX}-apim" diff --git a/hastelib/src/hastegeo/core/publishing/planetary_computer_provider.py b/hastelib/src/hastegeo/core/publishing/planetary_computer_provider.py index 3848e6a6..9ee0bbaf 100644 --- a/hastelib/src/hastegeo/core/publishing/planetary_computer_provider.py +++ b/hastelib/src/hastegeo/core/publishing/planetary_computer_provider.py @@ -1576,8 +1576,11 @@ def _stage_damage_class_asset( Returns the item asset dict (with a publish-store href), or ``None`` when the feature is disabled or there is nothing to rasterize. - Best-effort: a failure never fails the publish (the collection is still - created, just without Explorer visualization). + Best-effort: an ordinary failure is caught and degrades to a publish + without Explorer visualization (the collection is still created). The + one exception is resource exhaustion — an OOM-kill terminates the + worker rather than raising, so it can still fail the publish; the raster + is bounded by the per-side pixel cap to keep that path unlikely. """ if not self._explorer_render_enabled: return None diff --git a/hastelib/src/hastegeo/core/publishing/raster.py b/hastelib/src/hastegeo/core/publishing/raster.py index 68862fcb..865a9f4a 100644 --- a/hastelib/src/hastegeo/core/publishing/raster.py +++ b/hastelib/src/hastegeo/core/publishing/raster.py @@ -118,19 +118,24 @@ def rasterize_damage_cog( if damaged_mask is None: return None geoms = list(buildings_m.geometry.values) - # Damaged wins over undamaged on overlap: burn undamaged first, damaged last. damaged_flags = [bool(v) for v in damaged_mask] - damaged_count = sum(damaged_flags) - shapes = [ - (geom, DAMAGED_VALUE if flag else UNDAMAGED_VALUE) - for flag, geom in sorted( - zip(damaged_flags, geoms), key=lambda pair: pair[0] - ) - if geom is not None and not geom.is_empty - ] + # Partition into (geom, value) pairs, burning undamaged first so damaged + # wins on overlap. A two-pass partition is O(n) and avoids sorting shapely + # geometries (which aren't orderable) just to separate two classes. + undamaged_shapes = [] + damaged_shapes = [] + for flag, geom in zip(damaged_flags, geoms): + if geom is None or geom.is_empty: + continue + if flag: + damaged_shapes.append((geom, DAMAGED_VALUE)) + else: + undamaged_shapes.append((geom, UNDAMAGED_VALUE)) + shapes = undamaged_shapes + damaged_shapes if not shapes: return None + damaged_count = len(damaged_shapes) raster = rasterize( shapes, diff --git a/spec/features/data-publishing/explorer-visualization.md b/spec/features/data-publishing/explorer-visualization.md index 457fd75c..1d0cd1aa 100644 --- a/spec/features/data-publishing/explorer-visualization.md +++ b/spec/features/data-publishing/explorer-visualization.md @@ -14,6 +14,7 @@ PR (`-pc-explorer`) whose base is `prbatero/feat/data-publishing-pc`. - [Flow (PC publish)](#flow-pc-publish) - [Idempotency](#idempotency-re-publish--second-dataset) - [Configuration (env)](#configuration-env) +- [Resource use](#resource-use) - [Scope](#scope) - [Non-goals (v1)](#non-goals-v1) - [Open decisions](#open-decisions) @@ -136,6 +137,20 @@ Staging + cleanup reuse the `-pc` publish store and `finalize_unpublish` hook. | `PUBLISH_DAMAGE_RASTER_MAX_PIXELS` | `8192` | per-side cap; coarsen to fit | | `PUBLISH_DAMAGE_RASTER_MIN_ZOOM` | `13` | render/tile `minZoom` | +## Resource use + +- **Raster held in memory:** a `MemoryFile` GeoTIFF is copied to a COG on disk, bounded by the + per-side cap. At the `PUBLISH_DAMAGE_RASTER_MAX_PIXELS` ceiling (20000) that is + 20000² × 1 B ≈ 400 MB against a 4096 MB instance at `batchSize: 1`; the `0.5` m / `8192`-px + defaults are far smaller. +- **Vector-side memory is uncapped and scales with building count** — the reprojected + footprints plus the partitioned `shapes` pairs (references, not geometry copies). Modest at + realistic AOIs, but the bigger driver than the raster at very large footprint counts. + Revisit if the per-side cap ceiling rises or the encoding widens past single-band `uint8`. +- Rasterization is **best-effort**: an ordinary failure skips the Explorer layer without + failing the publish. The one gap is resource exhaustion — an OOM-kill terminates the worker + rather than raising, so it can still fail the publish (the per-side cap keeps this unlikely). + ## Scope - **PC target only.** Local publishing is unaffected (downloads only).