perf(ui): lazy-load routes and Azure Maps assets [5/5]

spec/features/perf-layer-loading/README.md

@@ -1,11 +1,18 @@
 # Feature: Image Layer & Model Run Loading Performance
 
-**Status:** draft
+**Status:** in-progress
 **Author:** prbatero
 **Date:** 2026-08-03
-**Target Release:** TBD
 **Priority:** P1
 
+## Contents
+
+- [Summary](#summary)
+- [Motivation](#motivation)
+- [Success Criteria](#success-criteria)
+- [HASTE Components Affected](#haste-components-affected)
+- [Document Index](#document-index)
+
 ## Summary
 
 The HASTE UI takes too long to load a project's image layers and their associated
@@ -15,8 +22,8 @@ patterns in the `GetProjectDetails` API path, fully-sequential (never paralleliz
 blob I/O in `hastelib`, full-container blob scans without prefix filtering, and a UI
 that re-fetches the entire project — every model of every layer — every 20 seconds
 while re-rendering the whole component tree. This spec catalogs the verified
-bottlenecks and lays out a phased plan to make layer/run loading fast and roughly
-constant-time regardless of project size.
+bottlenecks and lays out a phased plan to remove sequential amplification, bound
+process-wide concurrency, and avoid duplicate or idle refresh work.
 
 ## Motivation
 
@@ -33,21 +40,25 @@ constant-time regardless of project size.
 
 - [ ] `GET GetProjectDetails?includeModels=True` for a 50-layer / ~5-models-per-layer
       project returns in **< 1.5s p95** (from a current baseline measured in Phase 0).
-- [ ] Backend storage round-trips for that request drop from **O(layers × models)**
-      (~600 for the 50×5 case) to **O(1) small constant** (≤ ~6 partition reads).
+- [x] Logical data-layer calls for that request drop from **603 to 7**. This metric
+      counts processor operations, not Azure REST transactions; Blob downloads still
+      scale with the number of returned records.
+- [x] Aggregate blocking Blob I/O is capped by one process-wide executor (16 workers
+      by default), including concurrent HTTP requests.
 - [ ] UI time-to-interactive for the project page is **< 2s p95** on the same project
-      and no longer scales linearly with layer count.
-- [ ] Background refresh no longer refetches unchanged data or re-renders the full
-      tree; idle CPU and network on an open project page drop measurably.
+      and no longer scales linearly with layer count. Current production-bundle
+      observation is **2.12 s** with 58 ms post-response rendering.
+- [ ] Idle projects no longer poll and fresh conditional requests avoid storage via a
+      15-second process-local cache. Measure the resulting browser CPU/network delta.
 
 ## HASTE Components Affected
 
 | Component | Impact |
 |---|---|
 | `hastelib/src/hastegeo/core/data_layer/` | Add prefix-scoped listing, parallel/metadata-only reads, fix double-deserialize |
-| `hastelib/src/hastegeo/core/processors/` | `MetadataProcessor` filtered load + optional request-scoped cache |
-| `hastelib/src/hastegeo/core/artifact_storage/` | Parallelize multi-blob fetch |
-| `api/hastefuncapi/` | Rewrite `GetProjectDetails` layer loop; add cache headers |
+| `hastelib/src/hastegeo/core/processors/` | Keyed project-details loader and batch metadata reads |
+| `hastelib/src/hastegeo/core/artifact_storage/` | Bounded, atomic multi-blob fetch |
+| `api/hastefuncapi/` | Thin `GetProjectDetails` route; process-local TTL cache and ETags |
 | `api/hastefuncqueues/` | Parallelize independent loads; fix N+1 label lookup; batch saves |
 | `ui/src/Components/` | Smart polling, memoization, split context, lazy model expansion |
 
@@ -67,4 +78,5 @@ constant-time regardless of project size.
 | [impact-analysis.md](impact-analysis.md) | Risk, blast radius, backward compat | draft |
 | [test-plan.md](test-plan.md) | Benchmark harness & regression coverage | draft |
 | [results.md](results.md) | **Phase 0 measured baseline** (603 round-trips, 20.8 s API, 40.3 s UI TTI @ 50×5) | done |
+| [user-stories.md](user-stories.md) | User outcomes, acceptance criteria, and agent assignments | in-progress |
 | [tools/](tools/) | Seed + benchmarks: `phase0_baseline.py`, `bench_api_http.py`, `ui_bench.cjs`; `docker/docker-compose.perf.yml` overlay | done |

spec/features/perf-layer-loading/design.md

@@ -2,103 +2,86 @@
 
 ## Overview
 
-Collapse the `GetProjectDetails` request from `O(layers × models)` sequential storage
-round-trips to a small constant by (1) hoisting redundant reads out of loops,
-(2) batch-loading each metadata type **once per partition** and joining in memory,
-(3) parallelizing genuinely independent I/O, and (4) pushing filtering into the data
-layer. Add HTTP caching so unchanged data isn't recomputed. On the UI, stop the
-20-second full-project poll from thrashing the render tree by adding change-detection,
-context splitting, and memoization. Backend fixes are sequenced first because they are
-the dominant cost and are transparent to the UI.
+Collapse `GetProjectDetails` from hundreds of sequential logical data-layer calls to
+seven top-level operations. Preserve storage-key semantics for legacy records, overlap
+independent I/O behind one process-wide concurrency budget, and reuse Azure SDK clients.
+A bounded process-local cache deduplicates concurrent requests and serves fresh ETag
+checks without storage work. Underlying Blob GET transactions still scale with the
+records returned; a materialized project view would be required to make those constant.
+
+## Contents
+
+- [Architecture](#architecture)
+- [API Design](#api-design)
+- [Internal Interfaces](#internal-interfaces-hastegeo)
+- [Caching](#caching)
+- [UI Design](#ui-design-phase-4)
+- [Configuration](#configuration)
+- [Observability](#observability)
 
 ## Architecture
 
 ```
 ┌──────────────┐   GetProjectDetails    ┌────────────────────┐   1 read / type   ┌──────────────┐
 │   React UI   │───────────────────────▶│   hastefuncapi      │──────────────────▶│ Blob / Cosmos │
-│  Project.jsx │  (ETag / 304 aware)    │  GetProjectDetails   │  (batched+joined)  │  data layer  │
+│  Project.jsx │  (single-flight/ETag)  │  GetProjectDetails   │ keyed batch reads  │  data layer  │
 └──────────────┘                        └─────────┬──────────┘                    └──────────────┘
       ▲ smart poll (304 fast-path)                │ uses
       │                                  ┌─────────▼──────────┐
-      └── memoized rows, split context   │  MetadataProcessor  │  load_filtered() /
-                                         │  (+ load_map cache) │  load_all_from_partition (prefix)
+      └── active-job-only polling         │ ProjectDetailsProc. │ load_map() / exact
+                     │ + shared I/O budget │ metadata prefixes
                                          └────────────────────┘
 ```
 
 ## API Design
 
 ### `GET /api/GetProjectDetails` — reworked internals (contract unchanged by default)
 
-Same request/response shape by default, so the UI keeps working during rollout. New
-**optional** query params enable the lighter paths incrementally:
+The response shape remains unchanged. The only implemented query parameter is:
 
 | Param | Type | Default | Effect |
 |---|---|---|---|
-| `includeModels` | bool | existing | unchanged |
-| `includeArtifacts` | bool | `true` (compat) | when `false`, skip B2's per-model artifact/label expansion; return `modelCount` + model summaries only |
-| `summary` | bool | `false` | return per-layer counts (`modelCount`, `labelProjectCount`, `validationLabelCount`) without nested `models[]` — the shape the list view actually needs |
+| `includeModels` | bool | `false` | Include models, artifacts, and train-label URLs. |
 
-**New response headers:** `ETag` (hash of the serialized payload) and
-`Cache-Control: private, max-age=15`. On a conditional request whose `If-None-Match`
-matches, return `304` with an empty body.
+`summary` and `includeArtifacts` remain deferred. Do not send them until their response
+contracts are implemented and tested.
+
+Response headers are `ETag`, `Cache-Control: private, max-age=<ttl>`, and
+`X-Haste-Cache`. A matching `If-None-Match` returns `304` with an empty body. Request
+`Cache-Control: no-cache` or `max-age=0` forces a storage refresh before comparison.
 
 ### Reworked handler logic (replaces [function_app.py:534-638](../../../api/hastefuncapi/function_app.py#L534-L638))
 
 ```python
-# 1. Independent top-level reads in parallel
-project, image_layers, models, label_projects = await asyncio.gather(
-    _load(PROJECT.load, project_id),
-    _load(IMAGELAYER.load_all_from_partition),
-    _load(MODEL.load_all_from_partition) if include_models else _none(),
-    _load(LABELS.load_all_from_partition),          # ONCE, not per-layer (fixes B1)
-)
-
-# 2. Index once, join in memory — no per-item I/O
-models_by_layer   = group_by(models or [], "imageLayerId")
-labels_by_layer   = index_by(label_projects, "imageLayerId")
-
-# 3. Validation + (optional) artifacts: batch in parallel, bounded concurrency
-validation_by_layer = await gather_map(
-    {l["imageLayerId"]: VALIDATION.load for l in image_layers}
-)   # fixes B3
-if include_models and include_artifacts:
-    artifacts_by_model, labelsurl_by_model = await gather_models(models)  # fixes B2
-
-# 4. Assemble response purely in memory, then sort.
+project = await ProjectDetailsProcessor(project_id, config).load(include_models)
+payload = json.dumps(project)
+etag = sha256(payload.encode()).hexdigest()
 ```
 
-Key point: every metadata **type** is read at most once per partition (a handful of
-`load_all_from_partition` calls), plus one bounded-parallel batch for the per-key
-`VALIDATION`/artifact reads. Total round-trips become a small constant + at most two
-bounded-concurrency fan-outs, independent of `L × M` sequencing.
+`ProjectDetailsProcessor` loads the project first as the `404` gate, loads layers,
+labels, and models concurrently, then uses keyed `load_map` calls for validation and
+artifacts. Keyed maps preserve the old storage-key joins even when legacy document
+bodies omit optional `imageLayerId` or `modelId` fields.
 
 ## Internal Interfaces (hastegeo)
 
 | Module | Change | Signature | Purpose |
 |---|---|---|---|
-| `core/processors/metadata.py` | **new** | `load_filtered(self, predicate: dict) -> list` | Filter in the data layer, not the caller (fixes H3, B1, Q2). Backed by prefix scan + property match. |
-| `core/processors/metadata.py` | **new** | `load_map(self, keys: list[str], max_workers=8) -> dict` | Parallel multi-key load with bounded `ThreadPoolExecutor` (powers B2/B3 batches). |
-| `core/data_layer/azure_blob_storage_data_layer.py` | **fix** | `load_all(..., name_starts_with=None)` | Always pass a prefix; add optional metadata-only listing (fixes H1). |
-| `core/data_layer/azure_blob_storage_data_layer.py` | **fix** | parallel download loop | Bounded `ThreadPoolExecutor` in `load_all*` and artifact `fetch_artifact` (fixes H2). |
-| `core/data_layer/azure_blob_storage_data_layer.py` | **fix** | remove double `json.loads` | Single-serialize on save; delete re-parse (fixes H4). |
-| `core/blob.py` | **fix** | module-level `BlobServiceClient` singleton keyed by conn-string | Reuse client (fixes H5). |
-| `core/artifact_storage/azure_blob_artifact_storage.py` | **fix** | parallel `fetch_artifact` | Same `ThreadPoolExecutor` pattern (fixes H2). |
-
-`load_map` sketch:
-
-```python
-def load_map(self, keys, max_workers=8):
-    from concurrent.futures import ThreadPoolExecutor
-    out = {}
-    with ThreadPoolExecutor(max_workers=max_workers) as ex:
-        futs = {ex.submit(self._safe_load, k): k for k in keys}
-        for fut in futs:
-            out[futs[fut]] = fut.result()   # None on FileNotFoundError
-    return out
-```
-
-The API layer calls `await asyncio.to_thread(processor.load_map, keys)` so one thread
-hop wraps the whole bounded-parallel batch instead of one hop per key.
+| `core/processors/project_details.py` | **new** | `ProjectDetailsProcessor.load(include_models)` | Own storage orchestration and pure response assembly. |
+| `core/processors/metadata.py` | **new** | `load_map(keys, max_workers=None)` | Prefer one native Cosmos/PostgreSQL query; otherwise use bounded keyed loads. |
+| `core/processors/metadata.py` | **new** | `load_filtered(predicate)` | Explicit client-side partition scan and property filter; not a server-side optimization. |
+| `core/utils/parallel.py` | **new** | `parallel_map(...)` | One process-wide worker budget shared by requests and storage operations. |
+| `core/utils/async_cache.py` | **new** | `AsyncTTLCache` | Bounded TTL cache with per-key single-flight loading. |
+| `core/utils/blob.py` | **fix** | `get_blob_service_client(...)` | Reuse top-level clients and connection pools by credential target. |
+| Blob/local/Data Lake/Cosmos layers | **fix** | exact metadata matching | Longest known type wins, so `model` cannot consume `model_catalog`. |
+| Blob artifact storage | **fix** | atomic bounded download | Validate paths and replace temporary files only after successful download. |
+
+## Caching
+
+The cache key is `(projectId, includeModels)`. It stores only successful serialized
+responses, shares an in-flight load among concurrent callers, and evicts least-recently
+used entries beyond the configured bound. It is process-local and therefore does not
+provide cross-instance coherence. After the TTL, active-job polling refreshes storage.
 
 ## Queue design (Phase 3)
 
@@ -118,15 +101,12 @@ hop wraps the whole bounded-parallel batch instead of one hop per key.
 > latency and stop request pile-up. Memoization/virtualization (U3/U5) help poll-time
 > re-render churn, not first paint; `summary` payload mode (B6) is a minor win.
 
-- **Single-flight + cancellation (U7):** guard `fetchProjectDetails` so only one request
-  is in flight at a time — track the in-flight promise and reuse it, and use an
-  `AbortController` to cancel a superseded request on re-fetch/unmount. Removes the
-  duplicate concurrent load call (measured ~2× latency amplification).
-- **Smart poll (U1, U5):** send `If-None-Match` with the last `ETag`; on `304`, do
-  nothing. Otherwise shallow-compare (or hash-compare) before `setComponentState`.
-  **Do not start a poll while a request is in flight** (the measured response can exceed
-  the 20 s interval); pause polling when the tab is hidden (`document.visibilityState`)
-  and consider an interval that backs off toward the observed response time.
+- **Single-flight + cancellation (U7, implemented):** one keyed in-flight promise is
+  shared; a different project or real unmount aborts superseded work.
+- **Smart poll (U1, implemented):** send `If-None-Match`; return before state updates on
+  `304`; do not poll while hidden, in flight, or when all known jobs are terminal.
+- **Route assets (implemented):** route modules use `React.lazy`; Azure Maps control,
+  drawing, and swipe assets load in order only before a map-dependent route mounts.
 - **Split context (U2):** move volatile fields (`isLoading`, `dialogParams`,
   `bootstrapBreakpoint`) into a separate context/provider so a loading toggle doesn't
   re-render layer rows.
@@ -142,24 +122,32 @@ hop wraps the whole bounded-parallel batch instead of one hop per key.
 
 | Config Key | Type | Default | Where | Description |
 |---|---|---|---|---|
-| `HASTE_METADATA_LOAD_WORKERS` | int | 8 | App Settings / `local.settings.json` | Bounded parallelism for `load_map` fan-out |
-| `HASTE_PROJECTDETAILS_CACHE_SECONDS` | int | 15 | App Settings | `Cache-Control: max-age` for `GetProjectDetails` |
+| `HASTE_BLOB_DOWNLOAD_WORKERS` | int (1–64) | 16 | App Settings | Process-wide blocking I/O thread budget. |
+| `HASTE_METADATA_LOAD_WORKERS` | int (1–64) | 8 | App Settings | Per-map limit within the global I/O budget. |
+| `HASTE_ARTIFACT_DOWNLOAD_WORKERS` | int (1–64) | 8 | App Settings | Per-artifact limit within the global budget. |
+| `HASTE_PROJECTDETAILS_CACHE_SECONDS` | int (0–300) | 15 | App Settings | Process-local freshness and response `max-age`. |
+| `HASTE_PROJECTDETAILS_CACHE_ENTRIES` | int (1–512) | 64 | App Settings | Maximum process-local project response entries. |
 | queue `batchSize` | int | 1 → TBD | `host.json` | Concurrent messages per instance (load-test gated) |
 
 ## Observability
 
-- Log per-request storage round-trip **count** and total storage time in
-  `GetProjectDetails` (proves the O(N)→O(1) win and guards against regression).
+- Log logical data-layer call count/time and cache hit/miss in `GetProjectDetails`.
+  These metrics do not count Azure REST transactions; use Azure Storage metrics or SDK
+  pipeline instrumentation for transaction/cost analysis.
 - Emit request duration to App Insights; add a synthetic 50-layer project to the perf
   test to track p95 over time.
 - Track queue depth and dequeue/poison counts when changing `host.json`.
 
 ## Open Questions
 
-- [ ] Does the storage backend (Blob vs Cosmos, via `UnifiedDataLayer`) support a
-      server-side predicate for `load_filtered`, or is prefix-scan + in-layer filter
-      the best available? Determines how much H3 actually saves on the Blob path.
+- [x] `load_filtered` is an explicit partition-scan fallback and does not reduce
+  transferred records. A future backend query API is required for server filtering.
 - [ ] Is the double-serialization (H4) safe to fix in place, or are there existing
       blobs already double-encoded that need a migration/back-compat read path?
+      *(Partially answered 2026-08-20: legacy double-encoded blobs must be assumed, so
+      Phase 2 shipped a tolerant read (`_read_blob_content` parses twice when the first
+      parse yields a string) and deferred the save-side fix behind a migration. New
+      constraint from the data-publishing merge: `save()` now stamps `_index_metadata`
+      blob metadata, which any save-path rewrite must preserve.)*
 - [ ] Acceptable default polling interval / should we move to server-push (SignalR)
       instead of polling for run-status updates?

spec/features/perf-layer-loading/findings.md

@@ -1,5 +1,23 @@
 # Findings: Verified Performance Bottlenecks
 
+## Contents
+
+- [Cost Model](#cost-model-why-it-scales)
+- [HTTP API](#backend--apihastefuncapifunction_apppy)
+- [Core Library](#backend--hastelibsrchastegeocore)
+- [Queue Workers](#queue--apihastefuncqueuesfunction_apppy)
+- [UI](#ui--uisrc)
+- [Measured Priority](#measured-priority-adjustments-phase-0)
+
+> **Baseline terminology:** the Phase 0 counter records logical data-layer calls, not
+> Azure REST transactions. The N+1 findings and latency measurements remain valid, but
+> one partition call can contain a listing plus many Blob downloads.
+
+> **Implementation status (2026-09-01):** B1–B5, B7, H2, and H5 are addressed on the
+> cumulative branch. B6, server-side filtering, a materialized project view, and queue
+> configuration remain open. UI single-flight, ETag handling, in-flight/visibility
+> guards, and active-job-only polling are implemented.
+
 Each finding was confirmed by reading the referenced code. Severity reflects impact
 on the reported symptom (slow layer/run loading that scales with layer count).
 
@@ -13,7 +31,7 @@ For a project with **L** image layers and an average of **M** models per layer,
 + L × 1                (LABELS full-partition scan — once PER layer, see B1)
 + L × 1                (VALIDATION load per layer)
 + L × M × 2            (MODEL_ARTIFACTS load + TRAIN_LABELS export per model)
-= 3 + 2L + 2LM  sequential, blocking storage round-trips
+= 3 + 2L + 2LM  sequential, blocking logical data-layer calls
 ```
 
 For L=50, M=5 that is **~603 sequential round-trips**, each an `await asyncio.to_thread(...)`