Skip to content

demo: 07_minimal on the clean moczarr/mortie Moc/Toc API (issue #328) - #492

Closed
espg wants to merge 1 commit into
mainfrom
claude/328-demo-07-minimal
Closed

demo: 07_minimal on the clean moczarr/mortie Moc/Toc API (issue #328)#492
espg wants to merge 1 commit into
mainfrom
claude/328-demo-07-minimal

Conversation

@espg

@espg espg commented Aug 19, 2026

Copy link
Copy Markdown
Member

Refs #328.

Rewrites demo/07_minimal.ipynb against the clean reader API that landed today: mortie 0.9.10's Moc/Toc (espg/mortie#197, #199) and moczarr 0.5.0's coverage_moc / coverage_toc / candidate_leaves(..., when=) (espg/moczarr#48).

Landed as demo/07_minimal_clean.ipynb, not in place. demo/07_minimal.ipynb is untracked in the working tree and demo/07_minimal-Copy1.ipynb (a byte-identical Jupyter duplicate) was written at 08:39 this morning, minutes before this run started — the notebook looked live in a Jupyter session, and a notebook open in a kernel owns its file. Reconcile and rename at your convenience; the old file is untouched.

What the coverage step collapses to

The old coverage() helper hand-rolled load_root_coverage + root_coverage_and + candidate_leaves + a manual shard-id set intersection. It is now four lines:

cover = mz.coverage_moc(mz.load_root_coverage(sroot, store=sstore))
assert cover.contains(q), f"polygon leaves the {name} store's coverage"
leaves = mz.candidate_leaves(
    sroot, mz.read_manifest(sroot, store=sstore), aoi=q, when=when, store=sstore
)

with q = moc(aoi) built once above it. Moc and Toc ride the __morton_moc__() / __toc_words__() protocols straight into moczarr — no adapter, no words-shuffling. The 3-D view also lost ~25 lines of hand-rolled morton bit decoding to mortie.mort2geo + mortie.clip2order, and the 1850-epoch time arithmetic to mortie.to_datetime64.

Two regions, not one store plus a parked constant

The GEDI half was verified rather than assumed, and what is actually on sliderule-public changed the design:

region stores anonymous binder cryocloud
california (default) …/englacial/zagg/demo/atl03_tdigest_o9.zarr 19/h_tdigest_signal yes
serc sliderule-public/zagg-demo/serc_tdigest_strata.zarr 19/h_tdigest_signal + …/serc_gedi_flux.zarr 18/rx_flux no (403)

serc_gedi_flux.zarr and serc_tdigest_strata.zarr both exist, both cover the same four o9 shards (4331422233, 4331422234, 4331422411, 4331422412), and all four leaves are present in both — so the paired two-sensor view is real today, in Maryland, behind credentials. They are a different geography from the California store, so each region carries its own polygon and the notebook says that plainly rather than implying a CA pair that does not exist. REGION is the whole switch.

Phases

  • Coverage step on the typed APIcoverage_moc / moc() / contains / candidate_leaves, generalized over the region's sensors so one and two sensors read the same.
  • Temporal half with the None arm handled explicitlycoverage_toc returns Toc | None; None is printed as "publishes no temporal coverage, when= prunes nothing" with the §10 unlisted-is-unknown reason, never an AttributeError.
  • Region model + credential gateopen_region() probes reachability and, on failure, prints what is wrong and how to fix it, then raises a one-line RuntimeError instead of an obstore stack.
  • 3-D view and tensor export on mortie's decodersmort2geo / clip2order / to_datetime64; export uses subtree= so it fetches only the tile on screen.

How tested

Metadata-only against the live stores (NO EGRESS: coverage sidecars, manifests, zarr.json — kilobytes; no payload arrays, no shard loops):

  • california, live and anonymous: Moc(326 cells, orders 5-9, finest 12.734 km), 2,726 o9 shards, Yosemite box → 4 shards, coverage_tocNone.
  • serc, live and signed: both stores Moc(4 cells, order 9), SERC box → 4 shards each, intersection 4, coverage_tocNone on both.
  • serc without credentials: prints the explanation, raises RuntimeError: region 'serc' unreachable from this kernel. Confirms the negative that shapes the design — unsigned ListObjectsV2 on sliderule-public is AccessDenied, while the same call on us-west-2.opendata.source.coop succeeds.
  • load(), local_xy() and export() were run to completion offline against moczarr's in-tree fixtures (tests/data/strata_hive, located+untimed; tests/data/spec/temporal, located+timed), exercising both companion-channel branches and producing a real (2, 2, 16) tensor. That covers every line of the read path except the live bulk read itself.

ruff check / ruff format --check clean on the notebook; nbformat.validate passes.

The notebook ships unexecuted, deliberately. Its remaining cells need either a bulk read (egress from a laptop) or cryocloud credentials, and fabricating outputs for them would be worse than leaving them empty. Executing and saving is an operator run.

Questions for review

  1. §4 binder contract vs the paired view — flagged, not resolved. §4 requires notebooks to be binder-runnable on anonymously-reachable data. california satisfies it end to end. serc cannot: sliderule-public refuses unsigned reads, so the only existing paired store set is cryocloud-only. The notebook handles this by defaulting to the compliant region, documenting the split in a table up front, and failing the other region with an explanation rather than a traceback — but that is a mitigation, not compliance. The clean fix is a California GEDI store on source.coop (the ~$50 fleet run), which would make one region both paired and binder-clean and let serc be deleted. Want me to file that as its own issue?

  2. The export cell needs moczarr[zagg], so "no zagg" is not quite true. moczarr.hhdc.read_tensors imports zagg's t-digest algebra rather than vendoring it (moczarr's own zagg extra, espg/moczarr#19), so the last cell raises ImportError: HHDC rasterization needs zagg's t-digest algebra on a plain pip install moczarr. The install cell now says: base line for everything through the 3-D view, %pip install "moczarr[zagg]" for the final cell only. Alternative is dropping the export section to keep the notebook strictly reader-only — say the word.

  3. contains vs intersects. The ruled snippet asserts cover.contains(q), which is what shipped. It is False for a polygon that pokes over the coverage edge even though the overlap is real and readable, so an edge-of-California AOI raises rather than returning the partial roster. The prose points at intersects for that case. Fine as the headline, or should the demo model the partial-overlap path instead?

  4. API friction worth passing upstream (no action needed here):

    • candidate_leaves takes store= but no **store_kwargs, unlike read_manifest / load_root_coverage / open_leaf. An anonymous store therefore cannot use the bare ruled spelling — it needs an explicit open_object_store first.
    • open_leaf(root, shard, store=st) shares the handle only for the manifest GET; the leaf store is always a fresh open, so anonymous=True has to be passed again alongside store=. Easy to get wrong silently on a public bucket (the first draft of this notebook did).
    • mortie.clip2order(clip_order, words) takes its parameter before its data, the reverse of every neighbouring kernel.
    • No store anywhere publishes a §10 root temporal section yet, so coverage_toc is None on all three verified stores and the Toc branch of the demo has never run against real data. Worth a fixture-backed producer before the temporal story is demoed as working.

@espg espg added the implement label Aug 19, 2026
@espg

espg commented Aug 19, 2026

Copy link
Copy Markdown
Member Author

🤖 from Claude

CI is red for a pre-existing reason, unrelated to this PR (which touches one notebook and no src/ or tests/ file). Flagging rather than fixing, per §4's don't-fix-unrelated rule — it needs its own commit, and it blocks every open zagg PR, not just this one.

test (3.12) and test (3.13) both fail at collection, 4532 tests collected / 1 error:

ERROR collecting tests/test_time_axis.py
E   ModuleNotFoundError: No module named 'mortie.toc'
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!

Cause: tests/test_time_axis.py:12 is

from mortie.toc import Q_END_NS, Q_START_NS

and mortie 0.9.10 retired the mortie.toc submodulemortie.toc is now the Toc constructor. Its back-compat shim covers attribute access only (mortie.toc.Q_START_NS still resolves, with a DeprecationWarning); statement-form from mortie.toc import ... breaks at the rename by design, which is exactly what this is. CI resolves mortie from PyPI against the floor in pyproject.toml, so it started picking up 0.9.10 today and the whole suite stopped collecting.

Fix is one line — the constants are exported at top level:

from mortie import Q_END_NS, Q_START_NS

Verified against mortie 0.9.10: mortie.Q_START_NS / mortie.Q_END_NS both exist, and the shim's own deprecation message names that spelling as the replacement. ruff and build are green on this PR; only the two test jobs are affected, and only through this import.

build ✅ · ruff ✅ · check ✅ · test (3.12) ❌ · test (3.13) ❌ — all five would be green on this branch with that line changed.

@espg

espg commented Aug 19, 2026

Copy link
Copy Markdown
Member Author

this isn't matching the minimal example / terse notebook

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant