Re-pin benchmark shardmap fixtures under the authalic convention (mortie 0.9.8)

[espg]

Closes #438. Refs #432 (this PR unblocks repo CI), refs #392, refs #148, refs #202, refs #342.

What this does

mortie 0.9.8 makes latitude="authalic" the default (espg/mortie#186, landed in espg/mortie#188), so every morton word zagg derives from a lat/lon moves. Three classes of git-stored artifact were pinned under the old geodetic-as-spherical convention and now disagree with a live rebuild:

1. the five mortie-backed benchmark shardmap fixtures + their targets.json pins,
2. the kitchen_sink spec-conformance fixture's located point words + the O11 content hashes computed over them (this is what turns PR GEDI waveforms 2/3: generic vlen reader primitives, paired-asset shardmap, flux transform + gedi01b template #432's merge-commit CI red — test_content_hash.py::TestWorkerWiring::test_matches_pinned_fixture_hashes and TestStreamingLeafStaging::test_unsharded_leaf_reproduces_pinned_hashes_from_staged_values — and every CI run repo-wide stays red until it is re-pinned),
3. the mortie floor itself, which still allowed a pre-authalic resolve.

Approach: rebuild each pin through exactly the recipe its own drift guard uses, so the re-pin is reproducible by the guard rather than by a bespoke script.

• Shardmaps: the recipe in tests/test_benchmark_shardmap.py::test_pinned_shardmap_no_drift — manifest config resolution, per-entry AOI/temporal/CMR override with top-level fallback, backend from the committed pin's metadata.backend (all five are mortie), catalog_parquet offline snapshot for the 88S pair and a live CMR fetch for the NEON trio, bench_metrics.select_densest_shard for the pin, and the nested_in extraction (densest o10 shard inside the newly pinned o9 88S shard) for sm_healpix_o10_88s. The one-off driver lives in scratch and is not committed — the recipe is the test's, and the README already documents the CLI route (python -m zagg.catalog + select_densest_shard).
• Spec fixture: the documented route, uv run python tools/generate_spec_fixtures.py --only kitchen_sink (scoped with --only so the other fixtures' stale-by-design manifests are not churned).

Phases

• Phase 1 — re-pin the five mortie shardmap fixtures + targets.json (9d609ff3)
• Phase 2 — re-pin the kitchen_sink spec fixture + its content hashes (58eda825)
• Phase 3 — bump the mortie floor to >=0.9.8 (e82567d8)

Phase 1 — benchmark shardmaps: before / after

Densest-shard pins in tests/data/benchmark/targets.json:

pin	shard_key (pre-authalic → authalic)	n_granules	shards in map	granule membership
healpix_o9	5347395636851376137 → 5347294481781620745	66 → 66	4 → 4	identical (66 CMR granules)
healpix_o10	5347405532456026122 → 5347392338316492810	56 → 55	9 → 9	identical (66)
healpix_o11	5347398660508352523 → 5347392338316492811	50 → 50	22 → 22	identical (66)
healpix_o9_88s	11530494877603201033 → 11530143033882312713	5620 → 5536	576 → 564 (full ring)	same cat_88s.parquet snapshot, 35,639 catalog granules unchanged
healpix_o10_88s (nested)	11530494877603201034 → 11530145232905568266	4605 → 4991	1756 → 1692 (full ring)	same snapshot, 35,639 unchanged

Structural sanity — every non-morton field held across all five: grid_signature byte-identical, total_granules, backend, footprint, mortie_order, source, bbox, start_date/end_date unchanged; the NEON maps keep their exact shard counts (4 / 9 / 22) and their full 66-granule sets with zero granules added or removed; the 88S maps stay pruned to one shard with the same metadata.pruned marker. Of the morton-derived quantities, only total_pairs (NEON: 205→218, 361→360, 737→741) and the ring shard/pair counts moved — cell-boundary attribution, which is precisely what the convention change does.

The rebuild also refreshes the granule-record schema — disclosed here rather than reverted, since these maps are rebuilt through the drift test's own recipe and that recipe emits current-zagg.catalog output. Independent of the convention, every rebuilt map picks up:

• time_start / time_end on each granule record (the issue morton-hive/2 temporal machinery: windowed leaves, manifest temporal block, stamp time ranges, coverage full #246 fields; the committed maps predated them and carried {id, s3, https} only),
• a granules_assigned key in metadata (66 / 66 / 66 / 35639 / 35639),
• a fresh build_wall_s.

That, not the convention, is most of the byte growth: JSON size old→new is o9 0.07→0.10 MB, o10 0.13→0.17, o11 0.26→0.35, o9_88s 1.97→2.57, o10_88s 1.61→2.32 (~+1.3 MB total).

Consequence, verified rather than assumed: the five healpix maps were "legacy shardmaps" for the windowed fan-out (src/zagg/runner.py::_granule_time_span / _windowed_units, which raises "this shardmap predates it (no time_start/time_end on its granule records)") and after this commit they are not, while sm_rect_{3km,6km}.json stay on the old record shape — the committed fixture set is now split across two record schemas. The rect pair is not refreshed here because their spherely backend (the non-PyPI exact-S2 fork) is unavailable in this environment; they refresh whenever they are next re-pinned somewhere that has it. tests/data/benchmark/README.md's shard-map step now records the split and why.

No behavioral difference results. Driving _windowed_units (monthly schedule, no bounds.temporal) over each committed map directly: the five healpix maps now enumerate real window labels from the granule spans (o9 32 units / o10 34 / o11 37 over the first two shards; the pruned 88S pair 1 each), while sm_rect_{3km,6km} take the unchanged legacy branch and raise the pointed remedy. Nothing regresses — the healpix maps strictly gain a capability. No test asserts the old record shape or exercises a windowed read of these fixtures: tests/test_hive_windows.py::TestWindowedUnits builds its own synthetic timed records, and the only fixture consumer, tests/test_notebook.py (sm_healpix_o9.json), is non-windowed except for two raster cases that monkeypatch _windowed_units out. So the schema flip is uncovered by tests in both directions; flagged here rather than papered over with new machinery.

Two more things worth reading closely:

• o9 NEON is a tie, not a move. The pre-authalic pin 5347395636851376137 is still in the rebuilt map at 66 granules; 5347294481781620745 is also at 66, and select_densest_shard's lowest-key tiebreak picks the latter. The dispatched shard changes, the density does not.
• The o10 88S pin gains 8.4% granules (4605 → 4991). It is a different physical cell (the densest o10 inside the new o9 parent), and the ring now fans to 1692 o10 shards instead of 1756 — average granules per shard rises 2816 → 3050, i.e. +8.3%, which accounts for the pin's +8.4% almost exactly. The global o10 densest is 4,992 at 11530126541207896074 (another ring longitude), so the nested pin remains the runner-up by one granule, exactly the relationship the add 88 south stress test shard + benchmark #148 note describes. The catalog itself is byte-identical (same committed parquet, 35,639 granules), so nothing but attribution moved.

Also updated: three literal copies of the o9 pin in tests/test_benchmark.py (test_run_target_dry_run, the object-measure assertion, the hive/flat parity assertion). The fourth occurrence (line ~1224) is a synthetic "any valid packed morton word" for a shard map that carries no shard_key, not the pin, so it is left alone.

Phase 2 — spec fixture + content hashes

Regenerating kitchen_sink under 0.9.8 changes only the located point words. The generator jitters synthetic photons ±1e-5° around each cell center (tools/generate_spec_fixtures.py::_point_words: cell_centers → jitter → MortonIndexArray.from_latlon), so the order-29 point kind words shift while everything derived from cell arithmetic does not:

array	hash
6/morton	unchanged f6282635e373…
6/count	unchanged 29458cad5550…
6/composition	unchanged 04886a9dfb60…
6/h_tdigest_signal	unchanged 1de09e1b8058…
6/h_tdigest_noise	unchanged 201816825f7f…
6/h_tdigest_signal_locations	ffd8bcc3be76… → 11e0a598b7f6…
6/h_tdigest_noise_locations	05102b042df0… → c4f7ad04676c…
combined	57c413489b33… → 7ee8ad9278ea…

The fixture's decoded cells records show the same containment: only the h_tdigest_*_locations lists differ, and only in their low-order bits (e.g. 1234197404132064308 → 1234197404132064305). Cell mortons, counts, composition words and digest payloads are byte-identical.

Two frozen literals in tests/test_spec_conformance.py — the anti-circularity pins FROZEN_COMBINED["kitchen_sink"] and FROZEN_ARRAYS[("kitchen_sink", "6/h_tdigest_signal_locations")] — move with the fixture. minimal, column and flux are untouched: their leaves carry no located sibling, so their pins (including the minimal == column cross-fixture pin) still hold as literals.

Store-object churn in the regenerated fixture is exactly the two *_locations/c/0 chunk objects plus timestamps and the recorded zagg_version (written_at / generated_at in the leaf zarr.json, all.pyramid.zarr/zarr.json, all.pyramid.stats.json, morton_hive.json). No manifest or schema shape changed.

Phase 3 — mortie floor

mortie>=0.9.6 → mortie>=0.9.8, with the convention rationale as the comment block opener and the earlier-floor history preserved below it (the dangling "and the only reason for this bump" phrase, which belonged to the 0.9.6 bump, is removed). deployment/aws/build_layer.sh derives MORTIE_SPEC from pyproject.toml via tomllib (issue #322), so the layer floor follows automatically — no deploy-script edit here. tests/test_mortie_versions.py reads the floor dynamically: 12 passed.

How it was tested

• ZAGG_BENCHMARK_DRIFT=1 pytest tests/test_benchmark_shardmap.py -v -m slow → 5 passed (145 s). All five pins rebuild to their new values, live CMR for the NEON trio and offline parquet for the 88S pair.
• pytest tests/test_benchmark.py tests/test_benchmark_objects.py tests/test_deploy_lambda.py (every test reading targets.json shard keys, per grep -rl targets.json tests/) → 142 passed, 15 skipped.
• pytest tests/test_content_hash.py tests/test_spec_conformance.py → 140 passed, 2 skipped (the 2 skips are the optional moczarr cross-checks; moczarr is not installed here).
• pytest tests/test_mortie_versions.py → 12 passed.
• Full suite uv run --extra test pytest -q → 3863 passed, 38 skipped, 1 failed: only tests/test_lambda_build.py::TestFunctionBuild::test_function_build_succeeds, the known environmental Lambda-build test. Nothing else fails on morton words.
• ruff check / ruff format --check: two pre-existing findings in files this PR does not touch — N818 on src/zagg/registry.py:64 (UnknownCapability) and a reformat of a python code fence in tests/data/benchmark/README.md. Left alone per the "don't fix unrelated pre-existing failures" rule; flagging rather than silencing.

Questions for review

1. Spec fixtures were scoped OUT of issue Re-pin benchmark shardmap fixtures under the authalic convention (mortie 0.9.8) #438 ("regenerate with the zagg-side authalic spec-marker PR, refs declare the multiresolution ladder in machine-readable convention metadata (zarr multiscales + DGGS refinement) #392") but they are the CI-red artifact, so phase 2 regenerates kitchen_sink here. declare the multiresolution ladder in machine-readable convention metadata (zarr multiscales + DGGS refinement) #392 will still need to add the latitude-convention marker to the fixtures — this PR moves the values, not the grammar, so that work is unaffected except that it will regenerate on top of these bytes. Say the word if you'd rather this phase move to declare the multiresolution ladder in machine-readable convention metadata (zarr multiscales + DGGS refinement) #392 and CI stay red until then.
2. moczarr vendors these fixtures for its parity gates (Adopt zagg ragged decoding: generic leaf-decode layer + HHDC tensor profile (not the only contract) espg/moczarr#19/Backend support (i.e., local processing support) #20). The two changed *_locations hashes and the new combined are a cross-implementation change: moczarr's vendored copy needs the same refresh, or its parity gate goes red against a zagg store built on 0.9.8. Not opened as an issue there — that's an @espg call.
3. The o10 88S pin's +8.4% granules is explained (different cell, coarser effective ring fan-out — see above) but it does make that stress probe heavier than the number the add 88 south stress test shard + benchmark #148 notes were written against. If the intent was a like-for-like stress target across the convention change, the alternative is pinning the o10 shard nearest the old granule count instead of the nested densest — that would depart from the nested_in rule the drift test encodes, so I kept the rule.
4. The o9 NEON pin flips on a tie (two shards at 66; lowest key wins). The dispatched shard moves even though nothing about density did. Fine by the deterministic tiebreak, but it means the benchmark series' shard changes identity at this commit — worth a note in the series if you track shard identity across merges.
5. The un-committed driver. The regeneration ran from a scratch driver that mirrors test_pinned_shardmap_no_drift. It is not committed (no obvious home; the README documents the CLI route). If you'd rather have a tools/-side re-pin script beside the manifest, say so and I'll add one with a "re-pins deliberately" docstring.

[espg]

🤖 from Claude (review)

The ~21 km at 45° lat figure is wrong — mortie's own spec puts the authalic offset at 0.12830°, i.e. ~14.26 km.

espg/mortie docs/specification.md §9 (the section this comment cites):

conventions peaks near 45° latitude at `|beta - phi| ~= 0.12830°`
(~14.26 km along the meridian).

~21 km is the geodetic↔conformal peak (0.1924°), a different auxiliary latitude — authalic is beta - phi ≈ -(e²/3 + 31e⁴/180 + …) sin 2φ → 0.002239 rad → 0.1283° → 14.3 km on a 6371 km sphere. Everything else in the block checks out: latitude="authalic" is indeed the 0.9.8 default (mortie/convert.py:329 etc. in the installed 0.9.8), and the store-attr tokens authalic-wgs84 / geodetic-spherical are §9/§5 verbatim.

This block is the repo's only prose statement of the convention's magnitude and #392 will write the spec marker against it, so the number is worth fixing to ~14 km (0.12830°).

[espg]

🤖 from Claude

Fixed in ada43dbb. The comment now reads:

    # two conventions — the offset |beta - phi| peaks near 45° lat at
    # 0.12830°, ~14.26 km along the meridian (mortie spec §9) — so stores

Straight from espg/mortie docs/specification.md §9 (lines 697-698): conventions peaks near 45° latitude at |beta - phi| ~= 0.12830° (~14.26 km along the meridian). The trailing store-attr sentence now says "(same section)" rather than repeating the §9 cite.

Grepped the branch for any other copy of the 21 km / 0.1924° figure (grep -rn '21 km|21km' --include={*.md,*.toml,*.py,*.yaml,*.yml}): pyproject.toml:42 was the only one, so nothing else to fix.

[espg]

🤖 from Claude (review)

tests/data/benchmark/README.md still carries the pre-authalic ring numbers this note just moved — the two files sit one directory apart and now contradict each other.

README.md:215-219:

- **Pins.** o9: the ring's densest shard by granule count (5,620 granules over
  576 ring shards / 2.04M pairs / 35,639 catalog granules — ~95× NEON o9's 59).
  o10: the densest o10 shard **nested inside** the pinned o9 shard
  (`nested_in` in `targets.json`; 4,605 granules vs the global o10 max of
  4,642 at another longitude), ...

Rebuilding both maps offline from the same committed catalogs/cat_88s.parquet under 0.9.8 (the drift test's own recipe) reproduces this PR's pins exactly and gives the new figures:

[healpix_o9_88s]  rebuilt shards=564  pairs=2065142  global densest 11530143033882312713/5536  == pin
[healpix_o10_88s] rebuilt shards=1692 pairs=5160991  global densest 11530126541207896074/4992
                  nested densest 11530145232905568266/4991 == pin (parent == o9 pin, verified)

So the README's 5,620 / 576 ring shards / 2.04M pairs / 4,605 / 4,642 should read 5,536 / 564 / 2.07M / 4,991 / 4,992, and README.md:253's "10 cells instead of ~576 at o9" is now ~564. (The "~95× NEON o9's 59" on line 216 was already stale from the #202 re-pin — NEON o9 densest is 66 — so that one is pre-existing.)

[espg]

🤖 from Claude

Folded in eb496a9c. README.md:215-216 now reads 5,536 granules over 564 ring shards / 2.07M pairs, :218-219 reads 4,991 granules vs the global o10 max of 4,992, and :253 (now :266) reads 10 cells instead of ~564 at o9.

Cross-checked every figure against the committed artifacts rather than taking them on faith — metadata of sm_healpix_o9_88s.json gives total_shards: 564, total_pairs: 2065142; sm_healpix_o10_88s.json's metadata.pruned string carries the 4,991 nested / 4,992 global pair; targets.json pins n_granules: 5536 / 4991.

On the pre-existing ~95× NEON o9's 59: I did fix it, to ~84× NEON o9's 66, because it lives inside the same parenthetical whose granule count this commit changes — leaving it would have made one sentence self-inconsistent. Verified first: targets.json healpix_o9 pins n_granules: 66 (unchanged by this PR — the NEON maps rebuild byte-identical), and 5536/66 = 83.9.

grep -rn '5,620|5620|4,605|4,642|2.04M|576' README.md now returns nothing.

[espg]

🤖 from Claude (review)

The rebuilt fixtures also carry a granule-record schema refresh, which the PR body's "only total_pairs and the ring shard/pair counts moved" denies.

Diffing every map old vs new:

• every granule record in all five maps gains time_start / time_end (the issue morton-hive/2 temporal machinery: windowed leaves, manifest temporal block, stamp time ranges, coverage full #246 fields; old records were {id, s3, https} only),
• every map's metadata gains granules_assigned (66 / 66 / 66 / 35639 / 35639),
• build_wall_s changes.

That is most of the byte growth, not the convention: JSON size old→new is o9 0.07→0.10 MB, o10 0.13→0.17, o11 0.26→0.35, o9_88s 1.97→2.57, o10_88s 1.61→2.32 — ~+1.3 MB.

Behavioral consequence worth recording in the body rather than being discovered later: these five were "legacy shardmaps" for the windowing fan-out (src/zagg/runner.py:_granule_time_span / _windowed_units, which raises "this shardmap predates it (no time_start/time_end on its granule records)") and after this commit they are not, while sm_rect_{3km,6km}.json stay on the old record shape — the committed fixture set is now split across the two schemas.

The pins themselves verify clean: all five rebuild key-for-key (NEON trio live from CMR, map byte-identical to committed including shard membership; 88S pair offline from the committed parquet), and select_densest_shard over each committed map reproduces its targets.json shard_key/n_granules.

[espg]

🤖 from Claude

Folded as disclosure, not regeneration — cbd128f7 (README) plus a PR-body rewrite.

(a) PR body. The "only total_pairs and the ring counts moved" sentence is now scoped to morton-derived quantities, followed by an explicit block listing the schema refresh (time_start/time_end per record, granules_assigned in metadata, fresh build_wall_s), your byte-growth table (0.07→0.10 / 0.13→0.17 / 0.26→0.35 / 1.97→2.57 / 1.61→2.32 MB, ~+1.3 MB), and the split-fixture-set consequence naming runner._granule_time_span/_windowed_units and the two rect maps left on the old shape.

(b) No behavioral difference. Drove _windowed_units (monthly schedule, bounds_temporal=None) over each committed map directly:

sm_healpix_o9      no-span=0/60  -> 32 units, labels ['201901','201902','201904', ...]
sm_healpix_o10     no-span=0/60  -> 34 units
sm_healpix_o11     no-span=0/60  -> 37 units
sm_healpix_o9_88s  no-span=0/20  -> 1 unit  ['201810']   (pruned to one shard)
sm_healpix_o10_88s no-span=0/20  -> 1 unit  ['201810']
sm_rect_3km        no-span=60/60 -> ValueError: "...this shardmap predates it..."
sm_rect_6km        no-span=60/60 -> ValueError: "...this shardmap predates it..."

So the healpix five strictly gain the windowed path and the rect pair takes the unchanged legacy branch — no regression in either direction.

To your implicit question of whether that flip is covered: it is not, and I did not add machinery for it. tests/test_hive_windows.py::TestWindowedUnits is the only real exercise of _windowed_units, and it builds synthetic records via its own _timed_rec helper (tests/test_hive_windows.py:1208), never the fixtures. The only fixture consumer is tests/test_notebook.py (SHARDMAP = .../sm_healpix_o9.json, line 19), and its two windowed cases are raster ones that monkeypatch.setattr(runner, "_windowed_units", ...) to fail if the spatial path runs (lines 104-128). Nothing asserts the granule-record shape either. Flagging rather than backfilling — per the instruction not to add machinery here.

Suite after the fold: pytest tests/test_benchmark_shardmap.py tests/test_benchmark.py tests/test_mortie_versions.py tests/test_notebook.py tests/test_hive_windows.py -q → 251 passed, 20 skipped.

(c) README. New blockquote in the shard-map step of "Add a target" (tests/data/benchmark/README.md:182-193) states both schemas, which maps hold which, what each implies for windowing, and that the rect pair is unrefreshed because spherely is a non-PyPI fork unavailable here — so it refreshes whenever those two are next re-pinned in an environment that has it.

One unrelated note while in this file: ruff format --check still wants to squeeze the double space before # -> (shard_key, n_granules) in the select_densest_shard fence. That is pre-existing — git show main:tests/data/benchmark/README.md fails the same check at the same line — and outside my edits, so left alone per the PR body's existing flag.