Skip to content

Batch MOC densify: mocs_to_orders + the mortie/moc.py extraction (issue #156) - #160

Merged
espg merged 5 commits into
mainfrom
claude/156-mocs-to-orders
Aug 8, 2026
Merged

Batch MOC densify: mocs_to_orders + the mortie/moc.py extraction (issue #156)#160
espg merged 5 commits into
mainfrom
claude/156-mocs-to-orders

Conversation

@espg

@espg espg commented Aug 8, 2026

Copy link
Copy Markdown
Owner

Refs #156 (phase 1 of many — not Closes)

What

Two things that have to land together, because the first is where the second lives:

(A) mortie/moc.py — the MOC algebra split out of mortie/coverage.py. compress_moc, moc_to_order, moc_or/moc_and/moc_minus/moc_xor, moc_not, common_ancestor / moc_min and split_base_cells move verbatim (the old coverage.py:597-977), with _whole_sphere and the norm2mort import that only it used. The split axis is the domain, not the arity — so moc_to_order and mocs_to_orders sit next to each other and are read together, rather than a batch.py that would separate every twin (espg's ruling on #156, superseding the earlier batch.py approval). Rust follows the same axis: src_rust/src/moc/batch.rs alongside the existing src_rust/src/coverage/batch.rs.

The public surface is unchanged: mortie/__init__.py still re-exports every one of those names flat (mortie.moc_and, mortie.moc_to_order, …) and __all__ is identical apart from the one addition below.

(B) mocs_to_orders(values, offsets, order, max_cells=1 << 20) — the ragged batch of moc_to_order:

mocs, off = mortie.polygons_to_morton_mocs(lats, lons, off_in, order=8)
flat, flat_off = mortie.mocs_to_orders(mocs, off, 8)   # the pair, verbatim

Putting the budget in the serial pre-pass rather than the parallel one is deliberate and matches the scalar guard's whole point (#80): the estimate is an O(n) pass over the input words with no flat allocation, so an over-budget batch refuses before a single cell is materialized.

Module sizes

Both sides land comfortably under the ~1,000-line aim, no raise needed:

file lines
mortie/coverage.py 977 → 602
mortie/moc.py 541 (new)
src_rust/src/moc/batch.rs 404 (new)
src_rust/src/moc.rs 878 → 881 (one pub mod batch;)

Phases

  • Phase 1 — mortie/moc.py extraction + mocs_to_orders (Rust kernel, pyfunction, Python wrapper, tests, bench) (5d092f6), plus the phase-1 adversarial-review fold: scalar order guard (bc1aa67), order 0 accepted by the batch (2449040), max_cells domain parity (24c029d), memory-posture docstring correction (4de7665)
  • Phase 2 — the set-op family: mocs_and/mocs_or/mocs_minus/mocs_xor + the mocs_intersect predicate
  • Phase 3 — common_ancestors + children_of
  • Phase 4 — words_to_decimals + hive_paths
  • Phase 5 — moc_contains, mort2polygons, mort2bboxes, the WKB/geometry converters

Benchmarks (Apple Silicon, 10 cores)

python benchmarks/measure_mocs_to_orders.py N moc_order flat_order — the MOCs of N synthetic ~1° footprint quads (the same corpus measure_batch_coverage.py uses), scalar moc_to_order loop vs one mocs_to_orders call:

N MOC order → flat order flat cells scalar loop batch speedup
100,000 8 → 9 11.2 M 0.42 s (4.2 µs/moc) 0.14 s (1.4 µs/moc) 2.96×
100,000 8 → 10 44.7 M 2.32 s (23.2 µs/moc) 0.40 s (4.0 µs/moc) 5.80×
500,000 8 → 8 14.0 M 1.12 s (2.2 µs/moc) 0.37 s (0.7 µs/moc) 3.00×

Read this honestly: the speedup is smaller than #154's 19.9×, and it should be. to_order is a cheap kernel whose cost is dominated by writing the flat result, so the batch is bandwidth-bound on the output copy (which is serial, per chunk, by design of the memory posture) rather than compute-bound on a descent. The win grows with densify depth — 5.8× at order 10, where there is actual per-cell work to parallelize — and the per-item fixed cost still drops by ~3× (4.2 → 1.4 µs) because the scalar path pays two Python↔Rust crossings per MOC (rust_moc_to_order_count for the guard, then rust_moc_to_order).

Memory posture (maxrss delta across the call, each shape in its own process), re-measured in the review fold and now stated correctly in the docstrings — peak is input copy + result + one chunk, not result + one chunk:

shape input result peak Δ Δ / result input + result
100k MOCs, 8→10 12.9 MiB 342.1 MiB 398.3 MiB 1.16× 354.9 MiB
250k MOCs, 11→11 304.3 MiB 2097.5 MiB 2479.5 MiB 1.18× 2401.8 MiB
250k MOCs, 11→8 304.3 MiB 55.2 MiB 382.6 MiB 6.93× 359.5 MiB
250k MOCs, 11→4 (coarsen) 304.3 MiB 5.3 MiB 317.5 MiB 60× 309.5 MiB

The chunked assembly is doing its job — the model fits every row to within one chunk — but the omitted term is the full copy of the input that rust_mocs_to_orders must make (values.to_vec() / offsets.to_vec(): a &[u64] borrowed from a numpy array cannot cross py.allow_threads, which is why coverage/batch.rs's pyfunction does the same). Densifying it is noise; coarsening it is the entire peak. Consumer consequence: englacial/zagg#400's phase 3 feeds ~556k MOCs through this call, so that mandatory input copy is a real line item in that plan's memory budget rather than a rounding error — size the worker off input + result.

For the pipeline this was filed from, the relevant number is the timing table's third row: at catalog scale (~556k granules) the last mortie call in zagg's per-ring loop goes from ~2.2 µs/granule to ~0.7 µs/granule, and — more to the point — from inside a Python loop to a single call.

Testing

  • cargo test --lib: 277 passed, 1 ignored (266 before; 11 new in moc::batch::tests — per-MOC parity vs the scalar kernel, a >2×CHUNK batch across the chunk seam, empty batch, the budget refusal naming the lowest-index MOC, budget-is-per-item-not-per-batch, refusal-before-any-densify, exact-coverage offsets rejection, lowest-index offsets errors, injected-panic capture, order/layout rejection, order-0 parity with the scalar).
  • pytest -q: 1068 passed, 12 skipped after maturin develop --release (20 in mortie/tests/test_moc_batch.py, plus 2 in mortie/tests/test_coverage.py's TestMocToOrderGuard for the scalar order guard), against 1046 on main.
  • cargo fmt --check clean; cargo clippy --lib --benchesno new warnings (the 7 that fire are all pre-existing, in prefix_trie.rs, coverage/tests.rs and geo2mort.rs, none in touched code).
  • flake8 mortie benchmarks --select=E9,F63,F7,F82 clean; the --max-line-length=88 style pass and ruff check --select=E,F,W,I --ignore=E501 are clean on every file this PR touches (12 ruff findings remain repo-wide, all in files untouched here).
  • numpydoc lint mortie/moc.py mortie/coverage.py mortie/__init__.py clean.

New Python tests, in three groups:

  1. Parity — 40 randomized-ring MOCs; hand-built mixed-order MOCs including cells finer than the target (so the coarsen-and-dedup half of to_order is covered, not just the expand half); the antimeridian / pole-adjacent / basecell-edge rings; the Antarctic basin fixtures already in-tree (basins 24 and 2 — the pole+antimeridian one); determinism across runs; and the sorted-unique guarantee the docstring sells.
  2. Composition — the Batch polygon coverage: polygons_to_morton_mocs (issue #153) #154 fixture rings run through both stages, polygons_to_morton_mocsmocs_to_orders, asserted against the scalar chain; and the basin rings chained through both stages asserted equal to the flat morton_coverage of the same ring (the lossless-MOC identity).
  3. Contract — empty batch, single MOC, an empty MOC keeping its slot, exact-coverage offsets, offsets errors naming the MOC index, the budget refusal naming the lowest-index offender and carrying the scalar's escape text, the budget's strict > boundary pinned against the scalar's, the default being the single _FLAT_COVER_WARN_THRESHOLD for both functions (__defaults__ asserted equal, so a second ceiling cannot creep in unnoticed), refusal-before-densify, the accepted max_cells domain matching the scalar's (float budgets, 2 ** 64, negatives — the __defaults__ check pins the value, this pins the domain), order-0 parity, and the GIL-release check.

Pre-existing, unrelated, and left alone: plain cargo test (as opposed to cargo test --lib) fails to link on macOS — verified identical on a stashed, pristine tree at 814aea1 — which is the same extension-module linking issue #154 recorded for cargo bench. CI on Linux is unaffected. One transient cargo test --lib failure was seen early on and did not reproduce in 4 subsequent full runs; #154 recorded dissolve::tests::sub_hemisphere_cover_still_dissolves as intermittently failing on main at ~15%.

Questions for review

  1. mortie.coverage.Xmortie.moc.X for the moved names. The flat package surface is untouched, but the fully-qualified module path changed for anything that reached in through mortie.coverage. Two in-tree call sites needed updating (mortie/tests/test_geometry.py, and mortie/geometry.py:613's lazy from .coverage import moc_to_order). A sweep of zagg and moczarr found no consumer importing these from mortie.coverage — every one goes through the flat name — so no shim was added. Flag if you want a deprecation alias in coverage.py anyway.
  2. order divergence — resolved in the fold, both directions; one follow-up left to file. The two functions now share one domain, 0..=29, checked on both paths (bc1aa67, 2449040). The asymmetry was worse than this question stated: the scalar's out-of-range order did not merely shift past 64 — to_order_count's 1u64 << (2 * (order - depth) as u32) wraps mod 64 in release, so for depth-6 input the whole band order 38-48 estimated under the default budget and passed through to a Rust panic surfacing as pyo3_runtime.PanicException, whose MRO is (PanicException, BaseException, object) — uncatchable by except ValueError or except Exception, including the zagg catalog/shardmap.py handler #108's ruling cites as the no-migration argument. The wrapper guard raises the ValueError the contract already promises, pinned by a test written as a plain except ValueError. Untouched root cause: the shift overflow in Rust (to_order_count and to_order in src_rust/src/moc.rs) is still there — no input reaches it through the public API now, but the kernel remains panic-on-out-of-range with a fabricated estimate one call below, which is worth its own mortie small-fix issue. Not filed from here (opening an issue is side-effecting, CLAUDE.md §6) — say the word and it goes up.
    The batch's other end was a straight regression against its scalar twin: 1..=29 refused order = 0, which the scalar answers correctly (the base cells a MOC touches, first-class in mortie — _whole_sphere() is order-0 words). Widened to 0..=29 with parity pinned in both Rust and Python.
  3. No Arrow skin in this phase. Batch polygon coverage: polygons_to_morton_mocs (issue #153) #154 shipped mortie.arrow.polygons_to_morton_mocs as its phase 4. The equivalent here (ListArray of morton_index in, ditto out, with the slice re-basing) is a natural follow-up but is not in mocs_to_orders: ragged batch moc_to_order — plus an API sweep for bulk-by-default operators #156's phase-1 acceptance. Say the word and it becomes a phase.
  4. Deliberate structural parallel with coverage/batch.rs. BatchOrders::{new, extend_chunk, reserve_estimate} mirrors BatchMocs's by design (the issue asks the Rust side to follow the same axis), which means ~40 lines of the same shape in two modules. Factoring a shared ragged-u64 builder is possible but would refactor merged Batch polygon coverage: polygons_to_morton_moc over ragged (offsets) arrays — one call, rayon across polygons #153 code; left as-is deliberately, flag if the duplication should be collapsed now rather than when the third batch module (phase 2) needs it.

@espg espg added the implement label Aug 8, 2026
@espg

espg commented Aug 8, 2026

Copy link
Copy Markdown
Owner Author

🤖 from Claude

Status: phase 1 pushed (5d092f6), CI green, stopping here for the adversarial review pass. Phases 2-5 are unstarted; nothing is waiting on @espg beyond the four items under "Questions for review" in the body, none of which block phase 2.

CI on the head sha: test (3.10/3.11/3.12), ruff, numpydoc validation, arro3-no-pyarrow, Build (verify only) and all five wheel legs pass; only the codspeed benchmark jobs were still running at the time of writing.

Local gates, for the record: cargo test --lib 276 passed / 1 ignored, pytest 1064 passed / 12 skipped, cargo fmt --check and cargo clippy --lib --benches with no new warnings, flake8 --select=E9,F63,F7,F82 and numpydoc lint clean. Plain cargo test (not --lib) fails to link on macOS on an unmodified tree at 814aea1 too — the pre-existing extension-module linking issue #154 recorded for cargo bench.

@codecov

codecov Bot commented Aug 8, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.27%. Comparing base (4335b93) to head (4de7665).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #160      +/-   ##
==========================================
+ Coverage   95.22%   95.27%   +0.05%     
==========================================
  Files          11       12       +1     
  Lines        1570     1587      +17     
==========================================
+ Hits         1495     1512      +17     
  Misses         75       75              
Flag Coverage Δ
unittests 95.27% <100.00%> (+0.05%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
mortie/__init__.py 89.28% <100.00%> (+0.39%) ⬆️
mortie/coverage.py 97.16% <ø> (-0.98%) ⬇️
mortie/geometry.py 95.12% <100.00%> (ø)
mortie/moc.py 100.00% <100.00%> (ø)

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 814aea1...4de7665. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Comment thread mortie/moc.py
"""
morton = np.asarray(morton, dtype=np.uint64).ravel()
if max_cells is not None:
estimated = int(_rustie.rust_moc_to_order_count(morton, order))

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 from Claude (review)

[medium] Question (2) has more teeth than it states: the scalar's missing order guard does not merely "shift past 64" — the estimator silently wraps, the budget is bypassed, and the caller gets a PanicException that derives from BaseException, so except ValueError (and even except Exception) misses it.

to_order_count computes 1u64 << (2 * (order - depth) as u32) (src_rust/src/moc.rs). In a release build that shift wraps mod 64, so the estimate this line reads is fabricated for any out-of-range order. For a depth-6 input word:

  • order = 38 → shift 64 → wraps to 0 → estimate 1 cell → the default max_cells = 1 << 20 passes it → to_order runs → nested2mort asserts.
  • order = 255 → shift 498 → wraps to 50 → estimate 1125899906842624 → refuses, but the message quotes a count that was never real.

Measured on this branch (maturin develop --release, .venv156), scalar with the default guard on:

order= 38  PanicException: depth must be 0..=29, got 38
order= 40  PanicException: depth must be 0..=29, got 40
order= 70  PanicException: depth must be 0..=29, got 70
order=255  ValueError: moc_to_order would densify to ~1125899906842624 cells at order 255

and that exception is not catchable by convention:

>>> type(e).__mro__
(<class 'pyo3_runtime.PanicException'>, <class 'BaseException'>, <class 'object'>)
>>> isinstance(e, Exception)
False

For a depth-6 input the whole band order 38–48 slips the default budget this way (2 ** ((2 * (order - 6)) % 64) <= 1 << 20).

This matters to the ruling this function now carries. espg's #108 ruling keeps ValueError partly because consumers already catch it — the per-ring try/except ValueError around moc_to_order in zagg's catalog/shardmap.py is cited there as the no-migration argument — and that handler does not catch a PanicException.

So: the divergence is real and defensible, the batch's 1..=29 check is the right posture, and the scalar's hole is worth its own small-fix rather than leaving the asymmetry standing. The guard belongs in this Python wrapper, beside the max_cells check, where it costs nothing and raises the ValueError the contract already promises.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 from Claude

Folded in bc1aa67 — the guard is in the wrapper, beside the budget check, and the divergence with the batch is gone in both directions (this half and the order-0 half of the next thread).

morton = np.asarray(morton, dtype=np.uint64).ravel()
if not 0 <= order <= 29:
    raise ValueError(f"Order must be between 0 and 29, got {order}")

Reproduced your measurement on the rebuilt extension before changing anything (maturin develop --release, .venv156) — depth-6 word, default budget: order=38PanicException: depth must be 0..=29, got 38, MRO (PanicException, BaseException, object); order=255 → the budget ValueError quoting ~1125899906842624, a count the wrapped shift invented.

Pinned by TestMocToOrderGuard::test_out_of_range_order_is_a_catchable_valueerror (mortie/tests/test_coverage.py), written as the handler a consumer actually writes rather than pytest.raises, so an escaping PanicException fails the test rather than being caught by a broad matcher:

for order in (30, 38, 44, 48, 70, 255):
    try:
        mortie.moc_to_order(self.BASE_CELL, order)
    except ValueError as exc:          # the plain handler consumers write
        assert "between 0 and 29" in str(exc)
    else:
        raise AssertionError(f"order={order} did not raise")

38/44/48 are there specifically because they are the band that slipped the default budget on depth-6 input; test_order_zero_is_in_range pins that the new guard does not swallow order 0. The docstring now carries the reason (the BaseException MRO and the mod-64 wrap) rather than just the range, since that is what makes this a fix and not a new refusal.

Root cause left standing, deliberately: the wrap is in Rust — to_order_count's 1u64 << (2 * (order - depth) as u32) and to_order's matching shift (src_rust/src/moc.rs) — and this commit does not touch it. The wrapper guard means no input reaches it through the public API any more, but the kernel is still panic-on-out-of-range with a fabricated estimate one call below. Recorded in the PR body as its own mortie small-fix; not filed from here, since opening an issue is a side-effecting directive that needs espg (CLAUDE.md §6).

Comment thread src_rust/src/moc/batch.rs Outdated
order: u8,
max_cells: Option<u64>,
) -> Result<usize, String> {
if !(1..=29).contains(&order) {

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 from Claude (review)

[medium] This range also refuses order = 0, which the scalar accepts and answers correctly — so the batch is not a drop-in for coarsening a MOC to its base cells, and question (2) only discusses the upper end of the divergence.

Verified on this branch:

>>> m = np.concatenate([w(base=2, order=4), w(base=5, order=4)])
>>> mortie.moc_to_order(m, 0, max_cells=None)
array([3458764513820540928, 6917529027641081856], dtype=uint64)   # base cells 2 and 5, at order 0
>>> mortie.mocs_to_orders(m, np.array([0, 2], np.int64), 0, max_cells=None)
ValueError: Order must be between 1 and 29

Order 0 is first-class in mortie, not a degenerate case: moc.py's own _whole_sphere() builds the 12 base cells with norm2mort(zeros, base, 0), and to_order's coarsen branch (up = 2 * (depth - order)) handles order = 0 with no special casing.

Copying coverage/batch.rs:77's range verbatim is defensible there — an order-0 polygon cover is meaningless — but mocs_to_orders' order is a densify/coarsen target, where 0 answers a real question ("which base cells does this MOC touch"). The asymmetry is only visible because the scalar twin now sits in the same file.

Either widen to 0..=29 and add the parity case, or record the exclusion as a decision: the wrapper's Target HEALPix order (1-29) currently reads as an inherited coverage constraint rather than a choice, and question (2) names only the > 29 half.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 from Claude

Widened, not recorded as an exclusion — 2449040. validate_batch is now !(0..=29).contains(&order) with the message Order must be between 0 and 29, and the doc comment says why the range is not copied from coverage/batch.rs any more: order 0 is a real coarsen target ("which base cells does this MOC touch"), not a degenerate polygon cover.

Parity is pinned on both sides:

  • Rust — moc::batch::tests::order_zero_matches_scalar, the ragged() fixture through mocs_to_orders(.., 0, None) asserted slice-by-slice against to_order(.., 0).
  • Python — test_order_zero_parity (mortie/tests/test_moc_batch.py): 8 random-ring MOCs plus your exact case, a MOC spanning base cells 2 and 5, through _assert_batch_parity(mocs, order=0) and then asserted equal to norm2mort([0, 0], [2, 5], 0) outright.

The wrapper docstring's Target HEALPix order (1-29) and the Raises line both read 0-29 now, cross-referenced to the scalar's domain rather than standing as an independent number. The scalar's own missing upper guard — the other half of question (2) — is folded in bc1aa67 on the thread above, so the two functions now share one domain, 0..=29, checked on both paths.

cargo test --lib 277 passed (was 276), pytest 1068 passed / 12 skipped (was 1064).

Comment thread mortie/moc.py
"""
values = np.ascontiguousarray(np.asarray(values, dtype=np.uint64).ravel())
offsets = np.ascontiguousarray(np.asarray(offsets, dtype=np.int64).ravel())
out_values, out_offsets = _rustie.rust_mocs_to_orders(

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 from Claude (review)

[low] The wrapper coerces values and offsets but not max_cells, so the batch refuses budgets the scalar accepts — a parity gap in the one parameter this PR claims carries the scalar's semantics per item.

values and offsets get np.asarray(..., dtype=...).ravel() on the two lines above; max_cells goes to pyo3's Option<u64> raw. Measured on this branch, same MOC through both entry points:

max_cells mocs_to_orders moc_to_order
1e9 (float) TypeError accepted
2 ** 64 OverflowError accepted
-1 OverflowError ValueError (the documented refusal)

A float budget is a plausible spelling (max_cells=mem_bytes / 8), and the __defaults__ equality test does not cover this — it pins the default value, not the accepted domain.

int(max_cells) in the wrapper closes the float and 2 ** 64 rows (the latter then raising ValueError from the same overflow path rather than pyo3's); the negative row would still need an explicit check or a docstring note. Either fix it or say in the max_cells docstring that the batch takes a strict non-negative int where the scalar takes any comparable.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 from Claude

Fixed rather than documented as a divergence — 24c029d. The wrapper now coerces max_cells the way it already coerces values/offsets:

if max_cells is not None:
    max_cells = int(max_cells)
    if max_cells < 0:
        raise ValueError(f"max_cells must be non-negative, got {max_cells}")
    # The per-MOC estimate saturates at u64::MAX, so a budget at or past it
    # can never be exceeded -- clamping keeps the scalar's answer instead of
    # raising OverflowError out of the binding.
    max_cells = min(max_cells, (1 << 64) - 1)

All three rows of your table now agree with the scalar. The float row is exact parity, not an approximation: the scalar compares an integer estimate against the float, so flooring the budget gives the same decision at every value — pinned at the boundary (4095.5 refuses, 4096.5 does not, both entry points). The 2 ** 64 row clamps because the estimate saturates at u64::MAX, so a budget at or past it is unreachable either way. The negative row is the one place the messages differ — the batch says max_cells must be non-negative, the scalar reaches its densify refusal — but the type is the same ValueError and both refuse, which is the part a caller's handler sees.

Test extended as you asked, rather than the default-only check: test_budget_domain_matches_the_scalar walks 1e9, float(4 ** 6), 2 ** 64, 2 ** 70, np.uint64(...) asserting batch output equals the scalar's for each, then the two floor boundaries and the negative refusal. test_budget_default_is_the_single_flat_cover_threshold keeps pinning the default value; the two together now cover value and domain.

Comment thread mortie/moc.py Outdated
flat, flat_off = mortie.mocs_to_orders(cells, off, 8)

MOCs are densified in chunks and each chunk is copied into the ragged output
as it lands, so peak memory is about the returned ``values`` array plus one

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 from Claude (review)

[low] "peak memory is about the returned values array plus one chunk" holds in the densify direction — I independently reproduce 1.07–1.15× at 4× the PR's N — but it omits the full copy of the input, which is the entire peak whenever the result is smaller than the input.

Independent maxrss-delta measurement on this branch, same instrument as the PR body, each shape in its own process:

shape input result peak Δ Δ / result
100k MOCs, 8→10 11.8 MiB 303.6 MiB 349.7 MiB 1.15×
400k MOCs, 8→10 47.4 MiB 1214.4 MiB 1304.8 MiB 1.07×
400k MOCs, 8→10, batch sorted ascending by MOC size 47.4 MiB 1214.4 MiB 1230.6 MiB 1.01×
250k MOCs, 11→11 279.5 MiB 1805.8 MiB 2151.0 MiB 1.19×
250k MOCs, 11→4 (coarsen) 279.5 MiB 5.1 MiB 295.5 MiB 57×

The chunked assembly is doing its job. I attacked reserve_estimate specifically — row 3 reorders the batch ascending by MOC size so the first chunk's mean extrapolates from the smallest items and the reserve must keep being revised — and it did not degrade (1.01×).

The last row is rust_mocs_to_orders' values.to_vec() / offsets.to_vec(), and that copy is required, not gratuitous: a &[u64] borrowed from a numpy array cannot cross py.allow_threads, which is why coverage/batch.rs's pyfunction does the same. So this is a docstring correction rather than a code change — peak is result + one chunk + a second copy of the input, and in the coarsen direction the input term is the whole of it. Worth stating because the guarantee as written invites a caller to size a worker off the result alone.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 from Claude

Corrected in 4de7665 — docstring only, in both places that made the claim: the Python wrapper's memory paragraph and src_rust/src/moc/batch.rs's # Memory posture. The copy stays, for the reason you give (&[u64] borrowed from numpy cannot cross py.allow_threads), so the posture is now stated as input copy + result + one chunk.

Re-measured independently before rewriting the text — same instrument (maxrss delta across the call), each shape in its own process, on the rebuilt extension:

shape input result peak Δ Δ / result input + result
100k MOCs, 8→10 12.9 MiB 342.1 MiB 398.3 MiB 1.16× 354.9 MiB
250k MOCs, 11→11 304.3 MiB 2097.5 MiB 2479.5 MiB 1.18× 2401.8 MiB
250k MOCs, 11→8 304.3 MiB 55.2 MiB 382.6 MiB 6.93× 359.5 MiB
250k MOCs, 11→4 (coarsen) 304.3 MiB 5.3 MiB 317.5 MiB 60× 309.5 MiB

Your 57× reproduces at 60× on my run, and the added middle row (11→8) shows it is a continuum rather than a coarsen/densify switch — 6.9× at a mild coarsen. The input + result column is the point: the model fits every row to within one chunk (+43, +78, +23, +8 MiB), which is what the docstring now says, with the ratio quoted in both directions and an explicit "size a worker off input + result, not the result alone".

Two things left standing rather than folded:

cargo fmt --check clean, cargo clippy --lib --benches unchanged at the 7 pre-existing warnings (none in moc/), cargo test --lib 277 passed, pytest 1068 passed / 12 skipped, numpydoc lint clean on the touched modules.

@codspeed-hq

codspeed-hq Bot commented Aug 8, 2026

Copy link
Copy Markdown
Contributor

Merging this PR will not alter performance

✅ 71 untouched benchmarks
⏩ 1 skipped benchmark1


Comparing claude/156-mocs-to-orders (4de7665) with main (4335b93)2

Open in CodSpeed

Footnotes

  1. 1 benchmark was skipped, so the baseline result was used instead. If it was deleted from the codebase, click here and archive it to remove it from the performance reports.

  2. No successful run was found on main (814aea1) during the generation of this report, so 4335b93 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report.

@espg
espg marked this pull request as ready for review August 8, 2026 02:41
@espg espg added the waiting label Aug 8, 2026
@espg
espg merged commit 42db670 into main Aug 8, 2026
24 checks passed
@espg
espg deleted the claude/156-mocs-to-orders branch August 8, 2026 04:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant