diff --git a/CHANGELOG.md b/CHANGELOG.md index 639913d7..ad2c8115 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,70 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +- **BREAKING: `mortie.toc` is the `Toc` constructor, not a submodule** (issue + #198). `mortie/toc.py` is now `mortie/_toc.py`, which frees the `mortie.toc` + name for a callable — the same move issue #196 made for `mortie.moc`. + **Statement-form `import mortie.toc` and `from mortie.toc import x` break at + this rename** — the module does not exist any more, and no import-system + shim is possible because a callable cannot also be a module. The flat + package names are unchanged and are the supported spelling: + `mortie.time2toc`, `mortie.span2toc`, `mortie.toc2time`, `mortie.toc_merge`, + `mortie.toc_reduce`, `mortie.tocs_reduce`, `mortie.toc_is_range`, + `mortie.toc_overlaps`, `mortie.toc_contains`, `mortie.from_datetime64`, + `mortie.to_datetime64`, `mortie.from_gps_ns`, `mortie.to_gps_ns` — the four + grid/epoch constants, which previously lived only on the submodule, are now + flat too: `mortie.Q_START_NS`, `mortie.Q_END_NS`, `mortie.TOC_MAX_NS`, + `mortie.GPS_EPOCH_NS` — and `mortie.toc_normalize` / `mortie.toc_and` are + **new in this release, flat from the start**: they never had a submodule + spelling, so they are not in the shim's roster and + `mortie.toc.toc_and` was never reachable. Attribute access to the old names + (`mortie.toc.toc_merge`, `mortie.toc.Q_START_NS`) still resolves through a + migration shim for **one minor version**, emitting a `DeprecationWarning` + on each access (deduplication is left to the standard warnings filters); + the attributes then drop. + +- **`Toc`: a time-first temporal coverage object** (issue #198). + `toc("2020-01-01", "2021-06-01")` builds a temporal coverage from ISO + strings or `datetime64` instants and pairs (via `time2toc` / `span2toc`, + broadcasting), a `uint64` toc word array, or anything exposing the new + `__toc_words__()` interchange dunder. The canonical form is a word **set** + — `toc_normalize`'s sorted maximal merges, kept eagerly and stored + read-only, so `==` and `hash()` are well defined and gappy coverage keeps + its gaps (the constructor docstring states the one-way lossy-toward- + coverage act: subsumed instants are absorbed and not recoverable from the + cover). Methods are `overlaps`, `contains`, and `intersection` / `&` — + **every public method a single delegation to `toc_and`**, the one set + operation the issue #177 call-site audit ruled in, pinned mechanically by + the same delegation test machinery as `Moc` (now shared in + `mortie/tests/delegation.py`); union is construction + (`Toc(np.append(a.words, b.words))`) and difference/xor deliberately do + not ship. The predicates are documented as envelope algebra with a + conservative-direction table; `repr` prints the span/instant counts, the + outward-rounded UTC extent, and the covered duration. See + [docs/api/toc_object.md](docs/api/toc_object.md). + +- **Toc set algebra: `toc_normalize` and `toc_and`** (issues #177 / #198). + The two entries the issue #177 call-site audit ruled in, both new public + flat names. `toc_normalize(words)` is the **canonical cover form**: the + sorted word set with the same decoded coverage as the input — ranges + coalesce iff their decoded half-open envelopes overlap or abut exactly (a + surviving gap is never bridged, however small, because outward rounding + only shrinks apparent gaps), a timestamp a range subsumes is absorbed, and + a free instant survives bit-identical. `toc_and(a, b)` is the **one set + operation** over that form: both operands canonicalized, then a sorted + sweep emitting `[max(starts), min(ends))`, with a timestamp surviving iff + genuinely covered on both sides. Conservative directions: normalize is + coverage-identical with no rounding arm anywhere (merged bounds are + min/max of on-grid values); intersection is **exact by grid closure** — + the max of two starts stays on the 2^31 ns start grid and the min of two + ends on the 2^32 ns end grid — and never under-covers the true + intersection, over-covering only by the operands' own inherited quantum. + Union needs no operator (concatenate, then `toc_normalize`); **difference + and xor deliberately do not ship**, because conservative covers + *under*-cover on subtraction and no audited call site exists. Both release + the GIL and carry `toc_merge`'s scope — an arbitrary bit pattern is + garbage in, garbage out, deterministically. + - **BREAKING: `mortie.moc` is the `Moc` constructor, not a submodule** (issue #196). `mortie/moc.py` is now `mortie/_moc.py`, which frees the `mortie.moc` name for a callable. **Statement-form `import mortie.moc` and diff --git a/docs/api/toc.md b/docs/api/toc.md index e80818b8..9ac482a0 100644 --- a/docs/api/toc.md +++ b/docs/api/toc.md @@ -1,4 +1,4 @@ -# mortie.toc +# mortie toc kernel The toc word — temporal order coverage (issue #175): one `uint64` packing either an exact nanosecond timestamp or a conservative time range, sortable @@ -9,11 +9,28 @@ IVOA T-MOC. These flat-array elementwise ops are the type's scalar surface (the same relationship [the MOC kernel](moc.md) has to its ops over one cover), plus one ragged operator — `tocs_reduce`, the segmented sibling of `toc_reduce` (issue #177), kept here because it folds the word type itself -rather than operating over covers. The many-*cover* plurals still land in -[mortie.batch](batch.md), and wait on the interval-set algebra, which stays -deferred for want of a consumer. The names stay flat on the package +rather than operating over covers. `toc_normalize` and `toc_and` are the +set-algebra entries the issue #177 call-site audit ruled in: the canonical +cover form and the one set operation over it. The many-*cover* plurals still +land in [mortie.batch](batch.md). The names stay flat on the package (`mortie.time2toc`, ...). +These are the **kernel layer**: words in, words out, no wrapping cost, and +nothing here is deprecated. The **object layer** over them is +[mortie.Toc](toc_object.md), where every public method is a single delegation +to a function on this page. + +!!! warning "`mortie.toc` is no longer a module (issue #198)" + + The implementation moved to `mortie/_toc.py` so that `mortie.toc` could + become the `Toc` constructor — the same move issue #196 made for + `mortie.moc`. `import mortie.toc` and `from mortie.toc import …` + **break**; the flat package names (`mortie.time2toc`, + `mortie.toc_merge`, …, and now `mortie.Q_START_NS`, `mortie.Q_END_NS`, + `mortie.TOC_MAX_NS`, `mortie.GPS_EPOCH_NS`) are unchanged and are the + supported spelling. `mortie.toc.toc_merge`-style attribute access still + resolves for one minor version, with a `DeprecationWarning`. + Worked example: [examples/toc_temporal_coverage.ipynb](https://github.com/espg/mortie/blob/HEAD/examples/toc_temporal_coverage.ipynb) walks the type end-to-end on synthetic data — encoding, the conservative @@ -21,13 +38,15 @@ merge, sorting without a comparator, the window predicates at a quantum boundary, and the UTC/GPS round-trip ([run it on Binder](https://mybinder.org/v2/gh/espg/mortie/HEAD?labpath=examples%2Ftoc_temporal_coverage.ipynb)). -::: mortie.toc +::: mortie._toc options: members: - time2toc - span2toc - toc2time - toc_merge + - toc_normalize + - toc_and - toc_reduce - tocs_reduce - toc_is_range diff --git a/docs/api/toc_object.md b/docs/api/toc_object.md new file mode 100644 index 00000000..f4c1ea55 --- /dev/null +++ b/docs/api/toc_object.md @@ -0,0 +1,56 @@ +# mortie.Toc — the temporal coverage object + +`mortie.toc(...)` builds a `Toc`: a temporal coverage as an object, so that +gappy time coverage reads as time. + +```python +from mortie import toc + +when = toc("2020-01-01", "2021-06-01") +assert store_toc.overlaps(when) +sliver = store_toc & when # the canonical cover of the overlap +``` + +## The two-layer rule + +mortie's temporal-coverage surface is two layers and stays that way, the same +split [the spatial object](moc_object.md) documents: + +- **The kernel functions are the array/batch layer.** The free `toc_*` + functions on [the toc kernel page](toc.md) are words in, words out, + unchanged and un-deprecated, and the segmented `tocs_reduce` stays + function-shaped permanently. Array-first consumers keep calling these + directly, at zero wrapping cost. +- **The object is ergonomics.** `Toc` is a thin view over the canonical + `uint64` word set — `toc_normalize`'s sorted maximal merges — never a new + representation: **every public method is a single delegation to a kernel + function** (all three delegate to `toc_and`, the one set operation the + issue #177 call-site audit ruled in). The array stays the interchange + format — `Toc.__toc_words__()` hands the canonical words back, and any + object exposing that dunder is accepted wherever a `Toc` is. + +## The canonical form is a word set + +A store observed in campaigns has *gappy* coverage: one merged envelope +papers over the gaps exactly where they are most informative, so the +canonical form keeps k disjoint spans (plus free instants, bit-identical). +Normalization is **lossy toward coverage, one way**: a timestamp subsumed by +a range's decoded span is absorbed at construction, and a cover can be +rebuilt from the sibling word arrays it came from — never the arrays from a +cover. Union needs no method (construction normalizes, so +`Toc(np.append(a.words, b.words))` is the union), and the difference / +symmetric-difference directions deliberately do not ship: conservative +covers under-cover on subtraction, and no audited call site exists. + +Two naming notes. `Toc.overlaps` / `Toc.contains` compare two whole covers +and answer once; the un-deprecated kernel predicates `toc_overlaps` / +`toc_contains` of the same names take a `[q_start_ns, q_end_ns)` query +window and answer elementwise, per word — a different question. And the +predicates are *envelope* algebra, not data algebra: the +conservative-direction table in the module docstring below says which way +each answer can err near a span edge (the quanta are ~2–4 s). + +::: mortie.toc_object + options: + members: + - Toc diff --git a/examples/toc_temporal_coverage.ipynb b/examples/toc_temporal_coverage.ipynb index f78bb226..ad99b0e3 100644 --- a/examples/toc_temporal_coverage.ipynb +++ b/examples/toc_temporal_coverage.ipynb @@ -74,7 +74,6 @@ "import numpy as np\n", "import matplotlib.pyplot as plt\n", "import mortie\n", - "from mortie import toc\n", "\n", "mortie.__version__\n" ] @@ -124,10 +123,10 @@ } ], "source": [ - "print(f'Q_START_NS = {toc.Q_START_NS:>21,} ns ({toc.Q_START_NS / 1e9:.3f} s)')\n", - "print(f'Q_END_NS = {toc.Q_END_NS:>21,} ns ({toc.Q_END_NS / 1e9:.3f} s)')\n", - "print(f'TOC_MAX_NS = {toc.TOC_MAX_NS:>21,} ns (ceiling '\n", - " f'{mortie.to_datetime64(toc.TOC_MAX_NS - 1)})')\n", + "print(f'Q_START_NS = {mortie.Q_START_NS:>21,} ns ({mortie.Q_START_NS / 1e9:.3f} s)')\n", + "print(f'Q_END_NS = {mortie.Q_END_NS:>21,} ns ({mortie.Q_END_NS / 1e9:.3f} s)')\n", + "print(f'TOC_MAX_NS = {mortie.TOC_MAX_NS:>21,} ns (ceiling '\n", + " f'{mortie.to_datetime64(mortie.TOC_MAX_NS - 1)})')\n", "\n", "# The epoch identity: 1850-01-01T00:00:00 UTC is internal ns 0, exactly.\n", "assert mortie.from_datetime64('1850-01-01T00:00:00') == 0\n" @@ -341,9 +340,9 @@ "print(f'real [{seg_start:,} .. {seg_end:,}]')\n", "print(f'envelope [{env_start:,} .. {env_end:,})')\n", "print(f'slack start {(seg_start - env_start) / 1e9:.3f} s '\n", - " f'(< {toc.Q_START_NS / 1e9:.3f} s) '\n", + " f'(< {mortie.Q_START_NS / 1e9:.3f} s) '\n", " f'end {(env_end - seg_end) / 1e9:.3f} s '\n", - " f'(<= {toc.Q_END_NS / 1e9:.3f} s)')\n", + " f'(<= {mortie.Q_END_NS / 1e9:.3f} s)')\n", "\n", "assert env_start <= seg_start and env_end > seg_end\n" ] @@ -690,8 +689,8 @@ } ], "source": [ - "qs = (base // toc.Q_START_NS) * toc.Q_START_NS # 2^31-aligned\n", - "grid = ((qs + 3600 * 10**9) // toc.Q_END_NS) * toc.Q_END_NS # a 2^32 grid line\n", + "qs = (base // mortie.Q_START_NS) * mortie.Q_START_NS # 2^31-aligned\n", + "grid = ((qs + 3600 * 10**9) // mortie.Q_END_NS) * mortie.Q_END_NS # a 2^32 grid line\n", "qe = grid + 1 # 1 ns past it\n", "\n", "# (1) real interval inside the window, envelope end spills past qe\n", @@ -709,7 +708,7 @@ "assert mortie.toc_overlaps(a, qs, qe) and not mortie.toc_contains(a, qs, qe)\n", "assert mortie.toc_overlaps(b, qs, qe)\n", "print(f'\\nboth errors are bounded by one quantum: '\n", - " f'{toc.Q_END_NS / 1e9:.3f} s')\n" + " f'{mortie.Q_END_NS / 1e9:.3f} s')\n" ] }, { @@ -856,11 +855,16 @@ "## Notes\n", "\n", "- **The API is flat on the package.** `mortie.time2toc`, `span2toc`,\n", - " `toc2time`, `toc_merge`, `toc_reduce`, `toc_is_range`, `toc_overlaps`,\n", - " `toc_contains`, `from_datetime64`, `to_datetime64`, `from_gps_ns`,\n", - " `to_gps_ns`; the constants live on the module (`mortie.toc.Q_START_NS`,\n", - " `Q_END_NS`, `TOC_MAX_NS`, `GPS_EPOCH_NS`). Rendered reference:\n", + " `toc2time`, `toc_merge`, `toc_reduce`, `tocs_reduce`, `toc_is_range`,\n", + " `toc_overlaps`, `toc_contains`, `toc_normalize`, `toc_and`,\n", + " `from_datetime64`, `to_datetime64`, `from_gps_ns`, `to_gps_ns`; the\n", + " constants live flat on the package (`mortie.Q_START_NS`, `Q_END_NS`,\n", + " `TOC_MAX_NS`, `GPS_EPOCH_NS`). Rendered reference:\n", " [docs/api/toc.md](../docs/api/toc.md).\n", + "- **`mortie.toc` is the `Toc` constructor now, not a module**\n", + " ([#198](https://github.com/espg/mortie/issues/198)): the kernel moved to\n", + " `mortie/_toc.py`, and `mortie.toc(...)` builds the object layer over these\n", + " words — see [docs/api/toc_object.md](../docs/api/toc_object.md).\n", "- **Range ends are exclusive.** `toc2time` returns a half-open envelope; a\n", " timestamp returns `(t, t)`, the one closed case.\n", "- **Ceiling.** Both encoders reject times at or past `TOC_MAX_NS`\n", @@ -871,10 +875,12 @@ " seconds), which is what pins the epoch identity\n", " `from_datetime64('1850-01-01') == 0`; conversion is exact and invertible\n", " from 1972 on. See the `from_datetime64` docstring.\n", - "- **Not an IVOA T-MOC**, and the interval-set algebra over collections of\n", - " words (unions of disjoint ranges, ragged per-cell temporal covers) is\n", - " deferred to [#177](https://github.com/espg/mortie/issues/177) — this\n", - " notebook is the flat-array elementwise surface only.\n" + "- **Not an IVOA T-MOC.** Of the interval-set algebra over collections of\n", + " words ([#177](https://github.com/espg/mortie/issues/177)), the entries the\n", + " call-site audit ruled in now ship as `toc_normalize` (the canonical cover\n", + " form) and `toc_and` (the one set operation); difference and xor\n", + " deliberately do not — conservative covers under-cover on subtraction.\n", + " This notebook is the flat-array elementwise surface only.\n" ] } ], diff --git a/mkdocs.yml b/mkdocs.yml index 71b6925c..075b89e0 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -51,7 +51,8 @@ nav: - coverage: api/coverage.md - moc kernel: api/moc.md - Moc object: api/moc_object.md - - toc: api/toc.md + - toc kernel: api/toc.md + - Toc object: api/toc_object.md - prefix_trie: api/prefix_trie.md - geometry: api/geometry.md - batch: api/batch.md diff --git a/mortie/__init__.py b/mortie/__init__.py index 0b800b1f..31524536 100644 --- a/mortie/__init__.py +++ b/mortie/__init__.py @@ -26,6 +26,32 @@ split_base_cells, ) +# toc word -- temporal order coverage (issue #175; the module is +# mortie/_toc.py since issue #198 freed the `toc` name for the Toc +# constructor, but the names stay flat on the package either way -- the four +# grid/epoch constants included, now that the submodule spelling is gone). +from ._toc import ( + GPS_EPOCH_NS, + Q_END_NS, + Q_START_NS, + TOC_MAX_NS, + from_datetime64, + from_gps_ns, + span2toc, + time2toc, + to_datetime64, + to_gps_ns, + toc2time, + toc_and, + toc_contains, + toc_is_range, + toc_merge, + toc_normalize, + toc_overlaps, + toc_reduce, + tocs_reduce, +) + # Bulk (plural) twins of the scalar operators, consolidated by arity out of # coverage / geometry / moc / orders (issue #170). The flat package names below # are unchanged -- only the submodule they live in moved. @@ -120,21 +146,14 @@ xy_to_rank, ) -# toc word -- temporal order coverage (issue #175) -from .toc import ( - from_datetime64, - from_gps_ns, - span2toc, - time2toc, - to_datetime64, - to_gps_ns, - toc2time, - toc_contains, - toc_is_range, - toc_merge, - toc_overlaps, - toc_reduce, - tocs_reduce, +# The temporal object layer over the toc kernel above (issue #198), sibling of +# `Moc`: `Toc` wraps the canonical normalized word set, and `toc` is a +# callable namespace rather than a submodule -- `toc("2020-01-01", ...)` +# builds a `Toc`, and `toc.toc_merge`-style attribute access is the +# deprecation shim for the `mortie/toc.py` -> `mortie/_toc.py` rename. +from .toc_object import ( + Toc, + toc, ) __all__ = [ @@ -186,6 +205,8 @@ 'split_base_cells', 'Moc', 'moc', + 'Toc', + 'toc', 'linestring_coverage', 'from_wkb', 'from_wkbs', @@ -207,6 +228,8 @@ 'span2toc', 'toc2time', 'toc_merge', + 'toc_normalize', + 'toc_and', 'toc_reduce', 'toc_is_range', 'toc_overlaps', @@ -216,6 +239,10 @@ 'to_datetime64', 'from_gps_ns', 'to_gps_ns', + 'Q_START_NS', + 'Q_END_NS', + 'TOC_MAX_NS', + 'GPS_EPOCH_NS', ] # morton_index datatype (phase 5) + Arrow interop (phase 4) for issue #35. The diff --git a/mortie/toc.py b/mortie/_toc.py similarity index 78% rename from mortie/toc.py rename to mortie/_toc.py index 91b8dfbe..1d152b42 100644 --- a/mortie/toc.py +++ b/mortie/_toc.py @@ -36,9 +36,21 @@ the relationship :mod:`mortie._moc` has to its ops over one cover. :func:`tocs_reduce` is the one ragged operator here: the segmented sibling of :func:`toc_reduce`, kept beside its scalar because it is a fold over the -word type itself rather than an op over covers (issue #177). The -many-*cover* plurals still land in :mod:`mortie.batch`, and wait on the -interval-set algebra, which stays deferred for want of a consumer. +word type itself rather than an op over covers (issue #177). +:func:`toc_normalize` and :func:`toc_and` are the set-algebra entries the +#177 call-site audit ruled in (issues #177 / #198): the canonical cover +form the ``Toc`` object builds on, and the one set operation over it. The +many-*cover* plurals still land in :mod:`mortie.batch`. + +Renamed from ``mortie/toc.py`` to ``mortie/_toc.py`` for issue #198, which +frees the ``mortie.toc`` name for the :class:`~mortie.toc_object.Toc` +constructor -- the same move issue #196 made for ``mortie.moc``. Nothing +here changed and nothing here is deprecated: these free functions are the +**kernel layer** -- words in, words out, no wrapping cost -- and the +array-first consumers keep calling them on plain ndarrays. +:class:`~mortie.toc_object.Toc` is the **object layer** over them, and +every one of its public methods is a single delegation to +:func:`toc_and`. """ import operator @@ -381,6 +393,162 @@ def tocs_reduce(words, offsets): np.ascontiguousarray(w.ravel()), _as_offsets(offsets))) +def toc_normalize(words): + """Canonicalize a toc word set: sorted maximal merges. + + The canonical cover form (issues `#177 + `_ / `#198 + `_): the unique sorted word + set with the same decoded coverage as the input. Range words coalesce + **iff** their decoded half-open ``[start, end)`` envelopes overlap or + abut exactly -- a surviving decoded gap is never bridged, however small, + because outward rounding only shrinks apparent gaps, so a gap that + survives encoding is a floor on the true gap. A timestamp subsumed by + a range's decoded span adds no coverage and is absorbed; a timestamp no + range subsumes survives bit-identical as an exact degenerate member, + and equal timestamps deduplicate. Timestamps never merge with each + other or extend a range: re-encoding an instant into a range would + round outward and change coverage, which normalize never does. + + The canonical-form laws -- uniqueness, sortedness, duplicate-freeness + -- are guarantees over **encoder-produced** words, the scope + :func:`toc_merge` carries. An arbitrary bit pattern can decode to an + empty envelope (a "range" whose decoded end falls below its decoded + start), which subsumes nothing and does not collapse even against a + copy of itself, so junk words can come back duplicated. Coverage is + still preserved exactly (an empty envelope covers nothing) and the + output is still deterministic and a fixpoint -- junk in is junk out. + + Conservative directions (envelope algebra): + + - **Coverage-identical, not conservatively identical**: the output's + decoded coverage equals the input's exactly. Merged bounds are + min/max of on-grid values (starts on the 2^31 ns grid, ends on 2^32), + so no rounding arm exists anywhere in the operation. + - The input envelopes themselves **over-cover** the real data they were + encoded from (the encoders round outward) and never under-cover; + normalize preserves that direction unchanged. + - **Lossy toward coverage** (word identity, not coverage): which + subsumed instants existed -- and how many times -- is dropped. Exact + instants live in the sibling word arrays a cover is built from; a + cover can be rebuilt from the arrays, never the arrays from a cover. + + Parameters + ---------- + words : array-like + Toc words (``uint64``), any order, duplicates allowed. + + Returns + ------- + numpy.ndarray + The canonical cover, sorted ``uint64`` words (a set, not a + per-element map -- always an array, possibly shorter than the + input; empty in, empty out). + + Raises + ------ + ValueError + If ``words`` is negative or non-integer-typed. + + See Also + -------- + toc_merge : the single-envelope semilattice join (one word out). + tocs_reduce : segmented single-envelope folds. + + Examples + -------- + Timestamps inside a covering range absorb; a free instant survives + exactly, and the gap before it is preserved: + + >>> import mortie, numpy as np + >>> r = mortie.span2toc(mortie.from_datetime64("2020-03-01"), + ... mortie.from_datetime64("2020-03-05")) + >>> t = mortie.time2toc(mortie.from_datetime64( + ... ["2020-03-02", "2020-03-03", "2020-07-04"])) + >>> got = mortie.toc_normalize(np.append(t, np.uint64(r))) + >>> got.tolist() == sorted([r, int(t[2])]) + True + """ + w = _as_u64(words, "words") + return np.asarray(_rustie.rust_toc_normalize( + np.ascontiguousarray(w.ravel()))) + + +def toc_and(a, b): + """Intersect two toc word sets: the canonical cover of the common coverage. + + The one set operation the `#177 + `_ call-site audit ruled in + beyond :func:`toc_normalize` (issue `#198 + `_). Both operands are + canonicalized internally, so raw unsorted word sets are accepted; the + intersection then runs as a sorted-interval sweep, each surviving piece + ``[max(starts), min(ends))``. A timestamp survives iff it is genuinely + covered on both sides -- inside the other cover's decoded ranges, or + present as the identical instant in both -- and it survives + bit-identical. Union needs no operator (concatenate, then + :func:`toc_normalize`); the difference/xor directions deliberately do + not ship -- conservative covers under-cover on subtraction, and no + audited call site exists. Junk words carry :func:`toc_normalize`'s + scope: garbage in, garbage out, deterministically. + + Conservative directions (envelope algebra): + + - **Exact by grid closure, no rounding**: the max of two starts stays on + the 2^31 ns start grid and the min of two ends on the 2^32 ns end + grid, so every intersection bound is exactly representable. + - **Never under-covers the true intersection**: ``A ⊇ X`` and ``B ⊇ Y`` + imply ``A ∩ B ⊇ X ∩ Y`` -- conservatism is preserved by construction. + - **May over-cover** near piece edges by up to one quantum per side, + inherited from the operands' outward-rounded envelopes; the operation + itself adds none. + + Parameters + ---------- + a : array-like + Toc words (``uint64``), any order, duplicates allowed. + b : array-like + Toc words (``uint64``), the other operand. + + Returns + ------- + numpy.ndarray + The canonical cover of the intersection, sorted ``uint64`` words + (a set, not a per-element map -- always an array, empty when the + covers share nothing). + + Raises + ------ + ValueError + If either input is negative or non-integer-typed. + + See Also + -------- + toc_normalize : the canonical cover form (and, with concatenation, + the union). + toc_overlaps : the boolean window predicate when only intersection + emptiness is asked. + + Examples + -------- + Two observing campaigns share exactly their overlap week: + + >>> import mortie + >>> a = mortie.span2toc(mortie.from_datetime64("2020-03-01"), + ... mortie.from_datetime64("2020-03-15")) + >>> b = mortie.span2toc(mortie.from_datetime64("2020-03-10"), + ... mortie.from_datetime64("2020-04-01")) + >>> both = mortie.toc_and([a], [b]) + >>> s, e = mortie.toc2time(int(both[0])) + >>> s == mortie.toc2time(b)[0] and e == mortie.toc2time(a)[1] + True + """ + wa = _as_u64(a, "a") + wb = _as_u64(b, "b") + return np.asarray(_rustie.rust_toc_and( + np.ascontiguousarray(wa.ravel()), np.ascontiguousarray(wb.ravel()))) + + def toc_is_range(words): """Test which variant each toc word is. diff --git a/mortie/tests/delegation.py b/mortie/tests/delegation.py new file mode 100644 index 00000000..96a034fe --- /dev/null +++ b/mortie/tests/delegation.py @@ -0,0 +1,186 @@ +"""The single-kernel-delegation pin shared by the object-layer tests. + +:class:`~mortie.Moc` (issue #196) and :class:`~mortie.Toc` (issue #198) both +promise that every public method is a *single* delegation to a kernel +function. This module is the falsifiable form of that promise: a **shape +whitelist** over each method's returned expression plus a **denied-node +sweep** over everything inside it. The blessed shapes are + +- a bare kernel call, +- a kernel call re-boxed by a wrapper (``Moc(...)`` / ``Toc(...)`` / + ``cls(...)``), +- the emptiness comparison ``(...).size 0`` (which operators are + blessed is per class -- ``==`` for ``Moc.contains`` / ``within``, ``>`` for + ``Toc.overlaps``), +- ``np.array_equal((...), (...))``, opt-in, for + ``Toc.contains``'s canonical-form equality -- where the right-hand + coercion must be the *identical* coercion node the kernel call took, so + ``array_equal(toc_and(self.words, _words(other)), _words(self))`` (a + ``contains`` that silently means ``within``) is not a blessed shape. + +Operand coercion through the class's ``_words(...)`` helper is allowed +anywhere. Anything else -- a comprehension, a slice, arithmetic, a branch on +values, a second kernel call -- is algebra the object promised not to have, +and each test file proves its pin rejects such bodies with synthetic +violations. +""" + +import ast +from pathlib import Path + +# Operators, not calls -- so the "exactly one kernel call" count cannot see +# them. Filtering, reindexing or branching on word values is exactly what a +# delegation-only method must not do, so the node types are refused outright. +# The one comparison a body may contain is a blessed shape from the whitelist +# above, which is checked structurally and then excluded from this sweep. +DENIED_NODES = ( + ast.ListComp, ast.SetComp, ast.DictComp, ast.GeneratorExp, ast.Lambda, + ast.IfExp, ast.BoolOp, ast.BinOp, ast.UnaryOp, ast.Subscript, ast.Compare, + ast.NamedExpr, ast.For, ast.While, ast.If, ast.Await, +) + + +def class_def(module, name): + """The ``ast.ClassDef`` for *name* in *module*'s source file. + + Parameters + ---------- + module : module + The imported module whose source holds the class. + name : str + The class name to find. + + Returns + ------- + ast.ClassDef + The class definition node. + """ + tree = ast.parse(Path(module.__file__).read_text()) + return next(n for n in tree.body + if isinstance(n, ast.ClassDef) and n.name == name) + + +def called_name(node): + """The name a ``Call`` node invokes, or ``None`` for a non-call. + + Parameters + ---------- + node : ast.AST + Any AST node. + + Returns + ------- + str or None + The called name (``f`` for ``f(...)``, ``attr`` for ``x.attr(...)``), + or ``None`` when *node* is not a call. + """ + if not isinstance(node, ast.Call): + return None + func = node.func + if isinstance(func, ast.Name): + return func.id + return getattr(func, "attr", "") + + +def _blessed_size_compare(expr, size_ops): + """The expression under a blessed ``.size 0``, or ``None``. + + Parameters + ---------- + expr : ast.Compare + The comparison to check. + size_ops : tuple of type + The ``ast.cmpop`` types blessed for this class. + + Returns + ------- + ast.AST or None + The expression whose ``.size`` is compared, or ``None`` when the + comparison is not of the blessed shape. + """ + left = expr.left + if (isinstance(left, ast.Attribute) and left.attr == "size" + and len(expr.ops) == 1 and isinstance(expr.ops[0], size_ops) + and len(expr.comparators) == 1 + and isinstance(expr.comparators[0], ast.Constant) + and expr.comparators[0].value == 0): + return left.value + return None + + +def delegation_violation(method, *, kernels, wrappers, coercers, + size_ops=(ast.Eq,), array_equal=False): + """Why *method* is not a single kernel delegation, or ``None`` if it is. + + The shape whitelist, not a call count: the returned expression must be a + kernel call, that call re-boxed by a wrapper, a blessed + ``(...).size 0`` emptiness test, or (opt-in) the canonical + equality ``np.array_equal((...), (...))`` -- with no + denied operator node anywhere inside. + + Parameters + ---------- + method : ast.FunctionDef + The method definition to check. + kernels : set of str + The kernel functions a delegation may call, exactly once. + wrappers : set of str + Names that may re-box the kernel's answer (the class, ``cls``). + coercers : set of str + Operand-coercion helpers allowed anywhere in the body. + size_ops : tuple of type, optional + The ``ast.cmpop`` types blessed in the ``.size`` comparison + (default: equality only). + array_equal : bool, optional + Whether ``np.array_equal((...), (...))`` is a + blessed return shape, the right-hand coercion matching a coercion + node inside the kernel call verbatim (default: no). + + Returns + ------- + str or None + The reason the body violates the invariant, or ``None`` when it + holds. + """ + body = [s for s in method.body + if not (isinstance(s, ast.Expr) and isinstance(s.value, ast.Constant))] + if len(body) != 1 or not isinstance(body[0], ast.Return): + return "is not a single return statement" + expr = body[0].value + inner, blessed, extra_allowed = expr, None, set() + if isinstance(expr, ast.Compare): + inner = _blessed_size_compare(expr, tuple(size_ops)) + if inner is None: + return "compares something other than a blessed `(...).size` against 0" + blessed = expr + elif (array_equal and isinstance(expr, ast.Call) + and called_name(expr) == "array_equal"): + if len(expr.args) != 2 or expr.keywords: + return "calls array_equal with a shape other than (kernel call, coerced operand)" + rhs = expr.args[1] + coerced = {ast.dump(n) for n in ast.walk(expr.args[0]) + if called_name(n) in coercers} + if called_name(rhs) not in coercers or ast.dump(rhs) not in coerced: + return "compares the kernel's answer against something other than the coerced operand" + inner, blessed = expr.args[0], expr + extra_allowed = {"array_equal"} + if called_name(inner) in wrappers: + if len(inner.args) != 1 or inner.keywords: + return "wraps more than a single kernel call" + inner = inner.args[0] + if called_name(inner) not in kernels: + return f"returns {called_name(inner) or type(inner).__name__}, not a kernel call" + + calls = [called_name(n) for n in ast.walk(expr) if isinstance(n, ast.Call)] + kernel_calls = [c for c in calls if c in kernels] + if len(kernel_calls) != 1: + return f"makes {len(kernel_calls)} kernel calls, not exactly one" + extra = set(calls) - kernels - wrappers - coercers - extra_allowed + if extra: + return f"also calls {sorted(extra)}" + for node in ast.walk(expr): + if node is blessed: + continue + if isinstance(node, DENIED_NODES): + return f"contains a {type(node).__name__} node" + return None diff --git a/mortie/tests/test_moc_object.py b/mortie/tests/test_moc_object.py index 9f56e796..8839252d 100644 --- a/mortie/tests/test_moc_object.py +++ b/mortie/tests/test_moc_object.py @@ -13,7 +13,6 @@ import inspect import pickle import warnings -from pathlib import Path import numpy as np import pytest @@ -21,6 +20,7 @@ import mortie from mortie import Moc, moc from mortie.moc_object import _KERNEL_NAMES +from mortie.tests.delegation import class_def, delegation_violation def box(west, east, south, north): @@ -413,97 +413,26 @@ def test_latitude_conventions_are_not_the_same_cover(self): # The kernel functions a Moc method is allowed to call, plus the two non-kernel # roles a delegation may use: a wrapper that re-boxes the kernel's answer # (`Moc(...)` / `cls(...)`) and the operand coercion `_words(...)`. Anything -# else in a method body is algebra the object promised not to have. +# else in a method body is algebra the object promised not to have. The +# machinery itself -- the shape whitelist and the denied-node sweep -- lives in +# mortie/tests/delegation.py, shared with the Toc pin (issue #198). _ALLOWED_KERNELS = { "moc_and", "moc_intersects", "moc_minus", "moc_or", "moc_to_order", "moc_xor", "morton_coverage_moc", } -_ALLOWED_WRAPPERS = {"Moc", "cls"} -_ALLOWED_COERCERS = {"_words"} - -# Operators, not calls -- so the "exactly one kernel call" count cannot see -# them. Filtering, reindexing or branching on cell values is exactly what a -# delegation-only method must not do, so the node types are refused outright. -# The one comparison a body may contain is the blessed `.size == 0` shape, -# which is checked structurally and then excluded from this sweep. -_DENIED_NODES = ( - ast.ListComp, ast.SetComp, ast.DictComp, ast.GeneratorExp, ast.Lambda, - ast.IfExp, ast.BoolOp, ast.BinOp, ast.UnaryOp, ast.Subscript, ast.Compare, - ast.NamedExpr, ast.For, ast.While, ast.If, ast.Await, -) - - -def _class_def(name): - """The ``ast.ClassDef`` for *name* in ``mortie/moc_object.py``.""" - source = Path(mortie.moc_object.__file__).read_text() - tree = ast.parse(source) - return next(n for n in tree.body - if isinstance(n, ast.ClassDef) and n.name == name) - - -def _called_name(node): - """The name a ``Call`` node invokes, or ``None`` for a non-call.""" - if not isinstance(node, ast.Call): - return None - func = node.func - if isinstance(func, ast.Name): - return func.id - return getattr(func, "attr", "") def _delegation_violation(method): """Why *method* is not a single kernel delegation, or ``None`` if it is. - The shape whitelist, not a call count: the returned expression must be a - kernel call, that call re-boxed by ``Moc(...)`` / ``cls(...)``, or the - emptiness test ``(...).size == 0`` that ``contains`` / ``within`` - are built on -- with no denied operator node anywhere inside. - - Parameters - ---------- - method : ast.FunctionDef - The method definition to check. - - Returns - ------- - str or None - The reason the body violates the invariant, or ``None`` when it holds. + The Moc-specific instantiation of the shared pin: the returned expression + must be a kernel call, that call re-boxed by ``Moc(...)`` / ``cls(...)``, + or the emptiness test ``(...).size == 0`` that ``contains`` / + ``within`` are built on -- with no denied operator node anywhere inside. """ - body = [s for s in method.body - if not (isinstance(s, ast.Expr) and isinstance(s.value, ast.Constant))] - if len(body) != 1 or not isinstance(body[0], ast.Return): - return "is not a single return statement" - expr = body[0].value - inner, blessed = expr, None - if isinstance(expr, ast.Compare): - left = expr.left - if not (isinstance(left, ast.Attribute) and left.attr == "size" - and len(expr.ops) == 1 and isinstance(expr.ops[0], ast.Eq) - and len(expr.comparators) == 1 - and isinstance(expr.comparators[0], ast.Constant) - and expr.comparators[0].value == 0): - return "compares something other than `(...).size == 0`" - inner, blessed = left.value, expr - if _called_name(inner) in _ALLOWED_WRAPPERS: - if len(inner.args) != 1 or inner.keywords: - return "wraps more than a single kernel call" - inner = inner.args[0] - if _called_name(inner) not in _ALLOWED_KERNELS: - return f"returns {_called_name(inner) or type(inner).__name__}, not a kernel call" - - calls = [_called_name(n) for n in ast.walk(expr) if isinstance(n, ast.Call)] - kernels = [c for c in calls if c in _ALLOWED_KERNELS] - if len(kernels) != 1: - return f"makes {len(kernels)} kernel calls, not exactly one" - extra = set(calls) - _ALLOWED_KERNELS - _ALLOWED_WRAPPERS - _ALLOWED_COERCERS - if extra: - return f"also calls {sorted(extra)}" - for node in ast.walk(expr): - if node is blessed: - continue - if isinstance(node, _DENIED_NODES): - return f"contains a {type(node).__name__} node" - return None + return delegation_violation( + method, kernels=_ALLOWED_KERNELS, wrappers={"Moc", "cls"}, + coercers={"_words"}) def test_every_public_method_is_a_single_kernel_delegation(): @@ -516,7 +445,7 @@ def test_every_public_method_is_a_single_kernel_delegation(): values, a second kernel call -- is the finding this test exists to catch; :func:`test_the_delegation_pin_rejects_violating_bodies` proves it does. """ - methods = [n for n in _class_def("Moc").body + methods = [n for n in class_def(mortie.moc_object, "Moc").body if isinstance(n, ast.FunctionDef) and not n.name.startswith("_")] assert {m.name for m in methods} == { "contains", "difference", "from_polygon", "intersection", diff --git a/mortie/tests/test_toc.py b/mortie/tests/test_toc.py index 989b8195..ae1ab00c 100644 --- a/mortie/tests/test_toc.py +++ b/mortie/tests/test_toc.py @@ -11,7 +11,7 @@ import numpy as np import pytest -from mortie.toc import ( +from mortie._toc import ( GPS_EPOCH_NS, Q_END_NS, Q_START_NS, diff --git a/mortie/tests/test_toc_object.py b/mortie/tests/test_toc_object.py new file mode 100644 index 00000000..4094b668 --- /dev/null +++ b/mortie/tests/test_toc_object.py @@ -0,0 +1,627 @@ +"""Tests for the time-first ``Toc`` object and the ``mortie.toc`` shim (issue #198). + +The object is a thin view over the canonical toc word set: the commitment is +that every public method is a *single* delegation to a kernel function, and +the tests here are written to falsify that rather than to restate it -- see +:class:`TestDelegationParity` (value parity, method by method) and +:func:`test_every_public_method_is_a_single_kernel_delegation` (the structural +pin over the source, on the machinery shared with the ``Moc`` pin in +``mortie/tests/delegation.py``). +""" + +import ast +import copy +import inspect +import pickle +import warnings + +import numpy as np +import pytest + +import mortie +from mortie import Toc, toc, toc_object +from mortie.tests.delegation import class_def, delegation_violation +from mortie.toc_object import _KERNEL_NAMES + + +def u64(values): + """Shorthand: a uint64 array from Python ints.""" + return np.asarray(values, dtype=np.uint64) + + +# Golden words for the ISO constructor forms -- the determinism pin ("same +# input + same version -> same words", byte for byte, on top of the kernel +# goldens in test_toc_setops.py). +GOLDEN_PAIR_WORD = 10729324834951646742 # Toc("2020-01-01", "2021-06-01") +GOLDEN_INSTANT_WORD = 10729324836993577984 # Toc("2020-01-01") + +MARCH = ("2020-03-01", "2020-03-15") +LATE_MARCH = ("2020-03-10", "2020-04-01") +APRIL = ("2020-04-05", "2020-04-20") +FREE_INSTANT = "2020-07-04" + + +@pytest.fixture(scope="module") +def pair(): + """Two overlapping covers, for the set-op and predicate parity tests.""" + return Toc(*MARCH), Toc(*LATE_MARCH) + + +class TestConstructorForms: + """Every documented constructor source, and what each resolves to.""" + + def test_iso_pair_matches_span2toc(self): + expected = mortie.span2toc(mortie.from_datetime64(MARCH[0]), + mortie.from_datetime64(MARCH[1])) + assert np.array_equal(Toc(*MARCH).words, u64([expected])) + + def test_iso_instant_matches_time2toc(self): + expected = mortie.time2toc(mortie.from_datetime64(FREE_INSTANT)) + assert np.array_equal(Toc(FREE_INSTANT).words, u64([expected])) + + def test_datetime64_and_string_agree(self): + assert Toc(np.datetime64("2020-07-04")) == Toc(FREE_INSTANT) + assert Toc(np.datetime64(MARCH[0]), np.datetime64(MARCH[1])) == Toc(*MARCH) + + def test_instant_array(self): + whens = np.asarray(["2020-01-01", "2020-06-01"], dtype="datetime64[ns]") + expected = np.sort(mortie.time2toc(mortie.from_datetime64(whens))) + assert np.array_equal(Toc(whens).words, expected) + + def test_pair_arrays_broadcast(self): + starts = np.asarray(["2020-01-01", "2021-01-01"], dtype="datetime64[ns]") + ends = np.asarray(["2020-02-01", "2021-02-01"], dtype="datetime64[ns]") + expected = np.sort(mortie.span2toc(mortie.from_datetime64(starts), + mortie.from_datetime64(ends))) + assert np.array_equal(Toc(starts, ends).words, expected) + # ... including a scalar start against an array of ends. + fanned = Toc(starts[0], ends) + assert fanned.words.size == 1 # overlapping spans coalesce + + def test_overlapping_spans_coalesce(self, pair): + a, b = pair + both = Toc(np.append(a.words, b.words)) + assert both.words.size == 1 + assert int(both.words[0]) == mortie.toc_merge(int(a.words[0]), + int(b.words[0])) + + def test_words_array_round_trip(self, pair): + a, _ = pair + assert Toc(a.words) == a + + def test_words_are_normalized_eagerly(self, pair): + a, _ = pair + subsumed = mortie.time2toc(mortie.from_datetime64("2020-03-05")) + raw = u64([int(a.words[0]), subsumed, subsumed, int(a.words[0])]) + cover = Toc(raw) + assert np.array_equal(cover.words, mortie.toc_normalize(raw)) + assert cover == a # the subsumed instant absorbed, duplicates gone + + def test_free_instant_survives_bit_identical(self, pair): + a, _ = pair + free = mortie.time2toc(mortie.from_datetime64(FREE_INSTANT)) + cover = Toc(np.append(a.words, u64([free]))) + assert free in cover.words + + def test_protocol_object_round_trip(self, pair): + a, _ = pair + + class Carrier: + def __toc_words__(self): + return a.words + + assert Toc(Carrier()) == a + assert Toc(a) == a + + def test_scalar_int_is_a_word(self, pair): + a, _ = pair + assert Toc(int(a.words[0])) == a + + def test_nd_integer_source_is_refused(self, pair): + # An (N, 2) array of [start_ns, end_ns] pairs is what toc2time hands + # back transposed; raveling it would read four garbage words as a + # plausible cover near the 1850 epoch instead of pointing at + # Toc(starts, ends). + a, _ = pair + with pytest.raises(ValueError, match="1-D integer array"): + Toc(np.asarray([[1000, 2000], [3000, 4000]], dtype=np.int64)) + with pytest.raises(ValueError, match="1-D integer array"): + Toc(a.words.reshape(1, -1)) + + @pytest.mark.parametrize("source", [[], (), np.array([]), u64([])]) + def test_empty_words(self, source): + # The empty cover is load-bearing (it has its own row in the + # conservative-direction table), and `Toc([])` is the first thing + # anyone types -- an untyped empty is float64, so without an explicit + # route it would be refused as *numeric*. + empty = Toc(source) + assert empty.words.size == 0 + assert empty.words.dtype == np.uint64 + assert empty == Toc(u64([])) + + def test_end_is_refused_for_an_empty_source(self): + with pytest.raises(ValueError, match="already"): + Toc([], "2021-01-01") + + @pytest.mark.parametrize("kind", ["words", "cover"]) + def test_end_is_refused_for_a_words_source(self, pair, kind): + # Silently ignoring end= would let Toc(a.words, "2021-01-01") read as + # "extend this cover to 2021" while doing nothing. + a, _ = pair + source = a.words if kind == "words" else a + with pytest.raises(ValueError, match="already"): + Toc(source, "2021-01-01") + + @pytest.mark.parametrize("source, end", [ + (1.5, None), # a float is neither words nor time + ("2020-01-01", 123), # numeric end: wrong-epoch trap + (np.asarray([1.5, 2.5]), None), + ]) + def test_numeric_times_are_refused(self, source, end): + # np.asarray(123, "datetime64[ns]") would silently read ns since 1970 + # -- the wrong epoch -- so numeric endpoints are refused, not guessed. + with pytest.raises(ValueError, match="datetime64 / ISO-string"): + Toc(source, end) + + def test_inverted_span_is_the_kernel_error(self): + with pytest.raises(ValueError, match="after its end"): + Toc(MARCH[1], MARCH[0]) + + def test_negative_words_are_the_kernel_error(self): + with pytest.raises(ValueError, match="non-negative"): + Toc(np.asarray([-1, 2])) + + def test_no_coverage_knobs(self): + # The constructor matrix is (source, end) and nothing else. + params = list(inspect.signature(Toc.__init__).parameters) + assert params == ["self", "source", "end"] + + +class TestNormalizationAndIdentity: + """Eager canonicalization, immutability, equality and hashing.""" + + def test_words_are_canonical_eagerly(self, pair): + a, _ = pair + cover = Toc(np.append(a.words, Toc(*APRIL).words)) + assert np.array_equal(cover.words, mortie.toc_normalize(cover.words)) + + def test_words_are_read_only(self, pair): + a, _ = pair + assert not a.words.flags.writeable + with pytest.raises(ValueError): + a.words[0] = 0 + + def test_instance_is_immutable(self, pair): + a, _ = pair + with pytest.raises(AttributeError, match="immutable"): + a.words = u64([1]) + with pytest.raises(AttributeError, match="immutable"): + del a.words + + def test_pickle_round_trip(self, pair): + # Workers marshal their arguments: a cover must cross a process + # boundary as easily as the word array it wraps. + a, _ = pair + restored = pickle.loads(pickle.dumps(a)) + assert restored == a + assert not restored.words.flags.writeable + + @pytest.mark.parametrize("clone", [copy.copy, copy.deepcopy]) + def test_copy_round_trip(self, pair, clone): + a, _ = pair + restored = clone(a) + assert restored == a + assert not restored.words.flags.writeable + + def test_slots_only(self, pair): + a, _ = pair + assert Toc.__slots__ == ("words",) + assert not hasattr(a, "__dict__") + + def test_equality_is_word_identity(self, pair): + a, b = pair + assert a == Toc(*MARCH) + assert a != b + # An instant and its degenerate span encode different words: the + # canonical form compares, not the timeline it conservatively covers. + assert Toc(FREE_INSTANT) != Toc(FREE_INSTANT, FREE_INSTANT) + + def test_equality_against_a_non_toc_is_not_an_error(self, pair): + a, _ = pair + assert a != 5 + assert a.__eq__(5) is NotImplemented + + def test_hash_matches_equality(self, pair): + a, b = pair + assert hash(a) == hash(Toc(*MARCH)) + assert len({a, Toc(*MARCH), b}) == 2 + + +class TestDelegationParity: + """Every method equals the kernel call on ``.words`` -- the thin-view pin.""" + + def test_overlaps(self, pair): + a, b = pair + assert a.overlaps(b) + assert a.overlaps(b) == (mortie.toc_and(a.words, b.words).size > 0) + assert not a.overlaps(Toc(*APRIL)) + assert not a.overlaps(Toc(FREE_INSTANT)) + + def test_overlaps_conservative_direction_at_the_quantum(self): + # The documented direction, both ways: a real gap smaller than the + # encoding quantum can over-report True (the envelopes graze), while + # a gap wider than a quantum survives encoding and is a decisive + # False -- outward rounding only shrinks apparent gaps. + after = Toc("2020-03-15", "2020-04-01") + assert Toc(MARCH[0], "2020-03-14T23:59:59.999").overlaps(after) + assert not Toc(MARCH[0], "2020-03-14T23:59:50").overlaps(after) + + def test_contains(self, pair): + a, b = pair + inside = Toc("2020-03-05", "2020-03-06") + assert a.contains(inside) + assert not a.contains(b) # b spills past a's end + assert a.contains(a) + assert a.contains(Toc("2020-03-05")) # a subsumed instant + assert not a.contains(Toc(FREE_INSTANT)) + + def test_contains_is_the_kernel_identity(self, pair): + a, b = pair + for other in (b, Toc("2020-03-05", "2020-03-06"), Toc(*APRIL)): + expected = np.array_equal( + mortie.toc_and(a.words, other.words), other.words) + assert a.contains(other) == expected + + def test_contains_normalizes_a_raw_operand(self, pair): + # A raw duplicated, unsorted operand must compare canonical-to- + # canonical, or containment would falsely fail on word inequality. + a, _ = pair + inside = Toc("2020-03-05", "2020-03-06") + raw = u64([int(inside.words[0]), int(inside.words[0])]) + assert a.contains(raw) + + def test_contains_normalizes_a_protocol_operand(self, pair): + # Nothing obliges a third-party __toc_words__() to be canonical, so + # the protocol door must canonicalize like the raw-array door does -- + # otherwise the same word set answers differently by which door it + # came through. + a, _ = pair + inside = Toc("2020-03-05", "2020-03-06") + raw = u64([int(inside.words[0]), int(inside.words[0])]) + + class Carrier: + def __toc_words__(self): + return raw + + assert a.contains(Carrier()) == a.contains(raw) is True + # ... and unsorted, which the sweep would also reject uncanonicalized. + two = Toc(np.append(inside.words, Toc("2020-03-10", "2020-03-11").words)) + jumbled = two.words[::-1] + assert not np.array_equal(jumbled, mortie.toc_normalize(jumbled)) + + class Unsorted: + def __toc_words__(self): + return jumbled + + assert a.contains(Unsorted()) == a.contains(jumbled) is True + + def test_set_methods_take_raw_words_and_protocol_objects(self, pair): + a, b = pair + assert a.overlaps(b.words) == a.overlaps(b) + assert a.contains(b.words) == a.contains(b) + assert a.intersection(b.words) == a.intersection(b) + + def test_intersection(self, pair): + a, b = pair + result = a.intersection(b) + assert isinstance(result, Toc) + assert np.array_equal(result.words, mortie.toc_and(a.words, b.words)) + # The dunder is the same bound method, not a re-implementation. + assert Toc.__and__ is Toc.intersection + assert (a & b) == result + + def test_intersection_keeps_a_shared_instant_bit_identical(self, pair): + a, _ = pair + instant = Toc("2020-03-05") + assert np.array_equal(a.intersection(instant).words, instant.words) + + def test_empty_operand_predicates(self, pair): + # An empty cover is where the predicates stop agreeing: contains is + # vacuously True while overlaps is False. Chosen, not inherited -- + # see the module docstring's conservative-direction table. + a, _ = pair + empty = Toc(u64([])) + assert a.contains(empty) + assert not a.overlaps(empty) + assert not empty.contains(a) + assert not empty.overlaps(empty) + assert empty.contains(empty) + + def test_empty_operand_intersection(self, pair): + a, _ = pair + empty = Toc(u64([])) + assert a.intersection(empty) == empty + assert len(a & a) == len(a) + + def test_union_is_construction(self, pair): + # No union method by design (#177 audit): concatenate + construct is + # the union, because construction normalizes. + a, b = pair + both = Toc(np.append(a.words, b.words)) + assert both.contains(a) and both.contains(b) + assert not hasattr(Toc, "union") + + +# The kernel functions a Toc method is allowed to call: the #177 call-site +# audit ruled exactly one set operation in, so the roster is one name and all +# three public methods delegate to it. Wrapper and coercer roles as in the +# Moc pin; the blessed comparisons differ (`.size > 0` for overlaps, +# `np.array_equal(, _words(...))` for contains -- there is no +# toc_minus to build an emptiness-shaped containment on). +_ALLOWED_KERNELS = {"toc_and"} + + +def _toc_violation(method): + """The Toc-specific instantiation of the shared delegation pin.""" + return delegation_violation( + method, kernels=_ALLOWED_KERNELS, wrappers={"Toc", "cls"}, + coercers={"_words"}, size_ops=(ast.Gt,), array_equal=True) + + +def test_every_public_method_is_a_single_kernel_delegation(): + """The falsifiable form of the thin-view commitment (issues #196 / #198). + + A public ``Toc`` method must be exactly one ``return`` of exactly one + kernel call, optionally re-boxed by ``Toc(...)``, coercing its operand + through ``_words(...)``, compared as ``.size > 0``, or equality-tested + against the coerced operand via ``np.array_equal``. Any other body is + the finding this test exists to catch; + :func:`test_the_delegation_pin_rejects_violating_bodies` proves it does. + """ + methods = [n for n in class_def(toc_object, "Toc").body + if isinstance(n, ast.FunctionDef) and not n.name.startswith("_")] + assert {m.name for m in methods} == {"contains", "intersection", "overlaps"} + for method in methods: + reason = _toc_violation(method) + assert reason is None, f"Toc.{method.name} {reason}" + + +@pytest.mark.parametrize("source", [ + # Blessed-shape near-misses: the wrong comparison operator, the wrong + # constant, an equality against something other than the coerced operand. + "def m(self, other):\n" + " return toc_and(self.words, _words(other)).size == 0\n", + "def m(self, other):\n" + " return toc_and(self.words, _words(other)).size > 1\n", + "def m(self, other):\n" + " return toc_and(self.words, _words(other)).size >= 0\n", + "def m(self, other):\n" + " return np.array_equal(toc_and(self.words, _words(other)), self.words)\n", + # ... and the sharp one: a coercer by name, of the wrong operand -- this + # body is `within`, not `contains`, and the parity tests would not catch + # it because they only re-spell the same expression. + "def m(self, other):\n" + " return np.array_equal(toc_and(self.words, _words(other)), _words(self))\n", + "def m(self, other):\n" + " return np.array_equal(_words(other), _words(other))\n", + "def m(self, other):\n" + " return np.array_equal(toc_and(self.words, _words(other)),\n" + " _words(other), equal_nan=True)\n", + # A kernel outside the audited roster, algebra, and multi-call bodies. + "def m(self, other):\n" + " return Toc(toc_normalize(np.append(self.words, _words(other))))\n", + "def m(self, other):\n" + " return Toc(toc_and(toc_and(self.words, _words(other)), self.words))\n", + "def m(self, other):\n" + " return Toc(np.unique(toc_and(self.words, _words(other))))\n", + "def m(self, other):\n" + " return Toc(toc_and(self.words, _words(other))[:-1])\n", + "def m(self, other):\n" + " return Toc(toc_and(self.words, _words(other)) if other else self.words)\n", + "def m(self, other):\n" + " out = self.words\n" + " for w in other:\n" + " out = toc_and(out, _words(w))\n" + " return Toc(out)\n", + "def m(self, other):\n" + " return self.words\n", +]) +def test_the_delegation_pin_rejects_violating_bodies(source): + """The pin has to fail on algebra, or it pins nothing (issue #198).""" + method = ast.parse(source).body[0] + assert _toc_violation(method) is not None + + +class TestReprAndProtocol: + """What the object tells you about itself, and hands to others.""" + + def test_repr_of_a_range_shows_the_conservative_envelope(self): + # Both bounds round outward -- the start floors (2020-02-29T23:59:58), + # the end ceils past its 2^32 ns grid point (...T00:00:00.979553280 -> + # ...T00:00:01): the repr shows what the cover *covers*, not what it + # was built from, so neither printed bound may land inside it. + cover = Toc(*MARCH) + assert repr(cover) == ( + "Toc(1 range, 2020-02-29T23:59:58 to 2020-03-15T00:00:01, " + "14 days covered)") + starts, ends = mortie.toc2time(cover.words) + printed_start, printed_end = repr(cover).split(", ")[1].split(" to ") + assert np.datetime64(printed_start) <= mortie.to_datetime64( + int(starts.min())) + assert np.datetime64(printed_end) >= mortie.to_datetime64( + int(ends.max())) + + def test_repr_of_an_instant(self): + assert repr(Toc(FREE_INSTANT)) == ( + "Toc(1 instant, 2020-07-04T00:00:00 to 2020-07-04T00:00:00, " + "0 ns covered)") + + def test_repr_of_a_mixed_cover_counts_both_kinds(self, pair): + a, _ = pair + mixed = Toc(np.append(a.words, Toc(FREE_INSTANT).words)) + text = repr(mixed) + assert text.startswith("Toc(1 range + 1 instant, ") + assert text.endswith("14 days covered)") + + def test_repr_of_an_empty_cover(self): + assert repr(Toc(u64([]))) == "Toc(0 spans)" + + def test_len_and_iter(self, pair): + a, _ = pair + mixed = Toc(np.append(a.words, Toc(FREE_INSTANT).words)) + assert len(mixed) == mixed.words.size == 2 + assert np.array_equal(np.fromiter(mixed, dtype=np.uint64), mixed.words) + + @pytest.mark.parametrize("method, window_kernel", [ + ("overlaps", "toc_overlaps"), ("contains", "toc_contains")]) + def test_predicate_docstrings_disambiguate_the_window_kernels( + self, method, window_kernel): + # Both names collide with an un-deprecated batch predicate that asks + # a different question, so each docstring has to name the kernel it + # really delegates to and the one it does not. + doc = getattr(Toc, method).__doc__ + assert "mortie.toc_and : the kernel this delegates to." in doc + assert f"mortie.{window_kernel} :" in doc + assert getattr(mortie, window_kernel) is not None + + def test_protocol_hands_back_the_canonical_words(self, pair): + a, _ = pair + handed = a.__toc_words__() + assert handed is a.words + assert not handed.flags.writeable + + +class TestDeterminism: + """Same input + same version -> same words, byte for byte.""" + + def test_repeated_construction_is_byte_identical(self): + first, second = Toc(*MARCH), Toc(*MARCH) + assert first.words.tobytes() == second.words.tobytes() + + def test_golden_words(self): + assert np.array_equal(Toc("2020-01-01", "2021-06-01").words, + u64([GOLDEN_PAIR_WORD])) + assert np.array_equal(Toc("2020-01-01").words, + u64([GOLDEN_INSTANT_WORD])) + + def test_word_order_does_not_matter(self, pair): + a, _ = pair + free = Toc(FREE_INSTANT) + assert (Toc(np.append(a.words, free.words)) + == Toc(np.append(free.words, a.words))) + + +# The public surface of `mortie/toc.py` as it shipped in 0.9.9 (`git show +# 7f747e0:mortie/toc.py`): thirteen module-level functions plus the four +# grid/epoch constants. The *released* surface, matching the `_MocNamespace` +# pin -- `toc_normalize` and `toc_and` land in this same PR (phases 1-2), so +# `mortie.toc.toc_and` was never a spelling any consumer could hold and is not +# deprecated out. Held here as an independent copy so that editing +# `_KERNEL_NAMES` fails this test rather than redefining the pin. +_RETIRED_SUBMODULE_SURFACE = { + "GPS_EPOCH_NS", "Q_END_NS", "Q_START_NS", "TOC_MAX_NS", + "from_datetime64", "from_gps_ns", "span2toc", "time2toc", + "to_datetime64", "to_gps_ns", "toc2time", "toc_contains", + "toc_is_range", "toc_merge", "toc_overlaps", + "toc_reduce", "tocs_reduce", +} + + +class TestMigrationShim: + """`mortie.toc` is the constructor now; the old attributes deprecate out.""" + + def test_toc_is_callable_and_builds_a_toc(self): + assert isinstance(toc(*MARCH), Toc) + assert toc(*MARCH) == Toc(*MARCH) + + def test_call_forwards_the_end_argument(self): + assert toc(FREE_INSTANT) == Toc(FREE_INSTANT) + assert toc(FREE_INSTANT) != toc(*MARCH) + assert toc(MARCH[0], MARCH[1]) == Toc(MARCH[0], MARCH[1]) + + def test_toc_is_not_a_module(self): + with pytest.raises(ModuleNotFoundError): + __import__("mortie.toc") + + def test_kernel_roster_is_the_frozen_pre_rename_surface(self): + # The roster the shim resolves is a *historical* fact -- what + # mortie/toc.py exported at the rename -- not a live property of the + # kernel, so it is pinned against an independent copy of that surface. + # A kernel function added later must NOT be enrolled in the deprecated + # namespace, which is exactly what an equality against the live module + # would force. + assert set(_KERNEL_NAMES) == _RETIRED_SUBMODULE_SURFACE + + def test_shimmed_names_all_still_exist_on_the_kernel(self): + # The direction that is actually true today: the shim can never + # dangle. Functions must be the kernel module's own; the four + # constants are plain ints, checked for presence. + from mortie import _toc + for name in _KERNEL_NAMES: + member = getattr(_toc, name) + if callable(member): + assert member.__module__ == _toc.__name__ + assert set(_KERNEL_NAMES) <= set(dir(_toc)) + assert set(_KERNEL_NAMES) <= set(mortie.__all__) + + def test_a_new_kernel_function_does_not_join_the_shim(self, monkeypatch): + # Falsifiability of the pin above: growing the kernel must leave the + # deprecated roster alone rather than forcing a new name into it. + from mortie import _toc + + def brand_new_kernel(words): + return words + + brand_new_kernel.__module__ = _toc.__name__ + monkeypatch.setattr(_toc, "brand_new_kernel", brand_new_kernel, + raising=False) + self.test_kernel_roster_is_the_frozen_pre_rename_surface() + self.test_shimmed_names_all_still_exist_on_the_kernel() + with pytest.raises(AttributeError, match="not the old"): + toc.brand_new_kernel + + @pytest.mark.parametrize("name", _KERNEL_NAMES) + def test_deprecated_attribute_still_resolves_to_the_kernel(self, name): + from mortie import _toc + with warnings.catch_warnings(): + warnings.simplefilter("ignore", DeprecationWarning) + assert getattr(toc, name) is getattr(_toc, name) + assert getattr(toc, name) is getattr(mortie, name) + + def test_warning_fires_on_every_access(self): + # Dedup is the warnings module's job (filters are the user's + # contract), not shim state: under `always` every access is visible. + from mortie.toc_object import _TocNamespace + + shim = _TocNamespace() + with warnings.catch_warnings(record=True) as caught: + warnings.simplefilter("always") + shim.toc_merge, shim.toc_merge + assert len(caught) == 2 + assert issubclass(caught[0].category, DeprecationWarning) + assert "mortie.toc.toc_merge is deprecated" in str(caught[0].message) + + def test_a_later_consumer_still_observes_the_warning(self): + # No process-wide budget: the first block *delivers* the warning + # normally -- exactly what a per-name budget would spend -- and a + # second consumer's test suite, later in the same interpreter and on + # the same singleton, must still record it. + from mortie.toc_object import _TocNamespace + + shim = _TocNamespace() + with warnings.catch_warnings(record=True) as first: + warnings.simplefilter("always") + shim.toc_merge + assert len(first) == 1 + with warnings.catch_warnings(record=True) as second: + warnings.simplefilter("always") + shim.toc_merge + assert len(second) == 1 + assert "mortie.toc.toc_merge is deprecated" in str(second[0].message) + + def test_unknown_attribute_raises(self): + with pytest.raises(AttributeError, match="not the old"): + toc.moc_and + + def test_dir_lists_the_deprecated_names(self): + assert dir(toc) == sorted(_KERNEL_NAMES) diff --git a/mortie/tests/test_toc_setops.py b/mortie/tests/test_toc_setops.py new file mode 100644 index 00000000..0fadac6b --- /dev/null +++ b/mortie/tests/test_toc_setops.py @@ -0,0 +1,311 @@ +"""Tests for the toc set algebra (issues #177 / #198). + +``toc_normalize`` is the canonical cover form the ``Toc`` object builds on: +sorted maximal merges, per the espg-confirmed #177 rulings — Q1 (merge on +decoded bounds, never bridge a surviving decoded gap) and Q2 (subsumed +timestamps absorb; free instants survive bit-identical). ``toc_and`` is +the one set operation over it — pairwise ``[max(starts), min(ends))``, +exact by grid closure (Q3). The goldens here are normative at the Python +surface the way the cargo fixtures are at the kernel: the canonical form +is what ``Toc.__eq__`` will compare. +""" + +import numpy as np +import pytest + +from mortie._toc import ( + Q_END_NS, + Q_START_NS, + from_datetime64, + span2toc, + time2toc, + toc2time, + toc_and, + toc_is_range, + toc_normalize, +) + + +def u64(values): + """Shorthand: a uint64 array from Python ints.""" + return np.asarray(values, dtype=np.uint64) + + +def covered(words, t): + """Reference membership: is instant t in the decoded coverage of words?""" + starts, ends = toc2time(np.atleast_1d(u64(words))) + rng = toc_is_range(np.atleast_1d(u64(words))) + return bool(np.any(np.where(rng, (starts <= t) & (t < ends), starts == t))) + + +def rand_words(rng, n, base=None): + """A mixed batch of valid words, clustered around one random base so + overlaps, exact abutments and absorptions all occur. + + Drawing instants and range starts uniformly over the whole span would + spread ~1e18 ns of domain over a dozen words a few quanta wide, so no + two envelopes would ever touch and ``toc_normalize`` would degenerate + to a sort -- the randomized laws below would pass vacuously. The + ``toc_and`` tests pass an explicit shared ``base`` so the two operands + actually intersect (independent bases would make every intersection + empty and those laws vacuous the same way). + """ + if base is None: + base = np.uint64(int(rng.integers(0, 1 << 25)) * Q_END_NS) + + def off(k): + return rng.integers(0, k * Q_END_NS, size=n, dtype=np.uint64) + + t = time2toc(base + off(40)) + a = base + off(40) + r = span2toc(a, a + off(12)) + pick = rng.integers(0, 2, size=n).astype(bool) + return np.where(pick, t, r) + + +# ── goldens (normative) ───────────────────────────────────────────────── + + +def test_golden_issue_177_absorption_example(): + # espg's Q2 example: t1, t2 inside a covering range R absorb; t3 months + # later survives bit-identical; the Mar–Jul gap is preserved. + r = span2toc(from_datetime64("2020-03-01"), from_datetime64("2020-03-05")) + t1, t2, t3 = time2toc(from_datetime64( + ["2020-03-02", "2020-03-04", "2020-07-04"])) + got = toc_normalize(u64([t1, r, t2, t3])) + assert got.tolist() == sorted([r, int(t3)]) + assert toc_is_range(got).tolist() == [True, False] + + +def test_golden_abutting_envelopes_merge(): + # r1's decoded end (2^32 grid) meets r2's decoded start (2^31 grid) + # exactly: end code 8 → 8 * 2^32 = start code 16 * 2^31. + r1 = span2toc(3 * Q_START_NS, 8 * Q_END_NS - 5) + r2 = span2toc(16 * Q_START_NS + 1, 20 * Q_END_NS - 5) + assert toc2time(r1)[1] == toc2time(r2)[0] + merged = toc_normalize(u64([r2, r1])) + assert merged.tolist() == [span2toc(3 * Q_START_NS, 20 * Q_END_NS - 5)] + + +def test_golden_one_quantum_gap_survives(): + # 2^31 ns is the smallest decoded gap the grids can express; it is + # never bridged (Q1: a surviving gap is a floor on the true gap). + r1 = span2toc(3 * Q_START_NS, 8 * Q_END_NS - 5) + r3 = span2toc(17 * Q_START_NS + 1, 20 * Q_END_NS - 5) + assert toc_normalize(u64([r3, r1])).tolist() == [r1, r3] + + +# ── Q1: range merging ─────────────────────────────────────────────────── + + +def test_overlapping_and_nested_ranges_coalesce(): + a = span2toc(10 * Q_END_NS, 40 * Q_END_NS) + b = span2toc(30 * Q_END_NS, 60 * Q_END_NS) + nested = span2toc(12 * Q_END_NS, 20 * Q_END_NS) + got = toc_normalize(u64([b, nested, a])) + assert got.size == 1 + s, e = toc2time(int(got[0])) + assert (s, e) == (toc2time(a)[0], toc2time(b)[1]) + + +# ── Q2: timestamps ────────────────────────────────────────────────────── + + +def test_absorption_boundaries_are_exact(): + r = span2toc(50 * Q_END_NS, 70 * Q_END_NS) + s, e = toc2time(r) + at_start, last_in, at_end, before = time2toc(u64([s, e - 1, e, s - 1])) + assert toc_normalize(u64([r, at_start])).tolist() == [r] + assert toc_normalize(u64([r, last_in])).tolist() == [r] + # The envelope end is exclusive: the instant at e is outside. + assert toc_normalize(u64([r, at_end])).tolist() == [r, at_end] + assert toc_normalize(u64([r, before])).tolist() == [before, r] + + +def test_timestamps_never_merge_and_equal_ones_dedupe(): + t = 9 * Q_END_NS + 3 + a, b = time2toc(u64([t, t + 1])) + assert toc_normalize(u64([b, a])).tolist() == [a, b] + got = toc_normalize(u64([a, a, a])) + assert got.tolist() == [a] + assert not toc_is_range(got)[0] + + +# ── canonical-form laws ───────────────────────────────────────────────── + + +def test_empty_and_singletons_pass_through(): + assert toc_normalize(u64([])).tolist() == [] + assert toc_normalize(u64([])).dtype == np.uint64 + t = time2toc(123_456_789) + r = span2toc(5 * Q_END_NS, 6 * Q_END_NS) + assert toc_normalize(u64([t])).tolist() == [t] + assert toc_normalize(u64([r])).tolist() == [r] + + +def test_order_independent_and_idempotent(): + rng = np.random.default_rng(198) + shrank = 0 + for _ in range(50): + words = rand_words(rng, int(rng.integers(1, 12))) + reference = toc_normalize(words) + shrank += reference.size < words.size + assert toc_normalize(reference).tolist() == reference.tolist() + for _ in range(3): + assert (toc_normalize(rng.permutation(words)).tolist() + == reference.tolist()) + # Guard against a vacuous generator: the laws are only interesting on + # sets where something actually merged or was absorbed. + assert shrank, "no merge or absorption exercised" + + +def test_coverage_is_preserved_exactly(): + # Membership at every decoded bound and its neighbors agrees between + # the raw set and its canonical form — coverage-identical. + rng = np.random.default_rng(177) + shrank = 0 + for _ in range(30): + words = rand_words(rng, int(rng.integers(1, 10))) + canon = toc_normalize(words) + shrank += canon.size < words.size + starts, ends = toc2time(words) + probes = set() + for s, e in zip(starts.tolist(), ends.tolist()): + probes.update((max(s - 1, 0), s, s + 1, max(e - 1, 0), e, e + 1)) + for t in probes: + assert covered(words, t) == covered(canon, t) + assert shrank, "no merge or absorption exercised" + + +def test_canonical_output_is_sorted_and_duplicate_free(): + rng = np.random.default_rng(41) + shrank = 0 + for _ in range(30): + words = rand_words(rng, int(rng.integers(1, 14))) + canon = toc_normalize(words) + shrank += canon.size < words.size + # Compare, never subtract: np.diff on uint64 wraps, so a descending + # pair reads as a huge positive difference and slips through. + assert np.all(canon[1:] > canon[:-1]) + assert shrank, "no merge or absorption exercised" + + +def test_validation_matches_the_word_ops(): + with pytest.raises(ValueError, match="integer-typed"): + toc_normalize(np.array([1.5, 2.5])) + with pytest.raises(ValueError, match="non-negative"): + toc_normalize(np.array([-1, 2])) + + +def test_scalar_input_yields_the_one_word_set(): + t = time2toc(42) + assert toc_normalize(t).tolist() == [t] + + +# ── toc_and (issue #198 phase 2) ──────────────────────────────────────── + + +def test_golden_and_calendar_overlap_is_exact(): + # Two campaigns share exactly their overlap week: the intersection + # bounds are b's decoded start and a's decoded end, verbatim (Q3 grid + # closure — no rounding arm). + a = span2toc(from_datetime64("2020-03-01"), from_datetime64("2020-03-15")) + b = span2toc(from_datetime64("2020-03-10"), from_datetime64("2020-04-01")) + both = toc_and([a], [b]) + assert both.size == 1 + s, e = toc2time(int(both[0])) + assert s == toc2time(b)[0] and e == toc2time(a)[1] + assert s % Q_START_NS == 0 and e % Q_END_NS == 0 + + +def test_and_disjoint_and_abutting_share_nothing(): + a = span2toc(3 * Q_START_NS, 8 * Q_END_NS - 5) + abutting = span2toc(16 * Q_START_NS + 1, 20 * Q_END_NS - 5) + assert toc2time(a)[1] == toc2time(abutting)[0] + assert toc_and([a], [abutting]).tolist() == [] + assert toc_and([a], [span2toc(100 * Q_END_NS, 200 * Q_END_NS)]).tolist() == [] + + +def test_and_timestamp_survival_is_exact(): + r = span2toc(50 * Q_END_NS, 70 * Q_END_NS) + s, e = toc2time(r) + inside, at_end = time2toc(u64([e - 1, e])) + assert toc_and([inside], [r]).tolist() == [inside] + assert toc_and([r], [inside]).tolist() == [inside] + assert toc_and([at_end], [r]).tolist() == [] + assert toc_and([inside], [inside]).tolist() == [inside] + assert toc_and([inside], [at_end]).tolist() == [] + + +def test_and_accepts_raw_word_sets(): + rng = np.random.default_rng(3) + hits = 0 + for _ in range(20): + base = np.uint64(int(rng.integers(0, 1 << 25)) * Q_END_NS) + a = rand_words(rng, int(rng.integers(1, 10)), base) + b = rand_words(rng, int(rng.integers(1, 10)), base) + both = toc_and(a, b) + hits += both.size + assert (both.tolist() + == toc_and(toc_normalize(a), toc_normalize(b)).tolist()) + assert hits > 0, "no nonempty intersection exercised" + + +def test_and_laws_identity_commutativity_empty(): + rng = np.random.default_rng(17) + hits = 0 + for _ in range(30): + base = np.uint64(int(rng.integers(0, 1 << 25)) * Q_END_NS) + a = rand_words(rng, int(rng.integers(1, 10)), base) + b = rand_words(rng, int(rng.integers(1, 10)), base) + hits += toc_and(a, b).size + assert toc_and(a, a).tolist() == toc_normalize(a).tolist() + assert toc_and(a, b).tolist() == toc_and(b, a).tolist() + assert toc_and(a, u64([])).tolist() == [] + assert toc_and(u64([]), b).tolist() == [] + # The shared ``base`` is what makes the operands meet at all (see + # ``rand_words``); without this guard a generator tweak reduces the + # laws to empty == empty and they pass vacuously. + assert hits > 0, "no nonempty intersection exercised" + + +def test_and_membership_matches_both_sides(): + # The defining property: covered by A ∩ B iff covered by A and by B. + rng = np.random.default_rng(29) + hits = 0 + for _ in range(30): + base = np.uint64(int(rng.integers(0, 1 << 25)) * Q_END_NS) + a = rand_words(rng, int(rng.integers(1, 8)), base) + b = rand_words(rng, int(rng.integers(1, 8)), base) + both = toc_and(a, b) + hits += both.size + starts_a, ends_a = toc2time(a) + starts_b, ends_b = toc2time(b) + probes = set() + for s, e in zip(np.append(starts_a, starts_b).tolist(), + np.append(ends_a, ends_b).tolist()): + probes.update((max(s - 1, 0), s, s + 1, max(e - 1, 0), e, e + 1)) + for t in probes: + assert covered(both, t) == (covered(a, t) and covered(b, t)) + assert hits > 0, "no nonempty intersection exercised" + + +def test_and_output_is_canonical(): + rng = np.random.default_rng(31) + hits = 0 + for _ in range(30): + base = np.uint64(int(rng.integers(0, 1 << 25)) * Q_END_NS) + both = toc_and(rand_words(rng, int(rng.integers(1, 10)), base), + rand_words(rng, int(rng.integers(1, 10)), base)) + hits += both.size + assert toc_normalize(both).tolist() == both.tolist() + assert bool(np.all(both[1:] > both[:-1])) + assert hits > 0, "no nonempty intersection exercised" + + +def test_and_validation_matches_the_word_ops(): + with pytest.raises(ValueError, match="integer-typed"): + toc_and(np.array([1.5]), u64([1])) + with pytest.raises(ValueError, match="non-negative"): + toc_and(u64([1]), np.array([-1])) + assert toc_and(u64([]), u64([])).dtype == np.uint64 diff --git a/mortie/toc_object.py b/mortie/toc_object.py new file mode 100644 index 00000000..898e4212 --- /dev/null +++ b/mortie/toc_object.py @@ -0,0 +1,562 @@ +"""The time-first ``Toc`` object over the toc kernel (issue #198). + +mortie's temporal-coverage surface is two layers, the same deliberate split +:mod:`mortie.moc_object` documents for space: + +* the **kernel** -- the free ``toc_*`` functions in :mod:`mortie._toc`, words + in and words out, unchanged and un-deprecated. Array-first consumers + (zagg's per-cell folds, the segmented :func:`~mortie.tocs_reduce`) keep + calling them on plain ndarrays at zero wrapping cost. +* the **object** -- :class:`Toc`, here. It is ergonomics and nothing else: a + thin view over the canonical ``uint64`` word set + (:func:`~mortie.toc_normalize`'s sorted maximal merges), never a new + representation. **Every public method is a single delegation to a kernel + function.** There is no algebra in this module; a method body that is not + one kernel call is a bug, not a feature. + +All three object methods delegate to the same kernel, +:func:`~mortie.toc_and` -- the one set operation the issue #177 call-site +audit ruled in. The method names are the MOCpy-adjacent vocabulary (issue +#198 commitment 6), so they deliberately do **not** line up with the batch +layer's :func:`~mortie.toc_overlaps` / :func:`~mortie.toc_contains`, which +stay un-deprecated and ask a different question: those take a +``[q_start_ns, q_end_ns)`` query window and answer elementwise, per word, +while :meth:`Toc.overlaps` / :meth:`Toc.contains` compare two whole covers +and answer once. "The object method is the kernel of the same name with +``self`` bound" is the wrong reading here; each method docstring names the +kernel it actually calls. + +The interchange format stays the array. :meth:`Toc.__toc_words__` hands back +the canonical words -- the temporal sibling of ``__morton_moc__()`` -- and any +object exposing that dunder is accepted wherever a ``Toc`` is: mortie owns the +word grammar, downstream stores own their own encodings, and the two meet at a +plain ``uint64`` array with neither importing the other's private grammar. + +**Conservative directions.** Every predicate below is *envelope* algebra, not +data algebra. A range word's decoded envelope dilates the real interval it +was encoded from (starts floor to the 2^31 ns grid, ends ceil to 2^32 ns), so +the covered timeline is always a superset of the real one, on both sides of +the comparison. Read the answers accordingly; each method's docstring points +back here. + +| call | the question it answers exactly | as a question about the real data | +| --- | --- | --- | +| `a.overlaps(b)` | do the two covers share any decoded instant? | may say `True` for data that only comes within a quantum (~2-4 s) of each other; a `False` is decisive. The safe direction for "must I read this store?" -- never a false skip. | +| `a.contains(b)` | is `b`'s decoded coverage inside `a`'s? | may err either way by up to a quantum at span edges (each side's envelope dilates its own data). Exact for the question that matters -- "will a store whose coverage is `a` answer a query for `b`?" | +| `a & b`, `a.intersection(b)` | the canonical cover of the shared coverage | never under-covers the true intersection (`A ⊇ X` and `B ⊇ Y` imply `A ∩ B ⊇ X ∩ Y`); may over-cover near piece edges by up to one quantum per side, inherited from the operands. | +| either side empty | an empty cover is contained in everything and overlaps nothing | `a.contains(empty)` is `True` (vacuously -- there is no coverage of `empty` outside `a`) while `a.overlaps(empty)` is `False`. The two predicates disagree here by definition, not by accident: ask `contains` about coverage and `overlaps` about work to do. | +| `a == b` | identical canonical words | **not** equality of the underlying timeline: an instant and the degenerate span at the same time encode different words and compare unequal. | +""" + +import warnings + +import numpy as np + +from . import _toc +from ._toc import ( + from_datetime64, + span2toc, + time2toc, + to_datetime64, + toc2time, + toc_and, + toc_is_range, + toc_normalize, +) + +# The public surface of the former ``mortie.toc`` submodule -- what the +# migration shim below still resolves (with a DeprecationWarning) for one minor +# version. A frozen historical roster (the 0.9.9 surface: thirteen +# module-level functions plus the four grid/epoch constants), not a live view +# of the kernel: pinned as a literal, and pinned in the tests against that same +# history, so a name added to the kernel later does not join the deprecated +# namespace. ``toc_normalize`` and ``toc_and`` are deliberately absent: born +# in this same PR, they never had a ``mortie.toc.`` spelling any release +# could be using, so there is nothing there to deprecate. +_KERNEL_NAMES = ( + "GPS_EPOCH_NS", + "Q_END_NS", + "Q_START_NS", + "TOC_MAX_NS", + "from_datetime64", + "from_gps_ns", + "span2toc", + "time2toc", + "to_datetime64", + "to_gps_ns", + "toc2time", + "toc_contains", + "toc_is_range", + "toc_merge", + "toc_overlaps", + "toc_reduce", + "tocs_reduce", +) + + +def _words(operand): + """Canonical ``uint64`` toc words of a set-operation operand. + + *Every* operand is canonicalized here (:func:`~mortie.toc_normalize`), + protocol carriers included: nothing obliges a third-party + ``__toc_words__()`` to hand back canonical form, and the equality-shaped + delegation in :meth:`Toc.contains` compares canonical form against + canonical form or it false-negatives. Normalizing already-canonical + words is a fixpoint, so the :class:`Toc`-to-:class:`Toc` path is + unchanged. + + Parameters + ---------- + operand : Toc or array_like + A :class:`Toc`, anything exposing ``__toc_words__()``, or a raw toc + word array. + + Returns + ------- + numpy.ndarray + The operand's canonical words as a 1-D ``uint64`` array. + """ + protocol = getattr(operand, "__toc_words__", None) + if protocol is not None: + operand = protocol() + return toc_normalize(operand) + + +def _as_time_ns(value, name): + """Coerce one time endpoint to internal ns, refusing numeric input. + + Integers are always *words* in the :class:`Toc` constructor, and numpy + would otherwise read a bare number as naive ns since 1970 -- silently, on + the wrong epoch -- so numeric endpoints are refused rather than guessed. + + Parameters + ---------- + value : str, datetime64, or array_like + The endpoint(s): anything :func:`~mortie.from_datetime64` accepts. + name : str + Argument name for the error message. + + Returns + ------- + int or numpy.ndarray + Internal ns since 1850-01-01 (scalar in -> ``int`` out). + + Raises + ------ + ValueError + If the endpoint is numeric rather than datetime64 / ISO string, or + :func:`~mortie.from_datetime64` rejects it. + """ + if np.asarray(value).dtype.kind in "iufcb": + raise ValueError( + f"{name} must be a datetime64 / ISO-string time, got a numeric " + f"value; integers are toc words in this constructor, and " + f"internal-ns integers go through time2toc/span2toc explicitly") + return from_datetime64(value) + + +def _source_words(source, end): + """Resolve a constructor source to its toc words, before normalization. + + Parameters + ---------- + source : object + Any :class:`Toc` constructor source; see that class for the matrix. + end : str, datetime64, array_like, or None + The span-end side, for the time-pair form only. + + Returns + ------- + int or numpy.ndarray + Toc words (``uint64``), not yet normalized. + + Raises + ------ + ValueError + If ``end`` is given for a source that is already words -- it + completes a time pair, which such a source is not, so it is refused + rather than ignored -- or if an integer source is not 1-D: an + ``(N, 2)`` array of ``[start, end]`` pairs is a natural thing to have + in hand, and raveling it into words yields a plausible-looking cover + of garbage, so the shape is gated the way :class:`~mortie.Moc` gates + its own words branch. + + Notes + ----- + A size-0 non-datetime, non-string source is the **empty word set** in + every spelling -- ``[]``, ``()``, ``np.array([])`` and a typed + ``uint64`` empty alike. Untyped empty containers are ``float64`` by + numpy's default, so without that route they would reach the time path + and be refused as *numeric*, advising the caller to do exactly what they + were already doing. + """ + protocol = getattr(source, "__toc_words__", None) + if protocol is not None: + if end is not None: + raise ValueError( + "end= completes a time pair and applies to a time source " + "only; this source is already a cover (__toc_words__)") + return np.asarray(protocol(), dtype=np.uint64).ravel() + arr = np.asarray(source) + if arr.dtype.kind in "iu": + if end is not None: + raise ValueError( + "end= completes a time pair and applies to a time source " + "only; this source is already toc words") + if arr.ndim > 1: + raise ValueError( + f"toc words must be a 1-D integer array or a single int, got " + f"shape {arr.shape}; an (N, 2) array of time pairs goes " + f"through Toc(starts, ends)") + return source + if arr.size == 0 and arr.dtype.kind not in "MUSO": + # An untyped empty container ([], (), np.array([])) is float64 by + # numpy's default, so it would fall to the time path and be reported + # as *numeric*. It is not numeric, it is empty: the empty cover. + if end is not None: + raise ValueError( + "end= completes a time pair and applies to a time source " + "only; this source is already toc words") + return arr.astype(np.uint64) + if end is None: + return time2toc(_as_time_ns(source, "source")) + return span2toc(_as_time_ns(source, "source"), _as_time_ns(end, "end")) + + +_COVERED_UNITS = ( + ("years", 31_557_600 * 10**9), + ("days", 86_400 * 10**9), + ("hours", 3_600 * 10**9), + ("minutes", 60 * 10**9), + ("seconds", 10**9), + ("ms", 10**6), + ("us", 10**3), +) + + +def _covered_display(total_ns): + """Render a covered duration in the largest unit it fills. + + Parameters + ---------- + total_ns : int + Total covered nanoseconds. + + Returns + ------- + str + The duration with its unit, e.g. ``"517 days"``. + """ + for unit, scale in _COVERED_UNITS: + if total_ns >= scale: + return f"{total_ns / scale:.4g} {unit}" + return f"{total_ns} ns" + + +class Toc: + """A temporal coverage as an object -- gappy time coverage that reads as time. + + A thin view over the canonical ``uint64`` toc word set and nothing more: + the words *are* the coverage, this class is the ergonomics, and every + public method is a single delegation to a kernel function (see the module + docstring for the two-layer rule and the conservative-direction table the + methods share). + + The canonical form is a word **set**, not a single envelope (issue #198): + a store observed in campaigns has *gappy* coverage, and one merged + envelope papers over the gaps exactly where they are most informative -- + k disjoint spans keep them. The words are normalized eagerly + (:func:`~mortie.toc_normalize`) and stored read-only, which makes ``==`` + and :func:`hash` well defined; the instance itself is immutable. + + Normalization is **lossy toward coverage, one way** (the #177 Q2 ruling): + a timestamp subsumed by a range's decoded span adds no coverage and is + absorbed at construction -- which subsumed instants existed, and how many + times, is dropped. Exact instants live in the sibling word arrays a + cover is built from; a cover can be rebuilt from the arrays, never the + arrays from a cover. A timestamp no range subsumes survives + bit-identical as an exact degenerate member, and a surviving decoded gap + is never bridged (Q1). + + Union needs no method: construction normalizes, so + ``Toc(np.append(a.words, b.words))`` is the union. The #177 call-site + audit ruled exactly one set operation in (:func:`~mortie.toc_and`, the + :meth:`intersection` / ``&`` delegate); the difference and + symmetric-difference directions deliberately do not ship -- conservative + covers under-cover on subtraction, and no audited call site exists. + + Parameters + ---------- + source : str, datetime64, array_like, or Toc + What the coverage is. A UTC instant -- an ISO 8601 string or + ``numpy.datetime64`` -- or an array of them, encoded exactly via + :func:`~mortie.time2toc`; with ``end`` given, the start side of one + or more closed real intervals ``[source, end]``, enveloped + conservatively via :func:`~mortie.span2toc` (``source`` and ``end`` + broadcast). A 1-D integer array (or a single int) is taken as toc + **words** as given; any object exposing ``__toc_words__()``, another + :class:`Toc` included, contributes its canonical words. Integers are + always words: internal nanoseconds go through + :func:`~mortie.time2toc` / :func:`~mortie.span2toc` explicitly, so a + bare number can never be misread as a time on the wrong epoch. Any + size-0 source that is not datetimes or strings -- ``[]``, ``()``, + ``np.array([])``, a typed empty -- is the empty cover. + end : str, datetime64, or array_like, optional + End side of the time-pair form, same forms as ``source``. Only a + time source takes it: with a words or protocol source it is refused + rather than ignored. + + Raises + ------ + ValueError + If a time endpoint is numeric rather than datetime64 / ISO string, a + span is inverted or outside the representable epoch range (see + :func:`~mortie.span2toc` / :func:`~mortie.from_datetime64`), words + are negative, non-integer-typed or not 1-D, or ``end`` is given for a + source that is already words. + + See Also + -------- + mortie.toc_normalize : the eager canonicalization applied to every + instance. + mortie.toc_and : the intersection kernel the set methods delegate to. + + Examples + -------- + >>> import mortie + >>> when = mortie.Toc("2020-01-01", "2021-06-01") + >>> mortie.Toc("2020-06-15").overlaps(when) + True + >>> when.contains(mortie.Toc("2020-03-01", "2020-04-01")) + True + """ + + __slots__ = ("words",) + + def __init__(self, source, end=None): + words = toc_normalize(_source_words(source, end)) + words.setflags(write=False) + object.__setattr__(self, "words", words) + + def __setattr__(self, name, value): + """Refuse attribute assignment -- a Toc is immutable.""" + raise AttributeError( + "Toc is immutable (its hash is its words); build a new one instead" + ) + + def __delattr__(self, name): + """Refuse attribute deletion -- a Toc is immutable.""" + raise AttributeError("Toc is immutable; build a new one instead") + + def __reduce__(self): + """Rebuild through the constructor -- pickle and copy cannot set slots. + + A ``__slots__`` class is restored by assigning its slot state, which + the immutability guard above refuses; reconstructing from the words + instead keeps ``Toc`` picklable (workers marshal their arguments) and + deep-copyable at the cost of one ``toc_normalize`` on already-canonical + words. + + Returns + ------- + tuple + The ``(callable, args)`` pair the pickle protocol rebuilds from. + """ + return (Toc, (self.words,)) + + def overlaps(self, other): + """Whether this cover and ``other`` share any coverage. + + Envelope algebra, not data algebra -- see the module docstring's + conservative-direction table: ``True`` can mean "within a quantum of + each other", ``False`` is decisive. + + Parameters + ---------- + other : Toc or array_like + The cover to test against. + + Returns + ------- + bool + ``True`` if the two covers share any decoded instant. + + See Also + -------- + mortie.toc_and : the kernel this delegates to. + mortie.toc_overlaps : the batch layer's *window* predicate of the + same name, a different question -- which individual words + intersect a ``[q_start_ns, q_end_ns)`` window, elementwise. + """ + return toc_and(self.words, _words(other)).size > 0 + + def contains(self, other): + """Whether ``other``'s coverage lies entirely inside this cover. + + Envelope algebra, not data algebra -- see the module docstring's + conservative-direction table. This is the "will a store whose + coverage is ``self`` answer a query for ``other``?" test: the + intersection with ``other`` must give ``other`` back whole, and both + sides of that comparison are canonical -- the kernel's output by + construction, the operand because :func:`_words` canonicalizes every + operand it is handed -- so word equality is coverage equality. The + single-delegation shape costs a second :func:`_words` call (a foreign + ``__toc_words__()`` is therefore invoked twice per test); binding it + to a local would be a second statement, which the delegation pin + refuses. + + Parameters + ---------- + other : Toc or array_like + The cover to test for containment. + + Returns + ------- + bool + ``True`` if ``other`` adds no coverage outside this cover. + + See Also + -------- + mortie.toc_and : the kernel this delegates to. + mortie.toc_contains : the batch layer's *window* predicate of the + same name, and the mirror direction -- which individual words + fall inside a ``[q_start_ns, q_end_ns)`` window, elementwise, + rather than whether one cover sits inside another. + """ + return np.array_equal(toc_and(self.words, _words(other)), _words(other)) + + def intersection(self, other): + """Intersect with ``other``: the canonical cover of the shared coverage (``&``). + + Never under-covers the true intersection and may over-cover near + piece edges by up to one quantum per side, inherited from the + operands' envelopes -- see the module docstring's table. + + Parameters + ---------- + other : Toc or array_like + The cover to intersect with. + + Returns + ------- + Toc + The intersection cover. + """ + return Toc(toc_and(self.words, _words(other))) + + __and__ = intersection + + def __toc_words__(self): + """Canonical toc words -- the interchange protocol (issue #198). + + Returns + ------- + numpy.ndarray + The read-only ``uint64`` word array backing this cover. + """ + return self.words + + def __eq__(self, other): + """Compare canonical words -- word identity, not timeline equality.""" + if not isinstance(other, Toc): + return NotImplemented + return np.array_equal(self.words, other.words) + + def __hash__(self): + """Hash the canonical words; sound because a Toc is immutable.""" + return hash(self.words.tobytes()) + + def __len__(self): + """Count the words (disjoint spans and free instants) in the cover.""" + return int(self.words.size) + + def __iter__(self): + """Iterate the cover's toc words.""" + return iter(self.words) + + def __repr__(self): + """Show the span count, the UTC extent, and the covered duration. + + Both printed bounds round **outward** to the second -- the start + floors, the end ceils -- so the extent shown is never narrower than + what the cover covers. A decoded end sits on the 2^32 ns grid + (~4.295 s) and so is essentially never a whole second; flooring it + would print a time strictly inside the envelope, the one direction + the module docstring's conservative-direction table rules out. + """ + if self.words.size == 0: + return "Toc(0 spans)" + ranges = toc_is_range(self.words) + n_ranges, n_instants = int(ranges.sum()), int((~ranges).sum()) + kinds = [] + if n_ranges: + kinds.append(f"{n_ranges} range" + ("s" if n_ranges > 1 else "")) + if n_instants: + kinds.append( + f"{n_instants} instant" + ("s" if n_instants > 1 else "")) + starts, ends = toc2time(self.words) + first = np.datetime64(to_datetime64(int(starts.min())), "s") + end = to_datetime64(int(ends.max())) + last = np.datetime64(end, "s") + if last != end: + last += np.timedelta64(1, "s") + covered = _covered_display(int((ends - starts).sum())) + return (f"Toc({' + '.join(kinds)}, {first} to {last}, " + f"{covered} covered)") + + +class _TocNamespace: + """Callable stand-in for the retired ``mortie.toc`` submodule (issue #198). + + Calling it builds a :class:`Toc`; attribute access to the kernel functions + and constants the submodule used to hold still resolves, with a + :class:`DeprecationWarning`, for one minor version. Statement-form + ``import mortie.toc`` and ``from mortie.toc import ...`` break at the + rename -- the module is gone -- which is why the shim covers attribute + access and not the import system. + + The warning is raised on **every** access and the shim keeps no state, so + policy is left entirely to the warnings filters. Under the interpreter + defaults that means ``DeprecationWarning`` is ignored outside ``__main__`` + and ``stacklevel=2`` charges it to the *calling* module, so a consumer + module sees nothing until its filters ask -- ``-W``, ``PYTHONWARNINGS``, or + a test runner that enables the category (pytest does). ``always`` and + ``error`` filters then see every occurrence, and a + :func:`warnings.catch_warnings` block starts from a fresh registry; with + no shim-side budget to exhaust, a downstream test suite still observes the + warning however late in the process it runs. + """ + + __slots__ = () + + def __call__(self, source, end=None): + """Build a :class:`Toc`; see that class for the argument matrix.""" + return Toc(source, end) + + def __getattr__(self, name): + """Resolve a retired submodule attribute, warning on every access.""" + if name not in _KERNEL_NAMES: + raise AttributeError( + f"'mortie.toc' is the Toc constructor (issue #198), not the " + f"old submodule, and has no attribute {name!r}" + ) + warnings.warn( + f"mortie.toc.{name} is deprecated: mortie.toc is now the Toc " + f"constructor, not a module. Use the top-level mortie.{name} " + f"instead; this shim is removed in the next minor release.", + DeprecationWarning, + stacklevel=2, + ) + return getattr(_toc, name) + + def __dir__(self): + """List the deprecated kernel names this shim still resolves.""" + return sorted(_KERNEL_NAMES) + + def __repr__(self): + """Say what this object is, so `mortie.toc` is not mistaken for a module.""" + return " is deprecated>" + + +toc = _TocNamespace() +"""The :class:`Toc` constructor, bound where the submodule used to be.""" diff --git a/pyproject.toml b/pyproject.toml index 4d9f19db..cfcfb540 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -144,9 +144,10 @@ exclude = [ # tests (mirroring ruff's per-file-ignores), plus private '\.tests\.', # members, dunders and private modules: CLAUDE.md '\.conftest', # mandates numpydoc for the *public* surface '\._', - # private top-level modules lint without the package prefix; _moc.py is - # public surface behind a private filename, so it stays in the gate (#196) - '^_(?!moc\.)', + # private top-level modules lint without the package prefix; _moc.py and + # _toc.py are public surface behind private filenames, so they stay in the + # gate (#196, #198) + '^_(?!moc\.|toc\.)', ] [tool.ruff] diff --git a/src_rust/src/lib.rs b/src_rust/src/lib.rs index 72e99eb3..3a3e06df 100644 --- a/src_rust/src/lib.rs +++ b/src_rust/src/lib.rs @@ -1963,6 +1963,8 @@ fn _rustie(m: &Bound<'_, PyModule>) -> PyResult<()> { m.add_function(wrap_pyfunction!(toc::rust_tocs_reduce, m)?)?; m.add_function(wrap_pyfunction!(toc::rust_toc_is_range, m)?)?; m.add_function(wrap_pyfunction!(toc::rust_toc_window, m)?)?; + m.add_function(wrap_pyfunction!(toc::set_ops::rust_toc_normalize, m)?)?; + m.add_function(wrap_pyfunction!(toc::set_ops::rust_toc_and, m)?)?; m.add_function(wrap_pyfunction!(rust_wkb_rings, m)?)?; m.add_function(wrap_pyfunction!(rust_wkbs_coverage_mocs, m)?)?; #[cfg(feature = "descent-stats")] diff --git a/src_rust/src/toc.rs b/src_rust/src/toc.rs index 6650ea75..7597f4a5 100644 --- a/src_rust/src/toc.rs +++ b/src_rust/src/toc.rs @@ -4,7 +4,7 @@ //! quantized, conservative **time range**. All internal times are u64 //! nanoseconds since **1850-01-01T00:00:00** on a continuous, leap-free, //! GPS-aligned timescale (leap seconds exist only at the UTC boundary in -//! `mortie/toc.py`). This is *not* an IVOA T-MOC — see the design record. +//! `mortie/_toc.py`). This is *not* an IVOA T-MOC — see the design record. //! //! Layout (decision ledger on issue #175; design rationale on //! englacial/zagg#410): @@ -38,6 +38,8 @@ //! and fixture-pinned below; how they are computed (parallelism, chunking, //! error text) is not. +pub mod set_ops; + use numpy::{IntoPyArray, PyArrayMethods, PyReadonlyArray1}; use pyo3::exceptions::PyValueError; use pyo3::prelude::*; @@ -161,7 +163,7 @@ pub fn merge(a: u64, b: u64) -> u64 { // --------------------------------------------------------------------------- // Python bindings (issue #175 phase 2). Array-in/array-out over the scalar // kernel; rayon under allow_threads per the house pattern. Validation -// (dtype, shape, scalar/array symmetry) lives in mortie/toc.py. +// (dtype, shape, scalar/array symmetry) lives in mortie/_toc.py. // --------------------------------------------------------------------------- /// Borrow a contiguous u64 numpy buffer, copying only when non-contiguous. @@ -467,7 +469,7 @@ pub fn rust_toc_is_range(py: Python<'_>, words: PyReadonlyArray1) -> PyResu /// Window predicates: `mode = 0` overlaps, `mode = 1` contains (vectorized). /// /// Both test the word's conservative encoded bounds against the half-open -/// query window `[q_start, q_end)`; see `mortie/toc.py` for the +/// query window `[q_start, q_end)`; see `mortie/_toc.py` for the /// over/under-report semantics. #[pyfunction] pub fn rust_toc_window( diff --git a/src_rust/src/toc/set_ops.rs b/src_rust/src/toc/set_ops.rs new file mode 100644 index 00000000..35fd2b3d --- /dev/null +++ b/src_rust/src/toc/set_ops.rs @@ -0,0 +1,774 @@ +//! toc set algebra — the canonical cover form (issues #177 / #198). +//! +//! [`normalize`] collapses a toc word set into its **canonical cover**: the +//! unique sorted word set with the same decoded coverage — maximal merged +//! ranges plus the exact instants no range subsumes. It is the equality and +//! construction basis for the `Toc` object (issue #198) and the root-word-set +//! normalizer for store sidecars (englacial/zagg#480). +//! +//! The rulings this implements are recorded on issue #177 +//! (, +//! confirmed in +//! ): +//! +//! - **Q1 — merge on decoded bounds; never bridge a decoded gap.** Two range +//! words coalesce iff their decoded half-open `[start, end)` envelopes +//! overlap or abut *exactly* (`end_a == start_b` in exact ns — well-defined, +//! both grids decode to exact integers). A surviving decoded gap is a floor +//! on the true gap (outward rounding only shrinks apparent gaps), so it is +//! real information and is preserved however small. Because decoded starts +//! sit on the 2^31 ns grid and decoded ends on the 2^32 ns grid, a merged +//! union's bounds are min/max of on-grid values — still on-grid, so the +//! merge is **exact**: no rounding arm exists anywhere in this module. +//! - **Q2 — coverage semantics.** A timestamp inside a covering range's +//! decoded span adds no coverage and is absorbed; a timestamp no range +//! subsumes survives **bit-identical** as an exact degenerate member. +//! Equal timestamps deduplicate. Timestamps never merge with each other +//! or extend a range — re-encoding an instant into a range would round +//! outward and *change* coverage, which normalize never does. +//! +//! Like [`super::merge`], everything here is total over arbitrary bit +//! patterns: junk words are garbage in, garbage out — deterministic, never a +//! panic. The canonical form is normative for encoder-produced words and +//! pinned by the fixtures below. + +use numpy::{IntoPyArray, PyArrayMethods, PyReadonlyArray1}; +use pyo3::prelude::*; + +use super::{decode, FLAG_MASK, LOW_MASK}; + +/// Re-splice an exact instant into its timestamp word, with no domain check. +/// +/// Bit-identical to [`super::encode_timestamp`] on its domain; total (no +/// ceiling rejection) so that normalize can round-trip a junk "timestamp" +/// unchanged instead of erroring on input it merely passes through. +#[inline] +fn splice_timestamp(t_ns: u64) -> u64 { + ((t_ns >> 31) << 32) | FLAG_MASK | (t_ns & LOW_MASK) +} + +/// Re-encode a decoded on-grid envelope `[start_ns, end_ns)` as a range word. +/// +/// The inverse of [`decode`] for range words: exact (no rounding) because +/// normalize only ever holds bounds that came off the grids — `start_ns` a +/// multiple of 2^31, `end_ns` a multiple of 2^32 — and min/max keep them so. +#[inline] +fn encode_envelope(start_ns: u64, end_ns: u64) -> u64 { + ((start_ns >> 31) << 32) | (end_ns >> 32) +} + +/// A cover in canonical parts (the shared intermediate of the set ops). +/// +/// `ranges` are maximal merged decoded envelopes — exact half-open +/// `[start_ns, end_ns)` intervals, sorted, pairwise separated by a real +/// decoded gap. `stamps` are the exact instants no range subsumes, sorted +/// and deduplicated. +pub(crate) struct Canonical { + pub(crate) ranges: Vec<(u64, u64)>, + pub(crate) stamps: Vec, +} + +/// Decompose a word set into canonical parts. +/// +/// One sorted-stream pass per part, the house pattern of +/// [`crate::moc::normalize`]: decode and split, sort ranges and sweep-merge +/// overlap-or-abut (Q1), then co-walk the sorted instants past the merged +/// ranges, dropping exactly the subsumed ones (Q2). +pub(crate) fn canonicalize(words: &[u64]) -> Canonical { + let mut ranges: Vec<(u64, u64)> = Vec::new(); + let mut stamps: Vec = Vec::new(); + for &w in words { + let (s, e, is_rng) = decode(w); + if is_rng { + ranges.push((s, e)); + } else { + stamps.push(s); + } + } + ranges.sort_unstable(); + let mut merged: Vec<(u64, u64)> = Vec::with_capacity(ranges.len()); + for (s, e) in ranges { + match merged.last_mut() { + // Overlap or exact abutment: `s <= last_end` in exact ns. A + // strict `<` here would split abutting envelopes; a tolerance + // would bridge a surviving decoded gap. Both are wrong (Q1). + Some((_, last_end)) if s <= *last_end => *last_end = (*last_end).max(e), + _ => merged.push((s, e)), + } + } + stamps.sort_unstable(); + stamps.dedup(); + let kept = select_stamps(&stamps, &merged, false); + Canonical { + ranges: merged, + stamps: kept, + } +} + +/// Select the sorted instants inside `ranges` (`want_inside`) or outside them. +/// +/// The one stamp/range walk both set ops share: canonicalize wants the +/// outside half (Q2 absorption) and [`intersect`] the inside half (an instant +/// survives intersection with a cover that subsumes it). Only the wanted +/// half is built — every call site discards the other, so partitioning into +/// two vectors would allocate and fill a walk's worth of instants nobody +/// reads, on both hot paths. +/// `ranges` is sorted by *start* (ends need not ascend — a junk word decodes +/// to an empty envelope). A range ending at or before t can subsume no +/// later instant either, and `stamps` ascends, so the cursor only ever +/// moves forward; the membership check is then a decision for *all* +/// remaining ranges, because their starts ascend. +fn select_stamps(stamps: &[u64], ranges: &[(u64, u64)], want_inside: bool) -> Vec { + let mut kept = Vec::with_capacity(stamps.len()); + let mut i = 0; + for &t in stamps { + while i < ranges.len() && ranges[i].1 <= t { + i += 1; + } + let inside = i < ranges.len() && ranges[i].0 <= t; + if inside == want_inside { + kept.push(t); + } + } + kept +} + +/// Encode canonical parts back to the canonical word set (sorted u64s). +pub(crate) fn to_words(c: &Canonical) -> Vec { + let mut out: Vec = c + .ranges + .iter() + .map(|&(s, e)| encode_envelope(s, e)) + .chain(c.stamps.iter().map(|&t| splice_timestamp(t))) + .collect(); + out.sort_unstable(); + out +} + +/// Collapse a toc word set into its canonical cover form. +/// +/// Sorted maximal merges: ranges coalesced iff their decoded envelopes +/// overlap or abut exactly (Q1), subsumed instants absorbed and free +/// instants kept bit-identical (Q2). The output's decoded coverage equals +/// the input's **exactly** — order-independent and idempotent. Empty in, +/// empty out. +/// +/// Sortedness and duplicate-freeness are the canonical form over +/// **encoder-produced** words. A junk word can decode to an empty +/// envelope (end below start), which subsumes nothing and does not +/// collapse even against a copy of itself, so junk can come back +/// duplicated — coverage is still exact and the output is still a +/// fixpoint, but junk in is junk out (pinned by +/// `arbitrary_bit_patterns_normalize_without_panicking`). +pub fn normalize(words: &[u64]) -> Vec { + to_words(&canonicalize(words)) +} + +/// Intersect two canonical covers (the shared kernel of [`toc_and`]). +/// +/// Ranges run a two-pointer sweep over the two sorted disjoint families: +/// each surviving piece is `[max(starts), min(ends))` — **exact by grid +/// closure** (Q3): the max of two 2^31-grid starts stays on the start grid +/// and the min of two 2^32-grid ends stays on the end grid, so every +/// intersection bound is exactly representable and no rounding arm exists. +/// Instants survive iff genuinely covered on both sides: a stamp inside the +/// other cover's ranges (the `inside` half of [`select_stamps`]) or present +/// as the identical stamp in both. +/// +/// The output is canonical without a re-normalize, because the inputs are: +/// output ranges are sub-intervals of one side's disjoint non-abutting +/// ranges, separated by the other side's surviving gaps, so they are +/// disjoint and non-abutting; a surviving stamp lies outside its own +/// side's ranges (canonical), hence outside the output ranges those +/// contain; and the three stamp sources cannot overlap — a stamp equal on +/// both sides is inside neither side's ranges, so exactly one source +/// claims each instant and the sorted union is duplicate free. +fn intersect(a: &Canonical, b: &Canonical) -> Canonical { + let mut ranges = Vec::new(); + let (mut i, mut j) = (0, 0); + while i < a.ranges.len() && j < b.ranges.len() { + let (sa, ea) = a.ranges[i]; + let (sb, eb) = b.ranges[j]; + let (s, e) = (sa.max(sb), ea.min(eb)); + if s < e { + ranges.push((s, e)); + } + // Advance whichever side's range ends first (both on a tie): the + // finished range can intersect nothing later on the other side. + if ea <= eb { + i += 1; + } + if eb <= ea { + j += 1; + } + } + let mut stamps = select_stamps(&a.stamps, &b.ranges, true); + stamps.extend(select_stamps(&b.stamps, &a.ranges, true)); + let (mut i, mut j) = (0, 0); + while i < a.stamps.len() && j < b.stamps.len() { + match a.stamps[i].cmp(&b.stamps[j]) { + std::cmp::Ordering::Less => i += 1, + std::cmp::Ordering::Greater => j += 1, + std::cmp::Ordering::Equal => { + stamps.push(a.stamps[i]); + i += 1; + j += 1; + } + } + } + // Each of the three stamp sources is sorted but they interleave, and + // `Canonical` promises sorted stamps to whoever holds one — not just to + // [`to_words`], which would re-sort the whole word vector anyway. + stamps.sort_unstable(); + Canonical { ranges, stamps } +} + +/// Intersect two toc word sets: the canonical cover of the common coverage. +/// +/// Both operands are canonicalized internally (the posture of +/// [`crate::moc::moc_intersects`]), so raw unsorted word sets are accepted. +/// Conservatism is preserved by construction — `A ⊇ X` and `B ⊇ Y` imply +/// `A ∩ B ⊇ X ∩ Y` — and the sweep itself is exact (see [`intersect`]); +/// per Q3, the difference/xor directions deliberately do not ship. +/// Total over junk like [`normalize`]: garbage in, garbage out, no panic. +pub fn toc_and(a: &[u64], b: &[u64]) -> Vec { + to_words(&intersect(&canonicalize(a), &canonicalize(b))) +} + +/// Canonicalize a toc word set: sorted maximal merges (issue #198 phase 1). +/// +/// # Arguments +/// * `words` - Toc words (u64 NumPy array), any order, duplicates allowed +/// +/// # Returns +/// The canonical cover as a sorted u64 NumPy array: maximal merged ranges +/// plus the exact instants no range subsumes. Coverage-identical to the +/// input; see `mortie.toc_normalize` for the direction table. +#[pyfunction] +pub fn rust_toc_normalize(py: Python<'_>, words: PyReadonlyArray1) -> PyResult { + let w = words.to_vec()?; + let out = py.allow_threads(|| normalize(&w)); + Ok(out.into_pyarray_bound(py).into_any().unbind()) +} + +/// Intersect two toc word sets (issue #198 phase 2). +/// +/// # Arguments +/// * `a` - Toc words (u64 NumPy array), any order, duplicates allowed +/// * `b` - Toc words (u64 NumPy array), the other operand +/// +/// # Returns +/// The canonical cover of the common coverage as a sorted u64 NumPy array; +/// see `mortie.toc_and` for the direction table. +#[pyfunction] +pub fn rust_toc_and( + py: Python<'_>, + a: PyReadonlyArray1, + b: PyReadonlyArray1, +) -> PyResult { + let wa = a.to_vec()?; + let wb = b.to_vec()?; + let out = py.allow_threads(|| toc_and(&wa, &wb)); + Ok(out.into_pyarray_bound(py).into_any().unbind()) +} + +// ── tests ──────────────────────────────────────────────────────────────── + +#[cfg(test)] +mod tests { + use super::super::{ + encode_range, encode_timestamp, is_range, Q_END_NS, Q_START_NS, TOC_MAX_NS, + }; + use super::*; + + /// Deterministic PRNG (splitmix64) — no rand dependency. + fn splitmix64(state: &mut u64) -> u64 { + *state = state.wrapping_add(0x9E37_79B9_7F4A_7C15); + let mut z = *state; + z = (z ^ (z >> 30)).wrapping_mul(0xBF58_476D_1CE4_E5B9); + z = (z ^ (z >> 27)).wrapping_mul(0x94D0_49BB_1331_11EB); + z ^ (z >> 31) + } + + fn rand_time(state: &mut u64) -> u64 { + splitmix64(state) % super::super::TOC_MAX_NS + } + + /// A random valid word, timestamps and ranges mixed (short ranges too, + /// so absorptions and near-misses both occur). + fn rand_word(state: &mut u64) -> u64 { + match splitmix64(state) % 4 { + 0 | 1 => encode_timestamp(rand_time(state)).unwrap(), + 2 => { + let a = rand_time(state); + let b = a + splitmix64(state) % (8 * Q_END_NS); + encode_range(a, b.min(super::super::TOC_MAX_NS - 1)).unwrap() + } + _ => { + let (x, y) = (rand_time(state), rand_time(state)); + encode_range(x.min(y), x.max(y)).unwrap() + } + } + } + + /// Reference coverage membership over raw (un-normalized) words. + fn covered(words: &[u64], t: u64) -> bool { + words.iter().any(|&w| { + let (s, e, is_rng) = decode(w); + if is_rng { + s <= t && t < e + } else { + s == t + } + }) + } + + // ── golden fixtures (normative — the canonical form persists) ──────── + + #[test] + fn golden_issue_177_absorption_example() { + // espg's Q2 example (issue #177 decision record): t1, t2 inside a + // covering range R absorb; t3 months later survives bit-identical; + // the gap between R and t3 is preserved. + let r = encode_range(100 * Q_END_NS, 200 * Q_END_NS).unwrap(); + let (rs, re, _) = decode(r); + let t1 = encode_timestamp(rs + 7).unwrap(); // inside R's decoded span + let t2 = encode_timestamp(re - 1).unwrap(); // last covered instant + let t3 = encode_timestamp(5000 * Q_END_NS + 123).unwrap(); // far later + let got = normalize(&[t1, r, t2, t3]); + assert_eq!(got, vec![r, t3]); + } + + #[test] + fn golden_abutting_envelopes_merge_and_a_one_quantum_gap_survives() { + // r1's decoded end (2^32 grid) meets r2's decoded start (2^31 grid) + // exactly: end code 8 -> 8 * 2^32 = start code 16 * 2^31. + let r1 = encode_range(3 * Q_START_NS, 8 * Q_END_NS - 5).unwrap(); + let r2 = encode_range(16 * Q_START_NS + 1, 20 * Q_END_NS - 5).unwrap(); + assert_eq!(decode(r1).1, decode(r2).0, "envelopes abut exactly"); + assert_eq!( + normalize(&[r2, r1]), + vec![encode_range(3 * Q_START_NS, 20 * Q_END_NS - 5).unwrap()] + ); + // One start quantum later the decoded gap is 2^31 ns — the smallest + // gap the grids can express — and it is never bridged. + let r3 = encode_range(17 * Q_START_NS + 1, 20 * Q_END_NS - 5).unwrap(); + assert_eq!(normalize(&[r3, r1]), vec![r1, r3]); + } + + #[test] + fn golden_top_of_span_keeps_the_range_variant() { + // At the top of the span the end field is completely full + // (`end_ns >> 32 == LOW_MASK`): one more bit of end would land on + // bit 31 and re-encode the range as a *timestamp*. `toc.rs` guards + // that edge on the encoder side (`top_of_span_is_rejected`), but + // `set_ops` re-derives every word with its own unchecked + // `encode_envelope`, so pin the maximal end here too. + let r = encode_range(TOC_MAX_NS - 3 * Q_END_NS, TOC_MAX_NS - 1).unwrap(); + assert_eq!(r & LOW_MASK, LOW_MASK, "end field full"); + assert_eq!(decode(r).1, TOC_MAX_NS); + // The last encodable instant is subsumed by r and absorbs; r comes + // back through decode/re-encode bit-identical, still a range. + let t = encode_timestamp(TOC_MAX_NS - 1).unwrap(); + assert_eq!(normalize(&[t, r]), vec![r]); + // Merging an overlapping lower range moves the start and keeps the + // maximal end — the arithmetic that would overflow into the flag. + let lower = encode_range(TOC_MAX_NS - 9 * Q_END_NS, TOC_MAX_NS - 3 * Q_END_NS).unwrap(); + let got = normalize(&[t, r, lower]); + assert_eq!( + got, + vec![encode_range(TOC_MAX_NS - 9 * Q_END_NS, TOC_MAX_NS - 1).unwrap()] + ); + assert!(is_range(got[0])); + assert_eq!(decode(got[0]).1, TOC_MAX_NS); + } + + // ── Q1: range merging ──────────────────────────────────────────────── + + #[test] + fn overlapping_and_nested_ranges_coalesce() { + let a = encode_range(10 * Q_END_NS, 40 * Q_END_NS).unwrap(); + let b = encode_range(30 * Q_END_NS, 60 * Q_END_NS).unwrap(); + let nested = encode_range(12 * Q_END_NS, 20 * Q_END_NS).unwrap(); + let (s, _, _) = decode(a); + let (_, e, _) = decode(b); + let got = normalize(&[b, nested, a]); + assert_eq!(got.len(), 1); + assert_eq!(decode(got[0]), (s, e, true)); + } + + #[test] + fn merged_bounds_stay_on_grid_no_rounding() { + // Read the expectation off the *inputs*: when two range words + // coalesce, the merged word decodes to their min start and max end + // verbatim — min/max of on-grid values, so no rounding arm. + let mut st = 0x198; + let mut merges = 0; + for _ in 0..500 { + let a = rand_word(&mut st); + let b = rand_word(&mut st); + let (sa, ea, a_rng) = decode(a); + let (sb, eb, b_rng) = decode(b); + let out = normalize(&[a, b]); + // Only a two-range collapse has a min/max expectation: two + // timestamps, or a timestamp/range pair, need not collapse. + if a_rng && b_rng && out.len() == 1 { + assert!(is_range(out[0])); + assert_eq!(decode(out[0]), (sa.min(sb), ea.max(eb), true)); + merges += 1; + } + } + assert!(merges > 0, "no two-range merge exercised"); + } + + // ── Q2: timestamps ─────────────────────────────────────────────────── + + #[test] + fn absorption_boundaries_are_exact() { + let r = encode_range(50 * Q_END_NS, 70 * Q_END_NS).unwrap(); + let (s, e, _) = decode(r); + let at_start = encode_timestamp(s).unwrap(); + let last_in = encode_timestamp(e - 1).unwrap(); + let at_end = encode_timestamp(e).unwrap(); // end exclusive: outside + let before = encode_timestamp(s - 1).unwrap(); + assert_eq!(normalize(&[r, at_start]), vec![r]); + assert_eq!(normalize(&[r, last_in]), vec![r]); + assert_eq!(normalize(&[r, at_end]), vec![r, at_end]); + // `before` sorts ahead of r: its start quantum precedes r's floor. + assert_eq!(normalize(&[r, before]), vec![before, r]); + } + + #[test] + fn timestamps_never_merge_with_each_other() { + // Adjacent instants stay two exact members — a merged range would + // round outward and change coverage, which normalize never does. + let t = 9 * Q_END_NS + 3; + let a = encode_timestamp(t).unwrap(); + let b = encode_timestamp(t + 1).unwrap(); + assert_eq!(normalize(&[b, a]), vec![a, b]); + // Equal instants deduplicate to the one word, still a timestamp. + let got = normalize(&[a, a, a]); + assert_eq!(got, vec![a]); + assert!(!is_range(got[0])); + } + + // ── canonical-form laws ────────────────────────────────────────────── + + #[test] + fn empty_and_singletons_pass_through() { + assert!(normalize(&[]).is_empty()); + let t = encode_timestamp(123_456_789).unwrap(); + let r = encode_range(5 * Q_END_NS, 6 * Q_END_NS).unwrap(); + assert_eq!(normalize(&[t]), vec![t]); + assert_eq!(normalize(&[r]), vec![r]); + } + + #[test] + fn order_independent_and_idempotent() { + let mut st = 0xCA11; + for _ in 0..300 { + let n = 1 + (splitmix64(&mut st) % 12) as usize; + let mut words: Vec = (0..n).map(|_| rand_word(&mut st)).collect(); + let reference = normalize(&words); + assert_eq!(normalize(&reference), reference, "idempotent"); + for _ in 0..4 { + // Fisher-Yates on the deterministic PRNG. + for i in (1..words.len()).rev() { + let j = (splitmix64(&mut st) % (i as u64 + 1)) as usize; + words.swap(i, j); + } + assert_eq!(normalize(&words), reference, "order-independent"); + } + } + } + + #[test] + fn coverage_is_preserved_exactly() { + // Membership at every decoded bound and its neighbors must agree + // between the raw set and its canonical form — coverage-identical, + // not conservatively-identical. + let mut st = 0xC0DE; + for _ in 0..200 { + let n = 1 + (splitmix64(&mut st) % 10) as usize; + let words: Vec = (0..n).map(|_| rand_word(&mut st)).collect(); + let canon = normalize(&words); + let mut probes: Vec = Vec::new(); + for &w in words.iter().chain(canon.iter()) { + let (s, e, _) = decode(w); + probes.extend([s.saturating_sub(1), s, s + 1, e.saturating_sub(1), e, e + 1]); + } + probes.push(rand_time(&mut st)); + for t in probes { + assert_eq!(covered(&words, t), covered(&canon, t), "probe {t}"); + } + } + } + + #[test] + fn canonical_form_is_structurally_canonical() { + let mut st = 0x5E7; + for _ in 0..200 { + let n = 1 + (splitmix64(&mut st) % 14) as usize; + let words: Vec = (0..n).map(|_| rand_word(&mut st)).collect(); + let canon = normalize(&words); + assert!(canon.windows(2).all(|p| p[0] < p[1]), "sorted, no dups"); + let parts = canonicalize(&canon); + for pair in parts.ranges.windows(2) { + let ((_, e0), (s1, _)) = (pair[0], pair[1]); + assert!(e0 < s1, "ranges disjoint with a surviving gap"); + } + for &t in &parts.stamps { + assert!( + !parts.ranges.iter().any(|&(s, e)| s <= t && t < e), + "no stamp subsumed by a range" + ); + } + } + } + + #[test] + fn arbitrary_bit_patterns_normalize_without_panicking() { + let mut st = 0xBADF00D; + let junk: Vec = (0..256).map(|_| splitmix64(&mut st)).collect(); + let once = normalize(&junk); + assert_eq!(normalize(&once), once, "junk output is still a fixpoint"); + // Scope of the canonical form: a junk "range" whose decoded end + // falls below its decoded start has an empty envelope, so it + // subsumes nothing and does not collapse against a copy of itself. + // Duplicate-freeness holds for encoder-produced words only. + let empty = (4u64 << 32) | 1; + assert_eq!(decode(empty), (4 * Q_START_NS, Q_END_NS, true)); + assert_eq!(normalize(&[empty, empty]), vec![empty, empty]); + } + + // ── toc_and (issue #198 phase 2) ───────────────────────────────────── + + #[test] + fn golden_and_is_exact_by_grid_closure() { + // Overlapping ranges: the intersection decodes to max(starts) / + // min(ends) verbatim — Q3's closure, no rounding. + let a = encode_range(10 * Q_END_NS, 40 * Q_END_NS).unwrap(); + let b = encode_range(30 * Q_END_NS + 5, 60 * Q_END_NS).unwrap(); + let (sa, ea, _) = decode(a); + let (sb, eb, _) = decode(b); + let got = toc_and(&[a], &[b]); + assert_eq!(got.len(), 1); + assert_eq!(decode(got[0]), (sa.max(sb), ea.min(eb), true)); + // The bounds land back on their grids exactly. + assert_eq!(sa.max(sb) % Q_START_NS, 0); + assert_eq!(ea.min(eb) % Q_END_NS, 0); + } + + #[test] + fn golden_disjoint_and_abutting_intersect_to_nothing() { + let a = encode_range(3 * Q_START_NS, 8 * Q_END_NS - 5).unwrap(); + // Abutting envelopes (decoded end == decoded start) share no + // instant: both are half-open, so the intersection is empty. + let abutting = encode_range(16 * Q_START_NS + 1, 20 * Q_END_NS - 5).unwrap(); + assert_eq!(decode(a).1, decode(abutting).0); + assert!(toc_and(&[a], &[abutting]).is_empty()); + let far = encode_range(100 * Q_END_NS, 200 * Q_END_NS).unwrap(); + assert!(toc_and(&[a], &[far]).is_empty()); + } + + #[test] + fn nested_range_intersects_to_the_inner() { + let outer = encode_range(10 * Q_END_NS, 60 * Q_END_NS).unwrap(); + let inner = encode_range(20 * Q_END_NS, 30 * Q_END_NS).unwrap(); + assert_eq!(toc_and(&[outer], &[inner]), vec![inner]); + } + + #[test] + fn one_range_against_many_fragments() { + // A long a-range cut by three disjoint b-ranges: three pieces out, + // each an exact pairwise intersection, gaps preserved. + let a = encode_range(0, 100 * Q_END_NS).unwrap(); + let bs: Vec = [10u64, 40, 70] + .iter() + .map(|&k| encode_range(k * Q_END_NS, (k + 5) * Q_END_NS).unwrap()) + .collect(); + assert_eq!(toc_and(&[a], &bs), normalize(&bs)); + } + + #[test] + fn golden_and_top_of_span_keeps_the_range_variant() { + // Same reason as `golden_top_of_span_keeps_the_range_variant`, for + // the and path: `intersect` re-derives `min(ends)` through the same + // unchecked `encode_envelope`, and the top of the span is where one + // more bit of end lands on bit 31 and silently re-encodes the range + // as a *timestamp*. `rand_time` is uniform over the span, so the + // randomized tests reach the top 2^32 ns with probability ~2^-31. + let wide = encode_range(TOC_MAX_NS - 9 * Q_END_NS, TOC_MAX_NS - 1).unwrap(); + let inner = encode_range(TOC_MAX_NS - 5 * Q_END_NS, TOC_MAX_NS - 1).unwrap(); + let got = toc_and(&[wide], &[inner]); + assert_eq!(got, vec![inner]); + assert!(is_range(got[0])); + assert_eq!(got[0] & LOW_MASK, LOW_MASK, "end field full"); + assert_eq!( + decode(got[0]), + (TOC_MAX_NS - 5 * Q_END_NS, TOC_MAX_NS, true) + ); + // The last encodable instant survives against a cover containing it, + // bit-identical — not rounded up into the flag bit. + let t = encode_timestamp(TOC_MAX_NS - 1).unwrap(); + assert_eq!(toc_and(&[t], &[wide]), vec![t]); + } + + #[test] + fn and_timestamp_survival_is_exact() { + let r = encode_range(50 * Q_END_NS, 70 * Q_END_NS).unwrap(); + let (s, e, _) = decode(r); + let inside = encode_timestamp(e - 1).unwrap(); + let at_end = encode_timestamp(e).unwrap(); + let at_start = encode_timestamp(s).unwrap(); + // A stamp inside the other cover's range survives bit-identical … + assert_eq!(toc_and(&[inside], &[r]), vec![inside]); + assert_eq!(toc_and(&[r], &[inside]), vec![inside]); + assert_eq!(toc_and(&[at_start], &[r]), vec![at_start]); + // … the exclusive envelope end is outside … + assert!(toc_and(&[at_end], &[r]).is_empty()); + // … identical stamps intersect to themselves, distinct ones to + // nothing (an instant has no extent to share). + assert_eq!(toc_and(&[inside], &[inside]), vec![inside]); + assert!(toc_and(&[inside], &[at_end]).is_empty()); + } + + #[test] + fn golden_and_stamp_survives_on_an_output_piece_end() { + // The tightest canonicality boundary the and path has: a stamp + // landing exactly on a piece's *exclusive* end. It is the only + // sweep-created boundary a surviving stamp can touch — a stamp is + // outside its own side's ranges (canonical), hence outside every + // output piece those contain — and it is where a closed-vs-half-open + // slip in `intersect` would surface as non-canonical output. The + // randomized tests cannot reach it: a random offset coincides with a + // decoded end with probability ~2^-32. + let r1 = encode_range(0, 100 * Q_END_NS - 1).unwrap(); + let cut = decode(r1).1; + assert_eq!(cut, 100 * Q_END_NS); + // t is outside r1 (end exclusive), so it survives a's canonicalize. + let t = encode_timestamp(cut).unwrap(); + let r2 = encode_range(50 * Q_END_NS, 300 * Q_END_NS - 1).unwrap(); + let piece = encode_range(50 * Q_END_NS, 100 * Q_END_NS - 1).unwrap(); + let got = toc_and(&[r1, t], &[r2]); + assert_eq!(got, vec![piece, t]); + assert_eq!(decode(piece).1, cut, "the stamp sits on the piece's end"); + assert_eq!(normalize(&got), got, "canonical without a re-normalize"); + } + + #[test] + fn and_accepts_raw_word_sets() { + // Operands are canonicalized internally: unsorted, duplicated, + // absorbable words give the same answer as their canonical forms. + let r1 = encode_range(10 * Q_END_NS, 30 * Q_END_NS).unwrap(); + let r2 = encode_range(25 * Q_END_NS, 50 * Q_END_NS).unwrap(); + let t = encode_timestamp(28 * Q_END_NS).unwrap(); // absorbed by r1|r2 + let q = encode_range(20 * Q_END_NS, 40 * Q_END_NS).unwrap(); + let raw = vec![t, r2, r1, r2]; + assert_eq!(toc_and(&raw, &[q]), toc_and(&normalize(&raw), &[q])); + // q's decoded envelope sits wholly inside raw's merged coverage, so + // the intersection is q itself, bit-identical. + assert_eq!(toc_and(&raw, &[q]), vec![q]); + } + + #[test] + fn and_laws_identity_commutativity_empty() { + let mut st = 0xA17D; + let mut nonempty = 0; + for _ in 0..300 { + let n = 1 + (splitmix64(&mut st) % 10) as usize; + let m = 1 + (splitmix64(&mut st) % 10) as usize; + let a: Vec = (0..n).map(|_| rand_word(&mut st)).collect(); + let b: Vec = (0..m).map(|_| rand_word(&mut st)).collect(); + nonempty += !toc_and(&a, &b).is_empty() as u32; + assert_eq!(toc_and(&a, &a), normalize(&a), "A ∩ A = normalize(A)"); + assert_eq!(toc_and(&a, &b), toc_and(&b, &a), "commutative"); + assert!(toc_and(&a, &[]).is_empty()); + assert!(toc_and(&[], &b).is_empty()); + } + // Guard against a vacuous generator: only `rand_word`'s whole-span + // arm draws words wide enough to meet an independent draw, so a + // narrowing tweak there would quietly reduce this to empty == empty. + assert!(nonempty > 0, "no nonempty intersection exercised"); + } + + #[test] + fn and_membership_matches_both_sides() { + // The defining property: an instant is covered by A ∩ B iff it is + // covered by A and by B — probed at every decoded bound ± 1. + let mut st = 0xB007; + let mut nonempty = 0; + for _ in 0..200 { + let n = 1 + (splitmix64(&mut st) % 8) as usize; + let m = 1 + (splitmix64(&mut st) % 8) as usize; + let a: Vec = (0..n).map(|_| rand_word(&mut st)).collect(); + let b: Vec = (0..m).map(|_| rand_word(&mut st)).collect(); + let both = toc_and(&a, &b); + nonempty += !both.is_empty() as u32; + let mut probes: Vec = Vec::new(); + for &w in a.iter().chain(b.iter()).chain(both.iter()) { + let (s, e, _) = decode(w); + probes.extend([s.saturating_sub(1), s, s + 1, e.saturating_sub(1), e, e + 1]); + } + probes.push(rand_time(&mut st)); + for t in probes { + assert_eq!( + covered(&both, t), + covered(&a, t) && covered(&b, t), + "probe {t}" + ); + } + } + assert!(nonempty > 0, "no nonempty intersection exercised"); + } + + #[test] + fn and_output_is_canonical() { + // No re-normalize runs on the way out; the sweep must land in + // canonical form on its own (idempotence pins it). + let mut st = 0xCAB; + let mut nonempty = 0; + for _ in 0..200 { + let n = 1 + (splitmix64(&mut st) % 10) as usize; + let m = 1 + (splitmix64(&mut st) % 10) as usize; + let a: Vec = (0..n).map(|_| rand_word(&mut st)).collect(); + let b: Vec = (0..m).map(|_| rand_word(&mut st)).collect(); + let both = toc_and(&a, &b); + nonempty += !both.is_empty() as u32; + assert_eq!(normalize(&both), both, "already canonical"); + assert!(both.windows(2).all(|p| p[0] < p[1]), "sorted, no dups"); + } + assert!(nonempty > 0, "no nonempty intersection exercised"); + } + + #[test] + fn arbitrary_bit_patterns_intersect_without_panicking() { + let mut st = 0xDEAD; + let junk_a: Vec = (0..128).map(|_| splitmix64(&mut st)).collect(); + let junk_b: Vec = (0..128).map(|_| splitmix64(&mut st)).collect(); + let got = toc_and(&junk_a, &junk_b); + // Stricter than normalize's junk contract (see + // `arbitrary_bit_patterns_normalize_without_panicking`): `intersect` + // only pushes a piece when `s < e`, so no empty envelope survives the + // sweep and junk output is *strictly* canonical — a fixpoint with no + // duplicate to carry through. + assert!(!got.is_empty(), "junk pair intersected to nothing"); + assert_eq!(normalize(&got), got, "junk output is still canonical"); + assert!(got.windows(2).all(|p| p[0] < p[1]), "sorted, no dups"); + } + + #[test] + fn quantum_constants_are_the_decode_grids() { + // The Q1 exactness argument leans on decoded starts being 2^31 + // multiples and ends 2^32 multiples; pin that against the constants. + let mut st = 0x9; + for _ in 0..200 { + let w = rand_word(&mut st); + let (s, e, is_rng) = decode(w); + if is_rng { + assert_eq!(s % Q_START_NS, 0); + assert_eq!(e % Q_END_NS, 0); + } + } + } +}