Spec: normative toc word grammar, frozen for 1.x (issue #193) - #206
Conversation
espg
left a comment
There was a problem hiding this comment.
🤖 from Claude (review)
Bit-level pass over §10 against the shipped kernels (src_rust/src/toc.rs, mortie/toc.py), re-run empirically in the built worktree. Every constant, formula and golden word checks out: TOC_MAX_NS = 2⁶³−2³² = 9223372032559808512; GPS_EPOCH_NS = 47,486×86,400×10⁹ (and 1850-01-01 → 1980-01-06 really is 47,486 proleptic days); the eight §10.8 words (0x0000000080000000, 0x71E014439BE20000, 0x932610CAE9813400, 0xFFFFFFFDFFFFFFFF, 0x0000000500000004, 0x0000000D00000008, 0x0000000500000008, 0x0000000038F00A22) reproduce exactly, as do their decoded bounds and UTC renderings; the timestamp splice, the merge codes (t≫31, (t≫32)+1), the §10.4 valid-domain characterization (timestamp high field ≤ 2³²−3, range s ≤ 2e−1) are exact in both directions — I brute-forced encoder-satisfies and word-is-reachable over 20k samples each and found no gap; the int64 claim is right (bit 63 flips at internal ns 2⁶² for timestamps, at s = 2³¹ for ranges); “two unequal valid words always merge to a range word” holds (and the result is always valid, see below); associativity/commutativity/idempotence survived 20k random triples; the 1972 −9 s step-back, the +18 s 2017 offset, the epoch identity 1850-01-01 → 0, and the tie-break order all match. The renumbering is clean: contents anchors resolve, §9's cross-ref now says §11, the legacy #10-frozen-for-1x anchor is kept, and a repo-wide grep finds no stale §10/#10- reference (the only in-repo link, docs/api/toc.md, points at the correct new slug). Seven findings below, none of them a wrong bit: one freeze that cannot hold (the 2142 row's UTC cell moves on any IERS leap append), one selection trap (toc_contains returns True for the reserved 0 word on any epoch-anchored window), an unstated closure property that external readers already rely on, a behavioral misstatement about tocs_reduce on an empty batch, an ambiguous “one quantum” where two quanta are defined, a “law” that is really a writer discipline, and a present-tense citation claim that is not yet true of the citing store.
| | timestamp: epoch | `time2toc(0)` | `0x0000000080000000` | 2147483648 | 0 | 0 | 1850-01-01T00:00:00.000000000 | | ||
| | timestamp: GPS epoch | `time2toc(4102790400000000000)` | `0x71E014439BE20000` | 8205580801679687680 | 4102790400000000000 | 4102790400000000000 | 1980-01-06T00:00:00.000000000 | | ||
| | timestamp: 2018-01-01 UTC | `time2toc(5301590418000000000)` | `0x932610CAE9813400` | 10603180836377408512 | 5301590418000000000 | 5301590418000000000 | 2018-01-01T00:00:00.000000000 | | ||
| | timestamp: last valid instant | `time2toc(9223372032559808511)` | `0xFFFFFFFDFFFFFFFF` | 18446744065119617023 | 9223372032559808511 | 9223372032559808511 | 2142-04-11T23:46:54.559808511 | |
There was a problem hiding this comment.
🤖 from Claude (review)
This row's UTC cell is not freezable, but §11 now freezes “the §10.8 conformance vectors”.
to_datetime64 picks the offset from the last leap row at or before the instant, so an IERS append — exactly what §10.1's “Leap-table appends are additive” bullet (line 820) permits — moves the rendering of every instant after the new step. Year 2142 is after every future step. Simulating one appended row (("2030-01-01", 38)) against the shipped kernel:
after append, last valid instant renders: 2142-04-11T23:46:53.559808511 # was ...54.559808511
2018 row unchanged: 2018-01-01T00:00:00.000000000
epoch rows unchanged: 1850-01-01T00:00:00, 1980-01-06T00:00:00
The other seven rows are genuinely frozen (each precedes any future step); this one's last column is dated, not normative. Two consequences: §11's new bullet (“…and the §10.8 conformance vectors”) contradicts §10.1's additive rule for exactly this cell, and test_spec_toc.py::test_table_matches_kernels will fail on the append and force an edit to a table the page calls immutable. The same exposure applies to the §10.1 prose at lines 826–827 (“the last valid instant renders as UTC 2142-04-11T23:46:54.559808511”).
Suggested: freeze the word/ns columns and mark start (UTC) as a rendering under the leap table as of a stated date — or drop the UTC cell for the 2142 row only.
There was a problem hiding this comment.
🤖 from Claude
Fixed by scoping the freeze rather than dropping the cell. §10.8 gains a "What is frozen, exactly" paragraph: the word / ns columns and the decode they express are the immutable part; start (UTC) is derived — rendered through the §10.1 boundary convention under the leap table as of the 2017-01-01 step — and §10.1's additive rule applies to it, so an IERS append moves renderings of instants after the new step (of these rows only the 2142 one) while every word/ns/decode value stays byte-for-byte fixed. The regeneration test failing on such an append is the intended signal: the page gets a deliberate re-render, not a silent drift. §10.1's 2142 sentence carries the same scoping phrase, and §11's bullet now reads "the §10.8 conformance vectors (their UTC renderings scoped to the shipped leap table, per §10.8)". Rows left as-is — they are correct for the shipped table (_LEAP_TABLE in mortie/toc.py ends at the 2017-01-01 row), and test_spec_toc.py still regenerates and compares them literally: 18 passed. — 3b881d5
| **The all-zero word is unreachable.** No encoder output is `0`: the epoch | ||
| instant encodes as `0x8000_0000` (the flag bit sits at position 31, not at | ||
| the bottom of the word), and every range word has end code `e ≥ 1`, so the | ||
| smallest range word is `1`. External conventions may therefore reserve `0` |
There was a problem hiding this comment.
🤖 from Claude (review)
The 0-as-sentinel blessing is only safe for toc_overlaps; toc_contains selects the zero word. Shipped behavior:
toc2time(0) -> (0, 0) # a range word, envelope [0, 0)
toc_overlaps(0, 0, 10**18) -> False
toc_contains(0, 0, 10**18) -> True
§10.7's contains test is q_start <= s and e <= q_end, and (0, 0) satisfies it for every window anchored at the epoch (q_start == 0) — which is the natural “select everything” query. zagg's §8.2 “cost-free reservation” argument is stated only over the overlap predicate (“it overlaps no window, including one containing the epoch”), so a store that reserves 0 on the strength of this sentence and then selects with toc_contains pulls its unobserved cells into the result.
Since §10.4 declares out-of-domain words to carry “no semantics”, this line is the only thing an external convention has to go on. It needs the scope: toc_overlaps never selects 0, toc_contains does for epoch-anchored windows, so a fill-reserving reader must mask fills before any predicate.
There was a problem hiding this comment.
🤖 from Claude
Fixed — the note now carries the predicate scope. Added paragraph: reserving 0 is free of encoder collisions, not of predicate hits; 0 decodes as the empty envelope [0, 0), so toc_overlaps(0, ·, ·) is false for every window (the property zagg §8.2 argues over), but it is vacuously inside any epoch-anchored window, so toc_contains(0, 0, q_end) is true and a fill-reserving store must mask its sentinel before a containment query. Reproduced against the built kernel here: toc2time(0) -> (0, 0), toc_overlaps(0, 0, 10**18) -> False, toc_contains(0, 0, 10**18) -> True, toc_contains(0, 1, 10**18) -> False. The claim about zagg stays scoped to what its §8.2 actually argues. — 078e0e8
| verbatim; a timestamp's merge envelope is `s = t ≫ 31` (its high field) | ||
| and `e = (t ≫ 32) + 1` — and emits the **range** word | ||
| `(min(s_a, s_b) ≪ 32) | max(e_a, e_b)`. Two unequal valid words always | ||
| merge to a range word. |
There was a problem hiding this comment.
🤖 from Claude (review)
Closure is never stated, and external readers already depend on it. §10.6 says two unequal valid words merge to a range word, but not that the result is itself a valid word — i.e. that the merge is closed on the §10.4 valid domain.
It is: for valid inputs s_i ≤ 2e_i − 1, so min s ≤ s_a ≤ 2e_a − 1 ≤ 2·max e − 1, and max e ≥ 1 survives; I also checked 20k random valid pairs against the kernel and every merged word satisfies s ≤ 2e − 1 (hence is never 0).
Two claims ride on it. (1) The fold-tree bit-identity guarantee in the next paragraph: a reduction re-merges its own intermediate results, which is only inside the stated scope if those intermediates are valid. (2) zagg §8.2's “toc_merge cannot introduce it [0]” clause. One added clause — “the merged word is itself valid (§10.4), so a reduction stays inside the domain and never yields 0” — closes both.
There was a problem hiding this comment.
🤖 from Claude
Added. §10.6 now states closure explicitly right after the join rule: for valid inputs the merged codes again satisfy §10.4 (min s ≤ s_a ≤ 2e_a − 1 ≤ 2·max e − 1, with max e ≥ 1), so the merged word is itself valid and never 0 (§10.3) — and closure is what lets a reduction re-merge its own intermediates inside the scope stated below. Independently re-checked against the built kernel here: 20k random valid pairs (timestamps and ranges mixed), every merge valid, none 0, and the second-level merges of those results likewise valid. — 60ed2bb
| every input instant and every input envelope (conservatism direction | ||
| preserved, never narrowed). The join has **no identity element**: a | ||
| reduction over zero words is an error, never a sentinel | ||
| (`toc_reduce` / `tocs_reduce` refuse empty input). |
There was a problem hiding this comment.
🤖 from Claude (review)
tocs_reduce does not refuse empty input — it refuses an empty group. Shipped behavior:
tocs_reduce(np.array([], dtype=np.uint64), [0]) -> array([], dtype=uint64) # accepted
tocs_reduce([1, 2], [0, 1, 1, 2]) -> ValueError: group 1: tocs_reduce of an empty segment ...
toc_reduce(np.array([], dtype=np.uint64)) -> ValueError: ... the merge has no identity element
That is deliberate and test-pinned in src_rust/src/toc.rs (segmented_reduce_refuses_an_empty_group_by_index: “An empty batch is fine: no groups, no answers required.”). As written, an implementer reading “a reduction over zero words is an error” would raise on the empty batch and diverge from the reference implementation.
Suggested: “(toc_reduce refuses an empty array; tocs_reduce refuses an empty group — an empty batch, with no groups at all, is not a reduction and returns no words)”.
There was a problem hiding this comment.
🤖 from Claude
Fixed — the clause was wrong as written. §10.6 now reads: the join has no identity element, so a reduction over zero words is an error — toc_reduce refuses an empty array and tocs_reduce refuses an empty group; an empty batch (no words, no groups) asks for no reduction at all and is accepted, returning no words. Verified against the built kernel here: tocs_reduce([], [0]) -> array([], dtype=uint64); tocs_reduce([1,2], [0,1,1,2]) -> ValueError: group 1: tocs_reduce of an empty segment (the merge has no identity element); toc_reduce([]) -> ValueError. Matches segmented_reduce_refuses_an_empty_group_by_index in src_rust/src/toc.rs. — 579e99e
| overlaps and contains nothing**. | ||
|
|
||
| - **overlaps** (`toc_overlaps`): true iff the envelope intersects the | ||
| window. It may **over-report** near window edges by up to one quantum (a |
There was a problem hiding this comment.
🤖 from Claude (review)
“one quantum” is ambiguous where §10.2 defines two, and the slack is genuinely asymmetric. Q_START_NS = 2³¹ (~2.15 s) and Q_END_NS = 2³² (~4.29 s), and the encoder's rounding gives s·2³¹ > start_ns − 2³¹ at the leading edge but e·2³² ≤ end_ns + 2³² at the trailing edge — so the over-report bound is < Q_START_NS before the window's start and ≤ Q_END_NS past its end, i.e. up to ~4.29 s on the trailing side, twice what a reader who assumes “the quantum” = the start quantum would budget.
The same word carries the under-report bound in the contains bullet (line 984), and zagg §8.1/§8.2 restate “over-reporting by at most one quantum at an edge” verbatim as their own contract clause — so the bound they inherit should name which quantum applies at which edge.
There was a problem hiding this comment.
🤖 from Claude
Quantified. §10.7 gains a slack paragraph before the bullets: a range word's envelope starts at most 2³¹ − 1 ns before the real start (floored; zero on-grid) and ends between 1 ns and 2³² ns past its last covered instant (strictly-greater ceiling; exactly 2³² on-grid), timestamps carry no slack — so the misreports reach < 2³¹ ns leading, ≤ 2³² ns trailing, and a symmetric "one quantum" budget under-budgets the trailing edge by 2×. The bullets now name which side applies: an overlap false-positive requires real content ending later than q_start − 2³² or starting earlier than q_end + 2³¹; a contains miss requires the real interval to start less than 2³¹ ns after q_start or end less than 2³² ns before q_end. Both bounds checked against the built kernel (200k random word/window pairs each): no under-reported overlap, no over-reported contains, observed worst reaches 4,276,186,541 ns back and 2,135,777,262 ns forward, and zero violations of the contains necessary condition. — 4f23335
|
|
||
| **The conservative direction is law**: encoding only ever **widens** — the | ||
| envelope contains the real interval, never the reverse; an instant is never | ||
| widened into a range by encoding (it has its exact timestamp form), and a |
There was a problem hiding this comment.
🤖 from Claude (review)
“an instant is never widened into a range by encoding” is a writer discipline, not a property of these encoders. span2toc(t, t) is in-domain and returns a range word:
span2toc(0, 0) -> 1 # the range word for [0, 2³²), not the epoch timestamp 0x80000000
Nothing in §10.3 prevents a producer from encoding an instant through span2toc; what actually holds is the narrower “time2toc gives an instant an exact form, so no widening is forced”. The real obligation lives on the writer, and it is zagg that states it as a MUST (§8.1: “a timestep whose real acquisition is a single instant MUST be encoded as a timestamp word”).
Because the clause sits under “The conservative direction is law”, a store implementer can read it as a grammar guarantee that instants round-trip as timestamps — which mortie does not enforce, and which would then be relied on when deciding whether a decoded (t, t) pair means “instant”. Suggest attributing it explicitly: the grammar offers the exact form; choosing it over span2toc is the writer's obligation.
There was a problem hiding this comment.
🤖 from Claude
Fixed. The conservative-direction paragraph now states only what the encoders do (widen, never narrow); the instant clause moved into its own paragraph and is attributed: the grammar provides an exact form (time2toc) so no instant need be widened, but span2toc(t, t) is in domain and returns a range word (span2toc(0, 0) == 1, confirmed against the built kernel here), so taking the exact form is writer discipline — zagg §8.1 makes it a MUST, mortie does not enforce it. — 4dd053d
|
|
||
| External stores cite this section as the word grammar behind their own | ||
| declarations — e.g. zagg's `zagg-toc/1` attrs blocks store the grammar | ||
| revision token `mortie-toc/1`, which resolves here (informative; the token |
There was a problem hiding this comment.
🤖 from Claude (review)
“which resolves here” is not yet true of the store this names. zagg main's docs/specification.md §8 pins the mortie-toc/1 documentation pointer at the release-pinned API page, and says so in as many words:
… the URL above is the documentation pointer … It is release-pinned deliberately … the pin is
0.9.6…
Mortie's frozen specification contains no toc section: the grammar above is normative as a module reference pinned by mortie's own golden fixtures, which is a weaker guarantee. [#193] tracks adding the frozen section; when it lands, this pointer swaps to it…
So today that token resolves to …/0.9.6/api/toc/, and will keep doing so until zagg re-points — which is a separate change in a separate repo. Since this sentence is the one that tells an external reader where the grammar lives, it is worth making it forward-looking (“is what such a citation is intended to resolve to”) rather than present-tense, so the page does not assert a cross-repo state it cannot itself effect.
There was a problem hiding this comment.
🤖 from Claude
Fixed — the sentence no longer asserts a cross-repo state. §10 intro now reads that mortie-toc/1 is the class of citation this section is meant to receive, that where a given store's documentation pointer points is that store's own release-pinned choice, and that zagg's swaps here once this section ships (linking issue #193, which is the tracker zagg's §8 names). — 846911a
Merging this PR will improve performance by 10.51%
|
| Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|
| ⚡ | fast_norm2mort_orders[6] |
298.6 ns | 269.4 ns | +10.82% |
| ⚡ | fast_norm2mort_orders[10] |
315.3 ns | 286.1 ns | +10.19% |
Tip
Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.
Comparing claude/193-toc-spec (cc9d9de) with main (2612e86)2
Footnotes
-
1 benchmark was skipped, so the baseline result was used instead. If it was deleted from the codebase, click here and archive it to remove it from the performance reports. ↩
-
No successful run was found on
main(f053fa0) during the generation of this report, so 2612e86 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩
|
🤖 from Claude Status: all three phases landed and all review findings folded; the PR stays draft only because CI has not settled yet — nothing is red. Phases (all complete, head
Adversarial self-review + fold. The review
No finding was declined. Every numeric claim was re-verified against the live Gates. Full suite green locally ( Why still draft. On head |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #206 +/- ##
=======================================
Coverage 96.67% 96.67%
=======================================
Files 20 20
Lines 2283 2283
=======================================
Hits 2207 2207
Misses 76 76
Flags with carried forward coverage won't be shown. Click here to find out more. Continue to review full report in Codecov by Harness.
🚀 New features to boost your workflow:
|
|
🤖 from Claude The one red check is CodSpeed's analysis gate, and it cannot be caused by this PR. Flagging rather than acting on it, per CLAUDE.md §4 (unrelated CI failures get flagged, not "fixed"). Evidence — the complete diff of this PR: Zero source files, zero Rust. The five benchmarks reported as regressed ( Everything else on head This is now genuinely @espg's call, so the PR carries |
# Conflicts: # CHANGELOG.md
|
🤖 from Claude Merged Conflict. Section numbering survived intact. Main changed no line of What main's rename did break, and the fix (
No golden moved. Every §10.8 vector still regenerates bit-identically from the post-merge kernels — the merge changed no toc word behavior (main's Gates on the merged head. Full local suite CodSpeed is not a gate this PR can answer. Its analysis flags a swing on |
|
🤖 from Claude Both review questions are resolved per espg's in-session rulings (2026-08-24), folded in three commits:
The golden-vector table between the |
Closes #193.
Adds the normative spec section for the toc word grammar to
docs/specification.md, frozen for 1.x — the section zagg'szagg-toc/1declaration (grammar tokenmortie-toc/1, englacial/zagg#443) and other external readers will cite in place of the mike-versioned API page. Everything documents the encoding as shipped in 0.9.9 (src_rust/src/toc.rs,mortie/toc.py); no kernel changes.Plan and section outline: the plan comment on #193.
What the section covers
New §11 "The packed 64-bit toc word" (stable anchor
#toc-word), mirroring §1's structure, appended after the freeze list: "Frozen for 1.x" stays §10 — number and#10-frozen-for-1xanchor untouched, so the most-cited anchor in the spec never moves and future sections keep appending after it (espg ruling, see Questions below) — and gains the toc bullet (citing §11 forward):GPS_EPOCH_NS, the UTC-boundary leap convention (pre-1972 zero offset, +18 s since 2017, additive table appends, steps cited to IERS Bulletin C),TOC_MAX_NSapplied to both encoders[start:32][flag:1][low:31], flag polarity, the timestamp splice, unsigned-storage rules ≤ 2e−1), and the garbage-in-garbage-out posturePhases
mortie/tests/test_spec_toc.pydrift pin against the live kernelsdocs/api/toc.md↔ spec cross-links, CHANGELOG entryHow it was tested
python -m pytest -q: 1655 passed / 16 skipped at baseline, rebuilt wheel from this branch's parent).table:toc_vectorsmarkers is byte-identical, so the phase-2 drift pin is unaffected.Questions for review
<a name>anchors added for the renumbering are removed (#10-frozen-for-1xis again the heading's own slug); the number-free#frozen-for-1xstable anchor is kept, matching#toc-word. No external deep link breaks.mortie/toc.pyrather than restated (duplicated normative text drifts, per the page's own preamble); the convention (offsets, pre-1972 rule, additive appends) is stated normatively. Added alongside thetoc.pypointer: the authoritative external citation to the IERS leap-second announcements (Bulletin C), so a non-Python implementer has a code-free normative path to the identical values.