Skip to content

spill: carry the temporal companion through the cross-block fold (issue #477) - #479

Merged
espg merged 8 commits into
mainfrom
claude/477-temporal-spillfold
Aug 18, 2026
Merged

spill: carry the temporal companion through the cross-block fold (issue #477)#479
espg merged 8 commits into
mainfrom
claude/477-temporal-spillfold

Conversation

@espg

@espg espg commented Aug 17, 2026

Copy link
Copy Markdown
Member

Closes #477. Refs #410/#463 (channel + law), #370 (fold regime), #474/#475 (the knob this unblocks), #279 (kway lineage).

What this does

SpillAggregator's cross-block close carried the located channel only, so _mergeable classified any temporal: field as unfoldable and block_bytes (#474/#475) refused every temporal store — including all of California — with

field 'h_tdigest_signal': temporal companions (temporal: 'per-centroid') have no cross-block fold state yet …

This carries the §8.3 per-centroid companion through the same seam, one channel over.

The carry. The per-block build already returns (digest, *words) and the merge laws already take both channels (merge_tdigests(..., temporal1=, temporal2=), merge_tdigests_kway(..., temporal=) — shipped in #463, the overview folds are the working call site). So the change is state + plumbing, not algebra:

  • _DigestField gains temporal: bool (the §8.3 declaration; unlike location there is no column name — the words are derived).
  • The two located-only dicts become per-channel ones keyed by reducer kwarg:
    _digest_locs_digest_words[field][kwarg][cell] (running, pairwise fields) and
    _digest_loc_parts_digest_word_parts[field][kwarg][cell] (k-way parts).
    One module-level table, _CHANNELS = (("location", "locations", "locations"), ("temporal", "temporal", "times")), carries the declaration attr, the reducer kwarg and the emission key in the kernel's fixed return order — mirroring sweep_overview.COMPANION_CHANNELS's shape. Every fold and emission site zips the kernel's returned words against that table, so the ordering claim is structural rather than dict-insertion order.
  • _fold_block derives the toc word column per cell, into cell_data before params resolve — exactly where calculate_cell_statistics does it (aggregate.py:487) — and passes each declared channel to build_tdigest / build_tdigest_where; every declared channel then rides one merge (pairwise) or one k-way collapse (_finalize_kway), so a field carrying both never gets one channel folded against a partition the other did not see.
  • _chunk_outputs_merged emits ragged_channels with _aggregate_chunk_cells's own keys (locations, times), so a folded chunk and a pooled one are indistinguishable to the writer.
  • validate_spill_fold lifts the per-centroid refusal (keeping a named one for per-cell, which is a scalar reducer over the word column with no per-block accumulator here); the merge-mode message now routes to mode: spill the way the located one does.

The derived column lands in cell_data, which keeps _resolve_param's namespace equal to the pooled one (which also carries toc_word), so a where reading it resolves identically. _check_fold_columns accordingly resolves params against available + toc_word while source: / location: still check the true spilled-column list, so neither the diagnostics nor their messages loosen. An absent clock column is named by _toc_word_column itself, with the pooled path's message.

Not touched: no wire format, no attrs grammar, no spec marker — so no docs/specification.md change and no committed fixture bytes move. The spill fold regime is documented as not byte-identical to single-block (§2.3 / #370), so no §7 conformance fixture covers it; tests/data/spec/ is untouched by design.

Phases

  • Phase 1 — carry the channel through the close: per-channel fold state, derived toc column, k-way + pairwise folds, emission keys; lift the per-centroid refusal and update the refusal text; re-pin the two probe tests.
  • Phase 2 — tests: envelope conservation across a forced block close, k-way order-independence, dual-channel (located + temporal), pooled cross-check within the fold-law bounds.
  • Phase 3 — empirical check on the local spill harness with the temporal fold arm on; adversarial review + fold (six findings, all answered on-thread).

Interaction with #475 (claude/474-spill-knobs)

#475 is open and also touches spill.py/streaming.py. This branch is based on main, which does not carry #475 yet, and the diff is kept minimal on the shared surface: #475 owns __init__'s knob plumbing / _default_block_bytes, while this PR owns the fold state and _fold_block/_finalize_kway/_chunk_outputs_merged. The one line they could both touch is the SpillOverflowError message. #475 merges first; expect to sync this branch onto it before it is ready (the #457 two-green-PRs lesson — two independently green PRs on one file are not a green merge).

Note also src/zagg/processing/spill.py lands at 1,198 lines, under the 1,200 raise trigger (espg standing ruling, #351/#358) — prose trimmed rather than the module split. See "Questions for review".

Testing

Full suite after the fold: 4,445 passed, 38 skipped, 1 failed — the failure being the known environmental test_lambda_build::TestFunctionBuild::test_function_build_succeeds. ruff check / ruff format --check clean on everything this PR touches. Known pre-existing and left alone: test_lambda_build (environmental), test_client_transport (wall-clock flake), ruff N818 in registry.py, and the tests/data/benchmark/README.md format drift.

The new coverage lives in tests/test_spill_crossblock.py::TestTemporalMultiBlock; each test asserts spill_blocks_closed == len(_CELL_LISTS), and each granule covers its own day, so every multi-granule centroid genuinely spans blocks. Details, mutation checks and the harness numbers are in the thread.

Questions for review

  • spill.py is at 1,198 lines — 2 under the 1,200 raise trigger. The self-review flagged that as "heading past 1,200" and worth a decision now rather than at the next PR that touches this file. The §4 ruling pre-approves overages below 1,200, so this PR is inside it and did not split anything; the next feature on this module will not be. Splitting it (the obvious seam is SpillBlock + _Partition + check_tmp_headroom into a spill_block.py, leaving SpillAggregator) is a mechanical but reviewer-visible change and is deliberately not in this PR.
  • The per-cell companion shape stays refused (named). Folding it is genuinely easy — the §8.2 join is a semilattice, so per-block cell envelopes join exactly — but it needs a third field class in the fold (a scalar-word accumulator beside _composition_fields) and no shipped config pairs it with mode: spill. Left out as scope; say the word if it should ride along.

@espg espg added the implement label Aug 17, 2026
@espg

espg commented Aug 17, 2026

Copy link
Copy Markdown
Member Author

🤖 from Claude

Phases 1 and 2 are landed (8b5c5fe2, 1ef778a4). Empirical check from phase 3 below; adversarial review is running next.

Harness evidence: the fold arm runs WITH temporal=True

scratch_agg_profile.py's spill arm list carried a dated comment — "temporal configs cannot cross-block fold (no temporal spill-fold state yet — found 2026-08-17); fold arm runs located-only". Re-run on this branch (29 granules, the track-1336 Sierra o9 shard 5512186041575931913, 1.50 M photons, atl03_tdigest_strata_healpix + temporal: per-centroid at δ=4096) with the fold arm flipped back to temporal=True:

=== spill arm spill_buf20_fold_temporal: buf=20 block_bytes=8388608 blocks_closed=1
  ingest+flush=0.2s reduce(84 chunks)=8.4s total=8.6s rss_hwm=697MB (start 440)
=== spill arm spill_buf20_fold_notemp:   buf=20 block_bytes=8388608 blocks_closed=1
  ingest+flush=0.1s reduce(84 chunks)=4.8s total=5.0s rss_hwm=697MB (start 697)

and, driving the threshold harder so several blocks close under a temporal config:

=== spill arm spill_buf5_fold8_temporal: buf=5 block_bytes=8388608 blocks_closed=4
  ingest+flush=9.7s reduce(84 chunks)=6.2s total=15.9s rss_hwm=609MB (start 441)
=== spill arm spill_buf5_fold1_temporal: buf=5 block_bytes=1048576 blocks_closed=4
  ingest+flush=9.8s reduce(84 chunks)=6.1s total=15.9s rss_hwm=614MB (start 609)

Before this branch every one of those arms raised SpillOverflowError at the first close. blocks_closed >= 1 is the fold regime engaged, and the block-close warning now names the channel:

… digests merge under t-digest semantics, located centroids coarsen to common ancestors, temporal words widen to their members' envelope, composition takes one k-way re-quantization; counts stay exact.

The temporal channel costs ~3.6 s of reduce on this shard (8.4 s vs 4.8 s) — the same _centroid_envelopes term the pooled cur/notemp arms measured on 2026-08-17, not something the fold adds. The harness copy used here is a scratch copy; the main checkout's dated comment is updated at the end of this PR as the record.

Test evidence

tests/test_spill_crossblock.py::TestTemporalMultiBlock — each granule covers its own day and _force_tiny_blocks closes a block per flush, so every multi-granule centroid spans multiple blocks (each test asserts spill_blocks_closed == len(_CELL_LISTS)):

  • Envelope conservation (parametrized over k-way and pairwise): for every emitted cell, the join over the stored per-centroid words equals the join over the raw observation words, and every stored word's [start, end) lies inside that envelope. Both are exact statements at any fold depth (the §8.2 join is a semilattice and a fold partitions the members), not bounds. The test also asserts at least one stored word is a toc_is_range — otherwise conservation would be trivially satisfied by untouched per-observation timestamps.
  • Below the knee (δ=512, 20 obs/cell): every centroid is weight 1 and its word is the exact observation instant — multiset equality plus a per-row value → instant map, since the multiset check alone is permutation-invariant and would pass a shuffled channel.
  • k-way order-independence: granules fed forward and reversed produce byte-identical payloads and byte-identical locations and times. This is what would break if the channel's ties were resolved by input position rather than by the word.
  • vs pooled, within the fold law: the folded centroid partition differs from pooled's, so the words differ — but cell_envelope over them is the same token either way, total weights are exact, and quantiles agree to < 1 m.
  • Dual channel (the blocked CA shape: build_tdigest_where strata + location: + temporal: per-centroid): per stratum, payload/locations/times row-aligned, locations inside that stratum's contributor hull, times conserving that stratum's instants.
  • Diagnostics: a missing clock column fails with the pooled path's named message; a where expression may read the derived toc_word column (namespace parity with pooled); the derived column is never spilled — the block schema is still just the read carrier.

Mutation check on the fold state (feeding every k-way part the first block's words) fails 5 of the 6 behavioral tests, so they are not passing by construction.

Full pytest: 4443 passed, 38 skipped, 1 failed — the failure is the known environmental test_lambda_build::test_function_build_succeeds. ruff check / ruff format --check clean on everything this PR touches (pre-existing on main and left alone: N818 in registry.py, the tests/data/benchmark/README.md format drift).

No bytes under tests/data/spec/ move: this changes no wire format, no attrs grammar and no spec marker, and the spill fold regime is documented as not byte-identical to single-block (§2.3 / #370), so no conformance fixture covers it.

Comment thread src/zagg/processing/spill.py Outdated
# that cell pooled, and the encode stays one pass over the
# block's rows. It also keeps ``_resolve_param``'s namespace
# equal to the pooled one, which carries the column too.
toc = _toc_word_column(col_arrays, self.config)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 from Claude (review)

Must-fix (or at least account for): the per-partition encode adds a transient peak the spill budget does not charge, on the one path that exists because memory is scarce.

toc = _toc_word_column(col_arrays, self.config)
col_arrays = {**col_arrays, TOC_WORD_COLUMN: toc}

The comment above this justifies per-partition on CPU grounds ("the encode stays one pass over the block's rows"). But per-cell encoding — what calculate_cell_statistics does at aggregate.py:487, and therefore what _chunk_outputs_exact does in the single-block regime of this same class — is also exactly one pass over the block's rows in total. The only thing per-partition buys is amortizing toc_source() / parse_utc() / mortie call overhead across cells. What it costs is peak memory, and observation_words is allocation-heavy: offsets, whole, (offsets - whole) * 1e9, frac_ns, the int64 combine, and the uint64 result are all full-length.

Measured in this worktree (uv run python, tracemalloc, 5M rows):

rows=5,000,000 input=40MB out=40MB peak_extra=243MB = 48.6 bytes/row

~49 bytes/row transient, ~8 bytes/row of which (toc) is then held for the whole cell loop. That lands at the worst moment: right after _group_columns' sort copies exist and del cells, cols has freed only the pre-sort originals.

Against the budget in _default_block_bytes: block ≈ 0.2 × mem × K, so one partition ≈ 0.2 × mem bytes. At ~20 B/row of spilled columns that is 0.01 × mem rows, i.e. 49 × 0.01 × mem ≈ 0.49 × mem of new transient; at ~40 B/row it is still ~0.24 × mem. _BUILD_MULT = 3 is documented as covering "read-back columns + sort copies + per-cell outputs" — this term is in none of those, and it is a factor-of-N_cells regression against the per-cell shape everywhere else in the codebase.

Concretely, one of:

  • (a) move the encode inside the for cell, (start, end) loop over cell_data[source["field"]] (exact parity with the pooled path, and with _chunk_outputs_exact one method up), or
  • (b) keep it per partition but write into a preallocated uint64 out array in fixed-size chunks, so the transient is bounded by the chunk not the partition, or
  • (c) if per-partition is deliberate, say so in _default_block_bytes' docstring and fold the term into _BUILD_MULT, so an operator sizing a tier is not surprised by a temporal config OOMing at a block size a located config survives.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 from Claude

Folded as option (a) — the encode moved into the per-cell loop, in 97b4a7e9.

_fold_block no longer builds a partition-wide toc_word column. The derivation now sits immediately after cell_data is materialized, which is the same point the pooled path derives it (calculate_cell_statistics, aggregate.py:487), and crucially before _resolve_param(f.where, cell_data) — so test_where_may_read_the_derived_word_column still passes:

cell_data = {col: arr[start:end] for col, arr in col_arrays.items()}
if self._needs_toc:
    # ... Per cell, not once per partition: this path exists because memory is
    # scarce, and the encode's transient allocation is unbudgeted by
    # ``_default_block_bytes`` / ``_BUILD_MULT``.
    cell_data[TOC_WORD_COLUMN] = _toc_word_column(cell_data, self.config)

The comment was rewritten to justify per-cell on parity + peak-memory grounds rather than per-partition on CPU grounds, and the _fold_block docstring line that said "encoded once per partition below" now says per cell. self._needs_toc is kept — it still gates the work here and in _check_fold_columns.

Comment thread src/zagg/processing/spill.py Outdated
#: same order ``sweep_overview.COMPANION_CHANNELS`` folds overviews by. The keys
#: are ``_aggregate_chunk_cells``'s (``times``, not ``temporal``, for the §8.3
#: sibling), which is the contract ``chunk_outputs`` returns under.
_CHANNEL_EMIT = {"locations": "locations", "temporal": "times"}

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 from Claude (review)

Judgment call: this is the fourth private table of the same channel order, and the fold now relies on dict insertion order — the exact thing sweep_overview.fold_digests refuses to do.

The order-of-record for these two channels now lives in four places:

  • sweep_overview.COMPANION_CHANNELS(("location", "locations", ragged_locations_name), ("temporal", "temporal", ragged_times_name))
  • write.RAGGED_CHANNELS("locations", "times")
  • aggregate._aggregate_chunk_cells — the inline declared["locations"] / declared["times"] build
  • and now spill._CHANNEL_EMIT + _channels()

sweep_overview.py:335 already spells out why that matters, for this same pair of channels:

The returned slots are always in COMPANION_CHANNELS order, whatever order the mapping was built in: every arm below normalizes through that table, because the k-way merge fixes its own return order by the same table and an arm that instead followed the caller's insertion order would hand back a toc word in the locations slot. The two word grammars are mutually accepting, so such a swap raises nowhere — it just writes false containment claims.

_fold_block and _finalize_kway do follow insertion order:

for kw, vec in zip(chans, words, strict=True):        # chans is a dict
...
**{kw: by_cell[cell] for kw, by_cell in word_parts.items()}
for kw, vec in zip(word_parts, folded[1:], strict=True)

This is correct today_channels() returns a fixed tuple and every dict is built from it, so insertion order is kernel order, and I traced all four zips. But strict=True checks length, never order, and both channels are same-length uint64 vectors, so the swap the precedent warns about is silent here too.

Cheap hardening, no behavior change: build chans / _digest_words / _digest_word_parts / ragged_channels by iterating one module-level ordered table (or import COMPANION_CHANNELS and key off it), rather than three separate comprehensions over _channels(f) plus a separate _CHANNEL_EMIT lookup — so adding a third channel is one edit instead of four, and the ordering claim is structural rather than a comment.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 from Claude

Folded in 21e6a3ea — collapsed to one module-level ordered table, mirroring sweep_overview.COMPANION_CHANNELS's 3-tuple shape so the two read the same:

_CHANNELS = (("location", "locations", "locations"), ("temporal", "temporal", "times"))


def _channels(f: _DigestField) -> tuple[tuple[str, str], ...]:
    """``(reducer kwarg, emission key)`` per channel ``f`` declares, in kernel order."""
    return tuple((kwarg, emit) for attr, kwarg, emit in _CHANNELS if getattr(f, attr))

_CHANNEL_EMIT is gone. Every site that zipped the kernel's returned words against a locally-built dict now zips against declared = _channels(f):

  • _fold_block — all four zips (k-way stash, first-write, the {kw}1/{kw}2 merge pairs, and the merged-words writeback), and chans is built from the table too;
  • _finalize_kway — branches on declared rather than the word_parts dict, and both the merge_tdigests_kway kwargs and the returned-words zip come off the table;
  • _chunk_outputs_merged — the ragged_channels keys and the per-cell append both iterate the table instead of self._digest_words[name].items().

The ordering claim is now structural rather than a comment; the strict=True/silent-swap reasoning moved into the _CHANNELS docstring, which is the one place it belongs. Behavior unchanged — tests/test_spill_crossblock.py, tests/test_spill.py, tests/test_sweep_overview.py and the full suite are green.


times = np.asarray(times, dtype=np.uint64)
contributors = np.asarray(contributors, dtype=np.uint64)
whole = int(cell_envelope(contributors))

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 from Claude (review)

Must-fix (test quality): this helper has no tightness lower bound, so a fold that widened every centroid to the whole-cell envelope passes every new assertion in TestTemporalMultiBlock.

Both claims here are upper-bound-shaped:

  • (a) cell_envelope(times) == cell_envelope(contributors) — satisfied by any partition, including "every word = the whole-cell join".
  • (b) every word's [start, end) lies inside that whole envelope — satisfied maximally by that same degenerate answer.

assert merged at line 648 doesn't close it either: an over-widening bug makes more words ranges, not fewer.

Compare the located sibling, _assert_ancestor_or_equal, which pins both ends — "(a) every location is the ancestor-or-equal of at least one contributor (it sits on a member's path)" is exactly the lower bound missing here. So a channel bug that folded, say, the whole partition's words per cell instead of per centroid would be caught on the located channel and not on the temporal one.

Concrete pin available for free, and verified on this branch's own fixture: a weight-1 centroid must carry that observation's exact word unchanged (_centroid_envelopes: "A single-member centroid returns that member's word unchanged"), so it must be a timestamp, never a range. Ran the seed=21, delta=16, obs=60 fixture from _run_spill:

weight-1 centroids: 18  of which RANGE words: 0
mean span fraction of whole-cell envelope per cell: [0.778, 0.882, 0.882, 0.724, 0.763, 0.882, 0.789, 0.882]

so there are 18 weight-1 centroids in the compressed run today, all timestamps. Adding to _assert_envelope_conservation (it would need the digest passed in) or to test_envelope_conservation_across_block_closes:

w1 = digest[:, 1] == 1.0
assert w1.any()
assert not mortie.toc_is_range(words[w1]).any()
np.testing.assert_array_equal(np.isin(words[w1], contributors), True)

That fails immediately under any over-widening, and it exercises the compressed regime (the delta=512, obs=20 below-knee test only covers the case where every centroid is weight 1).

Note also the second half of the span-fraction number above: at 0.72–0.88 of the whole envelope, the stored words are already nearly the cell envelope in this fixture, so a "narrower than the whole" check would not discriminate — the weight-1 pin is the one that bites.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 from Claude

Folded in dd207468 — the weight-1 lower bound now runs in test_envelope_conservation_across_block_closes' body (kept out of the helper, which test_located_and_temporal_ride_the_same_fold also calls without a digest in hand):

contributors = _contributor_times(dfs, grid, int(children[cell_i]))
_assert_envelope_conservation(words, contributors)
# Both claims above are UPPER bounds — a fold that widened every
# centroid to the whole-cell envelope would satisfy them. ...
singles = words[digest[:, 1] == 1.0]
assert len(singles) > 0, "no weight-1 centroid survived the fold"
assert not mortie.toc_is_range(singles).any(), "a weight-1 word widened to a range"
assert np.isin(singles, contributors).all(), "a weight-1 word is no contributor's"

Checked it bites: with the emitted words temporarily replaced by np.full(len(digest), cell_envelope(contributors)) — exactly the over-widening the upper bounds could not see — the test fails on a weight-1 word widened to a range. Mutation reverted; both parametrizations pass on the real fold.

_assert_envelope_conservation's docstring now says outright that both its claims are upper bounds and that a caller wanting a floor pins the weight-1 rows.

)


def _contributor_times(dfs, grid, cell, mask_fn=None):

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 from Claude (review)

Judgment call (test gap): this helper models the contributor set as "every row in the cell", which is only true because no temporal fixture has NaN heights.

build_tdigest drops non-finite values and drops each channel's rows with them (tdigest.py:351-359):

finite = np.isfinite(values)
if temporal is not None:
    temporal = _check_words(temporal, "temporal", values.shape)[finite]

so a NaN-height observation's instant is legitimately never stored. _contributor_times applies only the cell mask (and the optional stratum mask_fn), never np.isfinite(df["h_ph"]), and _assert_envelope_conservation's docstring asserts the stronger claim — "no instant dropped by a block close, none invented". That equality is exact only for the all-finite fixtures actually used.

Two consequences:

  1. The docstring overstates what is being pinned. Worth one clause: the equality holds over the finite-valued rows, because the reducer drops the rest.
  2. _granule_dfs already takes nan_cells=, and TestLocatedMultiBlock exercises it, but no test in TestTemporalMultiBlock passes it. I ran the mixed case by hand (nan_cells=(4,), located+temporal, 5 forced closes) and it behaves correctly — the all-NaN cell drops out of the ragged output while count stays 180 — so this is a coverage gap, not a bug. The interesting variant is a partially NaN cell, where the channel must drop exactly the NaN rows' instants; that one is currently untested on either channel, and it is precisely where a [finite]-mask misalignment between payload and companion would show up.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 from Claude

Both halves folded in 3df3741b.

(a) _contributor_times now masks to np.isfinite(df["h_ph"]), so the helper computes what its name claims, and its docstring says why (build_tdigest drops non-finite values and the companion rows that go with them, so a non-finite row's instant is never stored). _assert_envelope_conservation's docstring no longer overstates: the claims are exact "at any fold depth over the rows the reducer keeps", with an explicit note that "every observation" means every finite-valued one and that contributors must come from _contributor_times, which masks the same way.

(b) New TestTemporalMultiBlock::test_partially_nan_cells_keep_the_channel_row_aligned. _granule_dfs gained a nan_frac option that scatters NaNs through every cell's rows; it is an elif under the existing nan_cells branch and draws from the rng only when set, so TestLocatedMultiBlock's whole-NaN behavior and every existing seed are untouched. The test runs delta=512 over ≤60 rows/cell so nothing merges, then asserts:

  • the cell really is partially NaN (finite.any() and not finite.all());
  • count still covers every row while len(digest) == finite.sum();
  • words.shape == (len(digest),) and every weight is 1;
  • the word multiset equals the finite rows' instants, and the per-row map finite value -> instant holds for each (centroid, word) pair — the permutation-invariant multiset check alone would not catch a misalignment.

Checked it bites: rolling the temporal channel by one row inside _fold_block (np.roll(cell_data[TOC_WORD_COLUMN], 1)) fails the test on 18/24 mismatched instants. Mutation reverted.

Comment thread src/zagg/processing/spill.py Outdated
# there, as on the pooled path, so a ``where`` reading it is not a
# missing column. An absent CLOCK column is named by
# ``_toc_word_column`` itself, with the pooled path's message.
available = sorted([*available, TOC_WORD_COLUMN])

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 from Claude (review)

Minor: this makes three error messages assert something false, and quietly widens two checks that are not about where.

The comment justifies the addition for the where case only ("a where reading it is not a missing column"), and test_where_may_read_the_derived_word_column pins exactly that. But available is the same list the other two checks in this method consume, and the same list all three messages print:

f"column is not in the spilled block (available: {available})"

After this line, toc_word is advertised as a column of the spilled block in every one of those messages — while test_the_derived_word_column_is_never_spilled asserts, correctly, that it is not. An operator debugging a typo'd source on a temporal config gets a candidate list containing a column they cannot spill.

It also relaxes the source and location checks: source: toc_word or location: toc_word now pass _check_fold_columns where they previously raised. source: toc_word is a real declaration for the per-cell shape (see _toc_word_column's docstring), which validate_spill_fold refuses anyway, so this is not reachable in a mergeable config today — but location: toc_word would now sail past the named check and die inside mortie.common_ancestor on a toc word instead.

Cheapest fix keeping the where behavior: leave available alone and pass the widened list only to _unresolvable_names, e.g.

resolvable = sorted([*available, TOC_WORD_COLUMN]) if self._needs_toc else available
...
missing = _unresolvable_names(f.where, resolvable)

Separately, and for the same method: the clock column (output.time_source.field) is the one fold-resolved column that does not get the up-front named check this method exists to provide — it surfaces from _toc_word_column inside the partition loop, after the read-back, with a message that says "not in the cell data" when it is really partition data. test_missing_clock_column_named shows the message is fine; adding source["field"] to the loop above would just make it fail at the same place as every other missing column, and would let the message name the block schema.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 from Claude

Folded in bf0aca20available stays the block's true column list; a separate resolvable is used only by the two _unresolvable_names calls:

# ... It stays OUT of ``available``, which is the block's true contents: the
# source/location messages must not advertise ``toc_word``, and a
# ``location: toc_word`` must still raise here rather than reach
# ``mortie.common_ancestor``. ...
resolvable = sorted([*available, TOC_WORD_COLUMN]) if self._needs_toc else available

So the source: / location: / composition-source: membership checks and all four error messages are back on the real spilled columns, and only the where and conf_* resolution checks see toc_word. test_where_may_read_the_derived_word_column still passes.

New TestFoldColumnDiagnostics::test_location_may_not_name_the_derived_word_column pins the tightened half both ways: a location: toc_word field must raise the named location: ValueError, and the message's available: list must not contain toc_word. Checked it bites — with available = resolvable restored it fails DID NOT RAISE ValueError.

Declining the clock-column pre-check, on line budget rather than merit. spill.py is at 1,198 of the 1,200 §4 raise trigger and a named up-front check plus its comment is ~7 lines. The failure mode is already good: _toc_word_column raises the pooled path's own named ValueError citing output.time_source.field, _check_fold_columns is a diagnostic-quality improvement rather than a correctness gate, and test_missing_clock_column_named already pins that message through the fold path. Worth revisiting if the module gets a length raise.

Comment thread src/zagg/processing/spill.py Outdated
(issue #477): single-block through the pooled machinery unchanged,
multi-block through the per-channel fold state below — so a field may
declare ``location:`` and ``temporal: per-centroid`` together, each folded
in the same merge as the payload it describes.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 from Claude (review)

Judgment call (§4 module cap): spill.py lands this PR at 1,199 lines — one line under the 1,200 raise trigger, up from 1,142 on origin/main.

$ git show origin/main:src/zagg/processing/spill.py | wc -l   -> 1142
$ wc -l src/zagg/processing/spill.py                          -> 1199

CLAUDE.md §4: "If a file is heading past 1,200, stop and raise it (issue comment) before splitting it or continuing." +57 in one PR with 1 line of headroom left is the definition of heading past it — the next PR that touches this module is blocked before it writes a line, and it will not be the PR that chose to spend the budget.

Worth raising on issue #477 now rather than deferring. There is also slack to reclaim here without a split: the same fact about the two channels riding one merge is restated in four places in this diff — this docstring, the _digest_words block comment (~L623-630), the _fold_block docstring (~L843-852), and the inline comment at the elif chans: arm (~L913-914). One canonical statement plus pointers would give back a dozen lines and read better.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 from Claude

Folded the duplication half; leaving the length concern standing for review rather than acting on it.

Post-fold wc -l src/zagg/processing/spill.py = 1,198, down from 1,199. Moving the toc encode into the per-cell loop (97b4a7e9) removed the per-partition block, and I trimmed the repeated channel prose while collapsing the channel table (21e6a3ea): the fact that every declared channel rides the same merge in the kernel's fixed order is now stated once, in the _CHANNELS docstring, and the four restatements — the class docstring's multi-block bullet and closing paragraph, the _digest_words / _digest_word_parts comments, _finalize_kway's docstring, and the ragged_channels comment in _chunk_outputs_merged — now point at _CHANNELS instead of re-deriving it. _check_fold_columns's docstring was tightened in bf0aca20 for the same reason.

That keeps the module under the 1,200 raise trigger, but the margin is 2 lines and the reviewer's point stands on its own: the next change to this file has essentially no headroom. Whether to raise the cap or split the module is a §4 decision for review — a fold subagent applying it would be a scope change, and filing it on issue #477 would be a side-effecting directive I am not authorized to take from a review comment (CLAUDE.md §6). Flagged here so it is on the record with the PR.

@espg

espg commented Aug 18, 2026

Copy link
Copy Markdown
Member Author

🤖 from Claude

Adversarial self-review posted six inline findings; all six are answered on their threads, four as fold commits (one per finding, files scoped to that finding):

Finding Fold What changed
per-partition toc encode adds ~49 B/row of transient outside the spill memory budget 97b4a7e9 Encode moved into the per-cell loop, exactly where calculate_cell_statistics does it — the reviewer's option (a). Parity with pooled and with this class's own single-block arm, and the transient disappears.
channel order asserted by comment + carried by dict insertion order in four zips 21e6a3ea Collapsed to one module-level _CHANNELS table (declaration attr, reducer kwarg, emission key) mirroring sweep_overview.COMPANION_CHANNELS; every fold and emission site zips against the table. The ordering claim is now structural.
_check_fold_columns widened available for all checks, so source:/location: loosened and three messages listed a column that is never spilled bf0aca20 available stays the true spilled-column list for membership and messages; a separate resolvable feeds only the where/param resolution. New test pins that location: toc_word still raises the named error.
envelope conservation had only upper bounds — a fold that widened every centroid to the whole-cell envelope would pass dd207468 Weight-1 floor added: at least one weight-1 centroid exists, none of their words is a range, each is a member of the contributor set. Mutation-verified against an over-widening _centroid_envelopes.
the "no instant dropped" claim is exact only over finite-valued rows, and no temporal fixture exercised NaN 3df3741b _contributor_times masks to np.isfinite(h_ph), both docstrings corrected, and _granule_dfs gained nan_frac for a partially NaN cell — the case where a [finite]-mask misalignment between payload and companion shows up. Mutation-verified with np.roll on the word column (fails 18/24 instants).

Two halves declined, stated on-thread rather than silently: the optional up-front clock-column check in _check_fold_columns (line budget; _toc_word_column already raises the pooled path's named message and test_missing_clock_column_named pins it through the fold path), and the §4 module-length concern, which is a decision to make rather than something a review comment authorizes — it is now under "Questions for review" in the PR body.

Harness re-run on the folded code (same shard, 29 granules, 1.50 M photons), confirming the fold arm still runs with temporal=True after the encode moved:

=== spill arm spill_buf20_fold_temporal: buf=20 block_bytes=8388608 blocks_closed=1
  ingest+flush=0.2s reduce(84 chunks)=11.4s total=11.5s rss_hwm=691MB (start 444)
=== spill arm spill_buf5_fold8_temporal:  buf=5  block_bytes=8388608 blocks_closed=4
  ingest+flush=14.2s reduce(84 chunks)=6.9s total=21.1s rss_hwm=691MB (start 691)
=== spill arm spill_buf20_fold_notemp:    buf=20 block_bytes=8388608 blocks_closed=1
  ingest+flush=0.1s reduce(84 chunks)=4.9s total=5.0s rss_hwm=696MB (start 691)

The per-cell encode costs ~3 s more reduce on this shard than the per-partition one did (11.4 s vs 8.4 s) — that is the parity trade, and it is the same per-cell shape the pooled path has always paid, now charged against a bounded rather than a partition-sized working set.

State: spill.py 1,198 lines; nothing under tests/data/spec/ or in docs/specification.md moved; full suite 4,445 passed / 38 skipped / 1 known-environmental failure; ruff check and ruff format --check clean on everything touched.

@espg
espg marked this pull request as ready for review August 18, 2026 00:28
@espg espg added the waiting label Aug 18, 2026
@espg

espg commented Aug 18, 2026

Copy link
Copy Markdown
Member Author

🤖 from Claude

Sync-merge with main landed (eb15a53a), now that #475 and #478 are merged. The SpillOverflowError seam merged cleanly (no conflicts — git merge origin/main, ort auto-merge across spill.py, streaming.py, test_spill_crossblock.py). Local suite at the merge head: 4478 passed, 38 skipped (ruff clean on the CI select; the one local failure is the known environmental test_lambda_build wheel-resolution issue, unrelated).

§4 line-cap update on the open spill.py question: the sync moved the file from 1,198 → 1,234 lines — now past the 1,200 raise trigger. For attribution: main itself is at 1,178 after #475 (the block_bytes surface lives in the same module), so this PR's own delta is +56. Raising per §4 rather than splitting unilaterally: (a) accept the overage for this PR and split in a follow-up if wanted, or (b) split now (the block-state _CHANNELS table + fold helpers are the natural seam). Lean: (a) — a pre-merge split would churn the already-reviewed fold sites.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

spill: carry the temporal companion through the cross-block fold (block_bytes unusable on temporal stores)

1 participant