fix: make the chunk pool's "safe to take away" predicate true, not approximately true

[emfdavid]

Summary

Closes #33, closes #34, closes #35 — three bugs with one root cause. Also closes #38
(the sizing decision and the diagnostic that follows from it; second commit). Eviction eligibility
was k not in self._pinned and s.ready, spread across five loosely-coupled fields
(remaining, ready, claimed, error, _pinned), and each issue is a different field
lying:

• ChunkPool.fail() marks a slot ready while tile tasks are still writing it #33 — fail() had to set ready = True to wake a waiter, because that was the only
  lever available. That simultaneously declared a half-written slot a finished cache entry
  while sibling tile tasks were still writing into it, and left the poisoned slot resident,
  so the next epoch took try_admit's resident branch and re-raised the stale error forever
  instead of refetching.
• ChunkPool.unpin_all() clears pins globally, corrupting a concurrent iteration #34 — unpin_all() cleared the pin map globally. With two producers over one pool
  (zip(ds.train, ds.val), documented in buffers.py:238-247), one iteration's epoch
  boundary stripped the other's pins and its in-use chunks became eviction candidates
  mid-gather.
• _Slot.claimed is a single bool, so one iteration's claim satisfies another's wait_ready #35 — claimed was a single bool, so one iteration's claim satisfied another's
  wait_ready. That iteration then gathered a chunk it never referenced, and its release
  decremented someone else's count.

All three produce plausible data, which throughput, shapes and smoke tests all pass —
only byte fingerprints catch them.

The shape

A slot now carries one explicit SlotState (FILLING → ASSEMBLED → READY, FAILED
terminal) advanced in exactly one place (_advance), plus two counters that answer one
question each:

• writers — tile tasks running, so eviction is never racing a live write.
• pending — tiles not yet delivered, so completeness is separate from quiescence.

References become owner-scoped (dict[key, dict[owner, int]]) and a reference is that
owner's claim, so claimed is deleted. One iteration = one owner, minted by _iterate and
shared with its scheduler (Scheduler.owner).

Every tile write happens inside pool.tile_write, a scope that releases on every exit
path — including cancellation at an await, which an early break triggers via
cancel_futures=True and which no explicit call site can cover.

unpin_all() splits into release_owner() and reset_epoch_counters(). Releasing
references and zeroing counters are unrelated jobs on different clocks, and bundling them is
why the reset had to unpin globally. Cleanup also moves into _iterate's own teardown, so
an abandoned pass cleans up after itself rather than relying on the next pass's prologue —
which, now that owners are scoped, would be a different owner and could not.

For reviewers

Behavior is meant to be unchanged for a single iteration. The one intended behavioral
change is for concurrent iterations, below.

Most valuable second look, in order:

1. ChunkPool._advance — the only function that moves state. Worth checking the
   quiesced-but-incomplete case (a cancelled pass leaves an abandoned partial: stays
   FILLING, dropped by release_owner) and that a FAILED slot is deliberately not
   dropped there, because a consumer still has to observe the error.
2. ASSEMBLED is load-bearing, not ceremony. The chunk transform and the persist
   write-back run outside the lock between the last tile landing and publication, and
   another thread can observe that window. A predicate derived from a tile counter alone
   would call it evictable. This is the one place the design note's proposed
   assembled = writers == 0 and error is None would have introduced a bug.
3. writers counts started tasks, not planned ones. Initialising it to the tile count
   at admit is wrong: a cancelled pass leaves tiles never started, so the slot would never
   quiesce and its budget would never be reclaimable.
4. pin_keys now notifies. Now that a reference is the claim, pinning can be the event
   that satisfies a parked wait_ready; under the old shared flag it never could.

Two notes on the tests:

• The original ChunkPool.unpin_all() clears pins globally, corrupting a concurrent iteration #34 reproducer asserted that A's pins survive B's prologue. It was not a
  valid discriminator — A's own producer releases those pins in the background within
  ~0.3 s whether or not B ever starts (measured), so it would fail with the bug fixed. It
  is replaced by a deterministic pool-level test of owner-scoped release; the byte-fingerprint
  test remains the end-to-end statement.
• deliver_tile is sugar over tile_write, not a second lever — it opens the same scope, so
  the writer count still moves in exactly one place.

Intended behavioral change: concurrent iterations need a larger budget

Two concurrent iterations now need a budget covering both working sets; the concurrent
test went from 20 → 32 chunks. This is not a leak — verified that a single iteration grows
peak-pinned to whatever budget it is given and works at every size, unchanged. The old global
unpin was spuriously freeing the other iteration's pins, so any budget that "worked" for
zip(ds.train, ds.val) before was relying on the bug.

The auto-sized default is still sized for one iteration, and that is now the settled
decision (#38): the engine cannot know how many iterations a caller intends to run, so an
automatic multiplier would silently cost memory in the single-iteration case that is almost
every case. Running several is an explicit choice, so sizing for it is the caller's.

The second commit makes that decision liveable, since the whole cost then falls on the
diagnostic. Starvation previously advised "raise cache_budget_bytes, or lower batch_size /
block_chunks" — correct, but not actionable when every resident chunk is legitimately
referenced and the caller cannot see why. It now names the cause:

... Nothing can free a slot, so this would hang. The working set is larger than the budget it
was sized for -- raise cache_budget_bytes, or lower batch_size / block_chunks. NOTE: 2
iterations are sharing this pool (e.g. `zip(ds.train, ds.val)`, or two DataLoaders); each
needs its own working set resident at once, so the budget must cover all 2.

Plus docs/tuning.md §"Several iterations at once multiply the budget" with the sizing rule
and why the default is not multiplied, and the same caveat at the auto-sizing site and on the
docstring that advertises the shared-pool configuration — so it is not carried by prose alone.
No example uses zip(ds.train, ds.val) as a runnable pattern; the mentions are the docstrings
and tests that advertise it, which is where a reader learns it is supported, so those are what
carry the caveat.

active_owners counts from mint to release, not from first pin. My first implementation
counted owners currently holding a reference and the note never fired — the second iteration
starves before it pins anything, which is exactly the case the message exists for. The test
caught it.

Author attestation

• I have reviewed every change in this PR, I can explain why each one is correct, and I
  have verified the claims made in this description.

Left unchecked deliberately — this was drafted by Claude, and that box is the human
author's to tick.

Checklist

• Tests added or updated — tests/test_pool_lifecycle.py reproduces all three; confirmed
  failing before the change for the documented reasons
• uv run ruff check src tests bench examples, uv run mypy src bench examples and
  uv run pytest -q are green locally (313 passed, 12 skipped)
• Docstrings and API docs for any new or changed public surface
• User-facing behavior documented in docs/*.md — not done. The concurrent-iteration
  budget requirement is user-facing and belongs in docs/tuning.md; deferred to Auto-sized residency budget assumes one iteration, so concurrent iterations deadlock #38 so the
  guidance and the auto-sizing fix land together rather than documenting a number that is
  about to change.
• A bullet added under ## Unreleased in CHANGELOG.md
• No load-bearing invariant is broken — no hot-path change (the state transition is the
  same lock sections the completion counter already used), Batch untouched, parallelism
  still in the event loop
• Touches ChunkPool, the scheduler, and cross-thread readiness → free-threaded run
  passes: 80 passed, 1 skipped on 3.13t with PYTHON_GIL=0 (pool, lifecycle, scheduler,
  residency, prefetch, window); re-run after the second commit, 70 passed / 1 skipped
• No performance claim.

[emfdavid]

Looks clean

Ran:
PYTHON_GIL=0 uv run --python 3.13t python -m bench.probe_decode --max-chunks 64 --repeats 5 --decode-threads 1,2,4,8 --no-raw and uv run python -m bench --engines insitu,memory --chunk-sizes 1 --epochs 2 --repeats 5 - no regression

[emfdavid]

Pushed a second commit (`0ce254e`) after your review — flagging since it lands on an already-reviewed PR.

Decision recorded: size for one is the correct default, so #38 needed no auto-sizing change. That puts the whole cost on the diagnostic, which is what this commit fixes: the starvation error now names how many iterations share the pool and the pattern that produces it, docs/tuning.md gains the sizing rule and the reason the default is not multiplied, and the caveat also sits at the auto-sizing site and on the BatchBuffers docstring that advertises the shared-pool configuration.

No example uses zip(ds.train, ds.val) as a runnable pattern — examples/advection/train_tf.py:80 is zip(grads, ...) for TF gradients, unrelated. The mentions are docstrings and tests that advertise it as supported, which is where a reader learns it, so those carry the note.

One implementation note: active_owners counts from mint to release, not from first pin. My first version counted current pin-holders and the note never fired — the second iteration starves before it pins anything, which is exactly the case the message exists for. The test caught it.

Verified: 299 passed / 20 skipped against a 298 / 20 baseline on the same tree and extras (delta is exactly the new test); ruff, mypy, mkdocs --strict clean; free-threaded 70 passed / 1 skipped on 3.13t with PYTHON_GIL=0. This closes #38, so the PR now closes #33, #34, #35, #38.