feat: DAG compaction with receiver-side reclamation, perf and reliability improvements

[samyfodil]

This PR addresses the long-standing unbounded-growth problem (#22): a crdt.Datastore only ever grows, even when keys are deleted. It adds compaction (fold a DAG's live state into snapshot blocks and purge the covered history), receiver-side reclamation (replicas that witnessed the old history reclaim their local copy too), and a set of performance/reliability fixes found along the way.

Compaction here is coordination-free, like every other operation in this package: any replica may compact at any time, concurrently with puts, deletes, and other compactions, and all replicas converge to exactly the state the uncompacted history would have produced. No writer quiescence, no consensus, no membership knowledge.

Compaction (Datastore.Compact)

Compact(ctx, dagName) walks the DAG locally, computes every key's winning (value, priority) plus the tombstones that must survive, writes them as one or more snapshot blocks (new Delta.snapshot flag), replaces the covered heads, purges the old blocks/set entries, and broadcasts the new head. Key design points:

• Stable element ids. A snapshot element keeps its original element id — compaction re-homes the element's storage (its delta now lives in the snapshot block) without changing its identity. This is what makes compaction semantically invisible to the OR-set: a tombstone created concurrently with a compaction targets the id it observed, and that id still names the element after any number of compaction generations. Element markers record the hosting block as an alias in their value (varint(priority) || alias-CID, additive format, no migration); findBestValue and the purge/compaction walks resolve reads through alias-or-path.
• Per-element priorities are preserved (new Element.priority field, used only in snapshots): a snapshot reproduces exactly the same conflict-resolution state on every replica.
• Snapshot links are covered-heads bookkeeping only — walkers never descend them. A fresh replica syncs the full live state from the snapshot block(s) alone.
• Two-generation tombstone rule: a tombstone is carried into the snapshot while its target may still be held un-killed by a lagging replica, and dropped one generation later. Carried tombstones are written with a re-affirming alias so the kill survives the purge of its target's block.
• Sibling-split: state larger than MaxBatchDeltaSize splits into sibling snapshot nodes that all become heads and collapse on the next write.

Concurrency semantics (tested, not just claimed)

• Compact ∥ Delete (TestCompactConcurrentDeleteWins): a delete concurrent with an unseen compaction wins in both exchange orders — no resurrection, because the tombstone's target id survives compaction.
• Compact ∥ Compact, same view (TestConcurrentCompactSameView): compaction is deterministic (sorted keys, stable ids, generation id derived from covered heads), so two replicas compacting the same view produce byte-identical snapshot CIDs — content-addressing collapses the race into a single generation.
• Compact ∥ Compact, divergent views (TestCompactDivergentViewsConverge): replicas compacting different views (including unseen deletes and overwrites) converge to exactly the state of a never-compacting oracle replica pair fed the same operations.
• Late tombstones (TestSnapshotElementKilledByLateTombstone): a snapshotted element is killed by a tombstone arriving before or after the snapshot, on full-history and snapshot-only replicas alike.

Receiver-side reclamation

Compact only reclaims space on the replica that runs it. Snapshot deltas therefore carry generation metadata (snapshotTotal, snapshotId): once a replica has merged every sibling of a generation, it purges its own local copy of the covered history (Options.ReclaimOnSnapshot, default on). Waiting for all siblings avoids transiently dropping keys whose surviving value lives in a not-yet-merged sibling. The path is soft-failure (logged, never marks the store dirty); Datastore.ReclaimCompacted(ctx, dagName) is the explicit/recovery entry point (crash-missed generations, disabled auto-reclaim, snapshots without metadata).

Performance

Benchmarks (i9-12900HK, in-memory store, -benchtime=20x):

Benchmark	before	after
Delete of a key with 100 versions	276 ms/op	~1 ms/op
KeysOnly query, 2000×1KB keys	4.0 ms/op	~1.7 ms/op
Put	95 µs/op	~40 µs/op

• putTombs groups tombstones by key: one findBestValue scan per key instead of one per tombstone (a Delete() of a key with N versions was O(N²) DAG fetches).
• Element markers store their priority (and, post-compaction, hosting alias) in the marker value, so findBestValue only fetches delta blocks for max-priority candidates. Migration v1→v2 backfills existing markers (batch-written, idempotent, tolerates unfetchable blocks; empty markers keep a permanent read fallback).
• Elements() honors Query.KeysOnly.

Disk footprint

Measured effect of one Compact on a replica loaded with 1,000 keys × 5 versions × 1KB values, 200 keys then deleted (logical bytes; single replica):

Component	before	after
Blockstore (DAG blocks)	5,200 blocks — 5.52 MB	1 block — 0.95 MB	−83%
Datastore set state (/s)	7,600 keys — 1.32 MB	3,400 keys — 1.06 MB	−20%
Datastore processed markers (/b)	348 KB	348 KB	kept, by design
Total	7.19 MB	2.36 MB	−67%

The blockstore dominates (every version's value is embedded in its delta block); the reduction scales with version churn — at 100 versions/key the history collapses ~100:1. The /s remainder includes the carried tombstones for the deleted keys, which drop at the next generation (two-generation rule). Processed markers (~67 bytes/op) are intentionally kept so stale rebroadcasts of purged history stay no-ops. Receivers get the same reduction via reclamation; fresh replicas only ever download the snapshot. Note these are logical deletions — the underlying store releases the bytes on its own schedule (Pebble background compaction; Badger value-log GC).

Reliability

• processNode unreserves queued children on merge failure (previously a failed merge left the CID reserved forever and the branch stalled until the next repair interval).
• DAG fetches issued by the set (findBestValue, the migration) are bounded by DAGSyncerTimeout — a missing block can no longer hang a local Delete() indefinitely.
• purgeKeyBlocks skips no-op value rewrites, so Put/Delete hooks only fire on real changes.

Rollout notes (not coordination requirements)

• All replicas should run a compaction-aware version before Compact is called anywhere — an old-code replica would descend snapshot links into purged history and mis-attribute element priorities. This is release ordering, the same as any format addition.
• Compacting while replicas are reasonably synced folds the most history (efficiency only; correctness holds regardless).

Tests

Unit coverage goes from 75.2% to 90.8%; the full suite and the -race suite pass. New infrastructure includes a per-replica-blockstore harness (a purge on one replica is invisible to others, as in production — the shared-blockstore harness cannot express this), end-to-end compaction/reclaim scenarios (in-sync, lagging, fresh, sibling-split, crash recovery, legacy snapshots), the concurrency-semantics suite above, fault-injection suites, and benchmarks.

The proto changes are backward compatible (new fields only). The storage migration runs automatically on first open (v1→v2).