blob storage for large atoms

[matthew-levan]

Blob Store for Large Atoms

Summary

Introduces a blob store that lets Urbit handle atoms larger than 32 MiB
without ever holding their bytes on the loom. Such atoms are stored as
content-addressed files under $pier/.urb/bob/ and referenced on the loom by a
bob atom — an indirect atom whose body points at small loom-resident
metadata instead of the bytes themselves. Serialization, the event log, IPC,
Clay sync, HTTP, and Ames all handle bob atoms transparently, materializing
bytes from disk (zero-copy via mmap) only when a jet actually needs them.

Motivation

The loom is a fixed-size memory-mapped region. Before this change, committing a
2 GiB file via |commit %base required allocating a 2 GiB atom on the loom,
jamming it for the event log, and shipping the jammed bytes over the king/serf
pipe — multiplying peak memory and making large-file support impractical.

Design

Bob atoms and the blob bank (pkg/noun/allocate.h, imprison.c, retrieve.c)

A bob atom is an indirect atom whose len_w MSB is set (u3a_blob_flag); its
buf_w[0] holds the loom offset of a u3a_blob metadata record rather than
data. The bank blb_p is a HAMT in u3v_home keyed by
bid = (mug_h << 32) | seq_h, marked during GC and rewritten during pack.

The design uses a single reference counter. A blob file is deletable iff
use_w == 0, where use_w = eve_w + les_h + live bob-atom cardinality:

• eve_w — event-log refcount (rebuilt from the LMDB BLOBS table on
  replay/chop)
• les_h — active king-held leases (durable in the LMDB LEASES table; the
  loom copy is zeroed every boot and rebuilt from that table)
• atom cardinality — number of live bob atoms pointing at this record, updated
  only on atom alloc/free

u3i_blob(mug, seq) allocates a bob atom. Retrieve functions (u3r_met,
u3r_bytes, u3r_mug, u3r_sing, …) and the new zero-copy u3r_view detect
bob atoms and source bytes from disk on demand. _ca_take_atom masks len_w
with u3a_blob_mask so bob atoms copy correctly across road transitions.

Blob store (pkg/vere/blob.c, blob.h)

Content-addressed files at .urb/bob/<mug>/<seq>. Per-bucket lockfiles hold the
next sequence number under fcntl advisory locking; install dedups byte-for-byte
against existing files in the bucket. A staging area .urb/bob/stg/ holds
in-flight uploads (cleaned at boot). The API covers save, load, mmap/unmap,
wipe (removes empty buckets), walk (enumerates the store), and staging-file
install. Includes a Windows mmap shim. Earth (king) is the sole writer;
Mars (serf) is read-only.

Ram/tap serialization (pkg/noun/serial.c, serial.h)

ram extends jam's bit-encoding with a 2-bit tag — normal atom, cell, backref,
or blob ref (mat-encoded mug then seq) — so bob atoms cross IPC and the
event log as compact references, never as bytes. tap decodes, rebuilding bob
atoms via u3i_blob. The wire format prepends a 5-byte "RAM\0" + version
header. Jam/cue are unchanged; old VER1/VER2 jam data is still read via fallback.

A streaming cue (u3s_cue_xeno_blob + u3_blob_bsink) lets rock import
stream over-threshold atoms straight from the jam bitstream into the blob store
in 1 MiB chunks — the atom never touches the loom — so |cram/rock import of
large state no longer needs --loom headroom.

IPC and king leases (pkg/vere/newt.c, lord.c, mars.c, king.c, vere.h)

The newt wire protocol gains a version byte (0x00 jam, 0x01 ram); all sends
use ram, all receives try tap then fall back to cue.

Leases are king-acquired, mars-issued, keeping a blob alive only in the
window between a %blob install and the commit of the event that references it.
The king stages a large file and sends a %blob writ; mars installs it and
issues a lease (les_h += 1) with a 15-minute TTL tracked in a c-heap min-heap.
The king renews via %blas every 5 minutes for blobs it still references and
releases via %blrl when its last reference dies. Renewal never resurrects a
deleted blob's bank entry.

Leases are durable. Each is a row in an LMDB LEASES table (MDB_DUPSORT,
keyed by blob id; the value pairs the expiry with a unique lease id, so multiple
live leases coexist on one blob and a specific row can be deleted). The row is
committed before mars acks the %blob install, so a mars crash in the
install→commit window cannot lose a lease the king believes it holds. %blrl
and TTL expiry delete the corresponding row; %blrl retires exactly one lease
unit. At boot, _find_home zeroes the snapshot's les_h and _mars_play_leases
rebuilds it from the table, preserving each lease's remaining TTL and pruning
expired or file-missing rows. To make the durable row trustworthy,
u3_blob_move_stg now fsyncs the bucket directory after the staging rename.

LMDB ref tracking, chop, and boot GC (pkg/vere/disk.c, db/lmdb.c)

Each committed event's blob ids are written to a BLOBS table keyed by event
number. Replay and |chop rebuild eve_w from that table over the retained
log. u3_disk_blob_gc reconciles .urb/bob/ against the bank — deleting blobs
at use_w == 0 and on-disk files with no bank entry — and now runs both under
|chop and at boot (after replay rebuilds eve_w and _mars_play_leases
restores les_h). Durable leases are what make boot-time reclamation safe: a
genuinely orphaned file is no longer indistinguishable from one a live lease
still protects.

Versions and migration (pkg/noun/version.h, disk.c)

• U3V_VER6 (home-road format): adds blb_p. _find_home migrates a v5
  snapshot in place and stamps ver_d; idempotent across a crash before the
  next checkpoint. Bitness migrations are only defined for U3V_VERLAT, keeping
  paths linear rather than a version×bitness grid; both disk entry points reject
  older snapshots with an actionable error.
• U3E_VER3 (epoch format): image.bin + ram-encoded event log. VER2→VER3
  rolls over to a new epoch.

Blob-aware jets (pkg/noun/jets/...)

Atom jets that read raw bytes — cut, met, can, cat, end, rap,
rip, rsh, sew, swp, the hash/crypto jets (shax, sha1, blake, keccak,
ripe, muk, adler, crc, aes, chacha, ed25519), bytestream, zlib — source
bytes from blob files (via u3r_view mmap) instead of loom buffers. Adds a
jam_shax jet that fuses jam and shax for blob hashing.

IO: HTTP, Unix, Mesa (pkg/vere/io/{http,unix,mesa}.c)

• HTTP: response bodies backed by bob atoms mmap the blob file and slice
  the mapping into the body chain — bytes never enter the king's loom; inbound
  bodies over threshold go straight to the blob store.
• Unix/Clay: syncs of large files stage to .urb/bob/stg/ and install via
  the %blob writ, injecting %into from the install callback; %ergo
  write-back streams from the store file in fixed chunks; the soft "has it
  changed?" check compares mugs without reading bytes.
• Mesa: reads packet atoms through u3r_view, and stores reassembled jumbo
  frames over threshold as blobs.

Meld, cram, and replay integrity

• |meld/|pack: _melt_cmp_atoms compares bob atoms by bid (not a memcmp
  that would read off the end of the stub); a deep u3a_blob_sane() walk runs
  after every pack/meld, with a cheap floor check at u3e_save.
• |cram: serializes through the blob-aware jam, expanding bob bytes (mmap'd
  from the store) into the rock so rocks stay portable; the home road is
  untouched. Deprecated u3u_meld and the unused _cu_* machinery are removed.
• Replay: a blob file missing from the store while the log still references
  it now fails replay hard (_mars_play_blobs reports every missing file with
  its event number and exits nonzero) instead of silently deferring the error.

What this does not change

• Jam/cue format and Nock semantics are unchanged; bob atoms are a pure runtime
  optimization with no Hoon/Arvo changes.
• Rocks remain fully self-contained and portable (cram expands blob bytes into
  the jam).
• No protocol changes are visible outside the king/serf boundary.

Status

Implementation is feature-complete. Test suites accompany the major
subsystems: blob_tests.c (incl. _test_sane/_test_meld invariant checks and
_test_lease* covering durable-lease round-trip, persistence across a restart,
and the multi-lease-per-blob invariant), serial_tests.c (incl. streaming-cue
coverage), retrieve_tests.c, and newt_tests.c. CI runs full replay of
fixture piers in both 32- and 64-bit builds.

Known limitations

• u3i_bytes takes a c3_w (uint32_t) length, capping a single materialized
  loom atom at 4 GiB. Blobs larger than this live on disk but cannot be fully
  realized on the loom; jets that stream via u3r_view are not subject to this.
• Each retrieve maps the blob from disk; there is no cross-call materialization
  cache within a single Nock event. u3r_view (zero-copy mmap) keeps this
  cheap, but repeated access still re-maps.

Future work

This PR establishes the blob store but only populates it at I/O ingress. Several
follow-ups extend it into a general large-atom memory tier:

Big-atom arithmetic / computation. Operating on a blob atom currently
round-trips the whole thing through the loom: u3i_vint, u3r_mp, and friends
call u3r_blob_load to materialize before computing. We want operators and jets
that compute directly against the mmap'd blob via the existing u3r_view
zero-copy path, and — where a result is itself large — write it straight back to
the store instead of materializing on the loom. The cheap single-pass cases come
first (increment via a one-pass carry over the mmap, comparison, slicing/cut,
which already read zero-copy), expanding toward the operations that big atoms
actually flow through.

A %blob hint. A Nock 11 hint, emitted from Hoon with ~>, that lets app
developers tell the runtime "atoms produced in this scope are expected to be
large." It plumbs through the existing dynamic-hint machinery — whitelisted in
_n_bint, handled in _n_hint_fore / _n_hint_hind (pkg/noun/nock.c) —
alongside hints like %bout and %hela. It is purely advisory (no semantic
change to the computation): the runtime uses it to bias allocation and
blobification decisions for results computed under the hint, e.g.
blobify-on-produce even below the size threshold, or to prime the automatic
detector below.

Automatic blob detection in the event loop. Nothing blobifies during normal
event processing today. We want a post-poke pass — around _mars_sure_feck
(pkg/vere/mars.c), after u3v_poke runs the kernel and before effects route
to drivers — that walks the new state and effects and externalizes atoms over
the threshold straight to the store, so a computation that produces a large atom
doesn't bloat the loom. This reuses u3_blob_save (pkg/vere/blob.c),
including its byte-for-byte _blob_dedup, so a recomputed identical atom maps
back onto the existing (mug, seq) rather than writing a duplicate. The %blob
hint feeds this pass as an explicit signal.

A blobify sweep in |pack. Extend |pack (u3m_pack, pkg/noun/manage.c)
— or a sibling command — with a pass that, during loom compaction, scans
reachable atoms and blobifies any exceeding U3_BLOB_THRESH, reclaiming loom
space. It fits naturally between the existing seek (u3a_pack_seek) and move
(u3a_pack_move) phases. Unlike the inline detector, this runs over everything
reachable rather than just freshly-produced atoms, so it catches large atoms that
accumulated before the detector existed or slipped below its per-event view.

Together these let large atoms originate, persist, and be computed on without
ever being forced to occupy the loom in full.