Skip to content

Bin streaming: zero-copy object views, one decode path, and batch lookup - #216

Merged
Crauzer merged 3 commits into
mainfrom
feat/bin-stream-views
Aug 30, 2026
Merged

Bin streaming: zero-copy object views, one decode path, and batch lookup#216
Crauzer merged 3 commits into
mainfrom
feat/bin-stream-views

Conversation

@Crauzer

@Crauzer Crauzer commented Aug 30, 2026

Copy link
Copy Markdown
Member

Implements the PROP half of docs/design/bin-streaming.md (sections 4.2-4.5, 8, 9 and 14) on top of the foundation #213 landed.

What lands

One internal layout core. stream::layout owns the byte-level knowledge every reader shares: where a value starts, how far it runs, what its header declares, how to decode a leaf. A cursor carries the numbering its bytes were written under, so nothing threads a legacy flag through every call and a slice can never be read under the wrong numbering. Two renderers sit on it and cannot drift - the borrowed views, and an owned decode that builds through the value model's checked constructors, so InvalidNesting, InvalidKeyType and MismatchedContainerTypes stay the value model's business. The module is crate-internal; only Numbering is public, since the latch reports it.

Zero-copy views (#208). Descending buffers an object's declared byte range once and views it in place. ObjectView iterates properties and finds one by name hash, PropertyView answers the shape and the element count from the header bytes alone, and ValueView mirrors PropertyValueEnum variant for variant, so Elements[3].Position reads one container index and two properties and nothing beside them. A read-only consumer materializes nothing, so it never pays the 96 bytes an owned node costs. The file-level sweep is untouched and still skips by size.

Buffering is also where the numbering latch lives: the object is walked as it lands, which proves its declared size against its property counts and settles the numbering. A kind byte that only decodes as legacy re-walks bytes already in memory and latches the handle for good.

One decode path (#209). ObjectStream::read and BinStream::into_bin are the owned way out, and Bin::from_reader becomes mount plus into_bin, so the eager tree and the streaming surface are the same parser. The ReadProperty impls for the self-sized kinds read through the same core; gathering one value from an io::Read grows a buffer until the walk can cross it and winds the reader back, so the extent rules exist once rather than twice.

The lookup cache is opt-in and the provider is the policy: NoCache is the default and a real provider rather than a disabled state, LruObjectCache is shipped, and Arc<BinObject<M>> is the currency, so a hit is a clone, eviction invalidates nothing a caller holds, and a handle with a cache installed is still Send.

Batch lookup (#214). objects_batch takes the whole request up front so the reads can be scheduled. The key is the file offset, never the hash: a cold handle resolves the request during the one forward scan that builds its table of contents and stops at the last hit, and a warm handle visits the requested rows in offset order. Yield order is file order either way, and missing() reports the hashes the file does not hold.

Behaviour changes

  • A string that is not UTF-8 raises Error::Utf8Error rather than Error::ReaderError - the byte-level codec's answer replacing the reader's.
  • Bin::from_reader buffers internally, so it no longer leaves the reader at a defined position. No caller in the workspace reads from it afterwards.

Where the design was corrected

Section 16 of the design records this in full. In short: the numbering is cursor state rather than a per-call parameter, which is what lets the walks be methods on the cursor; fixed_width belongs on Kind, beside the wire facts it already carries; numbering() replaces the drafted is_legacy(); and the walk the views need before handing out bytes is not one the owned path has to pay for, so Bin::from_reader crosses each object's bytes once rather than twice.

Two divergences between the renderers are kept deliberately and documented on both methods: ObjectView::property returns the first name-hash match where the owned map keeps the last, and property_count is the wire count where the owned map is deduplicated. Both differ only for an object that declares one name hash twice, which no shipped bin does, and closing either would cost every lookup the early exit it has.

Verification

cargo fmt --check, clippy --all-targets, doc --no-deps and the full workspace suite are clean at each commit, so the history bisects.

The corpus sweep over an install covers 392 archives and 48,912 PROP chunks: every object views cleanly, and all 2,472,864 properties have their declared shape and decoded value checked against the eager parse, with a batch of sampled hashes per chunk opening the same objects the per-hash lookups do. The 238 PTCH chunks still re-write byte for byte.

LTK_LOL_GAME_DIR="C:/Riot Games/League of Legends/Game" \
    cargo test -p ltk_meta --release --test corpus -- --ignored --nocapture

Closes #208, closes #209, closes #214. #192 stays open for the PTCH stream (#210) and the delta write-back (#211).

stream::layout owns the byte-level knowledge every reader shares: where a
value starts, how far it runs, what its header declares, how to decode a
leaf. A cursor carries the numbering its bytes were written under, so
nothing threads a legacy flag through every call and a slice can never be
read under the wrong numbering. Two renderers sit on it and cannot drift:
borrowed views, and an owned decode that goes through the value model's
checked constructors. The module is crate-internal - only Numbering is
public, since the latch reports it.

Descending buffers an object's declared byte range once and views it in
place. ObjectView iterates properties and finds one by name hash,
PropertyView answers the shape and element count from header bytes alone,
and ValueView descends to any depth, so Elements[3].Position reads one
container index and two properties and nothing beside them. Buffering is
where the latch lives: the object is walked as it lands, which proves its
declared size against its property counts and settles the numbering.

ObjectStream::read and BinStream::into_bin are the owned way out, and
Bin::from_reader becomes mount plus into_bin, so the eager tree and the
stream are one parser. The ReadProperty impls for the self-sized kinds
read through the same core; reading one value from an io::Read gathers
bytes until the walk can cross them and winds the reader back, so the
extent rules exist once. A string that is not UTF-8 now raises
Error::Utf8Error rather than Error::ReaderError.

The lookup cache is opt-in and the provider is the policy: NoCache is the
default and a real provider, LruObjectCache is shipped, and Arc is the
currency, so eviction invalidates nothing a caller holds and a handle
with a cache installed is still Send.

Closes #208, closes #209.
objects_batch takes the whole request up front so the reads can be
scheduled. The schedule key is the file offset, never the hash: hash
order says nothing about where objects sit in a file. A cold handle
resolves the request during the one forward scan that builds its table
of contents and stops at the last hit, so a request for objects near the
front of a large bin never reads the rest; a warm handle visits the
requested rows in offset order, so every seek is forward.

Yield order is file order either way, and a caller that needs its own
order collects and reorders. A miss has no file position, so misses are
not yielded - missing() reports them once the cursor is exhausted.

Closes #214.
Section 16 keeps the corrections the way the PTCH design keeps its own:
where the proposal needed a decision it had not made. The numbering is
cursor state so the walks can be methods, the module is layout rather
than the wire core and is crate-internal, fixed_width belongs to Kind,
numbering() replaces the drafted is_legacy(), and the walk the views
need before handing out bytes is not one the owned path has to pay for.
It also records the two renderer divergences kept on purpose, and what
the corpus attests: 392 archives, 48,912 PROP chunks, every one of
2,472,864 properties viewed, shaped and decoded against the eager parse.

The status line said nothing was implemented, which stopped being true
two pull requests ago. Tickets 03, 04 and 07 come up to the shipped
surface, and their Blocked by #207 lines go now that #207 has landed.
@Crauzer
Crauzer merged commit 3225df4 into main Aug 30, 2026
1 check passed
@Crauzer
Crauzer deleted the feat/bin-stream-views branch August 30, 2026 14:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bin streaming: batch object lookup Bin streaming: owned decode, the single decode path, and the lookup cache Bin streaming: zero-copy object views

1 participant