Bin streaming: zero-copy object views, one decode path, and batch lookup - #216
Merged
Conversation
stream::layout owns the byte-level knowledge every reader shares: where a value starts, how far it runs, what its header declares, how to decode a leaf. A cursor carries the numbering its bytes were written under, so nothing threads a legacy flag through every call and a slice can never be read under the wrong numbering. Two renderers sit on it and cannot drift: borrowed views, and an owned decode that goes through the value model's checked constructors. The module is crate-internal - only Numbering is public, since the latch reports it. Descending buffers an object's declared byte range once and views it in place. ObjectView iterates properties and finds one by name hash, PropertyView answers the shape and element count from header bytes alone, and ValueView descends to any depth, so Elements[3].Position reads one container index and two properties and nothing beside them. Buffering is where the latch lives: the object is walked as it lands, which proves its declared size against its property counts and settles the numbering. ObjectStream::read and BinStream::into_bin are the owned way out, and Bin::from_reader becomes mount plus into_bin, so the eager tree and the stream are one parser. The ReadProperty impls for the self-sized kinds read through the same core; reading one value from an io::Read gathers bytes until the walk can cross them and winds the reader back, so the extent rules exist once. A string that is not UTF-8 now raises Error::Utf8Error rather than Error::ReaderError. The lookup cache is opt-in and the provider is the policy: NoCache is the default and a real provider, LruObjectCache is shipped, and Arc is the currency, so eviction invalidates nothing a caller holds and a handle with a cache installed is still Send. Closes #208, closes #209.
objects_batch takes the whole request up front so the reads can be scheduled. The schedule key is the file offset, never the hash: hash order says nothing about where objects sit in a file. A cold handle resolves the request during the one forward scan that builds its table of contents and stops at the last hit, so a request for objects near the front of a large bin never reads the rest; a warm handle visits the requested rows in offset order, so every seek is forward. Yield order is file order either way, and a caller that needs its own order collects and reorders. A miss has no file position, so misses are not yielded - missing() reports them once the cursor is exhausted. Closes #214.
Section 16 keeps the corrections the way the PTCH design keeps its own: where the proposal needed a decision it had not made. The numbering is cursor state so the walks can be methods, the module is layout rather than the wire core and is crate-internal, fixed_width belongs to Kind, numbering() replaces the drafted is_legacy(), and the walk the views need before handing out bytes is not one the owned path has to pay for. It also records the two renderer divergences kept on purpose, and what the corpus attests: 392 archives, 48,912 PROP chunks, every one of 2,472,864 properties viewed, shaped and decoded against the eager parse. The status line said nothing was implemented, which stopped being true two pull requests ago. Tickets 03, 04 and 07 come up to the shipped surface, and their Blocked by #207 lines go now that #207 has landed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements the
PROPhalf ofdocs/design/bin-streaming.md(sections 4.2-4.5, 8, 9 and 14) on top of the foundation #213 landed.What lands
One internal layout core.
stream::layoutowns the byte-level knowledge every reader shares: where a value starts, how far it runs, what its header declares, how to decode a leaf. A cursor carries the numbering its bytes were written under, so nothing threads alegacyflag through every call and a slice can never be read under the wrong numbering. Two renderers sit on it and cannot drift - the borrowed views, and an owned decode that builds through the value model's checked constructors, soInvalidNesting,InvalidKeyTypeandMismatchedContainerTypesstay the value model's business. The module is crate-internal; onlyNumberingis public, since the latch reports it.Zero-copy views (#208). Descending buffers an object's declared byte range once and views it in place.
ObjectViewiterates properties and finds one by name hash,PropertyViewanswers the shape and the element count from the header bytes alone, andValueViewmirrorsPropertyValueEnumvariant for variant, soElements[3].Positionreads one container index and two properties and nothing beside them. A read-only consumer materializes nothing, so it never pays the 96 bytes an owned node costs. The file-level sweep is untouched and still skips by size.Buffering is also where the numbering latch lives: the object is walked as it lands, which proves its declared size against its property counts and settles the numbering. A kind byte that only decodes as legacy re-walks bytes already in memory and latches the handle for good.
One decode path (#209).
ObjectStream::readandBinStream::into_binare the owned way out, andBin::from_readerbecomes mount plusinto_bin, so the eager tree and the streaming surface are the same parser. TheReadPropertyimpls for the self-sized kinds read through the same core; gathering one value from anio::Readgrows a buffer until the walk can cross it and winds the reader back, so the extent rules exist once rather than twice.The lookup cache is opt-in and the provider is the policy:
NoCacheis the default and a real provider rather than a disabled state,LruObjectCacheis shipped, andArc<BinObject<M>>is the currency, so a hit is a clone, eviction invalidates nothing a caller holds, and a handle with a cache installed is stillSend.Batch lookup (#214).
objects_batchtakes the whole request up front so the reads can be scheduled. The key is the file offset, never the hash: a cold handle resolves the request during the one forward scan that builds its table of contents and stops at the last hit, and a warm handle visits the requested rows in offset order. Yield order is file order either way, andmissing()reports the hashes the file does not hold.Behaviour changes
Error::Utf8Errorrather thanError::ReaderError- the byte-level codec's answer replacing the reader's.Bin::from_readerbuffers internally, so it no longer leaves the reader at a defined position. No caller in the workspace reads from it afterwards.Where the design was corrected
Section 16 of the design records this in full. In short: the numbering is cursor state rather than a per-call parameter, which is what lets the walks be methods on the cursor;
fixed_widthbelongs onKind, beside the wire facts it already carries;numbering()replaces the draftedis_legacy(); and the walk the views need before handing out bytes is not one the owned path has to pay for, soBin::from_readercrosses each object's bytes once rather than twice.Two divergences between the renderers are kept deliberately and documented on both methods:
ObjectView::propertyreturns the first name-hash match where the owned map keeps the last, andproperty_countis the wire count where the owned map is deduplicated. Both differ only for an object that declares one name hash twice, which no shipped bin does, and closing either would cost every lookup the early exit it has.Verification
cargo fmt --check,clippy --all-targets,doc --no-depsand the full workspace suite are clean at each commit, so the history bisects.The corpus sweep over an install covers 392 archives and 48,912
PROPchunks: every object views cleanly, and all 2,472,864 properties have their declared shape and decoded value checked against the eager parse, with a batch of sampled hashes per chunk opening the same objects the per-hash lookups do. The 238PTCHchunks still re-write byte for byte.Closes #208, closes #209, closes #214. #192 stays open for the
PTCHstream (#210) and the delta write-back (#211).