tools/lance-resolver: add standalone Lance dataset resolver for warmup - #7399
Open
hiltonhe wants to merge 40 commits into
Open
tools/lance-resolver: add standalone Lance dataset resolver for warmup#7399hiltonhe wants to merge 40 commits into
hiltonhe wants to merge 40 commits into
Conversation
Contributor
|
The warmup PR has already been merged. Please remove the changes that are no longer needed and resolve the conflicts. Also, the format supported by warmup has been adjusted. |
hiltonhe
force-pushed
the
lance-resolver
branch
from
August 25, 2026 04:57
893f2de to
a3706ff
Compare
Contributor
Author
|
@jiefenghuang Thanks for the review. Done:
Could you PTAL again? Thanks. |
jiefenghuang
left a comment
jiefenghuang
left a comment
Contributor
There was a problem hiding this comment.
Verified the tool against synthetic Lance datasets. The byte-range plumbing in #7398 looks good, but the resolver itself has a few blocking correctness issues — most importantly it fails on real Lance manifests, which the current tests don't catch because they write a layout Lance never produces. Details inline.
Contributor
|
The proto files can reference a specific version of the Lance project directly, so there is no need to introduce an additional copy. The upstream go_package configuration currently has some issues. I think we can add a shell script that downloads the proto files from an explicitly pinned Lance version and then uses a specific version of protoc to generate the Go code. |
Extract Lance format parsing logic into a standalone CLI tool under tools/lance-resolver/. This is the external resolver approach: JuiceFS stays format-agnostic, and this tool handles all Lance-specific format parsing. Usage: lance-resolver /mnt/jfs/data/ds.lance | juicefs warmup -f - lance-resolver --columns id,name /mnt/jfs/data/ds.lance | juicefs warmup -f - Features: - Resolve Lance dataset to file paths (data files, deletions, overlays) - Version selection (latest or specific version via --version) - Column-level byte range resolution (V2 format, --columns) - Manifest-only mode (--manifest-only) - Index file inclusion (--include-indices) - Output format compatible with warmup --file (PR 1) Dependencies: warmup-range-support branch (PR 1) for --file range support Tests: 14 test functions covering manifest parsing, byte range merging, dataset resolution, error handling, and file system operations.
The current Lance V2 writes manifest files with a two-layer structure: [transaction_len:4][Transaction][manifest_len:4][Manifest][footer:16] Update readAndParseLanceManifest to detect and parse this format, falling back to the V1 format for backward compatibility. Also fix footer magic check offset (now at footer[36:40] instead of footer[24:28]) and mark column-level byte range resolution as TODO since the V2 CMO table is now protobuf-encoded instead of raw u64 entries.
The warmup -f file format uses SPACE to separate paths from byte ranges
('path start-end;start-end;...'), not TAB. Update comments and help
text to reflect the correct format.
Also fix help examples: remove '| juicefs warmup -f -' since warmup
does not support reading from stdin (-f -). Use temp file pattern
instead: lance-resolver > /tmp/list.txt; juicefs warmup -f /tmp/list.txt
- parse manifest via trailing 16-byte footer and manifest_pos instead of the V2 heuristic; validate with 64-bit arithmetic to avoid overflow - fix V2 latest manifest version selection in findLatestManifestByListing - fall back to full-file warmup when --columns is requested (column-level byte ranges are not implemented yet) - use the stdlib flag package and error on unexpected arguments - add Makefile test.tools target and extend unit tests for real manifest layout, non-zero manifest_pos, and V2 version selection
- add tools/lance-resolver/gen-protos.sh to download file.proto and table.proto from Lance v11.0.0-rc.1 and regenerate the Go code with protoc-gen-go v1.36.11 - remove checked-in .proto copies and the unused file2 protobuf set - regenerate table.pb.go from the pinned version and add Makefile gen-lance-protos target
- generate file2.pb.go from pinned Lance v11.0.0-rc.1 - add --include-data-pages flag and implement column byte-range resolution for V2 data files using the file footer and column metadata offset table - output 'path [start-end;...]' targets so juicefs warmup can warm only blocks referenced by the requested columns - fall back to full-file warmup for V1 files or files without the requested columns - add unit tests for field-to-column mapping, byte-range merge, column buffer ranges, and synthetic V2 footer parsing
cmPos and cmLen are read directly from the CMO table entry and their sum can overflow uint64, defeating the bounds check and leading to make([]byte, int(cmLen)) panicking on corrupt input. Validate with subtraction instead, and skip buffer ranges that would overflow.
The Makefile targets are unnecessary: gen-protos.sh can be run directly and the tool is already covered by go test ./tools/.... Document usage in a README under tools/lance-resolver instead.
A REPEATED (list) or PARENT (struct) column spans multiple physical columns: a list's element data lives in nested child columns that are not reachable via DataFile.ColumnIndices. Warming such a column by name only reached the outer column and silently missed the data. Detect them via the schema field type and fall back to full-file warmup with a warning.
Clarify that --columns only supports leaf (primitive) columns and that list/struct columns fall back to full-file warmup with a warning.
Record the planned direction: delegate nested (list/struct) column warmup to the official Rust lance/lance-file crates via a sidecar, rather than reimplementing the encoding walker in pure Go.
columnWarmupPath builds output paths with path.Join (forward slashes); the test expected filepath.Join (backslashes on Windows). Match the code so the test is green on all platforms.
hiltonhe
force-pushed
the
lance-resolver
branch
from
August 27, 2026 17:48
5ea9a6b to
355eda9
Compare
Upstream Lance writers never populate the Field type enum on the wire (it always decodes to PARENT), so the Field_LEAF check made --columns fall back to full-file warmup on every real dataset while synthetic tests passed because their builders set the enum explicitly. Derive leaf-ness from the flat field list instead: a field is a leaf iff no other field references it via parent_id. parent_id = -1 marks top-level fields (upstream stores it explicitly); a non-negative parent_id is a real reference and is passed through verbatim by upstream readers. Test builders now mirror the real wire shape: no type enum and explicit parent_id.
gen-fixtures.py writes the golden dataset with pylance 10.0.0 (the pin closest to the v11.0.0-rc.1 sources used by gen-protos.sh; that version is not on crates.io/PyPI and the footer/CMO layout is shared across V2 file versions). dump_fixture.py is an independent ground-truth extractor: it parses the manifest footer, V2 file footer, CMO table and ColumnMetadata straight from the fixture bytes and writes expected.json. Regenerations are stable only after normalization: file names are random, the manifest carries a creation timestamp, and the writer permutes which content lands in which file (RAYON/TOKIO single-threading does not fix it). expected.json therefore masks volatile fields and orders files by content signature; the golden tests compare multisets accordingly.
The golden tests resolve the committed fixture end to end: full-dataset resolution, per-column byte ranges for every leaf column in both metadata-only and data-pages modes, and struct/unknown-column fallback to full-file warmup. Because the writer permutes file contents across regenerations, all assertions compare multisets of per-file values. Verified non-self-confirming by mutation: corrupting any of footer length, magic offset, CMO table offset, CMO entry size or num_columns offset makes the golden tests fail. The synthetic-byte tests in resolver_test.go are now explicitly scoped to negative/boundary coverage (note at the top of the file); resolver.go changes are gofmt comment reflow only.
Data-file footers now carry a (major, minor) whitelist — (2,0)/(2,1) legacy numbering plus the (0,3) pair dispatched by the upstream v11 reader — so an unknown future grammar fails loudly (and falls back to full-file warmup) instead of being silently misread. Manifest footer versions are only warned about when unknown: just (0,2) is observed in the wild so far and the semantics are not documented enough to reject. The overflow-guard test now writes a valid version pair so it still reaches the guard it was written for.
check-fixture-drift.sh regenerates the fixture into a temp dir with the pinned pylance and verifies the normalized ground truth (expected.json) is unchanged via git diff --exit-code. Raw fixture bytes cannot be diffed: file names and the manifest timestamp are random per run and the writer permutes content across files, but the dumper's normalization was verified to be byte-stable across regenerations. The workflow runs the drift check plus the golden tests on every change under tools/lance-resolver/. README gains a testing/fixtures section documenting the regeneration commands.
gen-protos.sh pinned protoc-gen-go but used whatever protoc happened to be on PATH, so regeneration depended on the host environment. The script now downloads a pinned protoc release binary for the host platform (unzip with python3 fallback) and passes its bundled include dir, making generation reproducible without a host protoc install. Regenerating with protoc 36.0 changes only the version comment in the generated files (previously v3.21.12 from PATH); descriptor bytes are identical.
Any column projection first reads the V2 file footer (to locate the CMO table) and the projected columns' 16-byte CMO entries (to locate their ColumnMetadata); warming only the metadata protobuf left those reads as guaranteed cache misses. Both are now part of every column-level warmup line and merge with adjacent ranges, so typical output grows by one merged range at the file tail. dump_fixture.py computes the same ranges so expected.json stays the independent ground truth.
Fragments may store their row-id / last-updated-at / created-at version sequences in dedicated files (ExternalFile with path relative to the dataset root and an explicit byte range) once the sequence exceeds the ~200KB inline threshold; the resolver skipped them entirely, leaving guaranteed cache misses on stable-row-id datasets after warmup. They are now always emitted, with the [offset, offset+size) range when the payload is a sub-range. Real fixtures cannot cover this yet (the inline threshold is far beyond fixture size even at 100k rows), so the resolution is covered by a synthetic manifest test and the dumper/golden scaffolding (external_files in expected.json, EXTFILE tokens) is ready for the day a fixture can produce them.
ColumnMetadata and its pages may reference their encoding via Encoding.indirect (deferred encoding, buffer_location/buffer_length). The upstream v11 reader rejects those outright, so there is no reference implementation that defines how buffer_location resolves to an actual buffer; emitting guessed ranges would risk incomplete warmup while claiming coverage. The resolver now detects indirect encodings on the column or any page and falls back to full-file warmup with a warning, mirroring the list/struct fallback. Direct and absent encodings are unaffected.
Data files, overlays and column warmup all joined paths against the dataset's data/ dir, ignoring base_paths/base_id — shallow clones, imported files and multi-base datasets resolved to nonexistent paths (and --columns then failed open and fell back to the same wrong path). Path resolution now mirrors upstream data_file_dir_for_base: no base_id means the dataset's own data/ dir; a set base_id looks up manifest.base_paths (distinct from unset since base_id is optional), where dataset-root bases keep files under data/ and direct-file bases use the base path as-is. A missing or empty base entry warns and falls back to the dataset root instead of failing the whole run. Deletion files keep resolving against the dataset root: upstream deletion_file_path ignores their base_id. pylance cannot produce such manifests locally, so coverage is a synthetic test of the resolution rules; the golden fixture (no base paths) is unaffected.
Nested requests used to fall back to full-file warmup on the assumption that spanning multiple physical columns requires Lance's full encoding model. Empirically that is not the case for warmup: list/struct parents have no physical column at all (verified on real datasets — only the terminal leaf fields carry column indices, and list offsets live inside the item column's own buffers), so warming a subtree is just the union of the descendant columns' ranges. A requested column now expands transitively via parent_id to its whole subtree, and every subtree field with a physical column in the file contributes its footer/CMO/metadata/buffer ranges. Fallback remains for requests that resolve to no physical columns and for indirect (deferred) encodings. Cycle-safe expansion guards against corrupt manifests. The golden fixture gains a list column so nested warmup is verified against real bytes for both struct and list; dump_fixture.py exports the matching per-name warmup map (subtree-union ranges).
Adversarial verification (zero every byte outside the emitted ranges, then read with the official pylance reader) exposed two real gaps that both the Go resolver and the dumper shared: 1. The official reader opens a file with ONE optimistic tail read from the first global buffer (the file schema) to EOF and parses ALL column metadata from it regardless of the projection (upstream v11 reader.rs). Column warmup now covers [gbo[0].pos, EOF) plus any global buffers positioned before it, via a new globalBufferTail helper reading the GBO table. 2. Readers coalesce adjacent buffer reads into spans that include the alignment padding between buffers; ranges separated by <= 4KiB are now filled (free on block-granular caches like JuiceFS's). With both fixes the verification passes for every projection: leaves, list, struct, 3-level nesting and multi-column combinations. verify_warmup_ranges.py is committed so the adversarial check is repeatable; golden expectations regenerated (dumper computes the same tail/gap rules), regen determinism re-verified.
The flag table still said --columns accepts leaf columns only, while nested struct/list support landed in e7a0704 and the section right below the table describes it. Align the table with the actual behavior.
fillByteRangeGaps filled gaps up to 4KiB, which chained across unrequested columns on compact files: --columns id,name --include-data-pages covered 100% of the fixture file, degrading column-level warmup to whole-file for a two-column projection. Readers only ever coalesce across alignment padding, and Lance aligns buffers to 64-byte boundaries (padding <= 63 bytes; the official reader was observed to need at most 62), so the fill threshold drops to 64. Two of five columns now warm 50% of the fixture file instead of 100%, and the adversarial verification still passes for every projection (leaves, nested struct/list, multi-column, 3-level nesting). README documents the merging behavior and the dense-projection caveat.
The GBO bounds check used gboStart+tableLen > limit, which a corrupt footer storing gbo_start near MaxUint64 wraps past; and the extras loop formed pos+size before validating size, so out-of-range entries could slip through as garbage warmup ranges. Neither could panic (ReadAt rejects the huge offsets), but the style diverged from the subtraction guards used for CMO entries. All bounds checks now subtract. Writing the boundary tests also exposed an inverted skip condition: extras skipped buffers entirely BEFORE the tail (the ones that need a separate range) and redundantly included buffers already inside the reader's [tailStart, EOF) tail read. Skip iff pos >= tailStart now; dump_fixture.py mirrors the same rule. No observable change on the fixture (single global buffer), golden expectations unchanged. Covers: happy path with extras, zero buffers, gbo_start = MaxUint64 (wrap), table past file end, schema position beyond file, corrupt pos/size pairs, zero-sized extras, file smaller than the footer.
The golden tests covered file-kind counts and column-range multisets, but the layout facts exported by dump_fixture.py were not load-bearing: manifest_pos, snapshot version, manifest footer version, every data-file footer field (column_metadata_start, cmo/gbo table starts, buffer and column counts, footer version) and each column's CMO entry. TestGolden_Structure decodes the fixture bytes with a raw third implementation (no production parsing) and asserts all of them, matching data files to expected entries by footer bijection. Verified by mutation: corrupting any of cmo_table_start, cm_pos, manifest_pos, manifest footer minor or the file footer version pair in expected.json now fails the suite.
parent_id = 0 is ambiguous on the wire (proto3 cannot distinguish unset from an explicit 0), and the earlier fix-plan note flagged writers that omit parent_id on top-level fields. The reference tree builder (lance-file datatypes, TryFrom<&Fields> for Schema) resolves this as a REAL reference to field 0 — only -1 marks a root, and a missing parent is a schema error — so such omitted-parent manifests are misread by the official reader itself and cannot be valid ecosystem files. fieldChildren keeps mirroring that behavior deliberately: treating 0 as root would diverge from upstream and break legitimate schemas whose field 0 is a struct parent. The comment now cites the upstream source, unknown parents stay tolerated (no edge rather than a hard failure), and a documentation test pins all three behaviors.
The workflow checked fixture drift and ran the golden tests, but range SUFFICIENCY — the property only the adversarial check can prove — was not exercised: zero every byte outside the emitted ranges and confirm the official reader still returns correct data. The workflow already installs Go, Python and the pinned pylance, so it now runs verify_warmup_ranges.py for a leaf column, a list column, a struct column and the full multi-column projection. The script also builds its resolver binary with a platform-neutral name for Linux runners.
A corrupt global-buffer-offset table only made globalBufferTail return false, so column warmup silently continued with partial ranges minus the tail — inconsistent with corrupt CMO entries, which degrade to full-file warmup. globalBufferTail now returns an explicit three-state result: a file legitimately declaring zero global buffers stays absent (upstream rejects such files on open, but that is not a table corruption and ranges continue), while any validation failure is corrupt and makes columnWarmupPath warn and fall back to the full file, matching the CMO semantics. Boundary tests updated to the three states; new test pins the end-to-end fallback on a corrupt gbo_start.
…cation verify_warmup_ranges.py leaked its temp dirs: the go build directory was never removed, the dataset copy was only cleaned on the success path, and a failing case (which is exactly when the copy matters for post- mortem-free CI) left both behind. Everything now lives under one TemporaryDirectory, cleaned on success and failure alike. Files were also matched against the resolver output by basename, which would apply the wrong ranges if a dataset ever carries same-named files in different directories (e.g. multi-base-path imports). Matching now uses the path relative to the dataset root, and ranges resolved outside the dataset are reported and skipped instead of silently ignored.
The fill threshold was only covered indirectly through the golden ranges. The new table test pins the exact boundary — a 64-byte gap merges (Lance buffer alignment means padding never exceeds 63 bytes), 65 does not — plus chaining across several padding gaps and the far-apart and degenerate cases, so a future threshold tweak fails a named test instead of silently changing multi-column warmup scope.
… as local paths Extending the adversarial verification beyond the committed fixture surfaced a real gap: pylance registers local bases as file:/// URIs, and the resolver emitted them collapsed (file:/C:/...), which os.Open and 'juicefs warmup -f' cannot consume — column warmup silently fell back to full files for imported datasets. Base paths now convert file:// URIs to plain local paths (Windows drive and POSIX forms); other schemes pass through since those bases are not warmable locally. verify_warmup_ranges.py gains a --multi-base mode that generates a temp dataset split across the dataset root and an imported base (proven writable locally via initial_bases/target_all_bases), zeroes everything outside the emitted ranges in place — over BOTH root and base files, with an unprojected padding column so the pass actually bites — and verifies the official reader still returns correct data. CI runs it. The remaining coverage boundary (metadata-only ranges, external row-id sequences, indirect encodings) is documented in the script and README with the upstream constraint that blocks each.
Each verify_warmup_ranges.py invocation built its own resolver binary, so the five CI cases compiled it five times. The script now honors LANCE_RESOLVER_BIN pointing at a prebuilt binary (erroring clearly if missing) and still builds into its temp dir when unset; the workflow builds once into RUNNER_TEMP and exports the variable.
… file
A footer declaring zero global buffers kept producing partial column
ranges on the argument that upstream rejects such files on open anyway
("schema expected"). That was the one remaining 'structurally invalid
file but partial ranges output' case, kept as a third state (absent)
next to ok/corrupt.
Since a V2 file without a schema buffer is never a normal data path,
the distinction bought nothing: globalBufferTail collapses back to two
states (tail ranges, or corrupt -> warn + full-file fallback), and the
invariant 'any structural anomaly degrades to the full file' holds with
no exceptions. Synthetic files that expect ranges now register a
minimal schema global buffer via addGBOTable, matching what every real
V2 file carries.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Depends on: #7398
Motivation
Lance datasets span multiple files (manifests, data, transactions,
deletions). Users need to warm up the exact set of files for a specific
version, not blindly traverse the directory.
Changes
Add
tools/lance-resolver/, a standalone CLI tool that:juicefs warmup -fcompatible format--version,--manifest-only,--columnsflagsUsage
lance-resolver /mnt/jfs/dataset.lance > /tmp/list.txt juicefs warmup -f /tmp/list.txtArchitecture
tools/addition)Test Coverage
resolver_test.go: 53 sub-cases covering manifest parsing, datasetresolution, error handling, and idempotency