Skip to content

ORC: fill initial defaults for missing id-bound fields - #261

Closed
cbb330 wants to merge 1 commit into
chbush/orc-column-default-apifrom
chbush/orc-initial-default-orc-fill
Closed

ORC: fill initial defaults for missing id-bound fields#261
cbb330 wants to merge 1 commit into
chbush/orc-column-default-apifrom
chbush/orc-initial-default-orc-fill

Conversation

@cbb330

@cbb330 cbb330 commented Jul 23, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Build on ORC: bind struct fields by Iceberg field id in generic and Spark 3.1 readers #256's id-bound Generic ORC reader to materialize a field's initial-default only when it is absent from an id-bearing ORC file.
  • Omit such fields from the read projection and fill them as a per-file constant through the reader's existing constant converter (not idToConstant).
  • Preserve physical values and explicit nulls; id-less / name-mapped files retain legacy null synthesis so a default is never name-matched onto a legacy/migrated file.

Stack (bottom -> top)

  1. ORC: bind struct fields by Iceberg field id in generic and Spark 3.1 readers #256 - ORC id-binding backport (base 1.2.x)
  2. API: add column-default model support #260 - API column-default model (base chbush/orc-id-binding-backport)
  3. this PR - ORC initial-default fill (base chbush/orc-column-default-api)
  4. Spark 3.1 wiring (base chbush/orc-initial-default-orc-fill)

Depends on #260. Review this PR's own delta against its base branch chbush/orc-column-default-api (2 commits above appear until the lower PRs merge).

Supported scope

  • Generic ORC row reader
  • Scalar defaults at any struct nesting level, including structs within list elements and map values

Vectorized ORC and predicate pushdown on omitted default columns remain deferred.

Testing Done

  • Projection tests: trusted-id omission, id-less null synthesis, required defaults, nested defaults
  • Generic ORC end-to-end tests: scalar types, required/nested/list/map defaults, present values/nulls, empty/null parent structs, id-less files
  • iceberg-orc and iceberg-data suites pass
  • ORC RevAPI, checkstyle, and formatting pass

Tests generated with unit-tests plugin

Made with Cursor

For the generic and id-bound ORC read path, omit an absent field that
declares an initial-default from the projection and materialize it as a
per-file constant through the reader. Present values and explicit nulls
are preserved; id-less/name-mapped files keep legacy null synthesis.

Co-authored-by: Cursor <cursoragent@cursor.com>
@cbb330

cbb330 commented Jul 23, 2026

Copy link
Copy Markdown
Collaborator Author

Closing: this was based on the wrong branch (1.2.x, which is essentially vanilla apache 1.2.x). The correct target is the OpenHouse dev branch openhouse-1.2.0, which already has the id-bound ORC readers and the NestedField column-default API (#250) — so the id-binding and API backports here were redundant.

Rebuilt as a clean 2-PR stack on openhouse-1.2.0:

@cbb330 cbb330 closed this Jul 23, 2026
@cbb330
cbb330 deleted the chbush/orc-initial-default-orc-fill branch July 23, 2026 20:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant