Skip to content

Spark 3.1: fill ORC initial defaults on the row reader - #262

Closed
cbb330 wants to merge 1 commit into
chbush/orc-initial-default-orc-fillfrom
chbush/orc-initial-default-spark31
Closed

Spark 3.1: fill ORC initial defaults on the row reader#262
cbb330 wants to merge 1 commit into
chbush/orc-initial-default-orc-fillfrom
chbush/orc-initial-default-spark31

Conversation

@cbb330

@cbb330 cbb330 commented Jul 23, 2026

Copy link
Copy Markdown
Collaborator

Summary

Stack (bottom -> top)

  1. ORC: bind struct fields by Iceberg field id in generic and Spark 3.1 readers #256 - ORC id-binding backport (base 1.2.x)
  2. API: add column-default model support #260 - API column-default model (base chbush/orc-id-binding-backport)
  3. ORC: fill initial defaults for missing id-bound fields #261 - ORC initial-default fill (base chbush/orc-column-default-api)
  4. this PR - Spark 3.1 wiring (base chbush/orc-initial-default-orc-fill)

Depends on #261 (and transitively #260, #256). Review this PR's own delta against its base branch chbush/orc-initial-default-orc-fill.

Supported scope

  • Spark 3.1 ORC row reader

Vectorized ORC, other Spark/Flink readers, and predicate pushdown on omitted default columns remain deferred.

Testing Done

  • Spark 3.1 row tests: declared, undeclared, and nested defaults
  • Full Spark 3.1 test suite passes
  • Spark 3.1 checkstyle and formatting pass

Tests generated with unit-tests plugin

Made with Cursor

Wire the Spark 3.1 row ORC reader to materialize a field's initial-default
when it is absent from an id-bearing file, reusing BaseDataReader's
constant converter.

Co-authored-by: Cursor <cursoragent@cursor.com>
@cbb330

cbb330 commented Jul 23, 2026

Copy link
Copy Markdown
Collaborator Author

Closing: this was based on the wrong branch (1.2.x, which is essentially vanilla apache 1.2.x). The correct target is the OpenHouse dev branch openhouse-1.2.0, which already has the id-bound ORC readers and the NestedField column-default API (#250) — so the id-binding and API backports here were redundant.

Rebuilt as a clean 2-PR stack on openhouse-1.2.0:

@cbb330 cbb330 closed this Jul 23, 2026
@cbb330
cbb330 deleted the chbush/orc-initial-default-spark31 branch July 23, 2026 20:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant