[ENG-41614] Discover and upload Iceberg tables alongside Hudi - #191
Merged
Conversation
Sub-PR 2 of 2 splitting #189. Layers the Iceberg detector + uploader on top of the TableFormatDetector SPI introduced in #190. Discovery - IcebergTableFormatDetector: matches a directory if it contains a metadata/ sub-dir AND that sub-dir holds a *.metadata.json pointer (so Spark checkpoint dirs / doc folders named metadata/ don't false- positive). Registered alongside HudiTableFormatDetector in the TableDiscoveryService detector map. - Database gains an optional tableHints map keyed on the per-tableId segment of basePaths; entries carry a metadataLocationHint that the control plane supplies when it knows the current metadata.json URI (e.g. AWS Glue's metadata_location). Hint-less tables fall back to plain discovery. - Table model gains tableFormat (defaults to HUDI for backward compat) and metadataLocationHint. Upload - IcebergMetadataUploaderService sibling of TableMetadataUploaderService. Hudi's active/archived timeline distinction, hoodie.properties bootstrap, and LSM manifest plumbing don't apply to Iceberg, so a separate service is clearer than a branching megaclass. Reuses OnehouseApiClient, PresignedUrlFileUploader, AsyncStorageClient, StorageUtils. - Pointer-file selection (in order of preference): metadataLocationHint -> metadata/version-hint.text (Hadoop catalog) -> numeric-aware filename comparison. - TableDiscoveryAndUploadJob.dispatchUpload partitions discovered tables by format and runs the two uploaders concurrently; both runOnce and processTables route through it. Hudi-only configs see one extra empty future per cycle. Wire - New TableFormat alias on Database / Table flows through to InitializeSingleTableMetricsCheckpointRequest.tableFormat (nullable; server treats absent as HUDI). - OBJECT_STORAGE_URI_PATTERN accepts s3a:// alongside s3:// (Glue metadata_location scheme). Deploy ordering This change spans three repos and MUST roll out in order: 1. idls (TableFormat proto enum + TableMetricsCheckpoint field) 2. gateway-controller (IcebergCommitMetadataParser) 3. LakeView (this PR) Out-of-order LakeView ship: server's proto-JSON parser drops the unknown enum value and routes Iceberg uploads through the Hudi back-compat path.
vishalk9
approved these changes
May 22, 2026
| * these tests) to the Iceberg uploader. Default the mock to a successful no-op so the dispatch | ||
| * combine doesn't NPE on the (empty Iceberg set) branch. | ||
| */ | ||
| lenient() |
The @beforeeach used lenient() to stub the iceberg uploader because tests that exercise only the discovery-failure branch never reach dispatchUpload, and strict-stubbing rejects unused stubs. Moving the stub into the four tests that actually invoke dispatchUpload (continuous-mode success path, runOnce, runOnceWithRetry, plus the existing dispatch-routing test which already had its own stubs) lets us remove the suppression.
|
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Sub-PR 2 of 2 splitting #189. Layers the Iceberg detector + uploader on top of the
TableFormatDetectorSPI introduced in #190 (already merged). The control plane parses the snapshot summary from each uploadedmetadata.json— cumulativetotal-records,total-files-size,total-data-files, per-snapshot deltas, and thesnapshot-log[]history — gateway-controller side is a separate PR.See
docs/iceberg-support.mdfor the full design.What changed
Discovery
IcebergTableFormatDetector: matches a directory if it contains ametadata/sub-dir AND that sub-dir holds a*.metadata.jsonpointer (so Spark checkpoint dirs / doc folders namedmetadata/don't false-positive). Registered alongsideHudiTableFormatDetectorin theTableDiscoveryServicedetector map.Databasegains an optionaltableHintsmap keyed on the#tableIdsegment ofbasePaths; entries carry ametadataLocationHintthat the control plane supplies when it knows the currentmetadata.jsonURI (e.g. AWS Gluemetadata_location). Hint-less tables fall back to plain discovery.Tablemodel gainstableFormat(defaults to HUDI for backward compat) andmetadataLocationHint.Upload
IcebergMetadataUploaderServiceas sibling ofTableMetadataUploaderService. Hudi's active/archived timeline distinction,hoodie.propertiesbootstrap, and LSM manifest plumbing don't apply, so a separate service is clearer than a branching megaclass. ReusesOnehouseApiClient,PresignedUrlFileUploader,AsyncStorageClient,StorageUtils.metadataLocationHintfrom parser YAML (catalog-driven, e.g. Gluemetadata_location)metadata/version-hint.text(Hadoop catalog) — integer content names the currentv{N}.metadata.jsonunambiguouslyvprefix) is the version number; works for bothv{N}.metadata.json(Hadoop) and00000-<uuid>.metadata.json(Hive/Glue/Spark)TableDiscoveryAndUploadJob.dispatchUploadpartitions discovered tables by format and runs the two uploaders concurrently; bothrunOnceandprocessTablesroute through it.Wire
TableFormat(added in [ENG-41614] Introduce TableFormatDetector SPI (Hudi-only) #190) now flows through toInitializeSingleTableMetricsCheckpointRequest.tableFormat(nullable; server treats absent as HUDI).TableType(COW/MOR) stays@NonNullto preserve the existing wire contract; Iceberg init passesCOWas a meaningless placeholder since the server discriminates ontableFormatfirst.OBJECT_STORAGE_URI_PATTERNacceptss3a://alongsides3://(Gluemetadata_locationscheme).Hudi safety
Database.tableFormatnull → HUDI fallbackTable.tableFormat@Builder.Default = HUDIdispatchUploadnull-tableFormat → HUDI bucketTableMetadataUploaderService(Hudi path) untouchedOBJECT_STORAGE_URI_PATTERNwidening —s3://,gs://,abfss://, Azure HTTPS all still match (verified with 13-case regex sanity check)./gradlew build+./gradlew testgreen across all modulesDeployment ordering⚠️
This change spans three repos and must roll out in order:
TableFormatproto enum /TableMetricsCheckpointfieldIcebergCommitMetadataParserparses the uploadedmetadata.jsonand emits to OpenSearchIf LakeView ships before the server understands
tableFormat, the server-side protobuf JSON parser drops the unknown enum value and persistsTABLE_FORMAT_INVALID, routing Iceberg uploads through the Hudi back-compat fallback. Do not deploy this until the running control plane toleratestableFormat=ICEBERGcleanly. Seedocs/iceberg-support.md.Test plan
./gradlew :lakeview:testgreen (full suite)TestIcebergMetadataUploaderService(~538 lines) covers all three pointer-resolution paths + upload + checkpoint flowsTestIcebergTableFormatDetectorcovers match / no-match / false-positive (metadata/without*.metadata.json) / Hudi-layout-rejection / trailing-slash CommonPrefixTableDiscoveryServiceTestextended withtestDiscoverIcebergTableByDeclaredFormatTableDiscoveryAndUploadJobTestcoversdispatchUploadroutingStorageUtilsTestcoverss3a://forgetPathFromUrl+getBucketNameFromUri./gradlew buildgreen across all modules (lakeview, lakeview-glue, lakeview-sync-tool)Related