[ENG-44755] Emit discovery_path label on lakeView_table_discovery_failure

[vishalk9]

Summary

metadataExtractorDiscoveryFailure fires per-org on lakeView_table_discovery_failure_total with no table-level dimension — so on-call can only silence the whole org's alert, which then blinds us to any NEW table failure for that org during the silence window. Hit live on ENG-44423 / ENG-44424 (Cardlytics): a customer-side S3 bucket-policy explicit-Deny hit 3 of hundreds of paths under Domain=CustomerOfferEvent/, and the only lever was an org-wide 2-week Alertmanager silence.

Fix

• Add a table_base_path tag to lakeView_table_discovery_failure, wired at TableDiscoveryService — the per-path .exceptionally is the only failure call site with the failing path in scope.
• Run-level failure sites (Main, TableDiscoveryAndUploadJob) route through the reason-only overload, which now supplies an "unknown" sentinel so the counter's tag set stays consistent across all call sites (needed for a clean Prometheus label family).
• Once emitted, an Alertmanager silence can target {orgId, table_base_path} instead of the whole org.

tableId is deliberately not used: discovery is the step that resolves a path into a tableId, so at discovery-failure time no tableId exists yet — the failing base path is the available table-level dimension (matches the ticket's "and/or table base path").

Follow-ups (not in this PR)

• infra: surface {{ $labels.table_base_path }} on the metadataExtractorDiscoveryFailure alert + update the runbook silence step to {orgId, table_base_path}.
• Cardinality: table_base_path is emitted only on failure (0 series in steady state) and is naturally bounded by the customer's Deny scope. A defensive per-round cap (collapse to __many__ above K distinct failing paths) is a candidate hardening if a wholesale-denial pathological case shows up — open for reviewer input.

Test plan

• LakeViewExtractorMetricsTest: reason-only cases now assert the "unknown" sentinel tag; new testIncrementTableDiscoveryFailureCounterWithTableBasePath asserts the real path tag.
• TableDiscoveryServiceTest.testTableDiscoveryEncountersRateLimitException updated to the path-aware signature.
• ./gradlew :lakeview:spotlessApply — clean, no rewrites
• ./gradlew :lakeview:test --tests LakeViewExtractorMetricsTest --tests TableDiscoveryServiceTest — all passed locally

References

• ClickUp: ENG-44755
• Source incidents: ENG-44423 / ENG-44424

🤖 Generated with Claude Code

[nimahajan]

Task linked: ENG-44423 [FIRING:1] production - Failure encountered during metadata extractor discovery run
Task linked: ENG-44424 [FIRING:1] production - Failure encountered during metadata extractor discovery run
Task linked: ENG-44755 Emit tableId / table base path on metadataExtractorDiscoveryFailure so alerts can be silenced per-table

[kroushan-nit]

nit: path here is whatever directory the failed listAllFilesInDir was on, which is the table root only sometimes. Discovery stops recursing once it sees .hoodie/, so if the Deny lands on a prefix above the tables (e.g. s3://.../Domain=CustomerOfferEvent/) that list fails before we ever descend, and the emitted value is that domain prefix — a container of many tables, not a table base path. The table_base_path label name over-promises for exactly the broad-Deny case this feature targets.

Since the metric is new (no back-compat cost), maybe rename the key to something like discovery_path / scan_path, or add a one-line note that the value is the scan path at failure depth and can be a prefix above the table. Non-blocking.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud