Add Lightdash converter by ota2000 · Pull Request #253 · apache/ossie

ota2000 · 2026-07-23T05:11:07Z

Resolves #227 (proposed there and welcomed by @khush-bhatia — thank you!)

What

Bidirectional converter between Ossie documents and Lightdash semantic definitions, which live as meta blocks in dbt schema.yml files.

Export (osi_to_lightdash): Ossie → Lightdash-flavoured schema.yml dict (columns with meta.dimension, column/model meta.metrics, meta.joins)
Import (lightdash_to_osi): existing Lightdash definitions → Ossie, as a migration path into Ossie
Lightdash presentation attributes with no OSI vocabulary (label, format, round, group_label, hidden, percentile, ...) round-trip through custom_extensions with a stable vendor_name: "lightdash" (JSON-string data, per the model contract). Structural keys (sql / type / description) are deliberately excluded from the extension so a stale copy can never win over the OSI expression on export.
Relationships map to Lightdash meta.joins (sql_on is built from / parsed back into column pairs).

Conventions

Follows the layout being standardized on dev@: package apache-ossie-lightdash, uv packaging (mirroring converters/dbt), osi_to_lightdash.py / lightdash_to_osi.py module naming, ossie-lightdash CLI entry point, ASF headers throughout.

Verification

21 tests, including a structural round-trip of the in-repo TPC-DS example (datasets/sources, fields + dimension-ness, single-dataset metric expressions, relationships all preserved; the two cross-dataset metrics are dropped on export with a CROSS_DATASET_METRIC_DROPPED issue, as documented — a Lightdash model metric cannot reference other tables)
CLI smoke: ossie-lightdash export examples/tpcds_semantic_model.yaml ... → import back reproduces 5 datasets / 3 single-dataset metrics / 4 relationships

Notes

README documents the full mapping table and known limitations (cross-dataset metrics, percentile handling, primary_key/ai_context not carried, standalone Lightdash YAML mode out of scope for now).
For documents also consumed by dbt-core's native OSI parsing, the README recommends importing without --database so one document stays valid across environments ([Feature] Allow database-less OSI dataset sources (schema.table) to bind by schema + alias, so one OSI document works across environments dbt-labs/dbt-core#15649).
ICLA: in progress on my side, as noted in Add a Lightdash converter #227.

Bidirectional converter between Ossie documents and Lightdash semantic definitions (dbt schema.yml meta blocks): datasets/fields/metrics/ relationships map to models/columns/meta.metrics/meta.joins, and Lightdash presentation attributes round-trip through custom_extensions with vendor_name "lightdash". Follows the apache-ossie-<vendor> uv packaging layout and ships tests off the TPC-DS example, including a structural round-trip.

…elity - Add converter-lightdash-ci.yml mirroring the other converters (tests were otherwise not exercised by CI) - Rewrite ${other_table.column} references into cross-dataset references on import, not just ${TABLE}.column - Keep inexpressible metric types (percentile) in the extension even when the metric carries custom SQL, on both column-level and model-level metrics - Drop the redundant per-converter .gitignore (covered by the root one) and unused helpers; document the not-yet-carried model-level meta keys in the README

Copilot

Pull request overview

Adds a new Python-based converter package (apache-ossie-lightdash) that enables bidirectional translation between Ossie OSI documents and Lightdash’s dbt schema.yml-embedded semantic definitions (meta.dimension, meta.metrics, meta.joins), including a CLI and round-trip tests.

Changes:

Introduces osi_to_lightdash and lightdash_to_osi converters plus shared SQL/aggregation rewriting utilities.
Adds a CLI (ossie-lightdash) for export/import workflows and a structured issue-reporting model (ConverterIssueType).
Adds a Lightdash-focused test suite including a structural TPC-DS round-trip.

Reviewed changes

Copilot reviewed 13 out of 14 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
converters/lightdash/uv.lock	Adds pinned dependency lockfile for the new converter package.
converters/lightdash/pyproject.toml	Defines the `apache-ossie-lightdash` package, dependencies, and CLI entry point.
converters/lightdash/.gitignore	Ignores local venv and Python cache artifacts for the new converter.
converters/lightdash/README.md	Documents mapping rules, usage, and limitations for Lightdash conversion.
converters/lightdash/src/ossie_lightdash/init.py	Exposes the converter public API from the package root.
converters/lightdash/src/ossie_lightdash/cli.py	Implements `ossie-lightdash` import/export commands and output formatting.
converters/lightdash/src/ossie_lightdash/converter_issues.py	Adds typed conversion-issue reporting used by both directions.
converters/lightdash/src/ossie_lightdash/expression_utils.py	Provides SQL rewriting and simple-aggregation parsing/building helpers.
converters/lightdash/src/ossie_lightdash/osi_to_lightdash.py	Implements OSI → Lightdash dbt `schema.yml` export (dimensions/metrics/joins).
converters/lightdash/src/ossie_lightdash/lightdash_to_osi.py	Implements Lightdash dbt `schema.yml` → OSI import (datasets/fields/metrics/relationships).
converters/lightdash/tests/init.py	Establishes the test package with ASF header.
converters/lightdash/tests/test_osi_to_lightdash.py	Unit tests for OSI → Lightdash mapping (dimensions/metrics/joins/issues).
converters/lightdash/tests/test_lightdash_to_osi.py	Unit tests for Lightdash → OSI mapping (sources, metrics, joins, issues).
converters/lightdash/tests/test_tpcds_roundtrip.py	End-to-end structural round-trip test using the in-repo TPC-DS example.

Comments suppressed due to low confidence (1)

converters/lightdash/src/ossie_lightdash/osi_to_lightdash.py:205

definition.update(extension_data) allows vendor extension keys (e.g. sql, description, or type) to override the OSI-derived structural fields. This can let a stale extension value win over the OSI expression/description, which the PR description explicitly aims to prevent.

            definition["type"] = extension_data.get("type", "number")
            definition["sql"] = osi_sql_to_lightdash(expression, target_dataset)
            target_column = None

        definition.update(extension_data)

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

ota2000 · 2026-07-23T05:22:05Z

+            expression = _pick_expression(field.expression, self._dialect)
+            if expression and expression != field.name:
+                dimension["sql"] = osi_sql_to_lightdash(expression, dataset.name)
+            dimension.update(_lightdash_extension_data(field, issues))


Good catch — fixed in 80c775d. The export overlay now drops structural keys (sql/label on dimensions, sql/description on metrics) so a hand-authored extension can never override the OSI-derived definition; type on metrics deliberately stays overridable as the documented channel for types OSI expressions can't express (percentile). Regression test added.

ota2000 · 2026-07-23T05:22:06Z

+            sql_on = " AND ".join(
+                f"${{{from_model_name}.{from_column}}} = ${{{to_model_name}.{to_column}}}"
+                for from_column, to_column in zip(
+                    relationship.from_columns, relationship.to_columns
+                )
+            )


Fixed in 80c775d — mismatched from_columns/to_columns now skip the relationship with a RELATIONSHIP_COLUMNS_MISMATCHED issue instead of silently truncating. I went with the issue-reporting pattern (rather than raising like the Snowflake converter) to stay consistent with this converter's ConverterResult contract, where malformed elements degrade gracefully with a report — happy to switch to a hard error if reviewers prefer the Snowflake behavior. Regression test added.

ota2000 · 2026-07-23T05:22:08Z

+- **Cross-dataset metrics are dropped on export** (with a
+  `CROSS_DATASET_METRIC_DROPPED` issue): a Lightdash model metric cannot
+  reference other tables.
+- **Percentile metrics** keep `type` / `percentile` in the `lightdash`
+  extension (Ossie expressions cannot express them faithfully) and re-export
+  as model-level metrics.


Agreed — documented in 80c775d as a known limitation: dataset names are not preserved when they differ from the source table name (references inside expressions and relationships are rewritten consistently, but name-stable round-trips are not guaranteed).

…ip columns - Drop structural keys (sql/label on dimensions, sql/description on metrics) from the extension overlay on export, so a hand-authored extension can never override the OSI-derived definition; metric type stays overridable as the documented channel for inexpressible types - Skip relationships whose from_columns/to_columns lengths differ, reporting RELATIONSHIP_COLUMNS_MISMATCHED instead of silently truncating the join - Document that dataset names are not preserved when they differ from the source table name

Copilot

Pull request overview

Copilot reviewed 13 out of 14 changed files in this pull request and generated 3 comments.

ota2000 · 2026-07-23T05:30:23Z

+        if extension.vendor_name == LIGHTDASH_VENDOR_NAME:
+            try:
+                data.update(json.loads(extension.data))
+            except (TypeError, ValueError):
+                pass


Fixed in bb09df3 — unparseable extension data now reports an EXTENSION_DATA_INVALID issue, with a regression test.

ota2000 · 2026-07-23T05:30:24Z

+    def _convert_sql_metric(
+        self, metric_name: str, definition: Dict[str, Any], *, dataset_name: str
+    ) -> OSIMetric:
+        sql = definition.get("sql") or ""
+        expression = lightdash_sql_to_osi(sql, dataset_name)
+        return self._build_metric(


Fixed in bb09df3 — model-level metrics without sql are skipped with a METRIC_SQL_MISSING issue instead of importing an empty expression, with a regression test.

ota2000 · 2026-07-23T05:30:26Z

+[tool.uv.sources]
+# apache-ossie is not yet published to PyPI; resolve it from the in-repo
+# package for now. Remove this block once apache-ossie published to PyPI.
+apache-ossie = { path = "../../python", editable = true}


Fixed in bb09df3 — added the [tool.uv] block mirroring the other converters, and verified from a clean checkout that uv sync + uv run pytest passes (27 tests).

… model metrics, add [tool.uv] - Surface unparseable lightdash extension data as an EXTENSION_DATA_INVALID issue instead of silently dropping it - Skip model-level metrics without sql on import, reporting METRIC_SQL_MISSING rather than emitting an empty OSI expression - Add the [tool.uv] block (required-version, default-groups) used by the other converters so uv sync reliably installs the dev group in CI

Copilot

Pull request overview

Copilot reviewed 13 out of 14 changed files in this pull request and generated no new comments.

Mirror the change from the recent CI actions path cleanup: trigger on this workflow's own file instead of the whole .github tree.

* upstream/main: CI for databricks converter (apache#261) fix(dbt): resolve SUM_BOOLEAN semantic model for qualified columns (apache#292) [OSSIE][NVIDIA_GSF] Add bidirectional GSF semantic model converter (apache#247) Fix typo in python README.md (apache#284) Bump Java from 17 to 21 (apache#283) Adding AI disclosures updates (apache#259) Fix Omni pytest (apache#263) Updating CI Actions path (apache#276) Bump java version from 11 to 17 for Polaris converter (apache#278) Make the spec's own examples validate against osi-schema.json (apache#209) [Ontology] Flatten concept declarations in ontology spec (apache#257) feat(cli): add plugin invocation protocol (apache#155) Add datatype field to Field and Metric; reframe is_time as role marker (apache#113) Cleanup impl dir (apache#252)

Spec change apache#113 added `datatype` to fields and metrics and reframed `dimension.is_time` as a role marker rather than a type flag, which broke the TPC-DS round-trip: the converter inferred `is_time` from the Lightdash type and wrote `false` where the source left it unset. - Map Lightdash dimension types to and from `datatype`, so the type is carried in standard vocabulary instead of a vendor extension - Leave `is_time` unset on import (Lightdash has no equivalent marker) and report TIME_ROLE_NOT_REPRESENTABLE when a non-temporal field is flagged as a time axis on export - Emit a Lightdash type only for fields that are dimensions, so a measure-only field is not turned into a dimension by a round-trip - Document that datatypes round-trip by category rather than exact type

ota2000 · 2026-08-20T02:53:46Z

Rebased onto current main and updated the converter for the spec change in #113 (datatype on fields/metrics, dimension.is_time reframed as a role marker), which landed after this PR was opened.

Without it the TPC-DS round-trip fails — the converter used to infer is_time from the Lightdash type and wrote false where the example leaves it unset ({'d_date': (False, True)} != {'d_date': (None, True)}).

What changed:

Lightdash dimension types now map to and from datatype, so the type travels in standard vocabulary instead of the vendor extension (String↔string, Integer/Decimal/Float→number, Date↔date, DateTime/DateTimeTz→timestamp, Boolean↔boolean).
is_time is left unset on import — Lightdash has no role marker to derive it from — and a non-temporal field flagged as a time axis (like TPC-DS d_year, an Integer with is_time: true) is reported with a new TIME_ROLE_NOT_REPRESENTABLE issue on export.
A Lightdash type is only emitted for fields that are dimensions, so a measure-only field is no longer turned into a dimension by a round-trip.
README documents that datatypes round-trip by category rather than exact type (Integer returns as Decimal).

30 tests pass, verified locally on Python 3.11–3.14.

Copilot AI review requested due to automatic review settings July 23, 2026 05:11

ota2000 mentioned this pull request Jul 23, 2026

Add a Lightdash converter #227

Open

Copilot started reviewing on behalf of ota2000 July 23, 2026 05:11 View session

Copilot AI reviewed Jul 23, 2026

View reviewed changes

ota2000 requested a review from Copilot July 23, 2026 05:22

Copilot started reviewing on behalf of ota2000 July 23, 2026 05:22 View session

Copilot AI reviewed Jul 23, 2026

View reviewed changes

ota2000 requested a review from Copilot July 23, 2026 05:30

Copilot started reviewing on behalf of ota2000 July 23, 2026 05:31 View session

Copilot AI reviewed Jul 23, 2026

View reviewed changes

ota2000 added 3 commits July 29, 2026 10:38

Align CI workflow path filter with the repo-wide update

41b8192

Mirror the change from the recent CI actions path cleanup: trigger on this workflow's own file instead of the whole .github tree.

ota2000 force-pushed the add-lightdash-converter branch from 36e0033 to 049a528 Compare August 20, 2026 02:53

Navigation Menu

Uh oh!

Add Lightdash converter - #253

Add Lightdash converter#253
ota2000 wants to merge 7 commits into
apache:mainfrom
ota2000:add-lightdash-converter

ota2000 commented Jul 23, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

ota2000 Jul 23, 2026

Uh oh!

ota2000 Jul 23, 2026

Uh oh!

ota2000 Jul 23, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

ota2000 Jul 23, 2026

Uh oh!

ota2000 Jul 23, 2026

Uh oh!

ota2000 Jul 23, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

ota2000 commented Aug 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

ota2000 commented Jul 23, 2026

What

Conventions

Verification

Notes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

ota2000 Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

ota2000 Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

ota2000 Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

ota2000 Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

ota2000 Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

ota2000 Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

ota2000 commented Aug 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants