diff --git a/.github/workflows/converter-databricks-ci.yml b/.github/workflows/converter-databricks-ci.yml index 54ec6f0f..d4061961 100644 --- a/.github/workflows/converter-databricks-ci.yml +++ b/.github/workflows/converter-databricks-ci.yml @@ -32,11 +32,14 @@ on: - '.github/workflows/converter-databricks-ci.yml' jobs: - build: + python: runs-on: ubuntu-latest strategy: matrix: python-version: ["3.11", "3.12", "3.13", "3.14"] + defaults: + run: + working-directory: converters/databricks/python steps: - name: Checkout project @@ -53,11 +56,30 @@ jobs: echo "${HOME}/.local/bin" >> "${GITHUB_PATH}" - name: Sync dependencies - working-directory: converters/databricks run: | uv sync - name: Unit Tests - working-directory: converters/databricks run: | uv run pytest + + java: + runs-on: ubuntu-latest + defaults: + run: + working-directory: converters/databricks/java + + steps: + - name: Checkout project + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + + - name: Set up JDK 21 + uses: actions/setup-java@03ad4de0992f5dab5e18fcb136590ce7c4a0ac95 # v5.6.0 + with: + java-version: '21' + distribution: 'temurin' + cache: 'maven' + + - name: Build and test + run: | + mvn -B verify diff --git a/converters/databricks/README.md b/converters/databricks/README.md index e2fe6ccb..ff7997ca 100644 --- a/converters/databricks/README.md +++ b/converters/databricks/README.md @@ -1,124 +1,17 @@ - +Layout +------ -# Apache Ossie Databricks Converter +| Path | Language | Role | +|------|----------|------| +| [`java/`](java/) | Java | The maintained implementation; also ships a command-line tool (`OssieDatabricksConverter`). | +| [`python/`](python/) | Python | The original reference implementation. To be deprecated. | -Bidirectional, offline conversion between an [Apache Ossie](https://github.com/apache/ossie) -semantic model and a Databricks -[Unity Catalog Metric View](https://docs.databricks.com/aws/en/metric-views/) (YAML -`1.1`). No Databricks connection required. - -- **Export** (`ossie-databricks export`): Apache Ossie -> Metric View (one fact - `source` with a nested `joins` tree and a flat `dimensions` list). -- **Import** (`ossie-databricks import`): Metric View -> Apache Ossie. Metric View features Apache Ossie has - no native field for are preserved in `custom_extensions[DATABRICKS]`, so - `MV -> Apache Ossie -> MV` is lossless. - -On **export** (Apache Ossie -> Metric View), Apache Ossie features with no Metric View slot -- relationship -`ai_context`, `dimension.is_time`, non-`DATABRICKS`/`ANSI_SQL` dialects, foreign-vendor -`custom_extensions` -- are **dropped with a warning**. On **import** (Metric View -> Apache Ossie), -Metric View only features (filter, window, format, rely, ...) are instead **preserved** in -`custom_extensions[DATABRICKS]`, so `MV -> Apache Ossie -> MV` is lossless. Any input that breaks a -[requirement](#requirements) **raises a `ConversionError`** -- the converter never -silently drops a field or produces an invalid result. - -## Installation - -```bash -pip install apache-ossie-databricks # once published to PyPI -# or, from a checkout of this directory: -pip install -e . -``` - -The only runtime dependency is `PyYAML`. Python 3.11+. - -## Usage - -### Command line - -```bash -ossie-databricks export -i model.yaml -o view.yaml [--source orders] # Apache Ossie -> Metric View -ossie-databricks import -i view.yaml -o model.yaml [--name my_model] # Metric View -> Apache Ossie -``` - -With no `-o`, output goes to stdout. `--source` (export) picks the fact/grain (default: -the FK-sink dataset; naming a coarser-grain dataset produces `one_to_many` joins); -`--name` (import) sets the Apache Ossie model name (default: the source's last identifier). - -### Python API - -```python -from ossie_databricks import convert_ossie_to_metric_view, convert_metric_view_to_ossie - -metric_view_yaml = convert_ossie_to_metric_view(ossie_yaml_str) # optionally choose the fact/grain, e.g. (ossie_yaml_str, source="orders") -ossie_yaml = convert_metric_view_to_ossie(metric_view_yaml_str, model_name="sales") -``` - -## Mapping - -Each row maps in both directions; the **Notes** flag where a behavior is specific to -**export** (Apache Ossie -> Metric View) or **import** (Metric View -> Apache Ossie). - -| Apache Ossie | Metric View (v1.1) | Notes | -|---|---|---| -| `semantic_model.description` | `comment` | Model-level description only. | -| root dataset | `source` | The fact/grain. | -| other `datasets` | nested `joins[]` | Export: the relationship graph is reassembled into the join tree; a dataset reached by two paths (a diamond) fans out into one aliased join per path. | -| `relationship` `from_columns`/`to_columns` | join `on` (differing names) / `using` (shared names) | Decomposed into columns on import; rebuilt into `on`/`using` on export. | -| `relationship.from`/`to` direction | join `cardinality` | Export: source on the many (`from`) side -> `many_to_one`; on the one (`to`) side -> `one_to_many`. | -| `dataset.primary_key` / `unique_keys` | join `rely.at_most_one_match` | Both directions: export sets `at_most_one_match` when a key covers the join columns; import recovers a `unique_keys` from it. | -| `dataset.fields[]` | `dimensions[]` | Export: fields flatten into one list and a joined column is qualified by its full join path (`customer.c_name`; `customer.region.r_name` when nested). | -| `field.expression.dialects[]` | `expr` | Export: prefer the `DATABRICKS` dialect, else `ANSI_SQL`. | -| `metrics[]` | `measures[]` | Export: fact columns are referenced bare (`SUM(amount)`). | -| `field.label` | `display_name` | | -| `field` / `metric` `description` | `comment` | | -| `ai_context.synonyms` | `synonyms` | | -| `custom_extensions[DATABRICKS]` | `filter`, `window`, `format`, `rely`, `materialization` | Import stashes Metric View only features here; export restores them -- keeping `MV -> Apache Ossie -> MV` lossless. | - -## Requirements - -Conversion raises a `ConversionError` (rather than guessing or emitting something -invalid) when an input breaks one of these: - -- the Metric View `version` is not `1.1`; -- a `source` is not a 3-part `catalog.schema.table` name or a `SELECT`/`WITH` subquery; -- the relationship graph is not acyclic and resolvable to a single fact -- a cycle, or - multiple candidate facts without `--source`, is rejected (a diamond is allowed and - fanned out); -- a join has no condition (a cross join has no Apache Ossie relationship form); -- a join condition is non-equi or otherwise can't be decomposed into equi-join columns - (Apache Ossie relationships are equi-joins, so the join has no Apache Ossie representation); -- the input YAML is malformed. - -## Development - -```bash -pip install -e ".[dev]" -python3 -m pytest tests/ -``` - -Example-based unit tests plus Hypothesis property-based round-trip tests -(`test_roundtrip_properties.py`, which skip if `hypothesis` is not installed). - -## Future effort - -Both the Apache Ossie specification and the Databricks Unity Catalog Metric View YAML are still -evolving. As either side adds or changes fields, this converter will be updated to track -them -- extending the mapping and coverage in both directions to keep the conversion -current and to support as much as each format allows over time. +See [`java/README.md`](java/README.md) and [`python/README.md`](python/README.md) for building and +using each implementation. diff --git a/converters/databricks/java/README.md b/converters/databricks/java/README.md new file mode 100644 index 00000000..b24e9895 --- /dev/null +++ b/converters/databricks/java/README.md @@ -0,0 +1,137 @@ +# Apache Ossie Databricks Converter + +Bidirectional, offline conversion between an [Apache Ossie](https://github.com/apache/ossie) +semantic model and a Databricks +[Unity Catalog Metric View](https://docs.databricks.com/aws/en/metric-views/) (YAML `1.1`). Pure +YAML text in, YAML text out: it reads and writes the two formats as parsed maps and lists. + +- **Export** (`MetricViewToOssie`): Metric View -> Apache Ossie. The direction is named from the + Metric View's point of view -- it takes a Metric View *out* to Ossie. Metric-View-only features + Apache Ossie has no native field for are preserved in `custom_extensions[DATABRICKS]`, so + `MV -> Apache Ossie -> MV` is lossless. +- **Import** (`OssieToMetricView`): Apache Ossie -> Metric View (one fact `source` with a nested + `joins` tree and a flat `dimensions` list). + +On **import** (Apache Ossie -> Metric View), Apache Ossie features with no Metric View slot -- +relationship `ai_context`, `dimension.is_time`, non-`DATABRICKS`/`ANSI_SQL` dialects, foreign-vendor +`custom_extensions` -- are **dropped with a notice**. On **export** (Metric View -> Apache Ossie), +Metric-View-only features (`filter`, `parameters`, `materialization`, per-column `format`, measure +`window` / `partition`) are instead **preserved** in `custom_extensions[DATABRICKS]`, so +`MV -> Apache Ossie -> MV` is lossless. Any input that breaks a [requirement](#requirements) +**raises a `ConversionException`** -- the converter never silently drops a field or produces an +invalid result. + +## Requirements + +- **Java 21+** +- **Maven 3.6+** -- required to build the jar + +## Building + +Build the self-contained executable jar from source: + +```bash +mvn clean package +``` + +This produces `target/ossie-databricks-converter-0.1.0-SNAPSHOT.jar` with all dependencies +(Jackson and SnakeYAML) bundled. + +## Usage + +### Command line + +```bash +# import: Apache Ossie -> Metric View +java -jar target/ossie-databricks-converter-0.1.0-SNAPSHOT.jar import model.yaml -o view.yaml + +# export: Metric View -> Apache Ossie +java -jar target/ossie-databricks-converter-0.1.0-SNAPSHOT.jar export view.yaml -o model.yaml +``` + +With no `-o`, output goes to stdout. `--source` (import) picks the fact/grain (default: the FK-sink +dataset; naming a coarser-grain dataset produces `one_to_many` joins); `--name` (export) sets the +Apache Ossie model name (default: the source's last identifier). Conversion notices (features +dropped on import) are written to stderr; a non-convertible input exits non-zero. + +### Java API + +```java +import org.apache.ossie.converter.databricks.OssieConverter; + +// export: Metric View -> Apache Ossie (optionally name the model; default: the source's last part) +OssieConverter.Result ossie = OssieConverter.convertMetricViewToOssie(metricViewYaml, "sales"); + +// import: Apache Ossie -> Metric View (optionally choose the fact/grain; default: the FK-sink +// dataset -- naming a coarser-grain dataset produces one_to_many joins) +OssieConverter.Result view = OssieConverter.convertOssieToMetricView(ossieYaml, "orders"); +``` + +Each `Result` carries the output YAML (`result.yaml`) and any notices raised (`result.notices`, +the features dropped on import). A broken [requirement](#requirements) throws a +`ConversionException` instead. + +## Mapping + +Each row maps in both directions; the **Notes** flag where a behavior is specific to +**export** (Metric View -> Apache Ossie) or **import** (Apache Ossie -> Metric View). + +| Apache Ossie | Metric View (v1.1) | Notes | +|---|---|---| +| `semantic_model.description` | `comment` | Model-level description only. | +| root dataset | `source` | The fact/grain. | +| other `datasets` | nested `joins[]` | Import: the relationship graph is reassembled into the join tree; a dataset reached by two paths (a diamond) fans out into one aliased join per path. | +| `relationship` `from_columns`/`to_columns` | join `on` (differing names) / `using` (shared names) | Decomposed into columns on export; rebuilt into `on`/`using` on import. | +| `relationship.from`/`to` direction | join `cardinality` | Import: source on the many (`from`) side -> `many_to_one`; on the one (`to`) side -> `one_to_many`. | +| `dataset.primary_key` / `unique_keys` | join `rely.at_most_one_match` | Both directions: import sets `at_most_one_match` when a key covers the join columns; export recovers a `unique_keys` from it. | +| `dataset.fields[]` | `dimensions[]` | Import: fields flatten into one list and a joined column is qualified by its full join path (`customer.c_name`; `customer.region.r_name` when nested). | +| `field.expression.dialects[]` | `expr` | Import: prefer the `DATABRICKS` dialect, else `ANSI_SQL`. | +| `metrics[]` | `measures[]` | Import: fact columns are referenced bare (`SUM(amount)`). | +| `field.label` | `display_name` | | +| `field` / `metric` `description` | `comment` | | +| `ai_context.synonyms` | `synonyms` | | +| `custom_extensions[DATABRICKS]` | `filter`, `parameters`, `materialization`, per-column `format`, measure `window` / `partition` | Export stashes Metric-View-only features here; import restores them -- keeping `MV -> Apache Ossie -> MV` lossless. | + +## Requirements + +Conversion throws a `ConversionException` (rather than guessing or emitting something invalid) when +an input breaks one of these: + +- the Metric View `version` is not `1.1`; +- a `source` is not a 3-part `catalog.schema.table` name or a `SELECT`/`WITH` subquery; +- the relationship graph is not acyclic and resolvable to a single fact -- a cycle, or multiple + candidate facts without a chosen source, is rejected (a diamond is allowed and fanned out); +- a join has no condition (a cross join has no Apache Ossie relationship form); +- a join condition is non-equi or otherwise can't be decomposed into equi-join columns (Apache + Ossie relationships are equi-joins, so the join has no Apache Ossie representation); +- the input YAML is malformed. + +## Development + +Run the test suite: + +```bash +mvn test +``` + +JUnit 5 suites (unit + round-trip) live under `src/test/java/`, with the YAML fixtures in +`src/test/resources/`. The source layout: + +``` +src/main/java/org/apache/ossie/converter/databricks/ + OssieConverter.java public facade: entry points + ConversionException/Notices/Result + OssieConverterCommon.java shared constants, YAML I/O, map accessors, the stash codec + MetricViewToOssie.java export: Metric View v1.1 -> Apache Ossie + OssieToMetricView.java import: Apache Ossie -> Metric View v1.1 + OssieDatabricksConverter.java command-line entry point (import / export) +``` + +The authoritative contract is Metric View YAML v1.1 as Databricks defines it; the checked-in +fixtures under `src/test/resources/` pin the expected output of both directions. + +## Future effort + +Both the Apache Ossie specification and the Databricks Unity Catalog Metric View YAML are still +evolving. As either side adds or changes fields, this converter will be updated to track them -- +extending the mapping and coverage in both directions to keep the conversion current and to support +as much as each format allows over time. diff --git a/converters/databricks/java/pom.xml b/converters/databricks/java/pom.xml new file mode 100644 index 00000000..84120eb1 --- /dev/null +++ b/converters/databricks/java/pom.xml @@ -0,0 +1,131 @@ + + + + + 4.0.0 + + + org.apache + apache + 39 + + + + org.apache.ossie + ossie-databricks-converter + 0.1.0-SNAPSHOT + jar + + Apache Ossie Databricks Converter + Converts between Apache Ossie semantic models and Databricks Unity Catalog Metric Views + + + 21 + UTF-8 + 2.2 + 2.18.9 + 5.10.2 + + + + + + org.yaml + snakeyaml + ${snakeyaml.version} + + + + + com.fasterxml.jackson.dataformat + jackson-dataformat-yaml + ${jackson.version} + + + com.fasterxml.jackson.core + jackson-databind + ${jackson.version} + + + + + org.junit.jupiter + junit-jupiter + ${junit.version} + test + + + + + + + org.apache.maven.plugins + maven-jar-plugin + 3.3.0 + + + + org.apache.ossie.converter.databricks.OssieDatabricksConverter + + + + + + + + org.apache.maven.plugins + maven-shade-plugin + 3.5.1 + + + package + + shade + + + + + + + + org.apache.rat + apache-rat-plugin + + + verify + + check + + + + + true + + **/target/** + + **/*.md + + + + + + diff --git a/converters/databricks/java/src/main/java/org/apache/ossie/converter/databricks/MetricViewToOssie.java b/converters/databricks/java/src/main/java/org/apache/ossie/converter/databricks/MetricViewToOssie.java new file mode 100644 index 00000000..39a269af --- /dev/null +++ b/converters/databricks/java/src/main/java/org/apache/ossie/converter/databricks/MetricViewToOssie.java @@ -0,0 +1,474 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.ossie.converter.databricks; + +import static org.apache.ossie.converter.databricks.OssieConverterCommon.CARD_MANY_TO_ONE; +import static org.apache.ossie.converter.databricks.OssieConverterCommon.CARD_ONE_TO_MANY; +import static org.apache.ossie.converter.databricks.OssieConverterCommon.DIALECT_DATABRICKS; +import static org.apache.ossie.converter.databricks.OssieConverterCommon.MAPPER; +import static org.apache.ossie.converter.databricks.OssieConverterCommon.MV_VERSION; +import static org.apache.ossie.converter.databricks.OssieConverterCommon.OSSIE_VERSION; +import static org.apache.ossie.converter.databricks.OssieConverterCommon.STASH_SOURCE_KEY; +import static org.apache.ossie.converter.databricks.OssieConverterCommon.asList; +import static org.apache.ossie.converter.databricks.OssieConverterCommon.asMap; +import static org.apache.ossie.converter.databricks.OssieConverterCommon.get; +import static org.apache.ossie.converter.databricks.OssieConverterCommon.isSimpleIdentifier; +import static org.apache.ossie.converter.databricks.OssieConverterCommon.loadYaml; +import static org.apache.ossie.converter.databricks.OssieConverterCommon.lastIdentifier; +import static org.apache.ossie.converter.databricks.OssieConverterCommon.replaceOutsideLiterals; +import static org.apache.ossie.converter.databricks.OssieConverterCommon.requireStr; +import static org.apache.ossie.converter.databricks.OssieConverterCommon.str; +import static org.apache.ossie.converter.databricks.OssieConverterCommon.strList; +import static org.apache.ossie.converter.databricks.OssieConverterCommon.truthy; +import static org.apache.ossie.converter.databricks.OssieConverterCommon.validateSource; +import static org.apache.ossie.converter.databricks.OssieConverterCommon.writeStash; + +import java.util.ArrayList; +import java.util.HashMap; +import java.util.HashSet; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Locale; +import java.util.Map; +import java.util.Set; +import java.util.regex.Matcher; +import java.util.regex.Pattern; + +import org.apache.ossie.converter.databricks.OssieConverter.ConversionException; +import org.apache.ossie.converter.databricks.OssieConverter.Notices; +import org.apache.ossie.converter.databricks.OssieConverter.Result; + +/** + * IMPORT direction: Metric View v1.1 YAML -> Apache Ossie semantic model. Shared helpers come + * from {@link OssieConverterCommon}; the public entry point is re-exported through + * {@link OssieConverter}. + */ +// Map-based YAML manipulation: casts of the parsed Object graph to Map/List are inherently +// unchecked; the asMap/asList helpers guard them, so unchecked warnings here are expected. +@SuppressWarnings("unchecked") +final class MetricViewToOssie { + + private static final String[] MODEL_STASH_KEYS = {"filter", "parameters", "materialization"}; + private static final String[] JOIN_STASH_KEYS = {"rely", "cardinality"}; + // Metric-View-only column fields with no Apache Ossie representation, preserved verbatim in the + // DATABRICKS stash so an import can restore them. `window` and `partition` are measure-only + // (they live on MeasureExpression); a dimension never carries them, so the shared key list is + // simply never hit for those on the dimension path. + private static final String[] COLUMN_STASH_KEYS = {"format", "window", "partition"}; + private static final Pattern NON_EQUI_RE = Pattern.compile("[<>!]=|<>|[<>]"); + private static final Pattern AND_SPLIT_RE = Pattern.compile("\\s+AND\\s+", Pattern.CASE_INSENSITIVE); + private static final Pattern EQ_CLAUSE_RE = Pattern.compile("^\\s*(.+?)\\s*=\\s*(.+?)\\s*$"); + private static final Pattern SOURCE_QUALIFIER_RE = Pattern.compile("\\bsource\\."); + + private MetricViewToOssie() {} + + static Result convertMetricViewToOssie(String mvYamlStr, String modelName) { + Notices notices = new Notices(); + Map view; + try { + view = asMap(loadYaml(mvYamlStr)); + } catch (Exception e) { + throw new ConversionException("Invalid Metric View YAML: " + e.getMessage(), e); + } + if (view.isEmpty()) { + throw new ConversionException("Invalid Metric View YAML: expected a mapping at the root"); + } + String version = str(get(view, "version")); + if (!MV_VERSION.equals(version)) { + throw new ConversionException("Unsupported Metric View version '" + version + + "'. This converter targets v" + MV_VERSION + " only."); + } + Map model = convertView(view, modelName, notices); + Map out = new LinkedHashMap<>(); + out.put("version", OSSIE_VERSION); + List models = new ArrayList<>(); + models.add(model); + out.put("semantic_model", models); + try { + return new Result(MAPPER.writeValueAsString(out), notices.toList()); + } catch (Exception e) { + throw new ConversionException("failed to serialize Apache Ossie YAML: " + e.getMessage(), e); + } + } + + private static Map convertView( + Map view, String modelName, Notices notices) { + Object source = get(view, "source"); + if (source == null || source.toString().isEmpty()) { + throw new ConversionException("Metric View is missing required 'source'"); + } + String s = source.toString().trim(); + String firstToken = s.split("\\s+", 2)[0].toUpperCase(Locale.ROOT); + boolean isSql = firstToken.equals("SELECT") || firstToken.equals("WITH"); + String lastId = lastIdentifier(source); + String factName; + if (modelName != null) { + factName = modelName; + } else if (!isSql && lastId != null && isSimpleIdentifier(lastId)) { + factName = lastId; + } else { + factName = "metric_view"; + } + validateSource(source, factName); + + List> datasets = new ArrayList<>(); + Map factDs = new LinkedHashMap<>(); + factDs.put("name", factName); + factDs.put("source", source); + datasets.add(factDs); + List> relationships = new ArrayList<>(); + Map aliasToDataset = new HashMap<>(); + aliasToDataset.put("source", factName); + aliasToDataset.put(factName, factName); + Set seenNames = new HashSet<>(); + seenNames.add(factName.trim().toLowerCase(Locale.ROOT)); + + walk(factName, "source", asList(get(view, "joins")), datasets, relationships, + aliasToDataset, seenNames); + + // `fields` is a v1.1 alias for `dimensions`: an empty `dimensions: []` falls through to + // `fields`, and the "both set" warning fires only when BOTH are non-empty (not merely + // present). + List dimList = asList(get(view, "dimensions")); + List fieldList = asList(get(view, "fields")); + if (!dimList.isEmpty() && !fieldList.isEmpty()) { + notices.warn("view", "both 'dimensions' and 'fields' are set; 'fields' is a v1.1 alias " + + "for 'dimensions', so the 'fields' list is ignored"); + } + Map> fieldsByDataset = new LinkedHashMap<>(); + for (Map d : datasets) { + fieldsByDataset.put((String) d.get("name"), new ArrayList<>()); + } + List dims = !dimList.isEmpty() ? dimList : fieldList; + for (Object dimObj : dims) { + Map dim = asMap(dimObj); + if (isWildcard(dim)) { + notices.warn("dimension", "wildcard column '" + str(get(dim, "expr")) + + "' has no Apache Ossie field representation; skipped"); + continue; + } + Object[] converted = convertDimension(dim, aliasToDataset, factName); + fieldsByDataset.get((String) converted[0]).add(converted[1]); + } + for (Map d : datasets) { + List flds = fieldsByDataset.get((String) d.get("name")); + if (!flds.isEmpty()) { + d.put("fields", flds); + } + } + + List metrics = new ArrayList<>(); + for (Object mObj : asList(get(view, "measures"))) { + Map m = asMap(mObj); + if (isWildcard(m)) { + notices.warn("measure", "wildcard measure '" + str(get(m, "expr")) + + "' has no Apache Ossie metric representation; skipped"); + continue; + } + metrics.add(convertMeasure(m, factName)); + } + + Map model = new LinkedHashMap<>(); + model.put("name", factName); + if (truthy(get(view, "comment"))) { + model.put("description", get(view, "comment")); + } + model.put("datasets", datasets); + if (!relationships.isEmpty()) { + model.put("relationships", relationships); + } + if (!metrics.isEmpty()) { + model.put("metrics", metrics); + } + + Map modelStash = new LinkedHashMap<>(); + for (String k : MODEL_STASH_KEYS) { + if (view.containsKey(k)) { + modelStash.put(k, view.get(k)); + } + } + if (hasOtm(asList(get(view, "joins")))) { + modelStash.put(STASH_SOURCE_KEY, factName); + } + writeStash(model, modelStash); + return model; + } + + private static void walk(String parentName, String parentAlias, List joins, + List> datasets, List> relationships, + Map aliasToDataset, Set seenNames) { + for (Object joinObj : joins) { + Map join = asMap(joinObj); + String child = requireStr(join, "name", "join"); + if (child.trim().equalsIgnoreCase("source")) { + throw new ConversionException( + "Join name 'source' is reserved for the fact source; rename the join."); + } + if (!seenNames.add(child.trim().toLowerCase(Locale.ROOT))) { + throw new ConversionException("Duplicate dataset/join name '" + child + + "'; Metric View join names and the source must be distinct (case-insensitively)."); + } + Map childDs = new LinkedHashMap<>(); + childDs.put("name", child); + childDs.put("source", requireStr(join, "source", "join '" + child + "'")); + datasets.add(childDs); + aliasToDataset.put(child, child); + Map rel = convertJoin(join, parentName, parentAlias, child); + relationships.add(rel); + // rely.at_most_one_match on a many_to_one join -> recover a unique_key on the child. + Map rely = asMap(get(join, "rely")); + List toCols = strList(get(rel, "to_columns")); + if (child.equals(str(get(rel, "to"))) && !toCols.isEmpty() + && Boolean.TRUE.equals(rely.get("at_most_one_match"))) { + List uk = new ArrayList<>(); + uk.add(new ArrayList<>(toCols)); + childDs.put("unique_keys", uk); + } + walk(child, child, asList(get(join, "joins")), datasets, relationships, + aliasToDataset, seenNames); + } + } + + private static boolean hasOtm(List joins) { + for (Object jObj : joins) { + Map j = asMap(jObj); + if (CARD_ONE_TO_MANY.equalsIgnoreCase(str(get(j, "cardinality")))) { + return true; + } + if (hasOtm(asList(get(j, "joins")))) { + return true; + } + } + return false; + } + + private static Map convertJoin( + Map join, String parentName, String parentAlias, String child) { + boolean hasUsing = get(join, "using") != null && !asList(get(join, "using")).isEmpty(); + boolean hasOn = str(get(join, "on")) != null && !str(get(join, "on")).isEmpty(); + if (!hasUsing && !hasOn) { + throw new ConversionException("Join '" + child + "' has no join condition (empty or " + + "absent 'on'/'using'); condition-less (cross) joins have no Apache Ossie " + + "relationship representation."); + } + Object[] decomposed = decomposeOn(join, parentAlias, parentName, child); + List parentCols = (List) decomposed[0]; + List childCols = (List) decomposed[1]; + String rawOn = (String) decomposed[2]; + if (rawOn != null) { + throw new ConversionException("Join '" + child + "' uses a non-equi or unsupported join " + + "condition ('on: " + rawOn + "') that an Apache Ossie relationship cannot represent. " + + "Apache Ossie joins are equi-joins of simple `alias.column` pairs (the fact side may " + + "be qualified with `source`, the source table name, or left bare). Cannot import."); + } + // Only fall back to `using` when there is no `on` to decompose (see decomposeOn: `on` wins). + if (!hasOn && hasUsing && parentCols.isEmpty()) { + List using = strList(get(join, "using")); + parentCols = new ArrayList<>(using); + childCols = new ArrayList<>(using); + } + + String cardinality = str(get(join, "cardinality")); + if (cardinality == null) { + cardinality = CARD_MANY_TO_ONE; + } + Map rel = new LinkedHashMap<>(); + if (cardinality.toLowerCase(Locale.ROOT).equals(CARD_ONE_TO_MANY)) { + rel.put("name", child + "_to_" + parentName); + rel.put("from", child); + rel.put("to", parentName); + rel.put("from_columns", childCols); + rel.put("to_columns", parentCols); + } else { + rel.put("name", parentName + "_to_" + child); + rel.put("from", parentName); + rel.put("to", child); + rel.put("from_columns", parentCols); + rel.put("to_columns", childCols); + } + Map stash = new LinkedHashMap<>(); + for (String k : JOIN_STASH_KEYS) { + if (join.containsKey(k)) { + stash.put(k, join.get(k)); + } + } + writeStash(rel, stash); + return rel; + } + + /** Returns {parentCols, childCols, rawOn}; rawOn non-null means reject. */ + private static Object[] decomposeOn( + Map join, String parentAlias, String parentName, String childAlias) { + // A join may carry both `on` and `using` (Metric View validation only requires that at least + // one is present). `on` takes precedence, matching how Databricks resolves the join criteria + // in DataModelUtils.getJoinCriteriaExpression: `case (Some(on), _) => ...`. Falling back to + // `using` when `on` is present would silently join on different columns than the view does. + String on = str(get(join, "on")); + if (on == null || on.isEmpty()) { + // No usable `on`; the caller derives the columns from `using`. + return new Object[] {new ArrayList(), new ArrayList(), null}; + } + Set parentAliases = new HashSet<>(); + parentAliases.add(parentAlias); + parentAliases.add(parentName); + boolean allowBare = "source".equals(parentAlias); + List fromCols = new ArrayList<>(); + List toCols = new ArrayList<>(); + for (String clause : AND_SPLIT_RE.split(on)) { + if (NON_EQUI_RE.matcher(clause).find()) { + return new Object[] {null, null, on}; + } + Matcher m = EQ_CLAUSE_RE.matcher(clause); + if (!m.matches()) { + return new Object[] {null, null, on}; + } + String[] left = splitAlias(m.group(1)); + String[] right = splitAlias(m.group(2)); + String la = left[0]; + String lc = left[1]; + String ra = right[0]; + String rc = right[1]; + if (!(isSimpleIdentifier(lc) && isSimpleIdentifier(rc))) { + return new Object[] {null, null, on}; + } + boolean lParent = parentAliases.contains(la) || (la == null && allowBare); + boolean rParent = parentAliases.contains(ra) || (ra == null && allowBare); + if (childAlias.equals(la) && rParent) { + fromCols.add(rc); + toCols.add(lc); + } else if (childAlias.equals(ra) && lParent) { + fromCols.add(lc); + toCols.add(rc); + } else { + return new Object[] {null, null, on}; + } + } + return new Object[] {fromCols, toCols, null}; + } + + /** `customer.c_custkey` -> {"customer","c_custkey"}; `x` -> {null,"x"}. */ + private static String[] splitAlias(String operand) { + operand = operand.trim(); + int dot = operand.indexOf('.'); + if (dot >= 0) { + return new String[] {operand.substring(0, dot).trim(), operand.substring(dot + 1).trim()}; + } + return new String[] {null, operand}; + } + + private static Object[] convertDimension( + Map dim, Map aliasToDataset, String factName) { + String name = requireStr(dim, "name", "dimension"); + String expr = requireStr(dim, "expr", "dimension '" + name + "'"); + String[] resolved = resolveColumn(expr, aliasToDataset, factName); + String dsName = resolved[0]; + String ossieExpr = resolved[1]; + + Map field = new LinkedHashMap<>(); + field.put("name", name); + field.put("expression", dialectExpr(ossieExpr)); + if (truthy(get(dim, "comment"))) { + field.put("description", get(dim, "comment")); + } + if (truthy(get(dim, "display_name"))) { + field.put("label", get(dim, "display_name")); + } + if (truthy(get(dim, "synonyms"))) { + Map ai = new LinkedHashMap<>(); + ai.put("synonyms", new ArrayList<>(asList(get(dim, "synonyms")))); + field.put("ai_context", ai); + } + Map stash = new LinkedHashMap<>(); + for (String k : COLUMN_STASH_KEYS) { + if (dim.containsKey(k)) { + stash.put(k, dim.get(k)); + } + } + writeStash(field, stash); + return new Object[] {dsName, field}; + } + + /** Map a dimension expression to {dataset_name, de-aliased_expression}. */ + private static String[] resolveColumn( + String expr, Map aliasToDataset, String factName) { + // The -1 limit keeps trailing empty segments, so a malformed `customer.` files under + // `customer` rather than under the fact. + String[] segments = expr.split("\\.", -1); + for (int i = 0; i < segments.length; i++) { + segments[i] = segments[i].trim(); + } + String ds = null; + int i = 0; + while (i < segments.length - 1 && aliasToDataset.containsKey(segments[i])) { + ds = aliasToDataset.get(segments[i]); + i++; + } + if (ds == null) { + return new String[] {factName, expr}; + } + StringBuilder rest = new StringBuilder(); + for (int j = i; j < segments.length; j++) { + if (j > i) { + rest.append("."); + } + rest.append(segments[j]); + } + String restStr = rest.toString(); + return isSimpleIdentifier(restStr) ? new String[] {ds, restStr} : new String[] {ds, expr}; + } + + private static Map convertMeasure(Map measure, String factName) { + String name = requireStr(measure, "name", "measure"); + String rawExpr = requireStr(measure, "expr", "measure '" + name + "'"); + String expr = replaceOutsideLiterals(rawExpr, SOURCE_QUALIFIER_RE, factName + "."); + Map metric = new LinkedHashMap<>(); + metric.put("name", name); + metric.put("expression", dialectExpr(expr)); + if (truthy(get(measure, "comment"))) { + metric.put("description", get(measure, "comment")); + } + if (truthy(get(measure, "synonyms"))) { + Map ai = new LinkedHashMap<>(); + ai.put("synonyms", new ArrayList<>(asList(get(measure, "synonyms")))); + metric.put("ai_context", ai); + } + Map stash = new LinkedHashMap<>(); + for (String k : COLUMN_STASH_KEYS) { + if (measure.containsKey(k)) { + stash.put(k, measure.get(k)); + } + } + writeStash(metric, stash); + return metric; + } + + private static Map dialectExpr(String expr) { + Map dialect = new LinkedHashMap<>(); + dialect.put("dialect", DIALECT_DATABRICKS); + dialect.put("expression", expr); + List dialects = new ArrayList<>(); + dialects.add(dialect); + Map expression = new LinkedHashMap<>(); + expression.put("dialects", dialects); + return expression; + } + + private static boolean isWildcard(Map col) { + return !col.containsKey("name"); + } +} diff --git a/converters/databricks/java/src/main/java/org/apache/ossie/converter/databricks/OssieConverter.java b/converters/databricks/java/src/main/java/org/apache/ossie/converter/databricks/OssieConverter.java new file mode 100644 index 00000000..278cdadc --- /dev/null +++ b/converters/databricks/java/src/main/java/org/apache/ossie/converter/databricks/OssieConverter.java @@ -0,0 +1,103 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.ossie.converter.databricks; + +import java.util.ArrayList; +import java.util.List; + +/** + * Bidirectional converter between Apache Ossie semantic models and Databricks Metric Views. + * This class is the public facade, re-exporting the two conversion entry points and the shared + * {@link ConversionException}/{@link Notices}/{@link Result} types. The implementation is split + * by direction so each half can be read on its own: + * + *
    + *
  • {@link OssieConverterCommon} -- shared constants, YAML I/O, and map accessors + *
  • {@link OssieToMetricView} -- EXPORT: Apache Ossie -> Metric View + *
  • {@link MetricViewToOssie} -- IMPORT: Metric View -> Apache Ossie + *
+ * + *

The authoritative contract is Metric View YAML v1.1 as defined by the Databricks serde + * ({@code com.databricks.sql.serde.v11}) and its validation rules; the checked-in YAML fixtures + * pin the expected output for both directions. + * + *

Conversion operates on parsed YAML as plain maps and lists rather than typed models, so the + * converter stays independent of the Databricks serde classes and runs standalone. Warnings + * ("drops") are collected into a Notices buffer and returned rather than written to stderr, so a + * SQL surface can present them to the caller. + */ +public final class OssieConverter { + + // Re-exported so callers can reference OssieConverter.OSSIE_VERSION / .MV_VERSION as before. + public static final String OSSIE_VERSION = OssieConverterCommon.OSSIE_VERSION; + public static final String MV_VERSION = OssieConverterCommon.MV_VERSION; + + private OssieConverter() {} + + /** Raised for any input the converter refuses to convert (Java twin of ConversionError). */ + public static final class ConversionException extends RuntimeException { + public ConversionException(String message) { + super(message); + } + public ConversionException(String message, Throwable cause) { + super(message, cause); + } + } + + /** Collects drop/rewrite notices during a conversion. */ + public static final class Notices { + private final List messages = new ArrayList<>(); + void warn(String scope, String msg) { + messages.add("[" + scope + "] " + msg); + } + public List toList() { + return new ArrayList<>(messages); + } + } + + /** Result of a conversion: the emitted YAML plus any drop notices. */ + public static final class Result { + public final String yaml; + public final List notices; + Result(String yaml, List notices) { + this.yaml = yaml; + this.notices = notices; + } + } + + /** EXPORT: Apache Ossie semantic model YAML -> Metric View v1.1 YAML. */ + public static Result convertOssieToMetricView(String osiYamlStr, String source) { + return OssieToMetricView.convertOssieToMetricView(osiYamlStr, source); + } + + /** IMPORT: Metric View v1.1 YAML -> Apache Ossie semantic model YAML. */ + public static Result convertMetricViewToOssie(String mvYamlStr, String modelName) { + return MetricViewToOssie.convertMetricViewToOssie(mvYamlStr, modelName); + } + + /** Parse YAML text into a plain value (YAML 1.2 booleans, matching the converter). */ + public static Object parseYaml(String s) { + return OssieConverterCommon.parseYaml(s); + } + + /** Serialize a value to YAML using the converter's write mapper (for tests without their + * own jackson-yaml import). */ + public static String dumpYaml(Object obj) { + return OssieConverterCommon.dumpYaml(obj); + } +} diff --git a/converters/databricks/java/src/main/java/org/apache/ossie/converter/databricks/OssieConverterCommon.java b/converters/databricks/java/src/main/java/org/apache/ossie/converter/databricks/OssieConverterCommon.java new file mode 100644 index 00000000..deefa677 --- /dev/null +++ b/converters/databricks/java/src/main/java/org/apache/ossie/converter/databricks/OssieConverterCommon.java @@ -0,0 +1,498 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.ossie.converter.databricks; + +import java.util.ArrayList; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Locale; +import java.util.Map; +import java.util.regex.Matcher; +import java.util.regex.Pattern; + +import com.fasterxml.jackson.databind.ObjectMapper; +import com.fasterxml.jackson.dataformat.yaml.YAMLFactory; +import com.fasterxml.jackson.dataformat.yaml.YAMLGenerator; + +import org.yaml.snakeyaml.LoaderOptions; +import org.yaml.snakeyaml.Yaml; +import org.yaml.snakeyaml.constructor.SafeConstructor; +import org.yaml.snakeyaml.nodes.Tag; +import org.yaml.snakeyaml.resolver.Resolver; + +import org.apache.ossie.converter.databricks.OssieConverter.ConversionException; + +/** + * Shared constants, YAML I/O, and typed accessors for the Apache Ossie <-> Databricks + * Metric View converter. The direction-specific logic lives in {@link OssieToMetricView} + * (export) and {@link MetricViewToOssie} (import), which static-import these members. The public + * entry points and shared types are re-exported through {@link OssieConverter}. + */ +// Map-based YAML manipulation: casts of the parsed Object graph to Map/List are inherently +// unchecked; the asMap/asList helpers guard them, so unchecked warnings here are expected. +@SuppressWarnings("unchecked") +final class OssieConverterCommon { + + // -- constants ------------------------------------------------------------- + static final String OSSIE_VERSION = "0.2.0.dev0"; + static final String MV_VERSION = "1.1"; + static final String VENDOR = "DATABRICKS"; + static final String DIALECT_DATABRICKS = "DATABRICKS"; + static final String DIALECT_ANSI = "ANSI_SQL"; + static final int SYNONYM_LIMIT = 10; + static final int STASH_VERSION = 1; + static final String STASH_SOURCE_KEY = "source_dataset"; + static final String CARD_ONE_TO_MANY = "one_to_many"; + static final String CARD_MANY_TO_ONE = "many_to_one"; + static final int MAX_JOIN_NODES = 200; + + static final Pattern IDENTIFIER_RE = Pattern.compile("^[A-Za-z_][A-Za-z0-9_]*$"); + static final Pattern SELECT_WITH_RE = + Pattern.compile("(?i)^(select|with)\\b"); + + static final ObjectMapper MAPPER = buildMapper(); + // Writer for the custom_extensions stash blob. The exact byte format is pinned by the + // checked-in fixtures: a space after ':' and ', ' between entries, as in + // {"_v": 1, "filter": "x"}. + static final com.fasterxml.jackson.databind.ObjectWriter JSON_WRITER = buildJsonWriter(); + + /** A MinimalPrettyPrinter (no newlines) using the stash blob's separators: ": " / ", ". */ + private static final class JsonDumpsPrinter + extends com.fasterxml.jackson.core.util.MinimalPrettyPrinter { + @Override + public void writeObjectFieldValueSeparator(com.fasterxml.jackson.core.JsonGenerator g) + throws java.io.IOException { + g.writeRaw(": "); + } + @Override + public void writeObjectEntrySeparator(com.fasterxml.jackson.core.JsonGenerator g) + throws java.io.IOException { + g.writeRaw(", "); + } + @Override + public void writeArrayValueSeparator(com.fasterxml.jackson.core.JsonGenerator g) + throws java.io.IOException { + g.writeRaw(", "); + } + } + + private static com.fasterxml.jackson.databind.ObjectWriter buildJsonWriter() { + // ESCAPE_NON_ASCII keeps the blob pure ASCII (every non-ASCII + // char is emitted as a \\uXXXX escape). Set on the JsonFactory so it takes effect on the + // generator. Jackson emits the hex in UPPERCASE; writeStash lowercases it so the blob is + // byte-identical to the checked-in fixtures. + com.fasterxml.jackson.core.JsonFactory jf = new com.fasterxml.jackson.core.JsonFactory(); + jf.enable(com.fasterxml.jackson.core.JsonGenerator.Feature.ESCAPE_NON_ASCII); + return new ObjectMapper(jf).writer(new JsonDumpsPrinter()); + } + + // Matches a real unicode escape in serialized JSON so writeStash can lowercase its hex digits. + // + // The escape must be preceded by an EVEN number of backslashes, otherwise the `u` belongs to an + // escaped backslash rather than to an escape sequence: a stashed value holding a literal + // backslash followed by "uABCD" serializes with a doubled backslash, where the `u` is ordinary + // text and must be left alone (lowercasing it there would corrupt the value). Group 1 captures + // the (possibly empty) run of escaped backslashes so it can be re-emitted verbatim; group 2 is + // the hex to lowercase. + private static final Pattern UNICODE_ESCAPE_RE = + Pattern.compile("(? YAML_READER = ThreadLocal.withInitial(() -> + new Yaml(new SafeConstructor(new LoaderOptions()), new org.yaml.snakeyaml.representer.Representer( + new org.yaml.snakeyaml.DumperOptions()), new org.yaml.snakeyaml.DumperOptions(), + new LoaderOptions(), new Yaml12Resolver())); + + static Object loadYaml(String s) { + return YAML_READER.get().load(s); + } + + private OssieConverterCommon() {} + + // -- typed accessors over parsed YAML (Object) ---------------------------- + @SuppressWarnings("unchecked") + static Map asMap(Object x) { + if (x instanceof Map) { + return (Map) x; + } + return new LinkedHashMap<>(); + } + + @SuppressWarnings("unchecked") + static List asList(Object x) { + if (x instanceof List) { + return (List) x; + } + return new ArrayList<>(); + } + + static Object get(Map m, String k) { + return m.get(k); + } + + static String str(Object x) { + if (x == null) { + return null; + } + return x.toString(); + } + + static List strList(Object x) { + List out = new ArrayList<>(); + for (Object o : asList(x)) { + if (o != null) { + out.add(o.toString()); + } + } + return out; + } + + // -- helpers --------------------------------------------------------------- + static boolean isSimpleIdentifier(Object expr) { + return expr instanceof String && IDENTIFIER_RE.matcher(((String) expr).trim()).matches(); + } + + /** + * Applies {@code pattern -> replacement} to {@code sql}, but only to the parts of the expression + * that are actual SQL code -- spans inside string literals ({@code '...'}, {@code "..."}), + * backquoted identifiers, {@code -- line} comments, and {@code /* block *}{@code /} comments are + * copied through untouched. + * + *

Measure expressions are rewritten to add or strip a fact qualifier, and a blind + * {@code replaceAll} over the raw text also rewrites any occurrence inside a literal: a measure + * such as {@code SUM(IF(source.region = 'source.us', amt, 0))} would silently become + * {@code ... = 'us'}, changing the predicate and therefore the measure's value. Only code spans + * may be rewritten. + * + *

{@code replacement} is treated as a literal string, not as a regex replacement template. + */ + static String replaceOutsideLiterals(String sql, Pattern pattern, String replacement) { + StringBuilder out = new StringBuilder(sql.length()); + int i = 0; + int codeStart = 0; + while (i < sql.length()) { + char c = sql.charAt(i); + int skipTo = -1; + if (c == '\'' || c == '"' || c == '`') { + skipTo = endOfQuoted(sql, i, c); + } else if (c == '-' && i + 1 < sql.length() && sql.charAt(i + 1) == '-') { + int nl = sql.indexOf('\n', i); + skipTo = nl < 0 ? sql.length() : nl; + } else if (c == '/' && i + 1 < sql.length() && sql.charAt(i + 1) == '*') { + int end = sql.indexOf("*/", i + 2); + skipTo = end < 0 ? sql.length() : end + 2; + } + if (skipTo < 0) { + i++; + continue; + } + // Rewrite the code span that precedes this literal/comment, then copy the span verbatim. + out.append(rewriteLiterally(sql.substring(codeStart, i), pattern, replacement)); + out.append(sql, i, skipTo); + i = skipTo; + codeStart = skipTo; + } + out.append(rewriteLiterally(sql.substring(codeStart), pattern, replacement)); + return out.toString(); + } + + /** Index just past the quoted span starting at {@code start}; handles doubled-quote escapes. */ + private static int endOfQuoted(String sql, int start, char quote) { + int i = start + 1; + while (i < sql.length()) { + char c = sql.charAt(i); + if (c == '\\' && quote != '`' && i + 1 < sql.length()) { + i += 2; + continue; + } + if (c == quote) { + // A doubled quote is an escaped quote, not the end of the span. + if (i + 1 < sql.length() && sql.charAt(i + 1) == quote) { + i += 2; + continue; + } + return i + 1; + } + i++; + } + // Unterminated literal: treat the remainder as part of the span rather than rewriting it. + return sql.length(); + } + + private static String rewriteLiterally(String code, Pattern pattern, String replacement) { + return pattern.matcher(code).replaceAll(Matcher.quoteReplacement(replacement)); + } + + // Presence is tested by key, so a legitimately falsy non-string value (0, false) is returned; + // a missing key, a null, or an empty/whitespace-only string is rejected. + static Object require(Map obj, String key, String what) { + if (!obj.containsKey(key) || obj.get(key) == null) { + throw new ConversionException(what + " is missing required '" + key + "'"); + } + Object value = obj.get(key); + if (value instanceof String && ((String) value).trim().isEmpty()) { + throw new ConversionException(what + " has an empty '" + key + "'"); + } + return value; + } + + // Like require(), but the value must be a string. + static String requireStr(Map obj, String key, String what) { + Object v = require(obj, key, what); + if (v instanceof String) { + return (String) v; + } + throw new ConversionException( + what + ": '" + key + "' must be a string, got " + v.getClass().getSimpleName()); + } + + static String validateSource(Object source, String datasetName) { + String s = source == null ? "" : source.toString().trim(); + if (s.isEmpty()) { + throw new ConversionException("Dataset '" + datasetName + "': missing/empty 'source'"); + } + if (SELECT_WITH_RE.matcher(s).find()) { + return s; + } + String[] parts = s.split("\\.", -1); + boolean ok = parts.length == 3; + if (ok) { + for (String p : parts) { + if (p.isEmpty() || containsWhitespace(p)) { + ok = false; + break; + } + } + } + if (ok) { + return s; + } + throw new ConversionException("Dataset '" + datasetName + "': source '" + source + + "' must be a 3-part catalog.schema.table identifier or a SELECT/WITH subquery"); + } + + private static boolean containsWhitespace(String p) { + for (int i = 0; i < p.length(); i++) { + if (Character.isWhitespace(p.charAt(i))) { + return true; + } + } + return false; + } + + static String pickExpression(Object osiExpression) { + // Keep the raw (possibly non-string) values so + // the type check below can fire; select DATABRICKS-or-ANSI by truthiness (`or`), so a + // null/empty DATABRICKS expr falls through to ANSI; and raise on a non-string chosen + // value rather than silently coercing it. + Map dialects = new LinkedHashMap<>(); + for (Object d : asList(get(asMap(osiExpression), "dialects"))) { + Map dm = asMap(d); + dialects.put(str(get(dm, "dialect")), get(dm, "expression")); + } + Object chosen = truthy(dialects.get(DIALECT_DATABRICKS)) + ? dialects.get(DIALECT_DATABRICKS) : dialects.get(DIALECT_ANSI); + if (chosen != null && !(chosen instanceof String)) { + throw new ConversionException( + "expression must be a string, got " + chosen.getClass().getSimpleName()); + } + return (String) chosen; // null when neither dialect present -> caller warns and skips + } + + /** + * Emptiness test used throughout the converter for optional YAML values: null, the empty string, + * an empty list/map, numeric zero, and boolean false are all treated as absent; everything else + * is present. Used for the DATABRICKS-or-ANSI expression fallthrough (so an empty or absent + * DATABRICKS expr falls through to ANSI) and for optional-field mapping (so an empty + * `comment`/`synonyms` is dropped rather than emitted as an empty value). + */ + static boolean truthy(Object v) { + if (v == null) { + return false; + } + if (v instanceof String) { + return !((String) v).isEmpty(); + } + if (v instanceof java.util.Collection) { + return !((java.util.Collection) v).isEmpty(); + } + if (v instanceof Map) { + return !((Map) v).isEmpty(); + } + if (v instanceof Number) { + return ((Number) v).doubleValue() != 0.0; + } + if (v instanceof Boolean) { + return (Boolean) v; + } + return true; + } + + static List synonymsOf(Object aiContext) { + if (aiContext instanceof Map) { + return strList(get(asMap(aiContext), "synonyms")); + } + return new ArrayList<>(); + } + + static String mergeDescription(Object description, Object aiContext) { + String desc = str(description); + if (aiContext instanceof String && !((String) aiContext).trim().isEmpty()) { + String s = (String) aiContext; + // When both are present they are joined with a newline; otherwise the + // `if description` is a truthiness test, so an empty (or null) description returns the + // ai_context alone rather than prepending a stray newline. + return (desc != null && !desc.isEmpty()) ? desc + "\n" + s : s; + } + return desc; + } + + static Map readStash(Map obj) { + for (Object extObj : asList(get(obj, "custom_extensions"))) { + Map ext = asMap(extObj); + if (VENDOR.equals(str(get(ext, "vendor_name")))) { + // A null or empty-string `data` is treated as an empty object rather than a parse error. + String data = str(get(ext, "data")); + if (data == null || data.isEmpty()) { + data = "{}"; + } + Map parsed; + try { + parsed = asMap(MAPPER.readValue(data, Object.class)); + } catch (Exception e) { + throw new ConversionException( + "DATABRICKS custom_extensions data is not valid JSON: " + e.getMessage(), e); + } + parsed.remove("_v"); + return parsed; + } + } + return new LinkedHashMap<>(); + } + + static List foreignVendorExtensions(Map obj) { + List out = new ArrayList<>(); + for (Object extObj : asList(get(obj, "custom_extensions"))) { + if (!VENDOR.equals(str(get(asMap(extObj), "vendor_name")))) { + out.add(extObj); + } + } + return out; + } + + /** Attach a DATABRICKS custom_extensions entry holding `data`; no-op when empty. */ + @SuppressWarnings("unchecked") + static void writeStash(Map obj, Map data) { + if (data.isEmpty()) { + return; + } + Map payload = new LinkedHashMap<>(); + payload.put("_v", STASH_VERSION); + payload.putAll(data); + String blob; + try { + blob = JSON_WRITER.writeValueAsString(payload); + } catch (Exception e) { + throw new ConversionException("failed to serialize stash: " + e.getMessage(), e); + } + // Jackson emits unicode escapes with uppercase hex; the stash format uses lowercase. Lowercase + // just the 4 hex digits of each real escape, preserving any escaped-backslash run in front of + // it (see UNICODE_ESCAPE_RE). + blob = UNICODE_ESCAPE_RE.matcher(blob) + .replaceAll(m -> m.group(1) + "\\\\u" + m.group(2).toLowerCase(Locale.ROOT)); + List exts = (List) obj.computeIfAbsent("custom_extensions", k -> new ArrayList<>()); + for (Object extObj : exts) { + Map ext = asMap(extObj); + if (VENDOR.equals(str(get(ext, "vendor_name")))) { + ext.put("data", blob); + return; + } + } + Map ext = new LinkedHashMap<>(); + ext.put("vendor_name", VENDOR); + ext.put("data", blob); + exts.add(ext); + } + + /** Last dotted part of a table reference: `samples.tpch.lineitem` -> `lineitem`. + * Trim the whole reference, take the final dotted + * segment, then strip any surrounding backticks (so `cat.sch.`t`` -> `t`). */ + static String lastIdentifier(Object source) { + if (source == null) { + return null; + } + String s = source.toString().trim(); + int dot = s.lastIndexOf('.'); + String last = dot >= 0 ? s.substring(dot + 1) : s; + int start = 0; + int end = last.length(); + while (start < end && last.charAt(start) == '`') { + start++; + } + while (end > start && last.charAt(end - 1) == '`') { + end--; + } + return last.substring(start, end); + } + + /** Parse YAML text into a plain value (YAML 1.2 booleans, matching the converter). */ + static Object parseYaml(String s) { + try { + return loadYaml(s); + } catch (Exception e) { + throw new ConversionException("failed to parse YAML: " + e.getMessage(), e); + } + } + + /** Serialize a value to YAML using the converter's write mapper (for tests without their + * own jackson-yaml import). */ + static String dumpYaml(Object obj) { + try { + return MAPPER.writeValueAsString(obj); + } catch (Exception e) { + throw new ConversionException("failed to serialize YAML: " + e.getMessage(), e); + } + } +} diff --git a/converters/databricks/java/src/main/java/org/apache/ossie/converter/databricks/OssieDatabricksConverter.java b/converters/databricks/java/src/main/java/org/apache/ossie/converter/databricks/OssieDatabricksConverter.java new file mode 100644 index 00000000..51e1acbf --- /dev/null +++ b/converters/databricks/java/src/main/java/org/apache/ossie/converter/databricks/OssieDatabricksConverter.java @@ -0,0 +1,179 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.ossie.converter.databricks; + +import java.io.IOException; +import java.io.PrintStream; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.List; + +/** + * Command-line entry point for the converter between Apache Ossie and Databricks Metric Views. + * + * 
{@code
+ *   ossie-databricks import  [-o ] [--source ]
+ *   ossie-databricks export   [-o ] [--name ]
+ * }
+ * + *

{@code import} converts an Apache Ossie semantic model to a Metric View; {@code export} + * converts a Metric View to an Apache Ossie model. Output goes to the {@code -o} file, or to stdout + * when omitted. Conversion notices (features dropped on import) are written to stderr. A broken + * input raises {@link OssieConverter.ConversionException}, reported as a non-zero exit. + * + *

This is a thin command-line wrapper around {@link OssieConverter}: it parses arguments, reads + * the input YAML, invokes the library, and writes the result. Programmatic callers should use + * {@link OssieConverter} directly. + */ +public final class OssieDatabricksConverter { + + private OssieDatabricksConverter() {} + + public static void main(String[] args) { + try { + run(args, System.out, System.err); + } catch (ExitException e) { + System.err.println(e.getMessage()); + System.exit(e.code); + } catch (OssieConverter.ConversionException e) { + System.err.println("Conversion failed: " + e.getMessage()); + System.exit(1); + } + } + + /** Testable core: parses args, runs the conversion, and writes output. */ + static void run(String[] args, PrintStream out, PrintStream err) { + if (args.length == 0) { + throw new ExitException(2, usage()); + } + String command = args[0]; + Args parsed = Args.parse(args); + + String input = read(parsed.inputPath); + OssieConverter.Result result; + switch (command) { + case "import": + // Apache Ossie -> Metric View. `--source` picks the fact/grain (optional). + result = OssieConverter.convertOssieToMetricView(input, parsed.option); + break; + case "export": + // Metric View -> Apache Ossie. `--name` sets the model name (optional). + result = OssieConverter.convertMetricViewToOssie(input, parsed.option); + break; + default: + throw new ExitException(2, "Unknown command '" + command + "'.\n" + usage()); + } + + write(parsed.outputPath, result.yaml, out); + List notices = result.notices; + if (!notices.isEmpty()) { + err.println("Conversion notices (" + notices.size() + "):"); + for (String notice : notices) { + err.println(" " + notice); + } + } + } + + private static String read(String path) { + try { + return Files.readString(Path.of(path), StandardCharsets.UTF_8); + } catch (IOException e) { + throw new ExitException(1, "Cannot read input file '" + path + "': " + e.getMessage()); + } + } + + private static void write(String path, String content, PrintStream out) { + if (path == null) { + out.println(content); + return; + } + try { + Files.writeString(Path.of(path), content, StandardCharsets.UTF_8); + } catch (IOException e) { + throw new ExitException(1, "Cannot write output file '" + path + "': " + e.getMessage()); + } + } + + private static String usage() { + return "Usage:\n" + + " ossie-databricks import [-o ] [--source ]\n" + + " ossie-databricks export [-o ] [--name ]"; + } + + /** Parsed command-line arguments: the input file, an optional output file, and the option. */ + private static final class Args { + final String inputPath; + final String outputPath; + final String option; + + private Args(String inputPath, String outputPath, String option) { + this.inputPath = inputPath; + this.outputPath = outputPath; + this.option = option; + } + + static Args parse(String[] args) { + String inputPath = null; + String outputPath = null; + String option = null; + for (int i = 1; i < args.length; i++) { + String arg = args[i]; + switch (arg) { + case "-o": + case "--output": + outputPath = requireValue(args, ++i, arg); + break; + case "--source": + case "--name": + option = requireValue(args, ++i, arg); + break; + default: + if (arg.startsWith("-")) { + throw new ExitException(2, "Unknown option '" + arg + "'.\n" + usage()); + } + if (inputPath != null) { + throw new ExitException(2, "Unexpected extra argument '" + arg + "'.\n" + usage()); + } + inputPath = arg; + } + } + if (inputPath == null) { + throw new ExitException(2, "Missing input file.\n" + usage()); + } + return new Args(inputPath, outputPath, option); + } + + private static String requireValue(String[] args, int index, String flag) { + if (index >= args.length) { + throw new ExitException(2, "Option '" + flag + "' requires a value.\n" + usage()); + } + return args[index]; + } + } + + /** Signals a clean CLI exit with a message and status code (kept out of the library core). */ + static final class ExitException extends RuntimeException { + final int code; + + ExitException(int code, String message) { + super(message); + this.code = code; + } + } +} diff --git a/converters/databricks/java/src/main/java/org/apache/ossie/converter/databricks/OssieToMetricView.java b/converters/databricks/java/src/main/java/org/apache/ossie/converter/databricks/OssieToMetricView.java new file mode 100644 index 00000000..c1940c50 --- /dev/null +++ b/converters/databricks/java/src/main/java/org/apache/ossie/converter/databricks/OssieToMetricView.java @@ -0,0 +1,903 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.ossie.converter.databricks; + +import static org.apache.ossie.converter.databricks.OssieConverterCommon.CARD_MANY_TO_ONE; +import static org.apache.ossie.converter.databricks.OssieConverterCommon.CARD_ONE_TO_MANY; +import static org.apache.ossie.converter.databricks.OssieConverterCommon.MAPPER; +import static org.apache.ossie.converter.databricks.OssieConverterCommon.MAX_JOIN_NODES; +import static org.apache.ossie.converter.databricks.OssieConverterCommon.MV_VERSION; +import static org.apache.ossie.converter.databricks.OssieConverterCommon.OSSIE_VERSION; +import static org.apache.ossie.converter.databricks.OssieConverterCommon.STASH_SOURCE_KEY; +import static org.apache.ossie.converter.databricks.OssieConverterCommon.SYNONYM_LIMIT; +import static org.apache.ossie.converter.databricks.OssieConverterCommon.asList; +import static org.apache.ossie.converter.databricks.OssieConverterCommon.asMap; +import static org.apache.ossie.converter.databricks.OssieConverterCommon.foreignVendorExtensions; +import static org.apache.ossie.converter.databricks.OssieConverterCommon.get; +import static org.apache.ossie.converter.databricks.OssieConverterCommon.isSimpleIdentifier; +import static org.apache.ossie.converter.databricks.OssieConverterCommon.loadYaml; +import static org.apache.ossie.converter.databricks.OssieConverterCommon.mergeDescription; +import static org.apache.ossie.converter.databricks.OssieConverterCommon.pickExpression; +import static org.apache.ossie.converter.databricks.OssieConverterCommon.readStash; +import static org.apache.ossie.converter.databricks.OssieConverterCommon.replaceOutsideLiterals; +import static org.apache.ossie.converter.databricks.OssieConverterCommon.require; +import static org.apache.ossie.converter.databricks.OssieConverterCommon.requireStr; +import static org.apache.ossie.converter.databricks.OssieConverterCommon.str; +import static org.apache.ossie.converter.databricks.OssieConverterCommon.strList; +import static org.apache.ossie.converter.databricks.OssieConverterCommon.synonymsOf; +import static org.apache.ossie.converter.databricks.OssieConverterCommon.truthy; +import static org.apache.ossie.converter.databricks.OssieConverterCommon.validateSource; + +import java.util.ArrayDeque; +import java.util.ArrayList; +import java.util.Deque; +import java.util.HashMap; +import java.util.HashSet; +import java.util.LinkedHashMap; +import java.util.LinkedHashSet; +import java.util.List; +import java.util.Locale; +import java.util.Map; +import java.util.Set; +import java.util.regex.Pattern; + +import org.apache.ossie.converter.databricks.OssieConverter.ConversionException; +import org.apache.ossie.converter.databricks.OssieConverter.Notices; +import org.apache.ossie.converter.databricks.OssieConverter.Result; + +/** + * EXPORT direction: Apache Ossie semantic model -> Databricks Metric View v1.1 YAML. This is the + * harder direction: it reassembles a relationship graph into a join tree. Shared helpers come from + * {@link OssieConverterCommon}; the public entry point is re-exported through + * {@link OssieConverter}. + */ +// Map-based YAML manipulation: casts of the parsed Object graph to Map/List are inherently +// unchecked; the asMap/asList helpers guard them, so unchecked warnings here are expected. +@SuppressWarnings("unchecked") +final class OssieToMetricView { + + private OssieToMetricView() {} + + // -- tree node ------------------------------------------------------------ + private static final class Node { + final String dataset; + final Map rel; // empty for the fact/root + final boolean parentIsFrom; + String alias; + boolean isOtm; + final List children = new ArrayList<>(); + + Node(String dataset, Map rel, boolean parentIsFrom) { + this.dataset = dataset; + this.rel = rel; + this.parentIsFrom = parentIsFrom; + } + } + + // -- public entry --------------------------------------------------------- + static Result convertOssieToMetricView(String osiYamlStr, String source) { + Notices notices = new Notices(); + Map root; + try { + root = asMap(loadYaml(osiYamlStr)); + } catch (Exception e) { + throw new ConversionException("Invalid Apache Ossie YAML: " + e.getMessage(), e); + } + if (root.isEmpty()) { + throw new ConversionException("Invalid Apache Ossie YAML: expected a mapping at the root"); + } + String version = str(get(root, "version")); + if (!OSSIE_VERSION.equals(version)) { + throw new ConversionException( + "Unsupported Apache Ossie version '" + version + "'. Supported: " + OSSIE_VERSION); + } + List models = asList(get(root, "semantic_model")); + if (models.isEmpty()) { + throw new ConversionException("'semantic_model' must be a non-empty list"); + } + if (models.size() > 1) { + notices.warn("model", "multiple semantic models found; converting only the first"); + } + Map view = convertModel(asMap(models.get(0)), source, notices); + try { + return new Result(MAPPER.writeValueAsString(view), notices.toList()); + } catch (Exception e) { + throw new ConversionException("failed to serialize Metric View YAML: " + e.getMessage(), e); + } + } + + private static Map convertModel( + Map model, String explicitSource, Notices notices) { + String name = model.containsKey("name") ? str(get(model, "name")) : ""; + List datasetList = asList(get(model, "datasets")); + if (datasetList.isEmpty()) { + throw new ConversionException("Model '" + name + "' has no datasets"); + } + Set seen = new HashSet<>(); + Map> datasets = new LinkedHashMap<>(); + for (Object dObj : datasetList) { + Map d = asMap(dObj); + String dsName = requireStr(d, "name", "Model '" + name + "': dataset"); + if (!seen.add(dsName.trim().toLowerCase(Locale.ROOT))) { + throw new ConversionException("Model '" + name + "': duplicate dataset name '" + dsName + "'"); + } + datasets.put(dsName, d); + } + List> relationships = new ArrayList<>(); + for (Object r : asList(get(model, "relationships"))) { + relationships.add(asMap(r)); + } + + Map modelStash = readStash(model); + String factHint = explicitSource != null ? explicitSource : str(get(modelStash, STASH_SOURCE_KEY)); + Object[] built = buildJoinTree(name, datasets, relationships, factHint, notices); + Node root = (Node) built[0]; + String fact = (String) built[1]; + Map counts = assignAliases(root, fact); + markOtm(name, root); + + Map factDs = datasets.get(fact); + Map view = new LinkedHashMap<>(); + view.put("version", MV_VERSION); + view.put("source", validateSource(get(factDs, "source"), fact)); + + String comment = str(get(model, "description")); + if (truthy(comment)) { + view.put("comment", comment); + } + if (modelStash.containsKey("filter")) { + view.put("filter", modelStash.get("filter")); + } + + List joins = new ArrayList<>(); + for (Node child : root.children) { + joins.add(buildJoin(child, "source", datasets, notices)); + } + if (!joins.isEmpty()) { + view.put("joins", joins); + } + + Set droppedDims = new HashSet<>(); + Set droppedMeasures = new HashSet<>(); + List> dimensions = new ArrayList<>(); + Set seenDims = new HashSet<>(); + // Dataset name -> the alias path that addresses its columns from the primary source, collected + // from the same walk the dimensions use so measures qualify identically (see qualifyMeasure). + Map datasetAliasPath = new LinkedHashMap<>(); + for (Object[] entry : nodeOrder(root)) { + Node node = (Node) entry[0]; + @SuppressWarnings("unchecked") + List joinPath = (List) entry[1]; + boolean isFact = node == root; + String prefix = counts.get(node.dataset) > 1 ? node.alias : null; + // Fixed per node, like `prefix`: hoisted so it is not rebuilt for every field. + String qualifier = String.join(".", joinPath); + if (!isFact) { + // A dataset reachable by more than one path (diamond) appears once per path; the first + // wins, matching the order dimensions are emitted in. + datasetAliasPath.putIfAbsent(node.dataset, qualifier); + } + for (Object fObj : asList(get(datasets.get(node.dataset), "fields"))) { + Map field = asMap(fObj); + String fname = requireStr(field, "name", "dataset '" + node.dataset + "': field"); + if (node.isOtm) { + notices.warn("field '" + fname + "'", + "column on a one-to-many-joined table cannot be a dimension " + + "(must resolve to one value per source row); dropped"); + droppedDims.add(fname); + continue; + } + Map dim = + convertField(field, fname, qualifier, isFact, prefix, notices); + if (dim == null) { + droppedDims.add(fname); + continue; + } + String dn = (String) dim.get("name"); + if (!seenDims.add(dn.toLowerCase(Locale.ROOT))) { + throw new ConversionException("dataset '" + node.dataset + "': dimension name '" + dn + + "' collides with another dimension/measure; Metric Views require unique " + + "dimension/measure names -- rename before use"); + } + dimensions.add(dim); + } + } + + List> measures = new ArrayList<>(); + // Depends only on the fact name, so compile it once rather than per metric. + Pattern factQualifier = Pattern.compile("\\b" + Pattern.quote(fact) + "\\."); + for (Object mObj : asList(get(model, "metrics"))) { + Map measure = + convertMetric(asMap(mObj), factQualifier, datasetAliasPath, seenDims, notices); + if (measure == null) { + droppedMeasures.add(str(get(asMap(mObj), "name"))); + continue; + } + measures.add(measure); + } + + cascadeDrop(dimensions, measures, droppedDims, droppedMeasures, notices); + + // A Metric View must define at least one dimension or measure + // (SingleSourceMetricView.validate rejects an empty `select`), so a view with neither is one + // Databricks refuses at CREATE. Fail here instead, naming the dropped columns: after + // cascadeDrop the emptiness is usually a consequence of earlier drops rather than an empty + // input, and those names are the actionable part. + if (dimensions.isEmpty() && measures.isEmpty()) { + StringBuilder msg = new StringBuilder("Model '" + name + + "' produced no dimensions or measures; a Metric View requires at least one."); + if (!droppedDims.isEmpty() || !droppedMeasures.isEmpty()) { + msg.append(" Dropped during conversion:"); + if (!droppedDims.isEmpty()) { + msg.append(" dimensions ").append(sortedNames(droppedDims)); + } + if (!droppedMeasures.isEmpty()) { + msg.append(" measures ").append(sortedNames(droppedMeasures)); + } + msg.append(" -- see the warnings for why each was dropped."); + } + throw new ConversionException(msg.toString()); + } + + if (!dimensions.isEmpty()) { + view.put("dimensions", dimensions); + } + if (!measures.isEmpty()) { + view.put("measures", measures); + } + if (modelStash.containsKey("parameters")) { + view.put("parameters", modelStash.get("parameters")); + } + if (modelStash.containsKey("materialization")) { + view.put("materialization", modelStash.get("materialization")); + } + + warnDroppedModel(model, notices); + return view; + } + + private static Object[] buildJoinTree( + String modelName, Map> datasets, + List> relationships0, String factHint, Notices notices) { + for (Map rel : relationships0) { + String scope = "Model '" + modelName + "': relationship '" + + (rel.containsKey("name") ? str(get(rel, "name")) : "") + "'"; + Object f = require(rel, "from", scope); + Object t = require(rel, "to", scope); + if (!datasets.containsKey(str(f)) || !datasets.containsKey(str(t))) { + throw new ConversionException("Model '" + modelName + "': relationship '" + + str(get(rel, "name")) + "' references an unknown dataset"); + } + } + List> relationships = new ArrayList<>(); + for (Map rel : relationships0) { + relationships.add(orientByKey(rel, datasets, notices)); + } + String fact = pickFact(modelName, datasets, relationships, factHint); + rejectDirectedCycle(modelName, datasets, relationships); + + Map> adj = new HashMap<>(); + for (String n : datasets.keySet()) { + adj.put(n, new ArrayList<>()); + } + for (Map rel : relationships) { + adj.get(str(get(rel, "from"))).add(str(get(rel, "to"))); + adj.get(str(get(rel, "to"))).add(str(get(rel, "from"))); + } + Map dist = new HashMap<>(); + dist.put(fact, 0); + Deque queue = new ArrayDeque<>(); + queue.add(fact); + while (!queue.isEmpty()) { + String cur = queue.poll(); + for (String nb : adj.get(cur)) { + if (!dist.containsKey(nb)) { + dist.put(nb, dist.get(cur) + 1); + queue.add(nb); + } + } + } + List unreachable = new ArrayList<>(); + for (String n : datasets.keySet()) { + if (!dist.containsKey(n)) { + unreachable.add(n); + } + } + if (!unreachable.isEmpty()) { + java.util.Collections.sort(unreachable); + throw new ConversionException("Model '" + modelName + "': datasets " + unreachable + + " are not reachable from fact '" + fact + "' via relationships."); + } + Map> childrenOf = new HashMap<>(); + for (String n : datasets.keySet()) { + childrenOf.put(n, new ArrayList<>()); + } + for (Map rel : relationships) { + String a = str(get(rel, "from")); + String b = str(get(rel, "to")); + if (dist.get(a).equals(dist.get(b))) { + throw new ConversionException("Model '" + modelName + "': relationship '" + + str(get(rel, "name")) + "' joins two datasets equidistant from the fact; " + + "the graph is not tree-shaped (it contains a cycle)."); + } + String parent = dist.get(a) < dist.get(b) ? a : b; + String child = dist.get(a) < dist.get(b) ? b : a; + childrenOf.get(parent).add(new Object[] {child, rel, parent.equals(str(get(rel, "from")))}); + } + int[] counter = {0}; + Node root = build(modelName, fact, new LinkedHashMap<>(), false, childrenOf, counter); + return new Object[] {root, fact}; + } + + private static Node build(String modelName, String dataset, Map rel, + boolean parentIsFrom, Map> childrenOf, int[] counter) { + counter[0]++; + if (counter[0] > MAX_JOIN_NODES) { + throw new ConversionException("Model '" + modelName + "': join graph fans out to more than " + + MAX_JOIN_NODES + " joins; check for an unintended diamond explosion."); + } + Node node = new Node(dataset, rel, parentIsFrom); + for (Object[] c : childrenOf.get(dataset)) { + @SuppressWarnings("unchecked") + Map crel = (Map) c[1]; + node.children.add(build(modelName, (String) c[0], crel, (Boolean) c[2], childrenOf, counter)); + } + return node; + } + + private static Map assignAliases(Node root, String fact) { + Map counts = new HashMap<>(); + countNode(root, counts); + Set used = new HashSet<>(); + used.add("source"); + assign(root, null, fact, counts, used); + return counts; + } + + private static void countNode(Node node, Map counts) { + counts.merge(node.dataset, 1, Integer::sum); + for (Node c : node.children) { + countNode(c, counts); + } + } + + private static void assign(Node node, String parentAlias, String fact, + Map counts, Set used) { + String alias; + if (node.dataset.equals(fact)) { + alias = "source"; + } else { + String base; + if (counts.get(node.dataset) == 1) { + base = node.dataset; + } else if (parentAlias != null && !parentAlias.equals("source")) { + base = parentAlias + "_" + node.dataset; + } else { + base = node.dataset; + } + alias = base; + int n = 2; + while (used.contains(alias)) { + alias = base + "_" + n; + n++; + } + } + node.alias = alias; + used.add(alias); + for (Node c : node.children) { + assign(c, alias, fact, counts, used); + } + } + + private static String pickFact(String modelName, Map> datasets, + List> relationships, String factHint) { + if (factHint != null) { + if (!datasets.containsKey(factHint)) { + throw new ConversionException( + "Model '" + modelName + "': requested source '" + factHint + "' is not a dataset"); + } + return factHint; + } + if (datasets.size() > 1 && relationships.isEmpty()) { + throw new ConversionException("Model '" + modelName + "': " + datasets.size() + + " datasets but no relationships; cannot determine the fact table."); + } + Map incoming = new LinkedHashMap<>(); + for (String n : datasets.keySet()) { + incoming.put(n, 0); + } + for (Map rel : relationships) { + incoming.merge(str(get(rel, "to")), 1, Integer::sum); + } + List roots = new ArrayList<>(); + for (Map.Entry e : incoming.entrySet()) { + if (e.getValue() == 0) { + roots.add(e.getKey()); + } + } + if (roots.isEmpty()) { + throw new ConversionException("Model '" + modelName + "': join graph contains a cycle " + + "(no root dataset). A Metric View requires an acyclic, tree-shaped graph."); + } + if (roots.size() > 1) { + java.util.Collections.sort(roots); + throw new ConversionException("Model '" + modelName + "': multiple candidate fact datasets " + + roots + ". Name the grain with --source."); + } + return roots.get(0); + } + + /** + * Marks each joined node with its branch's cardinality and rejects a branch that mixes the two. + * + *

A Metric View requires every join within one top-level branch to share a single cardinality: + * `Join.validateSubJoinCardinalities` seeds the expected value from the top-level join and fails + * any descendant that differs (an absent `cardinality` reads as `many_to_one`). So a mixed branch + * is rejected in *either* direction -- a many-to-one nested under one-to-many, and equally a + * one-to-many nested under many-to-one. Emitting one would produce a view Databricks refuses at + * CREATE, so reject it here with a converter-level error instead. + */ + private static void markOtm(String modelName, Node root) { + // Each top-level join starts a branch and sets that branch's expected cardinality. + for (Node top : root.children) { + boolean branchIsOtm = !top.parentIsFrom; + top.isOtm = branchIsOtm; + markOtmVisit(modelName, top, branchIsOtm); + } + } + + private static void markOtmVisit(String modelName, Node node, boolean branchIsOtm) { + for (Node child : node.children) { + boolean isOtm = !child.parentIsFrom; + if (isOtm != branchIsOtm) { + throw new ConversionException("Model '" + modelName + "': join '" + child.alias + "' is " + + cardinalityName(isOtm) + " but descends from a " + cardinalityName(branchIsOtm) + + " join; a Metric View requires every join within one top-level branch to share the " + + "same cardinality."); + } + child.isOtm = branchIsOtm; + markOtmVisit(modelName, child, branchIsOtm); + } + } + + private static String cardinalityName(boolean isOtm) { + return isOtm ? CARD_ONE_TO_MANY : CARD_MANY_TO_ONE; + } + + /** + * Rejects a directed cycle in the relationship graph. + * + *

Checked on the DIRECTED graph, and specifically for an edge back to a dataset on the current + * DFS stack: a dataset reachable by two distinct paths (a diamond, e.g. `a -> b -> d` plus + * `a -> c -> d`) is directed-acyclic and legitimately supported via the fan-out aliases, so a + * test for "reached twice" would wrongly reject it. Only a genuine directed cycle is an error. + * + *

This replaces relying on the equidistance heuristic below, which only detects a cycle whose + * closing edge happens to join two datasets at the same BFS distance from the fact: a cycle such + * as `a -> b -> c -> d -> e -> b` has no equidistant edge, so it used to expand into duplicate + * join paths (`d` emitted under both `c` and `e`) and fabricate a tree from a cyclic model. + * `pickFact` does not catch it either -- it only fails when no dataset has zero incoming edges, + * and here `a` has none. With this check, MAX_JOIN_NODES is purely a fan-out bound rather than + * the last defense against a cycle. + */ + private static void rejectDirectedCycle(String modelName, + Map> datasets, List> relationships) { + Map> out = new HashMap<>(); + for (String n : datasets.keySet()) { + out.put(n, new ArrayList<>()); + } + for (Map rel : relationships) { + out.get(str(get(rel, "from"))).add(str(get(rel, "to"))); + } + Set done = new HashSet<>(); + Set onStack = new LinkedHashSet<>(); + for (String n : datasets.keySet()) { + List cycle = findCycle(n, out, done, onStack); + if (cycle != null) { + throw new ConversionException("Model '" + modelName + "': relationships form a directed" + + " cycle " + String.join(" -> ", cycle) + + "; a Metric View join graph must be acyclic."); + } + } + } + + /** The cycle path (closing dataset repeated at the end), or null if this subtree is clean. */ + private static List findCycle( + String node, Map> out, Set done, Set onStack) { + if (done.contains(node)) { + return null; + } + if (!onStack.add(node)) { + // Back-edge: report from the first occurrence of `node` so the message shows just the cycle. + List cycle = new ArrayList<>(); + boolean seen = false; + for (String s : onStack) { + if (s.equals(node)) { + seen = true; + } + if (seen) { + cycle.add(s); + } + } + cycle.add(node); + return cycle; + } + for (String next : out.get(node)) { + List cycle = findCycle(next, out, done, onStack); + if (cycle != null) { + return cycle; + } + } + onStack.remove(node); + done.add(node); + return null; + } + + /** + * Rewrites `.` heads in a measure expression to the alias path that addresses that + * dataset's columns from the primary source (`parentJoin.nestedJoin.`). + * + *

A dataset joined directly to the primary already maps to its own alias, so those rewrites + * are no-ops; only a nested dataset actually changes. Datasets are processed longest-name-first + * so a shorter name is never rewritten inside a longer one (`nation` must not match + * `nation_x.`), and the rewrite skips string literals and comments, as the fact strip does. + */ + private static String qualifyMeasure(String expr, Map datasetAliasPath) { + List datasets = new ArrayList<>(datasetAliasPath.keySet()); + datasets.sort((a, b) -> b.length() - a.length()); + String out = expr; + for (String dataset : datasets) { + String aliasPath = datasetAliasPath.get(dataset); + if (aliasPath == null || aliasPath.isEmpty() || aliasPath.equals(dataset)) { + continue; + } + out = replaceOutsideLiterals( + out, Pattern.compile("\\b" + Pattern.quote(dataset) + "\\."), aliasPath + "."); + } + return out; + } + + /** Sorted so the error message is deterministic (the dropped-name sets are unordered). */ + private static String sortedNames(Set names) { + List sorted = new ArrayList<>(); + for (String n : names) { + if (n != null) { + sorted.add(n); + } + } + java.util.Collections.sort(sorted); + return sorted.toString(); + } + + private static List nodeOrder(Node root) { + List order = new ArrayList<>(); + nodeOrderVisit(root, new ArrayList<>(), order); + return order; + } + + private static void nodeOrderVisit(Node node, List path, List order) { + order.add(new Object[] {node, new ArrayList<>(path)}); + for (Node child : node.children) { + List childPath = new ArrayList<>(path); + childPath.add(child.alias); + nodeOrderVisit(child, childPath, order); + } + } + + private static Map buildJoin(Node node, String parentAlias, + Map> datasets, Notices notices) { + Map rel = node.rel; + String alias = node.alias; + Map join = new LinkedHashMap<>(); + join.put("name", alias); + join.put("source", validateSource(get(datasets.get(node.dataset), "source"), node.dataset)); + + Map stash = readStash(rel); + List fromCols = strList(get(rel, "from_columns")); + List toCols = strList(get(rel, "to_columns")); + validateJoinColumns(rel, fromCols, toCols); + List parentCols = node.parentIsFrom ? fromCols : toCols; + List childCols = node.parentIsFrom ? toCols : fromCols; + if (parentCols.equals(childCols)) { + join.put("using", new ArrayList<>(parentCols)); + } else { + List clauses = new ArrayList<>(); + for (int i = 0; i < parentCols.size(); i++) { + clauses.add(parentAlias + "." + parentCols.get(i) + " = " + alias + "." + childCols.get(i)); + } + join.put("on", String.join(" AND ", clauses)); + } + if (stash.containsKey("rely")) { + join.put("rely", stash.get("rely")); + } else if (node.parentIsFrom && coversUniqueKey(datasets.get(node.dataset), toCols)) { + Map rely = new LinkedHashMap<>(); + rely.put("at_most_one_match", true); + join.put("rely", rely); + } + if (stash.containsKey("cardinality")) { + join.put("cardinality", stash.get("cardinality")); + } else if (!node.parentIsFrom) { + join.put("cardinality", CARD_ONE_TO_MANY); + } + List nested = new ArrayList<>(); + for (Node c : node.children) { + nested.add(buildJoin(c, alias, datasets, notices)); + } + if (!nested.isEmpty()) { + join.put("joins", nested); + } + return join; + } + + private static boolean coversUniqueKey(Map dataset, List joinCols) { + Set cols = new HashSet<>(joinCols); + List> keys = new ArrayList<>(); + List pk = strList(get(dataset, "primary_key")); + if (!pk.isEmpty()) { + keys.add(pk); + } + for (Object k : asList(get(dataset, "unique_keys"))) { + keys.add(strList(k)); + } + for (List key : keys) { + if (!key.isEmpty() && cols.containsAll(key)) { + return true; + } + } + return false; + } + + private static Map orientByKey( + Map rel, Map> datasets, Notices notices) { + List fromCols = strList(get(rel, "from_columns")); + List toCols = strList(get(rel, "to_columns")); + if (fromCols.isEmpty() || toCols.isEmpty()) { + return rel; + } + Map toDs = datasets.get(str(get(rel, "to"))); + boolean toHasKeys = !strList(get(toDs, "primary_key")).isEmpty() + || !asList(get(toDs, "unique_keys")).isEmpty(); + boolean fromCovers = coversUniqueKey(datasets.get(str(get(rel, "from"))), fromCols); + if (fromCovers && toHasKeys && !coversUniqueKey(toDs, toCols)) { + notices.warn("relationship '" + str(get(rel, "name")) + "'", + "from/to looks mislabeled (the `from` columns are a declared key, the `to` columns " + + "are not); re-orienting so the key side is the `to`/one side"); + Map swapped = new LinkedHashMap<>(rel); + swapped.put("from", get(rel, "to")); + swapped.put("to", get(rel, "from")); + swapped.put("from_columns", toCols); + swapped.put("to_columns", fromCols); + return swapped; + } + if (fromCovers && !toHasKeys) { + notices.warn("relationship '" + str(get(rel, "name")) + "'", + "the `from` columns are a declared key but the `to` side declares none, so from/to " + + "orientation can't be verified; using it as-is -- check the join direction if " + + "the resulting cardinality looks inverted"); + } + return rel; + } + + private static void validateJoinColumns( + Map rel, List fromCols, List toCols) { + String name = str(get(rel, "name")); + if (fromCols.isEmpty() || toCols.isEmpty()) { + throw new ConversionException( + "Relationship '" + name + "': from_columns and to_columns are required"); + } + if (fromCols.size() != toCols.size()) { + throw new ConversionException("Relationship '" + name + "': from_columns (" + fromCols.size() + + ") and to_columns (" + toCols.size() + ") must have the same length"); + } + } + + private static Map convertField(Map field, String name0, + String qualifier, boolean isFact, String prefix, Notices notices) { + String scope = "field '" + name0 + "'"; + String expr = pickExpression(get(field, "expression")); + if (expr == null) { + notices.warn(scope, "no DATABRICKS/ANSI_SQL dialect; dropping field"); + return null; + } + if (!isFact) { + if (isSimpleIdentifier(expr)) { + expr = qualifier + "." + expr; + } else if (prefix != null) { + notices.warn(scope, "complex expression on a fanned-out (diamond) join cannot be " + + "unambiguously qualified; dropped"); + return null; + } else { + notices.warn(scope, "complex expression on a joined table; emitted as-is, verify qualification"); + } + } + String name = prefix != null ? prefix + "_" + name0 : name0; + Map dim = new LinkedHashMap<>(); + dim.put("name", name); + dim.put("expr", expr); + String comment = mergeDescription(get(field, "description"), get(field, "ai_context")); + if (truthy(comment)) { + dim.put("comment", comment); + } + String label = str(get(field, "label")); + if (truthy(label)) { + dim.put("display_name", label); + } + List syns = synonymsOf(get(field, "ai_context")); + if (!syns.isEmpty()) { + dim.put("synonyms", truncateSynonyms(syns, scope, notices)); + } + Map stash = readStash(field); + if (stash.containsKey("format")) { + dim.put("format", stash.get("format")); + } + warnDroppedField(field, scope, notices); + return dim; + } + + private static Map convertMetric( + Map metric, + Pattern factQualifier, + Map datasetAliasPath, + Set seenNames, + Notices notices) { + String name = requireStr(metric, "name", "metric"); + String scope = "metric '" + name + "'"; + if (!seenNames.add(name.toLowerCase(Locale.ROOT))) { + throw new ConversionException("metric '" + name + "' collides with another dimension/measure; " + + "Metric Views require unique dimension/measure names -- rename before use"); + } + String expr = pickExpression(get(metric, "expression")); + if (expr == null) { + notices.warn(scope, "no DATABRICKS/ANSI_SQL dialect; dropping metric"); + return null; + } + // Re-qualify a joined dataset's columns with the alias path that addresses them from the + // primary source. A Metric View addresses a nested join column by its full path + // (`parentJoin.nestedJoin.col`), so a bare nested alias at the head is read as struct access on + // a parameter rather than as a join column -- it fails silently. Dimensions already qualify + // this way via the joinPath handed to convertField, so this keeps the two directions + // consistent on the same input. + expr = qualifyMeasure(expr, datasetAliasPath); + // Strip a `.` qualifier so fact columns are bare in measures (the Metric View idiom). + // Only outside string literals / comments: a literal such as 'customer.us' must not be + // rewritten, or the measure's predicate changes. + expr = replaceOutsideLiterals(expr, factQualifier, ""); + Map measure = new LinkedHashMap<>(); + measure.put("name", name); + measure.put("expr", expr); + String comment = mergeDescription(get(metric, "description"), get(metric, "ai_context")); + if (truthy(comment)) { + measure.put("comment", comment); + } + List syns = synonymsOf(get(metric, "ai_context")); + if (!syns.isEmpty()) { + measure.put("synonyms", truncateSynonyms(syns, scope, notices)); + } + Map stash = readStash(metric); + if (stash.containsKey("format")) { + measure.put("format", stash.get("format")); + } + if (stash.containsKey("window")) { + measure.put("window", stash.get("window")); + } + if (stash.containsKey("partition")) { + measure.put("partition", stash.get("partition")); + } + return measure; + } + + private static String referencesDropped( + String expr, String selfName, Set droppedDims, Set droppedMeasures) { + for (String m : droppedMeasures) { + if (m != null && Pattern.compile("measure\\(\\s*" + Pattern.quote(m) + "\\s*\\)") + .matcher(expr).find()) { + return m; + } + } + for (String d : droppedDims) { + if (d != null && !d.equals(selfName) + && Pattern.compile("(?> dimensions, + List> measures, Set droppedDims, + Set droppedMeasures, Notices notices) { + boolean changed = true; + while (changed) { + changed = false; + changed |= cascadePass(dimensions, "dimension", droppedDims, droppedDims, droppedMeasures, notices); + changed |= cascadePass(measures, "measure", droppedMeasures, droppedDims, droppedMeasures, notices); + } + } + + private static boolean cascadePass(List> coll, String kind, + Set droppedSet, Set droppedDims, Set droppedMeasures, Notices notices) { + boolean changed = false; + List> survivors = new ArrayList<>(); + for (Map col : coll) { + String nm = (String) col.get("name"); + String ref = referencesDropped((String) col.get("expr"), nm, droppedDims, droppedMeasures); + if (ref != null) { + notices.warn(kind + " '" + nm + "'", + "references dropped '" + ref + "'; dropping (downstream of a dropped field/metric)"); + droppedSet.add(nm); + changed = true; + } else { + survivors.add(col); + } + } + coll.clear(); + coll.addAll(survivors); + return changed; + } + + private static List truncateSynonyms(List syns, String scope, Notices notices) { + if (syns.size() > SYNONYM_LIMIT) { + notices.warn(scope, syns.size() + " synonyms exceeds Metric View limit; keeping first " + SYNONYM_LIMIT); + return new ArrayList<>(syns.subList(0, SYNONYM_LIMIT)); + } + return syns; + } + + private static void warnDroppedModel(Map model, Notices notices) { + if (!foreignVendorExtensions(model).isEmpty()) { + notices.warn("model", "foreign-vendor custom_extensions dropped"); + } + if (truthy(get(model, "ai_context"))) { + notices.warn("model", "model-level ai_context dropped (only the description maps to the view comment)"); + } + for (Object dsObj : asList(get(model, "datasets"))) { + Map ds = asMap(dsObj); + String scope = "dataset '" + str(get(ds, "name")) + "'"; + if (!strList(get(ds, "primary_key")).isEmpty() || !asList(get(ds, "unique_keys")).isEmpty()) { + notices.warn(scope, "primary_key/unique_keys not stored as columns; used to set " + + "rely.at_most_one_match on a matching many_to_one join where applicable"); + } + if (get(ds, "ai_context") instanceof Map && !asMap(get(ds, "ai_context")).isEmpty()) { + notices.warn(scope, "dataset-level ai_context (object) dropped"); + } + if (truthy(get(ds, "description"))) { + notices.warn(scope, "dataset-level description dropped (no per-source comment field)"); + } + if (!foreignVendorExtensions(ds).isEmpty()) { + notices.warn(scope, "foreign-vendor custom_extensions dropped"); + } + } + for (Object relObj : asList(get(model, "relationships"))) { + Map rel = asMap(relObj); + if (truthy(get(rel, "ai_context"))) { + String rn = rel.containsKey("name") ? str(get(rel, "name")) : ""; + notices.warn("relationship '" + rn + "'", "relationship ai_context dropped"); + } + } + } + + private static void warnDroppedField(Map field, String scope, Notices notices) { + Object dim = get(field, "dimension"); + if (dim instanceof Map && asMap(dim).containsKey("is_time")) { + notices.warn(scope, "dimension.is_time has no Metric View counterpart; dropped"); + } + if (!foreignVendorExtensions(field).isEmpty()) { + notices.warn(scope, "foreign-vendor custom_extensions dropped"); + } + } +} diff --git a/converters/databricks/java/src/test/java/org/apache/ossie/converter/databricks/OssieConverterRoundTripSuite.java b/converters/databricks/java/src/test/java/org/apache/ossie/converter/databricks/OssieConverterRoundTripSuite.java new file mode 100644 index 00000000..c6dfdfae --- /dev/null +++ b/converters/databricks/java/src/test/java/org/apache/ossie/converter/databricks/OssieConverterRoundTripSuite.java @@ -0,0 +1,560 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.ossie.converter.databricks; + +import static org.junit.jupiter.api.Assertions.assertEquals; + +import java.util.ArrayList; +import java.util.HashMap; +import java.util.LinkedHashMap; +import java.util.LinkedHashSet; +import java.util.List; +import java.util.Map; +import java.util.Random; +import java.util.Set; +import java.util.TreeSet; + +import org.junit.jupiter.api.Test; + +/** + * Property-based round-trip tests. For any generated model in the round-trippable subset, + * converting + * one direction and back preserves content: + * + * MV -> Ossie -> MV : source, every dimension/measure name+expr+metadata, every join + * (name/source/condition/cardinality/rely) and nesting, and model filter/comment/ + * materialization. + * Ossie -> MV -> Ossie : dataset names+sources+fields, relationship from/to/columns, + * metric name+expr, and model description. + * + * Uses a seeded java.util.Random so no + * property-testing library is needed; each of NUM_SEEDS seeds is one generated model. + */ +public class OssieConverterRoundTripSuite { + + private static final int NUM_SEEDS = 300; + private static final String[] AGGS = {"SUM", "COUNT", "AVG", "MIN", "MAX"}; + + // --- Rnd: the small interface the builders depend on --------------------- + private static final class Rnd { + private final Random r; + Rnd(long seed) { + this.r = new Random(seed); + } + boolean chance(double p) { + return r.nextDouble() < p; + } + int count(int lo, int hi) { + return lo + r.nextInt(hi - lo + 1); + } + T pick(List seq) { + return seq.get(r.nextInt(seq.size())); + } + String text() { + String alnum = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789"; + int n = r.nextInt(11); + StringBuilder b = new StringBuilder(); + b.append(alnum.charAt(r.nextInt(alnum.length()))); + String alnumSpace = alnum + " "; + for (int i = 0; i < n; i++) { + b.append(alnumSpace.charAt(r.nextInt(alnumSpace.length()))); + } + String s = b.toString().trim(); + return s.isEmpty() ? "x" : s; + } + String colname() { + String lower = "abcdefghijklmnopqrstuvwxyz_"; + String rest = lower + "0123456789"; + StringBuilder b = new StringBuilder(); + b.append(lower.charAt(r.nextInt(lower.length()))); + int n = r.nextInt(8); + for (int i = 0; i < n; i++) { + b.append(rest.charAt(r.nextInt(rest.length()))); + } + return b.toString(); + } + } + + private static final class Names { + private final Map n = new HashMap<>(); + String next(String prefix) { + int i = n.getOrDefault(prefix, 0); + n.put(prefix, i + 1); + return prefix + i; + } + } + + private static String threePart(Rnd rnd) { + return rnd.colname() + "." + rnd.colname() + "." + rnd.colname(); + } + + private static void maybeMeta(Rnd rnd, Map target) { + if (rnd.chance(0.4)) { + target.put("comment", rnd.text()); + } + if (rnd.chance(0.3)) { + target.put("display_name", rnd.text()); + } + if (rnd.chance(0.3)) { + List syns = new ArrayList<>(); + int k = rnd.count(1, 3); + for (int i = 0; i < k; i++) { + syns.add(rnd.text()); + } + target.put("synonyms", syns); + } + if (rnd.chance(0.25)) { + Map fmt = new LinkedHashMap<>(); + String type = rnd.pick(List.of("number", "currency", "date")); + fmt.put("type", type); + if (type.equals("currency")) { + fmt.put("currency_code", "USD"); + } + target.put("format", fmt); + } + } + + // --- Metric View builder (for MV -> Ossie -> MV) ------------------------- + private static Object[] buildJoin( + Rnd rnd, Names names, String parentAlias, int depth, List ancestorPath) { + String name = names.next("j"); + List path = new ArrayList<>(ancestorPath); + path.add(name); + String qual = String.join(".", path); + Map join = new LinkedHashMap<>(); + join.put("name", name); + join.put("source", threePart(rnd)); + if (rnd.chance(0.5)) { + int ncols = rnd.count(1, 2); + List using = new ArrayList<>(); + for (int i = 0; i < ncols; i++) { + using.add("u" + i + "_" + rnd.colname()); + } + join.put("using", using); + } else { + int ncols = rnd.count(1, 2); + List clauses = new ArrayList<>(); + for (int i = 0; i < ncols; i++) { + String pc = "fk" + i + "_" + rnd.colname(); + String cc = "pk" + i + "_" + rnd.colname(); + clauses.add(parentAlias + "." + pc + " = " + name + "." + cc); + } + join.put("on", String.join(" AND ", clauses)); + } + if (rnd.chance(0.4)) { + join.put("cardinality", "many_to_one"); + } + if (rnd.chance(0.3)) { + Map rely = new LinkedHashMap<>(); + rely.put("at_most_one_match", true); + join.put("rely", rely); + } + List> dims = new ArrayList<>(); + int nd = rnd.count(0, 2); + for (int i = 0; i < nd; i++) { + String col = rnd.colname(); + String expr = rnd.chance(0.7) ? qual + "." + col + : qual + "." + col + " + " + qual + "." + rnd.colname(); + Map dim = new LinkedHashMap<>(); + dim.put("name", names.next("c")); + dim.put("expr", expr); + maybeMeta(rnd, dim); + dims.add(dim); + } + if (depth < 2 && rnd.chance(0.35)) { + Object[] childResult = buildJoin(rnd, names, name, depth + 1, path); + List childJoins = new ArrayList<>(); + childJoins.add(childResult[0]); + join.put("joins", childJoins); + @SuppressWarnings("unchecked") + List> childDims = (List>) childResult[1]; + dims.addAll(childDims); + } + return new Object[] {join, dims}; + } + + private static Map buildMetricView(Rnd rnd) { + Names names = new Names(); + Map mv = new LinkedHashMap<>(); + mv.put("version", OssieConverter.MV_VERSION); + mv.put("source", threePart(rnd)); + if (rnd.chance(0.4)) { + mv.put("comment", rnd.text()); + } + if (rnd.chance(0.3)) { + mv.put("filter", rnd.colname() + " > 0"); + } + List> fields = new ArrayList<>(); + List joins = new ArrayList<>(); + int nsrc = rnd.count(0, 3); + for (int i = 0; i < nsrc; i++) { + String col = rnd.colname(); + String expr = rnd.chance(0.7) ? col : "UPPER(" + col + ")"; + Map dim = new LinkedHashMap<>(); + dim.put("name", names.next("c")); + dim.put("expr", expr); + maybeMeta(rnd, dim); + fields.add(dim); + } + int njoins = rnd.count(0, 2); + for (int i = 0; i < njoins; i++) { + Object[] jr = buildJoin(rnd, names, "source", 0, new ArrayList<>()); + joins.add(jr[0]); + @SuppressWarnings("unchecked") + List> jdims = (List>) jr[1]; + fields.addAll(jdims); + } + List> measures = new ArrayList<>(); + int nmeas = rnd.count(0, 2); + for (int i = 0; i < nmeas; i++) { + Map m = new LinkedHashMap<>(); + m.put("name", names.next("c")); + m.put("expr", rnd.pick(List.of(AGGS)) + "(" + rnd.colname() + ")"); + if (rnd.chance(0.4)) { + m.put("comment", rnd.text()); + } + if (rnd.chance(0.3)) { + List syns = new ArrayList<>(); + int k = rnd.count(1, 3); + for (int j = 0; j < k; j++) { + syns.add(rnd.text()); + } + m.put("synonyms", syns); + } + if (rnd.chance(0.3)) { + Map w = new LinkedHashMap<>(); + w.put("order", rnd.colname()); + w.put("range", "trailing 7 day"); + List window = new ArrayList<>(); + window.add(w); + m.put("window", window); + } + measures.add(m); + } + if (!joins.isEmpty()) { + mv.put("joins", joins); + } + // A Metric View requires at least one dimension or measure, so a model with neither is outside + // the round-trippable subset (the converter rejects it, as Databricks would). Both counts can + // independently come out zero, so add one dimension when that happens. + if (fields.isEmpty() && measures.isEmpty()) { + Map dim = new LinkedHashMap<>(); + dim.put("name", names.next("c")); + dim.put("expr", rnd.colname()); + fields.add(dim); + } + if (!fields.isEmpty()) { + mv.put("fields", fields); + } + if (!measures.isEmpty()) { + mv.put("measures", measures); + } + if (rnd.chance(0.2)) { + Map mat = new LinkedHashMap<>(); + mat.put("schedule", "every 6 hours"); + mat.put("mode", rnd.pick(List.of("relaxed", "strict"))); + mv.put("materialization", mat); + } + return mv; + } + + // --- Ossie builder (for Ossie -> MV -> Ossie) ---------------------------- + private static Map ossieField(String name, String expr) { + Map dialect = new LinkedHashMap<>(); + dialect.put("dialect", "DATABRICKS"); + dialect.put("expression", expr); + List dialects = new ArrayList<>(); + dialects.add(dialect); + Map expression = new LinkedHashMap<>(); + expression.put("dialects", dialects); + Map field = new LinkedHashMap<>(); + field.put("name", name); + field.put("expression", expression); + return field; + } + + private static Map buildOssie(Rnd rnd) { + Names names = new Names(); + String fact = "fact"; + List> datasets = new ArrayList<>(); + Map factDs = new LinkedHashMap<>(); + factDs.put("name", fact); + factDs.put("source", "c.s." + fact); + datasets.add(factDs); + List> relationships = new ArrayList<>(); + + int nDims = rnd.count(0, 3); + List reachable = new ArrayList<>(); + reachable.add(fact); + for (int i = 0; i < nDims; i++) { + String dname = names.next("dim"); + String parent = rnd.pick(reachable); + Map ds = new LinkedHashMap<>(); + ds.put("name", dname); + ds.put("source", "c.s." + rnd.colname() + i); + datasets.add(ds); + reachable.add(dname); + Map rel = new LinkedHashMap<>(); + rel.put("name", names.next("r")); + rel.put("from", parent); + rel.put("to", dname); + if (rnd.chance(0.5)) { + List cols = new ArrayList<>(); + int k = rnd.count(1, 2); + for (int j = 0; j < k; j++) { + cols.add(rnd.colname()); + } + rel.put("from_columns", new ArrayList<>(cols)); + rel.put("to_columns", new ArrayList<>(cols)); + } else { + int n = rnd.count(1, 2); + List fcols = new ArrayList<>(); + List tcols = new ArrayList<>(); + for (int j = 0; j < n; j++) { + fcols.add("fk" + j + "_" + rnd.colname()); + tcols.add("pk" + j + "_" + rnd.colname()); + } + rel.put("from_columns", fcols); + rel.put("to_columns", tcols); + } + relationships.add(rel); + } + for (Map ds : datasets) { + List flds = new ArrayList<>(); + int nf = rnd.count(0, 3); + for (int j = 0; j < nf; j++) { + flds.add(ossieField(names.next("c"), rnd.colname())); + } + if (!flds.isEmpty()) { + ds.put("fields", flds); + } + } + List metrics = new ArrayList<>(); + int nm = rnd.count(0, 2); + for (int i = 0; i < nm; i++) { + metrics.add(ossieField(names.next("c"), rnd.pick(List.of(AGGS)) + "(" + rnd.colname() + ")")); + } + // The converted Metric View needs at least one dimension or measure, so a model whose datasets + // have no fields and which declares no metrics is outside the round-trippable subset. Give the + // first dataset a field when nothing else would produce a column. + boolean anyField = false; + for (Map ds : datasets) { + if (!asList(ds.get("fields")).isEmpty()) { + anyField = true; + break; + } + } + if (!anyField && metrics.isEmpty()) { + List flds = new ArrayList<>(); + flds.add(ossieField(names.next("c"), rnd.colname())); + datasets.get(0).put("fields", flds); + } + Map model = new LinkedHashMap<>(); + model.put("name", names.next("m")); + if (rnd.chance(0.4)) { + model.put("description", rnd.text()); + } + model.put("datasets", datasets); + if (!relationships.isEmpty()) { + model.put("relationships", relationships); + } + if (!metrics.isEmpty()) { + model.put("metrics", metrics); + } + Map out = new LinkedHashMap<>(); + out.put("version", OssieConverter.OSSIE_VERSION); + List models = new ArrayList<>(); + models.add(model); + out.put("semantic_model", models); + return out; + } + + // --- Round-trip assertions ----------------------------------------------- + + @SuppressWarnings("unchecked") + private static Map asMap(Object x) { + return x instanceof Map ? (Map) x : new LinkedHashMap<>(); + } + + @SuppressWarnings("unchecked") + private static List asList(Object x) { + return x instanceof List ? (List) x : new ArrayList<>(); + } + + private static String dumpYaml(Map obj) { + return OssieConverter.dumpYaml(obj); + } + + private static String condCanon(Map join) { + List using = asList(join.get("using")); + if (!using.isEmpty()) { + Set sorted = new TreeSet<>(); + for (Object u : using) { + sorted.add(u.toString()); + } + return "using:" + sorted; + } + Object on = join.get("on"); + if (on == null) { + return "none"; + } + Set pairs = new TreeSet<>(); + for (String clause : on.toString().split("(?i)\\s+AND\\s+")) { + String[] lr = clause.split("=", 2); + pairs.add(lr[0].trim() + "=" + lr[1].trim()); + } + return "on:" + pairs; + } + + private static void flattenJoins( + List joins, String parent, Map acc, Set edges) { + for (Object jObj : joins) { + Map j = asMap(jObj); + String name = (String) j.get("name"); + acc.put(name, j.get("source") + "|" + condCanon(j) + "|" + j.get("cardinality") + + "|" + j.get("rely")); + edges.add(parent + "->" + name); + flattenJoins(asList(j.get("joins")), name, acc, edges); + } + } + + private static List dims(Map mv) { + List d = asList(mv.get("dimensions")); + return !d.isEmpty() ? d : asList(mv.get("fields")); + } + + private static String dimNorm(Map d) { + return d.get("expr") + "|" + d.get("comment") + "|" + d.get("display_name") + + "|" + d.get("synonyms") + "|" + d.get("format"); + } + + private static String measNorm(Map m) { + return m.get("expr") + "|" + m.get("comment") + "|" + m.get("synonyms") + + "|" + m.get("format") + "|" + m.get("window"); + } + + private static Map byName(List items, boolean measure) { + Map out = new LinkedHashMap<>(); + for (Object o : items) { + Map m = asMap(o); + out.put((String) m.get("name"), measure ? measNorm(m) : dimNorm(m)); + } + return out; + } + + private void assertMvRoundTrip(Map mv, long seed) { + String ossieYaml = OssieConverter.convertMetricViewToOssie(dumpYaml(mv), null).yaml; + Map mv2 = + asMap(OssieConverter.parseYaml(OssieConverter.convertOssieToMetricView(ossieYaml, null).yaml)); + + String ctx = " (seed " + seed + ")"; + assertEquals(mv.get("source"), mv2.get("source"), "source" + ctx); + assertEquals(mv.get("comment"), mv2.get("comment"), "comment" + ctx); + assertEquals(mv.get("filter"), mv2.get("filter"), "filter" + ctx); + assertEquals(mv.get("materialization"), mv2.get("materialization"), "materialization" + ctx); + assertEquals(byName(dims(mv), false), byName(dims(mv2), false), "fields" + ctx); + assertEquals(byName(asList(mv.get("measures")), true), + byName(asList(mv2.get("measures")), true), "measures" + ctx); + + Map a1 = new LinkedHashMap<>(); + Set e1 = new LinkedHashSet<>(); + flattenJoins(asList(mv.get("joins")), "source", a1, e1); + Map a2 = new LinkedHashMap<>(); + Set e2 = new LinkedHashSet<>(); + flattenJoins(asList(mv2.get("joins")), "source", a2, e2); + assertEquals(a1, a2, "joins" + ctx); + assertEquals(e1, e2, "join nesting" + ctx); + } + + private static String exprOf(Map obj) { + for (Object dObj : asList(asMap(obj.get("expression")).get("dialects"))) { + Map d = asMap(dObj); + if ("DATABRICKS".equals(d.get("dialect"))) { + return (String) d.get("expression"); + } + } + return null; + } + + private static Map fieldsMap(Map ds) { + Map out = new LinkedHashMap<>(); + for (Object fObj : asList(ds.get("fields"))) { + Map f = asMap(fObj); + out.put((String) f.get("name"), exprOf(f)); + } + return out; + } + + private static Set relSet(Map model) { + Set out = new LinkedHashSet<>(); + for (Object rObj : asList(model.get("relationships"))) { + Map r = asMap(rObj); + out.add(r.get("from") + "->" + r.get("to") + "|" + asList(r.get("from_columns")) + + "|" + asList(r.get("to_columns"))); + } + return out; + } + + private void assertOssieRoundTrip(Map ossie, long seed) { + String mvYaml = OssieConverter.convertOssieToMetricView(dumpYaml(ossie), null).yaml; + Map ossie2 = + asMap(OssieConverter.parseYaml(OssieConverter.convertMetricViewToOssie(mvYaml, null).yaml)); + + String ctx = " (seed " + seed + ")"; + Map m1 = asMap(asList(ossie.get("semantic_model")).get(0)); + Map m2 = asMap(asList(ossie2.get("semantic_model")).get(0)); + + Map d1 = new LinkedHashMap<>(); + for (Object dsObj : asList(m1.get("datasets"))) { + Map ds = asMap(dsObj); + d1.put((String) ds.get("name"), ds.get("source") + "|" + fieldsMap(ds)); + } + Map d2 = new LinkedHashMap<>(); + for (Object dsObj : asList(m2.get("datasets"))) { + Map ds = asMap(dsObj); + d2.put((String) ds.get("name"), ds.get("source") + "|" + fieldsMap(ds)); + } + assertEquals(d1, d2, "datasets" + ctx); + assertEquals(relSet(m1), relSet(m2), "relationships" + ctx); + + Map met1 = new LinkedHashMap<>(); + for (Object x : asList(m1.get("metrics"))) { + met1.put((String) asMap(x).get("name"), exprOf(asMap(x))); + } + Map met2 = new LinkedHashMap<>(); + for (Object x : asList(m2.get("metrics"))) { + met2.put((String) asMap(x).get("name"), exprOf(asMap(x))); + } + assertEquals(met1, met2, "metrics" + ctx); + assertEquals(m1.get("description"), m2.get("description"), "description" + ctx); + } + + @Test + public void metricViewRoundTripAcrossSeeds() { + for (long seed = 0; seed < NUM_SEEDS; seed++) { + assertMvRoundTrip(buildMetricView(new Rnd(seed)), seed); + } + } + + @Test + public void ossieRoundTripAcrossSeeds() { + for (long seed = 0; seed < NUM_SEEDS; seed++) { + assertOssieRoundTrip(buildOssie(new Rnd(seed)), seed); + } + } +} diff --git a/converters/databricks/java/src/test/java/org/apache/ossie/converter/databricks/OssieConverterSuite.java b/converters/databricks/java/src/test/java/org/apache/ossie/converter/databricks/OssieConverterSuite.java new file mode 100644 index 00000000..349ce92d --- /dev/null +++ b/converters/databricks/java/src/test/java/org/apache/ossie/converter/databricks/OssieConverterSuite.java @@ -0,0 +1,774 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.ossie.converter.databricks; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.junit.jupiter.api.Assertions.assertTrue; + +import java.io.InputStream; +import java.nio.charset.StandardCharsets; +import java.util.List; +import java.util.Map; + +import org.junit.jupiter.api.Test; + +/** + * Example-based tests for the Apache Ossie <-> Metric View converter, plus the fixture + * comparisons that pin the expected output of both directions. + */ +public class OssieConverterSuite { + + private static Object export(String osi, String source) { + return OssieConverter.parseYaml( + OssieConverter.convertOssieToMetricView(osi, source).yaml); + } + + @Test + public void fixtureAStarSchemaExportsToExpectedMetricView() { + String osi = + "version: \"0.2.0.dev0\"\n" + + "semantic_model:\n" + + " - name: sales\n" + + " description: Sales orders with customer attributes\n" + + " datasets:\n" + + " - name: orders\n" + + " source: samples.tpch.orders\n" + + " primary_key: [o_orderkey]\n" + + " description: One row per order\n" + + " fields:\n" + + " - name: o_orderkey\n" + + " expression:\n" + + " dialects:\n" + + " - dialect: DATABRICKS\n" + + " expression: o_orderkey\n" + + " description: Order identifier\n" + + " - name: o_orderdate\n" + + " expression:\n" + + " dialects:\n" + + " - dialect: DATABRICKS\n" + + " expression: o_orderdate\n" + + " label: Order Date\n" + + " ai_context:\n" + + " synonyms: [order date, date]\n" + + " - name: customer\n" + + " source: samples.tpch.customer\n" + + " primary_key: [c_custkey]\n" + + " fields:\n" + + " - name: c_name\n" + + " expression:\n" + + " dialects:\n" + + " - dialect: DATABRICKS\n" + + " expression: c_name\n" + + " description: Customer name\n" + + " relationships:\n" + + " - name: orders_to_customer\n" + + " from: orders\n" + + " to: customer\n" + + " from_columns: [o_custkey]\n" + + " to_columns: [c_custkey]\n" + + " metrics:\n" + + " - name: total_revenue\n" + + " expression:\n" + + " dialects:\n" + + " - dialect: DATABRICKS\n" + + " expression: SUM(o_totalprice)\n" + + " description: Total order revenue\n" + + " ai_context:\n" + + " synonyms: [revenue, total revenue, sales]\n" + + " - name: order_count\n" + + " expression:\n" + + " dialects:\n" + + " - dialect: DATABRICKS\n" + + " expression: COUNT(*)\n" + + " description: Number of orders\n"; + + String expected = + "version: '1.1'\n" + + "source: samples.tpch.orders\n" + + "comment: Sales orders with customer attributes\n" + + "joins:\n" + + "- name: customer\n" + + " source: samples.tpch.customer\n" + + " on: source.o_custkey = customer.c_custkey\n" + + " rely:\n" + + " at_most_one_match: true\n" + + "dimensions:\n" + + "- name: o_orderkey\n" + + " expr: o_orderkey\n" + + " comment: Order identifier\n" + + "- name: o_orderdate\n" + + " expr: o_orderdate\n" + + " display_name: Order Date\n" + + " synonyms:\n" + + " - order date\n" + + " - date\n" + + "- name: c_name\n" + + " expr: customer.c_name\n" + + " comment: Customer name\n" + + "measures:\n" + + "- name: total_revenue\n" + + " expr: SUM(o_totalprice)\n" + + " comment: Total order revenue\n" + + " synonyms:\n" + + " - revenue\n" + + " - total revenue\n" + + " - sales\n" + + "- name: order_count\n" + + " expr: COUNT(*)\n" + + " comment: Number of orders\n"; + + assertEquals(OssieConverter.parseYaml(expected), export(osi, null)); + } + + @Test + public void unsupportedVersionIsRejected() { + OssieConverter.ConversionException e = assertThrows(OssieConverter.ConversionException.class, + () -> OssieConverter.convertOssieToMetricView("version: '9.9'\nsemantic_model: []\n", null)); + assertTrue(e.getMessage().contains("Unsupported Apache Ossie version")); + } + + @Test + public void multipleCandidateFactsWithoutSourceIsRejected() { + String osi = + "version: 0.2.0.dev0\n" + + "semantic_model:\n" + + "- name: m\n" + + " datasets:\n" + + " - {name: orders, source: c.s.orders}\n" + + " - {name: returns, source: c.s.returns}\n" + + " - {name: customer, source: c.s.customer, primary_key: [c_custkey]}\n" + + " relationships:\n" + + " - {name: oc, from: orders, to: customer, from_columns: [o_custkey], to_columns: [c_custkey]}\n" + + " - {name: rc, from: returns, to: customer, from_columns: [re_custkey], to_columns: [c_custkey]}\n"; + OssieConverter.ConversionException e = assertThrows(OssieConverter.ConversionException.class, + () -> OssieConverter.convertOssieToMetricView(osi, null)); + assertTrue(e.getMessage().contains("multiple candidate fact datasets")); + } + + @Test + @SuppressWarnings("unchecked") + public void oneToManyEmitsCardinality() { + String osi = + "version: 0.2.0.dev0\n" + + "semantic_model:\n" + + "- name: m\n" + + " datasets:\n" + + " - name: orders\n" + + " source: c.s.orders\n" + + " primary_key: [o_orderkey]\n" + + " fields:\n" + + " - {name: o_orderstatus, expression: {dialects: [{dialect: DATABRICKS, expression: o_orderstatus}]}}\n" + + " - name: lineitem\n" + + " source: c.s.lineitem\n" + + " relationships:\n" + + " - {name: lio, from: lineitem, to: orders, from_columns: [l_orderkey], to_columns: [o_orderkey]}\n" + + " metrics:\n" + + " - {name: qty, expression: {dialects: [{dialect: DATABRICKS, expression: SUM(lineitem.l_quantity)}]}}\n"; + Map view = (Map) export(osi, "orders"); + List joins = (List) view.get("joins"); + Map join = (Map) joins.get(0); + assertEquals("one_to_many", join.get("cardinality")); + } + + @Test + public void oneToManyNestedUnderManyToOneIsRejected() { + // A Metric View requires one cardinality per top-level branch, so a one-to-many join nested + // under a many-to-one parent is rejected by Databricks just as the reverse nesting is. Fact + // `d`; `f -> d` points at d (many-to-one from d's perspective), and `g -> f` points away from + // f (one-to-many), which would nest one_to_many inside the many-to-one branch. + String osi = + "version: 0.2.0.dev0\n" + + "semantic_model:\n" + + "- name: m\n" + + " datasets:\n" + + " - name: d\n" + + " source: c.s.d\n" + + " fields:\n" + + " - {name: dcol, expression: {dialects: [{dialect: DATABRICKS, expression: dcol}]}}\n" + + " - name: f\n" + + " source: c.s.f\n" + + " primary_key: [fk]\n" + + " - name: g\n" + + " source: c.s.g\n" + + " relationships:\n" + + " - {name: df, from: d, to: f, from_columns: [fk], to_columns: [fk]}\n" + + " - {name: gf, from: g, to: f, from_columns: [fk], to_columns: [fk]}\n" + + " metrics:\n" + + " - {name: c, expression: {dialects: [{dialect: DATABRICKS, expression: COUNT(1)}]}}\n"; + OssieConverter.ConversionException e = assertThrows(OssieConverter.ConversionException.class, + () -> OssieConverter.convertOssieToMetricView(osi, "d")); + assertTrue(e.getMessage().contains("share the same cardinality"), + "expected a mixed-cardinality rejection, got: " + e.getMessage()); + } + + @Test + public void directedCycleWithNoEquidistantEdgeIsRejected() { + // a -> b -> c -> d -> e -> b with fact `a`: every edge spans adjacent BFS levels, so the + // equidistance heuristic sees nothing, and `a` has zero incoming edges so pickFact finds a + // root. Without a real acyclicity check this expanded into duplicate join paths (`d` under + // both `c` and `e`), fabricating a tree from a cyclic model. + String osi = + "version: 0.2.0.dev0\n" + + "semantic_model:\n" + + "- name: m\n" + + " datasets:\n" + + " - name: a\n" + + " source: c.s.a\n" + + " fields:\n" + + " - {name: acol, expression: {dialects: [{dialect: DATABRICKS, expression: acol}]}}\n" + + " - name: b\n" + + " source: c.s.b\n" + + " - name: c\n" + + " source: c.s.c\n" + + " - name: d\n" + + " source: c.s.d\n" + + " - name: e\n" + + " source: c.s.e\n" + + " relationships:\n" + + " - {name: ab, from: a, to: b, from_columns: [k], to_columns: [k]}\n" + + " - {name: bc, from: b, to: c, from_columns: [k], to_columns: [k]}\n" + + " - {name: cd, from: c, to: d, from_columns: [k], to_columns: [k]}\n" + + " - {name: de, from: d, to: e, from_columns: [k], to_columns: [k]}\n" + + " - {name: eb, from: e, to: b, from_columns: [k], to_columns: [k]}\n"; + OssieConverter.ConversionException ex = assertThrows(OssieConverter.ConversionException.class, + () -> OssieConverter.convertOssieToMetricView(osi, "a")); + assertTrue(ex.getMessage().contains("directed cycle"), + "expected a directed-cycle rejection, got: " + ex.getMessage()); + } + + @Test + @SuppressWarnings("unchecked") + public void diamondIsStillAcceptedByTheCycleCheck() { + // A diamond (`a -> b -> d` plus `a -> c -> d`) is an UNDIRECTED cycle but directed-acyclic, and + // is supported via the fan-out aliases. The cycle check must not reject it. + String osi = + "version: 0.2.0.dev0\n" + + "semantic_model:\n" + + "- name: m\n" + + " datasets:\n" + + " - name: a\n" + + " source: c.s.a\n" + + " - name: b\n" + + " source: c.s.b\n" + + " - name: c\n" + + " source: c.s.c\n" + + " - name: d\n" + + " source: c.s.d\n" + + " fields:\n" + + " - {name: dcol, expression: {dialects: [{dialect: DATABRICKS, expression: dcol}]}}\n" + + " relationships:\n" + + " - {name: ab, from: a, to: b, from_columns: [k], to_columns: [k]}\n" + + " - {name: ac, from: a, to: c, from_columns: [k], to_columns: [k]}\n" + + " - {name: bd, from: b, to: d, from_columns: [k], to_columns: [k]}\n" + + " - {name: cd, from: c, to: d, from_columns: [k], to_columns: [k]}\n"; + Map view = (Map) export(osi, "a"); + assertEquals("c.s.a", view.get("source")); + // `d` is reached by two paths, so its column is emitted once per fan-out alias. + List dims = (List) view.get("dimensions"); + assertEquals(2, dims.size(), "diamond should fan out to one dimension per path, got: " + dims); + } + + @Test + @SuppressWarnings("unchecked") + public void nestedJoinColumnInMeasureGetsFullAliasPath() { + // orders -> customer -> nation: `nation` nests under `customer`, so its columns are addressed + // as `customer.nation.col`. A bare `nation.` head would be read as struct access on a + // parameter, so the measure must be re-qualified -- and identically to the dimension path, + // which already emits `customer.nation.population` for the same column. + String osi = + "version: 0.2.0.dev0\n" + + "semantic_model:\n" + + "- name: m\n" + + " datasets:\n" + + " - name: orders\n" + + " source: c.s.orders\n" + + " - name: customer\n" + + " source: c.s.customer\n" + + " primary_key: [c_custkey]\n" + + " - name: nation\n" + + " source: c.s.nation\n" + + " primary_key: [n_nationkey]\n" + + " fields:\n" + + " - {name: population, expression: {dialects: [{dialect: DATABRICKS, expression: population}]}}\n" + + " relationships:\n" + + " - {name: oc, from: orders, to: customer, from_columns: [c_custkey], to_columns: [c_custkey]}\n" + + " - {name: cn, from: customer, to: nation, from_columns: [n_nationkey], to_columns: [n_nationkey]}\n" + + " metrics:\n" + + " - {name: pop, expression: {dialects: [{dialect: DATABRICKS, expression: SUM(nation.population)}]}}\n"; + Map view = (Map) export(osi, "orders"); + + List dims = (List) view.get("dimensions"); + Map dim = (Map) dims.get(0); + assertEquals("customer.nation.population", dim.get("expr"), + "dimension path should qualify with the full alias path"); + + List measures = (List) view.get("measures"); + Map measure = (Map) measures.get(0); + assertEquals("SUM(customer.nation.population)", measure.get("expr"), + "measure must use the same full alias path as the dimension, not a bare nested alias"); + } + + @Test + public void modelWithNoFieldsOrMetricsIsRejected() { + // A Metric View requires at least one dimension or measure, so emitting a version+source-only + // view would just fail at CREATE. Fail at conversion time instead. + String osi = + "version: 0.2.0.dev0\n" + + "semantic_model:\n" + + "- name: m\n" + + " datasets:\n" + + " - name: d\n" + + " source: c.s.d\n"; + OssieConverter.ConversionException e = assertThrows(OssieConverter.ConversionException.class, + () -> OssieConverter.convertOssieToMetricView(osi, null)); + assertTrue(e.getMessage().contains("no dimensions or measures"), + "expected an empty-view rejection, got: " + e.getMessage()); + } + + @Test + public void emptyAfterDropsNamesTheDroppedColumns() { + // Here the input is non-empty but everything drops: the only metric has no DATABRICKS/ANSI_SQL + // dialect. The error must name the dropped column so the cause is actionable, since after the + // cascade the emptiness is a consequence of the drop rather than an empty input. + String osi = + "version: 0.2.0.dev0\n" + + "semantic_model:\n" + + "- name: m\n" + + " datasets:\n" + + " - name: d\n" + + " source: c.s.d\n" + + " metrics:\n" + + " - {name: only_metric, expression: {dialects: [{dialect: SNOWFLAKE, expression: SUM(x)}]}}\n"; + OssieConverter.ConversionException e = assertThrows(OssieConverter.ConversionException.class, + () -> OssieConverter.convertOssieToMetricView(osi, null)); + assertTrue(e.getMessage().contains("no dimensions or measures"), + "expected an empty-view rejection, got: " + e.getMessage()); + assertTrue(e.getMessage().contains("only_metric"), + "the message must name the dropped column, got: " + e.getMessage()); + } + + @Test + public void duplicateDimensionNameIsRejected() { + String osi = + "version: 0.2.0.dev0\n" + + "semantic_model:\n" + + "- name: m\n" + + " datasets:\n" + + " - name: orders\n" + + " source: c.s.orders\n" + + " fields:\n" + + " - {name: id, expression: {dialects: [{dialect: DATABRICKS, expression: id}]}}\n" + + " - name: customer\n" + + " source: c.s.customer\n" + + " fields:\n" + + " - {name: id, expression: {dialects: [{dialect: DATABRICKS, expression: id}]}}\n" + + " relationships:\n" + + " - {name: r, from: orders, to: customer, from_columns: [cid], to_columns: [id]}\n"; + OssieConverter.ConversionException e = assertThrows(OssieConverter.ConversionException.class, + () -> OssieConverter.convertOssieToMetricView(osi, null)); + assertTrue(e.getMessage().contains("collides")); + } + + @Test + public void foreignVendorExtensionDroppedWithNotice() { + String osi = + "version: 0.2.0.dev0\n" + + "semantic_model:\n" + + "- name: m\n" + + " custom_extensions:\n" + + " - {vendor_name: SNOWFLAKE, data: '{}'}\n" + + " datasets:\n" + + " - name: orders\n" + + " source: c.s.orders\n" + + " fields:\n" + + " - {name: s, expression: {dialects: [{dialect: DATABRICKS, expression: s}]}}\n" + + " metrics:\n" + + " - {name: n, expression: {dialects: [{dialect: DATABRICKS, expression: COUNT(*)}]}}\n"; + OssieConverter.Result r = OssieConverter.convertOssieToMetricView(osi, null); + assertTrue(r.notices.stream().anyMatch(m -> m.contains("foreign-vendor custom_extensions dropped"))); + } + + // -- import direction (Metric View -> Apache Ossie) ----------------------- + + private static Object importMv(String mv) { + return OssieConverter.parseYaml( + OssieConverter.convertMetricViewToOssie(mv, null).yaml); + } + + @Test + @SuppressWarnings("unchecked") + public void importDecomposesJoinIntoRelationship() { + String mv = + "version: '1.1'\n" + + "source: c.s.orders\n" + + "joins:\n" + + "- name: customer\n" + + " source: c.s.customer\n" + + " on: source.o_custkey = customer.c_custkey\n" + + " rely: {at_most_one_match: true}\n" + + "dimensions:\n" + + "- {name: o_status, expr: o_orderstatus}\n" + + "- {name: c_name, expr: customer.c_name}\n" + + "measures:\n" + + "- {name: revenue, expr: SUM(o_totalprice)}\n"; + Map out = (Map) importMv(mv); + List models = (List) out.get("semantic_model"); + Map model = (Map) models.get(0); + List rels = (List) model.get("relationships"); + Map rel = (Map) rels.get(0); + assertEquals("orders", rel.get("from")); + assertEquals("customer", rel.get("to")); + assertEquals(List.of("o_custkey"), rel.get("from_columns")); + assertEquals(List.of("c_custkey"), rel.get("to_columns")); + } + + @Test + public void importRejectsNonEquiJoin() { + String mv = + "version: '1.1'\n" + + "source: c.s.orders\n" + + "joins:\n" + + "- name: customer\n" + + " source: c.s.customer\n" + + " on: source.o_custkey >= customer.c_custkey\n"; + OssieConverter.ConversionException e = assertThrows(OssieConverter.ConversionException.class, + () -> OssieConverter.convertMetricViewToOssie(mv, null)); + assertTrue(e.getMessage().contains("non-equi or unsupported")); + } + + @Test + public void importRejectsCrossJoin() { + String mv = + "version: '1.1'\n" + + "source: c.s.orders\n" + + "joins:\n" + + "- name: customer\n" + + " source: c.s.customer\n"; + OssieConverter.ConversionException e = assertThrows(OssieConverter.ConversionException.class, + () -> OssieConverter.convertMetricViewToOssie(mv, null)); + assertTrue(e.getMessage().contains("no join condition")); + } + + @Test + public void importRejectsUnsupportedVersion() { + OssieConverter.ConversionException e = assertThrows(OssieConverter.ConversionException.class, + () -> OssieConverter.convertMetricViewToOssie("version: '0.1'\nsource: c.s.t\n", null)); + assertTrue(e.getMessage().contains("Unsupported Metric View version")); + } + + @Test + public void mvToOssieToMvRoundTripsStash() { + // A view with MV-only features (filter/rely/format/window) must survive + // MV -> Ossie -> MV unchanged (the custom_extensions stash carries them). + String mv = + "version: '1.1'\n" + + "source: c.s.orders\n" + + "filter: o_orderstatus = 'F'\n" + + "joins:\n" + + "- name: customer\n" + + " source: c.s.customer\n" + + " on: source.o_custkey = customer.c_custkey\n" + + " rely:\n" + + " at_most_one_match: true\n" + + "dimensions:\n" + + "- name: net\n" + + " expr: o_totalprice\n" + + " format:\n" + + " type: currency\n" + + " currency_code: USD\n" + + "measures:\n" + + "- name: running\n" + + " expr: SUM(o_totalprice)\n" + + " window:\n" + + " - order: net\n" + + " semiadditive: last\n" + + " range: cumulative\n"; + String ossie = OssieConverter.convertMetricViewToOssie(mv, null).yaml; + String back = OssieConverter.convertOssieToMetricView(ossie, null).yaml; + assertEquals(OssieConverter.parseYaml(mv), OssieConverter.parseYaml(back)); + } + + // -- fixture comparisons --------------------------------------------------- + // The fixtures under src/test/resources/ossie_*.yaml pin the expected outputs + // checked into apache/ossie. Asserting the Java output parses equal to them (structure + // + stash blob STRINGS) is the real "one behavior, two implementations" guarantee -- + // it catches divergences like stash JSON spacing that a Java->Java round-trip misses. + + private static String loadFixture(String name) { + try (InputStream in = + OssieConverterSuite.class.getClassLoader().getResourceAsStream("ossie_" + name)) { + if (in == null) { + throw new IllegalStateException("fixture not found: ossie_" + name); + } + return new String(in.readAllBytes(), StandardCharsets.UTF_8); + } catch (java.io.IOException e) { + throw new RuntimeException(e); + } + } + + @Test + public void fixtureAExportMatchesFixture() { + String out = OssieConverter.convertOssieToMetricView(loadFixture("fixtureA_ossie.yaml"), null).yaml; + assertEquals(OssieConverter.parseYaml(loadFixture("fixtureA_metric_view.yaml")), + OssieConverter.parseYaml(out)); + } + + @Test + public void fixtureBImportMatchesFixture() { + // fixtureB exercises the custom_extensions stash (format/rely/filter) -- the parse + // includes the blob strings, so this is what pins the stash blob's exact spacing. + String out = OssieConverter.convertMetricViewToOssie(loadFixture("fixtureB_metric_view.yaml"), null).yaml; + assertEquals(OssieConverter.parseYaml(loadFixture("fixtureB_ossie.yaml")), + OssieConverter.parseYaml(out)); + } + + @Test + public void tpcdsExportMatchesFixture() { + String out = OssieConverter.convertOssieToMetricView(loadFixture("tpcds_ossie.yaml"), null).yaml; + assertEquals(OssieConverter.parseYaml(loadFixture("tpcds_metric_view.yaml")), + OssieConverter.parseYaml(out)); + } + + @Test + @SuppressWarnings("unchecked") + public void stashBlobUsesExpectedSpacing() { + // The strongest byte-level check: the emitted stash blob string (the `data` value of a + // custom_extensions entry) must use the stash format's separators + // (", " / ": "). Pull the blob out of the parsed model rather than substring-matching + // the outer YAML (where it appears escaped). + Object out = OssieConverter.parseYaml( + OssieConverter.convertMetricViewToOssie(loadFixture("fixtureB_metric_view.yaml"), null).yaml); + Map model = + (Map) ((List) ((Map) out).get("semantic_model")).get(0); + List exts = (List) model.get("custom_extensions"); + String blob = (String) ((Map) exts.get(0)).get("data"); + assertTrue(blob.startsWith("{\"_v\": 1, "), + "stash blob must use the spacing '{\"_v\": 1, ...}', got: " + blob); + } + + // -- parity edge cases found in review round 3 ---------------------------- + + @Test + public void pickExpressionFallsThroughEmptyDatabricksToAnsi() { + // An empty DATABRICKS dialect must fall through to ANSI_SQL, + // not be selected as the (empty) expression. + String osi = + "version: 0.2.0.dev0\n" + + "semantic_model:\n" + + "- name: m\n" + + " datasets:\n" + + " - name: f\n" + + " source: c.s.f\n" + + " fields:\n" + + " - name: d\n" + + " expression:\n" + + " dialects:\n" + + " - {dialect: DATABRICKS, expression: ''}\n" + + " - {dialect: ANSI_SQL, expression: ansi_col}\n" + + " metrics:\n" + + " - {name: n, expression: {dialects: [{dialect: DATABRICKS, expression: COUNT(*)}]}}\n"; + Object out = export(osi, null); + @SuppressWarnings("unchecked") + List dims = (List) ((Map) out).get("dimensions"); + @SuppressWarnings("unchecked") + Map dim = (Map) dims.get(0); + assertEquals("ansi_col", dim.get("expr")); + } + + @Test + public void pickExpressionRejectsNonStringExpression() { + // A non-string dialect expression (e.g. a YAML number) must raise, not be coerced. + String osi = + "version: 0.2.0.dev0\n" + + "semantic_model:\n" + + "- name: m\n" + + " datasets:\n" + + " - name: f\n" + + " source: c.s.f\n" + + " fields:\n" + + " - name: d\n" + + " expression:\n" + + " dialects:\n" + + " - {dialect: DATABRICKS, expression: 123}\n" + + " metrics:\n" + + " - {name: n, expression: {dialects: [{dialect: DATABRICKS, expression: COUNT(*)}]}}\n"; + OssieConverter.ConversionException e = assertThrows(OssieConverter.ConversionException.class, + () -> OssieConverter.convertOssieToMetricView(osi, null)); + assertTrue(e.getMessage().contains("expression must be a string")); + } + + @Test + public void bareOnOffValuesStayStringsNotBooleans() { + // YAML 1.1 would read a bare `on`/`off`/`yes`/`no` value as a boolean, silently losing + // a join condition or turning a synonym into `true`. Confirm the converter's parser + // keeps them as strings (the reader uses YAML 1.2 boolean semantics). + Object parsed = OssieConverter.parseYaml("a: on\nb: off\nc: yes\nd: no\n"); + @SuppressWarnings("unchecked") + Map m = (Map) parsed; + assertEquals("on", m.get("a")); + assertEquals("off", m.get("b")); + assertEquals("yes", m.get("c")); + assertEquals("no", m.get("d")); + } + + @Test + @SuppressWarnings("unchecked") + public void importDropsEmptyOptionalFields() { + // Optional fields are mapped only when non-empty, so an empty + // comment / empty synonyms list are omitted, not emitted as `description: ""` or an + // empty ai_context. The Java port must match (empty string / empty list are falsy). + String mv = + "version: '1.1'\n" + + "source: c.s.orders\n" + + "comment: ''\n" + + "dimensions:\n" + + "- {name: o_status, expr: o_orderstatus, comment: '', display_name: '', synonyms: []}\n" + + "measures:\n" + + "- {name: revenue, expr: SUM(o_totalprice), comment: '', synonyms: []}\n"; + Map out = (Map) importMv(mv); + List models = (List) out.get("semantic_model"); + Map model = (Map) models.get(0); + assertFalse(model.containsKey("description"), "empty comment must not become a description"); + Map ds = (Map) ((List) model.get("datasets")).get(0); + Map field = (Map) ((List) ds.get("fields")).get(0); + assertFalse(field.containsKey("description"), "empty comment must not map to description"); + assertFalse(field.containsKey("label"), "empty display_name must not map to label"); + assertFalse(field.containsKey("ai_context"), "empty synonyms must not map to ai_context"); + Map metric = (Map) ((List) model.get("metrics")).get(0); + assertFalse(metric.containsKey("description"), "empty comment must not map to description"); + assertFalse(metric.containsKey("ai_context"), "empty synonyms must not map to ai_context"); + } + + @Test + public void stashEscapesNonAsciiAsLowercaseHex() { + // The stash blob is pure ASCII: non-ASCII is escaped to \\uXXXX. The + // stash blob must be byte-identical, so a non-ASCII stashed value (here a `filter` + // literal) escapes rather than emitting raw UTF-8. + String mv = + "version: '1.1'\n" + + "source: c.s.orders\n" + + "filter: \"region = 'café'\"\n" + + "dimensions:\n" + + "- {name: o_status, expr: o_orderstatus}\n"; + String ossieYaml = OssieConverter.convertMetricViewToOssie(mv, null).yaml; + // Extract the stash blob and assert the exact expected bytes: + // the non-ASCII char is escaped as lowercase \\u00e9 (a single backslash + 5 chars), + // not emitted raw. (Comparing the parsed `data` string sidesteps YAML's own quoting.) + @SuppressWarnings("unchecked") + Map out = (Map) OssieConverter.parseYaml(ossieYaml); + @SuppressWarnings("unchecked") + List models = (List) out.get("semantic_model"); + Map model = (Map) models.get(0); + @SuppressWarnings("unchecked") + List exts = (List) model.get("custom_extensions"); + @SuppressWarnings("unchecked") + String blob = (String) ((Map) exts.get(0)).get("data"); + assertEquals("{\"_v\": 1, \"filter\": \"region = 'caf\\u00e9'\"}", blob, + "stash blob must use a lowercase \\u escape"); + // And it still round-trips back to the original view. + String back = OssieConverter.convertOssieToMetricView(ossieYaml, null).yaml; + assertEquals(OssieConverter.parseYaml(mv), OssieConverter.parseYaml(back)); + } + + @Test + public void stashPreservesAValueContainingALiteralUnicodeEscape() { + // A stashed value may itself contain the text of a unicode escape. Serialized, that is a + // DOUBLED backslash, so the hex-lowercasing pass must not treat it as a real escape -- + // doing so silently lowercases the value's own characters. + // Single-quoted YAML: a backslash is an ordinary character there, so `filter` really holds + // the six characters \ u A B C D rather than the character U+ABCD. + String mv = + "version: '1.1'\n" + + "source: c.s.orders\n" + + "filter: 'tag = \\uABCD'\n" + + "dimensions:\n" + + "- {name: o_status, expr: o_orderstatus}\n"; + @SuppressWarnings("unchecked") + Map parsedIn = (Map) OssieConverter.parseYaml(mv); + String original = (String) parsedIn.get("filter"); + // Guard the fixture itself: the value must contain a real backslash for this to be a test. + assertTrue(original.indexOf('\\') >= 0, + "test setup: filter must hold a literal backslash, got: " + original); + + String ossieYaml = OssieConverter.convertMetricViewToOssie(mv, null).yaml; + String back = OssieConverter.convertOssieToMetricView(ossieYaml, null).yaml; + @SuppressWarnings("unchecked") + Map restored = (Map) OssieConverter.parseYaml(back); + assertEquals(original, restored.get("filter"), + "a literal unicode-escape sequence in a stashed value must survive unchanged"); + } + + @Test + public void joinOnTakesPrecedenceOverUsing() { + // Metric View validation requires only that one of `on`/`using` is present, so both may be + // set. Databricks resolves the criteria from `on` when it is present, so the converter must + // decompose `on` and ignore `using` -- otherwise the relationship joins on other columns. + String mv = + "version: '1.1'\n" + + "source: c.s.orders\n" + + "joins:\n" + + "- name: cust\n" + + " source: c.s.customer\n" + + " on: source.o_custkey = cust.c_custkey\n" + + " using: [nation_key]\n" + + "dimensions:\n" + + "- {name: c_name, expr: cust.c_name}\n" + + "measures:\n" + + "- {name: cnt, expr: COUNT(1)}\n"; + @SuppressWarnings("unchecked") + Map out = (Map) OssieConverter.parseYaml( + OssieConverter.convertMetricViewToOssie(mv, null).yaml); + @SuppressWarnings("unchecked") + List models = (List) out.get("semantic_model"); + @SuppressWarnings("unchecked") + Map model = (Map) models.get(0); + @SuppressWarnings("unchecked") + List rels = (List) model.get("relationships"); + @SuppressWarnings("unchecked") + Map rel = (Map) rels.get(0); + assertEquals(List.of("o_custkey"), rel.get("from_columns"), + "`on` must win over `using`: expected the o_custkey/c_custkey pair"); + assertEquals(List.of("c_custkey"), rel.get("to_columns"), + "`on` must win over `using`: expected the o_custkey/c_custkey pair"); + } + + @Test + public void measureRewriteLeavesStringLiteralsAlone() { + // The fact qualifier is added/stripped by rewriting the measure expression. That rewrite must + // skip string literals: rewriting inside one changes the predicate and therefore the value. + String mv = + "version: '1.1'\n" + + "source: c.s.orders\n" + + "dimensions:\n" + + "- {name: o_status, expr: o_orderstatus}\n" + + "measures:\n" + + "- name: tagged\n" + + " expr: \"SUM(IF(source.region = 'source.us', 1, 0))\"\n"; + String ossieYaml = OssieConverter.convertMetricViewToOssie(mv, null).yaml; + assertTrue(ossieYaml.contains("'source.us'"), + "a literal mentioning the qualifier must not be rewritten, got:\n" + ossieYaml); + // The literal also survives the trip back. Note the *code* qualifier is normalized on the + // way through (`source.region` -> bare `region`, the Metric View idiom for fact columns); + // only the literal is required to come back byte-identical. + String back = OssieConverter.convertOssieToMetricView(ossieYaml, null).yaml; + assertTrue(back.contains("'source.us'"), + "the literal must survive the round trip, got:\n" + back); + } +} diff --git a/converters/databricks/tests/fixtures/fixtureA_metric_view.yaml b/converters/databricks/java/src/test/resources/ossie_fixtureA_metric_view.yaml similarity index 100% rename from converters/databricks/tests/fixtures/fixtureA_metric_view.yaml rename to converters/databricks/java/src/test/resources/ossie_fixtureA_metric_view.yaml diff --git a/converters/databricks/tests/fixtures/fixtureA_ossie.yaml b/converters/databricks/java/src/test/resources/ossie_fixtureA_ossie.yaml similarity index 100% rename from converters/databricks/tests/fixtures/fixtureA_ossie.yaml rename to converters/databricks/java/src/test/resources/ossie_fixtureA_ossie.yaml diff --git a/converters/databricks/java/src/test/resources/ossie_fixtureB_metric_view.yaml b/converters/databricks/java/src/test/resources/ossie_fixtureB_metric_view.yaml new file mode 100644 index 00000000..723bcec8 --- /dev/null +++ b/converters/databricks/java/src/test/resources/ossie_fixtureB_metric_view.yaml @@ -0,0 +1,60 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# Fixture B -- stash round-trip (MV -> Apache Ossie -> MV, lossless). +# Exercises the stash at every placement level: model (filter), relationship +# (rely), dimension (format), measure (format, partition). Must parse under v1.1. +# +# `partition` has no Apache Ossie representation, so it round-trips through the +# DATABRICKS stash like `format`/`window`. It is attached to a measure with no +# window block, since a partition combined with a window is rejected by v1.1 +# validation, and `partition.include` names an existing dimension as required. + +version: '1.1' +source: samples.tpch.lineitem +filter: l_returnflag = 'N' +comment: Line item shipping metrics +joins: +- name: orders + source: samples.tpch.orders + on: source.l_orderkey = orders.o_orderkey + rely: + at_most_one_match: true +dimensions: +- name: line_number + expr: l_linenumber + format: + type: number + decimal_places: + type: exact + places: 0 +measures: +- name: revenue + expr: SUM(l_extendedprice * (1 - l_discount)) + comment: Net revenue + format: + type: currency + currency_code: USD + decimal_places: + type: exact + places: 2 +- name: order_count + expr: COUNT(DISTINCT l_orderkey) + partition: + include: + - line_number + outer_aggregate: SUM diff --git a/converters/databricks/java/src/test/resources/ossie_fixtureB_ossie.yaml b/converters/databricks/java/src/test/resources/ossie_fixtureB_ossie.yaml new file mode 100644 index 00000000..65abec77 --- /dev/null +++ b/converters/databricks/java/src/test/resources/ossie_fixtureB_ossie.yaml @@ -0,0 +1,75 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# yaml-language-server: $schema=../../../../core-spec/osi-schema.json +# +# Fixture B -- expected Apache Ossie produced from fixtureB_metric_view.yaml. MV-only +# features are stashed in custom_extensions[DATABRICKS], keyed by their exact v1.1 +# field name. Exporting this back must reproduce fixtureB_metric_view.yaml. +# Model name is derived from the fact table (`lineitem`). + +version: "0.2.0.dev0" + +semantic_model: + - name: lineitem + description: Line item shipping metrics + datasets: + - name: lineitem + source: samples.tpch.lineitem + fields: + - name: line_number + expression: + dialects: + - dialect: DATABRICKS + expression: l_linenumber + custom_extensions: + - vendor_name: DATABRICKS + data: '{"_v": 1, "format": {"type": "number", "decimal_places": {"type": "exact", "places": 0}}}' + - name: orders + source: samples.tpch.orders + unique_keys: + - [o_orderkey] + relationships: + - name: lineitem_to_orders + from: lineitem + to: orders + from_columns: [l_orderkey] + to_columns: [o_orderkey] + custom_extensions: + - vendor_name: DATABRICKS + data: '{"_v": 1, "rely": {"at_most_one_match": true}}' + metrics: + - name: revenue + expression: + dialects: + - dialect: DATABRICKS + expression: SUM(l_extendedprice * (1 - l_discount)) + description: Net revenue + custom_extensions: + - vendor_name: DATABRICKS + data: '{"_v": 1, "format": {"type": "currency", "currency_code": "USD", "decimal_places": {"type": "exact", "places": 2}}}' + - name: order_count + expression: + dialects: + - dialect: DATABRICKS + expression: COUNT(DISTINCT l_orderkey) + custom_extensions: + - vendor_name: DATABRICKS + data: '{"_v": 1, "partition": {"include": ["line_number"], "outer_aggregate": "SUM"}}' + custom_extensions: + - vendor_name: DATABRICKS + data: '{"_v": 1, "filter": "l_returnflag = ''N''"}' diff --git a/converters/databricks/tests/fixtures/tpcds_metric_view.yaml b/converters/databricks/java/src/test/resources/ossie_tpcds_metric_view.yaml similarity index 100% rename from converters/databricks/tests/fixtures/tpcds_metric_view.yaml rename to converters/databricks/java/src/test/resources/ossie_tpcds_metric_view.yaml diff --git a/converters/databricks/tests/fixtures/tpcds_ossie.yaml b/converters/databricks/java/src/test/resources/ossie_tpcds_ossie.yaml similarity index 100% rename from converters/databricks/tests/fixtures/tpcds_ossie.yaml rename to converters/databricks/java/src/test/resources/ossie_tpcds_ossie.yaml diff --git a/converters/databricks/python/README.md b/converters/databricks/python/README.md new file mode 100644 index 00000000..6ef4ea8b --- /dev/null +++ b/converters/databricks/python/README.md @@ -0,0 +1,123 @@ + + +# Apache Ossie Databricks Converter + +Bidirectional, offline conversion between an [Apache Ossie](https://github.com/apache/ossie) +semantic model and a Databricks +[Unity Catalog Metric View](https://docs.databricks.com/aws/en/metric-views/) (YAML +`1.1`). No Databricks connection required. + +- **Export** (`ossie-databricks export`): Apache Ossie -> Metric View (one fact + `source` with a nested `joins` tree and a flat `dimensions` list). +- **Import** (`ossie-databricks import`): Metric View -> Apache Ossie. Metric View features Apache Ossie has + no native field for are preserved in `custom_extensions[DATABRICKS]`, so + `MV -> Apache Ossie -> MV` is lossless. + +On **export** (Apache Ossie -> Metric View), Apache Ossie features with no Metric View slot -- relationship +`ai_context`, `dimension.is_time`, non-`DATABRICKS`/`ANSI_SQL` dialects, foreign-vendor +`custom_extensions` -- are **dropped with a warning**. On **import** (Metric View -> Apache Ossie), +Metric View only features (filter, window, format, rely, ...) are instead **preserved** in +`custom_extensions[DATABRICKS]`, so `MV -> Apache Ossie -> MV` is lossless. Any input that breaks a +[requirement](#requirements) **raises a `ConversionError`** -- the converter never +silently drops a field or produces an invalid result. + +## Installation + +```bash +pip install apache-ossie-databricks # once published to PyPI +# or, from a checkout of this directory: +pip install -e . +``` + +The only runtime dependency is `PyYAML`. Python 3.11+. + +## Usage + +### Command line + +```bash +ossie-databricks export -i model.yaml -o view.yaml [--source orders] # Apache Ossie -> Metric View +ossie-databricks import -i view.yaml -o model.yaml [--name my_model] # Metric View -> Apache Ossie +``` + +With no `-o`, output goes to stdout. `--source` (export) picks the fact/grain (default: +the FK-sink dataset; naming a coarser-grain dataset produces `one_to_many` joins); +`--name` (import) sets the Apache Ossie model name (default: the source's last identifier). + +### Python API + +```python +from ossie_databricks import convert_ossie_to_metric_view, convert_metric_view_to_ossie + +metric_view_yaml = convert_ossie_to_metric_view(ossie_yaml_str) # optionally choose the fact/grain, e.g. (ossie_yaml_str, source="orders") +ossie_yaml = convert_metric_view_to_ossie(metric_view_yaml_str, model_name="sales") +``` + +## Mapping + +Each row maps in both directions; the **Notes** flag where a behavior is specific to +**export** (Apache Ossie -> Metric View) or **import** (Metric View -> Apache Ossie). + +| Apache Ossie | Metric View (v1.1) | Notes | +|---|---|---| +| `semantic_model.description` | `comment` | Model-level description only. | +| root dataset | `source` | The fact/grain. | +| other `datasets` | nested `joins[]` | Export: the relationship graph is reassembled into the join tree; a dataset reached by two paths (a diamond) fans out into one aliased join per path. | +| `relationship` `from_columns`/`to_columns` | join `on` (differing names) / `using` (shared names) | Decomposed into columns on import; rebuilt into `on`/`using` on export. | +| `relationship.from`/`to` direction | join `cardinality` | Export: source on the many (`from`) side -> `many_to_one`; on the one (`to`) side -> `one_to_many`. | +| `dataset.primary_key` / `unique_keys` | join `rely.at_most_one_match` | Both directions: export sets `at_most_one_match` when a key covers the join columns; import recovers a `unique_keys` from it. | +| `dataset.fields[]` | `dimensions[]` | Export: fields flatten into one list and a joined column is qualified by its full join path (`customer.c_name`; `customer.region.r_name` when nested). | +| `field.expression.dialects[]` | `expr` | Export: prefer the `DATABRICKS` dialect, else `ANSI_SQL`. | +| `metrics[]` | `measures[]` | Export: fact columns are referenced bare (`SUM(amount)`). | +| `field.label` | `display_name` | | +| `field` / `metric` `description` | `comment` | | +| `ai_context.synonyms` | `synonyms` | | +| `custom_extensions[DATABRICKS]` | `filter`, `window`, `format`, `rely`, `materialization` | Import stashes Metric View only features here; export restores them -- keeping `MV -> Apache Ossie -> MV` lossless. | + +## Requirements + +Conversion raises a `ConversionError` (rather than guessing or emitting something +invalid) when an input breaks one of these: + +- the Metric View `version` is not `1.1`; +- a `source` is not a 3-part `catalog.schema.table` name or a `SELECT`/`WITH` subquery; +- the relationship graph is not acyclic and resolvable to a single fact -- a cycle, or + multiple candidate facts without `--source`, is rejected (a diamond is allowed and + fanned out); +- a join has no condition (a cross join has no Apache Ossie relationship form); +- a join condition is non-equi or otherwise can't be decomposed into equi-join columns + (Apache Ossie relationships are equi-joins, so the join has no Apache Ossie representation); +- the input YAML is malformed. + +## Development + +```bash +pip install -e ".[dev]" +python3 -m pytest tests/ +``` + +Example-based unit tests plus Hypothesis property-based round-trip tests +(`test_roundtrip_properties.py`, which skip if `hypothesis` is not installed). + +## Status + +This Python implementation is the original reference and is **deprecated** in favor of the Java +converter under [`../java/`](../java/), which is the maintained implementation. New behavior and +fixes land in Java; this copy is kept for reference and is no longer actively extended. diff --git a/converters/databricks/pyproject.toml b/converters/databricks/python/pyproject.toml similarity index 100% rename from converters/databricks/pyproject.toml rename to converters/databricks/python/pyproject.toml diff --git a/converters/databricks/src/ossie_databricks/__init__.py b/converters/databricks/python/src/ossie_databricks/__init__.py similarity index 100% rename from converters/databricks/src/ossie_databricks/__init__.py rename to converters/databricks/python/src/ossie_databricks/__init__.py diff --git a/converters/databricks/src/ossie_databricks/_common.py b/converters/databricks/python/src/ossie_databricks/_common.py similarity index 100% rename from converters/databricks/src/ossie_databricks/_common.py rename to converters/databricks/python/src/ossie_databricks/_common.py diff --git a/converters/databricks/src/ossie_databricks/cli.py b/converters/databricks/python/src/ossie_databricks/cli.py similarity index 100% rename from converters/databricks/src/ossie_databricks/cli.py rename to converters/databricks/python/src/ossie_databricks/cli.py diff --git a/converters/databricks/src/ossie_databricks/metric_view_to_ossie.py b/converters/databricks/python/src/ossie_databricks/metric_view_to_ossie.py similarity index 100% rename from converters/databricks/src/ossie_databricks/metric_view_to_ossie.py rename to converters/databricks/python/src/ossie_databricks/metric_view_to_ossie.py diff --git a/converters/databricks/src/ossie_databricks/ossie_to_metric_view.py b/converters/databricks/python/src/ossie_databricks/ossie_to_metric_view.py similarity index 100% rename from converters/databricks/src/ossie_databricks/ossie_to_metric_view.py rename to converters/databricks/python/src/ossie_databricks/ossie_to_metric_view.py diff --git a/converters/databricks/tests/_roundtrip_helpers.py b/converters/databricks/python/tests/_roundtrip_helpers.py similarity index 100% rename from converters/databricks/tests/_roundtrip_helpers.py rename to converters/databricks/python/tests/_roundtrip_helpers.py diff --git a/converters/databricks/tests/_util.py b/converters/databricks/python/tests/_util.py similarity index 100% rename from converters/databricks/tests/_util.py rename to converters/databricks/python/tests/_util.py diff --git a/converters/databricks/tests/conftest.py b/converters/databricks/python/tests/conftest.py similarity index 100% rename from converters/databricks/tests/conftest.py rename to converters/databricks/python/tests/conftest.py diff --git a/converters/databricks/python/tests/fixtures/fixtureA_metric_view.yaml b/converters/databricks/python/tests/fixtures/fixtureA_metric_view.yaml new file mode 100644 index 00000000..4ffe6819 --- /dev/null +++ b/converters/databricks/python/tests/fixtures/fixtureA_metric_view.yaml @@ -0,0 +1,53 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# Fixture A -- expected UC Metric View (v1.1, single-source) produced from +# fixtureA_ossie.yaml. Must parse under the v1.1 strict schema. + +version: '1.1' +source: samples.tpch.orders +comment: Sales orders with customer attributes +joins: +- name: customer + source: samples.tpch.customer + on: source.o_custkey = customer.c_custkey + rely: + at_most_one_match: true +dimensions: +- name: o_orderkey + expr: o_orderkey + comment: Order identifier +- name: o_orderdate + expr: o_orderdate + display_name: Order Date + synonyms: + - order date + - date +- name: c_name + expr: customer.c_name + comment: Customer name +measures: +- name: total_revenue + expr: SUM(o_totalprice) + comment: Total order revenue + synonyms: + - revenue + - total revenue + - sales +- name: order_count + expr: COUNT(*) + comment: Number of orders diff --git a/converters/databricks/python/tests/fixtures/fixtureA_ossie.yaml b/converters/databricks/python/tests/fixtures/fixtureA_ossie.yaml new file mode 100644 index 00000000..a53942f3 --- /dev/null +++ b/converters/databricks/python/tests/fixtures/fixtureA_ossie.yaml @@ -0,0 +1,79 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# yaml-language-server: $schema=../../../../core-spec/osi-schema.json +# +# Fixture A -- all-native round-trip (Apache Ossie -> MV -> Apache Ossie). +# Star schema; every field maps to a native MV field. Documented losses on the +# Apache Ossie -> MV -> Apache Ossie trip: the model name (MV carries none) and primary_key. + +version: "0.2.0.dev0" + +semantic_model: + - name: sales + description: Sales orders with customer attributes + datasets: + - name: orders # fact: no incoming relationship -> becomes `source` + source: samples.tpch.orders + primary_key: [o_orderkey] # dropped on export (Apache Ossie-only) + description: One row per order + fields: + - name: o_orderkey + expression: + dialects: + - dialect: DATABRICKS + expression: o_orderkey + description: Order identifier + - name: o_orderdate + expression: + dialects: + - dialect: DATABRICKS + expression: o_orderdate + label: Order Date + ai_context: + synonyms: [order date, date] + - name: customer + source: samples.tpch.customer + primary_key: [c_custkey] + fields: + - name: c_name + expression: + dialects: + - dialect: DATABRICKS + expression: c_name + description: Customer name + relationships: + - name: orders_to_customer + from: orders + to: customer + from_columns: [o_custkey] + to_columns: [c_custkey] + metrics: + - name: total_revenue + expression: + dialects: + - dialect: DATABRICKS + expression: SUM(o_totalprice) # fact columns are bare in measures + description: Total order revenue + ai_context: + synonyms: [revenue, total revenue, sales] + - name: order_count + expression: + dialects: + - dialect: DATABRICKS + expression: COUNT(*) + description: Number of orders diff --git a/converters/databricks/tests/fixtures/fixtureB_metric_view.yaml b/converters/databricks/python/tests/fixtures/fixtureB_metric_view.yaml similarity index 100% rename from converters/databricks/tests/fixtures/fixtureB_metric_view.yaml rename to converters/databricks/python/tests/fixtures/fixtureB_metric_view.yaml diff --git a/converters/databricks/tests/fixtures/fixtureB_ossie.yaml b/converters/databricks/python/tests/fixtures/fixtureB_ossie.yaml similarity index 100% rename from converters/databricks/tests/fixtures/fixtureB_ossie.yaml rename to converters/databricks/python/tests/fixtures/fixtureB_ossie.yaml diff --git a/converters/databricks/python/tests/fixtures/tpcds_metric_view.yaml b/converters/databricks/python/tests/fixtures/tpcds_metric_view.yaml new file mode 100644 index 00000000..b22fc698 --- /dev/null +++ b/converters/databricks/python/tests/fixtures/tpcds_metric_view.yaml @@ -0,0 +1,67 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +version: '1.1' +source: tpcds.public.store_sales +comment: Store sales enriched with date, item, and customer dimensions +filter: ss_net_profit > 0 +joins: +- name: date_dim + source: tpcds.public.date_dim + on: source.ss_sold_date_sk = date_dim.d_date_sk + rely: + at_most_one_match: true +- name: item + source: tpcds.public.item + on: source.ss_item_sk = item.i_item_sk + rely: + at_most_one_match: true +- name: customer + source: tpcds.public.customer + on: source.ss_customer_sk = customer.c_customer_sk + rely: + at_most_one_match: true +dimensions: +- name: ticket_number + expr: ss_ticket_number +- name: sold_year + expr: date_dim.d_year + display_name: Year + synonyms: + - year + - yr +- name: sold_date + expr: date_dim.d_date +- name: item_category + expr: item.i_category + synonyms: + - category + - product type +- name: item_brand + expr: item.i_brand +- name: birth_country + expr: customer.c_birth_country +measures: +- name: total_sales + expr: SUM(ss_ext_sales_price) + comment: Total sales revenue + format: + type: currency + currency_code: USD +- name: total_quantity + expr: SUM(ss_quantity) + comment: Total units sold diff --git a/converters/databricks/python/tests/fixtures/tpcds_ossie.yaml b/converters/databricks/python/tests/fixtures/tpcds_ossie.yaml new file mode 100644 index 00000000..e055eab1 --- /dev/null +++ b/converters/databricks/python/tests/fixtures/tpcds_ossie.yaml @@ -0,0 +1,89 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +version: "0.2.0.dev0" +semantic_model: + - name: tpcds_store_sales + description: Store sales enriched with date, item, and customer dimensions + datasets: + - name: store_sales + source: tpcds.public.store_sales + fields: + - name: ticket_number + expression: + dialects: [{dialect: DATABRICKS, expression: ss_ticket_number}] + - name: date_dim + source: tpcds.public.date_dim + primary_key: [d_date_sk] + fields: + - name: sold_year + expression: + dialects: [{dialect: DATABRICKS, expression: d_year}] + label: Year + ai_context: {synonyms: [year, yr]} + - name: sold_date + expression: + dialects: [{dialect: DATABRICKS, expression: d_date}] + - name: item + source: tpcds.public.item + primary_key: [i_item_sk] + fields: + - name: item_category + expression: + dialects: [{dialect: DATABRICKS, expression: i_category}] + ai_context: {synonyms: [category, product type]} + - name: item_brand + expression: + dialects: [{dialect: DATABRICKS, expression: i_brand}] + - name: customer + source: tpcds.public.customer + primary_key: [c_customer_sk] + fields: + - name: birth_country + expression: + dialects: [{dialect: DATABRICKS, expression: c_birth_country}] + relationships: + - name: store_sales_to_date_dim + from: store_sales + to: date_dim + from_columns: [ss_sold_date_sk] + to_columns: [d_date_sk] + - name: store_sales_to_item + from: store_sales + to: item + from_columns: [ss_item_sk] + to_columns: [i_item_sk] + - name: store_sales_to_customer + from: store_sales + to: customer + from_columns: [ss_customer_sk] + to_columns: [c_customer_sk] + metrics: + - name: total_sales + expression: + dialects: [{dialect: DATABRICKS, expression: SUM(ss_ext_sales_price)}] + description: Total sales revenue + custom_extensions: + - vendor_name: DATABRICKS + data: '{"_v": 1, "format": {"type": "currency", "currency_code": "USD"}}' + - name: total_quantity + expression: + dialects: [{dialect: DATABRICKS, expression: SUM(ss_quantity)}] + description: Total units sold + custom_extensions: + - vendor_name: DATABRICKS + data: '{"_v": 1, "filter": "ss_net_profit > 0"}' diff --git a/converters/databricks/tests/test_metric_view_to_ossie.py b/converters/databricks/python/tests/test_metric_view_to_ossie.py similarity index 100% rename from converters/databricks/tests/test_metric_view_to_ossie.py rename to converters/databricks/python/tests/test_metric_view_to_ossie.py diff --git a/converters/databricks/tests/test_ossie_to_metric_view.py b/converters/databricks/python/tests/test_ossie_to_metric_view.py similarity index 100% rename from converters/databricks/tests/test_ossie_to_metric_view.py rename to converters/databricks/python/tests/test_ossie_to_metric_view.py diff --git a/converters/databricks/tests/test_roundtrip.py b/converters/databricks/python/tests/test_roundtrip.py similarity index 100% rename from converters/databricks/tests/test_roundtrip.py rename to converters/databricks/python/tests/test_roundtrip.py diff --git a/converters/databricks/tests/test_roundtrip_properties.py b/converters/databricks/python/tests/test_roundtrip_properties.py similarity index 100% rename from converters/databricks/tests/test_roundtrip_properties.py rename to converters/databricks/python/tests/test_roundtrip_properties.py diff --git a/converters/databricks/uv.lock b/converters/databricks/python/uv.lock similarity index 100% rename from converters/databricks/uv.lock rename to converters/databricks/python/uv.lock