Skip to content

[OSSIE][DATABRICKS] Add Java Converter and Reformat #333

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Haoranli503 wants to merge 4 commits into
base: main
Choose a base branch
from
Open

[OSSIE][DATABRICKS] Add Java Converter and Reformat #333

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
ec0436e
Add Databricks Unity Catalog Metric View converter (converters/databr…
Haoranli503 Jul 17, 2026
efeb18e
[OSSIE] Add Java converter and restructure converters/databricks
Haoranli503 Aug 19, 2026
3c16456
[OSSIE] Mark the Python converter as deprecated in favor of Java
Haoranli503 Aug 19, 2026
45cfa8e
Merge upstream/main into databricks-converter-v2
Haoranli503 Aug 19, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 25 additions & 3 deletions .github/workflows/converter-databricks-ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,11 +32,14 @@ on:
- '.github/workflows/converter-databricks-ci.yml'

jobs:
build:
python:
runs-on: ubuntu-latest
strategy:
matrix:
python-version: ["3.11", "3.12", "3.13", "3.14"]
defaults:
run:
working-directory: converters/databricks/python

steps:
- name: Checkout project
Expand All @@ -53,11 +56,30 @@ jobs:
echo "${HOME}/.local/bin" >> "${GITHUB_PATH}"

- name: Sync dependencies
working-directory: converters/databricks
run: |
uv sync

- name: Unit Tests
working-directory: converters/databricks
run: |
uv run pytest

java:
runs-on: ubuntu-latest
defaults:
run:
working-directory: converters/databricks/java

steps:
- name: Checkout project
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1

- name: Set up JDK 21
uses: actions/setup-java@03ad4de0992f5dab5e18fcb136590ce7c4a0ac95 # v5.6.0
with:
java-version: '21'
distribution: 'temurin'
cache: 'maven'

- name: Build and test
run: |
mvn -B verify
133 changes: 13 additions & 120 deletions converters/databricks/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,124 +1,17 @@
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
Ossie <-> Metric View converters
=================================

http://www.apache.org/licenses/LICENSE-2.0
A bidirectional converter between [Apache Ossie](https://github.com/apache/ossie) semantic models
and Databricks Unity Catalog Metric Views (YAML v1.1). Conversion is pure YAML text in, YAML text
out: it reads and writes the two formats as parsed maps and lists, independent of any engine.

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
Layout
------

# Apache Ossie Databricks Converter
| Path | Language | Role |
|------|----------|------|
| [`java/`](java/) | Java | The maintained implementation; also ships a command-line tool (`OssieDatabricksConverter`). |
| [`python/`](python/) | Python | The original reference implementation. To be deprecated. |

Bidirectional, offline conversion between an [Apache Ossie](https://github.com/apache/ossie)
semantic model and a Databricks
[Unity Catalog Metric View](https://docs.databricks.com/aws/en/metric-views/) (YAML
`1.1`). No Databricks connection required.

- **Export** (`ossie-databricks export`): Apache Ossie -> Metric View (one fact
`source` with a nested `joins` tree and a flat `dimensions` list).
- **Import** (`ossie-databricks import`): Metric View -> Apache Ossie. Metric View features Apache Ossie has
no native field for are preserved in `custom_extensions[DATABRICKS]`, so
`MV -> Apache Ossie -> MV` is lossless.

On **export** (Apache Ossie -> Metric View), Apache Ossie features with no Metric View slot -- relationship
`ai_context`, `dimension.is_time`, non-`DATABRICKS`/`ANSI_SQL` dialects, foreign-vendor
`custom_extensions` -- are **dropped with a warning**. On **import** (Metric View -> Apache Ossie),
Metric View only features (filter, window, format, rely, ...) are instead **preserved** in
`custom_extensions[DATABRICKS]`, so `MV -> Apache Ossie -> MV` is lossless. Any input that breaks a
[requirement](#requirements) **raises a `ConversionError`** -- the converter never
silently drops a field or produces an invalid result.

## Installation

```bash
pip install apache-ossie-databricks # once published to PyPI
# or, from a checkout of this directory:
pip install -e .
```

The only runtime dependency is `PyYAML`. Python 3.11+.

## Usage

### Command line

```bash
ossie-databricks export -i model.yaml -o view.yaml [--source orders] # Apache Ossie -> Metric View
ossie-databricks import -i view.yaml -o model.yaml [--name my_model] # Metric View -> Apache Ossie
```

With no `-o`, output goes to stdout. `--source` (export) picks the fact/grain (default:
the FK-sink dataset; naming a coarser-grain dataset produces `one_to_many` joins);
`--name` (import) sets the Apache Ossie model name (default: the source's last identifier).

### Python API

```python
from ossie_databricks import convert_ossie_to_metric_view, convert_metric_view_to_ossie

metric_view_yaml = convert_ossie_to_metric_view(ossie_yaml_str) # optionally choose the fact/grain, e.g. (ossie_yaml_str, source="orders")
ossie_yaml = convert_metric_view_to_ossie(metric_view_yaml_str, model_name="sales")
```

## Mapping

Each row maps in both directions; the **Notes** flag where a behavior is specific to
**export** (Apache Ossie -> Metric View) or **import** (Metric View -> Apache Ossie).

| Apache Ossie | Metric View (v1.1) | Notes |
|---|---|---|
| `semantic_model.description` | `comment` | Model-level description only. |
| root dataset | `source` | The fact/grain. |
| other `datasets` | nested `joins[]` | Export: the relationship graph is reassembled into the join tree; a dataset reached by two paths (a diamond) fans out into one aliased join per path. |
| `relationship` `from_columns`/`to_columns` | join `on` (differing names) / `using` (shared names) | Decomposed into columns on import; rebuilt into `on`/`using` on export. |
| `relationship.from`/`to` direction | join `cardinality` | Export: source on the many (`from`) side -> `many_to_one`; on the one (`to`) side -> `one_to_many`. |
| `dataset.primary_key` / `unique_keys` | join `rely.at_most_one_match` | Both directions: export sets `at_most_one_match` when a key covers the join columns; import recovers a `unique_keys` from it. |
| `dataset.fields[]` | `dimensions[]` | Export: fields flatten into one list and a joined column is qualified by its full join path (`customer.c_name`; `customer.region.r_name` when nested). |
| `field.expression.dialects[]` | `expr` | Export: prefer the `DATABRICKS` dialect, else `ANSI_SQL`. |
| `metrics[]` | `measures[]` | Export: fact columns are referenced bare (`SUM(amount)`). |
| `field.label` | `display_name` | |
| `field` / `metric` `description` | `comment` | |
| `ai_context.synonyms` | `synonyms` | |
| `custom_extensions[DATABRICKS]` | `filter`, `window`, `format`, `rely`, `materialization` | Import stashes Metric View only features here; export restores them -- keeping `MV -> Apache Ossie -> MV` lossless. |

## Requirements

Conversion raises a `ConversionError` (rather than guessing or emitting something
invalid) when an input breaks one of these:

- the Metric View `version` is not `1.1`;
- a `source` is not a 3-part `catalog.schema.table` name or a `SELECT`/`WITH` subquery;
- the relationship graph is not acyclic and resolvable to a single fact -- a cycle, or
multiple candidate facts without `--source`, is rejected (a diamond is allowed and
fanned out);
- a join has no condition (a cross join has no Apache Ossie relationship form);
- a join condition is non-equi or otherwise can't be decomposed into equi-join columns
(Apache Ossie relationships are equi-joins, so the join has no Apache Ossie representation);
- the input YAML is malformed.

## Development

```bash
pip install -e ".[dev]"
python3 -m pytest tests/
```

Example-based unit tests plus Hypothesis property-based round-trip tests
(`test_roundtrip_properties.py`, which skip if `hypothesis` is not installed).

## Future effort

Both the Apache Ossie specification and the Databricks Unity Catalog Metric View YAML are still
evolving. As either side adds or changes fields, this converter will be updated to track
them -- extending the mapping and coverage in both directions to keep the conversion
current and to support as much as each format allows over time.
See [`java/README.md`](java/README.md) and [`python/README.md`](python/README.md) for building and
using each implementation.
137 changes: 137 additions & 0 deletions converters/databricks/java/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,137 @@
# Apache Ossie Databricks Converter

Bidirectional, offline conversion between an [Apache Ossie](https://github.com/apache/ossie)
semantic model and a Databricks
[Unity Catalog Metric View](https://docs.databricks.com/aws/en/metric-views/) (YAML `1.1`). Pure
YAML text in, YAML text out: it reads and writes the two formats as parsed maps and lists.

- **Export** (`MetricViewToOssie`): Metric View -> Apache Ossie. The direction is named from the
Metric View's point of view -- it takes a Metric View *out* to Ossie. Metric-View-only features
Apache Ossie has no native field for are preserved in `custom_extensions[DATABRICKS]`, so
`MV -> Apache Ossie -> MV` is lossless.
- **Import** (`OssieToMetricView`): Apache Ossie -> Metric View (one fact `source` with a nested
`joins` tree and a flat `dimensions` list).

On **import** (Apache Ossie -> Metric View), Apache Ossie features with no Metric View slot --
relationship `ai_context`, `dimension.is_time`, non-`DATABRICKS`/`ANSI_SQL` dialects, foreign-vendor
`custom_extensions` -- are **dropped with a notice**. On **export** (Metric View -> Apache Ossie),
Metric-View-only features (`filter`, `parameters`, `materialization`, per-column `format`, measure
`window` / `partition`) are instead **preserved** in `custom_extensions[DATABRICKS]`, so
`MV -> Apache Ossie -> MV` is lossless. Any input that breaks a [requirement](#requirements)
**raises a `ConversionException`** -- the converter never silently drops a field or produces an
invalid result.

## Requirements

- **Java 21+**
- **Maven 3.6+** -- required to build the jar

## Building

Build the self-contained executable jar from source:

```bash
mvn clean package
```

This produces `target/ossie-databricks-converter-0.1.0-SNAPSHOT.jar` with all dependencies
(Jackson and SnakeYAML) bundled.

## Usage

### Command line

```bash
# import: Apache Ossie -> Metric View
java -jar target/ossie-databricks-converter-0.1.0-SNAPSHOT.jar import model.yaml -o view.yaml

# export: Metric View -> Apache Ossie
java -jar target/ossie-databricks-converter-0.1.0-SNAPSHOT.jar export view.yaml -o model.yaml
```

With no `-o`, output goes to stdout. `--source` (import) picks the fact/grain (default: the FK-sink
dataset; naming a coarser-grain dataset produces `one_to_many` joins); `--name` (export) sets the
Apache Ossie model name (default: the source's last identifier). Conversion notices (features
dropped on import) are written to stderr; a non-convertible input exits non-zero.

### Java API

```java
import org.apache.ossie.converter.databricks.OssieConverter;

// export: Metric View -> Apache Ossie (optionally name the model; default: the source's last part)
OssieConverter.Result ossie = OssieConverter.convertMetricViewToOssie(metricViewYaml, "sales");

// import: Apache Ossie -> Metric View (optionally choose the fact/grain; default: the FK-sink
// dataset -- naming a coarser-grain dataset produces one_to_many joins)
OssieConverter.Result view = OssieConverter.convertOssieToMetricView(ossieYaml, "orders");
```

Each `Result` carries the output YAML (`result.yaml`) and any notices raised (`result.notices`,
the features dropped on import). A broken [requirement](#requirements) throws a
`ConversionException` instead.

## Mapping

Each row maps in both directions; the **Notes** flag where a behavior is specific to
**export** (Metric View -> Apache Ossie) or **import** (Apache Ossie -> Metric View).

| Apache Ossie | Metric View (v1.1) | Notes |
|---|---|---|
| `semantic_model.description` | `comment` | Model-level description only. |
| root dataset | `source` | The fact/grain. |
| other `datasets` | nested `joins[]` | Import: the relationship graph is reassembled into the join tree; a dataset reached by two paths (a diamond) fans out into one aliased join per path. |
| `relationship` `from_columns`/`to_columns` | join `on` (differing names) / `using` (shared names) | Decomposed into columns on export; rebuilt into `on`/`using` on import. |
| `relationship.from`/`to` direction | join `cardinality` | Import: source on the many (`from`) side -> `many_to_one`; on the one (`to`) side -> `one_to_many`. |
| `dataset.primary_key` / `unique_keys` | join `rely.at_most_one_match` | Both directions: import sets `at_most_one_match` when a key covers the join columns; export recovers a `unique_keys` from it. |
| `dataset.fields[]` | `dimensions[]` | Import: fields flatten into one list and a joined column is qualified by its full join path (`customer.c_name`; `customer.region.r_name` when nested). |
| `field.expression.dialects[]` | `expr` | Import: prefer the `DATABRICKS` dialect, else `ANSI_SQL`. |
| `metrics[]` | `measures[]` | Import: fact columns are referenced bare (`SUM(amount)`). |
| `field.label` | `display_name` | |
| `field` / `metric` `description` | `comment` | |
| `ai_context.synonyms` | `synonyms` | |
| `custom_extensions[DATABRICKS]` | `filter`, `parameters`, `materialization`, per-column `format`, measure `window` / `partition` | Export stashes Metric-View-only features here; import restores them -- keeping `MV -> Apache Ossie -> MV` lossless. |

## Requirements

Conversion throws a `ConversionException` (rather than guessing or emitting something invalid) when
an input breaks one of these:

- the Metric View `version` is not `1.1`;
- a `source` is not a 3-part `catalog.schema.table` name or a `SELECT`/`WITH` subquery;
- the relationship graph is not acyclic and resolvable to a single fact -- a cycle, or multiple
candidate facts without a chosen source, is rejected (a diamond is allowed and fanned out);
- a join has no condition (a cross join has no Apache Ossie relationship form);
- a join condition is non-equi or otherwise can't be decomposed into equi-join columns (Apache
Ossie relationships are equi-joins, so the join has no Apache Ossie representation);
- the input YAML is malformed.

## Development

Run the test suite:

```bash
mvn test
```

JUnit 5 suites (unit + round-trip) live under `src/test/java/`, with the YAML fixtures in
`src/test/resources/`. The source layout:

```
src/main/java/org/apache/ossie/converter/databricks/
OssieConverter.java public facade: entry points + ConversionException/Notices/Result
OssieConverterCommon.java shared constants, YAML I/O, map accessors, the stash codec
MetricViewToOssie.java export: Metric View v1.1 -> Apache Ossie
OssieToMetricView.java import: Apache Ossie -> Metric View v1.1
OssieDatabricksConverter.java command-line entry point (import / export)
```

The authoritative contract is Metric View YAML v1.1 as Databricks defines it; the checked-in
fixtures under `src/test/resources/` pin the expected output of both directions.

## Future effort

Both the Apache Ossie specification and the Databricks Unity Catalog Metric View YAML are still
evolving. As either side adds or changes fields, this converter will be updated to track them --
extending the mapping and coverage in both directions to keep the conversion current and to support
as much as each format allows over time.
Loading