diff --git a/README.md b/README.md index 98df014..e1e16c1 100644 --- a/README.md +++ b/README.md @@ -41,6 +41,7 @@ A **source** is read into the IR; a **target** is written from it. `dbt` and | `databricks-metric-view` | | ✓ | | `lightdash` | | ✓ | | `ossie` | ✓ | ✓ | +| `slayer` | | ✓ | Adding a dialect is small, self-contained work: implement a `Parser` (dialect files to IR), an `Emitter` (IR to dialect files), or both — the interfaces in @@ -157,20 +158,20 @@ Every dialect maps to the same neutral IR, but targets differ in how much of it they can express. This is what each **target** emits today (`dbt` and `ossie` are also sources; see [Semantic Layer Dialects](#semantic-layer-dialects) above). -| Feature | `dbt` | `cortex` | `snowflake-semantic-view` | `supersimple` | `nao-yaml` | `nao-context-rules` | `databricks-metric-view` | `lightdash` | `ossie` | -|-------------------------|:-----:|:--------:|:-------------------------:|:-------------:|:----------:|:-------------------:|:------------------------:|:-----------:|:-------:| -| Tables | ✓ | ✓ | ✓ | ✓ | | ~ | ✓ | ✓ | ✓ | -| Columns | ✓ | ✓ | ✓ | ✓ | ✓ | ~ | ✓ | ✓ | ✓ | -| Time dimensions | ✓ | ✓ | ~ | ✓ | ✓ | ~ | ~ | ✓ | ✓ | -| Descriptions | ✓ | ✓ | ✓ | ✓ | ~ | ✓ | ✓ | ✓ | ✓ | -| Data types | ✓ | ✓ | | ✓ | | | | ~ | ✓ | -| Primary keys | ✓ | ✓ | ✓ | ✓ | | | | ~ | ✓ | -| Relationships | ✓ | ✓ | ✓ | ✓ | | ✓ | ✓ | ✓ | ✓ | -| Metrics (aggregations) | ✓ | ✓ | ✓ | ✓ | ~ | ✓ | ✓ | ✓ | ✓ | -| Ratio & derived metrics | ✓ | ✓ | ✓ | ~ | ✓ | ✓ | ✓ | ~ | ✓ | -| Synonyms | ~ | ✓ | | | ≈ | ≈ | ✓ | ≈ | ✓ | -| Physical table source | | ~ | ~ | ~ | ≈ | ≈ | ~ | | ✓ | -| Enums / allowed values | ✓ | ~ | ≈ | ≈ | ✓ | ✓ | ≈ | ≈ | ≈ | +| Feature | `dbt` | `cortex` | `snowflake-semantic-view` | `supersimple` | `nao-yaml` | `nao-context-rules` | `databricks-metric-view` | `lightdash` | `ossie` | `slayer` | +|-------------------------|:-----:|:--------:|:-------------------------:|:-------------:|:----------:|:-------------------:|:------------------------:|:-----------:|:-------:|:--------:| +| Tables | ✓ | ✓ | ✓ | ✓ | | ~ | ✓ | ✓ | ✓ | ✓ | +| Columns | ✓ | ✓ | ✓ | ✓ | ✓ | ~ | ✓ | ✓ | ✓ | ✓ | +| Time dimensions | ✓ | ✓ | ~ | ✓ | ✓ | ~ | ~ | ✓ | ✓ | ✓ | +| Descriptions | ✓ | ✓ | ✓ | ✓ | ~ | ✓ | ✓ | ✓ | ✓ | ✓ | +| Data types | ✓ | ✓ | | ✓ | | | | ~ | ✓ | ✓ | +| Primary keys | ✓ | ✓ | ✓ | ✓ | | | | ~ | ✓ | ✓ | +| Relationships | ✓ | ✓ | ✓ | ✓ | | ✓ | ✓ | ✓ | ✓ | ~ | +| Metrics (aggregations) | ✓ | ✓ | ✓ | ✓ | ~ | ✓ | ✓ | ✓ | ✓ | ✓ | +| Ratio & derived metrics | ✓ | ✓ | ✓ | ~ | ✓ | ✓ | ✓ | ~ | ✓ | ✓ | +| Synonyms | ~ | ✓ | | | ≈ | ≈ | ✓ | ≈ | ✓ | ✓ | +| Physical table source | | ~ | ~ | ~ | ≈ | ≈ | ~ | | ✓ | ✓ | +| Enums / allowed values | ✓ | ~ | ≈ | ≈ | ✓ | ✓ | ≈ | ≈ | ≈ | ≈ | `✓` structured · `≈` rolled up as text in a description or comment · `~` partial · blank not emitted. @@ -200,6 +201,20 @@ drops it: the IR carries a label or synonyms and require Runtime 17.3+. On an older warehouse, a view containing them is rejected. - **`lightdash`** emits a dbt `schema.yml` with Lightdash `meta:` blocks (dimensions, metrics, joins). It emits single-column primary keys and reference-only ratios, degrading composite keys, filtered aggregates, and cross-table derived metrics to a leading `# semglot:` comment block. A `dbt-meta-key-path` profile option switches `meta:` (dbt 1.9 and earlier) to `config.meta:` (dbt 1.10 and later). +- **`slayer`** emits one [SLayer](https://github.com/MotleyAI/slayer) model + YAML per table (SlayerModel schema v8, the shape `slayer models create` + reads). SLayer chooses aggregations at query time, so measures become named + `col:agg` formulas, derived metrics lower to SLayer's formula DSL (including + `cumsum(...)` for unbounded cumulative windows), and filtered aggregates + become hidden columns carrying SLayer's aggregation-time `filter:`. A query + source has a first-class home (`sql:`, a query-backed model). Relationships + are `~`: SLayer permits one join per target model, so a role-playing + dimension keeps its first FK and reports the rest; the join's declared name + and synonyms fold into the join's description. Measures colliding with a + column name or a SLayer transform name are renamed with a note (columns and + measures share a namespace). SLayer can also import `ossie` output directly + (`slayer import-osi`); the native dialect exists for the fidelity listed + above. - **`ossie`** is a source as well as a target (see [Semantic Layer Dialects](#semantic-layer-dialects) above), reading and writing the Apache Ossie core-spec. Every row above is structured except enums, which have no diff --git a/dialect/README.md b/dialect/README.md index bcfa4d0..e0e8409 100644 --- a/dialect/README.md +++ b/dialect/README.md @@ -30,6 +30,7 @@ Each emitter writes: | `databricks-metric-view` | one `.yaml` metric view per model table, with direct joins to referenced tables (requires Databricks Runtime 17.2+; `display_name`/`synonyms` require 17.3+) | | `lightdash` | `schema.yml` — a dbt schema file carrying Lightdash `meta:` blocks, so Lightdash compiles it into explores. A `dbt-meta-key-path` profile option switches `meta:` (dbt <= 1.9) to `config.meta:` (dbt >= 1.10) | | `ossie` | `semantic_model.yaml` (Apache Ossie core-spec 0.2.0.dev0); a source dialect as well as a target | +| `slayer` | one `
.yaml` SLayer model per table (SlayerModel schema v8, the shape `slayer models create` reads), plus `NOTES.md` for anything deferred | ## Mapping @@ -38,21 +39,21 @@ where marked, ossie) reads it back into the IR too; `text` the value survives only as prose folded into a description or comment; `--` not emitted (see [Gaps vs. limits](#gaps-vs-limits)). -| IR concept | `dbt` | `cortex` | `snowflake-semantic-view` | `supersimple` | `nao-yaml` | `nao-context-rules` | `databricks-metric-view` | `lightdash` | `ossie` | -|---|---|---|---|---|---|---|---|---|---| -| Table | `models:` + `semantic_models:` `<->` | `tables[].base_table` | `tables (...)` | one file per model | `--` | "Table reference" (if described) | `source` (+ `joins[].source` for referenced tables) | `models[]` entry | `datasets[]` `<->` | -| Table source (physical address) | `--` (warned) | `base_table` `database`/`schema`/`table` (three-part references only) | the `TABLES` clause reference | `table:` | `text` (into `notes:`) | `text` (into the Table reference entry) | `source:` + `joins[].source` | `--` (warned) | `source:` on the dataset `<->` | -| Column / dimension | column + `dimensions type: categorical` `<->` | `dimensions[]` | `dimensions (...)` | `properties` | `dimensions[]` (deduped) | listed if described | `fields[]` | column + `meta.dimension` | `fields[]` with `dimension.is_time: false` `<->` | -| Time dimension | `dimensions type: time` + `agg_time_dimension` `<->` | `time_dimensions[]` | plain dimension (not marked as time) | `properties` (Date) | `dimensions type: date` | with dimensions | plain `fields[]` entry (not marked as time) | column + `meta.dimension type: date/timestamp` | `fields[]` with `dimension.is_time: true` `<->` | -| Data type | column `data_type` `<->` | `data_type` | `--` | property `type` | `--` | `--` | `--` | `meta.dimension.type` only where confidently inferable, else omitted | `datatype` (logical enum) `<->` | -| Primary key | `primary_key` constraint + primary entity `<->` | `primary_key` | `primary key (...)` | `primary_key` | `--` | `--` | `--` | `meta.primary_key` (single column only; composite degrades) | `primary_key: []` (composite supported) `<->` | -| Relationship / join | `relationships` test on the FK column `<->` | `relationships[]` | `relationships (...) references` | `relations` (hasMany, join_key) | `--` | "Joins & routing" | `joins[]` (quoted `"on":` condition) | `meta.joins[]` (`sql_on` with `${table.col}` refs) | `relationships[]` (`from`/`to`, composite supported) `<->` | -| Description | `description` `<->` | `description` | `comment='...'` | `description` | `description` (field/metric) | prose | `comment` (field/measure/view) | `description` | `description` `<->` | -| Table synonyms | model `meta.synonyms` `<->` | `synonyms:` on the table | `with synonyms (...)` on the table | `text` (into the model description) | `text` (into `notes:`) | `text` (into the Table reference entry) | `text` (into the view `comment`) | `text` (into the model description) | `ai_context.synonyms` on the dataset `<->` | -| Synonyms | `meta.synonyms` on the column `<->` | `synonyms:` | `with synonyms (...)` | `--` (gap) | `text` (into description) | `text` (into description) | `synonyms:` (capped at 10) | `text` (into the column description) | `ai_context.synonyms` `<->` | -| Enum / allowed values | `accepted_values` test + `meta.enum` `<->` | `sample_values` + `text` | `text` (into comment) | `text` (into description) | `values:` | "Allowed values" | `text` (into comment) | `text` (into the column description) | `text` (into the field description) | -| Simple metric (aggregation) | `measures` + `metrics type: simple` `<->` | `facts[]` | `metrics (...)` | metric aggregation | metric `source{table,column,aggregation}` | "Key metrics reference" | `measures[]` | column-level `meta.metrics` | model-level `metrics[]` + a `fields[]` entry for the column `<->` | -| Ratio / derived metric | `type: ratio` / `type: derived` `<->` | `expr` (rendered SQL) | inline SQL in `metrics (...)` | division ratio -> pipeline; other arithmetic -> `NOTES.md` | `type: derived`, `formula` | rendered SQL | inline SQL in `measures[].expr` | model-level `meta.metrics` `type: number` (reference-only; filtered/compound aggregates and cross-table refs degrade) | model-level `metrics[]` (rendered SQL) `<->` | +| IR concept | `dbt` | `cortex` | `snowflake-semantic-view` | `supersimple` | `nao-yaml` | `nao-context-rules` | `databricks-metric-view` | `lightdash` | `ossie` | `slayer` | +|---|---|---|---|---|---|---|---|---|---|---| +| Table | `models:` + `semantic_models:` `<->` | `tables[].base_table` | `tables (...)` | one file per model | `--` | "Table reference" (if described) | `source` (+ `joins[].source` for referenced tables) | `models[]` entry | `datasets[]` `<->` | one file per model | +| Table source (physical address) | `--` (warned) | `base_table` `database`/`schema`/`table` (three-part references only) | the `TABLES` clause reference | `table:` | `text` (into `notes:`) | `text` (into the Table reference entry) | `source:` + `joins[].source` | `--` (warned) | `source:` on the dataset `<->` | `sql_table:` (any reference verbatim; a query goes to `sql:`, a query-backed model) | +| Column / dimension | column + `dimensions type: categorical` `<->` | `dimensions[]` | `dimensions (...)` | `properties` | `dimensions[]` (deduped) | listed if described | `fields[]` | column + `meta.dimension` | `fields[]` with `dimension.is_time: false` `<->` | `columns[]` | +| Time dimension | `dimensions type: time` + `agg_time_dimension` `<->` | `time_dimensions[]` | plain dimension (not marked as time) | `properties` (Date) | `dimensions type: date` | with dimensions | plain `fields[]` entry (not marked as time) | column + `meta.dimension type: date/timestamp` | `fields[]` with `dimension.is_time: true` `<->` | `columns[]` `type: DATE/TIMESTAMP` + model `default_time_dimension` | +| Data type | column `data_type` `<->` | `data_type` | `--` | property `type` | `--` | `--` | `--` | `meta.dimension.type` only where confidently inferable, else omitted | `datatype` (logical enum) `<->` | column `type` (sqlglot names) | +| Primary key | `primary_key` constraint + primary entity `<->` | `primary_key` | `primary key (...)` | `primary_key` | `--` | `--` | `--` | `meta.primary_key` (single column only; composite degrades) | `primary_key: []` (composite supported) `<->` | `primary_key: true` per column (composite supported) | +| Relationship / join | `relationships` test on the FK column `<->` | `relationships[]` | `relationships (...) references` | `relations` (hasMany, join_key) | `--` | "Joins & routing" | `joins[]` (quoted `"on":` condition) | `meta.joins[]` (`sql_on` with `${table.col}` refs) | `relationships[]` (`from`/`to`, composite supported) `<->` | `joins[]` on the FK model (one per target model; a second FK to the same target degrades) | +| Description | `description` `<->` | `description` | `comment='...'` | `description` | `description` (field/metric) | prose | `comment` (field/measure/view) | `description` | `description` `<->` | `description` | +| Table synonyms | model `meta.synonyms` `<->` | `synonyms:` on the table | `with synonyms (...)` on the table | `text` (into the model description) | `text` (into `notes:`) | `text` (into the Table reference entry) | `text` (into the view `comment`) | `text` (into the model description) | `ai_context.synonyms` on the dataset `<->` | `meta.synonyms` on the model | +| Synonyms | `meta.synonyms` on the column `<->` | `synonyms:` | `with synonyms (...)` | `--` (gap) | `text` (into description) | `text` (into description) | `synonyms:` (capped at 10) | `text` (into the column description) | `ai_context.synonyms` `<->` | `meta.synonyms` on the column | +| Enum / allowed values | `accepted_values` test + `meta.enum` `<->` | `sample_values` + `text` | `text` (into comment) | `text` (into description) | `values:` | "Allowed values" | `text` (into comment) | `text` (into the column description) | `text` (into the field description) | `text` (into the column description) | +| Simple metric (aggregation) | `measures` + `metrics type: simple` `<->` | `facts[]` | `metrics (...)` | metric aggregation | metric `source{table,column,aggregation}` | "Key metrics reference" | `measures[]` | column-level `meta.metrics` | model-level `metrics[]` + a `fields[]` entry for the column `<->` | `measures[]` `formula: col:agg` (+ hidden expression columns) | +| Ratio / derived metric | `type: ratio` / `type: derived` `<->` | `expr` (rendered SQL) | inline SQL in `metrics (...)` | division ratio -> pipeline; other arithmetic -> `NOTES.md` | `type: derived`, `formula` | rendered SQL | inline SQL in `measures[].expr` | model-level `meta.metrics` `type: number` (reference-only; filtered/compound aggregates and cross-table refs degrade) | model-level `metrics[]` (rendered SQL) `<->` | `measures[]` formula DSL (arithmetic, bare refs, `cumsum`; filtered aggs via hidden `filter:` columns) | ## Gaps vs. limits @@ -93,6 +94,17 @@ and nao dimensions carry a structured `values:` list metric view declares `source`/`joins`/`fields`/`measures` only, no primary-key key; Unity Catalog constraints on the underlying table are the authority there instead. +- **Role-playing dimensions in `slayer`.** A SLayer model declares at most one + join per `target_model` (its engine rejects a duplicate target with different + `join_pairs`, and there is no alias construct to disambiguate with), so of two + FKs to the same dimension the first is emitted and the rest degrade to a note. + Also renamed rather than dropped: a measure whose name collides with a column + (columns and measures share one namespace per model) or shadows a SLayer + transform name (`cumsum`, `first`, `last`, ...) gets a `_measure` suffix and a + note. Per-metric agg-time grains and slice-by dimension restrictions have no + slot — SLayer groups any measure by any dimension at query time — and are + reported. Bounded time windows and conversion metrics are not expressible and + degrade; unbounded cumulative windows emit as `cumsum(...)`. - **Composite primary keys in `lightdash`.** `meta.primary_key` takes a single column name, so a multi-column key has nowhere to go and degrades to a note. - **Synonyms and enum values in `lightdash`.** Lightdash has no synonym or diff --git a/dialect/slayer.go b/dialect/slayer.go new file mode 100644 index 0000000..2a894a1 --- /dev/null +++ b/dialect/slayer.go @@ -0,0 +1,610 @@ +package dialect + +import ( + "bytes" + "fmt" + "os" + "path/filepath" + "slices" + "sort" + "strings" + + "github.com/benchouse/semglot/ir" + "gopkg.in/yaml.v3" +) + +func init() { Register(slayerDialect{}) } + +// slayerDialect emits one SLayer model YAML per table — the file shape +// `slayer models create ` reads and `slayer models show` prints +// (github.com/MotleyAI/slayer, SlayerModel schema version 8). +// +// SLayer chooses aggregations at query time (`revenue:sum` colon syntax), so +// the IR maps unusually directly: a dimension is a Column, a measure is a +// Column plus a named ModelMeasure `:`, and a metric lowers to +// SLayer's formula DSL — arithmetic over colon references, bare references to +// other measures, and `cumsum(...)` for unbounded cumulative windows. A +// filtered aggregation becomes a synthesized hidden Column carrying SLayer's +// `filter:` (applied inside CASE WHEN at aggregation time), which is exactly +// the filtered-aggregate semantics. +// +// Zero value is usable; the build command sets identity from the profile. +type slayerDialect struct { + // Database and Schema reconstruct a physical address for a table that + // declares no Source of its own (dbt models resolve theirs through ref()). + Database, Schema string + // DataSource is the SLayer datasource name the emitted models are keyed + // under ((data_source, name) is a model's identity in SLayer). Taken from + // the profile's model-name option; "" is valid — `slayer import` flows + // assign the datasource at import time. + DataSource string +} + +func (slayerDialect) Name() string { return "slayer" } + +// WithOptions lets the CLI pass the profile's identity fields. +func (slayerDialect) WithOptions(o Options) Emitter { + return slayerDialect{Database: o.Database, Schema: o.Schema, DataSource: o.Name} +} + +// slayerVersion is the SlayerModel schema version emitted. Pinned, never computed. +const slayerVersion = 8 + +// ---- SLayer YAML shapes (SlayerModel, slayer/core/models.py) ---- + +type slModel struct { + Version int `yaml:"version"` + Name string `yaml:"name"` + SQLTable string `yaml:"sql_table,omitempty"` + SQL string `yaml:"sql,omitempty"` + DataSource string `yaml:"data_source"` + Columns []slColumn `yaml:"columns"` + Measures []slMeasure `yaml:"measures,omitempty"` + Joins []slJoin `yaml:"joins,omitempty"` + DefaultTimeDimension string `yaml:"default_time_dimension,omitempty"` + Description string `yaml:"description,omitempty"` + Meta map[string]any `yaml:"meta,omitempty"` +} + +type slColumn struct { + Name string `yaml:"name"` + SQL string `yaml:"sql,omitempty"` + Type string `yaml:"type,omitempty"` + PrimaryKey bool `yaml:"primary_key,omitempty"` + Description string `yaml:"description,omitempty"` + Hidden bool `yaml:"hidden,omitempty"` + Filter string `yaml:"filter,omitempty"` + Meta map[string]any `yaml:"meta,omitempty"` +} + +type slMeasure struct { + Formula string `yaml:"formula"` + Name string `yaml:"name"` + Label string `yaml:"label,omitempty"` + Description string `yaml:"description,omitempty"` + Type string `yaml:"type,omitempty"` +} + +type slJoin struct { + TargetModel string `yaml:"target_model"` + JoinPairs [][]string `yaml:"join_pairs"` + JoinType string `yaml:"join_type"` + Description string `yaml:"description,omitempty"` +} + +// slayerAggs is SLayer's built-in aggregation vocabulary +// (slayer/core/enums.py BUILTIN_AGGREGATIONS). A source aggregation outside it +// degrades loudly rather than emitting a formula SLayer would reject. +var slayerAggs = map[string]bool{ + "sum": true, "avg": true, "min": true, "max": true, + "count": true, "count_distinct": true, "count_distinct_approx": true, + "first": true, "last": true, "weighted_avg": true, + "median": true, "percentile": true, + "stddev_samp": true, "stddev_pop": true, "var_samp": true, "var_pop": true, + "corr": true, "covar_samp": true, "covar_pop": true, +} + +// slayerReservedMeasureNames are SLayer transform names a saved measure must +// not shadow (rejected at model construction; docs/concepts/formulas.md). +var slayerReservedMeasureNames = map[string]bool{ + "cumsum": true, "change": true, "time_shift": true, "lag": true, + "lead": true, "rank": true, "percent_rank": true, "dense_rank": true, + "ntile": true, "first": true, "last": true, "change_pct": true, + "consecutive_periods": true, +} + +// slayerType maps an IR/dbt data type string to SLayer's DataType vocabulary +// (TEXT/INT/DOUBLE/BOOLEAN/DATE/TIMESTAMP — sqlglot type names). An empty +// source type falls back by role: TIMESTAMP for a time dimension, TEXT +// otherwise (TEXT stays groupable, SLayer's own lenient default). +func slayerType(t string, isTime bool) string { + switch strings.ToLower(strings.TrimSpace(t)) { + case "int", "integer", "bigint", "smallint", "tinyint": + return "INT" + case "number", "float", "double", "double precision", "real", "numeric", "decimal": + return "DOUBLE" + case "boolean", "bool": + return "BOOLEAN" + case "date": + return "DATE" + case "timestamp", "datetime", "time", "timestamp_ntz", "timestamp_tz", "timestamp_ltz": + return "TIMESTAMP" + case "": + if isTime { + return "TIMESTAMP" + } + return "TEXT" + default: + return "TEXT" + } +} + +// synonymMeta returns the meta map carrying synonyms, or nil when there are +// none. SLayer's free-form meta dict is the structural home for them — the +// same meta.synonyms convention the IR sources them from in dbt. +func synonymMeta(syn []string) map[string]any { + if len(syn) == 0 { + return nil + } + return map[string]any{"synonyms": append([]string(nil), syn...)} +} + +// unqualifyExpr strips table qualification from Col references owned by table, +// for rendering into SLayer's Mode-A SQL slots (Column.sql / Column.filter), +// where a same-model column is referenced bare (dots there denote join paths, +// not table qualifiers). +func unqualifyExpr(e ir.Expr, table string) ir.Expr { + switch n := e.(type) { + case ir.Col: + if n.Table == table { + return ir.Col{Name: n.Name} + } + return n + case ir.Binary: + return ir.Binary{Op: n.Op, Left: unqualifyExpr(n.Left, table), Right: unqualifyExpr(n.Right, table)} + case ir.Agg: + out := n + if n.Arg != nil { + out.Arg = unqualifyExpr(n.Arg, table) + } + if n.Filter != nil { + out.Filter = unqualifyExpr(n.Filter, table) + } + return out + default: + return e + } +} + +// Emit does not mutate m; it reads m.Notes and accumulates its own degrade +// notes locally before writing the combined text to NOTES.md. +func (s slayerDialect) Emit(m *ir.Model, dir string) ([]string, error) { + if err := os.MkdirAll(dir, 0o755); err != nil { + return nil, err + } + var degradeNotes []string + resolve := metricResolver(m) + + models := map[string]bool{} + for _, t := range m.Tables { + models[t.Name] = true + } + // A SLayer join is written on the model that holds the foreign key (the + // Left endpoint), from inside the table loop below, so a relationship + // whose Left endpoint names no table is iterated by nothing and must be + // reported here instead. + for _, r := range m.Relationships { + if !models[r.Left] { + degradeNotes = append(degradeNotes, relNotEmittedWarning("slayer", r, relEndpointMissing(r.Left, + "a slayer join is written on the model that holds the foreign key"))) + } + } + + // Measure names must be project-unique for cross-table bare references to + // resolve; rename ones SLayer would reject up front — a name that shadows + // a transform, or one that collides with a column on the owning model + // (columns and measures share one namespace per SLayer model, enforced at + // model construction) — so Ref lowering below can target the emitted name. + tableColumns := map[string]map[string]bool{} + for _, t := range m.Tables { + cols := map[string]bool{} + for _, d := range t.Dimensions { + cols[d.Name] = true + } + for _, d := range t.TimeDimensions { + cols[d.Name] = true + } + for _, meas := range t.Measures { + if isIdent(meas.Expr) { + cols[meas.Expr] = true + } + } + tableColumns[t.Name] = cols + } + // renameMeasure resolves what SLayer will accept name as on table owner, + // with a degrade note explaining each forced rename. + renameMeasure := func(kind, name, owner string) string { + out := name + if slayerReservedMeasureNames[strings.ToLower(out)] { + out += "_measure" + degradeNotes = append(degradeNotes, fmt.Sprintf( + "%s %q: renamed to %q; its name shadows a slayer transform, which slayer rejects on a saved measure", kind, name, out)) + } + for tableColumns[owner][out] { + renamed := out + "_measure" + degradeNotes = append(degradeNotes, fmt.Sprintf( + "%s %q: renamed to %q; columns and measures share one namespace in a slayer model, and %q names a column on %q", kind, name, renamed, out, owner)) + out = renamed + } + return out + } + measureName := map[string]string{} + metricOwner := map[string]string{} + for _, t := range m.Tables { + for _, mt := range t.Metrics { + measureName[mt.Name] = renameMeasure("metric", mt.Name, t.Name) + metricOwner[mt.Name] = t.Name + } + } + + for _, t := range m.Tables { + model := slModel{ + Version: slayerVersion, + Name: t.Name, + DataSource: s.DataSource, + Description: t.Description, + Meta: synonymMeta(t.Synonyms), + DefaultTimeDimension: t.Grain, + } + + // Prefer the source dialect's own declared physical address. SLayer's + // sql_table holds one opaque dotted reference, so a genuine table + // reference is used verbatim regardless of its dot-part count. A + // QUERY source (which the OSI spec permits) has a first-class home + // here that most targets lack: SLayer's query-backed models (`sql:`). + switch { + case t.Source != "" && looksLikeQuery(t.Source): + model.SQL = t.Source + case t.Source != "": + model.SQLTable = t.Source + default: + parts := []string{} + if s.Database != "" { + parts = append(parts, s.Database) + } + if s.Schema != "" { + parts = append(parts, s.Schema) + } + model.SQLTable = strings.Join(append(parts, t.Name), ".") + } + + pk := map[string]bool{} + for _, k := range t.PrimaryKey { + pk[k] = true + } + colIndex := map[string]int{} // column name -> index in model.Columns + addColumn := func(c slColumn) { + if _, ok := colIndex[c.Name]; ok { + return + } + colIndex[c.Name] = len(model.Columns) + model.Columns = append(model.Columns, c) + } + fieldColumn := func(f ir.Field, isTime bool) slColumn { + c := slColumn{ + Name: f.Name, + Type: slayerType(f.DataType, isTime), + PrimaryKey: pk[f.Name] || pk[f.Expr], + Description: appendClause(f.Description, enumClause(f.Enum)), + Meta: synonymMeta(f.Synonyms), + } + if f.Expr != "" && f.Expr != f.Name { + c.SQL = f.Expr + } + return c + } + for _, d := range t.Dimensions { + addColumn(fieldColumn(d, false)) + } + for _, d := range t.TimeDimensions { + addColumn(fieldColumn(d, true)) + } + + // mintColumn synthesizes a hidden expression-backed column (for a + // compound or filtered aggregation arg) and returns its name. Two + // mints with the same SQL and filter share one column — a dbt measure + // and a metric over the same CASE expression are common — and a + // synthesized column always aggregates, so an undeclared type + // defaults to DOUBLE (TEXT would drive a wrong CAST at query time). + minted := map[[2]string]string{} + mintColumn := func(base string, c slColumn) string { + sig := [2]string{c.SQL, c.Filter} + if name, ok := minted[sig]; ok { + return name + } + name := base + for { + if _, taken := colIndex[name]; !taken { + break + } + name += "_expr" + } + c.Name = name + c.Hidden = true + if c.Type == "" { + c.Type = "DOUBLE" + } + addColumn(c) + minted[sig] = name + return name + } + + // IR measures: the backing column plus a named ModelMeasure over it. + for _, meas := range t.Measures { + agg := mapAgg(meas.Agg) + if !slayerAggs[agg] { + degradeNotes = append(degradeNotes, fmt.Sprintf( + "measure %q not emitted: aggregation %q is not a slayer built-in aggregation", meas.Name, meas.Agg)) + continue + } + expr := meas.Expr + if expr == "" { + expr = meas.Name + } + var col string + if isIdent(expr) { + col = expr + addColumn(fieldColumn(ir.Field{Name: expr, Expr: expr, DataType: meas.DataType, Description: meas.Description}, false)) + } else { + typ := "" + if meas.DataType != "" { + typ = slayerType(meas.DataType, false) + } + col = mintColumn(meas.Name+"_expr", slColumn{SQL: expr, Type: typ}) + } + name := renameMeasure("measure", meas.Name, t.Name) + model.Measures = append(model.Measures, slMeasure{ + Formula: col + ":" + agg, + Name: name, + Description: appendClause(meas.Description, synonymClause(meas.Synonyms)), + Type: measureResultType(meas.DataType, agg), + }) + } + + // Metrics lower to SLayer's formula DSL. + for _, mt := range t.Metrics { + formula, ok := s.lowerFormula(mt.Def, t.Name, measureName, metricOwner, resolve, &model, mintColumn, °radeNotes, mt.Name) + if !ok { + continue + } + if mt.Grain != "" && mt.Grain != t.Grain { + degradeNotes = append(degradeNotes, fmt.Sprintf( + "metric %q: per-metric agg-time grain %q not emitted; slayer carries only a model-level default_time_dimension", mt.Name, mt.Grain)) + } + if mt.Dimensions != nil { + degradeNotes = append(degradeNotes, fmt.Sprintf( + "metric %q: slice-by dimension restriction %v not emitted; slayer measures group by any dimension at query time", mt.Name, mt.Dimensions)) + } + model.Measures = append(model.Measures, slMeasure{ + Formula: formula, + Name: measureName[mt.Name], + Label: mt.Label, + Description: appendClause(mt.Description, synonymClause(mt.Synonyms)), + }) + } + + // Joins live on the FK-holding (Left) model. The join's declared name + // and synonyms have no structural slot on a SLayer join, but its + // description is a real prose home, so they fold there rather than + // warn (the nao-yaml precedent). SLayer permits at most ONE join per + // target model (its engine rejects a duplicate target with different + // join_pairs), so a role-playing dimension keeps its first FK and + // reports the rest — there is no alias construct to disambiguate with. + joinTarget := map[string]bool{} + for _, r := range m.Relationships { + if r.Left != t.Name { + continue + } + if joinTarget[r.Right] { + degradeNotes = append(degradeNotes, relNotEmittedWarning("slayer", r, fmt.Sprintf( + "model %q already joins %q, and a slayer model can declare at most one join per target model", t.Name, r.Right))) + continue + } + if !relHasColumns(r) { + degradeNotes = append(degradeNotes, relNotEmittedWarning("slayer", r, + "it declares no column pairs, and a slayer join requires join_pairs")) + continue + } + joinTarget[r.Right] = true + pairs := make([][]string, len(r.Columns)) + for i, cp := range r.Columns { + pairs[i] = []string{cp.Left, cp.Right} + } + model.Joins = append(model.Joins, slJoin{ + TargetModel: r.Right, + JoinPairs: pairs, + JoinType: "left", + Description: relIdentityClause(r), + }) + } + + // default_time_dimension must name a column on the model. + if model.DefaultTimeDimension != "" { + if _, ok := colIndex[model.DefaultTimeDimension]; !ok { + degradeNotes = append(degradeNotes, fmt.Sprintf( + "table %q: default time dimension %q not emitted; it names no column on the model", t.Name, model.DefaultTimeDimension)) + model.DefaultTimeDimension = "" + } + } + + var buf bytes.Buffer + enc := yaml.NewEncoder(&buf) + enc.SetIndent(2) + if err := enc.Encode(model); err != nil { + return degradeNotes, err + } + if err := enc.Close(); err != nil { + return degradeNotes, err + } + if err := os.WriteFile(filepath.Join(dir, t.Name+".yaml"), buf.Bytes(), 0o644); err != nil { + return degradeNotes, err + } + } + + allNotes := append(slices.Clone(m.Notes), degradeNotes...) + if len(allNotes) > 0 { + sort.Strings(allNotes) + var sb strings.Builder + sb.WriteString("# Not transpiled to slayer\n\n") + for _, n := range allNotes { + sb.WriteString("- " + n + "\n") + } + if err := os.WriteFile(filepath.Join(dir, "NOTES.md"), []byte(sb.String()), 0o644); err != nil { + return degradeNotes, err + } + } + return degradeNotes, nil +} + +// measureResultType maps a measure's declared type through its aggregation: +// counts are INT regardless of input; otherwise the input type maps through +// slayerType, left empty (SLayer infers) when the source declared none. +func measureResultType(dataType, agg string) string { + if agg == "count" || agg == "count_distinct" || agg == "count_distinct_approx" { + return "INT" + } + if dataType == "" { + return "" + } + return slayerType(dataType, false) +} + +// lowerFormula lowers a metric-definition AST to SLayer's formula DSL: +// colon-aggregation references (`revenue:sum`, `*:count`, dotted for a joined +// model), bare references to other saved measures, arithmetic, and cumsum() +// for an unbounded cumulative window. Compound and filtered aggregation args +// synthesize hidden backing columns via mint. Returns ok=false (with a +// degrade note) for a definition SLayer cannot express. +func (s slayerDialect) lowerFormula(e ir.Expr, table string, measureName map[string]string, + metricOwner map[string]string, resolve func(string) (ir.Expr, bool), model *slModel, + mint func(string, slColumn) string, notes *[]string, metric string) (string, bool) { + switch n := e.(type) { + case ir.Lit: + return n.Value, true + case ir.Ref: + // A bare measure name only resolves against the OWNING model's saved + // measures, so it is emitted only for a same-model metric. A ref to a + // metric on another model (or one outside the model) inlines its + // definition instead: the inlined Aggs qualify their columns with the + // owning table, which the Agg case below lowers to SLayer's dotted + // join-path syntax. + if name, ok := measureName[n.Metric]; ok && metricOwner[n.Metric] == table { + return name, true + } + if def, ok := resolve(n.Metric); ok { + return s.lowerFormula(def, table, measureName, metricOwner, resolve, model, mint, notes, metric) + } + *notes = append(*notes, fmt.Sprintf("metric %q not emitted: it references unknown metric %q", metric, n.Metric)) + return "", false + case ir.Agg: + agg := mapAgg(n.Func) + if !slayerAggs[agg] { + *notes = append(*notes, fmt.Sprintf("metric %q not emitted: aggregation %q is not a slayer built-in aggregation", metric, n.Func)) + return "", false + } + crossTable := n.Table != "" && n.Table != table + var col string + switch a := n.Arg.(type) { + case nil: + if agg != "count" { + *notes = append(*notes, fmt.Sprintf("metric %q not emitted: %s(*) is not expressible in slayer", metric, agg)) + return "", false + } + if crossTable { + *notes = append(*notes, fmt.Sprintf("metric %q not emitted: count(*) over joined model %q is not expressible in slayer", metric, n.Table)) + return "", false + } + if n.Filter == nil { + return "*:count", true + } + // Filtered COUNT(*): count a synthesized always-1 column instead, + // so the filter has a column to hang off. + col = mint(metric+"_base", slColumn{SQL: "1", Type: "INT"}) + case ir.Col: + if a.Table != "" && a.Table != table { + // A joined model's column: SLayer's dotted join-path syntax, + // resolvable when the emitted join graph connects the models. + if n.Filter != nil { + *notes = append(*notes, fmt.Sprintf("metric %q not emitted: a filtered aggregation over joined column %s.%s is not expressible in slayer", metric, a.Table, a.Name)) + return "", false + } + return a.Table + "." + a.Name + ":" + agg, true + } + col = a.Name + case ir.Raw: + // A raw fragment references its owning table's columns bare, so a + // column synthesized from it is only correct on that same model. + if crossTable { + *notes = append(*notes, fmt.Sprintf("metric %q not emitted: an expression aggregation over joined model %q is not expressible in slayer", metric, n.Table)) + return "", false + } + col = mint(metric+"_expr", slColumn{SQL: a.SQL, Type: "DOUBLE"}) + default: + *notes = append(*notes, fmt.Sprintf("metric %q not emitted: aggregation argument is not a column or expression", metric)) + return "", false + } + if n.Filter != nil { + // SLayer's Column.filter applies inside CASE WHEN at aggregation + // time — exactly a filtered aggregate. Synthesize a hidden column + // carrying the base expression and the predicate. + base := model.columnSQL(col) + cond := renderSQL(unqualifyExpr(n.Filter, table), resolve) + col = mint(metric+"_filtered", slColumn{SQL: base, Type: "DOUBLE", Filter: cond}) + } + return col + ":" + agg, true + case ir.Binary: + l, ok := s.lowerFormula(n.Left, table, measureName, metricOwner, resolve, model, mint, notes, metric) + if !ok { + return "", false + } + r, ok := s.lowerFormula(n.Right, table, measureName, metricOwner, resolve, model, mint, notes, metric) + if !ok { + return "", false + } + if _, c := n.Left.(ir.Binary); c { + l = "(" + l + ")" + } + if _, c := n.Right.(ir.Binary); c { + r = "(" + r + ")" + } + return l + " " + n.Op + " " + r, true + case ir.Window: + if n.Window != "" { + *notes = append(*notes, fmt.Sprintf("metric %q not emitted: a bounded %q window is not expressible in slayer", metric, n.Window)) + return "", false + } + base, ok := s.lowerFormula(n.Base, table, measureName, metricOwner, resolve, model, mint, notes, metric) + if !ok { + return "", false + } + return "cumsum(" + base + ")", true + case ir.Conversion: + *notes = append(*notes, fmt.Sprintf("metric %q not emitted: conversion/funnel metric is not expressible in slayer", metric)) + return "", false + default: + *notes = append(*notes, fmt.Sprintf("metric %q not emitted: definition is not expressible in slayer", metric)) + return "", false + } +} + +// columnSQL returns the SQL expression behind an emitted column name (the +// column's own name when it is a plain physical column), for re-wrapping into +// a synthesized filtered column. +func (m *slModel) columnSQL(name string) string { + for _, c := range m.Columns { + if c.Name == name && c.SQL != "" { + return c.SQL + } + } + return name +} diff --git a/dialect/slayer_test.go b/dialect/slayer_test.go new file mode 100644 index 0000000..aabe798 --- /dev/null +++ b/dialect/slayer_test.go @@ -0,0 +1,211 @@ +package dialect + +import ( + "os" + "path/filepath" + "strings" + "testing" + + "github.com/benchouse/semglot/ir" + "gopkg.in/yaml.v3" +) + +func emitSlayer(t *testing.T, m *ir.Model) (dir string, warnings []string) { + t.Helper() + dir = t.TempDir() + w, err := (slayerDialect{Database: "ANALYTICS", Schema: "MAIN", DataSource: "warehouse"}).Emit(m, dir) + if err != nil { + t.Fatalf("Emit: %v", err) + } + return dir, w +} + +func slayerFixture() *ir.Model { + return &ir.Model{ + Tables: []ir.Table{ + { + Name: "fct_orders", Description: "Orders.", Synonyms: []string{"purchases"}, + PrimaryKey: []string{"order_id"}, + Grain: "order_date", + Dimensions: []ir.Field{ + {Name: "order_id", Expr: "order_id", DataType: "number"}, + {Name: "status", Expr: "status", Enum: []ir.EnumValue{{Value: "open"}, {Value: "closed", Description: "Fulfilled"}}}, + {Name: "is_refunded", Expr: "is_refunded", DataType: "boolean", Synonyms: []string{"refund flag"}}, + }, + TimeDimensions: []ir.Field{{Name: "order_date", Expr: "ordered_at", DataType: "date"}}, + Measures: []ir.Measure{ + {Field: ir.Field{Name: "order_net_booked_amount", Expr: "order_net_booked", DataType: "float"}, Agg: "sum"}, + {Field: ir.Field{Name: "refund_flag_value", Expr: "case when is_refunded then 1 else 0 end", DataType: "float"}, Agg: "sum"}, + // Named after its own backing column: columns and measures + // share a namespace in slayer, so this must rename. + {Field: ir.Field{Name: "order_net_booked", Expr: "order_net_booked", DataType: "float"}, Agg: "avg"}, + }, + Metrics: []ir.Metric{ + {Name: "net_revenue", Label: "Net revenue", Description: "Net booked revenue.", + Def: ir.Agg{Func: "sum", Table: "fct_orders", Arg: ir.Col{Table: "fct_orders", Name: "order_net_booked"}}}, + {Name: "orders", Def: ir.Agg{Func: "count", Table: "fct_orders"}}, + {Name: "large_orders", Def: ir.Agg{Func: "count", Table: "fct_orders", Arg: ir.Col{Table: "fct_orders", Name: "order_id"}, + Filter: ir.Binary{Op: ">", Left: ir.Col{Table: "fct_orders", Name: "order_net_booked"}, Right: ir.Lit{Value: "100"}}}}, + {Name: "refund_rate", Label: "Refund rate", + Def: ir.Binary{Op: "/", Left: ir.Ref{Metric: "refund_flag_metric"}, Right: ir.Ref{Metric: "orders"}}}, + {Name: "refund_flag_metric", + Def: ir.Agg{Func: "sum", Table: "fct_orders", Arg: ir.Raw{SQL: "case when is_refunded then 1 else 0 end", Columns: []string{"is_refunded"}}}}, + {Name: "cumulative_revenue", Def: ir.Window{Base: ir.Ref{Metric: "net_revenue"}, Grain: "month"}}, + {Name: "rolling_revenue", Def: ir.Window{Base: ir.Ref{Metric: "net_revenue"}, Window: "30 days"}}, + {Name: "first", Def: ir.Agg{Func: "min", Table: "fct_orders", Arg: ir.Col{Table: "fct_orders", Name: "order_date"}}}, + {Name: "restricted", Grain: "week", Dimensions: []string{"status"}, + Def: ir.Agg{Func: "avg", Table: "fct_orders", Arg: ir.Col{Table: "fct_orders", Name: "order_net_booked"}}}, + }, + }, + { + Name: "dim_customer", Source: "PROD.CRM.customers", Description: "Customers.", + PrimaryKey: []string{"customer_sk"}, + Dimensions: []ir.Field{{Name: "customer_sk", Expr: "customer_sk", DataType: "number"}}, + }, + { + Name: "stg_events", Source: "SELECT * FROM raw.events WHERE kind = 'web'", + Dimensions: []ir.Field{{Name: "event_id", Expr: "event_id"}}, + }, + }, + Relationships: []ir.Relationship{ + {Name: "order_customer", Synonyms: []string{"who ordered"}, + Left: "fct_orders", Right: "dim_customer", + Columns: []ir.ColumnPair{{Left: "customer_sk", Right: "customer_sk"}}}, + {Left: "fct_missing", Right: "dim_customer", + Columns: []ir.ColumnPair{{Left: "x", Right: "x"}}}, + }, + } +} + +func TestSlayerEmit(t *testing.T) { + dir, warnings := emitSlayer(t, slayerFixture()) + + orders := readFile(t, filepath.Join(dir, "fct_orders.yaml")) + for _, want := range []string{ + "version: 8", + "name: fct_orders", + "sql_table: ANALYTICS.MAIN.fct_orders", // no declared source -> profile reconstruction + "data_source: warehouse", + "default_time_dimension: order_date", + "description: Orders.", + "synonyms:", // table meta.synonyms + "- purchases", + "primary_key: true", + "sql: ordered_at", // time dim with expr != name + "type: DATE", + "type: BOOLEAN", + "Values: open; closed = Fulfilled.", // enum folded into description + // identifier-backed measure: column + named measure over it + "formula: order_net_booked:sum", + "name: order_net_booked_amount", + "type: DOUBLE", + // compound measure expr -> synthesized hidden backing column + "sql: case when is_refunded then 1 else 0 end", + "hidden: true", + "formula: refund_flag_value_expr:sum", + // metrics + "name: net_revenue", + "label: Net revenue", + "formula: '*:count'", + "name: orders", + // filtered aggregation -> hidden column with filter: + "filter: order_net_booked > 100", + "formula: large_orders_filtered:count", + // ratio over refs -> bare measure-name arithmetic + "formula: refund_flag_metric / orders", + // unbounded window -> cumsum + "formula: cumsum(net_revenue)", + "name: cumulative_revenue", + // reserved transform name renamed + "name: first_measure", + // measure named after its own backing column renamed + "name: order_net_booked_measure", + // join on the FK-holding model, with identity folded into description + "target_model: dim_customer", + "- - customer_sk", + "join_type: left", + "description: 'Join name: order_customer. Synonyms: who ordered.'", + } { + if !strings.Contains(orders, want) { + t.Fatalf("fct_orders.yaml missing %q:\n%s", want, orders) + } + } + + customers := readFile(t, filepath.Join(dir, "dim_customer.yaml")) + if !strings.Contains(customers, "sql_table: PROD.CRM.customers") { + t.Fatalf("dim_customer.yaml should carry the declared source verbatim:\n%s", customers) + } + + events := readFile(t, filepath.Join(dir, "stg_events.yaml")) + if !strings.Contains(events, "sql: SELECT * FROM raw.events WHERE kind = 'web'") || + strings.Contains(events, "sql_table:") { + t.Fatalf("stg_events.yaml should be a query-backed model (sql:, no sql_table:):\n%s", events) + } + + notes := readFile(t, filepath.Join(dir, "NOTES.md")) + for _, want := range []string{ + `metric "rolling_revenue" not emitted: a bounded "30 days" window is not expressible in slayer`, + `metric "first": renamed to "first_measure"`, + `measure "order_net_booked": renamed to "order_net_booked_measure"`, + `metric "restricted": per-metric agg-time grain "week" not emitted`, + `metric "restricted": slice-by dimension restriction [status] not emitted`, + `relationship fct_missing -> dim_customer: not emitted to slayer at all`, + } { + if !strings.Contains(notes, want) { + t.Fatalf("NOTES.md missing %q:\n%s", want, notes) + } + } + if len(warnings) == 0 { + t.Fatal("expected degrade warnings") + } +} + +// TestSlayerEmitValidYAML asserts every emitted model file decodes and carries +// the required SlayerModel fields with sane types. +func TestSlayerEmitValidYAML(t *testing.T) { + dir, _ := emitSlayer(t, slayerFixture()) + entries, err := os.ReadDir(dir) + if err != nil { + t.Fatal(err) + } + for _, ent := range entries { + if ent.Name() == "NOTES.md" { + continue + } + var doc map[string]any + b, err := os.ReadFile(filepath.Join(dir, ent.Name())) + if err != nil { + t.Fatal(err) + } + if err := yaml.Unmarshal(b, &doc); err != nil { + t.Fatalf("%s: invalid YAML: %v", ent.Name(), err) + } + if doc["version"] != 8 { + t.Fatalf("%s: version = %v, want 8", ent.Name(), doc["version"]) + } + if _, ok := doc["name"].(string); !ok { + t.Fatalf("%s: missing name", ent.Name()) + } + if _, ok := doc["columns"].([]any); !ok { + t.Fatalf("%s: missing columns list", ent.Name()) + } + } +} + +func TestSlayerZeroValueUsable(t *testing.T) { + m := &ir.Model{Tables: []ir.Table{{ + Name: "t", + Dimensions: []ir.Field{{Name: "id", Expr: "id"}}, + }}} + dir := t.TempDir() + if _, err := (slayerDialect{}).Emit(m, dir); err != nil { + t.Fatalf("Emit: %v", err) + } + got := readFile(t, filepath.Join(dir, "t.yaml")) + if !strings.Contains(got, "sql_table: t") { + t.Fatalf("zero-value emitter should fall back to the bare table name:\n%s", got) + } + if !strings.Contains(got, `data_source: ""`) { + t.Fatalf("data_source must be present (empty) for SlayerModel defaults:\n%s", got) + } +} diff --git a/test/integration_test.go b/test/integration_test.go index 4ad3076..9e19eef 100644 --- a/test/integration_test.go +++ b/test/integration_test.go @@ -478,6 +478,75 @@ func TestEcommerceSupersimpleGolden(t *testing.T) { } } +func TestEcommerceSlayerGolden(t *testing.T) { + e, err := dialect.AsEmitter("slayer") + if err != nil { + t.Fatalf("AsEmitter: %v", err) + } + if c, ok := e.(dialect.Configurable); ok { + e = c.WithOptions(dialect.Options{Database: "ANALYTICS", Schema: "MAIN", Name: "ecommerce"}) + } + p, err := dialect.AsParser("dbt") + if err != nil { + t.Fatal(err) + } + model, err := p.Parse(sourceDirs...) + if err != nil { + t.Fatalf("parse: %v", err) + } + out := t.TempDir() + if _, err := e.Emit(model, out); err != nil { + t.Fatalf("emit: %v", err) + } + + goldenDir := "models/ecommerce/dbt/slayer" + entries, err := os.ReadDir(out) + if err != nil { + t.Fatal(err) + } + if os.Getenv("UPDATE_GOLDEN") == "1" { + _ = os.MkdirAll(goldenDir, 0o755) + } + for _, ent := range entries { + got, err := os.ReadFile(filepath.Join(out, ent.Name())) + if err != nil { + t.Fatal(err) + } + gpath := filepath.Join(goldenDir, ent.Name()) + if os.Getenv("UPDATE_GOLDEN") == "1" { + if err := os.WriteFile(gpath, got, 0o644); err != nil { + t.Fatal(err) + } + } + want, err := os.ReadFile(gpath) + if err != nil { + t.Fatalf("read golden %s (UPDATE_GOLDEN=1 to create): %v", gpath, err) + } + if string(got) != string(want) { + t.Fatalf("%s != golden:\n--- got ---\n%s", ent.Name(), got) + } + } + + // Reverse direction: every golden must have been produced, so a regression + // that stops emitting a file (e.g. NOTES.md or a whole table) is caught; + // the loop above only checks produced->golden. + if os.Getenv("UPDATE_GOLDEN") != "1" { + goldens, err := os.ReadDir(goldenDir) + if err != nil { + t.Fatal(err) + } + produced := map[string]bool{} + for _, ent := range entries { + produced[ent.Name()] = true + } + for _, g := range goldens { + if !produced[g.Name()] { + t.Fatalf("golden %q was not produced by Emit (stopped emitting or renamed?)", g.Name()) + } + } + } +} + // emitDatabricks runs dbt -> databricks-metric-view over the ecommerce fixture // and returns every emitted file keyed by name. func emitDatabricks(t *testing.T) map[string]string { @@ -662,6 +731,19 @@ func TestRolePlayingDimensionAllTargets(t *testing.T) { } }) + t.Run("slayer", func(t *testing.T) { + // SLayer permits one join per target model, so the role-playing + // dimension keeps its first FK and reports the second in NOTES.md. + got := emitTarget(t, "slayer", "fct_orders.yaml") + if n := strings.Count(got, "target_model: dim_customer"); n != 1 { + t.Errorf("slayer fct_orders.yaml has %d joins to dim_customer, want exactly 1\n--- got ---\n%s", n, got) + } + notes := emitTarget(t, "slayer", "NOTES.md") + if !strings.Contains(notes, "at most one join per target model") { + t.Errorf("slayer NOTES.md missing the duplicate-target join note:\n%s", notes) + } + }) + t.Run("nao-context-rules", func(t *testing.T) { got := emitTarget(t, "nao-context-rules", "RULES.md") for _, want := range []string{ diff --git a/test/models/ecommerce/dbt/slayer/NOTES.md b/test/models/ecommerce/dbt/slayer/NOTES.md new file mode 100644 index 0000000..8d213af --- /dev/null +++ b/test/models/ecommerce/dbt/slayer/NOTES.md @@ -0,0 +1,6 @@ +# Not transpiled to slayer + +- measure "net_line_revenue": renamed to "net_line_revenue_measure"; columns and measures share one namespace in a slayer model, and "net_line_revenue" names a column on "fct_order_lines" +- measure "quantity": renamed to "quantity_measure"; columns and measures share one namespace in a slayer model, and "quantity" names a column on "fct_order_lines" +- metric "aov": renamed to "aov_measure"; columns and measures share one namespace in a slayer model, and "aov" names a column on "fct_orders" +- relationship fct_orders -> dim_customer: not emitted to slayer at all: model "fct_orders" already joins "dim_customer", and a slayer model can declare at most one join per target model diff --git a/test/models/ecommerce/dbt/slayer/dim_channel.yaml b/test/models/ecommerce/dbt/slayer/dim_channel.yaml new file mode 100644 index 0000000..1738c73 --- /dev/null +++ b/test/models/ecommerce/dbt/slayer/dim_channel.yaml @@ -0,0 +1,13 @@ +version: 8 +name: dim_channel +sql_table: ANALYTICS.MAIN.dim_channel +data_source: ecommerce +columns: + - name: channel_id + type: DOUBLE + primary_key: true + description: Channel surrogate key. + - name: channel_name + type: TEXT + description: Channel display name. +description: Sales channel dimension. diff --git a/test/models/ecommerce/dbt/slayer/dim_customer.yaml b/test/models/ecommerce/dbt/slayer/dim_customer.yaml new file mode 100644 index 0000000..12a455d --- /dev/null +++ b/test/models/ecommerce/dbt/slayer/dim_customer.yaml @@ -0,0 +1,20 @@ +version: 8 +name: dim_customer +sql_table: ANALYTICS.MAIN.dim_customer +data_source: ecommerce +columns: + - name: customer_sk + type: DOUBLE + primary_key: true + description: Customer surrogate key. + - name: customer_segment + type: TEXT + description: 'Marketing segment. Values: new = First-ever order not yet placed or just placed; returning = Has ordered before; vip = High-value repeat customer; prospect = Signed up, never ordered.' + meta: + synonyms: + - segment + - customer_type + - name: accepts_marketing + type: BOOLEAN + description: Whether the customer opted in to marketing. +description: Customer dimension. diff --git a/test/models/ecommerce/dbt/slayer/dim_product.yaml b/test/models/ecommerce/dbt/slayer/dim_product.yaml new file mode 100644 index 0000000..63f313b --- /dev/null +++ b/test/models/ecommerce/dbt/slayer/dim_product.yaml @@ -0,0 +1,16 @@ +version: 8 +name: dim_product +sql_table: ANALYTICS.MAIN.dim_product +data_source: ecommerce +columns: + - name: product_id + type: DOUBLE + primary_key: true + description: Product surrogate key. + - name: category + type: TEXT + description: Product category. + - name: title + type: TEXT + description: Product title. +description: Product dimension. diff --git a/test/models/ecommerce/dbt/slayer/fct_order_lines.yaml b/test/models/ecommerce/dbt/slayer/fct_order_lines.yaml new file mode 100644 index 0000000..897a643 --- /dev/null +++ b/test/models/ecommerce/dbt/slayer/fct_order_lines.yaml @@ -0,0 +1,53 @@ +version: 8 +name: fct_order_lines +sql_table: ANALYTICS.MAIN.fct_order_lines +data_source: ecommerce +columns: + - name: order_line_id + type: DOUBLE + primary_key: true + description: Line-item surrogate key. + - name: order_id + type: DOUBLE + description: Order the line belongs to. + - name: product_id + type: DOUBLE + description: Product sold on the line. + - name: order_date + type: DATE + description: Date the parent order was placed. + - name: quantity + type: DOUBLE + description: Units sold on the line. + - name: net_line_revenue + type: DOUBLE + description: Net revenue for the line. +measures: + - formula: quantity:sum + name: quantity_measure + description: Units sold on the line. + type: DOUBLE + - formula: net_line_revenue:sum + name: net_line_revenue_measure + description: Net revenue for the line. + type: DOUBLE + - formula: quantity:sum + name: units_sold + label: Units sold + description: Units sold. + - formula: units_sold / fct_orders.order_id:count_distinct + name: units_per_order + label: Units per order + description: Units per order (cross-table). +joins: + - target_model: fct_orders + join_pairs: + - - order_id + - order_id + join_type: left + - target_model: dim_product + join_pairs: + - - product_id + - product_id + join_type: left +description: Order-line grain. One row per line item. diff --git a/test/models/ecommerce/dbt/slayer/fct_orders.yaml b/test/models/ecommerce/dbt/slayer/fct_orders.yaml new file mode 100644 index 0000000..5b8789b --- /dev/null +++ b/test/models/ecommerce/dbt/slayer/fct_orders.yaml @@ -0,0 +1,90 @@ +version: 8 +name: fct_orders +sql_table: ANALYTICS.MAIN.fct_orders +data_source: ecommerce +columns: + - name: order_id + type: DOUBLE + primary_key: true + description: Order surrogate key. + - name: customer_sk + type: DOUBLE + description: Customer the order belongs to. + - name: is_refunded + type: BOOLEAN + description: Whether the order was refunded. + - name: billing_customer_sk + type: DOUBLE + description: Customer who is billed for the order; may differ from the ordering customer (role-playing dimension, e.g. a corporate account's billing contact). + - name: channel_id + type: DOUBLE + description: Sales channel the order came through. + - name: aov + type: DOUBLE + description: Precomputed average order value; superseded by the computed aov metric. + - name: order_date + type: DATE + description: Date the order was placed. + - name: order_gross + type: DOUBLE + description: Gross order revenue before tax and refunds. + - name: order_net_booked + type: DOUBLE + description: Net booked revenue (gross minus tax and refunds). + - name: refunded_orders_count_expr + sql: case when is_refunded then 1 else 0 end + type: DOUBLE + hidden: true +measures: + - formula: order_gross:sum + name: order_gross_amount + description: Gross order revenue before tax and refunds. + type: DOUBLE + - formula: order_net_booked:sum + name: order_net_booked_amount + description: Net booked revenue (gross minus tax and refunds). + type: DOUBLE + - formula: order_id:count_distinct + name: orders_count + type: INT + - formula: refunded_orders_count_expr:sum + name: refunded_orders_count + - formula: order_gross:sum + name: gross_revenue + label: Gross revenue + description: Gross order revenue. + - formula: order_net_booked:sum + name: net_revenue + label: Net revenue + description: Net booked revenue. + - formula: order_id:count_distinct + name: orders + label: Orders + - formula: refunded_orders_count_expr:sum + name: refunded_orders + label: Refunded orders + description: Count of refunded orders. + - formula: net_revenue / orders + name: aov_measure + label: Average order value + description: Average order value (net revenue / orders). + - formula: refunded_orders / orders + name: refund_rate + label: Refund rate + description: Refunded orders / all orders. +joins: + - target_model: dim_customer + join_pairs: + - - customer_sk + - customer_sk + join_type: left + - target_model: dim_channel + join_pairs: + - - channel_id + - channel_id + join_type: left +description: Order-grain finance fact. One row per order. +meta: + synonyms: + - purchases + - sales diff --git a/test/models/ecommerce/dbt/slayer/obt_sales.yaml b/test/models/ecommerce/dbt/slayer/obt_sales.yaml new file mode 100644 index 0000000..68ee89f --- /dev/null +++ b/test/models/ecommerce/dbt/slayer/obt_sales.yaml @@ -0,0 +1,44 @@ +version: 8 +name: obt_sales +sql_table: ANALYTICS.MAIN.obt_sales +data_source: ecommerce +columns: + - name: order_line_id + type: DOUBLE + primary_key: true + description: Line-item surrogate key. + - name: order_id + type: DOUBLE + description: Order the line belongs to. + - name: customer_segment + type: TEXT + description: Marketing segment. + - name: is_refunded + type: BOOLEAN + description: Whether the order line was refunded. + - name: order_date + type: DATE + description: Date the order was placed. + - name: quantity + type: DOUBLE + description: Units sold on the line. + - name: net_revenue + type: DOUBLE + description: Net revenue for the line. +measures: + - formula: quantity:sum + name: obt_units_sold + description: Units sold on the line. + type: DOUBLE + - formula: net_revenue:sum + name: obt_net_revenue_line + description: Net revenue for the line. + type: DOUBLE +joins: + - target_model: fct_orders + join_pairs: + - - order_id + - order_id + join_type: left +default_time_dimension: order_date +description: Wide sales table at order-line grain. One row per order line. Measures only, no metrics.