diff --git a/DECISIONS.md b/DECISIONS.md index f2ef5942..85c2bab2 100644 --- a/DECISIONS.md +++ b/DECISIONS.md @@ -122,3 +122,4 @@ implementation detail. Include issue refs when known. - 2026-08-16 — Aggregated slot-type and display-format inference share one classifier (DEV-1788, follow-up to DEV-1784's Option A). `aggregated_type` (slot `DataType`) and `_infer_aggregated_format` (response `NumberFormat`) had disagreed on the stat/parametric family: type said `DOUBLE` while format fell through to inherit the source column's format, so `revenue:stddev_samp` was typed `DOUBLE` yet displayed as currency. Both now read a single `classify_aggregation` (`core/enums.py`) returning one of four `AggregationValueClass` buckets, and each function maps the bucket to its own output — no per-name branching survives, so the two axes cannot drift. The four builtin frozensets (`INTEGER_AGGREGATIONS`, `PRESERVING_AGGREGATIONS`, `FLOAT_SOURCE_UNIT_AGGREGATIONS`, `FLOAT_PLAIN_AGGREGATIONS`) partition `BUILTIN_AGGREGATIONS`, pinned by a completeness test; custom/model-defined aggregations hit the `PRESERVING` fallback (inherit type & format), unchanged. **Semantics chosen (Option B, unit-correct):** `avg`/`median`/`weighted_avg`/`percentile`/`stddev*` are `DOUBLE` but keep the source's UNITS, so display format inherits the source (falling back to `FLOAT` when the source has none — keeping type `DOUBLE` and format `FLOAT` coherent for unformatted measures, and confining the change to formatted ones); `corr`/`var*`/`covar*` are dimensionless/squared/product units, so they display as plain `FLOAT` regardless of source. `aggregated_type` is behaviourally unchanged (only restructured). **Net user-visible change, all in `_infer_aggregated_format`:** avg-family of a FORMATTED measure now inherits that format (was `FLOAT`); `corr`/`var*`/`covar*` now `FLOAT` (was inherit); `stddev*`/`percentile` unchanged (already inherited). Drift guard extended to the full four-bucket table and routed through the public callers (`measure_key_type` / `measure_key_format_description`), plus response-metadata assertions for the stat/parametric family. - 2026-08-17 — Last four parser-only scalars admitted (DEV-1753, follow-up to DEV-1744; closes the tripwire `TestParserAndBinderScalarSetsAgree`). `greatest`, `least`, `trunc`, `mod` were in the parser's `SCALAR_PASSTHROUGH` but not the binder's `SCALAR_FUNCTIONS`, so each parsed then raised `UnknownFunctionError` at bind while the parser's own error text advertised it; all four are now in `SCALAR_FUNCTIONS` with arity entries, so `SCALAR_PASSTHROUGH - SCALAR_FUNCTIONS == set()`. Rulings: (1) **`greatest`/`least` NULL divergence RATIFIED, not normalised** — same class as the `concat → ||` ruling (2026-08-05). They pass through to each backend's native form, and those forms disagree on NULL: Postgres/DuckDB/SQL Server/Snowflake (sqlglot emits `GREATEST_IGNORE_NULLS` for Snowflake) IGNORE NULLs, while SQLite (scalar `MAX(a,b)`/`MIN(a,b)`), MySQL and BigQuery PROPAGATE NULL. Normalising is impractical (variadic ⇒ NULL-ignoring is not portably expressible) and per-dialect refusal breaks Mode B's dialect-agnostic promise (SQLite is the default store), so the divergence is documented and users wrap in `ifnull`/`coalesce` if they need one behaviour. Witnessed live: SQLite propagates (`tests/test_dev1753_last_four_scalars.py`), DuckDB ignores (`tests/integration/test_integration_duckdb.py`). Only those two are runtime-witnessed; the other rows are sqlglot-emission-contract assumptions (T-SQL `GREATEST` needs SQL Server 2022+). Arity `(2, None)`: SQLite's one-arg `MAX`/`MIN` parses as the AGGREGATE, and MySQL requires ≥2. (2) **`trunc` 1-arg only** — a 2-arg `trunc(x, digits)` SILENTLY drops the digits on SQLite (emits `TRUNC(x)`), a wrong answer rather than an error; ClickHouse emits the lowercase `trunc` on a case-sensitive backend (live check in `tests/integration/test_integration_clickhouse.py`). (3) **`mod` renders through the `%` composer** (`render_arithmetic(op="%")`), not a raw `exp.Mod`: `exp.func("MOD", …)` raises at build time, and a directly-built `exp.Mod` mis-groups a complex operand (`mod(a+b, c)` → `a + b % c`, which re-parses as `a + (b%c)`); the `%` composer runs the operand-precedence pass, and the pre-existing unconditional-Mod parenthesisation (DEV-1744) covers a `mod(...)` nested inside arithmetic. (4) **2-arg strip form of `ltrim`/`rtrim` DEFERRED to DEV-1793** — its second argument is a character SET on most backends but an exact SUBSTRING on MySQL (`TRIM(LEADING remstr FROM str)`), a silent string-value cross-dialect divergence needing its own ruling; the trims stay `(1,1)`. - 2026-08-17 — DEV-1756 identifier-length fitting ported to the DEV-1450 pipeline for the PROJECTION-ALIAS and CTE-NAME surfaces. Main's design (aliases stay canonical inside SLayer, fitted only at emission, restored on result keys) is kept: the pure `fit_identifier`/`substitute_quoted` module and the `SqlDialect` methods (`fit_alias`/`emit_alias`/`alias_rewrite_map`/`decode_alias_map`/`_rekey_row`, and the `aliases`-taking `rewrite_emitted_sql`/`decode_result_keys`) merged intact, as did the per-dialect `max_identifier_bytes` budgets. **Projection aliases:** the wiring differs from main because the DEV-1450 pipeline is structural, not enrichment-based — main threaded `all_projection_aliases`/`public_projection_aliases` (enriched.py, deleted); here the write and read sides share ONE plan-derived source, a new `response_meta.projection_result_keys(root_planned)` (mirrors `_slot_result_keys`). The WRITE side passes it into `generate_planned_stages(..., projection_aliases=...)` → `rewrite_emitted_sql(sql, aliases=...)` (NOT parsed off the SQL — `sqlglot.parse_one` chokes on a pre-mangle BigQuery backticked dotted alias); the READ side threads the same set into every `decode_result_keys` call site (`build_response_metadata`, `_run_data_query` via `_Prepared.expected_columns`, the `get_column_types` probe). The two agree by the generator's naming contract (`_slot_result_keys` ≡ emitted projection aliases). **CTE names:** rather than adopt main's `SQLGenerator._cte_name`/`_cte_names` (absent here), the branch's own `naming.cte_name_from_alias` now length-fits `prefix+sanitized` via `fit_identifier` BEFORE `AliasAllocator.allocate_cte`, and raises `IdentifierCollisionError` if the allocator's `_2` suffix pushes a fitted name back over the limit (only a forced digest collision reaches this). The dialect + limit are threaded from the three cross-model/window/ranked call sites and the two time-shift `shifted_`/`sjoin_` sites; `limit=None` (unbounded dialects) is byte-identical to before. `TestCteNames`/`TestSweep` were rewritten to this API and un-skipped. **NOT ported (one follow-up):** virtual-model short fitting (`_fit_short`) — there is no `engine._query_as_model` on this branch; query-backed shorts flow through `source_bundle` expansion, so `TestVirtualModelShorts` stays `pytest.mark.skip`-marked. The projection/CTE scenarios drive the real engine (`_engine_generate` / `SlayerQueryEngine`) and pass, as does the pure-module `tests/dialects/test_identifier_fit.py`. Two "no-churn" golden tests were rewritten from main's byte-exact SQL (which pinned `SUM(...)`, but this branch emits `CAST(SUM(...) AS FLOAT64)`) to assert the DEV-1756 intent (under-limit ⇒ no `__` marker). Short-alias quoting in `build_flat_rename_wrapper` was left as the branch had it (lenient `AS "?rev"?` assertions kept), not switched to main's quote-iff-mixed-case-or-reserved policy. +- 2026-08-17 — Mode-A expansion gates qualification on root scope (DEV-1752). In `_process_column_node_sync` (`slayer/engine/column_expansion.py`) the `root_scope_ids` gate now runs BEFORE the qualification branch, not after — so a column outside the root scope (inside a subquery, CTE, set-op branch, or `Values`) is left completely untouched, neither qualified against the outer root nor inlined. Previously qualification fired unconditionally, rebinding a subquery's own columns to the outer model: `amount IN (SELECT amount FROM other_tbl)` became `orders.amount IN (SELECT orders.amount FROM other_tbl)`, silently changing the predicate (in SQLite the inner `orders.amount` correlates to the outer row, making the filter always-true). The contract is now explicit: a Mode-A subquery must be **self-contained** — its columns bind in its own scope. Correlation to the outer model is not supported: an outer reference in a subquery is a scope leak flagged by the `assert_scope_closed` invariant (`slayer/sql/scope_check.py`), which runs under `SLAYER_VALIDATE_SCOPES` (on in CI, off in production for performance; the only legal correlated ref is the RLS session-policy EXISTS). Runtime enforcement of the self-contained-subquery contract is out of scope for this bugfix — correlation is unsupported/undefined rather than hard-rejected at runtime. Chose this one-spot scope gate over a correlation heuristic: correlation is undecidable from a schema-less Mode-A fragment (a bare inner name that collides with a root column is byte-identical to a correlated ref), sqlglot's `external_columns`/`is_correlated_subquery` over-report on such fragments, and the scope-closure guard already forbids the only case a heuristic would target. Reuses the single existing `_root_scope_column_ids` notion of scope rather than adding a second. diff --git a/docs/concepts/references.md b/docs/concepts/references.md index 9c0527dd..6db10959 100644 --- a/docs/concepts/references.md +++ b/docs/concepts/references.md @@ -18,6 +18,7 @@ SLayer has two distinct expression layers and the rules for what each one accept * Single-dot `t.col` is a literal `.` SQL reference (sqlglot's normal behavior). * User-supplied multi-dot input (`a.b.c`) is auto-rewritten to `a__b.c` at validation time with a warning. * Other derived columns of the same model (or of a joined model via `__`) are recursively expanded so chains like `A.ratio = "A.bar / B.foo_normalized"` (where `B.foo_normalized` is itself derived) work. +* A subquery in a Mode-A surface (`col IN (SELECT … FROM other)`, a scalar `= (SELECT … LIMIT 1)`) must be **self-contained**: its own columns bind against the subquery's own `FROM`, never against this model — reference resolution does not reach in to re-qualify them. Correlating back to the outer model is **not** supported (an outer reference inside the subquery is a scope leak, flagged by the scope-closure check SLayer runs over generated SQL under `SLAYER_VALIDATE_SCOPES`). * `ModelMeasure` names are not visible from SQL mode — saved measures are DSL-only. * `{variable}` placeholders are substituted into these Mode-A surfaces from the merged variable set (raise-on-missing once any variable is in play; a fully variable-free execution leaves braces as literals; Mode-A- and dialect-aware string escaping, so quoted values round-trip on backslash-escaping backends like MySQL/ClickHouse — DEV-1727). See [Variables in model SQL](models.md#variables-in-model-sql). diff --git a/slayer/engine/column_expansion.py b/slayer/engine/column_expansion.py index 6168b15e..ac89140c 100644 --- a/slayer/engine/column_expansion.py +++ b/slayer/engine/column_expansion.py @@ -256,6 +256,11 @@ def _process_column_node_sync( """ if col.args.get("db") or col.args.get("catalog"): return None + # DEV-1752: gate BEFORE qualifying. A column outside the root scope belongs + # to a subquery / CTE / set-op branch and binds in its own scope, so it is + # left untouched — neither qualified against the outer root nor inlined. + if id(col) not in root_scope_ids: + return None table_id = col.args.get("table") col_name = col.name table_alias = table_id.name if table_id is not None else alias_path @@ -272,8 +277,6 @@ def _process_column_node_sync( if target_col is None or _is_trivial_base(column=target_col): col.set("table", exp.to_identifier(canonical_alias)) return None - if id(col) not in root_scope_ids: - return None next_is_root = is_root and (target_model is model) key = (target_model.name, col_name) if key in visited: diff --git a/slayer/engine/cross_model_planner.py b/slayer/engine/cross_model_planner.py index 2438f1bf..b18ae317 100644 --- a/slayer/engine/cross_model_planner.py +++ b/slayer/engine/cross_model_planner.py @@ -927,7 +927,7 @@ def _plan_filtered_local( model=host_model, public_alias=public_alias, ), - grain_measures=list(_grain_declared_measures(host_prebound)), + grain_measures=list(host_prebound.grain_declared_measures), inherited_filters=[ routing_by_id[fid].bound for fid in host_rooted_routes.applied @@ -1016,13 +1016,6 @@ def _plan_filtered_local( # module does not import ``stage_planner`` (no cycle). -def _grain_declared_measures(prebound: PreboundQuery) -> List[DeclaredMeasure]: - """The host's dimension + time-dimension declarations — the grain prefix of - ``declared_measures``, which the nested plan groups by unchanged.""" - n = prebound.n_dims + prebound.n_time_dimensions - return list(prebound.declared_measures[:n]) - - def _aggregate_declared_measure( *, key: AggregateKey, @@ -1314,14 +1307,15 @@ def _is_forward(path: Tuple[str, ...]) -> bool: # On the host->target path (handled by the forward-path CTE already). return bool(path) and path == target_path[: len(path)] - n_dims = host_prebound.n_dims - n_tds = host_prebound.n_time_dimensions grain_declared: List[DeclaredMeasure] = [] grain_host_sids: List[str] = [] grain_rerooted_keys: List[ValueKey] = [] needs_reroot = False - for i, dm in enumerate(host_prebound.declared_measures[: n_dims + n_tds]): + # The public_projection[i] positional pairing (and its silent None + # fallback) is deliberately left as-is; the cardinality/ambiguity concern + # it encodes is out of scope here (DEV-1688). + for i, dm in enumerate(host_prebound.grain_declared_measures): host_sid = public_projection[i] if i < len(public_projection) else None host_key = dm.bound.value_key inner = ( diff --git a/slayer/engine/prebound.py b/slayer/engine/prebound.py index 7d813c21..b2a89426 100644 --- a/slayer/engine/prebound.py +++ b/slayer/engine/prebound.py @@ -56,6 +56,7 @@ "PreboundQuery", "StrictQueryCarrier", "aggregated_type", + "partition_declared_measures", "dimension_key_metadata", "measure_key_format_description", "measure_key_type", @@ -68,6 +69,23 @@ # --------------------------------------------------------------------------- +def partition_declared_measures( + *, + declared_measures: List[DeclaredMeasure], + n_dims: int, + n_time_dimensions: int, +) -> Tuple[List[DeclaredMeasure], List[DeclaredMeasure], List[DeclaredMeasure]]: + """Split ``declared_measures`` into its (dims, time_dims, aggregates) prefix + partition — the slice arithmetic the planners used to inline. ``n_dims`` / + ``n_time_dimensions`` are the grain prefix lengths (see ``PreboundQuery``).""" + grain = n_dims + n_time_dimensions + return ( + declared_measures[:n_dims], + declared_measures[n_dims:grain], + declared_measures[grain:], + ) + + class PreboundQuery(BaseModel): """The typed product of ``plan_query``'s bind block. @@ -136,6 +154,16 @@ def _filter_texts_are_parallel(self) -> "PreboundQuery": ) return self + @property + def grain_declared_measures(self) -> List[DeclaredMeasure]: + """The dimension + time-dimension grain prefix of ``declared_measures``.""" + dims, time_dims, _ = partition_declared_measures( + declared_measures=self.declared_measures, + n_dims=self.n_dims, + n_time_dimensions=self.n_time_dimensions, + ) + return dims + time_dims + class StrictQueryCarrier(BaseModel): """The post-bind ``query.*`` surface the §5.4 seam approves. diff --git a/slayer/engine/stage_planner.py b/slayer/engine/stage_planner.py index c102762b..8fe08602 100644 --- a/slayer/engine/stage_planner.py +++ b/slayer/engine/stage_planner.py @@ -117,6 +117,7 @@ StrictQueryCarrier, measure_key_format_description, measure_key_type, + partition_declared_measures, ) from slayer.engine.source_bundle import ( ResolvedSourceBundle, @@ -739,7 +740,10 @@ def bind_query_inputs( # NOSONAR(S3776) — one cohesive bind pass. The stages n_dims = len(query.dimensions or []) n_tds = len(query.time_dimensions or []) filter_alias_map: Dict[str, ValueKey] = {} - for dm in declared_measures[n_dims + n_tds:]: + _, _, _agg_dms = partition_declared_measures( + declared_measures=declared_measures, n_dims=n_dims, n_time_dimensions=n_tds, + ) + for dm in _agg_dms: for alias in (dm.public_name, dm.declared_name, dm.canonical_alias): if alias is not None: filter_alias_map.setdefault(alias, dm.bound.value_key) @@ -1021,8 +1025,9 @@ def bind_query_inputs( # NOSONAR(S3776) — one cohesive bind pass. The stages # not the raw timestamp — which would silently widen the grain). Runs BEFORE # interning so a rewritten key never leaves a stale slot behind (identity is # only touched on the rewritten rank transform). - _dim_dms = declared_measures[:n_dims] - _td_dms = declared_measures[n_dims:n_dims + n_tds] + _dim_dms, _td_dms, _ = partition_declared_measures( + declared_measures=declared_measures, n_dims=n_dims, n_time_dimensions=n_tds, + ) _dim_key_set = {dm.bound.value_key for dm in _dim_dms} # A source column carrying two time-dimension granularities (``created_at`` # at both month and day) maps to two distinct ``TimeTruncKey`` buckets — a @@ -1399,6 +1404,9 @@ def _windowed_phase(bf: BoundFilter) -> Phase: filters_by_phase: List[FilterPhase] = [] bound_filter_ids: List[str] = [] + # Capture the date-range fids as they are minted; the windowed ``_src`` + # row-filter routing below reuses this exact set instead of re-deriving it. + date_range_fids: set = set() for i, bf in enumerate(bound_filters[:n_date_range]): fid = f"f{i}" filters_by_phase.append( @@ -1408,6 +1416,7 @@ def _windowed_phase(bf: BoundFilter) -> Phase: ), ) bound_filter_ids.append(fid) + date_range_fids.add(fid) filters_by_phase.extend(text_filter_entries) for i, bf in enumerate(bound_filters[n_date_range:], start=n_date_range): fid = f"f{i}" @@ -1668,7 +1677,6 @@ def _windowed_phase(bf: BoundFilter) -> Phase: # POST-reclassified windowed-measure filters are already excluded # (phase != ROW). if windowed_plans: - date_range_fids = {f"f{i}" for i in range(n_date_range)} src_where_ids, src_rewrites = _plan_src_row_filters( filters_by_phase=filters_by_phase, date_range_fids=date_range_fids, diff --git a/slayer/sql/generator.py b/slayer/sql/generator.py index 581dbfd5..ee6efc81 100644 --- a/slayer/sql/generator.py +++ b/slayer/sql/generator.py @@ -12,7 +12,19 @@ import logging import re from collections.abc import Sequence -from typing import AbstractSet, Any, Dict, List, Literal, Optional, Set, Tuple, Union +from typing import ( + AbstractSet, + Any, + Callable, + Dict, + Iterable, + List, + Literal, + Optional, + Set, + Tuple, + Union, +) import sqlglot from sqlglot import exp @@ -1668,50 +1680,28 @@ def _generate_from_planned_impl( # NOSONAR(S3776) — top-level dispatch over c ) # --- Window batch (one step CTE per Kahn batch) ---------- if ready_window: - step_num += 1 - step_name = cte_allocator.allocate_cte(f"step{step_num}") - prev_cte = chain_tail - carry_aliases = self._carry_aliases_in_plan_order( - aliases_by_slot_id, - ) - step_parts = [ - exp.column(a, quoted=True) for a in carry_aliases + window_entries = [ + (slot_id, slots_by_id[slot_id]) + for layer in ready_window + for slot_id in layer.slot_ids ] - for layer in ready_window: - for slot_id in layer.slot_ids: - slot = slots_by_id[slot_id] - alias = ( - slot.public_aliases[0] - if slot.public_aliases - else slot.declared_name - ) - full_alias = f"{source_relation}.{alias}" - window_expr = self._render_window_transform_sql( - slot=slot, - slots_by_id=slots_by_id, - slot_id_by_key=slot_id_by_key, - available_alias_by_slot_id=available_alias_by_slot_id, - planned_query=planned_query, - ) - if slot.type is not None: - window_expr = _wrap_cast_for_type( - window_expr, slot.type, - ) - step_parts.append( - window_expr.as_(full_alias, quoted=True), - ) - aliases_by_slot_id.setdefault(slot_id, []).append( - full_alias, - ) - available_alias_by_slot_id.setdefault( - slot_id, full_alias, - ) - ctes.append(CteEntry( - name=step_name, - query=exp.Select().select(*step_parts).from_(prev_cte), - depends_on=[prev_cte], - )) - chain_tail = step_name + chain_tail, step_num = self._emit_step_cte( + ctes=ctes, + chain_tail=chain_tail, + step_num=step_num, + cte_allocator=cte_allocator, + aliases_by_slot_id=aliases_by_slot_id, + available_alias_by_slot_id=available_alias_by_slot_id, + source_relation=source_relation, + slot_entries=window_entries, + render=lambda slot: self._render_window_transform_sql( + slot=slot, + slots_by_id=slots_by_id, + slot_id_by_key=slot_id_by_key, + available_alias_by_slot_id=available_alias_by_slot_id, + planned_query=planned_query, + ), + ) # --- time_shift layers (each gets shifted_ + sjoin_ pair) - for layer in ready_time_shift: for slot_id in layer.slot_ids: @@ -1750,44 +1740,142 @@ def _generate_from_planned_impl( # NOSONAR(S3776) — top-level dispatch over c ) pending_layers = not_ready - # 7b.11 — materialise POST-phase ArithmeticKey / ScalarCallKey - # slots that the user projected but no transform layer rendered. - # ``change(amount:sum)`` lowers to ``amount:sum - time_shift(...)``; - # the time_shift slot is rendered as a self-join CTE pair, but - # the outer ArithmeticKey slot that subtracts them needs its - # own step CTE. Same shape covers ``change_pct`` (division of - # arithmetic operands) and any future POST-phase non-transform - # slot the planner emits. + return self._finalise_transform_chain( + ctes=ctes, + chain_tail=chain_tail, + step_num=step_num, + cte_allocator=cte_allocator, + aliases_by_slot_id=aliases_by_slot_id, + available_alias_by_slot_id=available_alias_by_slot_id, + source_relation=source_relation, + slots_by_id=slots_by_id, + slot_id_by_key=slot_id_by_key, + planned_query=planned_query, + ) + + # ----------------------------------------------------------------- + # Transform-chain step-CTE emission (shared by the host and cross-model + # chains). DEV-1777: one shell for the four window / unmaterialised-POST + # step-CTE sites; per-site Kahn batching stays at the call sites. + # ----------------------------------------------------------------- + + def _emit_step_cte( + self, + *, + ctes: List["CteEntry"], + chain_tail: str, + step_num: int, + cte_allocator, + aliases_by_slot_id: Dict[str, List[str]], + available_alias_by_slot_id: Dict[str, str], + source_relation: str, + slot_entries: Iterable[Tuple[str, Any]], + render: Callable[[Any], exp.Expression], + ) -> Tuple[str, int]: + """Emit one transform-chain step CTE and advance the chain. + + ``render`` is invoked once per ``slot_entries`` element, in order, and + each element's alias-map updates happen AFTER its render — so a window + render sees earlier same-step aliases. ``render`` must not mutate the + alias maps. Appends the CTE to ``ctes``, updates both alias maps in + place, and returns ``(new_chain_tail, step_num)``. + """ + step_num += 1 + step_name = cte_allocator.allocate_cte(f"step{step_num}") + prev_cte = chain_tail + carry_aliases = self._carry_aliases_in_plan_order(aliases_by_slot_id) + step_parts = [exp.column(a, quoted=True) for a in carry_aliases] + for map_key, slot in slot_entries: + alias = ( + slot.public_aliases[0] + if slot.public_aliases + else slot.declared_name + ) + full_alias = f"{source_relation}.{alias}" + rendered = render(slot) + if slot.type is not None: + rendered = _wrap_cast_for_type(rendered, slot.type) + step_parts.append(rendered.as_(full_alias, quoted=True)) + aliases_by_slot_id.setdefault(map_key, []).append(full_alias) + available_alias_by_slot_id.setdefault(map_key, full_alias) + ctes.append(CteEntry( + name=step_name, + query=exp.Select().select(*step_parts).from_(prev_cte), + depends_on=[prev_cte], + )) + return step_name, step_num + + @staticmethod + def _unmaterialised_post_slots( + planned_query, aliases_by_slot_id: Dict[str, List[str]], + ) -> List[Any]: + """Projected POST-phase Arithmetic / ScalarCall slots no transform + layer rendered. + + ``change(amount:sum)`` lowers to ``amount:sum - time_shift(...)``: the + time_shift slot is a self-join CTE pair, but the outer ArithmeticKey + that subtracts them needs its own step CTE. Same shape covers + ``change_pct`` and any future POST-phase non-transform slot. + """ from slayer.core.keys import ( ArithmeticKey as _ArithKey, ScalarCallKey as _ScalarKey, TransformKey as _TKey, ) - unmaterialised: list = [] + unmaterialised: List[Any] = [] for cslot in planned_query.combined_expression_slots: if isinstance(cslot.key, _TKey): - # Transform-key slots are materialised by transform_layers. continue if cslot.id in aliases_by_slot_id: continue if isinstance(cslot.key, (_ArithKey, _ScalarKey)): unmaterialised.append(cslot) + return unmaterialised + + def _inner_select_from_final_cte( + self, *, chain_tail: str, aliases_by_slot_id: Dict[str, List[str]], + ) -> exp.Select: + """Inner SELECT over the final chain CTE: all carried aliases in PLAN + order (B8).""" + inner_aliases = self._carry_aliases_in_plan_order(aliases_by_slot_id) + return exp.Select().select( + *(exp.column(a, quoted=True) for a in inner_aliases), + ).from_(chain_tail) + + def _finalise_transform_chain( + self, + *, + ctes: List[CteEntry], + chain_tail: str, + step_num: int, + cte_allocator, + aliases_by_slot_id: Dict[str, List[str]], + available_alias_by_slot_id: Dict[str, str], + source_relation: str, + slots_by_id: Dict[str, Any], + slot_id_by_key: Dict[Any, str], + planned_query, + ) -> str: + """Close a transform chain (host or cross-model): materialise any + leftover POST-phase slot, assemble the WITH chain, apply the POST-phase + filter wrap, and emit the outer user-projection wrap. + """ + # 7b.11 — materialise POST-phase ArithmeticKey / ScalarCallKey slots + # the user projected but no transform layer rendered. + unmaterialised = self._unmaterialised_post_slots( + planned_query, aliases_by_slot_id, + ) if unmaterialised: - step_num += 1 - step_name = cte_allocator.allocate_cte(f"step{step_num}") - prev_cte = chain_tail - carry_aliases = self._carry_aliases_in_plan_order( - aliases_by_slot_id, - ) - step_parts = [exp.column(a, quoted=True) for a in carry_aliases] - for cslot in unmaterialised: - alias = ( - cslot.public_aliases[0] - if cslot.public_aliases - else cslot.declared_name - ) - full_alias = f"{source_relation}.{alias}" - rendered = render_value_key( + chain_tail, step_num = self._emit_step_cte( + ctes=ctes, + chain_tail=chain_tail, + step_num=step_num, + cte_allocator=cte_allocator, + aliases_by_slot_id=aliases_by_slot_id, + available_alias_by_slot_id=available_alias_by_slot_id, + source_relation=source_relation, + slot_entries=[(cslot.id, cslot) for cslot in unmaterialised], + render=lambda cslot: render_value_key( key=cslot.key, ctx=RenderContext( dialect=self._dialect, @@ -1796,38 +1884,18 @@ def _generate_from_planned_impl( # NOSONAR(S3776) — top-level dispatch over c available_alias_by_slot_id=available_alias_by_slot_id, ), ), - ) - if cslot.type is not None: - rendered = _wrap_cast_for_type(rendered, cslot.type) - step_parts.append(rendered.as_(full_alias, quoted=True)) - aliases_by_slot_id.setdefault(cslot.id, []).append( - full_alias, - ) - available_alias_by_slot_id.setdefault( - cslot.id, full_alias, - ) - ctes.append(CteEntry( - name=step_name, - query=exp.Select().select(*step_parts).from_(prev_cte), - depends_on=[prev_cte], - )) - chain_tail = step_name - - # Inner SELECT inside _outer wrap: ALL carried aliases sorted - # in PLAN order (B8 — this list used to be sorted alphabetically to - # match the legacy renderer byte-for-byte). - final_cte = chain_tail - inner_aliases = self._carry_aliases_in_plan_order(aliases_by_slot_id) - inner_select = exp.Select().select( - *(exp.column(a, quoted=True) for a in inner_aliases), - ).from_(final_cte) + ), + ) + # Inner SELECT inside _outer wrap: ALL carried aliases in PLAN order (B8). + inner_select = self._inner_select_from_final_cte( + chain_tail=chain_tail, aliases_by_slot_id=aliases_by_slot_id, + ) chain_sql = assemble_with_chain( entries=ctes, final=inner_select, ).sql(dialect=self.dialect, pretty=True) - # POST-phase filter wrap (filters referencing transform / arith - # slots). Mirrors legacy _generate_with_computed:1627-1648 — + # POST-phase filter wrap (filters referencing transform / arith slots): # ``SELECT * FROM () AS _filtered WHERE ``. post_filter_conditions = self._render_post_phase_filter_conditions( planned_query=planned_query, @@ -1840,9 +1908,9 @@ def _generate_from_planned_impl( # NOSONAR(S3776) — top-level dispatch over c f"\nWHERE {_SQL_AND_JOINER.join(post_filter_conditions)}" ) - # Outer SELECT in user-projection order (public slots only). - # Per-slot index walks each slot's public_aliases so duplicate - # interned names (DEV-1450 C13) both surface in the result. + # Outer SELECT in user-projection order (public slots only). Per-slot + # index walks each slot's public_aliases so duplicate interned names + # (DEV-1450 C13) both surface in the result. public_aliases_user_order: list[str] = [] outer_alias_index: Dict[str, int] = {} for sid in planned_query.projection: @@ -2314,9 +2382,10 @@ def _resolve_agg_inputs_via_scope( # NOSONAR(S3776) — one cohesive Law-1 disc so its joins must be in the FROM. 4. **first/last explicit TIME ARGS** (``amount:last(customers.signup_at)`` — DEV-1710). Discovery only; the ranked subquery's ORDER BY re-renders - the arg via ``_resolve_explicit_time_col``. Replaces the legacy - ``_collect_joined_paths_for_base`` AGGREGATE arm. A path-bearing - derived (``ColumnSqlKey``) arg — the DEV-1526 residual — is skipped. + the arg from the plan (``RankedAggregatePlan.ranking_time_key``). + Replaces the legacy ``_collect_joined_paths_for_base`` AGGREGATE arm. + A path-bearing derived (``ColumnSqlKey``) arg — the DEV-1526 residual + — is skipped. Cross-model aggregates (non-empty ``source.path``) are skipped in every sub-pass: their inputs are owned by the per-plan ``_cm_*`` CTE @@ -2409,8 +2478,8 @@ def _resolve_first_last_time_arg(key) -> None: # a source / kwarg does; resolving it through the scope registers # that join (Law 1), so the ranked subquery's ORDER BY ref is in the # base FROM. Replaces the legacy ``_collect_joined_paths_for_base`` - # AGGREGATE arm. Register-only: the render spec re-resolves via - # ``_resolve_explicit_time_col``. + # AGGREGATE arm. Register-only: the ranked plan carries the resolved + # ranking time column (``RankedAggregatePlan.ranking_time_key``). arg = self._explicit_time_arg_of(key) if arg is None: return @@ -2428,6 +2497,22 @@ def _resolve_first_last_time_arg(key) -> None: _for_each_local_agg(_resolve_first_last_time_arg) return resolved + def _throwaway_frame(self, *, model, relation: str, bundle) -> ScopeFrame: + """A target-rooted ``ScopeFrame`` built purely to reproduce an anchored + expression. Its ``join_paths`` are inert — join discovery is owned by a + separate pass — so every caller discards them; only the re-anchored SQL + is used. A fresh allocator per frame keeps its scope id generation-local. + """ + allocator = self._new_allocator() + return ScopeFrame( + scope_id=allocator.next_scope_id(relation), + root_model=model, + root_relation=relation, + bundle=bundle, + dialect=self._dialect, + allocator=allocator, + ) + def _resolve_agg_kwargs_for_key( self, *, key, source_model, source_relation: str, bundle, ) -> "Optional[Dict[str, ResolvedAggKwarg]]": @@ -2448,14 +2533,8 @@ def _resolve_agg_kwargs_for_key( kwargs = getattr(key, "kwargs", None) if bundle is None or not kwargs: return None - allocator = self._new_allocator() - scope = ScopeFrame( - scope_id=allocator.next_scope_id(source_relation), - root_model=source_model, - root_relation=source_relation, - bundle=bundle, - dialect=self._dialect, - allocator=allocator, + scope = self._throwaway_frame( + model=source_model, relation=source_relation, bundle=bundle, ) resolved = { kname: ResolvedAggKwarg(kind="expr", value=scope.resolve(kval)) @@ -2767,14 +2846,14 @@ def _explicit_time_arg_of(key): """The explicit positional ranking-time arg of a ``first`` / ``last`` aggregate, or ``None``. - The SINGLE arg-selection contract shared by the three sites that must + The SINGLE arg-selection contract shared by the two sites that must never disagree on WHICH positional arg is the time column (DEV-1710 / - Codex F1): the ranked-plan builder in ``slayer/engine/ranked_planner.py``, - the join-discovery pass in ``_resolve_agg_inputs_via_scope``, and the - render seam ``_resolve_explicit_time_col``. Returns the FIRST positional arg - iff it is a ``ColumnKey`` / ``ColumnSqlKey``; ``None`` for a - non-first/last agg, empty args, or a first positional arg of any other - type (first/last never takes a leading non-column positional). + Codex F1): the ranked-plan builder in ``slayer/engine/ranked_planner.py`` + and the join-discovery pass in ``_resolve_agg_inputs_via_scope``. Returns + the FIRST positional arg iff it is a ``ColumnKey`` / ``ColumnSqlKey``; + ``None`` for a non-first/last agg, empty args, or a first positional arg + of any other type (first/last never takes a leading non-column + positional). """ from slayer.core.keys import ColumnKey, ColumnSqlKey @@ -2784,97 +2863,6 @@ def _explicit_time_arg_of(key): return a if isinstance(a, (ColumnKey, ColumnSqlKey)) else None return None - def _resolve_explicit_time_col( - self, - *, - key, - source_model, - source_relation: str, - bundle=None, - ) -> Optional[str]: - """Resolve the explicit positional time arg on a ``first`` / ``last`` - aggregate into a SQL string suitable for ``ORDER BY`` inside the - ranked subquery. - - Handles both bare-column refs (``ColumnKey`` — - ``amount:last(created_at)``) and derived-column refs (``ColumnSqlKey`` - — ``amount:last(net_amount_date)`` where ``net_amount_date`` has a - non-trivial ``Column.sql``). DEV-1710 Stage 6: when a ``bundle`` is - available the arg is anchored through a ``ScopeFrame`` (Law 1) — the - same resolver the host base / kwargs passes use — so a bare joined ref - qualifies to its ``__``-path alias, a derived expression's inner bare - refs qualify to ``source_relation`` (never ambiguous against a - same-named joined column), and reserved-word relations are quoted - (DEV-1686). Without a ``bundle`` (the render-spec unit path) it falls - back to bare-ident qualification / verbatim emit. - - Returns ``None`` for non-first/last aggs and when ``key.args`` is empty - or its first element is neither a ``ColumnKey`` nor a ``ColumnSqlKey`` - (see ``_explicit_time_arg_of``). A derived time arg (``ColumnSqlKey``) - whose ``path`` is non-empty AFTER the DEV-1707 cross-model reroot — a - column a hop PAST the target — raises ``NotImplementedError`` rather - than silently emitting against a relation the isolated CTE does not - join; that residual-hop case is tracked as DEV-1526 (Stage 4). The - analogous residual ``ColumnKey`` arg is caught loudly by the - scope-closure validator (``SLAYER_VALIDATE_SCOPES``) instead. - """ - from slayer.core.keys import ColumnKey, ColumnSqlKey - - arg = self._explicit_time_arg_of(key) - if arg is None: - return None - if isinstance(arg, ColumnSqlKey) and arg.path: - raise NotImplementedError( - f"Derived time column with a residual join path " - f"(path={arg.path!r}, column={arg.column_name!r}) on a " - f"first/last positional arg is not yet supported by " - f"the ranked-subquery builder: the isolated CTE does " - f"not pull the residual join. Post-DEV-1707 the " - f"cross-model reroot strips the target prefix, so this " - f"fires only for a time arg a hop PAST the target; " - f"tracked as DEV-1526 (Stage 4)." - ) - # Validate a derived arg's existence up front so the not-found case is a - # clear error rather than the resolver silently anchoring the bare name. - col = None - if isinstance(arg, ColumnSqlKey): - col = next( - (c for c in source_model.columns if c.name == arg.column_name), - None, - ) - if col is None: - raise ValueError( - f"Derived time column {arg.column_name!r} (positional " - f"arg of {key.agg!r}) not found on model " - f"{source_model.name!r}." - ) - if bundle is not None: - # Law 1 — anchor the arg through a throwaway host-rooted scope. Its - # ``join_paths`` are discarded (discovery is owned by the base - # aggregate-input pass, which registers the same join); this call is - # purely to reproduce the SAME anchored SQL the ORDER BY needs. Same - # throwaway-frame pattern as ``_resolve_agg_kwargs_for_key``. - allocator = self._new_allocator() - scope = ScopeFrame( - scope_id=allocator.next_scope_id(source_relation), - root_model=source_model, - root_relation=source_relation, - bundle=bundle, - dialect=self._dialect, - allocator=allocator, - ) - return scope.resolve(arg).sql(dialect=self.dialect) - # No bundle (defensive; the render-spec unit path): bare ColumnKey - # qualifies to its ``__``-path alias / source relation, a derived - # bare-ident qualifies to the source relation, else emit verbatim. - if isinstance(arg, ColumnKey): - relation = "__".join(arg.path) if arg.path else source_relation - return f"{relation}.{arg.leaf}" - col_sql = col.sql if col.sql else col.name - if col_sql.isidentifier(): - return f"{source_relation}.{col_sql}" - return self._parse(col_sql).sql(dialect=self.dialect) - def _composite_agg_builder( self, *, slot, source_model, source_relation: str, bundle, resolved_agg_kwargs, @@ -4770,139 +4758,41 @@ def _render_cross_model_transform_chain( # NOSONAR(S3776) — pre-existing comp f"dependencies could not be resolved; pending ops: " f"{pending_ops!r}.", ) - step_num += 1 - step_name = cte_allocator.allocate_cte(f"step{step_num}") - prev_cte = chain_tail - carry_aliases = self._carry_aliases_in_plan_order( - aliases_by_slot_id, + window_entries = [ + (slot_id, slots_by_id[slot_id]) + for layer in ready + for slot_id in layer.slot_ids + ] + chain_tail, step_num = self._emit_step_cte( + ctes=ctes, + chain_tail=chain_tail, + step_num=step_num, + cte_allocator=cte_allocator, + aliases_by_slot_id=aliases_by_slot_id, + available_alias_by_slot_id=available_alias_by_slot_id, + source_relation=source_relation, + slot_entries=window_entries, + render=lambda slot: self._render_window_transform_sql( + slot=slot, + slots_by_id=slots_by_id, + slot_id_by_key=slot_id_by_key, + available_alias_by_slot_id=available_alias_by_slot_id, + planned_query=planned_query, + ), ) - step_parts = [exp.column(a, quoted=True) for a in carry_aliases] - for layer in ready: - for slot_id in layer.slot_ids: - slot = slots_by_id[slot_id] - alias = ( - slot.public_aliases[0] - if slot.public_aliases - else slot.declared_name - ) - full_alias = f"{source_relation}.{alias}" - window_expr = self._render_window_transform_sql( - slot=slot, - slots_by_id=slots_by_id, - slot_id_by_key=slot_id_by_key, - available_alias_by_slot_id=available_alias_by_slot_id, - planned_query=planned_query, - ) - if slot.type is not None: - window_expr = _wrap_cast_for_type( - window_expr, slot.type, - ) - step_parts.append( - window_expr.as_(full_alias, quoted=True), - ) - aliases_by_slot_id.setdefault(slot_id, []).append(full_alias) - available_alias_by_slot_id.setdefault(slot_id, full_alias) - ctes.append(CteEntry( - name=step_name, - query=exp.Select().select(*step_parts).from_(prev_cte), - depends_on=[prev_cte], - )) - chain_tail = step_name pending_layers = not_ready - # Materialise any projected POST-phase ArithmeticKey / ScalarCallKey - # slot a window layer didn't render (``cumsum(x) + 1``-style combos). - from slayer.core.keys import ( - ArithmeticKey as _ArithKey, - ScalarCallKey as _ScalarKey, - TransformKey as _TKey, - ) - unmaterialised: list = [] - for cslot in planned_query.combined_expression_slots: - if isinstance(cslot.key, _TKey): - continue - if cslot.id in aliases_by_slot_id: - continue - if isinstance(cslot.key, (_ArithKey, _ScalarKey)): - unmaterialised.append(cslot) - if unmaterialised: - step_num += 1 - step_name = cte_allocator.allocate_cte(f"step{step_num}") - prev_cte = chain_tail - carry_aliases = self._carry_aliases_in_plan_order( - aliases_by_slot_id, - ) - step_parts = [exp.column(a, quoted=True) for a in carry_aliases] - for cslot in unmaterialised: - alias = ( - cslot.public_aliases[0] - if cslot.public_aliases - else cslot.declared_name - ) - full_alias = f"{source_relation}.{alias}" - rendered = render_value_key( - key=cslot.key, - ctx=RenderContext( - dialect=self._dialect, - aliases=AliasFacilities( - slot_id_by_key=slot_id_by_key, - available_alias_by_slot_id=available_alias_by_slot_id, - ), - ), - ) - if cslot.type is not None: - rendered = _wrap_cast_for_type(rendered, cslot.type) - step_parts.append(rendered.as_(full_alias, quoted=True)) - aliases_by_slot_id.setdefault(cslot.id, []).append(full_alias) - available_alias_by_slot_id.setdefault(cslot.id, full_alias) - ctes.append(CteEntry( - name=step_name, - query=exp.Select().select(*step_parts).from_(prev_cte), - depends_on=[prev_cte], - )) - chain_tail = step_name - - final_cte = chain_tail - inner_aliases = self._carry_aliases_in_plan_order(aliases_by_slot_id) - inner_select = exp.Select().select( - *(exp.column(a, quoted=True) for a in inner_aliases), - ).from_(final_cte) - chain_sql = assemble_with_chain( - entries=ctes, final=inner_select, - ).sql(dialect=self.dialect, pretty=True) - - post_filter_conditions = self._render_post_phase_filter_conditions( - planned_query=planned_query, - slot_id_by_key=slot_id_by_key, + return self._finalise_transform_chain( + ctes=ctes, + chain_tail=chain_tail, + step_num=step_num, + cte_allocator=cte_allocator, + aliases_by_slot_id=aliases_by_slot_id, available_alias_by_slot_id=available_alias_by_slot_id, - ) - if post_filter_conditions: - chain_sql = ( - f"SELECT *\nFROM (\n{chain_sql}\n) AS {FILTERED_ALIAS}" - f"\nWHERE {_SQL_AND_JOINER.join(post_filter_conditions)}" - ) - - public_aliases_user_order: list[str] = [] - outer_alias_index: Dict[str, int] = {} - for sid in planned_query.projection: - slot = slots_by_id[sid] - if slot.hidden: - continue - all_aliases = aliases_by_slot_id.get(sid, []) - if not all_aliases: - continue - idx = outer_alias_index.setdefault(sid, 0) - alias = ( - all_aliases[idx] if idx < len(all_aliases) else all_aliases[-1] - ) - outer_alias_index[sid] = idx + 1 - public_aliases_user_order.append(alias) - return self._emit_planned_outer_wrap( - chain_sql=chain_sql, - public_aliases=public_aliases_user_order, - planned_query=planned_query, + source_relation=source_relation, slots_by_id=slots_by_id, - available_alias_by_slot_id=available_alias_by_slot_id, + slot_id_by_key=slot_id_by_key, + planned_query=planned_query, ) def _canonical_cross_model_alias( @@ -5260,9 +5150,9 @@ def _render_cross_model_cte( # NOSONAR(S3776) — single conceptual unit: share # the host-rooted derived key renders against the wrong alias inside # the CTE — and the DEV-1476(c) explicit time arg # ``customers.amount:last(customers.signup_at)``, whose positional arg - # must strip the host prefix in lockstep with the source so - # ``_resolve_explicit_time_col`` qualifies the time column under the - # target relation. ``column_filter_key`` rides through unchanged + # must strip the host prefix in lockstep with the source so the ranked + # plan qualifies the time column under the target relation. + # ``column_filter_key`` rides through unchanged # (owner-anchored, invariant under reroot). cross_model_path = getattr(agg_slot.key.source, "path", ()) local_agg_key = reroot_aggregate_key( @@ -5724,14 +5614,8 @@ def _collect_routed_filters( return None wanted = set(filter_ids) - allocator = self._new_allocator() - scope = ScopeFrame( - scope_id=allocator.next_scope_id(target_relation), - root_model=target_model, - root_relation=target_relation, - bundle=bundle, - dialect=self._dialect, - allocator=allocator, + scope = self._throwaway_frame( + model=target_model, relation=target_relation, bundle=bundle, ) ctx = RenderContext( scope=scope, @@ -8029,20 +7913,9 @@ def _build_agg_render_spec_from_planned( # NOSONAR(S3776) — sequential isinst if isinstance(source, ColumnKey) else source.column_name ) - # ``first`` / ``last`` aggregations rank rows via a ROW_NUMBER - # ranked CTE (planned in ``slayer/engine/ranked_planner.py``, - # rendered by ``_render_ranked_cte_from_planned``) and pick - # ``rn = 1`` through ``MAX(CASE WHEN _rn = 1 THEN col END)``. - # An explicit positional arg (``latest_amount:last(created_at)`` - # or ``…:last(derived_time_col)``) overrides the query's default - # ranking time column; the helper handles both bare-column - # (``ColumnKey``) and derived-column (``ColumnSqlKey``) args. - explicit_time_col = self._resolve_explicit_time_col( - key=key, - source_model=source_model, - source_relation=source_relation, - bundle=bundle, - ) + # ``first`` / ``last`` render through RankedAggregatePlan (see + # ``_ranked_value_expr``), never through this spec builder, so no + # explicit ranking time column is resolved here. agg_def = self._resolve_aggregation_def( key=key, source_model=source_model, src_leaf=src_leaf, ) @@ -8120,7 +7993,7 @@ def _build_agg_render_spec_from_planned( # NOSONAR(S3776) — sequential isinst filter_sql=filter_sql, agg_kwargs=agg_kwargs_str, aggregation_def=agg_def, - time_column=explicit_time_col, + time_column=None, ) raise NotImplementedError( f"AggregateKey source {type(source).__name__} not supported.", @@ -8298,19 +8171,12 @@ def _filter_render_context( slot_by_key=None, aliases_by_slot_id=None, ) -> RenderContext: """A ``RenderContext`` for the WHERE/HAVING filter family, over a - render-scoped host ``ScopeFrame`` (the crossed joins are pulled into the - FROM by a separate pass, so this scope's ``join_paths`` are inert — the - same throwaway pattern as ``_resolve_agg_kwargs_for_key``; PR 6 - consolidates them). Carries the filter-side CAST policy and the DEV-1539 - comparison grouping.""" - allocator = self._new_allocator() - scope = ScopeFrame( - scope_id=allocator.next_scope_id(source_relation), - root_model=source_model, - root_relation=source_relation, - bundle=bundle, - dialect=self._dialect, - allocator=allocator, + render-scoped host ``ScopeFrame`` from ``_throwaway_frame`` (the crossed + joins are pulled into the FROM by a separate pass, so this scope's + ``join_paths`` are inert). Carries the filter-side CAST policy and the + DEV-1539 comparison grouping.""" + scope = self._throwaway_frame( + model=source_model, relation=source_relation, bundle=bundle, ) return RenderContext( scope=scope, diff --git a/tests/golden/dev1747_sql_baseline.json b/tests/golden/dev1747_sql_baseline.json index 2f1b8d97..585c9c85 100644 --- a/tests/golden/dev1747_sql_baseline.json +++ b/tests/golden/dev1747_sql_baseline.json @@ -1,4 +1,9 @@ { + "chain/cross_model_nested_window::bigquery": "SELECT\n `orders___created_at`,\n `orders___cc`\nFROM (\nWITH _base AS (\n SELECT\n DATE_TRUNC(orders.created_at, MONTH) AS `orders___created_at`\n FROM orders AS orders\n GROUP BY\n DATE_TRUNC(orders.created_at, MONTH)\n), _cm_orders__customers__spend_sum AS (\n SELECT\n SUM(customers.spend) AS `orders___customers___spend_sum`\n FROM customers AS customers\n), base AS (\n SELECT\n _base.`orders___created_at`,\n _cm_orders__customers__spend_sum.`orders___customers___spend_sum`\n FROM _base\n CROSS JOIN _cm_orders__customers__spend_sum\n), step1 AS (\n SELECT\n `orders___created_at`,\n `orders___customers___spend_sum`,\n SUM(`orders___customers___spend_sum`) OVER (ORDER BY `orders___created_at`) AS `orders____cumsum_inner`\n FROM base\n), step2 AS (\n SELECT\n `orders___created_at`,\n `orders___customers___spend_sum`,\n `orders____cumsum_inner`,\n SUM(`orders____cumsum_inner`) OVER (ORDER BY `orders___created_at`) AS `orders___cc`\n FROM step1\n)\nSELECT\n `orders___created_at`,\n `orders___customers___spend_sum`,\n `orders____cumsum_inner`,\n `orders___cc`\nFROM step2\n) AS _outer", + "chain/cross_model_nested_window::duckdb": "SELECT\n \"orders.created_at\",\n \"orders.cc\"\nFROM (\nWITH _base AS (\n SELECT\n DATE_TRUNC('MONTH', orders.created_at) AS \"orders.created_at\"\n FROM orders AS orders\n GROUP BY\n DATE_TRUNC('MONTH', orders.created_at)\n), _cm_orders__customers__spend_sum AS (\n SELECT\n SUM(customers.spend) AS \"orders.customers.spend_sum\"\n FROM customers AS customers\n), base AS (\n SELECT\n _base.\"orders.created_at\",\n _cm_orders__customers__spend_sum.\"orders.customers.spend_sum\"\n FROM _base\n CROSS JOIN _cm_orders__customers__spend_sum\n), step1 AS (\n SELECT\n \"orders.created_at\",\n \"orders.customers.spend_sum\",\n SUM(\"orders.customers.spend_sum\") OVER (ORDER BY \"orders.created_at\") AS \"orders._cumsum_inner\"\n FROM base\n), step2 AS (\n SELECT\n \"orders.created_at\",\n \"orders.customers.spend_sum\",\n \"orders._cumsum_inner\",\n SUM(\"orders._cumsum_inner\") OVER (ORDER BY \"orders.created_at\") AS \"orders.cc\"\n FROM step1\n)\nSELECT\n \"orders.created_at\",\n \"orders.customers.spend_sum\",\n \"orders._cumsum_inner\",\n \"orders.cc\"\nFROM step2\n) AS _outer", + "chain/cross_model_nested_window::postgres": "SELECT\n \"orders.created_at\",\n \"orders.cc\"\nFROM (\nWITH _base AS (\n SELECT\n DATE_TRUNC('MONTH', orders.created_at) AS \"orders.created_at\"\n FROM orders AS orders\n GROUP BY\n DATE_TRUNC('MONTH', orders.created_at)\n), _cm_orders__customers__spend_sum AS (\n SELECT\n SUM(customers.spend) AS \"orders.customers.spend_sum\"\n FROM customers AS customers\n), base AS (\n SELECT\n _base.\"orders.created_at\",\n _cm_orders__customers__spend_sum.\"orders.customers.spend_sum\"\n FROM _base\n CROSS JOIN _cm_orders__customers__spend_sum\n), step1 AS (\n SELECT\n \"orders.created_at\",\n \"orders.customers.spend_sum\",\n SUM(\"orders.customers.spend_sum\") OVER (ORDER BY \"orders.created_at\") AS \"orders._cumsum_inner\"\n FROM base\n), step2 AS (\n SELECT\n \"orders.created_at\",\n \"orders.customers.spend_sum\",\n \"orders._cumsum_inner\",\n SUM(\"orders._cumsum_inner\") OVER (ORDER BY \"orders.created_at\") AS \"orders.cc\"\n FROM step1\n)\nSELECT\n \"orders.created_at\",\n \"orders.customers.spend_sum\",\n \"orders._cumsum_inner\",\n \"orders.cc\"\nFROM step2\n) AS _outer", + "chain/cross_model_nested_window::sqlite": "SELECT\n \"orders.created_at\",\n \"orders.cc\"\nFROM (\nWITH _base AS (\n SELECT\n STRFTIME('%Y-%m-01', orders.created_at) AS \"orders.created_at\"\n FROM orders AS orders\n GROUP BY\n STRFTIME('%Y-%m-01', orders.created_at)\n), _cm_orders__customers__spend_sum AS (\n SELECT\n SUM(customers.spend) AS \"orders.customers.spend_sum\"\n FROM customers AS customers\n), base AS (\n SELECT\n _base.\"orders.created_at\",\n _cm_orders__customers__spend_sum.\"orders.customers.spend_sum\"\n FROM _base\n CROSS JOIN _cm_orders__customers__spend_sum\n), step1 AS (\n SELECT\n \"orders.created_at\",\n \"orders.customers.spend_sum\",\n SUM(\"orders.customers.spend_sum\") OVER (ORDER BY \"orders.created_at\") AS \"orders._cumsum_inner\"\n FROM base\n), step2 AS (\n SELECT\n \"orders.created_at\",\n \"orders.customers.spend_sum\",\n \"orders._cumsum_inner\",\n SUM(\"orders._cumsum_inner\") OVER (ORDER BY \"orders.created_at\") AS \"orders.cc\"\n FROM step1\n)\nSELECT\n \"orders.created_at\",\n \"orders.customers.spend_sum\",\n \"orders._cumsum_inner\",\n \"orders.cc\"\nFROM step2\n) AS _outer", + "chain/cross_model_nested_window::tsql": "WITH _base AS (\n SELECT\n DATETRUNC(MONTH, orders.created_at) AS [orders___created_at]\n FROM orders AS orders\n GROUP BY\n DATETRUNC(MONTH, orders.created_at)\n), _cm_orders__customers__spend_sum AS (\n SELECT\n SUM(customers.spend) AS [orders___customers___spend_sum]\n FROM customers AS customers\n), base AS (\n SELECT\n _base.[orders___created_at] AS [orders___created_at],\n _cm_orders__customers__spend_sum.[orders___customers___spend_sum] AS [orders___customers___spend_sum]\n FROM _base\n CROSS JOIN _cm_orders__customers__spend_sum\n), step1 AS (\n SELECT\n [orders___created_at] AS [orders___created_at],\n [orders___customers___spend_sum] AS [orders___customers___spend_sum],\n SUM([orders___customers___spend_sum]) OVER (ORDER BY [orders___created_at]) AS [orders____cumsum_inner]\n FROM base\n), step2 AS (\n SELECT\n [orders___created_at] AS [orders___created_at],\n [orders___customers___spend_sum] AS [orders___customers___spend_sum],\n [orders____cumsum_inner] AS [orders____cumsum_inner],\n SUM([orders____cumsum_inner]) OVER (ORDER BY [orders___created_at]) AS [orders___cc]\n FROM step1\n)\nSELECT\n [orders___created_at],\n [orders___cc]\nFROM (\n SELECT\n [orders___created_at] AS [orders___created_at],\n [orders___customers___spend_sum] AS [orders___customers___spend_sum],\n [orders____cumsum_inner] AS [orders____cumsum_inner],\n [orders___cc] AS [orders___cc]\n FROM step2\n) AS _outer", "chain/cross_model_window::bigquery": "SELECT\n `orders___created_at`,\n `orders___cs`,\n `orders___run`\nFROM (\nWITH _base AS (\n SELECT\n DATE_TRUNC(orders.created_at, MONTH) AS `orders___created_at`\n FROM orders AS orders\n GROUP BY\n DATE_TRUNC(orders.created_at, MONTH)\n), _cm_orders__customers__spend_sum AS (\n SELECT\n SUM(customers.spend) AS `orders___customers___spend_sum`\n FROM customers AS customers\n), base AS (\n SELECT\n _base.`orders___created_at`,\n _cm_orders__customers__spend_sum.`orders___customers___spend_sum` AS `orders___cs`\n FROM _base\n CROSS JOIN _cm_orders__customers__spend_sum\n), step1 AS (\n SELECT\n `orders___created_at`,\n `orders___cs`,\n SUM(`orders___cs`) OVER (ORDER BY `orders___created_at`) AS `orders___run`\n FROM base\n)\nSELECT\n `orders___created_at`,\n `orders___cs`,\n `orders___run`\nFROM step1\n) AS _outer", "chain/cross_model_window::duckdb": "SELECT\n \"orders.created_at\",\n \"orders.cs\",\n \"orders.run\"\nFROM (\nWITH _base AS (\n SELECT\n DATE_TRUNC('MONTH', orders.created_at) AS \"orders.created_at\"\n FROM orders AS orders\n GROUP BY\n DATE_TRUNC('MONTH', orders.created_at)\n), _cm_orders__customers__spend_sum AS (\n SELECT\n SUM(customers.spend) AS \"orders.customers.spend_sum\"\n FROM customers AS customers\n), base AS (\n SELECT\n _base.\"orders.created_at\",\n _cm_orders__customers__spend_sum.\"orders.customers.spend_sum\" AS \"orders.cs\"\n FROM _base\n CROSS JOIN _cm_orders__customers__spend_sum\n), step1 AS (\n SELECT\n \"orders.created_at\",\n \"orders.cs\",\n SUM(\"orders.cs\") OVER (ORDER BY \"orders.created_at\") AS \"orders.run\"\n FROM base\n)\nSELECT\n \"orders.created_at\",\n \"orders.cs\",\n \"orders.run\"\nFROM step1\n) AS _outer", "chain/cross_model_window::postgres": "SELECT\n \"orders.created_at\",\n \"orders.cs\",\n \"orders.run\"\nFROM (\nWITH _base AS (\n SELECT\n DATE_TRUNC('MONTH', orders.created_at) AS \"orders.created_at\"\n FROM orders AS orders\n GROUP BY\n DATE_TRUNC('MONTH', orders.created_at)\n), _cm_orders__customers__spend_sum AS (\n SELECT\n SUM(customers.spend) AS \"orders.customers.spend_sum\"\n FROM customers AS customers\n), base AS (\n SELECT\n _base.\"orders.created_at\",\n _cm_orders__customers__spend_sum.\"orders.customers.spend_sum\" AS \"orders.cs\"\n FROM _base\n CROSS JOIN _cm_orders__customers__spend_sum\n), step1 AS (\n SELECT\n \"orders.created_at\",\n \"orders.cs\",\n SUM(\"orders.cs\") OVER (ORDER BY \"orders.created_at\") AS \"orders.run\"\n FROM base\n)\nSELECT\n \"orders.created_at\",\n \"orders.cs\",\n \"orders.run\"\nFROM step1\n) AS _outer", @@ -14,6 +19,11 @@ "chain/local_multi_step::postgres": "SELECT\n \"orders.created_at\",\n \"orders.rev\",\n \"orders.cs\",\n \"orders.ch\"\nFROM (\nWITH base AS (\n SELECT\n DATE_TRUNC('MONTH', orders.created_at) AS \"orders.created_at\",\n CAST(SUM(orders.amount) AS DOUBLE PRECISION) AS \"orders.rev\"\n FROM orders AS orders\n GROUP BY\n DATE_TRUNC('MONTH', orders.created_at)\n), step1 AS (\n SELECT\n \"orders.created_at\",\n \"orders.rev\",\n SUM(\"orders.rev\") OVER (ORDER BY \"orders.created_at\") AS \"orders.cs\"\n FROM base\n), shifted__time_shift_inner AS (\n SELECT\n DATE_TRUNC('MONTH', CAST(orders.created_at + INTERVAL '1 MONTH' AS TIMESTAMP)) AS \"orders.created_at\",\n CAST(SUM(orders.amount) AS DOUBLE PRECISION) AS \"orders.rev\"\n FROM orders AS orders\n GROUP BY\n DATE_TRUNC('MONTH', CAST(orders.created_at + INTERVAL '1 MONTH' AS TIMESTAMP))\n), sjoin__time_shift_inner AS (\n SELECT\n step1.\"orders.created_at\",\n step1.\"orders.rev\",\n step1.\"orders.cs\",\n shifted__time_shift_inner.\"orders.rev\" AS \"orders._time_shift_inner\"\n FROM step1\n LEFT JOIN shifted__time_shift_inner\n ON step1.\"orders.created_at\" IS NOT DISTINCT FROM shifted__time_shift_inner.\"orders.created_at\"\n), step2 AS (\n SELECT\n \"orders.created_at\",\n \"orders.rev\",\n \"orders.cs\",\n \"orders._time_shift_inner\",\n \"orders.rev\" - \"orders._time_shift_inner\" AS \"orders.ch\"\n FROM sjoin__time_shift_inner\n)\nSELECT\n \"orders.created_at\",\n \"orders.rev\",\n \"orders.cs\",\n \"orders._time_shift_inner\",\n \"orders.ch\"\nFROM step2\n) AS _outer", "chain/local_multi_step::sqlite": "SELECT\n \"orders.created_at\",\n \"orders.rev\",\n \"orders.cs\",\n \"orders.ch\"\nFROM (\nWITH base AS (\n SELECT\n STRFTIME('%Y-%m-01', orders.created_at) AS \"orders.created_at\",\n CAST(SUM(orders.amount) AS REAL) AS \"orders.rev\"\n FROM orders AS orders\n GROUP BY\n STRFTIME('%Y-%m-01', orders.created_at)\n), step1 AS (\n SELECT\n \"orders.created_at\",\n \"orders.rev\",\n SUM(\"orders.rev\") OVER (ORDER BY \"orders.created_at\") AS \"orders.cs\"\n FROM base\n), shifted__time_shift_inner AS (\n SELECT\n STRFTIME('%Y-%m-01', DATE(orders.created_at, '1 months')) AS \"orders.created_at\",\n CAST(SUM(orders.amount) AS REAL) AS \"orders.rev\"\n FROM orders AS orders\n GROUP BY\n STRFTIME('%Y-%m-01', DATE(orders.created_at, '1 months'))\n), sjoin__time_shift_inner AS (\n SELECT\n step1.\"orders.created_at\",\n step1.\"orders.rev\",\n step1.\"orders.cs\",\n shifted__time_shift_inner.\"orders.rev\" AS \"orders._time_shift_inner\"\n FROM step1\n LEFT JOIN shifted__time_shift_inner\n ON step1.\"orders.created_at\" IS shifted__time_shift_inner.\"orders.created_at\"\n), step2 AS (\n SELECT\n \"orders.created_at\",\n \"orders.rev\",\n \"orders.cs\",\n \"orders._time_shift_inner\",\n \"orders.rev\" - \"orders._time_shift_inner\" AS \"orders.ch\"\n FROM sjoin__time_shift_inner\n)\nSELECT\n \"orders.created_at\",\n \"orders.rev\",\n \"orders.cs\",\n \"orders._time_shift_inner\",\n \"orders.ch\"\nFROM step2\n) AS _outer", "chain/local_multi_step::tsql": "WITH base AS (\n SELECT\n DATETRUNC(MONTH, orders.created_at) AS [orders___created_at],\n CAST(SUM(orders.amount) AS FLOAT) AS [orders___rev]\n FROM orders AS orders\n GROUP BY\n DATETRUNC(MONTH, orders.created_at)\n), step1 AS (\n SELECT\n [orders___created_at] AS [orders___created_at],\n [orders___rev] AS [orders___rev],\n SUM([orders___rev]) OVER (ORDER BY [orders___created_at]) AS [orders___cs]\n FROM base\n), shifted__time_shift_inner AS (\n SELECT\n DATETRUNC(MONTH, CAST(DATEADD(MONTH, 1, orders.created_at) AS DATETIME2)) AS [orders___created_at],\n CAST(SUM(orders.amount) AS FLOAT) AS [orders___rev]\n FROM orders AS orders\n GROUP BY\n DATETRUNC(MONTH, CAST(DATEADD(MONTH, 1, orders.created_at) AS DATETIME2))\n), sjoin__time_shift_inner AS (\n SELECT\n step1.[orders___created_at] AS [orders___created_at],\n step1.[orders___rev] AS [orders___rev],\n step1.[orders___cs] AS [orders___cs],\n shifted__time_shift_inner.[orders___rev] AS [orders____time_shift_inner]\n FROM step1\n LEFT JOIN shifted__time_shift_inner\n ON (\n step1.[orders___created_at] = shifted__time_shift_inner.[orders___created_at]\n OR (\n step1.[orders___created_at] IS NULL\n AND shifted__time_shift_inner.[orders___created_at] IS NULL\n )\n )\n), step2 AS (\n SELECT\n [orders___created_at] AS [orders___created_at],\n [orders___rev] AS [orders___rev],\n [orders___cs] AS [orders___cs],\n [orders____time_shift_inner] AS [orders____time_shift_inner],\n [orders___rev] - [orders____time_shift_inner] AS [orders___ch]\n FROM sjoin__time_shift_inner\n)\nSELECT\n [orders___created_at],\n [orders___rev],\n [orders___cs],\n [orders___ch]\nFROM (\n SELECT\n [orders___created_at] AS [orders___created_at],\n [orders___rev] AS [orders___rev],\n [orders___cs] AS [orders___cs],\n [orders____time_shift_inner] AS [orders____time_shift_inner],\n [orders___ch] AS [orders___ch]\n FROM step2\n) AS _outer", + "chain/local_nested_window::bigquery": "SELECT\n `orders___created_at`,\n `orders___cc`\nFROM (\nWITH base AS (\n SELECT\n DATE_TRUNC(orders.created_at, MONTH) AS `orders___created_at`,\n SUM(orders.amount) AS `orders___amount_sum`\n FROM orders AS orders\n GROUP BY\n DATE_TRUNC(orders.created_at, MONTH)\n), step1 AS (\n SELECT\n `orders___created_at`,\n `orders___amount_sum`,\n SUM(`orders___amount_sum`) OVER (ORDER BY `orders___created_at`) AS `orders____cumsum_inner`\n FROM base\n), step2 AS (\n SELECT\n `orders___created_at`,\n `orders___amount_sum`,\n `orders____cumsum_inner`,\n SUM(`orders____cumsum_inner`) OVER (ORDER BY `orders___created_at`) AS `orders___cc`\n FROM step1\n)\nSELECT\n `orders___created_at`,\n `orders___amount_sum`,\n `orders____cumsum_inner`,\n `orders___cc`\nFROM step2\n) AS _outer", + "chain/local_nested_window::duckdb": "SELECT\n \"orders.created_at\",\n \"orders.cc\"\nFROM (\nWITH base AS (\n SELECT\n DATE_TRUNC('MONTH', orders.created_at) AS \"orders.created_at\",\n SUM(orders.amount) AS \"orders.amount_sum\"\n FROM orders AS orders\n GROUP BY\n DATE_TRUNC('MONTH', orders.created_at)\n), step1 AS (\n SELECT\n \"orders.created_at\",\n \"orders.amount_sum\",\n SUM(\"orders.amount_sum\") OVER (ORDER BY \"orders.created_at\") AS \"orders._cumsum_inner\"\n FROM base\n), step2 AS (\n SELECT\n \"orders.created_at\",\n \"orders.amount_sum\",\n \"orders._cumsum_inner\",\n SUM(\"orders._cumsum_inner\") OVER (ORDER BY \"orders.created_at\") AS \"orders.cc\"\n FROM step1\n)\nSELECT\n \"orders.created_at\",\n \"orders.amount_sum\",\n \"orders._cumsum_inner\",\n \"orders.cc\"\nFROM step2\n) AS _outer", + "chain/local_nested_window::postgres": "SELECT\n \"orders.created_at\",\n \"orders.cc\"\nFROM (\nWITH base AS (\n SELECT\n DATE_TRUNC('MONTH', orders.created_at) AS \"orders.created_at\",\n SUM(orders.amount) AS \"orders.amount_sum\"\n FROM orders AS orders\n GROUP BY\n DATE_TRUNC('MONTH', orders.created_at)\n), step1 AS (\n SELECT\n \"orders.created_at\",\n \"orders.amount_sum\",\n SUM(\"orders.amount_sum\") OVER (ORDER BY \"orders.created_at\") AS \"orders._cumsum_inner\"\n FROM base\n), step2 AS (\n SELECT\n \"orders.created_at\",\n \"orders.amount_sum\",\n \"orders._cumsum_inner\",\n SUM(\"orders._cumsum_inner\") OVER (ORDER BY \"orders.created_at\") AS \"orders.cc\"\n FROM step1\n)\nSELECT\n \"orders.created_at\",\n \"orders.amount_sum\",\n \"orders._cumsum_inner\",\n \"orders.cc\"\nFROM step2\n) AS _outer", + "chain/local_nested_window::sqlite": "SELECT\n \"orders.created_at\",\n \"orders.cc\"\nFROM (\nWITH base AS (\n SELECT\n STRFTIME('%Y-%m-01', orders.created_at) AS \"orders.created_at\",\n SUM(orders.amount) AS \"orders.amount_sum\"\n FROM orders AS orders\n GROUP BY\n STRFTIME('%Y-%m-01', orders.created_at)\n), step1 AS (\n SELECT\n \"orders.created_at\",\n \"orders.amount_sum\",\n SUM(\"orders.amount_sum\") OVER (ORDER BY \"orders.created_at\") AS \"orders._cumsum_inner\"\n FROM base\n), step2 AS (\n SELECT\n \"orders.created_at\",\n \"orders.amount_sum\",\n \"orders._cumsum_inner\",\n SUM(\"orders._cumsum_inner\") OVER (ORDER BY \"orders.created_at\") AS \"orders.cc\"\n FROM step1\n)\nSELECT\n \"orders.created_at\",\n \"orders.amount_sum\",\n \"orders._cumsum_inner\",\n \"orders.cc\"\nFROM step2\n) AS _outer", + "chain/local_nested_window::tsql": "WITH base AS (\n SELECT\n DATETRUNC(MONTH, orders.created_at) AS [orders___created_at],\n SUM(orders.amount) AS [orders___amount_sum]\n FROM orders AS orders\n GROUP BY\n DATETRUNC(MONTH, orders.created_at)\n), step1 AS (\n SELECT\n [orders___created_at] AS [orders___created_at],\n [orders___amount_sum] AS [orders___amount_sum],\n SUM([orders___amount_sum]) OVER (ORDER BY [orders___created_at]) AS [orders____cumsum_inner]\n FROM base\n), step2 AS (\n SELECT\n [orders___created_at] AS [orders___created_at],\n [orders___amount_sum] AS [orders___amount_sum],\n [orders____cumsum_inner] AS [orders____cumsum_inner],\n SUM([orders____cumsum_inner]) OVER (ORDER BY [orders___created_at]) AS [orders___cc]\n FROM step1\n)\nSELECT\n [orders___created_at],\n [orders___cc]\nFROM (\n SELECT\n [orders___created_at] AS [orders___created_at],\n [orders___amount_sum] AS [orders___amount_sum],\n [orders____cumsum_inner] AS [orders____cumsum_inner],\n [orders___cc] AS [orders___cc]\n FROM step2\n) AS _outer", "order/combined_cross_model::bigquery": "WITH _base AS (\n SELECT\n orders.status AS `orders___status`,\n CAST(SUM(orders.amount) AS FLOAT64) AS `orders___rev`\n FROM orders AS orders\n GROUP BY\n orders.status\n), _cm_orders__customers__spend_sum AS (\n SELECT\n SUM(customers.spend) AS `orders___customers___spend_sum`\n FROM customers AS customers\n)\nSELECT\n _base.`orders___status`,\n _base.`orders___rev`,\n _cm_orders__customers__spend_sum.`orders___customers___spend_sum` AS `orders___cs`\nFROM _base\nCROSS JOIN _cm_orders__customers__spend_sum\nORDER BY\n _cm_orders__customers__spend_sum.`orders___customers___spend_sum` DESC", "order/combined_cross_model::duckdb": "WITH _base AS (\n SELECT\n orders.status AS \"orders.status\",\n CAST(SUM(orders.amount) AS DOUBLE) AS \"orders.rev\"\n FROM orders AS orders\n GROUP BY\n orders.status\n), _cm_orders__customers__spend_sum AS (\n SELECT\n SUM(customers.spend) AS \"orders.customers.spend_sum\"\n FROM customers AS customers\n)\nSELECT\n _base.\"orders.status\",\n _base.\"orders.rev\",\n _cm_orders__customers__spend_sum.\"orders.customers.spend_sum\" AS \"orders.cs\"\nFROM _base\nCROSS JOIN _cm_orders__customers__spend_sum\nORDER BY\n _cm_orders__customers__spend_sum.\"orders.customers.spend_sum\" DESC", "order/combined_cross_model::postgres": "WITH _base AS (\n SELECT\n orders.status AS \"orders.status\",\n CAST(SUM(orders.amount) AS DOUBLE PRECISION) AS \"orders.rev\"\n FROM orders AS orders\n GROUP BY\n orders.status\n), _cm_orders__customers__spend_sum AS (\n SELECT\n SUM(customers.spend) AS \"orders.customers.spend_sum\"\n FROM customers AS customers\n)\nSELECT\n _base.\"orders.status\",\n _base.\"orders.rev\",\n _cm_orders__customers__spend_sum.\"orders.customers.spend_sum\" AS \"orders.cs\"\nFROM _base\nCROSS JOIN _cm_orders__customers__spend_sum\nORDER BY\n _cm_orders__customers__spend_sum.\"orders.customers.spend_sum\" DESC NULLS LAST", @@ -74,6 +84,11 @@ "order/windowed_cte::postgres": "WITH _base AS (\n SELECT\n DATE_TRUNC('MONTH', orders.created_at) AS \"orders.created_at\"\n FROM orders AS orders\n GROUP BY\n DATE_TRUNC('MONTH', orders.created_at)\n), _wm_orders__w AS (\n SELECT\n _base.\"orders.created_at\",\n CAST(SUM(_src._w_value) AS DOUBLE PRECISION) AS \"orders.w\"\n FROM _base\n LEFT JOIN (\n SELECT\n orders.created_at AS _w_time,\n orders.amount AS _w_value\n FROM orders AS orders\n ) AS _src\n ON _src._w_time >= _base.\"orders.created_at\" + INTERVAL '1 MONTH' - INTERVAL '90 DAY'\n AND _src._w_time < _base.\"orders.created_at\" + INTERVAL '1 MONTH'\n GROUP BY\n _base.\"orders.created_at\"\n)\nSELECT\n _base.\"orders.created_at\",\n _wm_orders__w.\"orders.w\"\nFROM _base\nLEFT JOIN _wm_orders__w\n ON _base.\"orders.created_at\" IS NOT DISTINCT FROM _wm_orders__w.\"orders.created_at\"\nORDER BY\n _wm_orders__w.\"orders.w\" DESC NULLS LAST", "order/windowed_cte::sqlite": "WITH _base AS (\n SELECT\n STRFTIME('%Y-%m-01', orders.created_at) AS \"orders.created_at\"\n FROM orders AS orders\n GROUP BY\n STRFTIME('%Y-%m-01', orders.created_at)\n), _wm_orders__w AS (\n SELECT\n _base.\"orders.created_at\",\n CAST(SUM(_src._w_value) AS REAL) AS \"orders.w\"\n FROM _base\n LEFT JOIN (\n SELECT\n orders.created_at AS _w_time,\n orders.amount AS _w_value\n FROM orders AS orders\n ) AS _src\n ON _src._w_time >= DATETIME(DATETIME(_base.\"orders.created_at\", '+1 months'), '-90 days')\n AND _src._w_time < DATETIME(_base.\"orders.created_at\", '+1 months')\n GROUP BY\n _base.\"orders.created_at\"\n)\nSELECT\n _base.\"orders.created_at\",\n _wm_orders__w.\"orders.w\"\nFROM _base\nLEFT JOIN _wm_orders__w\n ON _base.\"orders.created_at\" IS _wm_orders__w.\"orders.created_at\"\nORDER BY\n _wm_orders__w.\"orders.w\" DESC", "order/windowed_cte::tsql": "WITH _base AS (\n SELECT\n DATETRUNC(month, orders.created_at) AS [orders___created_at]\n FROM orders AS orders\n GROUP BY\n DATETRUNC(month, orders.created_at)\n), _wm_orders__w AS (\n SELECT\n _base.[orders___created_at] AS [orders___created_at],\n CAST(SUM(_src._w_value) AS FLOAT) AS [orders___w]\n FROM _base\n LEFT JOIN (\n SELECT\n orders.created_at AS _w_time,\n orders.amount AS _w_value\n FROM orders AS orders\n ) AS _src\n ON _src._w_time >= DATEADD(DAY, -90, DATEADD(MONTH, 1, _base.[orders___created_at]))\n AND _src._w_time < DATEADD(MONTH, 1, _base.[orders___created_at])\n GROUP BY\n _base.[orders___created_at]\n)\nSELECT\n _base.[orders___created_at],\n _wm_orders__w.[orders___w]\nFROM _base\nLEFT JOIN _wm_orders__w\n ON (\n _base.[orders___created_at] = _wm_orders__w.[orders___created_at]\n OR (\n _base.[orders___created_at] IS NULL AND _wm_orders__w.[orders___created_at] IS NULL\n )\n )\nORDER BY\n _wm_orders__w.[orders___w] DESC", + "order/windowed_cte_date_range::bigquery": "WITH _base AS (\n SELECT\n DATE_TRUNC(orders.created_at, MONTH) AS `orders___created_at`\n FROM orders AS orders\n WHERE\n orders.created_at BETWEEN '2024-01-01' AND '2024-06-30'\n GROUP BY\n DATE_TRUNC(orders.created_at, MONTH)\n), _wm_orders__w AS (\n SELECT\n _base.`orders___created_at`,\n CAST(SUM(_src._w_value) AS FLOAT64) AS `orders___w`\n FROM _base\n LEFT JOIN (\n SELECT\n orders.created_at AS _w_time,\n orders.amount AS _w_value\n FROM orders AS orders\n ) AS _src\n ON _src._w_time >= _base.`orders___created_at` + INTERVAL 1 MONTH - INTERVAL 90 DAY\n AND _src._w_time < _base.`orders___created_at` + INTERVAL 1 MONTH\n GROUP BY\n _base.`orders___created_at`\n)\nSELECT\n _base.`orders___created_at`,\n _wm_orders__w.`orders___w`\nFROM _base\nLEFT JOIN _wm_orders__w\n ON _base.`orders___created_at` IS NOT DISTINCT FROM _wm_orders__w.`orders___created_at`\nORDER BY\n _wm_orders__w.`orders___w` DESC", + "order/windowed_cte_date_range::duckdb": "WITH _base AS (\n SELECT\n DATE_TRUNC('MONTH', orders.created_at) AS \"orders.created_at\"\n FROM orders AS orders\n WHERE\n orders.created_at BETWEEN '2024-01-01' AND '2024-06-30'\n GROUP BY\n DATE_TRUNC('MONTH', orders.created_at)\n), _wm_orders__w AS (\n SELECT\n _base.\"orders.created_at\",\n CAST(SUM(_src._w_value) AS DOUBLE) AS \"orders.w\"\n FROM _base\n LEFT JOIN (\n SELECT\n orders.created_at AS _w_time,\n orders.amount AS _w_value\n FROM orders AS orders\n ) AS _src\n ON _src._w_time >= _base.\"orders.created_at\" + INTERVAL 1 MONTH - INTERVAL 90 DAY\n AND _src._w_time < _base.\"orders.created_at\" + INTERVAL 1 MONTH\n GROUP BY\n _base.\"orders.created_at\"\n)\nSELECT\n _base.\"orders.created_at\",\n _wm_orders__w.\"orders.w\"\nFROM _base\nLEFT JOIN _wm_orders__w\n ON _base.\"orders.created_at\" IS NOT DISTINCT FROM _wm_orders__w.\"orders.created_at\"\nORDER BY\n _wm_orders__w.\"orders.w\" DESC", + "order/windowed_cte_date_range::postgres": "WITH _base AS (\n SELECT\n DATE_TRUNC('MONTH', orders.created_at) AS \"orders.created_at\"\n FROM orders AS orders\n WHERE\n orders.created_at BETWEEN '2024-01-01' AND '2024-06-30'\n GROUP BY\n DATE_TRUNC('MONTH', orders.created_at)\n), _wm_orders__w AS (\n SELECT\n _base.\"orders.created_at\",\n CAST(SUM(_src._w_value) AS DOUBLE PRECISION) AS \"orders.w\"\n FROM _base\n LEFT JOIN (\n SELECT\n orders.created_at AS _w_time,\n orders.amount AS _w_value\n FROM orders AS orders\n ) AS _src\n ON _src._w_time >= _base.\"orders.created_at\" + INTERVAL '1 MONTH' - INTERVAL '90 DAY'\n AND _src._w_time < _base.\"orders.created_at\" + INTERVAL '1 MONTH'\n GROUP BY\n _base.\"orders.created_at\"\n)\nSELECT\n _base.\"orders.created_at\",\n _wm_orders__w.\"orders.w\"\nFROM _base\nLEFT JOIN _wm_orders__w\n ON _base.\"orders.created_at\" IS NOT DISTINCT FROM _wm_orders__w.\"orders.created_at\"\nORDER BY\n _wm_orders__w.\"orders.w\" DESC NULLS LAST", + "order/windowed_cte_date_range::sqlite": "WITH _base AS (\n SELECT\n STRFTIME('%Y-%m-01', orders.created_at) AS \"orders.created_at\"\n FROM orders AS orders\n WHERE\n orders.created_at BETWEEN '2024-01-01' AND '2024-06-30'\n GROUP BY\n STRFTIME('%Y-%m-01', orders.created_at)\n), _wm_orders__w AS (\n SELECT\n _base.\"orders.created_at\",\n CAST(SUM(_src._w_value) AS REAL) AS \"orders.w\"\n FROM _base\n LEFT JOIN (\n SELECT\n orders.created_at AS _w_time,\n orders.amount AS _w_value\n FROM orders AS orders\n ) AS _src\n ON _src._w_time >= DATETIME(DATETIME(_base.\"orders.created_at\", '+1 months'), '-90 days')\n AND _src._w_time < DATETIME(_base.\"orders.created_at\", '+1 months')\n GROUP BY\n _base.\"orders.created_at\"\n)\nSELECT\n _base.\"orders.created_at\",\n _wm_orders__w.\"orders.w\"\nFROM _base\nLEFT JOIN _wm_orders__w\n ON _base.\"orders.created_at\" IS _wm_orders__w.\"orders.created_at\"\nORDER BY\n _wm_orders__w.\"orders.w\" DESC", + "order/windowed_cte_date_range::tsql": "WITH _base AS (\n SELECT\n DATETRUNC(month, orders.created_at) AS [orders___created_at]\n FROM orders AS orders\n WHERE\n orders.created_at BETWEEN '2024-01-01' AND '2024-06-30'\n GROUP BY\n DATETRUNC(month, orders.created_at)\n), _wm_orders__w AS (\n SELECT\n _base.[orders___created_at] AS [orders___created_at],\n CAST(SUM(_src._w_value) AS FLOAT) AS [orders___w]\n FROM _base\n LEFT JOIN (\n SELECT\n orders.created_at AS _w_time,\n orders.amount AS _w_value\n FROM orders AS orders\n ) AS _src\n ON _src._w_time >= DATEADD(DAY, -90, DATEADD(MONTH, 1, _base.[orders___created_at]))\n AND _src._w_time < DATEADD(MONTH, 1, _base.[orders___created_at])\n GROUP BY\n _base.[orders___created_at]\n)\nSELECT\n _base.[orders___created_at],\n _wm_orders__w.[orders___w]\nFROM _base\nLEFT JOIN _wm_orders__w\n ON (\n _base.[orders___created_at] = _wm_orders__w.[orders___created_at]\n OR (\n _base.[orders___created_at] IS NULL AND _wm_orders__w.[orders___created_at] IS NULL\n )\n )\nORDER BY\n _wm_orders__w.[orders___w] DESC", "reroot/host_local_filter::bigquery": "WITH _base AS (\n SELECT\n customers__regions.name AS `orders___customers___regions___name`,\n CAST(SUM(orders.amount) AS FLOAT64) AS `orders___rev`\n FROM orders AS orders\n LEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\n LEFT JOIN regions AS customers__regions\n ON customers.region_id = customers__regions.id\n WHERE\n orders.status = 'A'\n GROUP BY\n customers__regions.name\n), _cm_orders__customers__spend_sum AS (\n SELECT\n regions.name AS `customers___regions___name`,\n CAST(SUM(customers.spend) AS FLOAT64) AS `customers___spend_sum`\n FROM customers AS customers\n LEFT JOIN regions AS regions\n ON customers.region_id = regions.id\n GROUP BY\n regions.name\n)\nSELECT\n _base.`orders___customers___regions___name`,\n _base.`orders___rev`,\n _cm_orders__customers__spend_sum.`customers___spend_sum` AS `orders___cs`\nFROM _base\nLEFT JOIN _cm_orders__customers__spend_sum\n ON _base.`orders___customers___regions___name` IS NOT DISTINCT FROM _cm_orders__customers__spend_sum.`customers___regions___name`", "reroot/host_local_filter::duckdb": "WITH _base AS (\n SELECT\n customers__regions.name AS \"orders.customers.regions.name\",\n CAST(SUM(orders.amount) AS DOUBLE) AS \"orders.rev\"\n FROM orders AS orders\n LEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\n LEFT JOIN regions AS customers__regions\n ON customers.region_id = customers__regions.id\n WHERE\n orders.status = 'A'\n GROUP BY\n customers__regions.name\n), _cm_orders__customers__spend_sum AS (\n SELECT\n regions.name AS \"customers.regions.name\",\n CAST(SUM(customers.spend) AS DOUBLE) AS \"customers.spend_sum\"\n FROM customers AS customers\n LEFT JOIN regions AS regions\n ON customers.region_id = regions.id\n GROUP BY\n regions.name\n)\nSELECT\n _base.\"orders.customers.regions.name\",\n _base.\"orders.rev\",\n _cm_orders__customers__spend_sum.\"customers.spend_sum\" AS \"orders.cs\"\nFROM _base\nLEFT JOIN _cm_orders__customers__spend_sum\n ON _base.\"orders.customers.regions.name\" IS NOT DISTINCT FROM _cm_orders__customers__spend_sum.\"customers.regions.name\"", "reroot/host_local_filter::postgres": "WITH _base AS (\n SELECT\n customers__regions.name AS \"orders.customers.regions.name\",\n CAST(SUM(orders.amount) AS DOUBLE PRECISION) AS \"orders.rev\"\n FROM orders AS orders\n LEFT JOIN customers AS customers\n ON orders.customer_id = customers.id\n LEFT JOIN regions AS customers__regions\n ON customers.region_id = customers__regions.id\n WHERE\n orders.status = 'A'\n GROUP BY\n customers__regions.name\n), _cm_orders__customers__spend_sum AS (\n SELECT\n regions.name AS \"customers.regions.name\",\n CAST(SUM(customers.spend) AS DOUBLE PRECISION) AS \"customers.spend_sum\"\n FROM customers AS customers\n LEFT JOIN regions AS regions\n ON customers.region_id = regions.id\n GROUP BY\n regions.name\n)\nSELECT\n _base.\"orders.customers.regions.name\",\n _base.\"orders.rev\",\n _cm_orders__customers__spend_sum.\"customers.spend_sum\" AS \"orders.cs\"\nFROM _base\nLEFT JOIN _cm_orders__customers__spend_sum\n ON _base.\"orders.customers.regions.name\" IS NOT DISTINCT FROM _cm_orders__customers__spend_sum.\"customers.regions.name\"", diff --git a/tests/integration/test_dev1756_identifier_length_pg.py b/tests/integration/test_dev1756_identifier_length_pg.py index dcd3fe94..d342aecf 100644 --- a/tests/integration/test_dev1756_identifier_length_pg.py +++ b/tests/integration/test_dev1756_identifier_length_pg.py @@ -141,11 +141,10 @@ def chain_env(_chain_storage): class TestPostgresIdentifierLength: async def test_server_truncates_at_63_bytes(self, chain_env) -> None: """Pin the 63-byte truncation premise against the live server.""" - client = chain_env._get_client( + client = chain_env._client_for( await chain_env._resolve_datasource( model=await chain_env.storage.get_model("SandboxInvoiceV2", data_source=DS), ), - ("probe", "probe"), ) rows = await client.execute(sql=f'SELECT 1 AS "{LONG_EMAIL}"') assert list(rows[0])[0] != LONG_EMAIL, "server did not truncate; premise broken" diff --git a/tests/test_agg_render_spec.py b/tests/test_agg_render_spec.py index b0d7e754..9a239122 100644 --- a/tests/test_agg_render_spec.py +++ b/tests/test_agg_render_spec.py @@ -474,243 +474,6 @@ def test_column_not_found_raises(self): ) -# --------------------------------------------------------------------------- -# _build_agg_render_spec_from_planned — first / last (time_column derivation) -# --------------------------------------------------------------------------- - - -class TestBuilderFirstLast: - def test_first_with_explicit_time_column_local(self): - # ``first(amount, created_at)`` — the positional ColumnKey arg - # becomes ``spec.time_column``. - key = AggregateKey( - source=ColumnKey(path=(), leaf="amount"), - agg="first", - args=(ColumnKey(path=(), leaf="created_at"),), - ) - slot = _slot( - key, - declared_name="amount_first", - public_name="amount_first", - slot_type=DataType.DOUBLE, - ) - spec = _invoke( - slot=slot, - key=key, - source_model=_orders_model(), - source_relation="orders", - full_alias="orders.amount_first", - ) - assert spec.aggregation == "first" - assert spec.time_column == "orders.created_at" - - def test_last_with_joined_time_column_uses_path_alias(self): - # A joined positional time arg uses the ``__``-joined path alias. - key = AggregateKey( - source=ColumnKey(path=(), leaf="amount"), - agg="last", - args=(ColumnKey(path=("customers",), leaf="signup_at"),), - ) - slot = _slot( - key, - declared_name="amount_last", - public_name="amount_last", - slot_type=DataType.DOUBLE, - ) - spec = _invoke( - slot=slot, - key=key, - source_model=_orders_model(), - source_relation="orders", - full_alias="orders.amount_last", - ) - assert spec.aggregation == "last" - # Mirrors legacy: ``__``-joined path + ``.``. - assert spec.time_column == "customers.signup_at" - - def test_last_with_derived_bare_time_column_local(self): - # DEV-1452 Codex fix: a ``ColumnSqlKey`` positional arg whose - # ``Column.sql`` is a bare identifier (renamed column) must - # resolve to ``.`` — previously the - # spec-build loop skipped ``ColumnSqlKey`` entirely and the - # ranked subquery silently fell back to the query's default - # ranking column. - key = AggregateKey( - source=ColumnKey(path=(), leaf="amount"), - agg="last", - args=( - ColumnSqlKey( - path=(), model="orders", column_name="created_at_alias", - ), - ), - ) - slot = _slot( - key, - declared_name="amount_last", - public_name="amount_last", - slot_type=DataType.DOUBLE, - ) - spec = _invoke( - slot=slot, - key=key, - source_model=_orders_model(), - source_relation="orders", - full_alias="orders.amount_last", - ) - assert spec.aggregation == "last" - # Bare-identifier derived column expands to its underlying SQL - # (``created_at``), qualified under the source relation. The - # derived NAME (``created_at_alias``) isn't projected in the - # ranked subquery's inner SELECT, so ORDER BY must reference the - # expanded form that IS visible (``orders.created_at``). - assert spec.time_column == "orders.created_at" - - def test_first_with_derived_expression_time_column_local(self): - # A ``ColumnSqlKey`` arg whose ``Column.sql`` is a non-trivial - # expression (``DATE_TRUNC(...)``) is materialised verbatim - # (after the sqlglot round-trip); its inner bare refs resolve - # against the ranked-subquery's FROM (the source relation). - key = AggregateKey( - source=ColumnKey(path=(), leaf="amount"), - agg="first", - args=( - ColumnSqlKey( - path=(), model="orders", column_name="created_at_day", - ), - ), - ) - slot = _slot( - key, - declared_name="amount_first", - public_name="amount_first", - slot_type=DataType.DOUBLE, - ) - spec = _invoke( - slot=slot, - key=key, - source_model=_orders_model(), - source_relation="orders", - full_alias="orders.amount_first", - ) - assert spec.aggregation == "first" - assert spec.time_column is not None - # Postgres-dialect rendering of DATE_TRUNC('day', created_at). - # Don't pin the exact whitespace — pin the structural tokens. - tc = spec.time_column.upper().replace(" ", "") - assert "DATE_TRUNC" in tc - assert "'DAY'" in tc - assert "CREATED_AT" in tc - - def test_cross_model_derived_time_column_resolves_after_reroot(self): - # DEV-1452 Stage B (DEV-1476 bug (c) + bug (d-cross)) — once the - # cross-model reroot pass strips the path from ``key.args`` - # symmetrically to kwargs (bug (c)), the spec builder sees the - # post-reroot shape (path=()) and resolves the derived column on - # the rerooted source model (the target — ``customers`` here). - # - # Pre-Stage-B this raised ``NotImplementedError`` because the - # reroot did NOT strip the path; the helper guarded against the - # path-bearing shape. Post-Stage-B the helper never sees a - # path-bearing ColumnSqlKey in this position (the reroot pass - # handles the path strip), and the local-derived branch resolves - # ``signup_at_alias`` on the rerooted source model. - customers = SlayerModel( - name="customers", - data_source="prod", - sql_table="customers", - columns=[ - Column(name="id", type=DataType.INT, primary_key=True), - Column(name="signup_at", type=DataType.TIMESTAMP), - Column( - name="signup_at_alias", - sql="signup_at", - type=DataType.TIMESTAMP, - ), - Column(name="amount", type=DataType.DOUBLE), - ], - ) - # POST-REROOT shape: path stripped on both source AND args. - key = AggregateKey( - source=ColumnKey(path=(), leaf="amount"), - agg="last", - args=( - ColumnSqlKey( - path=(), - model="customers", - column_name="signup_at_alias", - ), - ), - ) - slot = _slot( - key, - declared_name="amount_last", - public_name="amount_last", - slot_type=DataType.DOUBLE, - ) - spec = _invoke( - slot=slot, - key=key, - source_model=customers, - source_relation="customers", - full_alias="customers.amount_last", - ) - assert spec.time_column is not None - # The derived ``signup_at_alias`` (sql=``signup_at``) expands to - # the underlying SQL, qualified under the rerooted source relation. - assert "signup_at" in spec.time_column.lower(), spec.time_column - - def test_unknown_derived_time_column_raises(self): - # ``ColumnSqlKey`` whose ``column_name`` is not on ``source_model`` - # raises ValueError, mirroring the source-column lookup-miss path. - key = AggregateKey( - source=ColumnKey(path=(), leaf="amount"), - agg="last", - args=( - ColumnSqlKey( - path=(), model="orders", column_name="not_a_real_col", - ), - ), - ) - slot = _slot( - key, - declared_name="amount_last", - public_name="amount_last", - slot_type=DataType.DOUBLE, - ) - with pytest.raises(ValueError, match="Derived time column 'not_a_real_col'"): - _invoke( - slot=slot, - key=key, - source_model=_orders_model(), - source_relation="orders", - full_alias="orders.amount_last", - ) - - def test_first_with_filter_propagates_both(self): - key = AggregateKey( - source=ColumnKey(path=(), leaf="amount"), - agg="first", - args=(ColumnKey(path=(), leaf="created_at"),), - column_filter_key=SqlExprKey(canonical_sql="status = 'paid'"), - ) - slot = _slot( - key, - declared_name="paid_amount_first", - public_name="paid_amount_first", - slot_type=DataType.DOUBLE, - ) - spec = _invoke( - slot=slot, - key=key, - source_model=_orders_model(), - source_relation="orders", - full_alias="orders.paid_amount_first", - ) - assert spec.time_column == "orders.created_at" - assert spec.filter_sql is not None - assert "status" in spec.filter_sql - - # --------------------------------------------------------------------------- # _build_agg_render_spec_from_planned — custom aggregations # --------------------------------------------------------------------------- diff --git a/tests/test_column_expansion_sync.py b/tests/test_column_expansion_sync.py index 064351d3..c6569e5f 100644 --- a/tests/test_column_expansion_sync.py +++ b/tests/test_column_expansion_sync.py @@ -129,3 +129,69 @@ def test_unknown_alias_left_untouched() -> None: # cte_x.value untouched; A.bar stays qualified to the host alias. assert "cte_x.value" in _norm(out) assert "A.bar" in _norm(out) + + +def test_dev1752_subquery_inner_ref_not_qualified() -> None: + """DEV-1752: qualification is gated on root scope. The OUTER ``bar`` + qualifies to the host alias; the inner ``bar`` (a column of the subquery's + own FROM) is left bare even though ``bar`` is also a model column.""" + a = _model_a() + out = expand_derived_refs_sync( + sql="bar IN (SELECT bar FROM other_tbl)", model=a, alias_path="A", + resolve_model=_resolver({"A": a}), dialect="sqlite", + ) + assert _norm(out) == "A.bar IN (SELECT bar FROM other_tbl)" + + +def test_dev1752_root_inlines_while_subquery_local_untouched() -> None: + """DEV-1752 over-gating guard (non-differential): the root ``c1`` still + inlines to its definition; the inner ``c1`` is untouched. Non-differential + because the buggy qualification branch never fired for a DERIVED inner + column — only for trivial-base / non-model columns — so this guards against + the fix over-gating root inlining, not the bug itself.""" + a = _model_a() + out = expand_derived_refs_sync( + sql="c1 IN (SELECT c1 FROM other_tbl)", model=a, alias_path="A", + resolve_model=_resolver({"A": a}), dialect="sqlite", + ) + assert _norm(out) == "(A.raw_a + 1) IN (SELECT c1 FROM other_tbl)" + + +def test_dev1752_nonroot_explicit_ref_not_requalified() -> None: + """DEV-1752 (differential, alias != model name): an inner ref explicitly + qualified with the model name is left byte-for-byte untouched — it is NOT + rewritten to the host alias. Proves non-root columns are untouched, not just + coincidentally unchanged when alias == model name.""" + a = _model_a() + out = expand_derived_refs_sync( + sql="bar IN (SELECT A.bar FROM other_tbl)", model=a, alias_path="A_host", + resolve_model=_resolver({"A": a}), dialect="sqlite", + ) + # Outer bare bar -> host alias; inner A.bar stays A.bar (NOT A_host.bar). + assert _norm(out) == "A_host.bar IN (SELECT A.bar FROM other_tbl)" + + +def test_dev1752_set_operation_branches_leave_inner_bare() -> None: + """DEV-1752: both legs of a UNION are non-root scopes; their inner columns + stay bare even though ``bar`` is a model column.""" + a = _model_a() + out = expand_derived_refs_sync( + sql="bar IN (SELECT bar FROM t1 UNION SELECT bar FROM t2)", + model=a, alias_path="A", + resolve_model=_resolver({"A": a}), dialect="sqlite", + ) + assert _norm(out) == "A.bar IN (SELECT bar FROM t1 UNION SELECT bar FROM t2)" + + +def test_dev1752_cte_body_leaves_inner_bare() -> None: + """DEV-1752: a CTE (WITH) inside Mode-A is a non-root scope; refs in its + body and its select stay bare.""" + a = _model_a() + out = expand_derived_refs_sync( + sql="bar IN (WITH c AS (SELECT bar FROM t) SELECT bar FROM c)", + model=a, alias_path="A", + resolve_model=_resolver({"A": a}), dialect="sqlite", + ) + assert _norm(out) == ( + "A.bar IN (WITH c AS (SELECT bar FROM t) SELECT bar FROM c)" + ) diff --git a/tests/test_dev1476_first_last_explicit_time.py b/tests/test_dev1476_first_last_explicit_time.py index 03555dda..e7ee0ddd 100644 --- a/tests/test_dev1476_first_last_explicit_time.py +++ b/tests/test_dev1476_first_last_explicit_time.py @@ -943,127 +943,6 @@ def test_scalar_first_arg_gate_requires_default_time(self) -> None: resolve_ranking_time_key(key=key, root_model=orders, bundle=bundle) -# --------------------------------------------------------------------------- # -# Group 3 — ``_resolve_explicit_time_col`` renders through the resolver when a -# bundle is available. Bare/joined outputs equal the pre-resolver f-string (so -# they pin equivalence); the reserved-word case is the new correctness gain. -# --------------------------------------------------------------------------- # -class TestResolveExplicitTimeColViaResolver: - def _gen(self) -> SQLGenerator: - return SQLGenerator(dialect="postgres") - - def test_bare_local_time_arg(self) -> None: - orders = _u_orders() - key = AggregateKey( - source=ColumnKey(leaf="amount"), agg="last", - args=(ColumnKey(leaf="created_at"),), - ) - tc = self._gen()._resolve_explicit_time_col( - key=key, source_model=orders, source_relation="orders", - bundle=_u_bundle(orders), - ) - assert tc == "orders.created_at" - - def test_joined_time_arg_uses_path_alias(self) -> None: - orders = _u_orders() - key = AggregateKey( - source=ColumnKey(leaf="amount"), agg="last", - args=(ColumnKey(path=("customers",), leaf="signup_at"),), - ) - tc = self._gen()._resolve_explicit_time_col( - key=key, source_model=orders, source_relation="orders", - bundle=_u_bundle(orders), - ) - assert tc == "customers.signup_at" - - def test_reserved_word_relation_is_quoted(self) -> None: - # DEV-1686 gain: a reserved-word relation must be quoted in the rendered - # ORDER BY column. The pre-resolver f-string emitted it bare (invalid - # SQL); the resolver quotes it — this is red until Stage 6 reroutes - # rendering through ``ScopeFrame``. - order = SlayerModel( - name="order", sql_table="orders_tbl", data_source="prod", - columns=[ - Column(name="id", type=DataType.INT, primary_key=True), - Column(name="amount", type=DataType.DOUBLE), - Column(name="created_at", type=DataType.TIMESTAMP), - ], - ) - bundle = ResolvedSourceBundle(source_model=order, referenced_models=[order]) - key = AggregateKey( - source=ColumnKey(leaf="amount"), agg="last", - args=(ColumnKey(leaf="created_at"),), - ) - tc = self._gen()._resolve_explicit_time_col( - key=key, source_model=order, source_relation="order", bundle=bundle, - ) - assert '"order"' in tc, tc - assert "created_at" in tc - - def test_derived_columnsqlkey_resolves_via_resolver(self) -> None: - # A local derived (ColumnSqlKey) time arg resolves through the resolver - # (bundle set): ``signup_at_alias`` (sql=``signup_at``) expands to the - # underlying column, qualified under the source relation. - customers = _u_customers() - key = AggregateKey( - source=ColumnKey(leaf="amount"), agg="last", - args=(ColumnSqlKey(path=(), model="customers", column_name="signup_at_alias"),), - ) - tc = self._gen()._resolve_explicit_time_col( - key=key, source_model=customers, source_relation="customers", - bundle=_u_bundle(_u_orders()), - ) - assert tc == "customers.signup_at", tc - - def test_missing_columnsqlkey_raises_value_error(self) -> None: - # A ColumnSqlKey arg naming a column not on the source model raises the - # clear ValueError BEFORE any resolve (bundle set — proves the not-found - # guard precedes the ScopeFrame path, which would otherwise fall back to - # the bare name and silently mis-rank). - orders = _u_orders() - key = AggregateKey( - source=ColumnKey(leaf="amount"), agg="last", - args=(ColumnSqlKey(path=(), model="orders", column_name="not_a_real_col"),), - ) - # Hoist the generator + bundle out so the ONLY call that can raise inside - # ``pytest.raises`` is the one under test (Sonar S5778). - gen = self._gen() - bundle = _u_bundle(orders) - with pytest.raises(ValueError, match="Derived time column 'not_a_real_col'"): - gen._resolve_explicit_time_col( - key=key, source_model=orders, source_relation="orders", - bundle=bundle, - ) - - def test_path_bearing_columnsqlkey_raises_dev1526_with_bundle(self) -> None: - # PINS A DEAD BRANCH. ``_resolve_explicit_time_col`` is still called - # for every aggregate, but no first/last reaches it since DEV-1748, so - # this guard cannot fire outside a direct unit call like this one. The - # ranked CTE resolves its ranking key through its own scope, which is - # what removed the limitation the guard describes (see the un-xfailed - # ``test_a_joined_derived_time_arg_ranks_by_the_joined_expression`` in - # tests/test_dev1748_first_last_matrix.py). Kept until PR 6 removes the - # branch, per P-J. - # - # The residual-hop guard fires BEFORE resolution even when a bundle is - # available — the existing guard pin (test_reroot_aggregate_key.py) runs - # bundle=None; this fixes the ordering with a bundle set. - orders = _u_orders() - key = AggregateKey( - source=ColumnKey(leaf="amount"), agg="last", - args=(ColumnSqlKey(path=("regions",), model="regions", column_name="opened_day"),), - ) - # Hoist the generator + bundle out so the ONLY call that can raise inside - # ``pytest.raises`` is the one under test (Sonar S5778). - gen = self._gen() - bundle = _u_bundle(orders) - with pytest.raises(NotImplementedError, match="DEV-1526"): - gen._resolve_explicit_time_col( - key=key, source_model=orders, source_relation="orders", - bundle=bundle, - ) - - # --------------------------------------------------------------------------- # # Group 4 — end-to-end regression pins over a live SQLite DB. The join a # first/last time arg crosses must land in the ranked subquery's FROM (so the diff --git a/tests/test_dev1745_mode_a_door.py b/tests/test_dev1745_mode_a_door.py index 95ef4bd8..ce62ee07 100644 --- a/tests/test_dev1745_mode_a_door.py +++ b/tests/test_dev1745_mode_a_door.py @@ -207,20 +207,10 @@ def test_unresolvable_alias_is_not_requalified_to_root(self) -> None: ) assert "some_cte.flag" in out, out - @pytest.mark.xfail( - strict=True, - reason=( - "DEV-1752: known defect, tracked separately and deliberately NOT " - "fixed in this PR: qualification in _process_column_node_sync happens " - "BEFORE the root_scope_ids gate, so it is not scope-aware. Only " - "derived INLINING is gated. A column inside a subquery with its " - "own FROM is therefore qualified against the OUTER root: " - "'amount IN (SELECT amount FROM other_tbl)' becomes " - "'orders.amount IN (SELECT orders.amount FROM other_tbl)', " - "silently rebinding the inner reference to the wrong table." - ), - ) def test_subquery_column_is_not_qualified_against_outer_root(self) -> None: + # DEV-1752 fixed: qualification is now gated on root scope, so a column + # inside a subquery is left to bind in its own scope. See + # tests/test_dev1752_subquery_scope.py for the full pack. out = _sql_of( _scope().enter_predicate("amount IN (SELECT amount FROM other_tbl)") ) diff --git a/tests/test_dev1747_golden_sql.py b/tests/test_dev1747_golden_sql.py index 26f2df3b..695eb6ee 100644 --- a/tests/test_dev1747_golden_sql.py +++ b/tests/test_dev1747_golden_sql.py @@ -86,6 +86,18 @@ def _cases() -> dict: measures=[{"formula": "amount:sum(window='90d')", "name": "w"}], order=[{"column": "w", "direction": "desc"}], ), + # DEV-1777 C-a: a windowed measure WITH a well-formed date_range mints a + # date-range filter routed via ``date_range_fids`` (the src-row-phase + # gate the coupling re-derives). Pins the emitted BETWEEN so capturing + # the minted fid set instead of re-deriving it stays byte-identical. + "order/windowed_cte_date_range": _q( + time_dimensions=[{ + "dimension": "created_at", "granularity": "month", + "date_range": ["2024-01-01", "2024-06-30"], + }], + measures=[{"formula": "amount:sum(window='90d')", "name": "w"}], + order=[{"column": "w", "direction": "desc"}], + ), "order/transform_chain_wrap": _q( time_dimensions=_MONTH, measures=[{"formula": "cumsum(amount:sum)", "name": "cs"}], @@ -166,6 +178,21 @@ def _cases() -> dict: {"formula": "cumsum(customers.spend:sum)", "name": "run"}, ], ), + # --- DEV-1777 A0: dependency-split step-CTE shapes (Codex finding 2). --- + # A window over a window -> two dependent batches -> step1 then step2 + # (the dependency-split the extracted helper's ordering invariant guards). + # Single aggregate per case, so the base CTE has one column and the + # emitted SQL is deterministic across processes (multi-aggregate base + # column order is hash-seed dependent — see the _emit_step_cte unit test, + # which pins the multi-slot batch body deterministically instead). + "chain/local_nested_window": _q( + time_dimensions=_MONTH, + measures=[{"formula": "cumsum(cumsum(amount:sum))", "name": "cc"}], + ), + "chain/cross_model_nested_window": _q( + time_dimensions=_MONTH, + measures=[{"formula": "cumsum(cumsum(customers.spend:sum))", "name": "cc"}], + ), } @@ -255,3 +282,51 @@ def test_reroot_cases_actually_reroot(baseline) -> None: f"{key} is a re-rooting case with no ``_cm_`` alias — it stopped " f"re-rooting:\n{value}" ) + + +#: DEV-1777 (Codex plan-review finding 2): the step-CTE structure each +#: transform-chain case must still emit. ``step2`` distinguishes a dependency +#: split (a second Kahn batch, or a window followed by an unmaterialised POST) +#: from a single batch — the ``_emit_step_cte`` extraction must not collapse or +#: multiply batches. ``consecutive_periods`` layers its own ``cp_`` CTEs, not +#: ``step``. +_CHAIN_STEP_EXPECTATIONS: dict[str, dict[str, bool]] = { + "chain/local_multi_step": {"step1": True, "step2": True}, + "chain/local_consecutive_periods": {"cp": True}, + "chain/cross_model_window": {"step1": True, "step2": False}, + "chain/local_nested_window": {"step1": True, "step2": True}, + "chain/cross_model_nested_window": {"step1": True, "step2": True}, +} + + +def test_chain_cases_emit_expected_step_ctes(baseline) -> None: + """Vacuity guard for the transform-chain half. A case that silently stopped + reaching a step block would still "match golden" forever once the step-less + form was blessed; pin the presence/absence of ``step1`` / ``step2`` so a + dropped or collapsed batch fails loudly.""" + seen = set() + for key, value in baseline.items(): + if not key.startswith("chain/"): + continue + case_id = key.split("::", 1)[0] + seen.add(case_id) + expect = _CHAIN_STEP_EXPECTATIONS.get(case_id) + assert expect is not None, ( + f"{case_id} has no step-CTE expectation — add one to " + f"_CHAIN_STEP_EXPECTATIONS so the vacuity guard covers it" + ) + assert isinstance(value, str), f"{key} records an error, not SQL: {value}" + if expect.get("cp"): + assert "cp_" in value, ( + f"{key} is a consecutive-periods case with no ``cp_`` CTE:\n{value}" + ) + continue + assert ("step1" in value) is expect["step1"], ( + f"{key} step1 presence != {expect['step1']}:\n{value}" + ) + assert ("step2" in value) is expect["step2"], ( + f"{key} step2 presence != {expect['step2']} (batch collapsed or " + f"multiplied):\n{value}" + ) + unseen = set(_CHAIN_STEP_EXPECTATIONS) - seen + assert not unseen, f"expectations name cases not in the matrix: {sorted(unseen)}" diff --git a/tests/test_dev1752_subquery_scope.py b/tests/test_dev1752_subquery_scope.py new file mode 100644 index 00000000..5cf735c3 --- /dev/null +++ b/tests/test_dev1752_subquery_scope.py @@ -0,0 +1,310 @@ +"""DEV-1752 — Mode-A expansion must not qualify subquery columns against the +outer root. + +A subquery's own columns belong to the subquery's scope, not the outer model's. +Before the fix, ``_process_column_node_sync`` qualified every column node against +the outer root BEFORE the root-scope gate, so ``amount IN (SELECT amount FROM +other_tbl)`` became ``orders.amount IN (SELECT orders.amount FROM other_tbl)`` — +silently rebinding the inner ``amount`` to the wrong table. + +Contract: a Mode-A subquery must be self-contained (scope-closed). A bare name +binds to the subquery's own FROM; the expander leaves it alone. Correlation to +the outer model is NOT a supported Mode-A feature — an explicit outer reference +(``orders.col``) is a scope leak flagged by the ``assert_scope_closed`` +invariant (enabled by ``SLAYER_VALIDATE_SCOPES``: on in CI, off in production for +performance; the only legal correlated ref in generated SQL is the RLS +session-policy EXISTS). The expander is scope-aware and does not rebind such +refs; the scope-closure guard, not the expander, is what flags correlation, so +correlation is unsupported/undefined rather than hard-rejected at runtime. +""" +from __future__ import annotations + +import os +import sqlite3 +import tempfile + +import pytest +import sqlglot +from sqlglot import exp + +from slayer.core.enums import DataType +from slayer.core.models import Column, DatasourceConfig, SlayerModel +from slayer.core.query import ColumnRef, SlayerQuery +from slayer.engine.column_expansion import collect_root_scope_joined_paths +from slayer.engine.query_engine import SlayerQueryEngine +from slayer.engine.source_bundle import ResolvedSourceBundle +from slayer.sql.scope_check import ScopeLeakError +from slayer.storage.yaml_storage import YAMLStorage +# The orders/customers/regions ScopeFrame fixtures are shared with the DEV-1745 +# door pack; import them rather than duplicate the fixture triple. +from tests.test_dev1745_mode_a_door import ( + _customers, + _orders, + _regions, + _scope, + _sql_of, +) + + +def _inner_select_sql(rendered: str, dialect: str = "postgres") -> str: + """The rendered SQL of the first nested SELECT (the subquery body).""" + tree = sqlglot.parse_one(rendered, dialect=dialect) + inner = tree.find(exp.Select) + assert inner is not None, f"no subquery SELECT found in:\n{rendered}" + return inner.sql(dialect=dialect) + + +# --------------------------------------------------------------------------- # +# Scope-awareness: a subquery's own columns stay in the subquery's scope +# --------------------------------------------------------------------------- # +class TestSubqueryColumnsStayLocal: + + def test_non_correlated_in_subquery_keeps_inner_bare(self) -> None: + out = _sql_of( + _scope().enter_predicate("amount IN (SELECT amount FROM other_tbl)") + ) + inner = _inner_select_sql(out) + assert "orders" not in inner, ( + f"inner subquery column was qualified against the outer root: {out}" + ) + # The OUTER amount still qualifies to the root. + assert "orders.amount IN" in out, out + + def test_scalar_subquery_keeps_inner_bare(self) -> None: + out = _sql_of( + _scope().enter_predicate("status = (SELECT status FROM lookup LIMIT 1)") + ) + inner = _inner_select_sql(out) + assert "orders" not in inner, out + assert "orders.status =" in out, out + + def test_inner_name_colliding_with_root_column_stays_bare(self) -> None: + # ``amount`` IS a real orders column; the fix must be scope-driven, not + # name-driven — the inner ``amount`` still must not bind to orders. + out = _sql_of( + _scope().enter_predicate( + "amount IN (SELECT amount FROM other_tbl WHERE other_tbl.k = 1)" + ) + ) + inner = _inner_select_sql(out) + assert "orders" not in inner, out + + def test_expression_surface_keeps_inner_bare(self) -> None: + # Not predicate-only: a Column.sql scalar containing a subquery behaves + # the same through enter_expression. + out = _sql_of( + _scope().enter_expression("(SELECT amount FROM other_tbl LIMIT 1)") + ) + inner = _inner_select_sql(out) + assert "orders" not in inner, out + + +# --------------------------------------------------------------------------- # +# Correlation contract: bare = local; an explicit outer ref is a scope leak +# --------------------------------------------------------------------------- # +class TestCorrelationContract: + + def test_expander_does_not_rebind_explicit_outer_ref(self) -> None: + # The expander is scope-aware: it leaves an explicit outer ref untouched + # (it does NOT invent a rebind). Whether that ref is legal is decided + # downstream by assert_scope_closed, not here — see + # TestExecution.test_correlated_subquery_is_rejected. + out = _sql_of( + _scope().enter_predicate( + "EXISTS (SELECT 1 FROM line_items " + "WHERE line_items.order_id = orders.id)" + ) + ) + inner = _inner_select_sql(out) + assert "orders.id" in inner, out + assert "line_items.order_id" in inner, out + + def test_bare_reference_in_subquery_stays_bare(self) -> None: + # Documented limitation: a BARE name inside a subquery binds to the + # subquery's own FROM (local), never the outer root. Pin it so the + # contract is asserted, not accidental. + out = _sql_of( + _scope().enter_predicate( + "EXISTS (SELECT 1 FROM line_items WHERE line_items.amount = amount)" + ) + ) + inner = _inner_select_sql(out) + assert "orders.amount" not in inner, ( + f"bare inner reference was rebound to the outer root: {out}" + ) + + +# --------------------------------------------------------------------------- # +# Root-scope refs are unaffected (guard against over-gating) +# --------------------------------------------------------------------------- # +class TestRootScopeUnaffected: + + def test_top_level_bare_column_still_qualifies(self) -> None: + out = _sql_of(_scope().enter_predicate("amount > 1")) + assert "orders.amount > 1" in out, out + + def test_top_level_derived_column_still_inlines(self) -> None: + out = _sql_of(_scope().enter_predicate("doubled > 1")) + # ``doubled`` derives to ``amount * 2`` and inlines, qualified to root. + assert "orders.amount * 2" in out, out + + +# --------------------------------------------------------------------------- # +# Join-path discovery stays root-only (a subquery ref registers no join) +# --------------------------------------------------------------------------- # +class TestJoinDiscoveryRootOnly: + + def test_subquery_join_target_ref_registers_no_join(self) -> None: + scope = _scope() + scope.enter_predicate( + "id IN (SELECT x FROM other WHERE other.k = customers.balance)" + ) + assert scope.join_paths.as_list() == [], ( + "a join-target-looking ref INSIDE a subquery must not register a " + f"root join path: {scope.join_paths.as_list()}" + ) + + def test_collect_paths_ignores_subquery_scope(self) -> None: + parsed = sqlglot.parse_one( + "id IN (SELECT x FROM other WHERE other.k = customers.balance)", + dialect="postgres", + ) + host = _orders() + bundle = ResolvedSourceBundle( + source_model=host, referenced_models=[host, _customers(), _regions()], + ) + paths = collect_root_scope_joined_paths( + parsed=parsed, source_model=host, + source_relation=host.name, bundle=bundle, + ) + assert paths == [], paths + + def test_collect_paths_ignores_two_hop_alias_in_subquery(self) -> None: + # A resolvable TWO-hop alias inside a subquery must register NEITHER + # prefix — path-prefix collection is root-only. + parsed = sqlglot.parse_one( + "id IN (SELECT x FROM other " + "WHERE other.k = customers__regions.population)", + dialect="postgres", + ) + host = _orders() + bundle = ResolvedSourceBundle( + source_model=host, referenced_models=[host, _customers(), _regions()], + ) + paths = collect_root_scope_joined_paths( + parsed=parsed, source_model=host, + source_relation=host.name, bundle=bundle, + ) + assert paths == [], paths + + +# --------------------------------------------------------------------------- # +# Execution: real SQLite differential +# --------------------------------------------------------------------------- # +def _seed_sqlite(db_path: str) -> None: + con = sqlite3.connect(db_path) + try: + con.executescript( + """ + CREATE TABLE orders ( + id INTEGER PRIMARY KEY, + customer_id INTEGER, + amount REAL, + status TEXT + ); + CREATE TABLE other_tbl (amount REAL); + CREATE TABLE line_items ( + id INTEGER PRIMARY KEY, + order_id INTEGER, + note TEXT + ); + """ + ) + con.executemany( + "INSERT INTO orders VALUES (?,?,?,?)", + [(1, 100, 10.0, "a"), (2, 100, 20.0, "b"), + (3, 101, 30.0, "c"), (4, 101, 40.0, "d")], + ) + # Only 10.0 and 30.0 are present -> filter must keep exactly orders 1 & 3. + con.executemany("INSERT INTO other_tbl VALUES (?)", [(10.0,), (30.0,)]) + # line_items reference orders 1 & 3 only; line_items.id is deliberately + # disjoint from orders.id so a wrong (local) bind of an outer ref would + # change the answer. + con.executemany( + "INSERT INTO line_items VALUES (?,?,?)", + [(501, 1, "x"), (502, 3, "y")], + ) + con.commit() + finally: + con.close() + + +async def _engine_with_filter(base_dir: str, db_path: str, *, model_filter: str) -> SlayerQueryEngine: + storage = YAMLStorage(base_dir=base_dir) + await storage.save_datasource( + DatasourceConfig(name="test", type="sqlite", database=db_path), + ) + orders = SlayerModel( + name="orders", sql_table="orders", data_source="test", + columns=[ + Column(name="id", type=DataType.INT, primary_key=True), + Column(name="customer_id", type=DataType.INT), + Column(name="amount", type=DataType.DOUBLE), + Column(name="status", type=DataType.TEXT), + ], + filters=[model_filter], + ) + await storage.save_model(orders) + return SlayerQueryEngine(storage=storage) + + +def _ids(resp) -> set: + return {int(r["orders.id"]) for r in resp.data} + + +class TestExecution: + + async def test_non_correlated_subquery_filters_correctly(self) -> None: + """Differential: before the fix the buggy ``SELECT orders.amount FROM + other_tbl`` correlates to the outer row (always-true) and returns ALL + orders; after the fix the inner ``amount`` binds to ``other_tbl`` and + only orders 1 & 3 match.""" + with tempfile.TemporaryDirectory() as d: + db = os.path.join(d, "corpus.db") + _seed_sqlite(db) + engine = await _engine_with_filter( + os.path.join(d, "store"), db, + model_filter="amount IN (SELECT amount FROM other_tbl)", + ) + resp = await engine.execute( + SlayerQuery(source_model="orders", dimensions=[ColumnRef(name="id")]), + ) + assert _ids(resp) == {1, 3}, resp.data + + async def test_correlated_subquery_is_rejected_under_scope_validation( + self, monkeypatch, + ) -> None: + """Contract: a correlated Mode-A subquery references the outer relation, + which the scope-closure invariant flags as a leak. That invariant is + SLayer's ``assert_scope_closed`` pass, enabled by ``SLAYER_VALIDATE_SCOPES`` + (on in CI; the conftest autouse fixture sets it — here it is set + explicitly so the test does not silently depend on the ambient default, + and is off in production for performance). The DEV-1752 fix leaves the + outer ref untouched; the guard, not the expander, is what rejects it — + so correlation is an unsupported Mode-A shape, not a supported feature.""" + monkeypatch.setenv("SLAYER_VALIDATE_SCOPES", "1") + with tempfile.TemporaryDirectory() as d: + db = os.path.join(d, "corpus.db") + _seed_sqlite(db) + engine = await _engine_with_filter( + os.path.join(d, "store"), db, + model_filter=( + "EXISTS (SELECT 1 FROM line_items " + "WHERE line_items.order_id = orders.id)" + ), + ) + query = SlayerQuery( + source_model="orders", dimensions=[ColumnRef(name="id")], + ) + with pytest.raises(ScopeLeakError): + await engine.execute(query) diff --git a/tests/test_dev1777_declared_measure_partition.py b/tests/test_dev1777_declared_measure_partition.py new file mode 100644 index 00000000..ed9c6c82 --- /dev/null +++ b/tests/test_dev1777_declared_measure_partition.py @@ -0,0 +1,61 @@ +"""DEV-1777 sub-item 3(b): the ``declared_measures`` prefix partition is +structural. ``partition_declared_measures`` single-sources the dim / time-dim / +aggregate slice arithmetic that ``stage_planner`` and ``cross_model_planner`` +used to inline as ``[:n_dims]`` / ``[n_dims:n_dims+n_tds]`` / ``[n_dims+n_tds:]``; +``PreboundQuery.grain_declared_measures`` exposes the grain prefix over it. +""" + +from __future__ import annotations + +import pytest + +from slayer.engine.prebound import PreboundQuery, partition_declared_measures + +# Sentinel elements — partition_declared_measures is pure list slicing, so the +# element type is irrelevant; strings make the boundaries readable. +_DMS = ["d0", "d1", "t0", "a0", "a1"] + + +@pytest.mark.parametrize( + "n_dims,n_tds,dims,tds,aggs", + [ + (2, 1, ["d0", "d1"], ["t0"], ["a0", "a1"]), + (0, 0, [], [], ["d0", "d1", "t0", "a0", "a1"]), + (5, 0, ["d0", "d1", "t0", "a0", "a1"], [], []), + (0, 2, [], ["d0", "d1"], ["t0", "a0", "a1"]), + (3, 2, ["d0", "d1", "t0"], ["a0", "a1"], []), + ], +) +def test_partition_matches_manual_slicing(n_dims, n_tds, dims, tds, aggs) -> None: + got_dims, got_tds, got_aggs = partition_declared_measures( + declared_measures=_DMS, n_dims=n_dims, n_time_dimensions=n_tds, + ) + assert (got_dims, got_tds, got_aggs) == (dims, tds, aggs) + # Byte-for-byte the slices the planners used to inline. + grain = n_dims + n_tds + assert got_dims == _DMS[:n_dims] + assert got_tds == _DMS[n_dims:grain] + assert got_aggs == _DMS[grain:] + + +def test_partition_empty_list() -> None: + assert partition_declared_measures( + declared_measures=[], n_dims=0, n_time_dimensions=0, + ) == ([], [], []) + + +def test_grain_accessor_is_dims_plus_time_dims() -> None: + # model_construct skips validation so the accessor's slicing can be pinned + # without hand-building heavy DeclaredMeasure / BoundExpr fixtures. + pq = PreboundQuery.model_construct( + declared_measures=_DMS, n_dims=2, n_time_dimensions=1, + ) + assert pq.grain_declared_measures == ["d0", "d1", "t0"] + assert pq.grain_declared_measures == _DMS[: 2 + 1] + + +def test_grain_accessor_empty_grain() -> None: + pq = PreboundQuery.model_construct( + declared_measures=_DMS, n_dims=0, n_time_dimensions=0, + ) + assert pq.grain_declared_measures == [] diff --git a/tests/test_dev1777_emit_step_cte.py b/tests/test_dev1777_emit_step_cte.py new file mode 100644 index 00000000..b1c29dc3 --- /dev/null +++ b/tests/test_dev1777_emit_step_cte.py @@ -0,0 +1,183 @@ +"""DEV-1777 sub-item 1: ``_emit_step_cte`` is the one shell shared by the four +transform-chain step-CTE sites (window / unmaterialised-POST, in the host and +cross-model chains). These pin the shell directly and deterministically: + +* the multi-slot-in-one-batch body (the caller controls slot order, so unlike + an end-to-end golden this is not exposed to the base-CTE column-order + non-determinism tracked in DEV-1795); +* the render-before-mutate ordering invariant a later same-step slot relies on; +* block D (F2 unmaterialised-POST), which no real query reaches — it errors + earlier in ``_render_outer_composite`` — so it has no golden pin; the shell it + would use is pinned here, and block C (F1 unmaterialised-POST) covers the same + shell end-to-end via ``chain/local_multi_step``. +""" + +from __future__ import annotations + +from decimal import Decimal +from types import SimpleNamespace + +from sqlglot import exp + +from slayer.core.enums import DataType +from slayer.core.keys import ( + ArithmeticKey, + ColumnKey, + LiteralKey, + ScalarCallKey, + TransformKey, +) +from slayer.sql.generator import SQLGenerator +from slayer.sql.render.cte_assembly import CteEntry + + +def _gen() -> SQLGenerator: + return SQLGenerator(dialect="postgres") + + +def _base_ctes() -> list[CteEntry]: + return [CteEntry(name="base", query=exp.Select().select(exp.column("x")).from_("t"))] + + +def _slot(name: str, *, type_=None) -> SimpleNamespace: + return SimpleNamespace(public_aliases=[name], declared_name=name, type=type_) + + +def test_single_slot_emits_step_cte_and_advances_chain() -> None: + gen = _gen() + ctes = _base_ctes() + aliases = {"d0": ["orders.d0"]} + avail = {"d0": "orders.d0"} + new_tail, new_step_num = gen._emit_step_cte( + ctes=ctes, + chain_tail="base", + step_num=0, + cte_allocator=gen._new_allocator(), + aliases_by_slot_id=aliases, + available_alias_by_slot_id=avail, + source_relation="orders", + slot_entries=[("sid1", _slot("s1"))], + render=lambda s: exp.column("v", quoted=True), + ) + assert (new_tail, new_step_num) == ("step1", 1) + assert ctes[-1].name == "step1" + assert ctes[-1].depends_on == ["base"] + sql = ctes[-1].query.sql(dialect="postgres") + assert '"orders.d0"' in sql # carried alias, in plan order + assert 'AS "orders.s1"' in sql # rendered column, source-qualified + assert "FROM base" in sql + assert aliases["sid1"] == ["orders.s1"] + assert avail["sid1"] == "orders.s1" + + +def test_multi_slot_one_batch_preserves_caller_order() -> None: + gen = _gen() + ctes = _base_ctes() + new_tail, _ = gen._emit_step_cte( + ctes=ctes, + chain_tail="base", + step_num=0, + cte_allocator=gen._new_allocator(), + aliases_by_slot_id={}, + available_alias_by_slot_id={}, + source_relation="orders", + slot_entries=[("A", _slot("a")), ("B", _slot("b"))], + render=lambda s: exp.column("v", quoted=True), + ) + assert new_tail == "step1" + sql = ctes[-1].query.sql(dialect="postgres") + assert '"orders.a"' in sql + assert '"orders.b"' in sql + # Deterministic: the caller's slot order is the emitted column order. + assert sql.index('"orders.a"') < sql.index('"orders.b"') + + +def test_render_runs_before_alias_map_mutation_per_slot() -> None: + gen = _gen() + avail: dict = {} + observed: list[dict] = [] + + def render(_slot_obj) -> exp.Expression: + observed.append(dict(avail)) # snapshot at render time + return exp.column("v", quoted=True) + + gen._emit_step_cte( + ctes=_base_ctes(), + chain_tail="base", + step_num=0, + cte_allocator=gen._new_allocator(), + aliases_by_slot_id={}, + available_alias_by_slot_id=avail, + source_relation="orders", + slot_entries=[("A", _slot("a")), ("B", _slot("b"))], + render=render, + ) + # A's own alias is not yet in the map when A renders. + assert "A" not in observed[0] + # B's render sees A (materialised after A's render) but not itself. + assert "A" in observed[1] + assert "B" not in observed[1] + + +def test_typed_slot_is_cast_wrapped() -> None: + # _wrap_cast_for_type skips a bare exp.Column, so render a non-column + # expression to exercise the type-enforcing CAST the helper applies. + gen = _gen() + ctes = _base_ctes() + gen._emit_step_cte( + ctes=ctes, + chain_tail="base", + step_num=0, + cte_allocator=gen._new_allocator(), + aliases_by_slot_id={}, + available_alias_by_slot_id={}, + source_relation="orders", + slot_entries=[("sid", _slot("s", type_=DataType.DOUBLE))], + render=lambda s: exp.func("ABS", exp.column("v", quoted=True)), + ) + assert "CAST" in ctes[-1].query.sql(dialect="postgres").upper() + + +def test_unmaterialised_post_slots_detects_arith_and_scalar_call() -> None: + # The detection loop feeding block C (F1) and block D (F2): Arithmetic and + # ScalarCall POST slots are detected; TransformKey (materialised by a layer), + # an already-materialised slot, and a plain ColumnKey are skipped. + arith = SimpleNamespace( + id="arith", key=ArithmeticKey(op="-", operands=(LiteralKey(value=Decimal(1)),)), + ) + scalar = SimpleNamespace( + id="scalar", key=ScalarCallKey(name="abs", args=(ColumnKey(leaf="x"),)), + ) + transform = SimpleNamespace( + id="xf", key=TransformKey(op="cumsum", input=ColumnKey(leaf="x")), + ) + materialised = SimpleNamespace( + id="done", key=ArithmeticKey(op="+", operands=(LiteralKey(value=Decimal(2)),)), + ) + column = SimpleNamespace(id="col", key=ColumnKey(leaf="y")) + pq = SimpleNamespace( + combined_expression_slots=[arith, scalar, transform, materialised, column], + ) + out = SQLGenerator._unmaterialised_post_slots(pq, {"done": ["orders.done"]}) + assert [s.id for s in out] == ["arith", "scalar"] + + +def test_step_num_and_names_increment_across_calls() -> None: + gen = _gen() + ctes = _base_ctes() + alloc = gen._new_allocator() + tail, n = gen._emit_step_cte( + ctes=ctes, chain_tail="base", step_num=0, cte_allocator=alloc, + aliases_by_slot_id={}, available_alias_by_slot_id={}, + source_relation="orders", slot_entries=[("A", _slot("a"))], + render=lambda s: exp.column("v", quoted=True), + ) + tail2, n2 = gen._emit_step_cte( + ctes=ctes, chain_tail=tail, step_num=n, cte_allocator=alloc, + aliases_by_slot_id={"A": ["orders.a"]}, available_alias_by_slot_id={"A": "orders.a"}, + source_relation="orders", slot_entries=[("B", _slot("b"))], + render=lambda s: exp.column("v", quoted=True), + ) + assert (tail, n) == ("step1", 1) + assert (tail2, n2) == ("step2", 2) + assert ctes[-1].depends_on == ["step1"] # chained onto the prior step diff --git a/tests/test_reroot_aggregate_key.py b/tests/test_reroot_aggregate_key.py index cd3c73cd..400b5c54 100644 --- a/tests/test_reroot_aggregate_key.py +++ b/tests/test_reroot_aggregate_key.py @@ -39,7 +39,6 @@ import pytest -from slayer.core.enums import DataType from slayer.core.keys import ( AggregateKey, ColumnKey, @@ -48,8 +47,6 @@ StarKey, reroot_aggregate_key, ) -from slayer.core.models import Column, ModelJoin, SlayerModel -from slayer.sql.generator import SQLGenerator # =========================================================================== @@ -457,91 +454,3 @@ def test_reroot_does_not_mutate_input_key() -> None: # host-rooted (no accidental in-place surgery / aliasing). assert key.source == ColumnKey(path=("customers",), leaf="amount") assert key.args == (ColumnKey(path=("customers",), leaf="signup_at"),) - - - - -# =========================================================================== -# Section G — reworded residual-hop guard (DEV-1526 pointer) -# =========================================================================== -# PINS A DEAD BRANCH. ``_resolve_explicit_time_col`` is still CALLED — every -# aggregate reaches it through ``_build_agg_render_spec_from_planned`` — but no -# first/last does since DEV-1748, so it returns ``None`` on every production -# call and the residual-hop guard below can no longer fire outside a direct -# unit call like this one. The ranked CTE resolves its ranking key through its -# OWN scope, which pulls the residual join this guard exists to refuse; the -# end-to-end proof is the un-xfailed -# ``test_a_joined_derived_time_arg_ranks_by_the_joined_expression`` in -# tests/test_dev1748_first_last_matrix.py. Kept until PR 6 removes the branch, -# per P-J, so the removal is reviewed as a removal. -# -# After the unified reroot, a path-bearing ``ColumnSqlKey`` reaching -# ``_resolve_explicit_time_col`` is the deeper-hop RESIDUAL case (source -# shallower than the derived time arg): the isolated CTE does not yet pull -# the residual join (Stage 4 / DEV-1526). The guard stays a loud -# ``NotImplementedError`` but its message must point at DEV-1526, not the -# now-closed DEV-1476. - - -def _guard_source_model() -> SlayerModel: - return SlayerModel( - name="customers", - sql_table="customers", - data_source="prod", - columns=[ - Column(name="id", type=DataType.INT, primary_key=True), - Column(name="amount", type=DataType.DOUBLE), - ], - joins=[ - ModelJoin(target_model="regions", join_pairs=[["region_id", "id"]]), - ], - ) - - -def test_residual_columnsqlkey_time_arg_raises_dev1526() -> None: - gen = SQLGenerator(dialect="postgres") - key = AggregateKey( - source=ColumnKey(path=(), leaf="amount"), - agg="last", - # Residual path survives reroot when the derived time col is a hop - # PAST the target. - args=( - ColumnSqlKey( - path=("regions",), model="regions", column_name="opened_day", - ), - ), - ) - # Hoisted out of the ``pytest.raises`` block so the ONLY call that can - # raise inside it is the one under test (Sonar S5778). - source_model = _guard_source_model() - with pytest.raises(NotImplementedError) as excinfo: - gen._resolve_explicit_time_col( - key=key, - source_model=source_model, - source_relation="customers", - bundle=None, - ) - assert "DEV-1526" in str(excinfo.value) - assert "DEV-1476" not in str(excinfo.value) - - -def test_non_first_last_agg_returns_none_before_residual_guard() -> None: - # The ``agg not in (first, last)`` short-circuit must precede the - # residual-path raise: a non-ranking aggregate with a path-bearing - # ColumnSqlKey arg returns None, never reaching the guard. - gen = SQLGenerator(dialect="postgres") - key = AggregateKey( - source=ColumnKey(path=(), leaf="amount"), - agg="sum", - args=( - ColumnSqlKey( - path=("regions",), model="regions", column_name="opened_day", - ), - ), - ) - assert gen._resolve_explicit_time_col( - key=key, - source_model=_guard_source_model(), - source_relation="customers", - bundle=None, - ) is None diff --git a/tests/test_sql_generator.py b/tests/test_sql_generator.py index d70cbdac..b325d003 100644 --- a/tests/test_sql_generator.py +++ b/tests/test_sql_generator.py @@ -7317,7 +7317,8 @@ async def test_derived_time_arg_pulls_in_referenced_join( ) -> None: """DEV-1501 (Codex round 8): a DERIVED first/last time arg whose ``Column.sql`` references a joined column must pull that join into - the base FROM. ``_resolve_explicit_time_col`` expands + the base FROM. Join discovery (``_resolve_agg_inputs_via_scope`` via + ``_explicit_time_arg_of``) registers it, and the ranked plan expands ``net_signed_at.sql = "customers.signed_up_at"`` so the ranked subquery's ``ORDER BY`` emits ``customers.signed_up_at``; without a corresponding ``LEFT JOIN customers`` the SQL is broken.