DEV-1777: mechanical consolidations — step-CTE extraction, throwaway ScopeFrames, positional couplings

DECISIONS.md

@@ -122,3 +122,4 @@ implementation detail. Include issue refs when known.
 - 2026-08-16 — Aggregated slot-type and display-format inference share one classifier (DEV-1788, follow-up to DEV-1784's Option A). `aggregated_type` (slot `DataType`) and `_infer_aggregated_format` (response `NumberFormat`) had disagreed on the stat/parametric family: type said `DOUBLE` while format fell through to inherit the source column's format, so `revenue:stddev_samp` was typed `DOUBLE` yet displayed as currency. Both now read a single `classify_aggregation` (`core/enums.py`) returning one of four `AggregationValueClass` buckets, and each function maps the bucket to its own output — no per-name branching survives, so the two axes cannot drift. The four builtin frozensets (`INTEGER_AGGREGATIONS`, `PRESERVING_AGGREGATIONS`, `FLOAT_SOURCE_UNIT_AGGREGATIONS`, `FLOAT_PLAIN_AGGREGATIONS`) partition `BUILTIN_AGGREGATIONS`, pinned by a completeness test; custom/model-defined aggregations hit the `PRESERVING` fallback (inherit type & format), unchanged. **Semantics chosen (Option B, unit-correct):** `avg`/`median`/`weighted_avg`/`percentile`/`stddev*` are `DOUBLE` but keep the source's UNITS, so display format inherits the source (falling back to `FLOAT` when the source has none — keeping type `DOUBLE` and format `FLOAT` coherent for unformatted measures, and confining the change to formatted ones); `corr`/`var*`/`covar*` are dimensionless/squared/product units, so they display as plain `FLOAT` regardless of source. `aggregated_type` is behaviourally unchanged (only restructured). **Net user-visible change, all in `_infer_aggregated_format`:** avg-family of a FORMATTED measure now inherits that format (was `FLOAT`); `corr`/`var*`/`covar*` now `FLOAT` (was inherit); `stddev*`/`percentile` unchanged (already inherited). Drift guard extended to the full four-bucket table and routed through the public callers (`measure_key_type` / `measure_key_format_description`), plus response-metadata assertions for the stat/parametric family.
 - 2026-08-17 — Last four parser-only scalars admitted (DEV-1753, follow-up to DEV-1744; closes the tripwire `TestParserAndBinderScalarSetsAgree`). `greatest`, `least`, `trunc`, `mod` were in the parser's `SCALAR_PASSTHROUGH` but not the binder's `SCALAR_FUNCTIONS`, so each parsed then raised `UnknownFunctionError` at bind while the parser's own error text advertised it; all four are now in `SCALAR_FUNCTIONS` with arity entries, so `SCALAR_PASSTHROUGH - SCALAR_FUNCTIONS == set()`. Rulings: (1) **`greatest`/`least` NULL divergence RATIFIED, not normalised** — same class as the `concat → ||` ruling (2026-08-05). They pass through to each backend's native form, and those forms disagree on NULL: Postgres/DuckDB/SQL Server/Snowflake (sqlglot emits `GREATEST_IGNORE_NULLS` for Snowflake) IGNORE NULLs, while SQLite (scalar `MAX(a,b)`/`MIN(a,b)`), MySQL and BigQuery PROPAGATE NULL. Normalising is impractical (variadic ⇒ NULL-ignoring is not portably expressible) and per-dialect refusal breaks Mode B's dialect-agnostic promise (SQLite is the default store), so the divergence is documented and users wrap in `ifnull`/`coalesce` if they need one behaviour. Witnessed live: SQLite propagates (`tests/test_dev1753_last_four_scalars.py`), DuckDB ignores (`tests/integration/test_integration_duckdb.py`). Only those two are runtime-witnessed; the other rows are sqlglot-emission-contract assumptions (T-SQL `GREATEST` needs SQL Server 2022+). Arity `(2, None)`: SQLite's one-arg `MAX`/`MIN` parses as the AGGREGATE, and MySQL requires ≥2. (2) **`trunc` 1-arg only** — a 2-arg `trunc(x, digits)` SILENTLY drops the digits on SQLite (emits `TRUNC(x)`), a wrong answer rather than an error; ClickHouse emits the lowercase `trunc` on a case-sensitive backend (live check in `tests/integration/test_integration_clickhouse.py`). (3) **`mod` renders through the `%` composer** (`render_arithmetic(op="%")`), not a raw `exp.Mod`: `exp.func("MOD", …)` raises at build time, and a directly-built `exp.Mod` mis-groups a complex operand (`mod(a+b, c)` → `a + b % c`, which re-parses as `a + (b%c)`); the `%` composer runs the operand-precedence pass, and the pre-existing unconditional-Mod parenthesisation (DEV-1744) covers a `mod(...)` nested inside arithmetic. (4) **2-arg strip form of `ltrim`/`rtrim` DEFERRED to DEV-1793** — its second argument is a character SET on most backends but an exact SUBSTRING on MySQL (`TRIM(LEADING remstr FROM str)`), a silent string-value cross-dialect divergence needing its own ruling; the trims stay `(1,1)`.
 - 2026-08-17 — DEV-1756 identifier-length fitting ported to the DEV-1450 pipeline for the PROJECTION-ALIAS and CTE-NAME surfaces. Main's design (aliases stay canonical inside SLayer, fitted only at emission, restored on result keys) is kept: the pure `fit_identifier`/`substitute_quoted` module and the `SqlDialect` methods (`fit_alias`/`emit_alias`/`alias_rewrite_map`/`decode_alias_map`/`_rekey_row`, and the `aliases`-taking `rewrite_emitted_sql`/`decode_result_keys`) merged intact, as did the per-dialect `max_identifier_bytes` budgets. **Projection aliases:** the wiring differs from main because the DEV-1450 pipeline is structural, not enrichment-based — main threaded `all_projection_aliases`/`public_projection_aliases` (enriched.py, deleted); here the write and read sides share ONE plan-derived source, a new `response_meta.projection_result_keys(root_planned)` (mirrors `_slot_result_keys`). The WRITE side passes it into `generate_planned_stages(..., projection_aliases=...)` → `rewrite_emitted_sql(sql, aliases=...)` (NOT parsed off the SQL — `sqlglot.parse_one` chokes on a pre-mangle BigQuery backticked dotted alias); the READ side threads the same set into every `decode_result_keys` call site (`build_response_metadata`, `_run_data_query` via `_Prepared.expected_columns`, the `get_column_types` probe). The two agree by the generator's naming contract (`_slot_result_keys` ≡ emitted projection aliases). **CTE names:** rather than adopt main's `SQLGenerator._cte_name`/`_cte_names` (absent here), the branch's own `naming.cte_name_from_alias` now length-fits `prefix+sanitized` via `fit_identifier` BEFORE `AliasAllocator.allocate_cte`, and raises `IdentifierCollisionError` if the allocator's `_2` suffix pushes a fitted name back over the limit (only a forced digest collision reaches this). The dialect + limit are threaded from the three cross-model/window/ranked call sites and the two time-shift `shifted_`/`sjoin_` sites; `limit=None` (unbounded dialects) is byte-identical to before. `TestCteNames`/`TestSweep` were rewritten to this API and un-skipped. **NOT ported (one follow-up):** virtual-model short fitting (`_fit_short`) — there is no `engine._query_as_model` on this branch; query-backed shorts flow through `source_bundle` expansion, so `TestVirtualModelShorts` stays `pytest.mark.skip`-marked. The projection/CTE scenarios drive the real engine (`_engine_generate` / `SlayerQueryEngine`) and pass, as does the pure-module `tests/dialects/test_identifier_fit.py`. Two "no-churn" golden tests were rewritten from main's byte-exact SQL (which pinned `SUM(...)`, but this branch emits `CAST(SUM(...) AS FLOAT64)`) to assert the DEV-1756 intent (under-limit ⇒ no `_<hash>_` marker). Short-alias quoting in `build_flat_rename_wrapper` was left as the branch had it (lenient `AS "?rev"?` assertions kept), not switched to main's quote-iff-mixed-case-or-reserved policy.
+- 2026-08-17 — Mode-A expansion gates qualification on root scope (DEV-1752). In `_process_column_node_sync` (`slayer/engine/column_expansion.py`) the `root_scope_ids` gate now runs BEFORE the qualification branch, not after — so a column outside the root scope (inside a subquery, CTE, set-op branch, or `Values`) is left completely untouched, neither qualified against the outer root nor inlined. Previously qualification fired unconditionally, rebinding a subquery's own columns to the outer model: `amount IN (SELECT amount FROM other_tbl)` became `orders.amount IN (SELECT orders.amount FROM other_tbl)`, silently changing the predicate (in SQLite the inner `orders.amount` correlates to the outer row, making the filter always-true). The contract is now explicit: a Mode-A subquery must be **self-contained** — its columns bind in its own scope. Correlation to the outer model is not supported: an outer reference in a subquery is a scope leak flagged by the `assert_scope_closed` invariant (`slayer/sql/scope_check.py`), which runs under `SLAYER_VALIDATE_SCOPES` (on in CI, off in production for performance; the only legal correlated ref is the RLS session-policy EXISTS). Runtime enforcement of the self-contained-subquery contract is out of scope for this bugfix — correlation is unsupported/undefined rather than hard-rejected at runtime. Chose this one-spot scope gate over a correlation heuristic: correlation is undecidable from a schema-less Mode-A fragment (a bare inner name that collides with a root column is byte-identical to a correlated ref), sqlglot's `external_columns`/`is_correlated_subquery` over-report on such fragments, and the scope-closure guard already forbids the only case a heuristic would target. Reuses the single existing `_root_scope_column_ids` notion of scope rather than adding a second.

docs/concepts/references.md

@@ -18,6 +18,7 @@ SLayer has two distinct expression layers and the rules for what each one accept
 * Single-dot `t.col` is a literal `<table>.<column>` SQL reference (sqlglot's normal behavior).
 * User-supplied multi-dot input (`a.b.c`) is auto-rewritten to `a__b.c` at validation time with a warning.
 * Other derived columns of the same model (or of a joined model via `__`) are recursively expanded so chains like `A.ratio = "A.bar / B.foo_normalized"` (where `B.foo_normalized` is itself derived) work.
+* A subquery in a Mode-A surface (`col IN (SELECT … FROM other)`, a scalar `= (SELECT … LIMIT 1)`) must be **self-contained**: its own columns bind against the subquery's own `FROM`, never against this model — reference resolution does not reach in to re-qualify them. Correlating back to the outer model is **not** supported (an outer reference inside the subquery is a scope leak, flagged by the scope-closure check SLayer runs over generated SQL under `SLAYER_VALIDATE_SCOPES`).
 * `ModelMeasure` names are not visible from SQL mode — saved measures are DSL-only.
 * `{variable}` placeholders are substituted into these Mode-A surfaces from the merged variable set (raise-on-missing once any variable is in play; a fully variable-free execution leaves braces as literals; Mode-A- and dialect-aware string escaping, so quoted values round-trip on backslash-escaping backends like MySQL/ClickHouse — DEV-1727). See [Variables in model SQL](models.md#variables-in-model-sql).
 

slayer/engine/column_expansion.py

@@ -256,6 +256,11 @@ def _process_column_node_sync(
     """
     if col.args.get("db") or col.args.get("catalog"):
         return None
+    # DEV-1752: gate BEFORE qualifying. A column outside the root scope belongs
+    # to a subquery / CTE / set-op branch and binds in its own scope, so it is
+    # left untouched — neither qualified against the outer root nor inlined.
+    if id(col) not in root_scope_ids:
+        return None
     table_id = col.args.get("table")
     col_name = col.name
     table_alias = table_id.name if table_id is not None else alias_path
@@ -272,8 +277,6 @@ def _process_column_node_sync(
     if target_col is None or _is_trivial_base(column=target_col):
         col.set("table", exp.to_identifier(canonical_alias))
         return None
-    if id(col) not in root_scope_ids:
-        return None
     next_is_root = is_root and (target_model is model)
     key = (target_model.name, col_name)
     if key in visited:

slayer/engine/cross_model_planner.py

@@ -927,7 +927,7 @@ def _plan_filtered_local(
                 model=host_model,
                 public_alias=public_alias,
             ),
-            grain_measures=list(_grain_declared_measures(host_prebound)),
+            grain_measures=list(host_prebound.grain_declared_measures),
             inherited_filters=[
                 routing_by_id[fid].bound
                 for fid in host_rooted_routes.applied
@@ -1016,13 +1016,6 @@ def _plan_filtered_local(
 # module does not import ``stage_planner`` (no cycle).
 
 
-def _grain_declared_measures(prebound: PreboundQuery) -> List[DeclaredMeasure]:
-    """The host's dimension + time-dimension declarations — the grain prefix of
-    ``declared_measures``, which the nested plan groups by unchanged."""
-    n = prebound.n_dims + prebound.n_time_dimensions
-    return list(prebound.declared_measures[:n])
-
-
 def _aggregate_declared_measure(
     *,
     key: AggregateKey,
@@ -1314,14 +1307,15 @@ def _is_forward(path: Tuple[str, ...]) -> bool:
         # On the host->target path (handled by the forward-path CTE already).
         return bool(path) and path == target_path[: len(path)]
 
-    n_dims = host_prebound.n_dims
-    n_tds = host_prebound.n_time_dimensions
     grain_declared: List[DeclaredMeasure] = []
     grain_host_sids: List[str] = []
     grain_rerooted_keys: List[ValueKey] = []
     needs_reroot = False
 
-    for i, dm in enumerate(host_prebound.declared_measures[: n_dims + n_tds]):
+    # The public_projection[i] positional pairing (and its silent None
+    # fallback) is deliberately left as-is; the cardinality/ambiguity concern
+    # it encodes is out of scope here (DEV-1688).
+    for i, dm in enumerate(host_prebound.grain_declared_measures):
         host_sid = public_projection[i] if i < len(public_projection) else None
         host_key = dm.bound.value_key
         inner = (

slayer/engine/prebound.py

@@ -56,6 +56,7 @@
     "PreboundQuery",
     "StrictQueryCarrier",
     "aggregated_type",
+    "partition_declared_measures",
     "dimension_key_metadata",
     "measure_key_format_description",
     "measure_key_type",
@@ -68,6 +69,23 @@
 # ---------------------------------------------------------------------------
 
 
+def partition_declared_measures(
+    *,
+    declared_measures: List[DeclaredMeasure],
+    n_dims: int,
+    n_time_dimensions: int,
+) -> Tuple[List[DeclaredMeasure], List[DeclaredMeasure], List[DeclaredMeasure]]:
+    """Split ``declared_measures`` into its (dims, time_dims, aggregates) prefix
+    partition — the slice arithmetic the planners used to inline. ``n_dims`` /
+    ``n_time_dimensions`` are the grain prefix lengths (see ``PreboundQuery``)."""
+    grain = n_dims + n_time_dimensions
+    return (
+        declared_measures[:n_dims],
+        declared_measures[n_dims:grain],
+        declared_measures[grain:],
+    )
+
+
 class PreboundQuery(BaseModel):
     """The typed product of ``plan_query``'s bind block.
 
@@ -136,6 +154,16 @@ def _filter_texts_are_parallel(self) -> "PreboundQuery":
             )
         return self
 
+    @property
+    def grain_declared_measures(self) -> List[DeclaredMeasure]:
+        """The dimension + time-dimension grain prefix of ``declared_measures``."""
+        dims, time_dims, _ = partition_declared_measures(
+            declared_measures=self.declared_measures,
+            n_dims=self.n_dims,
+            n_time_dimensions=self.n_time_dimensions,
+        )
+        return dims + time_dims
+
 
 class StrictQueryCarrier(BaseModel):
     """The post-bind ``query.*`` surface the §5.4 seam approves.

slayer/engine/stage_planner.py

@@ -117,6 +117,7 @@
     StrictQueryCarrier,
     measure_key_format_description,
     measure_key_type,
+    partition_declared_measures,
 )
 from slayer.engine.source_bundle import (
     ResolvedSourceBundle,
@@ -739,7 +740,10 @@ def bind_query_inputs(  # NOSONAR(S3776) — one cohesive bind pass. The stages
     n_dims = len(query.dimensions or [])
     n_tds = len(query.time_dimensions or [])
     filter_alias_map: Dict[str, ValueKey] = {}
-    for dm in declared_measures[n_dims + n_tds:]:
+    _, _, _agg_dms = partition_declared_measures(
+        declared_measures=declared_measures, n_dims=n_dims, n_time_dimensions=n_tds,
+    )
+    for dm in _agg_dms:
         for alias in (dm.public_name, dm.declared_name, dm.canonical_alias):
             if alias is not None:
                 filter_alias_map.setdefault(alias, dm.bound.value_key)
@@ -1021,8 +1025,9 @@ def bind_query_inputs(  # NOSONAR(S3776) — one cohesive bind pass. The stages
     # not the raw timestamp — which would silently widen the grain). Runs BEFORE
     # interning so a rewritten key never leaves a stale slot behind (identity is
     # only touched on the rewritten rank transform).
-    _dim_dms = declared_measures[:n_dims]
-    _td_dms = declared_measures[n_dims:n_dims + n_tds]
+    _dim_dms, _td_dms, _ = partition_declared_measures(
+        declared_measures=declared_measures, n_dims=n_dims, n_time_dimensions=n_tds,
+    )
     _dim_key_set = {dm.bound.value_key for dm in _dim_dms}
     # A source column carrying two time-dimension granularities (``created_at``
     # at both month and day) maps to two distinct ``TimeTruncKey`` buckets — a
@@ -1399,6 +1404,9 @@ def _windowed_phase(bf: BoundFilter) -> Phase:
 
     filters_by_phase: List[FilterPhase] = []
     bound_filter_ids: List[str] = []
+    # Capture the date-range fids as they are minted; the windowed ``_src``
+    # row-filter routing below reuses this exact set instead of re-deriving it.
+    date_range_fids: set = set()
     for i, bf in enumerate(bound_filters[:n_date_range]):
         fid = f"f{i}"
         filters_by_phase.append(
@@ -1408,6 +1416,7 @@ def _windowed_phase(bf: BoundFilter) -> Phase:
             ),
         )
         bound_filter_ids.append(fid)
+        date_range_fids.add(fid)
     filters_by_phase.extend(text_filter_entries)
     for i, bf in enumerate(bound_filters[n_date_range:], start=n_date_range):
         fid = f"f{i}"
@@ -1668,7 +1677,6 @@ def _windowed_phase(bf: BoundFilter) -> Phase:
     # POST-reclassified windowed-measure filters are already excluded
     # (phase != ROW).
     if windowed_plans:
-        date_range_fids = {f"f{i}" for i in range(n_date_range)}
         src_where_ids, src_rewrites = _plan_src_row_filters(
             filters_by_phase=filters_by_phase,
             date_range_fids=date_range_fids,